S3E: Semantic Symbolic State Estimation With Vision-Language Foundation Models

Authors: Guy Azran, Yuval Goshen, Kai Yuan, Sarah Keren
Primary area: applications to robotics, autonomy, planning
Keywords: Task Planning, Symbolic AI, Computer Vision, Vision-Language Models

Abstract:
In automated task planning, state estimation is the process of translating an agent's sensor input into a high-level task state. It is important because real-world environments are unpredictable, and actions often do not lead to expected outcomes. State estimation enables the agent to manage uncertainties, adjust its plans, and make more informed decisions. Traditionally, researchers and practitioners relied on hand-crafted and hard-coded state estimation functions to determine the abstract state defined in the task domain. Recent advancements in Vision Language Models (VLMs) enable autonomous retrieval of semantic information from visual input. We present Semantic Symbolic State Estimation (S3E), the first general-purpose symbolic state estimator based on VLMs that can be applied in various settings without specialized coding or additional exploration. S3E takes advantage of the foundation model's internal world model and semantic understanding to assess the likelihood of certain symbolic components of the environment's state. We analyze S3E as a multi-label classifier, reveal different kinds of uncertainties that arise when using it, and show how they can be mitigated using natural language and targeted environment design. We show that S3E can achieve over 90\% state estimation precision in our simulated and real-world robot experiments.

Decision: reject
Reviewer scores: 3, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729884529272

thread_type:
official_review

<thread_object>
creator: reviewer_tvcw

<content>
summary:
This paper proposed a state estimator that leverages Vision Language Models to translate sensor data into high-level task states. However, this paper provides limited technical contribution or insight, as it is a straightforward application of VLMs.

strengths:
The paper is easy to understand. 

The method is evaluated with both simulation and real-world images.

weaknesses:
This paper is a straightforward application of VLMs. There is already a lot of work in robotics using VLM for state estimation, and some use symbolic representation to work with existing task and motion planning algorithms. [1,2,3]. These existing works often with much more complicated scenarios and systems than the ones demonstrated in this work. I don't think this provides much new insight or knowledge. 

The paper does not provide sufficient discussion on related work.  

The scenarios tested are very simple: only simple rigid objects on a tabletop setting with simple spatial relationships. "state estimator" should provide significantly richer information than just simple object class and their spatial relations. 

The state estimation is not used or evaluated in realistic robotics tasks or systems. It only performs offline evaluation. 

[1] Manipulate-Anything: Automating Real-World Robots using Vision-Language Models”
[2]AHA: A Vision-Language-Model for Detecting and Reasoning over Failures in Robotic Manipulation
[3]REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction

questions:
The paper is clear, I don't think answering any question would significantly change my evaluation. I think here is a few questions to think about. 
1) how to extend the state estimation to more general scenarios, e.g., non-rigid object, where the object state is beyond the location and spatial relationship? 
 2) What are some existing works in robotics using VLM as state estimation? Summary their approach and limitations and see how the proposed work can address them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730184546193

thread_type:
official_review

<thread_object>
creator: reviewer_zvee

<content>
summary:
The paper addresses the problem of estimating the truth values of symbolic state predicates, which is essential for PDDL-style planning. While prior works has performed state estimation in a task-specific setting, the proposed approaches uses foundation models as universal estimator that can be conditioned on the task through prompts. First, natural language question is created for every for every possible grounded predicates in the given domain. The translated question along with the image is passed to a VQA model (a VLM) to get the truth value of the predicate. The evaluation is done on a table top manipulation environment which is fully observable and uncluttered, which shows that the proposed method can ground the symbolic predicates across different domain/tasks.

strengths:
- The paper attempts to address an important problem for semantic state estimation. 
- The consideration of both the predicates and their uncertainty is a promising direction. 
- The evaluation is at a task level showing utility of the approach for realistic applications.

weaknesses:
- The proposed approach of determining the validity of predicates with VQA addresses only a part of the problem of estimating the symbolic state. The proposed approach is restricted to simple predicates in small clutter free settings. Scalability to scenarios with complex object arrangements and to higher-order predicates is limited. 
- The problem of eliciting uncertainty from LLMs has received attention in the past. The authors are requested to highlight the technical gap they are filling in this regard. Please refer to works such as https://arxiv.org/pdf/2307.01928 and https://arxiv.org/pdf/2306.13063.
- The problem of inferring semantic properties of the robot's environment has also been addressed in works that extract a scene graph from raw sensor data. Please see Maggio et al. for an example (https://arxiv.org/abs/2404.13696). I request authors to situate their work along side such efforts.

questions:
1. Scalability of exhaustive enumeration. The translation stage of each predicate query involves posing predicate truth value assessment as answer to a textual question. It appears from the text that this conversion is done for each predicate independently and exhaustively. However, there may be correlation in the predicates. If the robot is far away from the table then it is far away from all blocks on the table. Does the approach take advantage of such logical connections?

2. Agent-dependent predicates. Robot interaction tasks require predicates such as in_hand, robot_close_to etc. predicates. How is the agent morphology and other aspects communicated to the VLM for accurate estimation of these predicates.  

3. Higher order predicates. Often in task planning there is a need for higher order predicates. For example, consider a stack of blocks. One would need to reason about the top of the stack as different from a block at the base. See Pasula et al. (https://people.csail.mit.edu/lpk/papers/zpk.pdf) for an example of such predicate (topstack()). Such predicates requires higher order reasoning over the atomic properties that may hold in the scene. Does the proposed method address computation of such predicates. 

4. Baseline VLM performance for direct plan synthesis. The authors have taken the symbolic planning view where knowledge of state predicates is fundamental. Alternatively, what if one asks the VLM to directly synthesise robot actions for a given goal without explicitly querying for the predicates. It would be insightful to analyse this approach as a baseline.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713293257

thread_type:
official_review

<thread_object>
creator: reviewer_nmf7

<content>
summary:
This paper introduces S3E, a novel approach that uses vision-language foundation models for automated state estimation in robotic task planning. While traditional approaches require hand-crafted state estimation functions for each task, S3E offers a zero-shot solution that translates symbolic predicates into natural language questions, uses vision-language models to answer these questions from visual input, and converts the answers back into symbolic states. The system's effectiveness is validated through experiments in grocery sorting tasks and photorealistic blocksworld domains, particularly when employing uncertainty mitigation strategies through natural language instruction and environment design.

strengths:
The key strength of this paper lies in its practical approach to solving a significant challenge in robotics - automated state estimation. The authors present a solution that eliminates the need for hand-crafted state estimators, traditionally a major bottleneck in deploying robotic systems across different tasks. The authors propose an approach that bridges symbolic task planning with modern vision-language models through natural language as an intermediate representation. They provide practical mitigation strategies through natural language instruction and environment design, demonstrating strong empirical results with over 90% precision.

weaknesses:
There are some major problems that concerns me:
1. While S3E is presented as a zero-shot solution, it still requires significant assumptions about the environment, including full observability, visually distinguishable objects, and unambiguous domain descriptions - constraints that may not hold in many real-world applications. 

2. The experiments, while showing promising results, are limited to relatively controlled environments and simple manipulation tasks. The evaluation could be strengthened by exploring more challenging scenarios such as partial observability conditions, dynamic environments, or multi-agent interactions. This would help better understand the method's full potential.

3. The paper's handling of uncertainty, while acknowledged, could be more rigorous - the proposed mitigation strategies of environment design and natural language instruction essentially sidestep rather than solve the underlying uncertainty challenges. Additionally, the method's heavy dependence on the quality of the vision-language model's training data suggests potential brittleness when encountering out-of-distribution scenarios, as evidenced by the poor performance with certain objects in the simulated environment.

questions:
1. Have you explored any cases where the domain descriptions contain partial or ambiguous information? What were the results?
2. Were there any failure cases in the controlled environments that could provide insights into the system's limitations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730841648706

thread_type:
official_review

<thread_object>
creator: reviewer_hvcq

<content>
summary:
This paper introduces the use of pre-trained VLM to classify symbolic states. Specifically, a symbolic predicate is first converted to language instruction and is then fed to a VLM together with the image; the token probabilities of the "yes" and "no" answers are used to determine the probability of the predicate state being true. The paper proposes to deal with two types of uncertainties that arise from partial observation and ambiguities in the definitions of the symbolic predicate. To combat this first type of uncertainty, the authors propose manipulating and moving the occluded objects to be in direct sight. For the second type of uncertainty, the authors propose to provide clearifying instructions.

strengths:
Traditional methods for classifying symbolic states usually need to be manually designed by domain experts or trained with a large amount of data. This paper investigates using pre-trained VLMs to classify states in zero-shot. This approach can potentially improve the generalizability of task-planning methods.

weaknesses:
1. It's unclear how the system automatically determines that there is uncertainty in the prediction, which then needs to be addressed by asking for clarifying instructions or better camera views.
2. The predicates this paper tests are limited to determining whether an object is on a specific table region or whether the object is in hand or in the air. These predicates probably can be determined using very simple heuristics based on gripper state or detected object bounding boxes. It's hard to assess whether the method can reliably classify more diverse predicates for real-world robotic tasks. I suggest the authors to look at other photo-realistic simulation environments that provide annotated object states. 
3. Even though the authors motivate the paper for task planning, there is no evaluation of the method integrated with a task planning framework. It's unclear how the improvement in classification performance translates to the improvement in overall task success.
4. In line 176, the authors mention that the assumption is unrealistic. However, I think this assumption is valid if the action sequences are correct (e.g., if the robots are being teleoperated by a human user)
5. In general, the writing of the paper can also be improved. I found the discussion on uncertainty and experiment section a little confusing.

questions:
I hope the authors can address my concerns above. I don't have additional questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732791773265

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We are grateful to all reviewers for the insightful reviews. We address the specific issues raised by each reviewer individual comments. Here, we  summarize the main changes and additions made to the paper below:

- We address related additional related work suggested by the reviewers. This is reflected in the "Introduction" and the "Related Work" sections. The newly introduced work includes
	- eliciting uncertainties from language modles
	- scene graphs
	- additional work that use VLMs for state estimation
- We distinguish and elaborate on the different kinds of uncertainties in section 5 (Uncertainty in Semantic State Estimation)
- We reorganized the results (section 6.2) to make our results more accessible.
- We made several clarifications in the Conclusion (section 7) to further emphasize the limitations of S3E and our suggested avenues for future work.

We believe that this revision addresses the reviewers' main concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Semantic Symbolic State Estimation (S3E), which uses pre-trained VLMs to determine the truth value of symbolic state predicates from visual observations. S3E creates a natural language question for each grounded predicate. The resulting question together with the image are passed to a VLM to determine the probability of the predicate being true. The paper analyzes different types of uncertainties that can arise, e.g., due to partial observability. The paper evaluates S3E usings a table-top manipulation domain.

The paper was reviewed by four referees who largely agree on its primary strengths and weaknesses. Several reviewers appreciate the benefits of using a a pre-trained VLM for state estimation, particularly when compared to traditional methods that require the involvement of domain experts or the need for extensive training data. In this regard, a key strength of the paper is in providing a potentially practical approach to state estimation, which demonstrates strong empirical results on the domains that are considered. However, as several reviewers pointed out, the evaluation considers a limited set of predicates with simplistic scenarios that involve little-to-no clutter and full observability. The reviewers question whether a simple heuristic would prove effective in these settings and raise doubts about whether the method could be extended to more challenging and practical settings. Meanwhile, at least two reviewers find that a more complete discussion of related work on semantic state estimation and uncertainty quantification is needed to clarify the paper's contributions. The AC appreciates the effort that the authors put into responding to the initial reviews, which included an honest acknowledgement of some of the paper's limitations.