ActiveVOO: Value of Observation Guided Active Knowledge Acquisition for Open-World Embodied Lifted Regression Planning
Authors: Xiaotian Liu
Ali Pesaranghader
Jaehong Kim
Tanmana Sadhu
Hyejeong Jeon
Scott Sanner
Venue: NeurIPS 2025 poster ()
Keywords: Open-world Planning
Embodied Agents
Active Knowledge Acquisition
Lifted Regression
Primary area: reinforcement_learning
TL;DR: We introduce the ActiveVOO framework for active knowledge acquisition to identify, quantify, and prioritize task-relevant information for open-world embodied planning.

Abstract: The ability to actively acquire information is essential for open-world planning under partial observability and incomplete knowledge. However, most existing embodied AI systems either assume a known object category or rely on passive perception strategies that exhaustively gather object and relational information from the environment. Such a strategy becomes insufficient in visually complex open-world settings. For instance, a typical household may contain thousands of novel and uniquely configured objects, most of which are irrelevant to the agent’s current task. Consequently, open-world agents must be capable of actively identifying and prioritizing task-relevant objects to enable efficient and goal-directed knowledge acquisition. In this work, we introduce ActiveVOO, a novel zero-shot framework for open-world embodied planning that emphasizes object-centric active knowledge acquisition. ActiveVOO employs lifted regression to generate compact, first-order subgoal descriptions that identify task-relevant objects, and provides a principled mechanism to quantify the utility of sensing actions based on commonsense priors derived from LLMs and VLMs. We evaluate ActiveVOO on the visual ALFWorld benchmark, where it achieves substantial improvements over existing LLM- and VLM-based planning approaches, notably outperforming VLMs fine-tuned on ALFWorld data. This work establishes a principled foundation for developing embodied agents capable of actively and efficiently acquiring knowledge to plan and act in open-world environments.

Active VOO: Value of Observation Guided Active Knowledge Acquisition for Open-World Embodied Lifted Regression Planning
Xiaotian Liu', Ali Pesaranghader', Jaehong Kim², Tanmana Sadhu?, Hyejeong Jeon', Scott Sanner
"University of Toronto, Toronto, Canada LG Electronics, Toronto Lab, Toronto, Canada
LG Electronics. Seoul. South Korea Veector Institute for Artificial Intelligence, Toronto, Canada
xiaotian. ienotian.liuumali www.malllaal....mmmmm www.aaillloooooooooommm ca ssannerdmie utoronto. ca
(ali, pesarangaaerr, jaehong02.kia kim, tanmana. sadhu, hyejeong jeon)jeonnn @lge.com com
Abstract
The ability to actively acquire information is essential for open-world planning under partial observability and incomplete knowledge. However, most existing embodied Al systems either assume a known object category or rely on passive perception strategies that exhaustively gather object and relational information from the environment. Such strategy becomes insufficient in visually complex open-world settings. For instance, a typical household may contain thousands of novel and uniquely configured objects, most of which are irrelevant to the agent's 's current task. Consequently, open-world agents must be capable of actively iden- tifying and prioritizing task-relevant objects to enable efficient and goal-directed knowledge acquisition. this work, we introduce ACTIVEVOO, a novel zero-shot framework for open- world embodied planning that emphasizes object- bejecct-centicc ac- tive knowledge acquisition. ACTIVE VOO employs lifted regression generate compact, first-order subgoal descriptions that identify task-relevan objects, and provides principled mechanism to quantify the utility of sensing actions based on commonsense priors derived from L.MSs and VLMs.. We evaluate ACTIVE on the visual ALFWorld benchmark, where it achieves substantial improvements over existing LLM- and VLM- planning approaches, notably outperforming VLMs fine-tuned on ALFWorld data. This work establishes principled founda- tion for developing embodied agents capable of actively and efficiently acquiring knowledge to plan and act in open-world environments.
Introduction
Open-world embodied planning is challenging due to partial observability and incomplete task- relevant knowledge. In the open world, knowledge acquisition becomes essential, agents must identify and obtain relevant information necessary to generate feasible plans. Existing embodied Al systems often assume known object categories and rely on passive strategies that exhaustively collect objects and relational information without contextual relevance [Camoriano et al., 201 7]. These approaches quickly become infeasible in complex, information-ie environments due to combinatorial explosion of objects and their relations to each other.
Active knowledge acquisition offers better alternative by enabling the agent to proactively seek information that is relevant to the task at hand. However, existing work in this area has largely focused on locating or recognizing predefined object classes [Zhu et al., 2017, Yang et al., 2018] or information gain [Zurbrügg et al., 2022], thereby overlooking a crucial preceding reasoning step: identifying which objects actually relevant the goal. To illustrate these challenges, Figure 1 shows a comparative example of active vs. passive knowledge acquisition.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Maga
Koowledge
Geler.
pritpe poano, sank, bxs
Knowledge Base
Baa
Lifted Regression Plarner
(18)
smp ond
Hi
(0
[4]
Okjacts:
Color:
Iridge mätroxcee, aidee crelt, but
Oi-dge
Othe (
Posttora:
Mo
Can Plat Da
21
Masorticc
Obeilig close) Heidee notafs (nfowwnesdtri))..
deeseelep
Saneen
jentop
Plaro
27)
Other Otnerabject types restorstne
Susichia?
Acqulsition
Look for releaatt otjecis
Paxaivo Knowwodge Acquraition cojecte
All Vasel Intoriaoo
15 Obadd Iramel
Regrecsion
Progrecsion Planning
Figure I: Active vs. Passive Knowledge Acquisition. The left panel illustrates the active approach, where lifted regression generates abstract subgoals that guide targeted VI.M queries for object discovery. The numbered arrows indicate the sequential order of reasoning and sensing steps. In the active flow: 1) lifted subgoals are regressed from the goal, 2) candidate objects and relations are extracted for subgoals, 3) only relevant objects are sensed and grounded update the agent's 's knowledge base, and 4) the agent explores the environment based on sensing target. This closed loop of reasoning and sensing continues until a plan can be executed. The right panel depicts the passive approach. where grounded progression planner relies on exhaustive object collection. Here, 1) the agent first gathers all visible objects, 2) attempts plan grounding, 3) if grounding fails due missing objects, it re-enters exploration to collect additional data, and 4) Recovery occurs through further exploration (potentially guided by the failed plan context), but this process remains inefficient in open- settings containing unbounded numbers objects and relations.
In this work, we introduce ACTIVEVOO, an open-world planning framework with active knowledge acquisition. ACTIVEVOO adopts an object-centric view of knowledge, and decomposes active knowledge acquisition into two sub-tasks: 1) The agent determines which types of objects and relational properties are relevant to the goal. For example, it should recognize that a kettle is relevant when preparing hot drink, while TV remote is not. 2) The agent quantifies and prioritizes among the relevant objects to sense. For instance, when preparing breakfast, the agent must decide whether locating eggs is more important than finding a baking tray.
Object relevance can determined by evaluating whether an object is necessary for executing a feasible plan. However, generating plans in open-world world settings is feasible with typical grounded progression [Helmert, 2006] methods, which typically assumes the Closed World Assumption (CWA) and fully specified initial state. An alternative is regression- based planning, where the agent searches backward from the goal to identify preceding subgoals without requiring explicit knowledge of the initial state [Reiter, 2001, Sanner and Boutilier, 2009]. Moreover, regression can be conducted the lifted level, allowing compact relational reasoning that abstracts away from specific object instances.
ACTIVE leverages lifted regression [Sanner and Boutilier, 2009, Ghallab et al., 2004] to obtain a set of relevant object descriptions, which then quantifies and prioritizes based on how "helpful" each is to achieving the overall objective. For example, when tasked with "prepare a steak dinner," the agent must weigh the utility of searching for a steak versus locating a wine glass. To this end, we propose Value of Observation (VOO) to formalize active knowledge acquisition as decision-making process that selects utility-mann maximizing objects to sense. VOO measures the expected improvement in utility obtained by observing specific object, relative to the utility of acting without attempting to sense the object.
We evaluate ACTIVEVOO on the ALFWorld benchmark [Shridhar et al., 2020], which provides a diverse set of tasks both visual and textual modalities. While prior work has primarily focused on the textual setting, the visual version of ALFWorld remains highly challenging due to partial observability, object diversity, and complex interaction dynamics. We demonstrate that ACTIVEVOO achieves significant improvements against the visual ALFWorld tasks, notably outperforming vision-

---

ActiveVOO Agent
21)
Iratuctin
Symbolic Goal Parner
(112
Knowledge Base
Symtocko Sooks
Yeu
City acte 3t prtat,, aink table) 0
At
Regrension Planner
Sadirted?
(1T)
Micorialo
Poell one (ontop polaro bet cbject typee and métrtommppeo
Oajecte
(9)
Swarch
VLM
Image
Object
Sulgoak deenkng
LLM
Plan: pidupp)) swangys)
uafigee77)
Ofjoct Occeriptions
[OPT-40)
Subggan 1o aClenchoo ahrddpe/h))
chevingteeees"y at aresgee (rg
Prorri homm Bulld
(1) Prompl based
Clareet GD Coteraai
Plan: polkppT)) pMrt7
Environment
Butgoat (ho naidects, Pwd iaPotaaoo?
40 10 sancho (a
Reltion
canDroorhh, (V)
(5) Ofject
(5)
Plmt speny pooes/7 (1
2. inatee 7h, (he
Dade!
Diject Deectotione:
(R) Symbollc
le ragae (50 0.3
(sCabinnt Thy
(18) Yos, cespute the plar
(13) No. soplons bexad on most hoobtle plan-
Gonerated??
Figure 2: Overview of the VCTIVEVOO Framework. Figure 2 illustrates the full workflow of our approach, which integrates lifted regression, object-relation extraction, and Value of Observation (VOO)-guided active sensing. 1) Lifted regression decomposes the goal into a set of lifted subgoals that describe feasible abstract world configurations. 2) From each subgoal, object and relational descriptions extracted. 3) The agent computes the VOO for each candidate object or relation based on the current KB. 4) Objects with high VOO are selected to guide exploration and sensing. This active cycle continues until all predicates in at least one lifted subgoal are satisfied, at which point the associated plan is instantiated and executed.
language models that are fine-tuned specifically for this benchmark. Figure 2 shows the overall workflow of ACTIVEVOO. To summarize, we make the following key contributions in this work:
We enable active knowledge acquisition by extracting relevant high-level object and relational descriptions through lifted regression for open-world planning.
We propose decision-theoretic Value of Observation (VOO) approach that quantifies the utility of knowledge acquisition using commonsense beliefs from LLMS and VLMs.
We achieve state-of-the-art performance on the visual ALFWorld benchmark in zero-shot manner, outperforming even fine-tuned VLM baselines.
2 Methodology
key challenge for open-world embodied agents is determining which objects are relevant the task prior to planning, as novel and previously unknown objects may appear in the environment. Furthermore, exhaustively enumerating all relational information the agent observes is inefficient, since most objects the agent encounters are irrelevant to the task. To address this challenge, we propose a framework that identifies and prioritizes task rask-relevant objects through the following steps: 1) The agent applies lifted regression to derive subgoals representing possible world configurations where feasible plans can be executed; it then extracts high-level (unary predicate) object descriptions from each subgoal; 3) for each identified object description, the agent estimates the probability of successful sensing by leveraging -.LeMd commonsense; 4) the agent computes the Value of Observation (VOO) to prioritize object sensing based on expected utility gain; and 5) guided by VOO,
3

---

the agent actively senses and collects object information, updating its knowledge base and executing the plan once all relations subgoal can satisfied.
An overview of ACTIVEVOO is shown in Figure 2. We assume the agent is equipped with high-level actions defined in PDDL [Haslum al., 2019] syntax, extension of STRIPS [Fikes and Nilsson, 1971]. but lacks complete knowledge of object types and their relational states. The full action model for our experimental domain is provided in Appendix the examples below are modified for purposes of compact exposition. The agent receives high-level natural-Language instruction without step-by hepp--y--eppppppe guidance and employs goal-com conversion methods from [Song al., 2023, Xie et al., 2023] to translate natural -laguagg goals into PDDL form through few-shot prompting.
2.1 Lifted Plan and Subgoal Derivation
One approach for generating plans in the open world (i.e., open domain of potential objects) is lifted regression [Sanner and Boutilier, 2009], which allows the agent to derive high-level plans and their necessary execution conditions without relying concrete object instances. This is achieved by reasoning in a lifted representation and regressing backward from the goal to infer the abstract preconditions required for plan feasibility. By planning with lifted representations parameterized by variables, the agent reasons in high-level relational descriptions of objects rather than specific instances. For example, instead of concrete object instances such as "oven" "sink", lifting enables the agent to reason about object properties and relations like isClean (?x) and canClean (?y, ?x), deferring the instantiation of these variables until the relevant objects observed. Lifted regression plans backward by successively regressing the goal through the action model to determine the abstract preconditions (i.e., subgoals) that must hold for executing particular action to achieve that goal.
Example (1) Suppose the agent's goal is to put clean potato in the fridge, represented symbolically as
G = isPotato (?x) A isClean(?x) A isFridge A inside(?x, ?2).
Consider the action model
Preconditions: holds (?x)
PutIn(?x, ?z)
Effects: inside(?x, ?z)
Regressing the goal G through PutIn ?z) replaces the achieved effect inside ?z) with action's preconditions, yielding the regressed subgoal
= isPotato (?x) A taClean(2)) A isFridge (?z) A holds (?x).
This expression states that the agent has to be holding potato ?x that has already been cleaned and has found fridge ?z complete the overall goal.
Lifted Backward Search. Lifted regression enables the agent to reason backward from the goal to produce set of high-level conditional plans capable of achieving the agent's objective under different possible environment configurations. In this work, we apply regression via Lifted Backward Search [Ghallab et al., 2004], adapted to open- open-workd world settings [Liu et al., 2025], to generate a set of conditional plans II {91, T1)..... (gn, kn), where each lifted subgoal 9i specifies set of minimal preconditions that must hold for the corresponding plan H; to be executable. The details of
the open-world lifted regression algorithm are provided in Appendix B.
Following the same setup as the previous example, suppose lifted regression yields two abstract subgoals that represent feasible world configurations.
Example (2)
91 = isPotato (?x) A isClean A isFridge (?z) ^ holds (?x), canClean (?y, ?w) A isFridge(?2).
92 isPotato(?w) A cleaningObject(P)
Each subgoal gi is associated with lifted conditional plan क,, which be executed when the sponding şi satisfied:
Fi : PutIn ?z)),
m2 : PPicupp (?w), Cle, ?w), PutIn(?u, ?z)]

---

2.2 Object and Relation Description Extraction
Extracting Object Descriptions. To identify which objects are relevant for executing a plan, we first extract their symbolic descriptions from the lifted subgoals. We assume that object's 's type and characteristics are captured by the unary predicates present
subgoal formula g. For every variable Vgrs(g), construct conjunctive formula ov,g(2) (v) that represents the properties associated with that object, and collectively define the set of all such object formulas as O,
Ov.g (v) =
p(u),
o = (o)o(()) E g, v Vars(g) },
(I)
p(o)Eg, arity(p)=1
where duplicates are removed via a-renaming of variable symbols. This compact formulation encodes all unary predicates applied to each variable within g. yielding the object-level descriptions used as sensing targets during active knowledge acquisition.
Example (3) Continuing from the previous example, we extract unary predicates for each variable form their corresponding object descriptions using Eq. 1. For gi, we obtain:
O2.01 (?x) = isPotato (?x) A (sClean(?x) A holds (?x), 02.91 (?z) = isFridge(22) (?z).
Similarly, for ga. the extracted object deseriptions are: Ow.g2 (?w) isPotato(?)) (?y) = cleaningObj (?y), 02.92 (?z) LsFridge(?z)
After removing duplicate objects, obtain the set of four object descriptions used in subgoals 91 and g2:
(iaPotato isPotato A isClean (?x) holds (?x),
O cleaningObj (?y), isFridge
Extracting Relational Descriptions. In addition to unary object properties, each subgoal formula y may contain relational predicates that describe dependencies among multiple objects- for instance, for variables v = (?y: ?w), one may ask whether the relation canClean (?y, ?w) holds. More precisely, for every relational predicate p(v) with (#i....; U&) and k 2 2 appearing in we construct a relational fornaafforru formula that links the corresponding object formulas, and collectively define the set of all such relational formulas as R,
(v) =
p(v)): R (PrggV)) Ig E p(v) Eg, 9. M 2}. (2)
where each p is an open first-order formula whose free variables correspond the participating objects; it evaluates to true under an assignment iff each instantiates an object satisfying its unary description and the relational predicate p holds among them. Together, and R form lifted representation that links individual object descriptions to their relational structure for active knowledge acquisition.
Example (4) Continuing from the previous example, the subgoal ga contains one relational predicate,
canClean (?y, ?w). Using Eq. 2, we construct corresponding relational formula:
Piaww), 02 (?y, ?w) = (cleaningObj (?y) A isPotato (?w) ^ canClean (?y, ?w)).
This formula specifies that the relation canClean (?y, ?w) needs to hold for object isPotato (?w) and cleaningObject (?y) We can then define the (singleton) set of relation descriptions for subgoal g2:
R (cleaningObject(Ty)) isPotato(?))
canClean ?w)).
2.3 Sensing Belief Estimation via LLM Commonsense Probability Elicitation
Building on prior work showing that LLM token logits can approximate commonsense belief prob- abilities [Liu al., 2023a, Chen et al., 2023], we estimate how likely it is for the agent to sense particular objects. For object description ş (x) (x)AA. A.... A qm (x)) we query the LLM with statement "Does there exist an object that .i......ae Om" ?" For a relational description over v = (#),... v&) with predicate p, query "Can p(v) hold given v such that 91 (v1)..... (v))) (vx))"
Then, given
5

---

fixed context с (e.g., "You are reasoning about a typical household kitchen.") ") and a statement we can obtain model model-specific logits of "Yes" and "No" for given that context holds, denoted as
"Yes (c. s) logit (c, "Yes"),
(c, s) == LLM (c, "No"]:
and apply the definition of conditional probability to estimate whether s is true when context c is true:
P(s = T, 7) P(s = T/c T) = P(s - P P(s 7, T)
exp(2res(c,s))
exp(2Yes (c, (c, expl Noo (c, 8))
Since some logit APIs are stochastic, we average k logits across samples (k=5 in our case). For each E O and p R, we obtain p. == Pr(s,- = T I) and p, := Pr(s, T T) which we use to independently estimate the probability of whether objects satisfying So relationships satisfying Sp can be sensed context c. Prompt templates and paraphrasing details are provided in Appendix
2.4 Value of Observation (VOO) Computation
After extracting object and relational descriptions from lifted subgoals, the agent must determine how to prioritize among candidate objects for sensing. To this end, we propose Value of Observation (VOO), utility-based measure that quantifies the expected improvement in the agent's s ability to achieve its goal with respect to sensing object candidates fitting specific descriptions. By evaluating the VOO for each potential sensing target, the agent can prioritize observations that are most informative, thereby enabling goal-directed and object-centric knowledge acquisition open-world world environments.
Problem Setup. Given the sets of object descriptions O and relational descriptions R. we define the agent's 's knowledge states as a set of sensing beliefs over objects and relations:
S -(00000,,** R+ R o CO, R C R, otno = Ø, ktnk==0) (5)
Here, O+ and R+ denote the sets of object and relational descriptions that the agent believes can be sensed, whereas O and R represent those that the agent believes cannot be sensed. Objects that are not included in either of these sets but are present the overall object set O are considered unknown to the agent. The agent can perform a sensing action a, for an object description e O, which transitions it into one two possible successor states depending on whether o can be sensed:
$ (0+ U (o}. O R )
s' = (0*, o Ufo), R RT),
(6)
The two outcomes occur with respective probabilities Pa and Po estimated in Section 2.3. We formalize this knowledge acquisition process belief-state Markov Decision Process (MDP), whose full specification, including transition dynamics and reward formulation, is provided in Appendix D.
Utility Estimation and VOO Calculation. We define utility function U(s) reflecting the agent's belief that any subgoal E G is satisfiable given the agent's 's knowledge state 8. For each subgoal first identify the sets of still-unknown object and relational descriptions associated with
objunk (g. 5) obi(g) \ (O+ UOT),
relunk (g,s) rel(g) (R+ UR)..
Here, obi(g) and rel(g) denote the sets of object and relational descriptions associated with subgoal g, respectively.
Example (5) Continuing from the previous example, suppose the agent's 's current knowledge state is (0+, O R² R where O+ = (isPotato (?=)) and R* O R Ø, indicating that only
the potato has been identified. Recalling previous subgoal
92 = isPotato
cleaningObject(?)
canClean (?y, ?w) ^ isFridge(?z).
the unknown object and relational descriptions are respectively:
obj (92, s) = (leeaning@bject(y))) (?y), isFridge(?2)
rel (g2, 8) = (cleaningObeett (?y) A iaPotato (?w) ^ canClean ?w)}

---

Assuming independent sensing outcomes with success probabilities teo for object descriptions and PER for relational descriptions, we define the belief probability a subgoal g can be satisfied after sensing all currently unknown descriptions as:
(r(g is satisfiable
II
P.
II
Pp
(8)
oCoblun (g.s)
(9.s
The utility of given knowledge state s is then defined as the maximum probability of any subgoal that can be made satisfiable from s.
(s)
max gEG
II
Po
II
Pp
dEobluu(g..
p (9.8)
Example (6) Continuing from the previous example, assuming independent sensing outcomes with success probabilities Pelwmingghjess 0.9, PiaFridge 0.8, and PeanCl 0.7. the belief that can be satisfied is
Pr(g2 satisfiable |s]
0.8 0.7 0.504
For gi = isPotato A isClean (?x) A isFridge (?z) A holds (?x), the unknown descrip- tions are (01, = {isFridge isPotato (?x) A LaClean(2)) A holds(?x)), ), with
probabilities Pistridge = 0.8 and PasPotato A ieclean A bolds 0.1 giving
Pr(g is satisfiable s) 0.8 0.1 0.08
Using Eq. the utility of the current knowledge is defined
U(s)
max pE9
II
Po
II
Pp
max(0.504, 0.08) 0.504
oCobjus (g.4)
(Erclua(9..)
Value of Observation. Given utility Us), when the agent attempts sense object description o E O, the expected utility of the outcomes of sensing vs. not sensing o can be calculated as follows:
U Usense
P. Us',)) + (1- (i))
senot_sense_ = Us'').
(10)
Here, 5'4 and s'_ denote the successor knowledge states corresponding to successful and unsuccessful sensing outcomes, respectively. The left equation represents the expected utility from sensing o, while the right shows the baseline utility when the agent does not sense and cannot make use of it. Then, the Value of Observation (VO0) quantifies the expected utility gain from actively sensing or
( (s) Unnot_sense( (s) = po (U(()) - U(s')))
(11)
VOO thus measures how much the agent's expected progress toward satisfiable subgoal improves by actively acquiring knowledge about a specific object.
While the astute reader may observe non-incidental similarity between VOO and Value of Infor- mation (VOI) [Howard, 1966], there are causal differences the computations, VOI, deciding whether or not to make an observation (weather report) does not causally affect the outcome (whether will rain). In contrast for VOO, an object must be actively observed causally affect the outcome.
2.5 Active Knowledge Acquisition and Plan Execution
The ACTIVEVOO agent acquires knowledge from both the physical environment, by navigating to new locations and collecting visual observations, and by issuing language vision-language queries that infer object relations using LLMs or VLMs. At each step, the agent selects the object description with the highest Value of Observation, o* arg max.eo VOO,(s), and treats the next exploration target. It then queries an LLM to obtain the optimal location the agent can explore based on the exploration target. After navigating to the corresponding region, the agent captures an RGB frame and queries the VLM to detect instances that satisfy the top-k (in our case, top 5) object descriptions ranked via VOO values. Successful detections are updated into agent's knowledge state which in turn updates the estimated utilities (s) and all VOO scores. The agent also infers relational predicates (via L.LM reasoning or direct observation) whose arguments correspond the sensed object types. Once every predicate in lifted subgoal g E G is grounded, the agent instantiates
7

---

the associated plan 7g and executes the corresponding plan. ACTIVEVOO enables the agent to allocate its limited sensing budget toward information that maximizes expected utiliiy, seamlessly integrating physical exploration with commonsense reasoning over object relations. The complete execution flow of the agent is illustrated in Figure 2, and detailed implementation steps are provided in Appendix F of the supplementary material.
Experiments
Experimental Setup. We evaluate ACTIVEVOO on the ALFWorld benchmark using raw vi- sual inputs in zero-shot setting, without additional data for training or fine-tuning. We compare ACTIVEVOO against state-of-the-art LLM/VLM- planners using both vision and language observations, including zero-shot methods as well as fine-tuned models on ALFWorld data. The performance is evaluated using Success Rate, defined as the agent's 's ability to complete the task in 30 steps. We average success rate over runs. We set the regression length to 10 and the LLM temperature to 0.1, except when computing belief probabilities, where the temperature set to O. All experiments are carried out on a 6-Core AMD CPU with 32GB of RAM, and we rely API calls for LLMs/VLMM. For ACTIVEVOO, we use GPT-4o both for belief probability estimation and as a vision-language model to extract relevant objects. We choose ALFWorld [Shridhar al., 2020] as our benchmark to evaluate both text and vision observations modalities.
Baselines. We compare ACTIVEVOO against a diverse set of baselines across three categories: vision only models, language only models, and vision language models. We include ResNet- 18 [Shridhar et al., 2020] and MCNN-FPN [Shridhar et al., 2020], which are standard computer vision architectures that take an image as input and output an action. For language- baased models, we focus on prompting-based methods, REACT (aao et al., 2022], REFLEXION [Shinn et al., 2024], DEPS [Wang al., 2023], and AUTOGEN [Wu al., 2023], which rely solely on L.LMs for reasoning and planning with few-shot examples. Vision language agents take both raw pixel observations and natural language instructions as input, requiring multimodal reasoning. We include models that require data collection and fine-tuning, which are B BUTLER [Shridhar et al., 2020]. MINIGPT-4 [Zhu et al., 2023a], BLIP- [Li et al., 2023], LLAMA- CAMMA---DPTRRR [Gao et al., 2023], INSTRUCT LIP |Panagopoulou et al., 2023], and EMMA [Yang al., 2024], well as advanced VLMs, GPT40 [Hurst et al., 2024a] and LLAVA-138 13B [Liu et al., 2023b], in the zero-shot setting. Note that EMMA and REFLEXION require successive attempts on the same task. Thus, report their performance for both 3 and 10 trials.
4 Results and Discussion
Table I presents ACTIVEVOO's performance against 6 ALFWorld tasks, compares it with 13 base- lines representing vision-only, language-only. and vision- language models. ACTIVEVOO achieves an overall success rate of 0.86, the highest among all agents with only visual observations. Our approach significantly outperforms SOTA vision-language models such as GPT-40 and LLAVA- in zero-shot settings, and also surpasses models that require extensive task-specific fine-tuning and data collection, including BUTLER, LLAMAA A- ADAPTER, INSTRUCTBLIP, and EMMA. While EMMA's 's performance is comparable, it achieves this level after 10 successive trials in the same envi- ronment, leveraging cross-trial information transfer. In contrast, ACTIVEVOO is strictly evaluated in zero-shot setting, based single trial, without relying any prior experience memory from earlier episodes.
4.1
RQ 1: Overall Performance Comparison
Even when compared to language- based methods, which operate on structured textual inputs devoid of visual noise, ACTIVEVOO outperforms all existing LLM-based planners, except for REFLEXION- 10, which relies on 10 trials that reflect on past failures in the same environment, while ACTIVEVOO significantly outperforms REFLEXION-3 with trials. Additionally, most existing LLLM- planners require few-shot examples of the same task, which may not be available when the task is new. More impressively, ACTIVRVOO outperforms all approaches that rely on large-scale supervised training environment-specific fine-tuning, which are often costly and impractical.
8

---

Table I: Success rates the ALFWorld tasks using template task instructions. "Env." indicates whether the agent uses visual textual environment. "Multi-Trail" indicates whether the agent requires multiple trials the same environment. Finn-TTuiing states whether the model is fine-tuned with data collected from ALFWorld. "eg indicates that reported results are used, and indicates results are not available.
Agent
Euv,
Multi- Trial
Fine- Tuming
Avg.
Pick
Clean
Heat
Cool
Look
Pick2
Human Performance* [Shridhar et al., 2020)
Visual
X
x
0.91
Vision Only Models
CCNN-TPNN [Shridhar et al., 2020]
Visual
X
0.05
RESNET-18
[Shridhar al., 2020)
Visual
0.06
Language Only Models
GPT-4o (Text) [Hurst et 2024h]
Textual
0.21
0.29
0.17
0.21
0.23
0.27
0.13
REACT I'ao et al., 2022]
Textual
0.69
0.78
0.75
0.72
0.58
0.77
0.56
RERLEXION- [Shinn nl., 2024]
Textual
((3)
X
0.83
0.89
0.30
0.78
0.85
0.86
0.81
REFLEXION- [Shion
2024]
Textual
((10)
0.91
1.00
0.88
DEPS* [Wang 2023]
Textual
X
0.76
0.93
0.50
0.80
1.00
1.00
0.00
AUTOGEN* (Wu et al., 2023]
Textual
X
0.77
0.92
0.74
0.78
0.86
0.83
0.41
Vision Languuge Models
BUTLER* (Shridharet et 2020]
Visual
X
/
0.26
0.31
0.41
0.60
0.27
0.12
0.29
GPT-44 (Vision) [Hurst 2024b]
Visual
0:08
0.16
0.06
0.12
LLLAVA-1BB [l.in al., 2023b]
Visual
X
0.11
0.13
0.15
0.12
0.07
0.11
0.09
LLAMA--DAPUUR (Yang et al., 2024]
Visual
0.13
0.17
0.10
0.27
0.22
0.00
0.00
INSTRUCTBLIP" [Yang
2024]
0.22
0.26
0.00
EMMA-3* (Yang al., 2024]
Visual
v(3)
0.37
0.55
0.41
0.45
0.13
0.65
0.4
EMMA-10* (Yngg al., 2024]
Visual
/(10)
0.82
0.71
0.94
0.85
0.83
0.88
0.67
(Ours)
Visual
X 0.86
0.93
0.87
0.83
0.80
0.89
0.86
For example, BUTLER and EMMA require 52K and 15K expert demonstrations, respectively. In contrast, ACTIVEVOO only requires a high-level symbolic action model typically available for agents with predefined skill sets (see Appendix A in the supplementary material). Given the diversity and variability of open-world task instances, models trained on fixed task distributions often contrast, ACTIVEVOO can reason over any task that can be expressed in PDDL [Haslum et al., 2019] or similar symbolic forms. ACTIVE strong performance underscores the value of combining structured symbolic reasoning with LLM-inferred commonsense probability estimates, pointing promising direction for hybrid planning in embodied Al systems.
struggle to generalize.
4.2
RQ 2: Impact of Active Knowledge Acquisition
One key research question we inves- tigate is how much our agent's per- formance can be credited to active knowledge acquisition (which lever- ages
Table 2: Impacts of Active Knowledge Acquisition and Ac- tiveVOO Components (RQ2 and RQ3 Results)
objects from vi- sual observations) in comparison to more passive strategies. Under the exhaustive object acquisition setup, the agent passively extracts informa- tion from each scene by prompting the
Settings
Success Rate
Episode Length
VOO
0.86
15.3
w/ Exhaustive Obj. Acq.
0.22
25.4
Guall-Direted Ohj. Acg.
0.48
23.0
w/ LLM Subgoals Obj. Aeq.
0.47
22.5
VLM to extract all possible objects and relations from visual input, with- out contextual filtering. This method achieves a success rate of only 0.22 as shown in Table likely due the overwhelming number of potential objects and relations present in each image. In the Goal-Directed Object Acquisition setting, we instead query the VLM using only the high-level task goal. This leads to an improvement over the fully passive baseline. We also evaluate a strategy where the VLM queried using a plan generated by LLM. Both of these weakly guided methods achieve similar success rates (0.47 and 0.48), suggesting that LLM-generated plans alone are insufficient to reliably identify task-relevant objects. Together, these results empirically demonstrate the value of our principled VOO-based approach, which significantly outperforms passive or weakly guided knowledge acquisition strategies. The results are shown in Table 2, and the experimental setup details for these ablations can be found in Appendix G in the supplementary material.
w/o VOO Calculation
0.75
19.2
w/o VOO and Object Partition
0.68
21.3
w/ LLaVA-138 138 as VLM
0.82
16.1
w/ GPT-3.5 as LLM
0.77
19.5

---

4.2.1 RQ 3: Ablation of AoOiveVOO Components
We conduct a detailed ablation study to examine the contributions of various components in the VCTIVEVOO framework as shown Table 2. We assess the effect of VOO by replacing VOO- selected objects with those chosen solely by highest belief probability. In this setting, if the agent's s initial belief is improperly calibrated, it then lacks a mechanism to effectively explore alternatives. This results in performance drop from 0.86 to 0.75 and an increase in the average episode length by 3.9 steps. We further analyze the performance by removing not only the VOO module but also the mechanism to extract relevant object descriptions. In this case, the agent directly queries the LLM to produce belief probabilities of an entire subgoal. Under this setup, the success rate declines further to 0.68. We observe that when a subgoal becomes sufficiently complex, i.c., containing many predicates, the LLM's logits become a worse approximation of the agent's 8 actual belief. Lastly, we investigate the effect of using different VLM and LLM models. Replacing the 40 with LLaVA- 13B for the VLM component slightly reduces performance to 0.82, and substituting GPT-4o with GPT-3. for the LLM component results in a further drop to 0.77, indicating that stronger LLM/VLM reasoning capabilities directly enhance ACTIVE VOO's performance.
Related Work
Language and Vision Models for Embodied Planning. Recent work on embodied planning with LLMs focuses on predicting the next action [Ahn et al., 2022, Valmeekam et al., 2023, Hazra et al., 2024] or generating step-by-step plans through prompting (aao et al., 2022, Shinn et al., 2024, Huang et al., 2022]. Hybrid approaches that integrate classical planners with LLM reasoning have also been proposed [Arora and Kambhampati, 2023, Guan et al., 2023, Hazra al., 2024], though they typically operate under closed-world assumptions or rely on iterative re- ee-planning. VLM- methods extend embodied reasoning to visual observations, either via ision-to-language translation [Gao et al., 2024, Huang et al., 2023] or direct visual grounding [Driess et al., 2023, Hurst et al., 2024b]. VLAS based approaches produce low-level action sequences [Kim et al., 2024, Brohan et al., 2022], but generally assume static, fully observable environments. While some open-world VLMs [Yang et al., 2024, Liu al., 2023b] have been proposed, they rely on costly task-specific fine-tuning and extensive data collection. In contrast, ACTIVEVOO generates lifted plans without assuming full observability or complete knowledge, enabling robust reasoning in open-world settings.
Active Knowledge Acquisition For Embudied Agents. Existing works tackle active knowledge acquisition from different perspectives. A notable area of research is embodied navigation, where the agent is tasked with autonomously discovering an object of a certain type [Yang et al., 2018], navigating to a specified image [Zhu al., 2017], following step-by-stey instructions [Anderson et al., 2018, Ku et al., 2020]. However, these approaches typically assume that the goal or target object predefined. In contrast, goal. Other lines of work explore active object recognition [Bohg et al., 2017, Caselles- -Duléé et al., 2021] or general exploration of the environment [Zhu al., 2023b]. While these methods emphasize interaction and perception, they are not task-driven and do not require assessing the relevance of objects, an essential ability for open-world agents. Active VOO explicitly addresses this gap by integrating goal-conditioned object relevance estimation with a principled knowledge acquisition framework base on VOO for open-world embodied planning.
work focuses determining object relevance the overall
6 Conclusion
We introduce ACTIVEVOO, a novel framework that enables active knowledge acquisition for open- world embodied agents based the Value of Observation (VOO). Our method integrates lifted regression for plan subgoal generation with commonsense reasoning and probabilistic inference via LLMs, allowing for structured inference without requiring complete domain specification. We demonstrate that ACTIVEVOO significantly outperforms state-of-the-ee LLM and VLM methods without the need for additional data collection, domain-specific fine-tuning, or few-shot examples.
This work evaluated on the ALFWorld simulator, which requires further fine-tuning to transfer effectively to real-world settings involving physical robots. Additionally, it does not address user or contextual preferences in goal specification, which could influence the agent's underlying utility function.
Limitation
10

---

Acknowledgments
This work was supported by LG Electronics, Toronto AI Lab Grant Ref No. 2024-0565.
References
Raffaello Camoriano, Giulia Pasquale, Carlo Ciliberto, Lorenzo Natale, Lorenzo Rosasco, and Giorgio Metta. Incremental robot learning of new objects with fixed update time. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 3207-3214. IEEE, 2017.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017.
Wei Yang. Xiaolong Wang. Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using priors. arXiv preprint arXiv: 18/000544, 2018.
René Zurbrügg. Hermann Blum, Cesar Cadena, Roland Siegwart, and Lukas Schmid. Embodied active domain adaptation for semantic segmentation via informative path planning, IEEE Robotics and Automation Letters, 7(4):8691-866 2022.
Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26: 191-246, 2006.
Raymond Reiter. Knowledge in action: logical foundations for specifying and implementing dynami- cal systems. MIT press, 2001.
Scott Sanner and Craig Boutilier. Practical solution techniques for first-order mdps. Artificial Intelligence, 1736-6))748-7888 2009.
Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: theory and practice. Elsevier, 2004.
Mohit Shridhar, Xingdi Yuan, Alex-nle Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint wwxii:2000000766. 2020.
Patrik Haslum, Nir Lipovetzky. Daniele Magazzeni, Christian Muise, Ronald Brachman, Francesca Rossi, and Peter Stone. An introduction to the planning domain definition language, volume 13. Springer, 2019.
Richard Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving to problem solving, Artificial intelligence, 2(3-4):189-208 1971.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009, 2023.
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with lang-aa models. arXiv preprint arXiv:2302.05122, 2023.
Xiaotian Liu, Ali Pesaranghader, Hanze Li, Punyaphat Sukcharoenchaikul, Jachong Kim, Tanmana Sadhu, Hyejeong Jeon, and Scott Sanner. Open-world planning via lifted regression with 1lm- inferred affordances for embodied agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 00881-20897, 2025.
Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannanch Hajishirzi. Vera: A arXiv:2305.0366 03695, 2023a.
plausibility estimation model for commonsense statements, arXiv preprint
Wenkai Chen, Sahithya Ravi, and Vered Shwartz. Case: commonsense-augmented score with an expanded answer space. arXiv preprint arXiv:2311.0168.1 2023.

---

Ronald A Howard. Information value theory. IEEE Transactions on systems science and cybernetics, 2(1):22-26, 1966.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXhv:2210.03629, 2022.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Zihao Wang. Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, arXiv preprint arXiv:2302.01560 2023.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint urXiv:2308.088 08150000.85,,,,, 3(4), 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pages 19730-19742. PMLR, 2023.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint ar arxiv:2304. 15010, 2023.
Artemis Panagopoulou, Le Xue, Ning Yu, Jumnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to Ilms and emergent cross-modal reasoning. arXiv preprint arXiv:2311.18799 2023.
Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied multi-modal agent trained by an 1lm from parallel textworld. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16275-26288 2024.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprins arXiv:2410.21276. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in 2023b.
neural information processing systems,
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276 2024b.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as say: Grounding language in robotie affordances. arXiv preprint arXiv:2204.0116 2022.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36.75993-76005, 2023.
Rishi Hazra, Pedro Zuidberg Dos Martires, and Lue De Raedt. Saycanpay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20123-20133 2024.
12

---

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pages 91 11989747. PMLR, 2022.
Daman Arora and Subbarao Kambhampati. Learning and leveraging verifiers to improve planning capabilities of pre-trained language models. CoRR, abs/2305. 17077, 2023. doi: 10.48550/ARXIV. 2305.17077. URL https::/ //doi. org/1r...... 4.00.4855//aaa /arivv 2305 17077.
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre- trained large language models to construct and utilize world models for model-based task planning. In NeurlPS, 2023.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 12462- 12469. IEEE, 2024.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-lagguage-action model. arXiv preprint arXiv:2406.0m 09246, 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for sal-world control at scale. arXiv preprint arXiv:22/2.068) 2022.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. Aision-and-languaae navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room Multilingual sision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07955 07954,, 2020.
Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme. Interactive perception: Leveraging action in perception and perception in action. IEEE Trunsactions on Robotics, 33(6): 2017.
Hugo Caselles-Dupré, Michael Garcia-Ortiz, and David Filliat. Scod: Active object detection for em- bodied agents using sensory commutativity of action sequences. arXiv preprint arXiv:2107.02069, 2021.
Hao Zhu, Raghav Kapoor, So Yeon Min, Winson Han, Jiatai Li, Kaiwen Geng, Graham Neubig, Yonatan Bisk, Aniruddha Kembhavi, and Luca Weihs. Excalibur: Encouraging and evaluating embodied exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14931-14942, 2023b.
13

---

A
Action Model
Below are the high-level skills/actions defined in PDDL syntax. Each action A E A is represented as
A = (params(A)) pre(A), add(A), del(A)):
corresponding to the action's 's parameters, preconditions, and effects. The sets (dd(A) and del( indicate which predicates are made true false. Our regression approach requires no negated preconditions or goals, no conditional effects, and no quantifiers in preconditions or effects.
(:action PickupObject
:parameters (?o obj)
:precondition (and
(handEmpty)
(is0bject ?o))
:effect (and
(holds ?o)
(not (handEmpty))))
(:action PudObject.InBeeeptaclle
:parameters (?o obj ?r obj)
:precondition (and
(canContain ?o)
(isObject ?r)
(holds ?o))
:effect (and
(inReceptacle ?r)
(handEmpty)
(not (holds ?o))))>
(:action HeatObject
:parameters (?r obj obj)
:precondition (and
(isfleatingOjectt ?r)
(canteett ?r ?o)
(holds ?o))
:effect (and
(isffot ?o)))
(:action CleanObject
:parameters (?r obj obj)
:precondition (and
(iaCleaningObject ?r)
(canClean ?r ?o)
(holds ?o))
:effect (and
(isClean ?o)))
(:action CoolObject
:parameters (?r obj ?o obj)
:precondition (and (iaCoolingObject ?r)
(canCool ?x ?o)
(holds ?o))
:effect (and
(isCool ?o)))
(:action ToggleObject
:parameters (?r obj)
:precondition (and (is0bject ?r))
:effect (and (isDn ?r)))
14

---

B Open-World World Lifted Regression
B.1 Open-World Planning Formalism
We formalize the open-world planning problem as the tuple IT (P,A, G, T), where P denotes the set of predicate symbols, A the set of action schemas, O the (incomplete) set of known objects, G the goal formula, and / the agent's initial knowledge. Unlike closed-world planning, which presumes complete knowledge of all objects and the initial state, open-world agents must accommodate missing object instances as well as uninstantiated variables or predicates. We represent an atom p(T) by pairing a predicate E P with an n-tuple of terms T = (t1....,tt, tn). where vars(T)) indicates the variables occurring in T. An atom is lifted if vars(T) # 0. Each action schema A A. A = (params(A), (re(A), add(A), del( in which params(A) are the action's 's parameters, (re(AA) its preconditions, add(A) the predicates added to the resulting subgoal, and del(A) the predicates removed. The objective is to generate a finite set S = ((9. л.) m:))=i where each pair (gi; Th) consists of a feasible lifted subgoal 9i and associated action sequence n, whose length does not exceed user-defined horizon T.
B.2 Open-Worke Lifted Regression
Once the planning problem is formalized, we leverage Lifted Backward Search [Ghallab al., 2004], adapted to open-world settings, to produce a set S = {(91, mi)...., (gn. lifted subgoals gi and their corresponding plans म;. Starting from the overall goal G, the algorithm iteratively applies REGRESS to each RELEVANT action until no further regression is possible or predefined length 7 is reached. We say that an action schema a A is RELEVANT to subgoal s if s n (add(a) U del(a)) * 0, s n del(a) = 0. and O add(a) = 0. Whenever a is relevant, the regressed subgoal computed as REGRESS (s, a) (s add (a) pre(a). To avoid variable clashes and ensure correctness, each regression step applies the standard logical operations STANDARDIZE, UNIFY, and SUBSTITUTE (see Appendix C). The algorithm maintains a frontier of subgoal-plan pairs (g. r), initialized with (G, ()). At each iteration, if | subgoal-plant pair; otherwise, for each action a E A, we STANDARDIZE and UNIFY a with g, check RELEVANT(a, 9). and enqueue the regressible action exists, the pair is deemed terminal and also added to S. The process repeats until the frontier is empty, at which point the algorithm returns all lifted plans and subgoals in S of length at most T. A complete pseudocode listing appears in Appendix C the supplementary material.
nn) of all feasible
the current pair is added to S as terminal
pair (REGRESS(g, a), a:: if has not been visited. If no
B.3
Open- Open-World Lifted Regression Algorithm
The Open World Lifted Regression algorithm is initialized with empty set S, which the algorithm will return, and a frontier that contains the overall goal G paired with an empty plan. The algorithm performs backward search by repeatedly REGRESS through subgoal and trace pair (g. T) from the frontier. If the trace length |-|| equals the maximum depth т, it adds (g. π) to S. For each action schema A. the algorithm applies STANDARDIZE generate fresh variables to obtain A' and uses UNIFY to match A' effect with the current subgoal If is determined to be RELEVANT, the algorithm regresses g through A' to produce a new subgoal g' and an extended trace T. aappend((AA) The new subgoal g' is added to the frontier for further regression. If no actions apply. the algorithm treats (g, πT) as a leaf and adds it to S. Once the frontier is exhausted, S contains all lifted subgoals and their corresponding regression paths up to depth T.
C Logical Operations
Here is a set of standard logical operations used in this work for Lifted Regression:
Substitution: SUBSTITUTE is an operator that replaces terms in one set of logical statements with terms from another set. We use represent mapping between variables/ terms in the form of dictionary:
(wr/,, 02/ 0a/13..... vn/bn).
Standardization:
The STANDARDIZE operator, written as STANDARDIZE(p)) replaces all variables in predicate p with new variables v' such that e' V. This ensures that variables in one logical
15

---

Algorithm Open-Woold Lifted- Regirssion
1: == ().A.O,C.I),T
2: S < 0
3: Frontier + {(G, |}}
Regressed
{G}
5: while Frontier * 0 do
6: 9. + POP(Frontier)
7: len()) = then
8: (gadd((g, m))
else
10:
RegressActtios = 0 for each A in A do
11:
12:
A' STANDARDIZE(A)
13:
A + UNIFY(A', .g)
14:
if RELEVANT(@(A"), {())) then
15:
RegressActtions.dde AA'
16:
n.ppeedd( (A')
17:
<- (EGRESS((()))
18:
if o' not Visited then
19:
Regressed. add(g")
20:
Frontier add (gg x))
21:
end if end end for if RegressActions = and g' & Visited then
22:
23:
24:
25:
S.add m)
26:
end if
27:
end if
28:
end while
29:
return S
statement do not conflict with those in the target statement it tries to unify. Standardization is carried out by applying substitution 0(p). which replaces each variable as follows:
Vu vars(p), & Buu(SSTANDARDIZE()) (p)).
Unification: UNIFY(p. g) an operator that checks whether two logical statements can be made
equivalent by applying substitution and standardization to their variables. In Lifted Regression, this is used to match an action with a subgoal. Given two logical statements p and it returns substitution 0, which is the Most General Unifier (MGU), such that:
UNIFY(P, 4) = 0 where 0(p) = 0(g)-
D Belief MDP Formalism
We cast object-centric active knowledge acquisition a sequential decision problem: the agent selects which candidate objects to sense to maximize progress toward the goal. We model this as an MDP over the extracted object descriptions O relational deseriptions R, and lifted subgoals Formally. we define Markov Decision Process as a tuple M = (S, A,
State Space. The state captures verified (true/false) knowledge about objects and relations:
= (= 0- RR
0# CO R C R. o+no- = Ø, R n R = =2} 2.
Here O+ and are, respectively, the sets of object descriptions verified true or false; R+ and R the analogous sets for relational descriptions. Any description not in (O+U 0 (R+U R is unobserved.
Action Space. At each state, the agent chooses one unobserved description (object or relation) to sense:
A(s) (Sensebj(()) |oE (o+ (oT)} u (SenseRel(p) p R (R U R )
16

---

Each action corresponds to information -aaarraiion-gathrring operation that attempts to confirm whether the chosen description holds in the current environment.
Transition Model. The agent may sense object description E O or relational description p E R only if is currently unobserved. Moreover, a relational sensing action is enabled only when all argument-dependen object descriptions of have already been sensed. We assume that the outcome of each sensing action depends only on the corresponding object or relational descriptions rather than the state and s' in which the sensing occurs.
The transition model is therefore
S' = (O+ (o) O R
Po
(s, (o))
=
(0+, o U (0)) Rt, R-)
Po
= (0+, 0- Rt u {p), R-)
Pp
T(s, SenseRel (p).
8' 10+ 0- R R U(l)
Rt), Po E [0, 1] is the agent's belief that object can be sensed, and where 8 (0+, , Pp [0, is the belief that p holds true.
Reward Function. The agent receives a terminal reward when its verified knowledge suffices to realize some lifted subgoal. Let obj(g) O and rel(g) C R. denote, respectively, the object and relational descriptions required by g E G. We define
1,
3g such that is satisfiable
R(s) =
(12)
0, otherwise,
Having defined the MDP, we now describe how the transition probabilities po and Pp are estimated using a large language model (LLM) to provide commonsense priors over object existence.
E
Extracting Belief Probability via LLMs
For belief probability extraction from LLM logits, we use OpenAl's API (GPT-4o) with a simple yes/no classification interface. The estimator takes input an object description in natural language (e.g., "There exists cooked piece of meat on a plate") and returns a scalar belief probability that reflects the model's confidence in the truth of the statement. To compute this, we prompt the LLM with an existential question and analyze its token-level response. We set the temperature to zero and enable logit tracking.
E.1 Prompt Construction
We first construct natural language description of the target object based on its descriptive predicates. Each lifted subgoal generated by regression is represented as a conjunction unary and binary predicates. After applying object-centric partitioning to extract individual objects from the subgoals, we obtain a set of unary predicates for each object. These predicates are then translated into a coherent natural language phrase that captures the object's type and its functional relationships. The resulting description is embedded into fixed prompt template that assigns the language model the role of home assistant robot operating in household environment.
Below an example prompt for object description query:
You are home assistant robot operating common household environment.
Is it true that you can find an object such that this object is a plate it clean?
Please answer the question using only ['Yes*, 'No']
17

---

Below is an example prompt for relational description query:
You are home assistant robot operating in common household environment.
Is it true that object canclean object Y, given that object cleaning object and object is a plate?
Please anawer the question using only ['Yes', 'No']
Each query is repeated 5 times.
F
Agent Overview
Algorithm 2 ACTIVEVOO Agent
Require: Instruction Gnt. action model A, interaction budget B
I: GOAI.PARSER
LIFTEDREORESSION(G.A..
3: <- {ir (g. π) E 5}
4: (24, R) < PAKTTIONSUBGOALS(S)
5: KB I; budget +- B
6: for all (0, S do
7: Mg + EXXRRACTBBECTT())
8: end for
9: while not SATISFIED KB) and budget > 0 do
10: p (P(p n, KB))
for all Ug
12: VOO(o) + COMPUTEVOO p)
13:
end for
14:
O+ - {ol VOO(o) 0)
15:
o' < arg maxogo, VOO(o)
16: EXPLORE(O")
17:
for all 0+ do OBSERVE(O); (PDATEKB(oKKB) budget + hudget- -1 if budget= = 0 then
19:
20:
21:
break end if end for
22:
23:
24:
for all (g. T) E S do
25:
if SATISFIED(J, KB) then
26:
for all a E do ACT(a) succeeds then PROGRESS(KB.a)
28:
29:
end if
30:
budget budget -1
31:
budget= then break end if end for
32:
33:
34:
35:
end if
36:
end for
37:
end while
38:
return KB
For every candidate plan π E II, the agent maintains belief score P(g KB) that reflects the likelihood that the plan can currently succeed. Using these scores, the agent computes the Value of Observation (VOO) for each object description extracted from every lifted subgoal g. All objects with strictly positive VOO are collected into the set O4. This positive VOO set represents all observable objects that can potentially improve or alter the agent's 's decision. The agent then queries the languaee model for all objects in O4. Each query updates the knowledge base (KB) with new unary and relational facts. After the KB is updated, the agent scans the set of lifted subgoals.
18

---

Whenever the predicates of a subgoal 9 are satisfied, the associated plan x becomes executable. The agent immediately executes each action иT, updating the KB after each successful step. The loop terminates when the overall goal G is achieved or the interaction budget is exhausted.
G
Ablations
Passive and Weakly Guided Baselines: We evaluate three alternative methods for extracting object information from the environment. Exhaustive Object Acquisition is fully passive approach in which the vision-language model (VLM) is prompted to list all observable objects. Object Acquisition is weakly guided approach that prompts the VLM using only the overall task goal. Plan-Directed Object Acquisition first prompts the VLM to generate plan and then queries for all objects mentioned in that plan. The exact prompts used in our experiments provided below.
Exhaustive Object Acquisition:
You are home assistant robot operating in common household environment Give all objects you observe from the scene.
Please answer in this format: ["obj_1", "obj_2", "obj_n"]
Goal-Directed Object Acquisition:
You are home assistant robot operating in a common household environment Give all objects you can observe from the scene that are related to the goal {GOAL}.
Please answer in this format: ["obj_1", "obj_2", "obj_n"]
LLM-Subgoal Object Acquisition:
Extract high-level plan
You are home assistant robot operating in common household environment Your goal to (GOAL).
Give me plan that can achieve the goal.
Extract objects from the plan:
You are home assistant robot operating in common household environment Your plan is (PLAN).
Give me all objects you can observe from the scene that are related to your plan.
Please
this format: ["obj_1", "obj_2", "obj_n"]
All extracted object names are post-processed to align with the vocabulary used in ALFWorld; however, the list of ALFWorld object names never exposed during extraction. We employ GPT-4o as the VLM in all of these experiments.
Component Alations within ACTIVEVO0: We next disable or swap individual components of ACTIVEVOO:
No VOO: Without VOO, the agent extracts objects only from the plan it believes is most likely to succeed, that is, p = arg max,' P(p'), where the belief probabilities are obtained from the LLM.
We retain the same object-partition method but limit extraction to the chosen plan p. No VOO + No Partition: VOO calculation requires partitioning objects by their first-order descriptions. When partitioning is disabled, we cannot compute per-object plan probabilities, so we ask the LLM for feasibility the entire plan. The prompt is:
19

---

You home assistant robot operating
household environment Is the plan (PLAN) feasible execute this environment?
Please answer using only ["Yes", 'No'].
VLM Swap: We replace 4P with LLaVA- 13B for vision- language object extraction while still using GPT-4o to compute belief probabilities.
LLM Swap: We substitute GPT-4o with GPT-3. Turbo for belief-rrbaability estimation, keeping 4P as the VLM for visual information extraction.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The 3 contribution we listed in the introduction are all answered in the results section and ablations in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation included section in the Discussion in Section 5,
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
The full mathematical derivation of our framework are shown extensively through section with explicit assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The Section 2 shows our methodology with formal mathematical language
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: Anonymous repo is provided in the technical supplementary material with source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/eddee/eeembmmii ssi ssionoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The implementation details and parameters are included Section under "assumptions" and Section 3 under "Experimental Setup"
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: All results are averaged over 3 runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Hardware used to run experiments is given under Section 3 under "Experimen- Setup".
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / pettps://neurpprr
Answer: [Yes]
Justification:
We have reviewed and follow NeurlPS's code ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work is done with simulated environment with no direct link communi- cate with external users.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No new data model are being released.
The work is done with simulated environment with direct link external enviorment.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All creator and owners of the material and assets used properly credited in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi twwode..ooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
All code and algorithm are well documented and provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiments and research with human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No animal or human subjects are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
26

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM are used only to check grammar and help with formatting text.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
cc/ Please refer to L.LM policy (https: 2ttp:/pps..c/oooeeeeeeeeee//2/20 for what should or should not be described.
27


Reviews:
[{"review_id": "Ge3Tvl9ycn", "reviewer": "Reviewer_YV56", "summary": "This paper introduces ActiveVOI, a zero-shot framework for open-world embodied planning that actively acquires task-relevant information using Lifted Regression and Value of Information (VOI). The agent identifies subgoals via symbolic regression, uses LLM/VLM commonsense to estimate object relevance, and then prioritizes sensing based on VOI. On the ALFWorld benchmark, ActiveVOI achieves strong performance, outperforming most baselines in a single-trial visual setting, without any fine-tuning.", "questions": "- **LLM/VLM failure handling**: In cases where the LLM or VLM provides incorrect or misleading priors (e.g., low belief for a relevant object), does ActiveVOI have any fallback or exploration mechanism? Could belief calibration, ensembling, or conservative VOI strategies improve robustness to noisy commonsense reasoning?\n\n- **Knowledge persistence and reuse**: Have you considered augmenting ActiveVOI with a memory system for storing verified object properties or relations across episodes? This could support lifelong learning and reduce redundant sensing in related future tasks. Would VOI still be usable in such a setting?", "limitations": "yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- **Strong empirical performance in single-trial, zero-shot settings**: ActiveVOI achieves top-tier success rates among all visual agents that operate in a single trial without fine-tuning, significantly outperforming state-of-the-art baselines like GPT-4O (vision), LLAVA-13B, and BUTLER. This highlights the practicality and robustness of the method in realistic, one-shot planning scenarios.\n\n- **Principled and novel framework**: The combination of Lifted Regression for generating symbolic subgoals and VOI-guided active sensing represents a creative and theoretically well-grounded integration of symbolic AI and modern pretrained models. The decomposition into object-centric descriptions and relational constraints allows for efficient and targeted exploration.\n\n- **Clear methodology with strong theoretical foundation**: The paper includes rigorous formalization of the planning setup, VOI estimation, and regression mechanics. The object-relevance partitioning and the assumptions underlying the belief factorization are clearly articulated.\n\n\nWeaknesses:\n- **Heavy reliance on LLM/VLM accuracy**: The method assumes that GPT-4 (or similar models) provides reliable estimates for object relevance. As shown in ablations, performance drops with weaker LLMs (e.g., GPT-3.5). This dependence may limit robustness, especially when commonsense priors are uncertain or incorrect—an issue not deeply explored in the failure analysis.\n\n- **Scalability and memory handling remain open questions**: The planning horizon is capped, and the system operates episodically without a persistent knowledge base. In lifelong or multi-task settings, the agent would likely re-discover the same information repeatedly, as no explicit mechanism exists for long-term memory, knowledge abstraction, or reuse.", "was_revised": true, "final_justification": "My questions and concerns have been addressed. With the inclusion of an illustrative example showing how the proposed approach, ActiveVOI, mitigates initial commonsense errors, as well as a discussion on scalability and long-term knowledge persistence as future work, I believe this work merits acceptance.", "rebuttal": "We thank the reviewer for the detailed and insightful feedback. Below, we address the reviewer’s concerns and questions regarding LLM/VLM accuracy, failure recovery, and the potential for knowledge persistence and reuse.\n\n## \\[Weakness 1, Question 1] Reliance on LLM/VLM Accuracy and Failure Handling\n\n**Reviewer comments:**\n\n> “The method assumes that GPT-4 (or similar models) provides reliable estimates for object relevance. As shown in ablations, performance drops with weaker LLMs (e.g., GPT-3.5). This dependence may limit robustness, especially when commonsense priors are uncertain or incorrect—an issue not deeply explored in the failure analysis.”\n\n> “LLM/VLM failure handling: In cases where the LLM or VLM provides incorrect or misleading priors (e.g., low belief for a relevant object), does ActiveVOI have any fallback or exploration mechanism? Could belief calibration, ensembling, or conservative VOI strategies improve robustness to noisy commonsense reasoning?”\n\n**Response:**\n\n**Reliance on LLM/VLM accuracy:**\nWe appreciate the reviewer’s concern regarding the reliance on LLM/VLM estimates for object relevance. In fact, ActiveVOI is not statically bound to these initial priors. The agent maintains a knowledge base (KB) that is incrementally updated with each new observation, and VOI is re-computed over this updated belief state. As the agent gathers more information, its KB progressively aligns with the actual environment. This incremental recalibration of VOI helps mitigate the impact of initial commonsense errors.\n\n**Fallback mechanisms and potential for improving robustness:**\nWhile the current version of ActiveVOI does not implement a fallback mechanism, this was a deliberate design choice to isolate and evaluate the contribution of VOI-driven acquisition. That said, we included ablations against three alternative acquisition strategies that may be seen as surrogates for a fallback mechanism: Exhaustive, Goal-directed, and LLM-Subgoal-guided acquisition (Table 2), all of which show over 40% performance degradation. The ablations suggest that ActiveVOI is already comparatively more robust than other strategies. We agree that robustness could be further improved by integrating fallback strategies, especially in safety-critical or real-world applications. One potential direction is to incorporate belief calibration or LLM/VLM uncertainty thresholds to trigger a conservative default strategy when model confidence is low.\n\nWe appreciate the reviewer’s comments on these important issues and will highlight this discussion in the revised manuscript.\n\n---\n\n## \\[Weakness 2, Question 2] Scalability and Memory Persistence\n\n**Reviewer comment:**\n\n> “Scalability and memory handling remain open questions: The planning horizon is capped, and the system operates episodically without a persistent knowledge base. In lifelong or multi-task settings, the agent would likely re-discover the same information repeatedly, as no explicit mechanism exists for long-term memory, knowledge abstraction, or reuse.”\n\n> “Knowledge persistence and reuse: Have you considered augmenting ActiveVOI with a memory system for storing verified object properties or relations across episodes? This could support lifelong learning and reduce redundant sensing in related future tasks. Would VOI still be usable in such a setting?”\n\n**Response:**\n\nWe thank the reviewer for raising the important points of planning horizon selection, scalability, and long-term knowledge persistence. Below, we clarify how the current ActiveVOI framework addresses these issues and highlight future extensions.\n\n**Planning horizon:**\nThe lifted-regression planner used in ActiveVOI requires a finite planning horizon $t$ to enumerate all lifted plans up to depth $t$. In practice, $t$ is chosen based on the upper limit of the planning horizon reflecting time and cost constraints. For all benchmarks in our paper, we set $t = 15$, which corresponds to the upper bound of the expected episode length.\n\n**Per-episode KB and VOI caching:**\nActiveVOI maintains a structured KB during each episode. The KB compactly represents all verified object properties and relations as grounded predicates and is continuously updated through the agent’s observations and interactions. Additionally, we cache computed VOI scores to prevent redundant reasoning, which improves both cost and efficiency.\n\n**Persistent memory and VOI for lifelong/multitask settings:**\nSupporting persistent memory across episodes could be an interesting future direction. Unlike LLM-based approaches that store full interaction trajectories, our framework enables structured transfer via compact symbolic KBs (and optionally their VOI caches). These representations are lightweight, abstract away irrelevant task-specific details, and can be easily reused across tasks in a lifelong learning setting.\n\nThe primary adaptation required to support VOI in lifelong settings is to provide a mechanism for Bayesian updating (recalibration) of object sensing probabilities in light of previous sensing experience. Existing comparison benchmarks do not evaluate this setting, but we will discuss this important future work direction in the revision."}, {"review_id": "WmuNMuqgvV", "reviewer": "Reviewer_z6oJ", "summary": "The paper presents ACTIVEVOI, a novel framework for active knowledge acquisition in open-world embodied planning. It addresses the challenges faced by AI agents in environments with partial observability and incomplete task-relevant knowledge. ACTIVEVOI introduces a two-step approach: (1) determining relevant object descriptions through Lifted Regression, and (2) quantifying and prioritizing knowledge acquisition based on the Value of Information (VOI) theory, using commonsense knowledge from large language and vision-language models (LLMs/VLMs). The framework is evaluated on the ALFWorld benchmark, showing substantial improvements over existing methods, especially in visual tasks. ACTIVEVOI does not require task-specific fine-tuning and demonstrates superior performance in zero-shot settings.", "questions": "Could the authors provide further details on how the method might scale in real-world environments? Specifically, how might the approach be adapted to handle environments with significantly more objects or dynamic interactions between objects?\n\nHow does the proposed VOI calculation handle dependencies between object attributes and relations in more complex environments, where the assumptions of independence may not hold? Would relaxing these assumptions improve performance?\n\nCan the authors elaborate on potential strategies for reducing the computational cost of VOI calculations, especially when scaling the method to larger environments with many objects?", "limitations": "Yes, the authors have addressed the limitations of their work, particularly in terms of the evaluation environment and the theoretical assumptions made. However, there could be more discussion on the real-world applicability of the framework and how it can be adapted to handle more complex, dynamic environments.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nNovelty: ACTIVEVOI offers a fresh approach to active knowledge acquisition by combining Lifted Regression with VOI, addressing a crucial gap in open-world embodied planning.\n\nEffectiveness: The framework demonstrates significant improvements over existing models, particularly in zero-shot settings where no prior data is available for training or fine-tuning. This is a noteworthy achievement in open-world environments, which are inherently challenging.\n\nPractical Application: The integration of commonsense knowledge from LLMs/VLMs enables the agent to make intelligent decisions on what information to acquire, potentially leading to more efficient planning in real-world applications.\n\nWeaknesses:\n\nLimited Benchmark Evaluation: The experiments are primarily conducted on the ALFWorld benchmark, which is a well-regarded testing environment, but additional evaluations across other benchmark tasks, such as Alfred or VirtualHome would help to confirm whether the observed performance improvements are consistent across different types of environments and tasks. Expanding the evaluation scope would provide more robust evidence of the framework's general applicability.\n\nAssumptions on Knowledge Independence: The framework makes several assumptions regarding the independence of object attributes and relational properties. In complex real-world scenarios with intricate object interactions, these assumptions may lead to inaccuracies in the VOI calculations, affecting the agent's decision-making process.\n\nScalability and Efficiency: While the method is effective in small-scale environments, it remains unclear how well the system scales to much larger, more dynamic real-world environments with a high number of objects and complex interrelationships.", "was_revised": true, "final_justification": "The applicable scenarios of the method have obvious limitations. Based on the current results, I am very concerned about the practical value of the method.", "rebuttal": "## Rebuttal\n\n### \\[Weakness 1] Benchmark Choice\n\n**Reviewer comment:**\n\n> \"The experiments are primarily conducted on the ALFWorld benchmark, which is a well-regarded testing environment, but additional evaluations across other benchmark tasks, such as Alfred or VirtualHome would help to confirm whether the observed performance improvements are consistent across different types of environments and tasks. Expanding the evaluation scope would provide more robust evidence of the framework's general applicability.\"\n\n**Response:**\n\nWe thank the reviewer for highlighting the importance of evaluating generalizability across diverse embodied benchmarks. However, we find most benchmarks inadequate for our evaluation purposes. Below, we explain our rationale for focusing on ALFWorld and clarify the limitations of alternative environments for evaluating open-world planning and object-centric knowledge acquisition.\n\n**ALFRED and VirtualHome:**\nWe considered both ALFRED and VirtualHome, but found them unsuitable for evaluating semantic open-world planning and object-centric active knowledge acquisition. Tasks in both environments assumes subgoals or step-by-step instructions are available to the agent which already contain all necessary objects required to achieve the goal. This makes object-centric active sensing unnecessary.\n\n**ALFWorld:**\nWe selected ALFWorld because it exhibits key characteristics essential for our evaluation:\n\n1. Task-relevant objects are not directly specified and require semantic inference.\n2. The environment features high diversity in objects, scenes, and tasks, with rich visual observations.\n3. It is partially observable and can be configured without access to ground-truth object types or relations.\n4. The benchmark remains challenging for state-of-the-art models (e.g., VLMs), even with extensive domain-specific fine-tuning (as shown in Table 1).\n\n---\n\n### \\[Weakness 2, Question 2] Assumptions on Object Knowledge Independence\n\n**Reviewer comment:**\n\n> \"The framework makes several assumptions regarding the independence of object attributes and relational properties. In complex real-world scenarios with intricate object interactions, these assumptions may lead to inaccuracies in the VOI calculations, affecting the agent's decision-making process.\"\n>\n> \"How does the proposed VOI calculation handle dependencies between object attributes and relations in more complex environments, where the assumptions of independence may not hold? Would relaxing these assumptions improve performance?\"\n\n**Response:**\n\nWe agree that assumptions of independence must be handled carefully. While we discussed the motivation in Section 2.1.3, we provide further elaboration below that we will include in the revision.\n\n**1. Justification for the Independence Assumption**\n\n* **Reduce sensing targets:** By assuming independence between object attributes and relations, we enable object-level sensing, drastically reducing the number of sensing candidates. Without this assumption, the agent would have to reason over a joint space of all object-relation configurations, which is computationally intractable at scale.\n* **Enable reliable LLM reasoning:** LLMs are more reliable at generating commonsense priors for concise object descriptions (e.g., \"a knife is likely in the kitchen\") than for complex relational queries (e.g., \"is it likely to find a dirty and metal knife near a sponge with a dirty plate?\"). As shown in the \"w/o VOI and Object Partition\" row in Table 2, reasoning over complex subgoals leads to a significant performance drop (−18%), emphasizing the value of object-level prioritization.\n\n**2. Robustness to Biased VOI Calculation**\nWhile initial VOI estimates may be biased due to the independence assumption and model limitations, the agent's belief state is updated incrementally with new observations. VOI is re-computed over this updated KB, allowing recalibration and alignment with the actual environment over time. This dynamic adjustment mitigates the impact of initial bias.\n\n---\n\n### \\[Weakness 3, Questions 1 & 3] Scalability and Cost\n\n**Reviewer comment:**\n\n> \"Could the authors provide further details on how the method might scale in real-world environments? Specifically, how might the approach be adapted to handle environments with significantly more objects or dynamic interactions between objects?\"\n>\n> \"While the method is effective in small-scale environments, it remains unclear how well the system scales to much larger, more dynamic real-world environments with a high number of objects and complex interrelationships.\"\n>\n> \"Can the authors elaborate on potential strategies for reducing the computational cost of VOI calculations, especially when scaling the method to larger environments with many objects?\"\n\n**Response:**\n\n**Handling Environments with Many Objects:**\nWe thank the reviewer for raising the important issue of scalability, which is one of the core motivations behind ActiveVOI. Real-world environments may contain thousands of objects, yet only a small fraction is typically relevant for a given goal. ActiveVOI is expressly designed to exploit this observation:\n\n* **Lifted relational abstraction:** Our planner uses lifted regression to perform goal-directed backward search, deriving relevant subgoal descriptions that filter out irrelevant objects.\n* **Object description consolidation:** When multiple subgoals share object descriptions, we consolidate them to eliminate redundant sensing queries, further improving efficiency.\n\nThese strategies ensure that the agent’s sensing and reasoning complexity scales with the number of goal-relevant objects, not the total number of visible objects.\n\n**Reducing VOI Computation Cost:**\nThe above design choices greatly reduce VOI computational cost by narrowing the search space. Another major contributor to cost is LLM query overhead. ActiveVOI minimizes this cost through:\n\n* **Symbolic planner offloading:** Planning is delegated to a symbolic planner, whose computation is negligible compared to LLMs.\n* **LLM token efficiency:** ActiveVOI uses \\~2,000 LLM tokens per task, compared to >50,000 in step-by-step LLM planning methods like ReAct.\n\n**Handling Dynamic Environments:**\nWe acknowledge that adapting to dynamic or probabilistic settings is an important future direction. However, doing so would require different methodologies and baselines (e.g., conformant or FOND planning \\[2,3]). These settings are orthogonal to our current focus, but we will include discussion of this in the final version.\n\n---\n\n**References:**\n\n\\[1] Valmeekam, Karthik, et al. \"On the planning abilities of large language models—a critical investigation.\" *NeurIPS* (2023).\n\n\\[2] Hoffmann, Jörg, and Ronen I. Brafman. \"Conformant planning via heuristic forward search: A new approach.\" *Artificial Intelligence* 170.6-7 (2006): 507-541.\n\n\\[3] Muise, Christian, Sheila McIlraith, and Christopher Beck. \"Improved non-deterministic planning by exploiting state relevance.\" *ICAPS* (2012)."}, {"review_id": "szu4dscz4p", "reviewer": "Reviewer_H9Rd", "summary": "This paper introduces ActiveVOI, a framework for open-world embodied planning under partial observability and incomplete knowledge. The key innovation is an active, object-centric approach to knowledge acquisition: instead of passively gathering all available information, the agent uses Lifted Regression to generate compact, relational subgoal descriptions and identifies relevant objects via the theory of Value of Information (VOI). Commonsense priors from LLMs/VLMs help estimate object utility. Experimental validation is performed on the visual ALFWorld benchmark, showing substantial improvements over previous (even fine-tuned) LLM/VLM-based baselines.", "questions": "- Are there concrete differences between using LLMs for task-relevant object prediction/information gain and previous LLM- or information-theoretic approaches to active perception?\n- Do the authors anticipate that this approach would transfer smoothly from simulated benchmarks (ALFWorld) to real-world robot settings? If not, why they choosing the ALFWorld simulator rather than other Open-World Embodied tasks?\n- One of the claims of the framework is its incorporation of LLMs/VLMs in quantifying the utility of knowledge acquisition targets—i.e., using LLM-based commonsense for Value of Information (VOI) calculations and prioritization. If a traditional symbolic/planning agent were given the same—fully specified and correct—knowledge base (i.e., a list of all objects, their types, and relations), could it not also generate an optimal plan using standard planning techniques (such as PDDL/STRIPS)? In other words, is the reliance on LLM commonsense truly necessary for high performance, or is it mainly compensating for incomplete or noisy object knowledge?", "limitations": "See ABOVE.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n- The paper tackles a crucial challenge in embodied AI: practical and scalable planning in open worlds, where passive information gathering is infeasible due to combinatorial complexity, although it's not a new concept. \n- The proposed framework combines Lifted Regression for global, object-centric reasoning with a decision-theoretic (VOI) framework for prioritizing which objects to sense.\n- The formalization of open-world planning with lifted predicates, explicit attention to handling incomplete knowledge, and proof-based (sound and complete) guarantees for the regression process. VOI application to active perception/planning is theoretically justified and well-motivated.\n\nWeaknesses:  \n- The writing and the figures of the manuscript requires further improvement. For example, fig.1 shows complex pipelines, while the differences of the two pipelines only mainly lines in the Active Knowledge Acquisition step. \n- ALFWorld is not an Open-World Embodied setting, at least from my point of view. There are some similar works tackling the active reasoning problems in environments like Minecraft or Crafter, which I think is more suitable to call as open-world. The author should consider testing their algrithims in more scenarios. \n-  Active perception and information-driven exploration are long-standing topics in AI and robotics, especially using VOI. Recent works have begun to use LLMs (and even VLMs) for goal decomposition, subgoal extraction, and commonsense reasoning for embodied agents. I'm not sure whether the integration of VOI with LLMs for active object-centric search significantly novel, or just an incremental composition of existing tools.", "was_revised": true, "final_justification": "While the paper demonstrates significant strengths, there remain a few points that I am not fully convinced by like the limited evaluating pipeline (only tested on a single benchmark modified by the authors) and the \"open-world\" setting argued by the author (which I think is \"partial observability\" but not really \"open-world\").", "rebuttal": "We sincerely thank the reviewer for the detailed and thoughtful feedback. Below, we address the concerns and questions raised.\n\n## \\[Weakness 1] Regarding Presentation and Clarity\n\n**Reviewer comment:**\n\n> \"The writing and the figures of the manuscript requires further improvement. For example, Fig. 1 shows complex pipelines, while the differences of the two pipelines only mainly lines in the Active Knowledge Acquisition step.\"\n\n**Response:**\nWe thank the reviewer for their feedback on Figure 1. Our intention was to illustrate the difference between 1) passive and active knowledge acquisition and 2) grounded progression and lifted regression. We can certainly revise the figure to improve clarity; to help us improve it, please let us know which of the following distinctions depicted in the figure seems unclear:\n\n* **Difference between planning strategy (Steps 4,6 Left, Steps 8,9 Right):** Grounded progression replans if actions fail or when knowledge is insufficient for plan generation. In contrast, lifted regression generates a compact set of feasible lifted plans up front.\n* **Difference in Knowledge Base Size (KB blocks):** Passive sensing requires the agent to store all object and relational information, resulting in a much larger KB than active sensing, which stores only task-relevant information.\n* **Active Perception Pipeline (Step 5 Left):** The active method uses lifted subgoals and the agent’s knowledge base to selectively acquire task-relevant information from observations.\n\n## \\[Weakness 2] \"Open-World\" Clarification\n\n**Reviewer comment:**\n\n> \"ALFWorld is not an Open-World Embodied setting ... similar works tackling the active reasoning problems in environments like Minecraft or Crafter ... The author should consider testing their algrithims in more scenarios.\"\n\n**Response:**\n\n**Open-World Clarification:** Our use of the term open-world follows the Open World definition from a model-theoretic perspective, where the agent operates with “incomplete knowledge of the model” \\[1], i.e., incomplete information about object types, configurations, and relational properties.\n\n**Minecraft-based benchmark:** Most Minecraft-based benchmarks are considered open-world due to their expansive environments and open-ended task structures. However, they are highly domain-specific and offer limited generalizability, as they focus on high-level actions like “mining” and “crafting” over a fixed set of predefined objects and materials unique to the Minecraft domain \\[2]. Minecraft recipes are also widely available in LLM training data, removing the need to infer new object types, relations, or affordances, which are core challenges in open-world planning and active sensing.\n\n**ALFWorld benchmark:** We configure ALFWorld to align with our definition of open-world settings, where agents operate under partial observability using only egocentric RGB input, without access to ground-truth object types, attributes, or relational information. This makes it well-suited for evaluating open-world active sensing. A comparison with other embodied AI domains is provided in the next section (see \\[Weakness 3, Question 1]).\n\n## \\[Weakness 3, Question 1] Novelty with Comparison of Existing Approaches\n\n**Reviewer comment:**\n\n> \"Active perception and information-driven exploration are long-standing topics in AI and robotics ... LLMs (and even VLMs) for goal decomposition, subgoal extraction ... not sure whether the integration of VOI with LLMs for active object-centric search significantly novel...\"\n>\n> \"Are there concrete differences between using LLMs for task-relevant object prediction/information gain and previous LLM- or information-theoretic approaches to active perception?\"\n\n**Response:**\n\nActiveVOI introduces a novel formulation of active sensing by explicitly reasoning over goal-driven object relevance and utility in the open-world setting. Here, we highlight the novelty of ActiveVOI with a comparison with existing active sensing work.\n\n**Novelty of ActiveVOI:** Most existing work on active sensing focuses on object or semantic navigation, where the goal is to (1) locate a known object, and (2) to maximize information gain \\[6]. In contrast, ActiveVOI tackles important but underexplored problems in active sensing: (1) determining what type of object the agent should sense, and (2) assessing how much utility (not just information) is gained from sensing an object. These two components constitute novel contributions to the field.\n\n**Comparison with LLM/VLM subgoal decomposition:** LLM/VLM subgoal decomposition suffers from several known issues: (1) lacking any correctness guarantee of generated plans, (2) requiring few short examples, (3) poor alignment with the environment and agent’s actions, (4) performance degradation over long-horizon. In contrast, ActiveVOI is based on a formal decision-theoretic framework of lifted regression that: (1) guarantees correctness of the plan, (2) can plan in a zero-shot setting, (3) is based on the agent’s action model, and (4) is capable of tracking long-horizon reasoning.\n\n**Comparison with Information-theoretical approaches:** Traditional information-theoretical approaches often define utility as generic information gain (such as entropy reduction) without directly connecting this gain to the agent’s goal \\[7]. In contrast, ActiveVOI is task-centric, where the utility is defined with respect to goal achievement. This distinction is crucial in goal-oriented tasks, where not all information is equally valuable, and task relevance must guide sensing.\n\n## \\[Question 2] Real-World Feasibility and ALFWorld Choice\n\n**Reviewer comment:**\n\n> \"Do the authors anticipate that this approach would transfer smoothly from simulated benchmarks (ALFWorld) to real-world robot settings? If not, why they choosing the ALFWorld simulator rather than other Open-World Embodied tasks?\"\n\n**Response:**\n\nWe want to clarify how the method can support real-world deployment as a high-level reasoning module, and justify our choice of ALFWorld over other embodied AI benchmarks in the context of open-world active sensing.\n\n**Real-world transfer:** The primary obstacles for the transfer of ActiveVOI lie in integrating low-level control and hardware-specific tuning. ActiveVOI, as a reasoning and planning module, should be readily adaptable to real-world applications due to (1) being zero-shot without the need for fine-tuning, (2) avoiding the combinatorial explosion of objects by using a compact lifted-representation of relevant objects, which increase scalability.\n\n**Suitable Embodied AI benchmarks:** We considered several other benchmarks based on Habitat, iGibson, and VirtualHome \\[3, 4, 5], but found them insufficient for evaluating open-world planning and active sensing. Tasks designed on these benchmarks are focused on 1) object navigation, 2) object rearrangement, 3) instruction following where all relevant objects are assumed known (given in the instructions), making semantic reasoning and object-centric active sensing irrelevant.\n\n**ALFWorld Rationale:** We selected ALFWorld tasks because they have several key characteristics essential for our open-world active sensing evaluation requirements: (1) task-relevant objects are not directly given and require semantic inference; (2) the environment features high diversity in objects, scenes, and tasks, along with rich visual observations; (3) it is partially observable and can be configured without access to ground-truth object and relational information; and (4) the benchmark poses significant challenges for SOTA models (VLMs), even with extensive domain-specific fine-tuning (show in Table 1).\n\n## \\[Question 3] Could a Classical Planner Succeed with Full KB? Is LLM Commonsense Really Needed?\n\n**Reviewer comment:**\n\n> \"One of the claims  ... If a traditional symbolic/planning agent ... could it not also generate an optimal plan ... is the reliance on LLM commonsense truly necessary ... or is it ... for incomplete or noisy object knowledge?\"\n\n**Response:**\n\nLLMs and VLMs are used not only for knowledge completion of the agent’s KB, but also to estimate the non-deterministic outcomes of sensing actions. As a result, a plan generated by a standard PDDL model, even with a fully specified and correct KB, may not be optimal, as it typically assumes deterministic outcomes. ActiveVOI uses LLM-derived commonsense priors to approximate the transition probabilities of sensing actions (the likelihood of encountering a relevant object (e.g., a heat source) in a given environment (e.g., a kitchen). This allows the agent to compute the expected utility of sensing different objects under our Value of Information framework. In summary, LLMs and VLMs are essential in our framework, both for completing missing knowledge and for modelling the non-determinism of sensing actions.\n\n## References\n\n\\[1] Finzi, Alberto, Fiora Pirri, and Raymond Reiter. \"Open world planning in the situation calculus.\" AAAI/IAAI. 2000.\n\n\\[2] Wang, Zihao, et al. \"Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents.\" Advances in Neural Information Processing Systems (2023)\n\n\\[3] Puig, Xavier, et al. \"Virtualhome: Simulating household activities via programs.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n\\[4] Savva, Manolis, et al. \"Habitat: A platform for embodied ai research.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019.\n\n\\[5] Shen, Bokui, et al. \"igibson 1.0: A simulation environment for interactive tasks in large realistic scenes.\" International Conference on Intelligent Robots and Systems (IROS) 2021.\n\n\\[6] Crespo, Jonathan, et al. \"Semantic information for robot navigation: A survey.\" Applied Sciences 2020\n\n\\[7] Stachniss, Cyrill, Giorgio Grisetti, and Wolfram Burgard. \"Information gain-based exploration using rao-blackwellized particle filters.\" Robotics: Science and systems  2005."}, {"review_id": "zDlmPWZ0Ol", "reviewer": "Reviewer_CBjT", "summary": "This paper proposes a novel method named ActiveVOI for embodied task planning with open-world and zero-shot capabilities. ActiveVOI uses Lifted Regression to generate subgoals which then help identify task-relevant objects. Then a quantification of object utility named Value of Information (VOI) is introduced, which is generated by LLMs and VLMs. Experiments on this framework are carried out on ALFWorld benchmark, showing its outperformance over state-of-the-art LLM/VLM based planners.", "questions": "1. Will there be experiments with other embodied environments like Habitat and iGibson?\n\n2. Will there be experiments under real-world settings?\n\n3. Is there any measure taken to overcome LLM’s problems of inconsistency and prompt-dependency, and/or help correct possible bias?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1. This paper presents its novel method with detailed explanation and illustration, along with clear and rigorous mathematical descriptions.\n\n2. This paper presents an organic combination of Lift Regression and VOI theory for open-world active knowledge acquisition, and cleverly makes good use of commonsense and reasoning abilities of LLMs/VLMs.\n\n3. Experiments prove this framework’s SOTA performance under ALFWorld environment, and multiple ablation studies clearly present the contribution of each component including VOI calculation and subgoal object acquisition.\n\nWeaknesses：\n\n1.  The VOI calculation highly relies on LLM and VLMs, which we know face the problems of inconsistency and prompt-dependency, especially when the scene is complicated, or the instruction is unclear.\n\n2. The experiment is conducted on ALFWorld exclusively, which lacks validation on generalizability.\n\n3. Some might have concerns about novelty, as notions of active knowledge acquisition and VOI are already established, and this paper just makes combination in a new context.", "was_revised": true, "final_justification": "As my concerns are well addressed, I am increasing my score", "rebuttal": "## Rebuttal\n\n### \\[Weakness 1, Question 3]: LLM/VLM Consistency and Sensitivity\n\n**Reviewer comment:**\n>“The VOI calculation highly relies on LLM and VLMs, which we know face the problems of inconsistency and prompt-dependency, especially when the scene is complicated, or the instruction is unclear.”\n\n>“Is there any measure taken to overcome LLM’s problems of inconsistency and prompt-dependency, and/or help correct possible bias?”\n\n**Response:**\n\nWe appreciate the reviewer’s concern about the reliability of LLM and VLM outputs in complex scenes and under prompt variability. Below, we describe the specific design choices we implemented to mitigate inconsistency, reduce prompt sensitivity, and ensure robustness in ActiveVOI’s commonsense reasoning pipeline.\n\n**Regarding LLM/VLM Prompt Sensitivity and Inconsistency:** To mitigate prompt sensitivity and reduce reliance on domain-specific prompt engineering, we adopt three key design choices.\n\n* We kept the prompts minimal and generic without additional task-specific prompt tuning or templates (shown in Appendix D.1).\n* We leverage test-time scaling techniques of self-consistency to improve reliability of probability estimates by aggregating LLM outputs 10 times, resulting in a more stable estimation of commonsense priors.\n* ActiveVOI operates entirely in a zero-shot setting, avoiding prompt-induced bias and ensuring better generalization across unseen tasks.\n\n**On Handling Complicated Scene:** We handle complex scenes by reducing the object search space through compact lifted object descriptions and by removing duplication of semantically equivalent objects. These strategies can reduce LLM/VLM inconsistencies by limiting queries to a small set of goal-relevant objects, rather than reasoning over all observed entities.\n\n**On Handling Unclear Instructions or Goals:** Unclear instructions or ill-defined goals pose fundamentally different challenges in open-world planning that require separate methodologies and benchmarks. Prior work frames these problems through contingent planning \\[1] or preference elicitation \\[2]. While important, these issues are orthogonal to our focus on object-centric knowledge acquisition in open-world planning.\n\n---\n\n### \\[Weakness 2, Questions 1, 2]: Motivation for ALFWorld Compared to Other Benchmarks\n\n**Reviewer comment:**\n>“The experiment is conducted on ALFWorld exclusively, which lacks validation on generalizability.”\n\n**Reviewer questions:**\n>“Will there be experiments with other embodied environments like Habitat and iGibson?”\n\n>“Will there be experiments under real-world settings?”\n\n**Response:**\n\nWe thank the reviewer for raising the important question of generalizability. Below, we clarify our motivation for choosing ALFWorld over other embodied benchmarks and explain the current limitations of evaluating in real-world settings.\n\n**Habitat, iGibson Feasibility:** While Habitat and iGibson are popular embodied AI benchmarks, they are primarily designed for object navigation and rearrangement tasks. In such settings, object relevance is trivial, as all necessary object knowledge is directly provided. As a result, these benchmarks are not suitable for evaluating open-world planning and active sensing methods, where agents must infer which objects are relevant under partial knowledge and semantic uncertainty.\n\n**Motivation for Using ALFWorld:** We selected ALFWorld tasks because they have several key characteristics essential for our open-world active sensing evaluation requirements:\n(1) task-relevant objects are not directly given and require semantic inference;\n(2) the environment features high diversity in objects, scenes, and tasks, along with rich visual observations;\n(3) it is partially observable and can be configured without access to ground-truth object and relational information; and\n(4) the benchmark poses significant challenges for SOTA models (VLMs), even with extensive domain-specific fine-tuning (shown in Table 1).\n\n**On Real-World Evaluation:** To the best of our knowledge, there is currently no real-world setup that can support the level of object diversity, action complexity, with object state changes required to test open-world planning and active knowledge acquisition. Building such a testbed would require significant engineering effort, which we view as outside the core contributions of our work which is our ActiveVOI framework based on formal decision-theoretic reasoning.\n\n---\n\n### \\[Weakness 3]: Regarding Novelty of ActiveVOI\n\n**Reviewer comment:**\n>“Some might have concerns about novelty, as notions of active knowledge acquisition and VOI are already established, and this paper just makes a combination in a new context.”\n\n**Response:**\n\nWe would like to clarify that ActiveVOI is not merely a combination of existing approaches, but addresses an overlooked and fundamental challenge in open-world planning: (1) determining which objects are relevant for sensing, and (2) estimating the expected utility of sensing those objects. This problem is particularly difficult in open-world settings, where the agent operates under partial observability and lacks prior knowledge about object types, configurations, and affordances, while the set of potentially observable objects is unbounded. In contrast, most existing work on active perception assumes that the set of relevant objects is given. Thus, ActiveVOI fills this important but often overlooked gap in existing active sensing approaches \\[3].\n\n---\n\n### References\n\n\\[1] Dearden, Richard, et al. \"Incremental contingency planning.\" 13th International Conference on Automated Planning and Scheduling. 2003.\n\n\\[2] Ren, Allen Z., et al. \"Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners.\" Conference on Robot Learning. PMLR, 2023.\n\n\\[3] Crespo, Jonathan, et al. \"Semantic information for robot navigation: A survey.\" Applied Sciences 10.2 (2020): 497"}]

Author rebuttal:

Meta-review: Initial reviews for this work were generally positive-mixed (1 accept, 2 borderline accept, 1 borderline reject). Key concerns were: (i) unclear technical contributions given reliance on established ideas like VOI and lifted regression with LLM priors, (ii) evaluation limited to ALFWorld with no test of generalization to other embodied environments, (iii) lack of prompt-sensitivity and failure mode analysis for LLM outputs, (iv) limited discussion of scalability and memory handling for long-term knowledge persistence in complex environments, and (v) unclear positioning relative to prior work, making it hard to attribute gains to the core contribution vs. stronger LLM/VLM priors.

The authors' response was thorough and provided experiments and clarifications to address many of these concerns. However, the reviewers remained split, with 2 "Accepts" and 2 "Borderline Rejects." The remaining concerns were more about the limited evaluation (Reviewers H9Rd and z6oJ) and terminology around "open world" (Reviewers H9Rd). After reviewing the paper, reviews, and the authors' response, the AC feels that the paper makes a good contribution whose strengths outweigh the weaknesses and recommends acceptance. The authors are strongly recommended to incorporate the discussion from the rebuttal into the final version for completeness.
Decision: Accept (poster)