Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following
Authors: Minjong Yoo
Jinwoo Jang
Wei-Jin Park
Honguk Woo
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual instruction
Embodied planning
Retrieval augmented planning
Integrated task planning
Primary area: reinforcement_learning
TL;DR: We propose an exploratory retrieval augmented planning framework that utilizes environmental context memory to address continual instruction tasks in non-stationary embodied environments.

Abstract: This study presents an Exploratory Retrieval-Augmented Planning (ExRAP) framework, designed to tackle continual instruction following tasks of embodied agents in dynamic, non-stationary environments. The framework enhances Large Language Models' (LLMs) embodied reasoning capabilities by efficiently exploring the physical environment and establishing the environmental context memory, thereby effectively grounding the task planning process in time-varying environment contexts. In ExRAP, given multiple continual instruction following tasks, each instruction is decomposed into queries on the environmental context memory and task executions conditioned on the query results. To efficiently handle these multiple tasks that are performed continuously and simultaneously, we implement an exploration-integrated task planning scheme by incorporating the information-based exploration into the LLM-based planning process. Combined with memory-augmented query evaluation, this integrated scheme not only allows for a better balance between the validity of the environmental context memory and the load of environment exploration, but also improves overall task performance. Furthermore, we devise a temporal consistency refinement scheme for query evaluation to address the inherent decay of knowledge in the memory. Through experiments with VirtualHome, ALFRED, and CARLA, our approach demonstrates robustness against a variety of embodied instruction following scenarios involving different instruction scales and types, and non-stationarity degrees, and it consistently outperforms other state-of-the-art LLM-based task planning approaches in terms of both goal success rate and execution efficiency.

Exploratory Retrieval-Augmented Planning For Continual Embodied Instruction Following
Minjong Yoo', Jinwoo Jang', Wei-jin Park', Honguk Woo
Department of Computer Science and Engineering, Sungkyunkwan University
2 AAryyl Inc.
mjyoo2@skku. edu, jinustarōskku. jingacryl.ai, hwoodskku. edu
Abstract
This study presents an Exploratory Retrieval-Augmented Planning (ExRAP) frame- work, designed to tackle continual instruction following tasks of embodied agents dynamic, non-stationary environments. The framework enhances Large Lan- guage Models' (LLMs) embodied reasoning capabiliites by efficiently exploring the physical environment and establishing the environmental context memory, thereby effectively grounding the task planning process in time-varying environ- ment contexts. ExRAP, given multiple continual instruction following tasks, each instruction is decomposed into queries on the environmental context memory and task executions conditioned on the query results. To efficiently handle these multiple tasks that performed continuously and simultaneously, an intoraationiitegratee task planning scheme by incorporating the information- based exploration into the LLM-based planning process. Combined with memory- augmented query evaluation, this integrated scheme not only allows for a better balance between the validity of the environmental context memory and the load of environment exploration, but also improves overall task performance. Further- more, devise temporal consistency refinement scheme for query evaluation to address the inherent decay of knowledge in the memory. Through experiments with VirtualHome, ALFRED, and CARLA, our approach demonstrates robustness against variety of embodied instruction following scenarios involving different instruction scales and types, and non-stationn con-stationarity degrees, and it consistently out- performs other state-of tate-offthe th-the--rt LLM- M-seeded task planning approaches in terms of both goal success rate and execution efficiency.
implement
Introduction
The application of Large Language Models (LLMs) embodied AI is essential for harnessing common knowledge and immediately applying it to unseen tasks and domains without requiring additional training or data. Researchers are further enhancing task adaptation by integrating environ-
mental information with the intrinsic common knowledge of LLMs 2 4 5 6 7). This capability proves invaluable fields such as home robotics and autonomous driving, where it enables embodied agents to learn across diverse instruction following tasks with minimal data requirements.
For embodied agents, these tasks are often not mere single, one-time instructions but are multiple and persistent, necessitating continuous access to environmental knowledge to reason and plan effectively for user needs. In such scenarios, the efficiency of repeatedly collecting environmental knowledge through interaction each time the agent plans can be suboptimal. Furthermore, there is a clear need to integrate and manage multiple user requirements effectively.
"Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we investigate continual instruction following for an embodied agent, where multiple tasks are contingent upon the real-time information of continuously changing environment. This setup requires the agent to engage in ongoing exploration of the environment to adaptively respond to dynamic changes and fulfill the required tasks. To address the problem of continual instruction following, we present an exploratory
planning (ExRAP) framework, designed to enhance LLMs' embodied reasoning capabilities by integrating environmental context memory.
In ExRAP, to improve effectiveness and efficiency in managing environment interaction and explo- ration loads for multiple embodied tasks, we employ an exploration- inntrrratee task planning scheme, in which the information-basee
exploration is incorporated into the LLM-based planning process. This integrated planning scheme establishes a robust policy that balances the validity of the environmental context memory with the demands of environment interaction and exploration. We also devise a temporal consistency- based refinement scheme to ensure the robustness of memory-augmented query evaluation on environmental conditions. Through experiments with VirtualHome (8), ALFRED [9] and CARLA [0),, we demonstrate that the ExRAP framework achieves competitive performance in both task success and efficiency compared to several state-of-the-art embodied planning methods,
including ZSP [ SayCan []. ProgPrompt (3), and LLM--Plnnner [12).
Our contributions are summarized as: First, we propose a novel ExRAP framework, systemati- cally combining LLMs' reasoning capabilities and environmental context memory into exploration- integrated task planning to tackle continuous instruction following tasks in non-stationary embodied environments. We also introduce two schemes tailored for opploration-interrattee task planning in EXRAP, information-base exploration estimation and temporal consistency- bassed refinement on query evaluation. Finally, we demonstrate superior performance and robust- ness of ExRAP via intensive experiments with home robots and autonomous driving scenarios in VirtualHome, ALFRED, and CARLA.
2 Related work
Embodied instruction following, Embodied instruction following involves executing complex tasks based on an understanding of embodied knowledge. This aims to grasp various aspects of the physical environment including objects, their relations, and dynamics, and to plan appropriate sequences of actions or skills to complete the tasks specified by instructions successfully. In the area of task planning, there have been many works combine LLMs' reasoning capabilities with environmental characteristics. Recent research explored the utilization of skills' affordances to compute their values 21. implemented code-driven policies (3,4) 4. and generated reward functions [13 14), while highlighting the use of LLMs' enhanced abilities in task planning. Moreover, LLM-driven environment modeling approaches, utilizing L.LMs' common knowledge and reasoning about real- world objects, have been introduced (5 6 77. These LLM-basee approaches to task planning have been applied across a range of embodied instruction following tasks, facilitated by repeated interactions with environments, humans, or other agents [5
16 12
18 19].
While these approaches underscore the versatility and depth of LLMs for task planning and embodied agent control, they often rely on -on-systematic integration of observations to the LLMs' reasoning process. Furthermore, they rarely consider continual instruction following scenarios, where an agent should handle a set of instructions continuously, adapting to real-time environmental conditions. In contrast, our work differentiates itself by incorporating agents' exploration capabilities, which are guided by information gains, into continual instruction following.
Retrieval- augmented generation for LLM. Research in retrieval -uggeetted generation (RAG) fo- cused on efficiently executing tasks by sourcing and utilizing task-related information from databases. In particular, enhancing the performance of retrieval, which suggests relevant data when an LLM requires specific knowledge for the tasks, involves training retrievers [20 21 22 23], fine-tuning the LLM to adapt the RAG process [242 or exploiting the LLM itself for dynamic query refor- mation (26 27]. In the area of embodied task planning, recent studies adopt the integration of RAG with task-specific demonstrations [12]. Our work also uses RAG for embodied task planning, but it uniquely emphasizes its dynamic aspects. For continual instruction following, we prioritize the relevance and significance of the agent's skills, not only to perform tasks but also to ensure continuous and efficient synchronization of its environmental memory with changes in the environment.
2

---

Integrated plun
Instructionww plan
Continual instruction:
0. All conditiaes rect
6.
All anstiton mel
Ii the temperature is high, open the wandow,
Walk bedneen
1. Walk bodsoers
When watching TV, sum off the light.
Clzcia lceap, & TV light exttch
[heck samp
box and sink close togriher, soparate them.
Welk keecher
on light owitc kichen
wislow
Embodied ageal
Environmentall contect memory
Embodied envireament
Opes wislow
sikk
(4) Query evaluation
Open
Check
Walk light owctch
Memory update
Pox tohie
tuble
10.
9. Tir owtcc
Walk kitcker
Query response
11
Walk eist
12.
Cinds her
Exploration-intogeen task planning
15. Walk tabli
Pur box on table
Figure 1: Concept of ExRAP. In the embodied environment, this framework manages continual instructions, set of instructions for embodied instruction following tasks that are conducted continu- query evaluation,
ously and simultaneously. At each step, it operates through (a) memory-augmente task planning coupled with environmental context memory updates
and (b) inllorationnnnterratee (as shown in the left side of the figure). By performing this integrated plan in conjunction with the memory, the ExRAP framework achieves more efficient task exccution in response the continual instructions, compared to the instruction-we wise planning (as in the right side of the figure).
Exploration in reinforcement learning (RL). In the field of RL., exploration methods, designated to efficiently gather environmental information, have been focus of research. Strategies were developed. that prioritize the exploration of new environmental information, by offering intrinsic rewards [28)
[2] or through navigation schemes derived from offline demonstrations (30]
32
Particularly,
in DREAM [33], an exploration policy formulated to adapt to varying conditions by using data gathered during initial exploration episodes. This policy is optimized by maximizing the information exploration strategy with RAG for embodied instruction following. Our proposed framework is the first to implement exploration-integrate task planning with LLMs, enabling the efficient execution of continuously- performed instructions and facilitating the dynamic adaptation changing environmental conditions.
gain to better adapt to changes. Our work adapts this mutual information-base
Approach
3.1 Continual instruction following
We consider a set of instructions for embodied tasks that are continuously and simultaneously conducted based on specific environmental contexts. In Figure[] the instructions entail conditional actions like "If the temperature is high, open the window" and "When watching TV, turn off the light." The agent continuously explores the environment to verify whether the conditions (e.g.. temperature and TVV status) are met. Upon confirmations, the agent executes the associated tasks within the environment. We refer to these scenarios, where multiple embodied tasks are conditioned on environmental contexts and conducted continuously, as continual instruction following. This concept aligns closely with continuous queries [34 (35] in database literature, which monitor updates of interest over time and return results when specific thresholds or conditions are met. This is in contrast to single in-situ instruction following, where each task is executed based on isolated, one-time directives.
For continual instruction following tasks with conditional instructions I {ii, iM we consider non-stationary embodied environment that changes over time. The conditions of continual instructions may or may not be satisfied over time, requiring continuous exploration in the environment. When certain conditions are met, the associated tasks should be performed promptly. For this continual instruction following tasks in the non-stationary environment, we evaluate agent performance in terms of task completion and efficiency. Our goal is to establish an embodied agent policy n² that maximizes the overall performance of continual instruction following tasks. Specifically, we formulate the reward as a combination of (i) the task success rate SR and (ii) the average pending step PS. SR is the rate of completed tasks whose conditions have been met, and PS the average steps required to accomplish the task associated with instruction E Ic whenever the condition is met. For instructions and timestep 4, we then formulate the agent policy म* performing a skill upon observation o, as
Σ SR(s ((,,, I)) + I-PSL,
π² argmax,
3

---

Optimizing both SR and PS allows the agent not only to appropriately plan for multiple instruction following tasks but also to strategically integrate these instructions improve overall efficiency. The resulting policy is able to minimize redundant skill executions by addressing multiple tasks in an integrated manner, taking into account the possible spatial and temporal overlap of the task requirements. For instance, as illustrated in Figure [] an integrated plan achieves pending step of 7, the average of required timesteps 4 and 10 for the three instructions. This significantly shorter than 9.2, the average of 5, 9, and 14 achieved by an instruction-wise plan.
3.2 Overall framework
To address the challenge of continual instruction following in a non-stationary embodied environment, develop the ExRAP framework. It is designed to minimize the necessity for environmental interaction, by utilizing nemory-augmented and exploration-integratee planning schemes while ensuring robust task performance.
In ExRAP, each conditional instruction E I decomposed into two primary components: query q and execution Queries function as conditions for task initiation and are evaluated against environmental information. Executions, on the other hand, involve physical interactive manipulations that are triggered based on the results of query evaluation. In non-stationary environment, evaluating queries poses a unique challenge due to the need for the agent to continuously synchronize with constantly changing information. This synchronization often necessitates continual exploration, resulting in intensive interaction with the environment.
As described in Figure 1 ExRAP addresses this challenge through two components: (a) query evaluation using environmental context memory and (b) exploration-integrated task planning. In (a), the environmental context memory is established via temporal embodied knowledge graph to effectively represent the dynamic environment. Augmented with this graph-based context memory, the LLM-based query evaluator responds to queries by checking if their conditions are met and provides confidence levels for these assessments. To address the information decay, which stems from synchronization uncertainty between the previously collected environmental context and the actual current state of the environment, we incorporate entropy-basee temporal consistency refinement into the query evaluation process. In (b), ExRAP plans skills that instrumental not only for achieving tasks from an exploitation perspective but also for boosting confidence in the query evaluations from an exploration perspective. To effectively plan skills that balance both perspectives, we integrate the exploitation value of skills, which is derived from the in-context learning ability of LLMs, with their exploration value, which is determined through information-based estimation.
3.3 Memory-augmented query evaluation with temporal consistency
We represent both the environmental context memory and the observations perceived by the agent using a temporal embodied knowledge graph (TEKG), where the memory is established through the accumulation of these observations. Queries, derived from given instructions, are evaluated against the context memory, with consideration for inherent information decay within the previously accumulated data. The query evaluation procedure is described on the upper side of Figure [
TEKG and retriever. The TEKG comprises source entity se, relation re, target entity te, and timesteps f. We represent the environmental context memory at a specific timestep within the TEKG, defined as
of quadruples 7 (sc, re, te, t) consisting of
G (7I, 72, " ,N) where T; (ses, res, le,, L4), l4 S I.
To integrate the current observation 0++1 into previously established up-to-date memory Gt, employ an update function as follows:
G++1 = p(Gr, 0141) = {r G, (c(r, = o, VT E or+ 1
Here, is function that detects the semantic contradictions between quadruples, such when and 7' indicate that a TV is both "off" and "on". It returns if there is a contradiction, otherwise 0.
Constructed memory G serves as knowledge database for continual instruction following, enabling the retrieval of environmental information related to specific task directives such as instructions, queries, and exccutions. Specifically, for language-specife task directives L = {2). the retriever &

---

Query evaluation
Environmental
memory
Continual indraction:
Terpord enbodied lnswlldgge (TEKG)
Memory update
Iri sey ortten a ligh coe
Toweh
Wirdow
Observasion
(rihe flon
Quris (EKD0)().:)))
Buthroo
Jkdben
Paicd
Non-statiosary embodied environment
sneppreterd
Mamac--agnenicc query cviluntee Gxy
Dess
Lasigritth Belnnan
Off
Retriover thg
(b) iettivaationnnnlegrate lask planning
User Vormmmon Tine
Imcernnd skill valse
Explocation planner Up
plae all
Dulad 2
Walk window
Excerioss
9uth.
Denoomttices
30 Welk kinhen
TE
TEKO d (G7-2)
Exploitatin Vy
Skill
Walk window
(:06,)) ((P(..6))
2(e,60r.0)
Figure Overall procedures ExRAP. (a) Query evaluation: The instruction interpreter $ produces queries and executions, well condition function from continual instructions. The memory- augmented query evaluator M then evaluates these queries probabilistically using the LLM with retrieved TEKG from the environmental context memory. (b) iuploration-integratee task planning: The LLM-based exploitation planner By estimates the value of skills based on their executions and relevant demonstrations in in-context manner. Simultaneously, the exploration planner vn evaluates these skills using the subsequent TEKG through information- based value estimation. At each step, skill then selected based the integrated skill value from the two estimations.
interacts with the memory G and samples & quadruples {f..... )... The sampling is based on the multinomial softmax distribution, where the likelihood of retrieving quadruple is determined by the highest sentence embedding similarity between т and any 7 L.
Instruction interpreter.
The instruction interpreter & processes continual instructions I {i,,... (M J. translating them into queries Q and corresponding task executions E:
+ (T) = (Q: (qM) & (eI....,eM) C) where C(q,) = eg for Vj.
Here, Cs conditional function that maps each query to its respective execution counterpart.
Query evaluator. The memory-augmentee query evaluator 4M estimates the likelihood P.g of query
being satisfied, using the historical memory accumulated over time, denoted as G = G U U... U G,- Leveraging the memory-augmented LLM (PLIM), we develop the query evaluator xr by incorporating the previous step's query evaluation G-g( and prior of query evaluation R()) which is defined in 77.
R(g]G:-1) @ Gra, R(g(Gr-1) otherwise
if G = G1:t-1 (5)
P(g[Gr) 4 Pnrggt G1a, P(glG.-1))
Here, GLa ~ 2 RG (g]) is retrieved quadruples, and the prior R(q[G;-1) is the retrospective query response evaluated at timestep t using G.-
Due to the inherent information decay in the memory over increasing timesteps, a decline in con- fidence should be considered for the likelihood estimation in (5). To address this, incorporate an antropy-based temporal consistency as an intermediate step in query evaluation. Specifically, when using the memory from G,-1. we posit that the entropy of the prior query response at timestep t should be higher than at timestep 1-1:
H H(R(g|G.- H(P(g)).-.)).
(6)
To enforce the temporal consistency, we compute the multiple query response priors using PLIM and discard any responses that do not align with the consistency constraint. Specifically, if the entropy of each prior of query response is smaller than previous step P(q)Gr), it removed.
R-1) = EG in We Gna
PLLM (q. (4.4 P(y(Gr-1)) with hold 6
(7)
5

---

Then, we select set of corresponding executions & that are likely to require manipulations in the environment, using a filtering threshold 0.
& = {C(1)) P.((G) > 8)
(8)
3.4 Exploration-integratee task planning with information-based estimation
To facilitate integrated task planning for continual instructions, we devise exploitation and exploration planners. The former focuses exploiting appropriate skills to complete tasks associated with given instructions using the LLM, and the latter focuses on exploring the environment to update the memory in direction that maximizes information gain. We then integrate their plans to prioritize the next skills to be executed. The resulting planning directs to complete specific task executions E, ensuring effective maintenance of the environmental context memory. This maintenance process involves synchronizing the memory with the current state of the environment. The exploration-interrated task planning procedure is described on the lower side of Figure
Exploitation planner. Given the memory G. and a language description a skill E Z, the exploita- tion planner vr is responsible for estimating the value of the skill with respect to its effectiveness in accomplishing the executions & To do so, we harness the retrieved memory- -ugmentd LLMs along with their in-context learning capabilities. Specifically, under the assumption that expert planning dataset De, we retrieve demonstrations from De based on the graph similarity between the current observation o, and the observation within D...
access
UT (G, z) (E & Rn(Gr, E,, D,=)
Exploration planner. In conjunction with the exploitation planner, the exploration planner UR is responsible for assessing the value of the skill with respect to its utility for reducing the response uncertainty of the query evaluator & M- This assessment is intended for efficient environmental explo- ration, thereby facilitating the swift and precise identification of query conditions and maintaining the memory up-to-date. Specifically, we define the exploration value of skill as the difference of mutual information for consecutive timesteps using the query evaluation result in (
en en(G,,, = I(Q: -(Q.))
T(Q: - - H(P(d)
(10)
QEQ
where 7 denotes mutual information and G++1 is the updated memory by execution of skill z.
Direct computation of G:-1 is impractical without actual skill execution. Therefore, to evaluate the exploration value of skills before their execution, we focus on the entropy related only to the retrieved knowledge pertinent to the evaluated queries Q. This approach is feasible under the mild assumption that the entropy of the query evaluator reaches zero (indicating no uncertainty), once the TEKG memory is fully synchronized with the environment. Thus, approximate the exploration value as
d(OR (1- d( {4}) (G,, {qb)
UR (G, = [H(P(GIG.)) 9EQ
(11)
where d represents the average distance function between the retrieved quadruples and the current agent's entity in TEKG, and Gē is the predicted partially updated knowledge graph after skill execu- tion, where only quadruples containing the agent as an entity are altered. Note that the exploration value increases as the agent moves closer to the query-related environment parts on the graph through the skill execution. Consequently, the skill selected by maximizing the integrated skill value, which is obtained by a weighted sum of exploitation and exploration values, as defined in ( and respectively:
2t argmax([wT UT (Gt, + wR UR z)]- =EZ
(12)
Experiments
We evaluate ExRAP across various degrees of non-stationarity, scales of instructions, and instruction types. We also provide ablation studies and qualitative analysis. Further analysis is in Appendix D.

---

Table I: Performance in VirtualHome, ALFRED, and CARLA w.r.t non-stationarity. We the 95% confidence interval, using 10 random seeds for VirtualHome and 5 random seeds for both ALFRED and CARLA.
Low non-stationarity
Medium non-stationarity
High non-stationarity
Model
SR (t)
PS (1)
SR (*)
PS ())
SR (t)
PS (1)
Evaluation VirtualHome
ZSP
20.59%:+4712
31,03=1.68
20.06%::1.957
32.06:1:1.66
17 17.28%o#3.16%
24.08
SayCan
12%-
21.67=181
33.69%%:15 30%
21.81:4.14
27.33%:24244
16. 184339
ProgPrompt
4..10%%::4.44
18.84=24088
30.51% 45.311
23043++1007
27. 27.19%##2.99%
18.6044422
LLM-PPanner
40.977%=:.00%%
17.61=1.40
39. 4-
15.93-2..1
34.60%%2..49
14.94+2.99
ExRAP
61.12%::7.03%%
11.75:+2.49
$5.14 % 6.55%%
11.33:1192
50.12%:1:.70
8.61::2.55
Evaluation in ALFRED
ZSP
18.22%%:5.33%
12.24
14.67%::...88
20.83:43.63
9.56%
22.53+33.57
SayCan
45.67%:2689%
8.25+1186
41.81 7.:6..6%9%
8.39+
35. 76331
7.42+1.11
ProgPrompt
47.15%##1.17%
9.8142.14
35.62%:+111
7.22+1.35
19.97%
7.52+:2.4
LLM-PPanner
3.994
7.28:1109
63.809
7.28
35.76 755.66%
1.6
ExRAP
69.90%
5,94:092
64.00 +507%
4.82:1103
59.11 2.488
4.42:11.36
Evaluation CALRA
ZSP
10.44%%==1.0%
29.35=7.21
6.89%
32.466116003
4.67%
33.00+
SayCan
37.55%++4744
20.73=5.36
35. 14.11% 11% 12%
22.44:5538
30.71 % 28%
21.71+2201
LLM-PPanner
50.83%++1.600 14.02=301
44.00%:+..70%%
14.39++1.4
41 11.58%
13.59+2265
ExRAP
65.25%:47.477
12.43-13900
62.25 6.772%2
11.50=2.24
58.83 14.33%-110.0m
10.00%
10.84=2.52
Environments. We evaluate ExRAP in the context of household planning and skill-based autonomous driving with VirtualHome 88] ALFRED (9). and CARLA [10), where we use 16 to 19 distinct instructions for continual instruction following tasks. Details provided in Appendix A.
Evaluation metric. We employ two evaluation metrics for the objective specified in (T). The task success rate (SR) measures the proportion of completed tasks for continual instructions whose conditions are satisfied at each timestep. Given the continual instructions, the pending step (PS) represents the average number of timesteps required complete the associated tasks from the moment the conditions of the instructions are actually satisfied in the environment. Note that the agent's detection time of such condition satisfaction may differ from its actual occurrence.
Datasets. We use 100 trajectories across different environment settings in VirtualHome, and 50 trajectories in ALFRED and CARLA. These are used for in in-context learning of the exploitation planner in ExRAP and the baselines. Note that we use different environment settings for evaluation.
Baselines. ZSP [I] is an LLM-based zero-shot task planner. Our experiments serve as a baseline evaluate based task planning approaches. CayCan [1] is a state-of-the-art embodied agent framework, which integrates both language affordance scores derived from an LLM and embodied affordance scores learned through RL. In our experiments, we use the optimal affordance function for each environment. ProgPrompt (3 is framework to enhance language models' capabilities in gener- ating structured and logical outputs, by incorporating programming-like prompts. I.I.MPllnner [12) is state-of-th state-of-the-art embodied agent framework that utilizes L.MSs embodied knowledge to infer subtask sequences. incorporates object detection information from the agent's interactions for enhanced task planning. To address continual instructions and adapt non-stationary environments, we implement a variant of the LLM-PPanner that infers skills step-wise manner.
4.1 Main results
Non-stationarity. Table I presents a performance comparison in terms of SR and PS VirtualHome, ALFRED, and CARLA, respectively, under varying degrees of non- con-stationarity. where environment changes range from low high. The higher degree of non- -oatioottn means the environment changes more rapidly, requiring the agent to focus more on environmental information to adapt effectively. ExRAP achieves superior performance across degrees of non-stationary. Specifically, ExRAP demonstrates performance gain in SR by 16.45% on average compared to the most competitive baseline, the ---M-ere Furthermore, ExRAP shows a reduction in PS by 3.40 on average compared to the LLM-Plannee Importantly, the advantage of ExRAP becomes more significant with
7

---

Table 2: Performance in VoreualHome w.r.t. instruction scale
Small continual inst. (=3)
Medium continual inst. (=5)
Large continual inst. (#7)
Model
SR (1)
PS (.)
SR (1)
PS (4)
SR (1)
PS (U)
ZSP
23.111%=2..55%
22. 86:1244
20. 1..96%
32.06=41.66
T.43%:22.2
29.16:+26.04
SayCan
40.38%==8.9%
19.7642.02
33. 33.69%%45.30
21.81
23.04%::12.266
37.05++17.43
ProgPrompt
43.15%==3.224
19.99
30.51%:45.31%
23.43=1007
21 7..200%
29.00
LLM-Planner
49.28%:=5:008
17.1311767
39.89%:1445529
15.93=2.13
11.82%2114300
17.63+22.4
ExRAP
67.77 56 1:4.56
12.29
55.14% 6.59%%
11.33-11.92
53,86 % 1.599%
8.76:10.44
increasing levels of non-stationarity; the performance gap in SR between the LLM-Planner and ExRAP widens from 15.35% at low non-stationarity to 18.58% at high non-stationarity. Similarly, the be attributed to
gap in PS grows from an average of 2.96 to 3.73. This increase in performance ExRAP's strong ability to promptly discern newly satisfied conditions through pegloration-interraan task planning, coupled with accurate memory- augmented query evaluation. This capability enables ExRAP to respond to rapid environmental changes effectively.
Instruction scale. Table 2 shows a performance comparison under the medium non-stationarity, as the scale of continual instructions increases. As the number of instructions grows, the agent needs to collect more knowledge and perform more tasks. ExRAP achieves average gain in SR of 18.78% compared to the most competitive baseline, the LLM-Planner. Additionally, ExRAP reduces PS by an average of 6.31. ExRAP exhibits widening performance gaps the complexity of tasks increases; the SR gap grows from 8.499 compared to the LLM-Planner with a small continual instruction scale, to 22.04% % with large continual instruction scale. Similarly. the PS gap expands from average of 4.84 to 8.87. This performance difference highlights ExRAP's effectiveness in addressing multiple continual instructions simultaneously, demonstrating its robust integrated task planning capabilities. In ExRAP, the skill selection is guided by an integrated value derived from queries and executions, enabling the efficient handling of multiple instructions and concurrent memory updates.
Instruction type. We also test the applicability of ExRAP for three types of continual instructions in VirtualHome with medium non-stationarity. Sentence wise type organizes each instruction into individual sentences. This the default setting in our experiments. Summarized type condenses multiple instructions into fewer number of sentences. Object Ambiguation type contains abstract forms of target objects such as "something to read.". Table demonstrates superior performance of ExRAP for those types. ExRAP adapts RAG with the environmental context memory to interpret instructions and decompose them into queries and executions. This -ugmeettd approach, retrieving and utilizing the information relevant given instructions, enables effective grounding of continual instructions in different types within the environment.
Table 3: Performance w.r.t. instruction types
Sentence-wise
Summarized
Object Ambiguation
Model
SR (1)
PS (1)
SR (t)
(1)
SR (1)
PS
SayCan
33. 6..6
21.8114.4
32.66%
22.13.1567
10%
28. 28.03.15544
J..M-Planner
199.89%
15.931213
37. 87.19%;21.8
16.45:187
30.88% 70%
19.092499
ExRAP
55.14 % +6.59%
11.33=1192
53.11 % 15.100
10.42-428
50,26 % + 10.118
13.51=6.17
Qualitative analysis. Figure (a) and (b) compare the exploration strategies of ExRAP and LLM- Planner. Given multiple instructions, ExRAP demonstrates broader exploration. This reflects the cnlloration-integraaee task planning in ExRAP, which rather focuses on the overall gain achieved from each exploration and skill exccution.
4.2 Ablation study
We conduct several ablation studies for ExRAP in VirtualHome with medium non-stationarity.
Temporal consistency. We compare the performance of our ExRAP and its variant EXRAP-TC that performs query evaluation without temporal consistency-base refinement, As in Table EXRAP outperforms EXRAP-TC by 15.56% on average. As observed in Figure 3 (a) and (c), with tempo- ral consistency in query responses, information decay is effectively managed, Icading to broader
8

---

Frogancy
(a) ExRAP
(b)
(c) EXRAP-TC
(d) ExRAP-EXP
Figure 3: Knowledge exploration heatmap. Darker color represents high frequency in exploration.
exploration areas that accommodate different instructions simultaneously (i.e., in (a)). Otherwise, without temporal consistency, exploration tends concentrate exclusively specific knowledge where multiple queries overlap. This renders largely neglected areas experiencing significant decay, often leading invalidated query evaluation (i.c., in (c)).
Table 4: Ablation for query evaluation with temporal consistency
Low non-stationarity
Medium non-stationarity
High non-stationarity
Model
SR (t)
PS (L)
SR (t)
PS (U)
SR (t)
PS (J)
EXRAP-TC
17.25%
15. 15.30
43.92%.17.7%
15.47
27.91%.11466
9.67
ExRAP
61.13% 11376%
11.66
55.14% 16.59%
11.33:112
49.73% 88.88%
8.74
Exploration strategy. We compare the performance of our ExRAP and variants EXRAP-LLM and ExRAP-EXP using different planning strategies. EXRAP-LLM directly employs the LLM as the exploration planner, which fully relies on the LLM's capability without information-based exploration, by prompting the LLM with specific exploration commands, e.g., "explore the home." On the other hand, ExRAP-EXI employs only the exploitation planner with specific exploration commands, used only when there are no executions. As shown in Table ExRAP demonstrates higher performance of 26. 76 and 17.46% on average than the variants, respectively. As in Figure (a) and (d), ExRAP-EXP exhibits reduced exploration capabilities, resulting in a narrower exploration area.
Table Ablation for exploration-integrated task planning
Low non-stationarity
Medium non-stationarity
High non-stationarity
Model
SR
PS ())
SR
PS (Q)
SR
PS
EXRAP-LLM
34. 34.23%
15.1715.11
29. 20.42%%
10.781384
21.81%11408
14.65.1240
EXRAP-EXP
43.33%
13.07
40.68%22777
13.03+3m
29.36%641435
12.1643.6
ExRAP
61.13 133.66
11.66=3.93
55.14 163899
11.33
49.73% 2RS8S
8.74:274
LLMs for planning. While tested several LLMs, ranging from relatively smaller models such as Gemma-2B [36] to larger ones such as Llama-3-708 (37), our experiments thus far have utilized LLaMA-3-8B for ExRAP and the baselines. In Table 6 evaluate ExRAP and two baselines using different LLMs. Both LLM-Planner and SayCan exhibit improved performance with the larger Llama-3-70B, but experience significant drop performance with the smaller Gemma-2B model. Unlike those, ExRAP maintains robust performance even with the smaller model, highlighting the benefits of its memory- mugory-augmented, integrated planning approach.
5 Conclusion
We introduced the ExRAP framework to facilitate efficient integrated planning for multiple instruction following tasks, which are conducted continuously and simultaneously in the embodied environment. With the extended RAG architecture, the framework incorporates vemory-augmentt tion and exploration-integratee task planning schemes, thereby achieving both efficient environment exploration and robust task completion. Via experiments conducted in VirtualHome, ALFRED, and CARLA, we demonstrated the robustness ExRAP across various continual instruction following scenarios, specifying its advantages over other LLM-driven task planning approaches.
query evalua-

---

Table 6: Impact of different LLMs
Gemma-2B
Llama-3-8B
Llma--370B
Model
SR (?)
PS (J)
SR (1)
PS (U)
SR (1)
PS (J)
SayCan
27.53%::5.44%
19.9816682
34.31%::14.05
21.43:331
45.11%648.528
14.39=3.04
L.IM-Planner
23.31%:
20.74
39.07% s9s
15.81
47. 47.18%448428
16.33=2.79
ExRAP 52.75% 19.11%
13.36:362
54.89% #13.39%
10.59
=5.12%=93329
10.07:261
Limitation. ExRAP leverages LLMs, which makes its performance dependent the capabilities of these models to some extent. Compared to other baselines, also requires increased computation effort due to the management of environmental context memory with temporal consistency. The ablation studies demonstrate that ExRAP is able to deliver robust performance even with a relatively lightweight LLM, yet further investigation into runtime overhead is desired.
6
Acknowledgement
This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT), 1--20022-1222044 (2022-0- 00043), Adaptive Personality for Intelligent Agents, RS-2022-11221045 (2022-0-01045), Self-directed multi-modal Intelligence for solving unknown, open domain problems, and 1-2011-11190422 Artificial Intelligence Graduate School Program (Sungkyunkwan University)). the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-002131 18), ITT--ITRC (Information Technology Research Center) grant funded by the Korea government (MIST) (ITTP-2024-RS. -0--0002-4000076633 10%), and by Samsung Electronics.
References
[1] Anthony Brohan al. "Do can, not as say: Grounding language in robotic affordances".
In: Proceedings of the 6th Conference on Robot Learning (CoRL). PMLR. 2023, pp. 287-318.
[2] Rishi Hazra, Pedro Zuidberg Dos Martires, and Luc De Raedt. "SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledge". In: Proceedings of the 42nd AAAI Conference on Artificial Intelligence. Vol. 38. 18. 2024, pp. 20123-20133.
[3] Ishika Singh al. "Progprompt: Generating situated robot task plans using large language models". In: Proceedings of the 40th International Conference on Robotics and Automation
(ICRA). IEFF. 2023, pp. 11523-11530.
[4] Jacky Liang et "Code as policies: Language model programs for embodied control". In: Proceedings of the 40th International Conference on Robotics and Automation (ICRA). IEEE. 2023, pp. 9493-9500.
[5] Lin Guan et al. "Leveraging pre-trained large language models to construct and utilize world models for model-based task planning". In: Proceedings of the 36th Advances in Neural Information Processing Systems (NewrIPS) 36 (2023), pp. 79081-79094
[6] Shibo Hao et al. "Reasoning with language model is planning with world model".. In: arXiv preprint arXiv:2305.1 14992 (2023).
[7] Kolby Nottingham et "Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling". In: Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR. 2023, pp. 26311-26325.
[8] Xavier Puig et al. "Virtualhome: Simulating household activities via programs". In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018 pp. 8494- 8502.
[9] Mohit Shridhar et al. "Alfred: A benchmark for interpreting grounded instructions for everyday tasks". In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020. pp. 10740-10749.
[10] Alexey Dosovitskiy al. "CARLA: An Open Urban Driving Simulator". In: Proceedings of the Ist Conference on Robot Learning (CoRL). 2017, pp. 1-16.
10

---

[1]] Wenlong Huang et al. "Language models as zero-shot planners: Extracting actionable knowl- edge for embodied agents". In: Proceedings of the 39th International Conference on Machine Learning (ICML). PMLR. 2022, pp. 9118-9147.
[12] Chan Hee Song al. "LLmmppaamnn large language models". In: Proceedings of the 19th International Conference on Computer Vision (ICCV). 2023, pp. 2998 99989
Few-shot grounded planning for embodied agents with
[13] Wenhao Yu et al. "Language to rewards for robotic skill synthesis". In: arXiv preprint (2023).
arXiv:2306.0800
[14] Ademi Adeniji et al. "Language reward modulation for pretraining reinforcement learning".
In: arXiv preprint arXiv:2308. 12270 (2023).
[15] Wenlong Huang et al. "Inner monologue: Embodied reasoning through planning with language
models". In: arXiv preprint arXiv:2207. .56608 (2022).
[16] Haotian Sun et al. "Adaplanner: Adaptive planning from feedback with language models". In: Proceedings of the 37th Advances in Neural Information Processing Systems (NeurIPS) 36 (2024).
[17] Boyi Li et al. "Interactive task planning with language models". In: arXiv preprint arXiv:2310.1066 (2023).
[18] Zhenyu Wu al. "Embodied task planning with large language models", In: arXiv preprint arXiv:2307.011 30.008888888888 (2023).
[19] Jingkang Yang et al. "Octopus: Embodied vasion-language programmer from environmental feedback". In: arXiv preprint arXiv:2310.0855 (2023).
[20] Weijia Shi et al. "Replug: Retrieval-augmeentd black-box language models". In: arXiv preprint arXiv:2301. 12652 (2023).
[21] Michael Günther et al. "Jina embeddings 2: 8192-token general- purppse text embeddings for long documents". In: arXiv preprint arXiv:2310. /992. (2023).
[22] John X Morris al. "Text embeddings reveal (almost) as much as text". In: arXiv preprint arXiv:2310. 06816 (2023).
[23] Jingcheng Deng et al. "Regavae: A retrieval-augmentee gaussian mixture variational auto- encoder for language modeling". In: arXiv preprint arXiv:2310. 10567 (2023).
[24] Wenhao Yu et al. "Chain-of-note: Enhancing robustness in reurieeal-augmented language models". In: arXiv preprint arXiv:231 1.092 (2023).
[25] Peng Xu et al. "Retrieval meets long context large language models". In: arXiv preprint
arXiv:2310.0302. (2023).
[26] Zhengbao Jiang et al. "Active retrieval augmented generation". In: arXiv preprint arXiv:2305.0643 (2023).
[27] Akari Asai et al. "Self-rag: Learning to retrieve, generate, and critique through self-reflection".
In: arXiv preprint arXiv:2310.11511 (2023).
[28] Özgür Şimşek and Andrew G Barto. "An intrinsic reward mechanism for efficient exploration".
In: Proceedings of the 23rd International Conference on Machine Learning (ICML). 2006, pp. 833-840.
[29] Yuri Burda et al. "Exploration by random network distillation". In: arXiv preprint a8Xiv: /2894 (2018).
[30] Taewook Nam et al. based meta-reiinforeemet learning". In: arXiv preprint arXiv:2204.11828 (2022).
[31] Karl Pertsch, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors". In: Proceedings of the 4th Conference on Robot Learning (CoRL). PMLR.
2021, pp. 188-204 204.
[32] Karl Pertsch et al. "Guided reinforeement learning with learned skills". In: arXiv preprint arXiv:2107. 10253 (2021).
[33] Evan Z Liu et al. "Decoupling exploration and exploitation for meta-reinforcemmnt learning without sacrifices". _ In: Proceedings of the 38th International Conference on Machine Learning (ICML). PMLR. 2021, pp. 6925-6935.
[34] Ling Liu, Calton Pu, and Wei Tang. "Continual queries for internet scale event-driven infor- mation delivery". In: IEEE Transactions on Knowledge and Data Engineering 11.4 (1999), pp. 610-628.
11

---

[35] Yousuke Watanabe and Hiroyuki Kitagawa. "Query result caching for multiple event-driven continuous queries". In: Information systems 35.1 (2010), pp. 94-110.
[36]
Gemma Team et al. "Gemma: Open models based on gemini research and technology". In: arXiv preprint arXiv:2403.08880
(2024).
[37] Aeta "LJama 3 Model Card". In: (2024). URL: sttpae///gthuub..ccmmaaaaalaa
11ma33/bloo/mmain/MMDLLL wwwaamaiiin/MMMOOMM CARD. md
1lama3/blob) wwtlooo/mmii//i/
12

---

A
Broader impact
Our work does not involve activities associated with negative societal impacts, such as disseminating disinformation, creating fake profiles, conducting surveillance. Therefore, we do not expect any negative societal impacts from our research.
B
Environment settings
B.1
VirtualHome
We employ VirtualHome a complex simulation environment designed for embodied Al research, which offers wide range of interactive household activities. This environment requires an agent to perform tasks by interacting with various objects and following high-level action commands. VirtualHome features 162 different object types (e.g., TV, sofa) and multiple room types (e-g., kitchen, living room) across various indoor scenes, providing complex environment through diverse combinations of rooms. Additionally, to standardize the time duration skill execution, we have imposed the following restrictions the 'walk' skill: walking is only possible to adjacent rooms, and only permitted towards objects that are present in that same room.
Figure A.I: Visualization of VirtualHome.
The embodied agent collects information about objects that come into its view and uses this as observations. Additionally, the agent utilizes seven skills to respond to the given continual instructions: walk object or walk room, grab object, switch object, put object, putin object, open object, and close object. For constructing TEKG, we use the graph based environment implemented in VirtualHome.
For non-stationarity, the environment condition involves a single continual instruction from the set changing every predefined timestep: 4 for high non-stationarity, 6 for medium non-stationarity, and for low non-stationarity. For continual instruction following tasks, we implement 19 continual instructions. Table A. shows the details of instructions.
B.2 ALFRED
(a) ALFRED
(b) CARLA
Figure A.2: Visualization of ALFRED and CARLA.
We utilize ALFRED 9 which provides vision-and- language navigation and rearrangement tasks for embodied AI. This environment requires agent to follow language formatted instructions to
13

---

Table A.1: Continual instructions in VirtualHome environment
Example
no one is watching the TV, turn on.
If you have an apple somewhere, bring it to your desk. If you see book somewhere unorganized. bring it to the sofa. The dishwasher must always be open dry the dishwasher. is good for maintenance if the microwave is always open. Always leave the stove open.
The mug should always be on the coffectable.
To wash dishes, placc the plates in the microwave as shown. If you towels, put them in the washingmachine. your towel isn't stored somewhere else, put it in closet. If your computer stays off, turn it on.
If the cabinet is open, close it.
If someone reads a book and doesn't 't tidy it up, put it back. the stove off. go Put paper on the floor or anywhere else in cabinet. Place all visible mug in the microwave to sterilize them. If the radio is off, turn it on If someone uses plate for washing dishes and leaves somewhere, put it in the dishwasher. your microwave off, turn on.
turn
preheat.
accomplish real-world-like household tasks. ALFRED features 58 different object types (e.g., bread) and 26 receptacle types (e-g.. plate) across 120 various indoor scenes (e.g., kitchen).
The embodied agent gathers information about objects that enter its view and uses this as observations. Moreover, the agent utilizes six skills to respond to the given continual instructions: goto object, grab object, toggle object, put object, open object, and close object.
For soatstttton,rtty the environment condition involves a single continual instruction from the set changing every predefined timestep: 4 for high non-stationarity, for medium son-stationarity. and 8 for low non-stationarity. For continual instruction following tasks, we implement 16 continual instructions. Table shows the details of instructions.
Table A.2: Continual instructions in ALFRED environment
Example
TV is off, turn it on.
If you have an apple somewhere, bring it to your coffeetable. book somewhere unorganized, bring the sofa.
If you
is good for maintenance the microwave is always open. The mug should always be on the coffectable.
To wash dishes, place the plates in the microwave as shown. If your towel isn't 't stored somewhere else, put it in the closet. If your computer stays on, turn it off.
the cabinet is open, close it. If someone reads a book and doesn't 't tidy it up. put it hack. Put paper on the floor or any where else in the cabinet. Place all visible mug in the mierowave sterilize them. your microwave on and spinning, turn off. If somcone uses a plate for washing dishes and leaves somewhere, put it the sink. If you see towels, put them in the sink. If the alarmclock is alone, turn it off
B.3 CARLA
CARLA is an opensource simulator for autonomous driving tasks that provides various environment settings for driving conditions and maps. For our experiment, have modified CARLA to function
14

---

as based environment. The agent can navigate within the environment using three skills: turn right, turn left, and go straight. Additionally, the agent is capable loading and offloading goods. The objective of the agent is to transport goods from designated building to a target building whenever the conditions specified in the given instructions are met.
For non-stationarity, the environment condition involves a single continual instruction from the set changing every predefined timestep: 4 for high non-stationarity, for medium non-stationarity and 8 for low non-stationarity. For continual instruction following tasks, we implement 16 continual instructions such as "If the green building calls, load the goods from the green building to the red building". We utilize set of instructions as continual instructions.
C Implementation details
In this section, we provide the implementation details of our proposed framework ExRAP and cach baseline. Our framework is implemented using Python v3.10 and trained system of an Intel(R) Core (TM) 19- 109800 processor and two NVIDIA RTX A6000 GPUs. Each experiment run takes hours. We employ 4 different methods: ZSP (TT), SayCan (T). ProgPrompt (3), and LLM-Planner [12). Although the format differs, the observed environmental knowledge used remains consistent across all baselines. This includes observations of objects, object positions, object states, room adjacencies, etc. For generating a plan, we utilize the Llama-3 (7] model.
C.1 Baseline
ZSP capitalizes on the abilities LLMs for embodied task planning by translating instructions into skill sequences, thereby enhancing the performance of embodied tasks. ZSP achieves this through the generation of detailed step-by-step prompts derived from examples of similar successful tasks, and then it utilizes the LLM to generate executable plans based on these examples. For implementation, we refer to the opensource
SayCan (T) integrates the affordance function with language models, generating plans that feasible to the context. SayCan achieves this by learning the environment affordance function derived from the LLM with the agent's experiences. Similarly, calculate optimal affordance scores using environmental information and domain-specific knowledge. For observation, we utilize the linearized retrieved knowledge graph the prompt for SayCan as same ExRAP. For implementation, refer to the opensource
ProgPrompt 3 utilizes the code-style policy for generating plans for the embodied environment. As it passes the available primitive actions such as walk object, grab object, etc., in the form of import statements in Python, an available object list, and example tasks to L.M,, the L.LM produces some plans to succeed in the task. ProgPrompt can also execute conditional action by assert-elst statements in the codes which is the output of the LLM. For experiments, we provide the code examples for various tasks in VirtualHome and ALFRED. For implementation, we refer to the opensource
LLM-PPlnneer 112 leverages the LLMs using the demonstrations with retrieved in-context examples, empowering embodied agents to perform complex tasks in environments with observed information, guided by natural language instructions. For our experiments, the L.LM-Planner performs inference at each timestep, gathering observed environmental knowledge through natural language. When skill execution fails, captures the error and infers an alternative action based on the error the next timestep. For observation, we utilize the detection results from the environment and incorporate them For implementation, we refer to the opensource
as prompts for the
The hyperparameter settings for baselines are summarized in Table A.3
hhtpp://githubb.com wtwgghhhwh..oo//////////////gggg planner
bbtpss//gttubbcoo///uUUXX Lro.LLLGro///LL--PPP Planner/
15

---

Table A.3: Hyperparameter settings for baselines
Hyperparameters
Value
Llama-3-8B (Default)
LLM
Gemma- Llama-3-70B (Ablation study) 0.33
Temperature
In-context example retriever
dpr-ctx_ennd encoder-single-ng base + BM25
(LL.M-llanner ZSP) 2 (ProgPrompt)
Number of prompts
Maximum new tokens
C.2 ExRAP
In ExRAP, we address multiple continual instruction-folloving tasks by decomposing each instruc- tion into queries and task executions based environmental context memory. To manage these tasks effectively, which are executed continuously and simultaneously. we introduce an exploration- integrated task planning scheme. This scheme incorporates ifformation-basee exploration into the LLM-based planning process, enhancing the balance between maintaining the validity the environ- mental context memory and the demands of environment exploration, ultimately boosting overall task performance. Additionally, we implement a temporal consistency refinement scheme in our query evaluation to counteract the inherent decay of knowledge within the memory.
For the update function i TEKG (3), we design an algorithmic refinement function tailored for embodied environments. This refinement function operates under two simple rules. First, only the most recent timestep data retained for quadruple related agents (i.e., where the source entity target entity is an agent). This is because information related to agents in the environment is observable. Second, semantically contradictory information is categorized into two cases: one where object states opposite, such as simultancous quadruples indicating that TV is both off and and another where an object exists in two places at TEKG. Both scenarios result the removal the outdated quadruple.
The hyperparameter settings for the baselines are summarized in Table A.4
Table A.4: Hyperparameter settings for ExRAP
Hyperparameters
Value
Llama-3-88 (Default)
LLM
Gemma-2B, Llama-3-70B (Ablation study)
Temperature
0.33
In-context example retriever
dprrux_eenooderssinllennnngooe BM25
Number of prompts
Number of retrieved quadruples & for 4 Number of iterations of query evaluator dR
12
10
Maximum new tokens
40
Filtering threshold 0 in
0.5
Weights for exploration value w/z in 12 Weights for exploitation value Wy in 12
1.0
0.01
Additional experiment
D.1 Detailed results of non-stationarity and scale of continual instructions
Table A.5 presents performance comparison in terms of SR and PS. The environmental settings consist of two variables: the size of continual instructions and the degree of non-stationarity. As the size of continual instructions increases, the number instructions that need to be addressed also grows, requiring the agent to collect more knowledge and perform more tasks. Furthermore, higher
16

---

Algorithm Detailed implementation of ExRAP framework
1: Continual instructions I
2: Env. context memory Gt, Timestep
3: Queries Q, Executions &
4: Instruction interpreter 1,
query evaluator 3M
5: Exploitation planner vy, Exploration planner
6: 2,E, C (Z)
7: loop
8: // (a) Query evaluation in
9: for in Q do
Lo -
11:
for 1. .... 10 do
G 4 &n (4})
R1 = L q,t, (,t,Crn,,,P((G----
14:
if hold 66 then
15:
Lg.aapend(P(g(GG..
16:
end if
17:
end for B[ G( D (q, t, Git-1, R(glG:-1))
18:
19:
20:
end for
21:
// (b) Exploration-integrate task planning in 12
22:
argmax //w vy (Gr, + wn un (G,
23:
observation 0++1 = EnvStep(2))
24:
G ((GG,,041)) / ++1
25:
end loop
degree of non-stationarity means the environment changes more rapidly, necessitating that the agent focuses more on environmental information to adapt effectively.
As indicated in Table A.5 ExRAP demonstrates increase in SR by 4.73% 27.50 27.50% average compared to the most competitive baseline, the LLM-PPlnner., Furthermore, ExRAP shows an average reduction in PS by 4.84 13.29 compared to the LLM-Planner. Similar the experiments in the main text, ExRAP exhibits widening performance gaps as the complexity of tasks increases: the SR gap grows from 19.12% compared to the LLM--Plnner with small continual instructions. to 19.93 with large continual instructions. Similarly, the PS gap expands from an average of 6.79 to 9.97. Moreover, ExRAP demonstrates robustness embodied environments across varying levels of non-stationarity: the SR gap remains consistent, ranging from 17.77% compared to the LLM-Planner in low non-stationarity environments 17.32% high non-stationarity. The PS gap expands from an average of 6.96 to 9.27.
D.2 Detailed results of ablation study
We compare the performance of ExRAP and EXRAP-TC, which evaluates queries without temporal consistency-based refinement, to understand the impact of this feature. Additionally, ExRAP-LLM operates by directly inputting the instruction "Explore the home" into the LLM, thereby enabling to function as an exploration planner. This approach contrasts with ExRAP-EXP, which evaluates skills using an exploitation-only planner. Here, EXRAP-EXP inputs "Find (query)" for each query that is not yet satisfied, focusing solely achieving specific task objectives without incorporating exploration. Table A. presents the detailed results of the ablation study for Tables and in Section 4.2 of the main text.
17

---

Table A.5: Performance comparison in VirtualHome
Low non-stationarity
Medium non-stationarity
High non-stationarity
Model
SR (t)
PS (L)
SR (1) PS ())
SR (1)
PS ())
Small continual instruction. Num, of continual instructions is
7SP
34.67%:10001
15.68+144 40
33.119:+255%
22.86=241
22. 1.9.0%94.999
24.84=12.500
SayCan
10.77%
16.5842.75
40.58%:1879%
19.76=202
28.11%:1:608
10.13::00.00
ProgPrompt
43. 43.40%
16.00:0.51
43.15%.13.27%
19.99=1.59
30.71%:44..99
21.93::9.50
I...MPlanner
54.28%%226477
19.54:2419
42.61 12.61%:47319
17.13-747
36.94%.499999
20.91
ExRAP
71.78 % 15.69%
12.35
67.77 % 44.6%
12.29:0096
51.67 81114
12.58
Medium continual Instruction. Num. of continual instructions is
7SP
20.59%::4.11
31.03=+466
20 20.06%:+1.999
32.06=4.66
17.28%.4..el 16%
24.08::4.61
SayCan
35.129::44889
21.67=3.8
633613
21.81=4.4
27.33%..4.244
16.18:33.0
ProgPrompt
32.10%%::4.41
18.84+4.08
30.51% +5.311
23.43-1077
27. 27.19%+42.99
18.60=4.22
L..M-Planner
40.97%:27:000
17.61
39.89 39.899:14.525
15.93=2.13
34.60%
14.94
ExRAP
61.12% 1:70%%
11.75::2.99
55.14% % 6.59%
11.33++:122
50.12% 45570%
8.61::2.25
Large continual instruction. Num. continual instructions
7SP
14.69%:21.66
26.72:8819
7.43%
59. 16126.64
10.32%:14 38%
57.07=8.68
SayCan
32.50%%:49455
19.87=7.19
23. 13.044%:+12..66
17.443
24.20%:144366
32. 32.42:444443
ProgPrompt
29.34%.+3.16%
20.89+5.00
21.20%%+74445
29.00=1.27
21.48%%:t4448
40.05-26.19
L..M-Planner
41. 4..109%:5.23
18.97:0035
31.82%::14 30%
17.63_2.34
26.39%
20.19 20.19
ExRAP
56.75% 18.19%
11,21:22.3
53.86 % 8.59%
8.76:20.4
48,50 % 77.4%%
6,90=10.28
Table A.6: Detailed performance for ablation study
Low non-stationarity
Medium non-stationarity
High non-stationarity
Model
SR (1)
PS (J)
SR (1)
PS (
SR (1)
PS (L)
Small continual instruction. Num. of continual instructions 3
EXRAP-LLM
41.98%-42.9%
19. 19.55
36.39%+11 49%
21.45:1.66
33. 33.60%++12.555
11.51=2.49
ExRAP-MA
46, 46.22%%.41.4%
16.24+287
42.74%:49.52%
18.41#297
16..00%:4119999 11.9%%
13.02=4.07
EXRAP-TC
49.52%
21.12+371
47.24%
23.37
40.41%
8.53-5565
ExRAP
71.78% 1..9%
12.35=1.63
67.77% 14.56%
12.29
51.67 % 1.14
12.58=2.72
Medium continual instruction. Num. continual instructions
EXRAP-LLM
31.23%++245%%
15.17+
29.42% 63%
10.78=
21.81% 1466
14.65=2.40
ExRAP-EXP
43.33%.:11.288.
13.07:4475
40.68%:49..7%
13.03=301
29.36% 35%
12. 12.16-386
EXRAP-TC
17.25%%
15.301697
43.92%%
15.47
27.91%+1446445
9.6745332
EXRAP
61.13%
11.66=3393
55.14% 56.5%%
11.33-+1.22
49.73%
8.74
Large continual instruction. Num. continual Instructions
EXRAP-LLM
40.88%
16.4243.61
12.11%
12.27=3.32
21.51%
19.68=3.97
ExRAP-MA
43.87%
13.31++304
32. 111 10.70%
10.98:2245
23.46% 38%
11.76=3.77
EXRAP-TC
46.21%%.9.00
11.21=2.46
11.077%11.69%
9.60
11.40=
13.83=4.18
ExRAP
66.75% +8.19%
11.21=2263
53,86 % 1.859%
8.76=00.4
44,50 % -7.4%%
6.90
E
Anaylsis
E.1
Analysis of refinement temporal consistency
In Table A.7 we show examples of refined query responses. Although the input for subsequent query responses remains the same, the current timestep varies. Naturally, as timestep progresses, information decay should occur, and the entropy of the query response is expected to increase. However, the examples show an unexpected reduction in entropy from 0.24 to 0.15, leading to the removal of these outdated responses. This approach allows us to effectively model information decay, thereby improving the quality of query responses.
A.7
E.2 Analysis of computation overhead of ExRAP
ExRAP employs sentence embedding techniques, such DPR and BM25, for retrieving knowledge graphs and demonstrations to enhance exploitation value. shown the table below, the average retrieval and LLM inference time is approximately 14 times faster compared to inferring actions
18

---

Table A.7: Examples for temporal consistency- based refinement
Query: Tv is off
Timesteps: 42
Environmental knowledge: (TV, inside, livingroom, 7), (TV, is. off, 7). (TV, is, on, 9), (TV, tvstand, 7)
(TV, is, off, 27), (kitchen, adjacent, hedroom, 1), (kitchen, adjacent, bathroom, 1)
Query response: Yes: 25%, No: 75%
Query: Tv is off
Timesteps: 43
Environmental knowledge: (TV, inside, livingroom, 7), (TV, is, off, 7), (TV, is, on, 9), (TV, on, tystand, 7)
(TV, is, off, 27), (TV, is, off, 33), (kitchen, adjacent, bathroom, 1)
Query response: Yes: 11 %, No: 89% - Removed
Exploration value
non-stationarity High non-stationarity
on 5
46
Timestep since observation
Figure A.3: Examples for exploration value w.r.t non-stationarity; we noted that the increase in explo- ration is more larger in environments with higher non-station stationarity, leading enhanced exploration. This, in turn, resulted in a more frequent drop in exploration value.
through an LLM without retrieval, using RTX A6000 GPU and an 19-10980X processor. By selecting relevant quadruples and demonstrations based on the query and instructions, ExRAP effec- tively reduces the context length, maintaining efficient LLM inference times. In our experiments, retrieve only 3 demonstrations and 12 quadruples to generate prompts for each continual instruction, regardless of the size of the knowledge graph or dataset.
Table A.8: Comparison of Retrieval and LLM Inference Times
Retrieval (Quadruple)
Retrieval (Demos.)
LLM Inference
LLM Inference (w/o retrieval)
Time (sec)
0.021 sec
0.024 see
2.203 sec
31.243 sec
E.3
Analysis of exploration dynamics
Figures A.3 illustrate the changes in exploration value for single query during the execution of continual instructions. We observed that the exploration value increased steadily over time and decreased rapidly once information relevant to the query was collected. Specifically, we noted that the increase in exploration was more larger in environments with higher non-stationarity. leading to enhanced exploration. This, in turn, resulted in a more frequent drop in exploration value. As the reviewer suggested, investigating the exploration dynamics is interesting analysis that demonstrates how ExRAP improves performance.
E.4 Analysis of behavior ExRAP
To analyze the behavior patterns of ExRAP, we map generated plans to heuristic strategies such as Greedy, and Multiple Instruction First, and Max Staleness First. In our work, these heuristics are utilized only for analysis purposes, showing how FxRAP able achieve superior performance and how its exploration-integraaee planning policy behaves differently in specific situations.
19

---

Table A.9: Examples for ExRAP behavior as Greedy heuristic
Case
Continual instructions:
1. If you have apple somewhere, bring your desk.
2. If no one is watching the TV, turn it on.
3. your towel isn't stored somewhere clse. put it in the closet.
Timesteps: 10
Environmental knowledge: (TV, inside, livingroom, 7), (desk, inside, bedroom, 3), (upple, inside, kitchen, 9),
(apple, on, kitchencounter, 9), (kitchen, adjacent, hedroom, 1), (kitchen, adjacent, bathroom, 1), (lowel, inside, hathroom)...
Query response for instruction I: Yes: 93%, No: 7%
Query response for instruction 2: Yes: 23%. No: 77%
Query response for instruction 3: Yes: 83%, No: 17%
Method
Plan (Step-wise inference)
EXRAP
walk kitchen, walk apple, grah apple, walk hedmom,, walk desk, put apple desk, ...
Greody
walk kitchen, walk apple, grab apple, walk bedroom, walk desk, put apple desk, ...
Multiple Instruction First
walk kitchen, walk apple, grab apple, walk bathroom, walk towel ... walk bedroom, walk livingroom, walk tv, walk bedroom, walk kitchen,
Max Staleness First
Greedy. When the entropy values of the queries are generally low and there are few executions required to complete, ExRAP operates in a manner similar to a greedy heuristic with respect to a single instruction. For example, in Table A.9 when there exist few executions required to complete and the entropy of the query responses is generally low, EXRAP operates similarly to a Greedy heuristic, executing instruction 1 independently. However, consistently applying this heuristic in all scenarios results in executing an instruction-wise plan, similar to the baselines.
Table A010: Examples for ExRAP behavior as Multiple Instruction First heuristic
Case
Continual instructions:
1. If you have apple somewhere, bring it to your desk,
2. If no one is watching the TV, tum it on.
3. If your towel isn't stored somewhere else, put in the closet.
Timesteps: 34
Environmental knowledge: (TV, inside, livingroom, 2), (desk, inside, bedroom, 4), (apple, inside, kitchen. 5) (lowel, inside, closet, II)
(apple, on, kitchencounter, 3), (kitchen, adjacent, hedroom, 4), (kitchen, adjacent, buthroom, 7), (tv, is, off, 8) ...
Query response for instruction Yes: 45% No: 55%
Query response for instruction 2: Yes: 57% No: 43%
Query response for instruction Yes: 48%, No: 52%
Method
Plan (Step wise inference)
ExRAP
walk kitchencounter, walk bedroom, walk desk, walk livingroom, walk tv
Greody
walk bedroom, walk livingroom, walk tv, switch tv.
Multiple Instruction First
walk kitchencounter. walk bedroom, walk desk, walk livingroom, walk tv ...
Max Staleness First
walk bathroom, walk bathroomcounter, walk kitchen, walk kitchencounter...
Multiple Instruction First. The Multiple Instructions First heuristic selects the skill based on the number of related instructions rather than focusing on a specific instruction. the entropy values of various queries are generally high and there are many executions required to complete, EXRAP prioritizes skills where multiple instructions are concentrated. For example, in Table T.b1eA..10 A. when the overall entropy of query responses is generally high, suggesting need for efficient planning to explore multiple queries, ExRAP operates similarly to a Multiple Instructions First heuristic. However, consistently applying this heuristic in all cases can lead to an integrated plan, but result in failing to properly conclude tasks.
Max Staleness First. The Max Staleness First heuristic prioritizes selecting skills based on the age of information of related queries. Similarly, if the entropy value of a particular query very high, ExRAP prioritizes exploring that specific query. This typically aligns with Max Staleness First policy because our query evaluator operates with temporal consistency. For example, in Table A. when the entropy of query responses for instruction 2 is particularly high, indicating need to explore such queries, ExRAP operates similarly to a Max Staleness First heuristic. While this heuristic focuses on specific queries and may reduce efficiency, it prevents the occurrence of instructions being left unexecuted, thereby avoiding starvation of certain instructions. This strategy essentially requires precise measurement of information decay to be effectively implemented. In ExRAP, we
20

---

Table A.11: Examples for ExRAP behavior Max Staleness First heuristic
Continual instructions:
1. you have apple somewhere, bring it to desk.
2. no one is watching the TV, turn it on.
3. your
stored
else, put
closet.
Timesteps: 92
Query response for instruction Yes: 92%, No: 8%
Query response instruction Yes: 87%, No: 13%
Query response for instruction 3: Yes: 50%, No: 50%
Environmental knowledge: (TV, inside, livingroom, 2). (desk, inside, bedroom, 4), (apple, inside, kitchen, 89), (tv. is, off, 91)
(apple, on, sink, 89), (kitchen, adjacent, bedroom, (kitchen, adjacent, bathroom, 7), (towel, inside, bathroomcounter, 35)
Method
Plan (Step-wise inference)
ExRAP
walk kitchen, walk bathroom, walk bathroomcounter, walk faucet,
Greedy
walk livingroom, walk tv, switch IV.
Multiple Instruction
walk livingroom, walk sofa, grab apple, switch ty,
Max Staleness First
walk kitchen, walk bathroom, walk bathroomeounter, walk fancet,
perform temporal seassistecc-baseed refinement to enhance the quality of query responses, leading to improved performance.
As discussed so far, by appropriately utilizing different strategies including those similar to the three heuristics depending on given situations, our ExRAP dynamically adjusts its policy in the response to changes in the environment for multiple tasks of continual instruction following.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction should clearly state the claims made, including the contributions made the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We write separate limitation paragraph in Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Yes)
Justification:
We provide implementation details in Appendix, and also release source code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open Access to Data and Code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
We release code for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https http//udddsss// Set///oodSSSeisseeeeeeeee 7/nips.com http:p/iii//giieeeeeeeeeeaaasiss
osssiiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide implementation details, including environment and hyperparame- ters, in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in the appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the results are accompanied by the confidence interval.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: The paper indicate the type of compute workers CPU or GPU, and provide the amount of compute required for each of the individual experimental runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics atttps:/neeuris.... http://www....eooooooo////ssssaa
Answer: [Yes]
Justification:
We read NeurlPS Code Ethics. The paper preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there no societal impact on the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for the responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make the best faith effort.
12. Lieenses for Existing Assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the basclines code, we write footnote for url.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "TtzyTwTHgl", "reviewer": "Reviewer_K6e1", "summary": "The paper considers a problem of continual instruction following where an instruction contains multiple query-execution pairs to be performed.\nFor this, the paper proposes ExRAP comprised of two components: query evaluation and exploration-integrated task planning.\nQuery evaluation incrementally updates the temporal embodied knowledge graph and selects a set of the most relevant executions.\nThe exploration-integrated task planning part uses both exploration and exploitation by weight-summing the predicted values of predefined skills whose weights are hyperparameters.\nThe performance in the experiment shows noticeable margins compared to the baseline models.", "questions": "See weaknesses above.", "limitations": "The authors do not address the potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally written well and easy to follow.\n- The paper tackles an important and challenging problem of continual instruction following.\n- Multiple benchmarks are explored for the proposed problem setup, implying generalizability of the setup, yet posing some scale issue (see weaknesses).", "weaknesses": "- While agreeing that addressing an instruction that contains multiple tasks, it is unclear why it should be in the explicit form of query(condition)-execution pairs. This is not well-justified. In addition, can't it be implicit?\n- Regarding the first question, it looks like the proposed approach exploits the query-execution format of the proposed task setup (*e.g.*, query evaluator).\n- The scale of the continual instruction following dataset is a bit limited. All the 3 datasets use less than 20 continual instructions, which may potentially raise a generalizability concern.\n- High errors are observed across many tables in the experiments.\n  - Why is this the case? Can this be related to the small scale of the dataset?\n  - One argued contribution is a good performance of the proposed approach, but due to the high errors, it is unclear whether the argued outperformance is valid.\n- The paper introduces many hyperparameters but it is unclear how much the proposed approach is sensitive to the choice of them.\n- How to obtain the knowledge graph? is it obtained as GT values or predicted by some learned perception modules?\n- In L184, why do the authors posit $H(R(q|G_{t-1})) > H(P(q|G_{t-1}))$? Any intuition behind this?\n\n- Considering that the term \"continual\" also refers to learning some new tasks sequentially (*e.g.*, RL [a], IL [b], or even EIF [c]). Discussion about this aspect seems needed to clarify the usage of \"continual\" in this paper.\n\nReferences\\\n[a] Xie and Finn. Lifelong robotic reinforcement learning by retaining experiences. CoLLAs, 2022.\\\n[b] Gao et al.  Cril: Continual robot imitation learning via generative and prediction model. IROS, 2021.\\\n[c] Kim et al. Online continual learning for interactive instruction following agents. ICLR, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and insights.\n\n**W1**. Should the continual instructions be explicit?\n\nThe continual instructions are not always in the explicit form of query-execution pairs; they can also be implicit, such as \"Always leave the stove open\" in Appendix A.1. In that case, they can be interpreted and executed conditionally, \"Query: stove is closed, Execution: open the stove\"  by an instruction interpreter. Additionally, In our main text, within section 4.1 on Instruction type (Lines 276-284), we demonstrated that tasks could still be performed with minimal performance degradation even when multiple continual instructions are summarized or when objects are ambiguous.\n\n**W2** Exploitation of the query-execution format.\n\nWe utilize an instruction interpreter to transform continual instructions, in which each instruction is either implicit or explicit, into a query-execution format. To address various format of continual instructions, we use LLM as the instruction interpreter.\nIn the fields of real-time databases, event-based query processing, and monitoring system applications, it is common to separate user requests into query and execution components [1, 2].\nInspired by these processing methods, we proposed an ExRAP framework that uses the query-execution structure for continual instructions.\n\n[1] Shaikh, Salman Ahmed, et al. \"Smart query execution for event-driven stream processing.\" 2016 IEEE Second International Conference on Multimedia Big Data (BigMM). IEEE, 2016.\n\n[2] Liu, Ling, Calton Pu, and Wei Tang. \"Continual queries for internet-scale event-driven information delivery.\" IEEE Transactions on Knowledge and Data Engineering 11.4 (1999): 610-628.\n\n**W3** The scale of the continual instructions.\n\nFor evaluation, we construct the continual instructions by sampling 3, 5, or 7 instructions from a set of 19 continual instructions.\nChoosing 5 of these 19 possible continual instructions results in 11,628 unique combinations.\nWe also evaluate the types of instructions to demonstrate the generalization performance across different scenarios, ensuring that our approach can effectively handle a diverse range of instructions. \n\n**W4** High errors are observed across many tables in the experiments.\n\nThe time-varying features of the environment and the large number of combinations of continual instructions contribute to a large variance in performance.\nDespite this, the experimental results remain valid. As shown in Table 1 of the main text, the confidence intervals for the experiments in the ALFRED and CARLA environments do not overlap with those of the baselines, indicating that our results are statistically significant.\nTo minimize experimental error, we will add to increase the random seeds from 5 to 10 used in the experiments. \nThe below table shows the performance of the ExRAP and other baselines with 10 random seeds in medium non-stationarity, VirtualHome.\nAs the error bars decrease sufficiently and the overlap with the baselines disappears, we can observe that the results become statistically significant.\n\n| Model         | SR ($\\uparrow$)           | PS ($\\downarrow$)      |\n|----|-----|------|\n| ZSP           | 20.06% ± 1.93%            | 32.06 ± 4.66           |\n| SayCan        | 33.69% ± 5.36%            | 21.81 ± 4.14           |\n| ProgPrompt    | 30.51% ± 5.31%            | 23.43 ± 1.07           |\n| LLM-Planner   | 39.89% ± 4.52%            | 15.93 ± 2.13           |\n| ExRAP     | 55.14% ± 6.59%            | 11.33 ± 1.92           |\n\n**W5** Hyperparameters sensitivity.\n\nIn our experiments, the hyperparameters that impact performance are the exploration value coefficient $w_R$ and the exploitation value coefficient $w_T$. \nThe table below exhibits robust performance across a wide range of hyperparameter settings.\nWhen the exploitation value coefficient is high, exploration may be reduced, leading to a slight decrease in performance.\nHowever, the advantage of integrated planning across multiple instructions still results in comparable performance.\nWhen the exploration value coefficient is high, an excessive focus on gathering information can also lead to a slight decrease in performance.\nWe will add these results to the Appendix for the final version.\n\n| $w_R: w_T$ | SR ($\\uparrow$) | PS ($\\downarrow$) |\n|------------|-----------------|-------------------|\n| 1000:1     | 49.70%          | 14.67             |\n| 100:1      | 55.14%          | 11.33             |\n| 10:1       | 51.33%          | 13.98             |\n| 1:1        | 42.62%          | 16.66             |\n\n**W6** How to obtain the knowledge graph? is it obtained as GT values or predicted by some learned perception modules?\n\nPlease see general response, Q1\n\n**W7** The intuition of $H(R(q|G_{t-1})) > H(P(q|G_{t-1}))$.\n\nThis is based on the intuition that the uncertainty of information is increased over time in time-varying environments. \nFor instance, if the location of an object is observed at step $t-1$, the uncertainty about the object's location may stay the same (not moved) or increase by step $t$ in a non-stationary environment. \nConsequently, the uncertainty associated with the knowledge graph $G_{t-1}$ increases monotonically by the time it reaches step $t$. Therefore, when evaluating queries based on this knowledge graph, the uncertainty in the query response also increases. This approach is the same as common practices in fields such as sensor databases and monitoring systems, where dealing with data uncertainty is crucial [3, 4].\n\nThe formula should be corrected to $ H(R(q|G_{t-1})) \\geq H(P(q|G_{t-1}))$. Thank you for detailed review.\n\n[3] Prasad Sistla, A., et al. \"Querying the uncertain position of moving objects.\" Temporal databases: research and practice (1998): 310-337.\n\n[4] Cheng, Reynold, and Sunil Prabhakar. \"Managing uncertainty in sensor database.\" ACM SIGMOD Record 32.4 (2003): 41-46.\n\n**W8** Considering the meaning of the word \"continual.\"\n\nPlease see general response, Q2", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70BIMEv3fk", "reviewer": "Reviewer_Jxmz", "summary": "This paper presents the Exploratory Retrieval-Augmented Planning (ExRAP) framework to address the challenge of continual instruction following in non-stationary embodied environments. ExRAP enhances the reasoning capabilities of Large Language Models (LLMs) by efficiently exploring the environment and maintaining an environmental context memory to ground the task planning process.\n\nThe paper's main contributions are:\n\n1.\tA novel ExRAP framework integrating LLMs, memory-augmented reasoning, and information-based exploration for continual instruction following.\n2.\tTemporal consistency refinement and information-based exploration schemes tailored for ExRAP's integrated planning approach.", "questions": "If authors can address points 2 and 3 from weaknesses, it will help in understanding whether there is a distinction between concurrent or multi-task instruction following and truly continual instruction following.", "limitations": "Authors have addressed some of the limitations of the work in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The memory-augmented query evaluation and exploration-integrated planning framework provide a principled way to bridge the gap between high-level language understanding and low-level embodied decision-making.\n\nThe temporal consistency refinement scheme is a novel contribution in embodied contexts that addresses the important issue of information decay in memory-based solutions over time. Using a temporal embodied knowledge graph (TEKG) as an environmental context memory enables the agent to efficiently assess the satisfaction of instruction conditions without constantly revisiting the environment.", "weaknesses": "1. ExRAP's performance still somewhat depends on the underlying LLM's reasoning abilities. While the ablation study shows robustness to smaller LLMs, it's unclear how the framework would perform with much weaker language models or in domains where the LLM's knowledge is limited.\n\n2. The environments' non-stationary aspects are limited to a single instruction changing at fixed intervals. This does not fully capture the complexity and unpredictability of real-world non-stationary environments, where goals, conditions, and constraints can evolve in more intricate ways. \n\n3. The instructions used in the experiments seem to be largely independent of each other, without any significant long-term dependencies or relationships between tasks. An example from Table A.1 is: \"If no one is watching the TV, turn it on.\" If you have an apple somewhere, bring it to your desk. \"If you see a book somewhere unorganized, bring it to the sofa.\" These instructions in the table appear to be independent tasks without any clear long-term dependencies or relationships between them. Each instruction can be completed independently of the others.\n\nThat being said, the ExRAP framework does take important steps by incorporating memory-augmented reasoning and exploration-integrated planning. These components provide a way for handling evolving instructions and non-stationary environments. The authors' claims may be somewhat overextended given the current experimental setup, but the work still makes valuable contributions to embodied instruction following.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and insights.\n\n**W1** ExRAP's performance still somewhat depends on the underlying LLM's reasoning abilities. While the ablation study shows robustness to smaller LLMs, it's unclear how the framework would perform with much weaker language models or in domains where the LLM's knowledge is limited.\n\nWhen LLM performance degrades to a level where knowledge-based reasoning becomes unfeasible, a decline in ExRAP's performance is inevitable. \nHowever, as the reviewer commented, ExRAP demonstrates robust performance even in smaller LLMs, as shown in the ablation study (Section 4.2). \nIf implementation with much weaker language models or in domains where the LLM's knowledge is limited is required, techniques such as imitation learning [1] or knowledge distillation [2] could be utilized to address these challenges.\n\n[1] Das, Abhishek, et al. \"Embodied question answering.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[2] Choi, Wonje, et al. \"Embodied CoT Distillation From LLM To Off-the-shelf Agents.\" Forty-first International Conference on Machine Learning.\n\n**W2** The environments' non-stationary aspects are limited to a single instruction changing at fixed intervals. This does not fully capture the complexity and unpredictability of real-world non-stationary environments, where goals, conditions, and constraints can evolve in more intricate ways.\n\nAt every change interval, the position or state of some objects related to an instruction is changed randomly. While the overall environment changes at fixed intervals, the specific objects that change at each interval are variable. \nThis means that for each single continual instruction, the changes do not occur at fixed intervals. We will add these details comprehensively to the Appendix to clarify this aspect. Thank you for your comment.\n\n**W3** The instructions used in the experiments seem to be largely independent of each other, without any significant long-term dependencies or relationships between tasks. An example from Table A.1 is: \"If no one is watching the TV, turn it on.\" If you have an apple somewhere, bring it to your desk. \"If you see a book somewhere unorganized, bring it to the sofa.\" These instructions in the table appear to be independent tasks without any clear long-term dependencies or relationships between them. Each instruction can be completed independently of the others.\n\nWe propose a framework that efficiently integrates and executes shared subtasks among continual instructions, even when each instruction appears to exist independently. \nFor example, while moving to check if no one is watching TV, ExRAP can simultaneously verify the location of a book, or pick up a nearby apple and place it on the table while en route to turn on the stove.\n\nIn response to the reviewer's comments, we conduct experiments on scenarios where query conditions explicitly overlap, or where the execution of one continual instruction explicitly impacts other instructions, such as \"When computer is off, the mug should always be on the coffeetable, If your computer stays on, turn it off\".\n\n| Model        | SR ($\\uparrow$) | PS ($\\downarrow$) |\n|--------------|-----------------|-------------------|\n| LLM-Planner  | 40.11%          | 15.80             |\n| ExRAP        | 51.36%          | 12.59             |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbSEMkE65N", "reviewer": "Reviewer_kQXj", "summary": "The paper introduces an Exploratory Retrieval-Augmented Planning (ExRAP) framework, that tackles the problem of embodied planning in dynamic environments. The study focuses on continual instruction following, which is when the task consists of multiple conditional subtasks. The execution of each subtask is dependent on the environment state. This requires continual exploration to keep up-to-date information and integrated planning abilities for efficiency. \n\nExRAP combines Large Language Models (LLMs) with environmental context memory represented as a temporal embodied knowledge graph (TEKG), that captures the states of objects in the scene and their pairwise relationships. The TEKG is updated at each step using the most recent observation and an update function $\\mu$. Each subtask is translated into a pair of a query about the scene and a corresponding execution command with an instruction interpreter $\\Phi_l$. Next, the memory-augmented query evaluator $\\Phi_{M}}$ estimates the likelihood of the query being satisfied, using quadruples retrieved by $\\Phi_R$ as environmental context information for the LLM. Authors enforce a temporal-consistency constraint on the evaluator, resulting in a continual steady decrease of the entropy of the answer. The execution commands corresponding to the satisfied queries are then passed to the execution part of the model. This part consists of exploitation and exploration planners. The former assesses the effectiveness of the skill in accomplishing the execution task. The latter one does the same with regard to reducing the uncertainty of the query evaluator $\\Phi_M$. The performed skill is chosen as the one providing the maximum weighted sum of the two scores, balancing exploration and exploitation.\n\nIn the experiments, the authors measure the Success Rate (SR) of the task completion and the Pending Step (PS), representing the average number of steps taken to complete the task. The evaluation is performed on the VirtualHome, ALFRED, and CARLA environments. Performance in various degrees of stationarity in the environment is assessed, showing the advantage of the method. The performance with respect to the number of instructions is also evaluated. The method seems to excel in integrated planning, effectively solving multiple tasks with fewer steps. Ablation studies demonstrate the importance of temporal consistency and exploration-integrated planning. Intrestingly, the study reveals a limited effect of choosing a larger LLM as a base model.", "questions": "- Are the environment settings used in the in-context learning retrieval dataset the same or different from the ones used for the test?\n- What does the observation $o_t$ consist of for ExRAP? Is it a list of quadruples? How would the setup be adapted for the real-world case?\n- How does the system handles contradicting instructions? Like “If there is a storm outside, turn off the electricity in the house. If the TV says that the storm is over, turn the electricity on.”\n- To me, it is not clear why the temporal consistency works. Would a simple baseline of reducing the temperature of prediction perform as good as the temporal consistency constraint in the paper?\n- It would be interesting to investigate the exploration dynamics of the agent. Does the exploration scores on average decrease steadily throughout the experience or more rapidly in some particular times?", "limitations": "- Runtime overhead not investigated, but mentioned.\n- Applicability to the real world is not addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- To the best of my knowledge, one of the first works to tackle conditional embodied planning in a dynamically changing environment.\n- The results of the paper suggest that the method is clearly excels in producing more effective plans than the baselines, integrating multiple tasks in parallel. The plans are being adapted on the fly from the belief about the environment, and the belief is updated by neccessary exploration. This is a valuable contribution.", "weaknesses": "- The paper shows that the method is quite robust to the size of the base LLM. It could be a strength if there would be some training procedure possible. However, ExRAP relies exclusively on the in-context learning abilities of the LLM, thus the experiment shows that it scales poorly. The paper would benefit from an ablation study on the size of the retrieval dataset, so it can be shown that the method can scale with the context length of the model by increasing the number of examples.\n- The method excels at following multiple simple parallel conditional instructions, but it is not clear if it will be as good at the long-term tasks that require multiple steps to achieve, like the ones on which the LLM-Planner was tested. The advantage of having an adaptable planner may become a disadvantage against the baselines, as it may require more queries to the environment context memory, which will create a large computational overhead. It is unclear whether the method is better than the baselines in integrating multiple complex tasks, each of which requires multiple steps, into an optimal plan. Unfortunately, this is not investigated in the paper.\n- The applicability to the real-world setting is questionable and not investigated. Would the method scale appropriately with the number of objects in the scene? Methods like ConceptGraphs do not show robustness in that case.\n- The termin \"continual embodied instruction following\" is a little misleading, as in the paper, the instructions are passed all at once, and not at random times, which would require replanning. The \"conditional embodied instruction following in the dynamic environment\" could be a better name, though this is not a significant issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and insights.\n\n**W1** The scalablity of ExRAP.\n\nExRAP uses the sentence embedding techniques such as DPR and BM25 for knowledge graph retriever and demonstration retriever for exploitation value.\nAs shown in the table below, the average time taken for retrieval and LLM inference is about 14 times lower than the time it takes to infer actions through an LLM without retrieval, using an RTX A6000 GPU and an i9-10980XE.\n\n| Retrieval (Quadruple) | Retrieval (Demos.) | LLM Inference | LLM Inference (w\\o retrieval) |\n|---|--|---|---|\n|0.021sec|0.024sec|2.203sec|31.243 sec|\n\nBy selecting quadruples and demonstrations relevant to queries and instructions, it effectively reduces the context length and maintains the inference time of the LLM.\nIn experiments, we retrieve only 3 demonstrations and 12 quadruples to generate prompts for each continual instruction, regardless of the size of the knowledge graph or the dataset.\nTo clarify the retrieval models used, we will add these computational overhead results in Appendix.\n\n**W2** The complexity of the environment and computational overhead of ExRAP.\n\nFor evaluation, we use continual instruction that involves multiple steps to achieve. For example, to address the single continual instruction \"If you see a book somewhere unorganized, bring it to the sofa,\" the agent would need to execute the following steps:\n\n0. initialized in livingroom\n1. walk kitchen\n2. walk bathroom\n3. walk kitchen\n4. walk bedroom (find book)\n5. walk book\n6. grab book\n7. walk kitchen\n8. walk livingroom\n9. walk sofa\n10. put book\n\nTo effectively address complex continual instructions that require exploration, ExRAP performs continual instructions in an integrated approach. By doing so, ExRAP maximizes the effectiveness of the agent's actions for continual instructions, as evidenced by a reduction of 3.40 in the path length during experiments and resulting in a 16.45\\% improvement in the success rate in experiments.\n\nIn addition, as responded to in W1, ExRAP utilizes a retrieval-based approach to efficiently manage information. Even as the knowledge graph grows, it maintains the inference time of the LLM and minimizes computational overhead through retreival-based approach.\n\n**W3** The applicability to the real-world setting.\n\nThe environments used in our experiments (VirtualHome, ALFRED, and CARLA) are highly similar to real-world settings and are frequently used in prior works proposing embodied planning or LLM-based frameworks [1, 2, 3]. For example, in VirtualHome, a single scene contains between approximately 20 to 100 objects, and there are 50 types of room structures, each with varying object states and positions.\n\nIn ExRAP, a temporal embodied knowledge graph is used to efficiently manage and retrieve information on changing object states, enhancing adaptability and operational efficiency in dynamic environments.\n\n[1] Song, Chan Hee, et al. \"Llm-planner: Few-shot grounded planning for embodied agents with large language models.\" Proceedings of the IEEE/CVF ICCV. 2023.\n\n[2] Singh, Ishika, et al. \"Progprompt: Generating situated robot task plans using large language models.\" 2023 IEEE ICRA. IEEE, 2023.\n\n[3] Yang, Ruoxuan, et al. \"Driving Style Alignment for LLM-powered Driver Agent.\" arXiv preprint (2024).\n\n**W4** Considering the term \"continual embodied instruction following\".\n\nPlease see general response, Q2\n\n**Q1** Are the environment settings used in the in-context learning retrieval dataset the same or different from the ones used for the test?\n\nThey are different. The demonstrations used in in-context learning and the test environments differ in the structure of rooms and positions of objects in VirtualHome and ALFRED, and the locations of buildings in CARLA.\n\n**Q2** What does the observation $o_t$ consist of for ExRAP? Is it a list of quadruples? How would the setup be adapted for the real-world case?\n\nPlease see general response, Q1\n\n**Q3** How does the system handle contradicting instructions?\n\nIn this paper, we focused on optimizing multiple continual instructions and did not address conflict resolution.  ExRAP can be expanded into a neural-symbolic approach [4] with a temporal embodied knowledge graph, utilizing various research to identify and resolve contradictions [5, 6].\nThe question by the reviewer addresses a critical aspect of the multiple continual instructions and real-world implementation, and it is one of the future works for ExRAP. We include this content in the conclusion section to emphasize its importance.\n\n[4] Liu, Xiaotian et al. \"A planning based neural-symbolic approach for embodied instruction following.\" Interactions 9.8 (2022): 17.\n\n[5] Li, Jierui et al.. \"ContraDoc: Understanding Self-Contradictions in Documents with Large Language Models.\" arXiv preprint (2023).\n\n[6] Wan, Alexander et al. \"What Evidence Do Language Models Find Convincing?.\" arXiv preprint (2024).\n\n**Q4** To me, it is not clear why the temporal consistency works. Would a simple baseline of reducing the temperature of prediction perform as well as the temporal consistency constraint in the paper?\n\nThe problem of maintaining temporal consistency in LLMs is not resolved simply by lowering the temperature setting. As shown in Appendix D.1, LLMs often fail to reflect the increasing uncertainty of information over time, and this issue can occur even when the temperature is set to zero, particularly in small LLMs.\n\nTo address this issue in LLMs, we employ repetitive reasoning and critic to enforce temporal consistency. \nThis approach helps to ensure that the model maintains consistency over time, effectively capturing and reflecting temporal dynamics even in small LLMs.\n\n**Q5** It would be interesting to investigate the exploration dynamics of the agent. Do the exploration scores on average decrease steadily throughout the experience or more rapidly in some particular times?\n\nPlease see the general response Q3, and PDF file", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response**

We thank the reviewers for their valuable feedback. We are encouraged that they found our approach is novel, well-written, easy to follow, adresses the important and challenging problem, and provides valuable contribution,  as well as providing extensive evaluations which demonstrate the effectiveness of our approach in multiple benchmarks. We addressed all weaknesses and questions to resolve the concerns raised. For those areas where the reviewer requested additional information, we conducted further studies and provided detailed explanations to enhance overall understanding.

**Q1**  Consideration the observation of environment. (Reviewer K6e1, Reviewer kQXj)

In alignment with prior works on LLM-based agents [1, 2, 3, 4], the agent collects text-based observations about objects visible within its field of view or through interaction with objects.
It is then simply transformed into quadruples (e.g., "book is on the table" becomes ("book", "on", "table")). ExRAP constructs a temporal embodied knowledge graph using these observations.

However, addressing the reviewer's concern, we conducted experiments under the assumption that the agent collects observations via camera and utilizes an object detection module or Vision-Language Models (VLMs) for text observation, resulting in only partial information being detected when observing the environment.
Below table demonstrates that even with a decrease in detection probability, sufficient performance is maintained.

| Detection Prob. | SR ($\uparrow$) | PS ($\downarrow$) |
|-----------------|-----------------|-------------------|
| 80%             | 50.77%          | 13.52             |
| 90%             | 52.97%          | 12.38             |
| 95%             | 54.71%          | 11.84             |
| 100%            | 55.14%          | 11.33             |

[1] Song, Chan Hee, et al. "Llm-planner: Few-shot grounded planning for embodied agents with large language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Xiang, Jiannan, et al. "Language models meet world models: Embodied experiences enhance language models." Advances in neural information processing systems 36 (2024).

[3] Lin, Bill Yuchen, et al. "On grounded planning for embodied tasks with language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.

[4] Singh, Ishika, et al. "Progprompt: Generating situated robot task plans using large language models." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

**Q2**  Consideration the term "continual embodied instruction following". (Reviewer K6e1, Reviewer kQXj)

As noted in the main text (Lines 100-108), "continual instruction" is defined as multiple instructions that need to be performed continuously and simultaneously. So, the term "continual" as we use it does not refer to continual learning, which learns some new task sequentially. Rather, this usage aligns closely with the concept of continual query [6, 7], which are standing queries that monitor update of interest and return results whenever the update reaches specific thresholds.
Therefore, continual instruction following refers not merely to following conditional instructions but to a broader problem definition. It includes the need to continuously execute instructions in response to the environment and to simultaneously address multiple instructions. 

**Q3**  It would be interesting to investigate the exploration dynamics of the agent. Do the exploration scores on average decrease steadily throughout the experience or more rapidly in some particular times? (Reviwer kQXj)

Figures 1 in the PDF file illustrate the changes in exploration value for a single query during the execution of continual instructions.
We observed that the exploration value increased steadily over time and decreased rapidly once information relevant to the query was collected. Specifically, we noted that the increase in exploration was more larger in environments with higher non-stationarity, leading to enhanced exploration. This, in turn, resulted in a more frequent drop in exploration value. As the reviewer suggested, investigating the exploration dynamics is an interesting analysis that demonstrates how ExRAP improves performance.
We will add this to the Appendix of the final version. Thank you for the suggestion.

**Q4** The potential negative societal impact.

Our work does not involve activities associated with negative societal impacts, such as disseminating disinformation, creating fake profiles, or conducting surveillance. Therefore, we do not expect any negative societal impacts from our research.

We will add this statement explicitly in the final version.

Meta-review: This paper presents the Exploratory Retrieval-Augmented Planning (ExRAP) framework to address the challenge of continual instruction following in non-stationary embodied environments. ExRAP enhances the reasoning capabilities of Large Language Models (LLMs) by efficiently exploring the environment and maintaining an environmental context memory to ground the task planning process. The paper provides insights into integrating LLMs, memory-augmented reasoning, and information-based exploration for continual instruction following. Authors have also replied and satisfactorily replied to the concerns raised by the reviewers during the rebuttal period. Overall a good paper.
Decision: Accept (poster)