SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Visual Language Navigation (VLN) involves navigating in unknown environments using language input and visual feedback. In this context, we focus on the ALFRED dataset and methods based on the AI2-Thor simulator, which serve as the foundation for various approaches to embodied agent tasks. Initial attempts on ALFRED employed end-to-end methods, such as a Seq2Seq model  $(\mathrm{STG}^{+}20)$ , and transformers (ZC21). However, due to the long episodic sequences and limited training data, most top-performing VLN models for ALFRED employ modular setups, incorporating vision and language modules coupled with a higher-level decision-making module  $(\mathrm{MCR}^{+}22, \mathrm{MC22}, \mathrm{IO22})$ . These methods typically rely on hand-crafted scripts with a predefined policy. However, this approach has several downsides. First, domain experts must manually specify policies for each new task type, regardless of the changes involved. Second, fixed-policy approaches do not allow the agent to recover from errors in the vision and language modules, as the predefined policy remains unchanged.
Recently, there has been growing interest in utilizing planning as a high-level reasoning method for embodied agents. Notable work in this area includes OGAMUS  $(\mathrm{LSS}^{+}22)$ , and DANLI  $(\mathrm{ZYP}^{+}22)$ . These methods demonstrate the effectiveness of planning in achieving more transparent decision-making compared to end-to-end methods while requiring less domain engineering than handcrafted approaches. However, these works primarily tested in simpler tasks or rely on handcrafted planners specifically tailored to address a subset of embodied agent problems.
Large Language Models (LLMs) have recently been harnessed to guide embodied actions, offering new problem-solving approaches. Works like Ichter et al.'s SayCan utilize LLMs' likelihoods and robots' affordances, grounding instructions within physical limitations and environmental context  $(\mathrm{iBC}^{+}23)$ . Song et al.'s  $(\mathrm{SWW}^{+}23)$  LLM-Planner and Yao et al.'s  $(\mathrm{YZY}^{+}23)$  ReAct have further demonstrated the potential of LLMs for planning and reasoning for embodied agents. Alternative methods not using LLMs, include Jia et al. and Bhambri et al.'s hierarchical methods utilizing task instructions  $(\mathrm{JLZ}^{+}22, \mathrm{BKMC}22)$ . However, the scalability and robustness of these methods is under study. Our approach, in contrast, is ready to capitalize on the expressivity of objects and actions for superior generalization in new tasks, ideal for rich environments with multiple domain constraints. This challenges of using LLM for symbolic planning is noted by Valmeekam et al.  $(\mathrm{VSM}^{+}23)$  who showed the weakness of prompting-based approaches, and Pallagani et al.  $(\mathrm{PMM}^{+}22)$ , who highlight the massive amount of data required for reaching high accuracy.
8 Experiments and Results
The ALFRED dataset contains a validation dataset which is split into 820 Validation Seen episodes and 821 Validation Unseen episodes. The difference between Seen and Unseen is whether the room
Test Seen	Test Unseen
SR	GC	PLWSR	PLWGC	SR	GC	PLWSR	PLWGC
Seq2Seq	3.98	9.42	2.02	6.27	0.39	7.03	0.08	4.26
ET	38.42	45.44	27.78	34.93	8.57	18.56	4.1	11.46
HLSM	25.11	35.15	10.39	14.17	24.46	34.75	9.67	13.13
FILM	28.83	39.55	11.27	15.59	27.8	38.52	11.32	15.13
LGS-RPA	40.05	48.66	21.28	28.97	35.41	45.24	15.68	22.76
EPA	39.96	44.14	2.56	3.47	36.07	39.54	2.92	3.91
Prompter	53.23	64.43	25.81	30.72	45.72	58.76	20.76	26.22
Table 1: Comparison of our method with other methods on the ALFRED challenge. The challenge declare as winner the method with higher Unseen Success Rate (SR). For all metrics, higher is better. EPA is our approach. Under the line are approaches submitted after the challenge leader board closed.
environment is available in the training set. We use the validation set for the purpose of fine-tuning. The test dataset contains 1533 Test Seen episodes and 1529 Test Unseen episodes. The labels for the test dataset are contained in an online server and are hidden from the users. Four different metrics are used when evaluating ALFRED results. Success Rate (SR) is used to determine whether the goal is achieved for a particular task. Goal-condition Success is used to evaluate the percentage of subgoal-conditions met during an episode. For example, if an objective is to "heat an apple and put it on the counter," then the list of subgoals will include "apple is heated" and "apple on counter." The other two metrics are Path Length Weighted by Success Rate (PLWSR) and Path Length Weighted by Goal Completion (PLWGC), which are SR and GC divided by the length of the episode. For all the metrics, higher is better. We compare our methods, Egocentric Planning Agent (EPA), with other top performers on the current ALFRED leaderboard. Seq2Seq and ET are neural methods that uses end-to-end training  $(\mathrm{STG}^{+}20, \mathrm{ZC}21)$ . HLSM, FILM, LGS-PRA are hybrid approaches that disentangles grounding and modeling  $(\mathrm{BPF}^{+}21, \mathrm{MCR}^{+}22, \mathrm{MC}22, \mathrm{IO}22)$ .
8.1 Results
Table 1 presents a comparison of our outcomes with other top performing techniques on the ALFED dataset. With an unseen test set success rate of  $36.07\%$  and a seen test set rate of  $44.14\%$ , our EPA method won the last edition of the ALFRED challenge. The ALFRED leaderboard ranks based on SR on unseen data, reflecting the agent's capacity to adapt to unexplored environments. At the moment of writing, our achievements rank us second on the ALFRED leaderboard. The only method surpassing ours is Prompt(IO22), which utilizes the same structure as FILM and incorporates search heuristics-based prompting queries on an extensive language model. However, their own findings indicate that the performance enhancements are largely attributed to increasing obstacle size and reducing the reachable distance represented on the 2D map. The other top performer is LGS-PRA, which is also based on the FILM architecture, that improves performance via techniques on landmark detection method and local pose adjustment. In contrast, EPA employs a semantic graph, a representation distinct from the top-down 2D map applied by FILM, HLSM, LGS-PRA and Prompt. Our findings suggest that the performance boost over FILM and similar methods stems from our iterative planning approach, which facilitates recovery from failure scenarios via flexible subgoal ordering. Techniques outlined in LGS-PRA and Prompt should be compatible with our approach as well. The integration of these techniques with EPA an interesting area to investigate for future works. Our method exhibits lower PLWSR and PLWGC due to our initial 500 exploration steps. These steps are convenient for our planner to amass a diverse of of objects clusters and action angles to determine a proper expandable initial state. Although a pure planning-based exploration would eventually provide us with all the information required to solve the current task, it tends to excessively exploit known object clusters before exploring unknown areas. We only save object information when they are within immediate reach which means the agent is acting on immediate observations. This is done for convenience of converting observation into symbolic state and to reduce the plan length generated by the planner. Additional integration with a top-down SLAM map should help decrease the initial exploration steps, consequently enhancing both PWSR and PLWGC.
Failure Modes	Seen	Unseen
Obj not found	19.36	23.54
Collison	9.14	11.34
Interactions fail	7.33	8.98
Obj in closed receptacle	18.21	16.33
Language error	17.92	21.31
Others	28.04	18.5
Table 2: Percentage Error of each types in the validation set.
Ablation	Valid Unseen	Valid Seen
SR	GC	SR	GC
Base Method	40.11	44.14	45.78	51.03
w/ gt segment	45.13	49.72	49.22	53.21
w/ gt depth	41.85	46.03	48.98	51.46
w/ gt language	52.29	56.39	60.55	64.76
w/ all gt	58.33	63.71	68.49	73.35
Table 3: Ablation study with ground truth(w/ gt) of different perceptions
Ablation - parameters	Valid Unseen
SR	GC	PLWSR	PLWGC
Base Method	40.11	44.14	2.04	3.71
w/o init observation	30.75	36.16	17.34	21.93
200 init observation	37.3	42.23	12.41	16.34
w/o object pref	32.89	35.95	3.05	3.81
w/o fault recovery	27.31	26.55	2.41	3.36
w/o symbolic planning	3.47	4.61	2.16	2.98
Table 4: Ablation study with different parameters
8.2 Ground Truth Ablation
Our model, trained on the ALFRED dataset, incorporates neural perception modules to understand potential bottlenecks in visual perception and language, as explored via ablation studies detailed in Table 3. We observed modest enhancements when providing the model with ground truth depth and segmentation, with ground truth language offering the most significant improvement (18.22% and 19.87%). Our egocentric planner, responsible for generating dynamic policies through observation and interaction, proves more resilient to interaction errors compared to policy-template based methods like HLSM and FILM  $(\mathrm{MCR}^{+}22, \mathrm{BPF}^{+}21)$ . Thus, having the correct interpretation of the task description impacts our method more significantly than ground truth perception. We examined common reasons for failed episodes using both the seen and unseen datasets. The most prevalent error (23.54% unseen) is the inability of the agent to locate objects of interest due to failed segmentation or an unreachable angle. Errors stemming from language processing, where the predicted task deviates from the ground truth, are given priority. The results can be seen in Table 2.
8.3 Parameter Ablation
Neural vision modules may struggle with out-of-distribution data, such as ALFRED's unseen environments, leading to potential errors in object segmentation and distance estimation. Unlike hybrid methods like FILM and HLSM, our goal-oriented egocentric planner reassesses the environment after failure, generating new action sequences. For instance, in a task to hold a book under a lamp, a failure due to obscured vision can be addressed by altering the action sequence, as illustrated by the drop in performance to  $27.31\%$  in our ablation study (Table 4).
FILM's semantic search policy, trained to predict object locations using top-down SLAM maps, differs from our symbolic semantic graph representation, which precludes us from adopting FILM's approach. Instead, we hand-crafted two strategies to search areas around objects and receptacles based on common sense knowledge, which improved our Success Rate by  $7.22\%$  with no training. However, our agent only forms a node when it visits a location, necessitating an initial observation phase. Neglecting this step resulted in a  $9.35\%$  performance drop. FILM's SLAM-based approach can observe distant objects, requiring fewer initial observation steps and leading to better PLWSR and PLWGC. Combining a top-down SLAM map with our semantic graph could mitigate the need for our initial observation step and improve these scores.
Our algorithm's frequently re-plans according to new observation which is also essential for real-world applications using online planning system. Using modern domain-independent planners, such
as Fast Downward and Fast Forward (Hof01, Hel06), we've significantly reduced planning time. We tested a blind breadth-first search algorithm but found it impractical due to the time taken. A 10-minute cut-off was introduced for each task, which didn't impact our results due to an average execution time of under 2 minutes. Yet, this constraint severely lowered the success rate of the blind search method, only completing  $3.47\%$  of the validation unseen tasks (Table 4).
8.4 Generalization to Other Tasks
Our approach allows zero-shot generalization to new tasks within the same set of objects and relationships, offering adaptability to new instructions without manually specify new task types and templates. While neural network transfer learning remains challenging in embodied agent tasks, and FILM's template system requires hand-crafted policies for new tasks, our egocentric planning breaks a task into a set of goal conditions, autonomously generating an action sequence. This allows our agent to execute tasks without transfer learning or new task templates. We have chosen environments in which all objects for each task have been successfully manipulated in at least one existing ALFRED task. This selection is made to minimize perception errors. We tested this on five new task types not in the training data, achieving an  $82\%$  success rate, as shown in Table 6, sectionC.2 of the Supplementary Material. We could also adapt to new objects, actions, and constraints, such as time and resource restrictions. However, the current ALFRED simulator limits the introduction of new objects and actions. This flexibility to accommodate new goals is a key advantage of our planning-based approach.
Our contribution poses Embodied Agents as acting in a Partially Observable Markov Decision Process (POMDP), which allows us to tackle a broad class of problems, and further expands on the potential applications of planning-based methods for embodied agents. By adopting this approach, our method offers a more flexible and adaptable solution to address the challenges and limitations associated with fixed-policy and handcrafted methods.
We found that using intentional models with a factored representation in embodied agents is crucial. Whereas a learned world model provides a degree of useful information, their associated policies might under-perform for long horizon high-level decision-making, unless they are trained on massive amounts of interactions  $(\mathrm{SSS}^{+}17)$ . Comparing a world model with classical planning uncovers interesting distinctions. While creating a PDDL-based solution requires an initial investment of time, that time is amortized in contexts that require quality control per type and skill. On the other hand, the cost spent on crowdsourcing or interacting with a physical environment can be significant  $(\mathrm{STG}^{+}20)$ .
Learning classical planning models is relatively straightforward with given labels for objects and actions  $(\mathrm{AFP} + 18, \mathrm{CDVA} + 22)$ . On the other hand, if we aim to learn a world model, we should consider the performance at planning time and end-to-end solutions may not be the ideal approach in this context. When compared to intentional symbolic-based methods, end-to-end strategies tend to compound errors in a more obscure manner. Furthermore, as learned world models rely on blind search or MCTS, in new tasks the model might waste search time in a subset of actions that are irrelevant if action space is large, while symbolic planning can obtain long plans in a few seconds. We also propose addressing challenging planning tasks using simpler planning techniques, providing a foundation for more complex planning paradigms. Our method could be further enhanced by encoding explicit constraints and resources, such as energy.
While other methods might outperform our model in the ALFRED benchmark, they are often limited in scope. For instance, despite its high unseen success rate, the Prompter model serves as a baseline rather than a comprehensive solution for embodied agents. Our contribution provides a method that can adapt to shifts in task distribution without changing the symbolic planning model, while also supports changes in the agent's capabilities by modifying the symbolic planning model.
We acknowledge certain limitations in our approach. While we support safe exploration assuming reversible actions, real scenarios beyond ALFRED could include irreversible actions and failures. Our model is also sensitive to perception errors, making end-to-end methods potentially more robust in specific scenarios. Furthermore, our perception model lacks memory, potentially hindering nuanced
belief tracking. Despite these challenges, our method offers a promising direction for future research in embodied agent applications. In section B, we expand on the discussion of our approach.
10 Conclusion and Future Work
Our work introduces a novel iterative replanning approach that excels in embodied task solving, setting a new baseline for ALFRED. Using off-the-shelf automated planners, we improve task decomposition and fault recovery, offering better adaptability and performance than fixed instruction and end-to-end methods. Moving forward, we aim to refine our location mapping for precise environment understanding, extract more relational data from instructions, and integrate LLMs as exploration heuristics. We plan to tackle NLP ambiguity in multi-goal scenarios by prioritizing and pruning goals based on new information. Moreover, we aim to upgrade our egocentric planner to manage non-deterministic effects, broadening our method's scope beyond ALFRED. This direction, leveraging efficient planning methods for non-deterministic actions, is promising for future research (MBM14).
We want to acknowledge the ServiceNow research for providing the research opportunity and computational infrastructure to conduct this research project. This work was also made possible due to the assistance of Cyril Ibrahim at ServiceNow during the experimental execution phases of our project. We would also like to thank members of Mulab at Queen's University for providing a supporting research environment. Lastly, we want to acknowledge the assistance from organizers of the CVPR Embodied Agent Workshop for their guidance and feedback throughout the competition  $(\mathrm{DBB}^{+}22)$ .
$\left[\mathrm{AFP}^{+}18\right]$  Ankuj Arora, Humbert Fiorino, Damien Pellier, Marc Métivier, and Sylvie Pesty. A review of learning planning action models. The Knowledge Engineering Review, 33(e20):1-25, 2018.
[AKFM22] Masataro Asai, Hiroshi Kajino, Alex Fukunaga, and Christian Muise. Classical planning in deep latent space. J. Artif. Intell. Res., 74:1599-1686, 2022.
[ALPK18] Ferran Alet, Tomás Lozano-Pérez, and Leslie P Kaelbling. Modular meta-learning. In Conference on robot learning, pages 856-868. PMLR, 2018.
[APG09] Alexandre Albore, Héctor Palacios, and Hector Geffner. A translation-based approach to contingent planning. In Twenty-First International Joint Conference on Artificial Intelligence, 2009.
[BKMC22] Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. Hierarchical modular framework for long horizon instruction following, 2022.
[BPF+21] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In CoRL, volume 164 of Proceedings of Machine Learning Research, pages 706-717. PMLR, 2021.
[CC04] Stephen Cresswell and Alexandra Coddington. Compilation of ltl goal formulas into pddl. In ECAI, volume 16, page 985, 2004.
$\left[\mathrm{CDVA}^{+}22\right]$  Ethan Callanan, Rebecca De Venezia, Victoria Armstrong, Alison Paredes, Tathagata Chakraborti, and Christian Muise. Macq: A holistic view of model acquisition techniques. In ICAPS 2022 Workshop on Knowledge Engineering for Planning and Scheduling, 2022.
[CMW13] Stephen N. Cresswell, Thomas L. McCluskey, and Margaret M. West. Acquiring planning domain models using LOCM. The Knowledge Engineering Review, 28(2):195-213, 2013.
$\left[\mathrm{VSM}^{+}23\right]$  Karthik Valmeekam, Sarath Sreedharan, Matthew Marquez, Alberto Olmo, and Subbarao Kambhampati. On the planning abilities of large language models (a critical investigation with a proposed benchmark). arXiv preprint arXiv:2302.06706, 2023. Previous versions accepted "Foundation Models for Decision Making Workshop at Neural Information Processing Systems, 2022" with title "Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change)".
[WDKM21] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
[WJK08] Chenggang Wang, Saket Joshi, and Roni Khardon. First order decision diagrams for relational mdps. Journal of Artificial Intelligence Research, 31:431-472, 2008.
[WK10] Chenggang Wang and Roni Khardon. Relational partially observable mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1153–1158, 2010.
[WPY05] Jason D Williams, Pascal Poupart, and Steve Young. Factored partially observable markov decision processes for dialogue management. In Proc. IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pages 76-82. CiteSeer, 2005.
[YFG07] Sungwook Yoon, Alan Fern, and Robert Givan. FF-Replan: A baseline for probabilistic planning. In Proc. ICAPS 2007, pages 352-360, 2007.
[YRT+22] Karmesh Yadav, Santhosh Kumar Ramakrishnan, John Turner, Aaron Gokaslan, Oleksandr Maksymets, Rishabh Jain, Ram Ramrakhya, Angel X Chang, Alexander Clegg, Manolis Savva, Eric Undersander, Devendra Singh Chaplot, and Dhruv Batra. Habitat challenge 2022. https://aihabitat.org/challenge/2022/, 2022.
[YZY+23] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023.
[ZC21] Yichi Zhang and Joyce Chai. Hierarchical task learning from language instructions with unified transformers and self-monitoring. In ACL/IJCNLP (Findings), volume ACL/I-JCNLP 2021 of Findings of ACL, pages 4202-4213. Association for Computational Linguistics, 2021.
[ZYP+22] Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Peter Yu, Yuwei Bao, and Joyce Chai. DANLI: deliberative agent for following natural language instructions. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1280-1298. Association for Computational Linguistics, 2022.
This Supplementary Material section provides additional details and discussions to complement our main paper. In section B, we expand on the discussion of our approach. Section C presents specific results for the seven tasks of the ALFRED challenge as well as new tasks that can be solved in the environment. Section D provides a formal definition of Object-Oriented Partially Observable Markov Decision Processes (OO-POMDPs), while section E provides a detailed background for our planning approach. Finally, section F provides an in-depth explanation of our method for the ALFRED challenge.
B Further Discussion
In section 9, we touched on various aspects of our methodology. This section provides more detail on these areas. Our approach to the ALFRED challenge hinges on egocentric planning and a user-defined symbolic planning model. We refer to these models as intentional, as they are purposefully crafted by the user, integrating specific environmental objects and properties. Symbolic models function as a form of world model that facilitates planning. Some methods seek to learn a latent representation, but this often comes with a high sample complexity (HPBL23). Modular meta-learning can reduce the sample complexity (ALPK18, ORCC21, MvSE22). If cost is not a constraint, world models can become components of efficient strategies trained to solve problems in specific domains  $(\mathrm{SAH}^{+}20)$