An Analysis of Visually Grounded Instructions in Embodied AI Tasks
Authors: Marco Grazioso, Alessandro Suglia (Interdepartmental Research Center Urban/Eco, Università di Napoli Federico II, Logogramma s.r.l, Heriot-Watt University, AlanaAI, 2023)
Topic: Language Models

Abstract
Thanks to Deep Learning models capable of learning from Internet-scale corpora, we have observed tremendous advances in both text-only and multi-modal tasks such as question answering and image captioning. However, real-world tasks require agents that are embodied in the environment and can collaborate with humans by following language instructions. In this work, we focus on ALFRED, a large-scale instruction-following dataset designed to develop artificial agents that can execute both navigation and manipulation actions in 3D simulated environments. We present a new Natural Language Understanding component for Embodied Agents as well as an in-depth error analysis of model failures for this challenge, going beyond the success-rate performance that has been driving progress on this benchmark. Furthermore, we provide the research community with important directions for future work in this field, which are essential to the development of collaborative embodied agents.

Introduction
In recent years, we have witnessed tremendous improvements in Natural Language Understanding (NLU) tasks thanks to powerful Large Language Models (e.g., [1, 2, 3]). These models are trained by leveraging internet-scale textual data. However, with access to text only, they utilize only a portion of the rich multi-modal training data available from interactions with the world and other agents [4]. Embodied Artificial Intelligence (EAI) is the field of AI focused on developing agents that can perceive the environment through multi-modal inputs and execute actions in the world. Many benchmarks have been proposed in EAI. For example, Vision+Language Navigation [5] aims to study the ability of EAI agents to follow natural language instructions in 3D simulated environments. However, the agent can only output navigation actions, limiting the richness of concepts it can learn. To simulate a scenario closer to real-world use of these systems, Shridhar et al. [6] propose ALFRED, a new instruction-following benchmark that facilitates the study of situated language understanding, visual memory, commonsense reasoning, and long-term action planning. So far, progress on ALFRED has been driven by accuracy-based metrics on the official leaderboard (e.g., [7, 8, 9, 10]). However, given that the success rate on this benchmark remains below production-level performance (~40%), a more in-depth analysis of model failures is warranted. In this paper, we make two main contributions: 1) we train a novel Natural Language Understanding component for an EAI agent using multi-task learning that achieves a 0.117 error rate on the ALFRED validation unseen set, an improvement over the model proposed by Min et al. [7]; 2) we provide an in-depth analysis of our model’s failures, highlighting gaps in critical situated language understanding capabilities essential for EAI agents, such as referential expression resolution and conversational grounding [11].

Conclusion
In this work, we used the ALFRED dataset as a benchmark to investigate the language understanding abilities of state-of-the-art EAI models. We began by improving the model originally proposed by Min et al. [7] through multi-task learning, demonstrating that even with this new model, several issues remain unresolved. We categorized these problems into different classes to facilitate our analysis, leading us to conclude that an EAI agent must leverage multi-modal signals, commonsense knowledge, and user interaction to effectively solve embodied tasks. According to Schlangen [21], situated interaction is a direct, purposeful encounter between free and independent, yet similar agents. Following this definition, in ALFRED tasks, there are two distinct agents: a follower and a leader. The leader acts as an oracle, providing instructions in one go without engaging in dialogue with the follower and assuming that the follower possesses perfect capabilities to follow instructions without accounting for uncertainty or possible errors. Additionally, there is no concept of conversational grounding, which is intended as a joint activity where both agents negotiate meanings necessary to solve the task effectively and efficiently. In this regard, while the ALFRED dataset presents a challenging task, it falls short of providing a benchmark suitable for developing artificial agents capable of collaboratively solving tasks through natural language.