ATLAS: Adaptive Landmark Acquisition using LLM-Guided Navigation

Authors: utteja  kallakuri
bharat  prakash
arnab  neelim mazumder
al  rashid
nicholas r waytowich
tinoosh  mohsenin
Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-03-29 23:21:01

Abstract: Autonomous navigation agents traditionally rely on predefined maps and landmarks, limiting their ability to adapt to dynamic and unfamiliar environments. This work presents ATLAS, a novel system that continuously expands its navigable landmark set and performs complex natural language-guided navigation tasks. ATLAS integrates three key components: a path planning module for navigating to known landmarks, an object detection module for identifying and localizing objects in the environment, and a large language model (LLM) for high-level reasoning and natural language understanding. We evaluate ATLAS in diverse virtual environments simulated in Gazebo, including indoor office spaces and warehouses. Results demonstrate the system's ability to steadily expand its landmark set over time and successfully execute navigation tasks of varying complexity, from simple point-to-point navigation to intricate multi-landmark tasks with natural language descriptions. Our comprehensive tests show that ATLAS expands its knowledge, achieving a 100% success rate in tasks involving known landmarks and up to 100% in semantically inferred goals for objects not in the initial Knowledge Base. We further demonstrate the system's capacity to enhance its navigational knowledge incrementally, showcasing its ability to dynamically adapt and accurately perform complex, natural language-driven tasks in diverse simulation environments.

Main paper: 
 introduction: 
 Navigation tasks involving natural language instructions pose a significant challenge for autonomous agents [2, 4] . While traditional navigation systems rely on predefined maps and landmarks, real-world environments are dynamic and often require the ability to continuously learn and adapt. This paper presents a novel approach that combines various modules, including natural language processing, object detection, and path planning, to create an intelligent navigation system capable of continuously expanding its knowledge base and performing complex navigation tasks.The motivation behind this work stems from the observation that humans excel at navigating unfamiliar environments by continuously updating their mental maps and exploiting their ability to understand natural language descriptions. For instance, when exploring a new city, humans can identify notable landmarks through visual cues and incorporate them into their mental map [7, 19] . Furthermore, they can follow directions like "Head towards the tall clock tower, then turn left at the fountain" by grounding the language instructions to the perceived landmarks. This remarkable ability to fuse perception, language understanding, and spatial reasoning enables humans to navigate efficiently in novel environments [10] . The proposed ATLAS system aims to emulate this human-like navigation capability by integrating key components that enable continuous learning and reasoning. It is designed to address three objectives: 1) continually growing the set of navigable landmarks, 2) performing navigation tasks involving one or more landmarks, and 3) utilizing a large language model (LLM) to assist with high-level reasoning, semantic navigation, and landmark set expansion [6, 22] . By leveraging these capabilities, the system bridges the gap between natural language instructions and physical navigation in dynamic environments. In this work we use Cohere as the LLM [5] .At the core of ATLAS is a navigation module that canKnowledge BaseThe LLM planner accesses various landmarks from the knowledge base to create plans for robot navigation.COO Nav queries the knowledge base for the actual locations of the landmarks based on the LLM plan COO Nav acquires the locations of the landmarks from the knowledge base and sequentially passes the goals to the action server for plan executionUser prompt "main_door": {"x": -5.9, "y": -17.1} "tv": {"x": -6.5, "y": -10.2} "couch": {"x": 4.8, "y": 1.1} "laptop": {"x": -0.9, "y": 7.1}The Knowledge Base contains a set of known landmarks (ID and location) and is dynamically updated through interactions with the Object Detector. The detections are filtered using an LLM to only push relevant landmarks to the knowledge base User starts the high level planning process by providing an abstract task for the robot to update the knowledge base and use LLM subsequentlyFigure 1 . High-level system architecture detailing the components and data flow between the LLM Planner, COO Navigation Package, ROS Action Server, Knowledge Base with Landmarks, and Object Detector. The agent is setup with an initial set of landmarks it can navigate to in its knowledge base. The agent receives a goal which is parsed by the LLM and looks up the knowledge to get navigation goals. This is then used by the ROS planner for navigation. The agent detects objects along the way which is then filtered through the LLM to get a relevant objects. These new objects are stored back in the knowledge base. The dictionary progressively keeps growing as the agent explores the environment.plan paths to known landmarks based on their coordinates. However, the true power of ATLAS lies in its ability to learn and expand its knowledge through an object detection module and an LLM [24] . The detection module enables the agent to identify objects in its environment, while the LLM provides high-level reasoning and natural language understanding capabilities. The integration of these components allows ATLAS to continuously grow its landmark set by interpreting natural language instructions, detecting and localizing new objects, and incorporating them as navigable landmarks. Furthermore, the LLM's semantic understanding capabilities enable ATLAS to perform complex navigation tasks involving multiple landmarks and high-level reasoning, such as interpreting and executing instructions like "Go to the kitchen and then proceed to the living room."The key contributions are summarized as follows: • A system architecture that integrates a path planning module, object detection, and a large language model for navigation tasks. • A novel method for continuously expanding the set of navigable landmarks by detecting objects in the environment and incorporating them into the system's knowledge base with the assistance of the LLM. • An approach to performing complex navigation tasks involving multiple landmarks and high-level reasoning by leveraging the LLM's semantic understanding capabilities. • Experimental evaluation demonstrating the system's performance in various navigation scenarios, highlighting its adaptability, continuous learning, and execution of complex tasks with natural language instructions. 
background and related works: 
 Edge deployment of large multimodal models [15] [16] [17] [18] along with the intersection of natural language understanding and robotics have led to the field of Vision-and-Language Navigation (VLN) [2, 14] , where the integration of linguistic instructions with visual perceptions aids autonomous agents navigate and interact within an environment. The advent of Large Language Models (LLMs) has introduced new dimensions to decision-making in VLN, di-verging into two main trajectories: utilizing LLMs as auxiliary helpers or primary planners.In the domain of using LLM as an auxiliary helper, works such as LM-Nav [20] leverage LLMs to generate contextual information, such as landmark lists from raw navigation instructions, aiding vision-language models in inferring joint probability distributions. These applications underscore LLMs' potential beyond mere information generation, suggesting their capability for comprehensive planning grounded in real-world environments.LLM as a primary planner domain explores LLMs' capability to craft executable plans, directly or through low-level planners, for embodied agents [1, 8, 9, 13, 21, 25] . While these investigations underline the feasibility of LLM-driven planning, they often rely on the premise of predefined admissible actions, a condition that could limit practical applicability due to the dynamic nature of real-world environments and the increasing complexity with the number of environment objects.Voyager [23] is one such work in which the authors present an LLM-powered agent in Minecraft, autonomously exploring and learning complex skills through a novel curriculum, skill library, and a code refinement process, showcasing unmatched proficiency and adaptability. ITP [12] presents a framework that uses language models for robots to perform tasks and adapt to new goals. ITP leverages LLMs for high-level planning and execution, integrating planning, vision, and skill execution. GOAT [3] is a universal navigation system for mobile robots with three key contributions: a multimodal approach for handling goals via category labels, images, and language descriptions; a lifelong learning capability that utilizes past environmental experiences for improved navigation; and a platformagnostic design enabling deployment across various robotic platforms.Our paper introduces a novel integration of natural language processing, object detection, and a LLMs. Unlike ITP [12] , which relies on predefined guidelines for task execution, our approach dynamically expands navigable landmarks through environmental interaction and language model insights, facilitating continuous learning. Contrary to VOYAGER's [23] focus on skill acquisition within simulated settings, our method emphasizes real-world landmark learning for navigation. Furthermore, our system employs a language model for semantic reasoning and complex task execution, a capability not present in GOAT [3] . This integration enables sophisticated navigation based on natural language instructions, showcasing our system's adaptability and superior performance in real-world scenarios. 
proposed approach: 
 We begin the proposed approach by describing the problem statement and the overall architecture shown in Figure 1 .We consider a system where the agent receives instructions in the form of natural language describing a navigation task. The agent has a navigation module which is capable of planning a path to a landmark provided it has the corresponding co-ordinates on the map. The agent is initialized with a small set of landmarks to which it can navigate. It is also equipped with an object detection module that can detect objects as well as their distances from its current position. Finally, the agent also has access to an LLM to help with high-level reasoning.Given these ingredients, our objective is to build a system which can 1) continually grow the set of landmarks it can navigate to, 2) perform navigation tasks that involve one or more landmarks. 3) Use the LLM to assist with navigation, grow the landmark set, and perform semantic navigation.The agent architecture consists of three main modules. First, a path planning module called the COO Nav (Coordinate Navigation), which is built on top of ROS. Given a map, this module is used to perform path planning and navigate to a landmarks given a set of co-ordinates. Second, the high level planner module is an LLM which parses a natural language input and provides a structured plan. The plan comprises of a sequence of waypoints or landmarks that the agent must navigate to. The prompt to the LLM is constructed such that it has access to the current set knowledge base(KB) of landmarks and the output is a subset of landmarks required to complete the given task.Third, the object detection module is responsible to detecting and localizing objects in the environment which is then used to acquire more landmarks in the environment. This module uses a YOLO-v5s model [11] -to identify new objects as the agent is navigating the environment. This coupled with a depth sensor is used to localize new objects in the environment. However, the object detector is likely to detect random objects that might not make sense as landmarks. Instead of using a human to filter theses, we use another instance of the LLM to perform this task. Since LLMs capture human intuition, we design a prompt that asks the LLM to decide if it makes sense to add an object as a landmark. These new filtered landmarks are added to the KB which keeps on growing with time.Given a natural language instruction, these three components act together to perform high level planning, navigate in the environment, and add new landmarks to the growing KB. This process is outlined in the algorithm 1.The COO Nav package acting as the low-level policy bridges the gap between high-level plans generated by the LLM and the low-level ROS Action Server that controls UGV movement. COO Nav interfaces directly with ROS's actionlib to execute navigational tasks providing the services navigate to location and spin service. The navigate to location service controls the movement of the UGV to specified waypoints. It retrieves the coordinates from the KB based on the input location identifier and sends a goal to the ROS Action Server, directing the UGV to navigate to these coordinates. The spin service on the other hand, ensures the robot is able to make a in-place rotation thereby scanning the environment for any objects of interest to be added into the KB.The YOLO object detector is used for real-time object detection and executes callback functions to manage image and depth data from appropriate ROS topics and to update ATLAS with the latest inputs and detections. This operates in parallel with the navigation components and it continuously scans the UGV's surroundings to identify and classify objects which are used to enrich the KB. This ensures that ATLAS's environmental understanding evolves with each deployment, enhancing its navigational intelligence. spin() Capture user prompt and parse via LLM Planner  
environment: 
 To evaluate the performance of ATLAS, we conducted extensive experiments in diverse virtual environments using the Gazebo simulator. Gazebo provides a realistic 3D simulation of robotic environments, enabling us to rigorously test ATLAS's capabilities in controlled yet complex scenarios. We constructed multiple virtual worlds with varying landmark configurations, object placements, and environmental conditions, mimicking real-world settings such as indoor office spaces, residential spaces, and natural landscapes.Within these simulated environments, we deployed a mobile robot equipped with sensors for perception and navigation, replicating the setup of ATLAS. The robot's perception module utilized a simulated RGB-D camera and a Li-DAR sensor to detect and localize objects, while the navigation module planned and executed paths based on the available landmark information. The LLM operated in parallel, processing natural language instructions and assisting the robot in expanding its landmark knowledge and reasoning about complex navigation tasks. This experimental setup allowed us to systematically evaluate ATLAS's performance across a range of navigation scenarios, from simple pointto-point navigation to complex multi-landmark tasks with natural language descriptions.The first set of results illustrates ATLAS's capability to continuously grow its navigable landmark set over time. We present a graph that plots the number of landmarks in the robot's KB as a function of time spent exploring the virtual environment. The graph demonstrates a steady increase in the set of landmarks, showcasing ATLAS's ability to successfully detect, localize, and incorporate new objects as navigable landmarks with the assistance of the LLM. For the second set of results, we evaluate ATLAS's performance on various types of navigation tasks, ranging from simple point-to-point navigation to complex multi-landmark tasks with natural language descriptions. We report the task success rate, a metric that measures the percentage of navigation tasks successfully completed by the robot. We also test ATLAS's ability in semantic goal reaching which evaluates the agent's ability to leverage the language model's semantic understanding to interpret and execute navigation instructions involving landmarks not present in the current landmark set.The effectiveness of a fully autonomous system can be determined by its adaptive learning capabilities, specifically its ability to continuously integrate new environmental data into its navigational framework. In our case, this is captured in the expansion of the KB. The KB, initialized with a basic set of landmarks (referred to as 'KL IN IT '), is progressively augmented as the UGV explores its environment and identifies new objects of interest, as shown in Figure 2 . The incremental discovery process is shown by the UGV's exploration in two different environments. In 'Env 1', a household environment, the UGV's KB grew from the initial three landmarks to a total of eleven. Each additional entry in the dictionary represents the inclusion of a newly identified landmark during a task execution, such as 'Go to Room 1' and 'Go to Room 2'.For 'Env 2', a warehouse setting, a more complex environment, this KB growth is more evident. Here we start with five pre-known landmarks and extending to a comprehensive suite of seventeen. This environment has a broader array of object classes, showing the UGV's capacity to adapt its KB in diverse settings.The growth of the KB over time has significant implications for the execution of navigational tasks. With an enriched set of landmarks, the UGV is more equipped to understand and navigate to new locations, facilitating a higher task success rate while enabling the high level planner (HLP) to make more efficient plans. The relation between the breadth of the KB and the UGV's navigational performance highlights ATLAS's ability to leverage its growing database for enhanced autonomous operation.This dynamic learning ability, combined with the language model's semantic understanding, allows the UGV to extend its navigational capabilities beyond the initial landmark set, demonstrating ATLAS's potential in adaptive long-term deployment and unseen scenarios.Learning in Varied SettingsFigure 3 presents a detailed visualization of an autonomous task execution within a residential setting by the UGV, resulting in the dynamic update of the KB. The task begins with a user prompt, which is sequentially processed by the high-level and low-level planning stages.The process initiates with a user prompt:"Investigate Rooms 1 and 2". The HLP interprets this high-level command to provide a sequential plan to investigate two specific rooms within the residential environment. The HLP translates the user prompt into actionable goals for the UGV: first, to Go to Room 1, and then Go to Room 2. These goals are then broken down into discrete actions by the Low-Level Planner, which commands the UGV to move to the corresponding locations using the navigate to location service and execute a spin service. This spin ensures that the UGV performs a 30 • in-place rotation in the clock-wise and counterclockwise directions to scan the environment for any objects of interest.As the UGV executes the planned navigation the perception component, identifies and localizes objects such as a refrigerator, suitcase, potted plant, couch, and laptop. These objects are highlighted by bounding boxes in the Camera Front View. Each detected object is assessed against the current KB; if an object is not already present in the KB and is within a specified distance threshold, it is added as a new landmark. This process is shown by the growing entries in the KB, where new objects like 'Refrigerator' and 'Laptop' are incorporated as the UGV explores Rooms 1 and 2. The Gazebo and camera views collectively illustrate the UGV's field of operation and its perception field, respectively. The integration of these views with the planning stages and the KB updates provides a comprehensive feedback loop that not only demonstrates the ATLAS's current operational state but also informs its future behavior by increasing its spatial understanding and navigational database.Similarly, Figure 4 depicts task-oriented navigation in a warehouse setting. The sequence again starts with a userdefined task that undergoes systematic processing through various stages of planning and execution. In this setting, the user instructs the UGV with a prompt:"Navigate to the garage to check if the delivery truck is still here." This initiates a series of operations in which the HLP devises a preliminary plan to visit the garage. Consequently, the Low-Level Planner dispatches commands to the UGV to execute the navigate to garage service coupled with a spin service. The scenario continues as the UGV is sequentially tasked with navigating to the cafe to fetch a water bottle and deliver to the lounge. The UGV adapts to this complex, multistep operation by leveraging the updated KB to facilitate navigation and object interaction. ATLAS's capability extends beyond the retrieval of known landmarks within the KB. It can also perform advanced semantic reasoning by navigating to the most proximate and contextually relevant landmark when tasked to find an object absent from the KB. Table 3 illustrates the ATLAS's performance on semantic goal-reaching tasks.In scenarios where the user poses queries like "What is the closest landmark near which I can find a sink," the system can intelligently suggest 'Refrigerator' as the landmark, leveraging the semantic clustering of objects in the environment. Despite 'Sink' not being a direct entry in the KB, the LLM deduces that a sink can often be found in proximity to refrigerators in residential layouts.The 'Landmark (suggested / actual)' column showcases the LLM-prompted landmark versus the actual landmark. The success rate measures the number of times the appropriate landmark is prompted while the experiment is run 10 times.This ability to perform semantic goal-reaching by connecting the dots between related objects significantly enhances ATLAS's navigational efficiency, particularly in environments rich with contextually grouped entities. This ability makes the UGV not just a follower of direct commands but an intelligent agent capable of making inferences about its surroundings. 
conclusion: 
 In this paper we introduced ATLAS an autonomous navigation system that integrates path planning, object detection and LLMs to navigate using natural language instructions. The key contributions include ATLAS's ability to dynamically expand its knowledge base with new landmarks and use them to execute complex multi-step navigation tasks. Our results demonstrate ATLAS's ability to adapt and learn incrementally achieving a high success rate across both known and semantically inferred navigation tasks in simulated environments.Future work will explore extending ATLAS with more complex object detectors with a strong open-vocabulary detection capability and grounding ability. This allows us to further bridge the gap between the way humans and robots perceive and interact with their surroundings making autonomous navigation more versatile and intelligent. 
 
 Appendix:  



Reviews: review: title: title: Review
main_review: main_review: The findings illustrate how the system consistently broadens its repertoire of landmarks over time, effectively completing navigation tasks of different complexities. These tasks range from basic point-to-point navigation to intricate multi-landmark objectives accompanied by natural language instructions. Given the experiments and ablation study, I suggest `weakly accept` for this paper.