One to rule them all: natural language to bind communication, perception
  and action
Authors: name: Simone Colombani; name: Dimitri Ognibene; name: Giuseppe Boccignone
Published: 2024-11-22 16:05:54

Abstract: In recent years, research in the area of human-robot interaction has focused
on developing robots capable of understanding complex human instructions and
performing tasks in dynamic and diverse environments. These systems have a wide
range of applications, from personal assistance to industrial robotics,
emphasizing the importance of robots interacting flexibly, naturally and safely
with humans. This paper presents an advanced architecture for robotic action
planning that integrates communication, perception, and planning with Large
Language Models (LLMs). Our system is designed to translate commands expressed
in natural language into executable robot actions, incorporating environmental
information and dynamically updating plans based on real-time feedback. The
Planner Module is the core of the system where LLMs embedded in a modified
ReAct framework are employed to interpret and carry out user commands. By
leveraging their extensive pre-trained knowledge, LLMs can effectively process
user requests without the need to introduce new knowledge on the changing
environment. The modified ReAct framework further enhances the execution space
by providing real-time environmental perception and the outcomes of physical
actions. By combining robust and dynamic semantic map representations as graphs
with control components and failure explanations, this architecture enhances a
robot adaptability, task execution, and seamless collaboration with human users
in shared and dynamic environments. Through the integration of continuous
feedback loops with the environment the system can dynamically adjusts the plan
to accommodate unexpected changes, optimizing the robot ability to perform
tasks. Using a dataset of previous experience is possible to provide detailed
feedback about the failure. Updating the LLMs context of the next iteration
with suggestion on how to overcame the issue.