# Robot Control Platform for Multimodal Interactions with Humans Based on ChatGPT
## 3 Software Architecture
### Multimodal Human-Computer Interaction Framework  
In Figure 1, we can see the high-level structure of the multi-platform intelligent system for multimodal human-computer interaction framework. The framework, among others, consists of the following modules: _Main_, _MQTT Broker_ (using _Message Queuing Telemetry_Transport), Pepper, Avatar, Microservices close and Microservices in cloud. Avatar module is an Android application that can be deployed on most Android devices. It displays a virtual assistant that behaves similarly to a robot and may be used for testing or performing experiments in a virtual environment. Microservices close are services fully developed by us that can be deployed anywhere. They consist of perception modules: _Emotion, Gaze, Question_ detection and utility services for _Discovery_ and application programming interface (API) gateway _Edge server_ for ease of use and development of the framework. Speech recognition can be offloaded to an external service via the _Speech recognition_ module or executed locally in _Pepper_ or _Avatar_ modules. The _ChatGPT_ module represents connection with the ChatGPT large language model. In future, we are considering using local LLMs like large language model meta AI (Llama). The response from ChatGPT is presented to users using two communication channels: audio (speech) and visual, using [PERSON]'s tablet. Behavior Planner is a part of the _Main module_. It is responsible for executing scripted scenarios and choosing appropriate paths in the script using the _Decision module_ according to the data from perception components. Other functions of the _Main module_ are communication and coordination of other modules and preparing the robot or avatar for scenario execution. The perception components are responsible for extracting higher level data about the user interaction with the robot based on the raw data from the robot sensors or (in future) external ones. It consists of components like face tracking, gaze direction estimation, question detection, and speech recognition. _MQTT Broker_ is responsible for communication between _Pepper_, _Avatar_ modules, and the _Main module_; some communication also happens using the native pepper API. The _Pepper_ module consists of the pepper native application to handle robot movements and data sources (cameras, microphones, etc.) and the Android application working on [PERSON]'s tablet that allows us to use Android services like _Text-ToSpeech_ and speech to text for speech processing and also for displaying conversation or other important messages to the user during interaction. Reverse data flow with collected data from tablet sensors is also possible. See [27] for more details. Our framework allows  
Figure 1: Robot Control Platform for Multimodal Interactions with Humans based on ChatGPT. for offloading heavy computational procedures from the _Main module_ executed on a robot platform to a local or global cloud depending on the available resources. A great example of such approach is the perception modules that work as Close microservices but have fallback implementation in the _Main module_. The framework is also capable of execution in many operating systems and can handle different types of robots or virtual avatars seamlessly. By utilizing robot abstraction, adding new robot types to our system is relatively simple. It requires implementing the robot interface using a robot native solution and creating a minimal configuration. An essential element of the user experience is conversation with the robot and articulation of responses through the robot, epitomizing the convergence of state-of-the-art technology and tangible human-robot interaction. The application architecture delineates distinct modules for speech input, text transfer, ChatGPT interaction, result processing, and speech output, each contributing to the overall effectiveness of the conversational paradigm.