# Robot Control Platform for Multimodal Interactions with Humans Based on ChatGPT
## 3 Software Architecture
### Multimodal Human-Computer Interaction Framework
Stringent security measures are implemented to safeguard the integrity and confidentiality of user interactions, while meticulous error-handling strategies preempt unforeseen situations. The procedural delineation unfolds across four discernible components: the human user, the Pepper robot, the specialized application, and the ChatGPT interface, see Figure 1.  
The solution will be implemented and tested on the Pepper robot. Therefore, achieving this involves intricate integration with the ChatGPT API using Kotlin on Android Studio. The application's foundational structure revolves around a robust voice input mechanism, meticulously designed to capture user queries with unparalleled precision. This mechanism seamlessly converts spoken input into text, ensuring a fluid and responsive experience. The recognized text is then seamlessly transmitted to the ChatGPT API, initiating dynamic interactions capable of comprehending complex natural language and generating coherent contextual responses. These insights seamlessly circulate back into the project, enhancing overall communication. Concurrently, the MQTT queue is employed to invoke [PERSON]'s API, enabling control over the Pepper robot to execute specified instructions, and allowing additional ways of control for the person conducting the experiment such as starting, stopping the experiment, and invoking user speech input in case of error. Such functionality is provided by the MQTT Broker module. This structural stratification is orchestrated to synergistically harmonize the constituent elements, culminating in the establishment of a comprehensive framework for human-computer interaction. Within this framework, the complex integration of the Pepper bot and the ChatGPT system is achieved seamlessly. The multifaceted collaboration among these elements affords a sophisticated and dynamic environment, fostering a fluid exchange between the user, the robotic interface, and the advanced conversational capabilities facilitated by the ChatGPT interface. Upon initiation of the application installed on the robot, the user engages in spoken dialogue with the robot by posing inquiries. The application's speech-to-text system captures and transcribes the user's question, presenting it on the interface for visibility. Subsequently, the transcribed query is checked against a list of known instructions that robot can perform if the instruction is recognized e.g., \"Please play me a music\" or \"can you dance\". The robot executes such actions and then continues the conversation. The instruction list contains predefined robot actions. The action execution can be short, like play the saxophone, which takes about 15 s, or long and complex, e.g., magic trick, in which the robot performs a simple perception trick for the user that takes about 300 s. If the instruction is not recognized, the query is dispatched to ChatGPT via the API. In the event of the API functioning optimally, the response to the user's question is acquired. The obtained results are then showcased on the application's interface and concurrently broadcasted audibly through voice output. This iterative process persists until such time as the user opts to cease further inquiries, thereby concluding the ongoing conversation. Interaction flow can be observed in Figure 2.