# Robot Control Platform for Multimodal Interactions with Humans Based on ChatGPT
## 3 Software Architecture
### Implementation  
The Android application running on [PERSON]'s tablet is written in Kotlin and meticulously structured into three distinct components: speech recognition, voice broadcast, and communication with the main module that handles ChatGPT and robot communication. The underlying design philosophy of the user interface revolves around transcending age and regional barriers, ensuring universal accessibility. The UI design is thoughtfully crafted to be simplistic and user-friendly, ensuring that users from diverse backgrounds can seamlessly navigate and utilize the technology. For every user message, there is a two-step process to make the conversation with logic: first, the message is added to the conversation history, and then, an API call is initiated. Upon receiving responses from the API, they are seamlessly integrated into the conversation history and displayed in the chat (see Figure 3). The intricacies of _TextToSpeech_ posed unique challenges. The default behavior of broadcasting in the system's default language led to nuanced issues. In scenarios where the default language was English, but the input text was in French, the resulting voice announcement adhered to English grammar, creating a discordant experience. This challenge extended to various language combinations, exemplified by the awkward intonation and  
Figure 3: Application working on Pepper robot, user view of the robot during conversation. pronunciation mismatches observed when the input text was in Chinese. The integration of the Google ML Kit, a machine learning algorithm, emerged as a transformative solution. This advanced algorithmic framework became the cornerstone of language recognition within the application. The shift from manual keyword identification to an automated, machine-learning-driven approach not only enhanced accuracy but also paved the way for accommodating a broader spectrum of languages. By extracting the recognized language as a parameter, the _TextToSpeech_ function was empowered to deliver voice playback that was not only linguistically accurate but also imbued with the richness and nuance of diverse languages. The decision to employ the Google speech-to-text APK for speech-to-text conversion underscored a commitment to versatility, enabling the application to recognize and convert an array of languages--English, Spanish, Polish, Chinese, and more. Unfortunately, the robot uses an older version of android that cannot handle automatic language changes for the speech-to-text operation as such language is set at the beginning of the conversation for system language (or any installed language set by the experimenter). The application was tested on a newer device that can handle language change and works correctly. However, if a user manually input a message in any of the installed languages on the device, the application language would change and said language would be used further into conversation. In Algorithm 1, we can see how the Android application main loop is working. The application works in an event-driven manner as such a loop is indirect and works by a series of callbacks and events. The loop starts when the \(startInteraction\) function is called. It can be called by the \(mqttClient\) in \(handleIncomingMessage\) callback when a message comes from the framework or by a user pressing a button visible on the tablet screen. Then, noise suppression is initiated. It helps in speech-to-text functionality. Next, a speech input prompt is created using standard Android intent \(RecoguzierIntent.ACTION\_RECOGNIZE\_SPEECH\). When the speech recognition results are ready, \(onSpeechResult\) is called by the event, then the recognized message is handled by \(handleInput\). If it is a known command, e.g., \"Thank you, goodbye\", a message is sent to \(mqttBroker\) and the appropriate action is executed by the framework. In this case, the conversation would end. Otherwise, the CH4 PT API is called in \(callChatGPTAPI\) and the response is handled when ready in \(callbackGPT\).