# Robot Control Platform for Multimodal Interactions with Humans Based on ChatGPT
## 3 Software Architecture
### Implementation
If we receive a successful response, the application displays it on the screen as a conversation and, using \(TextToSpeech\), says it. Then, it waits three seconds and starts the next cycle of the loop by calling \(promptSpeechInput()\). During application start, \(startMqtClient\) is called. It connects the android application with the main framework via \(mqttBroker\). \(mqttClient\) maintains the connection and handles incoming and outgoing messages. Incoming ones are handled by another callback function, \(handleIncomingMessage\). It recognizes the command and invokes the appropriate functions. For example, on command \"start interaction\", received on command MQTT topic, a \(startInteraction\) function is called. It works similarly in other cases. Assertions and error cases were not included in Algorithm 1 for simplicity. They work by logging debug messages and, if the user is expecting some reaction from the robot also, showing and saying to the user the appropriate simplified message. For example, if the user wants the robot to play saxophone and, for some reason, the framework cannot perform this action, the message \"Sorry I can't play saxophone at this moment\" would be presented to the user. During implementation, we encountered a few interesting challenges. The three most important ones were: noise, latency, and user privacy. Environmental noise can make the speech recognition process difficult. To mitigate such a risk, we used the noise suppressor available in the Android library. Potential delays in processing voice input pose a real risk to the seamless flow of conversations. By using coroutines for asynchronous processing, where it was possible, and minimizing the number of calls in the conversation loop, we minimize latency time. Next, the most time-consuming action is a call to the ChatGPT API. The delay can be lowered by using local LLMs, which we are planning to do in our next study. However, it requires a powerful GPU installed locally. User privacy security is an important challenge; we handle it by storing user data in a minimal number of places, only for duration of conversation and transmitting it via a secured channel like the hypertext transfer protocol secure (HTTPS) connection. For detailed information about framework implementation, please refer to our previous article [27]. ```
procedurestartInteraction \(initNoiseSuppression()\) \(promptSpeechInput()\) procedureonSpeechResult(\(rquestCode,resultCode,intent\)) if\(resultCode==\)RESULT_OKthen \(input\gets intent.getResult()\) \(handleInput(input)\) procedurehandleInput(\(input\)) \(data\gets input.trim(it<=^{\prime}).toLowerCase()\) if\(command=check1 fKnownCommand(data)\)then \(mqttClient\gets getMqttClient()\) \(mqttClient.sendMessageToFramework(command)\) \(showTextToUser(^{\prime}letterytoperform^{\prime\prime}+command)\) else \(language\gets detectLanguage\) \(setLanguage(language)\) callChatGPTAPI(\(data\)) procedurecallChatGPTAPI(\(data\)) \(header\gets prepareChatGPTHeader()\) \(body\gets prepareChatGPTBody(data)\) \(client\gets OkHttpClient.Builder().connectTimeout(30,TimeUnit.SECONDS).build()\) \(request=Request.Builder().url(APIURL).header(header).post(body).build()\) \(client.newCall(request).enqueue(callbackGPT)\) procedure callbackGPT(\(response\)) ifresponse.isSuccessful() then \(cleanResult\gets getCleanResult(response.body)\) \(showTextToUser(cleanResult)\) procedureshowTextToUser(\(message\)) \(tts\gets getTextToSpeech()\) \(messageList\gets getMessageList()\) \(messageList.add(message)\) \(showMessageList(messageList)\) \(tts.speak(message)\) \(delay(3000)\) \(promptSpeechInput()\) procedurestartMqttClient \(mqttClient\gets MqttHelper(context)\) \(mqttClient.setCallback(handleIncomingMessage)\) \(mqttClient.connect(BROKER_URL)\)
```  
**Algorithm 1** Android application main algorithm.