# Enhancing Human-Agent Interaction via Artificial Agents That Speculate About the Future
## 3 Materials and Methods
### Approach
#### 3.1.1 Avatar Speech System  
We used the Loomie application to create the 3D humanoid virtual avatar for the experiment, using the same ethnically ambiguous female avatar with all participants ([[https://www.loomielive.com/](https://www.loomielive.com/)]([https://www.loomielive.com/](https://www.loomielive.com/)) accessed on 12 December 2024). The avatar was capable of moving its lips synchronously with the synthesized speech, as well as some basic built-in gestures from the Loomie application, which we did not attempt to modify here. The speech system itself (both development and testing) has been described in detail in several previous publications [57; 58], so we provide a brief overview here for brevity. In short, the speech system was developed as a \"Social AI\" specifically for cooperative game paradigms, where the system has awareness of events occurring within the game in _real time_, including player actions, as well as the ability to detect and respond to human utterances (i.e., automatic speech recognition, ASR). The speech content of the system itself was essentially a knowledge base that comprised hundreds of different speech utterances covering 46 different utterance categories, each related to a particular game situation (e.g., collecting resources, fighting monsters, deciding where to go next). The agent's speech comprised 3 types: self-generated speech (regarding current/past game events), anticipatory speech (about future possibilities), and ASR responses (to human queries). These are terms we defined to distinguish between utterances the AI produces using its internal logic of what to say next based purely on the current situation (self-generated) versus utterances that are based on predictions of potential future events (anticipatory speech). In contrast to these two, ASR responses are direct responses to human queries. Some short example dialogues of speech interactions between the human user and the artificial agent are provided in Appendix A, including both examples with and without anticipatory speech. The idea was that anticipatory speech should hypothetically create more fluid interactions and provide better coordinated action between the human and the artificial agent. The anticipatory speech was generated via deep learning models in Keras ([[https://keras.io/accessed](https://keras.io/accessed)]([https://keras.io/accessed](https://keras.io/accessed)) on 12 December 2024), which have been described in detail previously elsewhere, showing an AUC (area-under curve) performance of roughly 0.81 in predicting future game situations (see Section 4.3 in [58]). Here, we focus on the _application_ of those models during user testing and thus provide only a brief description of the model details for brevity. In short, those models attempt to predict future game situations that are likely to occur in the next 5-10 s and then talk about them. The models ran in real time, producing new predictions every second. In terms of architecture, the models combined several layers of convolutional neural networks (CNNs) and recurrent neural networks (LSTM), followed by a final dense layer with a sigmoidal activation function to produce a set of binarypredictions (probabilities) for multiple game situations simultaneously. The idea behind the model architecture was that the CNN layers could parse out \"invariant representations\" of pattern signatures occurring anywhere in the interaction, followed by the LSTM layers detecting critical \"sequences\" of those patterns over time. By critical, we mean that those sequences are related to the aforementioned game situations (e.g., collecting resources, fighting monsters, deciding where to go next) occurring shortly thereafter. To limit anticipatory speech to occurring only above a certain predicted probability, a probability threshold was set to control their frequency (0.7 in our case), which was determined through trial and error during testing. If more than one game situation exceeded the threshold, then the game situation with maximum probability was selected.