# HRLB\({}^{\wedge}\)2: A Reinforcement Learning Based Framework for Believable Bots
## 4 HRLB^2: A Reinforcement Learning Based Framework for Believable Bots
### Heuristic Exploration Process
This approach let us quantify the difference between the behaviors of our participants in the third-person Turing test because the effect's size favors the difference of sizes instead of confounding this with sample size [42]. Specifically, we applied the bootstrap effect sizes algorithm (_bootES_) implementation of [43] to our data. Additionally, we standardized this effect's size for the Cohen's d measure. In Table 2, we present the estimated Cohen's d, 95% CIs and the size effect for [PERSON] against the rest of participants. From these measures, we conclude that the manner to play SFIV of [PERSON] is highly dissimilar to the CPU bots' playing style. On the other hand, our bot's behavior is comparable to the playing style of the three human participants. The size effect is minimal when compared to Human 2 and Human 3. In other words, [PERSON] plays in an analogous style to intermediate and advanced players. As we have stated before, our main goal in this research work was to create a believable agent that plays in an effective manner. In addition, we built violin plots to better evaluate the exhibit skill level of our HRLB\({}^{\star}\)2-based bot. In Figure 7, we present a violin plot that presents the participants' skill level and the self-evaluation skill level of the judges. In this plot, the median is shown as a white dot, the thick bar in the center represents the interquartile range and the thin line represents the 95% confidence interval. By visually inspecting our violin plot, we can notice that the density distribution for [PERSON] is in between Human 2 and Human 3. Because of this analysis--and the previous size effect study--we can assert that [PERSON] exhibited a higher skill level than Human 2 but not as good as Human 3. Although [PERSON] could not match its teacher's competence (Human 3), we believe that the attained skill level is good enough to be considered as effective. However, there is still room for improvement in this regard. Lastly, we have read the 117 comments, by 78 people, from the open-ended question. From the analysis of these comments, we formulated the main reasons why our HRLB\({}^{\star}\)2-based bot was

Figure 7: Third-person Turing test violin plot. considered as a human player or a computer. Additionally, we include the number of times each comment was observed. These conclusions are listed below:

[PERSON] was classified as a human mainly because it:

* Followed strategies and performed combos that are suitable for a person of its exhibited skill level (15 observations)
* Exhibited a wider range of attacks than its opponent (11 observations)
* Made execution mistakes that a human would make (8 observations)
* Looked like it adapted to its opponent's game style (7 observations)

[PERSON] was classified as a computer mainly because:

* Its anti-air strategies were very consistent (11 observations)
* Made errors that a person of its exhibited skill level would not make (9 observations)
* It seemed like it did not predict its opponents moves (4 observations)
* It did not expressed emotions, such as fear when it was about to lose (3 observations)

## 8 Evaluation of HRLB^2

In this section, we present the implemented experiment that are aimed to better understand the individual contribution of HRLB^2's modules to the human-likeness and performance of [PERSON]'. For this experiment, we could collect 97 responses of people that participated as judges. Furthermore, the main results of this study are presented in Figures 8 and 9 and Table 4.

### Third-Person Turing Test for HRLB\({}^{\wedge}\)2

We chose to perform a third-person Turing test for bots to evaluate the modules of our architecture. The participants in this believability analysis are the next bots:

* [PERSON]': This is our bot that is designed to employ all the modules of HRLB\({}^{\wedge}\)2. * Bot 1: This bot employs all the modules of HRLB\({}^{\wedge}\)2; however, the exploration process is implemented in a random manner.