# HRLB\({}^{\wedge}\)2: A Reinforcement Learning Based Framework for Believable Bots
## 4 HRLB^2: A Reinforcement Learning Based Framework for Believable Bots
### Heuristic Exploration Process
* Bot 2: This bot is exactly the same as [PERSON]', but without the online planning algorithm that adapts the playing style of the bot in real-time. * Bot 3: This bot is exactly the same as [PERSON]', but without the Boolean function that improves the use of anti-air techniques. * Bot 4: This bot is implemented with a random exploration process, without the playing style adaptation module and without the Boolean function. * Bot 5: This bot is implemented with our proposed heuristic exploration process, without the playing style adaptation module and without the Boolean function. We recorded two different videos per each participant in this Turing test. In these videos, all participants played against the built-in AI level 6 (CPU 6). Therefore, this Turing test included 12 unique samples of bot behaviors. We published our survey on specialized channels of the FGC, to ensure the respondents have a strong background in fighting games. Before evaluating the behavior of the participants, we explained to the respondents that in all videos a bot was playing against the CPU 6. Thereafter, we presented, in a random manner, two matches from our set of videos. After the completion of the two matches, we asked the respondent the obligatory fixed-choice questions:

* Which bot played in a more human-like manner against the CPU? with choices: Bot in first video, and Bot in second video
* How would you assess the fighting skill level of the bot in the first video? with choices: Beginner, Intermediate, Advanced and Professional

Figure 9: Evaluation of HRLB\({}^{\wedge}\)2 violin plot. * How would you assess the fighting skill level of the bot in the second video? with choices: Beginner, Intermediate, Advanced and Professional In addition, we included the next optional open-ended question:
* If you could kindly provide more insight about the reasons that made you choose a bot as more human-like, please leave a comment below Finally, this Turing test finished with the following obligatory fixed-choice question:
* How would you assess your skills as a fighting game player? with choices: Beginner, Intermediate, Advanced and Professional We could collect 97 responses and the optional open-ended question was answered by 53 people. With these data, we proceeded to analyze the differences in human-likeness, and skill level, of all the implemented bots. ### Results of the Third-Person Turing Test for HRLB\({}^{\wedge}\)2

In the same manner as presented in Section 7.2, we computed the human-likeness ratios for the six bots in the believability test; these ratios are presented in Figure 8. Furthermore, we applied the same statistical significance analyses presented in Section 7.2. We report the estimated p-values and CIs in Table 3. In regards to performance, we present in Figure 9 a violin plot for each participant in this survey. In addition, this plot includes the self-assessment skill level of the judges. Although [PERSON]' and Bot 3 got the highest human-likeness ratios, the performed significance analyses do not find a significant difference against the rest of bots in the survey for HRLB\({}^{\wedge}\)2. Nevertheless, the effect size is large when comparing [PERSON]' against Bot 2, Bot 4 and Bot 5. Therefore, this might imply that the online planning algorithm--described in Section 4.5--is the module that contributes the most to the believability of our HRLB\({}^{\wedge}\)2 bot--[PERSON]'. It is important to notice that, even if the human-likeness ratios are similar among the participants, the perceived skill level varies between the bots. By visually inspecting the violin plots in Figure 9, there is a noticeable difference between the density distributions of the bots. In addition, to better understand the magnitude of the variation in the skill level between the bots, we performed an effect size analysis and confidence intervals. The results of this analysis are presented in Table 4. A major feature to notice is that only Bot 3 has a comparable skill level to [PERSON]'.