# HRLB\({}^{\wedge}\)2: A Reinforcement Learning Based Framework for Believable Bots
## 4 HRLB^2: A Reinforcement Learning Based Framework for Believable Bots
### Heuristic Exploration Process
Furthermore, creating believable bots for SFIV represents an especially challenging task. This difficulty arises from the fact that both players are always on screen in fighting games, making it harder to maintain the illusion of an agent being controlled by a human player. In contrast, in the FPS named _Unreal Tournament_--the testbed for the BotPrize Competition [41]--the judges and players participating in the Turing test only have a few moments to examine how their opponents play and react to the environment. Therefore, SFIV represents a more advanced challenge for the creation of believable bots. ## 6 Mark': A HRLB\(\star\)2-Based agent for Street Fighter IV

The proposed case study for our architecture HRLB\(\star\)2 is the design and implementation of a bot with the objective of playing SFIV in a human-like manner. We called our bot [PERSON]; we chose this name after _[PERSON]_ and a fighting game character named \(K^{\prime}\). Since SFIV is a highly complex domain, as a first step to approach the creation of believable characters through reinforcement learning, we focused on learning how to control, and play against, one particular character--_[PERSON]_. This solution reduced the action space to 70. Additionally, to reduce the state space, we used a coarser discretization method for the variables that track the position of the characters on screen. Despite these simplifications, we still faced a problem with a upper bound for the state space of \(10^{1310}\). We computed SFIV state space complexity, using the number of possible states (512) and the number of cells (210), as \(\log_{10}(512^{210})\)--according to our discretization method that we present in Section 6.1. This simplified version of the state space of SFIV is still much larger than the state space of Go (\(10^{170}\)) or Chess (\(10^{47}\)). To apply HRLB\(\star\)2 to SFIV, we must first provide all the variables and subtasks that are needed to build the task hierarchy for the domain at hand. Figure 3: A snapshot of _Street Fighter IV_. ### Variables

In a fighting game such as SFIV, all the variables that represent the environment are associated with the features that define the state of both characters on screen. Since we considered the opponent as part of the environment, most of the variables that we present below have a variant for each of the characters:

* Position: The perception of the position of the NPCs is key to play a fighting game. In particular, [PERSON]' has to perceive the position of both agents in the vertical and horizontal axes. * Movement: Similar to the last variables, it is important to incorporate knowledge about the direction of the NPCs' movement in the horizontal and vertical axes. * Projectile: This variable indicates if the opponent's projectile is getting closer, getting farther or not moving, according to the other player perspective. * Bars: There are three different bars in the game that quantify stats of the NPCs. The first measures the health level of the characters. Then, we have variables that gauge the amount of energy that can be used for special moves. * Frame Data: Any of the actions in the game are not instantaneous and, during their execution time, they pass through three different phases: _start-up_, _active_ and _recovery_. We discretized the phase of each move using these three variables. * Attacks: In this category, we include all necessary variables to interpret the actions a NPC can execute. As our experiment limits to play with/against _[PERSON]_, we have 62 character specific actions. ### Hierarchical Decomposition

In Figure 4, we present the task graph for MarK'. This hierarchical decomposition was proposed by our expert, who is also the first author of this paper. Therefore, this task graph incorporates domain knowledge that exploits state abstractions of individual MDPs within the hierarchy. With this procedure, we can significantly reduce the complexity of subtasks by ignoring parts of the state space that are not relevant to accomplish its goal [27].