# HRLB\({}^{\wedge}\)2: A Reinforcement Learning Based Framework for Believable Bots
## 4 HRLB^2: A Reinforcement Learning Based Framework for Believable Bots
### Heuristic Exploration Process
Consequently, a well-designed hierarchical task is vital to achieve effective behaviors and tractable MDPs to solve. Next, we present an overview of the designed subtasks to build the task graph for MarK':

* _PrimitiveActions_: These ten actions are positioned in the lowest-level of the hierarchy. The eight-position joystick is used to control the movement of the NPC, while the rest of the buttons execute normals. * \(Normal(n)\), \(Cover(c)\), \(GoTo(x)\), \(JumpA(a,y)\), \(Special(s)\): Here, we have five different multi-step actions that we categorize as low-level subtasks. \(Normal(n)\) has the objective of performing the normal specified in the parameter \(n\). Subtask \(Cover(t)\) is intended to block opponent's attacks and takes parameter \(t\) as input, which represents the attack that the bot is about to receive so it can properly defend against it. \(GoTo(x)\) takes the bot to the specified position \(x\). \(JumpA(a,y)\) has the goal of performing a jump attack; therefore, the parameters \(\langle a,y\rangle\) represent the attack to perform and the elevation at which it has to be executed, respectively. Lastly, we have subtask \(Special(s)\), which executes the special attack \(s\). Special attacks (specials) are more powerful than normals but also riskier and slower. All low-level subtasks only consider the state variables of the bot itself. * \(AntiAir\), \(OppOTG\), \(Stunt\): Here, we have three different multi-step actions that we define as high-level subtasks. These high-level subtasks have in common that all of them are implemented in only one playing style (Neutral). \(AntiAir\) specializes in defending against opponent's jump attacks, that is attacks that are performed while the character is in a jump state. \(OppOTG\) is activated when the opponent is laying on the ground. Similarly, \(Stunt\) subtask activates when the opponent is in a stunt state. All high-level subtasks are designed to focus on specific situations that the bot may encounter; thus, they can ignore variables that are not relevant to accomplish their goal. * _CloseRange_, _LongRange_, _BotOTG_: These high-level subtasks differ from the last ones because they are implemented in two different playing styles. As we can see in Figure 4, these multi-step actions have two variants: defensive and aggressive. _CloseRange_ subtask focuses on viable strategies in a close range, while _LongRange_ only considers effective long-range scenarios. _BotOTG_ is activated when the bot is hit to the ground. Again, these kinds of subtasks ignore variables that are not relevant to accomplish their goal. * _Defensive_, _Neutral_, _Aggressive_: Here, we have the subtasks that specify the playing style of their child actions. The _Defensive_ playing style is intended to produce more cautious strategies for the bot. This results in an agent that is more passive and prefers to keep a longer distance between it and its opponent. Therefore, the agent spends more time performing subtask _LongRange_. On the other hand, the _Aggressive_ playing style favors strategies that lead to giving damage to the opponent, regardless of how risky it may be. Thus, this version of the bot adopts more often attacks over defensive options from the subtask _CloseRange_. All the variables that use the child subtasks of each playing style are relevant. * _Root_: This is the root task of the bot, that is the complete problem in a flat representation. Therefore, this MDP must consider all state variables. With the design of the hierarchical decomposition for MarK', we could proceed with the _learning by observation_ procedure explained in Section 4.3. ### Learning by Observation

The _learning by observation_ process is aimed to learn the model of SFIV. In particular, this process was achieved by observing how our expert played SFIV against the built-in AI in the game at the available difficulty levels of 6-8. We selected this level of difficulty because, according to our expert, the built AI has exhibited the most human-like behavior in this configuration.