# HRLB\({}^{\wedge}\)2: A Reinforcement Learning Based Framework for Believable Bots
## 4 HRLB^2: A Reinforcement Learning Based Framework for Believable Bots
The learning process of the model was divided into 2-h intervals. After an interval is completed, all MDPs \(M_{i}\) are solved through SPUDD. Then, for each interval, we evaluated the performance of the computed policies \(\pi_{i}^{*}\) for all subtasks in the task graph. [PERSON]' performance was estimated as the difference between the damage that it dealt and received in 10 rounds, fighting against the built-in AI level 6. Figure 4: The task graph for our HRLB\(\times\)2-based agent that plays _Street Fighter IV_. The first four epochs shown in Figure 5 are the learning rates of [PERSON]' over the 8-h period of learning by observation. We stopped the learning procedure after the fourth interval since the learning rate of our bot seemed to start slowing down. Next, we continued with the heuristic exploration process. ### Heuristic Exploration Process

The objective of following the heuristic exploration process, defined in Section 4.4, is to refine the previously computed model of SFIV by letting our bot to explore the environment by itself. In a similar fashion to the learning by observation process, the heuristic exploration process was divided into 2-h intervals. In addition, we computed the performance of our bot in the manner explained in Section 6.3. As we can see in Figure 5, from Epoch 5 to Epoch 12, there were two box plots per epoch. The box plots on the right display the performance of [PERSON]' in each epoch, while the box plots on the left represent the same measure for a bot that uses a random exploration process. For both bots, the total time spent in exploration was 16 h, or eight epochs. We stopped the exploration process at this point because the performance of [PERSON]' made a substantial improvement from the seventh to the eighth epoch. In the next subsection, we explain the composition of the reward functions that aim to foster specific behavior traits to generate diverse playing styles. ### Reward Functions

A well-designed reward function is key to achieve the desired behavior for our bot. Although there is no accepted definition of a proper design of reward functions, it is better when a reward function is kept straightforward. If a reward function remains simple enough, we can potentially use it for different problems, which is favorable for benchmarks. Consequently, our reward function only considers three universal elements in 2D fighting games that are essential to evaluate the performance of a character:

Figure 5: Box plots of the learning rates of three different bots. * Health: Health bars are virtually a must in fighting games since they indicate the remaining stamina for each character on screen. When a character runs out of stamina, it loses. The variables we use to represent the rewards of this element are: dealt damage by [PERSON]' (\(R_{D+}(s^{\prime})\)) and taken damage by [PERSON]' (\(R_{D-}(s^{\prime})\)). * Cover: Even though dealing damage is crucial to win, covering is an effective technique for reducing the amount of received damage. The variables we use to represent the rewards of this element are: attack covered by [PERSON]' (\(R_{C-}(s^{\prime})\)) and attack covered by opponent (\(R_{C+}(s^{\prime})\)). * Positioning: Keeping the right distance to your opponent is a fundamental ability to take advantage of the specific set of skills of each character. To cope with the requirements described above, the design of our reward functions is based on a combination of shaped and sparse rewards. Sparse rewards are appropriate to represent _health_ and _cover_ elements. Table 1 displays the neutral reward function \(R(s^{\prime})\) and the corresponding shaping functions \(F(s^{\prime})\) to create the fighting style described in Section 4.2. On the other hand, shaped rewards are necessary to describe a good positioning--we give increasing reward \(r\) in ranges that are closer to the optimal position for fighting. We use an exponential function to define shaped rewards.