# HRLB\({}^{\wedge}\)2: A Reinforcement Learning Based Framework for Believable Bots
## 4 HRLB^2: A Reinforcement Learning Based Framework for Believable Bots
### Heuristic Exploration Process
```
Input: a problem described in HRLB\({}^{\prime}\)2 and policies \(\pi_{i}^{*}\) Output:\(\langle T(s^{\prime}\mid s,a),R(s,a)\rangle\) \(s\leftarrow\) ObserveState() \(a\leftarrow\) SelectAction(\(s,\epsilon\)) if\(a\) is primitivethen  ExecuteAction(\(a\)) \(s^{\prime}\leftarrow\) ObserveState() \(v\gets R(s,a)\) \(\langle t,v\rangle\leftarrow\) UpdateM(\(s,a,s^{\prime},v\)) return\(\langle t,v\rangle\) else whilea is executingdo  ExecuteAction(primitive(\(a\))) \(s^{\prime}\leftarrow\) ObserveState() \(v\gets R(s,a)\) \(\langle t,v\rangle\leftarrow\) UpdateM(\(s,a,s^{\prime},v\)) return\(\langle t,v\rangle\)
```  
**Algorithm 3**Explore()```
Input: bot's current state \(s\) Output:\(a\) \(a\leftarrow\) using Equation (5) return\(a\)
```

**Algorithm 4**\(\text{SelectAction}(s,e)\)

### Online Planning Algorithm

The last component of HRLB\({}^{\prime}\)2 is intended to choose the best action \(M_{a}\) and playing style \(Mst_{i}\) to maximize the expected bot's reward. In Algorithm 5, the process to achieve this objective is shown. First, we observe the current state \(s\) and greedily select the best action \(M_{a}^{*}\) to perform according to the policy \(\pi_{i}^{*}(s)\). After performing action \(M_{a}^{*}\), we observe the environment again and compute the reward \(v=R(s,a)+R(s^{\prime})\) that the bot acquired, where \(R(s,a)\) is the reward for performing action \(a\) in state \(s\) and \(R(s^{\prime})\) represents the reward of reaching state \(s^{\prime}\) after completing action \(a\). ```
Input: a problem described in HRLB\({}^{\prime}\)2, policies \(\pi_{i}^{*}\) and action-value functions \(Q_{i}(s,a)\) Output: Updated \(Qst(s,a)\) \(Q\gets Qst_{i}(s,a)\) \(Qst\gets 0\) \(s\leftarrow\text{ObserveState}()\) \(a\leftarrow\pi_{i}^{*}(s)\) ifa is primitivethen  ExecuteAction(\(a\)) \(s^{\prime}\leftarrow\text{ObserveState}()\) \(v\gets R(s,a)+R(s^{\prime})\) \(Qst\gets Q+\eta[v-Q]\) return\(Qst\) else whilea is executingdo  ExecuteAction(primitive(\(a\))) \(s^{\prime}\leftarrow\text{ObserveState}()\) \(v\gets R(s,a)+R(s^{\prime})\) \(Qst\gets Q+\eta[v-Q]\) return\(Qst\)
```

**Algorithm 5**\(\text{Play}()\)

Then, with value \(v\), we proceed to update the action-value \(Q_{i}(s,a)\) of the corresponding playing style subtask \(M_{i}^{st_{j}}\). This update to the model is carried out by the next incremental learning rule

\[Q_{i}(s,a)\gets Q_{i}(s,a)+\eta[v-Q_{i}(s,a)] \tag{6}\]

where \(\eta\) is a constant value that represents the step-size of learning. This procedure empowers the bot with the ability to adapt to the playing style of its opponent. It is important to emphasize that this incremental learning rule only updates the action-value functions of the different playing styles. Since human opponents adapt their strategies to try to overcome our bot game plan, we believe that the proposed global learning rule improves the diversity of behaviors of our bot--both players are changing their fighting approach in real time. Besides, we believe that the diversity of behaviors is key to improve the human-likeness of a bot. ## 5 Experiment: Street Fighter IV

In this section, we explain in detail the design and implementation of a believable bot based on the HRLB\(\star\)2 architecture. Our bot was designed to play the fighting game _Street Fighter IV_, as shown in Figure 3. We assess its human-likeness with a third-person Turing. _Street Fighter IV as a Testbed for Believable Bots and Reinforcement Learning_

The testbed we chose for our HRLB\(\star\)2 architecture is the fighting game _Street Fighter IV_ (SFIV). This game is a particularly difficult challenge since its state and action spaces are extensive. In addition, we should consider that it is a fast-paced game which leaves only 50-100 ms to make a decision. Moreover, SFIV is an imperfect and incomplete information game. Therefore, from a machine-learning standpoint, SFIV provides an excellent testbed for fast planning under uncertainty and cognitive skills based on the Theory of Mind [40].