# Hierarchical Episodic Control
## 3 Hierarchical Episodic Control
### Theoretical Analysis
#### 3.3.2 Convergence Analysis
: _The value functions \(M(s,\omega,a)\) and \(M_{\Omega}(s,\omega)\) computed from Algorithm 1 satisfy the following inequalities:_

\[s\in\mathcal{S},\omega\in\Omega,a\ \in\mathcal{A},M^{*}(s,\omega,a)\leq M(s, \omega,a)\leq M_{\max}(s,\omega,a) \tag{15}\]

\[s\in\mathcal{S},\omega\in\Omega,M^{*}_{\Omega}(s,\omega,a)\leq M_{\Omega}(s, \omega)\leq M_{\Omega,\max}(s,\omega) \tag{16}\]

\(s\) _satisfies Theorem 2._

**Theorem 3**. : _In the nearly deterministic environment, the value function of OptionEM satisfies the following:_

\[V^{\pi}(s)\geq V^{*}(s)-\frac{2\mu}{1-\gamma},s\in\mathcal{S} \tag{17}\]

Theorem 3 ensures the applicability of the OptionEM method in nearly deterministic environments, which closely resemble real-world scenarios.The proof can be seen in Appendix C.

## 4 Experimental Results and Analysis

### Four-Room Game

In the classic four-room reinforcement learning environment, the agent navigates a maze of four rooms connected by four gaps in the walls. In order to receive a reward, the agent must reach the green target square. Both the agent and the goal square are randomly placed in any of the four rooms. The environment is contained in the mini-grid [17] library, which contains a collection of 2D grid world environments that contain goal-oriented tasks. The agents in these environments are red triangular agents with discrete action spaces. These tasks include solving different maze maps and interacting with different objects (e.g., doors, keys, or boxes). The experimental parameter settings for the four-room game are presented in Table 1.

In the four-room game environment, a single-layer fully connected neural network is employed to estimate the value function. The input dimension is 3, the hidden layer has 100 units, and the output dimension is 4. The neural network is trained using the error backpropagation method. Based on [PERSON]'s work [18], five different values of \(\eta=0.2,0.3,0.5,0.7,1.0\) were tested. The experimental results indicated that the best performance was achieved with \(\eta=0.3\). Therefore, this parameter value is used directly in this study. In Figure 2, assuming that options have been provided and are kept fixed throughout the learning process, the only element being updated is the option-value function \(\mathcal{Q}_{\Omega}(s,\omega)\). The option policy uses this function for selecting among options. The multi-option update rule from Equation (3) is compared to the case of updating only the sampled option in the state. Four options designed by [PERSON] et al. are used in this scenario. In this case, the OptionEM method achieves a significantly better sample efficiency, demonstrating how the proposed method accelerates learning the utility of a suitable set of options. For the case of learning all option parameters, the learning process is conducted using Equation (4). In this setting, the OptionEM method is compared against the option-critic (OC) algorithm and the actor-critic (AC) algorithm. Both the OC and AC algorithms outperform the baseline algorithm in terms of hierarchical agent performance. However, the agent is able to achieve similar performance to the OC algorithm in approximately half the number of episodes. Additionally, it is noteworthy that OptionEM exhibits lower variance across different runs, which is consistent with the anticipated reduction in variance effect of the expected updates from prior research. Figure 2 displays five paths taken \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline AC learning rate & \(2\times 10^{-1}\) \\ OC learning rate & \(8\times 10^{-1}\) \\ OptionEM learning rate & \(8\times 10^{-1}\) \\ \(\eta\) & 0.3 \\ Number of episodes & 500 \\ Max steps per episode & 1000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Experimental parameter settings for four-room game. Figure 2: Four-room game experiments. by the agent in the game, with fixed start and goal points. Training and exploration processes showcase the comparison experiments between OptionEM and the option-critic algorithm.