# Hierarchical Episodic Control
## 3 Hierarchical Episodic Control
### Theoretical Analysis
#### 3.3.2 Convergence Analysis
Compared with S1 SP1, S2 MP2 changes the map, target, and path at the same time, and the generalization results of all three algorithms for this are slightly insufficient compared to the previous two but can still track the target relatively stably, and the model has some generalization potential, which may need to be improved by migration learning or more environmental enhancements. In most cases, the trackers trained by the bootstrappedEM algorithm and the OptionEM algorithm outperform those trained by A3C. The experimental results of the OptionEM(4) algorithm outperformed OptionEM(8), and the performance of the model was instead reduced with eight options, which also indicates that four options is a more appropriate setting in tracking scenarios. Figure 7 shows the distribution of options learned by the OptionEM algorithm when the number of options is set to four. Observation reveals that the distribution of options can be roughly described as follows: When the target has its back to the tracker and is at a distance, the option with purple marking is selected; when the target has its back to the tracker and is at a closer distance, the option with yellow marking is selected; and when the target is not in the screen or is facing the tracker, there are two possible options. The option with orange marking bootstrapped the tracker to perform a self-rotating motion, while the option with blue marking appears more randomly and the action has no obvious law to be summarized. The model options trained by OptionEM are able to distinguish more clearly between the situation where the target is in the field of view and the situation where the target is lost; when the tracker loses the target in its field of view, the trained options can control the tracker to perform a rotational movement in order to find the lost target as soon as possible. Overall, the distribution of these four options is somewhat consistent with the logic of humans conducting target tracking in real scenes. Table 5 displays OptionEM alongside other baselines in the UE4-based active tracking task, showcasing the time required to train an agent to achieve a cumulative reward of 800. In comparison to the classical A3C algorithm and other episodic memory algorithms, the proposed OptionEM algorithm demonstrates improved efficiency. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Environment** & **A3C** & **BootstrappedEM** & **OptionEM(4)** & **OptionEM(8)** \\ \hline S1 SP1 & \(2495.7\pm 12.4\) & \(2653.1\pm 2.7\) & \(\mathbf{2791.4\pm 20.1}\) & \(2488.1\pm 3.0\) \\ S1 MP1 & \(2106.0\pm 29.3\) & \(2345.8\pm 21.6\) & \(\mathbf{2539.4\pm 23.1}\) & \(2142.4\pm 71.1\) \\ S1 SP2 & \(2162.5\pm 48.5\) & \(2401.0\pm 33.8\) & \(\mathbf{2519.5\pm 54.0}\) & \(2097.3\pm 45.8\) \\ S2 MP2 & \(740.0\pm 577.4\) & \(1624.9\pm 11.3\) & \(\mathbf{1741.2\pm 21.4}\) & \(1672.4\pm 42.6\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Rewards in experiments on UE4 tracking. Figure 6: UE4 environment and settings. Source image: End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning [23]. ## 5 Summary

In this article, we introduce a hierarchical episodic control model, which employs a hierarchical implicit memory planning approach for value estimation of counterfactual trajectories. The planning process starts from the last step and recursively moves backward along the trajectory, forming an implicit plan in the episodic memory. Experiences are aggregated both along trajectories and across trajectories, and the model is updated using a multi-headed backpropagation mechanism similar to bootstrapped neural networks. This model extends the episodic memory framework to the field of hierarchical reinforcement learning and is theoretically analyzed to demonstrate its convergence and effectiveness. The results from various experiments, including the four-room game, Mujoco, and UE4 target tracking, indicate that the hierarchical episodic control model effectively enhances training efficiency.