# Hierarchical Episodic Control
## 3 Hierarchical Episodic Control
### Theoretical Analysis
#### 3.3.2 Convergence Analysis
Furthermore, due to the distinct update rules of the OptionEM algorithm compared to OC baseline and flexible OC algorithms, it can effectively utilize limited sample data, resulting in a better performance. ### _UE4-Based Active Tracking_

The UE4-based target tracking experiment is mainly designed to verify the generalization of the episodic memory module. A highly realistic virtual environment based on Unreal Engine 4 (UE4) is used for independent learning. The virtual environment has a three-layer structure. The bottom layer consists of simulation scenes based on Unreal 4, which contains a rich variety of scene instances, and provides a general communication interaction interface based on UnrealCV [20], which realizes the communication between the external program and the simulation scene. The agent-environment interaction interface is defined by the specific task, along with the relevant environment elements, such as the reward function, action-state space, etc. The interaction interface design specification is compatible with the OpenAI Gym [21] environment. The agent actively controls the movement of the camera based on visual observations in order to follow the target and ensure that it appears in the center of the frame at the appropriate size. A successful tracking is recorded only if the camera continues to follow the target for more than 500 steps. During the tracking process, a collision or disappearance of the target from the frame is recognized as a failed tracking. Accurate camera control requires recognition and localization of the target and reasonable prediction of its trajectory. Each time the environment is reset, the camera will be placed anywhere in the environment and the target counterpart will be placed 3 m directly in front of the camera. The specific tracker reward is as follows:

\[r=A-\left(\frac{\sqrt{x^{2}+(y-d)^{2}}}{c}+\lambda|\omega|\right) \tag{18}\]

For \(A>0,c>0,d>0,\lambda>0\), \(c\) is set as a normalized distance. The environment defines the maximum reward to a position where the target is directly in front of the tracker (the tracker is parallel to the target character's shoulders) at a distance of \(d\). With a constant distance, if the target rotates sideways, the tracker needs to turn behind the target to obtain the maximum reward. In order to enhance the generalization ability of the model, it is necessary to increase the diversity of the surrounding environment and the target itself as much as possible during the training process. In response to the diversity of the surrounding environment, the environment enhancement in UE4 can be used to easily modify the texture, lighting conditions, and other elements of the surrounding environment, specifically, randomly selecting images from a common image dataset deployed on the surface of the environment and objects to modify the texture and randomly deploy the position, intensity, color, and direction of the light source in the environment to change the scene lighting conditions. Figure 5: Mujoco Ant-Wall game. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithms** & **2 Options** & **4 Options** & **8 Options** \\ \hline OC & \(343(72)\) & \(291(82)\) & \(207(61)\) \\ Flexible OC & \(601(93)\) & \(\mathbf{582(73)}\) & \(539(86)\) \\ OptionEM & \(\mathbf{612(110)}\) & \(571(102)\) & \(\mathbf{558(97)}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Different numbers of options in Mujoco Ant-Maze game. The randomization of texture and illumination prevents the tracker from overfitting the specific appearance of the target and background. The diversity requirements of the target itself can be realized by varying the target's trajectory and speed of movement. The start and end position coordinates of the target can be randomly generated, and the corresponding trajectories are generated using the UE4 engine's built-in navigation module (the trajectories generated by the built-in navigation module automatically avoid obstacles and do not crash into walls).