# Hierarchical Episodic Control
## 3 Hierarchical Episodic Control
### Theoretical Analysis
#### 3.3.2 Convergence Analysis
The option-critic algorithm exhibits some imbalance among options, where a single option is consistently preferred by the policy. This imbalance is more pronounced when using tabular value functions, leading to the degradation of the option-critic algorithm into a regular actor-critic algorithm. However, this imbalance is mitigated when using neural networks for value function approximation. Due to the shared information among options, the robustness of learning strategies on both the environment's stochasticity and option policy learning process is improved, allowing the OptionEM algorithm to achieve a balanced performance based on state space separation. The Figure 3 illustrates the option intentions of the OptionEM algorithm and the option-critic algorithm during the testing phase. In order to effectively showcase the learning process of different options, we have chosen to set the game's background to a white color. In the figure, the green color represents the target location, and the blue portion indicates the current option's position and distribution. It can be observed that the option-critic algorithm exhibits a noticeable imbalance in the use of options, which might lead to the degradation of the option-critic algorithm into an actor-critic algorithm. The options learned by the OptionEM, using the twin network mechanism, are more balanced compared to the option-critic algorithm. However, there still exists some degradation in option 0 within OptionEM. Figure 3: Samples of the options in the four-room game. ### Mujoco

MuJoCo [19] (multi-joint dynamics with contact) is a physics simulator for research and development in robotics, biomechanics, graphics and animation, machine learning, and other fields that require fast and accurate simulation of the interaction of articulated structures with their environment. The default Mujoco games do not effectively demonstrate the subgoal or option characteristics of hierarchical reinforcement learning. Therefore, in this section, the Ant-Maze and Ant-Wall environments are used for training and testing. The Ant-Maze game environment utilizes open-source code from GitHub (environment source code: [[https://github.com/kngwyu/mujoco-maze.git](https://github.com/kngwyu/mujoco-maze.git)]([https://github.com/kngwyu/mujoco-maze.git](https://github.com/kngwyu/mujoco-maze.git)), accessed on 27 August 2023), while the Ant-Wall game environment is custom-built, featuring randomly initialized walls. In addition to training actions for the ant agent itself, the agent also needs to learn path planning strategies to navigate through mazes or around obstacles/walls. This represents a typical scenario for hierarchical reinforcement learning. Table 2 presents the parameter settings for the Mujoco games in this section. When the number of options is set to 2, it is evident that, for both the Ant-Maze and Ant-Wall environments, the agent employs one option when it is far from obstacles/walls and another option when it approaches obstacles/walls. Learning methods based on options provide valuable insights for the agent's long-term learning/planning in such scenarios. Observing the similarity in path planning between the Ant-Wall and Ant-Corridor games, an attempt was made to transfer the model trained in the Ant-Wall environment to the Ant-Corridor environment. Figures 4 and 5 present the qualitative results of the agent's learned option categories. In this experiment, the performances from six independent runs were averaged. Additionally, the OptionEM algorithm was compared with two option-based algorithm frameworks, and the results for different numbers of options were compared. In Table 3, it is noteworthy that when using eight options the performance eventually drops significantly. This could be attributed to the lower-dimensional environment of the Mujoco experiments and the simplicity of the environment, making such a high number of options unnecessary and decreasing the sample efficiency. One way to validate this is to apply the same algorithm in a continuous learning environment.