# Hierarchical Episodic Control
## 2 Related Work
### Hierarchical Reinforcement Learning  
Reinforcement learning improves policies through trial-and-error interaction with the environment. Its characteristics of self-learning and online learning make it an essential branch of machine learning research. Typical reinforcement learning algorithms represent behavior policies using state-action pairs, leading to the \"curse of dimensionality\" phenomenon where the number of learning parameters exponentially grows with the dimensionality of the state variables. Traditional methods to tackle the curse of dimensionality include state clustering, finite policy space search, value function approximation, and hierarchical reinforcement learning. Hierarchical reinforcement learning introduces an \"abstraction\" mechanism to traditional reinforcement learning by decomposing the overall task into subtasks at different levels [7; 12; 13]. Each subtask is solved in a smaller problem space, and the policies learned for subtasks can be reused, accelerating the problem-solving process [14]. The option is a hierarchical reinforcement learning approach proposed by [PERSON] [15], which abstracts learning tasks into options. Each option can be understood as a sequence of actions executed with a certain policy defined on a specific substate space to complete a subtask. Each action can be either a basic primitive action or another option. Hierarchical control structures are formed by invoking lower-level options or primitive actions through higher-level options. In the hierarchical reinforcement learning system, these options are added as a special kind of \"action\" to the original action set. Options can be predetermined by designers based on expert knowledge or generated automatically. Consider the call-and-return option execution model, where the agent first selects an option \(\omega\) based on the option policy \(\pi_{\Omega}\) then selects an action policy based on the option's intra-policy \(\pi_{\omega}\), and when the termination function value \(\beta_{\omega}\) equals 1, the option terminates, and the control returns to the higher-level policy to select a new option. The parameters of the intra-policy and termination function of option \(\omega\) are parameterized as \(\pi_{\omega,\theta}\) and \(\beta_{\omega,\eta}\), where \(\theta\) and \(\eta\) are trainable model parameters [8]. The update of the policy is performed using policy gradients, and the return value is denoted as \(\rho(\Omega,\theta,\eta,s_{0},\omega_{0})=\mathbb{E}_{\Omega,\theta,\omega} \left[\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}\mid s_{0},\omega_{0}\right]\). Compared to the original policy gradient, here \((s_{0},\omega_{0})\) is used instead of \(s_{0}\), where \((s_{0},\omega_{0})\) is an augmented state corresponding to \(s_{0}\). Within the option-critic framework [8], various value functions are defined. The option-state value function is defined as corresponding to \(V(s)\), and the expected return when selecting option \(\omega\) in state \(s\) is as follows:  
\[Q_{\Omega}(s,\omega)=\sum_{a}\pi_{\omega,\theta}(a\mid s)Q_{U}(s,\omega,a) \tag{1}\]  
The state-option pairs \((s,\omega)\) can be seen as an augmented state space, and \(Q_{U}:\mathcal{S}\times\Omega\times\mathcal{A}\rightarrow\mathbb{R}\) represents the action-value function within the option. The intra-policy action-value function (corresponding to \(Q(s,a)\)) when choosing action \(a\) in state \(s\) and option \(\omega\) is as follows:  
\[Q_{U}(s,\omega,a)=r(s,a)+\gamma\sum_{s^{\prime}}\mathrm{P}\big{(}s^{\prime}\mid s,a\big{)}U\big{(}\omega,s^{\prime}\big{)} \tag{2}\]  
Here, \(U:\Omega\times\mathcal{S}\rightarrow\mathbb{R}\) represents the option value function upon arrival, analogous to \(V(s^{\prime})\) in standard reinforcement learning.