# Hierarchical Episodic Control
## 1 Introduction  
Training slowness has long been an inherent challenge in reinforcement learning [1]. However, reinforcement learning frameworks based on episodic memory have, to some extent, addressed this issue. Episodic reinforcement learning (ERL) [2, 3, 4] introduces a non-parametric memory mechanism in reinforcement learning, which relies on stored memory data for value function learning or decision making. This approach to some extent addresses the issue of the large sample requirements that were challenging for the first-generation deep reinforcement learning models [1]. In cognitive science and neuroscience research, episodic memory is a form of \"autobiographical\" subjective memory [5]. Episodic memory is a type of long-term memory, and, in neuroscience, memories lasting for more than two weeks are considered long-term. Episodic memory involves the recollection of personal experiences, events that occurred at a specific time and place. For instance, we often remember the content of a presentation from a previous meeting, which constitutes an episodic memory tied to a particular time, place, and individual's personal experience. Another type of long-term memory is semantic memory, which pertains to organized facts and is independent of time and space. For example, the memory that \"Nanjing is the capital of Jiangsu\" is a semantic memory. Semantic memories are stored in neural networks formed within the hippocampus, medial temporal lobe, and thalamus. The medial temporal lobe, including the hippocampus and the anterior temporal cortex, is involved in forming new episodic memories [5, 6]. Patients with damage to themedical temporal lobe struggle to remember past events. Patients with bilateral hippocampal damage exhibit significant impairment in forming new memories of both their experiences and new events. Damage to the anterior temporal cortex can lead to a lack of time or location information in memories. Some researchers believe that episodic memories are always stored in the hippocampus, while others suggest that the hippocampus stores them briefly before they are consolidated into the neocortex for long-term storage. The latter perspective is supported by recent evidence, suggesting that neurogenesis in the hippocampal region of adults might contribute to removing old memories and enhancing the efficiency of forming new ones. Recent research indicates that the hippocampus plays a significant role in value-based decisions (the term \"value-based decisions\" as referred to in the cited literature corresponds to model-free reinforcement learning). It supports adaptive functioning and serves value-based decisions, reflecting how memories are encoded and utilized. The principles of episodic memory offer a framework for understanding and predicting the factors influencing decision making. The hippocampus's influence on decision making can occur unconsciously, allowing for automatic and internal impacts on behavior. In value-based decision making, interactions between the hippocampus and the striatum, as well as other decision-related brain regions, like the orbitofrontal and prefrontal cortices, are crucial [5]. Hierarchical reinforcement learning (HRL) applies an \"abstraction\" mechanism to traditional reinforcement learning [7]. It decomposes the overall task into subtasks at different levels, allowing each subtask to be solved in a smaller problem space. The policies learned for these subtasks can be reused, thus accelerating the problem-solving process. In this article, we introduce hierarchical episodic control, which combines the advantages of episodic memory and the option-critic architecture [8] to further enhance the efficiency of reinforcement learning. The model utilizes a hierarchical implicit memory planning approach to estimate the value of counterfactual trajectories. It recursively traverses from the last step to the first step along trajectories, forming a hidden planning scheme within episodic memory.