Prioritized memory access explains planning and hippocampal replay

1Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA. 2Department of Psychology, Princeton University, Princeton, NJ, USA. *e-mail: mmattar@princeton.edu A hallmark of adaptive behavior is the effective use of experience to maximize reward1. In sequential decision tasks such as spatial navigation, actions can be separated from their consequences in space and time. Anticipating these consequences to choose the best actions thus often requires integrating multiple intermediate experiences from pieces potentially never experienced together2,3. For instance, planning may involve sequentially retrieving experiences to compose a series of possible future situations4,5. Recent theories suggest that humans and animals selectively engage in such prospective planning as appropriate to the circumstances and suggest that omitting such computations could underlie habits and compulsion6–8. However, by focusing only on whether or not to deliberate about the immediate future, these theories largely fail to address how the brain chooses which of the many possible experiences to consider during such evaluation, a selection process that ultimately governs which decisions are made. In addition to prospective planning, behavioral and neuroimaging data suggest that actions can also be evaluated by integrating experiences before decisions are needed. Indeed, future decisions can be predicted not only from prospective neural activity5 but also from neural reinstatement when relevant information is first learned9 and during subsequent rest10,11 (Fig. 1a). Yet this further highlights the selection problem: if actions can be evaluated long before they are needed, which experiences should the brain consider at each moment to set the stage for the most rewarding future decisions? Addressing this question requires a more granular theory of memory access for action evaluation, which takes forward planning as a special case of general value computation. A window into patterns of memory access is offered by the hippocampus12. During spatial navigation, hippocampal place cells typically represent an animal’s spatial position, though they can also represent locations ahead of the animal during movement pauses13–15. For instance, during ‘sharp wave ripple’ events, activity might progress sequentially from the animal’s current location toward a goal location14,15. These ‘forward replay’ sequences predict subsequent behavior and have been suggested to support a planning mechanism that links actions to their consequences along a spatial trajectory15. However, this pattern is also not unique: activity in the hippocampus can also represent locations behind the animal14,16–19 and can even represent altogether disjoint, remote locations (particularly during sleep20,21; Fig. 1a). Collectively, these three patterns (forward, reverse, and offline replay) parallel the circumstances discussed above, in which reinstatement in humans predicts choice. The various patterns of hippocampal replay have been suggested to support a range of distinct functions such as planning13,15, learning through credit assignment16,19,22, memory retrieval23,24, consolidation23,25, and forming and maintaining a cognitive map18. Yet we still lack a theory describing how these various functions of replay come together to promote adaptive behavior and predicting which memories are replayed at each time and in which order. To address this gap, we develop a normative theory to predict not just whether but which memories should be accessed at each time to enable the most rewarding future decisions. Our framework, based on the Dyna reinforcement learning (RL) architecture26, views planning as learning about values from remembered experiences, a perspective that generalizes and reconceptualizes work on tradeoffs between model-based and model-free controllers6,7. We derive, from first principles, the utility of retrieving each individual experience at each moment to predict which memories a rational agent ought to access to lay the groundwork for the most rewarding future decisions. This utility is formalized as the increase in future reward resulting from such memory access and is shown to be the product of two terms: a gain term that prioritizes states behind the agent when an unexpected outcome is encountered and a need term that prioritizes states ahead of the agent that are imminently relevant. Notably, this theory at present investigates which of all possible memories would be most favorable to access in principle; it is not intended as (but may help point the way toward) a mechanistic or process-level account of how the agent might efficiently find them. To test the implications of our theory, we simulate a spatial navigation task in which an agent generates and stores experiences that can be later retrieved. We show that an agent that accesses memories in order of utility produces patterns of sequential state reactivation that resemble place cell replay, reproducing qualitatively and with Prioritized memory access explains planning and hippocampal replay