Author response: The roles of online and offline replay in planning
2020-05-09

Article Figures and data Abstract eLife digest Introduction Results Discussion Materials and methods Data availability References Decision letter Author response Article and author information Metrics Abstract Animals and humans replay neural patterns encoding trajectories through their environment, both whilst they solve decision-making tasks and during rest. Both on-task and off-task replay are believed to contribute to flexible decision making, though how their relative contributions differ remains unclear. We investigated this question by using magnetoencephalography (MEG) to study human subjects while they performed a decision-making task that was designed to reveal the decision algorithms employed. We characterised subjects in terms of how flexibly each adjusted their choices to changes in temporal, spatial and reward structure. The more flexible a subject, the more they replayed trajectories during task performance, and this replay was coupled with re-planning of the encoded trajectories. The less flexible a subject, the more they replayed previously preferred trajectories during rest periods between task epochs. The data suggest that online and offline replay both participate in planning but support distinct decision strategies. eLife digest Studies show that humans and animals replay past experiences in their brain. To do this, the brain creates a pattern of electrical activity for each part of a multistep experience and then plays them back in order. Humans and other animals can replay scenarios either while the experience is still happening (i.e. online replay) or later when they are resting or sleeping (i.e. offline replay). Being able to replay an experience and its outcome may help a person or animal plan a better course of action in the future. However, it is poorly understood how online and offline replay each contribute to such planning. To answer this question, Eldar et al. used a brain imaging tool called magnetoencephalography (MEG for short) to measure the electrical activity inside the brain. This technique was able to detect replays in the brain of individuals performing a particular task, and later whilst they were resting. In the experiments, 40 healthy volunteers played a game in which each location in a space was associated with an image, for example a frog or a traffic sign, and each image was given a value. Participants got paid for moving to more valuable images in one or two steps. Eldar et al. found that people who replay their steps during a task are able to adjust their choices on the fly, whereas individuals who replay their choices during rests tend to approach a task with a less flexible, more preformed plan. Eldar et al. suggest that replaying an experience too much during rest and not enough in real-time might contribute to more rigid behaviors, a theory that could shed light on the mechanisms behind certain behavioral disorders such as obsessive compulsive disorder. However, more studies are needed to determine if these two different replay strategies play a causal role in human behavior. Introduction Online and offline replay are both suggested to contribute to decision making (Behrens et al., 2018; Diba and Buzsáki, 2007; Foster, 2017; Foster and Wilson, 2006; Gupta et al., 2010; Ji and Wilson, 2007; Kurth-Nelson et al., 2016; Louie and Wilson, 2001; Ólafsdóttir et al., 2017; Pezzulo et al., 2014; Skaggs and McNaughton, 1996; Ólafsdóttir et al., 2015; Stachenfeld et al., 2017), but their precise contributions remain unclear. Replay of experienced and expected state transitions during a task, either immediately before choice or following outcome feedback, is particularly well suited to mediate on-the-fly planning, where choices are evaluated based on the states to which they lead (this is known as model-based planning). Off-task replay might serve a complementary role of consolidating a model of a state space, which specifies how each state can be reached from other states as well as the values of each state. According to this perspective, both types of replay help subjects make choices that are flexibly adapted to current circumstances. However, an alternative possibility is that off-task replay also directly participates in planning, by calculating and storing a (so-called model-free) decision policy that specifies in advance what to do in each state (Gershman et al., 2014; Mattar and Daw, 2018; Momennejad et al., 2018; Sutton, 1991). Such a pre-formulated policy is inherently less flexible than a policy that is constructed on the fly, but at the same time it decreases a need for subsequent online planning when time itself might be limited. Thus, rather than online and offline replay both supporting the same form of planning, this latter perspective suggests a trade-off between them. In other words, online replay promotes on-the-fly (model-based) flexibility, whereas offline replay establishes a stable (model-free) policy. Despite the wide-ranging behavioural implications of a distinction between model-based and model-free planning (Crockett, 2013; Everitt and Robbins, 2005; Gillan et al., 2017; Kurdi et al., 2019), and much theorising on the role of replay in one or the other form of planning, to date there is little data indicating whether online and offline replay have complementary or contrasting impacts in this regard. However, recent advances in magnetoencephalography (MEG) analysis have made it possible to study replay in human subjects in relation to learning and decision-making behaviour (Eldar et al., 2018; Kurth-Nelson et al., 2016; Liu et al., 2019). This methodology involves three key steps. First, MEG signals are recorded before, during, and after the subject performs a task of interest. Second, the MEG time series are decoded so as to estimate a moment-by-moment probability a subject is neurally representing each task element, even when the sensory processing associated with that element has ceased. This is the minimal requirement for replay. However, a special feature of replay is the coordination between the non-sensory neural representations of multiple, related, task elements. Thus, finally, the relationships between elements' representational probability time series are examined to determine whether pairs of elements tended to be represented sequentially, one after the other. Here, we use this methodology to test the relationship between both online and offline replay and key aspects of decision flexibility that dissociate model-free and model-based forms of planning (Daw et al., 2011). For this purpose, we first recorded MEG signals from human subjects during rest and while they navigated a specially designed state space. We next characterised each individual subject's decision-making flexibility based on their task behaviour, and we then analysed their MEG signals to look for evidence of on-task (Eldar et al., 2018) and off-task (Kurth-Nelson et al., 2016; Liu et al., 2019) sequences of state representations. Results Individual differences in decision flexibility We used distinct visual images to represent eight unique states, where occupancy of each state provided a different amount of reward (Figure 1a). Subjects started each trial at a random state and had to choose a movement direction in order to collect reward from subsequent states (Figure 1b). Subjects learnt beforehand how much reward was associated with each state (Figure 1—figure supplement 1), but they did not know initially where states were in relation to one another. The latter aspect of task structure had to be acquired through trial and error learning in order for subjects to be able to implement subsequent moves that delivered the maximal amount of reward. We assessed subjects' flexibility in three ways. First, after the initial two blocks of trials, we changed the reward associated with each state (Figure 1a; grey numbers) such that persisting with previously optimal moves would result in below-chance performance. Second, after two additional blocks of trials, we informed subjects that two specified pairs of states had switched positions (Figure 1a; 'switch'), again rendering the previously optimal policy now suboptimal. A flexible model-based planner would be capable of re-planning their moves perfectly following each of these instructed changes, since such a planner has acquired knowledge as to how each state can be reached. Conversely, a pure model-free planner would need to learn a new policy from scratch via trial and error each time there is a change, since such an agent only possesses a now outdated policy that specifies where to move from each state. Figure 1 with 4 supplements see all Download asset Open asset Subjects differed in decision flexibility. (a) Experimental task space. Before performing the main task, subjects learned state-reward associations (numbers in black circles) and they were then gradually introduced to the state space in a training session. After performing the main task for two blocks of trials, subjects learned new state-reward associations (numbers in dark grey circles) and then returned to the main task. Before a final block of trials, subjects were informed of a structural task change such that 'House' switched position with 'Tomato', and 'Traffic sign' switched position with 'Frog'. The bird's eye view shown in the figure was never seen by subjects. Subjects only saw where they started from on each trial and, after completing a move, the state to which their move led. The map was connected as a torus (e.g., starting from 'Tomato', moving right led to 'Traffic sign', and moving up or down from the tomato led to 'Pond'). (b) Each trial started from a pseudorandom location from whence subjects were allowed either one ('1-move trial') or two ('2-move trial') consecutive moves (signalled at the start of each set of six trials), before continuing to the next trial. Outcomes were presented as images alone, and the associated reward points were not shown. A key design feature of the map was that in 5 out of 6 trials the optimal (first) move was different depending on whether the trial allowed for one or two moves. For instance, given the initial image-reward associations (black) and image positions, the best single move from 'Face' is LEFT (9 points), but when two moves are allowed the best moves are RIGHT and then DOWN (5+9 giving 15 total points). Note that the optimal moves differed also given the second set of image-reward associations. On 'no-feedback' trials (which started all but the first block), outcome images were also not shown (i.e., in the depicted trials, the 'Wrench', 'Tomato' and 'Pond' would appear as empty circles). (c) The proportion of obtainable reward points collected by the experimental subjects, and by three simulated learning algorithms. Each data point corresponds to 18 trials (six 1-move and twelve 2-move trials), with 54 trials per block. The images to which subjects moved were not shown to subjects for the first 12 trials of Blocks II to V (the corresponding 'Without feedback' data points also include data from 6 initial trials with feedback wherein starting locations had not yet repeated, and thus, subjects' choices still reflected little new information). All algorithms were allowed to forget information so as to account for post-change performance drops as best fitted subjects' choices (see Materials and methods for details). Black dashed line: chance performance. Shaded area: SEM. (d) Proportion of first choices that would have allowed collecting maximal reward where one ('1-optimal') or two ('2-optimal') consecutive moves were allowed. Choices are shown separately for what were in actuality 1-move and 2-move trials. Subjects are colour coded from lowest (gold) to highest (red) degree of flexibility in adjusting to one vs. two moves (see text). Dashed line: chance performance (33%, since up and down choices always lead to the same outcome). (e,f) Decrease in collected reward following a reward-contingency (e) and spatial (f) change, as a function of the index of flexibility (IF) computed from panel d. Measures are corrected for the impact of pre-change performance level using linear regression. p value derived using a premutation test. Examining how subjects' overall performance altered immediately following these changes revealed a decrement in average performance (Figure 1c). However, there were substantial individual differences in this regard, with some subjects seamlessly adapting to reward and position changes, and others showing drops in performance to chance levels. Subjects whose performance showed a strong decline following a reward change tended to cope poorly also with the position change (ρ=0.50, partial correlation controlling for performance levels before the changes; p=0.001, Permutation test). As a third, more continuous, test of a different aspect of decision flexibility, we interleaved sets of six trials in which only a single move was allowed ('1-move trials') with trials which allowed two consecutive moves ('2-move trials'; Figure 1b). In 2-move trials, subjects were rewarded for both states they visited, and thus, an optimal course of action often required subjects to move first to an initial low-reward state in order to gain access to a high-reward state with their second move. Thus, we defined an individual index (IF) of decision flexibility as the difference between the proportion of moves that were optimal given the actual number of allotted moves and the proportion of moves that would have been optimal given a different number of allotted moves (i.e., had 1-move trials instead involved two moves and 2-move trials involved one move). An IF value of zero implies no net adjustment, while positive IF values imply advantageous flexibility. The results indicate subjects adjusted their choices advantageously to the number of allotted moves (+0.21, SEM 0.05, p < 0.001, Bootstrap test), though there was evidence again of substantial individual differences (Figure 1d). Importantly, IF correlated with how well a subject coped with the reward-contingency (Figure 1e) and position (Figure 1f) changes as well with how accurately they could sketch maps of the state space at the end of the experiment (r=0.51, p<0.001, Permutation test; Figure 1—figure supplement 2), indicating these subjects acquired and utilised a model of the state space. Planning two steps into the future Having a cognitive model that specifies how states are spatially organised makes it possible to plan several steps into the future. To test whether subjects were able to do that, we challenged subjects with 12 'without-feedback' trials at the beginning of each of the last four blocks, during which outcome images were not shown. This meant that in 2-move trials subjects had to choose their second move 'blindly', without having seen the image to which their previous move had led (e.g., the tomato in Figure 1b). We found that subjects performed above chance on these blind second moves (proportion of optimal choices: 0.56, SEM 0.03; chance = 0.45; p<0.001, Bootstrap test), and this was the case even immediately following position and reward changes, when subjects could not have relied on previously tested 2-move sequences (0.52, SEM 0.03; p=0.01, Bootstrap test). Most importantly, such blind-move success was correlated with IF (Figure 1—figure supplement 3a). This result indicates that more flexible subjects were better able to plan two steps into the future when required. Examining response times suggested flexibility was associated with advance planning also when it was not required. Thus, we found that IF correlated with quicker execution of second moves in general (Spearman correlation with median reaction time: r=-0.61, p<0.001, Permutation test). To determine whether advance planning was indeed generally associated with flexibility, we examined at what point during a trial their choices became decodeable from MEG signals. For this purpose, we trained a decoder to decode chosen moves from MEG signals recorded outside of the main task (see Materials and methods for details). Validating the decoder on MEG data from the main task showed that chosen moves became gradually more evident over the course of the trial, their decodability peaking 140 ms before a choice was made (Figure 1—figure supplement 3b). Thus, we used the move decoder to test whether second-move choices began to materialise in the MEG signal even before subjects observed the outcomes of their first moves. We found that chosen second moves were indeed decodeable already during first-move choices (decodability: M= 0.006, 95% Credible Interval = 0.004 to 0.008, Bayesian Gaussian Process analysis; Figure 1—figure supplement 3c) and prior to the appearance of the first outcome (decodability: M=0.004, 95% Credible Interval = 0.002 to 0.006; Figure 1—figure supplement 3d). Importantly, this early decodability was correlated with IF (β: M=0.29, 95% Credible Interval = 0.24 to 0.34). By contrast, later decodability, following the onset of the second image, did not correlate with IF (β: M=0.02, 95% Credible Interval = −0.02 to 0.05). Thus, neural and behavioural evidence concur with the notion that flexibility was associated with planning second moves prospectively. Individual flexibility reflected MF-MB balance These convergent results suggest that IF reflected deployment of a model-based planning strategy. To test this formally, we compared how well different model-free (MF) and model-based (MB) decision algorithms, as well as a combination of both, explained subjects' choices. Importantly, we enhanced these algorithms to maximise their ability to mimic one another (see Materials and methods for details). Thus, for instance, the MF algorithm included separate 1-move and 2-move policies, which allow it to achieve optimal adjustment to trial type given sufficient experience. We found that a hybrid of MF and MB algorithms outperformed substantially either of them alone (Bayesian Information Criterion [Bishop, 2006]: MF = 40821, MB = 43249, MF-MB hybrid = 39908), suggesting that subjects employed a mix of model-free and model-based planning strategies. Simulating task performance using the hybrid algorithm showed it captured adequately behavioural differences evident between subjects (correlation between real and simulated IF: r=0.92, p<0.001, Permutation test; Figure 1—figure supplement 4a). When we examined each subject's best-fitting parameter values, to determine which of these covaried with IF, we found 84% of inter-individual variance in IF was explained by three parameters that control a balance between model-based and model-free planning (Figure 1—figure supplement 4b). Importantly, less flexible subjects had comparable learning rates and a higher model-free inverse temperature parameter (in 2-move trials), indicating that lower flexibility did not reflect a non-specific impairment, but rather, it was associated with enhanced deployment of a model-free algorithm. Thus, our index of flexibility specifically reflected the influence of model-based, as compared to model-free, planning. On-task replay is induced by prediction errors and associated with high flexibility In rodents, reinstatement of past states, potentially in the service of planning, is evident both prior to choices (Pfeiffer and Foster, 2013) and following observation of outcomes (Pezzulo et al., 2014). Thus, we determined firstly at what point states were neurally reinstated during our task. For this purpose, we trained MEG decoders to the images subjects were processing (Figure Such decoders reveal representations that are reinstated from and contribute to decision (Eldar et al., 2018; Kurth-Nelson et al., 2015; 2006; and Foster, image decoders were trained on MEG data collected prior to subjects having knowledge the task, that the was of to other task (Figure supplement these decoders to MEG signals from the main task, we found no evidence of of outcome states to which subjects at choice (Figure supplement we found strong evidence that following outcomes to new states to which subjects had subjects represented the states from which they had moved p=0.001, Permutation test; Figure supplement we examined in the MEG data recorded following each outcome for evidence of replay of state sequences that subjects had Figure with supplements see all Download asset Open asset (a) of the image MEG decoder used for the = 40 subjects. The the decodability of starting images from MEG data recorded during the main task at trial was computed as the probability to the starting image by an based on each spatial MEG chance probability (b) of the analysis used to determine whether representational of pairs of elements one another in is computed as the difference between the of two time series with positive and time it on in the correlation this measure is for relationships even between correlated time indicates signals are in To test for evidence of we a measure of to the decoded MEG time a we previously showed is in replay of experienced and sequences of states (Eldar et al., 2018; Kurth-Nelson et al., 2016; Liu et al., Figure Importantly, is not to and thus, it is only found if representations one another in time (Eldar et al., 2018; as in previous we allowed for of up to Thus, following each we computed between the decoded representations of the and the outcome state (Figure 3a). MEG signals recorded following the second outcome in 2-move trials were also tested for the first (i.e., between the starting state and first Figure 3b). Figure Download asset Open asset On-task replay of trajectories as a function of individual flexibility. = 40 subjects. (a) corresponding to a from the image the subject had in the at the the to the image to which they the following outcomes (i.e., state prediction In the the indicates the actual action on the the indicates the that is For alone, time series are shown separately for subjects with high and values indicate replay and values indicate replay. As in previous (Eldar et al., was over all time from ms to and each a moving time of ms at the given time (e.g., the 1 MEG data from to following outcome). Dashed show data by a Bayesian Gaussian Process and the dark indicate where the 95% Credible Interval zero and The IF as a function of for the where the average over all subjects was p value derived using a premutation test. flexibility (b) following the of 2-move trials corresponding to a from the starting image to the first outcome (c) in the probability of a different as a function of recorded at the For is into high (i.e., above and (i.e., A correlation analysis between and probability of policy change showed a relationship (Spearman Bootstrap test). was over the first of from and in subjects with than the by for the first time the subject each of policy was computed as the of a different move when the same state corresponds to the average probability of change an Bayesian Gaussian Process approach (see Materials and methods for we tested for at which was evident and correlated with individual flexibility. This directly for by for the between them 2014). replay is to be induced by and Daw, 2018; Momennejad et al., 2018; and we also included the outcome (i.e. the state prediction error by the hybrid as a of We found encoding the last experienced state to ms and from to ms following outcome Figure that the median is for actual on the flexibility and, at the of 2-move trials, also the ms before to ms following outcome Figure 3b). These sequences were in with an of ms between the and were encoded in a direction corresponding to the order later in the the was also replayed to ms following outcome As would be the of was associated with enhanced of previously states during the (Figure supplement importantly, we found this evidence of all was correlated with IF 95% Credible Interval = to with the outcome = to and with the of these two = to Thus, was evident following outcomes in subjects with high index of flexibility. This result is with online replay to model-based planning. On-task replay is associated with changes of policy theorising the role of replay in planning that replay is induced when there is to policy and Daw, policy can either or a chosen move, in our choices in our experiment were more by a policy than since a substantial proportion of the latter already reflected a policy by subjects' above chance performance in Figure 1c). Thus, a role for replay in planning that subjects be more to replay trajectories that they might not to choose rather than trajectories whose choice a policy. To determine whether on-task replay was associated with behavioural change, we tested the relationship between corresponding to each move that subjects and the probability of making a different choice when the same state later We found that moves after which high was evident to moves that were less to be (Figure These policy changes the proportion of reward Thus, we found evidence of online replay for a chosen was coupled with advantageous of Off-task replay is induced by prediction errors and associated with flexibility We next off-task MEG data recorded during the rest that each experimental block. each block included starting states, we computed for the transitions subjects before and after each rest choice = per As a control we also examined for the chosen transitions from the same starting states choice = per We found evidence for the rest periods for transitions = SEM = 0.001, p=0.01, Bootstrap test). By contrast, we found no evidence of for the transitions < 0.001, SEM = 0.001, Bootstrap test). transitions were replayed in a with an time of ms between and trajectories that induced more reward prediction errors in the previous