Foreground and Background in Mental Models

Though I “supervised” Mark Stokes’ PhD, the direction of influence was never entirely clear. On a memorable evening in Havana, in the second year of his studies, Mark sat me down at a bar and suggested we should “give these new multivariate fMRI methods a try. ” Though these methods are now very standard, at that time there was a lot to consider, and Mark more or less single-handedly led the development of multivoxel pattern analysis in the Cambridge community. A few years later, now well established in Oxford, Mark approached me again and suggested that similar multivariate methods might usefully be applied to our neurophysiological data from the behaving monkey. Having learned by now that the best bet was just to sit back and let him go, I was happy to hand over data from a large sample of frontal lobe neurons recorded during a working memory task. Every few months over the next year, he would get back to me with new results on overall population activity, population dynamics, movements in state space, and his first ideas on uncovering the hidden state of a network with a nonspecific “ping. ” When we sat down to write the paper (Stokes et al., 2013), Mark wanted to push the idea of an activity-silent aspect to working memory. In our task, each trial began with a cue instructing the animal to monitor a stream of stimuli for a specific target. In the delay between the cue and the first stimulus of the stream, there was a strong neural code of current target identity. Between subsequent stimuli, however, this code was much reduced, while the animal (of course) continued tomonitor correctly. Following recent theoretical papers (Mongillo, Barak, & Tsodyks, 2008), Mark suggested that the enduring target code must be in an activity-silent form. On this account, high-speed synaptic plasticity allowed the cue to reconfigure the frontal network such that each choice object led correctly to a target or nontarget decision state. Used to the idea that workingmemory was implemented in sustained neural firing, I was less certain; I thought we could hardly take weak active coding as strong evidence of an additional silent process. Of course, much stronger evidence now comes from the pinging method, with selective states of a network revealed by distinct multivariate activity patterns following a neutral visual stimulus (Wolff, Jochim, Akyurek, & Stokes, 2017) or TMS pulse (Rose et al., 2016). This adds to the earlier demonstration that, when two items are held in working memory, only a foregrounded item can be decoded from the pattern of fMRI activity, with the “hidden” pattern reawakening if the attentional state is changed (Lewis-Peacock, Drysdale, Oberauer, & Postle, 2012). Meanwhile, it now seems inconceivable to me that Mark could have been wrong. Just from first principles, we can see that “working memory” must have its active and silent states—and indeed, that a boundary between “working memory” and “long-term memory” makes no real sense. At the most general level, ongoing cognition may be seen as a process of describing some state of affairs in a mental model and using that model to guide thought and behavior (Craik, 1943). Perception, for example, is much more than a matter of recognizing objects in the sensory environment; rather, we build an elaborate model of the current scene, with many objects organized into their correct spatial, temporal, and functional relationships. In this sense, brain activity in perception must closely resemble activity in retrieving a specific episode from memory, or creating an imagined episode from general knowledge of entities in our world and how they interact. “Cognitive control” is a matter of directing behavioral choices by the contents of a current model, whether those contents are provided by verbal instructions or any other means. Very much as envisioned in the symbolic artificial intelligence (AI) of the last century (Newell, 1990), cognition is a matter of combining new inputs and long-term knowledge into a structured model of a current problem, allowing this world knowledge to direct successful cognition, and often in symbolic AI called “working memory. ” A little consideration of any current world model surely implies both active (neural discharge) and silent (synaptic) components. A person sitting in a cognitive control experiment knows the rules of the task, where they should look for stimuli and what keyboard they should use for responses, but there is much more than this to his or her model of the current situation. The person knows the layout of the room, the time of day, why they are here, who gave the instructions, and what they plan to do next when the experiment is finished. It does not make sense to imagine that all of this knowledge is reflected in active neural firing, but equally, it would be unprofitable to ask which parts of the model are “working” and which are “long term. ” Certainly, much of this knowledge is more accessible today than it will be University of Cambridge