Multimodal images in the brain

Mental imagery is like perceiving, but in the absence of an immediate appropriate sensory stimulation. As such, imagery is often identified with specific phenomenology, such as the experience of ‘seeing with the mind’s eye’, ‘hearing with the mind’s ear’, and so on. However, the experience itself is not the image; and, in fact, it is not even clear that mental images must always be accompanied by a specific experience. Rather, mental images consist of internal representations of the same types as those that arise during the early phases of like-modality perception. Mental images are internal representations that are based on information stored in memory. In contrast, perception occurs when information is registered directly from the senses. This conception of mental imagery leads us to characterize many sorts of imagery. First, each perceptual modality should, in principle, be accompanied by the ability to generate images in that modality. And, in fact, people commonly report visual, auditory, and kinaesthetic images (e.g., Kosslyn et al. 1990). Second, not all forms of imagery need give rise to distinct phenomenology. For example, spatial images may give rise to only the most impoverished sense of ‘where things are’. Third, mental images need not be simply the recall of previously perceived objects or events; they also can be created by combining and modifying stored perceptual information in novel ways. Mental imagery has played a central role in theories of mental function at least since the time of Plato. It has fallen in and out of fashion, in large part because it is inherently a private affair – by definition restricted to the confines of one’s mind. (In this context, by ‘mind’ we mean brain function.) Thus, imagery has been difficult to study. In fact, in 1913 the founder of Behaviourism (the school of psychology that focused solely on observable stimuli, responses, and the consequences of responses), John B. Watson, denied that mental images even existed. Instead, he suggested, thinking consists of subtle movements of the vocal apparatus (Watson 1913). Nevertheless, clever researchers developed empirical methods for studying imagery. Notably, Alan Pavio and his colleagues (see, for example, Paivio 1971) were able to show that the use of imagery dramatically improves memory. However, even in the face of such findings, some researchers were not convinced that imagery is a distinct form of thought. Indeed, Watson’s position was echoed 60 years later by Zenon Pylyshyn, who championed the view that mental images are not ‘images’ at all, but rather rely on mental descriptions no different in kind from those that underlie language. According to Pylyshyn (1973), the pictorial aspects of imagery that are evident to conscious experience are entirely epiphenomenal, like the heat from a light bulb when you read (which plays no role in the reading process). That is, according to this descriptivist view, the fact that we experience visual images as akin to pictures in the mind’s eye says nothing about the nature of the underlying mental representation. The emergence of cognitive neuroscience has opened a new chapter in the study of mental imagery. An enormous amount has been learned about the neural underpinnings of visual