Hearing how smooth it looks: selective attention and crossmodal perception in the arts

A broad range of behavior is associated with crossmodal perception in the arts. Philosophical explanations of crossmodal perception often make reference to neuroscientific discussions of multisensory integration in selective attention. This research demonstrates that superior colliculus plays a regulative role in attention, integrating unique modality specific visual, auditory, and somatosensory spatial maps into a common spatial framework for action, and that motor skill, emotional salience, and semantic salience contribute to the integration of auditory, visual, and somatosensory information in ordinary perceptual contexts. I present a model for multisensory integration in our engagement with artworks derived from a diagnostic recognition framework for object recognition and a biased competition model for selective attention. The proposed model attributes a role to superior colliculus in a broader fronto-parietal attentional network that integrates sensory information, primes perceptual systems to the expectation of stimulus features salient to particular sensorimotor or cognitive tasks at particular locations, and inhibits the perception of task irrelevant distracters. I argue that this model demonstrates that crossmodal effects are the rule not the exception in perception and discuss ways in which it explains a range of crossmodal effects in our engagement with pictures, dance, and musical performances. Can music sound like a gesture or a bodily posture? Can sense modalities be crossed that way? There certainly is plenty of room for doubt about it? A broad range of behavior is associated with crossmodal perception in the arts: we hear the sounds of a movie emanating from the location of the visual event it depicts, not one determined by the arrangement of the speakers that surround us in the theater; the perceived duration of a note we hear is influenced by the gesture with which we see it performed; motor skill and perceived task difficulty can influence the way we hear music and see the _____________________________ Corresponding Author: W. P. Seeley Bates College email – wseeley@bates.edu Essays Philos (2012) 13:2 Seeley | 499 spatial layout of events depicted in pictures; we often report that the music we hear feels sad or exuberant; and emotionally expressive music is used to direct attention to salient visible features of film scenes. Some of these examples are garden variety cases of crossmodal effects in ordinary perception. Others can seem quite surprising. However, on closer look they too are perhaps just what is to be expected. Perception did not arise in a vacuum. Perceptual systems evolved in lock step with bodies, and more importantly the structure and function of perceptual systems are intrinsically related to our goals, cognitive capacities, and bodily capabilities. Selectivity is, from this perspective, a critical feature of perception. Perceptual systems must enable organisms to quickly and efficiently identify, and orient themselves to, features salient to their current goals in an environment replete with sensory information. Evidence suggests that perceptual systems accomplish this task by selecting minimal sets of features diagnostic for, or sufficient to determine the identity, shape, location, and affordances of, objects and events in the environment. Crossmodal effects play a key role in these perceptual processes. For instance, in ordinary contexts sounds occur at the locations of, and in temporal register with, the objects, actions, or events that produce them. This entails that the conjunction of a minimal set of spatio-temporally coincident auditory and visual cues diagnostic for task salient aspects of an object or event are sufficient to identify and locate it. Similarly, the perceptible aspects of objects and their affordances change in predictable ways as we move through the environment and orient our bodies to interact with them. The systematic relationship between movement dynamics and perceptual change entails that motor planning, preparation, and simulation can be used to generate forward models of environmental dynamics that direct attention and influence sensory encoding of task salient aspects of perceived events. A critical question arises in these contexts. How do perceptual systems assign salience to environmental features? One means is simple perceptual salience. Novel perceptual features and those that contrast starkly with their surround draw attention to themselves because they stand out in a crowd. However, the critical features salient to a task are not always the most perceptually salient. Cognitive systems must therefore include a means to flexibly bias responses to sensory stimuli independent of perceptual salience. Biased competition models for selective attention demonstrate that the perceived salience of environmental features is mediated by top-down cortico-thalamic networks that converge on the superior colliculus (SC), a multisensory area in the midbrain where spatial maps from different modalities are brought into register, and thereby influence the encoding of those features at all levels of visual, auditory, and somatosensory processing. A similar story can be told about the perceived emotional salience of environmental features. Emotional attention biases the encoding of task salient aspects of objects and events via a fronto-amygdala network that includes projections to sensory cortices. Essays Philos (2012) 13:2 Seeley | 500 The suggestion that I am making is that perceptual and affective systems are deeply interconnected via crossmodal cortico-thalamic attentional networks that enhance the flexibility of goal directed behavior. What is the explanatory payout of this view for philosophical questions about art? Artists develop productive strategies through systematic explorations of the behavioral effects of different sets of formal and compositional arrangements within a medium. Artworks can, as a consequence, be conceptualized as stimuli intentionally designed to trigger perceptual, affective, and cognitive responses diagnostic for their content in viewers, spectators, listeners, and readers. This entails that questions about the understanding and appreciation of art include questions about the way viewers, spectators, listeners, and readers acquire, represent, manipulate, and use information embedded in the formal structure of artworks in order to recognize and evaluate their content. My suspicion is that these data can contribute to a broad range of debates about the nature of art and the character of aesthetic experience. I can, of course, imagine some skepticism about this claim. However, we can all agree that philosophical theories about art and aesthetics must be consistent with psychological facts about our engagement with artworks. Therefore, questions like these about the construction and reception of artworks are not philosophically trivial. In what follows I present a model for multisensory integration in our engagement with artworks derived from Phillipe Schyns’s diagnostic recognition framework for object recognition and a biased competition model for selective attention. My plan is to back my way into the issue of crossmodal perception in the arts using a grab bag of examples that illustrate the relationship between diagnosticity, attention, and biased competition in perception. I will then detail the model and discuss how it generalizes to include the roles of multisensory integration, motor simulation, and affective salience in our engagement with artworks. Along the way I will knowingly and willfully commit the crime of visuocentrism. My reasons are methodological. The philosophical crime of visuocentrism, as I understand it, has two related aspects. It is both the mistake of making unfounded ontological claims about things like sounds based on ontological claims about things like visible objects, and the mistake of making unfounded phenomenological claims about experiences in other modalities based on an understanding of visual phenomenology. These are good worries. However, the analogy that grounds the influence of vision research on our understanding of other sense modalities is neither phenomenological nor ontological in these senses. Rather it is architectural, based in the topological organization of critical areas at all levels of perceptual processing systems (e.g., retinotopic organization of geometric space in visual areas, tonotopic organization of pitch space in auditory areas, and somatotopic maps of body space in somatosensory and motor areas) and the hierarchical organization of top-down and bottom-up influences in sensory processing. These facts ground structural analogies between visual, auditory, and somatosensory processing systems that can be used as heuristic devices to promote a better understanding of our perceptual Essays Philos (2012) 13:2 Seeley | 501 engagement with the environment.