Listening to Talking Faces 1 Running head : Listening to Talking Faces Listening to Talking Faces : Motor Cortical Activation During Speech Perception

Neurophysiological research suggests that understanding the actions of others harnesses neural circuits that would be used to produce those actions directly. We used fMRI to examine brain areas active during language comprehension in which the speaker was seen and heard while talking (audiovisual) or heard but not seen (audio-alone) or when the speaker was seen talking but the audio track removed (video-alone). We also examined brain areas active during speech production. We found that speech perception in the audiovisual, but not audio-alone or video-alone conditions activated a network of brain regions overlapping cortical areas involved in speech production and proprioception related to speech production. These regions included the posterior part of the superior temporal gyrus and sulcus, the superior portion of the pars opercularis, the dorsal aspect of premotor cortex, adjacent primary motor cortex, somatosensory cortex, and the cerebellum. Activity in dorsal premotor cortex and posterior superior temporal gyrus and sulcus was modulated by the amount of visually distinguishable phonemes in the stories. These results suggest that integrating observed facial movements into the speech perception process involves a network of brain regions associated with speech production. We suggest that this distributed network serves to represent the visual configuration of observed facial movements, the motor commands that could have been used to generate that configuration, and the associated expected auditory consequences of executing that hypothesized motor plan. These regions do not, on average, contribute to speech perception when in the presence of the auditory or visual signals alone. Listening to Talking Faces 3 Introduction Neurobiological models of language processing have traditionally assigned receptive and expressive language functions to anatomically and functionally distinct brain regions. This division originates from the observation that Broca’s aphasia, characterized by nonfluent spontaneous speech and fair comprehension, is the result of more anterior brain lesions than Wernicke’s aphasia, which is characterized by fluent speech and poor comprehension, and is the result of more posterior brain lesions (Geschwind, 1965). As with many such simplifications, the distinction between expressive and receptive language functions within the brain is not as straightforward as it may have appeared. Comprehension in Broca’s aphasia is almost never fully intact nor is production in Wernicke’s aphasia normal (Goodglass, 1993). Electrical stimulation of sites in both the anterior and posterior aspects of the brain can disrupt both speech production and speech perception (Ojemann, 1979; Penfield & Roberts, 1959). Neuroimaging studies have also shown that classically defined receptive and expressive brain regions are often both active in tasks that are specifically designed to investigate either perception or production (Braun et al., 2001; Buchsbaum et al., 2001; Papathanassiou et al., 2000). Recent neurophysiological evidence from nonhuman primates suggests an explanation for the observed interactions between brain regions traditionally associated with either language comprehension or production. The explanation requires examination of motor cortices and their role in perception. Regions traditionally considered to be responsible for motor planning and motor control appear to play a role in perception and comprehension of action (Graziano & Gandhi, 2000; Romanski & Goldman-Rakic, 2002). Certain neurons with visual and/or auditory and motor properties in these regions discharge both when an action is performed and during perception of another person performing the same action (Gallese et al., 1996; Kohler et al., 2002; Rizzolatti et al., 1996). In the macaque brain, these neurons reside in area F5 which is the proposed homologue of Broca’s area, the classic speech production region of the human (Rizzolatti et al., 2002). The existence of “mirror neurons” suggests the hypothesis that action observation aids action understanding via activation of similar or overlapping brain regions used in action performance. If this is the case, perhaps speech understanding, classically thought to be an auditory process (e.g., Fant, 1960), might be aided in the context of face-to-face interaction by cortical areas more typically associated with speech production. Seeing facial motor behaviors corresponding to speech production Listening to Talking Faces 4 (e.g., lip and mouth movements) might aid language understanding by recognition of the intended gesture within the motor system, thus further constraining possible interpretations of the intended message. Audiovisual Language Comprehension Most of our linguistic interactions evolved, develop, and occur in a setting of face-to-face interaction where multiple perceptual cues can contribute to determining the intended message. Although we are capable of comprehending auditory speech without any visual input (e.g., listening to the radio, talking on the phone), observation of articulatory movements produces significant effects on comprehension throughout the lifespan. Infants are sensitive to various characteristics of audiovisual speech (Kuhl & Meltzoff, 1982; Patterson & Werker, 2003). By adulthood, the availability of visual information about speech production significantly enhances recognition of speech sounds in a background of noise (Grant & Seitz, 2000; Sumby & Pollack, 1954) and improves comprehension even when the auditory speech signal is clear (Reisberg et al., 1987). Furthermore, incongruent audiovisual information can change the identity of a speech percept. For example, when an auditory /ba/ is dubbed onto the video of someone making mouth movements appropriate for production of /ga/, the resulting percept is usually /da/. We are susceptible to this audiovisual integrative illusion from early childhood through adulthood (Massaro, 1998; McGurk & MacDonald, 1976). Our experience as talkers and as listeners may associate the acoustic patterns of speech with motor planning and proprioceptive and visual information about accompanying mouth movements and facial expressions. Thus experience reinforces the relationships among acoustic, visual, and proprioceptive sensory patterns and between sensory patterns and motor control of articulation, so that speech becomes an “embodied signal”, rather than just an auditory signal. That is, information relevant to the phonetic interpretation of speech may derive partly from experience with articulatory movements that are generated by a motor plan during speech production. The mechanisms that mediate these associations could provide a neural account for some of the observed interactions between acoustic and visual information in speech perception that may not be apparent by studying acoustic speech perception alone. The participation of brain areas critical for language production during audiovisual speech perception has not been fully explored. It may be that the observed effects on speech comprehension produced by observation of a speaker’s face involves visual cortical areas or other multisensory areas Listening to Talking Faces 5 (e.g., posterior superior temporal sulcus), and not actual speech production areas. However, the evidence from nonhuman primates with regard to “mirror neurons” suggests that production centers in concert with other brain regions are likely candidates for the neural structures mediating these behavioral findings. Neurophysiological Studies of Audiovisual Language Relatively little is known about the neural structures mediating the comprehension of audiovisual language. This may be because when language comprehension is not viewed as modalityindependent, spoken language comprehension is seen as essentially an auditory process, and that it should be investigated as such in neuropsychological and brain imaging studies. However, visual input plays an important role in spoken language comprehension, a role that cannot be accounted for as solely a cognitive bias to categorize linguistic units according to visual characteristics when acoustic and visual information are discrepant (Green, 1998). Neuroimaging studies of speech processing incorporating both auditory and visual modalities are often focused on the problem of determining specific sites of multisensory integration (Calvert et al., 2000; Mottonen et al., 2002; Olson et al., 2002; Sams et al., 1991; Surguladze et al., 2001). Other studies have focused on only one (potential) component of audiovisual language comprehension, speech (i.e., lip) reading (Calvert et al., 1997; Calvert & Campbell, 2003; Campbell et al., 2001; Ludman et al., 2000; MacSweeney et al., 2000; MacSweeney et al., 2002a; MacSweeney et al., 2001; Surguladze et al., 2001). However, few studies have investigated the extent of the entire network of brain regions involved in audiovisual language comprehension overall (Callan et al., 2001; MacSweeney et al., 2002b). Nonetheless these experiments have collectively yielded a fairly consistent result: Audiovisual speech integration and perception produce activation of auditory cortices, predominantly posterior superior temporal gyrus and superior temporal sulcus. Though studies have reported activation in areas important for speech production (e.g., MacSweeney et al., 2002b), there has not been much theoretical interpretation of these activations. This may be in part because some studies use tasks that require an explicit motor response (e.g., Calvert et al., 1997; MacSweeney et al., 2002b; Olson et al., 2002), which limit the inferences that can be drawn about the role of motor areas in perception (Small & Nusbaum, In Press). However, it would be surprising if brain regions important for language production (e.g., Broca’s area and the precentral gyrus and sulcus) did not play a role in audiovisual speech perception, given the known connectivity betw