Psychophysics of the McGurk and other audiovisual speech integration effects.

When perceivers can see as well as hear talkers, speech perception is audiovisual (AV) (e.g., Massaro, 1987; McGurk & MacDonald, 1976; Sumby & Pollack, 1954). Many experiments have been carried out with mismatched AV stimuli in an effort to understand the perceptual integration of auditory and visual components of AV speech stimuli. In such experiment, four main perceptual effects are obtained: The fusion effect or McGurk effect (McGurk & MacDonald, 1976) is said to have occurred when, for example, a video /gA/ and an acoustic /bA/ stimulus are presented together, and perceivers report hearing /dA/, a different syllable. That is, the perceptual response is different from the response to either of the uni-sensory stimuli. Numerous studies have shown fusion effects (Green & Kuhl, 1989; Green, Kuhl, Meltzoff, & Stevens, 1991; MacDonald & McGurk, 1978; Manuel, Repp, Studdert-Kennedy, & Liberman, 1983; Massaro, 1987; Sekiyama & Tohkura, 1991; Summerfield & McGrath, 1984). The combination effect occurs when both the acoustic and video stimuli are perceived sequentially (McGurk & MacDonald, 1976). For example, pairing acoustic /gi/ with a video /bi/ can result in the perceived combination /bgi/ (Green & Norrix, 1997). Mismatched stimuli can also result in either auditory correct or visual correct responses. We hypothesized that the different types of perceptual effects are a consequence of a perceptual processing system that has learned the normal relationships between auditory and visual stimulus components (Bernstein, Lu, & Jiang, 2008). When that relationship is disrupted by mismatching the stimulus components, systematic changes occur in perceptual responses. However, in order to test this hypothesis, the systematic physical stimulus relationships between components need to be characterized. Stimulus characterizations typically have been perceptual and not quantitative (e.g., Braida, 1991; Grant & Braida, 1991; Massaro, 1998). That is, stimuli are typically described in terms of phoneme identities of component auditory and visual stimuli, and in terms of the match or mismatch between the components. Gross physical stimulus characteristics, such as duration, acoustic amplitude, and video image resolution, are among the few quantitative measures that are commonly reported, although these characteristics are not usually directly relevant to the hypothesis being tested. Even quantitative models of AV perceptual integration have mostly, to our knowledge, used perceptual responses, such as, auditory-only, visual-only, and AV phoneme confusion matrices, as input to the models (Braida, 1991; Grant & Braida, 1991; Massaro, 1998). Stimulus onset asynchrony has been one of the few quantitatively measured independent variables in AV speech research (Massaro, Cohen, & Smeele, 1996; McGrath & Summerfield, 1985; van Wassenhove, Grant, & Poeppel, 2007), but internal features of the stimulus components are typically not quantified (c.f., Munhall, Gribble, Sacco, & Ward, 1996). An influential theoretical paper by Summerfield (1987) posited that the core of a comprehensive account of AV speech integration would be a perceptually relevant metric of AV speech stimuli. Summerfield proposed as possible metrics, (1) auditory and visual phonetic features (the only qualitative stimulus characterization), (2) the acoustic filter function of the vocal tract, (3) vectors representing the magnitudes of independent acoustical and optical parameters, (4) successive static vocal-tract configurations, and (5) time-varying kinematic patterns. Summerfield also reasoned that these metrics were mostly inadequate. Integration in terms of phonetic features was inadequate, because the empirical evidence led to ad hoc rules for integration, depending on whether the features were visually or auditorally presented. The notion that the acoustic vocal tract transfer function could be derived from viewing a talker was deemed inadequate, because too much of speech production cannot be seen. An alternative suggestion was that a look-up table with canonical acoustic values could be accessed through vision, because perceivers “’know’ the audio-visual [sic] structure of phonemes.” But this metric was noted to have several disadvantages, among them the arbitrary supposition that evidence from vision is translated into auditory representations. The vector combination of acoustic and optical parameters in terms of acoustic spectra and two-dimensional visual images of the lips and teeth viewed from a single direction were criticized because of, for example, the problem of non-invariance of optical signals with diverse views of the talker. The fifth theoretical metric was based on speech dynamics: “If talkers communicated simply by oscillating their lips at different rates, we should have no difficulty in describing audio-visual [sic] speech perception as the integration of audible and visual evidence of modulation rate” (p. 40). Kinematic descriptors implying speech dynamics were proposed as a modality-free representation, but they were also viewed as problematic. For example, “the mapping [to dynamics] depends on the position of the tongue, and recovering dynamic parameters from natural speech acoustics will be at best computationally complex, and at worst intractable.” Although all of the metrics were viewed as problematic, the notion of a common metric or representation to explain AV speech perception been extensively discussed in the literature (e.g., Bernstein, Auer, & Moore, 2004; Fowler, 2004; Green, et al., 1991; Massaro, 1998; Rosenblum, 2008). The fact that AV stimuli are rarely quantified for perceptual experiments is not because the stimuli have never been quantified. Studies have been carried out to quantify speech signal relationships between measures of the auditory and visual stimulus components (Chandrasekaran, Trubanova, Stillittano, Caplier, & Ghazanfar, 2009; Craig, van Lieshout, & Wong, 2008; Jiang, Alwan, Keating, Auer, & Bernstein, 2002; Yamamoto, Nakamura, & Shikano, 1998; Yehia, Rubin, & Vatikiotis-Bateson, 1998). What has generally not been done is to relate such measures directly to perceptual responses (c.f., Bernstein, et al., 2008).