Neuroscience: What You See and Hear Is What You Get Dispatch (2004)

You enter a crowded room and someone calls your name. You turn to see who it is. You now see several people in the general direction the voice came from. Many are talking. Which one called your name? You hear it again and now the sound seems to come from straight ahead or nearly so. There are still a handful of candidates in your field of view, so you look from one to the other. Finally, you see one whose lips move as you hear your name once more. Sound and sight have come together and you identify the speaker as your college roommate. How does this work? That is, how does the brain find the appropriate auditory–visual correspondence to determine that a sound and sight have come from the same source? In a study published recently in Current Biology, Alais and Burr [1] have demonstrated an important and seemingly pervasive rule for the combination of visual and auditory cues to spatial location. Before discussing their experiment, it is useful to describe what we know about visual and auditory localization. The direction of stimulus can be represented by two coordinates: the azimuth (horizontal direction in angular units) and the elevation (vertical). The eye is well suited for determining direction, because the direction in which a light ray enters the eye is directly indicated by the position it stimulates on the retina. As a consequence, the visual system can distinguish very small directional changes. The ‘just-noticeable difference’ (JND) in the position of one small stimulus relative to another is roughly 5–10 seconds of arc [2]. The auditory system is not nearly so well suited for determining direction, because the direction a sound enters the ear must be calculated from a variety of cues. One set of cues arises from the spatial separation of the two ears. Because of the separation, most sounds travel different distances to the left and right ears. A sound on the left, for example, reaches the left ear slightly before reaching the right ear, thereby creating an interaural time difference. Similarly, interaural intensity differences arise because the head acts as a sound shadow. A sound on the left is attenuated more on its way to the right ear than on its way to the left ear. The azimuth of a sound source can be determined from these interaural time and intensity differences, but the JND is relatively large, typically 1–2 degrees [3]. It is unclear how interaural differences alone can signal changes in elevation, yet people can reliably distinguish elevation changes of 5–10 degrees [4]. Batteau [5] showed how the filtering effect of the outer ear (pinna) can provide the missing information. Specifically, the spectrum of an approaching sound is filtered by interactions with the groves and ridges of the pinna, and the listener is able to use the filtered spectrum to judge elevation. The visual system is thus far better suited than the auditory system for estimating direction. In Alais and Burr’s [1] study, the visual JND was 5–10 times lower (more precise) than the auditory JND. It is widely believed that such differences in the precision of localization lead to visual capture, in which the apparent direction of an auditory stimulus is determined largely by the direction of a corresponding visual stimulus. Ventriloquism is an entertaining example of this phenomenon [6]. Alais and Burr [1] asked whether visual capture of a sound’s apparent direction derives from a rigid rule, in which the visual estimate always determines the overall percept, or from a more general procedure of weighting sensory evidence in a statistically optimal fashion. What would the best way be to combine noisy — variable — auditory and visual estimates of direction? The answer depends on the goal. If one wants an estimate that is unbiased and has minimum variability, the best combination rule in most cases is a weighted average. Suppose the brain has unbiased estimates DA and DV based on auditory and visual signals, respectively. Those estimates will have moment-to-moment variability because of fluctuations in the physical stimuli and noisiness in the brain’s measurements of them. The variability of DA and DV can be represented by variances σA and σV. For convenience, define the reliabilities of DA and DV as reciprocal variances, rA = 1/σA and rV = 1/σV. Under reasonable assumptions, the rule yielding the lowest-variance, unbiased estimate is a weighted average [7]: