On the rapid perceptual processing of speech : From signal information to phonetic and grammatical knowledge 1

There is one aspect of normal speech comprehension that simultaneous interpreting highlights in particular: its speed. It appears fair to say that speech researchers are still far from being able to propose detailed algorithmic accounts of how listeners go about perceptually decoding speech. While several influential theories have been proposed and pursued experimentally, there is still no agreement on the mechanisms which make speech perception so singularly fast and robust. However, in on-going research, certain developments can be discerned that eventually may shed some light on both the mapping problem and its temporal course. Here we pay special attention to a type of mechanism whose prospects of meeting the criterion of processing speed appear particularly favorable: the so-called exemplar-based models of speech perception. In such models, phonetic and grammatical entities & rules arise developmentally as emergent consequences of the listener’s cumulative perceptual experience. Our discussion leads us to conclude that, in comparison with many traditional approaches, exemplar models possess several attractive features. They provide a mechanism for automatically sorting out the acoustic context-dependence of phonetic segments. They permit grammatical analyses to extrapolate to previously unheard, but well-formed utterances. Phonetic and grammatical analyses are seen to occur without ‘inference making’ or ‘hypothesis testing’ operating with minimal demands on cognitive operations, i.e., without, strictly speaking, any ‘top-down’ processing. Last but not least, they have the necessary temporal characteristics. In our opinion, the proposed models appear to offer the kind of perceptual processing that both normal language use and simultaneous interpreting could use. 1. The problem and a phonetic perspective 1.1 From signal to percept: fast processing of a complex input The aim of this paper is to present a brief overview of current work on speech perception and to do so in the context of simultaneous interpreting. That goal provides an opportunity to focus on a rather incompletely understood aspect, namely the speed of perceptual processing. It is fair to say that modern phonetics does not yet offer a comprehensive, unified account of how listeners process speech signals. Several interesting theoretical frameworks have been proposed and examined experimentally (more on that topic anon), 1 Presented at the International Symposium on Language Processing and Interpreting , Stockholm University, 21-22 February, 1997. but nevertheless much remains to be discovered. We are still far from understanding the mechanisms that make normal speech perception so fast and robust. Rather a highpriority item on the agenda seems to be the formulation of simplified, heuristic models that can guide experimentation and suggest testable, ‘in-principle’ solutions to the problem of describing both the nature and origin of speech percepts. This state of affairs is linked to one of the classical issues of experimental phonetics: the infinite variety of natural speech patterns. It is against the background of variability that the speed of processing seems so striking. Reviewing work on the ear-voice span, Gerver (1976) mentions a study of simultaneous interpreting and shadowing from prerecorded French into English. Twelve professional conference interpreters participated and performed their task under several different conditions of signal-to-noise ratio. It was found that the ear-voice span increased as a function of decreasing S/N, but remained fairly stable independent of listening conditions. The lag was about 5.7 words for interpreting and about 2 words for shadowing. Other estimates are also mentioned. A lag of at least 2 seconds is reported in one study (Oléron and Nanpon 1964), a figure that tends to increase with increasing difficulty of the input (Treisman 1965). Simultaneous interpreting involves several stages, e.g., switching attention from input to output, comprehending the source message as well as the planning and execution of the utterance in the target language. Those activities take time. It appears natural to ask, How could there be time also for the process of converting the signal into a speech percept, particularly if the redundancy of the input serves as a trigger for the application of top-down knowledge. How could there be time for such top-down processes as ‘inference making’ and ‘hypothesis formation’? Or are those questions asked in the wrong way? What is the nature and origin of top-down and other real-time mechanisms? Towards the end of this chapter we shall come back to those questions. 1.2 The infinite variety of signal patterns The variability problem becomes evident when we contrast speech as a psychological phenomenon with speech as a physical signal. What language users (and linguists) call the “same” utterance normally appears in the guise of innumerable different physical shapes: the phrase ‘psychologically the same, but physically not the same’ is a nutshell statement of this paradox which is a variant of the classical problem of perceptual constancy in experimental psychology. We need not illustrate the variability issue in great technical detail. A few simple everyday observations will convince us that it is real. First consider speaker identity. We are good at recognizing people by simply listening to them. We know that different speakers sound different, even though they speak the same dialect and utter the same phonetic segments, the same syllables, words and phrases. The use of the word "same" here poses a challenge for the phonetician, for "identical" speech samples from different speakers are far from identical as physical patterns. For one thing, speakers are built differently. Age and gender are correlated with anatomical and physiological variations, such as differences in the size of the vocal tract and the properties of the vocal folds. Communicating by telephone, we are usually able to tell correctly from the voice alone whether the person is young or old, or is male or female. Evidently we are able to do so from the cues in the acoustic signal. So what are the acoustic events that account for the fact that we hear the "same words" whether they are produced by speaker A or by speaker B? That is in fact a very important question in all kinds of contemporary research on speech. When we limit our focus to the speech of a single person, we realize that the problem of variability still remains a major one. There is the factor of speaking style and situation. There is an extremely large number of ways in which the syllables and phonemes of the "same" phonetic forms could be spoken. Usually without being fully aware of it, we speak in a manner that depends on who we are talking to. We sound different talking to a baby or a dog, or to someone who is hard of hearing, has only a partial command of the language spoken, or speaks a different dialect. In response to noisy conditions, we spontaneously raise our vocal effort. We articulate more carefully and more slowly in addressing a large audience under formal conditions than when chatting with an old acquaintance. We sometimes tend to mumble and speak more to ourselves than to those present, often with drastic reduction of clarity and intelligibility as a result. The way we sound is affected by how we feel, our voices reflecting the state of our minds and bodies. Clearly, the speech of a given individual mirrors the intricate interplay of an extremely large number of communicative, social, cognitive and physiological factors. A third factor becomes evident as we narrow the scope still further. The term ‘lab speech’ is used with reference to samples of a single person's speech produced in a specific speaking style, at a particular vocal effort and fixed tempo and from a list of test items, not spontaneously generated by the speaker, but chosen by the experimenter. Even such restricted conditions do not prevent the articulatory and acoustic correlates of phonological units from exhibiting extensive variations. Here the variability arises because linguistic entities are not produced clearly delimited one by one in a neat sequence. Rather they form a seamless stream of movements that overlap in time and whose physical shapes depend crucially on how the language in question builds its syllables and how it uses prosodic dimensions such as timing, stress, intonation, and tonal phenomena. This interaction is present irrespective of the language spoken, and whether we observe speech as a motor or acoustic phenomenon. We are referring to the universal aspects of articulatory organization known as coarticulation. 2. Some traditional paradigms in speech perception research How have major theories of speech perception attempted to deal with the abovementioned variability problem? How do they address the ‘speed of processing’ issue? 2.1 “Speech is rather a set of movements made audible than a set of sounds produced by movements” (Stetson 1951). This quotation applies to so-called gestural accounts: the Motor Theory of Speech Perception (Liberman & Mattingly 1985, 1989), Direct Realism (Fowler 1986, 1991, 1994), Dynamic Specification (Strange 1989a, 1989b), Articulatory Phonology (Browman & Goldstein 1992) and its application to phonological development (StuddertKennedy 1987, 1989). Their common denominator is the ‘phonetic gesture’: “. the invariant source of the phonetic percept is somewhere in the processes by which the sounds of speech are produced” (Liberman & Mattingly 1985:21; our italics). "An event theory of speech production must aim to characterize articulation of phonetic segments as overlapping sets of coordinated gestures, where each set of coordinated gestures conforms to a phonetic segment. By hypothesis, the organization of the vocal tract to produce a phonetic segment is invariant over variation in segmental and suprasegmental contexts. " (