USING NEURAL NETWORKS TO INVESTIGATE THE RELATIONSHIP BETWEEN SPEECH PRODUCTION AND PERCEPTION

The relationship between speech perception and speech production is a complex one. Different views have been offered on this relationship in the past. In particular, the question of the extent to which perception mirrors production has been contested. For this relationship to be fruitfully investigated it would be advantageous to adopt some objective means of deriving perceptual hypothesis from production data. This paper shows that simple neural networks can be used for this purpose. Two simulation experiments will be presented and compared with experiments dealing with the rate-dependent perception of Voice Onset Time (VOT) and of duration as a cue for vowel and consonant quantity. Results show that a neural network containing no hidden units is able to predict the result of a perception experiment dealing with vowel and consonant durations as cues to quantity, even at variable speaking rates, but fails to predict the results of an experiment dealing with the rate-dependent perception of VOT. 1. RELATING PRODUCTION AND PERCEPTION It would appear to stand to reason that listeners perceive speech from the acoustic signal and therefore that one should expect a tight fit between the acoustics of speech and speech perception. As is well known, however, theories of sp eech perception have often questioned this fit, and been struck by the fact that speech acoustics and perception sometimes seem to “go their separate ways”. Much was made of this in early work at the Haskins Laboratories, leading Liberman [5] to pose the question “When articulation and soundwave go their separate ways, which way does the perception go? ” His answer was that “perception always goes with articulation. ” This sentiment was later reflected in the motor theory of speech perception [6]. This view, t hough, has by no means gone undisputed. Thus Fant [3] made a forceful claim to the effect that “articulation and sound waves never go separate ways”. One factor which has made it difficult to resolve the question as to the tightness of the fit between acoustics and perception, is the fact that speech is highly variable. This variability is caused by numerous factors having to do with differences in voice quality, the effect of coarticulation, different utterance rates and so on. Speaking rate is a particularly important cause of variability with respect to temporal speech cues. A wellknown cue of this kind is Voice Onset Time (VOT) which is a measure of the time between oral release of a stop consonant and the start of vowel voicing. In languages such as English and Icelandic — which have two series of stop consonants, /bdg/ and /ptk/ — the difference between these two categories is conveniently captured by the VOT metric. The stops of the first series have short-lag VOTs of perhaps 10–25 ms, the latter series show much longer values of VOT, typically longer than 60 ms. It has long been realized that a temporal sp eech cue such as VOT is susceptible to extraneous influences, caused e.g. by changes in speaking rate. These changes have nothing to do with the linguistic message which is conveyed by VOT and the question arises as to how the listener will cope with the problem of keeping the linguistic and the non-linguistic factors apart. Production studies of rate effects have commonly shown that with slower speech the contrast between temporally contrasting phonemes increases. Miller and Baer [8] showed this for transition durations as a cue for the [b-w] distinction, and Miller, Green and Reeves [9] reported a similar finding for VOT. In both cases the greatest change was seen in the segment cued by the longer transitions [w] or the longer VOTs (aspirated stops). Thus in the case of VOT, slower utterance rates will lengthen the VOT for aspirated stops more than the VOT for unaspirated stops, increasing the separation of the two categories. This finding also clearly emerged from a study of the effect of speaking rate on VOT in Icelandic [12]. Research on the perception of VOT has shown that listeners appear to be influenced by the perceived speaking rate. In the case of VOT phoneme boundaries separating short and long lag categories of stops move to longer values of VOT when the overall duration of the syllables is lengthened [10, 16]. Thus a study of American English speakers found that the phoneme boundary in a /bi-pi/ continuum was located at 43.9 ms when the syllables were 325 ms long, but at 35.6 ms when the syllables were 125 ms long [10]. Such shifts in the location of the phoneme boundaries as a function of the perceived rate are commonly thought to reflect a “taking-intoaccount” type of perceptual mechanism — in this case appropriately termed “ratedependent perception”. This notion of “taking-into-account” is well-established in perceptual psychology in general, where it has been used e.g. to account for the perceptual constancies [2]. In the case of visual perception it is thus commonly found that the perception of size depends on the perceived distance of the object from the viewer. Size constancy is, at least over not too large distances, remarkably robust. Assuming that a similar type of mechanism is operating in phonetic perception the question arises as to the extent of the taking-into-account. Will the listener compensate perfectly for page 2085 ICPhS99 San Francisco changes in speaking rate? This question has received much less attention than it deserves though some attempts have been made to explore quantitatively the degree of perceptual normalization in rate-dependent perception. In their study of the behavior of VOTs under transformations of rate, Miller et al. [9] introduced the notion of a “variable optimal boundary” Miller et al. calculated this variable optimal boundary by sorting their VOTs measured at various utterance rates into 50 ms bins and calculating hypothetical phoneme boundaries that would optimally separate the two categories of unaspirated and aspirated stops for each bin. Miller et al. showed that this optimal boundary increased with slower utterance rates (as judged from the overall duration of the syllables) from 14 ms of VOT in 100–149 ms long syllables to 52.5 ms of VOT in 650–700 ms long syllables. The perceptual shifts are typically considerably more limited, as shown by the findings of Miller and Volaitis [10]. It is clear from the studies referenced so far that the question of how perception mirrors production can be fruitfully investigated by attempting to predict the results of perceptual experiments assuming that the production data determine the results of perceptual experiments. Various methods can be used to derive such calculations. The purpose of this paper is to argue for the usefulness of using simple neural networks for this purpose. This will be illustrated with reference to two experiments where stimuli for perception experiments were carefully modeled on production data. 2. NEURAL NETWORKS Neural networks have revolutionized the study of pattern recognition in the last decade or so. They have also had great influence on modeling work in psychology and, to a lesser extent, linguistics. It is the position of this paper that neural networks can also be a useful tool for probing the relationship between speech production and perception. Neural networks come in various sizes and shapes and use a multitude of different learning algorithms. One particular kind of neural network is that popularized by Rumelhart, McClelland and co-workers [15] which makes use of the “backpropagation” algorithm for training the network. A backpropagation network contains a number of connected nodes in a layered net. A network would typically have a layer of input nodes and a layer of output nodes. It might also contain a “hidden” layer of nodes interposed between the input and output layers. The task for the network is to learn some mapping between input and output. The input might be a set of English verbs in the present tense, the output the corresponding set of past tenses of the English verbs. The network has only one way of learning the mapping of input to output and that is by modifying the weights connecting the nodes of the network. These weights are automatically set by the learning algorithm used by the network, such as the backpropagation algorithm. Backpropagation is a supervised learning algorithm which means that the network will learn by comparing the output of the net to the desired output. If these do not agree an error is calculated from the difference between the actual net output and the desired output. This error is then “backpropagated” through the net leading to adjustments of weights between nodes. The backpropagation net learns by “gradient descent”, i.e. by attempting to minimize the error by moving downhill in an “error landscape”. Backpropagation is the most popular learning algorithm employed in current neural networks. Numerous simulators are available which make it easy for researchers to design their own neural networks. One such simulator is the tlearn simulator developed by Jeffrey Elman, Kim Plunkett and associates and r ecently described in detail in a number of accessible publications [1, 7, 14]. This simulator, available for both Macintosh and Windows, is extremely easy to use and has been used in the following simulations. 3. FEEDING SPEECH DATA TO A SIMPLE NEURAL NETWORK The tlearn simulator will be used here to answer the following question: Based on production data, say on the duration of vowels and consonants in stressed syllables in Icelandic (these being either of the type V:C or VC:), what predictions can be made from the production data about the optimal way of distinguishing the two categories of speech segments? A simple neural network, shown in Figure 1, will be used to make this prediction. This will be done by feeding measurement data to the network (the input signal) along with a description of the intended category, V:C or VC:. The intended category forms the d