Phonetica 2000;57:128–138 Rectived: January 10, 2000 Accepted: February 23, 2000
Osamu Fujimura Ohio State University, Columbus, Ohio, USA
Abstract
The Converter and Distributor (C/D) model is a generative description of artic¬ ulatory gesture organization for utterances. Its input comprises specifications for syllables by features, a paraphonologically augmented metrical structure, and system parameters for utterance conditions. A syllable-boundary pulse train is computed as a time function representing the skeletal rhythmic structure of the utterance. Control functions for articulators are computed by superimposing con¬ sonantal elemental gestures onto the base function, which includes voicing, vocalic, mandibular, and tonal functions associated with the pulse train. A preliminary analysis of microbeam data for experimental dialogues, using jaw opening as an approximate measure of the syllable magnitude, inferred syllable and boundary durations consistently with durational characteristics observed in the acoustic wave¬ form. Copyright © 2000 S. Karger AG, Basel
Introduction
In the history of phonetic studies, production and perception of speech constituted two major approaches in parallel. The underlying assumption in virtually all studies [but see Browman and Goldstein, 1992; Levelt et al., 1999] was that the building blocks of speech signals were phonemes: autonomous and integral segments with asso¬ ciated physical properties. Target values were inherent to individual segments. Coartic- ulation governed signal variation [Lindblom, 1963]. Allophonic variation was under¬ stood to be ad hoc and language-dependent and therefore had to be treated in phonology. Larger units such as syllables were formed under well-formedness con- straints (i.e. phonotactic constraints) of phonemic strings. On the other hand, psy¬ cholinguistic behaviors such as speech errors show that phonemic segments cannot be treated as an unstructured simple linear string [see, among others, Levelt, 1989; Fujimura, 1990; Levelt et al., 1999]. Articulatory data also show that phonemes are not autonomous segmental units [Fujimura, 1990; Sproat and Fujimura, 1993; Krakow, 1999]. A basic deviation from the general principle of segment concatenation and coar- ticulation was proposed by Sven Öhman [1967]. 1 His consonantal perturbation theory treated consonantal gestures independently of the vocalic progression. The Converter and Distributor (C/D) model (see below) is a fuller development of this fundamental concept. Nonlinear phonology also has abandoned the notion that speech is a linear string of phonemic segments, allowing more or less independent organization of differ- ent phonetic control dimensions. Related to the loose linkage among different phono¬ logic variables, i.e. tiers, features are underspecified rather than fully specified segment by segment [Archangeli, 1988; Steriade, 1994; Itô et al., 1995; for a new theory of syl- lable-based phonology, see Haraguchi, 1999]. The C/D model retains the distinction between phonology and phonetics. How- ever, phonetics is dependent on the language as well as utterance situations. If we accept this new view, phonetics must be more abstract and complex than it used to be thought. However, it can be quantitatively explicit and empirically testable. The gen¬ eral principle of coarticulation remains; but phonetics must account for more than what the segment-based theory can [see Kohler, 1990, for relevant discussion].
The C/D Model
The C/D model is a phonetic theory for representing utterances in realistic conver- sational speech [Fujimura, 1994]. It is a computational description of articulatory ges- ture organization using syllables, rather than phonemes, as the concatenative segmental units. Feature specifications in the input representation of the linguistic from are based on phonological contrast considering the syllable as the contextual domain for redun¬ dancy reduction [Fujimura and Lovins, 1978; Fujimura, 1996a]. The Converter, the first component of the model in the sequence of processes for phonetic implementa¬ tion, evaluates the input specifications for the utterance. A base function is constructed to represent the prosodic organization of the given utterance, enhancing a phonological metrical structure with numeric prominence control of selected metrical nodes and numerically setting system parameters for the given utterance conditions. The base function as one of its multidimensional aspects includes vocalic feature specifications for the syllable nucleus, representing the relatively slow articulatory movement from one syllable nucleus to the next, along with mandibular and laryngeal (voicing and tonal) linear progressions, forming discrete phonetic status contours in parallel. Elemental gestures for syllable margins are selected by the Distributor for each syllable component, i.e. p-fix (if any), onset, coda, or s-fix (if any) [Fujimura, 1996a; Fujimura and Williams, 1999]. A multidimensional set of Actuators assemble stored impulse response functions (IRFs) and the IRFs are superimposed upon the base function at the abstract phonetic level of description by Control Function Generators, before the control functions are delivered to the Signal Generator, the last component of the phonetic implementation process [Fujimura, 1996a; Fujimura and Williams, 1999]. 1 A prosody-based description of speech organization was first advocated by Firth [1948, 1957]. A new syllable- based theory was recently published by Levelt et al. [1999]. C/D Model 129 © 2000 S. Karger AG, Basel
Fax +41 61 306 12 34 The Ohio State University, 1070 Carmack Road E-Mail karger@karger.ch
ww.karger.com/journals/pho Columbus, OH 43210-1002 (USA), Tel. +1-614-292 2769 Fax +1-614-292 7504, E-Mail fujimura.1670511.edu www.karger.com Prosodic effects are observed not only in the traditional suprasegmental aspects of speech signals, but also in supralaryngeal articulation, in particular vowel quality along with voice quality control. The C/D model blends phonological functional structure with utterance-specific parameter settings, handling a mixture of symbolic and numeric variables within the phonetic implementation process. The base function is exemplified in figure 1 for the tongue body fronting gesture, evoked as the results of the evaluation of the feature specification {front} 2 considering tautosyllabic feature specifications for the nuclei of the stressed syllables /daet.s/ and /rek.t/. The reduced syllable of 'correct' is assumed, in this figure, to return the under¬ lying vocalic contour value to the resting position. The thin step function, labeled 'fronting contour', represents the underlying phonetic status contour [Fujimura, 1996a] as a syllable-to-syllable flow of the vocalic dorsum gesture. (The time values of the sta¬ tus switching for this contour is assumed to coincide with the abstract edges of the tri¬ angles for the syllables.) A more concrete level of representation for the tongue body front-back movement control is represented by the thick smooth time function as the result of subjecting the status contour signal to a coarticulatory filtering process [see Fujimura and Erickson, 1997]. (The effects of phrase boundaries are not depicted in this figure.) In figure 1, at the top, the phonological structure of each syllable for the sentence is shown according to the C/D model specification scheme. The symbols surrounded by curly brackets represent phonological feature specifications: {9, v} for {interdental, voice), [T, τ] for {apical, stop}, [K, τ] for {dorsal, stop}, {ρ} for {rhotic}. The s-fixes in English are always (apical) and voicing is determined by (voice) (or its lack as default for voiceless obstruents] for the tautosyllabic coda, and therefore only the man- ner feature (in this case { t } for {stop}) is specified. The IRFs for the consonantal ges¬ tures involved in this utterance are also shown below the vocalic function. It should be noted that this figure is speculative, provided only for explanatory purposes. The actual microbeam data, of course, are crucial for making these specula- tions, but none of the curves are to be directly observable in the exact form because of the nonlinear nature of the signal generator. Different actuators excite their IRFs by the relevant pocs (p-fix, onset, coda, or s- fix) [see Fujimura, 1996a] pulses, the thin vertical bars in the bottom panel of figure 1. Upward arrows arising from the top of each pocs pulse point to the excited impulse response functions as ballistic movement patterns for consonantal elemental gestures that belong to each syllable component. For example, the voiceless dorsal stop gesture | K, τ | occurs twice in this utterance, once for the onset of the first (reduced) syllable of 'correct' and the second for the coda of the second syllable of the same word. The peak values of the articulatory gestures are different, directly reflecting the syllable pulse magnitude. It should be emphasized that IRFs are generally different in their inherent shape and amplitude, depending on whether the feature specification is for onset or coda. The timing of the occurrence of the peak relative to the pocs pulse, a time-shifted syllable pulse [Fujimura, 1995], is part of the inherent property of the IRF (note the directions of the upward arows suggesting the IRF excitation process). The use of a particular articulator (for example the tongue tip) may be critical for a consonant in onset but not in coda (for example, replacement of [t] by a glottal stop, no apical contact for [1], etc., depending on the dialect in English). Such ad hoc qualitative dif- ferences in manifestation are described as differences of IRFs corresponding to the same feature specification, e.g. {lateralo} vs. {lateralo}, rather than allophonic rules in phonology. Their difference with respect to sensitivity to prosodic conditions is accounted for, at present, by the combination of the difference in IRFs and the effects of nonlinear mapping from the abstract control function to articulatory or acoustic physical signals.
Prosodic Control
At the bottom of figure 1, the temporal succession of triangles computed from the syllable pulse train is shown. The three syllables are identified by numerals according to their sequential occurrence in the utterance. The magnitudes of the thick pulses rep¬ resent the phonetic strengths of the syllables. An augmented metrical grid is assumed to be computed from the input specification of the phonological metrical structure, as in Liberman and Prince [1977] (metrical tree), Prince [1983], Hayes [1984] (metrical grid), or Halle and Idsardi [1995] (parentheses representation), phonetically augmented by paralinguistic specifications such as prominence enhancement for contrastive emphasis [Fujimura, 1994; Erickson, 1998]. Such a numeric metrical grid (i.e. the sequence of thick vertical bars in the bottom panel of the figure) represents an ordered 2 The feature [front] may be considered equivalent to [-back] in this paper. 130. Fujimura C/D Model .131 series of pulses for syllables, the pulse height representing the syllable magnitude. The shadow computation constructing a triangle for each syllable using the same angles but different magnification factors, according to the pulse height, yields a series of trian- gles left to right [Fujimura et al., 1998]. The contiguity of syllable triangles is broken by occasional gaps, representing boundaries of continuously variable strengths [see Sproat anf Fujimura, 1993, for discussion of various boundaries]. The boundary magnitude is assumed to be computable from the augmented metri¬ cal structure. The shadow angles for syllables and boundaries remain to be determined according to empirical data for the given utterance conditions (see below). Boundary features may evoke manifestations of certain supralaryngeal gestures as well as laryn- geal and pulmonary gestures [see Fujimura and Williams, 1999, for discussion of Japa- nese geminate obstruents and Spanish rhotics]. The voicing contour as a control function may be assumed to always exhibit, in any language, a single island per syllable [Fujimura, 1996a]. In other words, if there is a voice cessation in the control function of voicing, that must involve a syllable bound¬ ary in the abstract phonetic sense. This implies that the abstract phonetic status contour is syllable-based, the voice onset in the beginning part of the syllable and the voice off- set in the ending part of the syllable are implemented at time values relative to the onset and coda pulses, according to the phonological feature specifications for onset and coda (or in some languages s-fixes or p-fixes), respectively. Voicing can continue across syllable boundaries if no voice cessation occurs in the control function within the contiguous syllables, or if such cessations do not appear in phonetic signals due to coarticulatory undershooting [Lindblom, 1963]. Vocal fold vibration is controlled in different dimensions. The glottal adduction/ abduction is one, tension (primarily cricothyroid muscle contraction) is another. Stiff- ness control (vocalis muscle contraction) can be independent of tension and many be implemented via an IRF corresponding to a consonantal feature specification. Fo is only one of the physical manifestations of the voice quality control, but is acoustically robust and plays the central role in prosodic perception along with temporal modula- tion of phonetic events.
Inferring Syllable-Boundary Pulses from Jaw Movement
Using the C/D model as a prosodic description for an utterance, an empirical test of the validity of the model is underway. Specifically, to test, in part, the concept of the syllable-boundary pulse train, the following hypotheses were adopted as an instantiation of the model to study the rhythmic organization of articulatory patterns of utterances in a fairly natural dialogue between the experimenter [Donna Erickson, see Erickson et al., 1998] and the subject, as recorded by the X-ray microbeam sys¬ tem. For the interpretation of data, we hypothesized as follows: (1) Syllables are repre- sented by symmetric triangles of different sizes but with the same angle throughout an utterance, arranged along the time axis (fig. 1). (2) In each utterance analyzed, there is at least one occurrence of contiguous syllables without a prosodic boundary in between. (3) The maximum angle causing no overlaps of contiguous syllable triangles is the shadow angle, the tangent of which determines the ratio of the syllable magnitude to an abstract syllable duration for the utterance condition. (4) The remaining gaps between contiguous syllable triangles represent boundaries, their lengths representing the magnitudes (strengths) of the boundaries. Now specifically referring to the articulatory manifestation of the abstract syllable magnitude, we further assume an empirical approximation as follows: (5) Syllable mag¬ nitude is manifested as relative jaw opening provided the syllable nuclei are specified with the same phonological features. This implies that the observed excursion of mandibular motion for the syllable nucleus /a/ in the key items /faJv/, /naJn/, /paJn/, in the context of the current experiment, reveals directly the prosodic component (as opposed to phonologically specified inherent gestures of jaw movement which are the same in all tokens) of jaw control function as first approximation [Fujimura et al., 1998]. Data were recorded for 2 male and 2 female native speakers of Midwestern Amer- ican English. The experimenter conducted a dialogue with the subject whose articula¬ tion was recorded, along with the acoustic speech signal, by the microbeam system at the University of Wisconsin, Madison. The dialogue used 'five', 'nine' and 'pine' as the target words in a street address phrase, but was relatively freely designed to allow 132 the subject to use varied expressions. S/he was instructed beforehand that the partner of the dialogue, i.e. the experimenter, might need corrections because she was in a noisy communication environment. The address to be used in the response was given to the subject in a monitor display, and the subject's response always included at least three tokens of 'five' or 'nine', but not a string of three identical numbers. The street name was always 'Pine Street'. Figure 2 illustrates an example of the results of our semiautomatic interpretation of the syllable magnitude and timing data. The key words are marked automatically as NINE of FIVE for the crucial articulator of the consonant involved, for the purpose of visually confirming demisyllabic iceberg patterns [Fujimura, 1986, 1996b]. The mid¬ point or the iceberg threshold crossing points of the crucial articulator for the demi¬ syllable was used as the temporal center of the syllable triangle. The simultaneously recorded acoustic signal is shown at the bottom as a compressed waveform. The top panel of figure 2 represents the inferred syllable magnitude (triangle height) by copying the extent of mandibular lowering. The syllable 'duration' was computed according to the hypotheses above. Usually, as seen in this example, we found multiple instances of syllables representing the target words (all with /aJ/) approximately contacting with each other, while other instances of contiguous target words in the same utterance left a considerable gap in time (indicating boundaries). We checked the acoustic signals to determine if the magnitude of each intertrian- gular gap reasonably compared with what we could judge as an acoustic pause, when- ever determinable including voiced or unvoiced consonantal segments. We also lis¬ tened to the speech signals carefully to compare the perceptual impression of phrasing patterns with the articulatorily inferred boundary strengths. The result was quite rea¬ sonable without exception for some randomly sampled dialogues by each of the 4 speakers. The phrasing strategy seemed to vary quite considerably from speaker to speaker, however [Fujimura et al., 1998]. A complete data analysis is underway.
Prosodic Effects on Articulation
We would like to emphasize here that, unlike the traditional concept of prosody and suprasegmentals, the so-called segmental articulation, in particular the vowel artic- ulation, varies considerably, depending on the prosodic condition of the word, even when the word sequence uttered is exactly the same, i.e. when the segmental context is identical [Fujimura, 1990]. Figure 3 compares the same sentence: 'It's six five seven America Street', uttered in two different prosodic conditions, as observed in one of the earlier microbeam data obtained at the University of Wisconsin Microbeam Facility (simulated dialogue read from a monitor screen by the subject). The utterance in figure 3a had a contrastive emphasis placed on 'six', as a response to a question (also read by the subject) which contained the 'wrong' digit. In figure 3b, the word 'America' was emphasized as a correcting response to the question which cited the 'wrong' street name. The curves represent time functions pertaining to the vertical component of the movement of the pellet attached on the tongue blade surface, about 1 cm behind the tip of the tongue. The subject was a young male native speaker of the Midwest American dialect of English. The upward arrow in each figure points to the lowest position of the tongue blade for the vowel [i] in 'six', while the bracket shows the time domain corre- sponding to the word 'America'. It is apparent that the high front vowel in 'six' is articulated with a considerably lower blade position when the word is emphasized than not emphasized. Similarly, the Fig. 3. Tongue blade height in a pair of utterances, 'It's six five seven America Street', with con- 134 C/D Model word 'America', when emphasized, is pronounced with more vigorous blade move¬ ment and also expanded time span. A particularly striking effect of emphasis is observed in the articulation of the word-initial schwa (see the deepest minimum in the bracketed time domain) in 'America'. The blade is considerably lower for schwa than for the low vowel in 'five' (unemphasized) in the same utterance (fig. 3b), while not only the low vowel in 'five' but also the mid front vowel in 'seven', both not empha¬ sized, show lower blade position than the schwa in figure 3a. It should be noted, however, that in figure 3a for the unemphasized 'America', the word/phrase-initial schwa is more distinctly pronounced than the word-final schwa, the latter showing only an inflection point rather than any local minimum, and this is true even in figure 3b even though the absolute vertical position is lower in figure 3b than in figure 3a. The scales for vertical position in figure 3a, b are calibrated and are the same from top down. In figure 3b, a strong but slow action of jaw opening (not shown here but replicated consistently in later experiments [Erickson, pers. commun.]) starts from the beginning of the word, anticipating the succeeding stress syllable. This is particu¬ larly true when it is emphasized. The mandibular movement is reflected in the deep downward excursion of the tongue blade for the word-initial schwa, indicating a gen¬ eral prominence effect of the unit-initial syntagmatic position [see Keating, 1995]. It is seen in figure 3b that the tongue blade for the mid front vowel, with main stress, is not as low as for the initial schwa in the same emphasized word 'America'. Presumably, the inherent tongue body gesture for this feature-specified mid front vowel is articu- lated with an enhanced movement because of the emphasis. Such intricate interactions between the inherent vocalic gestures and the jaw movement gesture [see Maeda, 1992] have been consistently observed also in a recent microbeam study of the effects of emphasis using different vowels [Erickson et al., 1999]. Thus, enhancing the inherent articulation of the vowel, vocalic gestures seem to be hyperarticulated [Lindblom, 1990] when the syllable prominence (i.e. syllable magni- tude) is increased by emphasis. At the input to the C/D model, the metrical tree node representing the emphasized word receives a numeric enhancement factor, which results in the magnification of the grid height, representing the metrical value of the stressed syllable as the head of the word. The increased syllable pulse height of the directly affected syllable causes three main effects in parallel. First, the deviation of mandibular position (opening excursion) from the contextually determined neutral position for the pertinent syllable is increased according to the numeric enhancement. The jaw opening excursion deviating from the natural course of movement for the computed syllable pulse height, reflecting both the lexical metrical value (i.e. the phonologically determined degree of stress) and the syntagmatically determined phonetic degree of jaw opening (for example, phrase and word-initial position makes it more open than for final position). Second, the enhancement of the inherent vocalic gesture for the specified phono- logical feature is implemented in the vocalic aspect of the base function. In the case of 'America', the feature specification is {front} for the mid (unmarked) front lax (unmarked) 3 vowel of the second syllable corresponding to the head node of the phono¬ logical word. The inherent phonetic gesture for this feature specification is tongue body advancement as a phonetic status [front], and this inherent gesture is enhanced via the syllable pulse enhancement. The third effect of the enhanced syllable magnitude is the proportional increase in amplitude of all tautosyllabic consonantal elemental gestures as their common excita¬ tion pulse is made larger. Such an increase of an abstract gesture (i.e. IRF amplitudes as the passive response to the syllable pulse excitation) is not necessarily directly observ- able because of the highly nonlinear nature of the signal generator, the last stage of the model [Fujimura, 1994; Fujimura and Williams, 1999], typically showing saturation of the movement of the critical articulator of the consonant against the roof of the mouth including the contact of the sides of the tongue blade and local feedback within muscle spindles, etc. [Fujimura, 1998]. A quantitative evaluation of our interpretation must await a computational simula¬ tion of the dynamic, nonlinear deformation of the tongue and other organs in large deformation, along the line Wilhelms-Tricarico [1995] has shown. The effect is semi¬ quantitatively observable, however, in the form of a durational expansion of the ele¬ mental gesture manifestation both in articulatory movement patterns, as in figures 2 and 3, and in the acoustic segmental durations. The enhancement effect seems more obvious for onset gestures than for coda gestures. This is partly explained by difference between onset vs. coda IRFs with respect to the relation between the inherent peak height of the IRF and the effective threshold value of the articulator's (including vocal folds') position for acoustic discontinuities such as stop formation and release, voice onset and offset, etc. [Lehiste, 1970; Umeda, 1977; Crystal and House, 1997]. It is possible, however, that, in the case of the word 'America' shown above, the underlying phonological vocalic specification is not exactly unmarked (i.e. no feature specification) for the first syllable. It may be that some vocalic feature specification is there at the output level of (phrasal) phonology (as a reducible low vowel) for the ini¬ tial syllable of 'America', but syllable reduction at the Converter level of the phonetic implementation process suppresses the phonetic manifestation of the underlying fea¬ tures in typical utterance situations, by assigning the reduced phonetic status according to the metrical value of the syllable. Such a matter needs to be studied systematically with much more data. It is the intention of this paper that the C/D model, unlike tradi- tional phoneme-based segmental and suprasegmental models, is in principle capable of handling such situations [see Leben, in press, for a related discussion]. The goal of this study is to account for the diversity of observable effects of prosodic variation by pro- viding an effective framework of phonetic description.
Acknowledgment
This research has been supported in part by NSF (SBR-9511998, BCS-9977018), ATR/HIP, and ATR/MIC. The author would like to acknowledge effective collaboration by Donna Erickson, Bryan Pardo, Corey Mitchell, and Caroline Menezes. John Westbury and his staff at the Microbeam Facility, University of Wisconsin, assisted in the data collection for the experiments discussed here. The author is also grateful for editorial comments by Klaus Kohler and John Kingston, which contributed to a sub- stantial improvement of this article. 136 C/D Model 137