Is Recurrence Redundant? Revisiting Allen and Christiansen

Is Recurrence Redundant? Revisiting Allen and Christiansen (1996) C. Anton Rytting (rytting@ling.ohio-state.edu) The Ohio State University Department of Linguistics 222 Oxley Hall 1712 Neil Ave. Columbus, OH 43210 USA Abstract Several influential connectionist models of the word segmenta- tion task (e.g. Cairns, Shillcock, Chater, & Levy, 1997; Chris- tiansen, Allen, & Seidenberg, 1998) follow Elman (1990) in using simple recurrent networks (SRNs). The use of SRNs in this context appears to be traditional rather than independently motivated. This paper investigates whether alternatives to SRNs can achieve similar performance. Specifically, it reports a repli- cation of part of Allen and Christiansen (1996), and a compar- ison with multi-layer perceptrons (MLPs). The major results are confirmed; however, the SRN topology is shown not to be necessary for the task addressed. MLPs perform equally well, even when they have access to less context. Introduction Word segmentation is the task of recovering word boundaries from an initially unsegmented stream of linguistic input. En- glish writing marks these boundaries, making the task trivial, but Chinese writing does not, neither does any variety of nat- ural speech. If a language learner is to make any headway in acquir- ing the words in a language, that learner must learn how to pull these words out of running speech and hear them as sep- arate words. By learning the indirect cues that signal word boundaries for a given language, the word learner solves the word segmentation task. This begins early in the language acquisition process: studies show that babies begin to learn word segmentation strategies around 7-9 months after birth (see e.g. Jusczyk, Houston, & Newsome, 1999, for a review). Many of the initial experimental studies of word segmen- tation focused on individual cues: prosody (Jusczyk, Cut- ler, & Redanz, 1993), allophonic variations (Jusczyk, Hohne, & Bauman, 1999), distributional regularities (Saffran, Aslin, & Newport, 1996), and phonotactics (Friederici & Wessels, 1993). Similarly, early computational simulations of the word segmentation task focused on single cues. Brent and Cartwright (1996) used distributional regularities as a cue, combined with minimum description length as a language- independent heuristic. Cairns et al. (1997) also modeled the use of distributional regularities, but as a precursor to a later stage using stress (which was not explicitly modeled). Aslin et al. (1996) modeled the generalization of segmental cues at the ends of utterances to the ends of words. Christiansen, Allen, and Seidenberg (1998) explicitly ad- dresses the interaction among several cues: distributional reg- ularities, stress, and utterance-boundary information. Their model reflects a particular conception of the word segmen- tation task within the larger context of language acquisition. The infant’s primary task is the comprehension of the speech around it (as well as producing comprehensible speech in turn). As the child attends to the speech input to which it is exposed, it notices patterns in the signal (Saffran et al., 1996). Christiansen et al. (1998) conceive of a second, immediate task in language acquisition: namely, to continually update a statistical representation of these salient patterns. This process is exemplified in a simple model presented in Allen and Christiansen (1996). In this scenario, a network receives input one phone at a time. Its immediate task is learn to predict the identity of the next phone. 1 The immediate task is “self-supervised” in that the next input provides immediate feedback as to what the preceding output should have been. To model the contribution of distributional regularities at the phone level to the discovery of word boundaries, a neural network with an input unit and an output unit for each phone may be constructed, and the likelihood of a word boundary approximated by measuring points at which the error in pre- dicting the next phone is high (see Cairns et al., 1997, for a model based on this idea). In order to model the interaction of utterance-boundary info and distributional regularity, an ad- ditional unit, symbolizing an utterance boundary or pause in speech, is added to the input layer, and corresponding unit to the output layer. The idea of having multiple output nodes learning multiple problems simultaneously is referred to as “hints” or “catalyst” nodes (see Suddarth & Kergosien, 1991). The intuition (following Aslin et al., 1996) is that since utter- ance boundaries are a subset of word boundaries with more or less representative properties, the utterance boundary out- put unit should be activated not only at utterance boundaries, but (to a lesser degree) at all word boundaries. Hence, above- average activation of the utterance boundary output unit is used to measure the network’s performance on the derived task: identifying word boundaries. The other output units, corresponding to phones, can be thought of as catalyst units to the utterance boundary unit. A major point of Allen and Christiansen (1996) is that word boundaries can be learned in this way only for languages with particular statistical properties. It is straightforward to con- struct artificial languages that lack these properties. Specifi- cally, for a language where each syllable is equally likely to follow any other (or to end a word or an utterance), the net- work can learn at best only syllable boundaries. But if certain segments or groups of segments (such as syllables) are more 1 The assumption that the identity of the phone is unambiguously observable is itself an idealization. In real life, the intended segment is not observable with perfect accuracy, particularly for infants who may be presumed to be still learning the language’s inventory of sounds. The automatic speech recognition community rightly treats phone identity as part of the hidden structure, not the observable. Fully addressing this problem would unduly complicate the model under discussion, although Christiansen and Allen (1997) addresses it in part.