Zipfian word frequencies support statistical word segmentation

Zipfian word frequencies support statistical word segmentation Chigusa Kurumada kurumada@stanford.edu Department of Linguistics Stanford University Stephan C. Meylan smeylan@stanford.edu Department of Psychology Stanford University Abstract Word frequencies in natural language follow a Zipfian dis- tribution. Artificial language experiments that are meant to simulate language acquisition generally use uniform word fre- quency distributions, however. In the present study we exam- ine whether a Zipfian frequency distribution influences adult learners’ word segmentation performance. Using two exper- imental paradigms (a forced choice task and an orthographic segmentation task), we show that human statistical learning abilities are robust enough to identify words from exposures with widely varying frequency distributions. Additionally, we report a facilitatory effect of Zipfian distributions on word seg- mentation performance in the orthographic segmentation task, both in segmenting trained material and in generalization to novel material. Zipfian distributions increase the chances for learners to apply their knowledge in processing a new speech stream. Keywords: Word segmentation; statistical learning; Zipfian frequency distributions. Introduction Humans and other animals extract information from the en- vironment and represent it so that they can later use these representations for effective recognition and inference. One striking example of this phenomenon is that adults, children, and even members of other species can utilize statistical infor- mation to segment an unbroken speech stream into individual words after a short, ambiguous exposure (Saffran, Aslin, & Newport, 1996; Saffran, Newport, & Aslin, 1996; Aslin, Saf- fran, & Newport, 1998; Hauser, Newport, & Aslin, 2001). In a now-classic segmentation paradigm, Saffran, Newport, and Aslin (1996) played adults a continuous stream of synthe- sized speech composed of uniformly-concatenated trisyllabic words. After exposure to this stream, participants were able to distinguish the original words from “part-words”—chunks that had occurred with lower frequency and lower statisti- cal consistency. This work, combined with demonstrations with infants, suggested that statistical segmentation could be a viable method for early language learners to learn the word forms of their native language. While the results of these experiments are impressive, the ways in which these findings can be applied to understand natural language learning are still unclear. Recent research has begun to close this gap. The outputs of this statistical segmentation process are now known to be good targets for word-meaning mapping (Graf Estes, Evans, Alibali, & Saf- fran, 2007), and experiments with natural language sample suggest that the processes observed in artificial language ex- periments generalize to highly-controlled natural language samples (Pelucchi, Hay, & Saffran, 2009). In addition, sta- tistical segmentation has been shown robust to variation in Michael C. Frank mcfrank@stanford.edu Department of Psychology Stanford University sentence and word lengths (Frank, Goldwater, Griffiths, & Tenenbaum, 2010). Nevertheless, there are many links be- tween statistical segmentation and natural language learning that still have not been made. One key difference between standard segmentation paradigms and natural language is the distribution of frequen- cies. The empirical distribution of lexical items in natural language follows a Zipfian distribution (Zipf, 1965), in which relatively few words are used extensively (“the”) while most words occur only rarely (“toaster”). In particular, the absolute frequency of a word tends to be approximately inversely pro- portional to its rank frequency. While Zipfian distributions are ubiquitous across natural language, their consequences for learning are only beginning to be explored (Goldwater, Griffiths, & Johnson, 2006). The current paper investigates the consequences of Zipfian frequency distributions for sta- tistical word segmentation. Saffran, Newport, and Aslin (1996) hypothesized that the mechanism underlying statistical word segmentation was the computation of syllable-syllable transitions. In a uniform dis- tribution, nearly every word follows every other word, so these transition matrices are quite well-populated, but in a Zipfian language, they are very sparse. Some combinations of frequent words will have high transition probability between them (especially if they co-occur together frequently). If syl- lables are used in multiple words, the within-word transition probabilities for low-frequency words could be considerably lower than the between-word transition probability for high frequency words. This factor may have led to the low perfor- mance of transition-based models in computational compar- isons (Yang, 2004; Brent, 1999). Thus, the first question we ask in the current study is whether human statistical learning abilities can succeed in segmenting Zipfian-distributed input. Examining the problem from another side, however, a Zip- fian distribution might actually provide more information for segmentation. Bannard and Lieven (2009) suggest that repet- itive use of restricted types of words and word combinations in input are a strong predictor of the order of acquisition. In addition, six-month-olds can already exploit highly familiar words to segment and recognize adjoining unfamiliar words from fluent speech (Bortfeld, Morgan, Golinkoff, & Rathbun, 2005). Thus, our second question is whether (and under what conditions) Zipfian input could facilitate word segmentation. To address these two questions, we compared segmen- tation performance in uniform and Zipfian contexts across three different large-scale web-based segmentation experi- ments. Since Frank, Arnon, Tily, and Goldwater (2010) pro- vided evidence that crowd-sourcing platforms reliably repli-