Word forms - not just their lengths- are optimized for efficient communication

The inverse relationship between the length of a word and the frequency of its use, first identified by G.K. Zipf in 1935, is a classic empirical law that holds across a wide range of human languages. We demonstrate that length is one aspect of a much more general property of words: how distinctive they are with respect to other words in a language. Distinctiveness plays a critical role in recognizing words in fluent speech, in that it reflects the strength of potential competitors when selecting the best candidate for an ambiguous signal. Phonological information content, a measure of a word’s string probability under a statistical model of a language’s sound or character sequences, concisely captures distinctiveness. Examining large-scale corpora from 13 languages, we find that distinctiveness significantly outperforms word length as a predictor of frequency. This finding provides evidence that listeners’ processing constraints shape fine-grained aspects of word forms across languages. Despite their apparent diversity, natural languages display striking structural regularities (Greenberg, 1963; Evans and Levinson, 2009; Futrell et al., 2015). How such regularities relate to human cognition remains an open question with implications for linguistics, psychology, and neuroscience (Hauser et al., 2002; Evans and Levinson, 2009; Kemp and Regier, 2012; Fedzechkina et al., 2012). Prominent among these regularities is the well-known relationship between word length and frequency: across languages, frequently-used words tend to be short (Zipf, 1935). In a classic work, Zipf (Zipf, 1935) posited that this pattern emerges from speakers minimizing total articulatory effort by using the shortest form for words that are used most often, following what he later called the Principle of Least Effort (Zipf, 1949). While the underlying cause has been the subject of debate (Yule, 1944; Miller, 1957; Cancho and Solé, 2003; Conrad and Mitzenmacher, 2004; Piantadosi, 2014), this relationship between word length and frequency remains one of the most robust statistical laws that describe human languages. Here, we propose a generalization of Zipf’s analysis and present two possible listener-focused explanations. We show that a word’s frequency is inversely related to its distinctiveness—how easily it can be identified as the intended message for a given speech signal. While speakers prefer easier-toproduce less distinctive forms, they are constrained by listeners’ need for sufficiently distinctive forms to differentiate each word from others, especially if a speaker’s intended word has higher-frequency competitors. If word recognition is modeled as Bayesian inference (Norris and McQueen, 2008; Balling and Baayen, 2012), the probability of successful recognition depends on both the prior probability of the intended word and on the number and strength of alternative words (“competitors”). We define a statistical measure of distinctiveness that succinctly captures the diagnosticity of a ar X iv :1 70 3. 01 69 4v 1 [ cs .C L ] 6 M ar 2 01 7 word form by assessing the aggregate strength of competitors in the language. We then show that distinctiveness should be inversely related to frequency, if languages are constructed to equalize error rates for low and high frequency words. Importantly, distinctiveness subsumes Zipf’s observation regarding the relationship of length and frequency as a special case. Length is a naïve approximation of the distinctiveness of a word form if longer strings are simply less probable. We demonstrate that a more comprehensive measure of distinctiveness that takes into account the sound-to-sound and letter-to-letter sequences in a language accounts for significantly more frequency-related variance than does length across a broad sample of natural languages. This relationship between frequency and distinctiveness adds to a growing body of evidence that cognitive constraints influence the structural properties of natural languages (Hawkins, 1994; Fedzechkina et al., 2012; Futrell et al., 2015). Model We define a probabilistic language model to characterize the distinctiveness of word forms in terms of their constituent sound-to-sound or letter-to-letter transitions. This model can be fit using a large written sample of a particular language. The starting point for the model is formalizing the task of the listener as a rational statistical inference. Bayesian inference and distinctiveness Upon hearing a string of sounds s, a listener has to infer what word w was intended by the speaker. This can be formulated as a problem of Bayesian inference. The listener should calculate a posterior distribution P (w|s) over words based on the sounds. Applying Bayes’ rule, this is given by P (w|s) = P (s|w)P (w) P (s) (1) where P (s|w) is the probability of hearing s if w is the intended word, P (w) is the prior probability of the word w intended by the speaker, and P (s) is the probability of hearing s. Assuming that sounds are produced faithfully, such that P (sw|w) is close to 1 for a particular string sw for each w and close to 0 otherwise, we obtain the approximation P (w|sw) ≈ P (w) P (sw) (2) which expresses the probability that the word w is correctly identified as a function of its normalized frequency, P (w), and the probability of the string sw in the language, P (sw). We define the distinctiveness of a word to be inversely related to P (sw): intuitively, a word that shares the same sound sequences with many other words is less distinctive. Following this logic, we use the phonological information content (PIC) of a word to measure the distinctiveness of its phone-to-phone (sound-to-sound) transitions, or its approximation in characterto-character transitions (Cohen Priva, 2008). Analogous to the metric of lexical surprisal (negative log conditional probability) used to measure the predictability of a word given preceding words (Levy, 2008; Piantadosi et al., 2011; Smith and Levy, 2013), PIC is the negative log probability of the sequence of phones (distinct meaningful sounds) or letters in the string comprising a word. To obtain a compact representation of the probabilities of various phone or character sequences in a language we estimate an n-phone or n-character model (analogous to an n-gram model over words (Manning and Schütze, 1999) but computed over individual phones or characters) from a corpus sample. To support a stronger test of the relationship between distinctiveness and frequency, we avoid the circularity that more probable words necessarily contain more probable sequences by estimating the transition probabilities using the type inventory (unique words) in the language. Under this model, the phonological information content PIC(w) of a word is defined as: PIC(w) = − logP (sw) (3) = − logP (l1, . . . , l|sw|) for l ∈ sw (4)