Revisiting the Optimality of Word Lengths
Authors: Tiago Pimentel
Clara Meister
Ethan Wilcox
Kyle Mahowald
Ryan Cotterell
Venue: EMNLP 2023 Main (2023)
Keywords: word length
uniform information density
zipf
law of abbreviation
TL;DR: We formalize the lexicalization problem: how to select wordforms in a lexicon to minimize a specific communicative cost function. We then use it to derive Zipf's law of abbreviation and optimal word lengths under the channel capacity hypothesis.

Abstract: Zipf (1935) posited that wordforms are optimized to minimize utterances' communicative costs. Under the assumption that cost is given by an utterance's length, he supported this claim by showing that words' lengths are inversely correlated with their frequencies. Communicative cost, however, can be operationalized in different ways. Piantadosi et al. (2011) claim that cost should be measured as the distance between an utterance's information rate and channel capacity, which we dub the channel capacity hypothesis (CCH) here. Following this logic, they then proposed that a word's length should be proportional to the expected value of its surprisal (negative log-probability in context). In this work, we show that Piantadosi et al.'s derivation does not minimize CCH's cost, but rather a lower bound, which we term CCH-lower. We propose a novel derivation, suggesting an improved way to minimize CCH's cost. Under this method, we find that a language's word lengths should instead be proportional to the surprisal's expectation plus its variance-to-mean ratio. Experimentally, we compare these three communicative cost functions:  Zipf's, CCH-lower , and CCH. Across 13 languages and several experimental settings, we find that length is better predicted by frequency than either of the other hypotheses. In fact, when surprisal's expectation, or expectation plus variance-to-mean ratio, is estimated using better language models, it leads to worse word length predictions. We take these results as evidence that Zipf's longstanding hypothesis holds.

Revisiting the Optimality of Word Lengths
Tiago Pimentel
Clara Meister
Ethan Gotlieb Wilcox
Kyle Mahowald Ryan Cotterell
'University of Cambridge "ETH Zürich
"University of Texas, Austin
(tiago. pimentel, clara. meister, ethan. wi wilcox, ryan. cotttteeellll)nnnf .ttzz.th mahowald@utexas. edu
Abstract
Zipf (1935) posited that wordforms are opti- mized to minimize utterances' communicative costs. Under the assumption that cost is given by an utterance's 's length, he supported this claim by showing that words' lengths are in- versely correlated with their frequencies. Com- municative cost, however, can be operational- ized in different ways. Piantadosi et al. (2011) I) claim that cost should be measured the dis- lance between an utterance's 's information rate channel capacity, which we dub the chan- nel capacity hypothesis (CCH) here. Following this logic, they then proposed that a word's length should be proportional to the expected value of its surprisal (negative log-probability context). In this work, we show that Pianta- dosi et al. derivation does minimize CCH's cost, but rather a lower bound, which we term CCH4. We propose novel derivation, suggest- ing improved way minimize CCH's cost. Under this method, we find that language's word lengths should instead be proportional to the surprisal's is expectation plus its variance-to- mean ratio. Experimentally, we compare these three communicative cost functions: Zipf's, CCH₁, and CCH. Across 13 languages and sev- eral experimental settings, we find that length is better predicted by frequency than either of the other hypotheses. In fact, when surprisal's 's ex- pectation, or expectation plus variance-to-mean ratio, is estimated using better language mod- els, it leads to worse word length predictions. We take these results as evidence that Zipf's longstanding hypothesis holds.
3.0
10
2,5
MSL
de
2.0
em
no
he
es
1.5
he
he
ko
1.0 ko
ko
Zipf
CCH
CCH
Figure 1: Mean squared error achieved by a linear model predicting real word lengths under the three hypotheses (lower is better).
In subsequent work, Piantadosi et al. (2011) offered a complementary account of commu- nicative cost. Starting from the hypothesis that information rate should be roughly constant during communication (UID; Fenk and Fenk, 1980; Levy and Jaeger, 2007), they argue that word lengths should make information rates as close as possible to a hypothetical channel capacity, where the channel refers to the means by which information is transferred from one person to another. We term this the channel capacity hypothesis (CCH).² They conclude that lengths should be proportional to word's expected surprisal instead.
/ttps://githubs com/ tpi mentelms/ aptimality-of-want -erd-eegthhs
As the communicative efficiency of language provides important insights into human cognition (Gibson et al., 2019), Piantadosi et al.'s 's finding that word lengths are better explained by average surprisal than frequency has been influential. How- ever, there are shortcomings: First, the manner in which Piantadosi et al. finds solution which mini- mizes the cost associated with CCH is not formally
1 Introduction
Zipf proposed the idea that languages are optimized to minimize their expected utterance length (Zipf, 1935).' Under this hypothesis, word's length should be inversely proportional to its frequency. Indeed, this relationship has been attested across wide variety of the world's languages (Grzybek, 2015; Bentz and Ferrer-i-Cancho 2016, inter alia).
2 Pcc is one of the many instantiations the uniform infor- mation density (UID) hypothesis. We introduce this new ter- minology here to make the hypothesis' name more descriptive. SSurprisal is defined as negative log-probabiliy in context.
We will refer to this hypothesis as ZIPF.

---

specified. And second, Piantadosi et al. 's empirical results have been shown to be sensitive a number of methodological decisions, such as the choice of text-encoding (e.g., ascii vs. unicode), the inclusion of non-conventional wordforms and other orthographic conventions of a language (Mcylan and Griffiths, 2021; Levshina, 2022). Thus, there remain fundamental open questions about the relationship between communicative efficiency and word length. Here, we aim clarify both the- oretical and empirical aspects of this relationship. Theoretically, we offer a novel, formal deriva- tion of Piantadosi et al.'s claim. We find that Piantadosi et al. (2011) optimize not for the objective under the CCH, but for a lower bound on it instead; we call this the CCH, objective. We then provide closed-form expression for the function that determines word lengths under CCH: Word lengths should be proportional to the expected surprisal plus its variance-to-ment ratio. Importantly, we derive the solution above by framing the problem of assigning wordforms as the optimization of cost function." 4 By instantiating this optimization problem with the objectives posited by each hypothesis (ZIPF, CCH, and CCH+), we can compute their word length predictions within a single, unified framework.
problem. First, we assume fixed (but potentially infinite) vocabulary W of words, each of which we denote as w E W, and a fixed alphabet Σ. Given a vocabulary and alphabet, we define a lexicon as a function that outputs a wordform for each word; we denote a lexicon as Φ W + Σ" and a wordform as dbw:) E E*, Note that we distinguish between a word, which is an abstract notion or concept, and its wordform, which is its ortho- phonological realization. Further, let p(w, c) be a language's ' joint probability distribution over these words and their prior linguistic context c E W Finally, let cost cost|o] (w, c) be a cost function that, given a lexicon, outputs the communicative cost of word in context. It is often suggested that the only attribute of a wordform <(w) that the function cost(Ø) is concerned with is its length (ó(w)), where 11: : E' + 2; We now define the optimization problem proposed by Zipf as follows.
Definition 1. The lexicalization problem the task of finding an optimal lexicon s', which minimizes cost This lexicon can be described formally as the solution to
argmin
E cost|ol(w, c) p(u:c)
(1)
subject to 6 E 42
Empirically, we offer a large-scale comparison of ZIPF's, CCH,'s 's, and CCH's word length predic- tions across 13 typologically diverse languages. Notably, we use neural language models to estimate words' surprisals, which provides more accurate estimates than the n-gram models relied on by prior work this topic (Piantadosi et al., 2011; Meylan and Griffiths, 2021; Levshina, 2022). We find strong evidence (see Fig. 1) that languages are optimized to minimize their utterance lengths: A word's frequency (ZIPF's prediction) offers stronger predictive power over word lengths than either the surprisal's expected value (CCHL's 's prediction) or expected surprisal plus variance-to- mean ratio (CCH's prediction). We conclude that Zipf's longstanding theory stands strong.
where De is the set of valid o for language f.
There are many assumptions that one could make about Dy's iss characteristics. We make a few explicit in the following remark.
Remark 1. We take the set oe to include all lexi- cons which: ① only produce phonotactically valid wordforms, 2 respect morphological composi-
tion, 7 and 3 are uniquely decodable."
Another implicit constraint 44 regarding valid d- which comes from our specification of the output
"W define this distribution formally in App. A.
'Phonotactics tells us how phones can be combined to create wordforms in language. If we denote the set of all possible phonotactically valid wordforms in language f as Lf C. E', this means that the image of o is contained in L.. 'Roughly, if the concepts represented by and w' overlap in dimension that captured by E's morphology (e.g.. plurality in English) then their wordforms o(w) and o(w") are likely also partially overlap.
2 The Lexicalization Problem
Zipf (1935, 1949) posited that the lexicon is optimized for communication, taking the needs of both speakers and listeners into account. In this section, formalize a slice of this optimization
"This condition is perhaps too strict. Homophony, for instance, is when o(w) = o(w")) for w A w' and will, in gen- eral, make natural languages not uniquely decodable. How- ever, if two words never appear in the same context, natural languages may still be uniquely decodable even in the presence of homophony. We note that whether or not natural languages are optimized for being unambiguous is contentions (Chom- sky, 2002; Piantadosi et al., 20 Pimentel et al., 2020, 202 bh; Trott and Bergen, 2020, 2022).
"As we will make explicit, we relax some optimization constraints to he able to derive closed-form form solutions. These solutions will thus lead to lower hounds on the total cost.

---

space of d-is that these mappings only produce integer-length wordforms.
which can be roughly approximated as ((sp)(w)] ~ - log p(w). Unfortunately, empiri- cal evidence suggests that this solution, which sug- gests the proportionality constant in eq. (2) equals 1, is not representative of how natural languages be- have (Pimentel et al., 2021c). It thus gives us little insight into how actual wordforms should behave. Fortunately, we can derive more interesting result where the proportionality in eq. (2) still holds by only partially relaxing ! from Remark We first assume a very simplistic model of phonotactics. Given an alphabet Σ of phones, let Le C Σ* be the set of phonotactically valid wordforms in language L. Note that this assumes deterministic phonotac- tics (Gorman, 2013; Dai and Futrell, 2021). 12 Fur- ther, define PREFIXES Lt) (ax a E Le, < | to be the set of all prefixes in this language. Assumption 1.
In the subsequent sections, we consider re- laxations of eq. (1) to arrive at simple solutions regarding the lengths provided by optimal lexica. Specifically, we partially relax constraint ! and fully relax constraint ② when deriving a lexicon with minimal utterance length. Further, when deriv- ing optimal results for both CCH and J we also fully relax constraints ① 3 and Note that, as in all optimization problems, removing constraints always yields a lower bound on the expected cost we obtain under an optimal lexicon." 10
3 Revisiting Zipf's Law of Abbreviation
Zipf (1935, 1949) posited a specific form that the cost function in eq. (1) should take. Concretely, he posited that lexica were optimized with the goal of minimizing speakers' utterance lengths, which can be written as cost[ [1) c) [o(w) in our notation. In an attempt to formalize his position, he proposed his eponymous law of abbreviation:
The constant phonotactic assump- tion assumes there exists a K E Zoy0 such that K I |_] and, for every string a E PREFIXES there exist exactly K symbols {ox] OOO;; E PREFIXES(L)).
k= for which
[etp x - logg p(w)
(2)
In words, Assumption 1 says that there are ex- actly K valid symbols with which every phono- tactically valid prefix can be extended. Given this assumption, we can now find a solution to eq. (3), which only partially relaxes the phonotactic con- straint in Remark 1.
Over the years, Zipf's law of abbreviation has been empirically investigated numerous times (Wimmer et al., 1994; Sigurd et al., 2004; Kanwal et al., 2017; Koplenig al., 2022; Levshina, 2022; Petrini al., 2022, 2023). We now present formal derivation of Zipf's law of abbreviation by viewing it as an instantiation of the lexicalization problem. Hypothesis 1. Zipf's hypothesis predicts that com- munication is made optimal by the mapping @xipf that satisfies:
Theorem 1. The minimization problem given in Hypothesis I with constraint ? relaxed can be solved by Huffman coding 13 with K symbols. The optimal solution is given by
[e)p = 11 x (w))
(5a)
@zipt argmin E
lo(w)] p(w.c)
S
log p(w) +1 (5b)
(3)
logjs
K
subject
4bg
Proof. The proof is available in App. C.
If we relax constraints ① and ② in Remark 1. then the optimal solution eq. (3) can be achieved by Huffman coding (Huffman, 1952)." We know that this optimal solution's 's word lengths respect:
Theorem makes precise the sense in which we
claim to have derived Zipf's law of abbreviation.
Under the rough approximation (Paip(()))
lognn p(w)
logny K the proportionality in eq. (2) is
&eip (w)) < log p(W)+
+1
(4)
realized through the unknown constant 1/108/s) K.
"Explicitly, by relaxing 4 we allow [o(-)] to take on continuous values. Such relaxation destroys 0's interpre- tation assigning wordforms that live in M", However, it turns combinatorial optimization problem into continuous one and allows apply tools from calculus.
NNonddcterministic models of phonotactics also popu- lar; see (Hayes and Wilson, 2008) for classic study.
Huffman coding is an efficient O (IW) [W])-- algorithm that returns the exact solution to this problem. Huff- man coding, however, requires a finite W., which may not always be the case in theory. Linder et al. (1997) proved the existence of an optimal code which respects eq. (4) for distri- butions with infinite support hut finite entropy. See Pimentel et al. (2021c) for more discussion.
"Pimentel al. (2021c) estimate eq. (1) while respecting four constraints, but restricted to cost[e](u,, c) = lo(w). "I "B Kraft-McMillan's 's inequality, constraining our solu- tion to not only he uniquely decodable, but to he prefix firee adds nothing in terms of length (Cover and Thomas, 2005).

---

4 Revisiting Piantadosi et al. (2011)
this perspective, and assume that its operationaliza- tion given by eq. (6).
What's wrong with Zipf's law of abbreviation? The solution in eq. (5) is only optimal if one believes that cost [o]] e) = [<(w) is the true objective underlying the lexicalization problem. However, more recent work on communicative efficiency (c.g., Piantadosi et al., 2009, 2011) has proposed that speakers may intend to optimize another objec- tive instead. Specifically, one can take the perspec- tive that language an exchange of information via a noisy communication channel, where infor- mation is operationalized as a word's surprisal H(w c) = - log p( w e). This channel has an inherent capacity E E IR at which information can be transmitted while the receiver is still able to effectively decode the underlying message. Under this perspective, optimal communication happens when a word's information rate H(w in bits per character) is kept as close as possible to C. A word's 's channel deviation is then the difference between its information rate and channel capac- ity. This hypothesis can thus be stated within the framework of the lexicalization problem by defin- ing the cost (ost(()(( [o] (ur, c) of a lexicon as a function of the channel deviation.
4.1 Optimal Word Lengths
The exact solution to eq. (6) depends on the choice of dist. this section, we assume a quadratic distance function, i.e., dist((r, C) = (x e)² Efficient lexica should thus minimize the expected value of the square of the channel deviation under p(w, e) (i.e., its mean squared error). We now derive closed-form expression for CCH-optimal word lengths under this cost function. As in The- orem 1, we relax the morphological 2 constraint. Beyond this, we also relax the phonotactic !. ciiuue-dccodability 3 and the integer-length constraints. Note that, unlike in Theorem 1, we need to relax + here because we have no efficient combinatorial algorithm to solve eq. (6).
T
Theorem 2. Under Hypothesis 2, if we relax 1 2
and ④ the optimal word lengths are given by
E p(c/w)
[H2
c)]
1
[ecll(w)] =
(7)
C
E
H 10 c)]
p(clu))
Proof. The proof is available in App. D.
Hypothesis 2. The channel capacity hypothesis predicts that communication is made optimal by the mapping occh that satisfies:
We note that Eq. (7) equivalent to the expected surprisal plus a variance-to-mest ratio."
4.2 Choices of Distance
H(w (
Φoch argmin E dist p(w,e)
(o(w))
In the above section, we assumed a quadratic penalty for a word's channel deviation. There is, however, no inherent reason why dist should be quadratic. We thus examine alternative ways to quantify the deviation between a word's infor- mation rate and the channel capacity. Different choices of dist will then each define cost function (H(
(6)
subject to & E &
where dist(x; 8) is a function that quantifies how far x is from y. 14
Intuitively, eq. (6) penalizes lexica where the length of a word causes its information rate to de- viate from the channel capacity. Thus, &ech will generate word lengths which produce information rates that as uniform possible. It follows that it can be categorized under the larger umbrella of the uniform information density hypothesis (UID; Fenk and Fenk, 1980; Levy and Jaeger, 2007). As discussed by Meister et al. (2021), however, UID has several potential interpretations, only one of which involves maximizing the use of a communi- cation channel. Here, we will only discuss it under
through cost cost[o](u) (w, <) dist
C
To(w)]
Any specific utterance should fall in one of three
cases: First, word's information rate may be at H(wle)
capacity, i.e., when [o(w] = C. In this case, there are no baH-based costs. As the capacity is a real number, however, this is virtually impossible in practice. Second, information rate may be below capacity. This will imply an opportunity cost on communication: speakers will need more time to produce their utterances than desired, which is not ideal from the perspective of communicative efficiency (Levy and Jaeger, 2007; Kanwal, 2018).
14 "We consider dist(-) functions which satisfy the first tWo axioms required by true distance metrics: dist(a; y) 0 + = y (identity of indiscemibles) and dist(z, v) 2. 0 (non-negativity). but which are not necessarily symmetric and do not necessarily satisfy the triangle incquality.
19This can be seen via the following manipulations: E|a2] E
= E [x]] -[= B + E

---

Third, information rate may be above capacity. This again implies a cost on communication; since communicating above a channel's capacity is provably noisy (Shannon, 1948), there may be communication faults which will either lead to the wrong meaning being conveyed, or will require potential retransmission of the message.
Hypothesis 3. Piantadosi et al. predict that com- munication made optimal by the mapping Pachy that satisfies:
H(w 10 :e)
Occh, = argmin E dist
[
p(u,e)
(10)
subject
E te
The quadratic distance function that we have proposed above assumes symmetric cost, where communication above or below capacity are equally harmful. It is, however, reasonable to as- sume that the cost associated with communicating above capacity may be higher than the opportunity cost of communicating below it. This leads us to propose costs based on the following generalized distance function:
We now give the connection between Hypothe- sis and eq. (9) in the following theorem.
Theorem 3. Under Hypothesis 3, if we further
relax ! @ 3 and ④ the optimal word lengths
are given by
@eechy (w)) H(w IC)
(11)
dist(z: (2, C) - { - E) (r C)²
No>e
Proof. Using o = occyy as given by eq. (11), we get dist(-, e) = 0 for all words when evaluating the objective in eq. (10). By definition, this is the minimum for any dist.
(8)
else
where À R50. Under this generalized distance function, any value > will imply larger penalty for communicating above than below capacity. Further, when À == we recover the sym- metric quadratic distance function proposed earlier. Notably, when assuming this generalized dis- tance function, there is no capacity-agnostic closed- form value to which word lengths should be propor- tional. Here, find CCHx -Cttimpt lengths with two step process: (i) given large set of surprisal values paired with their word lengths, we find what the optimal capacity is for a language; (ii) we then use a gradient descent-ben based optimizer to find the optimal lengths under that capacity.
Note that dist H(w[C) To(w) e is constant with
spect to individual contexts c. Thus, the expec- tation in eq. (10) can simply be taken over the unigram distribution, p(w).. Moreover, if dist is a convex function, then, we can use Jensen's 's incqual- ity show that eq. (10) lower-bounds eq. (6). 17 We therefore denote Piantadosi et al.'s hypothesis and solution CCH₁-
Proposition 1. Given a convex dist function and any Φ E 4g the cost optimized by CCH₁ in Hypoth- esis 3 lower-bounds CCH's cost in Hypothesis 2
4.3 Piantadosi al.'s (2011) Lower Bound
H(w c) [o(w]]
E dist p(mcc)
e
In their paper, Piantadosi et al. offer a similar argu- ment to the one proposed in this section. They state, however, that the optimal word lengths follow:
(12)
H(W C) [o(w)]
z E dist p(wsc)
@ (w, c)l x H(w ())
(9)
Proof. The proof available in App. E.
where H(w C) is the surprisal of word w.
We now provide an example to show how CCH 's solution does not minimize
marginalized over all contexts. While Piantadosi et al. intended to find a solution which minimizes the cost associated with CCH, they actually do something else. We find that Piantadosi et al. 's pro- posal optimizes a different instantiation of the lexi- calization problem, one that does not use the objec- tive that formally corresponds the CCH hypothe- sis. 16 We give the objective Piantadosi et al. 's pro- posal is the solution to below as its own hypothesis.
dist
H(wle) To(u)]
under the distribution p(w, c).
Example 1. Let there be word with surprisal of
2 bits in ten distinct contexis, and a surprisal of 24 bits in single context; assume all eleven contexts are equiprobable. The word's 's average surprisal is thus bits (i.e., 10:2424) ). Further, assume we
Note that this lower bound is with respect to the function heing minimized in our optimization problem. It is therefore in addition to the lower bound that comes from relaxing this optimization problem's 's constraints.
16 "See Cohen Priva (2015), however, for a discussion on how average surprisal could still predict word's duration beyond individual surprisal effects.

---

have a channel with capacity C According
5.2 Evaluation Metries
to Theorem 3, we have cochy (w)) = H(w 2,
e
Even with access to the true p, comparing the word length predictions of the different theories above would be non-trivial. Language evolution is dynamic and noisy process: Even if one of the above optimization pressures has acted during the creation of languages' lexica, it is unlikely that they are perfectly optimal with respect that pressure. We thus cannot simply evaluate whether languages match our predictions exactly, Rather, we can instead measure if the general trends predicted by the different hypotheses match the trends observed in natural language. We will rely on number of metrics to evaluate our results. Taken together these metrics should allow to draw conclusions which theory (if any) best correlates with observed word lengths.
which under the CCH objective (eq. (6)) gives us an 10
expected cost of 10 (i.e., 11 (3 2) ( 2) If we choose word lengths according to Theo- rem 2 instead, we get that the length should be (eech (w))
This results in cost under the CCH objective of roughty? 2.86.
5 Experimental Setup
5.1 Estimating Word Length Predictions
To evaluate the different hypotheses, we test how well their predictions about word lengths align with the lengths of real languages' wordforms. These predictions require computing surprisals (either unigram contextual), which are defined according to the true probability distribution p (either as a function of p(t), or p(w o); the distribution p is defined more precisely in App. A). While we do not have access to the true probability distribution p, we do have samples from it. We use the following estimators of eqs. (5), (7) and (11):
Spearman Correlation. First, we follow prior work (Piantadosi et al., 2011; Meylan and Griffiths, 2021; Levshina, 2022) and use the Spearman cor- relation to assess the quality of each word-length hypothesis. A positive attribute of this correlation is that it can account for nonlinear relationships, potentially accounting for non-linear optimization obstacles. This metric, however, has a significant drawback: Namely, all wordforms contribute equally to its computation. If we evaluate large enough corpora using Spearman correlations, we will therefore consider vocabularies W mostly dominated by low-frequency and uncommon wordforms, such as typos, specialized terms, and names. Yet arguably, when evaluating the different hypotheses, given word should be weighted according to its usage (i.e, frequency) in a given language, as this is the case in our various optimization problems; a word's impact on the lexicalization problem's is objective is function of its frequency. This is perhaps one of the reasons why prior work has limited their analyses to only consider subset of the most common words per language (Piantadosi et al., 2011), design choice that we likewise employ in our main experiments.
Oeipt (w)l = log g(w)
(13a)
@echy.(1) (w)) = DDail JED
(w |c)
(13b)
2
log q( cc)
1och (1)
LED
(13c)
Σ
log g(w)
c¹)
d'D
E D, w). and D is our corpus, which we assume to be sampled from the distribution In practice, our corpus D is com- posed of data from one out of 13 languages from 5 language families in Wiki40B (Guo et al., 2020). Distribution q is our estimate of p. which we implement using language models. We use: normalized count statistics to estimate the unigram distribution p(w), and transformer models for p(w c). Our data and models are described in detail in App. B. 18 Note that we omit unknown constants from eqs. (13a) to (13c) because we only consider scale-invariant evaluation metries.
where D',, fc
Pearson Correlation. As a second metric, we evaluate the Pearson correlation between our predictions and actual word lengths. Pearson's 's correlation has similar drawbacks to Spearman's ss differing from it only in that its value reflects the strength of linear relationships.
II our main set of experiments. we filter the set of words we analyze to only include the top 25k most frequent words in a language which have wordforms composed of charac- ters in the language's 's alphabet; we use alphabet's 's as defined in homoglyph: https:/ //pppi. wwpppp.org/ppp.....p..pp.pp.rr.p wtg/prooeccttttc sppooeet/homeee htohhlyposs///////// We also pre-tokenize data with language-specifie UnigramI. tokenizers, and sum subword surprisals when necessary to get per-word values.
Weighted Mean Squared Error (MSE). As a third metric, we use weighted MSE, which

---

avoids the main drawbacks of the previous metrics. We fit linear regression models (without a bias term) to predict a language's s word lengths using our ZIPF, CCH, or + estimators as the sole predictor. Importantly, we weight each squared error term by that words' frequeney (both during this model's training and evaluation). This design choice makes our method more robust the set of words being evaluated, since the inclusion of exponentially many low-frequency words should not substantially affect weighted MSE. Note that this procedure is equivalent to measuring the predictive power of each hypothesis, while assuming eqs. (5), (7) and (11) predict an expected length, and that word lengths are normally distributed around these expected values.
Pearson cott.
Spearman cott.
41 ko
0.2-
of
0.1-
0.0-
Zipf
CCHL
OCH
ZipÊ
CCHL
CCH
Figure 2: Pearson and Spearman correlation of the three hypotheses in the analyzed languages (higher is better).
longer words will naturally decompose into more subword units, and so when adding subword sur- prisals, the total error of a word's surprisal estimate may correlate with its number of subword units.
To assess the impact of this potential systematic error in our estimates, we thus re-train our models using a vocabulary of 32k full words, replacing any word not in this set with an unk symbol, which is necessary when working with finite vocabularies. Under this model, all analyzed words are encoded using single "subword" unit. We then re-analyze the three hypotheses as before. In Fig. 3 (top), we see that a word's frequency still a better predictor of its length than the quantities put forth by other hypotheses. Further, in the only case in which CCH, offers better predictions than ZIPF (English, as evinced by higher Spearman correlations), their performance difference is now lower than before." 21 We also estimate ZIPF's unigram surprisals using tokenized counts, i.e., where we count subword tokens to estimate frequencies instead of directly counting the full words our training set. We then estimate the suprisal of a word as the sum of the surprisals of its subwords, thus assuming independence between them. We display these new results in Fig. 3 (bottom) under the name Zipf (subwords). We see here that this tokenization scheme increases our measured correlations, and Zipf (subwords) presents the strongest correlations in languages. Perhaps surprisingly, tokenization seems to not influence MSE as much.
6 Results
Our main results are presented in Fig. I and 2. In short, Fig. 1 shows that words' frequencies offer stronger predictive power of word lengths (as evinced by smaller MSE) than either of the metries. This result provides evidence for ZIPF's hypothesis over either CCH or CCH4- This result is particularly surprising since we improve on CCH's optimal word length predictions, but ZIPF's hypothesis still provides the best predictions." 19 A similar result can be seen in Fig. 2, where frequency offers the strongest correlation with lengths (in terms of both Pearson and Spearman), in all languages but English. Notably, in our results, some languages even have a negative correlation between the two surprisal- measures and actual word lengths. We now turn to amalyzing different methodological choices that could impact our results.
6.1 Sensitivity to Tokenization
The first design choice that we analyze here is the choice of tokenizer that we use to preprocess our data. As cross-entropies are necessarily larger or equal to entropies," 20 it is reasonable to expect that our language model surprisal estimates may be, on average, larger than true surprisals. While we do not know the exact per-token nature of this difference, it is conceivable that using UnigramLM tokenization could compound it: On average.
6.2 Sensitivity to Word Filtering Protocol
Next, we analyze our results' sensitivity with re- spect to how we select the set of words we analyze. Specifically, for our analyses so far we have only considered words whose wordform is composed
We improve CCH's 's optimal word length predictions over prior work both theoretically, by optimizing CCH as opposed to CCH,, and empirically, by using stronger language models. 20 "Cross-entroppy is the setabbllityweightedd average of the surprisal estimates from our language model.
21 Thesee correlations are, respectively, 0.09 vs. 0.21 with ZIPF and CCH₂ when using UnigramI M. After switching to full words they are 0.09 vs. 0.14.

---

4
Spearman corr.
0.4
3-
MSE
2
0.2
en
he
ne
ko
Ko
Ko
0.0
0-
Zipt
CCHL
CCH Zipf (subword)
Zipf
CCHL
CCH Zipf (subword)
4
Spearman corr.
ko
0.4
3
MSE
en
0.2
es
he
hd
he
ko
KO
0.0
he
Zipf
CCH
CCH Zipf (subword)
Zipf
CCHL
CCH Zipf (subword)
Figure 3: MSE and Spearman correlation when surprisals are estimated using either: full words directly (top), or adding subword surprisals (bottom). Note that when using full words, 7ipf and Zipf (subword) are the same.
ZIPP
CCH
ZIPF
OC1l
CCH
3-
MSE
MSE
((w) e E>'
(w) 5%
50000
100000
150000
200000
Word Types
Figure 4: Average MSE across languages when hypothe- ses are evaluated using different word filtering protocols.
Figure 5; Average MSE across languages when hypothe- ses are evaluated on different number of word types.
exclusively of characters in its language's alphabet. We now run similar analyses, but including either: All chite-space-ssparree test set, or all white-space-eeparaaee words with punctuation symbols.22 22 We denote these conditions as: Σ; when selecting alphabet-only words, Σg when selecting no no-punctuation words, and Σ when selecting all words. We display results under each condition in Fig. 4. We see that across these various protocols, ZIPF's hypothesis remains the most predictive." 23
dictor of word length. In App. G's Fig. 10, display results per language for MSE and Spear- man correlation. There, we see that MSE rates frequency best on all languages and across all eval- uated setups. Spearman correlation evaluated on few word types similarly rates frequency over CCH or CCH, predictions (again, except on English). When evaluated full test-sets, Spearman relation shows less straightforward conclusion: While ZIPF still achieves the highest correlation in most languages, CCH² achieves stronger correla- tions in Italian. Spanish and Russian. At this stage, however, the evaluated sets are dominated by low- frequency words, which may not be representative of the evaluated languages.
words in a language's s
Additionally, we consider the impact of includ- ing only the top 25k most frequent words in our analysis. In Fig. 5, we present MSE values com- puted when using sets composed from the top 10k most frequent words, to entire test sets. Notably, we again see that frequency remains the best pre-
6.3 Sensitivity to Model Quality
We consider punctuation to any of: & +0* >=<=>?([]]" "')' 2' App. G's Fig. 9, we also see that Spearman correlation is considcrably more sensitive to filtering protocols than MSE,
Finally, we investigate how our model quality influences our results. We train new models on subsets of our training sets to get language models

---

7 Discussion
P OCH
CC11
The answer to what drives the distribution of word lengths in lexica has long been considered impor- tant for understanding the evolution and function of language (see Gibson et al., 2019 for a review). Across multiple languages and various methodolog- ical choices, results support Zipf's law of ab- breviation over other potential explanations as a driving factor in the development of lexica.
MSE
6
20
12
14
18
23
Lungunge Modell' Cross-uurropy
Figure 6: MSE correlation function of the cross- entropy of models used to get surprisal estimates.
These findings deviate from Piantadosi et al.. who found average surprisal to be a stronger pre- dictor of word lengths. We hypothesize that this is because of methodological choices. Specifically, Piantadosi et al. derive surprisal estimates from language models that are now outdated (in terms of their quality), and we found that, when CCH's pre- dictions were computed using worse surprisal esti- mates, they had stronger correlations with length than when using better estimates. Like prior work on this topic (Meylan and Griffiths, 2021; Levshina, 2022), our analyses suggest the sensitivity of Pi- antadosi et al. 's results to methodological choices.
of different qualities. We then use these models to assess whether there is a relationship between model quality and a hypothesis' predictive power. In addition to the models estimated using the full training sets, we thus train 7 new transformer and unigram models per language, cach using from 1 million 1 billion training tokens in log-uniform intervals. We plot the predictive power of each hypothesis (ZIPF's, CCH,'s 's and CCH) vs. the language model's cross-entropy in Fig. 6. 24 Unintuitively, surprisal estimates of better models (i.e., with lower cross-cntropies) provide worse predictors of word length. An additional analysis suggests that the surprisal estimates of worse more strongly correlated
What do these results tell about the commu- nicative optimization of natural language lexica? In short, our results suggest lexica are optimized to minimize expected utterance lengths. Notably, other linguistic properties may be optimized to- wards other notions of communicative efficiency. While a word's duration is mainly determined by its wordform, speakers can still modulate this duration to certain extent; such modulation could target CCH. In fact, prior work has shown correlation between surprisal and duration (Bell et al., 2003; Aylett and Turk, 2004; Pimentel et al., 2021a).
language models with frequency (see Fig. 7 in App. G), which may justify this unituitive result since frequencies are most predictive of word lengths in our experiments. ZIPF's hypothesis, on the other hand, is robust to the quality of the used unigram model.
6.4 Sensitivity to Cost Function
Conclusion
In our last set of experiments, we analyze the im- pact of our choice of quadratic cost function in our results. Using the generalized cost function in eq. (8), we derive optimal word length predictions using values of X from to 5 in 0.25 intervals. We present their MSE and Spearman correlations in App. G's Fig. 13. While there seems to be slight tendency for CCH to be more predictive for larger values of X, ZIPF still has the most predictive power of the different hypotheses.
In this paper, we formalize the problem of assigning wordforms based on different notions of communicative efficiency, which we term the lexi- calization problem. Under this framework, we de- scribe the optimization problem related to the chan- nel capacity hypothesis, and, in doing so, we show that Piantadosi et al.'s predictions optimized for only a lower bound on CCH, rather than on the true objective. Further, while considering relaxed ver- sions of the lexicalization problem, we derive op- timal word length values for Zipf's hypothesis and CCH. We then empirically evaluate CCH's, CCHy's 's and ZIPF's predictions 13 languages. Our results strongly support ZIPF's hypothesis: Word lengths are optimized to minimize utterance lengths.
"We show per-language plots evaluated with both MSE and Spearman correlation in App. G's Fig. 11. We do not quantily our unigram models' quality, but assume that they increase monotonically with the size of the corpus on which they were estimated. We show a similar plot, hut with the number of training tokens on the a-axis, in App. G's Fig. 12.

---

Limitations
tion for relationships between redundaney, prosodic prominence, and duration in spontaneous speech. Langugg and Speech, 47(1):31-56,
A limitation of our work is that, when deriving op- timal word lengths under CCH and J we relax:
Alan Bell, Daniel Jurafsky, Eric Fosler-Lussier, Cyn- thia Girand, Michelle Gregory, and Daniel Gildea. 2003. Effects of disfluencies, predictability, and ut- terance position on word form variation in English conversation. The Journal of the Acoustical Society of America, 113(2)1001-1024
the phonotactic !, morphological composition (2) unique decodability 2 and the integer-length requirements. In the case of @ if a language's channel capacity is large, this might lead to poorer predictions under both these theories. Deriving optimal word lengths while considering this con- straint is left as an open problem for future work. In the case of ④ it is arguably unrealistic to consider continuous-length wordforms. This issue could be addressed by using linear program to solve prob- lems of the form eq. (I). This, well consider- ing the role of phonotacties ! and morphological composition 2 in CCH, is likewise left for future work. Further, we note that while we relax all four constraints to derive CCH- and CCH4 -optimal word lengths, we only relax ② (and partially !) to de- rive zipi-optimal lengths. This could realistically impact the fact that Zipf's hypothesis seems to have more predictive power over word lengths.
Christian Bentz and Ramon Ferrer-i-Cancho. 2016. Zipf's law of abbreviation as a language universal. In Proceedings of the Leiden Workshop Capturing Phylogenetic Algorithms for Linguistics. Universität Tübingen.
Noam Chomsky. 2002. An interview on minimalism. Cambridge University Press.
Uriel Cohen Priva. 2015. Informativity affects conso- nant duration and deletion rates. Laboratory Phonol- ogy, 6(2):433-27.
Thomas M. Cover and Joy A. Thomas. 2005. Flements of Information Theory. John Wiley & Sons, Ltd.
Huteng Dai and Richard Futrell. 2021. Simple induction of (deterministic) probabilistic finite-state automata for phonotactics by stochastic gradient descent. In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 167-176, Online. Associa- tion for Computational Linguistics.
Another limitation that our analyses focus solely on written data from Wikipedia. We recommend future work investigates how these findings generalize to spoken or signed languages, and to other text genres. Finally, while we use a typologically diverse sample of languages, it is still skewed towards Eurasian languages. This is because the large amount of text needed for training state-of-the-art language models- necessary to estimate entropy- are not available in many languages. Expanding the set of languages analyzed here would be necessary to confirm the generality of our results.
August Fenk and Gertraud Fenk. 1980. Konstanz im Kurzzeitgedächhnis Konstanz im sprachlichen In- formationsfhuß? Zeitschrift für Experimentelle und Angewandte Psychologie, 2703):400-144,
Edward Gibson, Richard Futrell, Steven T. Piantadosi. Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):389- 407.
Kyle Gorman. 2013. Generative Phonotactics. Ph.D. thesis, University of Pennsylvania.
Acknowledgements
We thank the anonymous reviewers and meta- reviewer for their feedback on this paper. Tiago Pimentel also thanks Hope McGovern and Simone Teufel for helpful comments on different stages of writing this manuscript. Tiago Pimentel is funded by Facebook PhD Fellowship. Clara Meister is funded by Google PhD Fellowship. Ethan Gotlieb Wilcox would like to acknowledge support from an ETH Postdoctoral Fellowship.
Peter Grzybek. 2015. Word length. In The Oxford Handbook of the Word, pages 89-119. Oxford Uni- versity Press.
Mandy Guo, Zihang Dai, Denny Vrandečié, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2440- 2452, Marseille, France. European Language Re- sources Association.
Bruce Hayes and Colin Wilson. 2008. A maximum en- tropy model of phonotactics and phonotactic learning. Linguistic Inquiry, 39(3):379-440.
References
David A. Huffman. 1952. A method for the construction of mainimum-reduuddan codes. Proceedings of the IRE, 1098))
Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explana-

---

Jasmeen Kanwal. 2018. Word length and the principle of least effort: language as an evolving, efficient code for information transfer. Ph.D. thesis, The University of Edinburgh, Edinburgh, UK.
Sonia Petrini, Antoni Casas-i-Muñozz Jordi Cluet-i- Martinell, Mengxue Wang. Christian Bentz, and Ra- mon Ferrer-i- 2022. The optimality of word lengths. Theoretical foundations and an empirical study. arXiv preprint arXiv:2208, 10384.
Jasmeen Kanwal, Kenny Smith, Jennifer Culbertson, and Simon Kirby, 2017. Zipf"s law of abbreviation and the principle of least effort: Language users opti- mise miniature lexicon for efficient communication. Cognition, 165:45-52.
Sonia Petrini, Antoni Casas-i-Muñooz, Jordi Cluet-i- Martinell, Mengxue Wang, Christian Bentz, and Ramon Ferrer-i-Cancho. 2023. Direct and in- direct evidence of compression of word lengths. Zipf's law of abbreviation revisited. arXiv preprint arXiv:2303.1012
Diederik P. Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In International Conference on Learning Representations.
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2011. Word lengths are optimized for efficient com- munication. Proceedings of the National Academy of Sciences, 108(9):3526-355) 3529.
Alexander Koplenig, Mare Kupietz, and Sascha Wolfer. 2022. Testing the relationship between word length, frequency, and predictability based on the German reference corpus. Cognitive Science, 46(6):e13090
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 12203)920.-
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics VVolue 1: Long Papers), pages 66 Melbourne, Australia. Association for Computational Linguistics.
Steven T. Piantadosi, Harry J. Tily, and Edward Gibson. 2009. The communicative lexicon hypothesis. Proceedings of the Annual Meeting of the Cognitive Science Sociery, volume 31, pages 2582-2587
Tiago Pimentel, Rowan Hall Maudslay, Damian Blasi, and Ryan Cotterell. 2020. Speakers fill lexical se- mantic gaps with context. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 4004-4015, Online. Association for Computational Linguistics.
Natalia Levshina. 2022, Frequency, informativity and word length: Insights from typologically diverse cor- pora. Entropy, 24(2).
Roger P. Levy and Tim Florian Jaeger. 2007. Speakers optimize information density through syntactic reduc- tion. In Advances in Neural Information Processing Systems, pages 849-856.
Tiago Pimentel, Clara Meister, Elizabeth Salesky. Si- mone Teufel, Damián Blasi, and Ryan Cotterell. 2021a. A surprisal- suurrisalddurationn trade-off across and within the world's languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 949-962, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
T. Linder, V. Tarokh, and K. Zeger. 1997. Existence of optimal prefix codes for infinite source alpha- bets. IEFF Transactions on Information Theory, 2226)2202-
Tiago Pimentel, Clara Meister, Simone Teufel, and Ryan Cotterell. bb. On homophony and rényi entropy. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 8284-8293, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revis- iting the uniform information density hypothesis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 963- 980, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tiago Pimentel, Irene Nikkarinen, Kyle Mahowald. Ryan Cotterell, and Damián Blasi. 2021c. How (non- joptimal is the lexicon? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4426-4438, Online. Association for Computational Linguistics.
Stephan C. Meylan and Thomas L. Griffiths. 2021. The challenges of large-scale, web-based language datasets: Word length and predictability revisited. Cognitive Science, :5(6);e ((66))11993. 12833.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling, Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (D (Demonstrations) pages 33,53, Minneapolis, Minnesota. Association for Computational Linguistics.
Claude E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal. 27(3):379-423.
Bengt Sigurd, Mats Feg-Olofsson, and Joost Van Weijer. 2004. Word length, sentence length and frequency Zipf revisited. Studia Linguistica, 58(1):37-5..

---

Sean Trott and Benjamin Bergen. 2020. Why do human languages have homophones? Cognition, 104499
Norwegian, Russian, and Turkish. These span five language families: Afro-Asiatic, muro-Eur,pean, Koreanic, Turkic, and Uralic. The data for each language comes pre-split into a training, validation and test set. We our models using the first two sets, while performing our analyses exclusively on the test-sets. As discussed above, the set of analyzed words may make large difference in the measured correlations. In our main set of experiments, we filter the set of words we analyze to only include wordforms composed of characters in the language's alphabet. 25 Table 1 (in App. F) includes the number of word types and tokens used per language in our analyses.
Sean Trott and Benjamin Bergen. 2022. Languages efficient, but for whom? Cognition, 225:105094.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Gejza Wimmer, Reinhard Köhler, Rüdiger Grotjahn, and Gabriel Altmann. 1994. Towards theory of word length distribution. Journal of Quamtitative Linguistics, 1(1):98-106.
George K. Zipf. 1935. The Psychobiology of Language. London: Routledge.
Models. To estimate the unigram distribution p(w), we use a simple MLE estimator: the (normalized) count statistics from our training set. To estimate contextual probabilities p(w c). we use an autoregressive language model pa- Specifically, train monolingual transformers in each language using fairseq (Ott al., 2019) with its default language modeling hyper-parameters. Our transformers (Vaswani et al., 2017) have 6 layers, a hidden size of 512, and 8 attention heads per layer. Further, they can attend to a context size of at most 512 tokens, and we train them with a dropout of 0.1, and a batch size of 64. We optimize our models using Adam (Kingma and Ba, 2015) with a learning rate of 5 x 10-4, weight decay of 0.01, and 1k warmup steps. In our main set of ex- periments, we further pre-tokenize each language's text using language-specifer tokenizers fit (using the UnigramL. algorithm; Kudo, 2018) on their respective training sets, with a vocabulary of 32k subword units. We then compute per-word sur- prisals by adding the surprisals of all the subwords that the word is composed of. (We also consider other tokenization schemes, described in $6.1.)
George K. Zipf. 1949. Human Behavior and the Princi- ple of Least Effort. Addison- Weslly Press.
A Defining p(w,o
In this section, explicitly define p(w, c). We do this in terms of more standard notation in language modeling. We define a sequence words as E S where S W {eas}. We then assume a distribution over such sequences p(s). We can now define p(w.e as:
T {w== 8t, = sc) (14) 1=1
p(w,)) c) & Σ p(s)
SCS
In words, word-context pair is frequent as it would be in natural language, where once se- quence is uttered, all its word-context pairs are observed jointly. This is not the only possible defi- nition of p(w,c) c), but is the one we opt for here.
Note that the distribution p(w, c) might thus not be well defined for all distributions p(s), as the normalizing constant in this definition might diverge. For instance, this distribution will not be well-defined for language model over alphabet W {a}, where p(a") गे for > and p(s) as its sequences' average length diverges.
C Proof of Theorem 1
Before proving Theorem 1, we provide a lemma which will useful for it. words, we prove a prestripres bijection between Le and A for an alphabet K such that | Al=K. K.
B Data and Models
The corpora used throughout our analyses come from Wiki40B (Guo et al., 2020). This dataset is composed of cleaned text from Wikipedia articles in more than 40 languages, out of which we select a subset of 13 for our analysis. Our selection includes: German, Greek, English, Spanish, Estonian, Finnish, Hebrew, Italian, Korean, Dutch,
Data.
Lemma 1. Under the constant phonotactic assumption, there exists an alphabet A with cardinality K such that, for every 2 0, AN is isomorphic to [NN, where L is the set of
We use alphabet's as defined in homoglyph: https:// pypi. www.orgg//ooo htt...rr//oooee//ooggyyy.s

---

phonolactically valid wordforms with length N.
Proof. We can easily derive these optimal word lengths from eq. (6) by taking its derivative with respect to a specific word's length, and setting to zero. First, we rewrite for mathematical convenience as:
Proof. First, is clear that A I We now prove the same for 1 by induction.
| N KWW
Base case (NN 0). The set O-length phonotac- tically valid strings includes only the empty string {e}. It follows that: | = = k
(0)
(Hore)) [o(w)]
(H(w)) ( e
E E p(w) p(cla)
(17)
Inductive step (N 0). By the inductive hy- pothesis, we have that |
KN-1. By
where we make the quadratic cost function explicit. We note this function is convex, and if we find point where its derivative is zero, we also find its global minimum. We now take its derivative with respect to a specific word's length [o(w)] and set this derivative to zero:
Assumption 1, each element in _ has K possible continuations in T ( follows that L ( L ( K= K
Since AN and L have the same number of el- ements for every N 0, there exists an isomor- phism between them.
H c) [<(w)]
H(w c)
((w)
E p(clan)
e'
Given the lemma above, we are now in position to prove Theorem I.
[x)w)
(18)
where we note that all terms involving other words will have derivative zero (with respect to this specific word w's length). As the expectation a linear operation, we can rewrite this equation as:
Theorem 1. The minimization problem given in Hypothesis with constraint ② relaxed can be solved by Huffiman coding with K symbols. The optimal solution is given by
[0eip((w)] n (w)
(5a)
H2 H2(w (w (c) [
H(w c) e
E p(clw)
E p(clw)
(o(w) (19)
log p(w) (5b)
logis K
Proof. Since ! (N) is isomorphic to N for every N Z 0, there exists a length-preserving bijection e between Le and (by Lemma 1). By Huffman's 's
Note that both the length and capacity are constant with respect to the expectation over contexts. Isolating the length term, thus, we get:
(1952) algorithm, we can construct an encoding that satisfies
E
H2 w
[c)]
1
p(c/w)
[d(w)
(20)
(n(ipr (w))) < logja p(w) +1
(15)
C
E
H(w
c)]
p( p((
However, because &b is length-preserving. ((eeep((())) = (w)) As an upper bound,
This completes the proof.
we thus have
E Proof of Proposition 1
10zipt(1)) < - -log A p(w) +
(16a)
log p( (w) +1 (16b)
Proposition 1. Given a convex dist function and any & E & the cost optimized by CCH, in Hypoth- esis 3 lower-bounds CCH's cost in Hypothesis 2
log K
D Proof of Theorem?
H(w (c) [d(w)] .e)
E dist p(m,0)
Theorem 2. Under Hypothesis 2, if we relax 1 2 3 and ④ the optimal word lengths are given by
(12)
H(w C)
> E dist p(w.o)
[o(w)] e)
E
H2
w c)]
1 (w)l
plc
(7)
E
H
c)]
Proof. can be easily shown by Jensen's inequal-
p

---

ity that for any choice of do
H(ww) c))
E dist p(wne)
[o(w)] .e)
(21a)
H(w
- E
dist
e
(21b)
p(w) p(ole))
[o(w)
E
H( w [0
p(clw)
2 E p(m)
dist
(21c)
10(w)) (w)
H(w C)
E dist p(w)
[o(w) e
(21d)
which completes the proof.
F
Data Statisties
We provide dataset statistics in Table 1.
Further Results
For more detailed reading, we provide MSE and Spearman correlation plots similar to Fig. and 2's but as bar plots in Fig. 8. We also provide per- language results:
1.0
on1
CCH
Sprammon Commlalinn
0.0
as function of the word filtering protocol used in our analysis in Fig.
function of the number of word types included in analysis in Fig. 10;
0.2
10
12
10
18
Language Model's Croeseettrogy
as function of our language model's cross- entropy in Fig. 11; and
Figure Spearman correlation with unigram surprisal as function of the cross-entropy of models used to get surprisal estimates
as function of the number of tokens used to train our language models and to get word count statistics in Fig. 12.
We also provide results when CCH is defined using generalized distfunctions, i.e., for several values of >, in Fig. 13. Finally, we show the Spearman correlation between CCH and CCH₁ and unigram surprisal as a function of the used language model's quality in Fig. 7.

---

None
Punetuation
Only Alphabet
Language
Family
ISO code
BPC
# Types
# Tokens
# Types
# Tokens
# Types
Tokens
German
Luropen
de
0.99
2,093,524
32,142,911
1,027,594
27,565,044 .,565,045
896,752
26,301,145
Greek
Tndo-Fil Furoppan
el
1.06
267,625
2,244,964
145,073
1,954,950
120,361
1,862,380
English
Indo-Eure Bud--er
en 1.09
2,419,694
78,392,487
748,109
66,881,077
609,839
65,261,138
Spanish
Indo-Eun Europeann
es 1.04
993,894
21,472,091
380,407
18,852,688
332.668
18,458,823
Estonian
Uralic
el 1.23
250,860
999,296
145,003
797.817
123,863
736,860
Finnish
Uralic
fi 1,03
566,755
2,741.783
333,229
2,249.003
318,514
2,156,132
Hebrew
Afro-Asiatic
he 1.41
497,550
4,153,846
230,867
3,394,959
208,077
3,299,727
Italian
Buro-Eur
it 1,04
823,397
14,500,421
297.153
12,371,701
269,005
12,056,925
Korean
Kureanic
2.40
538,093
1,953,812
385,948
1,622,552
331,060
1,460,230
Dutch
Indo-Luropean
nl 1.00
534,101
6,811.124
246,458
5,989.335
227.466
5,768.607
Norwegian
Indo- Indo-European
no 1.14
325,983
2,672,869
176,089
2,318,082
164,401
2,253,985
Russian
Indo-Eue Hulo--ur
1,06
1,474,777
15,824,324
707,806
13,073,143
546,179
11,922,144 147
Turkish
Turkic
tr
1.14
285,988
1,705,030
163,352
1,371,183
148,522
1,306,751
Table
1: Wiki40B data statistics.
4-
ZIPF
CCH
Correlation
0.4
2IPF
OCIL
MSB
3-
0.2
2-
Sprerman
0.0
de
61
60
68
60
he
18
kco
m1
130
111
tr
de
he
1t
100
Figure 8: MSE and Spearman correlation of the three hypotheses in the analyzed languages.
de
2.5
Spearman Correlation
0.0
0.0
in
al
he
it:
n1
0:21
n2
0.2
D.2
1.2
MSR
0.0
0.0
(w) E
6(00) E (w))
((w) E (fw) (le)
0.2
0.2
2,5
0.0
ZIFF
ZPF COHL
ere) DV
((ee e
cCll
E
(fa))
6(w)
(ww)
((w)
(fe))
((w)
Rw)) G(w)
((w)
((w)
(w)
Figure 9: MSE and Spearman correlation when hypotheses are evaluated while filtering words based
different protocols.
el
4-
1-
0.25
0.25
3.25
0/25
0.25
Speamann Correlation
kn
ml
MSE
0:25
0.25
1.25
0.25
0.25
0.00
TTEE TITE
TIDE TImE
TITE
TITE FITE
Tim Tim
mm
TTR 10
10a
10P
10
10
10P
10"
10"10"
10"
10'
30
Tu
RD
10
ZIPP
ZIPP
0:25
0.25
3.25
OCH
0:00
OCH
10
10
10P13!
102
10Pin!
10"
THE 106
OCH
1nf
10F
10P1n! 10"
TerTE vrem
10"1n
102
109
Figure 10: MSE and Spearman correlation when hypotheses are evaluated on different number of word types.

---

el
0.25
0.25
0.25
0-25
0.21
2-
2
0.00
Spoarman Corvelation
7.5
10.0
12.55
7.5
10.0
12.5
7.5
10.0
10
15
10
12.5
10
12.5
1000
10
15
ho
0.25
0.25
0.25
0.25
09
2-
2
0.00
10
15
10
15
12.5
15
20
15
10.0
12.5
10
15
10
15
1,5
10
12.5
20
1000
1r
81
ПPP
0.25
0/25
0.27
2IP
0.00
CCB
COH
125
15
10
13
109
10
1255
15
15
Lenguage Model's Cross-culrupy
Language Model's Cross-culropy
Figure 11: MSE and Spearman correlation function of the cross-entropy of models used to get surprisal estimates.
de
el
em
18
68
de
68
et
0.25
0.25
0.25
0.25
0.25
Spearmnn Correlation
0.00
he
lac
0.25
0.25
0.25
25
0.25
MISE
2
10
10a
10a
10
116
m0
110
Zipf
Zipf
0.25
0.25
0.25
Pinatabbei et ol.
Plantadoe et al.
0.00
108
10
20
CCII
108
10
:09
CCI
# Train Tokens
# Train Tokens
Figure 12: Results as a function of the number of tokens used to train language models and get count statistics.
de
e]
e1.
de
2
Spearman Conrelation
0.0
nl
0.2
0.2
0.2
0.2
0.2
MISE
0.0
2
2
fr
fr
0.2
0.2
0.0
Zipl
Piantadoci
CCH
OCH,
Zipf
Piantadosi et
CCH
CCH,
Figure 13: MSE and Spearman correlation function of the generalized dist's parameter (in the w-axiis.


Reviews:
[{"review_id": "ImMoQI0Hpj", "reviewer": "Reviewer_zBRX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is very clear and well written. It presents a very important contribution to language processing and should be definitely accepted. ", "reasons_to_reject": "No", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores the classical question of word length and its correlation with frequency. This classical results has been challenged by Piantadosi and colleagues, proposing a correlation between length and word's surprisal. In a first step, the paper precises the question of what is minimised in  Piantadosi's proposal, leading to a the definition of a channel capacity hypothesis (a channel corresponding information transfer between interlocutors). The paper aims at exploring the question of the  relation between communicative cost and word length and provides  theoretical and empirical contributions, providing new elements arguing in favor of the initial Zipf's hypothesis, corrrelating word lengths and frequency.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4EjTjj2GOD", "reviewer": "Reviewer_Wi23", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review!\n\nWe will fix the typo in line 424 for camera ready, and we will try to find more distinguished notations for the context and capacity in our equations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- interesting problem\n\n-- paper clear and well organized\n\n-- convincing experimental assessment and analysis of the results\n\n-- the paper has a few limitations, which are discussed in the last section, but the paper is interesting even with such problems", "reasons_to_reject": "None", "questions_to_authors": "None", "paper_topic_and_main_contributions": "This paper starts from existing theories on which information helps in previewing the length of word forms and aims at verifying them by exploiting more recent and precise language models. The considered hypotheses are introduced and carefully discussed, by also introducing new notation where necessary. In particular, they consider the Zipf law,  the channel capacity hypothesis and one variation of the latter. Empirical assessment has been performed on 13 different languages taken from the data set Wiki40B by means of an unbiased MLE estimator for unigram and monolingual transformers for the conditional probability. Results are complete and show the superiority of the Zipf law in explaining the data. In particular, the effect of different filters applied to tokenization and to the choice of the words to consider on one side and of the model quality on the other has been analyzed.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "Expression 4: I find that denoting context and capacity with such similar C's can be misleading: wouldn't it be possible to find a clearer notation?\n\nline 424: hypotheses -> hypothesis\n"}, "extra_text": {}}, {"review_id": "G7xBGvCV32", "reviewer": "Reviewer_Vf5C", "summary": "", "questions": "A: L 204-205 \"In order to solve for the optimal word lengths according to hypothesis 1, we will relax the wordform uniqueness condition\" -- Can you explain why it was not taken into account in the other equations, but here this condition was relaxed? We know that in addition to phonotactics, languages tend to have quite some homography and homophony (in some languages, like French, homophony is rather high). I see that you mention this as a limitation in the end of the paper, but I wonder why sometimes this condition can be relaxed with no problem.\n\nB: L 218-219 It would be great to see an explanation to what \"index of dispersion\" means\n\nC: L 454 It's not very clear what was the task for the model, a standard masked language modelling task?\n\nD: L 461 When using a vocabulary of subword units set to 32,000 the actual subwords will be pretty long and will be mostly similar to the actual words. This probably influences your results presented in Fig 3, thus, making both plots similar to each other. What would be interesting, is to compare several subword tokenisations with different vocabularies as a goal, e.g. 500, 1000, 15000 and to see if the metrics will change; I'd expect to see an impact of small vocabularies; when subword units would be smaller in length, they would have lower surprisal on average compared to full words constructed out of them (an extreme case would be to have just 1-character subwords, i.e. letters, to test this).\n\nIn this context, also the effects of non-alphabetic scripts would be interesting to discuss, since 1 character in English is different from 1 character in e.g. Korean. Note that Korean is located higher in Fig 3 on the bottom in Zipf (subwords) setting, but this was not discussed in the paper. Also Korean an Hebrew behave different from other languages in Fig 8 (Korean on the left, and Hebrew on the right). I would expect languages like Japanese and Chinese behave differently as well, this would be great to add to the discussion of your methods.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review! We answer the reviewer's questions below.\n\n\n> A: L 204-205 \"In order to solve for the optimal word lengths according to hypothesis 1, we will relax the wordform uniqueness condition\" -- Can you explain why it was not taken into account in the other equations, but here this condition was relaxed?\n\n\nWe relaxed the assumption there because we were not able to derive a closed form solution to the channel capacity hypothesis (in equation 4) while considering it. We are currently still trying to find an efficient way of computing a solution to equation 4 while considering this assumption, but we were not yet able to. We will make the reason behind this limitation explicit in the paper for camera ready.\n\n\n> B: L 218-219 It would be great to see an explanation to what \"index of dispersion\" means\n\n\nWe will add this definition to the paper: the index of dispersion is defined as the ratio between a random variable’s variance and its mean. We will also show how eq. 5 relates to it. Explicitly, we will add the equations below to make it easier for readers to follow: $\\frac{\\mathrm{E}[\\mathrm{H}^2]}{\\mathrm{E}[\\mathrm{H}]} = \\frac{\\mathrm{E}[\\mathrm{H}^2]}{\\mathrm{E}[\\mathrm{H}]} + \\mathrm{E}[\\mathrm{H}] - \\mathrm{E}[\\mathrm{H}] = \\frac{\\mathrm{E}[\\mathrm{H}^2] - \\mathrm{E}^2[\\mathrm{H}]}{\\mathrm{E}[\\mathrm{H}]} + \\mathrm{E}[\\mathrm{H}] = \\frac{\\mathrm{Var}[\\mathrm{H}]}{\\mathrm{E}[\\mathrm{H}]} + \\mathrm{E}[\\mathrm{H}]$. \n\n\n\n> C: L 454 It's not very clear what was the task for the model, a standard masked language modelling task?\n\n\nWe train our models as autoregressive language models (line 453), so to predict a word given previous ones. We will try to make this clearer in the paper for camera ready.\n\n\n> D: L 461 When using a vocabulary of subword units set to 32,000 the actual subwords will be pretty long and will be mostly similar to the actual words. This probably influences your results presented in Fig 3, thus, making both plots similar to each other. What would be interesting, is to compare several subword tokenisations with different vocabularies as a goal, e.g. 500, 1000, 15000 and to see if the metrics will change; I'd expect to see an impact of small vocabularies; when subword units would be smaller in length, they would have lower surprisal on average compared to full words constructed out of them (an extreme case would be to have just 1-character subwords, i.e. letters, to test this).\n\n\nWe agree with the reviewer that these would indeed be interesting experiments. The training of all language models in the paper was quite expensive already, though. We will consider whether to add this extra experiment for camera ready, or to leave it for future work.\n\n\n> In this context, also the effects of non-alphabetic scripts would be interesting to discuss, since 1 character in English is different from 1 character in e.g. Korean. Note that Korean is located higher in Fig 3 on the bottom in Zipf (subwords) setting, but this was not discussed in the paper. Also Korean and Hebrew behave different from other languages in Fig 8 (Korean on the left, and Hebrew on the right). I would expect languages like Japanese and Chinese behave differently as well, this would be great to add to the discussion of your methods.\n\n\nThis is another good point. Non-alphabetic scripts should lead to shorter words, on average, which in turn may lead to the lower MSE values observed for, e.g. Korean, in our results. We will add a discussion on how a language’s script may influence our results for camera ready. \n\n\n> References + Typos\n\n\nWe will add the suggested reference and fix the pointed out typo. Thanks for bringing them to our attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper might have a big impact on the study of language complexity\n- The paper revisits a well-known and influential hypothesis by Piantadosi et al. 2011\n- The results can influence research on interpretability in multilingual NLP\n- The paper is well-written, provides many experiments confirming the predictive power of word frequency\n", "reasons_to_reject": "- There are small points that could be improved, which I point out below in the section Questions for the authors; otherwise many questions I had in mind were already replied by the authors in the paper", "questions_to_authors": "A: L 204-205 \"In order to solve for the optimal word lengths according to hypothesis 1, we will relax the wordform uniqueness condition\" -- Can you explain why it was not taken into account in the other equations, but here this condition was relaxed? We know that in addition to phonotactics, languages tend to have quite some homography and homophony (in some languages, like French, homophony is rather high). I see that you mention this as a limitation in the end of the paper, but I wonder why sometimes this condition can be relaxed with no problem.\n\nB: L 218-219 It would be great to see an explanation to what \"index of dispersion\" means\n\nC: L 454 It's not very clear what was the task for the model, a standard masked language modelling task?\n\nD: L 461 When using a vocabulary of subword units set to 32,000 the actual subwords will be pretty long and will be mostly similar to the actual words. This probably influences your results presented in Fig 3, thus, making both plots similar to each other. What would be interesting, is to compare several subword tokenisations with different vocabularies as a goal, e.g. 500, 1000, 15000 and to see if the metrics will change; I'd expect to see an impact of small vocabularies; when subword units would be smaller in length, they would have lower surprisal on average compared to full words constructed out of them (an extreme case would be to have just 1-character subwords, i.e. letters, to test this).\n\nIn this context, also the effects of non-alphabetic scripts would be interesting to discuss, since 1 character in English is different from 1 character in e.g. Korean. Note that Korean is located higher in Fig 3 on the bottom in Zipf (subwords) setting, but this was not discussed in the paper. Also Korean an Hebrew behave different from other languages in Fig 8 (Korean on the left, and Hebrew on the right). I would expect languages like Japanese and Chinese behave differently as well, this would be great to add to the discussion of your methods.", "paper_topic_and_main_contributions": "The paper addresses the well-known Zipf's law abbreviation and the ways of predicting word length in languages. In particular, the authors revisit a paper by Piantadosi et al. 2011, which claimed that word length is predicted better by surprisal than by frequency, as stated by Zipf's law. The authors find out that Zipf's law holds and that word frequency has a better predictive power than surprisal. In addition to that, the authors clarify the hypothesis by Piantadosi et al. 2011 and construct a more exact mathematical formula for their hypothesis (minimising the channel capacity cost). ", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "L 491-494 This idea comes from Menzerath-Altmann's law \"The longer the whole, the smaller the parts\", would be great to add a citation Menzerath 1954, Altmann 1980", "Typos_Grammar_Style_and_Presentation_Improvements": "page 7 footnote: \"11 We punctuation to be\" misses a verb"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is clear, well written, and presents significant results. It combines sound modeling, thorough linguistic understanding, and meaningful connections to human cognition. It is particularly well suited to represent this track.
Decision: Accept-Main