SlowGuess/NAACL_2021_OCR huggingface dataset:
In this equation,  $K_{<n}$  is the current number of populated clusters; while  $c_{<n}^{(z_n)}$  is the number of instances currently assigned to cluster  $z_n$ . The PY-CRP has two hyperparameters:  $0 \leq a < 1$  and  $b \geq 0$ . The parameter  $a$  controls the rate in which the clusters grow (Teh, 2006), while  $b$  controls an initial preference for dispersion. Together, these ensure the formation of a long-tail—concocting a power-law distribution for the cluster frequencies. This property allows a cluster with wordform make, for example, to have an exponentially larger frequency than its graphotactic neighbor cake.
Modeling Word Tokens. Finally, given the set of wordforms and the cluster assignments, defining the form associated with a token is deterministic. Since each cluster only contains instances of one wordform, the form of a token is defined looking up the label of the cluster it was assigned to  $\ell_{z_n}$ :
$$
p \left(W _ {n} = \boldsymbol {w} _ {n} \mid z _ {n}, \ell\right) = \mathbb {1} \left\{\boldsymbol {w} _ {n} = \ell_ {z _ {n}} \right\} \tag {11}
$$
This way, the adaptor captures the frequency information of the words in the corpus—whereas the generator can focus on learning the language's graphotactics and morphology.
Model training. Unfortunately, we cannot directly infer the parameters of our model with a closed form solution. We thus use a solution akin to expectation maximization (Wei and Tanner, 1990): We freeze our LSTM generator while learning the PYCRP parameters, and vice versa. The PYCRP is trained using Gibbs sampling. For each token, we fix all cluster assignments  $\mathbf{z}_{-n}$  except for one  $z_{n}$ . This cluster is then re-sampled from the marginal  $p(Z_{n} = z_{n} \mid \mathbf{z}_{-n}, \boldsymbol{\ell}, \mathbf{w}_{n})$ , where we have access to  $\mathbf{w}_{n}$  since it is an observed variable. During this optimization small clusters may vanish, and new clusters  $z_{n} = K + 1$  (previously with no samples) may be created. This procedure, thus, may also produce new sets of wordforms  $\{\ell_{k}\}_{k=1}^{K'}$ , composed of the populated clusters' labels (where  $K'$  is the new number of clusters). We assume the distribution of these wordforms—which have dampened frequencies—to be more balanced than in the original full set of word tokens. The LSTM is trained using stochastic gradient descent, minimizing the cross-entropy of precisely this set of cluster's wordforms. As such, it is expected to be a more representative model of a language's graphotics; the irregular common words are less dominant in this training set. We give a longer explanation of our
model training procedure, together with the used hyperparameters, in App. B.
4.2 A More Intuitive Explanation
Despite its slightly odd formulation, the two-stage model has an intuitive interpretation. Once we have learned (and fixed) its parameters, we obtain the marginal probability of a wordform as:
$$
p (\boldsymbol {w}) = \tag {12}
$$
$$
\underbrace {\frac {c _ {\boldsymbol {w}} - \overbrace {\boldsymbol {n} _ {\boldsymbol {w}} \cdot \boldsymbol {a}} ^ {\text {s m o o t h i n g f a c t o r}}}{| \mathbf {z} | + b}} _ {\text {s m o o t h e d u n i g r a m f r e q u e n c i e s}} + \underbrace {\frac {(a \cdot K + b)}{| \mathbf {z} | + b}} _ {\text {i n t e r p o l a t i o n w e i g h t}} \cdot \underbrace {p _ {\phi} (\boldsymbol {w})} _ {\text {L S T M}}
$$
In this equation,  $c_{w}$  is the count of tokens with form  $w$  in the training set, while  $n_{w}$  is the number of distinct clusters with this same form. The model interpolates between a smoothed unigram corpus frequency and the probability an LSTM gives the analyzed wordform. This interpolation enables the model to place a non-zero probability mass on all possible wordforms—thus modeling an open vocabulary and having infinite support—while also placing a large probability mass on frequent wordforms. Furthermore, the smoothing factors per word type, together with the interpolation weight, are holistically learned by the PYCRP model using the training set.[10]
The value in which we are interested in this work is the expected cost of a code, given in eq. (4). We can easily estimate this value for a natural code by using its sample estimate:
$$
\operatorname {c o s t} \left(\mathbb {C} _ {n a t}\right) \approx \frac {1}{N} \sum_ {n = 1} ^ {N} \left| \mathbb {C} _ {n a t} \left(m _ {n}\right) \right| = \frac {1}{N} \sum_ {n = 1} ^ {N} \left| \boldsymbol {w} _ {n} \right| \tag {13}
$$
For an optimal code, we can upperbound it using the entropy of the distribution, while the entropy itself can be upperbounded by the cross-entropy of a model on it. We can compute this upperbound with a sample estimate of the cross-entropy:
$$
\begin{array}{l} \operatorname {c o s t} \left(\mathbb {C} _ {\star}\right) \leq \mathrm {H} (W) + 1 \leq \mathrm {H} _ {\theta} (W) + 1 \tag {14} \\ \lesssim \frac {1}{N} \sum_ {n = 1} ^ {N} \log_ {| \Sigma |} \frac {1}{p _ {\theta} (\boldsymbol {w} _ {n})} + 1 \\ \end{array}
$$
10Our model consistently produced lower cross-entropies (on held out tokens) to the ones of an LSTM baseline naively trained on a language's tokens.
In practice, we get a tighter estimate by using the Shannon (1948) code's lengths directly:
$$
\operatorname {c o s t} \left(\mathbb {C} _ {\star}\right) \lesssim \frac {1}{N} \sum_ {n = 1} ^ {N} \left[ \log_ {| \Sigma |} \frac {1}{p _ {\theta} (\boldsymbol {w} _ {n})} \right] \tag {15}
$$
where  $\lceil \cdot \rceil$  is the ceiling operation.
5.2 Morphological Constraints
As mentioned in §2, we use Morfessor (Smit et al., 2014) to tokenize our corpus into morphological units. Morfessor is a method for finding morphological segmentations from raw text data. As an unsupervised model, Morfessor is inherently noisy, but we take it as a proxy for a language's morphological segmentation. To compare the robustness of our results across different unsupervised segmentation algorithms, though, we also run our experiments using byte pair encoding (BPE; Gage, 1994; Sennrich et al., 2016) and WordPieces (Schuster and Nakajima, 2012).
We train Morfessor on all pre-tokenized sentences in our language-specific Wikipedia corpus (described in §5.4). With this pre-trained model in hand, we tokenize all words in our training, development and test sets. We get a set of morpheme tokens  $\{\pmb{u}_{n,j}\}_{j=1}^{J_n}$  for each word  $\pmb{w}_n$ , where this word is split into  $J_{n}$  morphological units.
We can now get the optimal length of a morphologically constrained code. With this in mind, we first train a fresh version of our two-stage model on the full set of morphological unit tokens—i.e.  $\{\pmb{u}_{n,j} \mid n \leq N, j \leq J_n\}$ , as opposed to the set of full word tokens,  $\{\pmb{w}_n\}_{n=1}^N$ . We estimate the length of this code with the following equation:
$$
\begin{array}{l} \operatorname {c o s t} \left(\mathbb {C} _ {\text {m o r p h}}\right) = \sum_ {m \in \mathcal {M}} p (m) \left| \mathbb {C} _ {\text {m o r p h}} (m) \right| \tag {16} \\ \lesssim \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {j = 1} ^ {J _ {n}} \left\lceil \log_ {| \Sigma |} \frac {1}{p _ {\theta} (\boldsymbol {u} _ {n , j})} \right\rceil \\ \end{array}
$$
Note that this cost estimate is still the average code-length per word token, as such we take the expectation over the meanings distribution. Each word's code-length, though, is now defined as the sum of the length of each of its constituent morphemes.
5.3 Graphotactic Constraints
The second linguistic constraint we would like to impose on our codes is graphotactic well-formedness—i.e. we wish our code to be composed only by sequences of characters that comply with the regularities observed in the language,
such as e.g. vowel harmony, syllable structure, or word-initial and word-final constraints. We use our generator LSTM for this. As mentioned before, this model is trained on wordforms with dampened frequencies—we thus expect it to learn a language's graphotactic patterns above a minimum quality threshold. We use this character-level model to sample (without replacement) as many unique wordforms as there are word types in that language (see Tab. 3 in App. A).<sup>11</sup> We assign each of these sampled wordforms  $\boldsymbol{w}_n'$ , ordered by word length, to one of the languages meanings  $m_n$ , inversely ordered by unigram probability, i.e.  $\mathbb{C}_{graph}(m_n) = \boldsymbol{w}_n'$ —thus generating an optimally Zipfian frequency-length correlation. With these assignments, we estimate the cost of a graphotactically constrained code:
$$
\operatorname {c o s t} \left(\mathbb {C} _ {\text {g r a p h}}\right) \approx \frac {1}{N} \sum_ {n = 1} ^ {N} | \boldsymbol {w} _ {n} ^ {\prime} | \tag {17}
$$
Analogously, with the generator trained on morpheme units we get an optimal code under both morphological and graphotactic constraints.
$$
\operatorname {c o s t} \left(\mathbb {C} _ {\text {m o r p h} + \text {g r a p h}}\right) \approx \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {j = 1} ^ {J _ {n}} \left| \boldsymbol {u} _ {n, j} ^ {\prime} \right| \tag {18}
$$
5.4 Dataset
We use Wikipedia data in our experiments. The data is preprocessed by first splitting it into sentences and then into tokens using SpaCy's language-specific sentencizer and tokenizer (Honnibal et al., 2020). After this, all punctuation is removed and the words are lower-cased. We subsample (without replacement) one million sentences of each language for our experiments, due to computational constraints. We then use an 80-10-10 split for our training, validation and test sets.
We choose typologically diverse languages for our experiments, each from a different language family: English, Finnish, Hebrew, Indonesian, Tamil, Turkish and Yoruba.12 These languages vary in their graphotactic tendencies and morphological
Unfortunately, our LSTMs use a softmax non-linearity to assign probabilities and, as such, can't produce zeros. Furthermore, due to the compositional nature of wordform probabilities (see eq. (8)), short implausible forms may have larger probability mass than long plausible ones. To mitigate this effect, when sampling wordforms we impose a minimum threshold of 0.01 on each transition probability  $p(\ell_t \mid \ell_{<t})$ .
$^{12}$ Dataset statistics are presented in App. A.
Figure 3: Bar plots of the code lengths under different constraints. In this plot, morphology is constrained through the use of Morfessor segmentation. Length in the shuffle condition for Tamil and Finnish exceed the scale (11.5 and 11 respectively). Optimal, Morph, Zipfian, Graph, Morph + Graph, Natural, Shuffle.
complexity. In order to improve our data quality, we hand-defined an alphabet for each language and filter sentences with them, only considering sentences consisting exclusively of valid characters.[13]
In this paper we consider the following codes:
Optimal. An information-theoretically optimal code under our two-stage model, estimated as defined by eq. (15). This is our most compressed code and does not include either morphological or graphotactic constraints.
Morph. A morphologically constrained code, as defined by eq. (16).
Graph. A code constrained by graphotactics, as defined by eq. (17).
Morph+Graph. A code constrained by both morphology and graphotics; defined by eq. (18).
Natural. The natural code—equivalent to the average token length and defined by eq. (13). This is the code length actually observed in our corpora.
Zipfian. A code estimated by re-pairing wordforms with meanings based on their frequencies; we then compute eq. (13) in this new code. This would be equivalent to the natural code length if lexicons had a perfect word length-frequency correlation (i.e., a Spearman's rank correlation of 1).
Shuffle. A code estimated by randomly repairing wordforms with meanings and computing eq. (13) in this new code. This would be equivalent to the natural code length if Zipf's law of abbreviation did not exist, i.e. lexicons had no word length-frequency correlation.
The average length for each considered code is presented in Fig. 3 and Tab. 1. As expected, we find that the average code length across natural languages is shorter than the shuffle condition and longer than the optimal condition. Interestingly, the codes produced by the other conditions investigated here also have the same identical order across all analyzed languages.
Adding morphological constraints on the code incurs no more than one extra character over the optimal condition—except for Finnish, for which the cost of morphology is slightly above one character. Notably, the use of unsupervised morphological segmentation may introduce some noise into our measurements. Consistently with our expectations, though, Yoruba (a morphologically poor language) pays the smallest cost for its morphology, while Finnish (a morphologically rich one) pays the largest.
BPE and WordPiece systematically produce shorter codes than Morfessor. This is sensible, since the first two would keep most frequent wordforms intact, generating a unique code for each of them. This would lead to codes in which the morphological productivity of frequent and infrequent words differ, amplifying frequency effects encountered in natural languages (Lieberman et al., 2007).
The graphotactic condition yields systematically longer codes than the morphological one, although here there are important differences between languages: English, Hebrew and Indonesian have similar code lengths for both code constraints; in the other languages the graphotactic code is substantially longer than the morphological one.
In all cases, the natural code is longer than the one with both graphotactic and morphological constraints—suggesting languages are not opti
<sup>13</sup>We define these sets of valid characters based on Wikipedia entries for the languages and the alphabets available in https://r12a.github.io/app-charuse/.
Language	Optimal	Morph	Graph	Morph + Graph	Zipfian	Natural	Shuffle
Morfessor	BPE	WordPieces	Morfessor	BPE	WordPieces
English	3.09	3.82	3.34	3.31	4.39	5.34	4.67	4.70	3.93	6.11	8.91
Finnish	3.89	5.13	4.94	4.95	7.37	7.55	7.60	7.65	6.59	8.72	10.97
Hebrew	3.52	4.38	3.98	3.99	4.82	5.19	4.95	4.88	4.50	5.79	6.97
Indonesian	3.31	4.08	3.67	3.66	4.63	5.08	5.02	4.95	4.25	7.06	8.30
Tamil	3.38	4.15	4.07	4.01	7.52	8.01	8.16	8.22	6.41	9.21	11.48
Turkish	3.52	4.28	4.12	4.03	5.67	6.31	5.98	5.93	5.31	7.52	9.09
Yoruba	2.84	3.18	3.00	2.97	4.63	4.85	4.69	4.61	4.24	5.34	7.10
Table 1: The average code lengths under the different coding schemes.
Figure 4: Comparison of the distances—additive (left) and multiplicative (right)—between natural languages and either optimal or shuffled baselines.
mally compressed, even when accounting for these constraints. That said, all of the natural languages are considerably more compressed than a lexicon produced by randomly reassigning wordforms.
In this paper, we introduced a model-based strategy to assess the relative contribution of different constraints on word (code) length at large. In particular, we evaluated how much natural languages differ from systems optimized for Zipf's law of abbreviation. Our proposed model improves upon an old method used to consider the efficiency of the lexicon: random typing models (Miller, 1957; Moscoso del Prado, 2013; Ferrer-i-Cancho et al., 2020). Miller introduced the idea of monkeys typing randomly on a keyboard and analyzed the properties of its resulting language. The monkeys' text, however, has no morphological or graphotactic constraints (but see Caplan et al., 2020) and does not follow a language's unigram distribution (Howes, 1968). As such, it cannot directly encode the same meanings or messages as the original language.
Our results show that, while natural languages do tend to map frequent messages to shorter words, the magnitude of this effect varies widely across our set of diverse languages. Notably, the distance between natural languages and the optimal codes is
Figure 5: Fraction of code length accounted for by the combined morphology and graphotactics model
larger than the distance between natural languages and their corresponding shuffled code (see Fig. 4). In other words, natural codes are closer to not being optimized (in the Zipfian sense) than to being maximally compressed.
That said, our morphological and graphotactic baselines, when combined, yield codes that display mean code lengths that are (in most cases) closer to the natural code than to the optimal (see Fig. 5). If our models are indeed able to capture the true patterns in our data, then this means that (compositional) morphology and graphotactics, along with the law of abbreviation, are sufficient to account for most of the length of natural codes—as observed in real languages. Graphotactic (primarily) and morphological constraints are enough to derive a code with a similar complexity to that of natural languages, which suggests the other factors discussed above (associated with, e.g., surprisal and non-arbitrary form-meaning mappings) likely play a more modest role in pushing natural languages away from the optimal Zipfian code.
The optimality of the lexicon occupies a major place in the scientific study of the structure and functional evolution of languages (Bentz and Ferrer-i-Cancho, 2016; Gibson et al., 2019; Mahowald et al., 2020). We hope that the method presented here—which allows for a more precise quantification of the (non-)optimality of lexicons—will be used to further the goal of understanding why languages are structured in the ways that they are, while offering insight into the functional trade-offs that underlie language variation and change.
This paper concerns itself with investigating lexicons' optimality under the perspective of Zipf's Law of Abbreviation. As we focus on computational linguistic experiments, we see no clear ethical concerns here. Nonetheless, we note that Wikipedia (from where we collect data) is not a fully representative source of a language's data—the biases in the data will likely also be present in our results.
Damián E. Blasi acknowledges funding from the Branco Weiss Fellowship, administered by the ETH Zürich. Damián E. Blasi's research was also executed within the framework of the HSE University Basic Research Program and funded by the Russian Academic Excellence Project '5-100'.
András Antos and Ioannis Kontoyiannis. 2001. Convergence properties of functional estimates for discrete distributions. *Random Structures & Algorithms*, 19(3-4):163-193.
Evan Archer, Il Memming Park, and Jonathan W. Pillow. 2014. Bayesian entropy estimation for countable discrete distributions. Journal of Machine Learning Research, 15(81):2833-2868.
Matthew Aylett and Alice Turk. 2004. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47(1):31-56.
Alan Bell, Daniel Jurafsky, Eric Fosler-Lussier, Cynthia Girand, Michelle Gregory, and Daniel Gildea. 2003. Effects of disfluencies, predictability, and utterance position on word form variation in English conversation. The Journal of the Acoustical Society of America, 113(2):1001-1024.
Chris Bentz and Ramon Ferrer-i-Cancho. 2016. Zipf's law of abbreviation as a language universal. In Proceedings of the Leiden Workshop on Capturing Phylogenetic Algorithms for Linguistics, pages 1-4, Leiden, Netherlands. University of Tübingen.
Damián E. Blasi, Søren Wichmann, Harald Hammarström, Peter F. Stadler, and Morten H. Christiansen. 2016. Sound-meaning association biases evidenced across thousands of languages. Proceedings of the National Academy of Sciences, 113(39):10818-10823.
Phil Blunsom, Trevor Cohn, Sharon Goldwater, and Mark Johnson. 2009. A note on the implementation
of hierarchical Dirichlet processes. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 337-340, Suntec, Singapore. Association for Computational Linguistics.
Paul Boersma. 1998. Functional Phonology. Holland Academic Graphics.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4617-4624, Online. Association for Computational Linguistics.
Spencer Caplan, Jordan Kodner, and Charles Yang. 2020. Miller's monkey updated: Communicative efficiency and the statistics of words in natural language. Cognition, 205:104466.
Bernardino Casas, Antoni Hernández-Fernández, Neus Català, Ramon Ferrer-i-Cancho, and Jaume Baixeries. 2019. Polysemy and brevity versus frequency in language. Computer Speech & Language, 58:19-50.
Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing, 4(1).
Isabelle Dautriche, Laia Fibla, Anne-Caroline Fievet, and Anne Christophe. 2018. Learning homophones in context: Easy cases are favored in the lexicon of natural languages. Cognitive Psychology, 104:83-105.
Isabelle Dautriche, Kyle Mahowald, Edward Gibson, and Steven T. Piantadosi. 2017. Wordform similarity increases with semantic similarity: An analysis of 100 languages. Cognitive Science, 41(8):2149-2169.
Mark Dingemanse, Damián E. Blasi, Gary Lupyan, Morten H. Christiansen, and Padraic Monaghan. 2015. Arbitrariness, iconicity, and systematicity in language. Trends in Cognitive Sciences, 19(10):603-615.
August Fenk and Gertraud Fenk. 1980. Konstanz im Kurzzeitgedächtnis - Konstanz im sprachlichen Informationsfluß? Zeitschrift für Experimentelle und Angewandte Psychologie, 27(3):400-414.
Gertraud Fenk-Oczlon and August Fenk. 2008. Complexity trade-offs between the subsystems of language. Language Complexity: Typology, Contact, Change, 43:65.
Ramon Ferrer-i-Cancho, Christian Bentz, and Caio Seguin. 2020. Optimal coding and the origins of Zipfian laws. Journal of Quantitative Linguistics, 0(0):1-30.
Ramon Ferrer-i-Cancho and Michael S. Vitevitch. 2018. The origins of Zipf's meaning-frequency law. Journal of the Association for Information Science and Technology, 69(11):1369-1379.
Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23-38.
Susanne Gahl. 2008. Time and thyme are not homophones: The effect of lemma frequency on word durations in spontaneous speech. Language, 84(3):474-496.
Edward Gibson, Richard Futrell, Steven P. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. 2019. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):389-407.
Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2011. Producing power-law distributions and damping word frequencies with two-stage language models. Journal of Machine Learning Research, 12:2335-2382.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in python.
Davis Howes. 1968. Zipf's law and Miller's random-monkey model. The American Journal of Psychology, 81(2):269-272.
David A. Huffman. 1952. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098-1101.
Dan Jurafsky, Alan Bell, Michelle Gregory, and William Raymond. 2001. Probabilistic relations between words: Evidence from reduction in lexical production. In Frequency and the Emergence of Linguistic Structure. John Benjamins, Amsterdam.
Adam King and Andrew Wedel. 2020. Greater early disambiguating information for less-probable words: The lexicon is shaped by incremental processing. Open Mind, pages 1-12.
Roger P. Levy and Tim Florian Jaeger. 2007. Speakers optimize information density through syntactic reduction. In Advances in Neural Information Processing Systems, pages 849-856.
Molly L. Lewis and Michael C. Frank. 2016. The length of words reflects their conceptual complexity. Cognition, 153:182-195.
Erez Lieberman, Jean-Baptiste Michel, Joe Jackson, Tina Tang, and Martin A. Nowak. 2007. Quantifying the evolutionary dynamics of language. Nature, 449(7163):713-716.
Tamas Linder, Vahid Tarokh, and Kenneth Zeger. 1997. Existence of optimal prefix codes for infinite source alphabets. IEEE Transactions on Information Theory, 43(6):2026-2028.