SlowGuess/NAACL_2021_OCR huggingface dataset:
Tiago Pimentel*, Irene Nikkarinen*,,y Kyle Mahowald Ryan Cotterell, Damián E. Blasi,
$^{3}$ University of Cambridge  $^{1}$ Yle  $^{1}$ University of California, Santa Barbara  $^{1}$ ETH Zürich  $^{1}$ Harvard University  $^{9}$ MPI for Evolutionary Anthropology  $^{9}$ HSE University
tp472@cam.ac.uk, irene.nikkarinen@gmail.com, mahowald@ucsb.edu, ryan.cotterell@inf.ethz.ch, dbl@fas.harvard.edu
The mapping of lexical meanings to wordforms is a major feature of natural languages. While usage pressures might assign short words to frequent meanings (Zipf's law of abbreviation), the need for a productive and open-ended vocabulary, local constraints on sequences of symbols, and various other factors all shape the lexicons of the world's languages. Despite their importance in shaping lexical structure, the relative contributions of these factors have not been fully quantified. Taking a coding-theoretic view of the lexicon and making use of a novel generative statistical model, we define upper bounds for the compressibility of the lexicon under various constraints. Examining corpora from 7 typologically diverse languages, we use those upper bounds to quantify the lexicon's optimality and to explore the relative costs of major constraints on natural codes. We find that (compositional) morphology and graphotactics can sufficiently account for most of the complexity of natural codes—as measured by code length.
Communication through language can be modeled under Shannon's classic communication framework (Shannon, 1948). Under this perspective, linguistic utterances are codes—which need to be decoded by a receiver ( listener) who is interested in the message (meaning) they encode. Famously,Zipf (1949) posited that language users shape these codes so to accommodate the principle of least effort. The most widely discussed and investigated empirical evidence for this feature is the so-called law of abbreviation, an ostensive negative correlation between word frequency and word length (Zipf, 1935; Bentz and Ferrer-i-Cancho, 2016). Communication effort decreases by encoding frequent messages in shorter words.
Figure 1: The average code length—under our coding schemes—on a representative language (Finnish). The distance between the baselines can be thought of as the cost of each constraint added to the system.
This correlation, however, is characteristically modest. There are many instances of short low-frequency words, like wen and jib in English, $^{1}$  and long frequent words, like happiness and anything. While the lexicon might be shaped by economy of expression, it is clearly not fully optimized for it. There are multiple—possibly competing—reasons why this could be the case.
First of all, the sequence of speech sounds, signs, or orthographic characters that serve as building blocks in a language comply with specific rules. These are referred to as phonotactics (in the case of speech sounds) and graphotactics (in written language).2 On top of these constraints, the lexicons of many languages of the world re-use subparts of words; these sub-parts can be productively composed to produce new meanings—which is referred to as morphological composition. This largely determines the family of wordforms associated with a given basic meaning—for instance, given the wordform health and its meaning, the nominal morphology of English readily provides
These mean, respectively, a benign tumor on the skin and a triangular sail on a boat.
2All languages impose these constraints on their wordforms, which might be leveraged for production and learnability (Vitevitch and Luce, 1999; Boersma, 1998).
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4426-4438 June 6-11, 2021. ©2021 Association for Computational Linguistics
the forms for many of its derived meanings, including healthy, unhealthy, healthier, etc.
Beyond these well-attested constraints, it might be argued that the negative correlation between the length of a word and its frequency is not the locus of optimization given the economy of expression pressure. Instead, wordforms might be efficiently encoding meanings based on their contextual surprise rather than frequency (Piantadosi et al., 2011). Finally, there is no reason to expect lexicons to be fully optimized for the economy of expression—this factor might steer languages in a given direction, but there is certainly room for non-compliance. Languages are, after all, not engineered systems but cultural artifacts.
In this paper, we examine how marginally non-optimal the lexicon is by taking the vantage point of the law of abbreviation. We develop a method to quantify the role of different linguistic constraints on determining wordforms, and we produce estimates on how compressible the lexicon could be in their absence (including morphology and phonotactics/graphotactics). We thus define an upper bound for the compressibility of a lexicon optimized purely for word length efficiency.
2 (Non-)Optimality in the Lexicon
As stated above, our notion of optimality is derived from Zipf's principle of least effort in the form of the law of abbreviation (Zipf, 1949; Mandelbrot, 1953; Ferrer-i-Cancho et al., 2020). However, this is by no means the only theory under which wordforms are optimized for encoding their messages.
One influential hypothesis is that languages optimize for uniform information density (Fenk and Fenk, 1980; Aylett and Turk, 2004; Levy and Jaeger, 2007)—roughly keeping the amount of information conveyed in a unit of time constant. In an information-theoretic setting, this would be equivalent to maximizing the use of a noisy channel between the speaker and an audience—keeping the transmission rate close to the channel capacity.
Under this view, it is not necessarily the case that words should be as short as possible. Rather, words that are infrequent or typically less predictable in context should be longer and take more time to produce—perhaps because the increased duration makes them more robust to noise. Consistent with this perspective, it has been shown that, in production, words with higher information content take longer to pronounce (Bell et al., 2003;
Jurafsky et al., 2001; Gahl, 2008). Additionally, words which are typically predictable in context are shorter than words which are less predictable in context (Piantadosi et al., 2011).
On another note, a purely coding-theoretically efficient language could make the lexical codes context dependent (Piantadosi et al., 2012), since context often disambiguates words (Dautriche et al., 2018; Pimentel et al., 2020a). Additionally, the meaning or message being conveyed by a given word might bias its form. Within languages, there seems to be a pressure for more semantically similar words to also be more phonologically similar (Monaghan et al., 2014; Dingemanse et al., 2015; Dautriche et al., 2017; Pimentel et al., 2019). Across languages, words for the same referents exhibit detectable patterns in term of their phonological makeup (Blasi et al., 2016), phonotactics (Pimentel et al., 2021b), as well as word length (Lewis and Frank, 2016)—this is driven by semantic features such as size, quality or complexity. Finally, there is a cross-linguistic tendency for lexicons to place higher surprisal in word-initial segments (van Son and Pols, 2003a,b; King and Wedel, 2020; Pimentel et al., 2021a) making words more constrained in their choice of final segments. These aspects of language might also collide with a purely Zipfian conception of lexicon optimality.
In this work, however, we consider optimality exclusively in the Zipfian sense of compressibility, and we ask how far natural language lexicons are from accommodating to this paradigm. We build a number of models that differ in relation to whether they accommodate to the law of abbreviation, to compositional morphology and to graphotactics. The comparison among these systems allows us to explore the extent to which each part of the linguistic system contributes to the overall cost of the linguistic code. It should be noted, though, that the consequences of unmodeled sources of structure in the lexicon (such as persistent sound-meaning associations or the adaptation of the code to surprisal effects) will forcibly be confounded with the overall lack of fit between our models and the data.
The morphological cost -i.e. the cost of morphology to a code's length-is associated with the fact that, across many languages, words are often constructed of meaningful sub-parts that are productively reused across the lexicon. Practically, this means that the wordforms of different meanings might not be independent if they overlap in a partic-
ular dimension that is captured by the morphology of the language. For instance, most wordforms that express two or more referents of a kind share a word-final suffix -s in English (towers, cats, ideas, etc). We treat this cost by considering optimal codes where the basic unit in the lexicon is not the word but sub-pieces, as determined by the unsupervised morphological parser Morfessor (Creutz andLAGus,2007;Smit et al.,2014).Under this regime, a word like unmatched is parsed into the tokens un, match, and ed.
The graphotactic cost concurrently imposes a set of additional constraints, determining which sequences of grapheme segments can constitute a valid wordform in a given language. While the main driver of these lexical constraints is actually phonotactics—which imposes rules dictating the possible phoneme sequences—we focus on graphotactics because our object of study is written language corpora. The degree to which phonotactics and graphotactics mirror each other vary substantially across languages; thus, in this work (which uses corpora from Wikipedia) we make our claims about language in the written modality and leave it to future work to generalize this work to the phonological domain. This could be done by applying the same method to phonemic representations of words.
3 A Coding-theoretic View of the Lexicon
This paper treats the lexicon, which we define as a set of pairs:  $\mathcal{L} = \{(m_n,\boldsymbol {w}_n)\}_{n = 1}^N$ . In general, this set will be infinite;  $m_{n}$  refers to a lexical meaning, taken from an abstract set  $\mathcal{M}$ , and  $\boldsymbol{w}_{n}$  refers to a wordform, taken from  $\Sigma^{*}$ , the Kleene closure of a grapheme alphabet  $\Sigma$ . When the exact index is unnecessary in context, we will drop the subscripted  $n$ ; and we make use of uppercase letters to refer to random variables (e.g.  $M$  or  $W$ ) where necessary. We will write meanings in typewriter font, e.g. cat, and wordforms in italics: cat (English), kissa (Finnish).
Viewing the lexicon from a coding-theoretic perspective, we consider the mapping from meaning to form as a code:  $\mathbb{C}:\mathcal{M}\to \Sigma^{*}$ . Every language comes endowed with a natural code  $\mathbb{C}_{nat}$ , which
is the observed mapping from lexical meanings to forms. As an example, consider the meaning cat and its Finnish form: we have  $\mathbb{C}_{nat}(\mathrm{cat}) = kissa$ . The topic of interest in this paper is the efficiency of language's natural codes.
The space of meanings  $\mathcal{M}$  is non-trivial to define, but could be operationalized as  $\mathbb{R}^d$ , which is infinite, continuous and uncountable (Pilehvar and Camacho-Collados, 2020). Meanwhile, the space of wordforms  $\Sigma^{*}$  is also infinite, but discrete and countable. As such, many meanings  $m_{n}$  must be mapped to the same form, resulting in lexical ambiguity. See Pimentel et al. 2020a for a longer discussion on these operationalizations. In this work, though, we do not engage with such ambiguity, considering  $\mathcal{M}$  as an abstract set of meanings, each of which defined by a distinct wordform—i.e. the code  $\mathbb{C}_{nat}$  is a bijection. A consequence of this strategy is that we take the space of meanings to be infinite, but discrete and countable; we only distinguish as many meanings as there are words, therefore, we end up with a countable number of meanings. Additionally, by considering a distinct meaning  $m_{n}$  for each wordform  $w_{n}$  in the lexicon, we only consider codes with as much lexical ambiguity as in the original language. $^5$
3.1 Words as Meanings
The unigram distribution represents the frequency of each wordform in a text, i.e. the probability of a token without conditioning on context  $p(W = kissa)$ . In this work, though, we assume the unigram distribution is a distribution over  $\mathcal{M}$ , e.g.  $p(M = \mathrm{cat})$  —this way we can analyze how changing the code  $\mathbb{C}$  would affect its efficiency.
As stated above, though, we take  $\mathbb{C}_{nat}$  to be a bijection. Such an assumption implies there is a deterministic function from wordforms to meanings in a specific lexicon  $\mathbb{C}_{nat}^{-1}(\boldsymbol {w}) = m$ . Probabilistically speaking, we write
$$
p (M = m \mid W = \boldsymbol {w}) = \mathbb {1} \left\{m = \mathbb {C} _ {n a t} ^ {- 1} (\boldsymbol {w}) \right\} \tag {1}
$$
$$
p (W = \boldsymbol {w} \mid M = m \} = \mathbb {1} \left\{\boldsymbol {w} = \mathbb {C} _ {n a t} (m) \right\} \tag {2}
$$
5Lexical ambiguity allows the mapping of multiple meanings to the same wordform and, in doing so, it enables the preferential re-use of short words (Piantadosi et al., 2012). Thus, the mapping of multiple meanings to the same form could be a source of efficiency in the lexicon (Fenk-Oczlon and Fenk, 2008; Ferrer-i-Cancho and Vitevitch, 2018; Casas et al., 2019; Trott and Bergen, 2020; Xu et al., 2020). Nonetheless, we do not treat it explicitly here.
<sup>3</sup>We also present results using the additional sub-word tokenizers: byte pair encoding (Gage, 1994; Sennrich et al., 2016) and word piece (Schuster and Nakajima, 2012). See Bostrom and Durrett (2020) for a discussion of the tradeoffs of these schemes, in terms of performance and compressibility.
4This alphabet is augmented with an end-of-word symbol.
This mapping implies
$$
\begin{array}{l} p (M = m _ {n}) = \sum_ {\boldsymbol {w} \in \Sigma^ {*}} p (M = m _ {n}, W = \boldsymbol {w}) \tag {3} \\ = p (W = \boldsymbol {w} _ {n}) \\ \end{array}
$$
Given this equality, we can reduce the problem of estimating the unigram distribution over meanings  $p(m)$  to the one over wordforms  $p(\boldsymbol{w})$ .
3.2 Code-length and optimality
As stated above, we assume the unigram distribution to be a distribution over  $\mathcal{M}$ . We now define the cost of a code as its expected length:
$$
\operatorname {c o s t} (\mathbb {C}) = \sum_ {m \in \mathcal {M}} p (m) | \mathbb {C} (m) | \tag {4}
$$
A smaller cost, then, implies a more efficient code. The famous source-coding theorem of Shannon (1948) gives us a theoretical limit on coding cost:
$$
\mathrm {H} (M) \leq \operatorname {c o s t} \left(\mathbb {C} _ {\star}\right) <   \mathrm {H} (M) + 1 \tag {5}
$$
where we define  $\mathbb{C}_{\star}$  to be the most efficient code, and where  $\mathrm{H}(M)$  is the entropy of distribution  $p$ :
$$
\mathrm {H} (M) = \sum_ {m \in \mathcal {M}} p (m) \log_ {| \Sigma |} \frac {1}{p (m)} \tag {6}
$$
According to the source-coding theorem, if we know the true distribution  $p$  over lexical meanings, then we know how to optimally code them. This turns the problem of estimating the efficiency of the lexicon into the one of estimating the entropy of an unknown discrete distribution  $p$ , a well-defined task with a pool of previous work (Miller, 1955; Antos and Kontoyiannis, 2001; Paninski, 2003; Archer et al., 2014). Because the distributions over wordforms and meanings are equivalent, we estimate the entropy  $\mathrm{H}(M)$  using wordforms:
$$
\mathrm {H} (M) = \mathrm {H} (W) = \sum_ {\boldsymbol {w} \in \Sigma^ {*}} p (\boldsymbol {w}) \log_ {| \Sigma |} \frac {1}{p (\boldsymbol {w})} \tag {7}
$$
3.3 Finite and Infinite Support
This section reviews a few technical results as regards the construction of codes from a probability distribution. If  $p$  had finite support—i.e., there were a finite set of possible meanings or wordforms—a simple Huffman encoding (Huffman, 1952) would give us an optimal code for our lexicon. However, this is not the case— $p(\boldsymbol{w})$  has support on all of  $\Sigma^{*}$ —so we might need a more complex strategy to get such a code. Linder et al. (1997) proved the existence of an optimal encoding for a distribution with infinite support, given that it has finite entropy.
Proposition 1. If distribution  $p(\boldsymbol{w})$  has finite entropy, i.e.  $\mathrm{H}(W) < \infty$ , then there exists an optimal encoding for it such that:  $\mathrm{cost}(\mathbb{C}_{\star}) < \mathrm{H}(M) + 1$ .
Proof. See Linder et al. (1997).
Luckily, under a weak assumption, this is the case for a well-trained language model.
Definition 1. Language model  $p(\boldsymbol{w})$  is  $\varepsilon$ -smooth if for all histories  $\boldsymbol{h} \in \Sigma^{*}$  we have  $p(EOW \mid \boldsymbol{h}) \geq \varepsilon$ .<sup>6</sup>
This fairly weak assumption states that partial wordforms have a lowerbound on their probability of ending. As such, there is an upperbound on the probability of a wordform which decreases exponentially with its length. Armed with this assumption, we can now show that any  $\varepsilon$ -smooth language model has a finite entropy.
Proposition 2. If a language model  $p(\pmb{w})$  is  $\varepsilon$ -smooth, then its entropy is finite, i.e.  $\mathrm{H}(W) < \infty$ .
Proof. See App. C.
Safe-guarded by Propositions 1 and 2, we now train a model to capture the unigram distribution. We will then use this model to estimate the code-length of an optimal lexicon.
4 Modeling the Unigram Distribution and its Challenges
Zipf's (1935) law states that the frequency of a word in a corpus is inversely proportional to its rank, resulting in a power-law distribution where a small subset of the words dominate the corpus. As such, naively training a character-level model on a language's tokens (i.e. predicting non-contextual wordforms with their natural corpus frequencies) would be unlikely to capture morphological regularities (Goldwater et al., 2011). Furthermore, it would burden the model to learn a mostly arbitrary assignment between form and frequency. As an example, the English verb make is much more common than the nouns cake and lake, even if graphotactically they may be equally probable.
A closer inspection of English shows that most frequent words tend to come from closed lexical classes including articles, pronouns, prepositions, and auxiliaries, such as the, of, it and be (Sinclair, 1999). These words tend to be short and manifest fossilized graphotactics (and phonotactics) as well
Under this assumption our language model is also consistent, as defined by Welleck et al. (2020)—sequences with infinite length have asymptotically zero probability mass.
as a more abundant prevalence of otherwise rare segments, such as the voiced and voiceless dental fricatives (orthographically expressed with  $th$ ). These rare segments would be overrepresented in such a naïve training regime, making it hard for the character-level model to correctly represent the language's graphotactics.
In order to address the problem of skewed frequencies, we use a novel neuralization of Goldwater et al.'s (2011) two-stage model to capture the unigram distribution. This model consists of two components: a wordform generator and a token frequency adaptor. The generator is a character-level model which produces wordforms, for which we use an LSTM; this model should place similar probability mass on graphotactically "good" wordforms, such as make, cake, and lake. Meanwhile, the adaptor sets the frequency with which these wordforms will appear as tokens. Following Goldwater et al., we base our adaptor on the Pitman-Yor Chinese restaurant process (PYCRP; Pitman and Yor, 1997), which allows the adaptor to model a power-law distribution; this model is then responsible for capturing the fact that make is a more frequent token than cake, and lake.
4.1 A Two-stage Model
The generative process of our two-stage model is presented graphically in Fig. 2. Our generator is a character-level LSTM language model, which generates a potentially infinite number of i.i.d. wordforms  $\{\ell_k\}_{k=1}^K$ . Independently, the PYCRP adaptor assigns each observed token in a dataset to a cluster  $\{z_n\}_{n=1}^N$ . In the literature, the value of  $z_n$  is the "table assignment" of the  $n^{\text{th}}$  token. These clusters are then used as lookup indices to the wordforms, producing the observed word tokens  $\{\pmb{w}_n\}_{n=1}^N$  where  $\pmb{w}_n = \ell_{z_n}$ . In general  $N \gg K$ , so tokens with the same wordform are grouped in few clusters. In this way, the adaptor sets the frequency with which wordforms appear as tokens in a corpus by defining each cluster's probability.
Generating Wordforms. As mentioned above, wordforms are sampled i.i.d. from a distribution  $p_{\phi}$  over strings defined by the generator. Specifically, this distribution over forms is defined as follows:
$$
p _ {\phi} (\ell) = \prod_ {t = 1} ^ {| \ell |} p _ {\phi} \left(\ell_ {t} \mid \ell_ {<   t}\right) \tag {8}
$$
Figure 2: A diagram of the two-stage model. The LSTM generates wordforms  $(\ell_{k})$ . The PYCRP samples cluster assignments  $(z_{n})$ . Cluster assignments are then used to lookup a form for each token  $(w_{n} = \ell_{z_{n}})$ . In this Figure, models are in magenta, latent variables in green and observed variable in orange.
where  $\ell$  is a vector of characters forming a word and  $\ell_t$  is its  $t^{\mathrm{th}}$  character. Each of these characters is encoded with a lookup vector, producing representations  $\mathbf{e}_t\in \mathbb{R}^{d_1}$  where  $d_{1}$  is the embedding size. These embeddings are then used as input to an LSTM (Hochreiter and Schmidhuber, 1997), producing the representations  $\mathbf{h}_t\in \mathbb{R}^{d_2}$ , where  $d_{2}$  is the size of the LSTM's hidden layer. The LSTM output is further used to obtain the distribution over potential characters:
$$
p _ {\phi} \left(\ell_ {t} \mid \ell_ {<   t}\right) = \operatorname {s o f t m a x} \left(\mathbf {W h} _ {t} + \mathbf {b}\right) \tag {9}
$$
In this equation, both  $\mathbf{W} \in \mathbb{R}^{|\Sigma| \times d_2}$  and  $\mathbf{b} \in \mathbb{R}^{|\Sigma|}$  are learnable parameters and the zero vector is used as the initial hidden state  $\mathbf{h}_0$ . The distribution  $p_{\phi}$ , representing the generator, is then used to generate the set of wordforms  $\{\ell_k\}_{k=1}^K$ , which is expected to represent the graphotics and morphology of the language. Notedly, these wordforms do not explicitly capture any notion of token frequency.
Adapting Word Frequencies. The adaptor is responsible for modeling the word frequencies, and it has no explicit notion of the wordforms themselves. The PYCRP assigns each token  $n$  to a cluster  $z_{n}$ . Each cluster  $z_{n}$ , in turn, has an associated wordform  $\ell_{z_n}$ , sampled from the generator. Consequently, all instances in a cluster share the same wordform. The probability of an instance  $n$  being assigned to cluster  $z_{n}$  is defined as follows:
$$
\begin{array}{l} p \left(Z _ {n} = z _ {n} \mid \mathbf {z} _ {<   n}\right) \tag {10} \\ \propto \left\{ \begin{array}{l l} \mathrm {c c} _ {<   n} ^ {(z _ {n})} - a & 1 \leq z _ {n} \leq K _ {<   n} (o l d c l u s t e r) \\ a \cdot K _ {<   n} + b & z _ {n} = K _ {<   n} + 1 (n e w c l u s t e r) \end{array} \right. \\ \end{array}
$$
${}^{7}$  LSTMs have been shown to be able to model phonotactics well by Pimentel et al. (2020b), and so we expect them to also work well with graphotactics.
<sup>8</sup>We note two subscripts are used here:  $k$  refers to the  $k^{\text{th}}$  wordform, while  $t$  indexes the  $t^{\text{th}}$  character in the wordform.
<sup>9</sup>This generative process allows the same wordform to be sampled multiple times, as they are generated i.i.d.