SlowGuess/NAACL_2021_OCR huggingface dataset:
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Kyle Mahowald, Isabelle Dautriche, Mika Braginsky, and Edward Gibson. 2020. Efficient communication and the organization of the lexicon. *PsyArXiv* preprint *PsyArXiv:4an6v*.
Benoit Mandelbrot. 1953. An informational theory of the statistical structure of language. Communication Theory, 84:486-502.
George Miller. 1955. Note on the bias of information estimates. Information Theory in Psychology: Problems and Methods.
George A. Miller. 1957. Some effects of intermittent silence. The American Journal of Psychology, 70(2):311-314.
Padraic Monaghan, Richard C. Shillcock, Morten H. Christiansen, and Simon Kirby. 2014. How arbitrary is language? Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1651):20130299.
Fermin Moscoso del Prado. 2013. The missing baselines in arguments for the optimal efficiency of languages. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 35.
Liam Paninski. 2003. Estimation of entropy and mutual information. Neural Computation, 15(6):1191-1253.
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2011. Word lengths are optimized for efficient communication. Proceedings of the National Academy of Sciences, 108(9):3526-3529.
Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3):280-291.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2020. Embeddings in natural language processing: Theory and advances in vector representations of meaning. Synthesis Lectures on Human Language Technologies, 13(4):1-175.
Tiago Pimentel, Ryan Cotterell, and Brian Roark. 2021a. Disambiguatory signals are stronger in word-initial positions. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics.
Tiago Pimentel, Rowan Hall Maudslay, Damian Blasi, and Ryan Cotterell. 2020a. Speakers fill lexical semantic gaps with context. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4004-4015, Online. Association for Computational Linguistics.
Tiago Pimentel, Arya D. McCarthy, Damián Blasi, Brian Roark, and Ryan Cotterell. 2019. Meaning to form: Measuring systematicity as information. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1751-1764, Florence, Italy. Association for Computational Linguistics.
Tiago Pimentel, Brian Roark, and Ryan Cotterell. 2020b. Phonotactic complexity and its trade-offs. Transactions of the Association for Computational Linguistics, 8:1-18.
Tiago Pimentel, Brian Roark, Søren Wichmann, Ryan Cotterell, and Damián Blasi. 2021b. Finding concept-specific biases in form-meaning associations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Virtual. Association for Computational Linguistics.
Jim Pitman and Marc Yor. 1997. The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator. Annals of Probability, 25(2):855-900.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152. IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Claude E. Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 27(3):379-423.
John Sinclair. 1999. A way with common words. Language and Computers, 26:157-180.
Peter Smit, Sami Virpioja, Stig-Arne Gronroos, and Mikko Kurimo. 2014. Morfessor 2.0: Toolkit for statistical morphological segmentation. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 21-24, Gothenburg, Sweden. Association for Computational Linguistics.
Yee Whye Teh. 2006. A hierarchical Bayesian language model based on Pitman-Yor processes. In COLING/ACL 2006 - 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, volume 1, pages 985-992, Morristown, NJ, USA. Association for Computational Linguistics.
Sean Trott and Benjamin Bergen. 2020. Why do human languages have homophones? Cognition, 205:104449.
Rob J. J. H. van Son and Louis C. W. Pols. 2003a. How efficient is speech? In Proceedings of the Institute of Phonetic Sciences, volume 25, pages 171-184.
Rob J. J. H. van Son and Louis C. W. Pols. 2003b. Information structure and efficiency in speech production. In Eighth European Conference on Speech Communication and Technology, Geneva, Switzerland.
Michael S. Vitevitch and Paul A. Luce. 1999. Probabilistic phonotactics and neighborhood activation in spoken word recognition. Journal of Memory and Language, 40(3):374-408.
Greg C. G. Wei and Martin A. Tanner. 1990. A Monte Carlo implementation of the EM algorithm and the poor man's data augmentation algorithms. Journal of the American Statistical Association, 85(411):699-704.
Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. 2020. Consistency of a recurrent language model with respect to incomplete decoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5553-5568, Online. Association for Computational Linguistics.
Yang Xu, Khang Duong, Barbara C. Malt, Serena Jiang, and Mahesh Srinivasan. 2020. Conceptual relations predict colexification across languages. Cognition, 201:104280.
George KingsleyZipf.1935.ThePsycho-biologyof Language:AnIntroductiontoDynamicPhilology. HoughtonMifflin.
George KingsleyZipf.1949.Human Behavior and the Principle of Least Effort.Addison-Wesley Press.
A Dataset sizes
In this section, we present the number of word tokens (Tab. 2) and word types (Tab. 3) in our analyzed datasets.
Train	Validation	Test
English	4,630,371	578,510	578,796
Finnish	2,558,634	319,546	320,716
Hebrew	4,911,953	613,457	609,864
Indonesian	4,039,552	506,085	507,587
Tamil	3,286,075	412,776	412,416
Turkish	2,676,471	333,120	332,359
Yoruba	373,517	46,415	46,283
Table 2: The number of word tokens used in training, validation and testing.
Train	Validation	Test
English	242,030	66,668	66,243
Finnish	466,745	109,232	110,378
Hebrew	311,860	104,555	104,478
Indonesian	243,118	69,792	70,079
Tamil	479,668	116,196	115,422
Turkish	308,419	84,300	83,871
Yoruba	47,740	12,877	12,877
Table 3: The number of word types used in training, validation and testing.
As mentioned in the main text, we cannot directly infer the parameters of our model and we use a solution similar to expectation maximization (Wei and Tanner, 1990). We freeze our LSTM generator while learning the PYCRP parameters and then freeze the PYCRP to train the LSTM model.
Expectation step. This step uses a Gibbs sampling procedure to estimate the parameters of the PYCRP. For each token in our dataset, we fix all cluster assignments  $\mathbf{z}_{-n}$  except for the given token's one  $z_{n}$ . We then re-sample this token's cluster based on the marginal probability  $p(z_{n}|\mathbf{z}_{-n},\ell ,\boldsymbol{w}_{n})$ . We do this for 5 epochs, and use the assignments which result in the best development set cross-entropy. This process can both remove clusters and create new ones by replacing tokens. The set of populated clusters (together with their wordform labels) then allows creating a new wordform dataset of size  $K^{\prime}$ , where the distribution of the token frequencies is expected to be less skewed. In practice, this wordform dataset is thus created from the resulting set of cluster labels  $\{\ell_k\}_{k = 1}^{K'}$ , i.e. a word will appear in this new dataset as many times as it was assigned as a cluster label.
Maximization step. We use the set of populated cluster labels to train the generator LSTM—assuming that this allows learning a more representative model of a language's graphotactics as the irregular common words are less dominant in its training set. In other words, at each epoch, the generator will be trained in a wordform as many times as it has been assigned as a cluster label.
Hyperparameters and implementation details. For the PYCRP, we fix hyper-parameters  $a = 0.5$  and  $b = 10,000$ , and we use the optimized Gibbs sampling algorithm designed by Blunsom et al.
(2009). As our generator, we use a three layers LSTM with an embedding size of 128, a hidden size of 512 and dropout of .33. This LSTM is trained using AdamW (Loshchilov and Hutter, 2019) and we hotstart it by initially training on the set of word types in the training set (the set of unique wordforms in it).
We present here the proof of Proposition 2. This proposition is repeated here for convenience:
Proposition 2. If a language model  $p(\boldsymbol{w})$  is  $\varepsilon$ -smooth, then its entropy is finite, i.e.  $\mathrm{H}(W) < \infty$ .
Proof. To prove this, we will break the entropy of a string in two parts. The entropy of the first character, plus the entropy of the following ones given the first, as in:
$$
\mathrm {H} (W) = \mathrm {H} \left(W _ {1}\right) + \mathrm {H} \left(W _ {> 1} \mid W _ {1}\right) \tag {19}
$$
We first bound the entropy of the first character using a uniform distribution upperbound:
$$
\begin{array}{l} \mathrm {H} \left(W _ {1}\right) = - \sum_ {\boldsymbol {w} _ {1} \in \Sigma} p \left(\boldsymbol {w} _ {1}\right) \log p \left(\boldsymbol {w} _ {1}\right) \tag {20} \\ \leq - \log | \Sigma | \\ \end{array}
$$
We now use the  $\varepsilon$ -smoothness property to upper-bound the entropy of the following characters:
$$
\begin{array}{l} \mathrm {H} \left(W _ {> 1} \mid W _ {1}\right) \tag {21} \\ = \sum_ {\boldsymbol {w} _ {1} \in \Sigma} p (\boldsymbol {w} _ {1}) \mathrm {H} (W _ {> 1} \mid W _ {1} = \boldsymbol {w} _ {1}) \\ = p (\mathrm {E} \circ \mathrm {W}) \mathrm {H} (W _ {> 1} \mid W _ {1} = \mathrm {E} \circ \mathrm {W}) \\ + \sum_ {\boldsymbol {w} _ {1} \in \Sigma , \boldsymbol {w}! = \mathrm {E o W}} p (\boldsymbol {w} _ {1}) \mathrm {H} (W _ {> 1} \mid W _ {1} = \boldsymbol {w} _ {1}) \\ = \sum_ {\boldsymbol {w} _ {1} \in \Sigma , \boldsymbol {w}! = \mathrm {E} \circ \mathrm {W}} p (\boldsymbol {w} _ {1}) \mathrm {H} (W _ {> 1} \mid W _ {1} = \boldsymbol {w} _ {1}) \\ \leq \sum_ {\boldsymbol {w} _ {1} \in \Sigma , \boldsymbol {w}! = \mathrm {E} \circ \mathrm {W}} p (\boldsymbol {w} _ {1}) \mathrm {H} (W _ {> 1}) \\ \leq (1 - \varepsilon) \mathrm {H} (W) \\ \end{array}
$$
Given both these upperbounds, we can bound the full wordform entropy:
$$
\begin{array}{l} \mathrm {H} (W) = \mathrm {H} \left(W _ {1}\right) + \mathrm {H} \left(W _ {> 1} \mid W _ {1}\right) \tag {22} \\ \leq - \log | \Sigma | + (1 - \varepsilon) \mathrm {H} (W) \\ \end{array}
$$
Finally, with simple algebraic manipulations we complete the proof:
$$
\mathrm {H} (W) \leq - \frac {1}{\varepsilon} \log | \Sigma | \tag {23}
$$