SlowGuess/NAACL_2021_OCR huggingface dataset:
Tiago Pimentel $^{\delta}$  Brian Roark $^{\circ}$  Søren Wichmann $^{\dagger}$  Ryan Cotterell $^{\delta,\Omega}$  Damián Blasi $^{1,9,\mathrm{D}}$
<sup>3</sup>University of Cambridge <sup>1</sup>Google <sup>1</sup>Kazan Federal University <sup>1</sup>ETH Zürich <sup>1</sup>Harvard University <sup>9</sup>MPI for Evolutionary Anthropology <sup>9</sup>HSE University <sup>tp472@cam.ac.uk, roark@google.com, wichmannsoeren@gmail.com, ryan.cotterell@inf.ethz.ch, dblasi@fas.harvard.edu
This work presents an information-theoretic operationalisation of cross-linguistic non-arbitrariness. It is not a new idea that there are small, cross-linguistic associations between the forms and meanings of words. For instance, it has been claimed (Blasi et al., 2016) that the word for TONGUE is more likely than chance to contain the phone [l]. By controlling for the influence of language family and geographic proximity within a very large concept-aligned, cross-lingual lexicon, we extend methods previously used to detect within language non-arbitrariness (Pimentel et al., 2019) to measure cross-linguistic associations. We find that there is a significant effect of non-arbitrariness, but it is unsurprisingly small (less than  $0.5\%$  on average according to our information-theoretic estimate). We also provide a concept-level analysis which shows that a quarter of the concepts considered in our work exhibit a significant level of cross-linguistic non-arbitrariness. In sum, the paper provides new methods to detect cross-linguistic associations at scale, and confirms their effects are minor.
The arbitrariness of the sign, i.e. the principle that a word's form is unrelated to what it denotes, was one of the cornerstones in the structuralist revolution in linguistics (Saussure, 1916). While languages do seem to adhere to the principle to a large extent, researchers have repeatedly uncovered evidence that there are preferences in form—meaning matches (Perniss et al., 2010). Indeed, the notion that these small, but systematic, form—meaning relations hold across the world's languages has become a mainstream topic of research in the last couple of decades. $^{1}$
Figure 1: We used a sample of 5189 languages (9148 doculects) to study cross-linguistic systematicity. In this map, the colours represent the four macroareas to which languages were assigned.
Determining effective metrics to capture meaningful form-meaning associations is far from trivial, though, and researchers have explored a substantial number of statistical and heuristic approaches (Bergen, 2004; Wichmann et al., 2010; Johansson and Zlatev, 2013; Haynie et al., 2014; Gutierrez et al., 2016; Blasi et al., 2016; Joo, 2019). Previous studies differ from each other along (at least) three axes: (i) which unit is used to measure wordform similarity (e.g., phonemes, sub-phonemic features or arbitrary sequences); (ii) how they deploy a baseline for statistical comparison (e.g. permute forms with meanings, or propose a generative model that yields wordforms uninformed by their meaning) and (iii) whether they study non-arbitrariness within or across languages.
Pimentel et al. (2019) provide the first holistic measure of non-arbitrariness (in a large vocabulary sample of a single language) using tools from information theory, and apply their measure to discover phonesthemes. Our work extends their approach to the problem of discovering and estimating the strength of frequent cross-linguistic form-meaning associations (e.g. iconicity and systematicity) in individual concepts. We do this by adapting Pimentel et al.'s (2019) approach, modelling
2Phonesthemes are sub-morphemic units which are associated in a language with some small semantic domain.
<sup>1</sup>See §2 below for a brief literature review and Dingemanse et al. (2015) for a more comprehensive one.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4416-4425 June 6-11, 2021. ©2021 Association for Computational Linguistics
form-meaning associations in a large collection of basic vocabulary wordlists covering close to  $3/4$  of the world's languages (see Fig. 1 and Wichmann et al., 2020). By taking the words in these lists to be random variables and asking how much information within wordforms is explained by the meaning they refer to, we obtain a quantitative estimate of cross-linguistic non-arbitrariness.
Specifically, we propose to model a universal (language-independent) form distribution (using neural language models), and then we estimate concept-specific distributions. With these in hand, we are able to determine how much the meaning of a concept predicts its form cross-linguistically—by measuring the mutual information between them; see §4 for details. This method further allows us to identify which concepts exhibit stronger non-arbitrary form-meaning association and which form patterns are more likely to occur in them.
In order to maximise the reliability of the observed associations, we implement stringent controls for genealogical and areal effects, as well as for the size of each language family. See §4.5 for details on these controls. After introducing these controls, we find that wordlists display an average of around 0.01 bits of form-meaning mutual information explained by cross-linguistic non-arbitrariness  $(\approx 0.3\%)$  of the wordform uncertainty) with substantial variation among concepts and languages. Of the 100 basic concepts in our data, we find a statistically identifiable pattern in 26 of them  $(p < 0.01)$ . Inspection of the results show that our method recovers previously proposed associations, e.g. the association of [l] with the concept TONGUE and [p] with FULL (Blasi et al., 2016).
2 Non-Arbitrary Form-Meaning Associations
Several studies have looked at non-arbitrary patterns in languages, be it systematicity (Shillcock et al., 2001; Gutierrez et al., 2016; Dautriche et al., 2017; Pimentel et al., 2019) or iconicity (Dingemanse, 2012, 2018). With respect to cross-linguistic non-arbitrariness specifically, the hypothesised sources of form-meaning associations range from the fact that humans are endowed with the same neurocognitive architecture (Bankieris and Simner, 2015) to their encountering similar experiences within the world (Parise et al., 2014).
While global non-arbitrary form-meaning associations have been hypothesised to exist at different levels of linguistic description (Haiman, 1980), by far the component of language that has received the most attention in this respect is the lexicon. A few circumstances facilitate this type of research in contrast to other domains of grammar. For instance, the space of possible words that could be used in a given language to refer to an arbitrary referent is large, whereas the relative canonical order of a verb with respect to its object complement is substantially smaller (which renders cross-linguistic similarities less informative than in the first case). Additionally, the sheer amount of data available in the form of wordlists exceeds other types of linguistic data for the languages of the world.
As a consequence, some of the largest evaluations of non-arbitrary form-meaning associations involve systematic wordlists with comparable referents across languages (Wichmann et al., 2010; Johansson and Zlatev, 2013; Haynie et al., 2014; Blasi et al., 2016; Joo, 2019). Most of these studies were focused on the regular association between phonemic or phonetic units with meaning, occasionally controlling for other potential sources of form-meaning association such as phonotactics or word length (Blasi et al., 2016). While useful, the estimates emerging from this type of study can be regarded as lower bounds to the total amount of non-arbitrary associations found in the vocabulary.
Recent efforts have resulted in datasets with thousands of languages (Wichmann et al., 2020), with which linguists can look for universal statistical patterns (Wichmann et al., 2010; Blasi et al., 2016). These studies, though, only looked at the presence (or not) of individual phones in words, not accounting for their connections. Our methods rely on neural phonotactic models, similar to those used by Pimentel et al. (2020), thus capturing a broader range of potential correspondences.
An exceptional resource with substantial cross-linguistic representation is provided in the Automated Similarity Judgment Program, better known by its acronym ASJP (Wichmann et al., 2020). ASJP is a collection of basic vocabulary wordlists, i.e. lists of words with referents that are expected to be widely attested across human societies. It involves body parts, some colour terms, lower numerals, general properties (such as big or round), and flora and fauna that are usually found in places
3See §3 for a discussion on potential biases in the dataset that likely influence the actual numerical value derived.
where humans live (e.g. trees and dogs). The individual words in ASJP are transcribed by field linguists in a specific phonetic annotation scheme that involves 41 symbols, chosen in order to maximise cross-linguistic utility by merging rare phones with similar phonetic features within the same category. These wordlists are assembled with the purpose of studying the history of languages—following the tradition established by Swadesh (1955)—under the principles of the comparative method.
ASJP has gathered, in its latest iterations, data for close to  $3/4$  of the world's languages, which makes it an unparalleled resource for evaluating form-meaning associations across spoken languages. Furthermore, the vocabulary in its wordlists was chosen as so to be resistant to borrowings—making it especially interesting for our purposes of finding universal form-meaning biases. We leave out pidgin and creole data,[4] as defined by the World Atlas of Language Structures (Dryer and Haspelmath, 2013), since they are ambiguous with relation to their genealogical affiliation. We also omit constructed and fake languages (e.g. Esperanto and Taensa). This leaves us 9148 doculects (or wordlists) from 5189 languages.[5]
Form-meaning associations have been studied in earlier versions of this dataset. Firstly, Wichmann et al. (2010) studied the average form across different concepts in ASJP, and found a number of tentative patterns pointing to non-arbitrariness. Yet the lack of historical and statistical controls compromised the nature of such patterns: form-meaning associations could be due to widespread linguistic contact (e.g. the word for DOG, Pache et al. 2016) or to its fortuitous presence in large families. Blasi et al. (2016), however, provide a conservative evaluation of individual form-meaning associations by imposing a restrictive set of conditions. They looked for associations that were present in a minimum number of continents and language families. This resulted in a sizable number of non-arbitrary
associations, many of which had been highlighted as interesting based on behavioural and linguistic experiments in a handful of languages.
Data Disclaimer. As mentioned above, ASJP gathers lists of wordforms that are expected to be present across most human societies and their corresponding language(s). While this guarantees a fair coverage in our study, it limits the scope of our conclusions to those concepts present herein.
We describe each word as comprised by form and meaning, which we represent as a pair  $(\mathbf{w}^{(n)},\mathbf{v}^{(n)})$ . The form  $\mathbf{w}^{(n)}\in \Sigma^*$  is represented as a phone string where  $\Sigma$  is a phonetic alphabet. In this work, we take  $\Sigma$  to be the set of 41 phonetic symbols in ASJP plus the end-of-string symbol. We write  $W$  to denote a  $\Sigma^{*}$ -valued random variable. The meaning  $\mathbf{v}^{(n)}\in \{0,1\} ^K$  is represented by a one-hot vector, where  $K$  is the number of analysed concepts. We write  $V$  to denote a  $\{0,1\} ^K$ -valued random variable.
4.2 Non-Arbitrariness as Mutual Information
The goal of this work is to measure cross-linguistic form-meaning associations, operationalised as the mutual information (MI) between a form-valued random variable  $W$  and a meaning-valued random variable  $V$ . Symbolically, we are interested in computing (Cover and Thomas, 2012):
$$
\mathrm {I} (W; V) = \mathrm {H} (W) - \mathrm {H} (W \mid V) \tag {1}
$$
Intuitively, this quantity captures the uncertainty we have over the form, the entropy  $\mathrm{H}(W)$ , minus how much uncertainty we have over the form given the meaning, the conditional entropy  $\mathrm{H}(W\mid V)$ . Thus, if eq. (1) is zero, its minimum, we have the result that meaning tells us absolutely nothing about the wordform. On the other hand, if eq. (1) is  $\min \{\mathrm{H}(W),\mathrm{H}(V)\}$ , its maximum, we have that the form is a deterministic function of the meaning (or the opposite; the meaning being deterministically determined given the form).
<sup>6</sup>We note Pimentel et al. (2019) used high-dimensional distributional semantic vectors to represent meaning, while we use a one-hot vector. However, their work relied on a specific language's WORD2VEC—a choice which could potentially bias our results with that language's properties. We did, however, run an extra experiment with English WORD2VEC; this led to similar conclusions to the ones presented here.
4Pidgins are believed to rely particularly on iconicity due to their smaller degree of lexicalisation; this reliance then diminishes as it morphs into a creole (Romaine, 1988). Future work could expand the methods here to study this phenomenon.
5 When there is more than one wordlist for one language (as defined by their ISO-codes) one can sometimes refer to them as different dialects, but these are often just alternative versions of the same language as recorded by different linguists. There can be as much variation in such different recordings as among different dialects recorded by one and the same linguist. For those reasons, it is practical to use the term doculect, which we adopt here. This is a neutral term that refers to some dialect as recorded in some specific source.
That the mutual information may take values in  $[0, \min\{\mathrm{H}(W), \mathrm{H}(V)\}]$  together with the fact that, for our specific study,  $\mathrm{H}(W)$  is smaller than  $\mathrm{H}(V)$  suggests a more interpretable metric called the uncertainty coefficient:
$$
\mathrm {U} (W \mid V) = \frac {\mathrm {I} (W ; V)}{\mathrm {H} (W)} \tag {2}
$$
This quantity is the proportion of uncertainty in the form reduced by knowing the meaning. Both mutual information and uncertainty coefficients are general measures of non-arbitrariness. One might also inquire about how non-arbitrary a single formmeaning pair is. To measure this, we propose pointwise mutual information (PMI):
$$
\operatorname {P M I} (\mathbf {w}; \mathbf {v}) = \log \frac {p (\mathbf {w} \mid \mathbf {v})}{p (\mathbf {w})} \tag {3}
$$
4.3 Approximating Mutual Information
As noted above, we want to estimate the entropy of language agnostic wordforms, i.e.
$$
\mathrm {H} (W) = \sum_ {\mathbf {w} \in \Sigma^ {*}} p (\mathbf {w}) \log \frac {1}{p (\mathbf {w})} \tag {4}
$$
Unfortunately, we do not know the exact distribution of  $p(\mathbf{w})$  and, even if we did, we would need to sum over the infinite set of possible strings  $\Sigma^{*}$  to compute this entropy, which is intractable. If we have another probability distribution  $p_{\theta}(\mathbf{w})$ , though, we can calculate the cross-entropy between them as an approximation, i.e.
$$
\mathrm {H} (W) \leq \mathrm {H} _ {\pmb {\theta}} (W) \approx \frac {1}{N} \sum_ {n = 1} ^ {N} \log \frac {1}{p _ {\pmb {\theta}} (\tilde {\mathbf {w}} ^ {(n)})} \quad (5)
$$
where  $\{\tilde{\mathbf{w}}^{(n)}\}_{n = 1}^{N}$  are samples from the true distribution  $p$ . Throughout the paper, the tilde marks held-out data, i.e., data not used during model training. We note that the approximation becomes exact as  $N\to \infty$  by the weak law of large numbers. This cross-entropy estimate gives us an upper bound on the actual entropy. This bound is tighter the closer the distributions  $p(\mathbf{w})$  and  $p_{\theta}(\mathbf{w})$  are.
4.4 Estimating the Approximator  $p_{\theta}$
How should we train a model to estimate this universal phonotactic distribution  $p_{\theta}(\mathbf{w})$ , though? We train a phone-level language model to predict the next phone given previous ones in a word, i.e.
$$
p _ {\boldsymbol {\theta}} (\mathbf {w}) = \prod_ {t = 1} ^ {| \mathbf {w} |} p _ {\boldsymbol {\theta}} \left(w _ {t} \mid \mathbf {w} _ {<   t}\right) \tag {6}
$$
In this work, we use an LSTM as our language model (Hochreiter and Schmidhuber, 1997). Each phone  $w_{t}$  is represented using a lookup embedding  $\mathbf{z}_{t} \in \mathbb{R}^{d}$ . These are fed into the LSTM, outputting temporal representations of the sequence:
$$
\mathbf {h} _ {t} = \operatorname {L S T M} \left(\mathbf {z} _ {t - 1}, \mathbf {h} _ {t - 1}\right) \tag {7}
$$
where  $\mathbf{h}_0$  is the zero vector. These representations are linearly transformed and used in a softmax to approximate the probability distribution:
$$
p _ {\boldsymbol {\theta}} \left(w _ {t} \mid \mathbf {w} _ {<   t}\right) = \operatorname {s o f t m a x} \left(\mathbf {W h} _ {t} + \mathbf {b}\right) \tag {8}
$$
All parameters are learned via gradient descent, minimising the cross-entropy in the training set.
4.5 Cross-linguistic Controls
As mentioned before, salient regularities between form and meaning across languages might result from large groups of genealogically or spatially related languages. In particular it is practical to consider two independent problems in this respect:
(i) Eq. (5)'s inequality only holds if  $\mathrm{H}_{\theta}(W)$  is estimated on a set of datapoints sampled independently from the set of points on which the model  $p_{\theta}$  was trained. As such, the test set should only include languages that are not genealogically or areally related to those in the training set;
(ii) Within our dataset, the different size of areal and genealogical groups should be accounted for so that our results are not biased towards particularly large areas or language families.
Train-test split. To mitigate the problem referred to in the first item, we cross-validate our models by appealing to the notion of macroareas, large-scale regions of the world that simultaneously maximise internal historical dependency while minimising external ones. Striking a balance between historical independence and data availability, we consider the following four macroareas: the Americas, Eurasia, Africa, and the Pacific (which in this instantiation includes Papua New Guinea and Australia—see Fig. 1). We will use these macroareas as our folds. Two macroareas will be used at each time for training, while one other is used for validation and the last for testing. Some language families, though, might be present in more than one macroarea (e.g. many European languages are spoken natively in the Americas and Africa). These
families will be assigned to the one macroarea which contains most of its family members, since we believe reducing genealogical impact should be preferred over areal impact for our data and purposes, in cases for which such a choice is required.7
Family size bias. The second problem is tackled by weighting each example's contribution to our loss function by the inverse of its family size  $l^{(n)}$ :
$$
\mathcal {L} (\boldsymbol {\theta}) = \frac {1}{L} \sum_ {n = 1} ^ {N} \frac {1}{l ^ {(n)}} \log \frac {1}{p _ {\boldsymbol {\theta}} \left(\mathbf {w} ^ {(n)}\right)} \tag {9}
$$
where  $L = \sum_{n=1}^{N} \frac{1}{l^{(n)}}$  re-normalises the cross-entropy using the family sizes. This weighted cross-entropy loss function makes per instance contributions of large language families smaller, reducing their impact on the trained model.
To mitigate the same bias effect on the evaluation of validation and test sets, we first get cross-entropies per word. We subsequently average them per language, per family, and per macroarea. This way, each family will have the same effect per macroarea and each macroarea will have the same effect on the overall cross-entropy.
4.6 Concept-Specific Form Distributions
We want to compare per-concept phonotactic models with general ones to analyse sound-meaning associations. With that in mind, we condition phone-level language models on meaning:
$$
p _ {\boldsymbol {\theta}} (\mathbf {w} \mid \mathbf {v}) = \prod_ {t = 1} ^ {| \mathbf {w} |} p _ {\boldsymbol {\theta}} \left(w _ {t} \mid \mathbf {w} _ {<   t}, \mathbf {v}\right) \tag {10}
$$
These models are trained following the same procedures explained above, but conditioning the LSTMs on concept specific representations. Specifically, the one-hot representation is linearly transformed and fed into the LSTM as its initial state
$$
\mathbf {h} _ {0} = \mathbf {W} _ {0} \mathbf {v} \tag {11}
$$
where the linear transformation  $\mathbf{W}_0\in \mathbb{R}^{d\times K}$  is randomly initialised and learned with the rest of the model. We then use this distribution to estimate the conditional entropy, analogously to eq. (5), as in
$$
\mathrm {H} (W \mid V) \lesssim \frac {1}{N} \sum_ {n = 1} ^ {N} \log \frac {1}{p _ {\boldsymbol {\theta}} \left(\tilde {\mathbf {w}} ^ {(n)} \mid \tilde {\mathbf {v}} ^ {(n)}\right)} \tag {12}
$$
where  $\{\tilde{\mathbf{w}}^{(n)},\tilde{\mathbf{v}}^{(n)}\}_{n = 1}^{N}$  are held-out frommeaning pairs, sampled from the true distribution.8
4.7 Non-Arbitrariness as Information
The mutual information between wordforms and meaning can be decomposed into the difference of two entropy measures. Unfortunately, we have no way of directly measuring these entropy values without their probability distributions  $(p(\mathbf{w})$  and  $p(\mathbf{w}|\mathbf{v}))$ . We use the estimated cross-entropies as an approximation to this mutual information:
$$
\begin{array}{l} \mathrm {I} (W; V) = \mathrm {H} (W) - \mathrm {H} (W \mid V) (13) \\ \approx \mathrm {H} _ {\boldsymbol {\theta}} (W) - \mathrm {H} _ {\boldsymbol {\theta}} (W \mid V) (14) \\ \end{array}
$$
We note that eq. (14) is approximate because it is the difference of two upper bounds. Furthermore, while there are many ways to estimate mutual information, computing it as the difference between two cross-entropies seems to produce consistent results (McAllester and Stratos, 2020).
4.8 Bounds and Optimisation
As mentioned in §4.3, our entropy upper bounds will be tighter if our models  $p_{\theta}$  better capture  $p$ . With this in mind, we optimise the hyperparameters of our models using Bayesian optimisation with a Gaussian process prior (Snoek et al., 2012)—hyper-parameter ranges are presented in App. A. We train 25 models for each configuration and choose the best one according to the validation set, optimising our weighted cross-entropy loss using AdamW (Loshchilov and Hutter, 2019).
5.1 Analysis #1: Overall Mutual Information
We are interested in estimating the cross-linguistic mutual information between meaning and wordforms. With this in mind, we follow the steps described in §4.4, but instead of only 1 model, we train 25 models using different seeds for each fold (totalling 100 models). Average results—overall and per macroarea—are shown in Tab. 1.
<sup>7</sup>As mentioned in §3, the list of concepts in ASJP was chosen to minimise borrowings across languages. We further note here that loan words are annotated in this dataset and we drop those words for the purpose of our analysis.
This meaning conditioned model may potentially be better than the raw LSTMs (without conditioning on meaning; due to the extra parameters). To control for this fact, we ran an extra experiment where we estimated  $\mathrm{H}(W)$  using the meaning dependent model with shuffled concept IDs (so there is no form-meaning association). The results from this shuffled IDs model were very similar to the raw LSTM ones.
$^9$ Our code is available at https://github.com/rycolab/form-meaning-associations.
Figure 2: Uncertainty Coefficients for all 100 concepts in ASJP distributed according to their average word length.