Venue: NAACL 2021

# Finding Concept-specific Biases in Form-Meaning Associations

Tiago Pimentel\({}^{\text{\textregistered}}\)   Brian Roark\({}^{\text{\textregistered}}\)   Soren Wichmann\({}^{\text{\textregistered}}\)   Ryan Cotterell\({}^{\text{\textregistered},\text{\textregistered}}\)   Damian Blasi\({}^{\text{\textregistered},\text{\textregistered},\text{\textregistered}}\)

###### Abstract

This work presents an information-theoretic operationalisation of cross-linguistic non-arbitrariness. It is not a new idea that there are small, cross-linguistic associations between the forms and meanings of words. For instance, it has been claimed [1] that the word for tongue is more likely than chance to contain the phone [I]. By controlling for the influence of language family and geographic proximity within a very large concept-aligned, cross-lingual lexicon, we extend methods previously used to detect within language non-arbitrariness [40] to measure cross-linguistic associations. We find that there is a significant effect of non-arbitrariness, but it is unsurprisingly small (less than 0.5% on average according to our information-theoretic estimate). We also provide a concept-level analysis which shows that a quarter of the concepts considered in our work exhibit a significant level of cross-linguistic non-arbitrariness. In sum, the paper provides new methods to detect cross-linguistic associations at scale, and confirms their effects are minor.

+
Footnote †: 2Phonesthemes are sub-morphemic units which are associated in a language with some small semantic domain.

University of Cambridge \({}^{\text{\textregistered}}\)Google 'Kazan Federal University \({}^{\text{\textregistered}}\)ETH Zurich

\({}^{\text{\textregistered}}\)Harvard University \({}^{\text{\textregistered}}\)MPI for Evolutionary Anthropology \({}^{\text{\textregistered}}\)HE University tp472@cam.ac.uk, roark@google.com, wichmannsoeren@gmail.com ryan.cotterell@inf.ethz.ch, dblasi@fas.harvard.edu

## 1 Introduction

The arbitrariness of the sign, i.e. the principle that a word's form is unrelated to what it denotes, was one of the cornerstones in the structuralist revolution in linguistics [36]. While languages do seem to adhere to the principle to a large extent, researchers have repeatedly uncovered evidence that there are preferences in form-meaning matches [8]. Indeed, the notion that these small, but systematic, form-meaning relations hold _across_ the world's languages has become a mainstream topic of research in the last couple of decades.1

Footnote 1: See §2 below for a brief literature review and Dingemanse et al. (2015) for a more comprehensive one.

Determining effective metrics to capture meaningful form-meaning associations is far from trivial, though, and researchers have explored a substantial number of statistical and heuristic approaches [9, 2, 23, 24, 25]. Previous studies differ from each other along (at least) three axes: (i) which unit is used to measure wordform similarity (e.g., phonemes, sub-phoneomic features or arbitrary sequences); (ii) how they deploy a baseline for statistical comparison (e.g. permute forms with meanings, or propose a generative model that yields wordforms uninformed by their meaning) and (iii) whether they study non-arbitrariness within or across languages.

Pimentel et al. (2019) provide the first holistic measure of non-arbitrariness (in a large vocabulary sample of a single language) using tools from information theory, and apply their measure to discover phonesthemes.2 Our work extends their approach to the problem of discovering and estimating the strength of frequent _cross-linguistic_ form-meaning associations (e.g. iconicity and systematicity) in individual concepts. We do this by adapting Pimentel et al.'s (2019) approach, modelling

Figure 1: We used a sample of 5189 languages (9148 doculects) to study cross-linguistic systematicity. In this map, the colours represent the four macroareas to which languages were assigned.



form-meaning associations in a large collection of basic vocabulary wordlists covering close to \(3/4\) of the world's languages (see Fig. 1 and Wichmann et al., 2020). By taking the words in these lists to be random variables and asking how much information within wordforms is explained by the meaning they refer to, we obtain a quantitative estimate of cross-linguistic non-arbitrariness.

Specifically, we propose to model a universal (language-independent) form distribution (using neural language models), and then we estimate concept-specific distributions. With these in hand, we are able to determine how much the meaning of a concept predicts its form cross-linguistically--by measuring the mutual information between them; see SS4 for details. This method further allows us to identify which concepts exhibit stronger non-arbitrary form-meaning association and which form patterns are more likely to occur in them.

In order to maximise the reliability of the observed associations, we implement stringent controls for genealogical and areal effects, as well as for the size of each language family. See SS4.5 for details on these controls. After introducing these controls, we find that wordlists display an average of around \(0.01\) bits of form-meaning mutual information explained by cross-linguistic non-arbitrariness (\(\approx 0.3\%\) of the wordform uncertainty) with substantial variation among concepts and languages.3 Of the 100 basic concepts in our data, we find a statistically identifiable pattern in 26 of them (\(p<0.01\)). Inspection of the results show that our method recovers previously proposed associations, e.g. the association of [I] with the concept tongue and [p] with fullBlasi et al. (2016).

Footnote 3: See §3 for a discussion on potential biases in the dataset that likely influence the actual numerical value derived.

## 2 Non-Arbitrary Form-Meaning Associations

Several studies have looked at non-arbitrary patterns in languages, be it systematicity Shillcock et al. (2001); Gutierrez et al. (2016); Dautriche et al. (2017); Pimentel et al. (2019) or iconicity Dingemanse (2012, 2018). With respect to cross-linguistic non-arbitrariness specifically, the hypothesised sources of form-meaning associations range from the fact that humans are endowed with the same neurocognitive architecture Bankieris and Simner (2015) to their encountering similar experiences within the world Parise et al. (2014).

While global non-arbitrary form-meaning associations have been hypothesised to exist at different levels of linguistic description Haiman (1980), by far the component of language that has received the most attention in this respect is the lexicon. A few circumstances facilitate this type of research in contrast to other domains of grammar. For instance, the space of possible words that could be used in a given language to refer to an arbitrary referent is large, whereas the relative canonical order of a verb with respect to its object complement is substantially smaller (which renders cross-linguistic similarities less informative than in the first case). Additionally, the sheer amount of data available in the form of wordlists exceeds other types of linguistic data for the languages of the world.

As a consequence, some of the largest evaluations of non-arbitrary form-meaning associations involve systematic wordlists with comparable referents across languages Wichmann et al. (2010); Johansson and Zlatev (2013); Haynie et al. (2014); Blasi et al. (2016); Joo (2019). Most of these studies were focused on the regular association between phonemic or phonetic units with meaning, occasionally controlling for other potential sources of form-meaning association such as phonotactics or word length Blasi et al. (2016). While useful, the estimates emerging from this type of study can be regarded as lower bounds to the total amount of non-arbitrary associations found in the vocabulary.

Recent efforts have resulted in datasets with thousands of languages Wichmann et al. (2020), with which linguists can look for universal statistical patterns Wichmann et al. (2010); Blasi et al. (2016). These studies, though, only looked at the presence (or not) of individual phones in words, not accounting for their connections. Our methods rely on neural phonotactic models, similar to those used by Pimentel et al. (2020), thus capturing a broader range of potential correspondences.

## 3 Data

An exceptional resource with substantial cross-linguistic representation is provided in the Automated Similarity Judgment Program, better known by its acronym ASJP Wichmann et al. (2020). ASJP is a collection of basic vocabulary wordlists, i.e. lists of words with referents that are expected to be widely attested across human societies. It involves body parts, some colour terms, lower numerals, general properties (such as big or round), and flora and fauna that are usually found in places 

where humans live (e.g. trees and dogs). The individual words in ASJP are transcribed by field linguists in a specific phonetic annotation scheme that involves 41 symbols, chosen in order to maximise cross-linguistic utility by merging rare phones with similar phonetic features within the same category. These wordlists are assembled with the purpose of studying the history of languages--following the tradition established by Swadesh (1955)--under the principles of the comparative method.

ASJP has gathered, in its latest iterations, data for close to \({3}/{4}\) of the world's languages, which makes it an unparalleled resource for evaluating form-meaning associations across spoken languages. Furthermore, the vocabulary in its wordlists was chosen as so to be resistant to borrowings--making it especially interesting for our purposes of finding universal form-meaning biases. We leave out pidgin and creole data,4 as defined by the World Atlas of Language Structures Dryer and Haspelmath (2013), since they are ambiguous with relation to their genealogical affiliation. We also omit constructed and fake languages (e.g. Esperanto and Taensa). This leaves us 9148 doculects (or wordlists) from 5189 languages.5

Footnote 4: Pidgins are believed to rely particularly on iconicity due to their smaller degree of lexicalisation; this reliance then diminishes as it morphs into a creole Romaine (1988). Future work could expand the methods here to study this phenomenon.

Form-meaning associations have been studied in earlier versions of this dataset. Firstly, Wichmann et al. (2010) studied the average form across different concepts in ASJP, and found a number of tentative patterns pointing to non-arbitrariness. Yet the lack of historical and statistical controls compromised the nature of such patterns: form-meaning associations could be due to widespread linguistic contact (e.g. the word for dog, Pache et al. 2016) or to its fortuitous presence in large families. Blasi et al. (2016), however, provide a conservative evaluation of individual form-meaning associations by imposing a restrictive set of conditions. They looked for associations that were present in a minimum number of continents and language families. This resulted in a sizable number of non-arbitrary associations, many of which had been highlighted as interesting based on behavioural and linguistic experiments in a handful of languages.

Data Disclaimer.As mentioned above, ASJP gathers lists of wordforms that are expected to be present across most human societies and their corresponding language(s). While this guarantees a fair coverage in our study, it limits the scope of our conclusions to those concepts present herein.

## 4 Methods

### Notation

We describe each word as comprised by form and meaning, which we represent as a pair \((\mathbf{w}^{(n)},\mathbf{v}^{(n)})\). The form \(\mathbf{w}^{(n)}\in\Sigma^{*}\) is represented as a phone string where \(\Sigma\) is a phonetic alphabet. In this work, we take \(\Sigma\) to be the set of 41 phonetic symbols in ASJP plus the end-of-string symbol. We write \(W\) to denote a \(\Sigma^{*}\)-valued random variable. The meaning \(\mathbf{v}^{(n)}\in\{0,1\}^{K}\) is represented by a one-hot vector, where \(K\) is the number of analysed concepts.6 We write \(V\) to denote a \(\{0,1\}^{K}\)-valued random variable.

Footnote 6: We note Pimentel et al. (2019) used high-dimensional distributional semantic vectors to represent meaning, while we use a one-hot vector. However, their work relied on a specific language’s word?vec—a choice which could potentially bias our results with that language’s properties. We did, however, run an extra experiment with English word?vec; this led to similar conclusions to the ones presented here.

### Non-Arbitrariness as Mutual Information

The goal of this work is to measure cross-linguistic form-meaning associations, operationalised as the **mutual information** (MI) between a form-valued random variable \(W\) and a meaning-valued random variable \(V\). Symbolically, we are interested in computing Cover and Thomas (2012):

\[\mathrm{I}(W;V)=\mathrm{H}(W)-\mathrm{H}(W\mid V) \tag{1}\]

Intuitively, this quantity captures the uncertainty we have over the form, the entropy \(\mathrm{H}(W)\), minus how much uncertainty we have over the form _given_ the meaning, the conditional entropy \(\mathrm{H}(W\mid V)\). Thus, if eq. (1) is zero, its minimum, we have the result that meaning tells us absolutely nothing about the wordform. On the other hand, if eq. (1) is \(\min\{\mathrm{H}(W),\mathrm{H}(V)\}\), its maximum, we have that the form is a deterministic function of the meaning (or the opposite; the meaning being deterministically determined given the form).

 

That the mutual information may take values in \([0,\min\{\mathrm{H}(W),\mathrm{H}(V)\}]\)--together with the fact that, for our specific study, \(\mathrm{H}(W)\) is smaller than \(\mathrm{H}(V)\)-- suggests a more interpretable metric called the **uncertainty coefficient**:

\[\mathrm{U}(W\mid V)=\frac{\mathrm{I}(W;V)}{\mathrm{H}(W)} \tag{2}\]

This quantity is the proportion of uncertainty in the form reduced by knowing the meaning. Both mutual information and uncertainty coefficients are general measures of non-arbitrariness. One might also inquire about how non-arbitrary a single form-meaning pair is. To measure this, we propose **pointwise mutual information** (PMI):

\[\mathrm{PMI}(\mathbf{w};\mathbf{v})=\log\frac{p(\mathbf{w}\mid\mathbf{v})}{p( \mathbf{w})} \tag{3}\]

### Approximating Mutual Information

As noted above, we want to estimate the entropy of language agnostic wordforms, i.e.

\[\mathrm{H}(W)=\sum_{\mathbf{w}\in\Sigma^{*}}p(\mathbf{w})\log\frac{1}{p( \mathbf{w})} \tag{4}\]

Unfortunately, we do not know the exact distribution of \(p(\mathbf{w})\) and, even if we did, we would need to sum over the infinite set of possible strings \(\Sigma^{*}\) to compute this entropy, which is intractable. If we have another probability distribution \(p_{\boldsymbol{\theta}}(\mathbf{w})\), though, we can calculate the cross-entropy between them as an approximation, i.e.

\[\mathrm{H}(W)\leq\mathrm{H}_{\boldsymbol{\theta}}(W)\approx\frac{1}{N}\sum_{n =1}^{N}\log\frac{1}{p_{\boldsymbol{\theta}}(\tilde{\mathbf{w}}^{(n)})} \tag{5}\]

where \(\{\tilde{\mathbf{w}}^{(n)}\}_{n=1}^{N}\) are samples from the true distribution \(p\). Throughout the paper, the tilde marks held-out data, i.e., data not used during model training. We note that the approximation becomes exact as \(N\to\infty\) by the weak law of large numbers. This cross-entropy estimate gives us an upper bound on the actual entropy. This bound is tighter the closer the distributions \(p(\mathbf{w})\) and \(p_{\boldsymbol{\theta}}(\mathbf{w})\) are.

### Estimating the Approximator \(p_{\boldsymbol{\theta}}\)

How should we train a model to estimate this universal phonotactic distribution \(p_{\boldsymbol{\theta}}(\mathbf{w})\), though? We train a phone-level language model to predict the next phone given previous ones in a word, i.e.

\[p_{\boldsymbol{\theta}}(\mathbf{w})=\prod_{t=1}^{|\mathbf{w}|}p_{\boldsymbol{ \theta}}(w_{t}\mid\mathbf{w}_{<t}) \tag{6}\]

In this work, we use an LSTM as our language model Hochreiter and Schmidhuber (1997). Each phone \(w_{t}\) is represented using a lookup embedding \(\mathbf{z}_{t}\in\mathbb{R}^{d}\). These are fed into the LSTM, outputting temporal representations of the sequence:

\[\mathbf{h}_{t}=\mathrm{LSTM}(\mathbf{z}_{t-1},\mathbf{h}_{t-1}) \tag{7}\]

where \(\mathbf{h}_{0}\) is the zero vector. These representations are linearly transformed and used in a softmax to approximate the probability distribution:

\[p_{\boldsymbol{\theta}}\left(w_{t}\mid\mathbf{w}_{<t}\right)=\operatorname{ softmax}\left(\mathbf{W}\,\mathbf{h}_{t}+\mathbf{b}\right) \tag{8}\]

All parameters are learned via gradient descent, minimising the cross-entropy in the training set.

### Cross-linguistic Controls

As mentioned before, salient regularities between form and meaning across languages might result from large groups of genealogically or spatially related languages. In particular it is practical to consider two independent problems in this respect:

1. Eq. (5)'s inequality only holds if \(\mathrm{H}_{\boldsymbol{\theta}}(W)\) is estimated on a set of datapoints sampled independently from the set of points on which the model \(p_{\boldsymbol{\theta}}\) was trained. As such, the test set should only include languages that are not genealogically or areally related to those in the training set;
2. Within our dataset, the different size of areal and genealogical groups should be accounted for so that our results are not biased towards particularly large areas or language families.

#### Train-test split.

To mitigate the problem referred to in the first item, we cross-validate our models by appealing to the notion of macroareas, large-scale regions of the world that simultaneously maximise internal historical dependency while minimising external ones. Striking a balance between historical independence and data availability, we consider the following four macroareas: the Americas, Eurasia, Africa, and the Pacific (which in this instantiation includes Papua New Guinea and Australia--see Fig. 1). We will use these macroareas as our folds. Two macroareas will be used at each time for training, while one other is used for validation and the last for testing. Some language families, though, might be present in more than one macroarea (e.g. many European languages are spoken natively in the Americas and Africa). These 

families will be assigned to the one macroarea which contains most of its family members, since we believe reducing genealogical impact should be preferred over areal impact for our data and purposes, in cases for which such a choice is required.7

Footnote 6: The reader can refer to this paper as “the name of the term ‘\(\mathbf{W}_{0}\)”.

Family size bias.The second problem is tackled by weighting each example's contribution to our loss function by the inverse of its family size \(l^{(n)}\):

\[\mathcal{L}\left(\boldsymbol{\theta}\right)=\frac{1}{L}\sum_{n=1}^{N}\frac{1}{ l^{(n)}}\log\frac{1}{p_{\boldsymbol{\theta}}(\mathbf{w}^{(n)})} \tag{9}\]

where \(L=\sum_{n=1}^{N}\frac{1}{l^{(n)}}\) re-normalises the cross-entropy using the family sizes. This weighted cross-entropy loss function makes per instance contributions of large language families smaller, reducing their impact on the trained model.

To mitigate the same bias effect on the evaluation of validation and test sets, we first get cross-entropies per word. We subsequently average them per language, per family, and per macroarea. This way, each family will have the same effect per macroarea and each macroarea will have the same effect on the overall cross-entropy.

### Concept-Specific Form Distributions

We want to compare per-concept phonotactic models with general ones to analyse sound-meaning associations. With that in mind, we condition phone-level language models on meaning:

\[p_{\boldsymbol{\theta}}(\mathbf{w}~{}|~{}\mathbf{v})=\prod_{t=1}^{|\mathbf{ w}|}p_{\boldsymbol{\theta}}(w_{t}~{}|~{}\mathbf{w}_{<t},\mathbf{v}) \tag{10}\]

These models are trained following the same procedures explained above, but conditioning the LSTMs on concept specific representations. Specifically, the one-hot representation is linearly transformed and fed into the LSTM as its initial state

\[\mathbf{h}_{0}=\mathbf{W}_{0}\,\mathbf{v} \tag{11}\]

where the linear transformation \(\mathbf{W}_{0}\in\mathbb{R}^{d\times K}\) is randomly initialised and learned with the rest of the model. We then use this distribution to estimate the conditional entropy, analogously to eq.5, as in

\[\mathrm{H}(W~{}|~{}V)\lesssim\frac{1}{N}\sum_{n=1}^{N}\log\frac{1}{p_{ \boldsymbol{\theta}}(\tilde{\mathbf{w}}^{(n)}~{}|~{}\tilde{\mathbf{v}}^{(n)})} \tag{12}\]

where \(\{\tilde{\mathbf{w}}^{(n)},\tilde{\mathbf{v}}^{(n)}\}_{n=1}^{N}\) are held-out from-meaning pairs, sampled from the true distribution.8

Footnote 8: This meaning conditioned model may potentially be better than the raw LSTMs (without conditioning on meaning; due to the extra parameters). To control for this fact, we ran an extra experiment where we estimated \(\mathrm{H}(W)\) using the meaning dependent model with shuffled concept IDs (so there is no form-meaning association). The results from this shuffled IDs model were very similar to the raw LSTM ones.

### Non-Arbitrariness as Information

The mutual information between wordforms and meaning can be decomposed into the difference of two entropy measures. Unfortunately, we have no way of directly measuring these entropy values without their probability distributions (\(p(\mathbf{w})\) and \(p(\mathbf{w}~{}|~{}\mathbf{v})\)). We use the estimated cross-entropies as an approximation to this mutual information:

\[\mathrm{I}(W;V) =\mathrm{H}(W)-\mathrm{H}(W~{}|~{}V) \tag{13}\] \[\approx\mathrm{H}_{\boldsymbol{\theta}}(W)-\mathrm{H}_{\boldsymbol {\theta}}(W~{}|~{}V) \tag{14}\]

We note that eq.14 is approximate because it is the difference of two upper bounds. Furthermore, while there are many ways to estimate mutual information, computing it as the difference between two cross-entropies seems to produce consistent results McAllester and Stratos (2020).

### Bounds and Optimisation

As mentioned in SS4.3, our entropy upper bounds will be tighter if our models \(p_{\boldsymbol{\theta}}\) better capture \(p\). With this in mind, we optimise the hyperparameters of our models using Bayesian optimisation with a Gaussian process prior Snoek et al. (2012)--hyper-parameter ranges are presented in App. A. We train 25 models for each configuration and choose the best one according to the validation set, optimising our weighted cross-entropy loss using AdamW Loshchilov and Hutter (2019).

## 5 Experiments and Analysis9
Footnote 9: Our code is available at [https://github.com/rycolab/form-meaning-associations](https://github.com/rycolab/form-meaning-associations).

### Analysis #1: Overall Mutual Information

We are interested in estimating the cross-linguistic mutual information between meaning and wordforms. With this in mind, we follow the steps described in SS4.4, but instead of only 1 model, we train 25 models using different seeds for each fold (totalling 100 models). Average results--overall and per macroarea 

Across macroareas, results indicate a small average contribution of meaning into form (in all cases smaller than \(1\%\)).10 A simple permutation test (explained later in this section) indicates that, under standard levels of significance (\(\alpha=0.01\)) and after controlling for multiple comparisons,11 this average quantity is significant in 2 out of 4 of the macroareas. Nevertheless, this should not be overinterpreted, as unaccounted factors might be responsible for these effects; for instance, the impact of shared history across families in regions smaller than macroareas (almost all human languages have been in contact, directly or indirectly). Hence it is reasonable to conclude that there is no definitive evidence for an overall average association at this level of description of the data. We consider specific concept form-meaning associations next.12

Footnote 10: For comparison, Pimentel et al. (2019) estimate intralanguage systematicity only accounts for roughly \(3\sim 5\%\) of the entropy in wordforms in English, German and Dutch (given a characteristic sample of the vocabulary).

Footnote 11: All our experiments rely on Benjamini and Hochberg (1995) corrections

Paired Permutation Tests.For the permutation test, we first get the average MI over the 25 random seed results for a macroarea. We then permute the signs on these 25 results to create \(10^{5}\) new average MIs. By comparing the original result with these permutation ones we get the probability that our MI estimate is significantly larger than zero. A relevant detail is that these tests are performed on estimates--as opposed to real MI. The mutual information is always non-negative, but our estimate is not. If the MI is zero, we expect our estimates to be negative half the time, since both upper bounds should be roughly equivalent \(\mathrm{H}_{\mathbf{\theta}}(W)\approx\mathrm{H}_{\mathbf{\theta}}(W\mid V)\).

A note on the LSTMs' quality.Our results strongly rely on the quality of approximations. Our language independent \(\mathrm{H}(W)\) estimate is \(3.85\) bits per phone. Meanwhile, the per-language phonotactic cross-entropy found by Pimentel et al. (2020) is, on average, roughly 3 bits per phone--generally speaking, these results seem consistent.13 Furthermore, our model's cross-entropy on the training set is 3.73--while it may have overfit slightly, this is not an aberration.

Footnote 12: We ran an experiment changing the macroarea combinations in the train-validation-test sets and the results were stable, leading only to minor numerical changes to Tab. 1.

### Analysis #2: Per Concept

In this section we focus on concept-specific form-meaning associations. With this in mind we group all words for a specific concept \(\mathbf{c}\in C\) into a set:

\[S_{\mathbf{c}}=\left\{(\tilde{\mathbf{w}}^{(n)},\tilde{\mathbf{v}}^{(n)})\mid \tilde{\mathbf{v}}^{(n)}=\mathbf{c}\in C\right\} \tag{15}\]

For each such set, we run a permutation test on their approximated pointwise mutual information

\begin{table}
\begin{tabular}{l l l c c c} \hline \hline  & \multicolumn{2}{c}{Macroarea} & \multicolumn{2}{c}{Systematicity} \\ \cline{2-6} Train & Validation & Test & \(\mathrm{H}(W)\) & \(\overline{\mathrm{I}(W;V)}\) & \(\mathrm{U}(W\mid V)\) \\ \hline P, Am\({}_{\text{m}}\) & Eurasia & Africa & 3.773 & 0.011* & 0.279\% \\ E, Ar & Pacific & Americas & 3.901 & 0.007† & 0.173\% \\ Av\({}_{\text{f}}\), P & Americas & Eurasia & 3.999 & 0.015�+ & 0.376\% \\ Am\({}_{\text{m}}\), E & Africa & Pacific & 3.755 & 0.016�+ & 0.422\% \\ \hline Average & & & 3.857 & 0.012�+ & 0.312\% \\ \hline \hline \end{tabular} \({}^{\ddagger}p<0.01\)* \(p<0.1\)

\end{table}
Table 1: Mutual information (in bits per phone) between meaning and wordforms. In the train column, P: Pacific, E: Eurasia, Af: Africa, An: Americas.

Figure 2: Uncertainty Coefficients for all 100 concepts in ASJP distributed according to their average word length.

 

values \({\rm PMI}(\mathbf{w}^{(n)};\mathbf{v}^{(n)})\), assessing if a concept has a statistically significant sound-meaning association.14 Of the 100 concepts in our dataset, 26 of them have positive mutual information (\(p<0.01\)). This means that, at least in the set of concepts represented in our dataset, non-arbitrary form-meaning associations are not exceptions.

Footnote 14: This permutation test is similar to the one in §5.1, but uses the family size corrections discussed in §4.5 when averaging results—i.e., for each permutation (and the original one) we average words, languages, families, and macroareas, in this sequence, to get the MI estimate.

We present the average uncertainty coefficient per concept compared to average wordform length in Fig. 2. We do not find any correlation between these measurements. Analysing these results more closely, we see the pronouns \(I\) and _you_ present the highest coefficient values. Most colours in our dataset (_white_, _red_, _green_, _yellow_) show statistically positive MI. Furthermore, some concepts related to body parts (_tongue_, _skin_, _knee_, _heart_, _claw_) and several concepts related to the environment (_water_, _sand_, _star_, _cloud_, _dry_, _cold_) have statistically positive results.

Wichmann et al. (2010) also looked at how concepts differ in their degree of form-meaning associations, presenting them in an ordered list together with a measure of how much they deviate from a global average phone usage. They only look at isolated phone's frequencies, though, and do not control for word length--our mutual information metric controls for both factors. When we compare our results to Wichmann et al.'s (2010) top 10 list of concepts, we see both contain several body parts (_tongue_, _skin_, _knee_) and pronouns (_I_, _you_).

### Analysis #3: Per Language

In their position paper, Perniss et al. (2010) argue that non-arbitrariness is a general property of language, although sometimes believed to be an exception. They further state that:

"if we look at the lexicon of English (or that of other Indo-European languages), we might be forgiven for thinking that there could be anything but a conventionally determined, arbitrary connection between a given word and its referent. For the vast majority of English words there is an arbitrary relationship between form and meaning."

In fact, in our results we do not find positive MI values, on average, for English. In this section, we analyse results per language, trying to find signs of cross-linguistic non-arbitrary associations in them.

Analogously to what we did with concepts, we run permutations tests using the PMIs for the set of words in each language (i.e. sets \(S_{l}\) analogous to \(S_{\bf c}\) in eq. (15)). Fig. 3 presents the per-language uncertainty coefficient values in a world map. There are 5189 languages in ASJP, out of those we find that only 85 have significantly positive mutual information (\(p<0.01\)). Each language, though, has at most 100 values (the number of concepts), making this a hard statistical test after correcting for the multiple tests. If we relax our hypothesis testing thresholds to \(p<0.05\) (an admittedly much weaker test), then 242 languages present statistically positive MI--this suggests that, although maybe not common, form-meaning patterns are not a rare exception restricted to a small number of languages.

Figure 3: Uncertainty Coefficients for the 5189 languages (9148 doculects) in ASJP. Each language is represented by a point and larger points imply a larger coefficient. Significance was assessed through permutation tests.

 



### Analysis #4: Per Concept-Token Pair

We now turn to the relationship between concepts and the phones which appear in them, trying to assess specific concept-phone pairs which present positive MI. Such a positive value would indicate that concept informs on the presence of that specific phone, suggesting a non-arbitrary association between them. Similarly to before, we create sets of concept-token pairs:

\[S_{\mathbf{c},s}= \Big{\{}(\tilde{w}^{(n)}_{t},\check{\mathbf{v}}^{(n)})\mid \tag{16}\] \[\check{\mathbf{v}}^{(n)}=\mathbf{c}\in C,\tilde{w}^{(n)}_{t}=s \in\Sigma\Big{\}}\]

where \((\mathbf{c},s)\) is the analysed concept-token pair and \(\tilde{w}^{(n)}_{t}\) is the \(t^{\text{th}}\) token of word \(\tilde{w}^{(n)}\). During this analysis, though, we focus on concept-phone pairs which had statistically significant PMIs in _all four_ macroareas, following the controls introduced in Blasi et al. (2016), as a way of maximising the chances of finding true history-independent associations (under the risk of increasing the rate of false negatives). With that in mind, we split sets \(S_{c,t}\) per macroarea and got the PMI values for each of them, similarly to SS5.2. We threw away pairs which did not occur at least 1000 times together and ran a permutation test with \(10^{5}\) permutations for each concept-token-macroarea tuple. We note a concept-token association does not make a pair probable; the token is simply more likely to appear with the concept than would be without it.

Tab. 2 presents pairs which were significant in all macroareas (\(p<0.01\) after corrections). After analysis, we find a few interesting results. As mentioned in SS1, we see an association between [l] and the concept tongue and between [p] and full, similarly to Blasi et al. (2016). We also see an association between pronouns--e.g. I, we, you--and the end-of-string [\(\#\)].15 This was expected; promuons are very frequent words in most languages, and such words are usually shorter Zipf (1949).

Footnote 15: The association of a concept with the [\(\#\)] symbol means the model can more easily predict the end-of-word when conditioned on this concept. This means the length of that concept is not distributed as the average, being more predictable.

As previously found by Blasi et al. (2016), the concept breast has a significant association with both [m] and [u]. As they point out, these might be due to the mouth configuration of suckling babies or the sounds they produce when feeding Jakobson (1960); Traunmuller (1994). We further find several other pairs which are supported by their findings: horn--[k,r]; knee-[o,u,k]; leaf-[l,p]; we-[n]. Furthermore, a nice sanity check is that none of the negative concept-pair associations they found are present in our results.

### Analysis #5: Macroareas vs Family

As a final experiment, we analyse the importance of splitting train-test sets according to macroareas (as discussed in SS4.5) in order to minimise areal effects--versus simply splitting languages based on their families. Even though the list of concepts in ASJP was designed to be resistant to borrowings (and we further remove loan words from our analysis), language contacts beyond loan words could still impact results. One such example is the (potential) impact of Basque in Spanish phonology, which lost word initial /f/ in many words, e.g. _hablar_, during the late Middle Ages (see pg. 91 of penny, 2002, for a longer discussion).

We create 4 folds, splitting them based on glottocode language families, and use 4-fold cross-validation to get family-split results--in opposition to the macroarea-split results. Using family-splits we get an \(\mathrm{I}(W;V)=0.020\) bits, with an uncertainty coefficient of 0.53% (averaged over the 4

\begin{table}
\begin{tabular}{l c|l r l|l r|l r|l r} \hline \hline Concept & Tokens & Concept & Tokens & Concept & Tokens & Concept & Tokens & Concept & Tokens \\ \hline blood & s & eye & i & liver & k lr t & path & dt & tree & \(\#\) \\ bone & s u & fire & \(\#\) t & loose & m n & say & \(\#\) & two & r \\ breast & m u & fish & a s & mountain & b d g l o r & see & \(\#\) e & water & \(\#\) \\ come & \(\#\) e & full & l o p t & name & \(\#\) i & skin & kl pr t & we & \(\#\) e i \\ die & t & give & \(\#\) & neck & o & star & kl pr stuw w & who & \(\#\) \\ dog & k & horn & kr & new & a & stone & kt & you & \(\#\) ai n \\ drink & \(\#\) u & I & \(\#\) a n & night & N d j im pr t & sun & e & & \\ ear & e l t & knee & Nb g km o rt u & nose & Ni u & tongue & de ln r & & \\ eat & \(\#\) & leaf & a l p t & one & kT & tooth & ei & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Concept–Token pairs with statistically significant (\(p<0.01\) after Benjamini–Hochberg corrections) mutual information in all 4 macro areas. \(\#\) is the end-of-string token.



folds)--this is almost twice the overall MI found on the macroarea-splits. A Welch's \(t\)-test between both runs shows family-splits have a larger MI than the macroarea results (\(p<0.01\)), suggesting it is important to control for areal effects when evaluating sound-meaning associations.

## 6 Conclusion

In this paper we have provided a holistic assessment of form-meaning associations involving words found in the basic vocabulary in a large number of languages. In agreement with previous findings, we find that on average the meaning does not contribute substantially to the form of the words, but instead the most consistent associations were restricted to a specific subset of all of the words analysed. We find a list of 26 concepts (out of the 100 analysed) with statistically significant form-meaning associations--suggesting that cross-linguistic non-arbitrariness is not a rare exception. Finally, we also find a set of concept-phone pairs with a consistently positive relationship across the four analysed macroareas.

### Ethical Considerations

This paper concerns itself with investigating cross-linguistic form-meaning associations. We see no direct ethical concerns relating to this work, as it only involves computational experiments on previously collected data.

### Acknowledgments

Soren Wichmann's research was partly funded by a subsidy from the Russian government to support the Programme of Competitive Development of Kazan Federal University, Russia. Damian E. Blasi acknowledges funding from the Branco Weiss Fellowship, administered by the ETH Zurich. Damian E. Blasi's research was also executed within the framework of the HSE University Basic Research Program and funded by the Russian Academic Excellence Project '5-100'.

## Appendix A Hyper-parameter Optimisation Range

As mentioned in SS4.8, we used Bayesian optimisation to tune the model's hyper-parameters. We consider a log-uniform prior over the embedding size (from \(4\) to \(1024\)), and over the size of the hidden state (\(32\) to \(1024\)). We also considered a uniform prior over the number of layers (\(1\) to \(4\)) and dropout (\(0\) to \(0.5\)).

## References

* Bankieris and Simner [2015] Kaitlyn Bankieris and Julia Simner. 2015. What is the link between ynaesthesia and sound symbolism? _Cognition_, 136:186-195.
* Benjamini and Hochberg [1995] Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. _Journal of the Royal Statistical Society. Series B (Methodological)_, 57(1):289-300.
* Bergen [2004] Benjamin K. Bergen. 2004. The psychological reality of phoneasthemes. _Language_, 80(2):290-311.
* Blasi et al. [2016] Damian E. Blasi, Soren Wichmann, Harald Hammarstrom, Peter F. Stadler, and Morten H. Christiansen. 2016. Sound-meaning association biases evidenced across thousands of languages. _Proceedings of the National Academy of Sciences of the U.S.A._, 113(39):10818-10823.
* Cover and Thomas [2012] Thomas M. Cover and Joy A. Thomas. 2012. _Elements of Information Theory_. John Wiley & Sons, New York.
* Dautriche et al. [2017] Isabelle Dautriche, Kyle Mahowald, Edward Gibson, and Steven T. Piantdosi. 2017. Wordform similarity increases with semantic similarity: An analysis of 100 languages. _Cognitive Science_, 41(8):2149-2169.
* Dellert et al. [2020] Johannes Dellert, Thora Daneyko, Alla Munch, Alina Ladygina, Armin Buch, Natalie Clarius, Ilja Grigorjew, Mohamed Balabel, Hirziny Isabella Boga, Zalina Baysarova, Roland Muhlenbernd, Johannes Wahle, and Gerhard Jager. 2020. NorthEuraLex: A wide-coverage lexical database of Northern Eurasia. _Language Resources and Evaluation_, 54:273-301.
* Dingemanse [2012] Mark Dingemansse. 2012. Advances in the crosslinkingtic study of ideophones. _Language and Linguistics Compass_, 6(10):654-672.
* Dingemanse [2018] Mark Dingemane. 2018. Redrawing the margins of language: Lessons from research on ideophones. _Glossa: a journal of general linguistics_, 3(1).
* Dingemanse et al. [2015] Mark Dingemanse, Damian E. Blasi, Gary Lupyan, Morten H. Christiansen, and Padraic Monaghan. 2015. Arbitrariness, iconicity, and systematicity in language. _Trends in Cognitive Sciences_, 19(10):603-615.
* Dryer and Haspelmath [2013] Matthew S. Dryer and Martin Haspelmath, editors. 2013. _WALS Online_. Max Planck Institute for Evolutionary Anthropology, Leipzig.
* Gutierrez et al. [2016] E. Dario Gutierrez, Roger Levy, and Benjamin Bergen. 2016. Finding non-arbitrary form-meaning systematically using string-metric learning for kernel regression. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers_, pages 2379-2388. Association for Computational Linguistics.

 

John Haiman. 1980. The iconicity of grammar: Isomorphism and motivation. _Language_, 56(3):515-540.
* Haynie et al. (2014) Hannah Haynie, Claire Bowern, and Hannah LaPalombara. 2014. Sound symbolism in the languages of Australia. _PLoS ONE_, 9(4):e92852.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Jakobson (1960) Roman Jakobson. 1960. Why 'mama' and 'paqa'? In B. Kaplan and S. Wapner, editors, _Perspectives in Psychological Theory: Essays in Honor of Heinz Werner_, pages 538-549. International Universities Press, New York.
* Johansson and Zlatev (2013) Niklas Johansson and Jordan Zlatev. 2013. Motivations for sound symbolism in spatial deixis: A typological study of 101 languages. _Public Journal of Semiotics_, 5(1):3-20.
* Joo (2019) Ian Joo. 2019. Phonosemantic biases found in Leipzig-Jakarta lists of 66 languages. _Linguistic Typology_, 24(1):1-12.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* McAllester and Stratos (2020) David McAllester and Karl Stratos. 2020. Formal limitations on the measurement of mutual information. In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_, volume 108 of _Proceedings of Machine Learning Research_, pages 875-884. PMLR.
* Pache et al. (2016) Matthias Pache, Soren Wichmann, and Mikhail Zhivlov. 2016. Words for 'dog' as a diagnostic of language contact in the Americas. In Andrea L. Berez-Kroeker, Diane M. Hintz, and Carmen Jangy, editors, _Language Contact and Change in the Americas: Studies in honour of Mariame Mihun_, pages 385-410. John Benjamins, Amsterdam/Philadelphia.
* Parise et al. (2014) Cesare V. Parise, Katharina Knorre, and Marc O. Ernst. 2014. Natural auditory scene statistics shapes human spatial hearing. _Proceedings of the National Academy of Sciences of the U.S.A._, 111(16):6104-6108.
* Penny (2002) Ralph John Penny. 2002. _A History of the Spanish Language_, 2nd ed. edition. Cambridge University Press, Cambridge.
* Permiss et al. (2010) Pamela Permiss, Robin Thompson, and Gabriella Vigliocco. 2010. Iconicity as a general property of language: Evidence from spoken and signed languages. _Frontiers in Psychology_, 1:227.
* Pimentel et al. (2019) Tiago Pimentel, Arya D. McCarthy, Damian Blasi, Brian Roark, and Ryan Cotterell. 2019. Meaning to form: Measuring systematicity as information. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1751-1764, Florence, Italy. Association for Computational Linguistics.
* Pimentel et al. (2020)Tiago Pimentel, Brian Roark, and Ryan Cotterell. 2020. Phonotactic complexity and its trade-offs. _Transactions of the Association for Computational Linguistics_, 8:1-18.
* Romainie (1988) Suzanne Romaine. 1988. _Pidgin and Creole Languages_. Longman, London/New York.
* de Saussure (1916) Ferdinand de Saussure. 1916. _Course in General Linguistics_. Columbia University Press, New York. English edition of June 2011, based on the 1959 translation by Wade Baskin.
* Shillcock et al. (2001) Richard Shillcock, Simon Kirby, Scott McDonald, and Chris Brew. 2001. Filled pauses and their status in the mental lexicon. In _ISCA Tutorial and Research Workshop (ITRW) on Disfluency in Spontaneous Speech_.
* Snoek et al. (2012) Jasper Snoek, Hugo Larochelle, and Ryan Prescott Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In _Advances in Neural Information Processing Systems_, volume 25. Curran Associates, Inc.
* Swadesh (1955) Morris Swadesh. 1955. Towards greater accuracy in lexicostatistic dating. _International Journal of American Linguistics_, 21(2):121-137.
* Traunmuller (1994) Hartmutt Traunmuller. 1994. Sound symbolism in deic words. In Hans Auli and Peter af Trampe, editors, _Tongues and Texts Unlimited. Studies in Honour of More Jansson on the Occasion of his Sixtieth Anniversary_, pages 213-234. Dept. of Classical Languages, Stockholm University, Stockholm.
* Wichmann et al. (2010) Soren Wichmann, Eric W. Holman, and Cecil H. Brown. 2010. Sound symbolism in basic vocabulary. _Entropy_, 12(4):844-858.
* Wichmann et al. (2020) Soren Wichmann, Eric W. Holman, and Cecil H. Brown. 2020. The ASJP database (version 19). Accessed on April 2, 2020.
* Zipf (1949) George Kingsley Zipf. 1949. _Human Behavior and the Principle of Least Effort_. Addison-Wesley Press, Cambridge, Mass.