SlowGuess/NAACL_2021_OCR huggingface dataset:
Macroarea	H(W)	Systematicity
Train	Validation	Test	I(W; V)	U(W | V)
P, Am	Eurasia	Africa	3.773	0.011*	0.279%
E, Af	Pacific	Americas	3.901	0.007	0.173%
Af, P	Americas	Eurasia	3.999	0.015‡	0.376%
Am, E	Africa	Pacific	3.755	0.016‡	0.422%
Average	3.857	0.012‡	0.312%
$\ddagger p <   0.01^{*}p <   0.1$
Table 1: Mutual information (in bits per phone) between meaning and wordforms. In the train column, P: Pacific, E: Eurasia,  $A_{f}$ : Africa,  $A_{m}$ : Americas.
Across macroareas, results indicate a small average contribution of meaning into form (in all cases smaller than  $1\%$ ).<sup>10</sup> A simple permutation test (explained later in this section) indicates that, under standard levels of significance ( $\alpha = 0.01$ ) and after controlling for multiple comparisons,<sup>11</sup> this average quantity is significant in 2 out of 4 of the macroareas. Nevertheless, this should not be overinterpreted, as unaccounted factors might be responsible for these effects; for instance, the impact of shared history across families in regions smaller than macroareas (almost all human languages have been in contact, directly or indirectly). Hence it is reasonable to conclude that there is no definitive evidence for an overall average association at this level of description of the data. We consider specific concept form-meaning associations next.<sup>12</sup>
Paired Permutation Tests. For the permutation test, we first get the average MI over the 25 random seed results for a macroarea. We then permute the signs on these 25 results to create  $10^{5}$  new average MIs. By comparing the original result with these permutation ones we get the probability that our MI estimate is significantly larger than zero. A relevant detail is that these tests are performed on estimates—as opposed to real MI. The mutual information is always non-negative, but our estimate is not. If the MI is zero, we expect our estimates to be negative half the time, since both upper bounds should be roughly equivalent  $\mathrm{H}_{\theta}(W)\approx \mathrm{H}_{\theta}(W\mid V)$ .
A note on the LSTMs' quality. Our results strongly rely on the quality of approximations. Our language independent  $\mathrm{H}(W)$  estimate is 3.85 bits per phone. Meanwhile, the per-language phonotactic cross-entropy found by Pimentel et al. (2020) is, on average, roughly 3 bits per phone—generally speaking, these results seem consistent.[13] Furthermore, our model's cross-entropy on the training set is 3.73—while it may have overfit slightly, this is not an aberration.
5.2 Analysis #2: Per Concept
In this section we focus on concept-specific formmeaning associations. With this in mind we group all words for a specific concept  $\mathbf{c} \in C$  into a set:
$$
S _ {\mathbf {c}} = \left\{\left(\tilde {\mathbf {w}} ^ {(n)}, \tilde {\mathbf {v}} ^ {(n)}\right) \mid \tilde {\mathbf {v}} ^ {(n)} = \mathbf {c} \in C \right\} \tag {15}
$$
For each such set, we run a permutation test on their approximated pointwise mutual information
For comparison, Pimentel et al. (2019) estimate intralanguage systematicity only accounts for roughly  $3\sim 5\%$  of the entropy in wordforms in English, German and Dutch (given a characteristic sample of the vocabulary).
11 All our experiments rely on Benjamini and Hochberg (1995) corrections
12We ran an experiment changing the macroarea combinations in the train-validation-test sets and the results were stable, leading only to minor numerical changes to Tab. 1.
These results are not directly comparable, though, since words are encoded with different phonetic alphabets in ASJP and NorthEuraLex (Dellert et al., 2020).
Figure 3: Uncertainty Coefficients for the 5189 languages (9148 doculects) in ASJP. Each language is represented by a point and larger points imply a larger coefficient. Significance was assessed through permutation tests.
values  $\mathrm{PMI}(\mathbf{w}^{(n)};\mathbf{v}^{(n)})$ , assessing if a concept has a statistically significant sound-meaning association.14 Of the 100 concepts in our dataset, 26 of them have positive mutual information  $(p < 0.01)$ . This means that, at least in the set of concepts represented in our dataset, non-arbitrary form-meaning associations are not exceptions.
We present the average uncertainty coefficient per concept compared to average wordform length in Fig. 2. We do not find any correlation between these measurements. Analysing these results more closely, we see the pronouns  $I$  and you present the highest coefficient values. Most colours in our dataset (white, red, green, yellow) show statistically positive MI. Furthermore, some concepts related to body parts (tongue, skin, knee, heart, claw) and several concepts related to the environment (water, sand, star, cloud, dry, cold) have statistically positive results.
Wichmann et al. (2010) also looked at how concepts differ in their degree of form—meaning associations, presenting them in an ordered list together with a measure of how much they deviate from a global average phone usage. They only look at isolated phone's frequencies, though, and do not control for word length—our mutual information metric controls for both factors. When we compare our results to Wichmann et al.'s (2010) top 10 list of concepts, we see both contain several body parts (tongue, skin, knee) and pronouns ( $I$ , you).
5.3 Analysis #3: Per Language
In their position paper, Perniss et al. (2010) argue that non-arbitrariness is a general property of language, although sometimes believed to be an exception. They further state that:
"If we look at the lexicon of English (or that of other Indo-European languages), we might be forgiven for thinking that there could be anything but a conventionally determined, arbitrary connection between a given word and its referent. For the vast majority of English words there is an arbitrary relationship between form and meaning."
In fact, in our results we do not find positive MI values, on average, for English. In this section, we analyse results per language, trying to find signs of cross-linguistic non-arbitrary associations in them.
Analogously to what we did with concepts, we run permutations tests using the PMIs for the set of words in each language (i.e. sets  $S_{l}$  analogous to  $S_{\mathbf{c}}$  in eq. (15)). Fig. 3 presents the per-language uncertainty coefficient values in a world map. There are 5189 languages in ASJP, out of those we find that only 85 have significantly positive mutual information ( $p < 0.01$ ). Each language, though, has at most 100 values (the number of concepts), making this a hard statistical test after correcting for the multiple tests. If we relax our hypothesis testing thresholds to  $p < 0.05$  (an admittedly much weaker test), then 242 languages present statistically positive MI—this suggests that, although maybe not common, form-meaning patterns are not a rare exception restricted to a small number of languages.
14This permutation test is similar to the one in §5.1, but uses the family size corrections discussed in §4.5 when averaging results—i.e., for each permutation (and the original one) we average words, languages, families, and macroareas, in this sequence, to get the MI estimate.
Concept	Tokens	Concept	Tokens	Concept	Tokens	Concept	Tokens	Concept	Tokens
blood	s	eye	i	liver	k l r t	path	d t	tree	#
bone	s u	fire	# t	louse	m n	say	#	two	r
breast	m u	fish	a s	mountain	b d g l o r	see	# e	water	#
come	# e	full	l o p t	name	# i	skin	k l p r t	we	# e i n
die	t	give	#	neck	o	star	k l o r s t u w	who	#
dog	k	horn	k r	new	a	stone	k t	you	# a i n
drink	# u	I	# a n	night	N d i l m p r t u	sun	e
ear	e l t	knee	N b g k m o r t u	nose	N i u	tongue	d e l n r
eat	#	leaf	a l p t	one	k t	tooth	e i
Table 2: Concept-Token pairs with statistically significant ( $p < 0.01$  after Benjamini-Hochberg corrections) mutual information in all 4 macro areas.  $\#$  is the end-of-string token.
5.4 Analysis #4: Per Concept-Token Pair
We now turn to the relationship between concepts and the phones which appear in them, trying to assess specific concept–phone pairs which present positive MI. Such a positive value would indicate that concept informs on the presence of that specific phone, suggesting a non-arbitrary association between them. Similarly to before, we create sets of concept–token pairs:
$$
\begin{array}{l} S _ {\mathbf {c}, s} = \left\{\left(\tilde {w} _ {t} ^ {(n)}, \tilde {\mathbf {v}} ^ {(n)}\right) \mid \right. \tag {16} \\ \left. \tilde {\mathbf {v}} ^ {(n)} = \mathbf {c} \in C, \tilde {w} _ {t} ^ {(n)} = s \in \Sigma \right\} \\ \end{array}
$$
where  $(\mathbf{c}, s)$  is the analysed concept-token pair and  $\tilde{w}_t^{(n)}$  is the  $t^{\text{th}}$  token of word  $\tilde{w}^{(n)}$ . During this analysis, though, we focus on concept–phone pairs which had statistically significant PMIs in all four macroareas, following the controls introduced in Blasi et al. (2016), as a way of maximising the chances of finding true history-independent associations (under the risk of increasing the rate of false negatives). With that in mind, we split sets  $S_{c,t}$  per macroarea and got the PMI values for each of them, similarly to §5.2. We threw away pairs which did not occur at least 1000 times together and ran a permutation test with  $10^5$  permutations for each concept–token–macroarea tuple. We note a concept–token association does not make a pair probable; the token is simply more likely to appear with the concept than would be without it.
Tab. 2 presents pairs which were significant in all macroareas  $(p < 0.01$  after corrections). After analysis, we find a few interesting results. As mentioned in §1, we see an association between [l] and the concept TONGUE and between [p] and FULL, similarly to Blasi et al. (2016). We also see an association between pronouns—e.g. I, WE, YOU—and
the end-of-string [\#].<sup>15</sup> This was expected; pronouns are very frequent words in most languages, and such words are usually shorter (Zipf, 1949).
As previously found by Blasi et al. (2016), the concept BREAST has a significant association with both [m] and [u]. As they point out, these might be due to the mouth configuration of suckling babies or the sounds they produce when feeding (Jakobson, 1960; Traunmüller, 1994). We further find several other pairs which are supported by their findings: HORN-[k,r]; KNEE-[o,u,k]; LEAF-[l,p]; WE-[n]. Furthermore, a nice sanity check is that none of the negative concept-pair associations they found are present in our results.
5.5 Analysis #5: Macroareas vs Family
As a final experiment, we analyse the importance of splitting train-test sets according to macroareas (as discussed in §4.5) in order to minimise areal effects—versus simply splitting languages based on their families. Even though the list of concepts in ASJP was designed to be resistant to borrowings (and we further remove loan words from our analysis), language contacts beyond loan words could still impact results. One such example is the (potential) impact of Basque in Spanish phonology, which lost word initial /f/ in many words, e.g. haber, during the late Middle Ages (see pg. 91 of Penny, 2002, for a longer discussion).
We create 4 folds, splitting them based on glottocode language families, and use 4-fold cross-validation to get family-split results—in opposition to the macroarea-split results. Using family-splits we get an  $\mathrm{I}(W;V) = 0.020$  bits, with an uncertainty coefficient of  $0.53\%$  (averaged over the 4-
15 The association of a concept with the  $[\# ]$  symbol means the model can more easily predict the end-of-word when conditioned on this concept. This means the length of that concept is not distributed as the average, being more predictable.
folds)—this is almost twice the overall MI found on the macroarea-splits. A Welch's  $t$ -test between both runs shows family-splits have a larger MI than the macroarea results  $(p < 0.01)$ , suggesting it is important to control for areal effects when evaluating sound-meaning associations.
In this paper we have provided a holistic assessment of form-meaning associations involving words found in the basic vocabulary in a large number of languages. In agreement with previous findings, we find that on average the meaning does not contribute substantially to the form of the words, but instead the most consistent associations were restricted to a specific subset of all of the words analysed. We find a list of 26 concepts (out of the 100 analysed) with statistically significant form-meaning associations—suggesting that cross-linguistic non-arbitrariness is not a rare exception. Finally, we also find a set of concept–phone pairs with a consistently positive relationship across the four analysed macroareas.
This paper concerns itself with investigating cross-linguistic form-meaning associations. We see no direct ethical concerns relating to this work, as it only involves computational experiments on previously collected data.
Søren Wichmann's research was partly funded by a subsidy from the Russian government to support the Programme of Competitive Development of Kazan Federal University, Russia. Damián E. Blasi acknowledges funding from the Branco Weiss Fellowship, administered by the ETH Zürich. Damián E. Blasi's research was also executed within the framework of the HSE University Basic Research Program and funded by the Russian Academic Excellence Project '5-100'.
A Hyper-parameter Optimisation Range
As mentioned in §4.8, we used Bayesian optimisation to tune the model's hyper-parameters. We consider a log-uniform prior over the embedding size (from 4 to 1024), and over the size of the hidden state (32 to 1024). We also considered a uniform prior over the number of layers (1 to 4) and dropout (0 to 0.5).
Kaitlyn Bankieris and Julia Simner. 2015. What is the link between synaesthesia and sound symbolism? Cognition, 136:186-195.
Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1):289-300.
Benjamin K. Bergen. 2004. The psychological reality of phonaesthetics. Language, 80(2):290-311.
Damián E. Blasi, Søren Wichmann, Harald Hammarström, Peter F. Stadler, and Morten H. Christiansen. 2016. Sound-meaning association biases evidenced across thousands of languages. Proceedings of the National Academy of Sciences of the U.S.A., 113(39):10818-10823.
Thomas M. Cover and Joy A. Thomas. 2012. Elements of Information Theory. John Wiley & Sons, New York.
Isabelle Dautriche, Kyle Mahowald, Edward Gibson, and Steven T. Piantadosi. 2017. Wordform similarity increases with semantic similarity: An analysis of 100 languages. Cognitive Science, 41(8):2149-2169.
Johannes Dellert, Thora Daneyko, Alla Münch, Alina Ladygina, Armin Buch, Natalie Clarius, Ilja Grigorjew, Mohamed Balabel, Hizniye Isabella Boga, Zalina Baysarova, Roland Mühlenbernd, Johannes Wahle, and Gerhard Jäger. 2020. NorthEuraLex: A wide-coverage lexical database of Northern Eurasia. Language Resources and Evaluation, 54:273-301.
Mark Dingemanse. 2012. Advances in the crosslinguistic study of ideophones. Language and Linguistics Compass, 6(10):654-672.
Mark Dingemanse. 2018. Redrawing the margins of language: Lessons from research on ideophones. Glossa: a journal of general linguistics, 3(1).
Mark Dingemanse, Damián E. Blasi, Gary Lupyan, Morten H. Christiansen, and Padraic Monaghan. 2015. Arbitrariness, iconicity, and systematicity in language. Trends in Cognitive Sciences, 19(10):603-615.
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
E. Dario Gutierrez, Roger Levy, and Benjamin Bergen. 2016. Finding non-arbitrary form-meaning systematicity using string-metric learning for kernel regression. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2379-2388. Association for Computational Linguistics.
John Haiman. 1980. The iconicity of grammar: Isomorphism and motivation. Language, 56(3):515-540.
Hannah Haynie, Claire Bowern, and Hannah LaPalombara. 2014. Sound symbolism in the languages of Australia. PLoS ONE, 9(4):e92852.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Roman Jakobson. 1960. Why 'mama' and 'papa'? In B. Kaplan and S. Wapner, editors, Perspectives in Psychological Theory: Essays in Honor of Heinz Werner, pages 538-549. International Universities Press, New York.
Niklas Johansson and Jordan Zlatev. 2013. Motivations for sound symbolism in spatial deixis: A typological study of 101 languages. *Public Journal of Semiotics*, 5(1):3-20.
Ian Joo. 2019. Phonosemantic biases found in Leipzig-Jakarta lists of 66 languages. Linguistic Typology, 24(1):1-12.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
David McAllester and Karl Stratos. 2020. Formal limitations on the measurement of mutual information. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 875-884. PMLR.
Matthias Pache, Søren Wichmann, and Mikhail Zhivlov. 2016. Words for 'dog' as a diagnostic of language contact in the Americas. In Andrea L. Berez-Kroeker, Diane M. Hintz, and Carmen Jany, editors, Language Contact and Change in the Americas: Studies in honour of Marianne Mithun, pages 385-410. John Benjamins, Amsterdam/Philadelphia.
Cesare V. Parise, Katharina Knorre, and Marc O. Ernst. 2014. Natural auditory scene statistics shapes human spatial hearing. Proceedings of the National Academy of Sciences of the U.S.A., 111(16):6104-6108.
Ralph John Penny. 2002. A History of the Spanish Language, 2nd ed. edition. Cambridge University Press, Cambridge.
Pamela Perniss, Robin Thompson, and Gabriella Vigliocco. 2010. Iconicity as a general property of language: Evidence from spoken and signed languages. Frontiers in Psychology, 1:227.
Tiago Pimentel, Arya D. McCarthy, Damian Blasi, Brian Roark, and Ryan Cotterell. 2019. Meaning to form: Measuring systematicity as information. In
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1751-1764, Florence, Italy. Association for Computational Linguistics.
Tiago Pimentel, Brian Roark, and Ryan Cotterell. 2020. Phonotactic complexity and its trade-offs. Transactions of the Association for Computational Linguistics, 8:1-18.
Suzanne Romaine. 1988. *Pidgin and Creole Languages*. Longman, London/New York.
Ferdinand de Saussure. 1916. Course in General Linguistics. Columbia University Press, New York. English edition of June 2011, based on the 1959 translation by Wade Baskin.
Richard Shillcock, Simon Kirby, Scott McDonald, and Chris Brew. 2001. Filled pauses and their status in the mental lexicon. In ISCA Tutorial and Research Workshop (ITRW) on Disfluency in Spontaneous Speech.
Jasper Snoek, Hugo Larochelle, and Ryan Prescott Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc.
Morris Swadesh. 1955. Towards greater accuracy in lexicostatistic dating. International Journal of American Linguistics, 21(2):121-137.
Hartmut Traunmüller. 1994. Sound symbolism in deictic words. In Hans Auli and Peter af Trampe, editors, Tongues and Texts Unlimited. Studies in Honour of Tore Jansson on the Occasion of his Sixtieth Anniversary, pages 213-234. Dept. of Classical Languages, Stockholm University, Stockholm.
Søren Wichmann, Eric W. Holman, and Cecil H. Brown. 2010. Sound symbolism in basic vocabulary. Entropy, 12(4):844-858.
Søren Wichmann, Eric W. Holman, and Cecil H. Brown. 2020. The ASJP database (version 19). Accessed on April 2, 2020.
George KingsleyZipf.1949.Human Behavior and the Principle of Least Effort.Addison-Wesley Press, Cambridge, Mass.