SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Morris Alper and Hadar Averbuch-Elor  
Tel Aviv University
Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings across languages and demographic groups, a phenomenon known as sound symbolism. Among the many dimensions of meaning, sound symbolism is particularly salient and well-demonstrated with regards to cross-modal associations between language and the visual domain. In this work, we address the question of whether sound symbolism is reflected in vision-and-language models such as CLIP and Stable Diffusion. Using zero-shot knowledge probing to investigate the inherent knowledge of these models, we find strong evidence that they do show this pattern, paralleling the well-known kiki-bouba effect in psycholinguistics. Our work provides a novel method for demonstrating sound symbolism and understanding its nature using computational tools. Our code will be made publicly available<sup>1</sup>.
"What's in a name? That which we call a rose by any other name would smell as sweet." —William Shakespeare, Romeo and Juliet
Philosophers of language have long debated whether the mapping between sound and meaning in speech is arbitrary. Discussions on this topic date back to the Socratic dialogues of Plato, as well as early modern philosophers such as John Locke and Gottfried Wilhelm Leibniz [30]. Charles de Saussure, the seminal early 20th century linguist and semiotician, famously stated that le signe est arbitraire<sup>2</sup>. In Saussure's view, words are simply arbitrary conventions and their sounds have no inherent connection to their meanings; hence the French word chien and its English translation dog both equally denote the same animal despite sharing no sounds in common [15].
Although the concept of the arbitrariness of the sign was influential on modern linguistics, there are many evident cases where it does not hold which have attracted great interest among modern researchers. Cases of sound symbolism in language, where the sounds themselves in a word have some connection to what they describe, include onomatopoeic phrases such as kapow (a punch or banging sound) and glub-glub (water bubbling). Beyond direct imitation, it has been noted that English and other languages show correlations between certain phonetic structures and types of meaning, such as the array of English words beginning with cr- denoting brittleness (crush, crunch, crash, crack, crackle, crinkle, ...). This raises natural questions including how universal these patterns of iconicity are between languages and cultures, whether they are rooted in psychological tendencies that influence language or vice versa, what acoustic properties of speech sounds influence iconicity, and the extent to which less obvious iconic patterns shape language in general.
1via our project page https://kiki-bouba.github.io/ 2"the sign is arbitrary
Original Experiment
Figure 1: Illustration of the kiki-bouba effect. The shapes on the far left illustrate stimuli used in the classic kiki-bouba experiment. The remaining images are random generations from Stable Diffusion with the prompt a 3D rendering of a  $\langle w\rangle$  shaped object, where  $\langle w\rangle \in \{kiki,bouba\}$ . Which of these images do you think were generated using pseudoword kiki and which with bouba? See below<sup>3</sup> for the answer.
Random Image Generations
Perhaps the most well-known and conclusively demonstrated examples of sound symbolism is the kiki-bouba effect. In this experiment, subjects are shown a spiky object and a round object, such as the left-hand shapes in Figure 1. When asked to assign one the name kiki and the other the name bouba, subjects show an overwhelming preference (see footnote<sup>4</sup> for the preferred matching). This effect has been shown for both auditory stimuli [19] and for written text (which implicitly maps to speech sounds) [13, 10, 14]. Furthermore, an array of studies have demonstrated it to hold across various speech sounds and constructed pseudowords (beyond kiki and bouba) [34], between different languages and cultures [3], and even among prelingual infants [40]. These findings challenge the Saussurean assumption that sound maps purely arbitrarily to meaning.
In another domain, recent years have seen explosive progress in the field of machine learning applied to natural language and vision, mainly powered by transformer neural networks and training on web-scale datasets of captioned images [6]. This includes discriminative models such as CLIP [45] and generative text-to-image models such as Stable Diffusion [49]. Although these models have revolutionized many applied tasks, they are largely used as a black box. One line of research examines the emergent behavior and inner structure of neural networks in order to interpret their behavior [48], and several works investigate the visual knowledge encoded by vision-and-language models (VLMs) in the context of human cognition and visual semantics in language [2, 61]. Given these parallels along with fundamental differences between these models and the human brain, it is natural to ask whether they have also learned to associate sounds in language (as reflected in their written representations) with visual semantics. In other words, does the vision-and-language setting reflect the presence of sound symbolic patterns in language? If so, this would provide strong evidence against Saussure's legendary claim from a novel computational perspective.
As seen in Figure 1, generative text-to-image models can be applied to text containing pseudowords such as kiki or bouba. Consider the various images generated by Stable Diffusion shown on the right-hand side of the figure. The round objects were generated with one of these pseudowords and the sharp objects were generated with the other (see below for the correct mapping), suggesting that the model associates each pseudoword with visual semantic properties such as roundness and sharpness. It is not obvious a priori that VLMs would learn associations with text input that is seemingly not well-formed; this suggestive parallel to human cognition invites methodical study. In this work, we analyze this phenomenon rigorously with quantitative as well as qualitative criteria, including generalizing to a larger set of constructed pseudowords reflecting many different speech sounds in English, and examining correlations between form and meaning on the grapheme (letter) as well as whole-word level.
To test for sound symbolism in VLMs, we use zero-shot knowledge probing which allows for evaluating the models' inherent knowledge—i.e., not new knowledge acquired during further training. We leverage the ability of CLIP to embed text and image data in a shared semantic space in order to probe discriminative and generative (text-to-image) models with the same evaluation metrics. Our tests evaluate whether these models encode pseudowords similarly to humans with respect to known symbolic associations, comparing them to adjectives indicating properties related to "sharpness" and "roundness". To further ground our results in human cognition, we also conduct a user study testing the ability of subjects to reconstruct pseudowords used to condition text-to-image generations. Our results demonstrate that sound symbolism can indeed be observed in VLMs; the models under
3From left to right: bouba, kiki, bouba, kiki, kiki, bouba
4The vast majority of subjects prefer the name kiki for the spiky object and bouba for the round object.
consideration associate visual meanings with the written forms of particular speech sounds. Although we do not claim to answer exactly how this is learned by these models, our results suggest that VLMs could provide valuable insights regarding sound symbolism. In addition to providing a new perspective on this phenomenon in cognitive science, our work sheds light on what is learned by multimodal models in the context of interpretability in machine learning, demonstrating that these models also encode knowledge over individual characters beyond the semantics of words.
Sound symbolism in language. A wide array of studies in cognitive psychology and linguistics have demonstrated the existence of sound symbolic associations, including the famous kiki-bouba experiment as well as many variations [51, 34, 28, 1, 7, 18, 10, 13, 17, 19]. This effect has been shown to be robust cross-culturally and across various languages and writing systems [4, 3, 14, 8], and even present among infants and toddlers [33, 40]. Corpus analyses of English have suggested that sound symbolic trends are present in the basic lexicon of the language – words with particular meanings may be statistically more likely to contain certain sounds [38, 60, 56]. Interestingly, research on blind individuals has found these associations to be weaker or absent when examined between spoken and tactile modalities, suggesting that visual input plays a key role [20, 22].
Vision-and-language models. Recent years have seen rapid development of foundation models, powerful pretrained models that can be adapted to downstream tasks, largely powered by transformer architectures and pretraining on web-scale data [6]. In the field of multimodal learning, significant foundation models include dual encoder models such as CLIP [45] and its open-source implementation OpenCLIP [11], trained with a contrastive objective on captioned image data. The power of these discriminative models for paired text-image understanding has been further leveraged in recent meteoric development of text-to-image generative models. Large diffusion models such as DALL-E 2 [47] and Imagen [50] set a new standard for general text conditioned image generation. Latent diffusion models such as Stable Diffusion make comparable results attainable with a smaller computational footprint by applying denoising diffusion to the latent space of an image autoencoder [49]. In particular, the classifier-free guidance [23] of Stable Diffusion uses OpenCLIP. In our work, we use OpenCLIP and Stable Diffusion as reference SOTA models for discriminative and generative text-image understanding respectively.
Parallels between VLMs and cognitive science. A number of prior works have examined the types of knowledge learned by artificial neural networks, with connections to learning and knowledge in cognitive science sometimes left implicit and sometimes stated explicitly. In particular, models trained on vision-and-language tasks have been found to encode knowledge about the visual world in ways that have parallels to human cognition. Although unimodal (text-only) language models learn some degree of factual information [44, 48, 12] and commonsense reasoning [59], the inclusion of visual data in training may bolster models with visual commonsense knowledge which is typically not written explicitly in text [58, 29, 26, 25, 61, 2]. Alper et al. [2] explicitly connect this to similar findings in cognitive science, where studies have explored the effect of human vision impairment and blindness on color associations [41, 53, 52, 55, 31]. In addition, Orgad et al. [39] note that text-to-image models in particular display implicit assumptions about the world based on correlations and biases in captioned image datasets.
Our work also explores the knowledge learned by VLMs, but rather than investigating the semantics of words or utterances in text, we focus on the surface form of textual input to these models and show that they have learnt a non-trivial mapping between sounds encoded by the written text and visual semantics. We also note that the surface form of textual input to these models has been explored in the context of typographic attacks [21, 32], in which models learn to associate text with images containing the text written out visually; however, they do not address semantic associations with the surface form of the input beyond its literal appearance as text.
3 Computational Paradigm for Sound Symbolic Probing
To test for the presence of sound symbolism in VLMs, we design a paradigm using controlled comparisons between pseudowords - nonsense words constructed from English letters with desired properties - and visual semantic properties. In particular, we are interested in whether a VLM
associates pseudowords with known sound-symbolic associations among humans (e.g. kiki – sharp) with visual properties such as “sharpness”. For generative (text-to-image) models this translates to testing whether pseudowords containing letters that are known to have “sharp” associations tend to generate sharper images on average, and conversely for pseudowords containing letters with “round” associations. Recent progress in the field of multimodal learning makes it possible to test this directly, as CLIP [45] can be used to test images for semantic properties by comparing them to text prompts (e.g. a 3D rendering of a sharp object) via cosine similarity of embedding vectors in a semantic space shared between the textual and visual modalities. Moreover, CLIP can be tested directly as a discriminative model for sound symbolism by comparing the text embeddings of pseudowords to those of such properties.
This intuition is formalized below. We first describe how pseudowords with ground truth associations are constructed (Section 3.1). We then elaborate on our zero-shot knowledge probing techniques (Section 3.2) and evaluation protocol (Section 3.3).
3.1 Pseudoword Construction
Although studies using particular pseudoword stimuli such as kiki-bouba [46, p.19] or maluma-takete [24, p.133] have found that certain speech sounds show cross-modal associations, the precise nature of which acoustic phenomena give rise to these associations is an active topic of research. McCormick et al. [34] demonstrate that speech sounds form a spectrum from most "sharp" to most "round". Relevant phonetic distinctions include sonority and voicing in consonants, and vowel height and rounding. Fort and Schwartz [19] suggest that these properties can be aligned on the dimensions of "spectral balance and temporal continuity", which have also been found relevant for sharpness associations with non-speech sounds such as drum beats [19] and electronic sound waves [42].
To test for the presence of sound symbolism in the VLMs under consideration, we split speech sounds into "highly sharp" and "highly round" categories based on phonetic properties, leaving an investigation of more fine-grained distinctions within the spectrum of sounds to future research. Following the classification of speech sounds described in [34], we define the following subsets of English graphemes (letters) for use in our experiments, divided between consonants and vowels and split into classes based on their corresponding sounds:
$$
C _ {\text {☆}}: \langle p t k s h x \rangle C _ {O}: \langle b d g m n l \rangle
$$
$$
\mathrm {V} _ {\text {☆}}: \langle e i \rangle \quad \mathrm {V} _ {\mathrm {O}}: \langle o u \rangle \quad \mathrm {V} _ {-}: \langle a \rangle
$$
The classes above use  $\star$  and  $\bigcirc$  as shorthand for "sharp" and "round" associations respectively, and  $V_{-}$  indicates a neutral association (may appear in either class of pseudoword, as described below). The main phonetic distinctions used to determine these classes are voicing for consonants and backness for vowels; see the supplementary material for more information about the phonetic details motivating these classes.
We construct pseudowords for use in our experiments using the three syllable template  $(\mathrm{CV})_1(\mathrm{CV})_2(\mathrm{CV})_1$  , where the first and last syllables are the same (e.g. kitaki, bodubo, ...). In addition, all graphemes must be drawn either from  $C_{\star}\cup V_{\star}\cup V_{-}$  or  $C_{\bigcirc}\cup V_{\bigcirc}\cup V_{-}$  , disallowing forms like kiduki which mix graphemes from the two classes. We denote the set of all pseudowords with graphemes from the former set as  $\Psi_{\star}$  , the set of pseudowords with graphemes from the latter set as  $\Psi_{\bigcirc}$  , and  $\Psi = \Psi_{\star}\cup \Psi_{\bigcirc}$  . The choice of this template has a few motivations, including the large number of possible forms, few clashes with existing English words, and similarity to the maluma-takete stimuli used by Kohler [24, p.133]. Examples of such pseudowords include the following (in arbitrary order):
$$
\Psi_ {\text {☆}}: \quad k i t a k i \quad h a t i h a \quad p e p a p e \quad x i s i x i \quad h i p e h i \quad x a x a x a \quad t e x e t e \quad \dots \tag {324total}
$$
$$
\Psi_ {\bigcirc}: \quad g u g a u \quad b o d u b o \quad g u n o g u \quad d a l u d a \quad m o m o m o \quad l u n u l u \quad g a d a g a \quad \dots \quad (3 2 4 \text {t o t a l})
$$
We also report results for the pseudowords kiki and bouba used verbatim, to reproduce the most well-known form of the kiki-bouba effect with VLMs. In accordance with the principles used to construct our pseudowords, kiki corresponds to class  $\star$  and bouba corresponds to class  $\bigcirc$ .
3.2 Zero-shot Knowledge Probing
We probe VLMs for sound symbolism with a zero-shot linear probe applied to embedding vectors in the multimodal embedding space of CLIP  $(\subset \mathbb{R}^{1024})$ . We consider only the zero-shot regime since we are interested in the inherent knowledge of our models (acquired from pretraining) and not the dynamics of training on new data. Furthermore, our approach tests VLMs end-to-end and is agnostic to the source of these effects with respect to the relative contribution of different model-internal components (such as tokenization and hidden activations).
**Prompts used:** We use the following prompts to probe the models under consideration, where  $\langle w \rangle$  is the item (word or pseudoword) to be inserted into the prompt:
$P_{1}$ : “a 3D rendering of a  $\langle w\rangle$  object”
$P_{2}$  : “a 3D rendering of a  $\langle w\rangle$  shaped object”
We use  $P_{1}$  for adjectives (e.g. ...of a round object) and  $P_{2}$  for nouns and pseudowords (e.g. ...of a cactus shaped object, ...of a kiki shaped object). These prompts are chosen for visual simplicity and interpretability; in the supplementary material we compare results on other prompts.
Embeddings: All of our tests use embedding vectors in CLIP space  $(\subset \mathbb{R}^{1024})$  corresponding to pseudowords and adjectives inserted into prompts. These may either be text embeddings calculated using CLIP's text encoder, or image embeddings using CLIP's vision encoder applied to image generations conditioned on the given text. In either case, we embed pseudowords and adjectives in CLIP space to obtain vectors  $v_{\langle w\rangle},w_{\langle a\rangle}\in \mathbb{R}^{1024}$  respectively and unit-normalize them to  $\hat{v}_{\langle w\rangle},\hat{w}_{\langle a\rangle}$ . When evaluating CLIP we directly encode both as text; when evaluating Stable Diffusion we calculate  $v_{\langle w\rangle}$  using image generation, as detailed further in Section 4.1.
Geometric scores  $\gamma_{\langle w\rangle}$ : We propose a scoring method to measure geometric attributes such as sharpness or roundness, applied to images of objects or embeddings corresponding to pseudowords. To calculate this, we identify the one-dimensional semantic direction of interest in CLIP embedding space aligned with a collection of adjectives, similar to prior work on finding interpretable linear subspaces of word embeddings [35, 5, 16] and multimodal embedding spaces [57, 43]. We manually select 20 ground-truth adjectives, split evenly between those with sharp or round associations which we denote by  $\mathrm{A}_{\hat{\mathcal{S}}}$  and  $\mathrm{A}_0$  respectively. These are as follows:
$\mathrm{A}_{\text{☆}} = \{\text{sharp, spiky, angular, jagged, hard, edgy, pointed, prickly, rugged, uneven}\}$
$\mathrm{A}_{\bigcirc}$  = {round, circular, soft, fat, chubby, curved, smooth, plush, plump, rotund}
Using the embeddings of these adjectives, we construct probe vector  $w_{adj} \coloneqq \sum_{\langle a \rangle \in \mathrm{A}_{\Delta}} \hat{w}_{\langle a \rangle} - \sum_{\langle a \rangle \in \mathrm{A}_{\odot}} \hat{w}_{\langle a \rangle}$ , unit-normalized to  $\hat{w}_{adj}$ . This vector approximates the direction in CLIP space representing the visual semantic dimension of interest; in other words, this is a unit vector pointing in the direction in CLIP's latent space which distinguishes between the two sets of adjectives. We probe item  $\langle w \rangle$  by calculating the score  $\gamma_{\langle w \rangle} \coloneqq \hat{v}_{\langle w \rangle} \cdot \hat{w}_{adj}$ , corresponding to projection onto the 1D subspace spanned by  $w_{adj}$ . Intuitively, the scalar  $\gamma_{\langle w \rangle}$  measures whether the given item is closer to the "round" or "sharp" end of the scale of associations in our model's semantic space. Tangentially, while this work focuses on using these scores to place pseudowords along a sharp-round semantic axis, our geometric scoring method could be applicable for identifying words or images with such associations in more general settings as well.
Phonetic scores  $\phi_{\langle w\rangle}$ : We also propose a complementary scoring method to measure associations with categories of sounds reflected in English spelling. This is applied in order to quantify the phonetic or graphemic associations that our models show with real English words such as the adjectives given above. We construct probe vector  $v_{pw} \coloneqq \sum_{\langle w\rangle \in \Psi_{\hat{\mathcal{A}}}} \hat{v}_{\langle w\rangle} - \sum_{\langle w\rangle \in \Psi_{\circ}} \hat{v}_{\langle w\rangle}$ , unit-normalized to  $\hat{v}_{pw}$ . This is a unit vector which estimates the dimension in CLIP space that best distinguishes between the two ground-truth classes of pseudowords. We then score item  $\langle w\rangle$  via cosine similarity with this probe:  $\phi_{\langle w\rangle} \coloneqq \hat{v}_{pw} \cdot \hat{w}_{\langle w\rangle}$ . Intuitively, the scalar  $\phi_{\langle w\rangle}$  measures whether the given item (e.g., adjective) is closer to pseudowords like  $kitaki_{\hat{\mathcal{A}}}$  or to those like  $bodubo_{\circ}$  in our model's semantic space. We call this "phonetic scoring" because it is a semantic dimension determined solely by the letters used in pseudowords and their underlying sounds.
3.3 Evaluation Method
For quantitative analysis, we evaluate the scores  $\gamma_{\langle w\rangle}$  and  $\phi_{\langle w\rangle}$  using classification metrics;  $\gamma_{\langle w\rangle}$  for predicting the binary class ( $\diamond$  or  $\bigcirc$ ) of pseudowords, and  $\phi_{\langle w\rangle}$  for predicting the binary class ( $\diamond$  or  $\bigcirc$ ) of adjectives with ground-truth labels. As these are unnormalized scores, we use non-probabilistic and threshold-agnostic metrics, namely ROC-AUC and Kendall correlation  $\tau$  between the scores in question and the ground-truth classes as a binary indicator variable. These also have the desirable property of being symmetric with respect to our class labels, neither of which is privileged with respect to the other. Additionally, we provide an analysis of the correlation between  $\gamma_{\langle w\rangle}$  and the particular sounds present in the first syllable of the given pseudoword.
We also reproduce the classic kiki-bouba experiment using the scores  $\gamma_{kiki}$  and  $\gamma_{bouba}$ . To place these scores on an interpretable scale, we use their percentile ranks relative to scores of all pseudowords in  $\Psi$ , and report the difference in these percentiles  $\Delta P_{kb}$ . A high value for this metric indicates that the given model strongly aligns these pseudowords along the semantic axis of roundness and sharpness.
In this section, we present our main findings. We first discuss details of the experimental setup (Section 4.1). We then present quantitative results (Section 4.2), the results of our user study (Section 4.3), and qualitative results (Section 4.4).