SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Further experimental details and results are provided in the supplementary material, including results on additional prompts and model architectures (such as a GAN-based text-to-image model). All of these settings show results consistent with our primary findings on CLIP and Stable Diffusion. There we also provide results for unimodal (text-only) text encoder models, which show mixed results when probed for sound symbolism.
Our primary findings in this work use models trained and prompted with texts in the English language. Nonetheless, in the supplementary material we also probe for sound symbolism in a multilingual text-to-image model with prompts in four geographically and linguistically diverse languages, with positive results. While we do not directly address the universality of sound symbolism across languages, these results warrant further investigation of sound symbolism in multilingual VLMs.
4.1 Experimental Details
We investigate the presence of sound symbolic patterns in two types of VLM: generative text-to-image models as exemplified by Stable Diffusion [49], and discriminative models with dual text and vision encoders as exemplified by CLIP [45]. We use the open-source CLIP implementation OpenCLIP [11] throughout; Stable Diffusion uses OpenCLIP as its text encoder for classifier-free guidance, motivating this comparison. We use these models as-is and probe them in the zero-shot regime, without any further training or adjustment of their tokenizer (which is identical for both models).
We note the a priori possibility that Stable Diffusion's weights may encode additional visual commonsense that is not present in CLIP. The majority of its parameters are in its UNet component (866M vs. 340M in its CLIP text encoder). These weights are trained on an image denoising task which requires understanding local regions in images, while CLIP's pretraining knowledge is only acquired from global image-text matching.
For CLIP, we assign embeddings to pseudowords by inserting them into textual prompts and embedding them with CLIP's text encoder. For Stable Diffusion, we embed pseudowords by generating images using such prompts and embedding the generated images using CLIP's vision encoder. Since the image generation process is stochastic, we reduce variance by generating multiple images with the corresponding prompt and calculating their mean embedding vector. As CLIP embeds text and images in the same space, we evaluate the pseudoword embeddings yielded by both models with the same probing method, as described in Section 3.3.
Model	γ<w>	φ<w>
AUC	τ	ΔPkb	AUC	τ
Stable Diffusion	0.74	0.34	80%	0.97	0.68
CLIP	0.77	0.39	52%	0.98	0.70
(random)	0.50	0.00	0%	0.50	0.00
Table 1: Results of zero-shot linear probing. Results under  $\gamma_{\langle w\rangle}$  indicate metrics for predicting pseudoword class ( $\star$  or  $\bigcirc$ ) from geometric scores; results under  $\phi_{\langle w\rangle}$  indicate metrics for predicting adjective class ( $\star$  or  $\bigcirc$ ) from phonetic scores. Probing methods and evaluation metrics are as described in Section 3. (random) indicates the expected performance of a purely random scoring method, providing a lower bound for the performance of the models under consideration.
Figure 2: Graphemes sorted by average geometric score  $\gamma_{\langle w\rangle}$  for pseudowords  $\langle w\rangle$  whose first syllable contains the given grapheme, calculated with Stable Diffusion and CLIP. Characters are colored based on their ground-truth association (red for  $\star$ , blue for  $\bigcirc$ ). Consonants are shown above and vowels below the arrow. We see that the two classes are mostly well-discriminated by these scores, especially when calculated Stable Diffusion. In this visualization, consonants and vowels are displayed on separate scales and are not positioned absolutely with respect to each other.
We present the results of our quantitative tests in Table 1. Across all metrics, our probing methods are able to predict pseudoword and adjective classes ( $\star$  or  $\bigcirc$ ) significantly better than chance. In particular, the results on Stable Diffusion indicate that images generated from pseudowords in  $\Psi_{\star}$  are more likely to be visually "sharp" and images generated from pseudowords in  $\Psi_{\bigcirc}$  are more likely to be visually "round", consistent with the examples shown in Figure 4. Additionally, we see that the  $\Delta P_{kb}$  scores indicate a replication of the kiki-bouba effect in multimodal models, with kiki having a relatively higher affinity for "sharp" adjectives and the latter for "round" adjectives in such models.
In Figure 2, we see graphemes (letters), split into consonants and vowels and sorted by the average geometric score  $\gamma_{\langle w\rangle}$  of all pseudowords containing them in the first syllable (first two characters). Surprisingly, we see an emergent pattern that closely reflects psychological and phonetic phenomena, as both models perfectly or nearly perfectly differentiate between  $\star$  and  $\bigcirc$ -associated graphemes (as described in Section 3.1). This is despite the fact that these models were trained on text-image pairs on the full caption level, primarily saw valid English text during training, and did not have direct access to the auditory modality at all. We even see intriguing patterns within the intra-class
Figure 3: Ground-truth adjectives sorted by phonetic score  $\phi_{\langle w\rangle}$ , calculated with Stable Diffusion and CLIP. Adjectives are colored based on their ground-truth association (red for  $\star$ , blue for  $\bigcirc$ ). We see that the two classes are highly differentiated by phonetic score for both models, as further reflected in the corresponding metrics in Table 1.
POS	Lowest φw(⇒○)	Highest φw(⇒☆)
Stable Diffusion
Noun	butterball, yolk, pregnancy, booger, eggnog, turnip, bellyful, crybaby, doughboy	shard, kite, origami, hexagon, diamond, flake, octagon, triangle, protractor, lozenge, foldout
Adj.	obese, chubby, stinky, pudgy, overweight, fat, pregnant, plump, drowsy, soggy, squishy	triangular, diagonal, angular, shattering, jagged, rectangular, edgy, housebroken, geometrical
CLIP
Noun	doughboy, loudmouth, gumdrop, boogeyman, madwoman, lord, butterball, goddaughter	prefix, talkativeness, asker, flexibility, shears, shift, peek, slope, task, exit, hemline, tightness
Adj.	muggy, soggy, gloomy, grouchy, lumpy humongous, hungry, cloudless, unsmiling	apelike, flexible, diagonal, static, triangular, external, shipshape, interlocking, angular
Table 2: Real English words sorted by phonetic score  $\phi_{\langle w\rangle}$ . Results are split by part of speech (POS), from the  $\sim 5.5K$  nouns in N and  $\sim 1K$  adjectives in A. Columns indicate items with the lowest and highest scores out of the entire sorted lists, corresponding to more relative similarity to the pseudowords in  $\Psi_{\bigcirc}$  or  $\Psi_{\star}$  respectively. As seen above, these clearly depict "round" and "sharp" characteristics, particularly the words selected with the generative pipeline (i.e. Stable Diffusion).
ordering of graphemes, such as close vowels  $\langle \mathrm{i} \quad \mathrm{u} \rangle$  having more extreme association than close-mid vowels  $\langle \mathrm{e} \quad \mathrm{o} \rangle$  and voiced sonorants  $\langle \mathrm{m} \quad \mathrm{n} \quad 1 \rangle$  having among the most "round" associations, although we leave investigation of such fine-grained patterns to future research.
Figure 3 shows the ground-truth adjectives from  $\mathrm{A}_{\mathcal{L}}$  and  $\mathrm{A}_0$  sorted by phonetic score for both models, corresponding to the  $\phi_{\langle w\rangle}$  metrics in Table 1. The near-perfect discrimination between the two classes under both models is consistent with the high metric values seen in the table.
4.3 User Study
In order to ground our results in human cognition, we conduct a user study using images produced by Stable Diffusion prompted using pseudowords. We adopt the two-alternative forced choice paradigm, where participants are provided with image pairs and corresponding pairs of pseudowords, and are asked to match the pseudowords to the corresponding images. We perform this study in two settings: in the first setting, we test participants' ability to distinguish between images generated using the specific pseudowords kiki and bouba; in the second setting, we test their ability to distinguish between images generated using random pseudowords from  $\Psi_{\star}$  and  $\Psi_{\bigcirc}$ . One hundred subjects participated in the first setting, and seventy-five participated in the second setting. Participants provided correct answers with overall accuracy of  $73\%$  in the kiki-bouba setting and  $55\%$  in the general pseudoword setting. In order to determine the statistical significance of these results while accounting for variation between subjects and items within each survey, we adopt a mixed-effects logistic regression model. We regress whether a question is answered correctly while treating question identity as a fixed effect and respondent identity as a random effect. Analysis of the first setting (kiki vs. bouba) results in an intercept estimate corresponding to a  $89\%$  overall success probability ( $p < 0.001$ ); in the second setting (random pseudowords) this results in an intercept estimate corresponding to a  $78\%$  overall success probability ( $p < 0.001$ ). By isolating the overall success rate from the effects of individual respondents and questions, our mixed-effects model results indicate that the sound symbolism exhibited by our text-to-image model correlates with human sound symbolic associations (with a stronger effect size in the simpler setting, but having a significant effect in both settings). Please refer to the supplementary material for additional details, including participant sourcing and compensation, the full text of instructions and survey examples, and a full statistical analysis of survey results.
We provide a qualitative analysis using real words from English, filtered to select for basic words and split by part of speech:  $\sim 5.5K$  nouns denoted by N, and  $\sim 1K$  adjectives denoted by A. We select basic concrete words for both categories by using lemmas from WordNet [36, 37], filtering out
High  $\gamma \langle w\rangle$ $(\Rightarrow \star)$
tepite☆
kataka☆
gugugu
pitapi☆
daguda
setese☆
Low  $\gamma_{\langle w\rangle}$ $(\Rightarrow \bigcirc)$
nubanu
gomago
magama
bomabo
mamomaa
lobalo
Figure 4: Image generations for pseudowords with high (top  $20\%$ ) and low (bottom  $20\%$ ) geometric scores. We visualize random selections of pseudoword-image pairs for each category. Pseudowords with class ( $\star$  or  $\bigcirc$ ) that does not match its geometric score are indicated in red. As seen above, the shapes of the generated images noticeably correlate with the pseudoword class.
donudoO (doughnut)
Figure 5: Images generated from pseudowords reminiscent of real English words. For each pseudoword we display an associated image generation and the automatically detected closest English word. Pseudowords with high or low geometric scores (in top or bottom  $20\%$  relative to all pseudowords) which do not match their class ( $\star$  or  $\bigcirc$ ) are indicated in red.
$\text{banaba}_\text{O}$  (banana)
hexahe (hexagon)
balobaO (blob)
bamobaO (bamboo)
pipepi (pipe)
dogado (doggy)
obscure or highly abstract items using age of acquisition and concreteness scores from [27, 9]. We sort these items by their  $\phi_{\langle w\rangle}$  scores and display the words with the highest and lowest scores.
Qualitative results are shown in Table 2 and Figure 4. In Table 2, we see a striking pattern that the head and tail of this list represent adjectives which intuitively describe visual properties that are more "sharp" or "round". In other words, these properties are highly correlated with relative similarity to the two pseudoword classes  $\bigcirc$  and  $\star$ . This strengthens our quantitative results for ranking ground-truth adjectives with phonetic scores  $\phi_{\langle w\rangle}$ , as we see these scores are truly aligned with visual semantic properties for the VLMs under consideration (particularly for Stable Diffusion). The images in Figure 4 show a similarly striking pattern, with the sharper objects (as measured by  $\gamma_{\langle w\rangle}$  score) being more commonly generated by pseudowords from  $\Psi_{\star}$  and rounder objects by pseudowords from  $\Psi_{\bigcirc}$ .
Pseudowords that resemble real English words may generate images reminiscent of real objects, as seen in Figure 5. There we display such pseudowords along with close English words, detected automatically via an automatic search heuristic (described in the supplementary material) combining fuzzy string matching and CLIP text-image similarity. This suggests a possible correlation between sounds in real English words and visual semantics as explored in the psycholinguistic literature [38, 60, 56].
5 Discussion, Limitations and Future Work
We have shown that sound symbolism is reflected in VLMs by evaluating them on pseudowords with controlled phonetic properties. By comparing pseudowords built from known "sharp" or "round" speech sounds, we find that these models learn associations with corresponding sharp and round adjectival properties, parallel to the classic kiki-bouba experiment in psychology.
Further strengthening these findings, we provide strong evidence in the supplementary material that these models have not learned specifically from items illustrating the kiki-bouba effect, by showing that this concept is not well-represented in the LAION dataset [54] upon which our VLMs were
trained, and by showing that Stable Diffusion does not generate coherent content from prompts referring to the experiment itself. Rather, it appears that our VLMs have learned sound patterns from their training data via large-scale correlations between sounds and shapes. Future work could further explore complex generalization patterns and biases learned by VLMs, associating non-obvious visual or semantic meanings with particular input patterns.
Our findings have significance both in the field of multimodal machine learning as well as in psycholinguistics. From a computational perspective, our work reveals an emergent behavior in VLMs that has been hitherto unexplored, to the best of our knowledge. In general, VLMs are being used as black boxes without a full understanding of how they understand the visual semantics of language. Our work sheds light on how these models interpret and respond to language, as well as raising questions about how sound symbolic associations are inferred from our models' training datasets, whether additional dimensions of sound symbolic meaning are present in these models apart from our findings, and whether similar associations could exist with other modalities beyond vision.
From a cognitive science perspective, our work provides a new perspective to the extensive literature on sound symbolism, which has long been a topic of interest and debate. If VLMs learn sound symbolic association from valid text in caption data, this may be due to the presence of sound symbolism as reflected in the basic lexicon of English (or other languages), and the presence of sound symbolism in language itself is moderately controversial (notably denied by Ferdinand de Saussure as discussed in Section 1). In this vein, investigation into how VLMs infer sound symbolic associations from their training data could potentially shed light on how sound symbolism is learned during human language acquisition. Additionally, these methods could provide new insights into the classic questions of what aspects of sound are tied to meaning and to what extent the observed phenomena are culturally universal or specific to the English language. Our work provides a new line of inquiry, suggesting that VLMs and their training datasets should be researched carefully for further insight.
Regarding limitations of our findings, we do not assume that these models directly imitate human cognition, although our results do show that sound symbolic patterns are present in captioned image data with a strong parallel to the psychology of language and vision. Additionally, we do not answer why these associations are present in language or exactly how they are learned from the given data. In particular, we note that our results are agnostic to whether these models have memorized associations between particular letters and shapes, and to whether they have a deeper understanding of the phonetic or acoustic properties implied by these letters. We leave investigation of these questions to future work, and foresee them providing an important direction for research into sound symbolism in human cognition and machine learning.
This work was partially supported by the Alon Fellowship. We thank Gal Fiebelman and Taelin Karidi for their helpful feedback.
[1] James S. Adelman, Zachary Estes, and Martina Cossu. "Emotional sound symbolism: Languages rapidly signal valence via phonemes". In: Cognition 175 (June 1, 2018), pp. 122-130. ISSN: 0010-0277. DOI: 10.1016/j.cognition.2018.02.007. URL: https://www.sciencedirect.com/science/article/pii/S0010027718300374 (visited on 12/18/2022).
[2] Morris Alper, Michael Fiman, and Hadar Averbuch-Elor. "Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[3] Damián Blasi et al. "Sound-meaning association biases evidenced across thousands of languages". In: Proceedings of the National Academy of Sciences of the United States of America 113 (Sept. 12, 2016). DOI: 10.1073/pnas.1605782113.
[4] Damián E. Blasi et al. "Sound-meaning association biases evidenced across thousands of languages". In: Proceedings of the National Academy of Sciences of the United States of America 113.39 (Sept. 27, 2016), pp. 10818-10823. ISSN: 0027-8424. DOI: 10.1073/pnas.1605782113. URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5047153/ (visited on 12/26/2022).
[5] Tolga Bolukbasi et al. "Man is to computer programmer as woman is to homemaker? debiasing word embeddings". In: Advances in neural information processing systems 29 (2016).
[55] Roger N Shepard and Lynn A Cooper. "Representation of colors in the blind, color-blind, and normally sighted". In: Psychological science 3.2 (1992), pp. 97-104.
[56] David M Sidhu et al. "Sound symbolism shapes the English language: The maluma/takete effect in English nouns". In: Psychonomic Bulletin & Review 28 (2021), pp. 1390-1398.
[57] Yoad Tewel et al. "Zero-shot image-to-text generation for visual-semantic arithmetic". In: arXiv preprint arXiv:2111.14447 (2021).
[58] Ramakrishna Vedantam et al. "Learning common sense through visual abstraction". In: Proceedings of the IEEE international conference on computer vision. 2015, pp. 2542-2550.
[59] Peter West et al. "Symbolic knowledge distillation: from general language models to commonsense models". In: arXiv preprint arXiv:2110.07178 (2021).
[60] Bodo Winter and Marcus Perlman. "Size sound symbolism in the English lexicon". In: Glossa: a journal of general linguistics 6.1 (2021).
[61] Chenyu Zhang et al. "Visual Commonsense in Pretrained Unimodal and Multimodal Models". In: arXiv preprint arXiv:2205.01850 (2022).