Kiki or Bouba? Sound Symbolism in Vision-and-Language Models
Authors: Morris Alper
Hadar Averbuch-Elor
Venue: NeurIPS 2023 spotlight (2023)
Keywords: multimodal learning
computer vision
NLP
cognitive science

Abstract: Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings across languages and demographic groups, a phenomenon known as sound symbolism. Among the many dimensions of meaning, sound symbolism is particularly salient and well-demonstrated with regards to cross-modal associations between language and the visual domain. In this work, we address the question of whether sound symbolism is reflected in vision-and-language models such as CLIP and Stable Diffusion. Using zero-shot knowledge probing to investigate the inherent knowledge of these models, we find strong evidence that they do show this pattern, paralleling the well-known kiki-bouba effect in psycholinguistics. Our work provides a novel method for demonstrating sound symbolism and understanding its nature using computational tools. Our code will be made publicly available.

Kiki or Bouba? Sound Symbolism in Vision-and-Language Models
Morris Alper and Hadar Averbuch-Elor Tel Aviv University
Abstract
Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings across languages and demographic groups, phenomenon known sound symbolism. Among the many dimensions of meaning, sound symbolism particularly salient and well- demonstrated with regards to cross-modal associations between language and the visual domain. In this work, we address the question of whether sound symbolism is reflected in vision-and lan---nnd-- models such CLIP and Stable Diffusion. Using zero-shot knowledge probing to investigate the inherent knowledge of these models, find strong evidence that they do show this pattern, paralleling the well-known kiki-bouba effect in psycholinguistics. Our work provides a novel method for demonstrating sound symbolism and understanding its nature using computational tools. Our code will be made publicly available"
1 Introduction
"What's in a name? That which we call a rose by any other name would smell as sweet."
-William Shakexpeare, Romeo and Juliet
Philosophers of language have long debated whether the mapping between sound and meaning in speech is arbitrary. Discussions this topic date back to the Socratic dialogues of Plato, well as early modern philosophers such as John Locke and Gottfried Wilhelm Leibniz (30]. Charles de Saussure, the seminal early 20th century linguist and semiotician, famously stated that le signe est arbitraire In Saussure's ' view, words are simply arbitrary conventions and their sounds have no inherent connection to their meanings; hence the French word chien and its English translation dog both equally denote the same animal despite sharing no sounds common [5]..
Although the concept of the arbitrariness of the sign was influential on modern linguistics, there are many evident cases where does not hold which have attracted great interest among modern researchers. Cases of sound symbolism in language, where the sounds themselves in a word have some connection to what they describe, include onomatopoeic phrases such as kapow (a punch or banging sound) and glub-glub (water bubbling). Beyond direct imitation, it has been noted that English and other languages show correlations between certain phonetic structures and types of meaning, such as the array of English words beginning with denoting brittleness (crush, crunch. crash, crack, crackle, crinkle, ...). This raises natural questions including how universal these patterns of iconicity between languages and cultures, whether they rooted in psychological tendencies that influence language or vice versa, what acoustic properties of speech sounds influence iconicity, and the extent which less obvious iconic patterns shape language in general.
'via our project page https://kikii
github.iii io/
Pieh sign is arbitrary"
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Original Experiment
Random Image Generations
Figure I: Illustration of the kiki-bouba effect. The shapes on the far left illustrate stimuli used in the classic kiki-bouba experiment. The remaining images are random generations from Stable Diffusion with the prompt a 3D rendering of (w) shaped object, where (w) (kiki, bouba). Which of of these images do you think were generated using pseudoword kiki and which with bouba? See below for the answer.
Perhaps the most well-known and conclusively demonstrated examples of sound symbolism the kiki-boubi effect. In this experiment, subjects shown spiky object and round object, such as the left-hand shapes in Figure I When asked to assign one the name kiki and the other the name bouba, subjects show overwhelming preference (see footnote for the preferred matching). This effect has been shown for both auditory stimuli [19] and for written text (which implicitly maps to speech sounds) [13 10 [4]. Furthermore, array of studies have demonstrated it to hold across various speech sounds and constructed pseudowords (beyond kiki and bouba) (34), between different languages and cultures (3), and even among prelingual infants (40). These findings challenge the
Saussurean assumption that sound maps purely arbitrarily to meaning.
In another domain, recent years have seen explosive progress in the field of machine learning applied to natural language and vision, mainly powered by transformer neural networks and training on web-scale datasets of captioned images (6). This includes discriminative models such as CLIP [45
and generative text-to-image models such as Stable Diffusion [49]. Although these models have revolutionized many applied tasks, they largely used black box. One line of research examines the emergent behavior and inner structure neural networks in order to interpret their behavior [48) and several works investigate the visual knowledge encoded by vision-and- and-language models (VLMs) in the context of human cognition and visual semantics in language [2] [1]. Given these parallels along with fundamental differences between these models and the human brain, it is natural to ask whether they have also learned to associate sounds in language (as reflected in their written representations) with visual semantics. In other words, does the iision-and-languag setting reflect the presence of sound symbolic patterns in language? If so, this would provide strong evidence against Saussure's 's legendary claim from a novel computational perspective.
As seen in Figure generative text-to-image models can be applied to text containing pseudowords such as kiki bouba. Consider the various images generated by Stable Diffusion shown the hanht-hand side of the figure. The round objects were generated with one of these pseudowords and the sharp objects that the model associates each pseudoword with visual semantic properties such as roundness and sharpness. It is not obvious priori that VLMs would learn associations with text input that is seemingly not well-formed; this suggestive parallel to human cognition invites methodical study. In this work, analyze this phenomenon rigorously with quantitative as well as qualitative criteria, including generalizing to larger set of constructed pseudowords reflecting many different speech sounds in English, and examining correlations between form and meaning on the grapheme (letter) as well as whole-wor word level.
generated with the other (see below for the correct mapping). suggesting
To test for sound symbolism in VLMs, use zero-shot knowledge probing which allows for evaluating the models' inherent knowledge-i.e, not new knowledge acquired during further training. We leverage the ability of CL.IP to embed text and image data in shared semantic space in order probe discriminative and generative (text-to-image) models with the same evaluation metrics. Our tests evaluate whether these models encode pseudowords similarly humans with respect to known symbolic associations, comparing them to adjectives indicating properties related to "sharpness" and "roundness". To further ground our results human cognition, we also conduct user study testing the ability of subjects to reconstruct pseudowords used to condition text-to-image generations. Our results demonstrate that sound symbolism can indeed be observed in VLMs; the models under
3 From left to right: bouba, kiki, bouba, kiki, kiki, bouba
"The vast majority of subjects prefer the name kiki for the spiky object and bouba for the round object.

---

consideration associate visual meanings with the written forms of particular speech sounds. Although we do not claim answer exactly how this is learned by these models, our results suggest that VLMs could provide valuable insights regarding sound symbolism. In addition to providing new perspective on this phenomenon in cognitive science, our work sheds light on what learned by multimodal models in the context of interpretability in machine learning, demonstrating that these models also encode knowledge over individual characters beyond the semantics of words.
2 Related Work
Sound symbolism in language. A wide array of studies in cognitive psychology and linguistics have demonstrated the existence of sound symbolic associations, including the famous kiki kiki-bouba
experiment well as many variations [5T 34
28
18
TO 13
17 19). This effect has been shown
to be robust cross-culturally and across various languages and writing systems 4 3 g] and even present among infants and toddlers [33
14
Corpus analyses of English have suggested that sound symbolic trends are present in the basic lexicon of the language words with particular meanings may be statistically more likely to contain certain sounds (38, 60 [56]. Interestingly, research blind individuals has found these associations to be weaker or absent when examined between spoken and tactile modalities, suggesting that visual input plays key role [20
40
Vision-and -language models. Recent years have seen rapid development of foundation models, powerful pretrained models that can be adapted downstream tasks, largely powered by transformer architectures and pretraining on web-scale data [6]. In the field of multimodal learning, significant foundation models include dual encoder models such as CLIP (45 and its open-source implemen- tation OpenCLIP [1T). trained with a contrastive objective on captioned image data. The power of these discriminative models for paired text-image understanding has been further leveraged in recent meteoric development of text-to-image generative models. Large diffusion models such as DALL-E 2 [47] and Imagen (50) set new standard for general text conditioned image generation. Latent diffusion models such as Stable Diffusion make comparable results attainable with smaller compu- tational footprint by applying denoising diffusion to the latent space of an image autoencoder (49) In particular, the classifier-fret free guidance (23) of Stable Diffusion uses OpenCLIP. In our work, we use OpenCLIP and Stable Diffusion as reference SOTA models for discriminative and generative text-image understanding respectively.
Parallels between VLMs and cognitive science. A number of prior works have examined the types of knowledge learned by artificial neural networks, with connections to learning and knowledge in cognitive science sometimes left implicit and sometimes stated explicitly. In particular, models trained on vision-and-langune language tasks have been found to encode knowledge about the visual world in ways that have parallels to human cognition. Although unimodal (text-only) language models learn some degree of factual information [4 48) 12) and commonsense reasoning [59]. the inclusion of visual data in training may bolster models with visual commonsense knowledge which is typically
not written explicitly in text (58) 29 26 25 61 1 2]. Alper et al. (2) explicitly connect this to similar
findings in cognitive science, where studies have explored the effect of human vision impairment
and blindness on color associations [41 53 52 55 31]. In addition, Orgad et al. (39) note that
text-to-image models in particular display implicit assumptions about the world based on correlations
and biases in captioned image datasets.
Our work also explores the knowledge learned by VLMs, but rather than investigating the semantics of words or utterances in text, we focus on the surface form of textual input to these models and show that they have learnt non-trivial mapping between sounds encoded by the written text and visual semantics. We also note that the surface form of textual input to these models has been explored in the context of typographic attacks [21 32), in which models learn to associate text with images containing the text written out visually: however, they do not address semantic associations with the surface form of the input beyond its literal appearance as text.
3 Computational Paradigm for Sound Symbolic Probing
To test for the presence of sound symbolism in VLMs, we design paradigm using controlled comparisons between pseudowords nonsense words constructed from English letters with desired properties and visual semantic properties. In particular, we are interested in whether VLM
3

---

associates pseudowords with known sound-symbolis associations among humans (e.g. kiki - sharp) with visual properties such as "sharpness". For generative (text-to-image) models this translates to testing whether pseudowords containing letters that are known to have "sharp" associations tend to generate sharper images on average, and conversely for pseudowords containing letters with "round" associations, Recent progress in the field of multimodal learning makes it possible to test this directly, as CLIP [45] can be used to test images for semantic properties by comparing them to text prompts (e.g. 3D rendering of sharp object) via cosine similarity of embedding vectors in a semantic space shared between the textual and visual modalities. Moreover, CLIP can be tested directly as a discriminative model for sound symbolism by comparing the text embeddings of pseudowords to those such properties.
This intuition is formalized below. are constructed (Section 3.1) We then elaborate on our zero-shot knowledge probing techniques (Section 3.2) and evaluation protocol (Section 3.3
We first describe how pseudowords with ground truth associations
3.1 Pseudoword Construction
Although studies using particular pseudoword stimuli such as kiki-bouba [46] 1.] maluma- takete [24] p.133] have found that certain speech sounds show cross-modal associations, the precise nature of which acoustic phenomena give rise these associations is an active topic of research. McCormick et al. (34] demonstrate that speech sounds form a spectrum from most "sharp" to most "round". Relevant phonetic distinctions include sonority and voicing in consonants, and vowel height and rounding. Fort and Schwartz [19] suggest that these properties can be aligned on the dimensions of "spectral balance and temporal continuity", which have also been found relevant for sharpness associations with non-speech sounds such drum beats [19] and electronic sound waves [42]
To test for the presence of sound symbolism in the VLMs under consideration, we split speech sounds into "highly sharp" and "highly round" categories based on phonetic properties, leaving an investigation of more fine-grained distinctions within the spectrum of sounds future research. Following the classification of speech sounds described in [34], we define the following subsets of English graphemes (letters) for use in our experiments, divided between consonants and vowels and split into classes based on their corresponding sounds:
C+ (ptkshx) (e 1)
Co: (b dbdgm11> Vo:
(o u>
V=: (a)
The classes above use A and as shorthand for "sharp" and "round" associations respectively, and V_ indicates a neutral association (may appear in either class of pseudoword, as described below). The main phonetic distinctions used determine these classes are voicing for consonants and backness for vowels; see the supplementary material for more information about the phonetic details motivating these classes.
We construct pscudowords for use in our experiments using the three syllable template (CV)(())))))) where the first and last syllables are the same (e.g. kitaki, bodubo, In addition, all graphemes must be drawn either from CVvVVuuV or CoUVoUV. disallowwig forms like kiduki which mix graphemes from the two classes. We denote the set of all pseudowords with graphemes from the former set as V + the of pseudowords with graphemes from the latter set Vo, and V V U Vo- The choice of this template has few motivations, including the large number of possible forms, few clashes with existing English words, and similarity to the maluma- takete stimuli used by Köhler 24 P.133J. Examples of such pseudowords include the following (in
arbitrary order):
L V kitaki
hatiha bodubo
pepape
xisixi
hipehi
xaxaxa
texete
(324 total)
Vo:
gugagu
gunogu
daluda
momomo
lunulu
gadaga
(324 total)
We also report results for the pseudowords kiki and bouba used verbatim, to reproduce the most well-known form of the kiki-bouba effect with VLMs. In accordance with the principles used to construct our pseudowords, kiki corresponds to class 1 and bouba corresponds to class O.

---

3.2 Zero-shot Knowledge Probing
We probe VLMs for sound symbolism with a zero-shot linear probe applied to embedding vectors in the multimodal embedding space of CLIP (= I2 We consider only the zero-shot regime since we are interested in the inherent knowledge of our models (acquired from pretraining) and not the dynamics of training on new data. Furthermore, our approach tests VL.Ss end-to-end and is agnostic to the source of these effects with respect to the relative contribution of different model-internal components (such as tokenization and hidden activations).
Prompts used: We use the following prompts to probe the models under consideration, where (w) is the item (word or pseudoword) to be inserted into the prompt:
Pt: "a 3D rendering of a (w) object"
P2: "a 3D rendering of a (w) shaped object"
We use Pi for adjectives (e.g. .f.of a round object) and P2 for nouns and pseudowords (e.g. of cactus shaped object,...ff a kiki shaped object). These prompts are chosen for visual simplicity and interpretability; in the supplementary material we compare results on other prompts.
Embeddings: All of our tests use embedding vectors in CLIP space (c Ro corresponding to pseudowords and adjectives inserted into prompts. These may either be text embeddings calculated using CLIP's text encoder, image embeddings using CLIP's vision encoder applied to image generations conditioned on the given text. In either case, we embed pseudowords and adjectives in CLIP space to obtain vectors 2(n)), W(n) E RR 1024 respectively and noi-noorallize them to (w) (a)- When evaluating CLIP we directly encode both as text; when evaluating Stable Diffusion we calculate Plu) using image generation, as detailed further in Section
Geometric scores ?ws: We propose a scoring method to measure geometric attributes such as sharpness or roundness, applied to images of objects or embeddings corresponding to pseudowords. To calculate this, we identify the one-dimensional semantic direction of interest in CLIP embedding space aligned with a collection of adjectives, similar to prior work on finding interpretable línear
subspaces of word embeddings [35] S [6] and multimodal embedding spaces (57,43) We manually
select 20 ground-truth adjectives, split evenly between those with sharp or round associations which
we denote by A and Ao respectively. These are as follows:
A {sharp, spiky, angular, jagged, hard, edgy, pointed, prickly, rugged, uneven) {round, circular, soft, fat, chubby, curved, smooth, plush, plump, rotund)
Ao
Using the embeddings of these adjectives, we construct probe vector Wodj := E(ajeA (aa))
E(ajeAo w)a)). unit-normalized to Dadj. This vector approximates the direction in CLIP space rep- resenting the visual semantic dimension of interest; in other words, this is unit vector pointing in the direction in CLIP's latent space which distinguishes between the two sets of adjectives. We
probe item (w) by calculating the score T(m)= == ((w)) -Wady. corresponding to projection onto the 1D) subspace spanned by Wadj- Intuitively, the scalar Mur) measures whether the given item is closer to the "round" or "sharp" end of the scale of associations in our model's I's semantic space. Tangentially, while this work focuses using these scores to place pseudowords along sharp-round semantic axis, our geometric scoring method could be applicable for identifying words or images with such associations in more general settings well.
Phonetic scores (wn) We also propose a complementary scoring method to measure associations with categories of sounds reflected in English spelling. This is applied in order to quantify the phonetic or graphemic associations that our models show with real English words such as the adjectives given above. We construct probe vector Upw = E(w)e 2t ((w) Enwtew ((w): unit-normalized to upm This
a unit vector which estimates the dimension in CLIP space that best distinguishes between the two ground-truth classes of pseudowords. We then score item (w) via cosine similarity with this probe: ((m) := trw (.))). Intuitively, the scalar (ww) measures whether the given item (e.g., adjective) is closer to pscudowords like kitaki to those like bodubo in our model's semantic space. We call this "phonetic scoring" because it is semantic dimension determined solely by the letters used in pseudowords and their underlying sounds.
5

---

3.3 Evaluation Method
For quantitative analysis, we evaluate the scores Ywo and ((n)) using classification metries; Yw) for predicting the binary class (I77 or () pseudowords, and ((w) for predicting the binary class (57 or O) of adjectives with ground-truth labels. As these are unnormalized scores, we use probbabbbiiistcc and prreshold-agnostee metrics, namely ROC-AUC and Kendall correlation T between the scores in question and the ground-truth classes binary indicator variable. These also have the desirable property of being symmetric with respect to our class labels, neither of which is privileged with respect to the other. Additionally, we provide an analysis of the correlation between Yu)) and the particular sounds present the first syllable the given pseudoword.
We also reproduce the classic kiki-boubu experiment using the scores "Add and Thouba- To place these scores on interpretable scale, V, and report the difference in these percentiles AP4s- A high value for this metric indicates that the given model strongly aligns these pseudowords along the semantic axis of roundness and sharpness.
their percentile ranks relative scores of all pseudowords in
4 Results and Evaluation
In this section, we present our main findings. We first discuss details of the experimental setup (Section [..]). We then present quantitative results (Section 4.2p the results of our user study (Section 4.3). and qualitative results (Section 4.4).
Further experimental details and results are provided in the supplementary material, including results on additional prompts and model architectures (such as GAN-based text-to- -i--tee-magee model). All of these settings show results consistent with our primary findings on CLIP and Stable Diffusion. There we also provide results for unimodal (text-only) text encoder models, which show mixed results when probed for sound symbolism.
Our primary findings in this work use models trained and prompted with texts in the English language. Nonetheless, in the supplementary material also probe for sound symbolism in multilingual text-to-image model with prompts in four geographically and linguistically diverse languages, with positive results. While we do not directly address the universality of sound symbolism across languages, these results warrant further investigation of sound symbolism in multilingual VLMs.
4.1 Experimental Details
We investigate the presence of sound symbolic patterns in types of VLM: generative text-to- image models as exemplified by Stable Diffusion [49] and discriminative models with dual text and vision encoders as exemplified by CLIP 45) We use the open-source CLIP implementation
OpenCLIP [I] throughout; Stable Diffusson uses OpenCLIP as its text encoder for classifier-fre guidance, motivating this comparison. We use these models as-is and probe them in the zero-shot regime, without any further training or adjustment of their tokenizer (which is identical for both models).
We note the a priori possibility that Stable Diffusion's 's weights may encode additional visual common- sense that is not present in CLIP. The majority of its parameters are in its UNet component (866M vs. 340M in its CLIP text encoder). These weights are trained on an image denoising task which requires understanding local regions in images, while CLIP's pretraining knowledge is only acquired from global image-text matching.
For CLIP, we assign embeddings to pseudowords by inserting them into textual prompts and embed- ding them with CLIP's text encoder. For Stable Diffusion, we embed pseudowords by generating images using such prompts and embedding the generated images using CLIP's ss vision encoder. Since the image generation process is stochastic, we reduce variance by generating multiple images with the corresponding prompt and calculating their mean embedding vector. As CLIP embeds text and images in the same space, we evaluate the pseudoword embeddings yielded by both models with the same probing method, as described in Section 3.3

---

7(w)
P(w)
Model
AUC
T
APkb
AUC
7
Stable Diffusion
0.74
0.34
80%
0.97
0.68
CLIP
0.77
0.39
52%
0.98
0.70
(random)
0.50
0.00
0%
0.50
0.00
Table 1: Results of zero-shot linear probing. Results under 7w) indicate metrics for predicting pseudoword class (57r or O) from geometric scores; results under ((w)) indicate metrics for predicting adjective class (177 or O) from phonetic scores. Probing methods and evaluation metrics are as described in Section 3 (random) indicates the expected performance of purely random scoring method, providing a lower bound for the performance of the models under consideration.
x h xhtkpsgdinm
x <hktgspndiDs
12
0
Stable Diffusion
CLIP
Figure 2: Graphemes sorted by average geometric score Yio for pseudowords (w) whose first syllable contains the given grapheme, calculated with Stable Diffusion and CLIP. Characters are colored based on their ground-truth association (red for 17, blue for O). Consonants are shown above and vowels below the arrow. We see that the two classes are mostly well-discriminater by these scores, especially when calculated Stable Diffusion. In this visualization, consonants and vowels are displayed on separate scales and are not positioned absolutely with respect to each other.
4.2 Quantitative Evaluation
We present the results of our quantitative tests in Table Across all metrics, our probing methods able to predict pseudoword and adjective classes (i²r O) significantly better than chance. In particular, the results on Stable Diffusion indicate that images generated from pscudowords in V tz are more likely to be visually "sharp" and images generated from pseudowords in Vo are more likely be visually "round", consistent with the examples shown in Figure Additionally, we see that the Aess scores indicate a replication of the tiki-bouba effect in multimodal models, with kiki having relatively higher affinity for "sharp" adjectives and the latter for "round" adjectives in such models.
In Figure 2 we see graphemes (letters), split into consonants and vowels and sorted by the average geometric score Y(w) of all pseudowords containing them in the first syllable (first two characters). Surprisingly, we see an emergent pattern that closely reflects psychological and phonetic phenomena, as both models perfectly (as described in Section 3.1 D. This is despite the fact that these models were trained on text-image pairs the full caption level, primarily saw valid English text during training, and did not have direct access to the auditory modality at all. We even see intriguing patterns within the intra-class
nearly perfectly differentiate between x and O-associated graphemes
Aogntts Jerport edpor porlate colloted staarg aps
cur curveed
Lake
aaloet Fl riinstor
pesidely bardd aoid
dee pord connbboy
Stable Diffusion
CLIP
Figure 3: Ground-truth adjectives sorted by phonetic score ((w), calculated with Stable Diffusion
and CLIP. Adjectives are colored based on their ground-truth association (red for 'A, blue for O) We that the two classes are highly differentiated by phonetic score for both models, further reflected the corresponding metrics in Table I
7

---

POS
Lowest P(u)) (= O)
Highest (ww) (= +
Stable Diffusson
Noun
butterball, yolk, pregnancy. booger, eggnog, nurnip, bellyful, crybaby, doughboy
shard, kite, origami, hexagon, diamond, flake, octagon, triangle, protractor, lozenge, foldout
Adj.
obese, chubby, stinky, pudgy, overweight, fat, pregnant, plump, drowsy, soggy, squishy
triangular, diagonal, angular, shattering, jagged, rectangular, edgy, housebroken, geometrical
CLIP
Noun
doughboy, loudmouth, gumdrop, boogeyman, madwoman, lord, butterball, goddaughter
prefix, talkativeness, asker, flexibility, shears, shift, peek, slope, task, exit, hemline, fightness
Adj-
muggy, soggy, gloomy, grouchy, lumpy humongous. hungry, cloudless, unsmilling
fleiile, diagonal, static, triangular, external, shipshape, interlocking, angular
Table 2:
Real English words sorted by phonetic score (w)) Results are split by part of speech (POS), from the ~5.5K nouns in N and K adjectives in A. Columns indicate items with the lowest and highest scores out of the entire sorted lists, corresponding to more relative similarity to the pseudowords in Vo or V respectively. As seen above, these clearly depict "round" and "sharp" characteristics, particularly the words selected with the generative pipeline (i.e. Stable Diffusion).
ordering of graphemes, such as close vowels {i u) having more extreme association than close-mid vowels (e o) and voiced sonorants (m 1> having among the most "round" associations, although we leave investigation of such fine-grand grained patterns to future research.
Figure 3 shows the ground-truth adjectives from A and Ao sorted by phonetic score for both models, corresponding to the &ie metrics in Table I The near-perfect discrimination between the two classes under both models consistent with the high metric values seen in the table.
4.3
User Study
In order to ground our results in human cognition, we conduct user study using images produced by Stable Diffusion prompted using pscudowords. We adopt the two-alternative forced choice paradigm, where participants are provided with image pairs and corresponding pairs of pseudowords, and are asked to match the pseudowords to the corresponding images. We perform this study in two settings: in the first setting, we test participants' ability to distinguish between images generated using the specific pseudowords kiki and bouba; in the second setting, we test their ability to distinguish between images generated using random pseudowords from V& and Vr. One hundred subjects participated in the first setting, and seventy five participated in the second setting. Participants provided correct answers with overall accuracy of 73% in the kiki-bouba setting and 55% in the general pseudoword setting. In order to determine the statistical significance of these results while accounting for variation between subjects and items within each survey, we adopt a mixed-effects logistic regression model. We regress whether question is answered correctly while treating question identity a fixed effect and respondent identity as a random effect. Analysis of the first setting (kiki vs. bouba) results in intercept estimate corresponding to 89% overall success probability (p < 0.001); in the second setting (random pseudowords) this results in an intercept estimate corresponding to a 78% overall success probability (p 0.001). By isolating the overall success rate from the effects of individual respondents and questions, our mixed-effects model results indicate that the sound symbolism exhibited by our text-to-image model correlates with human sound symbolic associations (with a stronger effect size in the simpler setting, but having a significant effect in both settings). Please refer the supplementary material for additional details, including participant sourcing and compensation, the full text of instructions and survey examples, and full statistical analysis of survey results.
4.4 Qualitative Results
We provide a qualitative analysis using real words from English, filtered to select for basic words and split by part of speech: ~5.5K nouns denoted by N, and ~1 K adjectives denoted by A. We select basic concrete words for both categories by using lemmas from WordNet [36] 37]. filtering out
8

---

High Mu)
(=
tepite-jz
kataka-
gugugu
pitapi
dagudao
setese +
Low ((w)
(= O)
nubanuo
gomagoo
magamao
bomaboo
mamonaa
lobaloo
Figure Image generations for pseudowords with high (top 20 %) and low (bottom 20%) geometric scores. We visualize random selections of pseudoword- imade pairs for each category. Pseudowords with class (57 or O) that does not match its geometric score are indicated in red. As seen above, the shapes of the generated images noticeably correlate with the pscudoword class.
donudoo (doughnut)
banabao (banana)
hexahe + (hexagon)
balobao (blob)
bamobao (bamboo)
pipepi (pipe)
dogadoo (doggy)
Figure 5: Images generated from pseudowords reminiscent of real English words. For each pseudoword display associated image generation and the automatically detected closest English word. Pseudowords with high or low geometric scores (in top or bottom 20% relative to all pseudowords) which do not match their class (57 or O) are indicated in red.
obscure or highly abstract items using age of acquisition and concreteness scores from 27 9.. We
sort these items by their (())) scores and display the words with the highest and lowest scores.
Qualitative results are shown Table 2 and Figure Table we see striking pattern that head and tail of this list represent adjectives which intuitively describe visual properties that are more "sharp" or "round". In other words, these properties are highly correlated with relative similarity the two pseudoword classes O and 5². This strengthens adjectives with phonetic scores p(w), as we see these scores are truly aligned with visual semantic properties for the VLMs under consideration (particularly for Stable Diffusion). The images in Figure show a similarly striking pattern, with the sharper objects (as measured by Y(w) score) being more commonly generated by pseudowords from L & and rounder objects by pseudowords from Vo
quantitative results for ranking ground truth
Pseudowords that resemble real English words may generate images reminiscent of real objects, as seen Figure (5) There we display such pseudowords along with close English words, detected automatically via automatic search heuristic (described in the supplementary material) combining fuzzy string matching and CL.IP text-image similarity. This suggests a possible correlation between sounds in real English words and visual semantics explored in the psycholinguistic literature [38
60 (6].
Discussion, Limitations and Future Work
We have shown that sound symbolism is reflected in VLMs by evaluating them on pscudowords with controlled phonetic properties. By comparing pseudowords built from known "sharp" or "round" speech sounds, we find that these models learn associations with corresponding sharp and round adjectival properties, parallel to the classic kiki-bouba experiment psychology.
Further strengthening these findings, we provide strong evidence in the supplementary material that these models have not learned specifically from items illustrating the kiki-bouba effect, by showing that this concept is not well-represented in the LAION dataset [54] upon which our VLMs were

---

trained, and by showing that Stable Diffusion does not generate coherent content from prompts referring the experiment itself. Rather, it appears that our VLMs have learned sound patterns from their training data via large-scale correlations between sounds and shapes. Future work could further explore complex generalization patterns and biases learned by VLMs, associating non-obvious visual or semantic meanings with particular input patterns.
Our findings have significance both in the field of multimodal machine learning as well as in psycholinguisties. From a computational perspective, our work reveals an emergent behavior in VLMs that has been hitherto unexplored, to the best of our knowledge. In general, VLMs are being used as black boxes without full understanding of how they understand the visual semantics of language. Our work sheds light on how these models interpret and respond to language, as well as raising questions about how sound symbolic associations are inferred from our models' training datasets, whether additional dimensions of sound symbolic meaning are present in these models apart from our findings, and whether similar associations could exists with other modalities beyond vision.
From a cognitive science perspective, our work provides a new perspective the extensive literature on sound symbolism, which has long been a topic of interest and debate. If VLMs learn sound symbolic association from valid text in caption data, this may be due the presence of sound symbolism as reflected in the basic lexicon of English (or other languages), and the presence of sound symbolism in language itself moderately controversial (notably denied by Ferdinand de Saussure as discussed in Section DT. In this vein, investigation into how VLMs infer sound symbolic associations from their training data could potentially shed light on how sound symbolism is learned during human language acquisition. Additionally, these methods could provide new insights into the classic questions of what aspects of sound are tied to meaning and to what extent the observed phenomena are culturally universal or specific to the English language. Our work provides new line of inquiry, suggesting that VLMs and their training datasets should be researched carefully for further insight.
Regarding limitations of our findings, we do not assume that these models directly imitate human cognition, although our results do show that sound symbolic patterns are present in captioned image data with a strong parallel to the psychology of language and vision. Additionally, we do not answer why these associations are present in language or exactly how they are learned from the given data. In particular, we note that our results are agnostic to whether these models have memorized associations between particular letters and shapes, and to whether they have deeper understanding of the phonetic or acoustic properties implied by these letters. We leave investigation of these questions to future work, and foresee them providing an important direction for research into sound symbolism in human cognition and machine learning.
Acknowledgments and Disclosure of Funding
This work was partially supported by the Alon Fellowship. We thank Gal Fiebelman and Taelin Karidi for their helpful feedback.
References
[1] James S. Adelman, Zachary Estes, and Martina Cossu. "Emotional sound symbolism: Languages rapidly signal valence via phonemes". .. In: Cognition 175 (June 1, 2018), pp. 122-130. ISSN: 0010-0277. DOr:
10. 1016/j cognition. 2018. 02. 007 URL: https / ///////.../// sciencedirect. comm com/science science/ articleepia//80100277 1830077777133741111 (visited on 12/18/2022).
[2] Morris Alper, Michael Fiman, and Hadar Averbuch-Floe "Is BERT Blind? Exploring the Effect of Eision-and-Language Pretraining on Visual Language Understandin"" In: Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognitton (CVPR). 2023.
[3] Damián Blasi et al. "Sound-meaning association biases evidenced across thousands of languages" In: Proceedings of the National Academy of Sciences of the United States of America 113 (Sept. 12, 2016).
DO1: 10. 30733/pass 1605782113
[4] Damián E. Blasi et al. "Sound meaning association biases evidenced across thousands of languages". In: Proceedings of the National Academy of Sciences of the United States of America 113.39 (Sept. 27.
2016). pp. 10818 18 10823. ISSN: 8027-8424, DO1: 10. 1073 / 1605782113 URL: https: //www /www
ncbi m1m. nih. gov/ http wwwticleeee/cc54/555551
/visited on 12/26/2022).
[5] Tolga Bolukbasi et al. "Man is to computer programmer as woman is to homemaker? debiasing word embeddings". In: Advances in neural information processing systems 29 (2016).
10

---

[6] Rishi Bommasani et al. "On the opportunities and risks of foundation models". In: arXiv preprint arXiv:2108. 07258 (2021).
[7] Roberto Bottini, Marco Barilari, and Olivier Collignon. "Sound symbolism in sighted and blind. The
role of vision and orthography in sound-shape correspondences". In: Cognition 185 (Apr. 1, 2019). pp. 62-70. ISSN: 0010-0277. DOI: 10 1 cognition 2019, 01 006 URL: https: : // www ( (visited on 12/ 12/18/2022).
sciencedirect wtt.:/cccccccccceceeeee http///iiincaaaaiicee/paaaaeaaae http/////...e/////////////1///0/
[8] Andrew J. Bremner et al. ""Bouba" " and "Kiki" Namibia? A remote culture make similar shape-sound matches, but different shape-taste matches to Westemers" . In: Cognition 126.2 (Feb. 1, 2013). pp. 165-
172. ISSN: 0010-0277. DOL: 10 1016 / cogmition 2012 09 007 URI.: https / www sciencedirect.co www/sciemce.eeccemccccccccmccmce wttpee//ttiileeeeeeeeeeeeeeeeeee pttpppp.... pttppp//////000000200000000 (visited on 01/22/2023).
[9] Mare Brysbaert. Amy Beth Warriner, and Victor Kuperman. "Concreteness ratings for 40 thousand generally known English word lemmas". In: Behavior research methods 46 (2014), pp. 904-911.
[10] Léa De Carolis et al. "Assessing sound symbolism: Investigating phonetic forms, visual shapes and letter fonts an implicit bouba- kiki experimental paradigm". In: PLOS ONE 13.12 (Dec. 21, 2018). Publisher: Public Library of Science, c0208874. ISSN: 933-62033 DOL 10. 1 pone. 0208874 URI: https: htt////ooaaaaaa// plos. org/ wtggpp/aone//ooooooooooooooooe// wataaaoneartiileeeaaa attaaaariiccecceaaaae 1371/ journal pone. 0208874 (visited on 12 12/18/2022).
[11] Mehdi Cherti et al. "Reproducible scaling laws for contrastive danguage-image learning". In: arXiv preprint arXiv:2212.07144 (2022).
[12] Roi Cohen et al. "Crawling the Internal Knowledge- Base of Language Models". In: arXiv preprint arXiv:2301.12810 (2023).
[13] Christine Cuskley, Julia Simner, and Simon Kirby. "Phonological and orthographic influences in the bouba-kiki effect". In: Psychological Research 81.1 (Jan. 1, 2017), pp. 119-130. ISSN: 1430-2772. DOI: (visited
10. 1007/000426-01--0700----- URL: htL::/htts:://// on 12/18/2022).
org/
[14] Aleksandra Čwick et al. "The bouba/kiki effect is robust across cultures and writing systems". In: Philosophical Transactions of the Royal Society B: Biological Sciences 377.1841 (Jan. 3, 2022). Publisher: Royal Society, p. 20200390. DOr: 10 1098 rstb 2020 0390 URL: hhttps
royalsocietypubllahing owt//oi//1.oooooooooooooooooo 1098/rstb 2020 0390 (visited on 12/21/2022).
[15] Ferdinand De Saussure. Course in general linguistics. (Original publication. 1916). 1916. URL:https Fffr.wikisuuree
[16]
Sunipa Dev and Jeff Phillips. "Attenuating bias in word vectors". In: The 22nd International Conference on Artificial Intelligence and Statistics. PMLR. 2019, pp. 879 887.
[17] Mark Dingemanse et al. "What sound symbolism can and cannot do: Testing the iconicity of ideophones from five languages". In: Language 92.2 (2016). Publisher: Linguistic Society of America, 17-e133. ISSN: 1535-0665. DOI: [0. 1353 lann 2016 0034 URL: https / muse jhu edu/ / pub/ pub/22 /
article/621188 (visited on 12/18/2022).
[18] Linda Drijvers, Lorijn Zaadnoordijk, and Mark Dingemanse. -oouddsymmbllimm is disrupted in dyslexia: Implications for the role of cross-modal abstraction processes". In: 37th Annual Meeting of the Cog- nitive Science Society (CogSci 2015). Cognitive Science Society. 2015, pp. 602-607. URL: https
//pure.mpp www.dppp.pe......p.mmmmmmmme.p.p aadee//umaanffeeeeee wwwaaaaa/aaaeeeeeaaaaaaaaaeeeeee wtaacc/e/eiieeeeeedevvvveeeeeeee wttp:/aaaaaaeppppp.ee.ee.. jsp?itemId=item 2152234 on 12/18/2022).
(visited
[19] Mathilde Fort and Jean-Luc Schwartz. "Resolving the bouba-kiki effect enigma by rooting iconic sound symbolism in physical properties of round and spiky objects". In: Scientific Reports 12.1 (Nov. 10, 2022).
Number: 1 Publisher: Nature Publishing Group. 19172. ISSN: 2045-2322. doI: 10. 1038/ 4419988
022 23623- w URL: https::////// nature. com/ htt///attccceeeee s41598 022 23623 w (Visited on 12/18/2022).
[20] Louise Fryer. Jonathan Freeman, and Linda Pring. "Touching words is not emough: How visual experience influences haptic-auditory associations in the "Bouba-Kiki" effect". In: Cognition 132.2 (2014), pp. 164- 173.
[21]
Gabriel Goh et al. "Multimodal ncurons in artificial neural networks". In: Distill 6.3 (2021), c30. Giles Hamilton- Kleetton-Fletcher al. "The role visual experience in the emergence of cross mrosl- correspon- dences". In: Cognition 175 (2018), pp. 114-121.
[22]
[23]
Jonathan Ho and Tim Salimans. "Classifier-free diffusion guidance", In: arXiv preprint arXiv:2207. (2022).
[24]
Wolfgang Köhler. Gestalt Psychology: An introduction to new concepts in modern psychology. 1947. Noriyuki Kojima al. "What is learned in visually grounded neural syntax acquisition". In: arXiv preprint arXiv:2005.0166
[25]
(2020).
[26] Satwik Kottur et al. "Visual word2vee (vis-w2x); Learning visually grounded word embeddings using abstract scenes". In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, pp. 4995.
11

---

[27]
Victor Kuperman, Hans Stadthagen- Gonzzlezz and Mare Brysbaert. "Age-of-acquisition ratings for 30,000 English words". In: Behavior rexearch methods 44 (2012). pp. 978-990.
[28] Simon Lacey et al. "Stimulus Parameters Underlying Sound-Symbolic Mapping of Au- ditory Pseudowords to Visual Shapes". In: Cognitive Science 44.9 (2020).
eprint:
https//onlineelibaaryy wwl.e..co//ooippd//iii /eog://ppppp/...11ooogogsss.. e12883, ISSN: 1551-6709. DoI: 10
1111/cogs. 12883 URL: ptttps://oolimeliine (visited on 12/18/2022).
www.ww.wwwwwleeeeeee www/do//aas////////o//oooa http://abbb/..111//oggg....... 12883
[29] Xiao Lin and Devi Parikh. "Don't just listen, use your imagination: Leveraging visual common sense for non-visual tasks". In: Proceedings of the IEEE conference on computer vision and puttern recognition. 2015. pp. 2984-2993.
[30]
Margaret Magnus. "A history of sound symbolism". In: The Oxford handbook of the history of linguistics 18 (2013), pp. 194-195.
[31]
Gloria Strauss Marmor. "Age at onset of blindness and the development of the semantics of color names". In: Journal of experimental child psychology 25.2 (1978), pp. 267-278.
[32]
Joanna Materzyúska, Antonio Torralba, and David Bau. "Disentangling visual and written concepts in
CLIP". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2022, pp. 16410-16419.
[33] Daphne Maurer, Thanujeni Pathman, and Catherine J Mondloch. "The shape of boubas: Sound-shapp correspondences in toddlers and adults". In: Developmental science 9.3 (2006), pp. 316-322.
[34]
Kelly McCormick et al. Sound meaning mappings in the Bouba-Kiki effect. July 25, 2015.
[35]
Tomas Mikolov et al. "Efficient estimation of word representations in vector space". In: arXiv preprint
arXiv:1301.378) (2013).
[36] George A Miller. "WordNet: lexical database for English". In: Communications the ACM 38.11
(1995), pp. 39 39-41.
[37]
George A Miller. WordNet: An electronic lexical database. MIT press, 1998.
[38]
Padraic Monaghan al. "How arbitrary is language?" In: Philosophical Transactions of the Royal Society
B: Biological Sciences 369.165 (2014), p. 20130299
[39] Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. "Editing Implicit Assumptions in Text-to-Image Diffusion Models"". In: arXiv preprint arXiv:2303. 08084 (2023).
[40] Ozge Ozturk, Madelaine Krehm, and Athena Vouloumanos. "Sound symbolism in infancy: Evidence for sound shape cross- mooss-modal correspondences in 4-month-olds olds". In: Journal of experimental child psychology 11.2 (2013), pp. 173-186.
[41] Jeroen van Paridon, Qiawen Liu, and Gary Lupyan. "How do blind people know that blue is cold? Distributional semantics encode color-adjective associations." In: Proceedings of the Annual Meeting of the Cognitive Science Society. Vol. 43. 43. 2021.
[42] Cesare Valerio Parise and Charles Spence. "'When birds of feather flock together': synesthetic corre- spondences modulate audiovisual integration in snnesssssthetss. In: PloS one 4.5 (2009), c5664,
[43] OF Patashnik et al. "Styleclip: Text-driven manipulation of stylegan imagery". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 2085-2094.
[44] Fabio Petroni et al. "Language models as knowledge bases?" In: arXiv preprint arXiv:1909.0100 (2019).
[45] Alec Radford et al. "Learning transferable visual models from natural language supervision". In: Interna- tional conference on machine learning. PMLR. 2021, pp. 8748-8763.
[46] Vilayanur S Ramachandran and Edward M Hubbard. "Synacsthesia-- window into perception, thought and language". In: Journal of consciousness studies 8. 12 (2001), pp. 3-34.
[47] Aditya Ramesh et al. "Hierarchical text-conditional image generation with clip latents". In: arXiv preprint ar Xiv:2204.088 06125 (2022).
[48] Anna Rogers. Olga Kovaleva, and Anna Rumshisky. "A primer in bertology: What we know about how bert works". In: Transactions of the Association for Computational Linguistics 8 (2020), pp. 842-866.
[49] Robin Rombach et al. "High-resolution image synthesis with latent diffusion models". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 10695. Chitwan Saharia et al. "Photorealistic text-to-image diffusion models with deep language understanding". In: Advances in Newral Information Processing Systems 35 (2022), pp. 36479-36494
[50]
[51] E. Sapir. "A study in phonetic symbolism". In: Journal of Experimental Psychology 12 (1929). Place: US
Publisher: Psychological Review Company. pp. 225-239. ISSN: 0022-1015. DOI: 10. 1037 1037/h0070931
[52] Armin Saysani, Michael C Corballis, and Paul M Corballis. "Colour envisioned: Concepts of colour in the blind and sighted". In: Visual cogmition 26.5 (2018), pp. 382-392.
[53] Armin Saysani, Michael C Corballis, and Paul M Corballis. "Seeing colour through language: colour knowledge in the blind and sighted". In: Visual Cognition 29.1 (2021), pp. 63-71.
[54] Christoph Schuhmann et al. 5b: An open large-scale dataset for training next generation image- text models". In: arXiv preprint 221 12.000200 (2022).
12

---

[55] Roger N Shepard and Lynn A Cooper. "Representation of colors in the blind, color-blind, and normally sighted". In: Psychologicul science 3.2 (1992). pp. 97-104.
[56] David M Sidhu al. "Sound symbolism shapes the English language: The maluma/takete effect in English nouns". In: Psychonomic Bulletin Review 28 (2021), pp- 1390-1398.
[57] Yoad Tewel et al. "Zero-shot image-to-text generation for risual-semantic arithmetic", In: arXiv preprint arXiv:2111. 14447 (2021).
[58]
Ramakrishna Vedantam al. "Learning common sense through visual abstraction". In: Proceedings of the IEEE international conference on computer vision. 2015, pp. 2542-2550.
[59] Peter West al. "Symbolic knowledge distillation: from general language models commonsense models". arXiv preprint arXiv:2110.0711 (2021).
[60] Bodo Winter and Marcus Perlman. "Size sound symbolism in English lexicon". In: Glossa: a journal of general linguistics (2021).
[61] Chenyu Zhang et al. "Visual Commonsense in Pretrained Unimodal Multimodal Models". In: arXiv preprint arXiv:2205.01885 (2022).
13


Reviews:
[{"review_id": "C2G4UsiG2Z", "reviewer": "Reviewer_NTGx", "summary": "This paper determines the extent to which pretrained vision and language models encode phonetic information associated with sharp or round objects. Prior research has shown a cross-lingual tendency to associate sounds with shapes in human studies. In this work, the authors investigate whether this holds for a discriminative (OpenCLIP) or generative (StableDiffusion) machine learning model. The methodology starts by constructing a set of 648 pseudowords based on a set of sharp / round Latin letters. Two prompts are used with the models, which either have the pseudoword used as a noun or an adjective. The prompts and pseudowords are used to show that both models do indeed exhibit a similar effect to humans. A human study shows a much stronger effect for the original kibi-bouba pair than the 648 pseudowords. The paper also claims that this finding cannot be attributed to the models seeing these examples during pretraining.", "questions": "* Do these results hold with a multilingual CLIP model or in a multilingual human evaluation?", "limitations": "* The paper does a good job of discussing potential limitations of the research", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Interesting study that is well-motivated.\n* Claims are clearly supported in the experiments.\n* Human evaluation is used to further support the automatic evaluation metrics.", "weaknesses": "* No major weaknesses that I could identify", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments. \n\nTo investigate whether sound symbolism may exist in a multilingual V&L model, we test the Kadinsky [1] multilingual text-to-image model (which uses multilingual CLIP) with our methodology, on four geographically and linguistically diverse languages: Finnish, Indonesian, Hungarian, and Lithuanian. Please see Tables 2 and 3 in the PDF attached to our global response for full results including metrics and prompt texts in each language. We find non-trivial sound symbolism in this setting in each language; for example, Finnish displays geometric and phonetic AUC 0.69 and 0.94 respectively, significantly higher than the 0.50 expected from random chance. These results suggest that sound symbolism may be learned in a multilingual V&L setting.\n\nEven with the results of this additional experiment, we wish to emphasize that we do not claim to demonstrate the universality of sound symbolism (as suggested by reviewer Cwqz). Although this is a topic of interest in the psychological literature, our work focuses on showing the existence of this phenomenon in V&L models, and not on whether it is a universal phenomenon cross-linguistically. We will state this more explicitly in a revised version of our work.\n\n[1] kandinsky-community/kandinsky-2-2-decoder on Hugging Face Model Hub"}, {"review_id": "keiXtRB73N", "reviewer": "Reviewer_gvCK", "summary": "The authors present an investigation of the phenomenon of ‘sound symbolism’ in a pair of cutting edge Vision-and-Language Models. Sound symbolism is an intriguing phenomenon whereby the meaning of a word can be in part traced back to the way the word sounds. It is an important phenomenon in psychology and linguistics because it challenges a strict view of the mapping between form and meaning as arbitrary (stated most famously by Saussure). The authors probe CLIP and Stable Diffusion by creating a small dataset of pseudowords that were constructed to reflect phonetic features that generally map onto “sharp” vs. “round” speech sound categories. They project the embedding vectors in 1024-dimensional CLIP space onto a one-dimensional semantic dimension of interest, defined by two sets of antonym shape adjectives (one set corresponding to synonyms of ‘round’ and the other corresponding to synonyms of ‘sharp’). They also define a phonetic score to measure phonetic/graphemic associations of the shape adjectives with the pseudowords. They analyze the models using these geometric measures, as well as with a human evaluation where participants are asked to select which of two Stable Diffusion-generated images is best described by a pseudoword. Across these evaluations, they find relatively strong evidence for the presence of sound symbolism in these models. ", "questions": "My questions are the same as the weaknesses. I’m not sure all of questions 1-3 need to be answered, but at least 2 of them do.\n1. How can these results inform theories in Cognitive Science?\n2. What hypotheses can be derived from these models a priori about sound symbolism, presumably by thinking about their training data?\n3. Why is sound symbolism important for evaluating sophisticated, deployable AI systems?\n4. Can you unpack the Phonetic Scores in a little more detail, and explain their relevance?\n5. Please provide more details about the User Study and analyze the results in a more appropriate manner.\n", "limitations": "The limitations were okay but could be improved by thinking more about the societal impact of these (human) biases getting incorporated into deployable generative models. I would bet some of the impacts of sound symbolism have been discussing in the psychological literature, and it would be important to mention them and discuss them in the context of these things getting baked into generative AI.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "One of the strengths of this paper is that it demonstrates how ideas from Cogntive Science can drive investigations into modern AI models. The phenomenon of sound symbolism is particularly interesting both for its intuitive appeal and its relation to foundational ideas in Cognitive Science. The paper was well written and largely a pleasure to read. The methods were overall well described and I found the evidence relatively convincing. ", "weaknesses": "The weakest part of the paper for me were the motivations, hypotheses, and implications. Yes, Vision&Language Models are increasingly powerful and increasingly deployed, but why are they interesting objects of study for Cognitive Science? We are told that “these methods could provide new insights into the classic questions of what aspects of sound are tied to meaning”, but not explained how this might happen or what this could look like. As well, the conclusion about “cultural universality” seems like a very difficult and non-obvious question that these methods would provide insight into. \n\nAnd the flip-side, why is sound symbolism an important phenomenon to investigate in these powerful and deployable models? One specific weakness is that there were no hypotheses presented for the models, and you could imagine developing the hypotheses based on how these models were trained. I think there’s a lot to say about this, but all we got one sentence buried in the results section that mentioned that the models primarily saw valid English text during training and didn’t have access to the actual sound of the words.\n\nThe second major weakness for me was the description and analysis of the User Study. The authors write “Over two hundred volunteers participated in this study”. The use of the word “volunteers” here suggests that the participants were not paid, which I hope is not the case. Who are these participants? How were they recruited? Was there any ethical review of this study? How many participants exactly were recruited (this is in the supplement, but should be presented in the main text)?\n\nOn the analysis side, the use of a binomial test is not appropriate as the individual data points are not i.i.d. Participants gave multiple responses and, presumably, multiple items (pairs of images) were rated by multiple participants. Thus, something akin to a mixed-effects (multi-level / hierarchical) logistic regression model would be more appropriate to account for the participant- and item-wise variability, and provide a much more realistic analyses of the results (see Gelman & Hill, 2006 for background). \n\nI found the Phonetic Scores in 3.2 the most difficult to understand and still am not completely sure I get it. From the math, I would have thought this score was something akin to a control measurement, and a way to contextualize the Geometric scores, but the authors seem to have a different perspective, which I didn’t fully get.\n\nFinally, Figures 2 & 3 took up a lot of space for not that much information. They could easily be condensed to make room for some of the context in the response.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments. We address the reviewer’s concerns regarding our user study in our global response (item A).  In particular, we reiterate that we believe the user study to not be critical to our results, and we are willing to remove or replace it upon request.\n\nRegarding motivation, we first note that other reviewers mention the motivation of our work as a strength, referring to it as an “interesting study that is well-motivated” (NTGx), having a “clear explanation of motivation… [and telling] a beautiful story” (kkX7). We explicitly address the reviewer’s questions next, emphasizing aspects of our work which we will better illustrate in a revised version.\n\nRegarding question 1 (importance of our results in the context of cognitive science), we believe our results have an impact in this domain for a number of reasons. Firstly, they suggest that sound symbolism is reflected in the linguistic training data of these models, and the presence of sound symbolism in language itself is moderately controversial (notably denied by Saussure, L22-23). Additionally, a series of psychological and linguistic works have investigated whether sound symbolism is reflected in the basic lexicon of English (see citations on L94); if V&L models learn sound symbolic association from valid English text (or other languages) in caption data, it might be due to this effect and could confirm the results of these studies with a new methodology. Finally, investigation into how V&L models infer sound symbolic associations from their training data could potentially shed light on how sound symbolism is learned during human language acquisition. While we do not claim to directly answer these questions, our results suggest that V&L models could provide valuable insights for investigating them. We will state these points more explicitly in a revised version.\n\nRegarding question 2 (a priori hypotheses regarding models) – when considering text containing nonsense words as input, one might expect models such as Stable Diffusion (SD) to produce purely random or nonsensical output. The competing hypothesis, which we believe to be non-obvious but for which we find strong evidence, is that such models have learned associations between the individual characters used in these nonsense words, and moreover in accordance with associations known from the psycholinguistic literature.\n\nRegarding question 3 (importance of our results in evaluating AI systems) as well as the reviewer’s comment about limitations and bias in generative models – in general, we believe that it is important to investigate what is learned by these powerful and deployable models and to provide interpretability to their behavior. In particular, we see value in understanding how V&L models may learn patterns from their training data, particularly patterns which require complex generalization beyond the captions seen during training. While we do not claim to determine the source of the observed effects, we provide evidence that these models have not simply memorized specific pseudowords seen during training (Sec 4.5). Such generalization could shed light on mechanisms leading to various associations and biases exhibited by V&L models, suggesting that they cannot only be explained by looking at individual training instances but possibly requiring an understanding of generalization over complex patterns on a large scale. We will incorporate such a discussion in a revised version.\n\nRegarding the statistical analysis of our survey results, we thank the reviewer for the suggestion and re-analyze the data using a mixed-effects logistic regression model. In this setting, we regress whether a question is answered correctly by a respondent (categorical response variable: 0 for incorrect answer, 1 for correct answer), where the question identity corresponds to a fixed effect and the respondent identity corresponds to a random effect. We fit this model using the “Lmer” implementation in the lme4 R package for modeling of linear mixed-effects models. For example, analyzing our survey of “kiki / bouba” generations* results in an intercept estimate corresponding to 89% overall success probability (p≈0.001), which isolates the overall success rate from the effects of individual respondents and questions. We will use this statistical analysis in a revised version (contingent on use of our survey data, per the discussion above).\n\n*(treating questions from survey versions 1 and 2 as separate question categories, and only using data from respondents who had not previously heard of the kiki-bouba effect)\n\nRegarding the phonetic scoring method (Sec 3.2), we clarify that the probe vector (L217) measures the direction in CLIP’s embedding space between the centroids of the two pseudoword classes (either as text or image embeddings, depending on the model being tested). We call this “phonetic scoring” because it is a semantic dimension determined by the phonetics (sound classes) of pseudowords alone. In Table 1 (last two columns) and Figure 3 we examine correlation between ground-truth adjective class and position on this dimension. In Table 2, we show that this dimension in CLIP space corresponds to interpretable semantics of real English words. We will emphasize these clarifications in a revised version."}, {"review_id": "5giW1nNoec", "reviewer": "Reviewer_kkX7", "summary": "The work's goal is to study whether sound symbolism is reflected in vision-and-language (VL) models like CLIP and Stable Diffusion (SD). The work proposed a method called **zero-shot knowledge probing**, and verified a sound symbolism phenomenon like **kiki-bouba effect** by evaluating the outputs from CLIP and SD when inputs were a set of predefined pseudowords, adjectives, and nouns, which are related to sharpness or roundness.\n\nIn short, the study provides a simple paradigm for investigating sound symbolism in VL models from VL latent space and verifies the existence of sound symbolism in VL models.\n", "questions": "1. Could you give a result on how exactly $|w_{adj}|$ and $|v_{pw}|$ are in L206 and L217 since the distribution of the adjectives and pseudowords in latent space is quite a blur?\n\n2. The caption of Table 2 is not clear, how is the list of words sorted and what does the `POS` line mean?\n\n3. Why do you choose SD to map texts to images, rather than using retrieval methods with CLIP on the LAION-2B dataset? They have the exactly same inherent knowledge.\n\n4. What are the probable applications of this study?\n", "limitations": "Although the work emphasizes that its study should be taken in context (L309), the inner reasons why VL models have the same sound symbolism effect as humans are not investigated. It should be addressed in future works.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The work gives a clear explanation of motivation, methodology, and experimental results.\n\n2. The work discusses the experimental results thoroughly and with some insights.\n\n3. The work provides rich supplementary material, which answers a few questions raised when I was reading the main paper.\n\n4. The work is well organized and tells a beautiful story with linguistic and cognitive backgrounds.\n\nIn conclusion, I believe the work is solid and insightful.\n", "weaknesses": "1. The work only studies open-sourced VL models CLIP and SD, which may relate to narrow computational resources. But studying models with different backbones like GAN or ViT would be more convincing.\n\n2. The **zero-shot knowledge probing** is designed a bit intuitively without comparison with other baseline methods, such as training an individual classifier for text-text and text-image scores or human annotators.\n\n3. The work did not discuss the inner results sound symbolism phenomenon in perspectives of the mechanism of machine learning but attributed it to the inherent knowledge in the models. I think the reason for the phenomenon might be a result of tokenization, i.e. a letter-level tokenized VL model / language model might produce different results.\n\n4. The work only studies the phenomenon in English, given L41 saying the phenomenon is universal to different languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments. We address the reviewer’s question regarding tokenization in our global response (item C), and the concern about multilingual results in our global response (item B) and in the response to reviewer NTGx.\n\nTo address concerns about the generality of our method across different model architectures, we have also tested the SOTA text-to-image models DeepFloyd-IF [1] and Kadinsky [2]. We find significant sound symbolic effects in both models; please see Table 1 in the PDF attached to our global response for full results when evaluating on our pseudowords. For example, DeepFloyd-IF shows geometric score AUC 0.63 and phonetic score AUC 0.98, significantly higher than 0.50 expected from random chance. We note that these models differ in underlying architecture, training objectives, and with respect to training data. For example, DeepFloyd-IF uses a T5 transformer text encoder (rather than the CLIP text backbone used by Stable Diffusion) and uses a pixel-level denoising U-Net (rather than the latent diffusion architecture and objective used by Stable Diffusion). We will include these results and discussion in a revised version.\n\nRegarding probing methods, we focus on zero-shot knowledge probing rather than trained text or image classifiers because we are interested in the inherent knowledge of our models and not the dynamics of training on new data, (L183-184), consistent with prior work on probing models’ intrinsic knowledge [3].  To provide an additional zero-shot method to complement our results, we also analyze image generations using image-level geometric properties. In particular, we estimate the “sharpness” or “roundness” of an image by using a Harris corner detector, which looks for points of sharp discontinuity representing geometric corners. We estimate the number of corners in generated images and find a significant difference between images generated from the two pseudoword classes, with “sharp” pseudoword images having significantly more corners on average than “round” pseudoword images, confirming that the former are visually more sharp than the latter on average. We will include these results in a revised version.\nWe first wish to clarify the wording and terminology asked about in questions 1 and 2. $\\left|w_{adj}\\right|$ as defined on L206 is a unit vector pointing in the direction in CLIP’s latent space which best separates between the two sets of adjectives from L205 (inserted into the prompts from L187). $\\left|v_{pw}\\right|$ as defined on L217 is similarly a unit vector pointing in the direction in CLIP’s latent space which best separates between the two sets of pseudowords (L176, inserted into the prompts from L187). In Table 2, POS stands for “Part Of Speech” (in our case, noun or adjective) and the words are sorted by phonetic score $\\phi_{\\left<w\\right>}$. We  will include these clarifications in a revised version.\n\nRegarding the use of Stable Diffusion (SD) for text-to-image generation versus image retrieval, we believe that there is in fact an important difference in the intrinsic knowledge of SD versus CLIP-guided image retrieval on LAION. While SD and (Open)CLIP were both trained on LAION, the intrinsic knowledge learned by SD’s denoising goal (which requires understanding local regions within images) is likely to be different from knowledge gained from image-level retrieval. Indeed, our quantitative and qualitative results demonstrate that CLIP and SD do not behave identically with respect to intrinsic knowledge; with SD) showing stronger sound symbolic effects (e.g. L299). Our primary motivation for including SD in our tests is to investigate the knowledge learned by a popular generative model, in addition to probing the discriminative model CLIP directly. \n\nRegarding applications of our work, we see value in understanding how these models “interpret and respond to language” (L335); in general, V&L models are being used as black boxes without a full understanding of how they understand the visual semantics of language, and we believe that probing them to understand what they have learned may be relevant to model interpretability. We also believe that our findings could have applications to cognitive science and linguistics (L339-344), where sound symbolism has long been a topic of interest and debate; computational methods could be used to provide more evidence for the presence of sound symbolism and to understand its cognitive basis. We will make these points more explicit in a revised version.\n\nFinally, we wish to clarify the wording and terminology asked about in questions 1 and 2. $\\left|w_{adj}\\right|$ as defined on L206 is a unit vector pointing in the direction in CLIP’s latent space which best separates between the two sets of adjectives from L205 (inserted into the prompts from L187). $\\left|v_{pw}\\right|$ as defined on L217 is similarly a unit vector pointing in the direction in CLIP’s latent space which best separates between the two sets of pseudowords (L176, inserted into the prompts from L187). In Table 2, POS stands for “Part Of Speech” (in our case, noun or adjective) and the words are sorted by phonetic score $\\phi_{\\left<w\\right>}$. We will include these clarifications in a revised version.\n\n[1] DeepFloyd/IF-I-M-v1.0 on Hugging Face Model Hub\n\n[2] kandinsky-community/kandinsky-2-2-decoder on Hugging Face Model Hub\n\n[3] Petroni et al. “Language Models as Knowledge Bases?” EMNLP 2019"}, {"review_id": "3oUmkapt3w", "reviewer": "Reviewer_DdD9", "summary": "The Kiki-bouba effect is a well studied phenomena in humans to consistently associate sharp and smooth objects with certain phonetics. This paper explores whether such an effect is present within image and language models. Specifically it looks at Stable Diffusion (a text to image generative model) and CLIP (a discriminative model for text and images) to see whether these models form representations of made up \"Kiki\" and \"Bouba\" type words in similarly consistent ways as humans do. \nThe paper is well written, and concludes that these models do indeed demonstrate this same tendency. ", "questions": "Did you try other ways of scoring? Does distance from centroids of the sharp/smooth groups also display the same effect for example? How important to observing this effect are the details in how scoring is done?", "limitations": "The authors do address the fact that these models don't have the same full set of inputs as humans do. They also note that nothing is discussed about why these models display these traits. These are important to highlight, but the paper has value as an observation that the effect is present. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written. The methods used to investigate whether this effect is present in these different models appear valid. \nThe observed effect is strong. It's interesting that these models demonstrate this as strongly as they do given they have no auditory dimension, but then again they are models of language. Nice observation though!", "weaknesses": "It would have been nice to make use of a model that doesn't have have any exposure to images. For example taking BERT and seeing if the pseudo words also get separated with the same tendency or not? Something analogous to demonstrating kiki-bouba in blind people, which to my lay understanding I believe has been done but only to a limited extent compared to other validations of the effect in various human populations. \nI'm unsure if this paper opens up any further significant study, or unlocks anything in an engineering sense, so I think its impact is likely capped. \nMinor: citation on line 174 would be good, rather than just \"some prior works\". \nNitpick: Line 24 -- \"bœuf\" is often translated as \"beef\" I thought, in which case there is a lot of overlap in sounds. This link is no doubt cultural, rather than saying anything about objects having similar sounding terms across languages. Maybe there is a better example to use though?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments. We will clarify wording issues and add requested citations in a revised version.\n\nRegarding unimodal (text-only) models, we refer to our investigation of these models in the supplementary material (supp Sec 3.1), where we test encoder-only models (similar to BERT) that have been trained on text alone. Our findings show mixed results, with possible sound symbolic effects evident but seemingly weaker than those shown by multimodal models; we leave a thorough investigation of the relative contributions of text and image data to future work (supp L124).\n\nRegarding scoring methods –  see the response to reviewer kkX7 for an additional zero-shot scoring method. Regarding the question about cluster centroids for scoring, our phonetic scoring method measures distance along the axis which separates the two centroids of pseudowords clusters, effectively comparing relative distances to the centroids of each cluster. \nRegarding the effect of different scoring methods, we found sound symbolic effects across probing methods (geometric and phonetic scoring, and the additional method proposed in our response to reviewer kkX7). We also found scoring to be robust to the choice of prompt text used in probes (supp Sec 3.2).\n\nRegarding the French word “boeuf”, we clarify that this is being used on L23 as a direct citation from Saussure, who used this example verbatim in the source being cited. Additionally, “beef” is actually a loanword in English from Old French [1], making the similarity non-coincidental. Nevertheless, we can replace this with an unambiguous example to avoid confusion.\n\n[1] D. Harper. “Etymology of beef.” Online Etymology Dictionary."}]

Author rebuttal:
We thank the reviewers for their constructive comments. We respond here to shared concerns as well as items raised by reviewers JJeH and Cwqz, referring to individual responses where more information is given.

**(A) Survey – reviewers JJeH, Cwqz, gvCK**

The main focus of our work is automatic probing of computational models for sound symbolism; the user study suggests that these results are grounded in human cognition, but it does not test our central hypothesis (whether V&L models have learned sound symbolism). We saw value in showing this additional result and conducted the survey following the informal procedure described below, but we are willing to either remove it entirely or to replace it with a study using Mechanical Turk crowdworkers (following all relevant procedures) upon request.

Regarding the methodology used, we used volunteers to avoid concerns about crowdsourcing and contract work, under the good-faith interpretation of NeurIPS’s ethics code that human studies refer to paid workers rather than consenting volunteers. This interpretation was made following discussions with several peers in our institutions, as well as observing that this practice has been followed in papers published in NeurIPS 2022 such as:

* Zhang et al. Generalized One-shot Domain Adaptation of Generative Adversarial Networks. (“...we finally collect valid votes from 53 volunteers…”)
* Hu et al. Hand-Object Interaction Image Generation. (“...we conduct a user study... There are 20 volunteers participating in this study.”)

Following common practices in our institutions, our study was distributed among university graduate students who were not familiar with this research project, and we controlled for prior knowledge of the phenomenon (supp L196, L227). We controlled for native language (supp Table 5) as this would be a natural confounding factor. We did not collect any sensitive personal information or demographic data such as race or education level. While reviewer Cwqz mentions these as significant for showing the universality of sound symbolism, we clarify that this is not the stated aim of our work (noting that L32-35 are not discussing our research goals, but rather providing context from the psychological literature). We will clarify these points in a revised version (contingent on any decision regarding the inclusion of the user study).

**(B) Multilingual evaluation – reviewers Cwqz, NTGx, kkX7**

Please see the response to reviewer NTGx where we conduct an additional experiment to test the cross-lingual generality of our results. Full results for each language tested are shown in Table 2 of the attached PDF, and Table 3 shows the prompts used for each language. We find evidence for the effect in a multilingual V&L model for each of the four geographically and linguistically diverse languages, and we will include these results in a revised version.

**(C) Tokenization – reviewers Cwqz, kkX7**

Regarding tokenization, we agree that this could potentially contribute to the observed effect. However, we wish to emphasize that our approach tests V&L models end-to-end, agnostic to the source of these effects with respect to the relative contribution of different model-internal components. We do note that OpenCLIP and Stable Diffusion use the same tokenizer and yet show differences in the strength of their sound symbolic effects (e.g. L299), precluding this as the only source of the observed results. We also clarify that we do not adjust the vocabulary of the models or the tokenizers in any way; we use these models as-is and probe them in the zero-shot regime. We will further emphasize these clarifications in a revised version.

Regarding reviewer Cwqz’s concern about items being “correctly mapped to their corresponding word vectors in CLIP”, we clarify that we deliberately design our pseudowords to avoid valid English words (L173) and thus they are indeed not found in the tokenizer’s vocabulary. This does not mean that the models are incapable of dealing with these inputs. This can be observed with many common English words, which are also not found in its vocabulary. For instance, “handkerchief” is split into three subword tokens (“hand”+”ker”+”chief”) by the tokenizer, but Stable Diffusion generations for this word show that it does indeed understand what a handkerchief is as a concept. Text encoders are capable of processing words that are not found in their vocabulary by splitting them into subword tokens, and we leverage this property to probe sound symbolism in these models.

**(D) Pseudoword construction – reviewer JJeH**

We clarify that the construction of pseudowords was fully automatic with no involvement of human participants. These were constructed using the combinatorial procedure described on L167-168 to include all possible combinations of letters matching the given pattern.

**(E) Vision-audio models – reviewer Cwqz**

While we agree that investigating such models would be an interesting and promising line of research, we base our investigation on an abundance of prior works in psychology and linguistics which investigate sound symbolism in written language. There is an implicit mapping between text and sound in spoken and written language, and the term “sound symbolism” is frequently used to refer to the possible connection between graphemes in text and meaning, as stated explicitly by [1, 2, 3] (which we cite on L88-91). As such, studying models trained on acoustic data would be an interesting direction for further research, but is out of scope of our study. We will emphasize this point in a revised version.

[1] Cuskley et al. “Phonological and orthographic influences in the 384 bouba–kiki effect”

[2] De Carolis et al. “Assessing sound symbolism: Investigating phonetic forms, visual shapes and letter 375 fonts in an implicit bouba-kiki experimental paradigm”

[3] Cwiek et al. “The bouba/kiki effect is robust across cultures and writing systems”

Meta-review: This paper examines the extent to which vision-and-language models capture sound symbolism - particularly, the "kiki" vs. "bouba" effect, whether models encode a relation between phonetic sounds and sharp or round shapes. They study the OpenCLIP and Stable Diffusion models using a set of pseudoword prompts and probing techniques, and find a significant effect in both models. They also conduct a user study to validate that the pseudowords have a similar effect on humans.

Reviewers agree that this is a strong paper, with interesting, well-designed experiments and claims that are clearly supported by the data. There were some questions about motivation for this area of study, which the authors elaborate on in their response, and about multilingual and unimodal (text-only) experiments, which they include in their supplemental PDF. Reviewers also ask about testing a more diverse set of models, since OpenCLIP and Stable Diffusion are closely related.

The paper was flagged for ethics review, however, with some concerns about whether the authors followed appropriate protocols around consent, data, etc. for the user study. The authors state that the participants were volunteers; however, we note that this still counts as a human study and there should still be a formal agreement by which subject consent to participation. We ask the authors to please include in their camera-ready version information on consent and how any personal data was handled, in accordance with the ethical guidelines at https://neurips.cc/public/EthicsGuidelines.
Decision: Accept (spotlight)