Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
Authors: Tom Kouwenhoven
Kiana Shahrasbi
Tessa Verhoef
Venue: NeurIPS 2025 poster ()
Keywords: Cross-modal associations
Vision-and-Language Models
bouba-kiki effect
Cognitive science
Primary area: evaluation
TL;DR: We re-evaluate whether vision-and-language models exhibit the human-like bouba-kiki effects, using two methods modelled after human experiments. Compared to humans, VLMs fall short in aligning cross-modal associations with human intuitions.

Abstract: Recent advances in multimodal models have raised questions about whether vision-and-language models (VLMs) integrate cross-modal information in ways that reflect human cognition. One well-studied test case in this domain is the bouba-kiki effect, where humans reliably associate pseudowords like ‘bouba’ with round shapes and ‘kiki’ with jagged ones. Given the mixed evidence found in prior studies for this effect in VLMs, we present a comprehensive re-evaluation focused on two variants of CLIP, ResNet and Vision Transformer (ViT), given their centrality in many state-of-the-art VLMs. We apply two complementary methods closely modelled after human experiments: a prompt-based evaluation that uses probabilities as a measure of model preference, and we use Grad-CAM as a novel approach to interpret visual attention in shape-word matching tasks. Our findings show that these model variants do not consistently exhibit the bouba-kiki effect. While ResNet shows a preference for round shapes, overall performance across both model variants lacks the expected associations. Moreover, direct comparison with prior human data on the same task shows that the models’ responses fall markedly short of the robust, modality-integrated behaviour characteristic of human cognition. These results contribute to the ongoing debate about the extent to which VLMs truly understand cross-modal concepts, highlighting limitations in their internal representations and alignment with human intuitions.

Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
Tom Kouwenhoven', Kiana Shahrasbi, Tessa Verhoef"
Leiden Institute of Advanced Computer Science Leiden University, The Netherlands
{t. kouwenhoven, k. shahhaasbi,
t. verhoef @iiacs.eeidenuniv.ll
Abstract
Recent advances in multimodal models have raised questions about whether vision- and-language models (VLMs) integrate cross- cross-modal information ways that reflect human cognition. One well-studied test case in this domain is the bouba- kiki effect, where humans reliably associate pseudowords like 'bouba' with round shapes and 'kiki' with jagged ones. Given the mixed evidence found in prior studies for this effect in VLMs, we present a comprehensive re-evaluation focused two variants of CLIP, ResNet and Vision Transformer (VIT), given their trality in many state-of-the-ate VLMs. We apply two complementary methods closely modelled after human experiments: prompt-based evaluation that uses probabilities as a measure model preference, and we use Grad-CAM as a novel approach to interpret visual attention in shape- matching tasks. Our findings show that these model variants do not consistently exhibit the bouba-kiki effect. While ResNet shows preference for round shapes, overall performance across both model variants lacks the expected associations. Moreover, direct compar- ison with prior human data the same task shows that the models' responses fall markedly short of the robust, modality-integrated behaviour characteristic human cognition. These results contribute the ongoing debate about the extent to which VLMs truly understand cross-modal concepts, highlighting limitations in their internal representations and alignment with human intuitions.
Introduction
Recent advances in multimodal models that integrate vision and language have brought artificial intelligence step closer to understanding the world in ways that resemble human experience and cognition. These models, which learn from vast amounts of paired visual and textual data, have demonstrated impressive capabilities in tasks such as image captioning, visual question answering, and cross- modsl retrieval (Radford et al.) 2021: Li et al. 2022 2023). Yet it remains unclear whether these models integrate visual and linguistic information in ways that parallel human cognitive processes. In this paper, we investigate whether VLMs exhibit human-like patterns of association between abstract visual shapes and unfamiliar words. Particularly, we focus studied cross-modal test cases in human cognition, the bouba-kiki effect, in which people consistently associate pseudowords like 'bouba' with round shapes and those like 'kiki' with jagged shapes
of the most widely
(Ramachandran and Hubbard 2001 Maurer al. 2006a Cwiek al. 2022). These associations have recently become a test case for evaluating whether language models trained on large-scale data
show similar patterns (Alper and Averbuch-Elor 2023 Verhoef et al., 2024 Loakman al. 2024 lida and Funakura 2024). Results, however, have been mixed. While Alper and Averbuch-Elop
(2023) found overwhelming evidence for bouba-kiki effect CLIP and Stable Diffusion, other
* 'Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Kiki
0.5
KIki
Takete
Maluns Bcuba
8.1
Congruent?
Congruent?
X
Kiki 8.1
0.2
Takete
Maluna
Bouba 0.4
Bouba
Figure An overview of the two complementary methods used. On the left, calculate the probabilities for each label across the four original pseudowords (note that the number of labels varies per label source) for each image shape and select the label with the highest probability (values are exemplary). On the right, we use concatenated image pairs and their labels as targets to calculate attention patterns with -rad-CAM and select the shape with the highest sum of attention.
studies have found less convincing patterns across different VLMs, testing methods and datasets (Verhoef et al. 2024 Loakman et al. 2024 lida and Funakura 2024).
This work therefore revisits this effect by thoroughly testing versions of CLIP (Radford et al. 2021): ResNet (He et al. 2016) and ViT (Dosovitskiy et al. 2021). We focus on CLIP because, among four models (CL.P, BLIP2, ViLT,, and GPT-4o) tested, Verhoef al. (2024) found that CLIP demonstrated the most promising alignment with a human-liss bouba-kiki effect. This aligns with previous work demonstrating that CLIP outperforms other models in capturing human-like decision patterns (Demircan et al. 2024). Perhaps most importantly, CLIP often acts as a foundational model state-of-the-art VLMs (see (sectonction 3p. If 'base' model does exhibit human-like associations, it is difficult to imagine that model using that base model as backbone will show human like preferences without explicit fine-tuning on relevant cross-modal associations. Unravelling how these 'base' models represent cross cross-modal information additionally benefits our understanding of limitations in, for example, spatial reasoning (Thrush et al.) 2022), (in-context) shape-colour biases in VLMS (Allen et al. 2025). and emergent communication setups (Kouwenhoven et al.. 2024).
By probing these models' preferences for shape-word association in novel contexts, we aim shed light on the nature of their internal representations and their alignment with human intuitions. When these differ, training models develop human-like prior expectations instilling them into models may help and could even improve learning efficiency (Lake et al. (2117). Doing so is essential, since flexible, grounded understanding of this kind could enable conceptually fluent, natural human- machine interactions, in which machines intuitively understand what we mean, even in unfamiliar settings. To this end. we employ two complementary methodological approaches that address the same cross-modal associations from different perspectives, thereby minimising the limitations of relying on a single evaluative framework. Despite this comprehensive strategy, we do not find compelling evidence for the bouba-kiki effect. This work contributes in the following ways:
(I) We test for the bouba-kiki effect VLMs ways that are close as possible how it has been tested with humans in the past. This enables direct comparisons between our results and human findings, building and expanding the work by Verhoef et al. 2022). Tested comprehensively, find that models, compared to humans, do not make consistent cross-modal associations.
(2) We introduce novel way to test models using Grad-CAM (Selvaraju al. 2020). method from the model interpretability literature, to look more closely at visual processing in the bouba- kiki context. Strengthening the robustness of our results, it reveals that the models do not explicitly focus on bouba-kiki related shape-specific features.
Related work
First, we discuss how prior work on cross-modal associations in human language and cognitive processing has shown that non-arbitrariness is both pervasive and affects how we learn, process and develop language. We then present previous studies that have tested for the bouba-kiki effect in VLMs.

---

2.1 Cross-modal associations in human language
Traditionally, it has been argued that mappings between words and their meanings are largely arbitrary. Hocket (1960) uses the words 'whale" and "microorganism' as example: 'whale" is a short word for a large animal, while 'microorganism' is the reverse. However, growing evidence from fields like cognitive science, language evolution and (sign language) linguistics suggests non-arbitrary form-meaning mappings are more widespread in human language than initially thought. This suggests that it should be considered general property of human language (Perniss al.] (2110), shaped by cognitive mechanisms similar to those involved in the bouba-kiki effect. Especially when looking beyond Indo-European languages, 'iconic' mappings, where word forms seem to resemble their meanings, appear to play a significant role in many languages (Perniss et al.) 2010 Dingemanse 2012: Imai and Kita. 2014). Some languages have specific word classes where characteristics of the meaning are mimicked or iconically represented in the word. For example, Japanese ideophones allow speakers to depict sensory information through word forms, such as saying nuru nuru when describing something as 'slimy' fuwa fuwafawa when is 'fluffy' (Dingemanse et al.
(016). Similarly, in Siwu, pimbili means 'small belly', while pumbuluu refers to 'fat belly', and non-speakers of those languages can typically guess the meanings of such words (Dingemanse et al. (2016). Even in languages not considered rich in sound symbolism, such as English and Spanish, vocabulary items from specific lexical categories, such as adjectives, are rated relatively high in iconicity (Perry et al. 2013).
Evidence for strong associations between speech sounds and particular meanings has been found in a broad sample of vocabularies from two-thirds of the world's languages BBlasi et al. (2016). Iconic mappings are not only widespread in the world's languages. they also help both children Imai et al.] (2008); Perry et al. (2015) 2018) and adults Nielsen and Rendall (2012) learn new words more easily. They, moreover, allow us to communicate successfully even when a shared language is absent existing vocabulary is insufficient, because moosl associations lay the groundwork for the negotiation of novel words and their meanings (Ramachandran and Hubbard 2001 Cuskley and Kirby 2013 Imai and Kita 2014). Shared cross-modal representations enable people to instantly interpret unfamiliar words even on first exposure by directly linking sensory experiences with meaning. Indeed, experiments with humans that studied the influence of cross- modal preferences on the emergence of novel vocabularies show that iconic strategies are frequently adopted when word forms and meanings can be intuitively mapped, and they help to communicate successfully [Verhoef et al. 2015, 2016h a Tamariz et al. 2018). An understanding of such mappings, however, requires human-lit integration of multi multi-sensory information or an awareness of common cross-modal associations. While multimodal computational models have demonstrated remarkable capabilities across range of tasks, the internal mechanisms through which these systems form and link representations remain opaque. In particular, it is still an open question whether these models integrate visual and linguistic information in ways that mirror human cognitive processes.
2.2 Bouba-kiki effect in VLMs
The bouba-kiki effect, specific example of visuo-linguistic processing, has been studied in VLMs before; however, the results so far seem to be conflicting. One key study in this space is the innovative work by /Alper and Averbuch-Elon (2023), who convincingly demonstrated that patterns aligning with the bouba-kiki effect are reflected in the embedding spaces of CLIP and Stable Diffussin.. They used Stable Diffusion generate images based on pseudowords that were carefully designed to reflect phonetic properties associated with sharp or round shapes. Specifically. they used CLIP's text encoder embed prompts containing either pseudowords descriptive adjectives, while the images generated from souudooord-bssed prompts were passed through CLIP'' vision encoder. This setup allowed both text and image representations to be analysed within the same multimodal embedding space. The embedding similarity between the pscudowords and the adjective or image representations was then used to compute geometric and phonetic scores, indicating alignment with sharp or round associations. They concluded that their findings indicate strong evidence for the existence of cross-modal associations in VLMs. Typical explanations given for the presence of a bouba-kiki kiki effect in humans include experience with acoustics and articulation (Ramachandran and Hubbard 2001] Maurer et al. 2006b, Westbury| (005a)), sffective-semantic properties of human and non-human vocal communication (Nielsen and Rendall 2011), or physical properties relating to audiovisual regularities in the environment (Fort and Schwartz| 2022). This renders the conclusion by Alper and Averbuch-Elor (2023) somewhat surprising, these explanations all involve situated, real-
3

---

world experience with a body and the environment, something these models entirely lack. Moreover, limitations of VLMs in visual grounding have been observed in many other domains (Thrush et al.
2022
Diwan et al., 2022; Kamath et al. 2023 Jabri et al. 2016 Goyal et al. 2017, Agrawal et al. 2018 Jones et al. 2024), suggesting that these models do not integrate textual and visual data in a manner that is human-like. Nonetheless, this work presented an innovative method for testing the bouba-kiki effect in VLMs. It convincingly demonstrated that VLMs encode relationships between word forms and semantic concepts related to roundness and jaggedness. As described previously, these associations are indeed abundantly present in human languages, and prior work with text-only language models has also shown that these models can detect such regularities (Abramova and
Fernández, 2016 Pimentel et al. 2019 de Varda and Strapparava 2022 Marklová et al. (025))
Interestingly though, lida and Funakura (2024) replicated [Alper and Averbuch-Elor experiments for Japanese and found that Japanese VLMMs did not exhibit the expected bouba-kiki effect, even though Japanese is a language rich in sound-symbolism (Dingemanse) (2012), and speakers of Japanese display the strongest bouba-kiki effect compared to speakers of 25 other languages in study by Čwiek et al. (2022). These findings suggest that Alper and Averbuch- Elor's method may not capture true sensory mappings, but instead detects regularities between word forms and meanings that are iaguuage-specifii rather than universal. For example, sounds like /p/, which are typically linked to sharpness according to the bouba-kiki effect, are strongly associated with roundness in ideophonic Japanese words, like pocha-pocha 'chubby', puyo-puyo 'fat', and puku-puku 'puffing up' (lida and Funakura 2024). This suggests that the method used to disambiguate sharp and round pseudowords and images may pick up on relationships between semantic concepts and word forms- being heavily entangled with the choice of ground-truth adjectives-rather than capturing true sensory mappings in languages. Moreover, the vectors used to assign a geometric or phonetic score to pseudoword or image must be sufficiently dissimilar, for which lida and Funakura (2024) reported that this was not the case. The contradictory findings between Japanese and English VLMs highlight the need for a different approach that aligns with human experiments on cross- modal associations.
key ingredient of human bouba-kike experiments is that tests are centred around specifically designed pairs of visual images that minimally differ, but with one more rounded and one more jagged version, as shown in Figure T] For example, |Maurer et al.] (2006a) presented pair of jagged/round images along with pair of bouba/kikillike words and participants were asked to pair them up in the most fitting way. Other studies employed more stringent tests, as in Cwiek jet al. (2022), where only the images were presented side by side, and participants had to choose the best-fitting image after listening to only one of the spoken words at time. Finally, Nielsen and Rendall (2033) presented single images and asked participants to generate novel pseudowords to match the shapes. In all of these cases, humans exhibit strong bouba-kiki effect. Inspired by these studies, Verhoef et al. 2024) used images from existing human experiments and explored four prominent VLMs carefully designed image- word matching tasks. They directly used model probabilities generated by VLMs to match specific pseudowords with images as a measure of preference and found limited evidence for the bouba-kiki effect. Two out of four models (CLIP and GPT-4o) exhibit moderate alignment with human-likk like associations, but only in some of the tests they conducted, and not consistently. The study concludes that cross modsl associations in VLMs are highly dependent on factors such as model architecture, training data, and the specific test used.
Others have also investigated the bouba-kiki effect and other cross-modal associations, such as relation between perceived size and vowels (Loakman et al., 2024) and understanding shitsukan terms (a Japanese concept that captures the sensory essence of object) (Shiono et al.) 2025). However, none of these consistently find a resemblance between the human and model associations. Tseng et al. (224) used the embedding method from Alper and Averbuch-Elor to test sensitivity to sound-symbolic associations in audio-visual models and report that these models capture sound- meaning connections akin to human language processing. Yet, this method again seems to rely on the relationships between semantic concepts and word forms, as was mentioned before. Given the conflicting evidence in this domain, we revisit the bouba-kiki effect through thorough investigation using two versions of CLIP and a wide variety of prompts. This approach introduces a novel method for testing it using visual interpretability methods and directly compares the results with similar findings from human studies.

---

3 Methodology
Cross-modal associations are assessed by prompting two different versions of CLIP Radford et al. 2021), specifically, a ResNet-50 and a ViT version. We deem this as reasonable since state-of- the-art VLMs such as Molmo (Deitke et al.) (024), LLaVA (Liu et al.] 2023). BLIP2 (Li et al. 2023). and InternVL (Chen et al. 2024) commonly use pre-trained frozen vision models and train lightweight alignment layers to align visual features with existing linguistic embeddings present in (large) language models (Liu et al., 2024). The frozen ViT version of CLIP particularly often the basis for many current of-te-of-the-rr VLMs (BLIP2, Molmo, InternVI.. Va,,a, i.a.). If CLIP, as a backbone, does not consistently exhibit human-like cross-modal associations, it is difficult to imagine how additional alignment layers can capture human-like representations without extensive fine-tuning. Moreover, CLIP enables us to extend existing approaches with an interpretabillty- methodology that cannot be applied to proprietary models
Linguistic inputs Models are probed through combining images and labels, the latter of which are pseudowords originating from four different sources. First, two sets of 'original" labels are used, which have been traditionally used the most in human studies (bouba-kiki and madluma-takktte) allowing for explicit comparison of human results with those of VLMs. Second, we borrow English adjectives from Alper and Averbuch- Elor (2023) that are 10 synonyms of 'sharp' and 'curved'. These serve as a baseline informing us whether the models can, in principle, make correct cross-modal associations. This differs from AAlper and Averbuch-Elor (2023). who use adjectives to create a vector to calculate a geometric score. Third, two-syllable labels were constructed following NNelsen and Rendall 2013), using previously established cross-modal patterns in English. Sonorant consonants (M, N.L. L) and rounded vowels (O0, OH, AH) tend to match curved shapes, while plosive consonants (T, K, P) and non-rounded vowels (EE, AY, UH) align with jagged shapes. Combining these yields 36 syllables (e.g.. loo, nah, kee, puh), categorised into four types: sonorant-rounded (S- (S-R), plosive- (S-NR), and coosive-non-rondde into two-syllable pscudowords, loosely replicating the task humans did in Nielsen and Rendall (2013). For analysis, we focus on 'pure' pseudowords that are either fully round-associated (S-R-S-R) or fully sharp-associated (P-NR-P-NR), yielding 162 labels. Fourth, VLMs tested on pseudowords generated using the method from Aller and Averbuch-Elor (2023). Each pseudoword follows a three-syllable structure, where consonants (P, T, K. S, H. X, B. D. G. M, N, L) are combined with vowels (E, I, O, U, A) Pseudowords repeat the first syllable at the end (e.g., 'kitaki", "bodubo"), Only 'pure' composed entirely of syllables from a single class are used, excluding mixed forms like 'kiduki'.
rounded (P-R), conorant-non-rounn
(P-NR). These were paired
The linguistic inputs comprised of label of interest- which should induce certain prefer- ences-an and a prompt ('The label for this image is <label>') such that embedding the label happens at the sentence level and is closer the models' natural objective. Importantly, for each image, we only differ the pseudowords in question, so variation in probability must be a result of the pseudoword. Provided that the pseudowords are generated anew, it is unlikely that the models encountered them during training. Ten different prompts are used (see (pepedppiixx A) to ensure that the results are robust and not an artefact of peculiarities in the prompts. These prompts are sourced from Verhoef et al. 20224) Alper and Averbuch-Elon (2023), or are newly created such that half of the labels appear as a noun, and the other half as an adjective.
Visual inputs The images fed to the models are either curved or jagged. Some of these images are sourced directly from previous works involving human participants (Köhler 1929 1947 Maurer et al.) 2006al Westbury| 2005b)). while others are specifically generated to test cross modal associations in VLMs. These were inspired by the method described in Nielsen and Rendall (201) and already used by Verhoef et al. 2024). The generated images were created by randomly distributing points within circle and then connecting them sequentially using curved lines for curved images and straight line segments for jagged images. Hence, this method generated image pairs that subtly differ only in features that are seemingly important for cross-modal effects. This differs from AAlper and Averbuch-Elor (2023), who generated images using Stable Diffusion, leaving less experimental
"The source data and code are available at http:::://sfffo//gg////
'The letter A is included per Alper and Averbuch Elor 2023 despite that it is typically not regarded to evoke cross-mode moos--mdaal associations in humans.
5

---

Ergln Adechee
Homer 221
Aze etal 2003
01-
Propocten moaohing
02
Assaab eypen
the w rideet This
The sole La colodr
spper for claesn drowriy valato
This cob4e
The vey ddleds
sppon cilee cterd s-idde deseeriiy
The tosk oatat
This
Figure 2: The proportion of congruent responses for matching both images of an image pair correctly (Match 1). A result in which models consistently match images above chance (the grey dashed line) across prompts would suggest the presence of cross-modal associations. This is only the case for the English adjectives, which function baseline. Model: Match + Word_rr ord_type + (1 Word_type (rreppppp
Prompt). Diamonds are descriptive means, and the dots are posterior means.
control and potentially distorting the understanding of which features cause the observed effects, An overview of all image pairs is displayed in Appenddx B
Analyses All our analyses use Bayesian Regression Models as implemented in the brms package (Bürkner 2021) in R (R Core Team 2024). We fit models (using 4 chains of 4000 iterations and warm-up of 2000) to predict the proportion of correct guesses given a Word_/ype with fixed effects for model, prompt, image pair, label pair. The exact model formulas are displayed under each figure. The interpretation of our visualisation is straightforward: an effect is significant if the posterior means and their credible intervals are above the chance level.
4 Probing through probabilities
To assess the preferences of the VIT and ResNet versions, we extract probabilities for all possible labels (i.e., syllables and pseudowords) conditioned an image. For each image, we consider the label with the highest probability to be the model's preference (shown on the left in Figure I). This comparable to how Nielsen and Rendall 2013) tested for human preferences. While using the probability data for all labels would have been possible, previous work demonstrated that using only the best-matching label yielded small bouba- hiki effect, whereas using all probabilities did not ( /Verhoef et al. 2024); such, we use the most promising method. Importantly, we follow Cwiek itt a (2022), who rightfully treated human responses as bouba/kiki-congeeeee matched bouba to a round shape, and crucially, also matched kiki to a spiky shape. In our work, this means that an image pair of a sharp and round shape must be matched with a congruent label.
when participants
We begin by considering the English adjectives, which serve baseline and do not inform us about the bouba-kiki effect. Figure 2 reveals that these are successfully matched to images with round or sharp features. This result consistent across prompts and is also somewhat expected. Yet, performance of roughly 80% across prompts and models could also be considered low given the clear-cut distinctions between images and adjectives. The primary labels of interest (i.c., the pseudowords) can not be robustly and correctly matched to images with corresponding features by either model version. Despite some variability across prompts and models, with some combinations leading to above-chance performance (e.g., ViT, Nielsen and Rendall's s pseudowords, and This drawing <label> or ResNet, /Alper and Averbuch- Elor pseudowords, and A picture of a <label> object), the descriptive means are mostly at chance level. Hence, we conclude that neither model displays clear cross-modal preferences. Interestingly, the individual modalities are, in principle, separable by both models and B.I., suggesting that the difficulty lies in combining information from multiple modalities. This also visible when looking into which labels are most commonly chosen, revealing that model predictions do not differ much even though they are presented
Results

---

with different images Appendix C For the experimental pseudowords, models seem to rely on a few frequently selected labels (the percentage of unique labels for ResNet and ViT is 35%).
This strengthens our observation that no robust syllable-level cross-modal associations influence predictions.
5 Beyond behavioural observations
In addition to using probability data, are interested in knowing whether the predictions are wrong for the right reasons, i.e., do they fail to attend to curved or jagged regions when presented with our images? As such, we further analyse the behaviour and preferences of models when associating an image with a label, using BAM, a technique from the model interpretability literature (Selvaraju et al.) 2020). Grad-CAM offers a visual explanation of model's prediction by calculating the gradients of the target class score. This involves using the cosine similarity between the label and image embeddings in relation to the feature maps from the final convolutional layer or the last attention block. In the case of ViT. we applied Grad-CAN to the last attention layer, which retains spatial information via its attention heads. We specifically focused on the attention from the [CLS] token to the image patches, similar to the approach in Caron et al. (2021) and following public implementations of Grad-CAM (Zakka 202T; Mamooler] 20 Chefer al. 202T). Specifically, to identify which image regions contributed most to the decision, we computed the gradients of the target score with respect to the attention weights. These are averaged across heads, and we then extract the attention weights from the [CLS] token to the image patches by removing the [CLS] column. This is reshaped into 2D spatial grid that acts as a feature map. The values in these feature maps (typically displayed as heatmap) can be interpreted as the importance or contribution of specific region in the image to the model's prediction (see SseeAAdpeddxx D for some visual examples).
This technique enables to very closely mimic the cross-cultural study conducted by Cwiek et al. (002)) in which human participants were shown the two classical bouba-kiki shapes and listened to the spoken words bouba or kiki. Hereafter, they selected which of the two shapes they thought corresponded to the word. To simulate this experiment with VLMs, we concatenate all image pairs into single image containing a curved and jagged shape (n=17). We then identify, for each label-which defines the expected
image region (left or right) receives the most attention from a model using Grad-CAM (see FFigure lp. While the generated heatmaps visually indicate attention patterns, we quantify attention by summing the attention allocated to each curved and jagged part of the image. Taking the total attention allocated to each part, rather than focusing on very specific regions. aligns with how humans perceive faces, objects, and words holistically (Taubert et al., 2011 Zhao et al.] 2016) Wong al.) 201T]. Nevertheless, we also experimented with entropy and centroid attenttioo onfariidof--atteniionn as quantifying measures, but none of these changed the results meaningfully (Appendix D We compare the sum of importance values in the expected region for a given pseudoword within particular category with the sum of importance values in the non-expected region for each image and text label. Based on this comparison, we compute the percentage of trials in which the model consistently focuses more the expected region than on the non-expected region (as shown on the right in Figuur I). The way images are concatenated is balanced such that the target appears eight or nine times on the left or right. Doing this eliminates potential model bias toward objects on the left or right. Initial analyses across models, prompts, labels, and images revealed that both models are relatively consistent in their predictions, regardless of the target location, with ResNet being consistent for 77.0% of the predictions and the VIT variant 73.5% (Appendix D).
Comparing VLMs with humans Using Grad-CAM we compare the performance of both models to that of English-speaking participants as reported by Cwick et al. (0022). While they only inves- tigated the images shown in Figure I] was first described by Wolfgang Köhler (Köhler 1929 1947) and sparked wider interest in cross-moda associations. We use only these two label pairs here, since the larger pseudoword set does not contain paired labels. However, our image generation process, which only varies how points are connected, allows evaluating across all image pairs, beyond just the original images. Specifically, we evaluate whether models attend more to the expected region (e.g., the sharp shape for 'kiki' and 'takete' or the round shape for "bouba" and 'maluma"). A prediction is considered congruent only both labels are correctly matched to their corresponding shapes. This setup closely mirrors human experiments and, for the bouba- kiki label pair, enables direct comparisons.
label pair bouba-kiki. another pair, maluma- takete, combined with the two
7

---

075
Propodten
atnagh
139
Avebel
The
Tous
The took iae
Figure 3: The proportion of correctly matched responses for both labels of label pair (Correct = 1) given an image pair using Grad-CAM. The green line indicates human 'performance' reported in (Cwick et al. 2022). The grey line shows the chance level (25%) the model must map 'bouba' and "Kiki" correctly. Model: Correct descriptive means, and the dots are posterior means.
Label Pair + (1 LabelPair Prompt)) Diamonds
Strikingly, neither CLIP model consistently maps both labels their intended shapes at human- like level (Figure 3); in fact, performance does not reliably exceed chance. These results contradict previous claims that vision-language models exhibit human-like cross-modal associations (Alper and Averbuch-Eine Elor 2023), and instead reinforce earlier findings showing no such effect. If there any situation in which an effect would expected, would be here since these classic word pairs most dominantly present in the data. Yet, these results show that merely learning about an existing cross-modd effect from data distributions different from having mechanistic preference for cross- modal associations, which should not come as surprise.
Analysing pseudowords Extending the analyses beyond the original word pairs, we now assess all pscudowords to examine whether CLIP displays fundamental association between syllables and shapes. The proportion pseudoword-late responses displayed in Figure 4rrevaals that there is again some variability across prompts, but model preferences are relatively consistent. Using this method, the ResNet variant displays general preference to attend to round shapes across all word types (note that not attending jagged shape if the label is of the jagged category means the model attends
Engist Acadhi
2013
Aper 2920
1.30
0.75
8
Propetan
0.20
30
80 36
8 8 36
06 00
9
9
86
84
tettg of
Ties dayps
veyy odeits
coasnng dayets
A-dave addes
Austuee stant
ras datee
This sosa oliaes
This ddlyes
ver odbats
radees
the Bhe
This
Calegory Cance ebyces
Aggos edocaey
Notor
App' Aorr
Figure 4: The proportion of correct matches (Correctrrooortion given word type and category (i.e., curved or sharp). A cross-modal association is indicated when a model consistently matches images above chance (grey line) for both obseevee only with ViT and the English adjectives used for comparison. Model: Correct Proportion ~ 1 + W Word Wype + Category + Word type Calegoryl Prompt). The diamonds are descriptive averages, and the dots are posterior means.

---

to the round shape). Provided that this is also the case for the English adjectives, it is unsurprising that the pseudowords do not display a bouba-kiki effect. The ViT variant is relatively consistent and can reliably distinguish shapes for English adjectives. Yet, using experimental pseudowords collapses performance to chance levels, indicating the absence of human-like associations between shape features and syllables or characters. A qualitative inspection of several attention patterns provides additional evidence against the presence of cross-modal associations in CLIP. This reveals that attention patterns primarily do not focus sharp edges round attributes of images, but instead mainly focus on the centres of shapes or background areas (Appendix Dj. The latter is similar to earlier findings by Dareet al. (2024) who found that ViT networks create artefacts at low- informative background areas for computational purposes rather than describing visual information. Both observations are in contrast with what, at its core, is required for bouba-kiki-lit liki effect.
6 Discussion and conclusion
This work investigated whether vision-and- laaio--an-- models, specifically two versions of CLIP (ResNet and ViT), exhibit human-like patterns of cross-modal association, as reflected in the bouba- kiki effect. We aimed to rigorously evaluate the presence or absence of this phenomenon in model behaviour, using two methodological approaches closely aligned with human experiments and intro- ducing the use of interpretability tools in this domain to probe internal representations. Crucially, we posit that can only speak of consistent model preference when results across both method- ologies are congruent with human-like preferences. Our first experiment, which used image-label probabilities to gauge model preference, showed no clear tross-modal preference for either model. The second experiment utilised Grad-CAM to simulate experimental conditions in human studies. Here, the ResNet variant, which showed an overall preference for round shapes, did not demonstrate behaviour consistent enough to qualify as a bouba-kiki effect by human standards. The based version of CLIP, widely used foundational component in many current multimodal systems, proved capable of matching shapes and English adjectives, but failed to do so for pseudowords, which have even weaker alignment than ResNet. Contrary to prior work suggesting VLMs may encode bouba-kill kikbb-ikkiiikke associations (Alper and Averbuch-Eloo 2023). our findings reveal little to no evidence that CLIP models, under these experimental conditions, exhibit consistent human-likk like mappings between pseudowords and visual shape features. This lack of alignment suggests that, despite their impressive performance across many downstream vision-language tasks, models like CLIP do not appear to represent cross-modal associations in cognitively grounded manner internally. Together, this raises questions about how cross-modal grounding is encoded or inherited in many larger state-of ofe------rrrrrrr art architectures.
Our findings complicate earlier claims about the presence of sound-symbolic associations in model embeddings. While Alper and Averbuch- Elor (2023) found strong evidence for a bouba-kiki effect using embedding similarity scores in CLIP and Stable Diffusion, Lida and Funakura (2024) used the same method with Japanese VLMs, but failed to pick up robust bouba-kiki-like mappings, even though Japanese is rich in sound-symbolism (Dingemanse) (2012). Instead, the embedding similarity scores seem to reflect language-specift regularities rather than general cross-modal preferences. Our approach deliberately avoided this by mirroring psycholinguistic testing paradigms with carefully controlled pairs of novel images. Corroborating other recent studies (Verhoef et al.] 2024] Loakman al. 2024), this empirically grounded probing method yielded results opposite to those of other studies, showing no evidence of a bouba-kiki effect. Whereas Verhoef et al. (2024) report limited, though not consistent, evidence for a bouba-kiki effect, our work differs since it uses a more comprehensive set of pseudowords, amending their analyses and suggesting that there is no general effect. We also ran the same tests using English adjectives, and in contrast to pseudowords, the models are able to make the expected mappings in that case. Furthermore, by employing Grad-CAM in the domain of cross-modal associations, we examined whether models visually attend to shape features that correlate with pscudoword form profiles. This enabled us not only to assess whether the models exhibit human-like cross- modll associations, but also to determine whether they do so for the right reasons by probing the decision-make making pathways within the models. They largely did not attend to the expected shape features, which presents further evidence for the claim that VLMs may lack human-like cross-modal representations.
What causes this misalignment between humans and VLMs? The broader implication is that current VLMs, even those trained on massive paired datasets, lack key component of human like multisen-
9

---

sory understanding: grounded, flexible, and intuitive mappings between sensory modalities. This perhaps not surprising given the lack of embodied interaction in their training and reliance on statistical co-occurrence rather than perceptual salience. Still, it can be argued that the co-occurrences in the training data are full of human-like preferences, including cross-modal associations. So while we may expect VLMs to pick up on these associations, their internal mechanisms inherently differ from those of humans and do not necessarily learn the same preferences from aligned image-text pairs. Previous work on visual grounding (Jones et al. (022), and shortcut learning, i.e., solving a test using unexpected non-human-lit shortcuts (Schwartz and Stanovsky 2022] Mitchell and Krakauer 2023) argued similarly. However, interestingly, we observed that the individual modalities are, in principle, separable. This suggests that CLIP's training objective (i.e., a sentence- levell contrastive loss) and its method for aligning text and image modalities (i.e., cosine similarity) are the culprits. Neither seems to specifically promote learning relations hetween visual features and phonetic elements in language.
Another straightforward difference between humans and VLMs is tokenisation. Unlike humans, who perceive words holistically (Wong et al. 201T), tokenisation can distort word representations, reducing the potential for human-like associations with visual shapes. Inspection of the tokenised pseudowords, however, reveals that this does not occur frequently ("Bouba' becomes 'bou' and 'ba', 'kiki' is one token, 'sepise' becomes 'sep', 'ise', and 'kaykuh' becomes 'kay', 'ku', 'h'. See section A.2 for more examples.) Most phonological structures are preserved that could match shape features. However, tokenisation may still split some pseudowords in ways that wouldn't 't evoke the expected modal associations in humans (c.g.. 'H' in 'OH' might elicit jagged rather than curved associations). In the cases where tokenisation breaks pseudowords into syllables that also show no bouba-kiki effeets in humans, the set of alternative pseudowords is large enough to contain options that could invoke these associations. If models had genuine preference in the direction of bouba-kiki effect, they would prefer those pseudowords that are tokenised as complete syllables, and retain the potential cross-modal association of interest, over those split into non-syllables in which the association may be corrupted. Our results show this is not the case. As such, the potential value of developing models with shared preferences between humans and machines remains significant. Establishing alignment in human and machine understanding of (visual-auditory) form- meaniig mappings and mutual understanding can enhance their interactions (Kouwenhoven et al. 2022). helping in creating AI systems that resemble how humans process and communicate meaning. A potentially rich line of future work includes systematically exploring how design choices in VLMs affect the emergence of cognitively meaningful representations and consistently incorporating linguuiin studies to account for cultural variation (as demonstrated by [iia and Funakura (2024))
Our work has some notable limitations. First, we only test on CLIP versions. We deliberately chose to focus on the CLIP model instead of more capable proprietary models (GPT-4o and Gemini2-Flash, i.... since they are not sufficiently transparent and thus do not help advance our understanding of their internal mechanisms. Nevertheless, it remains interesting to further unravel cross modal associations in more contemporary models. Although they often use frozen ViT variants of CLIP, the alignment layer between CLIP's embedding and the language model's 's embedding may learn preferences that resemble those of humans. While the original bouba-kiki effect is rooted in sound symbolism. our work and that of others (e-g.. Aller and Averbuch-Elor 2023 Loakman et al.) 2024, Tida and Funakura rreies on text, which may influence the outcomes. Yet, prior human experiments present pseudowords both acoustically and in written form (e.g.. and Rendall] 2013), making it challenging to disentangle orthographic from auditory contributions. Moreover, Cuskley et al. (2017) showed that auditory presentation cannot eliminate orthographic effects in literate participants, as characters are strongly associated with particular sounds. These associations are non-arbitrary: writing systems often employ iconic strategies, in which characters representing rounded sounds (rounded vowels, sonorants) tend to have curved shapes (Koriat and Levy 1977 Turoman and Styles 2017). Given this tight coupling between sound and orthography, correspondences between vision and spoken words should be highly correlated with vision-text correspondences. This makes it unlikely that audio-visual models would show more substantial bouba-kikk effects than text- models. Nonetheless, exploring audio-visus models (Tseng et al. 2024) remains valuable direction for understanding how different modalities contribute to cross-modal associations.
Overall, our results reinforce the importance of interpretability and cognitively inspired evaluation when assessing model performance cross-modal reasoning. If VLMs are to serve as truly intuitive human-machine machine interactions, they must not only succeed on benchmark datasets but also exhibit a more human understanding of abstract, grounded concepts.
agents in real-world human
10

---

References
Abramova, E. and Fernández, R. (2016). Questioning arbitrariness in language: a data-driven study of conventional iconicity. In Knight, K., Nenkova, A., and Rambow, O., editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 343-352, San Diego, California. Association for Computational Linguistics.
Agrawal, A., Batra, D., Parikh, D., and Kembhavi, A. (2018). Don't just assume: look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971-4980.
Allen, K. R., Dasgupta, I., Kosoy, E., and Lampinen, A. K. (2025). The in-context inductive biases of vision-language models differ across modalities. In Second Workshop on Representational Alignment ICLR 2025.
Alper, M. and Averbuch-Elor, (2023). Kiki or bouba? sound symbolism in Vision-and-langee models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems, volume 36, pages 78347-78359, Curran Associates, Inc.
Blasi, D. E., Wichmann, S., Hammarström, H., Stadler, P. F., and Christiansen, M. H. (2016). Sound- meaning association biases evidenced across thousands of languages. Proceedings of the National Academy of Sciences, 133339):00188-0088
Bürkner, P.-.. (2021). Bayesian item response modeling in R with brms and Stan. Journal of Statistical Software, 10045):-554
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), pages 9650-9660.
Chefer, H., Gur, S., and Wolf, L.. (2021). Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 397-406.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L.., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. (2024). Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24185-24198.
Cuskley, C. and Kirby, S. (2013). Synesthesia, Cross-Modality, and Language Evolution. In Oxford Handbook of Synesthesia. Oxford University Press.
Cuskley, C., Simner, J., and Kirby, S. (2017). Phonological and orthographic influences in the bouba-kiki effect. Psychological research, 81:1 11-119.
Čwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., Hiovain, K., Kawahara, S., Koutalidis, S., Krifka, M.. Lippus, P., Lupyan, G., Oh. G. E., Paul, J.. Petrone, C., Ridouane, R., Reiter, S., Schümchen, N., Szalontai, Á., Unal-Logacev, Ö., Zeller, J., Perlman, M., and Winter, B. (2022). The bouba/kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B: Biological Sciences, 377(1841):2020033))
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. (2024). Vision transformers need registers. In The Twelfth International Conference on Learning Representations.
de Varda, A. G. and Strapparava, C. (2022). A cross-modal and cross-lingual study of iconicity in language: Insights from deep learning. Cognitive Science, 46(6)ee33177.
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, I.,, Lu, J., Anderson, T., Bransom, E., Ehsani, K., Ngo. H., Chen, Y., Patel, A., Yatskar, M., Callison-Buul Burch, C., Head, A., Hendrix, R., Bastani, F,, VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J.. Skjonsberg. S., Schmitz, M., Sarnat, A., Bischoff, B., Walsh, P., Newell, C..
11

---

Wolters, P., Gupta, T., Zeng, K.-H., Borchardt, J., Groeneveld, D., Nam, C.. Lebrecht, S., Wittlif, C., Schoenick, C., Michel, O., Krishna, R., Weihs, L., Smith, N. A., Hajishirzi, H., Girshick, R., Farhadi, A., and Kembhavi, A. (2024). Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models.
Demircan, C., Saanum, T., Pettini, L.., Binz, M., Baczkowski, B. M., Doeller, C. E. Garvert, M. M., and Schulz, E. (2024). Evaluating alignment between humans and neural network representations in image-based learning tasks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Dingemanse, M. (2012). Advances in the cross- wross-liggusstc study of ideophones. Language and Linguistics Compass, 6(10):654-672
Dingemanse, M., Schuerman, W., Reinisch, E., Tufvesson, S., and Mitterer, H. (2016). What sound symbolism can and cannot do: Testing the iconicity of ideophones from five languages. Language, 97
Diwan, A., Berry, L.,, Choi, E., Harwath, D., and Mahowald, K. (2022). Why is winoground hard? investigating failures in visuolinguistic compositionality. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2250,,25,, Abu Dhabi, United Arab Emirates, Association for Computational Linguisties.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X.. Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations.
Fort, M. and Schwartz, J.-L. (2022). Resolving the bouba-kiki effect enigma by rooting iconic sound symbolism in physical properties of round and spiky objects. Scientific reports, 12(1):19172.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. (2017). Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913.
He, K.. Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Hockett, C. (1960). The origin of speech. Scientific American, 203:88-990 96.
lida, H. and Funakura, H. (2024). Investigating iconicity in vision-ane and language models: A case study of the bouba/kikieffect in japanese models. In Proceedings of the 46th Annual Conference of the Cognitive Science Society, volume 46.
Imai, M. and Kita, (2014). The sound symbolism bootstrapping hypothesis for language acquisition and language evolution. Philosophical transactions of the Royal Society B: Biological sciences, 369(1651):22 (11::51)):2002222 302988.
Imai, M., Kita, S., Nagumo, M., and Okada, H. (2008). Sound symbolism facilitates early verb learning. Cognition, 109(1):54-65.
Jabri, A., Joulin, A., and Van Der Maaten, L. (2016). Revisiting visual question answering baselines. In European conference computer vision, pages 727-739. Springer.
Jones, C. R., Bergen, B.. and Trott, (2024). Do multimodal large language models and humans ground language similarly? Computational Linguistics, 50(3):1415-1440.
Kamath, A., Hessel, J., and Chang, K.W. (2023). What's "up" with vision-language models? investigating their struggle with spatial reasoning. Bouamor, H., Pino, J., and Bali, K., editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 9161-9175, Singapore. Association for Computational Linguistics.
Köhler, W. (1929). Gestalt Psychology. New York: Horace Liveright.
12

---

Köhler, W. (1947). Gestalt Psychology. (2nd ed.) New York: Horace Liveright.
Koriat, A. and Levy, I. (1977). The symbolic implications of vowels and of their orthographic representations in two natural languages. Journal of Psycholinguistic Research, 6(2):93-103.
Kouwenhoven, TT,, Peeperkorn, M., Van Dijk, B., and Verhoef, T. (2024). The curious case of representational alignment: Unravelling visio-linguistic tasks in emergent communication. In Kuribayashi, T., Rambelli, G., Takmaz, E.,, Wicke, P., and Oseki, Y., editors, Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 57-71, Bangkok, Thailand. Association for Computational Linguistics.
Kouwenhoven, T., Verhoef, T., De Kleijn, R., and Raaijmakers, S. (2022). Emerging grounded shared vocabularies between human and machine, inspired by human language evolution. Frontiers in Artificial Intelligence, 5:886349.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, J. (2017). Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253.
Li, J., Li, D., Savarese, S., and Hoi, S. (2023). Blip-2: bootstrapping language-imag pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMI.R.rg.
Li, Li. unified vision- language understanding and generation. In International conference on machine learning, pages 12900. PMLR.
Li. D., Xiong. C., and Hoi, S. (2022). Blip: Bootstrapping language- -aaggge--mmeeeee pre- pre-training for
Liu, H., Li, C., Li, Y., and Lee, J. (2024). Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages (6296-26306.
Liu, H., Li, C., Wu, Q. and Lee, Y. (2023). Visual instruction tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems, volume 36, pages 34892-34916, Curran Associates, Inc.
Loakman, T., Li, Y., and Lin, C. (2024). With ears see and eyes to hear: Sound symbolism experiments with multimodal large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pages 1867, Miami, Florida, USA. Association for Computational Linguistics.
Mamooler, S. (2021). Clip explainability.
Marklová, A., Milička, J., Ryvkin, L., Ludmila Lacková Bennet, and Kormaníková, L. (2025). Iconicity in large language models.
Maurer, D., Pathman, T., and Mondloch, C. J. (2006a). The shape of boubas: Sound-shape corre- spondences in toddlers and adults. Developmental science, 93):316-322.
Maurer, D., Pathman, T., and Mondloch, C. J. (2006b). The shape of boubas: Sound shape corre- spondences in toddlers and adults. Developmental science, 93):3116-33
Mitchell, M. and Krakauer, D. C. (2023). The debate over understanding in ai's large language models. Proceedings of the National Academy of Sciences, 120(13)ee2155007722
Nielsen, A. and Rendall, D. (2011). The sound of round: evaluating the sound-symbolic role of consonants in the classic takete-maluma phenomenon. Canadian Journal of Experúmental Psychology/Revue canadienne de psychologie expérimentale, 65(2):115.
Nielsen, A. and Rendall, D. (2012). The source and magnitude of sound-symbolic biases in processing artificial word material and their implications for language learning and transmission. Language and cognition, 11-2::15-
Nielsen, A. and Rendall, D. (2013). Parsing the role of consonants versus vowels in the classic takete-maluma phenomenon. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 65))
13

---

Perniss, P.. Thompson, R., and Vigliocco, G. (2010). Iconicity as general property of language: evidence from spoken and signed languages. Frontiers in Psychology. 1(227).
Perry, L. K., Perlman, M., and Lupyan, G. (2015), Iconicity english and spanish and its relation to lexical category and age of acquisition. PloS one, (019))e0137147.
Perry, L. K., Perlman, M., Winter, B., Massaro, D. W., and Lupyan, G. (2018). Iconicity in the speech of children and adults. Developmental Science, 215322
Pimentel, T., McCarthy, A. D., Blasi, D., Roark, B., and Cotterell, R. (2019). Meaning to form: Measuring systematicity as information. In Korhonen, A., Traum, D., and Mårquez, I.., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1751-1764, Florence, Italy. Association for Computational Linguisties.
Core Team (2024). R: Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763 PMLR.
Ramachandran, V. S. and Hubbard, E. M. (2001). Synaesthesia window into perception, thought and language. Journal of Consciousness Studies, 8(12):3-34.
Schwartz, R. and Stanovsky, G. (2022). On the limitations of dataset balancing: The lost battle against spurious correlations. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Findings of the Association for Computational Linguistics: NAACL 2022, pages 2182-2194, Seattle, United States. Association for Computational Linguistics.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2020). Grad-cam: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vision, 128(2):336-33) 359.
Shiono, D., Brassard, A., Ishizuki, Y., and Suzuki, J. (2025). Evaluating model alignment with human perception: A study on shitsukan in LLMs and LVLMs. In Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., and Schockaert, S., editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 11428-14444 Abu Dhabi, UAE Association for Computational Linguistics.
Tamariz, M., Roberts, S. G., Martínez, I., and Santiago, J. (2018). The interactive origin of iconicity. Cognitive Science, 42(1):334-344 349.
Taubert, J., Apthorp, D., Aagten-Murphy, D., and Alais, D. (2011). The role of holistic processing in face perception: Evidence from the face inversion effect. Vision Research, 51:1273-1278.
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., and Ross, C. (2022). Winoground: Probing vision and language models for visio-linguistic compositionality. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2238-5248.
Tseng, W.-C., Shih, Y.J.., Harwath, D., and Mooney, R. (2024). Measuring sound symbolism in audio-visual models. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 1165-1172.
Turoman, N. and Styles, S. J. (2017). Glyph guessing for "oo'an and 'ce': Spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts. Royal Society open science, 170822
Verhoef, T., Kirby, S., and De Boer, B. (2016a). Iconicity and the emergence of combinatorial structure in language. Cognitive science, 1968))19691199
Verhoef, T., Roberts, S. G., and Dingemanse M. (2015). Emergence of systematic iconicity: Transmission, interaction and analogy. In Proceedings of the Annual Meeting of the Cognitive Science Society, pages 2481-2486 Cognitive Science Society.
14

---

Verhoef, T., Shahrasbi, K., and Kouwenhoven, T. (2024). What does kiki look like? cross moosll asso- ciations between speech sounds and visual shapes in rision-and-langual models. In Kuribayashi. T., Rambelli, G., Takmaz, E., Wicke, P., and Oseki, Y., editors, Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 199-213, Bangkok, Thailand. Associa- tion for Computational Linguistics.
Verhoef, T., Walker, E., and Marghetis, T. (2016b). Cognitive biases and social coordination in the emergence of temporal language. In Proceedings of the Anmual Meeting of the Cognitive Science Society, volume 38, pages 2615-2620.
Westbury, C. (2005a). Implicit sound symbolism in lexical access: Evidence from interference task. Brain and language, 93(1):10-19
Westbury, C. (2005b). Implicit sound symbolism in lexical access: Evidence from an interference task. Brain and language, 93(1):10-19.
Wong, A. C.-N., Bukach, M., Yuen, C., Yang, L., Leung, S., and Greenspon, (2011). Holistic processing of words modulated by reading experience. PLOS ONE, 6(6):1-7.
Zakka, K. (2021). A playground for clip-like models.
Zhao, M., Bülthoff, H. H., and Bülthoff, (2016). Beyond faces and expertise: Facelike holistic processing of nonface objects in the absence of expertise. Psychological Science, 27(2):213-222. PMID: 26674129.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
section cllearl shows that both models perform below chance level. section directly compares computational results work by (Cwiek al. 2022).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The final section includes some limitations to our methodology, and reliance on text-based models instead of audio-visual models. We, moreover, discuss for each experiment how consistent the models' predictions are, regardless of whether they make human-like associations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification:
Our work does not focus on making a theoretical contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The method section (section 3) clearly describes how our linguistic inputs are made, or where they originate from. Idem for the images used. The Bayesian Regression Models used are also clearly specified in each analyses including the parameters used fit them.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
17

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our source code and the data will be published on OSF upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ca
swt.mmmmssssooococooooooc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7
httpssummssiiooooiioooooooio for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Similar to the result reproducibility, our methodology is clearly described in section 3 The methods to create the images and labels are also described detail. The tests used to make our contributions are also clearly specified underneath each figure. Moreover, the source code will be available upon publication, exposing any other experimental details. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide credible intervals in each figure and explain how analyses should be interpreted (section 3). Morcover, we explicitly state the models fitted and the parameters used to do the fitting.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiments do not need heavy computing since do not train new models or fine-tune anything. All the code and analyses ran on a single MacBook Air M1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httph::t///// :tttp::://neuiisee httparppp../ppp.bii///ttttttttte
Answer: [Yes]
Justification:
We do violate the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [No]
19

---

Justification: Our work does not discuss societal impact, as it primarily focuses on whether current CLIP models (already widely available) align with human cross-modal preferences. We do not develop new model that may be used by society, deal with biased data that may result discrimination, or any related issue. As such, we do not deem it necessary to discuss societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no need for safeguards since we only evaluate already available models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the code-base we used for Grad-CAM (eccton 5), and properly mention the sources of the images and labels that we did not generate ourselves (section 3).
20

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicablle as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not use human subjects but base our comparisons human performance on other work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer:
[NA]
Justification:
Our work did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

A Linguistic inputs
The experimental inputs tested across different prompts robustness. They originate
from earlier investigations (Alper and Averbuch-Elow 2023; Verhoef et al. 2024) and are extended
with additional prompts such that the label occurs equally frequently noun and adjective (Table 1
Prompt
Word type
Origin
The label for this image is <label>
Noun
Verhoef et (024
This is <label>
Noun
new
A drawing of a <label>
Noun
new
This drawing is <label>
Adjective
new
This thing is <label>
Adjective
Alper and Averbuch Elor
2023
A <label> object
Adjective
Alper and Averbuch- Elor
2023
A picture of a <label> object <label>
Adjective
Alper and Averbuch-Elor
2023
Noun
Alper and Averbuch-Fllor
2023
This looks like <label>
Noun
new
This is very <label>
Adjective
new
Table 1: Different linguistic inputs are used test our models for cross cross-modal associations.
A.1 Textual embeddings
To make correct associations, the models must, at minimum, be able to disambiguate labels (real and non-words) from each other. If they fail do so, is presumably also impossible to link certain words and their linguistic features to shape-specific features. |Figure djisplays a t-SNE visualisation of the models' embeddings and reveals that they, in principle, should be able to disambiguate labels from different categories within word type.
ResNet
VI
adjective
Sharp edjectives
Appe's roune
Alper's sharp
Niatsen's round
sharp
Figure 5: t-SNE plot showing how the language models of different CLIP variants interpret labels from different categories. The colour shades indicate which word type label in a category belongs to. In order to correctly match labels to images with shape-specific features, a model must be able to discriminate word types between labels of the same category. This is clearly possible. This plot shows the embeddings for the prompt: The label for this image <label>, Different plots result in similar distributions.
A.2 Tokenisation
To determine whether byte-pair encodings or our pseudowords break phonological structures, we qualitatively assess set of tokenised examples. These pseudowords parsed as elements of the
23

---

Figure 6: Image pairs from (Maurer al. 2006b)
sentence prompts, but we only focus the pseudowords themselves. The list below reveals that most of the target words are tokenised way that preserves at least some phonological structures that could be matched to shape features.
Bouha - 'bou' and 'ba'
Kiki > 'kiki'
Takete + 'take', 'te'
Maluma - 'mal', 'uma'
Xehaxe + 'xe', 'ha', 'xe'
Lohlah 'loh', 'lah'
Sepise + 'sep', 'ise"
Kaykuh "kay', 'ku', 'h'
Loomoh 'loom', 'oh"
Mohmah + 'moh', 'mah'
B
Visual inputs
The full set of images with visual shapes that used in the experiments is shown here. Besides the original image pair from Köhler [9299 1947). we used four image pairs from Maurer et al.) (2006b). displayed in Figure 6) four from (Westbury] 2005a Figure 7). and eight generated pairs following the method described by Nielsen and Rendall (2013) and used in (Verhoef et al., 2024, Figure 8).
For each image pair, the Curved version is displayed on the left and the Jagged version the right,
B.1 Visual embeddings
Similar to the textual embeddings presented before, Figure displays a t-SNE visualisation of models' visual embeddings. It is visible that the models should, in principle, be able to disambiguate our target images based on their condition, even when they only differ how the randomly distributed points are connected.
C Unique labels
To test whether the models the first experiment (section change their predictions according to set of images, report the ratio of unique labels Table 2 In this case, not concerned with

---

Figure 7: Image pairs from (Westbury 2005a)
15
Figure 8: Newly generated image pairs
ResNet
VIT
Condition Jagged Curced
Figure 9: t-SNE plot showing how the vision models of different CLIP variants interpret images from conditions. The colour shades indicate which image condition an image belongs to. In order to correctly match labels to image, model must be able discriminate the images based on their shape. This is clearly possible.
25

---

Model
Word type
Ratio
ResNet
Initial words
.655
ResNet
English adj.
.255
ResNet
Nielsen et al.
.329
ResNet
Alper al.
.44
ViT
Initial words
.800
VIT
English adj.
.320
VIT
Nielsen et al.
.331
VIT
Alper et al. .450
Table The average ratio of unique labels chosen for each image (n=34) across different prompts (n=10) in different label sets. The latter means that we divide the unique labels by the length of the set of possible labels to gauge diversity. high ratio indicates that the corresponding model assigned different labels to different images.
whether the models make predictions that align with the bouba-kiki effect, but rather with the diversity of their predictions. We calculate the ratio of unique labels for each word type across all images and the ten prompts. These ratios should, in a scenario where cross-modal associations are present, be high, and the number of correct trials, as seen in Figure 2] would be above chance. However, it is clear that the models somewhat change their predictions (mainly for the initial pseudowords) when they conditioned on different images, but mostly collapse onto the same labels for different images. Table 2] confirms this by showing that there is only slight variation among picked labels for each prompt. Given that responses are only counted as congruent when both predictions (i.e. the prediction for a jagged and curved shape) are correct, this explains why we find that both models perform at chance levels. A qualitative example is provided in Table 3] which shows the unique labels and their ratios for a single prompt. The model and word type that are most affected by different images (i.e. have a high ratio of unique words) also exhibit the strongest bouba-kiki like effect Figure 2 This appears to happen even though, in the case of both ViT and ResNet, and Alper and Averbuch-Eloo labels, the majority of the labels would be associated with sharp images by humans.
Model
Word type
Ratio
Uniquely chosen labels
ResNet
Initial words
.200
bouba
ResNet
English adj.
300
angular, circular, curved, prickly, rotund, spiky kaykee, kubpay, kuhpec, kuhpuh, lahmoo, lohloh, lohmah, Iohmoo, mohmah, nahmoo, nohmoo, paykuh, peepay, teepee kehake, kehike, ladula, lunulu, malama, nulunu, paxapa, pihapi, pikepi, pikipi, pixapi, sepise, tatata, tekete, xaxixa, xehexe
ResNet
Nielsen et al.
.412
ResNet
Alper et al.
.471
VIT
Initial words
500
bouba, takete
VIT
English adj.
.50
angular, circular, curved, rotund, spiky keekay, kuhtay, lahmoo, lohlah, loomoh, mahloh, mahnoh, mohloo, mohnoh, nohloh, noonoo, taypay, taypuh, teepee dododo, hexehe, kixiki, lamola, lubalu, lunulu, mamuma, monomo, mubomu, patipa, pisipi, pixapi, sahisa, satasa, tehite, texate, xahexa, xakixa, xasixa, xehexe, xikexi, xipaxi, xisixi
VIT
Nielsen et al.
.412
VIT
Alper al.
.676
Table 3: The set of unique labels chosen across all images (n=34) and its ratio for example prompt ('The label for this image is <label>'). The latter means that we divide the unique labels by the length of the set of possible labels to gauge diversity. A high ratio indicates that the corresponding model assigned different labels to different images.
26

---

D Grad-Cam visualisations, consistency, and additional results
Figure 10 provides example visualisations of the attention patterns for different randomly selected image parrs and randomly selected labels across both models. Here, sampled different image pairs for the models as to give a complete view of the images used. The pseudowords are consistent for the columns. The images reveal that models mostly attend to the centres of shapes and/or focus non-informative background areas. The latter is also described by Darcet et al. (2024). Although this small sample, it clear that the target labels do not steer models towards shape-specific features.
edgy
ViT
Figure 10: Visualising the attention pattern for the text prompt: "The label for this image is <label>' in both models.
D.1 Prediction consistency
To test whether the models in the second experiment (section 5) change their predictions resulting from different shape positions, we report the ratio of consistency in Table 4, In this case, we are not interested in whether the models make predictions that align with the bouba-kiki effect, but rather in the consistency of their predictions. Both models are rather consistent in the mappings they make between labels and shapes.
D.2 Quantifying model preference
The results presented in section SJutilise the sum of attention values to quantify the models' pref- erences, as this aligns with humans' holistic perception (Taubert et al. 2011 Zhao et al.) 2016 Wong al., 2011)) Though artificial models may show their preference differently. To this end, we additionally experimented with the entropy and centroid-of-attention as quantifiers of model preference.
Comparing the predictions (averaged over all images, labels, and prompts) quantified by the centroid of attention with the sum of attention, find that the predictions using the centroid of attention strongly overlap with those resulting from the sum of attention (ViT: 85.6% and ResNet: 88.0%). Given this considerable overlap, it is not surprising that the results remain highly similar when using
Model
Word type
Ratio
ResNet
Initial words
.799
ResNet
English adjectives
.761
ResNet
Nielsen et
.758
ResNet
Alper et al.
.765
VIT
Initial words
.723
ViT
English adjectives
.749
ViT Nielsen et al.
.722
ViT
Alper et al.
.739
Table 4: The ratio of consistently attending to the same shape in a different position when presented with the same label. Values are averages across image pairs, prompts, and labels. A high ratio indicates that the corresponding model consistently focuses on the same shape, even when the target location is different.
27

---

Model
Sum
Entropy
Centroid
ResNet
.519
.488
.515
ViT
.522
.514
.515
Table The correctness of all model prediction types using different quantifications averaged across all images, labels, and prompts.
this alternative method TTble 5 Yet, for the entropy of attention, the predictions (where the image half with the lower entropy acts as the model's preference) for ViT overlap strongly (80. 88.2%) but not for ResNet (21.1%). This entropy method, despite yielding different predictions, still results in slightly lower number of correctly predicted images, with only 48.8% of the shapes correctly identified, compared with 51.9% for the quantification method that uses the sum of attention. So were entropy -aaedd predictions, the performance would be even worse than predictions based on the sum of attention.
28


Reviews:
[{"review_id": "o09XtAgdc0", "reviewer": "Reviewer_BFRt", "summary": "The paper investigates whether vision-and-language models (VLMs), specifically two variants of CLIP (ResNet and Vision Transformer), exhibit human-like cross-modal understanding by testing the bouba-kiki effect—a phenomenon where people associate round shapes with the word \"bouba\" and jagged shapes with \"kiki.\" Using prompt-based evaluation and Grad-CAM for visual attention, the study finds that:\n* ResNet shows some preference for round shapes, but\n* The models do not consistently demonstrate the bouba-kiki effect.\n* Their performance falls short of human-like cross-modal integration.\nThese findings suggest that current VLMs have limited internal representations of such intuitive, cross-modal associations, raising questions about the depth of their \"understanding.\"", "questions": "- Please motivate why the work is valuable for the machine learning community as you do not propose strategies to improve cross-modal associations. \n- What is the practical impact of investigating the bouba-kiki effect?", "limitations": "The limitations of the work are discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper is well written and easy to follow. \n- The paper provides a good overview of the literature regarding the bouba-kiki effect.\n- The results are obtained both with synthetic and natural data.  \n \nWeaknesses: \n- The paper does not contribute to novel machine learning models. Its contribution is interesting for the natural language processing and computer vision domains, so EMNLP, CoNLL and CVPR might be better venues for its publication. \n- It is strange that a CLIP model was chosen as underlying vision-language foundation model. It is well known that alignment in CLIP is computed at the sentence/scene level and not at the level of individual words, which devalues the findings of the paper.", "was_revised": true, "final_justification": "The answers to my questions mitigate my concerns.", "rebuttal": "First of all, we would like to thank you for the feedback. We are happy that you had only a few weaknesses/questions, that the paper was easy to follow, and that you enjoyed the overview of the literature. Please find our response to the weaknesses and questions below. \n\n# Weaknesses\n**Contribution to machine learning models –** We acknowledge that our work is primarily in the domain of vision and natural language. Nevertheless, there are several reasons why we think it is as interesting and relevant for the NeurIPS community as for venues like EMNLP or ACL. Our work fits squarely within several themes listed in the CfP, such as ‘deep learning’, ‘neuroscience and cognitive science’.  Moreover,  the NeurIPS community oftentimes advances deep learning techniques (generative AI, foundation models, LLMs, etc.). Our work reveals an important shortcoming in current approaches, which will hinder the continued development of models that aim to align human and computer visio-linguistic, and more generally multimodal, processing. This is arguably equally important as introducing new methods, as evidenced by the fact that Alper and Averbuch-Elor (2023)—whose conclusions appeared to have been drawn somewhat prematurely, given our new findings—also published their findings (spotlight) at NeurIPS. \nFinally, as also pointed out as a strength by reviewer xGW7, our work evaluates the same problem via different methods to provide a more comprehensive and nuanced picture. Such practices are, in our opinion, of value to the NeurIPS community as we believe that our findings not only could inspire others to develop models that take cross-modal associations into account, but also introduce several ways to test for such associations in a human-aligned manner.\n\n**CLIP as tested model –** There seems to be a misunderstanding. You are completely correct that alignment in CLIP is calculated at the sentence level. As spelled out in lines 202-204 and shown in table 1 (appendix A), we indeed embed the pseudowords into sentences (10 versions) such that this alignment score can be calculated at the sentence level. Importantly, for each image, we *only* differ the pseudoword in question, so variation in probability *must* be a result of the pseudoword. We hope that this alleviates your concerns about the use of CLIP. \n\n# Questions:\n**Contribution and value of this work –** Please see our answer as a response to weakness 1. \n\n**Practical impact –** The importance of shared perception/experiences/assumptions is often overlooked in the machine learning community (see also reviewer ZTfi). The bouba-kiki effect, or cross-modal associations in general, is one example for which we would like there to be alignment between humans and models since this results in a more intuitive understanding of each other’s prior expectations and assumptions. A lack thereof may hinder natural interactions between humans and machines (Verhoef et al. 2024), especially when considering the dynamic and adaptive nature of human language. Training models in a way that allows them to develop human-like prior expectations or instilling models with them may help alleviate this and could even improve learning efficiency (Lake et al., 2017). The practical impact of our work will be incorporated in the updated version of our paper.\n\n# References\nLake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building machines that learn and think like people. Behavioral and Brain Sciences. 2017;40:e253. doi:10.1017/S0140525X16001837"}, {"review_id": "wpKEQsSNBm", "reviewer": "Reviewer_xGW7", "summary": "This paper revisits the classic bouba-kiki effect within the context of vision-language models (VLMs), particularly two CLIP variants (ResNet and ViT). Unlike previous studies that rely on embedding similarity or generation-based probing, this paper introduces a methodology tightly aligned with human psycholinguistic experiments. \n\nThe findings show that, despite some sensitivity to semantic adjectives (e.g., \"round\", \"spiky\"), CLIP models do not consistently exhibit the bouba-kiki effect for pseudowords, failing both in behavior and visual attention alignment.", "questions": "- Did you explore whether the CLIP models consistently output the same label regardless of the image input (i.e., label collapse), and if so, how does this affect interpretability?\n- Did you analyze the spatial distribution of attention beyond total intensity? Your Grad-CAM analysis uses summed activation across curved vs jagged halves, but this could overlook spatial patterns within each region. For example, did you consider using centroid-of-attention, entropy, or overlap with curvature-dense regions to better quantify alignment with shape features?\n- How does tokenization affect the model's ability to form shape-word associations, particularly for pseudowords? Since CLIP and other Transformer-based models operate on subword or byte-pair encodings (e.g., \"bouba\" might be split into “bou” and “ba”), this fragmentation may distort the phonological structure that underlies the bouba-kiki effect in humans, where sound symbolism operates over holistic auditory forms. Did you analyze how the pseudowords are tokenized by the CLIP text encoder, and whether token splits align with phoneme boundaries or disrupt the intended phonetic profiles? Could this explain the models' failure to form shape-sound associations, especially compared to their robust performance on real adjectives?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- This paper stands out not for discovering a new bouba–kiki effect, but for disproving its presence in CLIP with a much more rigorous and cognitively aligned methodology. It raises the bar for testing cross-modal iconicity in vision-language models. Prior work (e.g., Alper & Averbuch-Elor) used embedding similarities or generation-based tasks with less experimental control. This paper brings direct comparability to human data, especially from Ćwiek et al. (2022).\n- Human-aligned evaluation protocol: The authors closely mirror experimental designs used in psycholinguistics (e.g., forced-choice tasks with minimal shape differences), allowing for principled comparison between human and model behavior.\n- A strong plus to use Grad-CAM for cross-modal interpretability: By examining visual attention patterns during label-shape association, the paper moves beyond accuracy to probe the internal mechanisms of cross-modal processing in VLMs.\n\n## Weaknesses\n\n- Sound-text-vision entanglement is underexplored: Since the original bouba-kiki effect is rooted in auditory perception, the textual substitution (e.g., \"bouba\" as a string token) may be too weak a proxy. The paper acknowledges this but does not systematically explore text-to-speech models or phoneme-level grounding.\n- Model scope is limited to CLIP: While CLIP is foundational, the paper does not evaluate whether more modern VLMs (e.g., BLIP-2, LLaVA) or large multimodal models (e.g., GPT-4o) show similar or different patterns, which weakens the generality of the conclusions.\n- While the paper introduces novel methodological angles, the core question of whether VLMs exhibit the bouba-kiki effect has already been substantially explored in prior work, particularly by Alper & Averbuch-Elor (2023) and Verhoef et al. (2024). As such, the contribution may feel incremental, and the bouba-kiki phenomenon, though conceptually interesting, may be too narrow in scope to carry a full conference paper on its own.", "was_revised": true, "final_justification": "My concerns are addressed indeed, but no more exciting results are added during rebuttal. I would keep my current ratings.", "rebuttal": "First of all, we wish to thank you for the positive feedback. It gives us joy to read that you found our work interesting. \n\n# Weaknesses\n**Sound-vision entanglement –** We agree with you that sound-vision entanglement is underexplored in our work and that it is a very interesting future direction. There are, however, some points that should be noted. While it is widely assumed that the bouba-kiki effect in humans is largely driven by sound-shape correspondences, many prior experiments with humans actually provide both an acoustic and written form for tested pseudowords (which was also the case in the study by Nielsen and Rendall (2013) which formed the basis for one of our word sets), so many prior studies cannot clearly disentangle the roles of orthography versus sound in the bouba-kiki effect. One study by Cuskley et al. (2017) explicitly explored the role of orthography and showed that auditory presentation cannot eliminate potential effects of orthography in literate human participants, since characters are typically strongly associated with a particular speech sound. Moreover, these associations are often not arbitrary but also shaped by human word-shape biases, resulting in systems where characters that represent rounded sounds (like rounded vowels or sonorant consonants) tend to have more curves (Koriat, 1977;Turoman & Styles, 2017). Correspondences between vision and spoken words are therefore expected to be highly correlated with correspondences between vision and written form. This makes it unlikely that a model trained on spoken language would be more likely to exhibit the bouba-kiki effect than a model trained on text would.  \nGiven the strong relationship between sound and wordforms and the fact that these words cannot be reliably associated with images, it would be surprising if an audio-visual model would pick this up. Despite our scepticism, we definitely think that this is worth exploring further with models that differ in how they combine multiple modalities.\n\n**Limited model scope –** We indeed only test two CLIP variants for two reasons. First, CLIP is a foundational model that is often used in more contemporary models (lines 176-178). If this model does not show human-like associations, it is difficult to imagine that further-tuned models do show human-like preferences. Second, Verhoef et al. (2024) tested CLIP, VILT, BLIP-2, and GPT-4o in a similar vein to the first experiment in our paper. Out of these models, only CLIP and GPT-4o showed limited evidence of a bouba-kiki effect. Given the fact that we extend the existing approaches with an interpretability-based methodology to unravel the attention preferences of VLMs, we focus only on CLIP here because GPT-4o does not provide the required openness for this method. It goes without saying that newer methods are released as we speak (e.g., the pointing behaviour of AI2’s Molmo—which also uses CLIP as a visual backbone), which require further investigation.\n\n**Incremental contribution –** Since the topic of our paper is relatively new in this community (introduced by Alper and Averbuch-Elor at NeurIPS 2023), we would argue that it is still maturing, and we believe our contributions are more than incremental. This is also corroborated by reviewer 74nD, who states that our work differentiates itself from prior work. Given the strengths you identified (points 2 & 3), it seems to us that you agree with this. Finally, our methodology is strongly aligned with humans and can inspire others to align their methods with human experiments as well. \n\n\n# Questions \n**Label collapse –** This is super interesting, and we did indeed explore this in Appendix C. Table 2 shows that both CLIP versions tend to pick some words more often than others. However, out of 34 trials (images), the average ratio of ±33% across ten prompts indicates that around 10 different pseudowords are picked across all the images. As such, it seems like there is a preference for some pseudowords but no such thing as label collapse. \n\nAs a response to your question, we also looked into the most frequently chosen pseudowords and provide some examples. For ViT and Alper et al’s pseudowords, some of the labels that appear most are “manuma”, “kiteki”, “gologo”, “xehaxe”, “lunulu”, “mogumo”. For ResNet these were: “xekaxe”, “sepise”, “xesixe”, “xehexe”, “gologo”. In the case of ResNet and Nielsen et al. pseudowords: “lahloh”, “mohmah”, “kaykuh”, “mohmoh”, “puhkuh”. And for ViT: “teepee”, “lohlah”, loomoh”, “lohlah”, “looloh”, “loomah”. The updated version of our paper will include this extra information.\n\n**Spatial distribution of Grad-CAM –** Thank you for the suggestion. This is indeed something we explored but did not mention in the paper since we argue that the sum of attention aligns with the holistic view humans have (lines 272-274). Nonetheless, we also used different spatial distributions, such as the centroid of attention and the entropy method you mentioned.\n\nComparing the predictions chosen by the centroid of attention with the sum of attention, we find that 86.8% of the predictions are the same. Given this large overlap, the results likely would not change if we used this alternative method. \n\nIn the case of entropy as a source of prediction (where the image half with the lower entropy acts as the model’s preference), we find that only 50.6% overlaps with the predictions based on the sum of attention. Yet, this method results in a lower number of correctly predicted images, with only 50.7% correctly identified shapes as opposed to 55.4% for the method which uses the sum of attention. So if we were to use entropy-based predictions, the performance would be even worse than predictions based on the sum of attention.\n\n**Tokenisation –** This is an interesting point that requires further clarification since it is not immediately clear whether tokenisation hurts or benefits bouba-kiki-like associations (see also the question of reviewer 74nD).\n\nInterestingly, analyses reveal that humans even display a bouba-kiki effect at the level of single syllables (Nielsen and Rendall, 2013), suggesting that the associations do *not* rely on complete words. Since tokenisation often preserves syllabic structures, our analysis is not so different from investigating the bouba-kiki associations in humans.\n\nTo add strength to this argument, we tokenised the most common labels mentioned in our answer to your first question. Some examples:\n\nBouba → “bou” and “ba”\n\nKiki → “kiki”\n\nTakete → “take”, “te”\n\nMaluma → “mal”, “uma”\n\nXehaxe → “xe”, “ha”, “xe”\n\nLohlah → “loh”, “lah”\n\nSepise → “sep”, “ise”\n\nKaykuh → “kay”, “ku”, “h”\n\nLoomoh → “loom”, “oh”\n\nMohmah → “moh”, “mah”\n\n\nThese examples, which we will add to an updated version of our paper, show that most of the target words are tokenised in a way that preserves at least some phonological structures that could be matched to shape features. \n \nNonetheless, the tokenisation process in language models may split syllables or pseudowords into tokens that would not necessarily evoke the expected cross-modal associations in humans either (e.g., a separate evaluation of ‘H’ in OH may invite a jagged association instead of a curved one). \nIf it is indeed the case that the tokenisation process breaks a pseudoword into syllables that would also show no bouba-kiki effects in humans, we would argue that the set of possible other pseudowords is large enough to contain other pseudowords that *could* invoke the cross-modal associations. If there would truly be a model preference, another pseudoword (which is tokenised into complete syllables) would then be preferred over the pseudoword that is broken up into non-syllables. Yet, our results show that this is not the case. \n\n# References\nKoriat, A. (1977). The symbolic implications of vowels and of their orthographic representations in two natural languages. Journal of Psycholinguistic Research, 6(2), 93–104.\n\nNora Turoman and Suzy J Styles. 2017. Glyph guessing for ‘oo’and ‘ee’: Spatial frequency information in\nsound symbolic matching for ancient and unfamiliar scripts. Royal Society open science, 4(9):170882."}, {"review_id": "KriTSH72CH", "reviewer": "Reviewer_ZTfi", "summary": "Authors test bouba-kiki effect, which is observed by humans, to vision language models. The authors throughly investigate the bouba-kiki effect on the vision language models, and conclude that the models do not present bouba-kiki effect, which implies that the models does not perceive the world as humans do.", "questions": "As I pointed out in the weakness, I would like to highlight two things: \n\n1. Grad-CAM for transformer architecture and multimodal setting. I am curious if there is no issue of applying grad-cam to the transformers and multimodal models. \n2. Why the current study is meaningful as the text-vision model is fundamentally different from human's world perception in the context of bouba-kiki effect? Especially when humans rely on audio instead of text, and when the audio plays a critical role in bouba-kiki effect, is it still meaningful to use text-vision model, not audio-vision model?\n\nI am quite positive on this paper and its results, but if authors can make justifications on these points, I will gladly increase my score.", "limitations": "Yes, authors listed limitations of the paper. But it would be further talk about the limitations as I described in the Question section. I believe those questions are crucial for this paper's validity.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Overall \nThis is well written paper and readers would easily follow the logic. The topic of this paper covers an important topic that is often neglected in modern machine learning community. And the way this paper explores the topic is quite clear and through. \n\n## Strength\n1. Important topic and significance: Compared to the prior works, this paper conduct thorough study on bouba-kiki effect on the vision language models in more comprehensive and controlled ways. It challenges the belief that the vision language models work in a similar way to humans. \n2. Rigorous methods: The authors employ a comprehensive set of pseudowords, multiple prompt templates, and image pairs inspired by human experiments, along with Bayesian regression analysis for statistically credible conclusions. \n3. Clear writing: This paper is well written and easy to follow. \n\n## Weakness\n1. Grad-CAM for transformer architecture: The Grad-CAM method is originally proposed for convolutional operations. The use of grad-cam for the transformer architecture and multi-modal setting is not quite clear to me. If you could provide some justification on this, that would be helpful. \n2. Connecting text and images, not speech: If my understanding is correct, the human studies listed in this paper are done by the speech, not text. Although humans are given texts, we can connect text to speech, and this speech in audio should deliver lots of precious information of whether the text or word is more like bouba or kiki. Although I think this work and the series of prior works are fascinating, I want to ask a fundamental question on this approach, that uses text instead of speech or audio. I understand that the typical models of multimodal processing are text and vision, not audio and vision. But is it truly fair comparison to human's world perception as we can connect text to audio, while vision-language models don't?", "was_revised": true, "final_justification": "The response from the authors resolved my concerns. Specifically, the missing link between the written words to the sound, was pointed out by the authors, which made me understand the value of this work. The technical concern about the GRAD-CAM for transformer architecture is also answered. I believe this is a good work for Neurips.", "rebuttal": "First of all, we want to thank you for your feedback and are delighted to see that you are positive about our paper. Please find our justifications for your questions below. \n\n# Weaknesses\n**Grad-CAM for ViT –** It is indeed correct that Grad-CAM was originally introduced for CNNs. However, recent works (Chefer et al. 2021, Mamooler, 2021) have shown that it can be adapted to transformer-based models by treating attention maps as similar to convolutional feature maps. In our paper, following the approach in these previous works, we applied Grad-CAM to the last attention layer of the ViT variant of CLIP, which contains spatial information through its attention heads. We specifically focused on the attention from the [CLS] token to the image patches (like in Caron et al. (2021)). To identify which image regions contributed most to the decision, we computed the gradients of the similarity score between the image and text embeddings (which we used as the target for classification) with respect to the attention weights. Then, we averaged the attention maps across heads, extracted the attention weights from the [CLS] token to the image patches by removing the [CLS] column, and reshaped the resulting vector into a 2D spatial grid. We hope that this justification is satisfactory.\n\n**Audio-Vision methodology –** \nThis is a very legitimate question. We agree with you that sound-vision entanglement is underexplored in our work and that it is a very interesting future direction. There are, however, some points that should be noted. While it is widely assumed that the bouba-kiki effect in humans is largely driven by sound-shape correspondences, many prior experiments with humans actually provide both an acoustic and written form for tested pseudowords (which was also the case in the study by Nielsen and Rendall (2013) which formed the basis for one of our word sets), so many prior studies cannot clearly disentangle the roles of orthography versus sound in the bouba-kiki effect. One study by Cuskley et al. (2017) explicitly explored the role of orthography and showed that auditory presentation cannot eliminate potential effects of orthography in literate human participants, since characters are typically strongly associated with a particular speech sound. Moreover, these associations are often not arbitrary but also shaped by human word-shape biases, resulting in systems where characters that represent rounded sounds (like rounded vowels or sonorant consonants) tend to have more curves (Koriat, 1977;Turoman & Styles, 2017). Correspondences between vision and spoken words are therefore expected to be highly correlated with correspondences between vision and written form. This makes it unlikely that a model trained on spoken language would be more likely to exhibit the bouba-kiki effect than a model trained on text would.  \nGiven the strong relationship between sound and wordforms and the fact that these words cannot be reliably associated with images, it would be surprising if an audio-visual model would pick this up. Despite our scepticism, we definitely think that this is worth exploring further with models that differ in how they combine multiple modalities. \n\nAll in all, we believe that we can make a fair comparison. We will elaborate further on the dichotomy between human sound-shape associations and the current study, focusing on text-shape associations, in the limitations section of our updated paper.\n\n# Questions\nWe believe that your questions are addressed in the response above, which addresses the weaknesses you identified. We are happy to answer any additional questions that may arise.\n\n# References\nChefer, H., Gur, S., & Wolf, L. (2021). Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 397-406).\n\nMamooler, S. (2021). CLIP explainability. Github repository\n\nCaron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9650-9660).\n\nKoriat, A. (1977). The symbolic implications of vowels and of their orthographic representations in two natural languages. Journal of Psycholinguistic Research, 6(2), 93–104.\n\nNora Turoman and Suzy J Styles. 2017. Glyph guessing for ‘oo’and ‘ee’: Spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts. Royal Society open science, 4(9):170882."}, {"review_id": "0pAPI0wlf8", "reviewer": "Reviewer_74nD", "summary": "This paper addresses previous inconsistencies in evaluating whether VLMs can capture cross-modal associations. It presents a comparative analysis of two CLIP variants across two experiments, using a more comprehensive and diverse set of pseudowords. To probe the models’ behavior, the authors apply a gradient-based explainability method to assess the influence of visual features. Their findings reveal that the cross-modal associations learned by VLMs diverge significantly from those observed in humans.", "questions": "**Reflection on the reasons behind misalignment**: the results showing that English adjectives lead to consistent cross-modal associations are compelling, especially since they suggest that VLMs can connect semantic meanings to visual features. However, this pattern does not extend to sound-symbolic associations like bouba-kiki. Do the authors attribute this discrepancy to the nature of CLIP's training data? Since image captions rarely reflect symbolic or phonetic associations, it seems plausible that such mappings are simply not represented during pretraining. Could the authors elaborate on this potential limitation and whether alternative datasets or finetuning could help reveal such associations?\n\n**Clarification on tokenization effects**. The claim that tokenization hinders the analysis of sound-symbolic effects deserves further clarification. While tokenization may segment words, syllabic structures are often preserved, especially in pseudowords designed to evoke phonetic patterns. Therefore, it is not immediately clear how tokenization alone would disrupt bouba-kiki-like associations. Could the authors specify more precisely how tokenization interferes with these effects?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- This work offers a novel and systematic investigation into cross-modal associations between language and vision. It clearly differentiates itself from prior work with relevant citations.\n- It uncovers Insightful findings, e.g., that model predictions often collapse to a few labels despite a large candidate set.\n- It introduces the use of GradCAM to interpret which visual features influence the model’s decisions, adding novelty to the analysis.\n\nWeaknesses:\n\n- The analysis focuses solely on the image encoder, without exploring the multimodal layers responsible for aligning text and vision in VLMs. Applying interpretability techniques across the full model could offer a more comprehensive understanding of cross-modal associations.\n- While the study is original, its significance is limited. It largely confirms existing suspicions about inconsistencies across models rather than investigating the reasons behind those. A deeper investigation into the causes of this misalignment (e.g., the surprising lack of symbolic alignment in Japanese) would have added more impactful insight.\n\nMinor: on line 333, the claim that the ViT variant underperforms compared to ResNet is unclear. Figure 4 suggests that ViT yields more consistent results and performs better on English adjectives. Clarification is needed here.", "was_revised": true, "final_justification": "We believe that this score is adapted to the quality of the work, and our concerns were addressed. I would keep my current ratings.", "rebuttal": "First of all, we want to thank you for taking the time to review our paper and share your opinion about it. \n\n# Weaknesses \n**Analyses on multi-modal layers –** More comprehensive investigation into intermediate layers is indeed an interesting avenue to explore. This would especially be interesting for dual-stream models that learn additional multi-modal layers between the text and vision modality (e.g. BLIP2) or single-stream models like VILT. However, the results of Verhoef et al. 2024 indicate that these models do not display behaviour consistent with a bouba-kiki effect. We, therefore, do not explore these models in this work and focus on CLIP. \n\nSince CLIP has been trained to map textual and visual inputs into a single joint multi-modal embedding, it does not have intermediate multi-modal layers that must explicitly align text and vision. In an attempt to better understand why we observe misalignment between human and CLIP’s behaviour, we explored whether the individual modalities are in principle separable in Appendix A & B and found that this seems to be the case. This indicates that CLIP’s method to align texts and images (cosine similarity) seems to be the culprit (please also see our response to your first question below).\nNevertheless, your suggestion of investigating intermediate layers in dual-stream models is an exciting idea for future work. \n\n\n**Limited significance –** We argue that the significance is not limited. First of all, our Grad-CAM method is novel (which you identified as a strength of this paper) and goes beyond confirming inconsistencies by showing that CLIP does not attend to human-like image features (Figure 1 and Figure 10). Secondly, our work nuances the findings by Alper and Averbuch-Elor (2023), who make claims that require more substantiated argumentation. Our work offers contrary findings and accompanying argumentation that we see as a significant message for the community. Finally, we do address the surprising finding that Japanese VLMs do not exhibit expected bouba-kiki effects in lines 140-145. We draw on the findings by Iida and Funakura (2024) and conclude that investigating sound-symbolism in VLMs requires methodologies aligned with human experiments. \n\n\n**Clarification line 333 –** Thank you for this comment. Our phrasing should have been more concise here. The ViT variant indeed performs better on English adjectives, but *crucially* does not do so for the targets of interest: the pseudowords. The English adjectives merely serve as a baseline (line 187) to inform us whether the models can, in principle, recognise the relevant visual features and map them to meaningful words (which they can). The claim in line 333 corresponds to the performance on the pseudowords; we will make this clear in the revised version of this paper. \n\n# Questions\n**Reflection on misalignment –** This is a very interesting suggestion. We think that there are two reasons for the observed discrepancy. First, CLIP’s training data indeed does not explicitly reflect symbolic or phonetic associations. Nevertheless, human language and its orthography are not arbitrary either (see also our response to reviewer xGW7). Characters that represent rounded sounds (like rounded vowels or sonorant consonants) tend to have more curves (Koriat, 1977;Turoman & Styles, 2017). Since these sounds are related to shape features, CLIP’s training data could (but it doesn’t) still provide enough information to learn human-like cross-modal associations.\nSecondly, CLIP combines textual and visual information through cosine similarity and is trained with a contrastive loss at the sentence level. Neither seems to specifically promote learning relations between visual features and phonetic elements in language. Interestingly, however, as mentioned at line 247, the individual modalities seem to be separable in CLIP’s embedding space. We therefore expect that finetuning on (parts of) targeted datasets could reveal human-like associations. However, the bouba-kiki effect is merely one of many cross-modal associations present in human cognition. Instead of finetuning for each specific human-like bias, research towards a more fundamental approach that combines multi-modal information more naturally seems more desirable. \n\n**Tokenisation –** We want to thank you for your question and your argument that tokenisation oftentimes keeps syllabic structures intact. This increases its relevance and need for clarification.\n\nInterestingly, analyses reveal that humans even display a bouba-kiki effect at the level of single syllables (Nielsen and Rendall, 2013), suggesting that the associations do not rely on complete words. Since, as you mention, tokenisation often preserves syllabic structures, our analysis is not so different from investigating the bouba-kiki associations in humans.\n\nTo add strength to this argument, we tokenised the most common labels mentioned in our answer to your first question. Some examples:\n\nBouba → “bou” and “ba”\n\nKiki → “kiki”\n\nTakete → “take”, “te”\n\nMaluma → “mal”, “uma”\n\nXehaxe → “xe”, “ha”, “xe”\n\nLohlah → “loh”, “lah”\n\nSepise → “sep”, “ise”\n\nKaykuh → “kay”, “ku”, “h”\n\nLoomoh → “loom”, “oh”\n\nMohmah → “moh”, “mah”\n\nThese examples, which we will add to an updated version of our paper, show that most of the target words are tokenised in a way that preserves at least some phonological structures that could be matched to shape features. \n \nNonetheless, the tokenisation process in language models may split syllables or pseudowords into tokens that would not necessarily evoke the expected cross-modal associations in humans either (e.g., a separate evaluation of ‘H’ in OH may invite a jagged association instead of a curved one). \nIf it is indeed the case that the tokenisation process breaks a pseudoword into syllables that would also show no bouba-kiki effects in humans, we would argue that the set of possible other pseudowords is large enough to contain other pseudowords that *could* invoke the cross-modal associations. If there would truly be a model preference, another pseudoword (which is tokenised into complete syllables) would then be preferred over the pseudoword that is broken up into non-syllables. Yet, our results show that this is not the case.\n\nTo answer your question: While tokenisation may have an impact on the results, we think that the extent to which this influences them is somewhat limited. We did, therefore, not discuss this in detail in our paper, but we will do so as a response to your rightful question.\n\n# References\nKoriat, A. (1977). The symbolic implications of vowels and of their orthographic representations in two natural languages. Journal of Psycholinguistic Research, 6(2), 93–104.\n\nNora Turoman and Suzy J Styles. 2017. Glyph guessing for ‘oo’and ‘ee’: Spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts. Royal Society open science, 4(9):170882."}]

Author rebuttal:

Meta-review: (a) **Summary:** This paper explores the extent to which the bouba-kiki effect – where humans associate “bouba” with round shapes and “kiki” with jagged shapes – is present in CLIP models. They use both model probabilities and GradCAM to test model preference and visual attention, respectively. They found CLIP with the ResNet backbone has some preference for round shapes but overall, the models do not demonstrate the bouba-kiki effect. Importantly, CLIP models do not show the same cross-model behavior for this effect as humans do. It suggests models do not demonstrate the same human intuitions.

(b) **Strengths:**
- This work uses both synthetic and natural data (**Reviewer BFRt**), showing the findings are generalizable to different types of data.
- The methodology for the models is a forced choice task, which closely aligns with existing psycholinguistics literature (**Reviewer xGW7**).
- By using GradCAM, this paper explores more of the interpretability angle that is not possible from model probabilities alone (**Reviewer xGW7, Reviewer 74nD**).
- The paper is well-written overall and easy for readers to understand (**Reviewer BFRt, Reviewer ZTfi**).

(c) **Weaknesses:**
- Only CLIP models are used (**Reviewer BFRt**), while more SOTA VLMs are excluded from the analysis. I agree that additional VLMs should be tested, especially given the title and wording in the abstract.
- The findings are a negative result in some ways (CLIP models do not show the bouba-kiki effect) without giving insight into why it happens and what the significance of this is (**Reviewer 74nD**). The paper would benefit from a deeper analysis about why the misalignment with humans happens.

(d) **Recommendation:** Reviewers overall view this work positively, with scores of 4, 4, 5, 4. The strengths are in the experimental setup, where both probabilities and attention are used to probe the bouba-kiki effect. Additionally, the work thoughtfully parallels existing human studies. The weaknesses are that only CLIP models are used. While I appreciate that the authors used two backbones, one CNN-based and one transformer, I do agree that more VLMs should have been used. This would strengthen the conclusions and give more insight into other widely used, SOTA models. With all of this considered, this work has solid findings supported by the reviewers. I suggest acceptance.

(e) **Discussion/Changes during Rebuttal:**
- **Point:** GradCAM was originally proposed for convolutional operations and therefore may not be transferable to the ViT backbone. In the rebuttal, the authors present additional work that uses GradCAM for Transformer models. The reviewer suggests they are convinced by the rebuttal. Impact: This was fully addressed and therefore did not impact my suggestion.
- **Point:** CLIP embeds image-text similarity at the sentence level, so **Reviewer BFRt** is concerned about how the individual words’ impacts are tested. The authors clarify that only an individual word is toggled between sentences, such that the impact must be from said word. Impact: This clarification makes sense and this concern did not impact my recommendation.
Decision: Accept (poster)