Title: Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect
Venue: NeurIPS 2025
Accepted: yes
Keywords: Cross-modal associations

Vision-and-Language Models

bouba-kiki effect

Cognitive science
Abstract: Recent advances in multimodal models have raised questions about whether vision-and-language models (VLMs) integrate cross-modal information in ways that reflect human cognition. One well-studied test case in this domain is the bouba-kiki effect, where humans reliably associate pseudowords like ‘bouba’ with round shapes and ‘kiki’ with jagged ones. Given the mixed evidence found in prior studies for this effect in VLMs, we present a comprehensive re-evaluation focused on two variants of CLIP, ResNet and Vision Transformer (ViT), given their centrality in many state-of-the-art VLMs. We apply two complementary methods closely modelled after human experiments: a prompt-based evaluation that uses probabilities as a measure of model preference, and we use Grad-CAM as a novel approach to interpret visual attention in shape-word matching tasks. Our findings show that these model variants do not consistently exhibit the bouba-kiki effect. While ResNet shows a preference for round shapes, overall performance across both model variants lacks the expected associations. Moreover, direct comparison with prior human data on the same task shows that the models’ responses fall markedly short of the robust, modality-integrated behaviour characteristic of human cognition. These results contribute to the ongoing debate about the extent to which VLMs truly understand cross-modal concepts, highlighting limitations in their internal representations and alignment with human intuitions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses previous inconsistencies in evaluating whether VLMs can capture cross-modal associations. It presents a comparative analysis of two CLIP variants across two experiments, using a more comprehensive and diverse set of pseudowords. To probe the models’ behavior, the authors apply a gradient-based explainability method to assess the influence of visual features. Their findings reveal that the cross-modal associations learned by VLMs diverge significantly from those observed in humans.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This work offers a novel and systematic investigation into cross-modal associations between language and vision. It clearly differentiates itself from prior work with relevant citations.\n- It uncovers Insightful findings, e.g., that model predictions often collapse to a few labels despite a large candidate set.\n- It introduces the use of GradCAM to interpret which visual features influence the model’s decisions, adding novelty to the analysis.\n\nWeaknesses:\n\n- The analysis focuses solely on the image encoder, without exploring the multimodal layers responsible for aligning text and vision in VLMs. Applying interpretability techniques across the full model could offer a more comprehensive understanding of cross-modal associations.\n- While the study is original, its significance is limited. It largely confirms existing suspicions about inconsistencies across models rather than investigating the reasons behind those. A deeper investigation into the causes of this misalignment (e.g., the surprising lack of symbolic alignment in Japanese) would have added more impactful insight.\n\nMinor: on line 333, the claim that the ViT variant underperforms compared to ResNet is unclear. Figure 4 suggests that ViT yields more consistent results and performs better on English adjectives. Clarification is needed here.\n\nQUESTIONS:\n**Reflection on the reasons behind misalignment**: the results showing that English adjectives lead to consistent cross-modal associations are compelling, especially since they suggest that VLMs can connect semantic meanings to visual features. However, this pattern does not extend to sound-symbolic associations like bouba-kiki. Do the authors attribute this discrepancy to the nature of CLIP's training data? Since image captions rarely reflect symbolic or phonetic associations, it seems plausible that such mappings are simply not represented during pretraining. Could the authors elaborate on this potential limitation and whether alternative datasets or finetuning could help reveal such associations?\n\n**Clarification on tokenization effects**. The claim that tokenization hinders the analysis of sound-symbolic effects deserves further clarification. While tokenization may segment words, syllabic structures are often preserved, especially in pseudowords designed to evoke phonetic patterns. Therefore, it is not immediately clear how tokenization alone would disrupt bouba-kiki-like associations. Could the authors specify more precisely how tokenization interferes with these effects?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors test bouba-kiki effect, which is observed by humans, to vision language models. The authors throughly investigate the bouba-kiki effect on the vision language models, and conclude that the models do not present bouba-kiki effect, which implies that the models does not perceive the world as humans do.\n\nSTRENGTHS AND WEAKNESSES:\n## Overall \nThis is well written paper and readers would easily follow the logic. The topic of this paper covers an important topic that is often neglected in modern machine learning community. And the way this paper explores the topic is quite clear and through. \n\n## Strength\n1. Important topic and significance: Compared to the prior works, this paper conduct thorough study on bouba-kiki effect on the vision language models in more comprehensive and controlled ways. It challenges the belief that the vision language models work in a similar way to humans. \n2. Rigorous methods: The authors employ a comprehensive set of pseudowords, multiple prompt templates, and image pairs inspired by human experiments, along with Bayesian regression analysis for statistically credible conclusions. \n3. Clear writing: This paper is well written and easy to follow. \n\n## Weakness\n1. Grad-CAM for transformer architecture: The Grad-CAM method is originally proposed for convolutional operations. The use of grad-cam for the transformer architecture and multi-modal setting is not quite clear to me. If you could provide some justification on this, that would be helpful. \n2. Connecting text and images, not speech: If my understanding is correct, the human studies listed in this paper are done by the speech, not text. Although humans are given texts, we can connect text to speech, and this speech in audio should deliver lots of precious information of whether the text or word is more like bouba or kiki. Although I think this work and the series of prior works are fascinating, I want to ask a fundamental question on this approach, that uses text instead of speech or audio. I understand that the typical models of multimodal processing are text and vision, not audio and vision. But is it truly fair comparison to human's world perception as we can connect text to audio, while vision-language models don't?\n\nQUESTIONS:\nAs I pointed out in the weakness, I would like to highlight two things: \n\n1. Grad-CAM for transformer architecture and multimodal setting. I am curious if there is no issue of applying grad-cam to the transformers and multimodal models. \n2. Why the current study is meaningful as the text-vision model is fundamentally different from human's world perception in the context of bouba-kiki effect? Especially when humans rely on audio instead of text, and when the audio plays a critical role in bouba-kiki effect, is it still meaningful to use text-vision model, not audio-vision model?\n\nI am quite positive on this paper and its results, but if authors can make justifications on these points, I will gladly increase my score.\n\nLIMITATIONS:\nYes, authors listed limitations of the paper. But it would be further talk about the limitations as I described in the Question section. I believe those questions are crucial for this paper's validity.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the classic bouba-kiki effect within the context of vision-language models (VLMs), particularly two CLIP variants (ResNet and ViT). Unlike previous studies that rely on embedding similarity or generation-based probing, this paper introduces a methodology tightly aligned with human psycholinguistic experiments. \n\nThe findings show that, despite some sensitivity to semantic adjectives (e.g., \"round\", \"spiky\"), CLIP models do not consistently exhibit the bouba-kiki effect for pseudowords, failing both in behavior and visual attention alignment.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- This paper stands out not for discovering a new bouba–kiki effect, but for disproving its presence in CLIP with a much more rigorous and cognitively aligned methodology. It raises the bar for testing cross-modal iconicity in vision-language models. Prior work (e.g., Alper & Averbuch-Elor) used embedding similarities or generation-based tasks with less experimental control. This paper brings direct comparability to human data, especially from Ćwiek et al. (2022).\n- Human-aligned evaluation protocol: The authors closely mirror experimental designs used in psycholinguistics (e.g., forced-choice tasks with minimal shape differences), allowing for principled comparison between human and model behavior.\n- A strong plus to use Grad-CAM for cross-modal interpretability: By examining visual attention patterns during label-shape association, the paper moves beyond accuracy to probe the internal mechanisms of cross-modal processing in VLMs.\n\n## Weaknesses\n\n- Sound-text-vision entanglement is underexplored: Since the original bouba-kiki effect is rooted in auditory perception, the textual substitution (e.g., \"bouba\" as a string token) may be too weak a proxy. The paper acknowledges this but does not systematically explore text-to-speech models or phoneme-level grounding.\n- Model scope is limited to CLIP: While CLIP is foundational, the paper does not evaluate whether more modern VLMs (e.g., BLIP-2, LLaVA) or large multimodal models (e.g., GPT-4o) show similar or different patterns, which weakens the generality of the conclusions.\n- While the paper introduces novel methodological angles, the core question of whether VLMs exhibit the bouba-kiki effect has already been substantially explored in prior work, particularly by Alper & Averbuch-Elor (2023) and Verhoef et al. (2024). As such, the contribution may feel incremental, and the bouba-kiki phenomenon, though conceptually interesting, may be too narrow in scope to carry a full conference paper on its own.\n\nQUESTIONS:\n- Did you explore whether the CLIP models consistently output the same label regardless of the image input (i.e., label collapse), and if so, how does this affect interpretability?\n- Did you analyze the spatial distribution of attention beyond total intensity? Your Grad-CAM analysis uses summed activation across curved vs jagged halves, but this could overlook spatial patterns within each region. For example, did you consider using centroid-of-attention, entropy, or overlap with curvature-dense regions to better quantify alignment with shape features?\n- How does tokenization affect the model's ability to form shape-word associations, particularly for pseudowords? Since CLIP and other Transformer-based models operate on subword or byte-pair encodings (e.g., \"bouba\" might be split into “bou” and “ba”), this fragmentation may distort the phonological structure that underlies the bouba-kiki effect in humans, where sound symbolism operates over holistic auditory forms. Did you analyze how the pseudowords are tokenized by the CLIP text encoder, and whether token splits align with phoneme boundaries or disrupt the intended phonetic profiles? Could this explain the models' failure to form shape-sound associations, especially compared to their robust performance on real adjectives?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates whether vision-and-language models (VLMs), specifically two variants of CLIP (ResNet and Vision Transformer), exhibit human-like cross-modal understanding by testing the bouba-kiki effect—a phenomenon where people associate round shapes with the word \"bouba\" and jagged shapes with \"kiki.\" Using prompt-based evaluation and Grad-CAM for visual attention, the study finds that:\n* ResNet shows some preference for round shapes, but\n* The models do not consistently demonstrate the bouba-kiki effect.\n* Their performance falls short of human-like cross-modal integration.\nThese findings suggest that current VLMs have limited internal representations of such intuitive, cross-modal associations, raising questions about the depth of their \"understanding.\"\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The paper is well written and easy to follow. \n- The paper provides a good overview of the literature regarding the bouba-kiki effect.\n- The results are obtained both with synthetic and natural data.  \n \nWeaknesses: \n- The paper does not contribute to novel machine learning models. Its contribution is interesting for the natural language processing and computer vision domains, so EMNLP, CoNLL and CVPR might be better venues for its publication. \n- It is strange that a CLIP model was chosen as underlying vision-language foundation model. It is well known that alignment in CLIP is computed at the sentence/scene level and not at the level of individual words, which devalues the findings of the paper.\n\nQUESTIONS:\n- Please motivate why the work is valuable for the machine learning community as you do not propose strategies to improve cross-modal associations. \n- What is the practical impact of investigating the bouba-kiki effect?\n\nLIMITATIONS:\nThe limitations of the work are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}