CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions
Authors: Yuchen Huang
Zhiyuan Fan
Zhitao He
Sandeep Polisetty
Wenyan Li
Yi R. Fung
Venue: COLM 2025 (2025)
Keywords: Vision-Language Models
Cultural Understanding
Fine-Grained Recognition
Contextual Knowledge
Synthetic Data Generation
Contrastive Learning
TL;DR: We introduce CulTwin, a synthetic cultural dataset of visually similar concept pairs with contextualized captions, and CultureCLIP, a CLIP-based model fine-tuned to better distinguish visually similar yet culturally distinct concepts.

Abstract: Pretrained vision-language models (VLMs) such as CLIP excel in general multimodal comprehension but often struggle to capture nuanced, context-dependent visual cues. This makes it difficult to distinguish between similar-looking concepts with potentially different cultural meanings. Such deficiencies are mainly due to a limited amount of high-quality cultural data, contextual information, and the lack of negative examples that highlight subtle differences. To mitigate this, we design a data curation pipeline leveraging open-sourced VLMs and text-to-image models to construct CulTwin, a synthetic cultural dataset. This dataset consists of paired concept-caption-image triplets, where concepts visually resemble each other but are culturally different. Then, we fine-tune CLIP on CulTwin to develop CultureCLIP, which aligns cultural concepts with contextually enhanced captions and synthetic images through tailored contrastive learning. Experiments on culture-specific benchmarks show that CultureCLIP outperforms the base CLIP, achieving up to a notable 5.49\% improvement in fine-grained concept recognition on certain tasks while preserving CLIP's original generalization ability, validating the effectiveness of our data synthesis and VLM backbone training paradigm in capturing subtle cultural distinctions.

Published as a conference paper at COLM 2025
CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions
Yuchen Huang', Zhiyuan Fan', Zhitao He², Sandeep Polisetty?, Wenyan Lia
Yi R. (May) Fung!
Hoong Kong University of Science and Technology
University of Massachusetts Amherst, UUniversity of Copenhagen [yhuanggn, yrfung) (aang)@cceeuust ussh.
Abstract
Pretrained vision-language models (VLMs) such as CLIP excel in gen- eral multimodal comprehension but often struggle to capture nuanced, context-dependent visual cues. This makes it difficult to distinguish be- tween similar-looking concepts with potentially different cultural mean- ings. Such deficiencies are mainly due to limited amount of high-quality cultural data, contextual information, and the lack of negative examples that highlight subtle differences. To mitigate this, we design data cura- tion pipeline leveraging open-sourced VLMs and text-to-image models to construct CulTwin, a synthetic cultural dataset. This dataset consists of triplets, where concepts visually resemble each other but are culturally different. Then, we fine-tune CLIP on CulTwin to develop CultureCLIP, which aligns cultural concepts with contextually enhanced captions and synthetic images through tailored contrastive learn- ing. Experiments on culture-specific benchmarks show that CultureCLIP outperforms the base CLIP, achieving up to a notable 5.49% improvement in fine-grained concept recognition on certain tasks while preserving CLIP's original generalization ability, validating the effectiveness of our data syn- thesis and VLM backbone training paradigm in capturing subtle cultural distinctions.
paired
1 Introduction
Recent advancements in vision-language reasoning (Zhang et al., 2024b; Lu et al., 2024) have revolutionized multimodal understanding by efficiently integrating visual and linguistic semantics within a shared feature space (Jia et al., 2021; Radford et al., 2021). By leveraging large-scale image-text corpora, models such as Contrastive Language-Image Pretraining (CLIP) (Radford et al., 2021) exhibit remarkable generalization capabilities across diverse downstream tasks, including visual question answering (VQA)(Shen et al., 2021; Song et al., 2022), mooas retrieval (Koukounas et al., 2024; Baldrati et al., 2023), and zero-shot image classification (Zhou et al., 2022; Saha et al., 2024). CLIP is built on a contrastive learning objective, where two separate encoders are trained to bring matching image-text pairs closer in the feature space while pushing apart non-matching pairs within the same batch. Due to the concise nature of the text in its training data, CLIP is effective at coarse- grained semantic alignment, particularly in identifying the general type of the main object (Radford et al., 2021; Zhang et al., 2024a). However, often struggles with tinengrained alignment, especially when contextual visual details, such as specific accessories, stylistic cues, or symbolic elements that convey meaning only within particular cultural contexts, are required to distinguish between visually similar but culturally distinct concepts.
For example, CLIP might correctly identify both Yuelao (the Chinese god of love and marriage) and Taishang Laojun (a Daoist patriarch) as elderly Chinese deities, but it often
'Qur code is publicly available at https: //gs::://tttte http:/ggttttttt/tee
wtg.......coo....... com/ www.hhum/huhlullaacacaa IP.
1

---

Published as a conference paper at COLM 2025
* MOGE ENIDDER Lo8d
INAGE
Aurnet
IMAGE SYNTHESIS
SHAVEED FEATURE SPADE
POBITVE CAPTION Yunloo, alterry daery holds red thread, tranecme
NECCATIVE CAPTION
OAPTOON
al Tacket mostter
Teshang Leounn,
optannain
TEXT ENCODER
ENCZDER
CAPTION GENERATION
CONNEPT
Taishang
SHWED SPACE
Yuetoo
Talshang Loojun
Chinese Elderty
Image Errts
Nag Imigge Emd
CLIP: 4, d, recogrize visually similar yet outturally distinot concepts.
CuttureCLIP (Ours): 4, <dd allgn Images groinod
Caption
sonceptthrought beseeptttruggh.ootssseeeeee contextuallzed coptione and synthetic images.
Por Cancess Emr.
Map Conoect Eib.
The original CLIP model fails capture fine-grained contextual visual cues (highlighted by the green box), leading to mismatches with cultural concepts. Although both the image and concept projections fall within the region outlined by the purple dashed box-ie., the semantic space of Chinese elderly deities-th distance between the image of Yuelao (pink circle) and correct concept (pink square) is greater than that to an incorrect one such as Taishang Laojun (blue square), i.e., d1 >> d2. Right: CultureCLIP improves fine-grained cultural understanding by jointly aligning concepts with contextualized captions and their corresponding synthetic images, while repelling unrelated concepts and captions in the embedding space (see details in Section 4.2).
Figure CLIP vs. CultureCLIP. Left:
struggles to tell them apart because of its insensitivity to capturing subtle yet crucial visual details, like the red Ihread for Yuelao, which stands for love, the alchemy furnace for Taishang Laojun, which stands for immortality (both shown in green in Figure 1). These culturally specific visual cues, however, are exactly what people in particular cultural groups use to distinguish line-grained concepts. This raises an intriguing question: How can we teach CLIP capture such details so that it can differentiate between cultural concepts that share visual similarities?
natural approach is curate large-scale dataset comprising visually similar cultural con- cept pairs (i.e., original concepts alongside their hard negatives) accompanied by image-text pairs enriched with contextual cultural knowledge and illustrating subtle visual distinc- tions. However, collecting such data presents three major challenges: First, high-quality cultural image-text pairs are scarce and costly to annotate. Existing manually curated cultural datasets are typically limited in scale, often containing only few thousand samples (Nikandrou et al., 2024; Bhatia et al., 2024; Romero et al., 2024), due to the labor-intensive nature of data collection and annotation, tise. While web-scraped data might serve as an alternative (Xu et al., 2023), it frequently introduces substantial noise: images may be ethically inappropriate, copyright-protected oww-resolution, or culturally misleading, and the corresponding text may be mismatched or entirely absent. Second, CLIP's original training strategy inherently favors concise captions. Specifically, it imposes a 77-token limit, with the majority of alignment achieved within the first 20 tokens (Zhang et al., 2024a). This design makes it particularly challenging to incorporate lengthy, information-rich cultural context directly into the text for training, as doing so may disrupt the original image-text alignment learned by the model. Third, fine-grained hard negatives specifically tailored for visually similar concepts are lacking. In the original CLIP framework, negative samples are randomly drawn within each batch, which significantly reduces the likelihood of including conceptually similar but visually distinct examples. Although recent works (Yuksekgonul et al., 2022; Patel et al., 2024) intro- duce harder negatives by making minor modifications to captions images, they still lack
well as the need for domain- sppccffc exper-
2

---

Published as a conference paper at COLM 2025
structured negative samples that highlight both coarse-grained similarities and fine-grained differences, which are crucial for enabling culturally grounded visual distinctions.
Motivated by these challenges, we introduce CulTwin, synthetic dataset of paired concept- caption-image triplets, which we call Twin Cards. In each card, two similar concepts are paired together, and their captions enriched with cultural background knowledge using a vision-language model. The corresponding images are then generated from these captions by text-to-image model. Building on CulTTwin we propose CultureCLIP, contrastive learning framework that jointly aligns concepts, captions, and images in a shared embedding space, As illustrated in Figure 1 (right), our training objective works like a magnetic field: attracts each concept toward its corresponding caption and image, while repelling it from the captions and images of its culturally contrasting counterpart. Experiments on culture- specific benchmarks show that CultureCLIP significantly outperforms the original CLIP model, achieving over 5% improvement in fine-grained concept recognition on specific tasks. These results highlight the effectiveness of our synthetic dataset and training methodology in capturing subtle cultural nuances.
2 Related Work
Advancements in Vision-Language Models Recent studies on multimodal reasoning (Peng et al., 2025; Zhao et al., 2025; Zhang et al., 2025; He et al., 2025) have significantly ad- vanced vision-language models and broadened their capabilities across various downstream tasks and domains. For instance, Med-Flamingo (Moor et al., 2023) unlocks medical VQA abilities through continued pre-training on paired and interleaved medical image-text data, while Chem VLM (Li et al., 2024), trained on bilingual image-text data, enhances the joint un- derstanding of textual and visual chemical information. In the cultural domain, Culture VLM (Liu et al., 2025) improves cultural understanding by fine-tuning on a large-scale multimodal benchmark, CultureVerss
Despite these advancements, existing vision-language models still struggle to capture fine-grained visual cues and often misclassify visually similar but culturally distinct concepts. We propose CultureCLIP, which aligns cultural concepts with enriched captions and synthetic images through contrastive learning, improving cultural differentiation while preserving generalization capabilities.
Data for Vision-Language Pre-training Cross-modal mutual information maximization relies on large-scale, diverse training data that captures real-world concepts and relation- ships. With more than 5 billion internet-derived image-caption pairs, the LAION dataset (Schuhmann et al., 2022) serves as critical training resource. MetaCLIP (Xu et al., 2023) formalizes CLIP's implicit data selection via explicit metadata balancing, creating 400M CommonCrawl pairs. SynthCLIP (Hammoud et al., 2024) reduces reliance on web-scraped data by generating over 30 million synthetic pairs. LaCLIP (Fan et al., 2023) enhances text augmentation through in-context language model rewriting. For human-centric AI, fine- grained cultural understanding is essential, yet culturally relevant multimodal data is scarce. While Nayak et al. (2024); Liu et al. (2025) introduced cultural datasets, the diversity of images hinders vision-language models from learning cultural distinctions. In this work, we construct CulTwwin, a synthetic dataset comprising concept-caption-imag triplets enriched with cultural contextual knowledge.
Contrastive Pre-training Contrastive learning has become a strong method for multimodal representation learning, with CLIP (Radford et al., 2021) demonstrating scalability and zero- shot transfer potential. More efficient contrastive pre-training methods have been proposed for finer-grained multimodal representations learning (Zhang et al., 2024c; Patel et al., 2024). BLIP-2 (Li et al., 2023) introduces a lightweight Querying Transformer for cost-efficient pre-training. NegCLIP (Yuksekgonul et al., 2022) generates hard negative captions through semantic perturbations. TripletCLIP (Patel et al., 2024) uses hard negative pairs with a triplet contrastive loss. Existing contrastive pre- pre-training methods focus on caption-image pairs and their negative samples, but fail to capture culturally relevant information due to its multidimensional nature. To address this, CultureCLIIP enhances cultural understanding
3

---

Published as a conference paper at COLM 2025
Stagel: CulTwin Data Curation
Stage2: CultureCLIP Fine-tuning
CONCEPT MINING AND TWIN MATCHING (3.9)
PREDEF MED TAXGNOWY
POGITIVE IMAGGE
NEGATING BAGGE
DOPCSPP Shangg Bao
Teconory-Cuhiee
CountryChre
Fastarne
Top-dww Generation
Courtty Chino
TANOROWY
W
Charglaon Dao
R Gultural rdlvanes? Umiihhouuntrg Wrich taccromr??
CONTEXT
CAPTION
wTT chipe goidas Domain
IMAGE ENCOGER
LoRA
Filerig us!
Estract Contect
Grtirgg boticen
ESiiE TOM Shengar a of Srub.
por-iby
Twin Mutching
Bettomnu Collection
CONCEPT
Sondi Pi
A
CAPTION GENERATION (3.2)
Country Chink Tapanamy Culrine
POGITIVE CAPTION
Cancopt
Acpprra
Bisting rrseitesd anll Shangi TOOO II cntipy boTero
DONTEXT
Coer da pigge
TEXT ENCODER
LaRA
extoree setereeoo wnddingeeenccomm soading s past, oigget Stanghh
CONDEPT
POSHTIVE CAPTION
IMAG SYNTHEBIS AND QUALITY FILTEING (3.5)
Twin Card
NEGATIVE CORREPT C
REOAT THE CAPTION
POBITIVE CONDEPT
Fatdryy
Xiwilog
CONCEPT
CAPTION
POBIIVV
NEGATTIVE CAPTION
Fragg
Irage Erid,
Pix. Camsonn fenn W
Cxptoon Frih $7
Geneesson
INAGEE
PAAGE
Cp Carood Erds of"
Concagt Drnb, x.""
Figure Left: Data curation pipeline in CulTwin. Right: Architecture of CultureCLIP.
by aligning cultural concepts with contextualized captions and synthetic images, while separating unrelated concepts in the embedding space.
3 CulTwin: A Three-stage Cultural Data Curation Pipeline
High-quality and diverse data has been shown to be essential for training models like CLIP (Nguyen et al., 2023; Fang et al., 2022). In this section, we present three-stage data curation pipeline for constructing CulTwin, synthetic cultural dataset composed of triplets that are visually similar but culturally distinct. The pipeline begins by collecting culturally grounded concepts, their background
Twin Cards-pairs of concept-caption-imag
knowledge, and visual features, and performing twin matching to identify negative samples that are visually similar but culturally different (Section 3.1). Next, diverse captions are generated by leveraging cultural context and key visual features through large language model (LLM) (Section 3.2). Finally, images are synthesized from the captions and evaluated using Vision-Language Model (VLM), which scores each image based on authenticity, consistency, and cultural fidelity to guide data quality filtering (Section 3.3). The resulting triplets are then organized into Twin Cards for later fine-grained contrastive training. Figure 2 (Left) provides an overview of the full CulTwwi data curation pipeline.
3.1 Concept Mining and Twin Matching
We begin with a manually predefined taxonomy covering 229 countries and 8 cultural categories, including Cuisine, Clothing, Animals & Plants, Art, Architecture, Daily Life, Symbols, and Festivals. This taxonomy is designed to capture a broad and representative set of cultural elements (definitions of each category are provided in Appendix A). We then collect culturally meaningful concept candidates through both bottom-up collection and top-down generation.
4

---

Published as a conference paper at COLM 2025
Table 1: Filtering Outcomes and Image Quality Scores Across Diverse Cultural Taxonomies
Image Quality Scores After Filtering
Filtering Outcomes
Taxonomy
MLLM-a-atJudge
Human Evaluation
Pase %
Retained
Auth
Cons
Fid
Auth
Cons
Fid 1
Cuisine
4,403 0.528
3.548 0.627
3,750
0.719
4.517 0.866
3.533
0.957
3.550
1,023
77.46
24824/32,046
Clothing
4,469
3.62
0.422
4.002
0.155
0.956
1.202
7250/9,
Animal & Plants
4,350 + 0.415
3.79 + 0.528
4,284
0.3366
4.183
+1.162
3.783 + 1.185
3.800 + 1.137
75.63
11866/14,00
Act
4,390 0.464
4032 0:45
0.255
4.43
0.761
2,90 1.189
2917 2917=1114
72.67
15806/21,755 221,730
Architecture
4.222
3,87
0.402
3,997
4.183
2.917
6377/10 A00
Daily Life
4.239 2 0.436
3.882
0.489
4.152 a 0.300
4.483
= 0.904
1,050 = 1.132
4,100 =: 1.179
72,80
6188/ / 88500
Symbol
3.802 a: 0.371
3.830 = 0.291
3.977 a 0.109
4,467
0.718
3,950 a: 1.310
3.833 : 1.227
85.12
681/800
Festival
3 0.332
0.061
4.167 0.986
3.535 1.067
3,533 a 1,008
72.12
1731/2,400
Scores are averaged over three dimensions-Auut and Fid (cultural fidelity-ratted from 1 to 5. Pass % indicates the proportion of images passing quality thresholds; Retained shows the number of remaining samples per category.
(image authenticity), Cons (concept consistency),
Bottom-up Concept Collection In this approach, we first collect candidate concepts and their background information (definitions, images, captions) from Wikipedia. We then use Qwen2.5- (Bai et al., 2025) to assess each concept's cultural relevance, discarding any that are not strongly related the predefined categories. To ensure strict filtering, we designed prompts (Appendix F) that favor rejection over uncertain inclusion, prioritizing data quality over quantity. For the retained concepts, we assign metadata (country and cultural category) and extract key contextual and visual features.
Top-down Concept Generation In this approach, we further expand the concept pool using the predefined taxonomy. For each cultural category and country, Qwen2.5-VL is employed generate culturally grounded concepts, along with associated context and visual features.
After obtaining these filtered concepts from both approaches, we perform twin matching to identify culturally distinct but visually similar hard negatives for each concept, using Qwen2.5-VL. conditioned on its context and key visual features. These final concept pairs, together with their metadata and contextual information, form the foundation for generating detailed captions and images in subsequent stages.
3.2 Diverse Caption Generation
In the second stage, we generate diverse, culturally rich captions for each concept in the paired sets. These captions are designed to highlight both key visual elements and cultural nuances, preserving subtle distinctions before image synthesis. We leverage Qwen2.5- VL to incorporate cultural context and salient visual features, producing contextualized descriptions. To mitigate the risk of limited diversity and potential overfitting in synthetic data (Hammoud et al., 2024), we guide the model to vary aspects such as artistic style, scene setting, and compositional details, thereby enriching the visual representation of each concept.
3.3 Image Synthesis and Quality Filtering
After caption generation, we synthesize images using Stable Diffusion 3.5 (Rombach et al., 2022), followed by quality filtering with MLLMMas-a-Judge (Chen et al., 2024), implemented using Qwen-VL-2.5, Each synthesized image is evaluated across three key dimensions on a scale from 1 to 5:
1) Authenticity: Evaluates the physical realism and adherence to common human under- standing. 2) Consistency: Assesses alignment between the image and its caption, ensuring accurate concept representation. 3) Cultural Fidelity: Examines the preservation and correctness of cultural features specific to the concept.
Images receiving a score in any dimension or an average score below 3 are discarded. To validate the automated filtering, we also perform human evaluation on sampled subset,
5

---

Published as a conference paper at COLM 2025
CUISINE
CLOTHING
ANIMAL & PLANES
XIACLONG BAO A chet cambully places Xanlonghao in nemboo translacent wrappers sevory inaide.
THENGJAAR BAOO
KIMONO
KEBAYA
GLASSWORT
CHAMOMILE Chamoriia flowers with white petals and centers bloom amidse gress, coudiing-lllmees callmesss
Aiizoing Shengjamben a hot iron sidlict, its golden the
Lady in black kimono with delicate cakura the vibrant obi tiad Sumiicaa River Tokyo.
Thres women in vibmant. embro kebayos, reaturing intricate desigrs, traditional showessing
Green glasssert plants with succulent, needle-
glastening apae sparkting crystiia.
Indonesien drese.
Figure 3: Twin Card examples from the Cuisine, Clothing, and Animals & Plants categories. Each Twin Card contrasts two culturally distinct yet visually similar concepts, with each side comprising concept-caption-imagg triplet (see Section 3.3). Additional examples are provided in Appendix B.
where three PhD-level experts independently score the images using the same criteria. Summary statistics of the automated and human evaluations are shown in Table 1.
Categories like Cuisine and Clothing exhibit high Authenticity scores, reflecting their tangible and universally recognized nature, making them easier to evaluate accurately by both models and human judges. In contrast, more abstract categories such as Festival tend to have lower Authenticity and Cultural Fidelity scores, particularly in human evaluations, due to their complex and diverse cultural elements. Furthermore, Art and Archilecture show noticeable gap between automated and human evaluations, especially in Consistency and Cultural Fidelity. These categories involve nuanced cultural and conceptual details that automated models struggle to capture, highlighting the importance of human judgment for these intricate domains.
Each final Twin Card is constructed by assembling two triplets each consisting of concept, its caption (from Section 3.2), and the corresponding synthesized image-to explicitly highlight cultural contrast while maintaining visual similarity. Example Twin Cards are illustrated in Figure 3, and additional details of CulTwwn can be found in Appendix B.
CultureCLIP: Fine-Grained Cultural Alignment
4.1 Preliminary
CLIP learns joint image-text representations via an image encoder F : I -> Rd and a text encoder G : T + Rd, projecting inputs into shared embedding space V of dimension d. Given batch of N image-text pairs {(4, T.) representations are computed as /! F()) and w G(T)). CLIP constructs similarity matrix E RNN, where each entry Si = sim(y) denotes the cosine similarity between image I, and text Tj- The contrastive loss encourages each aligned pair (on the diagonal of 5) to have higher similarity than all mismatched pairs in the same row or column. Formally:
fcup((, T) = LIZT (1, (12 (T,
(1)
exp(sim (.. )/z)
(Iz((ITT)
Σ log
(2)
N
exp sim wt 4/T))
exp(sim w /T)/
CTl(L, = N
log
(3)
E exp(sim //z) \/Tr))
Here, T temperature parameter that controls the sharpness of the similarity distribution.

---

Published as a conference paper at COLM 2025
NegCLIP (Yuksekgonul et al., 2022) builds on this framework by introducing hard negative captions T, derived through semantic perturbations of the original texts T These hard negatives are added alongside the standard in-batch negatives, forming an extended candidate set 7 {T+ (T-)
and resulting in similarity matrix Š RNx2N, The contrastive objective is thus extended beyond standard in-batch negatives to also include explicitly constructed hard negatives:
NegCLIIP (1, . TT = LyT. (1, ,T + (T,((T,T),,
(4)
exp(sim( ( 47 )/T)
aeg (1,T TT
log -==1 exp(sim(4) )/T) + Em (xp((ii(())))))m 4 )/T)
(5)
TripletCLIP (Patel et al., 2024) further incorporates hard negative images 1_ The training objective encourages I+ to be closer to T than T and symmetrically, I to align more with T than with T This is achieved by summing two NegCLIP-style losses:
ITrpleeCLI , ) NegCLIP(T* ,T,,))+++ (NT
(6)
While prior work enhances image-text alignment using modality-spediin hard negatives, it tends to focus on coarse-grained semantic differences. We extend this framework by introducing abstract concepts anchors to better connect detailed captions and images. Through a refined training objective (Section 4.2), our model intends capture fine-grained cultural semantics with greater precision.
4.2 CultureCLIP
To better capture fine-grained cultural semantics, we build on the Twin Cards introduced in Cultwin, where each card contains two triplets-(C*, I+, and (C ""),,T,)--- -epreeeennnnningg visually similar but culturally distinct concepts. These pairs serve as hard negatives for each other. Our goal is to align each concept with its corresponding caption and image while distinguishing it from its cultural counterpart.
To achieve this, we propose CultureCLIP, a contrastive learning framework that jointly embeds concepts, captions, and images into shared semantic space. As illustrated in Figure 1, we encourage semantic attraction within each triplet and repulsion across its cultural counterpart. In addition, the overall architecture of our pipeline, including this learning scheme, shown in Figure (Right). We employ a shared text encoder for both concepts and captions to preserve the original alignment between images and captions while aligning concepts with images. This shared encoder design ensures that fine-tuning for cultural distinctions does not degrade the model's general cross-modal alignment ability. Our overall training objective is defined as:
CCultunCLII d₂ Leoncopt + Aạ Leaption
where Ae and A balance the contributions concept-level and caption-leved objectives. Both objectives are symmetrically formulated to promote attraction within positive triplets and repulsion from their cultural counterparts:
capptonn = NNeeULI TT, + CNegCLIP ,T,
(8)
Leoncept
Coccuur(tttttcC... (Nneem((1,CC,CC))
(9)
This structure jointly anchors abstract cultural concepts to specific visual-textual cues, enhancing the model's ability to differentiate subtle cultural semantics. To further preserve the original model's generalization ability, we apply the parameter-efficient LoRA method for fine-tuning on this loss, rather than directly training both the visual and text encoders.
Experiment
5.1 Experimental Setup
Model Choices and Prompting Strategies In this work, we leverage Qwen2.5-VL. (Bai et al., 2025) as our LLM/VLM (note that the specific choice model is not the primary focus of this paper). Although more powerful LLMs could potentially further improve data quality, we consider Qwen2.5- VL to strike a good balance between performance and cost, making it a practical choice for our experiments. Given the significant impact that prompt design has on LLM performance (Brown et al., 2020), we meticulously craft prompt templates for each LLM in the pipeline, employing a few-shot learning approach. This includes presenting set of example input-output pairs to guide the model in various tasks. All prompts used in this study are provided in Appendix F. For the text-to-image generation task, use Stable Diffusion 3.5 (Rombach et al., 2022) as the model for image synthesis.
7

---

Published as a conference paper at COLM 2025
Benchmarks We evaluate the models on both culture-specific and culture-agnostic tasks. To assess cultural understanding, we adapt three (Bhatia al., 2024), and CROPE (Nikandrou et al., 2024)-intb statement-ranking tasks suitable for CLIP-based models. In each task, the model must select the most semantically accurate description for given image from set of culturally grounded statements, thereby testing its ability to capture fine-grained, culture-specific visual cues (reported as Accuracy). To assess general vision-language capabilities, we further evaluate the models on MS COCO (Lin et al., 2015) and Flickr30k (Plummer et al., 2016), reporting the average Recal for both image-to-texx and text-to- rext-to-image retrieval tasks. Details of the benchmarks and additional evaluation results on widely used image classification datasets are provided in Appendix D.
GlobalRG-Retrieval
Baseline We first evaluate the performance of CLIP (Radford al., 2021), NegCLIP (Yuksekgonul et al., 2022), and TripletCLIP (Patel et al., 2024) on our tasks to establish a baseline. Building on this, we introduce CLIP++, NegCLIP++, and TripletCLIP++ as enhanced baselines. In these ++ versions, we train the base CLIP model using our own dataset, aligning the synthetic images with contextualized caption-image pairs, without incorporating the concept.
Implementation Details All fine-tuned CLIP models use ViT-B/32 (Dosovitskiy et al., 2021) as the image encoder F and the default CLIP text encoder (Radford et al., 2021) as the text encoder G. We fine-tune the model using and G are frozen, and additional LoRA parameters for these two encoders are applied and trained for 10 epochs with global batch size of 2048, learning rate of 3 x 10 weight decay of 0.1, and cosine learning rate schedule. We employ LoRA primarily to maintain the model's general capability rather than reduce memory requirements. In our preliminary experiments with full parameter fine- tuning, we observed significant performance degradation on both culture-specific and culture-agnostic benchmarks, likely due to the distribution gap between the general data used in pretraining and the culture-specific data used during fine-tuning, which disrupted knowledge the model had already acquired. Before inference, LoRA parameters are merged with the backbone transformers, ensuring both efficiency and the preservation of zero-shot transfer capabilities. All models are fine-tuned on Nvidia H20 GPUs using the official Hugging Face Transformers codebase (Wolf et al., 2020).
method, LoRA (Hu et al., 2021), during which F
5.2 Main Results
Evaluation on Culture-Specift Tasks As summarized in Table 2, CultureCLIP significantly outperforms all baseline models on cultural benchmarks, achieving 5.49% improvement over CLIP on GlobalRG-G, thus demonstrating strong tnne-grained cultural understanding, Directly fine-tuning CLIP on our cultural dataset (CIIP++) leads to a substantial performance drop of 17.99% highlighting the limitations of naive fine-tuning without explicit negative samples or concept-level alignment. In contrast, our enhanced variants NegCLIP++ and TripletCLIP++ (which incorporate hard negatives) achieve improvements of 2.34% and 0.80% over CLIP, respectively. When further combined with CuItureCLIP, we observe a large net improvement, emphasizing the effectiveness of our design in leveraging hard negatives and concept supervision for cultural feature discrimination. Similar trends are observed for GlobalRG-R and CROPE. We note that NegCLIP and TripletCLIP are pre-trained based on much smaller datasets (e.g. CC3M, CC12M) and do not include culturally relevant data, resulting in substantially lower performance on general benchmarks. For fairness, we do not consider them direct baselines the cultural evaluation but include them the table for completeness. All models in Table 2 are trained on the same unfiltered 100k synthetic dataset using LoRA with rank 4 to ensure fair comparison.
concept-level alignment
Evaluation on Aulstre-Agnostic Tasks On general vision-language tasks, CultureCLIP main- tains strong performance and even slightly improves over the baseline, with gains of 0.90% on MS COCO and 0.30% on Flickr30k. This indicates that cultural fine-tuning does not compromise, and may even enhance, general retrieval capabilities.
5.3 Ablations
Which contributes more to the model and alignment, caption or concept? As shown in Table 3, concepts act as abstract semantic anchors, providing stronger cultural alignment capacity and enabling the model to distinguish subtle differences. Captions refine the model's I's understanding of specific details but are less critical for recognizing cultural nuances.
Can a higher-quality cultural dataset improve performance? As shown in Table 4, using quality-filtered data ("+QF"), which contains 73.8k high-quality samples after image filtering, consis-
8

---

Published as a conference paper at COLM 2025
Table 2: Experimental results on culture-specific and culture-agnostic tasks. All models are trained on the same unfiltered 100k dataset using LoRA with rank Best scores are in bold. Second best scores are underlined.
Culture- Splture-Specific Tasks
Culture-Agnostic Tasks
Methods
Neg
Con
GlobalRG-G
GlobalRG-R
CROPE
MS COCO
Flickr30k
CLIP
63.98
78.22
74.69
65.40
89.0
NegCLIP TripletCLIP
6,50
2.70
10.80
22.00
CLIP+ (ours)
46.05
49.98
73.62
28.80
50.80
+ (ours)
66.32
78.41
79.25
65.50
89.20
TripleCCIIP++ (ours)
64.78
78.45
79.25
65.50
89.30
CultureCLIP (ours)
69.47
78.60
78.84
66.30
89.30
Table 3: Ablation study on loss configurations. All models are trained on the same unfiltered 100k dataset using LoRA (rank 4) to preserve general multimodal alignment capabilities.
Configuration
Mcca//cn))
Speeife-Specific Tasks
Culture-Agnostic Tasks
GlobalRG-G
GlobaIRG-R
CROPE
COCO
Flickr30k
Single Branck (No Negative)
Captiun-unly w/o neg
1.0 --
66.95
77.43
79.19
65.60
89.20
Concept-only w/ neg
-/ 1.0
64.24
77.70
79.19
65.50
89.00
Single Branch (With Negufive)
Caption-only w/ neg
1.0 /- -/ 1.0
66.27
77.53
29.25
65.50
89.20
Concept-only w/ neg
65.83
77.29
79.19
65.60
88.90
Mixed Branches
Cap (w/o neg) Con (w/o neg)
0.5
67.29
77.27
79.25
65.50
89.20
Cap (w/ neg) Con (w/o neg)
0.5 / 0.5
67.12
78.70
79.25
65.60
89,30
Cap (w/o neg) Con (w/ neg)
0.5 / 0.5
68.81
76.87
79.19
65.50
89.20
(Both with Negative)
Both w/ neg (Ours)
0.7 / 0.3
65.93
78.80
79.37
66.80
89.50
Both w/ neg (Ours)
0.5 / 0.5
67.12
78.25
78.60
66.30
89.20
Both w/ neg (Ours)
78.60
78.84
89.30
tently improves performance despite using fewer samples compared to the full 100k unfiltered set. For example, comparing Config (+QF, LoRA r=4) to Config 3 (LoRA r=4), and Config 6 (+QF, LoRA r=8) to Config (LoRA 1=8), observe clear performance gains across all cultural benchmarks. This underscores the effectiveness of our data curation pipeline in providing cleaner and more informative supervision for fine-grained cultural alignment.
What role does LoRA play in adapting a pretrained model to downstream tasks? Our ablation results in Table show that without LoRA (e.g.. Configs
Table 4: Ablation study on quality filtering (QF) and LoRA. "+QF" uses the 73,8k filtered samples; otherwise, the full 100k dataset is used.
Configuration
LoRA Rank
GlobalRG-G
GlobaIRG-R
CROPE
1 and 2), directly fine-tuning
Beseline
46.95
49.98
73.62
pretrained model on domain- specific cultural dataset leads to a substantial performance drop-sseciffiallly 22.52% and 26.03% lower compared to Con-
43.64
50.68
74.33
+ LORA (r=4)
69.47
78.60
78.84
+ LORA (r-8)
65.29
78.03
79.37
LoRA (r=4)
QF LORA (r-8)
66.48
78.21
79.37
figs and 5, respectively. This suggests that, without LoRA, the model struggles to effectively absorb specialized supervision, resulting in significant loss of generalization ability. In contrast, when LoRA applied (Configs 3-6), the model not only achieves improved performance but also maintains capabilities on general tasks. These findings highlight LoRA's essential role mitigating catastrophic forgetting: enables the model to flexibly adapt to new cultural signals while preserving the broad vision-language alignment learned during pretraining, ensuring robustness across both domain-specific and general scenarios.
cultural benchmarks
Conclusion
In this paper, introduce CulTwin, high-quality synthetic dataset of paired concept-caption-maan triplets verified by humans, where captions enriched with cultural background knowledge using

---

Published as a conference paper at COLM 2025
Vasion-language model, and images generated by text-to-image model to reflect fine-grained visual features. Building on CulTwin, we propose CultureCLIP, a novel contrastive learning frame- work that jointly aligns cultural concepts, captions, and images in shared embedding space. Our experiments demonstrate that CultureCLIP surpasses baseline models on culture-specific benchmarks, achieving 5.49% improvement while simultaneously showing performance gains rather than degra- dation culture-agnostic benchmarks. These results underscore the effectiveness of synthetic dataset and training methodology capturing nuanced cultural distinctions while preserving and even enhancing the model's generalization capabilities across broader contexts.
Limitations and Future Work
While CultureCIIP significantly improves fine-grained cultural understanding, several limitations remain. Both CLIP and CultureCLIP still struggle with cases where the visual distinction is highly abstract or stylistic (see error case analysis in Appendix E). In addition, our current pipeline, for practical considerations, adopts Qwen2.5-V2 as an MLLM-as-a-Judge to assess cultural relevance through multidimensional scoring and to guide filtering. However, compared to more advanced models such as GPT-4o, this choice may introduce biases, lead misjudgments, or lack interpretability Furthermore, despite the diversity of Cul it fundamentally fully synthetic dataset, and there may still exist distributional gap between synthetic and real-world images. Looking ahead, future work may explore several directions: (1) Improving abstract visual reasoning, by enhancing the model's capacity to recognize subtle visual cues, such as artistic styles symbolic meanings; (2) Enhancing the robustness of cultural understanding, by mitigating vulnerability visual variations and ensuring stable performance across diverse conditions (Fan al., 2025); (3) Developing inter- pretable assessment modules, to enable more robust and transparent cultural data evaluation beyond the current Owen2.5-VL-basee judge; and (4) Bridging the synthetic-real domain gap, by integrating real and synthetic data or applying domain adaptation strategies to improve generalization and visual grounding.
10

---

Published as a conference paper at COLM 2025
References
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025, URL https://arxis wttpaaawwi..../aaaaaeee org/abs/ abs//502.. 13923.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto del Bimbo. Composed image retrieval using contrastive learning and task-oriented clip-based features, 2023. URL https://aaxxii.. orrg/ abs/2308. 11485.
Mchar Bhatia, Sahithya Ravi, Aditya Chinchure, Eunjeong Hwang, and Vered Shwartz. From local concepts to universals: Evaluating the multicultural understanding of vision-language models. arXiv preprint ar Xiv::400.00000 000:2240.00023,, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vort Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https:
/ttpss//arxivv http//aaaivrgg//////////////tl org/ abs/2085. 14165.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark, 2024. URL https::/arxxv..rg////// 2402. 04788.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv. org/bbgg/////oo httg/bb//2000 0010 11999.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36:35544-35577 2023.
Zhiyuan Fan, Yumeng Wang, Sandeep Polisetty, and Yi R. Fung. Unveiling the lack of Ivlm robustness to fundamental visual variations: Why and path forward, 2025. URL https: //axxiv. org/ a552/44. 16727.
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip), 2022. https/tps://ark/ www.:br///bbb/////// htt.:tppp//aavvv..rr//////222111 81397.
Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Berard Ghanem. Synthclip: Are we ready for a fully synthetic clip training? arv preprint 200::2202..1832 2024.
Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Yuchen Huang, Shujin Wu, and Yi R. Fung. Mmboundary: Advancing milm knowledge boundary awareness through reasoning step confidence calibration, 2025. URL. https://arxiv. org/ abs/ 2 23224.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//arxiv. //arkiv.org/abs/211 2106 09685.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. URL http::://aaxiv.rr////aae/ae 2 210..020200002 85918.
Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martinez, Saahil Ognawala, Susana Guz- man, Maximilian Werk, Nan Wang, and Han Xiao. Jina clip: Your clip model is also your text retriever, 2024. URL https://arxiv http:/://aaxi.orro/a/sssss 20224
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
11

---

Published as a conference paper at COLM 2025
Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, al. Chemvim: Exploring the power of multimodal large language models in chemistry area. Xie preprint ar xfx::22408.072446 2024.
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URL httpp::/aarxv..orrrr org. https////4......
Shudong Liu, Yiqiao Jin, Cheng Li, Derek F Wong, Qingsong Wen, Lichao Sun, Haipeng Chen, Xing Xie, and Jindong Wang. Culturevim: Characterizing and improving cultural understanding of vision-language models for over 100 countries, ariie preprint ar arxii:2201.01222 01282, 2025.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. 2024.
ar arviv preprint ar Xiv:2403.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Aishwarya Agrawal, et al. Benchmarking vision language models for cultural understanding. ar Xiv preprint (tfo:2407.10920,, 2024.
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip, 2023. URL https: //arxiv. wrgp/b//////....//////////////// .5516.
Malvina Nikandrou, Georgios Pantazopoulos, Nikolas Vitsakis, loannis Konstas, and Alessandro Suglia. Crope: Evaluating in-context adaptation of vision and language models to culture-specific concepts. arviv preprint ar Xiv:2410.15453. 2024.
Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives. Advances in Neural Information Processing Systems, 37:32731 2024.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reason- ing abilities through two-stage rule-based rl, 2025. URL h org/abs/2503. 255./56633...6...
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016. URL httpp://arkxw..rg/aab///// 058.0/045500.00.....
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL https://arxiv. http:///aaavv.ooo////////// ar//22//212 10752.
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hemán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D'Haro, Marcelo Viridiano, Marcos Satecha-Garitagoitte Maria Camila Buitrago Cabrera, Mario Cadriiuez-Cantelar, Mélanie Jouitteau, Mihail Mohaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Mónica Silva, Pran- jal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijay,, Santiago Góngora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Fhsan, Vladimir Araujo, Yova Kementched- jhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea,
12

---

Published as a conference paper at COLM 2025
Thamar Solorio, and Alham Fikri Aji. Cvqa: Culturally-divers, multilingual visual question answering benchmark, 2024. URL https://arxiv. org htg//bb//2406 059677
Oindrila Saha, Grant Van Horn, and Subhransu Maji. Improved zero-shot classification by adapting vlms with text descriptions, 2024. URL httpp::/arrxv..or//aae 02460
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models, Advances in neural information processing systems, 01522278-25294 2022.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks?, 2021. URL https: //trkiv.org/aas////// 2ttpb////117..00. 06383.
Haoyu Song, Li Dong, Weinan Zhang, ling Liu, and Furu Wei. CLIP models are few-shot learners: Empirical studies on VQA and visual entailment. In Smaranda Muresan, Preslav Nakov, and
Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6108, Dublin, Ireland, May 2022. Association for URL
Computational Linguistics. doi: 10.18653/ 21233//122222aclloooo/ acl-locl 421/.
org/
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. emnlp-demos. 6.
URL https://www aclweb.
Hu Xu, Saining Xie, Xiaoging Ellen Ian, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. ar preprint arXio:2309.166 16671, 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint ar arXio:2210.01966, 2022.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pp. 310-325. Springer, 2024a.
Gengyuan Zhang, Yurui Zhang, Kerui Zhang, and Volker Tresp. Can vision-language models be good guesser? exploring vlms for times and location reasoning. Proceedings of the IEF.F/CVF Winter Conference on Applications of Computer Vision, pp. 636-645, 2024b.
Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, and Yi R. Fung. V1m2-bench: closer look at how well vlms implicitly link explicit matching visual cues, 2025. URL https: ttttps://arka http:///xxxvvvvveee .rgg /bs 2502. 2284.
Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio- lisouliigciitic compositional understanding. In Proczedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13774-13784, 2024c.
Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning, 2025. URL haatps://arriv..rr//bb///////// 253.2
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337- -3448, July 2022. ISSN 1573- 1405. doi: 10. 1213-1222222222 020---0222--66622 01633-1. URL httt:://dxx.o//// 10./10. 1887//11226--2222
13

---

Published as a conference paper at COLM 2025
A Country List and Cultural Taxonomy
In this study, the list of country names is based data from the GeoNames database: GeoNames.org
The cultural taxonomy includes the following categories, each representing a significant aspect of cultural identity:
Cuisine: Refers to the foods, culinary practices, and cooking methods that are unique to specific regions or cultures, This includes iconic dishes, preparation techniques, and the cultural background behind eating habits, as well as the importance of food in social and religious practices.
Encompasses traditional garments, accessories, and adornments from various cultures. It includes not only clothing but also items like jewelry, headwear, and footwear that hold cultural significance, reflecting identity, status, and traditions.
Clothing:
Describes the native species, both fauna and flora, that hold cultural importance. This category includes the use of animals and plants in mythology, cuisine, traditional medicine, and environmental practices, well as their roles in folklore and symbolism.
Animal Plants:
Art: Includes visual arts, sculptures, and other forms of artistic expression that represent culture's aesthetic and artistic heritage. This encompasses paintings, sculptures, performance arts, and crafts that reflect the identity, beliefs, and historical evolution of community.
Architecture: Refers to the design, style, and structures built by particular culture. This includes traditional houses, temples, monuments, and public buildings that showcase the engineering, material use, and aesthetic values of the culture.
Covers the everyday activities, routines, and practices that define how people in a particular culture live. This includes family roles, work habits, and leisure activities, as well practices around health, education, and community.
Daily Life:
Symbol: Involves the symbols, logos, and imagery that carry cultural meaning. This category includes national flags, religious icons, mythological figures, and colors that convey beliefs, values, and identity in various contexts.
Festival: Encompasses cultural festivals, holidays, and ceremonies, along with the associated customs, rituals, and practices. Examples include events like Chinese New Year, Diwali, and Christmas, each rich in traditions, foods, and rituals that symbolize community and heritage.
14

---

Published as a conference paper at COLM 2025
B CulTwin Details
We collected total of 99,996 Twin Cards, each consisting of two concept-captioniiant triplets with culturally distinct negatives. After applying the image quality filtering described in Section 3.3, 73,823 high-quality samples were retained scale significantly larger than existing cultural benchmarks such CROPE (approximately corresponding captions have an average length of 14.55 words. All images are synthetically generated from these captions using Stable Diffusion 3,5 Large Turbo, with efficient generation throughput of approximately 3,000 images per hour single H20 GPU. Additional examples of Twin Cards illustrated Figure
samples). Each concept is represented
single word, and
CUISINE
ARCHITECTURE
Xlaetong Bao: chef carefullyy places Xiaolongtao in bamboo showtasing translucant waaperr savory broth inside.
Shenglian Beo: sizzling Shanglianbao hotiiron skillee, its goldlnn crispy bottom contrasting with the sesame seeds top.
Siheyuen: Courtyard siheyuan with lantems green plants, symbolizing prosparity and vitality.
Hut: Weathared wonden aurrounded greanary, typical of rurai landscapes.
CLOTHING
DAILY LIFE
Lady black kimono with delicate sakura blossoms, the vibrant obi sash elegantly, walking allng the Sumida Riverin Tokyo.
Kebaya: women embroidered kabayas, fosturing intricate floral desigrs, gracafully. showesssing g treditional Indonesian dress.
Traditional grinding stona with sylindricsi pestle, used for grinding grains spices.
brown rolling pin with handle wrappeed falbric, found traditional Korean bakery.
ANIMAL & PLANTS
SYMBOL
Glasswort: Green glasswort lik leaves away the coastal
Chamomile: Chamomile
Cenotaph: Cenotaphi flags, surrounded by biooming poupies at ANZAC Day
Hero Conved Nedu, India, with inscriptions and symbolic figures.
petail yellow centers bioom emidst üreen grass, exuding calmnees.
follen
breece, their surfaces glistening sparkling
ARI
FESTIVAL
Shiboris Indigo- rrddgodyydd kumn shibori fabric spider patterns, reflecting Japanese textile craftamanship
Tie-dye: Woman arranging vibrant tie tabrics, showcssing patterss and bolci colors.
Chhath: A weman
Andi Perukkur Children in traditional attire float floating candles during Perukku, Tamii festival honoring water.
fervently family watches the riverbank.
Figure 4: Additional Twin Card examples showcasing diverse cultural concepts beyond the main text examples, further illustrating fine-grained cultural distinctions and visual similarity.
15

---

Published as a conference paper at COLM 2025
C
CultureCLIP Pseudocode
turecl ieliiploosss loss (possscccccee..em pos_caption_embs
pos_img iem_imm_ebb,,.....
neg_concept_embs, neg_caption_embs, neg_img_ logit_scale, umbda_ccation=00.0
combda_concept
Full CultureCLIP Loss: Caption's negclip + Concept 's negclip Aligns concepts with their corresponding captions and images, while distinguishing them from their culturally opposite counterparts.
Calculate Caption's NegCLIP loss
(pos__m_eemm pos_caption_emb
neg-caption_ent logit_scales negclip_loss (neg_img_embs veg_caption_embl pos_caption_embs logit_scale)
# Calculate Concept NegCLIP loss concept_loss_val
negelip_loss imm___m_emmb,, pos_concept_embs,
neg ceeceptnnept_eeee embs, logit_ oggit_scale) +
reg_concept_emi (ww.llssssenenmmmme_eeeeeee pos_conccetteemb,, logit_scale)
# Total Loss
total_los, = lambda_caption caption_loss__al lambda_concept
concept_
return total_loss
D Benchmark Details and Additional Results
In this section, provide detailed descriptions of the benchmarks used to evaluate models, covering both culture-specific and culture-agnostic tasks.
Culture-Specific Tasks
GlobalRG-Grounding (Bhatia al., 2024): Each data point consists of image, concept, and country. We generate four statement-based options for the model choose from, such as "The item in the picture is (concept) in (country)." The correct option corresponds to the appropriate concept for that country, while incorrect options are created by selecting concept from the same country that does not match the image.
GlobalRG-Retrieve G-RR--Reeti Retrieaal (Bhatia et al., 2024): Each data point consists of an image, category, and country, without specifying particular concept. The task focuses identifying the correct country for the depicted category. Options are phrased "The picture depicts kind of {category) in [country)," " with incorrect options generated by randomly substituting the country.
CROPE (Nikandrou al., 2024): The original dataset asks whether the image displays the defined concept ("yes" or "no"). We filter out "no" cases where the question concept and definition concept differ, and reformulate the task as two-choice classification: "There is {question concept) in the image" or "There is (definition concept) in the image," " requiring the model to correctly identify the depicted concept.
16

---

Published as a conference paper at COLM 2025
Culture-Agnostic Tasks
COCO el,, 2015): This large-scale dataset comprises over 330,000 images annotated with approximately five captions each. We evaluate using bidirectional retrieval metrics, specifically Text2Image and Image2Text Recall@5, and report their arithmetic mean as the final score.
Flickr30k (Plummer et al., 2016): This dataset contains 31,000 images primarily focused on human activities, each paired with five descriptive captions. Similar to MS COCO, we adopt bidirectional retrieval metrics to assess moos- alignmment.
More General Image Classification Benchmarks: To further assess generalization capabili- ties, we evaluate our models on several widely used image classification benchmarks, includ- ing FER2013, ImageNet-1k, ImageNet-A, ImageNet-O, ImageNet-R, VOC2007, CIFAR-10, and CIFAR-100. These datasets collectively cover a broad spectrum of visual tasks, such as facial expression recognition, large-scale object classification, out-of-distribution robustness, multi-label classification, and both coarse- and fine-grained category recognition. Empirical results demonstrate that CultureCIIIP consistently maintains, and in some cases slightly improves, performance these benchmarks. This suggests that the incorporation of cultural fine-tuning does not compromise general vision-language alignment or classification capabil- ities but rather enhances overall robustness and versatility, A comprehensive summary of these results, including Top-1 Accuracy (Accl), Top-5 Accuracy (Acc5), and Mean Per-Class Recall (MPCR), is provided in Table 5.
Table Performance on general image classification benchmarks (%). We report Top- 1 Accuracy (Accl), Top-5 Accuracy (Acc5), and Mean Per-Class Recall (MPCR) across various datasets. CultureCLIP maintains or slightly improves general performance despite additional cultural training.
Model
FERMIT3
ImageNet-I1 ImageNet-A
ImageNet-O
ImageNNetRR
VOC2877
CIFAR-10
CIFAR-100
Jap-2 Ancwrac (Acc])
OpenAt CLIP
41.22
63.37
31.51
47.35
69.31
76.45
80:77
64.24
Caplion (w/e)
41.57
63.37
$1.61
47.50
69.28
76.52
87.84
64.48
Caption (w/o neg) Cancept (wr/ neg)
41.20
63.35
31.59
17.65
69.33
76.
59.86
61.40
Caption (w/ neg!
41.99
63.39
31.61
47 65
69.26
76.52
30.80
61:45
CultureCLIP
41.26
63.37
31.60
47
69.31
76:45
89.83
64.47
Aep-SAe Arconcy (AceS)
OpenAT CLT
94.75
68.82
64.23
78.30
88.81
95.95
90.61
88
Ciption (w/o neg)
94.65
53.50
64.31
78.
$8.50
95.95
90.62
88.91
Caption (w/o neg) Concopt (w//o neg)
91.82
$5.83
61.20
78.25
$8.00
95.94
99.63
85.8
Caption (w/ neg!
91.69
$3.82
61.33
10
$8.79
95.95
99.63
88.87
CultureLiPP
94.80
88.90
64.29
78.10
$8.55
96.95
99.63
88.65
Moon For
For-Clas Clas Rocall (MPCR)
OpenAl CLI
36.10
63.36
32.63
18.55
67.92
$0.59
89.83
64.21
Caption (w/
36.00
63.36
32.75
49.09
67.92
80.60
59.85
64.46
Caption (w/o begg Concopt (w/o neg)
36.00
63.37
32.72
19.01
67.94
90.64
99.87
66.40
Caplion (w/
36.42
63.99
92.72
49.04
67.57
80.58
89.83
64.41
CultureCLIP
36.21
63.37
32:77
49.00
67.96
$0.60
89.85
64.41
17

---

Published as a conference paper at COLM 2025
E
Error Case Analysis
Distinguishing between Gongbi and Xieyi styles One illustrative failure case involves dif- ferentiating between two classic Chinese painting styles: gongbi and xieyi. The gongbi style is known for its meticulous brushwork, fine lines, and realistic details, often used to depict flowers, birds, and other subjects in highly controlled and precise manner. contrast, the xieyi style (literally "writing ideas") emphasizes freehand expression, bold strokes, and abstract or suggestive forms rather than realistic details. In this example, both CLIP and CultureCLIP misclassified an input gongbi painting as xieyi. However, CultureCLIP assigned lower confidence to the incorrect label (72% vs. CLIP's 78%), indicating modest calibration improvement. This suggests that while our model still struggles with highly abstract stylistic distinctions, it demonstrates better uncertainty awareness compared to the original CLIP, which a step toward more nuanced cultural reasoning.
Model Prediction
CulturoUPP
This
0.28
0.70
Kay painting
0.2
0.
0.
Input Image: An image of Gongbi Painting
Figure 5: Example failure case comparing CLIP and CultureCLIP on a gongbi painting. While both models misclassified it as xieyi, CultureCLIP exhibited lower confidence in its wrong prediction, suggesting better calibration.
Prompt Engineering
Bottom-up Filtering Prompt for Qwen2.5-VL
Please determine whether the concept '[title)' clearly and unambiguously belongs to one of the eight cultural categories ( Cuisine, Clothing, Animals & Plants, Art, Architecture, Daily Life, Symbol, Festival). If the concept is only loosely related. culturally ambiguous, or does not strongly align with any of the categories, please select "A' to ensure strict filtering. The concept does not clearly belong to any of the eight categories. The concept is clearly and directly related to one of the eight categories. Answer only with 'A" 'B'.
Output the result in JSON format as follows:
{
"concept_type": "A" "B"
Figure 6: Bottom-up filtering prompt for Qwen2.5-VL.
18

---

Published as a conference paper at COLM 2025
Bottom-up Classification Prompt for Qwen2.5-VL
Given a cultural concept, definition, definition caption, and definition image, your task is to classify and extract the following information in English:
country or region associated with the concept. Cultural Category: IMPORTANT You MUST choose exactly ONE category from these eight predefined categories:
-Countr::
Cuisine
Clothing Animal Plants
Art
Architecture
Daily Life
Symbol
Festival
Do not use any other categories. If the concept doesn 't clearly fit into one of these categories, choose the closest match.
English that provides insight into the cultural and functional use of the concept. Key Visual Features: The visual features that distinguish the concept (e.g., shape, material, color, size) described in English.
Context: A short 20-word description
Examples:
Input:
Concept:
Kimono
Definition: traditional Japanese garment with long, wide sleeves and an obi sash, worn for formal occasions and festivals.
Definition Caption: colorful kimono with crane patterns, displayed at Kyoto's textile museum.
Definition Image: example_ imalle__mmaeejjpp jpg
Output:
Country: Japan
Category: Clothing
Context: traditional Japanese garment with long, wide sleeves and an obi sash, worn for formal occasions and festivals.
Key Visual Features: Long wide sleeves, obi sash, colorful patterns, traditional Japanese style.
Current Task:
Concept: (concept)
Definition: (definition) Definition Caption: (caption)
Definition Image: (image_url)
Generate the Country, Cultural Category (MUST be one of the eight predefined categories), Context, and Key Visual Features in English:
Figure 7: Bottom-up classification prompt for Qwen2.5-VL.
19

---

Published as a conference paper at COLM 2025
Top-down Generation Prompt for Qwen2.5-VL
Given country and cultural category, your task is to generate the following:
Concept: cultural concept that fits the provided country and category.
Context: A short 20-word description that provides insight into the cultural and functional use of the concept.
Key Visual Features: The visual features that distinguish the concept (e.g., shape, material, color, size).
Examples:
Input:
Country: China
Cultural Category: Food
Output:
Concept: Mantou
Context: Steamed wheat bun symbolizing prosperity and wisdom, commonly served during festivals and family gatherings.
Key Visual Features: Pillowy white appearance, round or rectangular shape, smooth surface, typically palm-sized. Input:
Country: Japan
Cultural Category: Art
Output
Concept: Ukiyo-e
Context: Traditional woodblock prints depicting scenes from everyday life, nature, and historical events.
Key Visual Features: Flat color blocks, bold outlines, vibrant pigments, rectangular format, detailed patterns.
Current Task:
Country: (country)
Cultural Category: (category)
Generate the concept, context, and key visual features:
Figure 8: Top-down generation prompt for Qwen2.5-VL.
20

---

Published as a conference paper at COLM 2025
Twin Matching Prompt for Qwen2.5-VL
Given a cul cultural concept from a certain category, its context, and its visual features, your task is to generate the following: New Concept: A visually similar but culturally different concept from the same category.
New Context: short 20-word description English that provides insight into the cultural and functional use of the generated concept.
New Key Visual Features: The visual features that distinguish the concept (e.g., shape, material, color, size) from the original concept.
Examples:
Input:
Category: Art
Concept: Erhu
Context: A two-stringed Chinese musical instrument played with bow, often used in traditional Chinese music. Key Visual Features: Two strings, bow, wooden body, and horsehair bow.
Output:
New Concept: Guzheng
New Context: Chinese zither-like instrument with large, rectangular wooden body and multiple strings, played with plucking.
New Key Visual Features: A large, rectangular wooden body, multiple strings, and a plucking mechanism.
Current Task:
Category: (category)
Concept: (concept)
Context: (context)
Visual Features:
Generate new concept, context, and visual features:
Figure 9: Twin matching prompt for Qwen2.5-VL.
21

---

Published as a conference paper at COLM 2025
Diverse Caption Generation Prompt for Qwen2.5-VI
Given cul tural concept, context, and key visual features, your task generate different captions that describe the concept various scenarios while preserving its cultural significance. The captions must reflect different styles or settings, but they should all clearly include the key visual features. Follow these guidelines:
key visual differences shape, size, cooking method, setting).
Emphasize
scenes (e.g.,
Retain the cultural or functional context (e.g., everyday use ceremonial purpose, tradition).
Ensure the differentiating features (e.g., shape, texture, size, material, use) are clearly reflected each caption. Each caption should be under 15 words.
Each caption should be unique, showing different perspectives settings, but should always include the key visual features
Examples:
Input:
Concept: Xiaolongbao (Soup Dumplings)
Context: Steamed soup dumplings with delicate, thin wrappers, filled with savory broth and pork, typically steamed in bamboo baskets.
Key Visual Features: Delicate, thin wrappers and steamed in bamboo baskets.
Output:
1. A chef carefully places Xiaolongbao in bamboo steamer, showcasing their thin, translucent wrappers and savory broth inside.
2. A steaming bamboo basket of Xiaolongbao, delicate wrappers holding savory broth, served in an elegant Shanghai restaurant.
3. Steamed xiaolongbao resting in bamboo baskets, ready to served during family meal.
4. Crispy fried xiaolongbao, golden-brown and served with dipping sauce, sitting in bamboo baskets.
5. Miniature xiaolongbao filled with crab roe, elegantly presented in bamboo baskets at Cantonese restaurant.
Steaming xiaolongbao with delicate skin, served bamboo baskets during a traditional Chinese New Year meal. 7. Bamboo steamed xiaolongbao, filled with savory broth, served alongside hot tea in Beijing teahouse.
8. Translucent, plump xiaolongbao, freshly steamed in
bamboo baskets for a cozy brunch setting.
9. Steamed xiaolongbao with pork filling, served in bamboo baskets with chili oil at a street food stall.
10. Elegant xiaolongbao, arranged in bamboo baskets, presented at lavish festive feast.
Current Task:
Concept: {concept)
Context: [context)
Key Visual Features: (visual_features
Generate 10 different captions, each reflecting a different style or scene, but all incorporating the key visual features:
Figure 10: Diverse caption generation prompt for Qwen2.5-VL.
22

---

Published as a conference paper at COLM 2025
Prompt for Quality Evaluation on Authenticity
Please analyze this image and rate its authenticity on scale of to 5. You can refer to the context to help you make the decision.
Focus whether the concept shown is realistic and follows common
sense. Consider:
Are all elements anatomically and physically correct?
Does everything look natural and possible in the real world?
Are there any unrealistic or deformed features?
Examples:
Input:
Concept: Erhu
Context: The Erhu is traditional Chinese musical instrument played with bow. consists a wooden body and two strings, and is known for its expressive and resonant sound. Often used in Chinese classical and folk music, it is typically performed in both solo and ensemble settings.
Image: image showing person playing Erhu with three hands in mid air
Output
Input:
Concept: Erhu
Context: The Erhu is a traditional Chinese musical instrument played with a bow. It consists of a wooden body and two strings, and is known for its expressive and resonant sound. Often used in Chinese classical and folk music, it is typically performed in both solo ensemble settings.
Image: An image showing person playing Erhu with two hands sitting on the chair;
Output:
Now, give you the image, concept and its corresponding context: Concept: (concept)
Context: (context)
Output only score:
Figure 11: Prompt for quality evaluation on authenticity.
23

---

Published as a conference paper at COLM 2025
Prompt for Quality Evaluation on Consistency
Please analyze this image and rate its consistency with the concept on a scale of 1 to 5. You can refer to the context to help you make the decision. Focus on whether the image accurately depicts the specified concept without showing wrong concepts. Consider: Does the image show exactly the concept mentioned? Are there any mismatched or wrong elements?
Is the concept clearly and accurately represented?
Examples:
Input:
Concept: Tang Sancai
Context: type of Chinese glazed pottery from the Tang Dynasty, known for its colored glazes (green, yellow, and brown) often used for tomb figurines and decorative pieces. Image: An image showing blue -uue-and-wiie white porcelain bowl
Output
Input:
Concept: Tang Sancai
Context: type of Chinese glazed pottery from the Tang Dynasty, known for its colored glazes (green, yellow, and brown) often used for tomb figurines and decorative pieces. Image: An image showing exactly Tang Sancai horse
Output
Now, give you the image, concept and its corresponding context: Concept: [concept)
Context: [context]
Output only the score:
Figure 12: Prompt for quality evaluation on consistency.
24

---

Published as a conference paper at COLM 2025
Prompt for Quality Evaluation on Cultural Fidelity
Please analyze this image and rate cultural fidelity on scale
of 1 to 5, Focus on whether the cultural elements are accurate and appropriate for the specific context. Consider:
Are all cultural elements accurate for this context?
Are there any mixed or incorrect cultural elements?
Does everything align with the cultural background specified?
Examples:
Input:
Concept: Mexican Day of the Dead
Context: Mexican holiday celebrating deceased loved ones, featuring marigold flowers, sugar skulls, and candle altars.
Image: image showing marigold flowers, skull paintings, and candle altars.
Output: 5
Input:
Concept: Mexican Day of the Dead
Context: Mexican holiday celebrating deceased loved ones, featuring marigold flowers, sugar skulls, and candle altars.
Image: An image showing showing marigold flowers, skull paintings, and Chinese joss paper money.
Output
Now, give you the image, concept and its corresponding context: Concept: [concept)
Context: [context]
Image: (image) Output only the score:
Figure 13: Prompt for quality evaluation on cultural fidelity.
25


Reviews:
[{"review_id": "wjhe5n5t1H", "reviewer": "Reviewer_9Kid", "summary": "The paper addresses the issue of lack of cultural representation in CLIP, i.e. the model's inability to distinguish similar but culturally distinct concepts. The authors create a synthetic dataset of pairs of visually similar concepts. Each concept is accompanied by an image and caption which feature the concept. CLIP is finetuned on this data using LoRA, using the contrastive loss from NegCLIP (which incorporates hard negatives), applied not just to image-caption pairs but also to image-concept pairs. Results on various culture-specific benchmarks indicate that the training has effectively instilled cultural knowledge into CLIP, and ablations show that this is in larger part thanks to the concept-image pairing. \n\nThe work is clearly written, although some details are missing (see below). I can't say that it is particularly original, as there is nothing about the objective, the creation of the data or the training of the model that is particularly innovative. That being said, a CLIP model with improved cultural knowledge is needed, and this work improves considerably on some relevant metrics, so perhaps members of the community will find the model (if publicly released), a useful tool for their work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is mostly well written.\n\nThe data creations procedure sounds reasonable.\n\nIt addresses a timely issue and shows very good results on appropriate benchmarks.", "reasons_to_reject": "The evaluation of how well the culture-specific training preserved the general capabilities of the CLIP base model is limited to retrieval on COCO and Flickr30k. At the very least, some image classification benchmarks should be included, such as ImageNet and CIFAR, as they are indicative of the generalization of the model to a broad range of objects. On a related note, I think it's important to introduce the cultural benchmarks in at least two-three sentences, so the reader can make better sense of the results and discussion. \n\nKey details about the dataset are missing, pertaining to its size, caption length, computational cost of its creation (e.g. in terms of time).\n\nThe authors do not state whether they will make the dataset and model public. \n\nError analysis or outlook for future work is missing. Some examples of CLIP v CultureCLIP outputs on the culture benchmarks would be nice to get a sense for how off-target CLIP was and where CultureCLIP might still be lacking in performance.", "questions_to_authors": "The authors state that \"A natural approach is to fine-tune CLIP on a large-scale dataset containing image-text pairs that explicitly describe culturally significant details of concepts.\". It is not immediately obvious to me why that's the natural way to go. The mention of the significant details, if anything, might give the model less incentive to store knowledge about the distinctive features of entity X, since those details are provided in the textual context and as such do not need to be memorized. Training with hard negatives is the natural way to go, as I see it, without providing any textual cues to the distinctive features, in order to force the model to actually _learn_ them from the data. Indeed at the end we see that concepts are more useful than captions, which I intepret to mean that when the model fully has to rely on the concept with no additional signal from the textual context to be used as a shortcut, it learns the visual features of the entity best. So I'm not sure why the authors chose to introduce the approach in this way.  \n\nI don't understand why the \"Efficient Contrastive Pre-training\" is called that. \n\n\"A VLM then evaluates their cultural relevance, associating each concept with a specific country and cultural category.\" -  what are the possible categories? How is the list compiled?\n\n\"Concepts that do not meet the criteria are discarded.\" -> what are the criteria?\n\nIt is not clear what the authors mean by front side and back side on l 144-146\n\nWhat is the final size of the dataset? What is the average length of the captions?\n\nWhat text-to-image model is used to generate the images?\n\nIs Qwen2.5-vl used for all steps involved in the generation of the data? What gives the authors confidence that this model is powerful enough to reliably carry out all tasks? Was any human validation performed?\n\nWhy is it a better idea to generate images of the concepts rather than source natural images from the web/existing image repositories? Is there any concern around the synthetic nature of the data? This should be discussed in the paper.\n\n\"This structure jointly anchors abstract cultural concepts to specific visual-textual cues\" - since there is no loss that concerns concepts and captions, I think the cues are only visual, right?\n\nI assume concepts are encoded with the same text encoder as used for captions. Are they presented as solo words or contextualized in a template of some sort? Is there concern for using the same module to encode two distinct types of inputs, effectively doing multi-tasking?\n\nWould it make sense to extend the pairs to cross-cultural concepts which may look alike but are in fact highly distinct? Such as a clay vessel which in one culture might be use to store food and in another to collect waste? Such examples were presented in the CVQA paper by Romero et al. 2024. \n\nThe data creation section is somewhat tidious and could be shortened, with parts of it moved to the appendix.", "extra_scores": {}}, {"review_id": "Yn5TnpSJoc", "reviewer": "Reviewer_25A8", "summary": "This paper introduces a synthetic dataset containing culture-specific concept-caption-image triplets and presents contrastive approaches inspired by NegCLIP and TripletCLIP to train a cultureCLIP using the proposed dataset. The dataset is designed to amplify similar yet distinct cultural concept pairs that current VLMs struggle with. They show that using hard images and caption/concepts negatives and LoRA, they can improve the baseline CLIP on culturally specific tasks without sacrificing general task performances.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I find the focus of cultural awareness in VLMs still a relatively new topic and deserves more attention in the field. I also appreciate the design of the dataset construction pipeline and the corresponding training strategy. These insights can be helpful for people who are interested in adapting VLMs for specific domains.", "reasons_to_reject": "**Reliability of the Proposed Dataset** I think certain descriptions in the paper are too high-level and missing specific details:\n- Where do the images come from (real images from Wikipedia or synthetically generated)?  How do you guarantee that the pair constructed is culturally appropriate? Please provide more details of MLLM-as-a-Judge and why Qwen2.5-VL is proficient in judging culturally related questions. For human eval, how many examples are annotated? Are the annotators qualified to annotate concepts across many countries? \n\n**Unclear Model results**\n- What is the CultureCLIP set up in Table 2? Some ablation results have even better performance compared to what’s reported in Table 2.  (E.g. Table 3 max performance on GlobalIRG-G is 69.76 compared to 69.47 in table 2) \n- One of the conclusions in ablation is filtered dataset enhances model performance. Could the authors please clarify what the ablation experiments were done for this section?\n- It seems that the culturally specific data tested are biased towards questions involving concepts. Could the author explain why datasets like CultureVerse or CVQA are not included as a benchmark? Or VLMs like CultureVLM and other closed-source VLMs are not tested?", "questions_to_authors": "- Why are NegCLIP and TripleCLIP not evaluated on GlobalIRG and CROPE datasets?\n- Why CLIP++ trained on more cultural data but the performance is significantly worse?\n- The metrics in Table 2 are not consistent between cultural-specific and agnostic tasks. It’s better to use percentages for both for consistency. \n- On line 144, “Finally, the back side is generated by the LLM, which matches visually similar concepts with the front side within the same cultural category, following the same format.” Could you clarify how you measure visual similar matches?", "extra_scores": {}}, {"review_id": "EB2sn0ntNj", "reviewer": "Reviewer_n9zq", "summary": "The paper introduces CultureCLIP, an enhanced version of the CLIP vision-language model (VLM), designed to improve fine-grained cultural concept differentiation while maintaining generalization capabilities.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Reasonable Motivation and Novelty**: The motivation behind the paper is well-articulated and addresses a critical gap in vision-language models (VLMs). By empowering CLIP to distinguish culturally distinct yet visually similar concepts through fine-grained visual cues, the work presents an interesting and valuable contribution to multimodal learning. The introduction of CultureCLIP, which enhances cultural awareness, aligns with the growing need for culturally sensitive AI systems.\n\n**High-Quality CulTwin Dataset**: The CulTwin dataset is a significant strength, constructed through a meticulous three-stage data curation pipeline. The use of synthetic concept-caption-image triplets, enriched with cultural context via vision-language models and text-to-image diffusion models, ensures high quality and diversity. This dataset not only supports the proposed method but also has the potential to advance multimodal alignment research in cultural contexts.\n\n**Comprehensive Experimentation**: The experimental setup is robust, featuring a variety of culture-specific and culture-agnostic benchmarks. The ablation studies systematically validate the effectiveness of the proposed contrastive learning framework.", "reasons_to_reject": "**Need for Concrete Examples in Cultural Taxonomies**: While the cultural taxonomy is well-defined across eight categories, the paper would benefit from providing specific examples for each category to enhance clarity. \n\n**Lack of Visualization Results**: The paper claims improved fine-grained cultural understanding, but it lacks visualizations to substantiate these claims. For example, showing matching scores between an image (e.g., Yuelao with a red thread) and captions describing similar cultural concepts would provide concrete evidence of CultureCLIP’s ability to distinguish subtle differences. Such visualizations would make the results more interpretable and compelling.\n\n**Limited Innovation Beyond CLIP Framework**: Although CultureCLIP introduces a novel dataset and training objective, the core methodology heavily relies on the existing CLIP architecture and contrastive learning paradigm. The current approach, while effective, risks being perceived as an incremental extension of CLIP.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CultureClip, a culturally aware CLIP that is fine-tuned on a synthetic cultural dataset with triplets that represent different cultural contexts. The reviewers highlighted several strengths of the paper, including the timeliness of the need for culturally aware models, the dataset construction methods and the strong experiments. The reviewers also had several issues that they highlighted, which were all addressed by the authors either by additional results or by promises to modify the paper. For example, the authors provided multiple satisfying clarifications, generalizations of their results to several other image classification benchmarks, visualizations, and committed to adding sections analyzing the errors of their model. The authors are asked to add all the promised changes to their final paper.
Decision: Accept