CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning
Authors: George Ibrahim
Rita Ramos
Yova Kementchedjhieva
Venue: COLM 2025 (2025)
Keywords: Image Captioning
Concepts
Retrieval
RAG
Multilingual
TL;DR: Image captioning with concept and captions retrieval augmented generation.

Abstract: Multilingual vision-language models have made significant strides in image captioning, yet they still lag behind their English counterparts due to limited multilingual training data and costly large-scale model parameterization. Retrieval-augmented generation (RAG) offers a promising alternative by conditioning caption generation on retrieved examples in the target language, reducing the need for extensive multilingual training. 
However, multilingual RAG captioning models often depend on retrieved captions translated from English, which can introduce mismatches and linguistic biases relative to the source language. We introduce CONCAP, a multilingual image captioning model that integrates retrieved captions with image-specific concepts, enhancing the contextualization of the input image and grounding the captioning process across different languages. 
Experiments on the XM3600 dataset indicate that CONCAP enables strong performance on low- and mid-resource languages, with highly reduced data requirements. Our findings highlight the effectiveness of concept-aware retrieval augmentation in bridging multilingual performance gaps.

Published as a conference paper at COLM 2025
CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning
George Ibrahim' Rita Ramos² Yova Kementchedjhiiva'
!Department of Natural Language Processing, MBZUAI
2INESC-ID, Instituto Superior Técnico, University of Lisbon {george. gbbiimm,,,, yova. wwwakkeeentcheddhiieaa/llaiaaai embzuui ac.
Abstract
Multilingual vision-language models have made significant strides in im- age captioning, yet they still lag behind their English counterparts due to limited multilingual training data and costly large-scale model param- eterization. -aurieeel- generation (RAG) offers a promising alternative by conditioning caption generation the target language, reducing the need for extensive multilingual training. However, multilingual RAG captioning models often depend on retrieved captions translated from English, which can introduce mismatches and linguistic biases relative to the source language. We introduce CONCAP a multilingual image captioning model that integrates retrieved captions with image-specific concepts, enhancing the contextualization of the input image and grounding the captioning process across different languages. Experiments on the XM3600 dataset indicate that CONCAP enables strong performance on low- and mid-resource languages, with highly reduced data requirements. Our findings highlight the effectiveness of concept- aware retrieval augmentation in bridging multilingual performance gaps.
retrieved examples in
1 Introduction
Vision Language Models (VLMs) have achieved increasingly strong performance in image captioning, driven by the scaling of model size and the use of extensive training data (Liu et al., 2023; 2024; Tschannen et al., 2025). Although VLMs demonstrate strong performance in image captioning, most remain English-centric, lacking the ability to generate high- quality captions for other languages. Growing efforts aim to build high-quality multilingual datasets for training and evaluation and develop multilingual VLMs on these (Geigle et al., 2024; Yue et al., 2025). However, even with massive, costly multilingual training and high parameterization to accommodate multiple languages, recent models continue to exhibit substantial performance gaps between English and other languages (Yue et al., 2025).
To alleviate the need for expensive multilingual training, recent work has explored retrieval- augmented generation (RAG) a promising direction (Ramos et al., 2023a; 2024). In this approach, models are conditioned not just on the image, but also on a set of retrieved captions in the target language. By leveraging these retrieved examples, the model requires far less multilingual training data, as it benefits from demonstrations of how to generate captions across different languages. However, key limitation of multilingual retrieval- augmented captioning models is that retrieved captions often come from translated datasets (Thapliyal et al., 2022) and datasets sourced from Western- et al., 2015), which may not fully capture the nuances of the target language and culture. Furthermore, the retrieved captions come from images similar to the input image, but almost certainly differ from it in some aspects, which inherently introduces noise through the retrieval augmentation.
image repositories (Chen
In this work, we introduce CONCAP, multilingual image captioning model that integrates retrieved captions and image-specific concepts. CONCAP builds upon the mBLIP architecture (Geigle et al., 2024), augmenting generation with retrieved information from captions of sim-

---

Published as a conference paper at COLM 2025
ilar images and concepts relevant to the input image. This enriches the captioning process with more informative and precise context and improves the quality of generated captions. Finetuning CONCAP on just 0.6M multilingual image-caption pairs yields considerably better results on low- and mid-resource languages in the XM3600 multilingual caption- ing benchmark (Thapliyal et al., 2022), compared to the original mBLIP model trained 4M samples. Despite its lightweight and data-efficient training setup, CONCAP achieves stronger multilingual capabilities on average, even compared to Pangea, state-of-thh-art multilingual VLM with 7B parameters trained on 6M data points (Yue et al., 2025). Through ablation studies of the caption and concept retrieval augmentations, we show that each component makes a valuable contribution, and the two complement each other.
2
Related work
2.1 Image Captioning
Image captioning has seen significant progress in recent years, driven by large-scale pretrain- ing and powerful vision-and-language models (VLMs) that leverage off-the-shelf unimodal components such as CLIP encoders and language decoders (Li et al., 2023; Alayrac et al., 2022; Mokady et al., 2021; Panagopoulou et al., 2023). Very recently, captioning performance has been further enhanced by the rise of large language models (L.Ms) (OpenAI, 2024; Touvron et al., 2023; Taori et al., 2023; Chiang et al., 2023) and visual instruction-turee (Liu et al., 2023; Laurençon et al., 2024). These models generally follow similar architectural trends. The unimodal vision and language components, typically S kept frozen, are connected either through a cross-attention mechanism (e.g., Flamingo; (Alayrac et al., 2022)), trainable Q-former networks (e-g., BLIP2; (Li et al., 2023)), fully autoregressive architecture, where the output of the vision encoder is directly mapped to sequence of text embeddings (e.g., LLaVA; (Liu et al., 2023)).
data
Despite progress in captioning quality, most models remain English-centric, while multilin- gual image captioning has been largely overlooked. Notable multilingual efforts include Thapliyal et al. (2022) that created XM3600, an established human-annotated captioning benchmark in 36 languages. Thapliyal et al. (2022) also proposed COCO-35 and CC3M-35L, translated versions of the standard COCO (Chen et al., 2015) and Conceptual Captions (Sharma et al., 2018) datasets in 35 languages, aimed at training multilingual captioning models. Using these datasets, they introduced BB+CC, the first large-scale multilingual image captioning model trained from scratch. Following this, mBLIP, based on BLIP2 (Li et al., 2023), introduced more efficient multilingual vision and language model, em- ploying the CLIP encoder (Radford et al., 2021) and multilingual LLMs. Despite having orders of magnitude fewer parameters, mBLIP is also trained machine-translate multilingual data. Very recently, Yue et al. (2025) developed Pangealns, culturally-aware instruction dataset with 6M samples across 39 languages, and Pangea, multilingual multimodal large-scale model designed to bridge language and cultural gaps. However, despite extensive multilingual training, these models still demonstrate a bias toward English captioning, with substantially lower performance for other languages. In contrast, we propose a method that effectively bridges the gap between languages and bypasses the need for massive training data by leveraging retrieval.
mixture of massive
2.2 Retrieval-augmenter Image Captioning
Retrieval-uugmented generation (RAG) methods provide additional context to the model by incorporating information retrieved from an external datastore (Lewis et al., 2020). RAG models have demonstrated improved performance not only across various NLP tasks (Fan et al., 2024; Huang& & Huang, 2024) but also in vision-anddlanguuage tasks (Yasunaga et al., 2022), such as image captioning Li et al. (2024b). Closely related to our work, Ramos et al. (2023b) proposes SmallCap, showing that augmenting an image captioning model with retrieved captions not only improves captioning performance but also reduces the number of trainable parameters and facilitates adaptation to out-of-domain settings. In contrast, Li et al. (2024a) introduces the EVCap model, which incorporates retrieved concepts (e-g., object
2

---

Published as a conference paper at COLM 2025
Similar Images shde:
* beautiful view of lake at the base of a sountaio, view of a lake and nountain en a sunry day,
Retrieved
3 cross country skter high ie the hills above lake,
a large body of water surrounded by mountate range.
image contain:
carpathian, hikee,
+ Preffx pivtto of
lagoon, trecking,
Caption English:
Text Encoder
Image Enoeder
Teit Eneodar
Retrieved
Concapts
Coapt)
Coepll)
Snap
Atarest Weiighor Captions
sunducches and some vegetables wiile plate topped with two sandhishes and salad plate two sanduiches and side
meal consisting
waterfall
hillside.
CONCAP: lake surrounded by
hearest Neighbor concepts
cardsiches,, hotdlegs, topoligs, liites, platter, serviing neal, casty, luech, colest
nountains
hillside.
Figure 1: (Left) CONCAP architecture combining visual features with retrieved captions and concepts via Q-Former and LLM. (Right) Multilingual image-text retrieval using cosine similarity with prefixed concept prompts and cross-lingual caption mapping.
names) instead of full captions avoid redundancy and mitigate misleading information in the retrieved text. Zeng et al. (2024) also explores augmenting image captioning with key concepts related to the image, but zero-shot setup.
While RAG has gained traction in image captioning, its use in multilingual image captioning has been limited, with only two models so far. Ramos al. (2023a) explored retrieval- augmented generation in training-free setup using in-context learning. Ramos et al. (2024) introduced PAELLA, a supervised multilingual retrieval-augmented image captioning model that is not only parameter-efffiient, like SmallCap, but also data-efficient through the incorporation of retrieval augmentation. It is important to note that both of these models rely the standard approach of using retrieved captions. However, concept-based augmentation, as seen in (Li et al., 2024a; Zeng et al., 2024), has yet to be explored in multilingual context. Moreover, unlike previous works, explore the integration of both retrieved captions and concept retrieval to enhance multilingual caption generation.
3 Proposed Approach
CONCAP is an efficient multilingual retrieeal-augmeeted image captioning model that com- bines retrieved captions and retrieved concepts to generate more accurate and contextually grounded captions across languages. An overview of the approach shown in Figure
3.1
Architecture
The method proposed here is applicable to any vision-language modeling architecture, as only modifies an input-specific prompt passed to the language decoder. In our experiments, we concretely adopt the mBLIP architecture (Geigle et al., 2024), one of the established state-of-the-art models for multilingual image captioning, while being parameter-efficient
3.2 Caption Retrieval
CONCAP builds on standard retrieval strategies (Ramos et al., 2023b; 2024), which use vision-language representation models (Radford et al., 2021; Zhai et al., 2023) to align images and text in a shared representation space. Given corpus of image captions in specific language, the text encoder CLIP-style model is used to pre-compute and store the caption embeddings in datastore. This indexed datastore is then queried using the
3

---

Published as a conference paper at COLM 2025
image encoder of the same model, with cosine similarity computed to find the most relevant captions based on the image embeddings.
CONCAP enhances caption generation by incorporating the top-1 most similar captions to the input image, guiding the language decoder towards generating captions in the target language with syntactic and semantic relevance to the intended prediction. However, we note that these retrieved captions come from similar images and may not fully capture the content of the input image, potentially introducing noise or mismatches in terms. For example, while a caption might correctly describe a bus in the street, it could mistakenly specify a "red bus," which may not be accurate for the input image.
3.3 Concept Retrieval
Concept retrieval follows similar process to caption retrieval, but instead of full sentences, concerns individual lexical items. To contextualize them and enhance their semantic representation, each concept is wrapped in short, language-specific template (e.g. "a photo of a dog"). We used GPT-4o to generate the templates (OpenAI, 2024), then verified them using Google Translate. This latter step was particularly important for morphologically rich and syntactically distinct languages, where direct translations may not always yield fluent or semantically accurate prompts. A complete list of language-specific templates is provided in Figure 9. While concepts are wrapped in template for retrieval, they are inserted into the final prompt in raw form (e.g., "dog").
Concept retrieval is performed analogously to caption retrieval, taking the top-m most relevant concepts per image. The intuition here is that augmenting the captioning process with highly relevant retrieved concepts will counteract noise from the retrieved captions and fill in possible gaps in the concept coverage of retrieved captions.
3.4 Prompt Format
The final prompt for the language decoder of CONCAP combines both retrieved captions and concepts to provide informative and semantically relevant context for caption generation. The prompt is organized into three segments:
1. Similar images show: caption_1, caption.2, ..., caption.n.
2. This image might contain: concept_1, concept.2, concept.m.
3. Caption in {lang]:
The first segment presents captions retrieved from visually similar images, providing sentence-level context. The second segment includes m concepts retrieved from language- specific wordlists. Together, these components guide the model toward producing accurate and semantically grounded captions in the target language. The language of the prompt is fixed English, and the full name of the target language used (e.g., "Caption in Chinese".)
During training, the language decoder receives the full prompt as input and generates caption conditioned on the image features V and the task prompt X. The model is optimized using teacher forcing, minimizing the cross-entropy loss over the reference caption tokens.
4 Experiments
4.1 Experimental Setup
Base Model CONCAP adopts the mBLIP architecture as well as its initialization strategy, i.e,, we carry out multilingual vision- bision-language training from scratch. The vision encoder, the Q-Former, and the projection layer are initialized from BLIP-2 (Li et al., 2023), while the language decoder is initialized from mTT0XL (Muennighoff et al., 2023). Following Geigle et al. (2024), we freeze the vision encoder and language decoder, and inject LoRA layers in the language decoder (Hu et al., 2021). Consequently, only the Q-Former, the projection layer, and LoRA layers are updated during training. This setup maintains training efficiency with approximately 111 million trainable parameters.
4

---

Published as a conference paper at COLM 2025
Training and Evaluation Data For training, we use the COCO-35L dataset (Thapliyal et al., 2022), a multilingual extension of the English COCO dataset (Chen et al., 2015), where each English caption is translated into 34 additional languages using Google Translate. The dataset is based on the Karpathy split, which includes 113,287 training images, each paired with five captions, resulting in total of 19.8M image-caption pairs across 35 languages. Following PAELLA (Ramos et al., 2024), we subsample the training split down to 566K image-caption pairs, ensuring equal representation across all languages.
We perform evaluation on the XM3600 dataset (Thapliyal et al., 2022), a human-annotated multilingual benchmark featuring captions in 36 languages. The dataset consists of 3,600 distinct images sourced from countries where the target languages are spoken, yielding total of 261,375 captions. On average, each image is annotated with two captions.
The Pangea project (Yue et al., 2025) introduced XM100, a subset of the XM3600 benchmark, designed enable faster and more focused multilingual evaluation while preserving diversity across languages and visual content. It consists of 100 selected images from the original set, each paired with its captions in all 36 languages.
Caption Retrieval Caption retrieval is implemented differently during training and evalu- ation to reflect the characteristics of each dataset. COCO-35L, used for training, contains images primarily sourced from English-speakinn tion, includes images from 36 geographically diverse countries. In both settings, we retrieve the top-4 captions per image, following prior work (Ramos et al., 2023b).
contexts, while XM3600, used for evalua-
For training on COCO-35L, we adopt an English-pivot retrieval strategy following prior work (Ramos et al., 2023b; 2024). We use CLIP (Radford et al., 2021) to retrieve captions from datastore populated with the English portion of COCO. Retrieved captions are then mapped into the target language using shared caption IDs. In $4.5, we also explore language-specific retrieval during training, which may be necessary scenarios where English-parallel captions are unavailable.
For evaluation, retrieval is performed using the mSigLIP model (Zhai et al., 2023) directly in the target language. Since XM3600 contains geographically and culturally diverse images and captions, we expect a multilingual retriever to work better.
Concept Retrieval We construct list of concepts for each language by tokenizing the COCO-35L training captions and extracting all unique tokens. The resulting lists contain tokens that are not necessarily semantically meaningful (e-g., subword tokens) or useful (e.g,, stopwords). However, curating a clean, high-quality concept list for each language is a non-trivial task, especially for low-resource languages where reliable tools for filtering or validation are limited or unavailable. Rather than relying on pre-filtering, we let the retrieval process handle relevance, trusting that the learned similarity space will naturally favor semantically meaningful and visually grounded tokens.
We maintain separate datastores per language and perform retrieval using mSigLIP. Follow- ing hyperparameter tuning over m = 4, 10, 20, we find that m = 10 yields best performance on the COCO-35L development set after 5 training epochs (see Table 5 in Appendix A.3), with results for m = 20 being very close.
In $5 we explore strategies for enrichment of the concept lists, and in $4.5 we compare different multilingual vision-language representation models. For efficient large-scale retrieval, we use the FAISS library (Douze et al., 2024), which performs fast nearest neighbor search over dense embeddings.
Tokenization When extracting wordlists and running evaluations, we apply language- specific tokenization strategies for languages that lack explicit word boundaries, such as Chinese, Japanese, Thai, and Hindi. These languages require specialized tokenizers to accurately segment text into meaningful units. For Chinese, we use the Jieba library (Junyi, 2012); for Thai, the Py ThaiNLP library (Phatthiyaphaibun et al., 2024); for Hindi, the Indic NLP Library (Kunchukuttan, 2020); and for Japanese, a tokenizer built on MeCab via the Fugashi wrapper (McCann, 2020; Kudo, 2006).
5

---

Published as a conference paper at COLM 2025
Models
Train
Total 0
Dataset
en
es
hi
zh
L
L6
mBLIP
111M
4.84B
2.71M
80.3
62.5
17.1
6.5
9.9
25.9
PAELLA
34M
3B
566K
57.3
44.9
20.8
25.9
20.7
26.9
BB + CC
0.8B
0.8B
135M
58.4
42.5
19.7
20.2
22.4
29.3
Pangea 7B
7B
6M
75.9
64.6
16.2
29.0
12.5
31.8
CONCAP
111M
4.84B
566K
72.4
58.6
24.4
21.7
18.2
34.2
Table 1: CIDEr scores on the XM3600 evaluation set, including results for four core languages (English, Spanish, Hindi, and Chinese), the average across five low-resource languages (Lş: Bengali, Mãori, Quechua, Swahili, and Telugu), and the overall average across all 36 languages (L36). We also report model size (Total f), the number of trainable parameters (Train f), and the size of the datasets used for training.
Training Details All models were trained with a learning rate of 3e and a weight decay of 0.1. We applied Low-Rank Adaptation (LoRA) with rank 8, scaling factor 16, and dropout rate 0.05, following the configuration used in mBLIP (Geigle et al., 2024) (see Table in the Appendix for more training details.) Training was conducted for 10 epochs with epoch-based checkpointing using 2x A100 40GB GPUs and an effective batch size of 256. We selected the best checkpoint based on CIDEr scores on the COCO-35L development split (see Table 4 in Appendix for results). For all models, the final epoch gave the best results.
Evaluation We evaluate model performance using the widely established CIDEr met- ric (Vedantam et al., 2015), which measures the similarity between generated and reference captions based on n-gram overlap, placing greater weight on n-grams that are frequently agreed upon across references. For inference, we use beam search with beam size of 5, length penalty of 1, and a maximum output length of tokens.
Baselines We compare our proposed method against several state-of-the-art multilinguul image captioning models. Since CONCAP builds on top of mBLIP (Geigle et al., 2024), begin by comparing with the original mBLIP. Next, we compare it with PAELLA (Ramos et al., 2024), multilingual retrieval-augmentte retrieved captions. PAELLA uses CLIP-ViT-B/33 32 encoder (Radford et al., 2021) and the multilingual XGLM (Lin et al., 2021) as decoder. We then compare with BB+CC (Thapliyal et al., 2022), fully-supervised multilingual captioning model that combines the mT55base (Xue et al., 2020) and ViT-B/ B16 models (Zhai et al., 2022), trained end-to-end and COCO-35L datasets. Finally, we compare against the recent Pangea (Yue et al., 2025), large multilingual model with 7B trainable parameters based on VLaVe--ext architecture and trained on Pangealns, a multilingual and culturally relevant multimodal dataset with million instruction examples.
model similar to ours, but with only
CC3M-35L
4.2 Main Results
Table presents the main results, comparing CONCAP to several strong baselines on the XM3600 benchmark. We report CIDEr scores on the four core languages (English, Spanish, Hindi, and Chinese), the low-resource set Lş (Bengali, Mãori, Quechua, Swahili, and Telugu), and the full set of 36 languages (Le6), all as defined in (Thapliyal et al., 2022). We present results using language codes, with their mappings listed in Table 10.
Averaged Performance (L36) We observe that, on average, CONCAP outperforms all baselines by a large margin, scoring 2.4 CIDEr points more than the next best model, Pangea. This is impressive given the large gap in trainable parameters (111M vs. 7B) and the amount of vision-language training data (566K vs. 6M data points). Comparing CONCAP to mBLIP, where both models share the same architecture and number of trainable parameters, observe a gap of 8.3 CIDEr points, despite an almost five-fold reduction in training data with CONCAP. The CONCAP approach enables highly data-efficient training.

---

Published as a conference paper at COLM 2025
Models
en
es
hi
zh
L5
L36
CONCAP
72.4
58.6
24.4
21.7
18.2
34.2
NoRAG
66.0
48.9
20.4
17.4
17.2
26.9
ConRAG
70.9
55.0
19.5
20.9
17.1
30.3
CapRAG
66.2
53.3
23.9
20.2
16.9
31.4
ConRAGRich
71.4
51.2
19.2
17.9
16.5
28.6
CapRAGM
38.3
30.4
16.4
13.9
13.1
20.4
Table 2: CIDEr scores XM3600 across CONCAP and various ablations: NoRAG (no retrieval), CapRAG (caption retrieval), ConRAG (concept retrieval only), and two augmen- tations: CapRAGM (target language retrieval), and ConRAGRich (extended concepts list).
Per-language Performance Looking at individual languages, we see more nuanced picture, where nearly every language ranks models differently. mBLIP appears to prioritize English, but is also strong Spanish. Pangea ranks best Spanish and Chinese but struggles on the five low-resource languages, where BB+CC, on the other hand, excels. CONCAP is best Hindi and not particularly weak on any of the languages considered here. We show in $4.6 that CONCAP provides better coverage of low- and medium--reoorre languages, whereas Pangea offers better coverage of high-resource languages.
4.3 The Contribution of Concepts and Captions
Having established the strong overall performance of CONCAP, now turn to analysis of the individual contribution of its components, looking at models trained without re- trieval augmentation (NoRAG), with retrieved captions (CapRAG), and retrieved concepts (ConRAG). Results are shown in Table 2.
The comparison to NoRAG isolates the strength of our proposed approach in otherwise identical experimental setup. We find that CONCAP 's impressive performance indeed attributed the retrieval augmentation, as its non-augmented counterpart performs on par with weaker baselines from Table 1.
Looking at the two forms of retrieval augmentation in isolation, find that each improves performance over NoRAG on average, with CapRAG being slightly more effective than ConRAG. While the gain from caption retrieval corroborates prior work (Ramos et al., 2024), it is interesting to see that concepts alone can also provide highly effective signal.
The most insightful finding here is that CONCAP considerably outperforms both ConRAG and CapRAG, indicating that the gains from these two forms of retrieval augmentation are additive: captions help guide generation with fluent language patterns, while concepts ensure broader content coverage and more accurate visual grounding.
4.4 The Role of English Pivot
Previous work has demonstrated that retrieving captions in English yields good results in multilingual settings, as shown in COCO-35L. However, this exploits a particularity of the COCO-35L dataset, while in practice, English translations might not be available. This motivates exploring retrieval strategies that work directly each target language.
We thus experiment with retrieval directly in the target language (using mSigLIP) at training time.! We carry out this experiment in the CapRAG setting and report results in Table 2 (CapRAGM). CapRAGM considerably underperforms CapRAG across all individual lan- guages and language groupings and results in worse performance even than NoRAG, iee, in this scenario, caption retrieval actively hurts performance. This indicates that the quality of the retriever is of utmost importance for the success of caption-based retrieval augmenta- tion in multilingual image captioning. In this sense, concept- retrieval augmentation
Recall that at inference time, retrieval is always done in the target language (see $4.1.)

---

Published as a conference paper at COLM 2025
100
120
13.554.
mCUP
CLIP Translated
mSigliip
mSigli
80
T6.
Jina
100
69.3
50
60
C0.
07.3
Score
Score
F2.0
55.88
23.7
25.
8n
L
hi
zh
Le
(a) Concepts Retrieval
(b) Captions Retrieval
Figure 2: Retrieval performance on the COCO development set across two settings: (a)
compares mCLIP, mSigLIP, and Jina retrieved concepts, (b) compares English CLIP retrieval against mSigLII' on retrieved captions.
(ConRAG) offers a more reliable alternative, as it does not depend on translation and aids performance regardless of possible shortcomings, i.e,, noise, in the retrieval process.
4.5
Best Retriever
To identify the most effective multilingual vision-language representation model for concept retrieval, we conducted a comparative study of three widely-used models: mSigLip, mCLIP, and Jina-CLIP (Zhai et al., 2023; Chen et al., 2023; Koukounas al., 2024). For faster iteration, we used a reduced training set of 113k samples, equally distributed across languages. We trained ConRAG model for 5 epochs, using retrieved concepts from each of the three retrievers, and evaluated performance on the COCO development split. As shown in Figure 2a, mSigLIP achieved the highest performance on average, and was therefore selected as the default multilingual retriever for all our experiments.
PANGEA
60
8m
es
in
H
nd
ja
no
ru de
sw
he
ar
t
di
da
fa
uk
m
bn
al
m
hr
hu
Figure 3: Per-language performance between CONCAP and Pangea on XM3600.
4.6 Per-language Performance
In Figure 3 we provide breakdown of performance per language, comparing CONCAP and the next-best model from Table 1, Pangea. The languages are arranged descending order based their Pangea scores. We observe that CONCAP consistently outperforms l'angea across many low- and mid-resource languages. Notable gains are observed in languages such Telugu, Hindi, Mãori, Filipino, Farsi, and others. Meanwhile, Pangea performs strongly on regh-resource languages such as English, Spanish, and Portuguese, reinforcing the divide between high- and low-resource languages even further.
8

---

Published as a conference paper at COLM 2025
9
CIDEr points
fil
mi
hu
do
pt
Figure 4: Performance difference (A CIDEr) the XM100 test set when changing the wordlists: COCO-5L+XM3300" (CX), CX+Pangea (CXP), CXP+Wikipedia (CXPW). The indicates that XM3600 has been filtered to exclude any words and images related to the test subset (XM100) to avoid contamination. Positive values indicate improvement over the COCO baseline; negative values indicate drop.
Concept Enrichment
The concept retrieval, in its base form discussed above, done using concept list built from the training data, which, as reminder, is translated from English and grounded in Western images. The natural next step is the integration of culturaall-reeresennta Enriched concept retrieval could result in better generalization and out-of-domain coverage. Below, we present early results in this direction showing that this is not a trivial task.
lexicons.
We enrich the default COCO-35L concept lists for each language with additional concepts sourced from the PangeaIns dataset (Yue et al., 2025). Specifically, we focus on the culfural portion of the data, aiming to accommodate the geographic diversity of XM3600 with relevant cultural concepts. The results in Table 2 (ConRAGRie) indicate a negative impact from concept enrichment: performance drops by 1.7 CIDEr points relative to ConRAG. This suggests that adding more terms from external datasets may introduce noisy, less relevant concepts, which dilute the retrieval quality.
To better understand this finding, we test ConRAG on XM100 with varying concept list configurations, focusing on sensitivity to datastore size and makeup. We compare: (1) CX, which adds filtered XM3600 lexicons (excluding the XM100 captions); (2) CXP, which includes Pangealns cultural terms; and (3) CXPW, which adds Wikipedia and Common Crawl entries for broader but less focused coverage. Per-language results are presented in Figure 4, as a change in CIDEr score from the base lexicon (COCO-35L) the different augmented lexicon configurations.
This addition of the filtered XM3600 lexicon (CX) shows a mixed effect, with roughly half of the languages seeing improvements and the other half experiencing a performance drop. Expanding this setup with a culturally-relevant lexicon (CXP) leads to a further decline in performance for larger portion of the languages. Finally, incorporating broad web-based lexicon (CXPW) results in the majority of languages showing a degradation in performance. While expanding the lexicon pool could, in theory, improve the coverage of retrieved concepts with geographically diverse and culturally relevant terms, in practice, it seems to add noise, which distracts the generation process and yields lower-quality captions.
ConRAG (COCO-35L)
.6 صشيية i سساارة ططر علل بحلس Jحل (Aman sitting the back of small car.)
ConRAG (Oracle)
ب الطروقة علي و لكك كر د بخود رحل (A man driving a tuk-tuk on the road.)
فحم مز بق طريق في صغير ة سيار يغرد Jحل busy street.)
PANGEA
(A man driving small
Figure 5: Oracle-augmenter ConRAG correctly identifies the tuk-tuk in the image.

---

Published as a conference paper at COLM 2025
Lastly, we conduct an oracle experiment on 200 images from the JEEM dataset for cross- cultural Arabic image captioning (Kadaoui et al., 2025). The subset consists of highly cultural images, manually selected by native Arabic speaker based solely visual content, and manually annotated with highly relevant cultural concepts. The ConRAG model, augmented with these oracle concepts, achieves CIDEr score of 17.9, compared to 13.9 when retrieving concepts from à general-purpose concept list (COCO-35L). A qualitative example of improved cultural awareness is shown in Figure 5. These gains can be observed only in the early stages of training (the results above correspond to one epoch of training). Continued finetuning seems to shift the decoder's ' output distribution, making unseen lexical items harder to predict, even when prompted.
6 Conclusion
We introduced CONCAP, a multilingual image captioning model that enhances caption generation by integrating retrieved captions with image-specific concepts. This approach improves caption quality while reducing the need for extensive multilingual training. Experiments on XM3600 show that CONCAP outperforms mBLIP and Pangea despite using fewer training resources. Ablation studies confirm the additive benefits of caption and concept retrieval. Further analysis informs the choice of retriever and highlights the brittleness of standard caption retrieval strategies. Concepts retrieved from enriched lexicons prove to be noisier and to hinder the accurate generation of image captions. Oracle experiments with cultural concepts show that this form of retrieval augmentation improves captioning initially, but the benefits fade as training progresses: the model increasingly depends its learned vocabulary, limiting integration of new cultural concepts. Future work should investigate more effective ways for concept enrichment and consider targeting datasets with more pronounced cultural representation, where well-informed concept enrichment may prove more effective.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a Visual Language Model for Few-Shot Learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Adoances in Neural In- formation Processing Systems, 2022. URL https: //penreview wwtevvvwww.eeeefereeeee mett/forumidd-EbMuimAPPPP
Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, and Wenping Wang. mCLIP: Multilingual clip via cross-lingual transfer. In Proceedings of ACL, Toronto, Canada, 2023.
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arxiv preprint xxx1:11504.,0355, 2015. URL https://arxxv..r http://wwwwwg////////////s///s// 00325.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open- source chatbot impressing GPT-4 with 9 ChatGPT quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library. 2024.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards auggeeedl- large language a models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024.
10

---

Published as a conference paper at COLM 2025
Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavaš. mBLIP: Efficient boot- strapping of multilingual vision-LLMs. In Jing Gu, Tsu-Jui (Ray) Fu, Drew Hud- son, Asli Celikyilmaz, and William Wang (eds.), Proceedings of the 3rd Workshop on Ad- vances in Language and Vision Research (ALVR), Pp. 7-25, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ v1//3////2244llv-1222 ÜRL orgp:///oooo.../22244// alvr-1.2/.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, 2021. URL https: http:///aaxxi..oora///11//// 09665
Yizheng Huang and Jimmy Huang. A survey on retrieval-uugmente text generation for large language models. arXiv preprint arXiv:2404.10981, 2024.
Sun Junyi. Jieba Chinese text segmentation. httpss//g..tuub..o///sss////eee 2012. Ac- cessed: 2025-03-27,
Karima Kadaoui, Hanin Atwany, Hamdan AI-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, and Yova Ke- mentchedjhieva. JEEM: Vision-Language Understanding in Four Arabic Dialects, 2025. 21910.
URL
Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Is- abelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan Wang, and Han Xiao. Jina CLIP: Your CLIP Model Is Also Your Text Retriever, 2024. URL https: htt///aix....gaa/see/////e
20204.
Taku Kudo. MeCab: Yet Another Part-of-Speech and Morphological Analyzer. https: //taku910. gi thub. io/mecab/, 2006. Accessed: 2025-03-27.
Anoop Kunchukuttan. The IndicNLP Library. https://gitte tttb. com/ com//anoookuuncukutttan/ pdf, 2020.
indic
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37: 87874-87900 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmeented Generation for Knowledge-Intensive NLP tasks. In Advances in Neural Information Processing Systems, 2020. URL https://prooeeddngg. neurips. wttps.ccppapppe htpppppper/2222
780e 1b266445d774111e5-aper htw.:/22290077001122226666777610 pdd.
Jiaxuan Li, Duc Minh Vo, Akihiro Sugimoto, and Hideki Nakayama. Evcap: Retrieval- augmented image captioning with external visual-name memory for open-world com- prehension. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13733-13742, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language- Image Pre-training with Frozen Image Encoders and Large Language Models, 2023. URL fitps:///axiin wtg//bs///////ee 12597.
Wenyan Li, Jiaang Li, Rita Ramos, Raphael Tang, and Desmond Elliott. Understand- ing retrieval robustness for retrieval-augmented image captioning. arXiv preprint arXio:2406.02225,, 2024b.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. Few-shot learning with multi- lingual language models. arXiv preprint drXiv:2112.10668, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. Advances in neural information processing systems, 36:34892-34916 2023.
11

---

Published as a conference paper at COLM 2025
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, January 2024. URL https://llawa-vl.. thub. iw/bllg///////////o (etp/lloog///-----2 01 - 30- llxva-next/.
Paul McCann. fugashi, a tool for tokenizing Japanese in python. In Eunjeong L. Park, Masato Hagiwara, Dmitrijs Milajevs, Nelson F. Liu, Geeticka Chauhan, and Liling Tan (eds.), Proceedings of Second Workshop for NLP Open Source Softwvare (NLP-OSS), pp. 44-51, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v/ 2020.nlposs-1.7. URL https:://tcccatttt
org.0020.....oooooo nlposs-1.2 1.7/.
Ron Mokady, Amir Hertz, and Amit H. Bermano. ClipCap: Clip prefix for image captioning, 2021. URL https: http:///aaxxv...gabb//s//sss 09741444.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xian- gru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual Generalization through Multitask Finetuning, 2023. URL
01786.
OpenAL. GPT-4o System Card, 2024. URL http://aarkxv.orr//b///11112211
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-InstructBLIIP: A framework for aligning x-modal instruction-aware representations to Ilms and emergent cross-modal reasoning. arXiv preprint ar xffo:23311.1879, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting on Association for Computational Linguistics, ACL '02, pp. 311-318, USA, 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1000 3115/1073883, 1073135.
URL https: //dpii org/10.
Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriya- wongkul, Lalita Lowphansirikul, and Pattarawat Chormai. PyThaiNLP: Thai natu- ral language processing in Python, June 2024. URL https://gi thub. com/PyThaiNLP/ pythainlp/.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, 2021. URL https::tp/// //attp: org/abs////// a2120 00020.
Rita Ramos, Bruno Martins, and Desmond Elliott. LMCap: Few-shot multilingual im- age captioning by retrieval augmented language model prompting. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Com- putational Linguistics: ACI. 2023, pp. 1635-1651, Toronto, Canada, July 2023a. Associ- URL
ation for Computational Linguistics. doi: org//023. findings-acl. 104/.
nttps://aclantholl/
Rita Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. SmallCap:
Lightweight image captioning prompted with retrieval augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023b.
Rita Ramos, Emanuele Bugliarello, Bruno Martins, and Desmond Elliott. PAELLA: Parameter-efficient lightweight language-agnostic captioning model. In Findings of the Association for Computational Linguistics: NAACL 2024, June 2024.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captioon
A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2018.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Stanford Alpaca: An instruction-following LLLMA model, 2023.
12

---

Published as a conference paper at COLM 2025
Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Com-
putational Linguistics, 2022. doi: 10.18653/v /2022...mnlpmmainn5555. URL https:
TTaclantholoog.org/222202 www.:ogggg.eee..eo wttp://rr//222.eemmmmaaaaaa emn1p-main.45/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigL.PP 2: Multi- lingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, 2025, URL http:://aaaxi...rg/aaa//////////
14786.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based Image Description Evaluation, 2015. URL
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m massively multilingual pre-trained text-to-text transformer. Xiv preprint ar arii:20100.19344 2020.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented multimodal language modeling. arviv preprint ar (rXi:22211.12611, 2022.
Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. Pangea: A Fully Open Multilingual Multimodal LLM for Languages, 2025. URL https: wwwpvvv..rrgbb//411... 16153.
Zequn Zeng, Yan Xie, Hao Zhang, Chiyu Chen, Bo Chen, and Zhengjue Wang. MeaCap: Memory-augmenter zero-shot image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 14100-14110, 2024.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision trans- formers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 12104-12113, 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Tra-rrainng, 2023. URL https:
15343.
13

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Training Configuration and Checkpoint Selection
The training configuration detailed in Table 3 indicates that we consistently used a batch size of 256 for all experiments. Our models were trained using A100s 40G GPUs, with the CONCAP taking approximately 66 hours to complete 10 epochs.
Model
Batch Size
Grad Accum
Seq Len
Eff. Batch
Time (hrs)
Normal
64
8
256
38
ConRAG
32
56
256
45
CapRAG
16
111
256
58
CONCAP 16
155
256
66
Table 3:
Training configurations for different models. All experiments used 2x A100 40GB GPUs and ran for 10 epochs with checkpointing.
Table 4 presents the CIDEr scores for each checkpoint on the COCO-35L development set. It evident that the final checkpoint achieved the highest score for all models.
Model
2
3
4
5
6
7
10
NoRAG
66.3
78.6
85.1
89.8
92.9
95.9
97.3
98.2
98.8
99.2
CapRAG
78.6
89.6
92.8
94.1
95.3
97.0
99.1
100.3
101.2
102.3
ConRAG
66.3
76.2
81.4
83.7
85.1
88.7
92.0
92.8
93.4
95.1
CONCAP
79.3
92.7
97.5
99.9
101.2
102.7
105.5
107.3
108.2
109.4
Table 4: Average CIDEr scores on the COCO-35L dev split across four core languages for each checkpoint.
A.2 Prompts For Concept Retrieval
In Table 9, we present the multilingual prompts used for concept retrieval. Initially, we translated the prefix from English other languages using GPTT-4o (OpenAI, 2024). These translations were later verified with Google Translate to ensure semantic consistency and linguistic accuracy. For Chinese and Japanese, an additional suffix was required to add nuance to the sentence and preserve natural phrasing.
A.3 Number of Concepts
We conducted an ablation study on the COCO development set to determine the ideal number of retrieved concepts. ConRAG was trained on 20% of the data, and evaluated different numbers of shots on the COCO validation split. As shown in Table 5, performance for 10 and 20 retrieved concepts is virtually identical, but augmenting generation with more retrieved concepts increases the inference cost. Accordingly, we chose to use m = 10.
en
hi
zh
L
90.4
69.1
26.6
47.9
58.5
10
92.4
74.6
26.2
49.2
60.6
20
92.5
77.1
24.2
47.0
60.2
Table 5: Captioning performance by language across 10, and 20 retrieved concepts.
14

---

Published as a conference paper at COLM 2025
A.4
Memory Usage
Table 9 reports the COCO concept list length for each language, with an average of ap- proximately 43,940 words. Given that MS-COCO contains 566,000 captions, the memory footprint of the concepts datastore is less than 10 percent of that of the captions datastore. A visual comparison of the caption and concept datastore sizes across languages is shown in Figure 6. Since concepts and captions be retrieved in parallel, with the newly added concept datastore being smaller and thus faster to access, this addition introduces no latency to the retrieval process.
Captions Store (MBs) Concepts (MBs)
33
200
130
20
11
2L3
20
00 11
8
mi
no
59
MS
D
Figure 6: Comparison between the file sizes of the caption and the concepts datastores per language.
A.5 Per-Language Performance
Figure details per-language performance for CONCAP, ConRAG, and CapRAG. We see that CONCAP consistently outperforms the other two variants on all languages except for Maori (mi), which seems to benefit more from retrieved concepts than captions.
CapRAG
ConRAG
CIDE Score
30
20
10
65
pt.
no
tr
do-
vi Il
he- pl
al
Ja de
ru ar-
SW
hn
hi
nu
mi
th
an
to
ko
quz
Figure 7: Per-Language performance between CONCAP, ConRAG, and CapRAG.
A.6 Concept List Sizes and Performance Scores
Table 9 shows the sizes of the concept lists for each language across different dataset combinations. We began with the COCO-35L words, then filtered the XM3600 to exclude the image-caption pairs present in XM100 to prevent leakage. After that, we included the remaining words. Next, incorporated the Pangealns cultural subset to add more words not found the COCO-35L translated data. Finally, we appended words from the Fasttext library, which was trained on Wikipedia and Common Crawl.
15

---

Published as a conference paper at COLM 2025
A.7 Results on All Languages
Table 10 presents the CIDEr scores across all languages, while Table 11 shows the BLEU scores (Papineni et al., 2002), both evaluated the XM3600 dataset. We compare the performance of all our approaches against the state-of-the-art model, Pangea (Yue et al., 2025). L refers to the core languages English (en), Spanish (es), Chinese (zh), and Hindi (hi), as defined in prior work (Thapliyal et al., 2022).
A.8 Linguistic Prior
Retrieval-ugmmented generation (RAG) introduces a linguistic prior that, in our context, functions a complementary signal rather than a substitute for visual grounding. To assess this, we include two key baselines in our experiments: NoRAG, which leverages only visual inputs, and CONCAP, which integrates both image and retrieved textual information. Additionally, we conduct Text-only ablation in which the original image is replaced with a solid-color version computed from its mean RGB value, effectively removing all visual content while preserving layout and input structure. This ablation is designed to isolate the model's dependence on the retrieved text. Preliminary results, summarized in Table 6, indicate that CONCAP consistently outperforms both NoRAG and Text-only across the four primary languages. These findings suggest that the model benefits from a combination of modalities and does not exhibit overreliance the retrieved text alone.
Model
English
Spanish
Chinese
Hindi
L
NoRAG
66.0
48,9
17.4
20.4
38.2
Text-only
29.1
9.6
4.11
10.8
13.4
CONCAP
72.4
58.6
21.7
24.4
44.3
Table 6: Comparison between Image-only (NoRAG), Text-only, and CONCAP across the four core languages
16

---

Published as a conference paper at COLM 2025
Lang
COCO
+XM3600
+Pangea
+Wiki
en
27456
27878
27878
204316
28187
29193
101099
223659
zh 34971
37622
153658
270470
hi 13502
13891
17546
24723
ar 52423
55558
138879
235593
fr 28630
30179
100639
226474
de
76307
81124
257457
354984
67761
70790
249805
331528
ja
18375
20109
71197
71197
pt 27150
28635
104681
225965
27150
31624
118497
234006
nl 55286
57027
83447
241859
tr 49688
51533
201970
294533
68342
70449
70449
244036
no 68342
68561
275367
391581
da 64403
66429
66429
239378
fi 116243
120918
120918
282930
pl
65313
67677
121040
243183
cs
64368
65982
104444
234808
uk 65293
69289
116724
245061
ro 65293
39233
140965
254239
el
40270
43154
83401
227733
he 50846
53321
101299
212993
bn 23825
24431
27381
32391
fa 17555
19612
29802
170328
38794
39586
40964
193375
sw
34741
37141
122410
276114
vi
8597
8943
39866
214706
th 6414
8114
22497
40005
id
18734
19921
88607
229330
ko 21221
30353
346315
346315
hu 90319
94465
94465
257909
hr 56070
58227
58227
221033
fil 32680
34503
34503
211716
mi
13352
14799
14799
14799
Table 9: Size of the concept list for each language across different dataset combinations.
17

---

Published as a conference paper at COLM 2025
Language
Prompt Format
ar
صررف <concept>
bn
C 2 प <concept>
Cs
fotografie <concept> et foto af en <concept> ein Foto von ein <concept>
da
de
el
iIl ςooooyp<pia aTÓ <concept> a photo of <concept> una foto de una <concept>
en
es
fa
وك jز عكك <concept> valokuva <concept>
fi
fil
isang larawan ng <concept>
fr
une photo de <concept>
he
7 minn <concept>
hi
Uक तरबार की <concept>
hr
fotografija od <concept>
hu
egy fotó a <concept>
id
sebuah foto dari <concept>
it
una foto di <concept>
ja
<conceect
ko
인 O A <concept>
mi
he whakaahua o te <concept> een foto van een <concept> et bilde av en <concept>
nl
no
pl
zdjęcie <concept> uma foto de <concept>
pt
quz
huk fotota a <concept>
ro
o fotografic a unui <concept>
ru
ςоотoрраςии <concept>
Sv
ett foto på en <concept>
sw
picha ya <concept> 2 @7 2J <concept>
te
th
HUnnuua <concept> bir fotografi <concept> dororpaais a <concept> môt búrc ånh cua môt <concept>
tr
uk
vi
zh
*<concept>($)j
Table 9: Language-speciffcc prompt templates used for concept retrieval. Each row shows the prefix and, where applicable, the suffix in Chinese and Japanese used to wrap a concept token in grammatically correct and semantically natural form for that language.
18

---

Published as a conference paper at COLM 2025
Models
en
es
zh
hi
fa
fi
ko
te
fil
mi
mBLIP
80.3
62.5
17.1
6.5
4.2
16.9
11.1
11.9
19.2
9.5
Pangea
75.9
64.6
16.2
29.0
23.0
3.5
17.5
11.4
8.2
0.2
NoRAG
66.0
48.9
20.4
17.4
28.9
16.6
8.7
15.0
25.1
30.8
ConRAG
70.9
55.0
19.5
20.9
36.3
20.3
11.8
14.9
28.0
28.6
ConRAGRich
71.4
51.2
19.2
17.9
32.1
19.1
11.2
14.4
25.7
29.5
CapRAG
66.0
50.6
23.2
19.8
35.8
19.2
11.9
22.8
37.3
44.8
CapRAGM
66.2
53.3
23.9
20.2
38.2
20.0
13.3
16.1
33.1
21.3
CONCAP
72.4
58.6
24.4
21.7
40.7
22.2
13.7
17.7
35.4
23.0
Models
hu
id
hr
fr
vi
da
mBLIP
21.5
38.7
4.9
58.4
48.7
14.3
46.3
35.8
4.3
Pangea
4.4
55.8
8.2
66.9
19.0
36.6
46.8
64.3
24.7
NoRAG
18.6
33.8
13.1
36.7
42.6
22.3
43.1
31.8
37.7
ConRAG
21.2
37.7
17.0
39.9
49.3
24.9
47.6
36.8
42.8
ConRAG Rich
21.7
34.5
15.0
37.6
47.7
21.6
45.9
32.2
41.5
CapRAG
20.8
36.9
22.4
38.1
44.4
26.5
47.0
42.8
41.4
CapRAGM
22.0
38.8
23.4
40.2
47.9
18.7
27.2
32.9
25.6
CONCAP
24.3
41.1
25.7
45.9
53.1
28.3
51.3
45.1
45.6
Models
ja
nl
he
th
it
uk
de
pt
tr
mBLIP
0.2
55.3
18.5
0.3
26.6
45.8
20.3
31.6
54.1
24.0
Pangea
47.0
51.4
35.7
60.9
38.4
51.9
19.1
37.4
62.3
29.5
NoRAG
23.6
47.4
18.9
34.0
22.3
31.5
18.5
24.8
45.8
18.3
ConRAG
27.3
52.6
26.5
35.1
24.8
39.1
21.5
26.6
46.8
21.8
ConRAGRioh
23.2
50.7
23.8
33.0
23.3
34.5
19.9
27.8
45.1
21.4
CapRAG
28.2
51.7
29.1
36.6
25.7
40.6
21.3
27.1
48.0
25.8
CapRAGM
21.3
28.6
18.8
25.3
16.9
27.3
14.9
17.6
29.7
17.1
CONCAP
30.5
55.3
32.4
38.9
29.4
43.5
24.4
29.6
52.0
27.7
Models
CS
pl
bn
ar
ro
el
quz
L44
L
L36
mBLIP
29.6
30.9
12.9
21.3
22.6
23.8
1.1
41.6
9.9
25.9
Pangea
23.1
25.9
14.2
31.2
31.5
9.7
0.0
46.4
12.5
31.8
NoRAG
28.0
22.0
16.6
17.9
17.3
21.8
1.1
38.2
17.2
26.9
ConRAG
31.7
26.2
15.9
23.3
20.7
25.5
1.1
41.6
17.1
30.3
ConRAGRich
29.2
25.4
16.0
22.0
19.5
23.0
1.1
39.9
16.5
28.6
CapRAG
34.5
27.1
19.7
27.0
23.5
26.9
1.1
40.9
16.9
31.4
CapRAGM
19.9
17.3
15.4
17.0
16.3
16.8
1.1
24.8
13.1
20.4
CONCAP 38.7
31.4
20.8
29.2
25.5
30.5
44.3
18.2
34.2
Language codes: en=English, es=Spanish, zh=Chinese, hi=Hindi, fa=Farsi, fi=Finnish, ko=Korean, te=lelugu, fil=Filipino, mi=Maori, hu=Hungarian, id=Indonesian, hr=Croatian, fr=French, sv=Swedish, sw=Swahill, no=Norwegian, vi= vi=Vietnamese, da=Danish, ja=Japanese, nl=Dutch, he= th=Thai, ru=Russian, it=ltalian, uk=Ukrainian, de=German, pt=Portuguese, tr=Turkish, cs=Czech, pl=Polish, bn=Bengali, Arabie,,,, ro= el=Greek, quz=Quechua.
Table CIDEr Scores: XM3600 Evaluation on 36 languages.
19

---

Published as a conference paper at COLM 2025
Models
en
es
zh
hi
fa
fi
ko
te
fil
mi
NoRAG
11.7
8.0
2.1
2.8
4.7
1.2
0.4
0.7
7.1
8.5
ConRAG
11.7
8.5
2.1
2.5
5.0
1.2
0.4
0.7
6.8
8.2
ConRAGRial
11.9
8.0
1.7
2.4
4.7
1.2
0.4
0.6
6.6
8.4
CapRAG
11.0
8.5
1.9
2.6
5.2
1.3
0.6
0.5
7.8
7.6
CapRAGM
5.5
4.4
1.0
1.6
3.3
0.6
0.3
0.0
4.7
5.8
CONCAP
11.9
9.5
2.0
2.9
5.9
1.4
0
0.5
8.4
8.1
Models
hu
id
hr
fr
SW
no
vi
da
NoRAG
1.9
5.1
1.2
7.1
6.7
2.6
8.1
9.5
6.7
ConRAG
2.0
5.1
1.2
7.1
7.2
2.7
8.3
9.7
6.7
ConRAGRial
2.2
4.6
1.2
6.6
6.8
2.5
8.0
9.3
6.4
CapRAG
2.0
5.0
2.4
7.4
7.1
3.0
7.6
10.4
6.3
CapRAGM
1.0
3.2
1.4
4.9
3.4
1.9
4.0
7.9
3.7
CONCAP
2.5
5.5
2.8
8.6
7.9
3.2
8.7
11.0
7.3
Models
ja
nl
he
th
it
uk
de
pt
tr
NoRAG
7.2
8.8
2.1
6.6
2.5
5.6
2.3
3,9
7.9
2.1
ConRAG
7.8
8.9
2.4
7.1
2.5
6.3
2.4
3.4
7.6
1.9
ConRAGRick
6.7
8.7
2.1
6.9
2.2
5.7
2.2
3.8
7.2
1.9
CapRAG
7.8
9.0
3.0
6.3
2.7
6.7
2.2
3.8
8.2
2.3
CapRAGM
5.2
4.1
1.7
3.0
1.6
4.1
1.4
2.4
4.4
1.4
CONCAP
10.6
9.3
3.3
6.8
3.4
7.3
2.9
4.3
8.9
2.5
Models
pl
bn
ar
ro
el
quz
L4
Ls
La6
NoRAG
3.1
2.7
0.6
1.6
4.3
1.8
0.0
6.2
2.5
4.4
ConRAG
2.8
3.0
0.5
2.0
4.5
1.7
0.0
6.2
4.5
2.4
ConRAGRid Rich
2.5
2.9
0.5
1.8
4.3
1.5
0.0
6.0
4.3
2.4
CapRAG
3.2
3.2
0.5
2.5
4.8
1.8
0.0
6.0
2.3
4.6
CapRAGM
1.5
1.7
0.3
1.3
2.7
1.0
0.0
3.1
1.6
2.7
CONCAP
4.0
3.8
0.5
2.6
5.2
2.1
0.0
6.6
2.5
5.1
Language codes: en=English, es=Spanish, zh=Chinese, hi=Hindi, fa=Farsi, fi=Finnish, ko=Korean, te= TelTe,,gu, fil-Filipino, mi=Maori, hu=Hungarian, id=Indonesian, hr=Croatian, fr=French, sv=Swedish, sw=Swahili, no=Norwegian, vi=Vietnamese, da=Danish, ja=Japanese, nl=Dutch, Hebrew,w, th=Thai, ru=Russian, it=Italian, uk=Ukrainian, de=German, pt=Portuguese, tr=Turkish, cs=Czech, pl=Polish, bn=Bengali, Ar=bra, ro=Romanian, el=Greek, quz=Quechua.
Table 11: BLEU scores: XM3600 Evaluation on 36 languages.
20


Reviews:
[{"review_id": "UZvZiQUZTZ", "reviewer": "Reviewer_5aBu", "summary": "The paper proposes a multilingual image-captioning framework that augments an mBLIP backbone with two retrieval channels: (i) top-k captions from visually similar images and (ii) top-m lexical “concepts” retrieved from language-specific word lists. Both signals are concatenated into a prompt fed to the frozen mT0-XL decoder fine-tuned via LoRA. Training uses only 566k machine-translated COCO-35L pairs, yet CONCAP attains the highest average CIDEr on the 36-language XM3600 benchmark, surpassing the 7B parameter model while keeping few number of trainable weights.\n\n-----------------After rebuttal------------------\nApologies for replying late while in the hospital - thanks for the rebuttal, which addresses my concerns. I have decided to increase my score.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clear.\n2. The ablation is comprehensive. \n3. The method achieves desirable performance with few number of trainable weights.", "reasons_to_reject": "1. The evaluation is limited.\n- Only the image captioning experiment on a single dataset XM3600 is provided. No retrieval, VQA or cross-modal generation tasks are provided.\n\n2. English-pivot retrieval unrealistic: CapRAG’s strong gains hinge on English IDs unavailable in most real-world corpora; per-language retrieval (CapRAG-M) collapses. \n\n3. Inference cost unreported: retrieving 4 captions + 10 concepts per image from 35 language-specific FAISS indices introduces latency and large memory footprints. \n\n4. Several important analyses are missing.\n- How sensitive is the model to the number of retrieved captions or concepts (e.g., what happens when n or m is increased or reduced dramatically)?\n-  Could multilingual caption retrieval performance be improved by combining embeddings from multiple retrievers (e.g., ensemble of mSigLIP and mCLIP)?\n- How does CONCAP perform in true zero-resource settings (i.e., no training data for a given language, only retrieval)?", "questions_to_authors": "See weaknesses.", "extra_scores": {}}, {"review_id": "2jEw6pUbVi", "reviewer": "Reviewer_umB5", "summary": "The paper introduces a multilingual image captioning model called CONCAP. Prior approaches faced a couple of issues including lack of large scale data and bias in translated captions. CONCAP improves the captioning for non-English languages by using retrieved captioned and image-specific concepts to augment caption generation. This approach outperforms state-of-the-art models on XM3600 dataset despite requiring much less training data. The performance is especially good on low and mid resource langauges.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Substantial improvement on XM3600 while being highly data efficient.\n- Novel approach for contextualization and grounding of captions using image-specific concepts helps address gaps in multilingual captioning performance.", "reasons_to_reject": "- CONCAP fails to perform better than other approaches on certain high resource languages like en, es etc. The positioning of the paper could potentially be different to focus on mid and low resource languages since the approach is more effective for that setting.\n- Retrieval is dependent on having a good English pivot. This could be a limitation if high quality translations are not available in certain scenarios.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "92BJApzmPb", "reviewer": "Reviewer_HrUv", "summary": "The paper proposes to improve the performance of multilingual image captioning by conditioning the generation on retrieved captions as well as retrieved image concepts (single words), and applies this idea to the mBLIP architecture.\nThe comparison with several baselines show improved performance on low and medium resourced languages. Ablating either caption or concept retrieval components shows that they both contribute to the preformance of the modified mBLIP.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed methods is very simple while at the same time seems to improve captioning performance on the majority of languages.", "reasons_to_reject": "The contibution is relatively minor modification of existing methods. The approach relies on retrieving the captions and concepts associated with related images, and thus may lead the model to rely more on generic (linguistic) priors rather than the specific content of the image itself. Given this risk, I would have liked an evaluation which focuses on ensuring that the model uses the visual input and does not \"hallucinate\".\n---\nUpdated: The authors have addressed the concern about \"hallucination\" to some extent, and thus I'm upgrading my rating.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ovmotlH7O", "reviewer": "Reviewer_H9Ar", "summary": "The paper presents CONCAP, a parameter-efficient* architecture for generative multilingual image captioning based on retrieval-augmented generation, with retrieval of captions over a datastore of images and of concepts over a datastore of terms. The authors instantiate their architecture over an mBLIP image captioning model, training it over the dataset COCO-35, and evaluating it on XM3600 and the subset XM100. The presented model achieves good performance overall, with remarkable improvement on low-resource languages compared to the base mBLIP model, while maintaining good performance in other languages, surpassing the overall performance of other multilingual models, such as PANGEA and PAELLA. The authors present substantial evaluations, along with clarifying ablation studies investigating different methods for retrieval and their impacts on the performance.\n\n\nIt is important to notice that the model is in some sense over-reliant on the English language, as shown in their results, in the sense that, by training on a dataset constituted of English captions translated into other languages, linguistic and cultural biases in descriptions may be learned by the model, which may be reproduced in the generated captions during evaluation. In fact, it is not completely clear if a multilingual and multicultural image-language dataset can be consistent, in the terms proposed by Thapliyal et al in the proposal of XM3600, since, as discussed for example by Liu et al. [1], people from different cultures may evoke different concepts to describe scenes (e.g. a bowl of fruits can be seen as 'a bowl of exotic fruits' or  'an offering' depending on cultural and religious background). As such, by training on COCO-35 and evaluating on XM3600 the authors may underrepresent the performance of their model. In fact, we can see some variation in the description of the images in different languages on XM3600. It would be an interesting exercise to train the model on part of XM3600 and test on XM100, for example. \n\nThere are some typos in the text and duplicate entries in the bibliography that should be revised (listed below):\n - several citations in the text should not be direct citations Author (year) =, but indirect citations (author, year), as the authors are not directly referencing the work, only talking about it (page 4, base model -Hu et al;  page 6, baselines - several)\n - section 4.2: should it be L_{36} instead of L_{35}?\n - what is L_4 in Tables 7 and 8? It has not been introduced\n - Ramos et al 2023 (LMCap) appears twice - keep the peer-reviewed version\n - Liu et al 2023 appears twice - keep the peer-reviewed version\n\n\n* In the sense that only a part of the architecture is trained end-to-end, thus only a fraction of the overall parameters are trainable, as it takes advantage of pretrained image-encoders and language-decoders.\n\n[1] LIU, Fangyu et al. Visually Grounded Reasoning across Languages and Cultures. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. p. 10467-10485.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A parameter-efficient architecture that is easily adaptable for new and improved image encoders and language decoders, trained and evaluated on a multitude of typologically diverse languages with competing results;\n\n- The work investigates the use of concept-based augmentation in image captioning, which to my knowledge, is new.", "reasons_to_reject": "- While the work is interesting and novel in the sense of incorporating concept augmented retrieval to the problem, conceptually it can be seen as an incremental contribution. If we compare it to PAELLA - a much smaller model - it is not clear if the increased performance is due to the concept augmentation of simply as a function or the increased number of trainable parameters in the  model.", "questions_to_authors": "- Why the two different dataset chosen for training and evalutation, with training over a translation-based dataset? Did the authors consider using two manually annotated and culturally diverse datasets, such as XM3600 and PageaIns for training/evaluation? How are the results expected to vary in this scenario?\n\n- Why in the investigation of the use of English as pivot, the authors compare the results of the model with CLIP (CapRAG) and with mSigLIP (CapRAG_M), instead of using msiglip in both cases? How can we compare both results and achieve any meaningful conclusion on the role of English as pivot in the experiment if the encoder has changed?\n\n- Since the size of the concept list seems to have no direct correlation with the improvement in performance of the model, how can we evaluate the contribution of the concept lists? Why all unique tokens are considered valid concepts, when most tokens cannot be directly related to concepts? From the concept list, in the evaluation experiments, which concepts have been more often retrived and how do they relate to the images (this evaluation could be feasibly be done for the XM100 dataset)? this distribuition has changed considerably in the experiments with different concept lists? how? It would be interesting for future work to validate or order concept candidates by importance - e.g. through notions such as lexical salience [1] (often related to surprisal or predictability).\n\n\n[1] Erkan, G., & Radev, D. R. (2004). Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22, 457-479.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to augment the context of multilingual image captioning models with relevant retrieved image concepts in addition to retrieved captions. By doing so, the authors argue that it can counteract noise and improve the quality of generated captions. The proposed technique has been empirically validated through extensive experiments and ablation studies.

## Identified weaknesses
- The contribution of the paper is relatively incremental but reviewers acknowledge that while the approach is simple, it empirically shows that it is effective.
- The technique might over-rely on linguistic priors and ignore visual features, which could be a limitation in some cases. The authors addressed that issue by running additional ablation experiments which seem to indicate the models still rely on visual features to work.

## Positive aspects

- This work has clear motivation, provides thorough empirical evidence, and is well-structured.
- The authors show that their approach can surpass the SOTA model while training only a fraction of the parameters (111M vs. 7B) and with 10x fewer training data.
- This work identifies the dependence on English pivots in prior methods as a key limitation, particularly when high-quality translations or English identifiers are unavailable. It demonstrates that concept-level retrieval in the target language remains effective without English pivots, offering a more robust solution for multilingual contexts.

## Recommendation
Overall, the paper is a solid contribution to the field of multilingual image captioning. While the contribution may not be groundbreaking, it provides valuable insights and empirical evidence that can benefit both application and future research. The paper is recommended for acceptance, with the understanding that it is an incremental but meaningful advancement in the area of retrieval-augmented captioning. The authors have also made efforts to address reviewer concerns through additional experiments and clarifications.
Decision: Accept