Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning (2024)

Abstract: Recent advances in retrieval-augmented models for image captioning highlight the benefit of retrieving related captions for efficient, lightweight models with strong domain-transfer capabilities. While these models demonstrate the success of retrieval augmentation, retrieval models are still far from perfect in practice: the retrieved information can sometimes mislead the model, resulting in incorrect generation and worse performance. In this paper, we analyze the robustness of a retrieval-augmented captioning model SMALLCAP. Our analysis shows that the model is sensitive to tokens that appear in the majority of the retrieved captions, and the input attribution shows that those tokens are likely copied into the generated output. Given these findings, we propose to train the model by sampling retrieved captions from more diverse sets. This decreases the chance that the model learns to copy majority tokens, and improves both in-domain and cross-domain performance.

heading: 1 Introduction
text: Recent retrieval-augmented image captioning models have shown success in strong image captioning performance while reducing model parameters by retrieving related captions for a given image (Ramos et al., 2023b; Sarto et al., 2022; Yang et al., 2023). These models use retrieved information as additional context besides the input image. However, similar to retrieval-augmented language models (Yoran et al., 2023), image captioning models enhanced with retrieval can sometimes be misled by irrelevant information. For example, in Figure 1 the captioning model is misled by the token “elephant” in the retrieved captions, and generates captions that do not match the given image.
For retrieval-augmented language models, Yoran et al. (2023) have studied the cases where retrieval misled the model prediction, and address this problem with a retrieval-robust LLM by continuous
training with synthetic data for question answering tasks. However, in their approach, the retrieval system returns only one passage at each step. Considering that LLMs can be sensitive to the order of prompts (Lu et al., 2022), the robustness of using multiple retrieved results has not been fully studied. Evaluating and improving the robustness of retrieval-augmented image captioning models remains under-explored, specifically when the model is augmented with multiple retrieved results.
To bridge this gap in the literature, we study the robustness of the SMALLCAP retrieval-augmented captioning model (Ramos et al., 2023b). By the definition of retrieval robustness proposed in Yoran et al. (2023), retrieved context should boost model performance when relevant, and should not adversely affect it when irrelevant. We thoroughly examine the robustness of the model with regards to the order of the retrieved captions, and the relevance of the retrieved content. We also present a novel analysis of model behaviour based on majority voting, supported by input attribution and
9285
attention analyses to investigate how the retrieved tokens influence the model generation. And finally, inspired by Hoang et al. (2022), we propose to sample the retrieved captions from a larger list during training to prevent the model from overfitting to the top relevant captions. Our evaluation shows improved model robustness and better out-of-domain generalization.
The main findings of this paper are: 1) We study the robustness of an existing retrieval-augmented captioning model SMALLCAP and find it is not robust to processing randomly retrieved content. 2) We identify that tokens that frequently occur in the retrieved captions, i.e. majority tokens, have high attribution scores with regard to the tokens generated by the model. This phenomenon suggests heightened sensitivity and copying. 3) Training with sampled retrieved captions from a larger list instead of with fixed top-k relevant captions improves model robustness, yielding better generalization and out-of-domain performance.1

heading: 2 Related Work
text: Robustness of retrieval-augmented models. Retrieval-augmented generation (RAG) involves enhancing the generation process by incorporating retrieved information from an external datastore as additional context to the input (Lewis et al., 2020). RAG models have shown to improve performance across a variety of NLP tasks (Mialon et al., 2023). However, RAG models can overly rely on retrieved information, resulting in inaccurate generation when the retrieved context is flawed (Yan et al., 2024; Yoran et al., 2023).
Recent efforts aim to enhance RAG model robustness against misguided or hallucinated generations. One approach involves filtering retrieved content (Wang et al., 2023; Yoran et al., 2023; Yasunaga et al., 2023; Yan et al., 2024; Asai et al., 2023) by applying or training an additional evaluator. Another direction focuses on improving robustness during the training of the generation model itself. Specifically, for retrieval-augmented question answering with large language models, Yoran et al. (2023) propose continued training with a synthetic dataset that contains both relevant and irrelevant context, while Cuconasu et al. (2024) suggests incorporating irrelevant documents. In retrieval-augmented translation, robustness is
1We release the code at https://github.com/ lyan62/RobustCap
improved through shuffling retrieved translations (Hoang et al., 2022), ensemble model decoding (Hao et al., 2023), and controlled interactions between source and retrieved translations (Hoang et al., 2023).
Retrieval-augmented image captioning. Image captioning is the task that describes the visual contents of an image in natural language (Xu et al., 2015; Osman et al., 2023). Recent studies have integrated RAG into this field. Sarto et al. (2022) and Zhou and Long (2023) experimented with retrieving similar or style-aware images before generating captions. Li et al. (2023) introduced a lightweight image captioning model that utilizes retrieved concepts. More related to our work, Ramos et al. (2023a) developed end-to-end encoder-decoder models that attend to both the image and retrieved caption embeddings.
In particular, the SMALLCAP model (Ramos et al., 2023b), presenting retrieval augmentation in image captioning could reduce trainable parameters and adapt to out-of-domain settings. The model utilizes frozen unimodal models, incorporating a pre-trained encoder and decoder connected by trainable cross-attention layer.
However, it still remains unclear how retrieved captions influence the generation of captions in retrieval-augmented image captioning, especially concerning visual inputs. Additionally, the evaluation and enhancement of the robustness of these models are still under-explored.

heading: 3 Robustness of Retrieval-Augmented Image Captioning
text: To evaluate the robustness of the SMALLCAP retrieval-augmented caption model (Ramos et al., 2023b), we conduct controlled experiments and observe its resilience to changes in (1) the order of the retrieved captions and (2) the content relevance of the retrieved captions.

heading: 3.1 Robustness Evaluation
text: For a given image, SMALLCAP is augmented with a sequence of k retrieved captions that are combined into an input for the language model decoder: “Similar images show cap1, cap2, ..., capk. This image shows ...”. The retrieved captions are obtained through image-to-text retrieval using CLIP embeddings (Radford et al., 2021), and are sorted according to their relevance, i.e., cosine similarity. From the sorted retrieved captions, we retain the most
similar captions as the retrieval list. In this regard, the top-k retrieved candidates are the first k captions in the list, and the low-ranked captions are the last-k captions in the list. SMALLCAP uses the top-k retrieved captions in the prompt by default.
Context order. When prompting the model to generate a caption for a given image, we can change the order of the retrieved captions by permuting or reversing them. We evaluate the effect of the order changes in two settings: one with a model trained using the top-k retrieved captions (default), and another that is also trained with permuted or reversed retrieved captions.2
Content relevance. To evaluate how robust the model is towards noise in the retrieved captions, we are curious to see how the model performs when (1) captions are randomly retrieved, i.e. likely to be irrelevant for the given image (2) only lowranked retrieved captions are available. Here the randomly retrieved captions are those retrieved with another image. For low-ranked captions, we take the lowest-ranked k captions from the retrieval list that consists of top seven relevant captions.

heading: 3.2 Experimental Setup
text: In the experiments, we set k = 4 as it has been demonstrated as the optimal number of captions
2For the model trained with default order—top four captions, we use the pretrained checkpoints from HuggingFace: https://huggingface.co/Yova/ SmallCap7M, https://huggingface.co/Yova/ SmallCapOPT7M
by Ramos et al. (2023b). We evaluate SMALLCAP models with both OPT-350M (Zhang et al., 2022) and GPT-2 (Radford et al., 2019) as the decoder models. For the image encoder, we use ResNet50x64 (He et al., 2016) and CLIP-ViT-B/32 (Radford et al., 2021) as the retrieval encoder. We keep the same model setting in the following sections unless stated otherwise.
Data and metrics We first evaluate the robustness of SMALLCAP on COCO validation set for in-domain evaluation. Then we evaluate on NoCaps (Agrawal et al., 2019), which contains In, Near and Out-of-domain data, and serves as a challenging dataset designed to assess the generalization capabilities of models trained on COCO. For both datasets we use the validation set experimenting with different number of retrieved captions, i.e. different k values. We report peformance using CIDEr score (Vedantam et al., 2015).

heading: 3.3 Order Robust but Content Sensitive
text: Order robust. From the results in Table 1 and Table 2, we observe that SMALLCAP is indeed robust to the order of the retrieved texts. Permuting the order of the captions during training and evaluation show 1 CIDEr point improvement for COCO (Lin et al., 2014) and 2− 3 CIDEr score increase for NoCaps (Agrawal et al., 2019). This indicates that if multiple captions are used for augmentation, then permuting their order helps.
Content sensitive. Figure 2 shows that when using randomly retrieved captions instead of the top-k most relevant captions, performance drops drastically compared to the no-retrieval baseline.3 This
3Here the top and low ranked captions are obtained from a list of top-seven captions retrieved captions ordered by their cosine similarity to the image embedding.
implies that SMALLCAP lacks resilience to noise in the retrieved captions, and the irrelevant context has the potential to mislead the model, resulting in inaccurate predictions. When prompting with low-ranked retrieved captions, while performance slightly decreases, the retrieval-augmented model still outperforms the one without retrieval.

heading: 4 Majority Tokens Explain Behavior
text: To better understand how each token of the retrieved content relates to the observed sensitivity discussed in the previous section, we hypothesize that the model is driven by the presence of majority tokens. In other words, when the model is prompted with retrieved captions, we assume that the predicted tokens are influenced by the tokens that appear in the majority of the retrieved captions. To test this assumption, we propose a majority voting analysis, followed by input attribution, and an attention analysis of the model behavior.

heading: 4.1 Majority Tokens
text: We first introduce the definition of majority tokens. Let R = [T1, · · · , Tn] represent a retrieved caption R, which contains a sequence of n tokens. For a given image, we assume that a total of K retrieved captions are used in the model prompt: R1, R2, . . . , RK . For each token Ti in the set of unique tokens from the retrieved captions, we define Ti as a majority token (denoted as TM ) if Ti appears in more than half of the retrieved captions4, i.e., CTi > K 2 where CTi is the number of retrieved captions that contains token Ti as in Equation 1:
4Note that we remove the stop words in the retrieved captions when determining the majority tokens. The stop words are filtered from the top-100 most frequent tokens in the COCO dataset, where we manually remove meaningful tokens such as “man”, “two” from the list. Please see the Appendix A for the complete list.
CTi =
K∑
l=1
1[Ti ∈ Rk] (1)
For a generated caption Y = [y1, · · · , yn] in the evaluation data, we can calculate the majority-vote probability PTM∈Y as the probability of the majority token TM appearing in the generated caption.
We expect that the higher the value of PTM∈Y , the more likely it is that the model is generating captions based on the majority tokens.

heading: 4.2 Experimental Setup
text: We test our majority vote assumption with a controlled experiment. Specifically, we analyze the predictions of the model in two settings, each provided with K = 3 retrieved captions to ensure the presence of a majority token:
2 Good 1 Bad (2G1B): The retrieval set contains two relevant captions and one irrelevant caption;
2 Bad 1 Good (2B1G): The retrieval set contains two irrelevant captions and one relevant caption.
The assumption is that, if there is a majority voting behavior with respect to the retrieved captions, the model will copy such majority tokens to the final output. The distinction will be clear in this setting — in the setup 2B1G, if the model is robust to the retrieved context, the model will focus more on the good caption instead of the majority tokens in the two bad captions.
We use the COCO evaluation set and the pretrained checkpoint with the OPT decoder of Ramos et al. (2023b) for this analysis. Good captions are obtained using the top-two and top-one retrieved captions, respectively, for a given image. Bad captions are obtained by retrieving one or two captions, respectively, from a randomly selected image.
Results. We find that the probability of majority vote in the 2G1B setting is 86.47%. This high probability suggests that the majority tokens in the good captions could be being used to guide the model generation. In the 2B1G setting, the model is much less likely to generate majority tokens from the bad captions, indicating some robustness in not always following them. However, 20.84% of the time, the model can still be misled by their appearance, resulting in the majority tokens being copied into the model output.

heading: 4.3 Input Attribution with Integrated Gradients
text: To better understand the role of majority tokens in model generation, we use integrated gradients (Sundararajan et al., 2017) for input attribution analysis. This enables us to examine the influence of each individual token in the retrieved captions on the model prediction.
Attribution visualization. Figure 3 shows an example of an attribution visualization, where the attribution score of each input token (x-axis) is computed at each generation step (y-axis). Bright color cells correspond to high attribution to the input token. High attribution scores to the same tokens seen in the retrieved captions may indicate copying. Negative attribution scores are observed at contradicting tokens observed in the retrieved captions to the current generation. Negative scores are observed at token “her” when model is predicting the token “boy” and at token “small” when predicting “herd”. Additional input attribution visualizations can be found in Appendix B.1.
Quantitative analysis. We also quantitatively analyze the impact of majority tokens by calculating pairwise attribution scores between tokens in retrieved captions and those predicted by the model. Higher attribution values suggest greater sensitivity to the input token (Ancona et al., 2018). Figure 4 shows the distribution of the pairwise attribution scores for the 2B1G setup. It is clear that the model is sensitive to the majority tokens, especially when the generated token exists in the retrieved captions. Such behavior indicates weak robustness: we would not expect a robust model to be distracted
by the tokens from the two irrelevant retrieved sentences at inference time. To better visualize the impact, we show distribution of original attribution values and the absolute values (Ancona et al., 2018) across all evaluation samples.

heading: 4.4 Attention and Model Behavior
text: Finally, we visualize the self-attention and crossattention to locate the heads and layers in the SMALLCAP-OPT125M model that may contribute to the majority voting behaviour when generating a caption. This is crucial because all interactions between captions (self-attention) and images (crossattention) take place in this stage.
Distribution of max attention occurrence. We partition the text input prompt into five distinct segments: begin of the sentence token (<BOS>),
prompt tokens before retrieved k captions (prefix), i.e. “Similar image shows”, the retrieved captions (retrieval) cap1, · · · capk, prompt tokens before generation (suffix), i.e. “This image show”, and the generation itself. For image patches, we segment them into two pieces – the CLS output embedding, and the set of patch output embeddings.
Let Sn denote the sets of indices, where n = 1, 2, . . . , 5 for five segments. For the text input, each segment Sn contains the indices of the tokens in each segment. To track the occurrence of max attention values in Sn, we define the indicator function 1[In(i, j)] as follows:
1[In(i, j)] = { 1 if argmaxz Att(j, z)i ∈ Sn 0 otherwise , (2)
where argmaxz Att(j, z)i is the index of the input with the maximum attention score for sample i.
For self-attention between the textual tokens, Att(j, z) represents the attention score between the jth generated token and the zth text context token, denoted as SAtext(j, z).
For cross-attention between the decoder and the image representations, we report both a text-centric and an image-centric analysis. The text-centric analysis XAtext(j, z) measures the attention between the jth image patch and the zth text token, to identify which segments of the text have the highest cross-attention scores in relation to the image. In the image-centric analysis XAimg(j, z),
we measure the attention between the jth generated token and the zth image patch. We now redefine the Sn notation to let S1 represent the CLS output embedding, and S2 represent the set of image patch embeddings, respectively. This allows XAimg(j, z) to identify if the CLS patch embedding receives the highest cross-attention scores in relation to the generated tokens, or if it is the actual image patch embeddings.
For each analysis, SAtext(j, z), XAtext(j, z), and XAimg(j, z), we calculate the proportion of occurrences of the maximum score in Sn by averaging through all generated tokens for a dataset.
Self-attention. We gather attention scores between the generated tokens and context tokens, and categorize the distribution of the maximum scores into the five text segments (BOS, prefix, retrieval, suffix, and generation).
Figure 5(a) illustrates the changes in the distribution of maximum self-attention scores in each layer of the decoder language model. Notably, at the initial layers, a majority of attention heads exhibit heightened focus on retrieved captions or the current context for generation. However, after the second layer, we observe an increased emphasis on the beginning of sentence token (
). This behavior is consistent with prior research on the attention mechanism of GPT-2 (Vig and Belinkov, 2019). Figures 9a and 10a show the behaviour for all self-attention heads in for the GPT and OPT model variants, respectively.
Cross-attention. Similar to the self-attention behaviour, we categorize the occurrence of the maximum cross-attention to the five text segments. As shown in Figure 5(b), in most attention heads, the cross-attention attains its maximum value between the image and the retrieved captions or between the image and the generated tokens. Figures 9b and 10b show the text-centric analysis for all crossattention heads for the GPT and OPT backbones.
Finally, we inspect whether the model focuses on the CLS patch or actual image patches. In Figure 5(c), we observe that the model only pays maximum attention to the image patches in the final layers (the blue line). Figures 9c and 10c show the full results for the image-centric analysis.
Overall, these observations show that the model attends to both modalities during the caption generation process. However the lack of strong crossattention to actual image patches suggests that the
model is misled by text prompts, even when irrelevant information is absent in the provided image.

heading: 5 Improving Robustness to Retrieval via Sampling
text: In order to improve the robustness of the model to potentially noisy captions, we propose to randomly sample the captions from a larger retrieval list for a given image, instead of training with only the top-k retrieved captions. In this manner, the model can learn from more diverse context that includes both top- and lower-ranked captions.

heading: 5.1 Experimental Setup
text: Inspired by Hoang et al. (2022), we experiment with two sampling methods during training to improve retrieval robustness.
Sample-k training. We sample k captions randomly from the top-N=7 retrieved captions during training5. Following Ramos et al. (2023b), we train SMALLCAP with the OPT-350M decoder on the COCO captioning dataset (Chen et al., 2015) for 10 epochs on a NVIDIA A100 40GB GPU with the default learning rate of 1e-4 and batch size of 64. We experiment with k in the range of 1–4.
Controlled sample-k training (c-sample-k). Aiming to train the model that better distinguishes irrelevant context, we design a controlled sampling process — selecting k−1 randomly from the larger list while keeping the top relevant caption of the image during training. We train the model with same hyperparameters and dataset as sample-k.

heading: 5.2 Evaluation and Results
text: In addition to the COCO and NoCaps validation set, we evaluate the Out-domain performance of the model using VizWiz caption dataset (Gurari et al., 2020) and report CIDEr scores.
Sample-k training improves model robustness to random retrieved captions. As shown in Table 3, incorporating sampled retrieved captions into training consistently enhances performance across various k values. The improvement is particularly notable when captions are randomly retrieved, suggesting the model is now better able to ignore irrelevant context. If we compare across different values of k, sampling mitigates the model’s sensitivity to the number of retrieved captions, outperforming
5We sample from the top-N=7 for alignment with the baseline; see the Appendix for an ablation on varying N .
top-k training. For instance, it achieves comparable performance with a smaller k value than in the case of top-k training. Furthermore, the gap between using the top-k vs. the last-k retrieved captions is reduced with sample-k training: the maximum gap is reduced from 3.0 to 1.0 CIDEr points, indicating increased model robustness, even with lowerranked retrieved captions. Figure 6 and 12 show qualitative examples of the improved robustness to randomly retrieved examples.
Sampling improves cross-domain evaluation. We also evaluate on VizWiz and NoCaps to measure cross-domain performance (Table 4). This is a more realistic setting where retrieved captions are out-of-domain and could be more noisy and less relevant. The application of sampling improves across all values of k for Vizwiz. On the NoCaps dataset, with the COCO datastore, sampling consistently improves near and out-domain performance, suggesting increased robustness to noisy retrieval context. This is consistent with the benefits of sampled training demonstrated in cross-domain machine translation by Hoang et al. (2022). If we use a larger datastore that incorporates internet-derived captions (+Web), this consistently improves indomain performance. Retrieval constraints are alleviated for near and out-domain samples with the larger datastore, where we see smaller gains with sample-k. See qualitative examples in Figure 11 in
Appendix C.
Controlled sampling further improves crossdomain evaluation. Finally, on top of our best performing sample-k model, controlled sample-k further improves performance for both NoCaps and VizWiz. This suggests that incorporating both toprelevant and low-ranked captions during training aids the model in distinguishing irrelevant context.

heading: 6 Discussion
text: Majority tokens are reliable hints during training. To better understand why the model relies on majority tokens during generation, we calculate the probability that majority tokens in the retrieved captions overlap with the ground truth captions (TM ∈ GT ), and with the predicted tokens
(TM ∈ Pred). Table 5 shows that in 88%–99% of the training examples, the majority tokens in the retrieved captions are also present in the ground truth captions. This suggests that the model can develop a bias towards majority tokens due to the fact that they are so often present in the ground truth during training. This analysis also clarifies the decrease in the model’s robustness as k increases when randomly retrieving captions. This is because a higher k only adds noise without providing useful majority tokens. The use of sampling during training exposes the model to more diverse context, which leads to a slightly increased level of selectivity.
In Figure 7, we show the variation in the distribution of majority tokens across various evaluation datasets. When captions are randomly selected for the COCO evaluation data, there are fewer majority tokens in the retrieved captions. This presents a challenge for the model in making use of the retrieved captions, which accounts for the performance decrease shown in Figure 1. For evaluation, with the same value of k, the fewer the number
of majority tokens in the retrieved captions, the harder it is for the model to “copy” those tokens to the final output. In such scenarios, we obtain bigger improvements with the sample-k training.

heading: 7 Conclusion and Future Work
text: We studied the robustness of the state-of-theart retrieval-augmented image captioning model SMALLCAP and provide an through analysis and explanation of how retrieved captions effect the final prediction. Our exploration shows that SMALLCAP is robust to the order of the retrieved captions, but it is sensitive to retrieval noise, which has implications for using retrieval-augmented models in new domains. With extensive input attribution analysis, we show that such sensitivity is due to majority tokens in the retrieved captions. We demonstrate a more retrieval robust model can be trained with sampling methods during training. We expect that our analysis can inspire better retrieval-robust captioning models in the field.
In the future, we will investigate whether the majority voting behaviour is exploited in other retrieval-augmented captioning models. We hope to further explore if other techniques such as tokendropping or prefix-tuning would further improve retrieval robustness.
Ethics Statement
We acknowledge the potential risks of hallucination and biases introduced by retrieval augmentation in captioning models. Misleading tokens from the retrieved captions could cause the model to generate
captions describing nonexistent entities or objects in images (Liu et al., 2024; Rohrbach et al., 2018). This could have adverse effects, such as propagating systematic biases present in the datastore used for retrieval (Foulds et al., 2024).
Despite the exploration in our work, we acknowledge that no system is perfect, and undesirable biases may still be present with our methods. We emphasize the need for continued research into techniques for identifying and mitigating hallucination and bias in retrieval-augmented models (Foulds et al., 2024; Deng et al., 2024). We also stress the importance of responsible deployment, with human oversight and content moderation pipelines.
As researchers, we have an ethical obligation to be transparent about the potential risks and limitations of our work. We welcome further scrutiny and discussion around these critical issues within the research community.
Limitations
We evaluate the robustness of a single retrievalaugmented image captioning model in this study. Given variations in training process and model structures, the observed model behavior may be specific to our chosen model. Applying the same analysis to other models would be useful for a deeper understanding regarding explainability and interpretation of retrieval augmented image captioning models, which we leave for future work.
For all experiments in our study, we employ the same CLIP-ViT-B/32 backbone as the image encoder. Investigating how model robustness varies with different visual encoders would enhance the scope of our study.
While training with sampling improves model robustness, it is intuitive that introducing more noise during training makes the task more challenging. In all our experiments, we train the model for same number of epochs as SMALLCAP, therefore it is not clear if the model would gain more robustness if trained longer. We are curious if there exists an optimal balance between training time and the level of noise exposure for achieving model robustness.

heading: Acknowledgments
text: We thank Lei Li and the CoAStal and LAMP groups for feedback. Wenyan Li is supported by the Lundbeck Foundation (BrainDrugs grant: R279-2018-1145) and a research grant (VIL53122) from VILLUM FONDEN. Jiaang Li is sup-
ported by Carlsberg Research Foundation (grant: CF221432). Rita Ramos is supported by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (i.e., the Center For Responsible AI), and also by Fundação para a Ciência e Tecnologia (FCT), through the project with reference UIDB/50021/2020 (DOI:10.54499/UIDB/50021/2020) and the Ph.D. scholarship with reference 2020.06106.BD.

heading: A Majority Tokens
text: A.1 Stop words list
In this section, we present the stop words that were filtered from the COCO dataset in the experiments described in Section 4.2:
[’out’, ’some’, ’of’, ’is’, ’while’, ’are’, ’with’, ’down’, ’has’, ’over’, ’the’, ’next’, ’up’, ’near’, ’several’, ’other’, ’at’, ’top’, ’from’, ’in’, ’on’, ’a’, ’there’, ’an’, ’to’, ’and’, ’her’, ’front’, ’by’, ’for’, ’his’, ’it’]

heading: B More Visualization
text: B.1 Input Attribution with Integrated Gradients
In Figure 8, we show more attribution visualization for the experiment setup 2B1G in Section 4 where high attribution scores are observed in the majority tokens and mislead the model to generate incorrect captions.
B.2 Attention
In Figure 9 and Figure 10, we depict the distributions of both self-attention and cross-attention scores across various heads and layers for SMALLCAP with GPT-2 and OPT decoder variants.

heading: C Qualitative examples
text: We show more qualitative examples in Figure 11 and Figure 12.

heading: D More results
text: Order robustness evaluation In Table 6 and Table 7, we provide both CIDEr and BLEU4 scores for order robustness evaluation (Section 3).
Number of retrieved captions for sample-k training We experiment with different size of the retrieval candidate list from which we randomly select captions for sample-k training (Table 8).
Percentage of tokens that are likely to be copied In Table 9 we show the percentage of tokens that are likely to be copied from retrieved captions averaging through all samples in the validation set. Majority tokens takes more than half of the copied tokens.
Comparison with other methods Inspired by Yoran et al. (2023), we have considered intentionally including less relevant captions by including one irrelevant caption, one low-ranked caption,
and top-2 relevant captions instead of using top-4 retrieved captions. However, in our preliminary experiments, this strategy does not perform as well as the sampling approach, likely due to the high noise level it introduced.
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
Prefix Retrieval Suffix Generation
HEAD 0 HEAD 1 HEAD 2 HEAD 3
HEAD 4 HEAD 5 HEAD 6 HEAD 7
HEAD 8 HEAD 9 HEAD 10 HEAD 11
(a) Self attention distribution.
0 5 10 0
0.2
0.4
0.6
0.8
0 5 10 0
0.2
0.4
0.6
0.8
0 5 10 0
0.2
0.4
0.6
0.8
0 5 10 0
0.2
0.4
0.6
0.8
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
0.8
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.5
0 5 10 0
0.5
1
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
Prefix Retrieval Suffix Generation
HEAD 0 HEAD 1 HEAD 2 HEAD 3
HEAD 4 HEAD 5 HEAD 6 HEAD 7
HEAD 8 HEAD 9 HEAD 10 HEAD 11
(b) Cross attention distribution. Distribution of max attention scores of the interaction between various part of text prompt and image patches.
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
CLS Others
HEAD 0 HEAD 1 HEAD 2 HEAD 3
HEAD 4 HEAD 5 HEAD 6 HEAD 7
HEAD 8 HEAD 9 HEAD 10 HEAD 11
(c) Cross attention distribution. Distribution of max attention scores of the interaction between two type of image patches (cls, others) and all text tokens.
Figure 9: Statistics of max attention scores in self and cross attentions from different different layers and heads with SMALLCAP (GPT2 variant). Compute the proportion of each attention scores from self and cross attention belongs to which parts.
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
<BOS> Prefix Retrieval Suffix Generation
HEAD 0 HEAD 1 HEAD 2 HEAD 3
HEAD 4 HEAD 5 HEAD 6 HEAD 7
HEAD 8 HEAD 9 HEAD 10 HEAD 11
(a) Self attention distribution.
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
0 5 10 0
0.2
0.4
0.6
0.8
0 5 10 0
0.2
0.4
0 5 10 0
0.2
0.4
0.6
<BOS> Prefix Retrieval Suffix Generation
HEAD 0 HEAD 1 HEAD 2 HEAD 3
HEAD 4 HEAD 5 HEAD 6 HEAD 7
HEAD 8 HEAD 9 HEAD 10 HEAD 11
(b) Cross attention distribution. Distribution of max attention scores of the interaction between various part of text prompt and image patches.
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
0 5 10 0
0.5
1
CLS Others
HEAD 0 HEAD 1 HEAD 2 HEAD 3
HEAD 4 HEAD 5 HEAD 6 HEAD 7
HEAD 8 HEAD 9 HEAD 10 HEAD 11
(c) Cross attention distribution. Distribution of max attention scores of the interaction between two type of image patches (cls, others) and all text tokens.
Figure 10: Statistics of max attention scores in self and cross attentions from different different layers and heads with SMALLCAP (OPT-125M variant). Compute the proportion of each attention scores from self and cross attention belongs to which parts.

heading: Future Work
text: In the future, we will investigate whether the majority voting behaviour is exploited in other retrieval-augmented captioning models. We hope to further explore if other techniques such as tokendropping or prefix-tuning would further improve retrieval robustness