Understanding Retrieval Robustness for Retrieval-Augmented Image
  Captioning
Authors: name: Wenyan Li, name: Jiaang Li, name: Rita Ramos, name: Raphael Tang, name: Desmond Elliott

Recent advances in retrieval-augmented models for image captioning highlight
the benefit of retrieving related captions for efficient, lightweight models
with strong domain-transfer capabilities. While these models demonstrate the
success of retrieval augmentation, retrieval models are still far from perfect
in practice: the retrieved information can sometimes mislead the model,
resulting in incorrect generation and worse performance. In this paper, we
analyze the robustness of a retrieval-augmented captioning model SmallCap. Our
analysis shows that the model is sensitive to tokens that appear in the
majority of the retrieved captions, and the input attribution shows that those
tokens are likely copied into the generated output. Given these findings, we
propose to train the model by sampling retrieved captions from more diverse
sets. This decreases the chance that the model learns to copy majority tokens,
and improves both in-domain and cross-domain performance.