IC3: Image Captioning by Committee Consensus
Authors: David Chan
Austin Myers
Sudheendra Vijayanarasimhan
David A Ross
John Canny
Venue: EMNLP 2023 Main (2023)
Keywords: Image Captioning
Large Language Models
Prompt Engineering
Visual Description
TL;DR: IC3 is a visual description method which generates a set of candidate captions, then summarizes them to generate high quality image descriptions.

Abstract: If you ask a human to describe an image, they might do so in a thousand different ways. Traditionally, image captioning models are trained to generate a single "best" (most like a reference) image caption. Unfortunately, doing so encourages captions that are "informationally impoverished," and focus on only a subset of the possible details, while ignoring other potentially useful information in the scene. In this work, we introduce a simple, yet novel, method: "Image Captioning by Committee Consensus" (IC3), designed to generate a single caption that captures high-level details from several annotator viewpoints. Humans rate captions produced by IC3 at least as helpful as baseline SOTA models more than two thirds of the time, and IC3 can improve the performance of SOTA automated recall systems by up to 84%, outperforming single human-generated reference captions, and indicating significant improvements over SOTA approaches for visual description. Code is available at [https://davidmchan.github.io/caption-by-committee/](https://davidmchan.github.io/caption-by-committee/)

IC Image Captioning by Committee Consensus
David M. C Austin Myers?, Sudheendra Vijayanarasimhan' David A. Ross" .ohn Canny
University of California, Berkeley, 'Google Research
(davidchan, Jeelyy edu, {aom, svnaras,
Abstract
Deseribe
polie officr is nitng The Growm norael
O-
If you ask a human to describe an image, they might do a thousand different ways. Image captioning models, on the other hand, are tradi- tionally trained to generate a single "best" (most like reference) caption. Unfortunately, doing encourages captions that are informationally impoverished. Such captions often focus on only subset of possihle details, while ignoring other potentially useful information in the scene. this work, we introduce simple, yet novel, method: "Image Captioning by Committce Consensus" (IC"), designed to generate a single caption that captures details from multiple viewpoints by sampling from the learned semantic space of a base captioning model, and carefully leveraging large language model to synthesize these samples into single comprehensive caption. Our evaluations show that humans rate captions produced IC' more helpful than those produced by SOTA models more than two-thirds of the time, and IC improves the performance of SOTA automated recall systems by up to 84%, outperforming single human- gennenaed reference captions and indicating significant improvements over SOTA approaches for visual description. Code/Re- sources available https: htts:://daiddmhhaaaa github. io/caption by- committee.
"Reople in frott oflo lootth alore'
'A polceemaa riding heree nost to man bicooe
man is näin horso down o city rmeer"
polrn office neting brown down oity
B
In the IC (Image Captioning by Committee Consensus) method. we first leverage standard image captioning models to generate descriptions covering a range of content within the image, similar to how human raters describe the image from independent and unique points of view. We then summarize the group of captions using vision-free summarization model into single, high-quality description of the image, suitable for use visual description applications.
Figure 1:
While a set of image captions can be useful, many applications, such as alt-text generation, demand single succinct sentence that summarizes the information present in the image. This "summarized" caption usually takes a different structural form compared the captions sourced from crowd workers that make up the datasets. While contain a subset of the relevant information in an image, it is unlikely that they contain everything (MacLeod et al., 2017; Stangl et al., 2020)-
captions may
1 Introduction
Unfortunately, while the development of large vision and language models (VLMs) has led progress on a variety of tasks including image captioning, models are trained to produce samples from the reference distribution of a captioning dataset such as MS-COCO (Li et al., 2022; Wang et al., 2022b; Alayrac et al., 2022; Yu et al., 2022; Chen et al., 2022). While not inherently flawed, this approach reproduces the dataset's single annotator viewpoint captions, containing some, but not all, of the semantic information present in the image.
Generating high-quality description of an image is not only an open research problem, but is also a challenging task for humans (Lin et al., 2014; Sharma et al., 2018; Young et al., 2014). Image captioning datasets usually acknowledge this fact; rather than providing single gold standard caption for each image, they instead rely human anno- tators provide multiple captions for each image, hoping that the set of collected captions collectively captures all of the relevant semantic information.
Thus we seek to answer the question: "How can we combine many single-viewpoint captions into
Corresponding author

---

collective summary of the image containing the relevant semantic information?"
ance in the caption space, and introduced human, statistical controls reduce the variance, turning the multi-modal problem into several uni-modal prob- lems. While these methods are effective at describ- ing the same image multiple times from multiple per- spectives, they have not demonstrated an effective approach that generates a single caption covering all of the information in each of the diverse captions.
One way to obtain a more comprehensive caption, given a set of single-viewpiint captions from annotators, would be to have another human expert consider the set of captions from the committee of individual annotators, and create a new caption that combines complementary information while filter- ing out any syntactic or semantic errors. Motivated by this idea, we propose the Image Captioning by Committee Consensus (IC²) approach, which uti- lizes off-the-shelf VLMs in conjunction with large language models (LLMs) to generate higher quality captions than would be possible with VLMs alone. Our key contributions summarized follows:
Dense captioning methods (Johnson et al., 2016 6; Yang et al., 2017: Li et al., 2019; Yin et al., 2019; Kim et al., 2019) attempt to generate a full descrip- tion of all of the objects in the image, however, dense image captions are long and unwieldy, and often contain redundant or repetitive information. Similar long- long-form captions have been explored in the form of paragraph captioning (Zha et al., 2019; Krause et al., 2017: Liang et al., 2017: Mao et al., 2018; Chatterjee and Schwing, 2018; Luo et al., 2019), however all cases, no efforts have explored using additional post-processing or models to distill the relevant information for alt-text or for downstream applications. In this work, we explore beyond single- view captioning and move towards captions that are short, succinet summaries of the full visual context.
1. We introduce IC², a simple, yet novel, approach leveraging pre-trate pre-trained image captioning models and large language models to generate semantically complete image captions from a collection of "single-viewpoing captions.
We perform an extensive human evaluation of our method which demonstrates that human raters rate image captions generated by IC3 higher in both helpfulness and correctness.
A natural way of summarizing and filtering a dense caption, paragraph caption, or set of captions, is with a pre-trained model for summarization. While end-to-end methods for abstractive and extractive text summarization exist (Allahyari et al., 2017), recently, large language models (LLMs) such GPT-3 (Brown et al., 2020), LAMDA (Thoppilan et al., 2022) and PALM (Narang and Chowdhery, 2022) have demonstrated remarkable zero-shot performance when performing language- only summarization tasks (Brown et al., 2020; Liu et al., 2021: Goyal et al., 2022: Chintagunta et al., 2021; Kieuvongngam et al., 2020), so it is natural that they would be capable of summarizing multimodal information in zero-shot way. Indeed, recently, large-scale vision and language models (VLMs) and large-scale language-only models (LLMs) have revolutionized a number of sub-fields in AI, including in the image captioning space. Models such as BLIP (Li et al., 2022), OFA (Wang et al., 2022b) and Flamingo (Alayrac et al., 2022) have all demonstrated strong performance in single-view image captioning tasks, and indeed, many of these approaches are rated good better than human users in some evaluations.
3. We demonstrate through several automated measures that captions generated using IC2 contain significantly more semantic information than baseline captions. Notably, CLIP-recall with IC captions can be improved by 22-84%, with improved coverage of objects and actions in the image.
2 Background and Related Work
The idea that captioning models tend to produce single-viewpoint captions has been prevalent in the image captioning community for many years under several names (Wang and Chan, 2019). Notably, research has focused on quantifying and improving the "diversity" of output captions, including specific methods (Klein et al., 2022; Aneja et al., 2019; Dai at, 2017; Mahajan et al., 2020; Mahajan and Roth, 2020; Wang et al., 2017, 2016) and metrics (Holtz- man et al., 2020; Zhu et al., 2018; Wang et al., 2020; Shetty et al., 2017; Deshpande et al., 2019; Chan al., 2022b; van Miltenburg et al.. 2018: Wang and Chan, 2019). As an alternate approach to increasing and quantifying diversity, some methods (Gan et al., 2017; Yang et al., 2020; Zha et al., 2019; 9 Fang et al., 2022) have focused on explicitly modeling the vari-
Surprisingly, vision-blind LLMs have also

---

forperattne Eased Sammpling
People satch the zherry blossces at the Japanese ir enndsti garden in pii Ladelpe on saturday.
Peosle enjey the cherry blossoss in the us nat nonional arboretur in sashington
People warvel
beautiful weeping
cherry trees in fand
gatnering place under tree full of pink flovers.
Peosle are shown standing in park under canppy of cherry trees.
OFA/BLIP Seed Model
group people standing People pose for photo white watching cherry trees blossons;
(Generates semantic mantiold)
Nolsy Caption Samples
(Representative somples from sanantic dietrbution)
Large Language Model + Summary Prompt This 1s hard problen. Carefully sumsarize in OKE detailed sentence the following captions different (pessibly incorrect)
Output Caption:
LLM Decoding
A group of people gathered in pank aomining the cherry blossoms and taking photos them.
[Reaes Seare))
describing the ident ity you'
describe everything,
exanple:
Capttons: (forratted_capp captions]- Summary:
I'n not sure. but the image is likely of...
The IC approach. Every captioning model defines a distribution across a caption semantic space. This distribution is unlikely to be unimodal, thus, while maximum likelihond decoding approaches such as beam search will capture a local maximum, this point is not likely to be representative of the full distribution of captions. Instead, IC first gemerates a representative sample of captions from the semantic manifold using temperature- based sampling. This set naturally captures any means as well as the variance of semantic information in the image. Because this group of captions can be large, hard to parse, noisy, or incorrect, we use a large-scale language model, such as GPT-3, paired with prompt engineering, to summarize and filter the noisy group of captions. The resulting captions are more dctailed and often more useful than captions generated by beam search alone.
Figure 2:
become particularly prevalent in multimodal image/language spaces, primarily using language- only prefix generated by set of pre-trained tools. Mokady al. (2021) explores using a continuous embedding as a prompt for a GPT-style language model and demonstrate strong single-viewpoint image captioning performance, while Hu et al. (2022) and Tiong et al. (2022) leverage natural language prompts along with GPT to achieve SOTA performance on visual question answering.
noisy) information about the image from multiple sampled viewpoints. This avoids repetitive (and computationally expensive) QA loop, which with imperfect models can not only introduce significant noise, but also can fail to uncover detail outside the questioning distribution. Also related to our work is Xie et al. (2022), which uses similar tags to generate paragraph-captinn but does not explore filtering the image, or using existing caption distributions.
3 Methods
Closest to our approach are (Zhu et al., 2023) (developed concurrently with the proposed work) and Zeng et al. (2022). Zeng et al. (2022) leverages a CLiP-based model to extract key tags from the image, and then uses GPT-3 along with a specialized prompt to generate stylized image caption, in attempt to emulate the Socratic method. Zhu al. (2023) further employs the Socratic method by employing Chat-GPT and BLIP-2 (Li et al., 2023) to ask and answer questions about the image, respectively. Finally, Chat-GPT summarizes the QA transcript into the image description. Our proposed approach primarily differs from Zhu et al. (2023) and Zeng et al. (2022) in the method of visual data extraction. Instead of using the Socratic method, which requires repeated high-quality questioning and high-quality VQA models to elicit data, or imprecise image tagging models, our approach relies on existing image-captioning mod- els augmented with temperature based sampling, which are able to generate diverse set of (possibly
In this work, we introduce a simple framework for visual description, based on a committee generation then summarization process, which we call "Image Captioning by Committee Consensus" (IC*). The approach consists of two stages. In the first stage, leverage a standard pre- meettrineed image captioning model sample several (potentially noisy) captions using temperature- based sampling from the caption distribution. This generates a representative sam- ples from the caption distribution, each possibly de- scribing different parts of the image. In the second stage, we leverage a summarization model to sum- marize the information in each of the captions in a short and succinct way that can be presented to user. An overview of our method is given in Figure 2.
3.1
IC Image
Captioning by Committee Consensus
The goal of is to generate an output caption from a given image, by first sampling from a frozen image

---

captioning model, and then summarizing these generated captions into single "summary caption". More formally, given an image I, we aim to produce a sequence of m tokens 21...Im describing the im- age. Formally, an image captioning model can be de- scribed as a function. M which takes and a set of to- kens 11..0k=1 in some vocabulary V, and produces probability distribution P(a VII,a1 (i. (..0a...))) the probability of the next token in the sentence given all previous tokens and the image.
IC the summarization model visually blind that is, the image is not taken into account at all during the summarization phase. We discuss our choice of summarization model in subsection 3.3. In our work, C' is generated using beam-search from the summa- rization model, giving maximum likelihood estimate of the best summary of the input captions.
3.2 Image Captioning Models
The first stage of the method is to sample a set K of candidate captions from an underlying pre-trained image captioning model, M. In this work, we explore two underlying image captioning engines, the BLIP model (Li et al., 2022), and the OFA model (Wang al., 2022b), which both represent high- high-quality image-captionng models pre-trained on large-scale image and language data, then fine-tuned for captioning on the MS-COCO dataset (Lin et al., 2014), More details on the specific image captioning models can be found in Appendix D.1.
Traditionally, image captioning models generate a caption C, where:
C= argmax
I [((a)a (1) "I.SRSN E=1
Finding the argmax is particularly challenging, as it is an optimization over all possible sequences. Usually, to avoid these challenges, a technique such as beam search (Li et al., 2016; Shen al., 2017) is used to reduce the number of possible candidates. Recently, however, has been shown by several papers, including Chan et al. (2022a) and Caglayan et al. (2020) that captions generated using beam search contain only the mutual information between the references, and that such captions are often bland, and uninteresting. To avoid this, we instead take a different approach. Instead of maximizing the likelihood, we generate a set of samples, = {k- ... ...k.,, kj), from the model using temperature-based sampling of the distribution:
Temperature Selection: We want to generate a sample of captions that lies as close to the reference distribution as possible. For most models, this will be at or close to temperature 1. To validate this, we use the TRM-CIDEr metric introduced in Chan et al. (2022b) measure the distance between the reference distribution and the generated captions at set of temperatures between 0 and 2. We found that for the BLIP model, the optimal temperature was 1.15, and for the OFA model, the optimal temperature was 0.95.
log P(a1...mm.!!)
k; =u....mmm x xexp
(2)
T
Selecting size of K: To select the number of cap- tions that are generated, we used small validation set and found that 10 captions represented good trade-off between the length of the prompt, and the quality of the model. Sampling larger numbers of candidate captions can improve the total captured information but can decrease the performance of the summarization model, and be more costly evaluate (See Appendix C.3 for an ablation).
where T's a temperature parameter. At temperature 1, the resulting samples K k; k; are an unbiased estimate of the distribution of reference captions. This means that, unlike the maximum likelihood estimate caption, the sampled captions will contain variance in information commensurate with the variance of human descriptions.
Unfortunately, while the caption set is a good description of all of the details that we might care about in the image, a set of captions can be hard to parse for downstream user. Thus, we would like to summarize the set K single caption, by remov- ing any redundant (or incorrect) information, and combining any details mentioned in one description, and not in one of the others. To do this, we leverage a summarization model S, which maps the set of captions to our final single output caption C. In
3.3 Summarization Models
The choice of the summarization model S is key decision when implementing IC', as the model should be capable of high-quality zero-shot abstractive summarization. We found that using a large language model (LLM) for zero-shot sum- marization is effective in generating high-quality output summaries of the input captions (See

---

Appendix C.1), For the results in section 4, use GPT-3 (Brown et al., 2020), as it produced strong abstractive summaries of the candidate captions, however we discuss (and explore) in detail the choice of summarization model Appendix C.I.
of prompt can improve zero-shot performance, we also add the short sentence "this is hard problem". We found that this generally improved the quality of the model by a small amount, with diminishing returns as the quality of the candidate captions improved as scen in the ablation in Table A3.
3.4 Prompt Selection
Use of capitalization: In our exploration of the prompt space, we found that in some cases, the models choose to generate long concatenations of the input captions instead of generating a single short and concise sentence. Thus, to help alleviate this issue, we found that capitalizing the "ONE" when asking for sentences encouraged the GPT models to produce shortened captions, usually consisting of single sentence (reducing the average caption length from 120.03 to 107.89 characters).
To use a large-scale language model for summa- rization, we follow the techniques in Brown et al. (2020), and design input text prompt passed to the language model, to encourage the generation of correct output. The choice of prompt is defined by several motivations, including encouraging the model to summarize the information in the sampled captions, particularly the uncertainty of the captions, encouraging the model to convey this uncertainty in a natural manner (if the model is unsure about what is in the scene, it should identify when this is the case) and making sure that the generated caption is comprehensive. and contains all of the unique viewpoints presented in each of the sampled descriptions. Our final prompt used for experiments with GPT-3 was:
Style Transfer and Contextual Captions: addition to the above goals, it is interesting future work to explore how the prompt can be used to shape the generated output caption, either for zero-shot transfer to other languages, to guide the generation of the text pay attention to specific details in the image. While we do cursory exploration of such an approach in Appendix H and Appendix I, future work in this area is essential.
This is hard problem. Carefully summarize in ONE detailed sentence the following captions by different (possibly incorrect) people describing the same scene. Be sure to describe everything, and identify when you' re not sure. For example: Captions: [formatted captions). Summary: I'm not sure, but the image is likely of...
3.5 Evaluation
n-gram matching scores such as CIDEr (Vedantam et al., 2015) do a poor job comparing distributions of texts. An example of this is single caption which is the concatenation of two non-overlapping references. Because for each reference, there exist n-grams in the candidate that do not overlap with that reference, the candidate will score poorly. However the candidate has the same (or more) information than either of the two original reference sentences alone. Thus, along with extensive human evaluation, introduce two novel automated measures of caption quality, which directly address information retrieval.
Encouraging and surfacing uncertainty: In our prompt design, we aim to encourage the model to account for potential uncertainty/noise in the sam- pled captions. In many cases, high disagreement among the captions can indicate uncertainty in the language distribution, so we encourage the model to identify when the individual captions differ using language such as possibly incorrect, is likely of and I not sure in the prompt. The effect of encouraging and surfacing uncertainty is demonstrated in Table A2 in the appendix. This shows that choosing this language significantly increases the likelihood that models generate uncertain language, and that such captions are rated as more correct on average by human raters.
CLIP Recall: One measure of the quality of a caption is its ubility to distinguish between images in dataset (Hessel et al., 2021; Wang et al., 2022a). In this work, we leverage CLIP (Radford et al., 2021) recall model, and use it estimate approximate specificity of each of the captions generated by our approach. Specifically, for each image i, we compute the CLIP embeddings I; and the corresponding caption C,. We then compute
This is a hard problem: Following Kojima et al. (2022), who showed that adding short interrogaaiveeinsttutttve sentences to the beginning

---

the CLIP-Score (Hessel al., 2021) between Z, and every other generated caption Cj. and from this, compute the mean reciprocal rank (MRR) of the caption C₁ and the recall at 1, 5 and 10. High values of MRR suggest that the captions are more discriminative within the test set (and thus, are often more detailed in nature).
Table I: Head-To To Io-Head human evaluation performance of models augmented with IC' the MS-COCO dataset. Table shows % of instances preferred by users.
MODEL
HELPFULNESS
CORRECTNESS
BLIP-2+ Ic² BLIP-2
51.97% %
44.10
37.49%
42.67%
TIE
9.78%
11.6%
BLIP+IC" IC.
52.05 % 33.44%
42.90
Content Coverage: of the individual caption, we also would like to mea- sure how much of the total information provided by all of the references is included in the summarized caption. To do this, we first compute the caption C, for each image and fetch the references R م N for each image. Let.N(C,)) be the of nouns in a caption, G, and V(C:) be the set of verbs. Let
In addition to the specificity
BLIP
36.28
TIR
14.5 1 %
20.82%
OFA IC²
52.91 % 32.72% 14.37%
48.93
OFA
33.94%
TIB
17.12%
REF IC?
55.79 % 36.65%
48.80
REF TIE
36.27%
7.46%
13.97%
Reference Baseline: Because IC can be used to augment any existing captioning method, we also explore augmenting the human reference captions with IC3. To do this, we use the reference captions (RFF) as candidates for the summary pipeline, which are then summarized by the LLM to generate the REF+IC² caption. Such an approach removes the additional variance introduced by the candidate captioning model and demonstrates the potential of I applied to near-perfect captioning approach.
i(TE
Iva(n)= = 0
(3)
otherwise
We compute exact noun overlap for C, as:
Noun Overlap
E IN((()
(4)
((RR) (E)(())))
Verb overlap is defined analogously for V. We compute fuzzy overlap similar to exact overlap, however instead of Equation 3, we use:
Results & Discussion
In this section, we compare captions generated using our baseline seed image captioning models, BLIP (Li al., 2022), BLIP-2 (Li et al., 2023), and OFA (Wang et al., 2022b), to captions generated using IC We leverage two image captioning datasets for evaluation: MS-COCO (Lin al., 2014) and the Flickr-30K dataset (Young et al., 2014) (see Appendix D.2).
IN((n) - I1, -f1
- otherwise
(5)
where E word-embedding function (we use beddings from the Spacy package (Honnibal et al., 2020)), and Φ 0.1 is threshold.
Human Evaluation: To test the performance of model in real-world conditions, leverage human evaluations on the Amazon Mechanical Turk platform. We perform two styles of human evaluation. In "context-free" evaluation, raters are asked to rate two components: The "Helpfulness" of the caption to somebody who cannot see the image (on a scale of O to 4), and the factual "Correctness" of the caption (on a scale of 0 to 5). In "head-to toeaadd" evaluation, raters are presented with captions and asked which is more "helpful" for somebody who cannot see the image, as well as which is more factually "correct". Full details on the exact questions asked and the experimental design are given in Appendix F.
Figure 3 and Appendix G give some qualitative examples of our method compared to several baseline methods. We can see that descriptions using IC3 are often longer, and contain more detail than their counterpart baseline models. Further, most display uncertainty about the content of the image natural way, which the baselines are not able to accomplish (see Appendix C.2).
4.1 Human Evaluation
Recent works (Chan et al., 2022b; Caglayan al., 2020) have confirmed that human evaluation remains the gold standard for visual description evaluation, despite progress automated evaluation

---

Image siiens stting plote, possibly aticking our although it could on
Image Sortas
Image 203787
O IC' caño chocolote and
OFA, IC', AettAyour young
orange park anciosed pool at night.
OFA small figurines, possibly Stor Wars sittng
performing
getting ready pizzo couple figurines sitting lop fabie pizza
cream cake, pancokie, crope cako, or stock of pancokee
OFA: man naing skateboard up the side of romp.
Socratic Models: A peoceful scana of stote park in the city.
Models: group people enjoying delicious
OFA: coke sitingg on top of plate. Socratic Modela: voriety awwel treats on displyy of a delt.
pizzo at local pizzenia
Humani A person doing skateboard trick up bowl
Human: Two loys are sitting with a foy cup and real pizzo.
Human: A plate that has ciessert it.
Figure 3: Some qualitative examples of IC' paired with the OFA model. We can see in all of these cases that OFA IC' surfaces significantly more detail about the image, including both foreground and background details, as well as maintains the syntactic quality, relevant world-knowledh knowledge, and high-level details of the
caption.
Table 2: Human rater mean opinion score for IC on
Table 3: Ma-d-To-Head human evaluation performance of IC on hard MRR MS COCO splits.
MS-COCO. Helpfulness (H. 0-4), Correctuess 0-5).
CANDIDATE GENERATOR
BASELINE
+ +CC
MODEL
HELPFULNESS 1 CORRECTNESS
HI
C1
H
C1
BLIP +
48.06 %
41.78%
BIII
28.50%
30.43%
OFA
2.876
3.891
2.965
4.010
TIE
21.01 %
25.60%
BLIP
2.901
3.951
2.921
3.881
REFERENCES
2.932
3.966
2.985
3.985
OFA + IC²
51.10% %
48.90
OFA
32.04% %
29.52%
TIE
15.06%
19.78%
of image captioning. As discussed in subsection 3.5, we perform two experiments: head-to-head experiments and mean opinion score evaluation. The results of the head-to-head experiments on MS- COCO are shown in Table 1, where we can see that IC' augmented models significantly outperform the baselines on both helpfulness and correctness (Helpfulness: OI'A + 3 vs. OI'A, 0.0008; BLIP 3 vs. BLIP, p 0.008; BLIP2 IC vs. p p 0.003; REF IC vs. REF, REF,p-1.73e
Table 4: Human ruter opinion for IC Hard- MRR subsets. Helpfulness (H, 0-4), Correctness (C, 0-5).
CANDIDATE GENERATOR
BASELINE
IC
H1
C1
H1
C1
OFA HARD SUBSET
OFA REFERENCES
2,452 2.649
3.651
2.713
3.713
3.675
2.728
3.902
BLIP HARD SUBSET
Correctness: OFA IC vs. OFA p= 0.04288 BL.IP
BLIP REFERENCES
2.708
3.827
2.648
3.704
IC vs. BLIP, p= 0. 02008 BLIP2 + 3 vs. BLIP2, p= 0.898; REF + IC3 vs. REF, 0.
2.887
3.887
2.934
3.918
Table shows the performance of IC in terms of mean opinion score, and demonstrates that even in a calibration-free setup, where no extra evidence is presented, IC methods significantly outperform their baseline counterparts when rated for helpful- ness (Helpfulness: OFA + 3 p= 0.0237: BLIPP + IC = REF + IC3, = 0.0293; n 121). Numerically, I outperforms baselines on the correctness measure, however we found in all three cases that the difference was not statistically significant. We believe the the reduction in margin is caused by several effects: (1) without point of reference for the potential quality of the captions, AMT workers cannot tell which captions are deserving of high scores and (2) both OFA and
BLIP are strong captioning models, so a random sample of MS-COCO images may not contain difficult images that separate the two methods.
To investigate this hypothesis, we ran several additional human studies on a set of challenging examples, which we call the Hard MRR splits (see Appendix D.2), which contain the 200 most challenging images for CLIP to recall. We show the head-to head-to-headd to-head experiments in Table 3, and see that again, in head-to- to-head experiments, IC signif- icantly outperforms baseline methods (OFA IC p= 0.0225, 28, BLIP + I p 1=000074., 52). In MOS experiments (Table 4), IC augmented OFA and Reference captions both significantly outper- form their baselines (p < 0.05.n = 41) on both BLIP

---

Table 5: CLIP Recall for IC' augmented captions in the MS-COCO Dutaset (Karpathy Test Split). MRR: Mean Reciprocal Recall, R@K Recall @ K.
Table 7: Content coverage performance on IC² augmented captions the MS-COCO Dataset (Karputhy Test Split). N: Noun Recall, Verb Recall
MODEL
MRR
@@
R@5
@@ 10
EXACT
FUZZY
MODEL
N1
V1
Nf
V
REF + Ic²
0.776
0.691
0.883
0.930
REF
0.593
0.480
0.724
0.808
REF IC²
0.552
0.354
0.767
0.616
REF
0.255
0.137
0.567
0.398
OFA IC"
0.748
0.656
0.857
0.914
BLIP IC!
0.734
0.639
0.848
0.908
BLIP2 IC²
0.364
0.229
0.667
0.529
B1,P2 IC²
0.746
0.652
0.863
0.921
BLI++IC
0.353
0.223
0.663
0.534
OPA
0,586
0.472
0.717
0.798
OFA + IC
0.351
0.211
0.656
0.498
BLIP
0,501
0.382
0.634
0.736
BLIP2
0.277
0.185
0.582
0.442
BLIP2
0.589
0.473
0.725
0.811
BLIP
0.266
0.196
0.573
0,486
OFA
0.275
0.171
0.583
0.412
Table 6: CLIP Recall for IC captions in the Flickr-30K test set. MRR: Mean Reciprocal Recall, R@K: Recall @ K.
MODEL.
MRR
60 f
R 6
R 6
REF + IC3
0.856
0.836
0.836
0.938
Table Content coverage performance on IC" augmented
REF
0.708
0.679
0.679
0.798
captions in the Flickr-30K Test Dataset.
OFA IC.
0.806
0.782
0.782
0.889
BLIP + Ic
0.736
0.707
0.707
0.829
EXACT
FUZZY
OFA
0.658
0.629
0.629
0.745
MODEL
NOUNt
VERB
NOUN f
VERB
BLIP
0.499
0.463
0.463
0.581
REF I
0.548
0.350
0.763
0.684
REF
0.246
0.147
0.543
0.490
and OFA Hard MRR sets, but the experiments with BLIP on the BLIP Hard MRR set are inconclusive (p=0.682, == 41), suggesting that in some cases, IC's is unable to overcome all of the challenges with the seed captioning model. The fact that the head-to-head performance on the BLIP Hard MRR split in Table 3 is stronger for IC coupled with the fact that reference captions augmented with IC² perform better on this set suggests that IC² can manage some of the underlying noise, does not fully compensate for lack of calibration.
BLIP+IC'
0.283
0.200
0.604
0,585
OFA + IC?
0.296
0.195
0.607
0.571
BLIP
0.205
0.134
0.505
0,507
OFA
0.230
0.147
0.533
0.495
and search. On all datasets, IC3 outperforms single human reference captions, suggesting that summarizing multiple viewpoints is essential for strong automated recall performance.
Table 7 and Table 8 both demonstrate the summarization ability of IC3 augmented methods, as IC' outperforms all bascline methods in recalling content from the dataset, often by relatively large margins. The verb recall is often lower (though still improved) across all approaches, suggesting that IC focuses recalling content over action in image. We further quantify IC'ss summarization capability in Appendix C.5 where we find that increasing the diversity of the input candidates can improve noun/verb recall, however has little impact on MRR. These results suggest that IC² summarizes any salient information as required.
4.2 Automated Evaluation
As discussed in subsection 3.5, we also perform automated evaluations of the method on both the MS-COCO and Flickr-30K datasets. The perfor- mance of IC' in CLIP recall is first demonstrated in Table 5, where for MS-COCO, CLIP recall MRR is improved by 27.6% under OFA, and by 46.5% under BLIP, suggesting that IC² augmented captions significantly outperform SOTA captions in indexing scenarios. Similar improvements exist in Table 6, where IC² improves CLIP MRR by 22.49% for OFA and up to 84.46% for BLIP. These results suggest that IC3 surfaces significant additional detail compared individual baseline and reference sentences, leading to strong recall performance, and suggesting that IC augmented captions can lead to benefits when applied to indexing
While Appendix D.3 discusses the performance of our methods N-Gram measures, such measures are relatively misleading, as we generate captions that differ significantly from reference captions, thus, the N-Gram metries are naturally lower compared to maximum -ikeelhooodieooooddooooooo baselines.

---

5 Limitations
such as the Karpathy test set for MS-COCO and the Flicker-30K datasets (which consists of 5k images each). Using GPT-3, IC3 costs about $0.0 m0.01e9//mae, and with GPT-3.5, that cost falls to 50.001/lmage less than Chat Captioner (Zhu et al., 2023), which can cost as much as 50.27/1mage, which made it infeasible to run large-scale experiments. The ex- periments/ablatioos/aal GPT-3 tuning in this paper was performed for $250 (USD). Our approuch, while not necessarily cheap, is several orders of magnitude less expensive than training/evaluating fine-tuned vision and language models such as Flamingo (1536 TPUs/15 days, roughly $1,780,531 using on-demand TPU pricing) or BLIP-2 (16 A 100 GPUs, 6 days, $11,796 using AWS on-demand pricing). Furthermore, we hope that this cost will be prohibitive long-term. GPT-3.5 is an order of magnitude cheaper, and has similar performance GPT-3, and open-weight models such as Koala and Vicuna, seem promising for the future affordable LLMs (see Appendix C.I), making IC3 even more accessible to students and researchers.
While IC' significantly outperforms baseline captioning approaches, as well can outperform single human image captioning references, also suffers from several distinct limitations.
Notably, this is significantly
Hallucination: While IC3 often produces high-quality summaries of the associated captions, it has several distinct failure modes, mostly coming down to hallucinations induced by the underlying captioning model. In some cases, objects that are hallucinated by the model can propagate the summary, even if they are internally inconsistent with other captions in the candidate set K. Another distinct failure mode of the captions is when uncertainty in the samples is interpreted as two distinct parts of the image. For example, if 50% of the captions refer to dog, and 50% of the captions refer cat, the model may infer that both dog and are present in the image, even though only a single, unknown, animal is there. Examples of these failure cases are shown in Appendix I. We believe that such failure cases can largely be solved by introducing a visually aware summarization model, however, as of writing, no sufficiently large-scale general-purpose multi-modal model exists which is capable of serving this purpose.
Conclusion
In this work, introduce IC a method for image captioning that first samples captions from multiple viewpoints and then summarizes and filters them to create high-fidelity descriptions. As far we are aware, IC3 is the first work demonstrate pipeline for generating a single caption by integrat- ing distributionaall-faathh candidate captions, and does so without changing model architecture or retraining by leveraging summarization to produce single omnibus caption capturing the full distribu- tion of information. Further, IC is the first work for paragraph captioning image captioning that uses summarization of distributionally-faitte samples and the first to demonstrate in human experiments that long-form captions encoding this distribution are preferable to single reference captions. Human users rate IC3 captions at least as helpful as baseline captions up to 80% of the time, and such IC' captions are capable of inducing up to 84% relative improvements in recall approaches bascline captioning methods. While our implementation of IC is relatively simple, it demonstrates significant gains over traditional paradigms, suggesting that this only the beginning for caption sampling and summary methods.
Controllability: One of the key applications of image captioning systems alt-text generation. As discussed in recent work (Archibald, 2023), alt-text generation is largely contextual, which means that for each image, the alt-text of the image should depend on the context that such an image is included in. While IC' introduces natural pathway for including context through the summarization model, we have found (see Appendix I.2), that I is somewhat resistant to prompts that encourage surfacing background information. Exploring how to make IC3 surface arbitrary information in the image instead focusing primarily on foreground information key direction for future work.
caption
The use of many closed source large language models can represent significant financial, human, and environmental cost (Bender al., 2021). We recognize that for some researchers and students, the financial cost of using a large zero-shot model such as GPT-3 can be prohibitive, making IC' difficult to compare against, especially for large-scale experiments
The Cost of using LLMs:

---

References
IEEE 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Abhaya Agarwal and Alon Lavie. 2008. Meteor, M-BLEU and M-TER: Evaluation metries for high-coreen high-correlation with human rankings of machine translation output. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 115-118, Columbus, Ohio. Association for Computational Linguistics.
Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2020. Curious case of language generation evaluation metrics: A cautionary tale. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 2322-2322 Barcelona, Spain (Online). In- ternational Committee on Computational Linguistics.
Jean- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. ArXiv preprint, abs/2204. 14 198.
David M. Chan, Austin Myers, Sudheendra Vijaya- narasimhan, David A. Ross, Bryan Seybold, and John F. Canny, 2022a. What's 's in a caption? dataset- specific linguistic diversity and its effect on visual description models and metrics. In IEEE/CVF Con- ference on Computer Vision and Puttern Recognition Workshops, CVPR Workshops 2022, New Orleans, LA, USA, June 19-20, 2022, pages 4739447444540 4748, IEEE,
Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi As- sefi, Sacid Safaei, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut. 2017. Text summa- rization techniques: a brief survey. ArXiv preprint, abs/1707.02265
David M Chan, Yiming Ni, Austin Myers, Sudheendra Vijayanarasimhan, David A Ross, and John Canny, 2022b. Distribution aware metries for conditional natural language generation. ArXiv preprint, Ahs/2209.07518
Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexan- der G. Schwing. 2019. Sequential latent spaces for modeling the intention during diverse image captioning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV Seoul, Korea (South), October 27- November 2, 2019, pages 4260-4269. IEEE.
Moitreya Chatterjee and Alexander G Schwing. 2018. Diverse and coherent paragraph generation from images. In Proceedings of the European conference on computer vision (ECCV), pages 729-744.
Jake Archibald. 2023, Writing great alt text: Emotion matters.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergio- vanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. 2022. Pali: A jointly-scaled multilingual language- image model. ArXiv preprint, hts/2209.06994.
Yuntao Bai, Andy Jones. Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedhack. ArXiv preprint, abs/2204.0555 22.0666..
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. pages 610-623.
Bharath Chintagunta, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2021. Medically aware GPT-3 as data generator for medical dialogue summarivation. In Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations. pages 66-76, Online. Association for Computational Linguistics.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. If you use this software, please cite it using these metadata.
Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. 2017. Towards diverse and natural image descriptions via a conditional GAN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages Computer Society.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Newral Information Processing Systems
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexan- der G. Schwing, and David A. Forsyth. 2019. Fast, diverse and accurate image captioning guided by part- of-speech. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pages 10695-10704. Computer Vision Foundation / IEEE.

---

Zhiyuan Fang. Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang. Yezhou Yang, and Zicheng Liu. 2022. Injecting semantic concepts into end-to- end image captioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans. LA, USA, June 18-24, 2022, pages 1998-17998. LEEE.
Virapat Kieuvongngam, Bowen Tan, and Yiming Niu, 2020. Automatic text summarization of covid-19 medical research articles using bert and gpt-2. ArXiv preprint, abs/2006.01997.
Dong-Jin Kim, Jinsoo Choi, Tac-Hyun Oh, and So Kweon. 2019. Dense relational captioning: Triple- stream networks for relationship-based captioning. In IEEE Conference on Computer Vision and Patter Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6271-6280. Computer Vision Foundation IEEE.
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2017. Semantic compositional networks for visual captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1141-1150. IEEE Computer Society.
Franz Klein, Shweta Mahajan, and Stefan Roth, 2022. Diverse image captioning with grounded style. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28- October 1, 2021, Proceedings, pages 421 236. Springer.
Xinyang Geng, Arna Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. Blog post.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv preprimt, abs/2205. 11916.
Mark E Glickman. 1995. The glicko system. Boston University, 16:16-17.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, ArXiv preprint, abs/2209, 12356..
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for gener- ating descriptive image paragraphs. In 2017 IEEE Conference on Computer Vision and Puttern Recog- nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3337-3345, IEEE Computer Society.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras. and Yejin Choi. 2021. CLIPScore: reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tomer Levinboim. Ashish V. Thapliyal, Piyush Sharma, and Radu Soricut. 2021. Quality estimation for image captions based on large-scale human evaluations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 3157-3166, Online. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26- 30, 2020. OpenReview.nes
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spacy: cetusstiallstrenngeee natural language processing in python, zenodo. 2020.
Jiwei Li. Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Austin, Texas, Association for Computational Linguisties.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022. Promptcap: Prompt-guided guided task- image captioning. ArXiv preprint, abs/2211 11/22221.000 1.09699.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping anguage-image pre-training with frozen image encoders and large language models. ArXiv preprint, abs/2301.125977.
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2016. Densecap: Fully convolutional localization networks for dense captioning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. Las Vegas, NV. USA, June 27-30, 2016, pages 4574-4574. IEEE Computer Society.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping anguage-image pre-training for unified vision-language understand- ing and generation. In International Conference on Machine Learning, 1CML 2022, 17-23 July 2022. Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18888-22900 PMLR.
Jungo Kasai. Keisuke Sakaguchi, Lavinia Dunagan, Ja- cob Morrison, Ronan Le Bras, Yejin Choi, and Noah A. Smith. 2022. Transparent human evaluation for image captioning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, pages 3464-322 3478, Seattle, United States. Association for Computational Linguistics.
Xiangyang Li, Shuqiang Jiang, and Jungong Han. 2019. Learning object context for dense captioning. The Thirty-Thirl AAAI Conference on Artificial

---

Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educa- tional Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawait, USA, January 27- February 1, 2019, pages 8650-8657 8657. AAAI Press.
Sharan Narang and Aakanksha Chowdhery. 2022. Pathways language model (palm): Sealing to 540 billion parameters for breakthrough performance.
OpenAL. 2022. Introducing chatgpt.
OpenAL. 2023, Gpt-4 technical report.
Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P. Xing. 2017. Recurrent topic-transition GAN for visual paragraph generation. In IEEE International Conference on Computer Vision, 1CCV 2017, Venice, Italy October 22-29, 2017, pages 3382-3391. IEEE Computer Society.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalua- tion of machine translation. In Proceeding gs of the 40th Annual Meeting of the Association for Computational Linguistics, pages 31 1-318, Philadelphia, Pennsylva- nia, USA. Association for Computational Linguistics.
Chin-Yes Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 4, 2021, virtual, pages 48 1 6-4828.
Association for Computational Linguistics.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects context. In European conference on computer vision, pages 740-755. Springer.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nyya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICM1, 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research. pages 8748-88888888 8763.8773..73 PMLR.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. ArXiv preprint, 0os/2103.10388
Yadan Luo, Zi Huang, Zheng Zhang, Ziwei Wang, Jingjing Li, and Yang Yang. 2019. Curiosity-driven reinforcement learning for diverse visual paragraph generation. In Proceedings of the 27th ACM Inter- national Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019. pages 2341-2350.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Haley MacLeed, Cynthia I. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people's is experiences with -eeeerrte-eee captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA, May 06-11, 2017, pages 5988-5999. ACM.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67
Shweta Mahajan, Iryna Gurevych, and Stefan Roth. 2020. Latent normalizing flows for many-to-many -o-myny cross-domain mappings. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia. April 26-30, 2020. OpenReview.net.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv preprint, abs/1910.01108.
Shweta Mahajan and Stefan Roth. 2020. Diverse image captioning with context-object split latent spaces. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. ArXiv preprint, abs/221 1.05
Yuzhao Mao, Chang Zhou, Xiaojie Wang, and Ruifan Li. 2018. Show and tell more: Topic-oriented multi-sentence image captioning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 4258-44-808 4264. ojg.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.
Proceedings the 56th Annual Meeting of the Association for Computational Lin- guistics (oolume I: Long Papers), pages 2556-2565, Melbourne, Australia, Association for Computational Linguistics.
Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. ArXiv preprint, abs/2 11.0933..

---

Tianxiao Shen, Tao Lei. Regina Barzilay, and Tommi S. Jaakkola. 2017. Style transfer from non- poralalllll text by cross-alignment. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Nearal Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6830-6841.
Jiuniu Wang, Wenjia Xu. Qingzhong Wang, and An- toni B Chan. 2022a. On distinctive image captioning via comparing and reweighting. IEFF Transactions on Pattern Analysis and Machine Intelligence.
Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik. 2017. Diverse and accurate image description using variational auto-encoder with additive gaussian encoding space. In Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 5756-5766.
Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. 2017. Speaking the same language: Matching machine human captions by adversarial training. In IEEE International Conference on Computer Vision ICCV 2017, Venice, Itly. October 22-29, 2017, pages 4155-4164 IEEE Computer Society.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b. OFA: unitying architectures, tasks, and modalities through simple learning framework. Inter- national Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 233 13318-23340, PMLR.
Atability 2023. Stability ai launches the first of its stablelm suite of language models.
Abigale Stangl, Meredith Ringel Morris, and Danna Gurari. 2020. "person, shoes, tree. is the person naked?" what people with vision impairments want in image descriptions. In CHI 220: CHII Conference on Human Factors in Computing Systems, Honolulu, HI. USA, April 25-30, 2020, pages 1-13. ACM.
Qingzhong Wang and Antoni B. Chan. 2019. Describing like humans: On diversity in image captioning. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 4195-4203, Computer Vision Foundation IEEE.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An cestruction-followee //gi thgb. cooo//ttsu-laat lttaab//stafffrd_aapssss
Ilama model. https:
Qingzhong Wang, Jia Wan, and Antoni B Chan. 2020. On diversity in image captioning: Metries and methods. IEFE Transactions on Pattern Analysis and Machine Intelligence
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, al. 2022. Lamda: Language models for dialog applicatioon. ArXiv preprint, abs/2201. .08239.
Zhuhao Wang, Fei Wu, Weiming Lu, Jun Xiao, Xi Li. Zitong Zhang, and Yueting Zhuang. 2016. Diverse image captioning via grouptalk. In Proceedings of the Twenty-Fifth International Joint Conference on Ar- rificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 2957-2964 IJCAT/AAAI Press.
Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C.H. Hoi. 2022. Plug-and-play VQA: Shot VQA by conjoining large pretrained models with zero training. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 951-967, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. LingutationnaLLnnuititessssss
Yujia Xie, Luowei Zhou, Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, and Michael Zeng. 2022. Visual clues: Bridging vision and language foundations for image paragraph captioning. ArXiv preprint, abs/2206.01844 01843.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. ArXiv preprint. abs/2302.13971.
Linjie Yang. Kevin D. Tang, Jianchao Yang, and Li-Jia Li. 2017. Dense captioning with joint inference and visual context. In 2017 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Honolulu, HI, USA, July 21-26, 2017, pages 1978-1988 IERE Computer Society.
Emiel van Miltenburg, Desmond Elliott, and Pick Vossen. 2018. Measuring the diversity of automatic image descriptions. In Proceedings of the 27th Inter- national Conference on Computational Linguistios, pages 1730-1741, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Yang. Chongyang Gao, Hanwang Zhang, and Jianfei Cai. 2020. Hierarchical scene graph mecoder-decoder for image paragraph captioning. In MM '20: The 28th ACM International Conference on Mulrimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 4181-4189.
Ramakrishna Vedantam, C. Lawrence Zitmick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566-4575. IEBE Computer Society.
Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, and Jing Shao. 2019. Context and attribute

---

grounded dense captioning. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6241-6250. Computer Vision Foundation IEEE.
Peter Young. Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung, Mojtaba Scyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. ArXiv preprini, abs/2205.01917
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. 2022. Socratic models: Composing zero-shot multimodal reasoning with language. ArXiv preprint, abs/2204.00555
Zheng-Jun Zha, Daqing Liu, Hanwang Zhang, Yongdong Zhang, and Feng Wu. 2019. Context-aware visual policy network for fine-grained image captioning. IEEE transactions on pattern analysis and machine intelligence, 4402).710-722.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. 2023. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. ArXiv preprimt, abs/2303.06594.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08- 18-12,, 2018 pages 1097-1 100. ACM.

---

Appendix
C.1 Choice of Summarization Model
Appendix gives the acknowledgements for our work.
The choice of the summarization model S is a key decision when implementing IC'. Table A1 demonstrates the performance of IC with several models, both using prompting for large language models, and using summarization of the captions directly. Generally we found that models from OpenAT (Such as GPT-3 and GPT-4) were strong performers, however models from Anthropic (such as CLAUDE), have strong summarization performance as well. The strongest open-source models are Koala and Vicuna, both Chat-style models, with LLama and StableLM following, While Table A1 seems to imply that T-5 is strong model (and it likely is in terms of content-coverage in recall), T-5 often copy oapy-pastes several of the candidate sentences instead generating a strong abstractive summary, leading to decreased fluency.
Appendix B describes the code release, and links to available released resources associated with the paper.
Appendix C describes several explorations of the hyperparameters, including the language model, number of candidate samples, and prompts.
Appendix D describes additional expeirmental details including the image captioning models and datasets.
Appendix describes an ELO scoring system which we use in some additional appendix experiments.
Appendix F describes the human studies, and analysis of the human studies in detail.
C.2 Prompts
In this section, we present several explorations of possible prompts. First, Table A2, we present ploration of the prompt with and without language which encourages the model to produce uncertainty- specific language (the green text in subsection 3.4). To evaluate this, we use two approaches: a head-to-head experiment where captions generated by the two prompts are evaluated directly by human raters for helpfulness and correctness (following subsection 3.5), and automated measure of "likely-language occurrence", LLOP. To compute LLOP, we compute the number of captions contain- ing words that indicate some uncertainty including "likely", "probably", "possibly" and others. We find that without explicitly encouraging the model to produce uncertain language, the model seldom does while doing so improves both the helpfulness and correctness when measured by human raters.
Appendix G gives additional qualitative results for the method.
Appendix H explores how I can be used in zero- shot style and language transfer situations.
Appendix I describes some additional failure modes and limitations of IC in detail.
A Acknowledgements
Authors, as part of their affiliation with UC Berkeley, were supported in part by the Berkeley Artificial Intelligence Research (BAIR) industrial alliance program.
B
Code Release
Our code is available at https:///tthhi com/ daviddcchan//aptiion-----om ttee, and is made publicly available on Github with an MIT license, and contains the implementation. as well as the validation results for each of the models, the evaluation server/framework, and other necessary artifacts, to encourage further research in the domain of diverse/summarized image captioning.
In the second exploration in Table A3, we explore the question of using prefix similar to one explored in Kojima et al. (2022). We find that while the prefix does help. it is not a key component of our method, and increases automated measures by small, but perceivable, levels.
C.3 Choosing the number of candidate samples
C Hyperparameter Exploration
Choosing the number of captions to summarize is highly dependent on both the abilities of the lan- guage model, and the tolerance for exccution cost of the method. Adding more captions can increase
In this section, we provide additional experimental details regarding the choice of the hyperparameters for our method discussed in section 3.

---

Table A1: Exploration of the choice language model, when holding the prompt and candidate captions stable, using BLIP on 200-element randomly sampled subset of the MS-COCO dataset.
EXACT
FUZZY
RECALL
MODEL
NOUN
VERB
NOUN
VERB
MRR
R@1
R@5
LANGUAGE MODELS
IC' (SCAO AI.,, 2022)
0.248
0.16
0.551
0.402
0.834
0.725
0.98
0.995
IC + DISTILGPT2 (SANH ET AL., 2019)
0.208
0.146
0.517
0.488
0.643
0.535
0.795
0.825
IC + GPT2 (RADFORD ET AL., 2019)
0.272
0.159
0.602
0.542
0.638
0.51
0.79
0.83
IC GPT2 (RADFORD 2019)
0.28
0.164
0.583
0.486
0.735
0.64
0.85
0.89
+ GPT2 MED (RADFORD ET AL., 2019)
0.299
0.187
0.606
0.531
0.79
0.705
0.9
0.935
IC + GPT2 XL (RADFORD ET AL., 2019)
0.28
0.18
0.58
0.473
0.849
0.755
0.975
0.99
IC² GPT3 (BROWN ET AL., 2020)
ADA
0.282
0.18
0.585
0.463
0.866
0.78
0.975
0.985
BABBAGE
0.199
0.115
0,504
0.341
0.83
0.735
0.97
1.0
CURIB
0.218
0.111
0.519
0.319
0.827
0.71
0.975
0.995
DAVINCI2
0.321
0.207
0.622
0.491
0,939
0.895
1.0
1.0
DAVINCI3
0.381
0.251
0.675
0.547
0.958
0.925
1.0
1.0
IC' + GPTNEO 125M (BLACK AL., 2021)
0.235
0.157
0.521
0.447
0.777
0.69
0.895
0.915
IC¹ + GPTNEO IB (BLACKE ET AL.,, 2021)
0.253
0.155
0.546
0.403
0.844
0.75
0.985
0.995
IC¹ GPTNEO 2B (BLACK AL.,, 2021)
0.242
0.15
0.536
0.393
0.844
0.74
0.98
1.0
IC
(TOUVRON AI.,, 2023)
0.224
0.128
0.517
0.324
0.777
0.65
0.945
0.995
IC + LLAMA 13B (TOUVRON ET AL.,, 2023)
0.257
0.175
0.554
0.419
0.834
0.725
0.99
1.0
IC + STABLE LM 3B (STABILITYAL, 2023)
0.247
184
0.552
0.454
0.873
0.785
0.985
0.995
CHAT MODELS
IC + ALPACA 7B (TAORI ET AL., 2023)
0.324
0.216
0.63
0.503
0.912
0.85
1.0
1.0
IC' + CHATOPT (OPENAI, 2022)
0.401
0.27
0.692
0.595
0.954
0.920
1.0
1.0
IC CLAUDE (BALET 2022)
0.38
0.262
0.677
0.583
0.962
0.935
1.0
1.0
+ GPT4 (OPENAI, 2023)
0.42
0.29
0.713
0.609
0.96
0.925
1.0
1.0
IC + KOALA 7B (GENG ET AL., 2023)
0.284
0.178
0.586
0.455
0,899
0.825
0.99
1.0
IC KOALA 13B v1 (GENG AL., 2023)
0.418
0.323
0.692
0.637
0,916
0.865
0.985
0,985
+ KOALA
V2 (GENG ET AL., 2023)
0.376
0.264
0.67
0.592
0.923
0.87
0.995
0.995
IC + STABLE LM 3B (STABILITYAL, 2023)
0.077
0.06
0.299
0.144
0.265
0.23
0.28
0.295
IC² + STABLE LM 7B (STABILITYAL, 2023)
0.263
0.197
0.564
0.489
0.873
0.785
0.995
1.0
IC VICUNA 7B (CHIANG AL.,, 2023)
0.361
0.247
0.658
0.548
0.938
0.89
1.0
1.0
+ VICUNA 13B (CHIANG ET AI.,, 2023)
0,384
0.272
0.676
0.584
0.927
0.87
0.995
0.995
SUMMARY MODELS
IC² (RAFFEL AI.,, 2020)
0.353
0.25
0.646
0.587
0.903
0.845
0.98
0.99
IC + SMALL (RAFFEL AL.,, 2020)
0,402
0.289
0.681
0.609
0.944
0.9
0.995
1.0
BASELINES
+ REFERENCES
0.434
0.305
0.684
0.564
0.939
0.895
0.995
BLIP BASELINE (LIET AL., 2022)
0.266
0.196
0.567
0.491
0.865
0.77
0.995
1.0
CHAT CAPTIONER (T5-XXL + CHATGPT)
0.361
0.207
0.669
0.564
0.947
0.905
1.0
1.0
(ZHU ET AL., 2023)
Table A2: Exploration of "uncertainty--ecouraggii language in the prompt, using BLIP and GPT-3 on 200 element randomly sampled subset of the MS-COCO dataset. See Appendix C.2 for discussion of LLOP, the "likely-language occurrence percentage". Helpfulness and correctness are given head-to-head win percentage following subsection 3.5.
MODEL
LLOP HELPFULNESS
CORRECTNESS
CANDIDATES WITH CANDIDATES WITHOUT
62.5%
52.01%
72.32%
4.0%
43.63%
18.16%
REFERENCES WITH REFERENCES WITHOUT
52.0%
34.65%
53.00%
0%
28.79%
26.20%

---

Table A3: Content coverage and CLIP recall demonstrating the use of "This is hard problem" in the prompt, using BLIP on 200 element randomly sampled subset the MS-COCO dataset.
EXACT NOUN
FUZZY NOUN
CLIP RECALL
MODEL
VERB
VERR
MRR
R
R
@ 10
CANDIDATES WITH
0.322
0.216
0.647
0.503
0.770
0.645
0.96
0.99
CANDIDATES WITHOUT
0.316
0.208
0.638
0.496
0.765
0.635
0.94
0.99
REFERENCES WITH REFERENCES WITHOUT
0.516
0.308
0.744
0.560
0.833
0.745
0.97
0.995
0.518
0.305
0.745
0.558
0.830
0.745
0.97
0.99
vs candidate jummary clip recall mr
V candidate summary noun, recall
0.00
corsisate Suntrary 200R J0c4l
0.32
0.75
0.30
0.28
0.85
oeditatte
0.60
0.5
0.34
0.0
0.22
(a) K CLIP MRR
(b) Noun Recall
Figure A1: Exploration of the number of candidate captions CLIP MRR and Noun Recall for the GPT-3 language model.
the amount of information discovered by the visual model, and generate better representative samples of the input data distribution, however it can in- crease the context length passed to the large lan- guage model, straining the summarization capabil- ities of the model, and leading increased cost for the LLM. We ablate the choice in Figure A1. Here, we can see that increasing the number of captions can lead to increases in automated measure perfor- mance (as more captions will capture more infor- mation), however more captions can also increase execution time linearly with the number of candidate captions. We can see here that much of the benefit is captured 10 candidate captions, which we chose for this work, since it represents good raae-offf between execution time, and caption quality.
C.4 BLIP+ OFA
Because the caption summarization process is independent of the caption generation process, it is natural question ask if multiple different sources of caption generation could be used during the generation phase. The results of combining the sampled candidates from both BLIP and OFA are shown in Table A5
C.5 How
is caption diversity related to IC3 outputs?
One reasonable question to ask is: does the diversity of the input captions impact the quality of the output summarized caption? In Figure A2, we plot the Self- BLEU (Zhu et al., 2018) the candidate captions (a measure of caption-sed diversity), against the auto- mated evaluation measures. We find that in general, there are very weak correlations between the CLIP MRR and the diversity of the candidate set (OFA, 0.079, BLIP, 0.094, BLIP-2, 0.059): when more diversity is needed to express the content high specificity, the model is including it. When less diversity is required, the model does not include it. We do however see correlation between the content recall scores of the model, and the diversity
Is performance just due to LLMs correcting caption errors? Table Table A4 shows both the automated performance of the GPT-3 model, and the human performance of the GPT-3 model for several values of K. We can see that while GPT can help to improve single captions through error correction (as evidenced by slightly higher scores for GPT-3 (K=1) vs. the baseline), the best scores are achieved with higher values of K.

---

Table A4: Exploration of the choice of K for the GPT-3 language model and the BLIP-2 captioning engine on
randomly sampled 200 element MS-COCO subset. Human results are given as Glicko-2 scores (See Appendix E).
EXACT
FUZZY
CLIP RECALL
HUMAN GLICKO
K
NOUN
VERB
NOUN
VERB
MRR
R@1
R@5
BASELINE
0.264
0.162
0.564
0.423
0.885
0.805
0.985
1367.28
0.258
0.161
0,562
0.456
0.872
0.790
0.985
1534.48
10
0.346
0.212
0.646
0.516
0.956
0.920
1.000
1674.51
100
0.368
0.223
0.665
0.526
0.948
0.905
1.000
1623.22
Table A5: Content coverage and CLIP recall demonstrating the combination of caption engines on 200 element randomly sampled subset the MS-COCO dataset.
EXACT
FUZZY
CLIP RECALL
MODEL
NOUN
VERB
NOUN
VERB
MRR
R@1
R@5
R 10
OFA + IC
0.341
0.204
0.648
0.485
0.796
0.685
0.945
0.985
REFS IC
0.517
0.308
0.744
0.561
0.833
0.745
0.970
0.995
BLIP IC
0.313
0.206
0.637
0.493
0.770
0.645
0.960
0.99
IC
0,300
0.184
0.623
0.474
0.770
0.660
0,97
BLIP
0,230
0.178
0.542
0.439
0,551
0.400
0.91
OFA
0.212
0.150
0.531
0.387
0.341
0.115
0.630
0.89
REFS
0.214
0.537
0.099
0.337
0.683
0.540
0.965
of the input candidates (Noun Recall: OFA.
OFA: OFA (Wang al., 2022b) is unified paradigm for multimodal pre-training, which is both task and modality agnostic. For pre-training,
r 0.233, BLIP r 0.238, BLIP-2, r 0.252,
Verb Recall: OFA, r 0.199, BLIP 0.193,
BLIP-2, r 0.185). This suggests that when the candidates are more diverse, this information is captured in the output summary sentence.
OFA unifies several vision and language tasks including image generation, visual grounding, image captioning, image classification and lan- guage modeling among others, and is pre-trained using 20M publicly available image-text pairs,
D Additional Experimental Details
The OFA (Huge) model that use is fine-tuned on MS-COCO for image captioning, and unless otherwise specified, we generate baseline captions using beam search with 16 beams, and generate candidate captions for IC3 using temperature sampling as described in subsection 3.2.
D.1 Image Captioning Methods
this work, we explore two image captioning models as seed models for IC': BLIP (Li et al., 2022) and OFA (Wang et al., 2022b).
BLIP: BLIP et al., 2022) is vision-language pre-training framework designed to effectively noisy web data at scale for pre-training. The model operates by using a large dataset of synthetic image-caption pairs, generated by a seed captioning model, and a filter to remove low-quality synthetic captions. BLIP has demonstrated strong transfer performance to many vision-language tasks, and performs particularly well when transferred to image captioning in zero-shot and fine-tuning scenarios. The BLIP (Large) model that fine-tuned MS-COCO for image captioning. and unless otherwise specified, we generate baseline captions using beam search with 16 beams, and gen- crate candidate captions for IC3 using temperature sampling as described in subsection 3.2.
D.2 Datasets
We explore image captioning across several datasets in this paper.
MS-COCO Dataset: The MS-COCO dataset (Lin et al., 2014) dataset for image description containing 328K images, each with 5 ground truth descriptions. MS-COCO is licensed under a Creative Commons Attribution 4.0 license. All of the results in this work presented on the test-split of the Karpathy splits of the COCO-2014 dataset.
is
Flickr-30K Dataset: The Flickr-30K dataset (Young et al., 2014) is an image description dataset containing 30K images and 150K captions (5

---

Figure A2: Plots showing diversity of candidate captions plotted against automated evaluation measures when using 10 candidate captions, and GPT-3 (Davinci v3) as a LMM
ground truth captions per image), and is licensed un- a non-conneeoooooo commercial (for research use only) license. All of the results are presented on the test- split of the Karpathy splits of the Flickr-30K dataset.
different from the reference distribution.
E ELO Scoring for Human Ratings
The results shown in subsection 4.1 indicate a challenging reality: humans often find it difficult to calibrate to the quality of image captions when viewing single image captions alone, but can find much easier understand any differences in quality when presented with two pairs of captions. in a head to head fashion. Unfortunately, since human caption ratings can be expensive, it tricky to perform all head to head caption evaluations across values of K, language models, captions, etc. In order to compensate for this, in some situations instead of running the full head to head experiment, we instead use tournament, which measures the quality of model through an Glicko-2 score- ccore-based rating system (Glickman, 1995). some cases, we report the Glicko-2 scores of each of the models in our human-rating tournament, as proxy for the overall quality of the model.
Hard MRR Splits: In some situations, want to be able explore the performance of our model vs. baselines on the most challenging captions. We call these splits the "Hard MRR" splits, and they consist the set of 200 samples for which the MRR of the underlying captioning model lowest. Thus, HARD MRR - BLIP contains the 200 samples minimizing MRR for the baseline BLIP model, with a caption generated using beam search (16 beams), and sim- ilarly HARD MRR OFA contains the 200 samples minimizing MRR for the baseline OFA model with a caption generated using beam search (16 beams).
D.3 N-Gram Metric Scores
The performance of the model on traditional n-gram measures is demonstrated in Table A6. In this work, IC3 models are designed to produce captions which are the combination of all of the viewpoints presented by each of the individual captions, suggesting that they contain more information on average than any single reference sentence. Because of this, often the n-gram performance of the model is significantly worse,
F Human Studies
In our work, we run two different human rating stud- ies, a head-to-head comparison between methods, and a context-free method which generates mean opinion scores. A screenshot of our evaluation tool for mean opinion scores is given in Figure A11, and screenshot of the evaluation tool for head-to-heate rating is given in Figure A12. Both of these experiments have been approved as exempt under category by the SOUUTTED-FOR-REVIISS IRB, protocol ID 2022-11-15846 For any questions, contact OMITTED-FOR--EEVIE
while the overlap of content n-grams may be higher (suggested by Table and Table in the main paper), there are lot extra n- n-grams per caption, which will decrease metric scores. We explore four n-gram measure: BLEU (Papineni et al., 2002), CIDEr: (Vedantam et al., 2015), METEOR (A (Agarwal and Lavie, 2008) and ROUGE-L (Lin, 2004). We also compute the MAUVE (Pillutla et al., 2021) score between all samples generated and the reference samples, which measures the deviation in the language space, and notice that the MAUVE score is extremely low, suggesting that we have succeeded in producing a language distribution which is significantly
Significant prior work has explored the collection of human judgments of the quality of visual descriptions. Human judgment is considered the gold standard for visual description evaluation, and previous studies typically rely on human annotators to rate caption quality on one or multiple

---

Table A6; Performance of models augmented with IC on traditional N-gram measures.
MODEL
BLEU@41
CIDERT t ROUGE-L1 MAUVE
MS-COCO (KARPATHY SPLIT)
OFA +I?
0.159
0.495
0,483
0,091
ICC
0.118
0.325
0,445
0.074
0.292
1.323
0,598
0.254
0.292
1.315
0,595
0.158
FLICKR-30K (TEST SPLIT)
OFA + IC²
0.132
0.392
0,449
0,004
IC*
0.092
0.277
0,401
0.004
OFA
0.212
0.872
0,541
0.004
0.160
0.501
0.727
0.004
axes (Levinboim et al., 2021; Kasai et al., 2022).
head-to-head Caption Rating
Viewing given image, and a pair of captions, and deciding which caption better describes the image.
While automated methods exist for the evaluation of caption quality (Agarwal and Lavie, 2008;
Vedantam et al., 2015; Papineni et al., 2002), recent work including THUMB (Kasai et al., 2022), which has run human evaluations on captions produced by models based on "Precision", "Recall", "Fluency", "Conciseness" and "Inclusive Language", has shown that humans produce captions which score significantly higher when judged by human raters, than when judged by existing measures (and further, that human judgments of quality correlate poorly with existing measures), necessitating the need for human evaluation as opposed to evaluation of captioning methods using automated measures for caption quality. Our model quality evaluation method is closely aligned with the work in (Levin- similar questions
While there several possible activities, each session consists of sequences of the same type of activity, and each task is presented in randomized order for each subject.
Subjects are linked to the data collection interface on our server developed by us in frame directly from an Amazon Mechanical Turk internal HIT using the ExternalQuestion API which allows external web content to be displayed within the internal HIT. No third-party software is used with the HITs and no reviewing data is collected by Amazon or any third-parties with the use of this API,
The subjects are shown consent form on the Amazon Mechanical Turk HIT prior entering our data collection interface. Subjects then required to click the "I Accept" button confirm their agreement with the consent information of the study. They are then redirected to the data collection interface. For each image, users are presented with an image, and an associated image description. Images are drawn from the MSCOCO dataset (Lin et al., 2014). Human generated captions drawn from the references collected by the authors of (Lin al., 2014).
boim et al., 2021), where to determine the "helpfulness" and "correctness" of visual descriptions. Our work differs that instead collecting set human ratings of captions for the purpose of training statistical models, we aim to evaluate the quality of both human and machine generated captions, in an effort to determine if the machine generated captions from recently proposed methods in our group outperform existing human and machine generated captions for the images.
In this study, subjects participate sessions of reviewing visual media with corresponding visual descriptions, e.g. pairs of images and captions. These sessions consist of sequences of rating tasks, with session consisting of not more than ten tasks. The types of tasks, which we call activities, are as follows:
For task (Caption Rating), users are first asked to rate the "helpfulness" of the caption, with the prompt: "Does the caption provide useful information for person who cannot see the image (and who is trying to understand what is in the image)?", among the options: Not helpful all", "Slightly helpful", "Moderately helpful", "Helpful", " "Very helpful".. The user can also select the option can't 't tell". The user is then asked to rate the "Correctness" of the caption with the prompt "How
Caption Rating Viewing a given image and caption pair, and rating the quality the caption on several axes (described below).

---

correct is the caption?", among the options "Com- pletely wrong". "Mostly wrong" "Slighll wrong", "Slightly right", "Mostly right", "Completely right". The user can also select the option can't tell".
with an expected completion time of 1.5 minutes per session), and should be able to complete the session in under one and half minutes (based on several pilot examples). Subjects can participate in the task a maximum of 100 times. The maximum time commitment for each subject over two months of our study is 2 hours.
The user is then asked to select the "submit" button, move to the next task in the HIT, which is composed of 10 tasks. The user can also skip the task by selecting the "Image/Caption not visible button".. If the user selects "I can't tell" or "Image/Caption not visible" for any option, the tasks remaining are not decreased, but if the user selects submit, and passes a valid rating, then the number of tasks remaining are reduced by 1.
To compute p-values, we first aggregate each users' session scores for each model (in the case of MOS, we take the mean for cach model, and in the case of thed-tt-hhatte we assign a a++1 value for a win, and a -1 value for a loss, and take the mean). For MOS, we compute a --sided t-test on the aggregated samples (which should be independent) to the base- line, while for the head-to-head scores, compute I-sided single-sample t-test against mean of zero.
For task B (head-to-head Caption Rating), the user presented with two imgge captions instead of one image caption, and asked to select "Caption A better represents the content in the image" or "Caption B better represents the image". The user can also select "I can't t tell". The user is then asked to select the "submit" button, to move to the next task in the HIT, which is composed of 10 tasks. The user also skip the task by selecting the "Image/- Caption not visible button". the user selects "I can't tell" or "Image/Caption not visible" for any option, the tasks remaining is not decreased, but the user selects submit, and passes a valid rating, then the number of tasks remaining is reduced by 1.
G Additional Qualitative Examples
Additional qualitative examples are given in Figure A3, Figure A4, Figure A5, Figure A6 and Figure A7. From these examples, it is clear that IC outperforms the baseline in many situations. Examples this section are randomly selected from the test set when indicated.
After completing all of the tasks in the session, users given randomly generated code, which is entered in the Amazon MTTurk HIT page, and links the user's survey results the Amazon worker ID. We collect these linkings to perform analysis on inter-rater agreement, as while the session itself is anonymous, users may complete multiple sessions, and some method is required to maintain identity between the sessions.
After each of these sessions, subjects are given a brief survey regarding the task difficulty (Select from the options: "Very Easy", "Basy", "Normal", "Hard", "Very Hard") and prompted for any additional comments on the session in general for each session in an (optional) open-response format. Users are also encouraged to protect their privacy with the prompt: "After submitting your responses, you can protect your privacy by clearing your browser's history, cache, cookies, and other browsing data. (Warning: This will log you out of online services.)" Subjects were compensated with $0.18 USD per session (based on the recommended Amazon wage (federal minimum wage, $7.25/Hr).

---

EWIST
(a) BLIP+IC²: A plate with orange. crackers, lettuce, and possibly other items such nuts book. BLIP: A close up of plate of food on table.
(b) BLIP+IC": A man standing ten- nis court holding a tennis raequet, possi- bly wearing orange outfit raincoat. BLIP: man standing top of tennis court holding a racquet.
(c) BLIP+IC;: A woman riding brown horse and jumping over hurdles in competition, with other people watching.
woman riding the back of brown horse.
Figure A3: Additional qualitative examples of BLIP + IC on the MS-COCO dataset.
kriend
swa
A
(a) OPA+IC?: A woman in bikini jumping in the air to hit a volleyball on beach, possibly while playing game of beach ball.
(b) OFA+IC?: Two women in kimonas standing in front of an information board with umbrellas, possibly in the rain.
(c) OFA+IC?: A lacrosse player in white jersey running down the field with the ball during game match. OFA: A lacrosse player runs with the ball.
OFA: A woman in bikini jumping in the air to hit volleyhall.
OFA: Two women with umbrellas stand- ing front of an information board.
Figure A4: Randomly selected qualitative examples of OFA IC. the Flickr30K dataset.

---

(a) BLIP+IC:: A man in striped trunks riding surfboard on large wave near group of people.
(b) BLIP+IC': A young boy standing outside of building, possibly front a window or doorway, holding cell phone to his ear and wearing a red shirt. BLIP: A little boy standing outside of a building talking on a cell phone.
(c) BLIP++C': Two people, possibly hikers. sitting on top mountain. possihly icy or rocky, overlooking a snowy valley,
BLIP: A man riding wave on top of a surfboard.
BLIP: A couple of people sitting on top of a mountain.
Figure A5: Randomly selected qualitative examples of BLIP + IC? on the Flickr30K dataset.
CORY HATTER AVE
SEETE PUCKEER
NEW
Pizza A
Rolls G0
CARNELL PITTS AVE
(a) OFX+IC": A group of people sitting a bench under a tree, with four green street signs hanging from it. OFA: A group of people sitting on bench under tree.
(b) OFA+IC A plate of food on table with rice, heans, and possibly meat dish, such as chicken mashed potatoes.
(c) OFA+IC": A plate with two items of food on possibly sandwich and burrito, or two empanadas, sitting table counter.
OEA: A plate of food a table.
OFA: plate of food table.
Figure A6; Randomly selected qualitative examples of OFA IC on the MS- AS-COCO dataset.

---

VISIIAB
SOJA
122 In
Crlweb
BLIP+IC male tennis player wearing white, walking across tennis court while holding racquet. possibly after losing a match.
(b) BLIP+IC banana, bowl cereal, cup coffee sitting on a table counter.
(c) BLIP+IC:: man wearing either red or green white, holding lennis racquet and swinging it tennis ball on tennis court.
BLIP: banana sitting next a bowl cereal and cup of coffee.
BLIP: A man walking across tennis court holding a racquet.
BL.IP: man holding a tennis racquet on tennis court.
Figure A7: Randomly selected qualitative examples of BLIIP + IC on the MS-COCO dataset.

---

H Zero-Shot Style and Language Transfer
with toilet and sink and cat", and "A cat walking around in bathroom", and the LLM is not visually aware, there is no reason to doubt the existence of the cat, and it is included in the caption. Luckily, this failure case the model prefaces the existence of the cat with "may", however there are situations where this is not the case.
is well known that models such as GPT3 (Radford et al., 2021) are capable of many zero-shot tasks, such as language style transfer and translation. By modifying the prompt in the summarization approach, IC can be used to generate captions in different styles and languages. For example, can modify the prompt generate captions in different languages, for example, to generate captions in Japanese, we could use the prompt:
In Figure A9c, case of the model: treating uncertainty as multiple objects. Because the captioning model not awarc of the visual content of the image, when there a high amount of noise in the captions, such as here, where the actual contents of the plate are unclear, the model often ascribes the noise to several objects in the scene, instead of a single uncertain object. This sometimes be automatically detected by counting the number of commas in the caption, and we have found empirically that re-generating any caption with more than't 7 commas can reduce or eliminate these effects (though we do not use this prst-processing step in the paper).
the third major failure
This is a hard problem. Carefully summarize in Japanese in ONE detailed sentence the following captions by different (possibly incorrect) people describing the same scene. sure to describe everything, and identify when you' you're re not sure. For example: Captions: (formatted captions). Summary (in Japanese): TH t $ x 5
We see the performance of the model for such prompts in Figure A8. Such captions represent an easy way to transfer knowledge to different languages, however may not outperform translating the English caption alone.
1.2 Controllable Alt-Text Generation
While our model is capable of generating high- fidelity descriptions of the image, as discussed in section 5, the model can struggle when asked to describe background and contextual details that differ significantly from the reference dataset distribution. To demonstrate this, we perform a case study with the image in Figure A10.
I Failure modes & Limitations
In this section of the appendix, explore some of the limitations of the method, and provide some insight into how the model could be improved.
In the case study, we take the prompt:
This is a hard problem. Carefully summarize in ONE detailed sentence the following captions by different (possibly incorrect) people describing the same scene. Be sure to describe everything, and identify when you're not sure. For example: Captions: (formatted captions]. Summary: I'm not sure, but the image is likely of...
1.1 Hallucination
Some examples of failure cases are shown in Figure A9, In Figure A9a, we can see the effect of "4th-wall breaking," one of the key failure modes of the method. Because the prompt suggests that that the model should combine several underlying captions, the output caption references the fact of this combination in hidden way, when it says "other details varying". In some cases, the model might produce captions that end with words such as "... as stated by the captions" or ".. but the captions differ." " which both reference the prompt, and interfere with the flow of the caption.
and replace the colored prompt with a set of different prompts, to generate potential alt-text for the image in Figure 100. From these prompts, we can see that in many cases, the model fails to be controllable, and only repeats the key information in the scene. While in some cases the prompts can elicit additional information, we helieve there is significant work to be done to explore how we can sample enough relevant information from the base
In Figure A9b, we can see situation where the model passes through hallucination from the underlying captioning model. Because of the 10 captions in the candidate set K mention a cat: "A cat in a bathroom staring into a sink", "A bathroom

---

Stogrppiic
Figure A8: Examples generated caption (BLIP + Ic") for different languages:
ENGLISH: A woman riding a brown horse and jumping over hurdles in a com- petition, with other people watching. Una mujer montando caballo un caballo marrón mientras salta obstáculo campo verde, posiblemente una competición con espectadores mirando.
SPANISH:
FRENCH: Quelqu' un qui monte à cheval sur le dos d' un cheval brun et saute par-dessus un obstacle dans un champ, avec des gens arriere-plan.
JApanese: t T
(a) Generated BI,IP+IC² Caption: sitting at desk in front of least computer, possibly two, with other details such as clothing and accessories varying.
(b) Generated BLIP+IC? Caption: A small bathroom with white toilet, sink, counter, and possihly marble tile floor, and there may be cat present.
(e) Generated BLIP+IC. Caption: A white plate topped with sandwich and cup of coffee, possibly accompanied by other food items such as french toast and/or meat.
Figure A9; Examples of BLIP IC failure modes (MS-COCO Dataset).
captioning model (it seems that increasing the num- ber of samples is not always sufficient), and surface that information in the generated summary (simple modifications to the prompt seem ineffective).
Prompt: "Focus on describing the buildings" Result: A woman riding a bicycle past some
buildings, possibly hotel, with gate in the background.
Prompt:
"Only describe the buildings" A person riding a bicycle past a tall building.
The results for BLIP are below: Prompt: With original prompt
Result:
Result: A woman riding bicycle past a building
with cafe, bar, hotel, and/or restaurant, possibly with fence nearby.
"do not mention any people in the summary"
Prompt:
Result: A person riding a bicycle down street past buildings.
Prompt: "Focus on describing what the person is doing"
Prompt: "Focus describing the color and architecture of the buildings"
Result: person riding bicycle past a building or establishment of some kind.

---

Cofe Sar. HataH
Figure A10: Photo by Zhang Kaiyv'. Alt-text is often contextual. From the reference: "If [the image] is used in arti- cle about a public bicycle hire scheme named Hire-a-bike, then the bike is the focus, and the alt text could be "A woman rides a Hire-a-bike along city road". However, it's used in an article about dispute between the café and the restau- rant, the alt text might be "The storefronts of the 'Café Bar Hotel' and Alpen Hotel Restaurant", Although if the dis- pute about color of the buildings, the guttering, or the window frames, then those details should be part of the alt text."
Result: A person riding bike past hotel with colorful wooden shutters and other buildings nearby.
Result: A person riding bicycle in front of building, fence, or restaurant.
Prompt: "Focus on describing what the person is doing" "I'm not sure, but the image is likely of a person, who is"
Prompt: "Focus on describing what the woman is wearing"
Result: woman wearing casual clothing riding a bike past two colorful buildings.
Result: Possibly a woman, riding a bicycle on street in front of building, fence, metal gate, restaurant, hotel, or other structures in various cities such Tallinn, Vienna, Albany, Alphen aan den Rijn, Berlin, Strasbourg, Frankfurt am Main.
Prompt: "Make sure describe the appearance of the buildings in detail"
Result: person riding bicycle in front of building, which could an old hotel, cafe, apart- ment building, restaurant, or other type of building with many windows and possibly a fence or gate. Prompt: "Make sure to describe the person's appearance in detail"
Prompt: "Make sure include details about what the person is wearing and doing"
Result: A woman riding her bicycle past tall
fence, possibly with restaurant behind it, in front of a building or hotel in a city such Kitzingen, Albéliyah, Breckenridge. Frankfurt, Innbruck. Bad Schwartau..
Result: A person, possibly woman, riding a bicycle past buildings, fence, restaurant, and a hotel.
Prompt: "Make sure describe the person's appearance in detail, such what they are wearing and doing"
Prompt: "Make sure to only include details about what the person is wearing and doing"
Result: A woman riding a bicycle on city street in front of a building, fence, or restaurant.
Result: woman in a floral dress riding a red bike past an old, historic building with a restaurant across the street, while an older man is riding bi- cycle past several other Ander-fashioned buildings. The results for OFA are below:
Prompt: With original prompt
Result: woman riding bicycle in front of building, fence, or restaurant a variety of cities in Europe.
"do not mention any people in the summary"
Prompt:
Result: A person riding bicycle in front of a building, fence, or restaurant.
Prompt: "Focus on describing what the person is doing"

---

Description Rating Tool
Instructions: Imagine that we would like write text description of image below so that somebody who cannot the image understand the Look the image and the description, then answer the questions below rate the description's 's helpfulness and correctness. Make press "Image/Caption not visible".
all of the questions. If you can't the image caption,
HIT Images Remaining: 12
LA2
Image Caption: old rowboat beached on coastline near an ocean or large body water, possibly at sunrise.
Helpfulness: Does caption provide useful information for person cannot see the image (and who is trying to understand what is in the image)? The caption is...
Not helpful all
Slightly helpful
Moderately helpful
Very helpful
Helpful
can't tell
Correctness: How correct the caption? The caption is...
Completely wrong
Mostly wrong
Slightly wrong
Slightly right
Mostly right
Completely right
can tell
Image/Caption Not Visible
Submit
Figure All: Description rating tool (Mean Opinion Scores).

---

Description Rating Tool
Instructions: Imagine that we would like to write text description of the image below so that somebody who cannot see image can understand scene. Look image and description, then answer the questions below to rate the description's helpfulness and correctness. Make sure to answer all of questions. you can't see image or caption, press "Image/Caption not visible".
Images
JAL
Image Caption A Japan Airlines plane taking off from an airport, possibly in Japan, with mountains in the background. Image Caption A plane taking off from runway with mountains the background.
Which caption provide more useful information for person who cannot see the image (and who trying understand what is the image)?
Helpfulness:
Caption
Caption
Both are the same
can't 't tell
Correctness: Which caption more factually accurate? The caption is...
Caption A
Caption B
Both are the same
can't tell
Image/Caption Not Visible
Submit
Figure A 12: Description rating tool (head-to-hcad),


Reviews:
[{"review_id": "5ycuWA4pyz", "reviewer": "Reviewer_9GXH", "summary": "", "questions": "What is the improvement of IC3 on top of the recent image captioning model, such as BLIP-2?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments, and that the reviewer highlights our simple and exciting approach to generating fine-grained image captions, and the detailed experiments that we perform to demonstrate their value. We'd like to take this opportunity to respond to some of the weaknesses and questions discussed by the reviewer:\n\n**Performance on BLIP-2/SOTA Models:** We provide the results on BLIP-2 (both the automated measures and an additional human head to head evaluation) on the Karpathy Test Set of MS-COCO below, and will include them in the camera ready version of the paper. The results for BLIP-2 are in line with BLIP-1 and OFA; humans prefer IC3 augmented captions in terms of helpfulness, and find them as correct as BLIP-2 captions (no regression in correctness), and IC3 results in significantly improved recall over baselines.\n\n_Human Evaluation (N=108, p=0.003 (helpfulness), p=0.898 (correctness)):_\n\\begin{array}{lcc}\n\\textbf{Method} & \\textbf{Helpfulness} & \\textbf{Correctness} \\\\\\\\\n\\text{BLIP-2 + IC3} & 51.97\\\\% & 44.10\\\\% \\\\\\\\\n\\text{BLIP-2} & 37.49\\\\% & 42.67\\\\% \\\\\\\\\n\\text{Tie} & 9.78\\\\% & 11.6\\\\% \\\\\\\\\n\\end{array}\n\n_Automated Measures:_\n\\begin{array}{lcccccccc}\n\\textbf{Method} & \\textbf{Noun Recall} & \\textbf{Verb Recall} & \\textbf{Fuzzy Noun} & \\textbf{Fuzzy Verb} & \\textbf{MRR} & \\textbf{R@1} & \\textbf{R@5} & \\textbf{R@10} \\\\\\\\\n\\text{BLIP-2 + IC3} & 0.364 & 0.229 & 0.667 & 0.529 & 0.746 & 0.652 & 0.863 & 0.921 \\\\\\\\\n\\text{BLIP-2} & 0.277 & 0.185 & 0.582 & 0.442 & 0.589 & 0.473 & 0.725 & 0.811 \\\\\\\\\n\\end{array}\n\nIn the case where real human captions are used instead of captioning model output, we still demonstrate in both human studies and automated measures that IC3-summarization outperforms baseline references. In addition, we provide a further head-to-head investigation of references vs. references + IC3 on the Karpathy Test Set of MS-COCO  (N=108, p=1.732e-5 (helpfulness), p=0.0019 (correctness)):\n\n\\begin{array}{lcc}\n\\textbf{Method} & \\textbf{Helpfulness} & \\textbf{Correctness} \\\\\\\\\n\\text{Ref + IC3} & 55.79\\\\% & 48.8\\\\% \\\\\\\\\n\\text{Ref} & 36.65\\\\% & 36.27\\\\% \\\\\\\\\n\\text{Tie} & 7.46\\\\% & 13.97\\\\% \\\\\\\\\n\\end{array}\n\n**Method is primarily prompt engineering:** We want to emphasize that there are two key lessons from this work (i) that there is more useful information in _collections_ of human captions than in a single caption (the “C” for Committee in the paper title), and that (ii) optimizing a distributionally-faithful baseline model generating captions at temperature near 1.0 reproduces this human diversity, allowing an LLM to integrate this information into a better single caption. Furthermore, beyond the summarization stage, we pioneer the use of distributional metrics (TRM-based in section 3.2) to allow optimization of the first-stage captioning model for the IC3 application. \n\nIn summary, as far as we are aware:\n- No existing work demonstrates a pipeline for generating a single caption by integrating distributionally-faithful candidate captions. IC3 does this without changing model architecture or retraining by leveraging summarization to produce a single omnibus caption capturing the full distribution of information.\n- No existing work for paragraph captioning or image captioning uses summarization of distributionally-faithful caption samples.\n- No existing work demonstrates in human experiments that long-form captions encoding this distribution are preferable to single reference captions.\n\nWe will clarify these details in the camera ready version of the paper by expanding our discussion in Sections 1, 2 and 3 to include a more thorough description of the motivation/approach. If space allows, we will add an additional figure to demonstrate the key technical motivation of our approach  to complement Figure 2 (the method figure).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a simple but exciting approach to generating fine-grained image captions using OpenAI GPT APIs. They conducted detailed experiments to show the values of generated captions on both human and automated evaluations. The description rating tool is potentially helpful for similar tasks.", "reasons_to_reject": "The approach is prompt engineering and lacks comparison to LLM-based image captioning models such as BLIP-2.", "questions_to_authors": "What is the improvement of IC3 on top of the recent image captioning model, such as BLIP-2?", "paper_topic_and_main_contributions": "This paper uses a prompt engineering approach to generate a summarized long caption from short captions generated by human or image captioning models (BLIP and OFA) using OpenAI GPT APIs. The authors showed that the summarized captions are better than the ones from human, BLIP, and OFA on both human evaluation (AMT) and automated evaluation.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HIP6KGeuDz", "reviewer": "Reviewer_k3RY", "summary": "", "questions": "Suggestion:\n1. Adjust the structure of the article. In terms of content, it is very substantial, and the author has analyzed it from multiple perspectives but hopes to highlight the key points.\n2. Highlight theoretical highlights. Conduct more theoretical analysis and improve Figure 2 (or add a framework diagram to introduce the model).\n3. Supplementary experiments. Add experiments on some cutting-edge indicators in the current image captioning field, as mentioned earlier. Unlike many articles, some commonly used indicator experiments are in the appendix, and it is recommended to provide corresponding supplementary introductions, which improve the readability of this article.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s detailed comments and that the reviewer highlights our simple and performant approach and detailed evaluations. We'd like to take this opportunity to respond to some of the weaknesses and questions discussed by the reviewer:\n\n**Lack of Comparison on Common Indicators (BLEU/METEOR/CIDEr/ROUGE)/Mainstream Image Captioning Models:** As we discuss in L572-L577 of the paper, and in detail in Appendix C.3, traditional “n-gram” based measures such as BLEU(1/4), METEOR, CIDEr, and ROUGE when applied using standard references caption sets are biased towards candidate captions which are impoverished, favoring only “safe” descriptions which are shared among many references. When evaluated using more task- or human-centered metrics, IC3 captions perform better. We argue that this highlights a weakness in those metrics. We nevertheless included those metrics in appendix C3. \n\nIn terms of models, the IC3 meta-approach should be transferable to any distributionally-faithful image captioning method (which we validate with TRM measures in Section 3.2). According to the official MS-COCO online benchmark (https://competitions.codalab.org/competitions/3221#results), OFA (discussed in the paper) is ranked #2 only behind GIT which is not publicly available. We also will provide additional results on BLIP-2 (a recent model outperforming BLIP) on the Karpathy Test Set of MS-COCO in the camera ready paper:\n\n_Human Study (N=108, p=0.003 (helpfulness), p=0.898 (correctness)):_\n$$\n\\begin{array}{lcc}\n\\textbf{Method} & \\textbf{Helpfulness} & \\textbf{Correctness} \\\\\\\\\n\\text{BLIP-2 + IC3} & 51.97\\\\% & 44.10\\\\% \\\\\\\\\n\\text{BLIP-2} & 37.49\\\\% & 42.67\\\\% \\\\\\\\\n\\text{Tie} & 9.78\\\\% & 11.6\\\\% \\\\\\\\\n\\end{array}\n$$\n\n_Automated Measures:_\n$$\n\\begin{array}{lcccccccc}\n\\textbf{Method} & \\textbf{Noun Recall} & \\textbf{Verb Recall} & \\textbf{Fuzzy Noun} & \\textbf{Fuzzy Verb} & \\textbf{MRR} & \\textbf{R@1} & \\textbf{R@5} & \\textbf{R@10} \\\\\\\\\n\\text{BLIP-2 + IC3} & 0.364 & 0.229 & 0.667 & 0.529 & 0.746 & 0.652 & 0.863 & 0.921 \\\\\\\\\n\\text{BLIP-2} & 0.277 & 0.185 & 0.582 & 0.442 & 0.589 & 0.473 & 0.725 & 0.811 \\\\\\\\\n\\end{array}\n$$\n\nIn the case where real human captions are used instead of captioning model output, we still demonstrate in both human studies and automated measures that IC3-summarization outperforms baseline references. In addition, we provide a further head-to-head investigation of references vs. references + IC3 on the Karpathy Test Set of MS-COCO (N=108, p=1.732e-5 (helpfulness), p=0.0019 (correctness)):\n\n$$\n\\begin{array}{lcc}\n\\textbf{Method} & \\textbf{Helpfulness} & \\textbf{Correctness} \\\\\\\\\n\\text{Ref + IC3} & 55.79\\\\% & 48.8\\\\% \\\\\\\\\n\\text{Ref} & 36.65\\\\% & 36.27\\\\% \\\\\\\\\n\\text{Tie} & 7.46\\\\% & 13.97\\\\% \\\\\\\\\n\\end{array}\n$$\n\n**Lack of Theoretical Reasoning:** Our paper uses novel theoretical tools introduced in prior work, namely sparse distributional metrics, and applies them to the practical problem of caption generation. Since the implementation and performance of those metrics was described in another paper, we did not repeat it here. However, those metrics have not previously been applied to captioning, and that requires a robust summarization method, which  is the subject of the present paper. We want to emphasize that there are two key lessons from this work (i) that there is more useful information in _collections_ of human captions than in a single caption (the “C” for Committee in the paper title), and that (ii) optimizing a distributionally-faithful baseline model generating captions at temperature near 1.0 reproduces this human diversity, allowing an LLM to integrate this information into a better single caption. Furthermore, as mentioned previously, we pioneer the use of distributional metrics (TRM-based in section 3.2) to allow optimization of the first-stage captioning model for the IC3 application. \n\nIn summary, as far as we are aware:\n- No existing work demonstrates a pipeline for generating a single caption by integrating distributionally-faithful candidate captions. IC3 does this without changing model architecture or retraining by leveraging summarization to produce a single omnibus caption capturing the full distribution of information.\n- No existing work for paragraph captioning or image captioning uses summarization of distributionally-faithful caption samples.\n- No existing work demonstrates in human experiments that long-form captions encoding this distribution are preferable to single reference captions.\n\nWe will clarify these details in the camera ready version of the paper by expanding our discussion in Sections 1, 2 and 3 to include a more thorough description of the motivation/approach. If space allows, we will add an additional figure to demonstrate the key technical motivation of our approach  to complement Figure 2 (the method figure).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Although the author didn’t compare the core indicators in the experiment, they also conducted many experiments to prove their performance from other perspectives.\n2. Compared to other single-viewpoint methods, the author's committee consensus method, although relatively simple, has good results.", "reasons_to_reject": "1. I personally believe that the entire article seems to have been improved and summarized from practical applications. There is relatively little theoretical reasoning in the article, which does not reflect the innovation in the theoretical aspect. And the content of Figure 2 is not concise and clear, it is recommended to supplement the Framework diagram to introduce the model.\n2. The biggest problem with the experiments is that although the model proposes a method based on a large model, general indicator analyses such as BLEU-1, BLEU-4, METR, and CIDEr are not shown in the experimental section for Image Captioning tasks. Moreover, no BLIP has online test scores based on the MSCOCO dataset, which are not mentioned in the article. At the same time, there is also a lack of performance comparison with mainstream Image Captioning models in recent years, and the baseline performance of BLIP is not the best in Image Captioning. Without these key experiments and indicators, it is impossible to show the innovation and progressiveness of the methods in the field of Image Captioning.\n3. Through Rebuttal, the author provided some supplementary explanations for Weakness 2 (experimental design), but objectively speaking, the readability of this article is somewhat lacking, and some essential details are not handled carefully enough.", "questions_to_authors": "Suggestion:\n1. Adjust the structure of the article. In terms of content, it is very substantial, and the author has analyzed it from multiple perspectives but hopes to highlight the key points.\n2. Highlight theoretical highlights. Conduct more theoretical analysis and improve Figure 2 (or add a framework diagram to introduce the model).\n3. Supplementary experiments. Add experiments on some cutting-edge indicators in the current image captioning field, as mentioned earlier. Unlike many articles, some commonly used indicator experiments are in the appendix, and it is recommended to provide corresponding supplementary introductions, which improve the readability of this article.", "paper_topic_and_main_contributions": "For most \"single-viewpoint\" captions methods, this article proposes a relatively simple, committee consensus image captioning method, which performs well in some artificial experiments. This manuscript has a lot of content and has designed many experiments. However, this article lacks experiments on core indicators, and it is also recommended to supplement more theoretical analysis.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The many aspects of this article are very substantial, but there are still some shortcomings in the experimental setup and structural layout. I personally believe that these works will also be helpful to the author, which can refer to the design of these articles in terms of experiments and chapter layout:\n\n[1] Adaptive path selection for dynamic image captioning\n[2] Dual global enhanced transformer for image captioning"}}, {"review_id": "gBy0GE1gOX", "reviewer": "Reviewer_5Y4s", "summary": "", "questions": "Q1. What is the size of the dataset used for human evaluation?\n\nQ2. How many subjects participated in the human evaluation?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments, and thorough understanding of the paper and the appendix, and that the reviewer highlights our well thought out approach and detailed evaluations. We'd like to take this opportunity to respond to some of the weaknesses and questions discussed by the reviewer:\n\n**LLM Controllability/Hallucination:** While we thoroughly discuss the limitations of our implementation (Sec 6) in terms of hallucination in appendix H.1 and controllability in Appendix H.2, it’s important to distinguish controllability of our method vs the current implementation of it with specific languages models. Both our caption-generating VLM and our summarizing LLM admit prompts which we optimized. So arbitrary controls/direction can be passed to those models. The fact that the models do not always follow such instructions, and regularly hallucinate is a well-known challenge of current LLMs and is the subject of a great amount of research on instruction-tuning, RLHF, and other methods. As hallucination and instruction-following performance of the base models improves, we should inherit similar improvements in IC3. \n\n**Human Evaluation Questions:** While the setup varies slightly between experiments, in general we use the 5000 samples from the MS-COCO karpathy test set (or the 200 sample “hard” subset of that constructed according to our discussion in appendix C.2). The number of subjects is different for each experiment (given by $n$ values in Section 4.1), but ranges from 41 to 121 subjects.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Following are the strengths of the paper -\n\n1. The paper is well written and easy to follow.\n\n2. The approach is very well thought and backed by a detailed human evaluation study which validates it.\n\n3. The automatic evaluation results also support that the IC3 approach significantly outperform SOTA captions.\n", "reasons_to_reject": "Following are the weaknesses of the paper -\n\n1. Since the approach uses LLM's, hence it will also suffer from its weakness like hallucinations. \n\n2. The model isn't controllable and fails to describe the background or contextual information.\n\n3. Since the model isn't controllable, it might not generalize to the real world scenarios.\n", "questions_to_authors": "Q1. What is the size of the dataset used for human evaluation?\n\nQ2. How many subjects participated in the human evaluation?\n", "paper_topic_and_main_contributions": "This paper introduces a simple, yet novel method - “Image Captioning by Committee Consensus” which is designed to generate a single caption that captures details from multiple viewpoints by sampling from the learned semantic space of a base captioning model, and also leveraging a LLM to synthesize these samples into a single comprehensive caption.\n\nThe key contributions of the paper are –\n\n1. An approach to leverage pretrained image captioning models, and LLM’s to generate semantically complete image captions from a collection of “single-viewpoint” captions.\n\n2. Human evaluation of the method demonstrates that captions generated by IC3 are rated higher in helpfulness (without image context) and correctness.\n\n3. Several automated measures also show that captions generated by IC3 contain significantly more semantic information than baseline captions.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rRhjviWIjk", "reviewer": "Reviewer_WL9Q", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments, and that the reviewer highlights our simple, fluid and exciting approach to generating image captions, and the human evaluation experiments that we perform to demonstrate their value.  \n\nWhile some may argue that the approach lacks overall novelty, we strongly believe that the technical contribution is significant. We would like to highlight that as far as we are aware:\n- No existing work demonstrates a pipeline for generating a single caption by integrating distributionally-faithful candidate captions. IC3 does this without changing model architecture or retraining by leveraging summarization to produce a single omnibus caption capturing the full distribution of information.\n- No existing work for paragraph captioning or image captioning uses summarization of distributionally-faithful caption samples.\n- No existing work demonstrates in human experiments that long-form captions encoding this distribution are preferable to single reference captions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While a simplistic approach overall (wisdom of the crowd), the detailed presentation of the model will be beneficial to the controllable image captioning community as a whole. While some papers have looked at slightly similar approaches (as ackknowledged by the authors), this model brings togeter a number of key issues in a fluid way. The human evaluation is also positive. ", "reasons_to_reject": "Some might argue that the approach lacks overall novelty, but I feel that this is outweighed by the quality of some of the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a relatively simple idea. Generate captions in the traditional way for a given image. Then fold those captions together into a single coherent and detailed caption through the use of what is essentially an LLM summarisation algorithm. While this is a simple idea, it is a good idea and the authors present a nice overview of the approach and detailed human evaluation of the outputs achieved. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple method to generate a single coherent and detailed image caption through the use of LLM to summarize outputs from multiple generated short image captions. The authors have done a nice job of rebuttal.

In general, 3 reviewers are positive about the paper, and one reviewer is slightly negative about the paper. Reviewers found that (1) the proposed method is simple and exciting; (2) experiments are comprehensive with detailed human evaluation study; (3) the paper is well written. One reviewer shared the concern that the standard evaluation metrics like BLEU/CIDEr scores are not reported and there is a lack of comparison with most SoTA image captioning models. The AC thinks that this is not a major concern, since the proposed method aims to provide a detailed comprehensive caption, rather than a short caption that aims to chase SoTA CIDEr score. Overall, given the reviews, the AC is positive about the paper.
Decision: Accept-Main