Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning
Authors: Yujia Xie
Luowei Zhou
Xiyang Dai
Lu Yuan
Nguyen Bach
Ce Liu
Michael Zeng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Image paragraph captioning
vision-language models
zero-shot learning

Abstract: People say, "A picture is worth a thousand words". Then how can we get the rich information out of the image? We argue that by using visual clues to bridge large pretrained vision foundation models and language models, we can do so without any extra cross-modal training. Thanks to the strong zero-shot capability of foundation models, we start by constructing a rich semantic representation of the image (e.g., image tags, object attributes / locations, captions) as a structured textual prompt, called visual clues, using a vision foundation model. Based on visual clues, we use large language model to produce a series of comprehensive descriptions for the visual content, which is then verified by the vision model again to select the candidate that aligns best with the image. We evaluate the quality of generated descriptions by quantitative and qualitative measurement. The results demonstrate the effectiveness of such a structured semantic representation. 

Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning
Yujia Xie, Luowei Zhou', Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, Michael Zeng
Microsoft (yujiaxie, luowei. zhou, xiyang.dal luyuan, nguyenbach, ce liu, nzeng/microsoft.com
Abstract
People say, "A picture is worth a thousand words". Then how can we get the rich information out of the image? We argue that by using visual clues to bridge large pretrained vision foundation models and language models, we can do so without any extra cross-modal training. Thanks to the strong zero-shot capability foundation models, we start by constructing a rich semantic representation the image (e.g., image tags, object attributes locations, captions) structured textual prompt, called visual clues, using a vision foundation model. Based on visual clues, we use large language model produce series of comprehensive descriptions for the visual content, which is then verified by the vision model again select the candidate that aligns best with the image. We evaluate the quality of generated descriptions by quantitative and qualitative measurement. The results demonstrate the effectiveness of such structured semantic representation.
Introduction
This image is of a family celebrating Christmas. They are all gathered around a dinner table, with turkey and other food on The family is smiling and seems to be enjoying themselves. There is Christmas tree in the background and some Christmas lights on the walls.
This image features a horse and buggy travelling down a road in the town of Holguin, Cuba. The horse is harnessed to the buggy and is pulling along, while a farmworker rides behind. This image captures the everyday life of Cubans, with their traditional horse-drawn carts still use.
Figure I:
Examples of generated image paragraph.
"Vision is a process that produces from images of the external world description that is useful to the viewer and not cluttered with irrelevant information." -David Marr, Vision, p31
What makes good "description" for vision? Over the past several decades, computer vision pioneers drew inspiration from neural science, cognitive science, and psychophysics (Marr) 2010), pointing us the North Stars (Fei-Fei and Krishna 2022), some among them being image classification and object detection. Despite the tremendous progress that has been made, much of these object-centtie works remain a proxy for an eventual task application that requires a holistic view of the visual content, involving concepts beyond objects: actions, attributes, and relations, to name a few.
"Currently Google Brain.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In our work, we argue that textual representation suffices such "description". It brings forth a more holistic visual representation than categorical labels. It allows machines to interpret visual signals through descriptive captions (Zhou et al. 2020b, Li et al. 2022), and perform more language-hear tasks such as question-answers (Rajpurkar et al. 2016), or multi-round dialogues (Li et al., 2017). On the other hand, the access to abundant web multimodal language data (e.g., image alt-text, video subtitles) provides us with the fuel for powering neural visual representations from contrastive language-image pre-training (CLIP, Yuan et al. (202 Radford et al. (202I)) The marriage of the two renders a new computer vision system that is faithful, generic, and versatile.
We name this new computer vision system BEST, for Bridging with Explicit Structured Textural clues. We start by constructing a semantic representation of the image. This semantic representation, which we referred to as visual clues, comprises rich semantic components, from object and attribute tags to localized detection regions and region captions. Powered by the recent advances in vision foundation model Florence (Yuan et al. (221), the visual clues are rich in open-vocabulan lary expressions, marking a major difference compared to existing symbolic approaches (e.g.. scene graphs (aaphss/Kishaaaa et al. (2017) with closed-set vocabularies.
The visual clues are interpretable, not only for humans, but also for machines. Take the generative language model GPT-3 (Brown et al. 2020). The visual clues could be digested by GPT-3, which in return produces crisp language descriptions that are sensible to the viewer while not cluttered with irrelevant information from the visual clues. Whereas this open-loop process could potentially suffer from object hallucination issues (Maynez et al., 2020 Zhou al.] 2020a) as the outputs from GPT-3 are not governed by any means, we further deploy a closed-loop verification procedure that grounds descriptions back to the original image.
To evaluate the quality of the language descriptions, we resort to an existing task named Image Paragraph Captioning (IPC), but with a twist. IPC aims to address the demand for generating long, coherent, and informative descriptions of the whole visual content of an image (Krause et al.
2 which can eventually be used for many applications including poetry composition (Liu et al. 2018 automatic recipe generation (Salvador et al.] 2019), visual storytelling (Huang et al. 2016)) advertisement generation, or help blind or visually- -impiiree people see better. The existing metrics for IPC such as BLEU (Papineni et al. 2002). METEOR (Denkowski and Lavie, 2014), and CIDEr /Vedantam et al. 2015) encourage exact matching between semantics in generated captions and those in the reference. However, they over penalize visual details that are not annotated thus compromising their qualifications for measuring overall representation quality. Inspired by Anderson et al. (2016); Krishna al. (2017), we propose to measure the accuracy on scene graphs extracted from generated text against human-annotated graphs, which, as suggested by Anderson et al. (2016), co-relates better with human judgment.
The contributions are twofold. First, we propose general framework for semantic visual representa- tion and showcase its application to image paragraph captioning. The framework is simple yet highly extendable, allowing new components to be plug-in and supporting other use scenarios that require a holistic view of the visual content. Second, we benchmark the effectiveness of the proposed model on its capacity for representing visual concepts (e.g., scene graphs) and set new state-of-the-ar results.
Notations. We denote () as inner product between two vectors, |A| as the cardinality of set A.
2 Related Works
Image paragraph generation. The task of generating image paragraphs is first introduced by Krause et al. (2017). Conditioned on the visual features, they first train a sentence recurrent neural network (RNN) output sentence topics, and then feed each of the topics into another RNN to generate the paragraphs. Liang et al. (2017) further improve the hierarchical RNN framework by introducing an adversarial discriminator for smoother transitions between sentences. Chatterjee and Schwing (2018) also address cross-sentence topic consistency by a global coherence vector. Melas-Kyrizzi et al. (2018) add repeat penalty to the optimization, to prevent the appearance of repeated sequences. Wang et al. (2019) use convolutional auto auto-encoder for topic modeling based on region-level image features. Along this line, many other works have been done (Dai et al.] [017]] Luo et al. 2019 Mao et al.) 2018 Xu et al.) 2020 Guo et al.) 2021: Shi et al. (2021). Most of the proposed models, however, are trained on Stanford image-paragraph datasel (Krause et al. 2017). which only contains 14 thousand of training paragraphs for its expensive nature to collect. Due to lack of data, the generated paragraphs usually lack coherence both locally and globally. Therefore, many of the above works aim to make the best use of data to improve the coherence. Yet nowadays,

---

Visual Information
Visual Class
Open sugger
Togio orgple sunnter vest nend
Otfects
wearing andt is lower mattle image modivate the image Attribude:
Caption: holding a ned apple
sacotor vest
Open
Object::
raple et of imoge and smallo lmogee.
vocab
Caption:
Aftiuuter gai apple
detectoor
Aomibutte aweatter vest
Location: (225, 354, 461.
Caption
Tags opaie
eueson holdinng end apple
Cape on
Arr bute apale
Locatin: [52, 208, 254]
image argule chevisingg orgyle sweater, sweater
Tagi
Captioner
image
Candidate Paragraphs
Open topper
Language
Mode
Output Paragraph
This image featwer a mos dreered in and
This image is of mas holding opolt hls hand....
holling
shaviing and fomile profersientt
Figure 2: Framework demonstration. The orange open vocab tagger box corresponds to the image encoder f-(-) and the text encoder as fi(-). The blue captioner box is the caption model c(-). large language models can generate long coherent paragraphs by default. Our work, leveraging recent progress of large pretrained models, focuses more on how to guide and constrain the generated text instead.
Constrained text generation In recent years, rapid progress has been made in vision-language pretraining ( (VLP). CLIP (Radford et al. 2021), ALIGN (Jia et al. 2021), and Florence (Yuan et al. 2021) are proposed encode vision and language into jooint representation space for crossmodal alignment tasks, e.g., zero-shot image classification. Another line of research, e.g., Sim VLM (Wang al., 202T). FLAVA (Singh al. 202T. BLIP (Li al. (022). CoCa (Yu et al.) 2022) and many others (Cho et al., 2021] Wang et al. 2022 Zhu et al. 2021 1 Alayrac et al. 2022) adopt encoder- decoder models trained with generative losses. Those models are capable of performing image captioning in zero-shot manner. A concurrent work, Socratic Models (SM, Zeng et al. (2022), ), also use textual data to bridge the domain gap between vision-language models and language models. The model, however, is stronger in retrieval tasks than captioning tasks also other works leveraging large language models to solve vision tasks, egg. PICa (Yang et al. 202T)
will show later. There
uses GPT-3 (Brown et al. 2020) to extract commonsense knowledge for visual question answering tasks, MAGIC (Su et al. 2022) uses a CLIP-induced score to regularize the language generation so that it is semantically related to the given image, and VisualGPT (Chen et al. 2022) employs relf-resurrecting encoder-decoder attention mechanism to adapt the language models with a small amount of in-domain image-text data.
Framework
Given an image I, our goal to generate long and coherent descriptive text based on image inputs, leveraging only the existing pretrained models. Our framework can be divided into three stages:
1. Represent 7 with visual clues S, which contain the rich visual information;
2. Feed the visual clues into language model generate candidate paragraphs {T:)/ K
3. Select the best paragraph 7 from the candidates (T: x=
The overall framework is illustrated in Figure We will then elaborate on each of them.
3.1 Visual Clue Extraction
We leverage three state-of-the-ar models with the open-vocabulary capability to extract the visual information, namely, the concise tags, the short captions, and the local descriptions.
Concise tags, The first model we use is the contrastively trained lasianelanguage models, egg. CLIP (Radford et al., 2 (022), Florence (Yuan al., (02T). Such models pretrained image-text pairs (2,, Vi J. and is composed of the image encoder f.).) and the text encoder f((-). Given a minibatch B, the models are optimized by contrastive loss
1
Σ
exp((f((x)) f(li)//) EzyeB exp((((xx,)) ((v)))
C
B
E exp (f)();,, (x)),, M(n))//))
EB
3

---

where т is the temperature. This loss explicitly uses inner product (..) to measure the similarity between the encoded image fz;) and encoded text fi(y;), and higher similarities are encouraged the images and texts are paired. Therefore, such pretrained model is capable of selecting the tags that describe the image / from a set of customized tags by computing the similarities. Given a set of tags (t i= we compute the similarities between the input image I and the tags, and adopt the tags with top-M similarities,
T = [t = arg Mogg f,r=1,,... .N
Short captions. The second model is a caption model c(-). We use it to generate an overall image description c(7).
Local descriptions. The third model is an object detection model. We adopt a well-trained object detector, to provide us with the locations of the possible objects in the format of bounding boxes. The bounding boxes are processed with the non-maximum suppression technique to filter out repetitions. Denote the object proposals {b,} and image regions cropped from corresponding boxes as {p, ر customized tag set,
We first select the indices of the bounding boxes with objects that can be named by our
= 1.......... RR. (4112 (((F(()) > B,i 1, NN.
Here, 3 is a threshold certifying whether t; is aligned with Pj. Given set of customized attribute {as W each selected proposal & from P is then assigned to an attribute
argmax (: a..i=1....,
(3)
and the corresponding tags
O₂ (PLK( f((+)) > 1,... .N.
(4)
In addition to the tags and attributes to the bounding boxes, also use the caption model c(-) to provide some more descriptive texts (c(pp,)))
In summary, we collect a tag set T and caption c(1) as global descriptions to the image, and a quadruple (be,, o2e Oz, c(pt.)) )) as local descriptions for each selected bounding box.
3.2 Candidate Synthesis
We then format the collected visual information into the structured visual clues, which can be directly used as the prompt of the language model. Figure shows an example of the visual clues. We observe that the information near the end of the prompt will have more significant influence on the language model output. As the tags T are usually more informative and the local extractions are noisier, we input the visual clues with the order of local descriptions, caption, and tags.
To incorporate each local description, a naive way is to inject the coordinates of the bounding boxes directly into the prompt. However, find the current language models still lack the capability to handle the inference task with numbers, especially in a zero-shot manner. Therefore, we reformat the bounding boxes be, into plain language by describing its location and size. Specifically, rule-based method to divide the locations into 9 classes ("upper "upper middle", "upper right", "leff". "middle", "right", "lower left". "lower middle", "lower right"). and divide the sizes into classes ("large", "moderate-sized"" "small"), }, and incorporate these descriptions into the prompt.
adopt
The other visual clues are inputted straightforwardly in the format as showed Figure 2 The prompt then fed into a large-scale language model to synthesize candidate paragraphs [Ti full of descriptive details.
3.3 Candidate Selection
Finally, we use the vision-language model again, select the candidate that aligns best with the image.
S argmnax (F((T,, Ti.==1,... KK
To further rule out the unrelated concepts S, we filter the output again in sentence level. This because large language models sometimes have hallucination issues, i.c.. it might generate unrelated

---

graph and sentences in the paragraphs. For example, paragraph beginning with "A couple is hugging on the beach." " is likely to be followed with "It's a beautiful day and they 'e enjoying the sun and each other's 's company." even if there is no visual clue suggesting the weather. Therefore, we split it into sentences (81, $2,'"' BU), and use a threshold y to remove the sentences with lower similarities,
T = (s J((s)) > 7, 1,. .U).
(6)
In this way, we obtain the final output T*
4 Automatic Evaluation Metric: SPIPE
As indicated by Figure T the gener- ated paragraphs of images can be very flexible. This makes the n-gram based metrics, e.g., BLEU (Papineni et al. 2002). ROUGE (Li 2004). CIDEr Vedaanta et al., 2015), METEOR (Denkowski and Lavie 2014), unsuit- able for evaluating the generated text. Instead, we focus on the semantic propositional content. For example, given an image with content "A man sitting in front of a blue snowboard", a good evaluation metric for IPC should evaluate whether each of the semantic propositions is correct, namely, a). a man is sitting; b). man is in front of a snow board: c). the snowboard is blue, instead of the exact words used in the text. To do so, SPICE (Anderson et al. 2016) extracts the scene graphs (Johnson et al. 2015) from the generated texts and the reference texts, respectively, and computes an F-score between the graphs. SPICE targets image caption tasks, where there are usually multiple good references for each image, and the generation is less flexible. However, IPC tasks usually only have one reference (Krause al.) (017), which is not enough to evaluate the flexible generation. Therefore, we propose to directly compare the scene graphs extracted from the generated text to human-annotated graphs. 3 an example of the generated graph from text and the human-annotated graph for the image.
Semantic Compartoon bonn
man sits on the ground on the of mountess antiigg snsw and wvaz ful wenter gour oost groy hunds which
The
stowbsord
otisched to hi feet. He foals out mounten somee os the run refocts on the snowy mosyiaai perto. A pors treas are ths oeft jurt dises tut The ground that te als oni poshhed atowe.
Figure 3: An example of the the text extracted graph.
Specifically, scene graph consists of the objects, the attributes of the objects, and the relationships between the objects. To parse the generated text into a scene graph, we use a two-stage approach following Anderson al. (2016). First, use the pretrained dependency parser (Klein and Manning 2003) to establish the synthetic dependency between the words. Then we map from the dependency trees scene graphs using a rule-based system (Schuster al., 2015). Given scene graphs extracted from the text and the based on the synonym match") Denkowski and Lavie (2014) between the two graphs among the conjunction of three sets of concepts: (object), (object, attribute), and (object, relationship. subject). Paying homage to Anderson et al. (2016), we name our approach SPIPE, Semantic Propositional Image Paragraph Evaluation.
graphs (Krishna et al. 2017)), our metric computes an F-score
5 Empirical Analysis
The basic evaluation of the generated output should include three aspects:
1. Accuracy. Most of the contents appearing the paragraph should be from the image;
2. Completeness. Most of the contents appearing in the image should be included in the paragraph;
3. Coherence. Paragraphs should be more than concatenating the sentences together.
We evaluate the accuracy and completeness of the generated descriptions using the proposed automatic evaluation metric SPIPE, and do human evaluation to quantify the coherence. We include 500 randomly sampled outputs in output. html in the Supplement Materials for readers to perform qualitative study.
'Tuples are considered to be matched if their lemmatized word forms are equal or if they are found in the same WordNet Milllrr 1995) synset.
5

---

Table 1: Comparison between different methods using SPIPE metric on the test of the Stanford
dataser (Krause et al. 2017).
Name
F-score
Precision
Recall
BLIP-large
7.6
38.0
4.4
Socratic model
3.2
13.9
1.9
Models
BEST-general domain
8.8
15.3
6,6
BEST-VG domain
10.0
17.5
7.6
Oracle
BEST with human extracted visual clues
22.9
32.8
19.0
Stanford dataset
17.3
27.7
14.0
Annotation
Concatenation of VG captions
18.9
40.0
14.1
5.1
Model Specification
Models. We adopt Florence-H (uaan al. 2021) the vision-language model, BLIP-large (Li et al. 2015) as the captioner with its default setting, and one-stage detector general object detector. To be more specific about the detector,
2022) finetuned on COCO captions dataset CChen et al.
first omit the category information from COCO (Chen et al. 2015) datatset and train Dynamic Head (Dai al. 2021) on the bounding boxes only to formulate a class-agnostic object detector. We then use non-maximum suppression (NMS) to select the top 100 object proposals.
We use GPT-3 (Brown al. 2020) Davinci-text-00) model the language model. To enable more difference in the generated candidates, we adopt temperature as 0.8, as a higher temperature encourages the model have more creative outputs. We adopt the frequency penalty as 0.5 and the maximum number of tokens as 100.
Customized sets. To construct a general domain tag set, collect the most frequently searched 400 thousand queries in Bing Search as the tags (4) We adopt the attribute set of the Visual Genome dataset (Krishna al. 2017) as the attribute set {a.)
y y = 0.2, and number of candidates K 40. Among K 40 candidates, half of them are generated without caption information while the remaining half are with them. This is because we notice the caption model sometimes cannot output good captions due too small bounding boxes. We also remove the bounding boxes that smaller than 1/400 of the image sizes.
Parameters. We adopt number tags M = 5, thresholds 3
5.2 Automatic Evaluation
In this section, evaluate our framework on the test set of Stanford dataset (Krause et al.] 2017). The dataset is subset of Visual Genome (VG) datasef" 3 (Krishna et al. 2017). and therefore can obtain the human-annotated scene graphs from VG as well. We compare the following frameworks.
SPIPE to benchmark the accuracy and completeness of our framework. We
BLIP Li et al. 2022). This is the BLIP-large model finetuned on COCO captions dataset.
Socratic model Zeng
2022). We adopt the image captioning code without alternation. BEST-general domain. This is our framework with the customized set listed above.
BEST-VG domain. With open-vocabulary capability, our framework can adapt to specific domain. Here, we replace the customized tag set [t, for the local objects the object set of VG datasets.
The results are shown in Table [ Our general domain framework significantly outperforms the BLIP model and the Socratic model. With the domain specified to VG, the performance is further boosted.
Figure Segure4show shows an example with image cropped from the Socratic model paper (Zeng al.) (022) directly. We find that caption generation does not require the complex prompt used in Socratic model. Our framework with only tagging information T can generate texts with similar degree of detail, See Appendix D for more discussion.
2 We remark that VG caption data included the pretraining data BLIP model. Therefore we not claim our framework as a zero-shot method, despite that can handle images in wild in zero-shot way. 'https:// tgtb.t....ttttgg com/ google-reseseeeee /oogle wwssearch/treee/mstteeeeeoeeeeee

---

This image features group of people standing under tree with pink blossoms. The people are all dressed in various types of clothing, and some carrying bags backpacks. The tree is sakura tree, and the blossoms are in full bloom. There is bench in the background, and the ground is covered in fallen petals.
BEST:
SM: People gather under a blossoming cherry tree, enjoying the beauty of natural together.
BLIP: A group of people standing around a blue table.
BEST with only tags: The image is of people gathered in park, looking at cherry blossoms. The parks are full of people enjoying the festival and the beautiful view.
Figure 4: An example cropped from Zeng et al. 2022) paper, with other outputs for comparison.
To evaluate the representation capability of our visual clues, we also compare it to naive scene graph generation method. We use the vision-language model to assign objects, attributes, and relationships between the objects, using the object set, attribute set, and relationship set of VG. And then we compare the generated scene graph to the 0.8 and recall 0.2. We discuss more on why this does not work in Appendix [
graph. The F-score is 0.3, with precision
We also build an oracle model to see the limit of our framework. The oracle model in Table[usess the ground truth objects with ground truth attributes to replace the corresponding concepts in the visual clues of our framework. It significantly outperforms the human annotation, either from Stanford dataset or from VG. This reveals the large potential of BEST with the development of object detectors.
5.3 Ablation
Table Ablation on each components. The metries are F-score (F), Precision (P), and Recall (R).
We perform an ablation study to see how each of the components contributes to the performance. Especially, we consider replacing the open-vocabulary object de-
Name
F
P
R
BEST-VC domain
10.0
17.5
7.6
tector with YOLO v5 (Jocher) 2020), which is a closed-set detector trained with COCO classes. Table 2 shows the results. The performance of the YOLO v5 alternation is competitive compared to our general domain version. The pre- cision is higher. which may be conse- quence that YOLO models tend to recog- nize fewer objects (Zou et al. 2019). However, it still inferior to our VG domain model.
Extraction with YOLOV5
9.0
19.0
6.3
Remove local information Remove caption model c(-)
8.0
14.4
6.0
8.7
15.0
6.6
Input tags T only
5.9
10.7
4.4
Smaller language model (curie)
8.9
15.9
6.7
Weaker tagger (CLIP ViT-L/14)
7.8
16.4
5.6
5.4 Human Evaluation
To further evaluate our framework, we perform human evaluation. We first compare BEST to human annotation. Specifically, we randomly sample 200 descriptions from the test set of the two sources. For each assignment, we present one image and two corresponding descriptions, and ask human evaluators to evaluate on accuracy, completeness, coherence, and ask an additional question "which of the descriptions is written by human" for the humanlikeness aspect. They will choose one answer from (Description 1, Description 2. Cannot determine). ). For each assignment, we hire 5 workers using the Amazon Mechanical Turk platform. More details can be found in Appendix E As the difference between the long texts can be subtle, we perform two statistical tests to see whether the difference is statistically significant. Please refer to Appendix E.2 for the hypotheses.
Table shows the results. There no significant evidence (p value 0.5) showing human annotation is better than BEST in terms of completeness and humanlikeness. However, BEST still falls behind terms of accuracy and coherence. The failure cases are usually because the BEST outputs might contain small mistakes that cannot be easily filtered out, mostly from the hallucination of the language model. We show more examples in Appendix C
We then compare BEST to BLIP and the Socratic model using similar hypothesis tests. The results show BEST are significantly better than BLIP and Socratic models under most of the metries (p value < 0.05). Note that here accuracy is defined slightly different than the precision used in Table In human evaluation, providing background information about concepts in the image is not viewed as inaccurate, while in Table it will hurt the precision.
7

---

Table Human evaluation. p-value with the binomial test, and p-value 2 with Mann Wonn-Wh U test. The blue regions in the voted proportion section represent the proportion that the descriptions from the first source are better than the second, while the orange ones represent the second are better than the first. The 1 1.2% and 0.6% in the middle of row 4 and 12 represent " "Cannot determine".
Sources
Criteria
Voted proportion
p-value
p-value 2
Accuracy
61.0%
39.0%
x 10
9 x 10-6
Anno. / /EST
Completeness
0.38
0.50
Coherence
59. 7%
40.3%
10-3
8 10-4
Humanlikeness
54.7%
1.2% %
44 2%
0.10
0.50
Accuracy
57.2%
42.8% 2%.8%
0.03
10-3
BEST /LLIP
Completeness
3.66%
26.4%
2 x 10-11
2 10-28
Coherence
56.2%
43.8%
0.05
5 x 10
Humanlikeness
58.3%
41.7%
9 x 10-3
5 10
Accuracy
59
40.6%
6 10-3
10-5
BEST/ Socratic
Completeness
71.3%
28.7%
2 x 10-9
1 10
Coherence
68.7%
31.3%
3 x 10-7
2 x 10-12
Humanlikeness
53.4%
0.6%
46.0%
0.18
0.35
Variants and Real-world Applications
The proposed framework opens up many creative real-world applications. For example, People with vision deficiencies may not be able to view images easily. BEST can help convert into precise and comprehensive descriptions for general domain images.
Another example is the closed-loop training of the large models. The large-scale vision- language model and language model used in BEST are trained tremendous amounts of data, and thus can memorize knowledge beyond human capability. We cally annotate data, which is easy to scale up. Furthermore, it can incorporate commonsense knowledge into the text naturally. For example, in the second example of Figure | the text contains "This image captures the everyday life of Cubans, with their Iraditional horse-drawn carts still in use." This is because
Table 4: Finetune BLIP-large on different data.
Name
R
No finetune
7.6
38.0
4.4
With Socratic generated data
17.5
2.3
it automati-
With BEST generated data
11.6
23.1
8.2
tags contain "Cuba" and "buggy", and the language model knows traditional horse-drawn carts are still in use in Cuba. We finetune BLIP-large model on our BEST generated data. The training images are similar Stanford dataset KKaause et al., 2017), which is around 15 thousands. After finetuning, the F-score improves more than 50%.
With small modifications, the proposed framework enables us to free human labor for even more applications. To list few examples,
Visual storytelling. shown in Figure (a), the framework can generate charming stories based on the input image. To do so, we simply change the end of the prompt to be "Tell me creative story:".
Automatic ads generation: As shown to upload an image, and make small modifications to the generated advertisement as wanted. As there usually one product in an image, we adopt the number of input tags M We also change the end of the prompt to be "Write a product description to sell in eBay or Amazon marketplace to get lots of engagement:".
( with the framework, the merchants only need
Social media post. As shown in Figure (c), the framework can be a social media bot, which may alleviate the workload of internet celebrities. We change the end be "Social media post:".
Background generation. As shown 5 the framework can also be used to provide some background knowledge. To do so, change the end of the prompt to be "Textbook text:".

---

As the Hogwarts Express made its way through the Scottish countryside, a young wizard named Harry peered out the window the stunning Glenfinnan Viaduct. He had always dreamed of traveling on the venerable steam train, and now he was finally doing it! The locomotive chugged along, its great iron wheels clanking over the rails, as passengers in every car gazed wonder at the beautiful scenery.
(a)
Looking for a stylish and practical coat to keep you warm this winter? Look no further than our long parka! Made from high quality beige fabric, this coat is sure to keep you warm and confortable all winter long. It features stylish and flattering fit, as well as warm and cozy hood, making it perfect for cold days out.
(b)
(c)
What do you think of this Chevy Impala Lowrider? I spotted it at the Sema Car Show and it's definitely a head-turner!
A tench is a fish that lives in fresh water. It is common fish that can be found in many lakes and rivers. The tench is a large fish, and can grow to be quite large. It has greenish- brewnshh-brown body, and is covered in scales. The tench feeds on insects, plants, and small fish.
(d)
The image is of cover of The Economist magazine. The headline is " MMarch of the machines". The cover features drawing of the Maschi- nenmensch, or machine man, in the center. The background is art deco in style. This image represents the rise of robots and artificial intelligence, they become increasingly important in our society.
(e)
Figure 5: Examples for real-world applications with variants of the proposed framework.
Applications with scene texts. As shown in Figure we plug in an OCR (Optical Character Recognition) model (Farley) 2022). and insert the OCR output into the prompts with prefix "This image contains text:" before the captions.
5 (c). when the input image has rich scene text,
Applications to Visual Question Answering
The visual clues is a faithful and detailed description of the image, which can be used to answer visual questions leveraging the question answering ability of language models. Specifically, we replaced the ending of the prompt to be the question, e.g. we replace the "Describe the image in detail:" by "What is the man holding?" ?", We benchmark its performance in two Visual Question Answering (VQA) datasets we use the GQA (Hudson and Manning (019)) dataset for probing the capability of scene understanding, and the OK-VQA (Marino et al.) 2019) dataset for the awareness of the commonsense knowledge.
Since no training is performed, BEST generated answer usually have different formatting from the ground truth, causing difficulty in evaluation. For example, for question "Is the ground blue or brown?", the ground truth answer in GQA is "brown", but the BEST answer is "The ground in the image is brown.". Therefore, we use GPT-3 model again reformat the answer. We refer this evaluation method as Generative. Furthermore, for the GQA dataset, the answers in the training set and test set have large overlap. So adopt the nearest embedding from the training answers the final answer, and refer the method as Discriminative. More details can be found in Appendix FF
Table 5 shows the evaluation results. BEST outperforms Socratic models significantly, suggesting visual clues are better image representations. We also benchmark the accuracy on BLIP (finetuned on VQA v2 dataset (Goyal et al. 2017) and Visual Genome dataset (Krishna et al. (017)) for reference, which is not directly comparable since its pretrain and finetune datasets have a significant overlap with the evaluation datasets. Figure o Figure 7 show some success and failure cases of from the VQA datasets.

---

Table 5: The VQA accuracy on GQA and OK- VQA datasets.
Method
Evaluation
GQA
OK- OK-VQA
Generative
24.95
16.50
Socratic
Discriminative
26.89
Generative
37.00
28.89
Visual Clues
Discriminative
39.93
BLIP
Exact Match
47.58
43.62
Question: What mate-
Question: What ingredient is missing from the picture to make pb and sandwich? GT: Jelly
rial is the crosswalk in front of the stores?
GT: Concrete
BEST: Concrete
BEST: Jelly
Figure Examples of the success cases from the GQA (left) and the OK- VQQ (right).
8 Limitations and Further Improvements
Prompt tuning. As suggested in Brown et al. (2020), language models can infer better when they are shown examples in the prompt. In our experiments, however, this results in model directly copying sentence pieces from example paragraphs, introducing unnecessary noise. We suspect this prompt generated one. This may be
tuning approach may work better the input examples are similar promising direction as can better control writing style.
Visual clues as scene graph. Our visual clue extraction process is motivated by the fact that an image can be comprehensively represented by scene graph (Johnson et al.) 2015). As mentioned in Section a scene graph contains objects, attributes of objects, and the relationship between objects. In BEST, however, we do not include the relationships,
we observe in our initial study that the current vision- language models, although powerful, are not good at inferring relationships (echoing findings from Thrush et al. )2022)) Yet, relationships among the objects are important components of an image. This can be plugged into our framework if better vision-language models are developed.
well-trained filter model. We find that the current filtering strategy (6) is not immune to certain types of mistakes. As also mentioned in Thrush al. (2022). the vision-language model cannot accurately associate attributes to their corresponding objects. For example, in the second image of Figure [ if there is sentence "The man wears black shirt.", ", it will lead high image-text relevance score, since there is a man, shirt, and dark bush in the image. To handle this issue, we crop the image into local regions and pair each region with an attribute. Still, it the language model who hallucinates new attributes and the attributes happen to be in the image, these captions cannot filtered out. We suspect an adversarially trained filter is needed to perform the task.
Broader Societal Impact. BEST inherits the risks of large vision and language models. BEST can potentially output offensive language and propagate social biases and stereotypes. For real applications, we can use rule-based methods or train a specific filter to filter out the offensive text. This is an area that we plan explore gain more insights further.
Question: What place is pictured? GT: Shore
Question: Where in the world would you find this structure?
GT: California
BEST: Africa
BEST: Los Angeles, Califor- mia
Figure 7: Examples of the failure cases from the GQA (left) and the OK-VQA (right).
10

---

References
ALAYRAC, DONAHUE, J., LUC, P., MIECH, A., BARR, I,, HASSON, Y., LENC, K., MENSCH, A., MILLICAN, K., REYNOLDS, M. ET AL. (2022). Flamingo: visual language model for few-shot learning. arXiv preprint 22Xi::2204111995
ANDERSON, P., FERNANDO, B., JOHNSON, M. and GOULD, S. (2016). Spice:
Semantic proposi- tional image caption evaluation. In European conference on computer vision. Springer.
BROWN, T., MANN, B., RYDER, N., SUBBIAH, M., KAPLAN, J. D., DHARIWAL, P., NEBLAKAN-
TAN, A., SHYAM, P., SASTRY, G., ASKELL, A. AL. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33 1877-1901.
CHATTERJEE, M. and SCHWING, G. (2018). Diverse and coherent paragraph generation from images. In Proceedings of the European Conference Computer Vision (ECCV).
CHEN, J., GUO, H.,, YI, K., LI, B. and EL.HOSI,Y,, M. (2022). Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
CHEN, X., FANG, H., LIN, T.-Y., VEDANTAM, R., GUPTA, S., DOLLÁR, P. and ZITNICK, C. L. (2015). Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.0032..
CHO, J., LEI, J., TAN, H. and BANSAL, M. (2021). Unifying vision-and iisio--an--annguae tasks via text generation. International Conference on Machine Learning. PMLR.
DAI, B., FIDLER, S., URTASUN, R. and LIN, D. (2017). Towards diverse and natural image descriptions via conditional gan. In Proceedings of the IEEE international conference on computer vision.
DAL, X., CHEN, Y.. XIAO, B.. CHEN, D., LIU, M., YUAN, and ZHANG, L. (2021). Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
DENKOWSKI, M. and LAVIE, A. (2014). Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation.
FARLEY, P. (2022). What is optical character recognition? azure cogmitive services.
https://docs mi crosoff. com/ en us/ www.ue/coooiiiieeeeeerii....o computer- wwwiooioverviee......././.
FEI-FEI, I.. and KRISHNA, R. (2022). Searching for computer vision north stars. Daedalus, 151 85-99.
GOYAL, Y., KHOT, T., SUMMERS-SE -TAAY D., BATRA, D. and PARIKH, D. (2017). Making the v in vqa matter: Elevating the role of image understanding in visual question answering. Proceedings of the IEEE conference on computer vision and pattern recognition.
GUO, D., LU, R., CHEN, B., ZENG, Z. and ZHOU, M. (2021). Matching visual features hicrarchical semantic topics for image paragraph captioning. arXiv preprint arXiv:2105.04145
HE, P., LIU, X... GAO, J. and CHEN, W. (2020). Deberta: recoding-enhaneed bert with disentangled attention. arXiv preprint arXiv:2006.03654.
HUANG, T.-H., FERRARO, F., MOSTAFAZADEH, N., MISRA, I., AGRAWAL, A., DEVLIN, J., GIRSHICK, R., HE, X., KOHLI, P., BATRA, D. ET AL. (2016). Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
HUDSON, D. A. and MANNING, C. D. (2019). Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition.
11

---

JIA, C., YANG, Y., XIA, Y., CHEN, Y.-T., PAREKH, Z., PHAM, H., LE, Q. SUNG, Y.-H., LI, Z. and DUERIG, T. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning. PMLR.
JOCHER, G. (2020). ultralytics/yolos:: v3.1 1- Bug Fixes and Performance Improvements. https
//github.com www.com///llaa wtw.:///iliraaaliscs/ooooooooooa
https://doi 10. 5ww1/2eeo002ooo 1rw.o0...2.////oooo....o 4154370
JOINSON, KRISHNA,, R., STARK, M., LI, L.-J., SIIAMMA, D., BERNSTEIN, M. and FEI-FBI, L. (2015). Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pallern recognition.
KLEIN, and MANNING, C. D. (2003). Accurate unlexicalized parsing. In Proceedings of the s anmual meeting of the association for computational linguistics.
KRAUSE, J., JOHNSON, J., KRISHNA, R. and FEI-FEI, L. (2017). A hierarchical approach for gencrating descriptive image paragraphs. In Proceedings of the IEEE conference on computer vision and pattern recognition.
KRISHNA, R., ZHU, Y., GROTH, O., JOHNSON, J., HATA, K., KRAVITZ, J., CHEN, S., KALAN- TIDIS, Y., LI, L..J., SHAMMA, D. A. ET AL. (2017). Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123 32-73.
L1, J., LI. D., XIONG, C. and HOl. S. (2022). Blip: unified vision- languaee understanding and generation. arXiv preprint arXiv:2201.12086.
Bootstrapping language-image pre-training for
LI, Y., SU, H., SHEN, X., LI, W., CAo, Z. and NIU, S. (2017). Dailydialog: manually labelled multi-turn dialogue dataset. arXiv preprint arXiv: 1710.03957.
LIANG, X., HU, Z., ZHANG, H., GAN, C. and XING, E. P. (2017). Recurrent copic-transition gan for visual paragraph generation. In Proceedings of the IEEE international conference on computer vision.
LIN, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out.
LIU, B., FU, J., KATO, M. P. and YOSHIKAWA, M. (2018). Beyond narrative description: Generating poetry from images by aavei-adversarial training. In Proceedings of the 26th ACM international conference on Multimedia.
LUO, Y.. HUANG, Z., ZHANG, Z., WANG, Z., LI, J. and YANG. Y. (2019). Curiosity-driven reinforcement learning for diverse visual paragraph generation. In Proceedings of the 27th ACM International Conference Multimedia.
MAO, Y., ZHOU, C., WANG, X. and LI, R. (2018). Show and tell more: Topic-oriented multi- sentence image captioning. In IJCAI.
MARINO, K., RASTEGARI, M., FARHADI, A. and MOTTAGHI, R. (2019). Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/evf conference on computer vision and patlern recognition.
MARR, D. (2010). Vision: A computational investigation into the human representation and processing of visual information. MIT press.
MAYNEZ, J., NARAYAN, S., BOHNET, B. and McDONALD, R. (2020). On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661.
MELAS-KYRIAZI, L., RUSh, A. M. and HAN, G. (2018). Training for diversity in image paragraph captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
MILLER,, G. A. (1995). Wordnet: lexical database for english. Communications of the ACM, 38 39-41.
12

---

PAPINENI, K., ROUKOS, S., WARD, T. and ZHU, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguisties.
RADFORD, A., KIM, J. W., HALLACY, C., RAMESH, A., GOH, G., AGARWAL, S., SASTRY, G., ASKELL, A., MISHKIN, P., CLARK, J. ET AL. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR.
RAJPURKAR, P., ZHANG, J., LOPYREV, K. and LIANG, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 160.000000.02500
SALVADOR, A., DROZDZAL, M., GIRÓ-I NIETO, X. and ROMERO, A. (2019). Inverse cooking: Recipe generation from food images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition.
SCHUSTER, S., KRISHNA, R., CHANG, A., FEI-FEI, L. and MANNING, C. D. (2015). Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language.
SHI, Y., LIU, Y., FENG, F., LI, R., MA, Z. and WANG, X. (2021). S2td: A tree-structured decoder for image paragraph captioning. ACM Multimedia Asia. 1-7.
SINGH, A., HU, R., GOSWAMI, V., COUAIRON, G., GALUBA, W., ROHRBACH, M. and KIELA, D. (2021). Flava: A foundational language and vision alignment model. arXiv preprint arXiv:211..0442..
SU, Y., LAN, T., LIU, Y., LIU, F., YOGATAMA, D., WANG, Y., KONG, L. and COLLIER, N. (2022). Language models can see: Plugging visual controls in text generation. arXiv preprint arXiv:2205.02655.
THRUSU, T., JIANG, R., BARTOLO, M., SINGII, A., WILLIAMS, A., KIELA, D. and Ross, C. (2022). Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR.
VEDANTAM, R., LAWRENCE ZITNICK, C. and PARIKH, D. (2015). Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition.
WANG, J., PAN, Y., YAO, T., TANG, J. and MEI, (2019). Convolutional auto-encoding of sentence topics for image paragraph generation. arXiv preprint arXiv: 00249.
WANG, P., YANG, A., MEN, R., LIN, J., BAL, S., LI, Z., MA, J., ZhOU, C., ZhOu, J. and YANG, H. (2022). Unifying architectures, tasks, and modalities through a simple sequence-to-sequeeee learning framework. arXiv preprint arXiv:2202.03005.
WANG, Z., YU, J., YU, A. W., DAI, Z., TSVETKOV, Y. and CAO, Y. (2021). Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904.
XU, C., LI, Y., LI, C., AO, X., YANG, M. and TIAN, J. (2020). Interactive key-value memory- augmented attention for image paragraph captioning. Proceedings of the 28th International Conference on Computational Linguistics.
YANG, Z., GAN, Z., WANG, J., HU, X., LU, Y., LIU, Z. and WANG, L. (2021). An empirical study of gpt-3 for few-shot knowledge-hasee vqa. arXiv preprint arXiv:2
YU, J., WANG, Z., VASUDEVAN, V., YEUNG, L., SEYEDIOSSEIII, M. and Wu, Y. (2022). Coca: Contrastive captioners are image-text foundation models.
YUAN, L.. CHEN, D., CHEN, Y.-L., CODELLA. N., DAI, X., GAO, J., HU, H., HUANG, X., LI, B., LI. C. ET AL. (2021). Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.1143.
ZENG, A., WONG, A., WELKER, S., CHOROMANSKI, K., TOMBARI, F., PUROHIT, A., RYOO, M., SINDHWANT, V., LEF, J., VANHOUCKE, V. ET AL.. (2022). Socratie models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598
13

---

ZHOU, C., NEUBIG, G. GU, J., DIAB, M., GUZMAN, P., ZETTLEMOYER, L. and GHAZVININEJAD, M. (2020a). Detecting hallucinated content conditional neural sequence generation. arXiv preprint arXiv:201 1.02593.
ZHOU, L., PALANGI, H., ZHANG, L., HU, H., CORSO, J. and GAo, (2020b). Unified vision- language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34.
ZHU, X., ZHU, J., LI, H., WU, X., WANG, X., LI, H., WANG, X. and DAI, J. (2021). Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. arXiv preprint arXiv:2112.01522
Zou, Z., Sm1, Z., GUo, Y. and YE, J. (2019). Object detection in 20 years: A survey,
https://arxiv.orr//abb 15550 05055
References
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes] As much as possible. Some of the models and data are proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] The proposed method does not involve training.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, cloud provider)? [No] The proposed method does not involve training. The training used in application section is not significant. The overall usage of computing resources not significant.
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're
using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes]
14


Reviews:
[{"review_id": "raVOIUU3f6i", "reviewer": "Reviewer_b7uK", "summary": "The visual clues paper tries to leverage the recent advances in both vision-and-language and natural language understand fields to generate comprehensive and rich descriptions of the images. The model works in three steps: (i) generate a set of visual clues (i.e. information) about the image to embed or encode rich visual information (ii) pass these visual clues as a prompt to a large language model to generate candidate paragraphs (iii) select the best paragraph from the candidates. The paper also purposes a new metric to better evaluate long descriptions using scene graph comparisons. The human and automatic evaluations both suggest that the descriptions generated by this method are almost on par with the human annotations.", "questions": "- It would be great to see the exact visual clues as well for the images in the supplementary. Without that context, it is hard to understand what was passed to GPT3 to generate the description.\n- Will any information about the 400K top bing queries be provided?\n- There is no basis behind claiming that n-gram based metrics don't work for larger paragraph generations for images.\n- When the tags are matched with the images, are the tags directly matched or is some sort of prompt used?\n- What is source/intuition behind the paragraph on line 38\n- I disagree with footnote 2 and believe that this overlap is somewhat impactful. The Visual Genome captions are not just semantic propositions but are full captions. Also, in a single image, there are multiple crops for which we have the captions. Now, when these crops along with their captions compound, they do provide a lot of information about the image which is non-trivial leak\n- On line 292, how much BEST-generated data was the BLIP-large model trained on?", "limitations": "Yes, the authors have discussed the limitations. In general, this model because it is developed on proprietary models will not be available so can't have direct impact. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Legend: S (Strength), W (Weakness), C (General comment)\n- C: The paper is very well written and easy to understand. Though, there are some things which are not clear and not well-mentioned probably due to the fact that some of the models used in this paper are proprietary. \n- S: The paper connects the recent advances in the vision-and-language and NLP fields really well by leveraging almost only the pretrained models to generate very coherent and comprehensive descriptions of the images. Since there is almost no training involved, it further suggests how unimodally strong models can be bridged across modalities leveraging abundantly available unimodal data in the world.\n- S: The approach to generate visual clues is quite exhaustive and covers various sources of visual information: tags using a contrastively trained VL model, object detection model whose bounding box classes are again calculated by the contrastively-trained VL model, and a full-image caption generator. \n- S: One important contribution of this paper is a new automated metric for evaluating longer paragraph descriptions of the images as existing n-gram based metrics are not suitable for the variety seen in longer descriptions. The metric uses scene graph based similarity and is an extension to SPICE metric.\n- S: Human evaluations on the outputs of the model suggest that model's outputs are almost on par in terms of completeness and humanlike but fall behind in accuracy and coherence.\n- W: The previous point brings me to one of the weaknesses of this approach is that since we are relying on pretrained models in the complete pipeline, any errors are eventually compounding as we go across the pipeline which means that if any wrong content is passed via visual clues to the text generator, it would add a compound on top which is why maybe the accuracy and coherence are low.\n- W: It is hard to understand the significance of SPIPE without seeing results on other metrics for the baselines as well. As of now, only SPIPE results are reported and there is no discussion on why ngram metrics probably don't work on the paragraph comparison.  \n- W: The model seems to ignore scene text in the image which might be an issue in real world scenarios as scene text is omnipresent. Even when you are reading this review, you are reading scene text.\n- W: There have been works in past such as VisualGPT [1] which try to adapt the language model directly to image captioning. A comparison or discussion with them would be helpful.\n- W: The baselines except BLIP are somewhat weak. Even BLIP when finetuned on COCO captions would be more inclined to generate shorter captions. This is also evident from the examples in supplementary. Since COCO captions are small, it would make more sense to use and compare against captioners trained on dataset which contain larger image descriptions such WIT and Localized Narratives which the paper doesn't do as of now. Most of the models used as baselines are tuned for generating single sentence captions.\n- W: Most of the models used in this paper are proprietary and not available to the public so it is hard to make any comparison down the line. What would be interesting is to use CLIP instead of Florence along with a publicly available large language model to see how this approach performs.\n\n[1] Chen, Jun, et al. \"Visualgpt: Data-efficient adaptation of pretrained language models for image captioning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AnC9BY2BOO_", "reviewer": "Reviewer_ErrM", "summary": "The paper presents a general framework (BEST) for generating semantic visual clues by capturing information through tags, captions, object detections by leveraging existing pre-trained visual LMs and textual LMs. The framework enters them in an ascending order of their quality (highest quality predictions at the end): local descriptors, captions, and tags as input to a large-scale LM to generate K candidate paragraphs. The last step leverages the vision language model to select the top K candidates that align best with the image.  The paper presents an extensive set of experiments against state-of-the-art to show the effectiveness of the proposed framework. The paper has both ablation and qualitative results to demonstrate how each component contributes to the final performance and quality of the generated paragraphs. The paper also proposed an evaluation metric: SPIPE that captures Accuracy, Completeness and Coherence.   \n", "questions": "Detailed questions/comments\n\nIn Figure 2, could authors highlight what color is being used to represent the text encoder as f_t(.)? \n\nLine 35: Florence reference is missing. Please refer at the first introduction. There is a reference later e.g. on Line 106. \n\nLine 112: How are tags generated {t_i}_{i=1}^{N}? \nSection 3.1 add/create sub-subsections or bold headings to improve organization. \n\nLine 147/148: Could more intuition be added to what sentences are kept or removed from Candidate selection Step?\n\nLine 204: Some details on how tau temperature needs to be adopted in practice (intuition is needed)? What happens if \\tau is 100 vs 0.8 as used in the paper? \n\nLine 206: Tags are discussed. How you refer this back on Line 113 where it was unclear how these N tags are selected. \n\nSome more clarification what are the exact inputs to BEST will improve reproduction of the results. What models need to be re-trained/re-run etc? As the paper is leveraging most  of the pre-trained models. It will be good to summarize in a Table/Figure how each component is participating in the final BEST model. \n  ", "limitations": "Already captured in Section 7 of the paper. Authors could  add some \"additional societal biases that textual language models suffer from\". Add some details on pros/cons of training these large models to environment. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. Overall, the paper is well written and is easy to read.\n2. The paper presents a general framework (BEST) for generating detailed image caption paragraphs. The quantitive/qualitative evaluation results show the effective of BEST against BLIP-large and Socratic model in Table 1. \n3.  Ablation results in Table 2 show the effectiveness of each component of BEST and Table 3 captures Accuracy, Completeness and Coherence components of BEST against Ground-truth/BLIP/Socratic.\n4. Fine Tuning BLIP-large also improves results as shown in Table 4.\n5. The paper presents many real-world setting where BEST could be useful. May detailed results are presented in Figure 5. \n6. The Section 7 clearly calls out limitations and further improvements to the BEST. \nWeakness: \n1. Given Socrate model is solving a similar problem: it will be good to discuss how and why BEST is so much better than Socrates model in evaluation results. \n2. In related work, more details around how BEST model is different than Socrate model with be different. \n3. Given, there are many pre-trained model involved and also fine tuning is presented. How much it costs in terms of time/resources to train BEST? How much more expensive is training of BEST model when compared to Socrate model?  \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9PDdsUkyp9A", "reviewer": "Reviewer_8CQ9", "summary": "The authors propose a new framework for generating image paragraph captioning in this paper. The framework extracts text descriptions from images using different pretraining modules, subsequently feeds these extracted modules to a language model for paragraph generation, and finally selects the optimal paragraph from several candidate results using a cross-modal selection module as the final result. Also, to better measure this task's results, the authors propose a new evaluation method SPIPE based on the SPICE evaluation method.", "questions": "See the “weakness”.\n\nHope the authors improve this paper according to the “weakness”.", "limitations": "The authors have addressed some of the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The framework proposed in this paper can generate a paragraph for a given image, which is much longer than the previously widely used image caption.\n2. The authors propose a new evaluation metric SPIPE to measure the results of IPC.\n\nWeakness\n1. Lack of comparison with other traditional IPC methods. There are many previous IPC methods mentioned in the Related Works section.   but their results are not compared with the proposed methods' results in the experimental section.\n2. I think this work seems like a direct extension of PICa to the IPC task. The novelty may be limited.\n3. There is no experimental proof in the article that the new evaluation metric SPIPE could evaluate IPC better than the previous metrics, such as SPICE and METEOR. There is also no analysis of whether the SPIPE evaluation metrics can evaluate different aspects. Besides, SPIPE needs human-annotated graphs for each paragraph in the dataset, which limits the generalizability of this metric. As a result, only using the SPIPE metric in all the experiments is not convincing.\n4. In Table1, compared with BLIP-large, BEST–general domain improve 1.2 in F-score. Due to more processes (Candidate Synthesis&Selection) and higher demands on computing resources (GPT-3), I don't think the improvement is significant.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5TYyT_N3awm", "reviewer": "Reviewer_WzAU", "summary": "This paper introduces a new method for image paragraph captioning, where the goal is to generate long, coherent and informative textual descriptions of an image. Their method uses several large pretrained models, combined without the need for further training. From the image, open-vocabulary models like CLIP generate tags, object detectors generate a list of objects along with their locations, and image-captioning models generate (shorter) captions of the entire image or image regions. All of these outputs are combined into a natural language prompt (called \"visual clues\" by the authors), which is then fed to a language model like GPT-3. The language model then generates multiple candidate captions from the visual clues, which are then ranked by a contrastive model like CLIP. The authors also propose SPIPE, a new evaluation metric for measuring performance on image paragraph captioning. Using this metric, the authors show that their method outperform other baselines. The authors also conduct human evaluations, showing that their method typically outperforms previous ones, and can be competitive with human annotations.", "questions": "1) Is SPIPE correlated with human perception?\n\n2) Have authors considered exploring tasks beyond IPC? For instance, VQA.", "limitations": "To the best of my knowledge, authors adequately discuss the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1) The ideas presented in this work are (to the best of my knowledge) novel, and the research direction is timely.\n\n2) The experimental setup and results are solid.\n\n3) The paper is very clear and well written.\n\n**Weaknesses**\n\n1) One concern is that a great portion of the claims that their method is better than previous work rely on a metric proposed by this own work.  The authors explain why other evaluation metrics might not be ideal, but I believe it would still be informative to report them in the paper, especially since there exists image paragraph captioning benchmarks that use other evaluation metrics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept