ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions
Authors: Deyao Zhu
Jun Chen
Kilichbek Haydarov
Xiaoqian Shen
Wenxuan Zhang
Mohamed Elhoseiny
Venue: Accepted by TMLR ()

Abstract: Asking insightful questions is crucial for acquiring knowledge and expanding our understanding of the world. However, the importance of questioning has been largely overlooked in AI research, where models have been primarily developed to answer questions. With the recent advancements of large language models (LLMs) like ChatGPT, we discover their capability to ask high-quality questions when provided with a suitable prompt. This discovery presents a new opportunity to develop an automatic questioning system. In this paper, we introduce ChatCaptioner, a novel automatic-questioning method deployed in image captioning. Here, ChatGPT is prompted to ask a series of informative questions about images to BLIP-2, a strong vision question-answering model. In ChatCaptioner, we investigate whether two AI models, unable to individually describe images in detail, can collaborate through an automated, visually guided dialogue to generate a better and more enriched image description than a single AI model. We conduct human-subject evaluations on common image caption datasets such as COCO, Conceptual Caption, and WikiArt, and compare ChatCaptioner with BLIP-2 as well as ground truth. Our results demonstrate that ChatCaptioner's captions are significantly more informative, receiving three times as many votes from human evaluators as BLIP-2 alone for providing the most image information.
Besides, ChatCaptioner identifies 53% more objects within the image than BLIP-2 alone measured by WordNet synset matching.

Published in Transactions on Machine Learning Research (03/2024)
ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions
Deyao Zhu, Jun Chen", Kilichbek Haydarov', Xiaoqian Shen, Wenxuan Zhang, Mohamed Elhoseiny
King Abdullah University of Science and Technology fdeyao. zhu, jun.chen chen, kilichbek. haydarov, ziaogion. shen, santuon.zhang. mohamed, elhoseiny) @kaust. edu.
Reviewed on OpenReview: https: // openrrpeeeeeee ew. net/ net/forunt id= 1Lo VwFkZNo
Abstract
Asking insightful questions crucial for acquiring knowledge and expanding our under- standing of the world. However, the importance of questioning has been largely over- looked in AI research, where models have been primarily developed to answer questions. With the recent advancements of large language models (LLMs) like ChatGPT, we dis- cover their capability to ask high-quality questions when provided with a suitable prompt. This discovery presents a new opportumity to develop an automatic questioning system. In this paper, we introduce ChatCaptioner, a novel automaatic-questionnng method de- ployed in image captioning. Here, ChatGPT prompted to ask a series of informative questions about images to BLIP-2, strong vision question-answering model. In Chat- Captioner, investigate whether two AI models, unable to individually describe images in detail, can collaborate through automated, visually guided dialogue to generate better and more enriched image description than a single Al model. We conduct human- subject evaluations on common image caption datasets such as COCO, Conceptual Cap- tion, and WikiArt, and compare ChatCaptioner with BLIP-2 as well as ground truth. Our results demonstrate that ChatCaptioner's captions are significantly more informative, re- ceiving three times as many votes from human evaluators as BLIP-2 alone for providing the most. image information. Besides, ChaiCaptioner identifies 53% more objects within the image than BLIP-2 alone measured by WordNet synset matching. Code is available at https://gi thub. http///iiioonnaa//aaaaaaaaaaaaaa
1 Introduction
Asking good questions is not only an essential component of effectively acquiring knowledge, but also plays a pivotal role in enhancing our intelligence and expanding our understanding of the world. Taking medical diagnoses as an example, doctors must ask patients targeted questions about their symptoms to gather relevant information and make accurate diagnoscs. Likewise, in scientific research, asking insightful questions is paramount to advancing knowledge and discovering new findings that may have far-reaching implications.
However, the primary focus in recent Al research has been on developing models that can better answer questions, like InstructGPT (Ouyang et al. 2022) in Domain Question Answering (Yang et al. 2015: Rajpurkar et al. 2016 Joshi et al. 2017) and BLIP-2 (Li et al. 2023) in Visual Question Answering (Atool et al. 2015 Goyal et al. 2017 Hudson & Manning 2019). Despite the significant progress in the question- answering models, their offectiveness in providing useful information heavily reliant on the quality of the questions they receive. In essence, these models depend on humans to ask insightful questions that can direct their generation of informative answers. If have an automatic questioning machine that keeps asking informative questions, the human questioners can be replaced and the question-answering models can be guided to provide more valuable knowledge automatically.
* "Equal contribution

---

Published in Transactions on Machine Learning Research (03/2024)
Instructions
ChatGPT Questioner
BLIP-2
Answerer
Describe this Image detail.
basket kittens
What the color of the basket?
brown
How many kittens the basket?
four
the
the
the basket?
black and white
What the texture of the basket?
wicker
What the background of the image?
grass
What the size the kittens the basket?
small
What is the position of the kittens the basket?
They are sitting
What is the shape of the basket?
round
What the breed the kittens?
not sure
Caption: The image depicts round wicker basket with four small black white kittens sitting it. The basket iss brown in color and is placed on green grass.
Figure 1: Example of the dialog between ChatGPT and BLIP-2: BLIP-2 fails to provide a detailed description in the first message exchange. More details about the image highlighted in red are obtained through multiple conversational interactions between the two models. Eventually, the questioner is able to produce a more detailed caption about the image by focusing on multiple aspects of the image.
Recent studies (Wei et al. 2022a Ouyang et al. 2022 Wei et al. 2020 Kojima et al. have highlighted the impressive zero-shot learning abilities of Large Language Models (LLMs) that are fine-tuned to follow instructions. These LLMs can perform new tasks in zero-shot manner when presented with well-crafted instruction prompts. We discover that such LLMs like ChatGPT (OpenAT 2022 have the ability to keep ask- ing new and contextually relevant questions when properly designed prompts given. With this capability in place, building an elfective automalic questioning machine is now feasible lask.
Based on our findings, we design an automatic questioning system on ChatGPT and integrate it into im- age captioning, where strong vision-langunge models like BLIP-2 [Li et al. 2023) are available to answer image-related questions. Our method, named ChalCaptioner, generates more informative and enriched im- age captions by asking relevant questions to incrementally gain more information. In detail, we design a prompting system that encourages ChatGPT to ask a serics of informative questions that maximize its knowledge of the image, building on previous questions and answers. Note that ChatGPT is pure language model and cannot. "see" any visual information. We present the inquired image to BL.I--2 and set it as the question answerer. At the end of the conversation, Chat is prompted to summarize the discussion into a few sentences as the final enriched image description. An example of the conversation between ChatGPT and BLIP-2 and the final caption is shown in F
In our experiments, we aim to investigate whether ChatCaptioner, an automated visual dialogue system between large language model ChatGPT and visual language model BLIP-2, acquire more visual information compared to the output of the visual language model alone. We evaluate ChatCaptioner's captions on sampled images from COCO [Lin et al. 2014), WikiArt (Salch & Elgamma] 2015), and CC (Sharma el. al. 2018) datasels based on the human subject evaluation experiment. Compared to BLIP-2's state-of-the-art direct captioning performance, ChatCaptioner receives three times as many votes from human evaluators for providing richer image information. Besides, ChatCaptioner identifies 53% more objects than BLIP-2 alone within the image. Results verify the benefit of good questions to acquire more knowledge from existing AT models and the effectiveness of modern LLMs to serve zero-shot. automatic questioners.
2

---

Published in Transactions on Machine Learning Research (03/2024)
BLI-2 Image Ipp
CheldPt
ChatGPT: Questioner
BLIP-2: Answerer
Laase,
ChaSSPT
BLIP
Task Instruction
Instruction
Task
Lag Paw
Describe this image in detail,
A man is stending in front of a ber with e lot of alcohol
Answets
Visual Question Answering
BLP
What the wearing in the image?
shirt and suspenders
Question Insinuction de
(ext Quest Question...
ChalgPt
What expression on the man's face?
Automatic Questioning
he is looking the camera
Summarization Nov sunmarize the ii
ChaIGPT
The Image shows a man standing in frant of bar with multiple alcohol bottles visible, including vodka, rum, gin, and tequila. man wearing blue shirt and suspenders and is looking
Conversation Summarization
Figure 2: ChatCaptioner contains a questioner ChatGPT and an answerer BLIP-2. We start with a task instruction to set the context and outline the problem for ChatGPT. Once the answer from BLIP-2 is received, we prompt the chat log and question instruction to ChatGPT to continue asking information-acquiring questions. Finally, we provide summarization instruction to ChatGPT to conclude the conversation as the image captions. Enriched details are highlighted in red.
2 Related Works
Learning to Ask Questions There is vast work in NLP community that focuses on question generation.
Question generation (Mostow & Chen 2009 Heilman & Smith (010)) is the task of generating a question
from a given passage and an answer. [Sun al. (2018): Duan et al. (2017); [Sun et al.
(2023) have concentrated
the generation of question words such as 'when', 'how', and 'why'. [Zhao et al. (2018) Tuan et al. (2020)
considered leveraging paragraph-level context to enhance question relevance and coherence. Recent hods
like Jia et al. (2020): Xiao et al. (2020): Liu et al. (2019); Ghanem et al. (2022); ; Wang et al.
(2022a Liu
et al. (2020); Sultan et al. (2020); Wang et al. (2020); Gou et al. (2023); Nema et al. (019)) have explored
different neural network architectures and training strategies for better performance. Tang et al. (2018)
and Wang et al. (2022c) have explored question generation within collaborative settings, suggesting a broad
spectrum of approaches refinc and diversify question gencration methodologics.
However, in cases when do not have the answer and need to ask questions for the answers, such methods are not applicable. Visual Question Generation (Mostafazadeh et al.) 2016 Zhang al. 2016) is a task aimed at gencrating natural and engaging questions for a given image. Several works like Patro et al. (2018) 2020); Li et al.] 22018;; Jain et al.] ;; Vedd et al. 22021 ; Shen el al. (019) have been proposed to solve this task. They focus on generating independent questions only and do not have the ability to keep asking new questions based on the previous questions. Our work differs from previous studies significantly. First, we focus on acquiring more knowledge via the generated questions, instead of just generating them. Secondly, our method can keep asking new and relevant questions based on previous questions. Third, our approach leverages modern large language models and requires zero training for questioning.
Large Language Model and Prompting Recent research (Brown et al. 2020 Kojima et al. Wei al,
2020 2022b Chung et al. 2022 Ouyang et al.
2022] has revealed the abilities of Large Language Models Chowdhery et al. 2022 to solve versatile tasks specified by 2020 shows the capability to learn new tasks by providing a Mhaii--of-Thouee methods few task examples provided in the prompt, named in-context learning. Moreover,
(LLMs) like GPT-3 (Brown et al. 2020) or PaLM
prompting. For example, GPT-3 [Brown et al.
[Kojima et al. Wei et al.] (220)) demonstrate that explicitly asking LLM to solve tasks stop-by-step in the prompt improves the performance significantly. Additionally, FLAN (Wei et al., 2022a Chung al.] 2022) demonstrates that. L.Ms with ruction tuning can accomplish tasks in zero-shot manner. Zhou et. al.
3

---

Published in Transactions on Machine Learning Research (03/2024)
2022 proposes the Automatic Prompt Engineer (APE) by utilizing LLMs to generate instruction-followin Zero- and few-shot prompts. Further studies, including InstructGPT (Ouyang et al., 2022) and Chat GPT (OpenAl 2022), show that the performance of LLMs can be enhanced even further by using reinforcement learning from human feedback (Christiano et al.) 2017 Stiennon et al. 2020 In our work, we leverage the instruction- rolloooon-oooo ability of LLMs and design prompls that enable ChatGPT to keep asking new questions
about images.
Image Captioning and Visual Question Answering Several works (Lin et al. 2014 Antol et al, 2015 Sharma ct al. 2018: Young ct al. 2014; Das et al.) 2017 de Vrics et al.
2017] have been curated
to explore the link between visual and linguistic information, facilitating the investigation of tasks such as image captioning, visual question answering (VQA), and visual dialog. Recent rescarch vision and language pertaining (Chen et al. 2022 Tsimpoukelli et al.) 2021 Alayrac et al., 2022 Wang et al., 2022b; Li et al. 2022 2023) has advanced the performance for image captioning and visual question answering
(VQA) by a large margin. For example, VisualGPT (Chen et al.) 2022) shows the benefits of initialization
with pretrained language models for more data-efficient training. Frozen (Tsimpoukelli et al.
2021 extend
it by finetuning a vision encoder and aligning it with frozen LLM. BEiT-3 (Wang et al. 2022b) and
BLIP Li et al. 2022) pretrain models using unified transformer architecture. Flamingo (Alayrac et al, 2022] proposes a cross-attention design to align vision and language. BLIP-2 (Li et al., 2023) introduces a lightweight Q-Former that converts visual features into tokens that can be directly understood by frozen LLM, and achieves impressive results on both image captioning and VQA tasks. Tn our work, our automatic questioning mechanism leverages the VQA capability of BLIP-2 to extract additional image information and enrich the image captions beyond the original BLIP-2 captions.
3 ChatCaptioner
In ChatCaptioner, we design an automatic questioning mechanism based on ChatGPT's zero-shot instruction-follwiii ability lo keep asking informative questions about images. BLIP-2, the vision-language model, then provides new image information according to the asked questions. Finally, ChatGPT is prompted to summarize the chat history and generate the final image captions with rich details. An overview of our method demonstrated in Fig 2
3.1 Automatic Questioning
To activate the questioning ability of ChatGPT, we design a prompting system that enables ChatGPT to generate questions based on previous chat logs. Our prompting system for ChatGPT contains three components: a task instruction for explaining the task PtaskQ: chat log to store previous questions and answers Pchat, a question instruction for generating high-quality questions Pg. Each question is generated given the context + Py. In addition, we design question-trimming mechanism for the automatic post-processing of the generated questions.
ChatGPT Task Instruction PtaskQ The task instruction PlaskQ sets the context and outlines the task that ChatGPT is required to perform. PtaskQ directs ChatGPT to generate questions that aim to extract as much information as possible about an image. PtaskQ is designed as follows:
have an image. Ask me questions about the content of this image. Carefully asking me informative questions to mazimize your information about this image content. Each time ask one question only without giving an answer. Avoid asking yes/no questions. I'll put my answer beginning with "Answer:".
The prompt "Fach time ask one question only without giving answer" in PtaskQ is designed to instruct ChatGPT to generate only one question per round. Without this prompt, ChatGPT may generate set of questions at once, rather than generating them one by one in response to each answer.
Chat Log Pehet After the task instruction PtaskQ, we include a chat log Pehot containing all the previous questions and answers. Pchat follows template for each Question-Answer round, which is as follows:
4

---

Published in Transactions on Machine Learning Research (03/2024)
Describee this imge in deteil
Deserbbe this image in detai.
Describe this image detail
Deazrite this image in detai.
A.dog looking out the window of cat What fiwee the image?
man surting in the coean
oa ning red, painting?
paint
imager?
What subject wbstract painting
what doing
a curny day
She siting chai
What color the dog's furt
What dominant colors the painting
the
blke
black, rod, Muc
commnn' the image?
brown
Wher type car the deg in?
What the man weartig in the im emgerr
What is the size of the painting?
What type
stting
yellow shirt
cheir with red cualion
what the sconery outside the car?
What the man holding hand
while surting
Are there any idontifialle shapes objects painting?
What is the woinnn's facial exp rsssion?
no
she is locking ot the vicner What the woman's posture like?
What day is in image?
man's position surfboare?
what the painting?
abstract
not suru
What texture of the
stiting chatr
What the expression on the dogl' tace??
surfboard?
smooth
What is the background of the image like?
Whet the dog looking nutside the window
What man'ss fectal image?
white
focling conveyed by the sainting?
sente of mystery
What is the woman's age range image?
red wall
srrkiing
What the composition the oe ming Beer?
betweeen 40 and 50
traffic light
What the distar co betwoen the man and the shore?
Wher the woman's ethnicity the image?
What light the la look
traffic
red painting
French
far from
What is the arraggement of crorr the painting?
What is the woman's hair
green
What ard the other th tggs in
What the dog's prsttton the car?
Mack and blue
long
back window
from and the
beach
This abstract parningg with dommnnnt
cxtors of black, red, and blus. The parnting
with dark brown long heir sitting on chail"
depicts
back cress
The image a white wooking out
The
man surfing
has ideetallable shapes sbjects
looking the The background of the image red wall.
with red custton,
day the bue ocean, He is wearing
oventil conveyed by the paintinng is sence of
smooth texturs
looking mg a green traffic light a city. The dog tooks curouus.
yallro ahrr holding white iouadd
mystery. compostion the painiing is
The woman French ethnictty.
the badiground, there a beach
mantly
Describbe Lhie mage in deteil.
Desr be this image in dataell
image status of building
Describe INs image
group zhildeen are running weound
boy de ving cur thruugh fence What the age the in the image?
[A men plaging guitee from of emood What is the man wearing while
chair in yardd
What is the gender of the chiichen the image?
What is the color the ion statue?
playing the gultar?
What is the ago range of the childrn the imager?
boys and girts
What type car is driking
image?
Wind ng baskgroune the image?
gray
What is the pender of the crowd watching the man play the guitor?
blue shirt
What the ethnicit the childros in the image?
between 3 and 10 years sld
What is the con tion the fence
after the car possed through
What is the pase of the lion statue?
tample
What is the excression on the face of the man playnng the gultar?
female
frn sbotue is standing front of temple
damaped
What is the size of the lion statue
heppy
what the bockground of the Imaaee?
What the uttinictty of the imaeai?
in relation to the tumpler
What tyge of guitar is the man playing?
What the weather like Image?
yard
What is the roprocimate time of day the image?
Adoon
What is the positiun the atatum in on the temple?
the lin statue small
What is the her color of the man
electric gutter
surny
maming
the statue is front of the templl the
playing the putar??
the image??
dathing like
What is the boy wearing the mage?
atatua??
What is ethrucity of slaying
They are wearing shorts and shirts
shirt
Ion statue front of temppte
Hizaanic
What the or of the chair in the image?
What's is the color the fence that was comaged by the boy diiving the cor?
What the material make the linn statue?
What is the age range of the people in the watching the play guttw??
What the expression the lonen's faces
white
brown
What is the strle of the temple
20 40
cosiczed the image?
the backgroun of the image?
What the setting the image?
What is the pase of the children in the image?
happy
What is the make and model of the that the boy is driving in the image?
What is the name location
ENiessC
What is the day tie image?
rurning
grean
chinese zomple
early evening
The shows group boys and girls, aped between 3 and 30 years old, running a white in sunny chridren wearing spower to be happy.
The mape depicts young boy drivigg
statue shanding front a Chinese
The image shows small stone loon
The image depicts Hispontc man playing
rad elediric gui park during early
temple
has
momning on a reed. The fence was camaged
the ion status standing posc. The
heppy expression on Nis fhoe. "The crowd
anirts and
by the car. The bey weating shirt. The
exp ression on the face of the ton status is
watching him cons fermates in the age
temole
Figure 3: Qualitative examples of the chat log and the captions from ChatCaptioner in various images from COCO (Lin et al. 2014), WikiArt (Salch & Elgammal 2015). and CC (Sharma ct al. 2018). Compared to the initial description from 2LIP-2, questions from ChatGPT extract additional image information and lead to enriched final captions.
5

---

Published in Transactions on Machine Learning Research (03/2024)
Question: (question) Answer: (answer)
Note that we hard-code the first question as "Describe the image in detail" to start the conversation. Based on the first response of BLIP-2, which provides brief initial description of the image, ChatGPT is prompted to ask follow-up questions to extract more information about the image.
Question Instruction Py To guide ChatGPT in generating new questions, we provide a question instruc- tion Py before each question generation. Pq. located after the chat log, cues ChatGPT to generate a new question and aims to ensure that the questions are of high quality. It's designed as follows:
Next Question. Avoid asking yes/no questions. Question:
The prompt "Next Question" in Py is critical to ensure that ChatGPT contimues to ask questions. Without it, ChatGPT may produce undesired sentences after few Question-Anss Answer rounds. Additionally, we notice that ChatGPT prefers asking yos/no questions which are usually not as informative as other questions. We therefore add the prompt "Avoid asking yes/no questions" reduce the generation of yes/no questions.
Question Trimming Despite our explicit instruction to not answer the question itself, we observe that sometimes ChatGPT fabricates an answer after asking the question. Fortunately, we find that these fab- ricated answers always begin with the text. "Answer:", following the template specified in the prompt. Therefore, we automatically remove these fabricated answers by discarding the generated text starting from "Answer:".
3.2 Question Answering
Similar to ChatGPT, our BLIP-2 prompting mechanism consists of three components: task instruction PlaskA, the chat log Pehot same as the ChatGPT one, and an answer instruction Pa- Each answer generation is prompted by PtaskA + Pchat + Pa- Also, we have an answer-Irimming mechanism for post-processing.
BLIP-2 Task Instruction PtaskA We design the BLIP-2 task instruction PlaskA to alleviate the issue of hallucinating non-existent information in the image. PlaskA includes an uncertainty prompt "If you are not about the answer, say you don't. 't know honestly" that encourages BLIP-2's honest. admission of lack of knowledge. The instruction is as follows:
Answer given questions. If you are not sure about the answer, say you don't know honestly. Don't imagine any condents that are not in the image.
Answer Instruction
After the chat log Pehar, we provide a straightforward answer instruction to guide
BLIP-2's answering process. The instruction is structured as follows:
Question: (question) Answer:
Answer Trimming Similar ChatGPT, BLIP-2 occasionally gencrates a question after providing an answer. As the LLM backend of BLIP-2, the FLAN-T5 model (Chung et al.) 2022), has much weaker questioning ability than ChatGPT shown later in the question analysis in the experiment section, we auto- matically filter out these questions by discarding any texts starting with "Question:".
3.3 Context Summarizing
To obtain a concise summary of the conversation between ChatGPT and BLIP-2 as the final image caption, we use a summarization instruction after the conversation. This instruction, located after the chat log, prompts ChatGPT to generate a summary using the following structure:
Now summarize the information you get in a few sentences. Ignore the questions with answers no or not sure. Don't add information. Don't miss information. Summary:

---

Published in Transactions on Machine Learning Research (03/2024)
BLIP-2:
person is snowboarding down mountain
ChatCaptioner:
The image depicts male snowboarder wearing blue jacket and pants, snowboarding down snowy mountain under sunny weather. The snowboard and the mountain are both blue and holding snowboard.
person
CIDEr
ROUGE
METEOR
BLIP-2
83.7
61.1
24.5
ChatCaptioner
0.0
23.9
18.7
An example of the limitation of traditional image caption metrics. Although Chat Captione extracts more image details highlighted in red compared to BLIP-2, it receives much lower scores under traditional metrics.
Figure 4:
4 Experiments
We explore ChatCaptioner from various perspectives through series of experiments, including assessment of the informativeness and accuracy of its captions, as well as analysis of the quality of its generated questions. Qualitative results including chat logs and final captions on various images from different datasets are shown in Fig
Details of Model Deployment. For our experiments, we use the ChatGPT model "gpt-3.5-turbo available on the OpenAI API (OpenAT, 2023a). This model the most powerful GPT-3.5 model accessible through the API during our project. For BLIP-2, we the biggest. version containing a FLAN-T5 (Chung et al. language model with 11 billion parameters and VIT-G/14 model from EVA-CLIP (Fang et al, In all experiments, BLIP-2 answers 10 questions per image, with the first question being hard-coded
2022
2022
88
Describb the image in detail.". " The remaining questions are from ChatGPT, unless otherwise specified.
Limitation of Traditional Metrics. The conventional image captioning metrics such as Vedantam et al, 2013), ROUGE (Lin & Hovy 2002), and METEOR (Banerjec & Lavic 2005)) are frequently employod measure the quality of captions. However, the usage of these metrics to evaluate ChatCaptioner can be limiting, because these metrics designed to measure the similarity between the tested caption and reference captions, assuming that the reference captions perfect image descriptions. Since Chat generates captions that contain a greater level of detail than the reference captions, the metrics may yield low similarity scores, leading to inferior overall performance scores compared to other image captioning approaches like BLIP-2. This issue is depicted in Fig Thus, in our experiments, we primarily rely on human assessments to comprehensively analyze the performance of Chat from various perspectives. The design of all the human evaluation interfaces presented in the supplementary.
4.1 Information Analysis
Does Chat ChatCaptioncs extract more information from the image? We design experiment to evaluate whether Chai is able to generale captions with more information about the image than BLIP-2 alone and ground truth captions. We randomly selected 100 photos from the COCO (Lin et al. 2014 validation set, 100 artworks from WikiArt (Saleh & Elgammal 2015 dataset with ground truth captions from ArtEmis (Achlioptas et al. 2021). 100 internet images from the Conceptual Captions (CC) Sharma et al. 2018 validation dataset, and 100 images with detailed and long ground truth captions from the Open Image
Table 1: Human votes on the captions containing the most image information.
Mothods
COCO
WikiArt
CC
OI-LN
Avg.
GT
26%
14%
8.5%
33.5%
20.5%
BLIP-2
21%
12.5%
23%
6.5%
15.8%
Ours
53%
73.5%
68.5%
63.8%
65%

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Numbers objects discovered by captions.
Methods
Covered/ All
Ratio
Improved
BLIP-2 383
33.2%
Ours
586 195/
50.8%
53.0%
Table 3: Correctuess Analysis. BLIP-2 correctly answers 66.7% of ChatGPT's questions. 81% of the final captions are deemed correct by humans. Besides, 94% of the wrong captions are caused by BLIP-2's wrong answers.
COCO
WikiArt
CC
Avg.
Answer Correct Rate
64%
73%
63%
66.7%
Caption Correct Rate
77%
78%
88%
81 %1%
Tssues From BLII--2
100%
82%
100%
94%
Localized Narratives (OI-LN) (Pont-Tuset et al. 2020g dataset. Human evaluators on Amazon Mechanical Turk are presented with an image and lour caplions one from our method, one from BLIP-2, one ground truth caption, and one fake caption for quality control. Evaluators are asked to pick the caption that offers the richest information about the image. Results are demonstrated in Tab On average, Chat receives three to four times as many votes as pure BLIP-2's captions and ground truth captions, showing that by combining two AT models via questioning and asking, the system can provide more detailed image description than single Al model alone.
How many objects in images can Chat Captioner discover? We randomly sampled 200 images from Pascal VOC (Everingham et al. 2010) and considered all class labels in the segmentation masks as the ground truth objects. We then assessed how many of these objects are included in the captions. We utilize WordNet. from NLTK (Bird et al., 2009) to find words with similar semantic meanings based on the Wu- Palmer Similarity of their synsets. Tab 2 presents the experimental results, where 1154 objects are identified in the 200 sampled images. BLIP-2 covers only 383 of them, while ChatCaptioner increases the coverage by 53% to 586, suggesting that the automatic questioning helps BLIP-2 find more objects in the images.
4.2 Correctness Analysis
How accurate are the captions from Chat Captioner? We conducted human evaluation where evaluators were presented with an image and generated caption, as well as all Q&AA between ChatGPT and BLIP-2. The evaluators need to verify the correctness of the caption, select incorrect answers from BLIP-2, and judge whether the incorrectness can be altribuled to the wrong answers. The experiments were performed on samples from COCO [Tin et al. 2014), WikiArt. (Salch & Flgammal 2015). and CC (Sharma et al. 2018) datasets. Each image was evaluated by 4 different evaluators. Results presented in Tab reveal that about 80% of the captions correctly. Among the incorrect captions, 94% are caused by BL.IP-2's s wrong answers, suggesting that BLIP-2 is the primary source of incorrectness. This implies that using a more powerful VQA model may help to enhance the overall performance of the system in the future.
deemed correct. Moreover, BLIP-2 answers around 67% of the questions
Does BLIP-2 know doesn't know? BLIP-2 usually makes up answers the question cannot be answered based on the image. In other words, BLIP-2 doesn't know that doesn't know this information. To mitigate this issue, we incorporate an uncertainty prompt "If you are not sure about the answer, say you don't know honestly." " in our BLIP-2 task instruction PlaskA. Two examples showing the effectiveness of the uncertainty prompt are demonstrated in Fig and more examples are in Fig and the supplementary.
How effective is the uncertainty prompt? We randomly selected 200 images from the CC (Sharma et al. 2018) and collected 1,800 questions based on these images. We then identify 147 questions that BLIP-2 is uncertain about, present these questions human evaluators, and ask them answer based on the image content. Results in demonstrrae that approximately 60% of these questions are doemed unanswerable based the image. For the remaining answerable questions, BLIP-2 cannot correctly answer 30 of them.
8

---

Published in Transactions on Machine Learning Research (03/2024)
What is the gender and age of
What the name of the store green awning? Uncertainty Prompt
person peppers?
Uncertainty Prompt Male, young
A gas station With Uncortainty Prompt
With Uncortainty Promat Don't know
Not sure
Figure 5: Examples of BLIP-2's answers with and without the uncertainty prompt. The uncertainty prompt helps BLIP-2 avoid making up answer when it encounters questions that cannot be answered based the image.
Table 4: Analysis questions that BLIP-2 unsure about. 60% deemed unanswerable by humans. 20% cannot be correctly answered by BLIP-2. Overall, BLIP-2 makes mistakes on 80% of these questions.
Total Uncertain Questions
Unansweabbl Questions
Answerable But Wrong
Avoided
Answers
Num.
147
83
30 20%
118
Ratio
60%
80%
In total, without the uncertainty prompt, BLIP-2 will generate 118 incorrect answers out 147 uncertain questions, resulting an error rate of approximately 80%. In addition, out of the original 1800 questions, BLIP-2 has 674 wrong answers. Taking the 147 potential wrong answers avoided by the uncertainty prompt into account, the uncertainty prompt. reduces about 15% of the wrong answers.
4.3 Question Analysis
How diverse are the automatically generated questions? We analyze 1419 unique questions filtered from 1800 generated questions on 200 random CC Sharma et al. 2018 samples. To visualize the diversity
there any vegetable the image?
What is the expression the thelgifls face?
there any writing or logo truck?
How many chandeliers are in the lobby?
What the overall mood the image?
Are there any people the image?
What size the status compared the temple?
What type of drone is the soldier using?
What the gender of the person the image?
What material the clothes in cloest made
What is the sky in the image like?
Are any of the runners wearing hal?
What is the woman holding in her hand?
What color are men's beards?
What the wearing?
Figure 6: Visualization of question embeddings and question examples printed at the corresponding positions. Our method can ask diverse questions focusing on various perspectives of the image.

---

Published in Transactions on Machine Learning Research (03/2024)
Table 5: Number of unique questions. InstructGPT and ChatGPT excel at generating diverse questions and rarely repeating questions within a dialogue.
Unique Q/Total Q
OPT 6.78
FLAN-115
InstructGPi
ChatGPP
Per Dialogue
1.75/9
2,03/9
9/9 1400/1800 1800
8, 8.98/9
All Questions
166/1800
169/1800
1419/1800
0UP-2: man in hard hat looking of yellow valve
DPT 6.78
DUP-2
BUP-2
What name the man the hand hot?
wearing?
don'tt know
hard hat
What color is the valve??
the hard hot?
don't know
What holding his hand?
the
hard
clipboard
know
What the name the man
protective gear?
don't know
the doing the image? inspecttng a large yellow tank
What the name the man in the hard hat?
don't
FLAN-TS 13B
BLIP-2
BLIP-
ChateIPT
BUP-2
What is the man wearing?
yellow
What is the man doing with the valve?
value
Inspecting
What is the man looking st?
Next Question,
What the the mun's hand hat?
peper
industrial syserm
What is the man ng?
many messages on twiller yes yestendrr
red
What the materfal of the valve!
clipboard
What
hdtp
What the size the valve?
a hard hat
http
smail
What is the man wearing?
an express train
the cutdours?
frain travelx a high speed
Figure 7: Question examples from various LLMs. ChatGPT and InstructGPT are able to ask new and informative questions. FLAN-T5 11B and OPT 6.7B tend to repeat old questions. GPT-3 fails to ask any related questions.
of the questions, we compute the question embedding based on Sentence-BEie [Reimers & Gurevych, 2019) and reduce the dimension to two for visualization via UMAP (McInnes et al. 2018 The resulting visualization is presented in Fig
Our analysis shows that the questions from ChatCaptioner are diverse and cover various perspectives, in- cluding ligh-level information like "What is the overall mood of this image?", object properties like "What material are the clothes in the closet made of?", environmental information like "What is the sky in the image like?", and so on. This suggests that our automatic questioning mechanism is able to generate diverse questions, explore various aspects of the image, and acquire more informative responses.
Can we use other LLMs as questioners? In addition to ChatGPT, we evaluate the automatic ques- tioning capabilities of several other large language models (LLMs), including InstructGPT (Ouyang et al. 2022), GPT-3 (Brown et al. 2020) without instruction finetuning, FLAN-T5 11B (Chung et al. 2022) (11B parameters), and OPT 6.7B (Zhang et al., 2022) (6.7B parameters). Fig showcases the different question- asking abilities given the initial description from BLIP-2. We observe that InstructGPT shows similar question-asking ability as ChatGPT and is able to generate meaningful and diverse questions. While FLAN- T5 11B and OPT 6.7B also exhibits the ability to ask questions, it repeats a single question several times. Interestingly, GPT-3 fails to generate any meaningful questions related to the image.
We further analyze their 1800 generated questions on 200 CC (Sharma al. 2018) samples. We skip the poorly performed GPT-3 here. LLMs' questioning abilities are evaluated in two ways. Firstly, we measure the average number of unique questions per image, which indicates whether the model can keep asking new
10

---

Published in Transactions on Machine Learning Research (03/2024)
ChatCaptioner (ChatGPT+BLIP-2))
In the image, baby is standing in front of woodon table. The baby is wearing a brown hoodle and appears to be curlous. On the table, there is a small brown cupcake with white candle. The baby is holding fork and the cupcake, but not looking direcily at the camera.
GPT-4V:
The image shows a young child staring at small birthday candle on top of what appears to be muffin or a small cake. The child holding fork and standing close to a wooden table with the cake placed on a white plate. There's mug the left side the plate, and you part of another person to the right side of the photo, though they are mostly frame. scene suggests birthday celebration. child seems curious or thoughtful about the lit candle.
Figure 8: An qualitative comparision between GPT-4V and ChatCCpttioee (ChatGPT + BLIP-2).
Table 6: Ablation study with the GPT-4V answerer.
Questioner
Answerer
Human Votes
ChatGPT GPT-4V
92%
ChatGPT
BLIP-2
8%
questions in a single dialogue. Secondly, we count the total unique questions to see if the questioner could customize the questions according to the given contexts or just ask fixed predefined questions.
Results in Tab reveal that ChatGPT and InstructGPT almost never repeat their question in single dialogue and generate around 1400 unique questions out of 1800, suggesting that they able to ask new questions according to the contexts. In contrast, FLAN-T5 11B and OPT 6.7B tend to repeat old questions, averaging about only 2 unique questions per image and generating less than 170 unique questions in total. Our study suggests that to develop a questioning machine that can generate novel and customized questions, it may be necessary to utilize instruction-finttnnc LLMs with dozens of billions of parameters.
4.4 Exploration with GPT-4V
How are the results when combined with GPT-4V? With the rapid advancement in this field, advanced vision-language models with robust capabilities for detailed image captioning, such as GPT-4MOpenAT (2023b), are now accessible. To evaluate the caption quality between ChatCaptioner (ChatGPT+BLIP-2) and GPT-4V, we randomly selected 20 COCO images and instructed human anno- tators to choose their preferred captions. The results indicate a unanimous preference for captions generated by GPT-4V in all 20 instances. A qualitative example is provided in Figure S Generally, captions from GPT-4V encompass greater number of objects and location details, illustrating features such "There's mug to the left side of the plate", in comparison to those produced by ChatCaptioner (ChatGPT+BLIP-P)
Can a better vision answerer improve ChatContioner? In this study, we substituted GPT-4V for BLIP-2 as the answerer, while retaining the original questioner. The experimental results, detailed in Table D reveal that human annotators prefer the pairing of the ChatGPT questioner with the GPT-4V answerer over the original ChatGPT questioner and BLIP-2 answerer in 92% of instances. This preference highlights the significance of proficient answerer within our system. The suboptimal performance of the Chat Chptioene system when paired with the BLIP-2 answerer may be attributable to the incorrect responses provided by BLIP-2, discussed in Section 4.2
Can ChatCaptiones further improve GPT-4V? ChatCaptioner is an automated question-answering system designed to extract more information from images than a single model alone. The design of Chat- Captioner is orthogonal to the choice of underlying models. In our experiment, we integrated ChatCaptioner
Table 7: Human votes on the captions containing the most image information.
Methods
GPT-4V alone
GPT-4V + ChatCaptioner
Votes
3%
97%
11

---

Published in Transactions on Machine Learning Research (03/2024)
with GPT-4V, where GPT-4V serves as both the questioner and answerer within the ChatCaptioner frame- work, with all other settings remaining constant. Human evaluators were tasked with selecting captions that most accurately capture the details of an image, comparing the standalone output of GPT-4V against its performance within the ChatCaptioner framework. The results, presented in Table 7 indicate that in 97% of cases, captions produced by GPT-4V combined with Captoonr contain more detailed image infor- mation, thus confirming the effectiveness of our proposod framework eliciting additional details through its question-and-answee mechanism.
5 Conclusion
In this work, we discover that advanced large language models possess the ability to pose insightful and diverse questions when provided with well-crafted prompts. Based on our findings, we develop an automatic questioning system named ChatCaptioner for the task of image captioning. By prompting ChatGPT to keep asking questions that expand its understanding of an image, ChatCaptioner guides BLIP-2 to provide com- prehensive image information, resulting in image captions that are significantly more detailed and enriched than those from BLIP-2 alone. ChatCaptioner demonstrates the power of automatic questioning systems to effectively extract desired information. Through our work, we aim to draw atlention lo the potential of automatic questioning systems in Al and inspire further research in various domains.
References
Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J Guibas. Artemis: Affective language for visual art. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 11569 11579, 2021.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 2423-2433, 2015.
Satanjeev Banerjee and Alon Lavie Meteor: An automatic metric for mt evaluation with improved corre- lation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation mensures for machine translation und/or sumomrrizalinn, pp. 65 72, 2005.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Pylhon: analyzing terl with the natural language toolkit. O'Reilly Media, Inc.", 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in newral information processing systems, 33:1877 1901, 2020.
Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion, pp. 18030-18040, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Youghao Zhuang, Joseph E. Gouzalez, lon Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URI. 9023-03-30-vicuna//
org/blogg
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ariio preprint Xxx::2244.0211 2022.
12

---

Published in Transactions on Machine Learning Research (03/2024)
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment. learning from human preferences. Advances in neural information processing systems, 30, 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetueed language models. arXiv preprint ar arX1::2210.114 114,1, 2022.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 326 335, 2017.
Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pielquin, Hugo Larochelle, and Aaron Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. Question generation for question answering. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Meth- ods in Natural Longuage Processing, pp. 868-8666-777 Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10. 1..11863v1/DDD/ / 1090. URL http:s///ccaatthoooggoorr///////
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303 338, June 2010.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint ar w:Xiv:2211007632 2022.
Bilal Ghanem, Lauren Lutz Coleman, Julia Rivard Dexter, Spencer McIntosh von der Ohe, and Alona Fyshe. Question generation for reading comprehension assessment by modeling how and what to ask. arXiv preprint XrXi::2204, 02900,, 2022.
Qi Gou, Zehua Xia, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li, and Nguyen Cam-Tu, Diversify question generation with retrieval-augie -angeeal-angmmnneed style transfer. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1677-1690, Singapore, December 2023. Association for Computational Linguisties. doi: 10.18653/v1/ 2023.emnlp-mainn main. 104. URL httpps://aclaathoolggg. .rg.lloooo.ooogoggeoeeeee wrw.//23...22/// 2023. emnlp canlp-main. 104
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 6904-6913, 2017.
Michael Heilman and Noah Smith. Good question! statistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 609-617, 2010.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Unnal Jain, Ziyu Zhang, and Alexander G Schwing, Creativity: Generating diverse questions using varia- tional autoencoders. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. (485- 2017.
Xin Jia, Wenjie Zhou, Xu Sun, and Yunfang Wu. How to ask good questions? try to leverage paraphrases. In Proceedings of the 58th Arneual Meeting of the Association for Computational Linguistics, pp. 6130 6140, 2020.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly super- vised challenge dataset for reading comprehension. arXiv preprint arvi: 1xxx:1705.000 170503355, 2017.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imag pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: with frozen image encoders and large language models. arXiv preprint Xrx::23011. 12597, 2023.
Bootstrapping language image pre-training
Yikang Ti, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. Visual question generation as dual task of visual question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognation (CVPR), June 2018.
Chin-Yew Lin and Eduard Hovy. Manual and automatic evaluation of summaries. In Proceedings of the ACL-02 workshop on aulomalic summarizalion, pp. 45 51, 2002.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference computer vision, pp. 740-755. Springer, 2014.
Bang Liu, Mingjum Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu. Learning to generate questions by learning what not to generate. In The world wide web conference, pp. 1106-1118, 2019.
Bang Liu, Haojie Wei, Di Niu, Haolan Chen, and Yancheng He. Asking questions the human way: Scalable question-answer generation from text corpus. In Proceedings of The Web Conference 2020, pp. 2032 2043, 2020.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approxima- tion and projection. Journal of Open Source Software, 3(29), 2018.
Nasrin Mostafazadeh, Tshan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. Generating natural questions about an image. In Proceedings of the 54th Annual Meeting of the Association for Compululional Linguislics (Volume 1: Long Papers), pp. 1802 1813, 2016.
Jack Mostow and Wei Chen. Generating instruction automatically for the reading strategy of self-questioning. In AIED, pp. 465-472 2009.
Preksha Nema, Akash Kumar Mohankumar, Mitesh M Khapra, Balaji Vasan Srinivasan, and Balara- man Ravindran. Let's ask again: Refine network for automatic question generation. arXiv preprint arXiv:1909.05388 2019.
OpenAT. Introducing chatgpt. http:///peeud.ccm//laaa/tbba/u 2022.
OpenAL. Openai api: Gpt-3.5. https://platform.pponnt htt.://aw...coooooc///////////// http://docs//odlll/gg--3--53 2023a. Accessod:
March 7, 2023.
R OpenAL. Gpt-4 technical report. arXiv, pp. 2303 08774, 2023b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. Tn Advances in Neural Information Processing Systems, 2022.
Badri Patro, Vinod Kurmi, Sandeep Kumar, and Vinay Namboodiri. Deep bayesian network for visual question generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
Badri Narayana Patro, Sandeep Kumar, Vinod Kumar Kurmi, and Vinay Namboodiri. Multimodal differen- tial network for visual question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4002-4012, Brussels, Belgium, Computational Linguistics. doi: 10. 111533/11DD18-1433 URI. htt.:ttpp///claattoooooooo//////
2018. Association for
14

---

Published in Transactions on Machine Learning Research (03/2024)
Jordi Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part V 16, pp. 647 664. Springer, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint Xrv: 1606. 05250, 2016.
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint Xix::11088 10084, 2019.
Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric the right feature. arXiv preprint (=Xi::150500055 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Tingke Shen, Amlan Kar, and Sanja. Fidler. Learning to caption images through a lifetime by asking questions. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 10393 10402, 2019.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008 3021, 2020.
Md Arafat Sultan, Shubham Chandel, Ramón Fernandez Astudillo, and Vittorio Castelli. On the importance diversity in question generation for QA. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5651-56660 155615656,60 Online, July 2020. Association for Computational Linguistics. doi: 10. 00.18633/v1/0000 acl-main. URL https:///aclannhologyy
500
Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. Answer-focused and position- aware neural question generation. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 3930-3939, 2018.
Xingwu Sun, Hongyin Tang, et al. Inflected forms are redundant in question generation models. arxiv preprinl Xrx::22301.000 00397, 2023.
Duyu Tang, Nan Duan, Zhao Yan, Zhirui Zhang, Yibo Sun, Shujie Liu, Yuanhua Lv, and Ming Zhou. Learning to collaborate for question answering and asking. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Länguisties: Human Language Technologies, Volume 1 (Long Papers), pp. 1564 1574, 2018
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 200- 212, 2021.
Luu Anh Tuan, Darsh Shah, and Regina Barzilay. Capturing greater context for question generation. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 9065 9065-9072, 2020.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 4566- 4575, 2015.
Nihir Vedd, Zixu Wang, Marek Rei, Yishu Miao, and Lucia Specia. Guiding visual question generation. arXiv preprint arXio:2110.0822 082622 2021.
Qifan Wang, Li Yang, Xiaojun Quan, Fuli Feng, Dongfang Liu, Zenglin Xu, Sinong Wang, and Hao Ma. Learning to generate question by asking question: A primal-dual approach with uncommon word gener- ation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
46-61, 2022a.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXie:2208.104 2022b.
Xu Wang, Simin Fan, Jessica Houghton, and Lu Wang. Towards process-oriented, modular, and versatile question generation that meets educational nceds. In Marine Carpuat, Maric- Cathe-Ca de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 291-202100 191-302, Seattle, United States, July 2022c. Association for Computational Linguistics. doi: 10.18653/vv (33633/vl/// /20222 naacl-main 22. URL https://aclantholl/
org 2022. naacl-main, 22
Zhen Wang, Siwei Rao, Jie Zhang, Zhen Qin, Guangjian Tian, and Jun Wang. Diversify question generation with continuous content selectors and question type modeling. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguisties: EMNLP 2020, pp- 2134-2143, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1 http//20022.nnnngeeeeemmee URL httpe://aclanttooooyyooom http://22..finnniinie eertiings-ennl.....
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2020.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Lo. Finetuned language models are zero-shot learners. In International Conference on Learning Represenlations, 2022a.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Trans- actions on Machine Learning Research, 2022b.
Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-gen: An en- hanced multi-flow pre-training and fine-tuning framework for natural language generation. arXie preprint arXiv:2001.1331, 2020.
Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2013-2018 2015.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual de- notations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Shijic Zhang, Lizhen Qu, Shaodi You, Zhenglu Yang, and Jiawan Zhang. Automatic generation of grounded visual questions. arxiv preprinl ar Xr:: 161.20630,, 2016.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint Xiv:2200 01068, 2022.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 3901 3910, 2018.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human- level prompt engineers. orXiv preprint arXio:2211.019. 2022.
16

---

Published in Transactions on Machine Learning Research (03/2024)
A
Appendix
A.1 Cost
Our method is based on the ChatGPT model, specifically the apt-3.5-turlo version which we access through OpenAT's API. At the time of our project, the cost for using 1000 tokens in 9p4-8.5-tarbo was 0.002 US Dollars. On average, we spent approximately 2500 tokens for each image for ten Question-Anse rounds, which translates to cost of approximately 0.005 US Dollars per image.
A.2 Yes/ No Question Ablation
Usually, ycs/no questions contain relatively less information. To reduce the generation of yes/no questions from ChalGPT, we explicitly add a prompt "Avoid asking yes/no questions" " in the lask instruction PtoskQ and the question instruction Py. Our ablation study in shows that this prompt reduces the generation of yos/no questions from 33% of the cases to 2% in 1800 questions 200 random CC [haarma et al. (0018 samples, verifying its effectiveness.
Table & Effectiveness of the yes/no prompt.
Total Question
Yee/No Question Promps
Yag/No Prompt
Ratio
35%
2%
A.3 Using Vicuna Questioners
In addition to the open-sourced LLMs Flan-T5 11B and OPT 6.7B in the main paper, here conduct experiments with one of the latest SOTA open-sourced LLMs, Vicuna 13B Chiang et al. (2023), as the questioner and evaluate the generaled question quality as shown in Tab Experimental results suggest that Vicuna is able to generate diverse questions like ChatGPT and rarely repeats the questions in a single 9-question dialogue.
Table 9: Number of unique questions.
Unique Total Q OPT 6.7B
FLAN-T5
Vicuna
ChasGPT
Per Dialogme
1.75/9
2.03/9 169/1800
8.88/9
8.98/9
All Questions
166/1800
1710/1800
1419/1800
17

---

Published in Transactions on Machine Learning Research (03/2024)
A.4 Human Evaluation Interface
Ploce doale inforantion
decde comairs monk Infermation
For and uden youers and rightggnt rnd
lmap
Capton:: The imace slown group nour young Anin man weatng short wahirs poann kasall tora
strecheed
ocanssic eyes. sutada mondow satas
Oepori oo people woll Fissbees cansess
(a) Human evaluation interface of the informa- tion experiments.
(b) Human evaluation interface correctness experiments.
berouaton und the and
What is the of the people in the image?"
(c) Human evaluation interface of the question-answerable experiments.
Figure 9: Human evaluation interfaces
18

---

Published in Transactions on Machine Learning Research (03/2024)
A.5 Random Examples of Generated Questions in Conceptual Captioning Dataset
What is the material of the pier in the image?
What is the position of the sign that says "No Swimming Allowed" on the dock?
What is the material of the valve?
What is the color of the plate on which the cake is placed?
What is the expression on the man's face?
What is the boy's expression while playing with the sailboat?
What is the angle of the camera in the image?
What type of flower is it?
What is the color of the woman's glasses?
What is written on the quote on the mountain?
What is the design on the cake?
What is the woman's 's hair color the image?
Are the man and woman standing sitting in the image?
What is the location of the scene depicted in the image?
What is the boy's expression?
What is the material of the pink running shocs?
What is the expression on the man's face?
What type vegetation surrounds the pond in the image?
What is the size of the fountain in the image?
What is the name of the mountain range seen in the background of the image?
What is the name of the park?
What is the design of the woman's dress?
What is the color of the chainsaw?
What is the ethmicity of the two men in the image?
What is the woman's 's pose in the photo?
What modifications, any, have been made to the car in the image?
What kind of donuts are in the box?
What is the woman's age range in the image?
What is the weather like in the image?
What is the man's 's posture like in the image?
What kind of lighting is in the room?
What is the woman's 's hair color in the image?
What is the woman wearing in the image?
What is the woman's pose the image?
What is the type of the lightning bolt? (e.g. cloud-to-ground, cloud-to-cloud)
What is the context setting of the image?
What type of event. is taking place where the man is performing?
What is the shape of the base stand of the ice sculpture?
What is the size of the tent?
What is the boy wearing?
What is the man doing?
What is the material of the sink in the image?
What is the age range of the people in the image?
What is the style of the hat?
What is the background of the image?
What is the surrounding environment like in the image?
What is the woman wearing?
What is the shape of the backyard?
What is the material of the backpack the man is carrying?
What is the texture of the granny square?
What is the style of the bridge?
What is the size of the sun in the image?
19

---

Published in Transactions on Machine Learning Research (03/2024)
A.6 More Qualitative Results
Dess the image drooil
The cacte-lic suroanded Gnow and
Desorbe Ih image dutars
Dnd rogc shous city CuBding
image
Decerbe tns arage iittiiile
bette phone
Whatlsth aecheertunt agle
What arand the pinti nurninng
What the
crial
Whatis the color the
Vertous
WMit is the pretarmunt color the the
Vhhaa the design of the wareer
costle the imege)
the
What wosther
city Kin-in puage?
The sursunnding wns the
Wier volor IN prvise in image??
sure
the caable image?
What the Senner dieritution
Imager
you imagge
Imaget
1: a day
WhLL the cut the doigge
the shope the windows castle image?
Wher pre people wnoving
FM runte shoos
not
upui the buildigg the images
sermunninn landuccpe lik the Image
What the color the starts wom people
"the imoge?
They and restioon Whit mese! diyii Retrettnds
not
th prsition of the siotic the trageet
WT al His Shoper of the sose hackt
the
not
namo Image?
inace?
guongrouns
1h fari no shoe??
Jencoeetav cancll
neuschwwarcie
Mhut is the rure yhd builing
the image?
the Image?
Maachwars sattn Gastle?
The mast depict the Weupch wandter on
The image cookits provd of people shitt and ara produrtieurty onlly make. Time are not sancited.
iagee coality fry Iiakie leed
Imoge fustures pair pire rumnisg choes, wash bettle glioe, siithe Tottte the bo of bull materta
surtor ded beowe mus etaime atrround ape soney.
and
waather is surry and the suriage
peeppe tesage
foregrs and of the tmage.
pholos a not losoe
Discribs detail
weterfll
Desoris this mage 9eta with Fi
DESORS ThiS 161A
Dise be Lha Image In petala.
'Shat the co of the watco the
soand corpae
zake with
Wt the meotiag
What womann doing imceel
What coloria the calid the Image?
Whe Imaige?
telggh
What the maan doingg with tarda??
What poiin for protto
trandred
What saidin pultor
imoge?
Inogect
What womare Pair color!
Ekinde
Hraren lglied and therlles
What the topee of the Evilke
Illo lecon
What Ig
What ooker boockgreund
Whet the becige of the inage??
tyow wig itatiooio serund the materaa
expressiin worrs
Blue
i the wor an woaring les des the
coucoooor hurmningane the date?
Farect
black lacket
rackground
scceaus
Whir to the wor en hokii hee
What colciio Exctaettion Oaee
canmiyed by imrage)
stsge?
water
"S0 the disciiinn the low
chare
Wthul type 31 ovent the waman
sodure of trosting OPeC
doveroerd
shoved fraad
prammenn
"Wi us di locat of satoriel
imege?
What the skon color the mun!
What the name the movem)
the shape of the caloeity imoge/
skinn
"margo/
larra MiCorra salf."
the shan to sokiing
Hrager
Whee fauor the raks??
fender
carpet
the mrepo, a Faan will a shornd Ficid and
The Faage Shows lnannd Ad secaring
cake
Mideri fela in Bambil, "The mauortal
wear a blace an andd lopica nervous
the mowe "Marga Dhs is holddng bottle buckggruue cslerts
thee bod sam the flauiing on calle huiviiggalee pia
The Soned and wasttrr faaiy. and ftrme Veiiat Fatx Zerbieed Frams
aaclogruund behiin the itage is mo3m
flove the untensee..
Figure 10: Random samples from CC Sharma al.
(2018
20

---

Published in Transactions on Machine Learning Research (03/2024)
Describee th image dotalls
cainiigg of boot and boot What the the bestin the pekccig?
Deponte this Iragge in deta (RAS%
Describe this image cetails.
Describe inage detsils A younng bay hokting a bird
Resket
Whath the color bost pekring?
net sure
aehaa owwre ucherw pokktnng
with to and wheen bockgrundd
holliing a basket
imagu?
the weneing?
black shirt
them the par sting?
whi incite the backet that the wemen in hottinng
white
What bacogiounn cubriig Mh?
expressiion
What norm etherihy??
holdiing the tlle?
What everail color anlette Laed the painting?
14C
the paini rg?
sacres sine phinteg!
pine
the wameer's expression?
ish
What the Loy's on cenkmatt
alfteed canvad
happy
What the Eac Backgecurd of the imarge
uscror skae the pol/11
what is the rickroound the imose
mountais
What the main outgect the
forrett
What is the undt on arqund the buy and the lie?
surs
the appres mate slse the parttigg
mountain
What the shape of the boot the
ao canvas
betweee 40 und H0
What the bay's wilerictty?
What the pooition the bosti:
Wi-t is tie pnse the ieople
plance
What the boy's halr coker?
white
seated
whee sirid the weering image?
blonds
What the med um used to create the
rot sure
Whhaa in the dee of the people
make
what the color the bwitont woman drngg
neets
What bay daing with biid?
haiding
int sure
Wetur is the gloua the seaple
The imagge of vourin hos, app matimnaes
paze hax backgraund shape bee uneessen the allo orkeenn The ina sutyeet ol the unddar. creste thes peinting a ebc untinoe..
okder
holding white bind The say his cod
betoren and years oid alearing white wiile buskat with uppe frardee and the wemas tas blonde hair
mo uesin The boy of white ethnicity and
LAU laan Just with pf ne trees backemendd
write
and grwan. painting affowd to
Dnscrioo thic mncee in setans
sing other
Desuric UI inagge br.c.ab.
stamei of whidon with Noreer
Desorite This imagge desall
pain of being towweed into à grave
Desrrie this image Dets
csuple of people siting
schome
What the wammn WeaT BVY
buck gourdd esagi Bia"
W ai (he wowning ti
singing rowiy
usti
nting?
the
the siul unll glven witikiee
hair
What bockground (ali liaa
robs
What ore the people waating in
sreard
visiii
dark
lomo
tingtng
hed glans dild
What the sitti ecutioo
A pleno quita th ai
the mar in painting?
What is anainge snangesent the
Gureee"
hardss the stameed glis windowr
at the color of the esstis ratel?
What people stiin
siyle puinti coed
bock
rnd
tt Inog??
the Staree glasi windoes
mon
gender the people
abstract
bock
crucfte
1HE troper
Whet the gerddrr the person who
<De rind glees vi dile?
palniied this image?
la ctrnliity poodic
slle?
wersn rocd fece the fndd glass window?
knockrg
parntinc
male
the image?
jasswase
What viblis
is cloar
the trogee aper the Image
postare
Whete the wanking noel plaso BlEW
JN" at the style of the pairiing
clear
What thes posiion prac
3 r924
What io the tope of aataaatttrsee used
the peocle in tnvoge
Whst the ostor the apmannss
The pla.eiiiinnnn milccle ofthe
it's clear
Seoll vodi cliing with ska sullen
(Ten and wonat, The Yoat averrno
liose
brews
The
Ioen sed Into of ane. The Hoat to wainingg exceggaand of the 24r1in rinn it dari
the
deccle weeciigg limonos 4ttii
couple (up arspi
between 43 and
aod the 127 (ve ano
ceniing rolu: lak fataal and bachigheend color
style of the painting sei an the typee
ia mare and temuie. and they apasel
poodls
Figure 11: Random samples from WikiArt Salch & Elgammal 2015
21

---

Published in Transactions on Machine Learning Research (03/2024)
Descriee thi Irage deteils
Decatte SHE tranee in cetalls
DONSCTOO ThiS iregg in dera
Descrize this image in detalls
apowboard in maniin Agvinet
Iota the crowbeare the
past Whet locadon of the trauge?
What color trak??
is Mite Whot the color vite?
The trein Blcck
beigth
Maue
What the tox eockartane of the irase"
Whil's the calor of octress
The (onq)
What the weather lice the
lance and
blace and white
What bype of train i
fencel
What the diw of jwed
Whhtl the cargo being can by Irain)
Whht the wesornn?
lsaning opoirst the fence bockgmond of the imgge
Sesi Facrd)
What the woither the in the Image?
Nelisnn
Whet mart
Ane there any other obtects visilee
wooden fence
Cleer blue
tradder
Whates the shape kot
Imager
disettoo?
zearas
sure
envrromean around
whot is the shape of the clouds in
trade Tane
Une tmagee
peteos and dlisting froi??
nnst
in the facenion the mii.
the color the marn's
the imapel
disign anovloarrd
the expression subres" facest
Inado
NeEsure
net.surc
What io the time of cay in the Imoge?
beach?
Whet the shops of the anavbbardd inoger
Whet's the sursourding eivcomeett
in the materia snoubaaerd the
image"
mucely pand
image?
the wouther
What the soe berlidd the mon?
I'm weiling for your mani queeiicet
The imagee chapi by black fiss carrted by the tudi unltasnn b. Mabble imsae basides the train and tracka. timm of day. and the westher ore untinnen..
The on sundy day a beach with write sand treed the backartund, und The coler of the shhpe diouus city are uniorowm
The Imago phaws sentt de and showboord
The Imageerl of smell herd of bluck and sabnes fnciing
in the nartppounn on Skrnyy dade The reattrall eade anl
pudkie diractorra uppeariig quie
antrroinn
soy
Deottbe this tmagge in Seialls
Discrite this detail
Descrice trage ditalls
Descriaa image darcalls
cibyratec atectteett a towe
Wo nomw the the trage?
words thene difirer Cn
bocksmann pess
What's the colar the bears it the
Heno te control poute) weartng
What Uhe color of the man shot!
Whit is the Shis of lears the
brown
large
PRITADE cos pichee??
men hold mo the oxcie
Ihat ore the beara dking the
the
The teiliing wiidan in tnagee
dleck
yycling el othir
What The tmreee?
The holsinn the epele?
Wetutis the surrund eg envrorment
What viidde the backgreund
whibe
seople
racka and woter
What the the
In the backggr of imc
Teatt is the puetice the beare in
Whet 8: the bockshelves?
Blue
apple hold
They amt water
What the seine of skeet White
the man)
What the shape of the rodks the
bascohellen)
What type vetiicees deliiii
at gander man heiding the acelc?
What the pouture the boars
gander parson 580 the
What dive can NwW the
airaat the age of man tolding the acolc?
Wh the distunp bedaveann beoo
They and in water
the persor Eectal coanssiionn
buildiin AOaCIIS to chickkik towar?
Wh'ati the nair dayle the holding
What is the dupness taoms beass images
in len fsst
WTA: 32w raage the person couch??
sursings clack tower
inoce showe cic street Alt wiite
The Image conalats "eaac man prpar med Luruetii apple with the wirea Thank rocm with The tacaa on
the wathr aedl et each dinan The recks bears for aporn engry axpress
arayy
couo is weading whit and helddng remict heacy Tho 391 rorge persin
Image decida
aitiing
wost bow aee car and truc0s on the
The
pasale
a
the ane
on poeek clock ttwedy
Figure 12: Random samples from COCO Lin et al. 2014
22


Reviews:
[{"review_id": "XNDBvsLWM1", "reviewer": "Reviewer_G3WG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I think that the findings in the paper are quite interesting. I think the paper would be a lot stronger if there was more analysis to identify the degree of importance of each component in the system (prompt sensitivity, LLM, captioning model).\n\n## Strengths:\n- The paper proposes a simple method of leveraging vision-language models and strong LLMs to create a dense captioning system.\n- They show that they can use the ChatCaptioner system to identify a greater coverage of image objects in an image (Sec 4.1), suggesting that this method is better at extracting information from BLIP-2, which is an interesting finding. It’s reminiscent of several automatic prompting approaches, which might be useful to discuss in related work.\n\n\n## Weaknesses:\n- The authors tested several other LLM backbones in 4.3, but only appear to report these results qualitatively (describing that only InstructGPT can generate meaningful questions). Could these results be quantitatively measured? For example, running human evaluations on a small subset of these to see the effects of different LLM backbones.\n- Similar to the experiment in 4.3, can we do the same with other VLMs? It would be valuable to measure the performance of other captioning models, e.g., BLIP-1, MAGMA, Fromage, or LLaVA. It would be ideal to have some sort of chart showing the impact of different VLMs and LLMs: is one or the other more important?\n- Missing references: VisDial [1] is highly relevant to this work. [2] on using LLMs for automatic prompt engineering seems highly relevant to the approach. It would be useful to discuss these papers (and corresponding follow up works) in the related work section.\n- Minor typos: “BLLIP-2” in abstract, OpenAI API citation “(cha, 2023)” in Section 4\n\n\n**References**\n\n[1] Das, Abhishek, et al. \"Visual dialog.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n[2] Zhou, Yongchao, et al. \"Large language models are human-level prompt engineers.\" arXiv preprint arXiv:2211.01910 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a system called ChatCaptioner, which leverages a pretrained vision-language model (VLM), such as BLIP-2, and a strong text-only LLM backbone (gpt-3.5-turbo in this work). They use the LLM to generate questions over 10 dialogue rounds, and use BLIP-2 to answer each generated question. Finally, they generate a dense caption by prompting the LLM to summarize the information from the dialogue in a few sentences. This system allows them to generate more information visual descriptions compared to BLIP-2 (which usually generates shorter captions).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Would strengthen the work:\n- The authors report that GPT-3.5 was the best API-based model at the time of the paper. However, I’m curious if we can perform an ablation with the GPT-4 backbone. It would be great to compare the performance of GPT-4V (which can also do dense captioning) and ChatCaptioner (BLIP-2 + GPT-4 text-only). Does the proposed approach come close to the multimodal GPT-4? Even if not, ChatCaptioner seems like it would have significantly lower cost, so it may be an argument to use it (BLIP-2 + GPT-3.5) over the more expensive multimodal APIs (GPT-4V).\n- How sensitive is the system to the number of questions? Does it saturate at 10 questions, or before that? Can we see an ablation experiment with different rounds of questioning? (maybe at least 3, 7, rounds, and 12 if it doesn’t saturate at 10)\n- Would the automated caption metrics on Localized Narratives work better, since those captions are more detailed and presumably closer in distribution to the dense captions that ChatCaptioner generates?\n- Instead of generating questions from an LLM, can we use the questions from the VisDial dataset to achieve similar results? What would be the difference in performance?", "broader_impact_concerns": "There are no major issues with the ethical implications of the work. It combines off-the-shelf LLM and VLMs, and is focused on image captioning which is not very high risk. However, it would be useful to describe how the human evaluators were sourced and compensated (i.e., do they meet local standards for minimum wage).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2oKXmsU7U", "reviewer": "Reviewer_ch9K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper makes a good point in arguing that AI systems need to learn to ask good questions, so they can acquire knowledge or help diagnose problems.\n- The authors make a good effort in running Human evaluation experiments and present strong results.\n- The paper is generally well written.\n\nWeaknesses\n- The paper is already out of date - ChatGPT-4V would presumably react quite differently and you could maybe run the entire study with GPT-4V, or now Bard (Gemini) -- how do results look like here?\n- The paper does not go technically deep. Prompts have been manually generated, with ChatGPT in mind, and it is not clear if there was an attempt to optimize prompts for the other LLMs that were tested, or if prompts were simply rerun on different LLMs.\n-", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a system (code is being made available) that will use ChatGPT (or InstructGPT and other related models) to repeatedly ask questions about an image, which BLIP-2 will then \"answer\" in a dialog. At the end, ChatGPT summarizes the acquired information. Using rule-based prompting, ChatGPT will thus extract more information from an image than a single invocation of BLIP-2 would yield. ChatGPT is using BLIP-2 like a \"tool\" to extract information, which it then summarizes. \nThe paper evaluates this idea on a number of datasets (e.g. COCO). Since the output of ChatGPT does not correspond to typical captioning patters (so CIDEr, ROUGE, etc cannot easily be applied), Human evaluation is used, which shows that this process yields more information (e.g. more objects or more properties) with a reasonably high level of \"correctness\". An \"uncertainty\" prompt is introduced to improve results and authors present some plots that show the diversity of the generated questions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- There is a body of work in question generation for NLP (diversity, quality, etc), which the paper does not cite - it would be helpful to put this work in context\n- Please discuss the impact of recent releases (esp MM-LLMs) on this work -- is this still relevant? Which parts of the results will carry over to other models/ tasks?\n- How were the prompts designed? Was there any attempt to optimize the questioning strategy with ChatGPT, or when porting to other LLMs?\nAs-is, the paper reads a bit like a snap-shot of an interesting & relevant idea, but given the speed with which the field moves forward, and the emergency of multi-modal LLMs, I am wondering if that paper's", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IyJlOVaFLv", "reviewer": "Reviewer_ekHZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nOverall, I liked the paper. The idea is simple, yet interesting and works well. I thought the experiments/analysis was pretty complete (especially given the difficulty of evaluating text generations in general). There were several times I was writing out comments for analysis I thought were missing only to find it later in the paper - how accurate are the captions, effectiveness of the uncertainty prompts. And these analyses all had quantitative results. The paper was well written and easy to follow and the diagrams were easy to follow. \n\nWeakness: \nI don’t think there were many weaknesses. One question I had was how do you think this method is situated in light of the release of GPT-4, which has a vision component?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new image-captioning method where ChatGPT is used to generate questions that a VL model answers. This dialog is then summarized to form a caption.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions (Minor) \n- Given how much longer it appears the ChatCaptioner response is than the BLIP-2 response (5 lines compared to 1 line in Figure 4), how can the BLIP-2 response be voted to contain more information than the ChatCaptioner response? \n- In the Correctness Analysis, if the incorrectness was not attributed to the wrong answers, then does it have to be due to the summarization from ChatGPT?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were fairly aligned in their views of the paper, and I agree with them about it. The central idea of the paper, that of asking questions with an LLM, is an interesting one. However, the implementation in this work is a bit mundane, stapling existing prompted models together. As one reviewers pointed out in the private comments, the introduction of GPT-4V steals a little of this work's thunder, although the fact that asking questions still leads to higher performance is interesting. The addition of GPT-4V exploration to the work during the discussion period is very nice.

The evaluation is fairly thorough and sound, in my judgment. One reviewer pointed out that automatic evaluation could help future researchers evaluate on this task more consistently and cheaply.

Minor: I spotted one typo going through the paper. Section 4.4: CapCaptioenr <- typo in the section heading. Please fix for the final version.
Decision: Accept (poster)