Chatting Makes Perfect: Chat-based Image Retrieval
Authors: Matan Levy
Rami Ben-Ari
Nir Darshan
Dani Lischinski
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Retrieval
Multi-modal learning
TL;DR: We study the a new approach for Image Retrieval using an interactive chat with the user

Abstract: Chats emerge as an effective user-friendly approach for information retrieval, and are successfully employed in many domains, such as customer service, healthcare, and finance. However, existing image retrieval approaches typically address the case of a single query-to-image round, and the use of chats for image retrieval has been mostly overlooked. In this work, we introduce ChatIR: a chat-based image retrieval system that engages in a conversation with the user to elicit information, in addition to an initial query, in order to clarify the user's search intent. Motivated by the capabilities of today's foundation models, we leverage Large Language Models to generate follow-up questions to an initial image description. These questions form a dialog with the user in order to retrieve the desired image from a large corpus. In this study, we explore the capabilities of such a system tested on a large dataset and reveal that engaging in a dialog yields significant gains in image retrieval. We start by building an evaluation pipeline from an existing manually generated dataset and explore different modules and training strategies for ChatIR. Our comparison includes strong baselines derived from related applications trained with Reinforcement Learning. Our system is capable of retrieving the target image from a pool of 50K images with over 78% success rate after 5 dialogue rounds, compared to 75% when questions are asked by humans, and 64% for a single shot text-to-image retrieval. 
Extensive evaluations reveal the strong capabilities and examine the limitations of CharIR under different settings. Project repository is available at https://github.com/levymsn/ChatIR.

Chatting Makes Perfect: Chat-based Image Retrieval
Matan Levy!
Rami Ben-Ari
Nir Darshan
Dani Lischinski
Thee Hebrew University of Jerusalem, Israel OriginAI, Israel sevyees.huji.
Abstract
Chats emerge as an effective user-friendly approach for information retrieval, and are successfully employed in many domains, such as customer service, healthcare, and finance. However, existing image retrieval approaches typically address the case of single query-to-image round, and the use of chats for image retrieval has been mostly overlooked. this work, we introduce ChatlR: chat-based image retrieval system that engages in conversation with the user to elicit information, in addition an initial query, order to clarify the user's search intent. Motivated by the capabilities of today's s foundation models, we leverage Large Language Models to generate follow-up questions to an initial image description. These questions form dialog with the user in order retrieve the desired image from a large corpus. In this study, we explore the capabilities of such a system tested on a large dataset and reveal that engaging in dialog yields significant gains in image retrieval. We start by building an evaluation pipeline from an existing manually generated dataset and explore different modules and training strategies for ChatlR. Our comparison includes strong baselines derived from related applications trained with Reinforcement Learning. Our system is capable of retrieving the target image from a pool of 50K images with over 78% success rate after dialogue rounds, compared 75% when questions are asked by humans, and 64% for single shot text-to-image retrieval. Extensive evaluations reveal the strong capabilities and examine the limitations of CharlR under different settings. Project repository is available at
Introduction
Users have always been the central focus of information retrieval. Conversational search offers opportunities to enhance search effectiveness and efficiency. The tremendous growth in the volume of searchable visual media underscores the need for fast and reliable retrieval systems. Retrieval capabilities are indispensable in general internet image search, as well as in specific domains, such as e-commerce or surveillance. Current approaches image retrieval in computer vision primarily focus on image-to mmage-to-image [10,46], text-to-image [30,31] ] and composed-iman retrieval [19,27]. However, single query might fail fully convey the search intent, and multiple trials may be required before satisfactory result is retrieved. Furthermore, it is to the user to decide how to modify the query in each trial, while the retrieval system processes each attempt independently.
Motivated by these difficulties and inspired by recent progress in Large Language Models (LLM),, which have demonstrated unprecedented natural language chat capabilities [36-38,48], we introduce and explore a new image retrieval "protocol":: Chat-based Image Retrieval, which we dub ChatlR. A schematic view of ChatlR and the system that we propose in this paper is provided in Figure 1. The process starts with user-provided short description of the desired image, similarly to text-to-image retrieval. However, from this point on, the retrieval system is able to progressively refine the query by
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Dialog Query
Image Search
Round
Stuplightt shine deserted, wel city
Ground Truth
Round
what color the light?
Image Retriever (F)
2000 cmb, space
rad
Image Embedder
big city?
looks a smaller town
are there signes?
Dialog Building
are there building??
Question Generation
Humren Interasiinn
few the cistance
OLO
How many ianes?
Only and bike fane
what time of day?
Round
daytmm
Aocumuiate Hound 86 for next search
Figure I: An overview of Chat Image Retrieval. The pipeline consist of two stages: Image Search (IS) and Dialog Building (DB). The IS stage takes an input the ongoing dialog, composed of image caption and a few rounds of Q&As, in order to find the target image. Note that a dialog of length is solely the image caption, equivalent to Text-to-Image retrieval task. The DB stage provides the follow-up question to the current dialog.
actively polling the user for additional information regarding the desired result. Idcally, a ChatIR system should avoid gathering redundant information, and generate dialogues that steer towards the desired result as quickly as possible. Note that this gradual progress scenario is very different and more natural from providing at the outset an overly descriptive caption, which hypothetically contains all of the required information. In contrast, ChatlR proactively obtains the information from unified and continuous manner in order to retrieve the target
the user and able to process image within a few question answering (Q&A) rounds.
Specifically, the ChatlR system that we propose in this work consists of two stages, Image Search (IS) and Dialog Building (DB), as depicted in Figure 1. Image search is performed by an image retriever model F. which is a text encoder that was trained project dialogues sequences (of various lengths) into a visual embeddings space. The DB stage employs a question generator G, whose task generate the next question for the user, taking into account the entire dialog up to that point. The two components of ChatIR are built upon the strong capabilities of instructional LLMs (where the model is instructed about the nature of the task) and foundation Vision and Language (V&L) models.
What dataset do we use and is it necessary create and manually label such a dataset? 2. How do independently evaluate different components of the system? 3. How do we define benchmark and performance measure for this task make further progress in this domain measurable?
Addressing this task we are faced with three main questions: 1.
To mitigate the cumbersome and costly process of collecting numan-machinn conversations we use the VisDial dataset [8]. Although this dataset was designed and generated to create chats about images without any retrieval goal, we use the specific image related to each dialog as our retrieval target, and the dialog as chat. In our case the questioner an agent and the answerer is human while in the Visual Dialog task [8] it is vice versa.
Considering the goals of ChatlR as a conversational retrieval system, we evaluate its performance by measuring the probability of a successful retrieval up to each round in the dialog. We use this metric to systematically study the major components of the framework and to examine the impact of different questioner models G, and training strategies for F' on retrieval performance.
For example, when training F using various masking strategies, we found that masking the initial descriptions proved be the most effective method (elaborated in Section 5). Since the retrieval performance of ChatIR also depends on the quality of the questions generated by G, we evaluate several alternatives for G based on their impact on F's retrieval ranking. One of the problems in this evaluation is the need for user in the loop, to answer G's questions (at inference time), while taking into account the chat history. Such evaluation of ChatlR is obviously costly and impractical at scale. To mitigate this, we replace the user with multi malti-purrpos vision- language model BLIP2 [21], as

---

Visual Dialog Model (VDM) that answers questions. We further harvest human answers testing our system in the real scenario with a human providing the answers, and show a comparison between the VDM and humans in terms of impact on the performance of ChatlR.
We find that ChatlR can retrieve the target image from corpus of 50K images, within the top- 10 results, with success rate of 78.3% and 81.3%, after 5 and 10 Q&A rounds, respectively. Overall, Chatl increases retrieval success by 18% over single-shot text-to-image retrieval where the user provides only text description.
In summary, our contributions are follows:
Introduction of ChatlR, a novel framework for visual content search guided by an interactive conversation with the user.
We explore the Chatl idea leveraging foundation V&L models, with several LLM ques- tioners and image retrieval training strategies.
We suggest evaluation protocols suitable for continual progress and assessment of question- ers using a Visual Dialog model in place of a human.
We test our framework on real human interactions by collecting answers from users and further evaluate our method against strong baselines generated from prior art.
2
Related Work
Man-machine dialogue has been used for information retrieval for decades [35]. More recent works involving chatbots for large knowledge corpus include [11,28, 21,28,3,, 39J. Below we survey related tasks that involve visual modality and dialogues.
Visual Conversations: In visual domain, there are many applications that cope with only one- step human-image interactions, e.g., VQA [1, 13, 29, 50, 56], Image Retrieval (in its variations) 56,, and Composed Image Retrieval [2,4, 12, 16,49]. While the output of these methods either images or answers, Visual Question Generation tackles a counter VQA task, and tries to generate a single question about the image [25,42]. Visual Dialog [8,33] is the task of engaging in dialog about image, where the user is the questioner and the machine is the answerer that has access to the image. With the bloom of text-based image generation models, some approaches suggest to initiate chat for image generation. For instance, Mittal et al. [32] propose method generate an image incrementally, based on a sequence of graphs of scene descriptions (scene-graphs). Wu et al. introduced VisualCharGPT [51], an "all-in-one" system combining the abilities of multiple V&L models by prompting them automatically, for processing or generating images. Their work focuses on access management for different models of image understanding and generation. A concurrent work [59] uses ChatGPT in conjunction with BLIP2 [21] to enrich image captioning. Although all the above studies deal with combinations of vision and language, none of them target image retrieval.
slightly different line of work addresses the problem of generating dialogues about images, called Generative Visual Dialogues [9, (9,34,5 3,,4,,54 57J. They generally focus training two agents, Questioner- bot (Q-bot) and Answerer-bot (A-bot), in order to generate the dialog. The idea is to test the ability of machines in generating natural conversation about an image. To this end, both bots had access to the dialog history while the A-bot can further "see" the image, for providing the answers. The training strategy based on Reinforcement Learning (RL) while for evaluation used, named "Cooperative Image Guessing Game" with a reward where the Q-bot should predict the image pool of 9.5K images. However, the recent foundation V&L models outperform these methods in all aspects e.g. question answering [33] and question diversity (see also Section 4) and are shown to be effective also for various downstream tasks 15, 15, 19, 20, 22-22,22222 29, 33,44, 50, 50, 58]. Motivated by these capabilities, we build our model on L.L and V&I foundation models to create our ChatIR system. We compare our method to the related prior art in [9,34], although these tasks are different from ours and do not directly target image retrieval.
auxiliary task is
Visual Search: An important aspect of information retrieval visual search and exploration. Involving the user in search for visuals is a longstanding task of Image Retrieval that has been previously studied by combining human feedback [14, 17, 18, 52]. The feedback types vary from binary choice (relevanViirelevan)) e.g. [43, 45] through pre-defined set of attributes [18, 40],
3

---

and recently by open natural language form, introduced as Composed Image Retrieval (CoIR) (19,77,9,522 52]. The ColR task involves finding a target image using a multi-modal query, composed of an image and text that describes a relative change from the source image. Some studies construct a dialog with the user [14.52] by leveraging such ColR models where the model incorporates the user's textual feedback over an image to iteratively refine the retrieval results. However, these particular applications differ from ChatlR in not involving user interaction through questions, nor does explicitly utilize the history of the dialogue. One way or another, this type of feedback requires the user to actively describe the desired change in an image, time after time without relation to previous results, as opposed to ChatlR where the user is being pro-actively and continually questioned, with including all the history in each search attempt.
3 Method
We explore a ChatIR system comprising two main parts: Dialog Building (DB) and Image Search (IS), as depicted in Figure 1. Let denote the ongoing dialog as D; (C, Q1, A1. ..., Qi, A.). where C is the initial text description (caption) of the target image, with (Qk) denoting the questions and {A their corresponding answers at round i. Note that for = 0, Dg == (C), thus the input to IS is just the caption, i.e., special case of the Text-to-Image Retrieval task.
The dialog building stage consists of two components, the Question genera- tor G and the Answer provider, which in practice a human (the user) who presumably has a mental image of the target T. In our case G is an LLM that generates the next question Qi+1 based on the dialog history D;, i.e. G D; - Qi+1- We assume that G' operates without the benefit of knowing what the target T is. In this paper, we examine various approaches for the questioner G, exploring the capabilities and limitations as well as their failure modes (reported in Section 4). In order to enable experimenting with these different alternatives at scale, we cannot rely on user-provt providovdd answers to the questions proposed by G. Thus, in these experiments, all of the questions are answered using the same off shels model (BLIP2 [2]]). A smaller scale experiment (reported in Section 4.3) evaluates the impact of this approach on the performance, compared to using duman--prvvine answers.
Dialog Builder Model
Image Retriever Model: Following common practice in image retrieval [15, 19,112,,124,
52]. our IS process searches for matches in an embedding space shared by queries and targets (see Figure 1). All corpus images (potential targets) are initially encoded by an Image Embedder module, resulting in single feature representation per image / E R2, with d denoting the image embedding space dimension. Given dialog query D,, the Image Retriever module F, a transformer our case, maps the dialog F D, Ed to the shared embedding space. The retrieval candidates are ranked by cosine-similarity distance w.r.t the query embedding. As our F we use BLIP [22] pre-trained image/text encoders, fine-tuned for dialog-based retrieval with contrastive learning. We leverage the text encoder self-attention layers to allow efficient aggregation of different parts of the dialog (caption, questions, and answers), and for high level perception of the chat history. Motivated by previous work [20,29 29,33], we concatenate D, elements with a special separating token [SEP], and an added [CLS] token to represent the whole sequence. The latter is finally projected into the image embedding space.
We train F using the manually labelled VisDial [8] dataset, by extracting pairs of images and their corresponding dialogues. We train F to predict the target image embedding, given a partial dialog with rounds D., concatenating its components (separated with special [SEP] token) and feeding F with this unified sequence representing D, In Section we demonstrate that randomly masking the captions in training boosts the performance.
Implementation details: we set an AdamW optimizer, initializing learning rate by 5 x 10-5 with a exponential decay rate of 0.93 to x 10 10-6 We train the Image Retriever F on VisDial training set with batch size of 512 for 36 epochs. The Image Embedder is frozen, and is not trained. Following previous retrieval methods [19,41], we use the Recall @K surrogate loss as the differentiable version of the Recall metric. Training time 114 seconds per epoch on four NVIDIA-A 100 nodes. In
testing, retrieve target images from an image corpus of 50, 000 unseen COCO [26] images.

---

4 Evaluation
In this section we examine various aspects of ChatlR. We start by showing the benefit of ChatlR over existing Text-To-Image (TTI) retrieval methods. We conduct experiments on established TTI benchmarks (Flickr30K [55] and COCO [26]), and then proceed to evaluate our model on the human- annotated dialogue dataset (VisDial). Next, we compare ChatlR performance various Questioner models (G). Finally, we evaluate our top performing model with real humans as answerers, small validation subset. In all experiments, a large search space is used (50K images for VisDial, 30K for Flickr30K, and 5K for COCO).
VisDial using
4.1 Comparison With Existing Text-to-lmage Retrieval Methods
Here we compare the retrieval performance of ChatlR with existing Text-To-Image methods. First, we generate two synthetic image-dialle mmage-diloguue datasets (using ChatGPT as a questioner, and BLIP2 [21] as an answerer) from the two established TTI benchmarks: Flickr30K and COCO. In Figure 2 we compare our method to two TTI methods, CLIP and the publicly available SoTA baseline for TTI, BLIP [22], in a zero-shot setting (i.e., none of the compared methods have been fine-tuned on either dataset). We find that our method surpasses the two baselines by a large margin, on both datasets. Furthermore, when we provide the baselines with the concatenated text of the dialogues, instead of just caption, they exhibit significant improvement over the single-hop TTI attempt. Nevertheless, the gap in favor of our method is maintained (Fig. 2a) or increased (Fig. 2b). These zero-shot results show that: 1) dialogues improve retrieval results for off-the-sheif TTI models. Although the CLIP and BLIP baselines have only been trained for retrieval with relatively short (single-hop) text queries, they are still capable of leveraging the added information in the concatenated Q&A text. Note that CLIP becomes saturated at a certain point due to the 77 token limit on the input. 2) Our strategy of training an Image Retriever model with dialogues (as described in Sec. 3) further improves the retrieval over the compared methods, raising accuracy from 83% to 87% at singlehhpp retrieval, and surpassing 95% after 10 dialogue rounds (on COCO). Next, we fine-tune TA TTI BLIP on VisDial (by providing with images and their captions only) and compare it to our method on the VisDial validation set. Results presented in Figure 3. We make two main observations: 1) The retrieval performance of the fine-tuned single-hop TTI baseline of BLIP, is nearly identical to our dialogue-trained model (63.66% vs. 63.61% %). This corresponds to dialogues with 0 rounds in Figure 3a. 2) Using a dialogue boosts performance, while increasingly longer dialogues with ChatIR eventually achieve retrieval performance over 81% % (Fig. 3a), showing a significant improvement a single-hop TTI.
4.2 Comparison Between Questioners
We examine various Questioner models (G)) and their relations with F by evaluating the entire system. As previously discussed, we use a Visual Dialog (VD) model to imitate the user by answering G's questions. More specifically, use BLIP2 that was previously showed to be effective in zero-shot VQA and VD [7.21]. Using the same VD model as answerer in our experiments allows a fair comparison between different questioner models. As retrieval measure we compute the rate of images that were successfully retrieved among the top-k ranked results up to each dialog round (we opt for k 10 here). We stop the chat for each image soon as it reaches the top-k list and add it to success pool, since in practice, the user will stop the search at this stage. We examined the following LLM models for G in experiments:
Few-Shot Questioner: We test four pre-trained LLMs to generate the next question, based on few-shot instructions [3]. We explicitly instruct the model with the prompt "Ask new question in the following dialog, assume that the questions are designed to help us retrieve this image from a large collection of images" alongside with a few train examples (shots) of dialogues with the next predicted question. Examples for such prompts can be found in our supplementary material. Specifically, we tested FLAN-TS-XXL [6], FLAN-ALPACA,
[47], FLAN-ALPACA-XXI and ChatGPT [36]. The examples in Figures 5 and were generated using ChatGPT as questioner.
Unanswered Questioner: in this method we use an LLM to generate 10 questions at once, based solely on the given caption and without seeing any answers. Thus, although the actual retrieval conducted using the full dialogues (questions and answers), the questions generated in this manner are not affected by the answers. Here we provide ChaiGPT with the following prompt: "Write 10 short questions about the image described by the following caption. Assume that the questions
5

---

Zero-Shot Performance COCO
Zero-shot Performance on Flickr30k
95
88
86
84
89
62
Nits010 (in 40
Hits01 (In
83
82
76
74
77
72
73
10
68
Zaro-shat
Ours Zero-shot
BUP Zero-shot
BLIP Zero-shot
Zero
10
4 Dialog Rounds
10
Dialog Rounds
(a) Hits @ 10 success COCO (higher is better).
(b) Hits @ 10 success Flickr30k (higher is better).
Figure 2: In ChatlR, retrieval is attempted after every Q&A round, while in traditional TTI retrieval, there is single attempt, without any Q&A involved (the leftmost dot in the green and orange curves). To examine whether the TTI baselines would benefit from the extra information conveyed in the dialogue, we also plot (as hollow points) retrieval attempts made by these baselines using concatenation of increasing numbers of Q&A rounds. It may be seen that these concatenated queries improve the retrieval accuracy, even though CLIP and BLIP were trained with such queries.
are designed to help us retrieve this image from a large collection of images: [CAPTIONJ". This experiment demonstrates how the influence of the answers the question generation affects the retrieval performance.
Human:
Here use the dialog to simulate a human question generator.
VisDial dataset [8]. We extract the questions from each
In Figure 3 we present the performance of different Questioners with the same Image Retriever F. While Figure 3a presents the retrieval success rate (Hit rate), Figure 3b presents per-round performance in terms of Average Target Rank (ATR), where lower is better. The first observation shows a consistent improvement retrievals with the length of the dialog, demonstrating the positive impact of the dialog on the retrieval task. Looking at the top-performing model, we already reach high performance 73.5% Hit rate a corpus 50K unseen images, after just rounds of dialog, 10% improvement over TTI (from 63%, round 0 in the plot). We also observe that questioners from the previous work of 343 based on RL training are among the low performing methods.
Next, we see wide performance range for FLAN models with questioners in early rounds, while their success rate diminishes as the dialog rounds progress, causing them to underperform compared to humans. However, the success rate the ChatGPT (Unanswered) questioner (pink line), that excludes answer history, is comparable to that of humans, with less than ~ 0.5% gap. By allowing full access to the chat history, the ChatGPT questioner (blue line) surpasses other methods, mostly by a large margin (with ~2% over Human). Perhaps more importantly, in both ChatGPT questioners we see a strong and almost monotonic decrease in Average Target Rank (Fig. 3b) as the dialog progresses (blue and pink lines), similarly to the Human case (green dashed line). Other questioners fail to provide progressive improvement of the target image rank (lower ATR) with the dialog length, implying saturation of the relevancy of their questions. Interestingly, as dialogues progress, only the human questioners consistently lower the target rank to a minimum of almost 25/50, 000, demonstrating highest quality of questions.
surpassing human
Next, we measure question repetitions for each questioner as a suspected cause behind the perfor- mance levels of different questioners (previously addressed in [34]). For each questioner we calculate the average number of exact repetitions of questions per dialog (i.e., out of 10). While the average number of repetitions nearly 0 for either human or ChatGPT questioners, the other methods exhibit an average of 1.85- 3.44 repeated questions per dialog. We observe that these findings are correlated

---

Retrieval Performance for Different Questioners
Average Target Rank for Different Questioners
49
80
47
45
78
76
Aver seraaee Target Rank
42
Hits@10 (In
74
39
72
ChatGPT
FLAN-ALPACA-XXL
-5
70
Human
ChatGPT (Unanswered)
31
FLAN-ALPACA-XL
66
RL1s
29
FLAN-TS-XXL
27
64
RL17
25
0
6
7
8
9
10
2
5
6
10
Dialog Rounds
Dialog Rounds
(a) Questioners Hits@ 10 success (higher is better).
(b) Target image average rank (lower is better).
Figure 3: Left: Evaluation of different chat questioner methods on VisDial. For all cases (including "Human") the answers are obtained from BLIP2. Note that dialog with 0 rounds is only the image caption, a special case of the text-to-image retrieval task. Top and super-human performance is obtained by ChatGPT with a significant gap over several versions of FLAN and previous RL based methods, RL17 [9] and RL19 [34]. Right: Average rank of target images after each round of dialog.
with the overall performance of questioners in Figure 3, since repetitive questions add little or no information about the target image. More details available in our suppl. material, where also measure uniqueness at the token level (repeated questions result in fewer unique tokens per-dialog).
4.3 Comparison to Human In The Loop
While above we examined all the questioner methods in conjunction with BLIP2 answers, here we evaluate the performance with as human as the answer provider. Our incentive for this experiment is 1) To evaluate how answers of BLIP2 compare to those of humans. Is there any domain gap? 2) To test our nop-performing questioner in a real ChatIR scenario. To this end, we conduct dialogues on ~ 8% of the images in the VisDial [8] validation set (through a designed web interface), between ChatGPT (Questioner) and Human (Answerer). Note we also have fully human-labelled dialogues (Q: Human, A: Human) collected in the dataset. We refer the reader to suppl. material for further information about the data collection.
Figure 4 shows the results where we present three combinations of Questioner and Answerer A C & A (blue dot line). 2 & BLIP2 (blue solid line) and the reference of Human @ & Human (red dot line). We observe that while all experiments perform similarly, up to 5 dialog rounds. beyond that point, Human & Auman A (red dot line) outperforms both ChatGPT alternatives. Considering the previous experiment in Figure 3, showing the advantage of ChatGPT over Human as questioner, we observe that Human generated answers are of better quality than BLIP2 (in terms of final retrieval results). This advantage boosts the Human full loop to become more effective than ChatGPT. The results imply small domain gap between BLIP2 and Human answerer (but with similar trend). justifying the usage of BLIP2 in our evaluations. Furthermore, we observe that in the real use-case of ChatGPT & Human A the model reaches a high performance of 81% Hit rate using only 5 dialog rounds. Similar to previous results, we observe a saturation in the marginal benefit of the last dialogues rounds.
Next we show some qualitative examples. More examples as well as dialogues egg. Human vs. BLIP2 answers can be found in the suppl. material. Figure 5 describes a search for traffic light, where the answerer is BL.I2. At round 0 (search by the caption), the target image is ranked as 1, 149 due to existence of many traffic lights in the corpus and also erroneous candidates. The results show the rapid decrease in the rank reaching at the top of the candidates after only dialog rounds. Note that
7

---

Impact of Different Questioner Answerer
Average Rank for Different Questioner and Answerer
40
Q: ChatGPT, A: Human
Q: ChatGPT, BLIP2
82
Human, Human
81
Avarage Target Rank
-0
Hits@1o (In
25
73
73
Q: ChatGPT, A: Human
15
07
Q: ChatGPT, A: BLIP2
Q: Human, Human
10
10
10
Dialog Rounds
Dialog Rounds
(a) Questioners Hits@ 10 performance (higher is better).
(b) Target image average rank (lower is better).
Figure 4: Human-AL. 1. We want to show performance on real scenario. 2. Verify the validation method BLIP2 vs. Human answerer 3. Human-Human is the best. BLIP2 answers lower quality according to (a) We conducted this experiment to evaluate the difference, but still BLIP2 still presents reasonable performance although inferior to human
Predicidd
Pank $1
Pars 42
Rack
Rank sc
Pask 95
there obunc wabblein opert mritkc ghu?
Figure 5: ChatlR example: a dialog is conducted between ChatGPT and a user imitator (BLIP2) on the ground truth image (green frame). Top-5 retrievals are presented after each dialog round.
since the first round, when the location is specified as "a house", the retrieved images are more likely traffic lights with a house in the background. Figure 6 shows another case with search for a specific train. This example is drawn from our test case against Human answerer. Note how the train at top of the list changes from black to green and blue after question about the color is asked (second round), boosting the rank from 22 to 2. The last round fine-tunes the top-2 results by distinguishing between parking forms of on track vs on platform.
We conducted further analysis of using our pipeline for generating additional data, wherein the questioner and answerer collaborated to generate dialogues on more images. The inclusion of these dialogues and their corresponding images in the training set resulted in an improvement in the Average Rank metric, but did not provide any benefit to the specific retrieval measure of Hit@ 10. Due to lack of space, discuss these results in the suppl. material.

---

Predeted 100
Part
Park 4J
Rast
Hash od
Pack
pho3d?
foreground
West hps of Inan io?
Whst
amply oraw thre paooengarx board??
Figure 6: ChatlR example: dialog is conducted between ChatGPT and Human on the ground truth image (green frame). Top-5 retrievals are presented after each dialog round.
5
Ablation Study
In this section we conduct an ablation and examine different strategies for training the Image Retriever model F. We further examine few Questioner (G) baselines and discuss our evaluation protocol.
Impact Masking in Train
Impact of Questioner vs Answerer Methods
82
80
35
78
50
76
Averoge Target Rank
Hits@ 10 (in %0
35
30
20
Solid:
ChatGPT
Dahsed: Human
25
ChatSPT.A: BLIP2
Masking: Caption
Human,
Ma Masking: Questions
Q. Human, BLP2
Rl.., BUPP
Masking: Answers
15
RLn AlRla
Masking: Rounds
C: RLus A: BUP2
Masking Tokens
20
Rlus Rlas
Masking
Questioner,
Dialog Rounds
Rounds
(a) Comparison of masking strategies.
(b) Rank performance for different Q&A models.
Figure 7: Left: Retrieval performance of image retriever models (F) trained with different masking strategies. Results are reported for two different answerers G. Right: Impact of different Questioner and Answerer models on the average target ranking, as the dialog progresses (lower is better). RL17 [9] and RL19 [34] represent previous RL methods.
Masking strategy: In this experiment we examine five different masking strategies for training of the Image Retriever model (F). Figure 7a shows evaluations of the resulting F models using two different questioners G: ChatGPT (solid lines) and Human (dashed lines). As baseline we train F using dialog sequences (concatenated as described in Section 3), without masking any parts of the dialog. Next, we randomly mask different components of the training dialogues: captions, questions, answers, entire Q&A A rounds, or individual tokens. In each strategy, we randomly select 20% of the components of a certain type for masking. The results in Figure 7a show that among these strategies, masking the captions improves the performance by 2 3% regardless of the questioner type. By

---

hiding the image caption during training, F is forced to pay more attention to the rest of the dialogue in order to extract information about the target image. Thus, F is able learn even from training examples where retrieval succeeds based on the caption alone.
Question A Answering methods: In Figure 7b, we examine different combinations of questioner and answerer in terms of Average Target Rank (ATR) and observe some interesting trends. We observe that Human and ChatGPT questioners are the only cases that improve along the dialog. As expected, Human answerer generates higher quality answers resulting in lower ATR. For this comparison we also consider a "pool" questioner, i.e., a classifier that selects a question from a closed pool of 40K questions, as well as dialogues generated by two previous RL-based methods [9,34].
We first examine the impact of using BLIP2 as substitute to human answerer, in our evaluation protocol. The same set of human generated questions (from the VisDial validation subset) is used to compare the changes in average target rank when using BLIP2 (green dash-dot) dash-dot) answerer. While BLIP2 is less accurate than Human, both follow the same trend. This justifies our use of BLIP2 to compare between different questioners at scale (Figure 3). In fact, the performance measured using BLIP2 can be considered as an underestimation of the true retrieval performance. Figure 8 shows example with answers of BLIP2 and human (to human questions).
human (orange
a bride and groom cutting into their wedding cake together
can tell it's night day time?
day time
nD
anyone else in the photo?
no
just bride and groom
do they look excited happy?
yes
yes
what color is the cake?
while
while with gold accent
how many tiers it?
three
only hiers photo cuts off
are there flowers anywhare?
nO
no
Figure 8: Dialog example with two different answerers: BLIP2 (left) and Human (right).
Next, we examine the rank performance of dialogues generated by two previous RL-based methods, RL17 [9], and RL19 [34]. We compare dialogues entirely generated by each of the two methods, as well as dialogues where the questions of RL19 are answered by BLIP2. We observe that the RL- based answerer (dubbed A-bot in (34]) performs poorly, since replacing it with BLIP2 significantly improves the results (dashed gray and brown lines vs. dashed pink and magenta). However, even with the BLIP2 answerer, the RL methods still struggle with improving the average retrieval rank as the dialog progresses, and the rank remains nearly constant, similarly the pool questioner model.
Summary and Discussion
Since conversation is natural means of human information inquiry, framing the visual search process within a dialog is expected to make the search process more natural, in terms of query entry and interaction to locate relevant content. In this paper, we proposed ChatiR for image retrieval, a model that chats with the user by asking questions regarding a search for images, being capable of processing the emerged dialog (questions and answers) into improved retrieval results. We showed through extensive experiments that using foundation models we are able to reach a performance level nearly as good as a human questioner. Our analysis yields some interesting insights and results: e.g., failure cause in many alternatives appear to be the inability in continuously generating new genuine questions. Yet, some limitations still exist, e.g., our current concept uses questioner that relies solely on the dialog as an input, to generate the follow-up question. An optimal questioner however, may further consider the retrieved results or a set of candidates in order to extract the most distinguishing attribute for narrowing down the options. We believe that our framework and benchmark will allow further study of the demanding application of chat-based image retrieval, as a tool for improving retrieval results along with continuous human interactions.
10

---

This work was supported in part by the Israel Science Foundation (grants 2492/20 and 3611/21). We would like to thank Elior Cohen, Oded Ben Noon and Eli Groisman for their assistance in creating the web interface. We also thank Or Kedar for his valuable insights.
Acknowledgments:
References
[1] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: Visual Question Answering. In /CCV, 2015.
[2] Alberto Baldrati, Marco Bertini, Tiberio Uriechio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining CLIP-based features. In CVPR, pages 21434-21442. IECE, 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Leamners In NeurlPS, volume 33, pages 1877-1901, 2020. 5
[4] Yanbei Chen, Shaogang Gong. and Loris Bazzani. Image Search With Text Feedback by Visiolinguistic Attention Learning. In CVPR. pages 1999-3008. 2020.
[5] Yen CanChhnn Chen. Linjie Li. Licheng Yu. Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal Image-TExxt Representation I. Learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, volume 12375 of Lecture Notes in Computer Science, pages
104-120. Springer, 2020. 3
[6] Hyung Won Chung. Le Hou. Shayne Longpre. Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma. Albert Webson, Shixiang Shane Gu, Zhuyun Dai. Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu. Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Lc, and Jason Wei. Scaling rnstructionFinetuned Language Models.
CoRR, abs/2210.1 114416,, 2022. 5
[7] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning, 2023. 5
[8] Abhishck Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In CVPR, 2017. 2, 3, 4, 6,
[9] Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learnnng cooperative visual dialog agents with deep reinforcement learning. In ICCV, pages 2951-2960, 2017. 3, 6, 7, 10
[10] Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou. Training vision transformers for
image retrieval. arX0v:2/02.05644, 2021. 1
[11] Jianfeng Gao, Chenyan Xiong. Paul Bennett, and Nick Craswell. Neural approaches to conversational
information retrieval, volume 44. Springer Nature, 2023. 3
[12] Sonam Goenka, Zhaoheng Zheng, Ayush Jaiswal, Rakesh Chada, Yue Wu, Varsha Hedau, and Pradeep Natarajan. VaP Vision Language Transformer for Fashion Retrieval with Feedback. In CVPR,
pages 14085-14095. IEEE, 2022.
[13] Yash Goyal, Tejas Khot, Douglas Summers-Stayy Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In CVPR, 2017. 3
[14] Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennic, Gerald Tesauro, and Rogério Schmidt Feris. Dialog-
based Interactive Image Retrieval. In NewrlPS. pages 676-686, 2018. 3,4
[15] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In Marina Meila and Tong Zhang, editors, ICML, volume 139 of Proceedings of Machine Learning Research, pages 4904 4916 PMLR, 2021. 3.4
[16] Jongseok Kim, Youngjae Yu. Hoeseong Kim, and Gunhee Kim. Dual Compositional Learning Interactive Image Retrieval. AAAI, 35(2):
2021.
[17] Adriana Kovashka and Kristen Grauman. Attribute Pivots for Guiding Relevance Feedback in Image Search. In ICCV, 2013. 3
[18] Adriana Kovashka, Devi Parikh, and Kristen Grauman. WhittleSeach:: Image search with relative attribute feedback. In CVPR, pages 2973 2980. IEEE Computer Society, 2012. 3
[19] Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Data Roaming and Early Fusion for Composed Image Retrieval. arXiv preprint 3rXiv:2303.004429 2023. 1, 3,3, 4
[20] Matan Levy, Rami Ben-Ari, and Dani Lischinski. ClasiffiationnReeeessiion for Chart comprehension. In
ECCV, pages 469-484, 2022. 3,4
[21] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping Enguage-Image
Pre- Pre-training with Frozen Image Encoders and Large Language Models. CoRR, abs/2301. 125597, 2023. 2, 3,
4, 5
11

---

[22] Junnan Li, Dongxu Li. Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping Linguage-Imag Pre-training for Unified Vision-Language Understanding and Generation. In ICML, pages 12888-12900,
2022. 3.4.5
[23] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Goumare, Shafiq R. Joty. Caiming Xiong, and Steven Chu- Hong Hoi. Align before Fusc: Vision and Language Representation Leamiing with Momentum Distillation. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NewrIPS, pages 9694-979990 2021. 3,4
[24] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for
vision-language tasks. In ECCV, August 2020. 3,4
[25] Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. Visual question generation dual task of visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6116612, 2018. 3
[26] Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmcck. Microsoft COCO: Common Objects in Context. In ECCV, volume 8693, pages 740-755, 2014, 5,5
[27] Zheyuan Liu, Cristian Rodriguez- Opazzo, Damien Teney, and Stephen Gould. Image Retrieval on Real-life
Images with Pre-trained Vision-and- and-Language Models. In ICCV, pages 2105 2214, 2021. 1,4
[28] Andreas Lommatzsch and Jonas Katins. An information retrieval- based approach for building intuitive
chatbots for large knowledge bases. In LWDA, pages 143-352, 2019. 3
[29] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. ViLBERT: Pretraining Task-Agnostic Visiolinguistic
Representations for Vision-and- Aioon-nnd-Languee Language Tasks. In NeurlPS, pages 13-23 23, 2019. 3,4 4
[30] Xiaopeng Lu, Tiancheng Zhao, and Kyusong Lec. VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, ACL., 2021.
[31] Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and Stéphane Marchand-Maillet. Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders. ACM, 2021.
[32] Gaurav Mittal, Shubham Agrawal, Anuva Agarwal, Sushant Mehta, and Tanya Marwah. Interactive image generation using seene graphs. In ICLR, 2019. 3
[33] Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. Large-Scale Pretraining for Visual Dialog: A Simple State-of thee-of-th--Ar the-Art Baseline. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- an-Michael Frahm, editors, ECCV, volume 12363 of Lecture Notes in Computer Science, pages 336 352. Springer, 2020. 3,4
[34] Vishvak Murahari, Prithvijit Chattopadhyay, Dhruv Batra, Devi Parikh, and Abhishek Das. Improving generative visual dialog by answering diverse questions. EMNLP, 2019. 3, 6, 7,9, 10
[35] Robert Oddy. Information retrieval through man-machine dialogue. Journal of documentation, 33(1):1-
14, 1977. 3
[36]
OpenAL. ChatGPT. https: / http///pppaaaaa//////e//ee com wwtp///////cc..////aaa////// 2023. 1,5
[37]
OpenAI. GPT-4 Technical Report. CoRR, abs/2303.088744 2023.
[38]
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke
Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback. In NeurIPS, 2022. 1
[39] Vishal Pallagani and Biplav Srivastava. A generic dialog agent for information retrieval based on auto- mated planning within a reinforcement learning platform. Bridging the Gap Between Al Planning and Reinforcement Learning (PRL), 2021.
[40] Devi Parikh and Kristen Grauman. Relative attributes. In Dimitris N. Metaxas, Long Quan, Alberto Sanfeliu, and Luc Van Gool, editors, ICCV, 2011. 3
[41] Yash Patel. Giorgos Tolias, and Jifí Matas. Recall@k surrogate loss with large batches and similarity mixup. In CVPR, pages 7502 7511, 2022. 4
[42] Badri Patro, Vinod Kurmi, Sandeep Kumar, and Vinay Namboodiri. Deep bayesian network for visual question generation. In WACV, pages 1566-1576. 2020. 3
[43] Lorenzo Putzu, Luca Piras, and Giorgio Giacinto. Convolutional neural networks for relevance feedback in content based image retrieval: A content based image retrieval system that exploits convolutional neural networks both for feature extraction and for relevance feedback. Multimedia Tools and Applications, 79:26995-27021, 2020. 3
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Marina Meila and Tong Zhang, editors, ICML, 2021. 3,
[45] Yong Rui, Thomas S. Huang, Michael Ortega, and Sharad Mehrotra. Relevance feedback: a power tool for interactive content-based based image retrieval. IEEE Trans. Circuits Syst. Video Technol. 1998. 3
12

---

[46] Chull Hwan Song, Jooyoung Yoon, Shunghyun Choi, and Yannis Avrithis, Boosting vision transformers for image retrieval. In WACV, pages 107-117, 2023.
[47]
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A Strong, Replicable Cetrructioo-Followiii Model. https: / //orfm. wtw.fora.dd..r.ro wdw//023002 /0//////// 33/3133// ///paaaaaaaa..a..aa..aaaaaaaaaaa htmmh,m11........ 2023.
[48]
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du. YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng. Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier- Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Areas Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. LaMDA: Language Models for Dialog Applications. CoRR, abs/2201.082333 2022.
[49]
Nam Vo, Lu Jiang. Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing Text and Image Image Retrieval an Empirical Odyssey. CVPR, pages 6432 6441, 2019. 3,4
[50] Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying Architectures. Tasks, and Modalities Through Simple
Sequence-to-Se -e-uueecce Learning Framework. ICML, volume 162. pages 1334400, 2022.
[51] Chenfei Wu, Shengming Yin. Weizhen Qi. Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. S0tI:23003.06777 2023. 3
[52] Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogério Feris. Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback. In CVPR, pages
11307-1131,, 2021. 3. 4
[53] Yiling Wu, Shuhui Wang. Guoli Song, and Qingming Huang. Learning Fragment Self- Attentien Embed- dings for Image-Text Matching. In Laurent Amsaleg, Benoit Huet, Martha A. Larson, Guillaume Gravier, Hayley Hung. Chong Wah Ngo, and Wei Tsang Ooi, editors. ACM, pages 2088 2096. ACM, 2019. [54] Zipeng Xu, Fandong Meng, Xiaojie Wang, Duo Zheng. Chenxu Lv, and Jie Zhou. Modeling Explicit Concerning States for Reinforcement Learning in Visual Dialogue. In BMVC, 2021.
[55] Peter Young, Alice Lai. Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67-78, 2014. 5
[56] Yan Zeng, Xinsong Zhang, and Hang Li.
Vision Language Pre- Training: Aligning Texts with Visual Concepts. In JCML, volume 162, pages 25994 26009, 2022. 3
[57] Duo Zheng. Zipeng Xu, Fandong Meng. Xiaojie Wang. Jiaan Wang, and Jie Zhou. Enhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser. In EMNLP, 2021.
[58] Luowei Zhou, Hamid Palangi, Lei Zhang. Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified Vision LanguaLe Pre Pre-Tnain for Image Captioning and VQA. In AAAI, pages 13041-13049 2020. 3 [59] Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. ChatGPT asks, BLIP-2 answers: Automatic questioning towards enriched visual descriptions. arXiv:2303.c 065944 2023.
13


Reviews:
[{"review_id": "lK6I0T1Lib", "reviewer": "Reviewer_6Lfv", "summary": "This paper proposes a chat-based image retrieval framework, which can clarify users’ search intent. Authors design a question generation model based on LLM to generate questions based on dialog history. After user answer the question, an image retriever which is a transformer model is trained to extract text embedding to search image. Authors also use a LLM to answer the questions taking the place of users for fast training and evaluation. Authors use existing dataset to evaluate the method.", "questions": "See weakness", "limitations": "See weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed framework is useful for clarifying user search intent, thus it’s practically valuable.\n+ The proposed framework is well evaluated.\n+ Authors use a existing dataset to avoid collecting a new dataset.", "weaknesses": "--- Components of the proposed framework are existing models, this weakens the novelty of this paper.\n--- Lack comparison with sota image-text retrieval methods on image-text retrieval datasets in experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **6Lfv**, thank you for your insightful feedback!\n\n**Lack comparison with SoTA image-text retrieval methods on image-text retrieval datasets in experiments:**\n\nPlease see the global part of the rebuttal and the figures in the accompanying pdf, where we address this concern.\n"}, {"review_id": "Uz1mm30mu5", "reviewer": "Reviewer_Z6DK", "summary": "This study introduces ChatIR, a chat-based image retrieval system that engages in a conversation with the user to clarify their search intent and retrieve the desired image from a large corpus. The system leverages Large Language Models to generate follow-up questions to an initial image description and achieves a success rate of over 78% after 5 dialogue rounds, compared to 75% when questions are asked by humans and 64% for a single shot text-to-image retrieval.\n", "questions": "(1) Missing reference for \"common practice in image retrieval\" in line 148.\n\n(2) The readability of Section 4 is currently hindered by the complexity of the annotations, such as the format of \"Q: XXX & A: YYY.\" To enhance readability, I recommend using a different font or formatting approach to simplify the annotations in the revised version. Simplifying the annotations will make the section more accessible and easier to follow for readers, ensuring a smoother comprehension of the methodology and findings.\n\n", "limitations": "The authors have included discussions of the limitations.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The strength of this submission lies in its clear and compelling motivation, which focuses on utilizing chat interactions to refine search and enhance image retrieval. The authors effectively articulate the significance of this research direction, highlighting the potential of chat-based interactions to improve the accuracy and relevance of image search results. Additionally, the submission features a nice illustration that visually communicates the proposed approach, providing a clear representation of the underlying concept. This visual aid aids in understanding the methodology and reinforces the clarity of the paper.\n", "weaknesses": "One notable weakness of the submission is the lack of baseline comparison. While the paper introduces a novel conversation-based setup for image retrieval, it only reports results within this specific setup. The absence of results on the traditional single-hop text-to-image retrieval using the same dataset raises concerns about the necessity of introducing conversation into the retrieval process. Without a comparison to traditional single-hop methods, it is difficult to fully understand the advantages and potential improvements offered by the conversation-based approach (what if traditional methods already achieve comparable success rates?). Such comparisons would help address the question of why the system should be made more complex with conversation, and provide a stronger rationale for the proposed approach.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **Z6DK**, thank you for your insightful feedback!\n\n**The absence of results on the traditional single-hop text-to-image retrieval:**\n\nPlease see the global part of the rebuttal and the figures in the accompanying pdf, where we address this concern.\n\nQ1: Missing reference: Thank you. We will add it.\n\nQ2: Readability Q: XXX & A: YYY: We appreciate this suggestion and will change it in the revised version.\n"}, {"review_id": "5xhrvEyT8J", "reviewer": "Reviewer_34K9", "summary": "In this paper, the authors present a dialog-based image retrieval system and show strong performance against baseline models.", "questions": "Check my comments for weaknesses.", "limitations": "It is not clear how this works could be expanded beyond the current setup majorly because it is not very clear what type of dialogue one needs to have with the system.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The author's attempt to augment the image retrieval process with dialogue is interesting and largely under-explored. \n\n2. The paper is well-written and easy to read.\n", "weaknesses": "1. The authors are largely missing out on what type of dialogue one needs to have (given the caption) such that it helps to better visualize the caption. Why one even needs to do that? Is it because the captions are not useful/detailed?\n2. The experiments are solely designed around the availability of instruction-tuned LLMs (for question generation), BLIP2 (for answering), and the VisDial dataset. Once you answer my previous comment, please justify why VisDial (alone) is enough for this experiment. Are instruction-tuned LLMs generating the type of questions you expected? \n3. The paper is not well motivated. Specifically, the need to have a dialogue for image retrieval. Though the results are encouraging the reason why we are doing these experiments in the first place is not clear. I request the authors to give some more thought to this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **34K9**, thank you for your insightful feedback!\n\n1: **Regarding the motivation and “What type of dialogue one needs to have (given the caption) …”**:\nThe motivation is that, typically, a short query is not sufficient to retrieve the correct images from a large corpus, certainly not in a single attempt, as we discussed in lines 27-30 and 35-39. Commonly, people search an image with a short description, that might fail to fully convey the search intent, or sometimes it may result in many images that comply with the same description (please see Fig 4,5 and examples in suppl. material). Our idea is to engage in a conversation with the user in order to elicit additional information, and clarify the user’s search intent (lines 35-39). Gradually eliciting and accumulating the information from the user and being able to process it in a unified way is the essence of our ChatIR.\n\n**Type of questions:** Since the end goal of ChatIR is to retrieve the image, we expect the questioner to generate questions that when answered, improves the retrieval results (which is the instruction we give to ChatGPT). These questions typically relate to color, location, time of day, number of certain objects and more (see examples in Figs 4 and 5 in the main paper, and Fig. 1-5 in suppl. for LLM and human questioner).\n\nWe show that such dialogues are able to significantly boost the retrieval performance (lines 173-176 and 208-210). Please also see the global part of the rebuttal and the figures in the accompanying pdf, where we discuss the advantages of our approach.\n\n2.1: **“please justify why VisDial (alone) is enough for this experiment”**: Since our task requires data consisting of images paired with dialogues, we use the VisDial dataset that contains such annotations (although it was annotated for a different task), as we describe in lines 55-59. Thanks to VisDial we are able to test ChatIR in many different scenarios, such as combining a human questioner and a machine or human answerer. These results are presented in Fig 2 (paper) and Fig 8 (suppl. material).\n\n2.2: **“Are instruction-tuned LLMs generating the type of questions you expected?”**: Yes. The end goal of ChatIR is to retrieve the image, thus we expect the questioner to generate a question that when answered, improves the retrieval results. This is indeed the case as our evaluations show (Fig 2, 3 and in suppl. material). More specifically, as we discuss in Sec 4.1, some Instruction-tuned LLMs generate the type of questions we expected (e.g. ChatGPT, see examples in Fig 4, 5 and Suppl.material). On the other hand, some others (e.g. FLAN-T5, FLAN-ALPACA) struggle with long context, which results in question repetitions (lines 224-231) resulting degraded performance (Fig 2).\n\n3: **Motivation**\n\nPlease see our response above\n"}, {"review_id": "fPrDhOsHcb", "reviewer": "Reviewer_L59s", "summary": "The paper tackles the problem of chat-based image retrieval. The goal is that the system, being chat-based and powered by a LLM, engages in a conversation with the user to elicit information, in addition to an initial query, in order to clarify the user’s search intent by asking follow up questions. These questions form a dialog with the user in order to retrieve the desired image from a large corpus. The authors also suggest an evaluation protocols suitable for continual progress and assessment of questioners using a Visual Dialog model in place of a human and test their framework on real human interactions which involves collecting answers from users and further evaluate the method against strong baselines generated from prior art.", "questions": "1. Are there other methods than BLIP2 that can be used to answer the questions? Based on the results from Fig 3, up to a certain dialog round of 4-5 there is no difference between human answers and BLIP2. Do the authors have any insights on what happens beyond that point?\n\n2. Did the humans have access to the image when answering the questions?\n\n3. What data do you use for evaluation since it's not clear? VisDial?\n\n4. Will all the data and all the evaluation protocol be made available online?\n\n5. Can you elaborate a bit on the Unanswered setup? The model has available the caption + several questions but no answer when making the retrieval?", "limitations": "Various limitations are discussed throughout the paper and there the societal impact is somehow discussed, but briefly.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper tackles an interesting problem and proposes an interesting setup as opposed to the text-image retrieval task. If all data and the protocol is made available, I think this can be very valuable for the research community.", "weaknesses": "I really like the overall goal of the paper and I think it's an interesting direction. However, I have few concerns in terms of clarity (please see questions below) that I think need to be further detailed in the paper. Also, what I think it's lacking the current version is a comparison with classical text-image retrieval methods. If I understood correctly, the method can be applied to any dataset, so I think that applying it to some common benchmarks for text-image retrieval and showing how the dialog rounds affect the results can further validate this approach. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **L59s**, thank you for your insightful feedback!\n\n**Lack of comparison with classical text-image retrieval methods:**\n\nPlease see the global part of the rebuttal and the figures in the accompanying pdf, where we address this concern.\n\n### Answers to questions: ###\n\n1.  **“Are there other methods than BLIP2 that can be used to answer the questions?”**:  Theoretically, any generative Visual-Dialog model might serve as an answerer. We chose the SoTA model for this task (BLIP2).\n**“Fig 3, up to a certain dialog round of 4-5 there is no difference between human answers and BLIP2. Do the authors have any insights on what happens beyond that point?”**:  Our analysis shows that human answers are longer (see analysis in lines 29-34 in suppl. material) implying that humans tend to volunteer additional information, while BLIP2 provides short and concise answers (e.g. Fig 7, and suppl. Fig 5). Another source of this discrepancy might be the less accurate and less elaborate BLIP2 answers at long dialogues.\n\n2.  **“Did the humans have access to the image when answering the questions?”**: Yes, in all experiments that involved ground truth human answers, users had access to the image.\n    \n3.  **“What data do you use for evaluation since it's not clear? VisDial?”**: Yes, our evaluation was primarily based on different measures for the image retrieval task. We use the VisDial dataset (lines 55-59), where each image is associated with a dialog. Thanks to VisDial we are able to test ChatIR in many different scenarios, such as combining a human questioner and a machine or human answerer. We test these different dialog options and measure the retrieval capabilities of images from the pool (50k). As mentioned earlier, for this rebuttal we also use COCO and Flickr30k with synthetic dialogues .\n    \n4.  **“Will all the data and all the evaluation protocol be made available online?”**: Yes, the entire data and protocols will be made available upon acceptance.\n5. **“Can you elaborate a bit on the Unanswered setup?”**: Following the description in lines 194-200, in this setting the *questioner* (ChatGPT) is provided with the caption only. It is asked to generate at once 10 different questions based solely on the caption (without seeing any answers). Thus, although the retrieval is conducted using the full dialogues (questions and answers), in this setting the answers have no effect on question generation in the dialog. We will clarify this in the revision. This experiment also shows the answers’s influence on the question generation, namely, questions conditioned on the prior dialogue history are more effective for retrieval.\n"}]

Author rebuttal:

Dear Reviewers and ACs,  
We were happy to see that the reviewers have found that our paper presents: “interesting problem and proposes an interesting setup” (L59s), “process with dialogue is interesting and largely under-explored” (34K9), The strength of this submission lies in its clear and compelling motivation (Z6DK), The proposed framework is practically valuable and well evaluated (6Lfv). Your input is instrumental for improving our paper.

  

The main concern of the reviewers seems to be regarding lack of comparisons with existing Text-to-Image retrieval baselines and evaluations on additional datasets. We would like to stress that we indeed compare our retrieval method to an existing Text-To-Image (TTI) retrieval baseline (on VisDial), reporting results in lines 174-176 and 208-210. More specifically, the baseline we compare to is BLIP, since this is the publicly available SoTA model for TTI [1]. For a fair comparison we further fine-tuned this baseline model on the VisDial dataset for the TTI task (providing it with images and their captions only). We find that the retrieval performance of the fine-tuned single-hop text-to-image baseline, is nearly identical to our dialogue-trained model (63.66% vs. 63.61%). This corresponds to dialogues with 0 rounds in Figure 2a. However, using increasingly longer dialogues in our method, eventually achieves retrieval performance over 81% (Fig. 2a), showing a huge improvement over the single-hop TTI SoTA baseline.

  

As for the choice of dataset, since VisDial is the only dataset containing image-dialogue pairs, we found it to be the most suitable dataset and benchmark for evaluating our method. However, due to the raised concerns and for further validation, we generated two synthetic image-dialogue datasets (using ChatGPT as a questioner, and BLIP2 [2] as an answerer) from two Text-To-Image (TTI) benchmarks, Flickr30k and COCO (see figures in the PDF attachment). We then compared our method to two TTI baselines, namely CLIP and BLIP, in a zero-shot setting (i.e., none of the compared methods have been fine-tuned on either dataset). We find that our method surpasses the two baselines by a large margin, on both datasets. Furthermore, when we provide the baselines with the concatenated text of the dialogues, instead of just a caption, they exhibit a significant improvement in success rate (please see attached figures) over the single-hop TTI attempt. Nevertheless, the gap in favor of our method is maintained (COCO) or increased (Flickr30k).

Our zero-shot results on COCO and Flickr30k (in attached figures) show that:

1.  Dialogues improve retrieval results for off-the-shelf TTI models. Although the CLIP and BLIP baselines have only been trained for retrieval with relatively short (single-hop) text queries, they are still capable of leveraging the added information in the concatenated Q&A text. Note that CLIP becomes saturated at a certain point due to the 77 token limit on the input.
    
2.  Our strategy of training an Image Retriever model with dialogues (as described in Sec. 3) further improves the retrieval over the compared methods, raising accuracy from 83% to 87% at single-hop retrieval, and surpassing 95% after 10 dialogue rounds (on COCO).
    

We believe the above addresses the main concerns in the reviews, and we welcome further discussion.

References:

[1] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML, pages 12888–12900, 2022

[2] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. CoRR, abs/2301.12597, 2023


Meta-review: The work is about an approach to chat-based image retrieval, leveraging a language model for user interactions and aiming to enhance search results through dialog. The motivation for this research direction is clearly articulated, emphasizing the potential of chat-based interactions to refine search queries and improve image retrieval relevance. 

However, reviewers pointed out some weaknesses in the paper. 
- The lack of baseline comparisons to traditional single-hop text-to-image retrieval methods using the same dataset raises questions about the necessity of introducing conversation into the retrieval process. Such comparisons would better justify the complexity of the proposed approach and its potential advantages. Even though the authors mentioned they compared with BLIP in a few lines, the details of how BILP baseline was set up are missing, as well as some other baselines (e.g., BILP-2).
- The paper would benefit from more detailed explanations regarding the types of dialogues needed to aid in visualizing captions and the rationale for employing conversation in image retrieval. 

Regarding datasets to be evaluated, it worth checking multi-modal dialogue datasets as well, for examples, I would recommend instead of simulated dataset in the future version, the authors can check some of these datasets for experiments (https://github.com/ImKeTT/Awesome-Multi-Modal-Dialog).
Decision: Accept (poster)