SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we measure question repetitions for each questioner as a suspected cause behind the performance levels of different questioners (previously addressed in [34]). For each questioner we calculate the average number of exact repetitions of questions per dialog (i.e., out of 10). While the average number of repetitions is nearly 0 for either human or ChatGPT questioners, the other methods exhibit an average of  $1.85 - 3.44$  repeated questions per dialog. We observe that these findings are correlated
(a) Questioners Hits@10 success (higher is better).
Figure 3: Left: Evaluation of different chat questioner methods on VisDial. For all cases (including "Human") the answers are obtained from BLIP2. Note that dialog with 0 rounds is only the image caption, a special case of the text-to-image retrieval task. Top and super-human performance is obtained by ChatGPT with a significant gap over several versions of FLAN and previous RL based methods,  $RL_{17}$  [9] and  $RL_{19}$  [34]. Right: Average rank of target images after each round of dialog.
(b) Target image average rank (lower is better).
with the overall performance of questioners in Figure 3, since repetitive questions add little or no information about the target image. More details are available in our suppl. material, where we also measure uniqueness at the token level (repeated questions result in fewer unique tokens per-dialog).
4.3 Comparison to Human In The Loop
While above we examined all the questioner methods in conjunction with BLIP2 answers, here we evaluate the performance with as human as the answer provider. Our incentive for this experiment is 1) To evaluate how answers of BLIP2 compare to those of humans. Is there any domain gap? 2) To test our top-performing questioner in a real ChatIR scenario. To this end, we conduct dialogues on  $\sim 8\%$  of the images in the VisDial [8] validation set (through a designed web interface), between ChatGPT (Questioner) and Human (Answerer). Note we also have fully human-labelled dialogues (Q: Human, A: Human) collected in the dataset. We refer the reader to suppl. material for further information about the data collection.
Figure 4 shows the results where we present three combinations of  $\text{Questioner}^Q$  and  $\text{Answerer}^A$ :  $\text{ChatGPT}^Q \& \text{Human}^A$  (blue dot line),  $\text{ChatGPT}^Q \& \text{BLIP2}^A$  (blue solid line) and the reference of  $\text{Human}^Q \& \text{Human}^A$  (red dot line). We observe that while all experiments perform similarly, up to 5 dialog rounds, beyond that point,  $\text{Human}^Q \& \text{Human}^A$  (red dot line) outperforms both ChatGPT alternatives. Considering the previous experiment in Figure 3, showing the advantage of ChatGPT over Human as questioner, we observe that Human generated answers are of better quality than BLIP2 (in terms of final retrieval results). This advantage boosts the Human full loop to become more effective than ChatGPT. The results imply a small domain gap between BLIP2 and Human answerer (but with similar trend), justifying the usage of BLIP2 in our evaluations. Furthermore, we observe that in the real use-case of  $\text{ChatGPT}^Q \& \text{Human}^A$  the model reaches a high performance of 81% Hit rate using only 5 dialog rounds. Similar to previous results, we observe a saturation in the marginal benefit of the last dialogues rounds.
Next we show some qualitative examples. More examples as well as dialogues e.g. Human vs. BLIP2 answers can be found in the suppl. material. Figure 5 describes a search for a traffic light, where the answerer is BLIP2. At round 0 (search by the caption), the target image is ranked as 1, 149 due to existence of many traffic lights in the corpus and also erroneous candidates. The results show the rapid decrease in the rank reaching at the top of the candidates after only 3 dialog rounds. Note that
(a) Questioners Hits@10 performance (higher is better).
Figure 4: Human-AI. 1. We want to show performance on real scenario. 2. Verify the validation method BLIP2 vs. Human answerer 3. Human-Human is the best. BLIP2 answers are lower quality according to (a) We conducted this experiment to evaluate the difference, but still BLIP2 still presents a reasonable performance although inferior to human
Figure 5: ChatIR example: a dialog is conducted between ChatGPT and a user imitator (BLIP2) on the ground truth image (green frame). Top-5 retrievals are presented after each dialog round.
since the first round, when the location is specified as "a house", the retrieved images are more likely traffic lights with a house in the background. Figure 6 shows another case with search for a specific train. This example is drawn from our test case against Human answerer. Note how the train at top of the list changes from black to green and blue after a question about the color is asked (second round), boosting the rank from 22 to 2. The last round fine-tunes the top-2 results by distinguishing between parking forms of on track vs on platform.
We conducted further analysis of using our pipeline for generating additional data, wherein the questioner and answerer collaborated to generate dialogues on more images. The inclusion of these dialogues and their corresponding images in the training set resulted in an improvement in the Average Rank metric, but did not provide any benefit to the specific retrieval measure of Hit@10. Due to lack of space, we discuss these results in the suppl. material.
Figure 6: ChatIR example: a dialog is conducted between ChatGPT and Human on the ground truth image (green frame). Top-5 retrievals are presented after each dialog round.
In this section we conduct an ablation and examine different strategies for training the Image Retriever model  $F$ . We further examine a few Questioner ( $G$ ) baselines and discuss our evaluation protocol.
(a) Comparison of masking strategies.
Figure 7: Left: Retrieval performance of image retriever models  $(F)$  trained with different masking strategies. Results are reported for two different answerers  $G$ . Right: Impact of different Questioner and Answerer models on the average target ranking, as the dialog progresses (lower is better).  $RL_{17}$  [9] and  $RL_{19}$  [34] represent previous RL methods.
(b) Rank performance for different Q&A models.
Masking strategy: In this experiment we examine five different masking strategies for training of the Image Retriever model  $(F)$ . Figure 7a shows evaluations of the resulting  $F$  models using two different questioners  $G$ : ChatGPT (solid lines) and Human (dashed lines). As a baseline we train  $F$  using dialog sequences (concatenated as described in Section 3), without masking any parts of the dialog. Next, we randomly mask different components of the training dialogues: captions, questions, answers, entire Q&A rounds, or individual tokens. In each strategy, we randomly select  $20\%$  of the components of a certain type for masking. The results in Figure 7a show that among these strategies, masking the captions improves the performance by  $2 - 3\%$  regardless of the questioner type. By
hiding the image caption during training,  $F$  is forced to pay more attention to the rest of the dialogue in order to extract information about the target image. Thus,  $F$  is able to learn even from training examples where retrieval succeeds based on the caption alone.
Question Answering methods: In Figure 7b, we examine different combinations of questioner and answerer in terms of Average Target Rank (ATR) and observe some interesting trends. We observe that Human and ChatGPT questioners are the only cases that improve along the dialog. As expected, Human answerer generates higher quality answers resulting in lower ATR. For this comparison we also consider a "pool" questioner, i.e., a classifier that selects a question from a closed pool of 40K questions, as well as dialogues generated by two previous RL-based methods [9, 34].
We first examine the impact of using BLIP2 as a substitute to a human answerer, in our evaluation protocol. The same set of human generated questions (from the VisDial validation subset) is used to compare the changes in average target rank when using BLIP2 (green dash-dot) or a human (orange dash-dot) answerer. While BLIP2 is less accurate than Human, both follow the same trend. This justifies our use of BLIP2 to compare between different questioners at scale (Figure 3). In fact, the performance measured using BLIP2 can be considered as an underestimation of the true retrieval performance. Figure 8 shows an example with answers of BLIP2 and a human (to human questions).
a bride and groom cutting into their wedding cake together
Figure 8: Dialog example with two different answerers: BLIP2 (left) and Human (right).
Next, we examine the rank performance of dialogues generated by two previous RL-based methods,  $RL_{17}$  [9], and  $RL_{19}$  [34]. We compare dialogues entirely generated by each of the two methods, as well as dialogues where the questions of  $RL_{19}$  are answered by BLIP2. We observe that the RL-based answerer (dubbed A-bot in [34]) performs poorly, since replacing it with BLIP2 significantly improves the results (dashed gray and brown lines vs. dashed pink and magenta). However, even with the BLIP2 answerer, the RL methods still struggle with improving the average retrieval rank as the dialog progresses, and the rank remains nearly constant, similarly to the pool questioner model.
6 Summary and Discussion
Since conversation is a natural means of human information inquiry, framing the visual search process within a dialog is expected to make the search process more natural, in terms of query entry and interaction to locate relevant content. In this paper, we proposed ChatIR for image retrieval, a model that chats with the user by asking questions regarding a search for images, being capable of processing the emerged dialog (questions and answers) into improved retrieval results. We showed through extensive experiments that using foundation models we are able to reach a performance level nearly as good as a human questioner. Our analysis yields some interesting insights and results: e.g., a failure cause in many alternatives appear to be the inability in continuously generating new genuine questions. Yet, some limitations still exist, e.g., our current concept uses a questioner that relies solely on the dialog as an input, to generate the follow-up question. An optimal questioner however, may further consider the retrieved results or a set of candidates in order to extract the most distinguishing attribute for narrowing down the options. We believe that our framework and benchmark will allow further study of the demanding application of chat-based image retrieval, as a tool for improving retrieval results along with continuous human interactions.
Acknowledgments: This work was supported in part by the Israel Science Foundation (grants 2492/20 and 3611/21). We would like to thank Elior Cohen, Oded Ben Noon and Eli Groisman for their assistance in creating the web interface. We also thank Or Kedar for his valuable insights.