Vision-by-Language for Training-Free Compositional Image Retrieval
Authors: Shyamgopal Karthik
Karsten Roth
Massimiliano Mancini
Zeynep Akata
Venue: ICLR 2024 poster (2024)
Keywords: Vision-Language Models
Large Language Models
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A simple method using off-the-shelf foundation models for Composed Image Retrieval without any training

Abstract: Given an image and a target modification (e.g an image of the Eiffel tower and the text “without people and at night-time”), Compositional Image Retrieval (CIR) aims to retrieve the relevant target image in a database. While supervised approaches rely on annotating triplets that is costly (i.e. query image, textual modification, and target image), recent research sidesteps this need by using large-scale vision-language models (VLMs), performing Zero-Shot CIR (ZS-CIR). However, state-of-the-art approaches in ZS-CIR still require training task-specific, customized models over large amounts of image-text pairs. In this work, we proposeto tackle CIR in a training-free manner via our Compositional Image Retrieval through Vision-by-Language (CIReVL), a simple, yet human-understandable and scalable pipeline that effectively recombines large-scale VLMs with large language models (LLMs). By captioning the reference image using a pre-trained generative VLM and asking a LLM to recompose the caption based on the textual target modification for subsequent retrieval via e.g. CLIP, we achieve modular language reasoning. In four ZS-CIR benchmarks, we find competitive, in-part state-of-the-art performance - improving over supervised methods Moreover, the modularity of CIReVL offers simple scalability without re-training, allowing us to both investigate scaling laws and bottlenecks for ZS-CIR while easily scaling up to in parts more than double of previously reported results. Finally, we show that CIReVL makes CIR human-understandable by composing image and text in a modular fashion in the language domain, thereby making it intervenable, allowing to post-hoc re-align failure cases. Code will be released upon acceptance.

Published as a conference paper at ICLR 2024
VISION-BY-LANGUAGE FOR
TRAINING- COMPOSITIONAL IMAGE RETRIEVAL
Shyamgopal Karthik Karsten Roth--x, Massimiliano Mancini", Zeynep Akata 1334
'Tübingen AI Center & University of Tübingen. University of Trento
'Helmholzz Munich, 'Technical University of Munich equal contribution
ABSTRACT
Given an image and a target modification (e.g g an image of the Eiffel tower and the text "without people and at night-time"), Compositional Image Retrieval (CIR) aims to retrieve the relevant target image proaches rely on annotating triplets that is costly (i.e. query image, textual modi- fication, and target image), recent research sidesteps this need by using large-scale rision-languaee models (VLMs), performing Zero-Shot CIR (ZS-CIR). How- ever, state-of-the-art approaches in ZS-CIR still require training task-specific, cus- tomized models over large amounts of image-text pairs. In this work, we propose to tackle CIR in training-free manner via our Compositional Image Retrieval through UisionbyyLanguage (CIReVL), a simple, yet human- undrrasaanndess and scalable pipeline that effectively recombines large-scale VLMs with large lan- guage models (LLMs). By captioning the reference image using a pre-trained generative VLM and asking LLM to recompose the caption based on the textual target modification for subsequent retrieval via e.g. CLIP, we achieve modular language reasoning. In four ZS-CIR benchmarks, we find competitive, in-part state-of-the-ar art performance improving over supervised methods. Moreover, the modularity of CIReVL offers simple scalability without re-training, allow- ing us to both investigate scaling laws and bottlenecks for ZS-CIR while eas- ily scaling up to in parts more than double of previously reported results. Fi- nally, we show that CIReVL makes CIR human-understn understatddtt by composing image and text in modular fashion in the language domain, thereby making it intervenable, allowing to post-hoc re-align failure cases. Code available at githhbbcom//xpaannbeM///iii Languuua
database. While supervised ap-
INTRODUCTION
Compositional Image Retrieval (CIR) necessitates nuanced coupling between the image content and the semantics the textual query to retrieve new image that accurately embodies the relevant image elements and the modifications described in the textual query. To achieve this, previous works require curated triplets (query image, modifying text, target image) to train a specific CIR system. However, annotating such triplets is both difficult and labor-intensive. To tackle this problem, recent research proposed Zero-Shot CIR (ZS-CIR) (Saito et al. 2023 Baldrati et al. (023). Based on large-
scale pre-trained vision-language models (VLMs) (e.g. CLIP (Radford al.
2021 ), these methods
use image-caption pairs to train textual inversions (Gal et al., 2023 Cohen et al., 2022) mapping
images to text tokens. A static template merges tokens and textual modifications
obtain target
captions, performing CIR without explicit supervision. Thus, even when leveraging large-scale VLMSs 7S-CIR methods still train additional mapping networks on large image-caption datasets.
In this work, we propose to achieve training-free ZS-CIR by leveraging ubiquitously available, off- the-shelf models already trained with large-scale training data. Our Compositional Image Retrieval through Vision-by- -Lnnguugee (CIReVI) follows the vision-by-language paradigm (Zeng et al.) 2023 Zhu et al. 2023 Berrios et al.- 2023 Levy et al. 2023a), which uses language as an ab- straction layer for reasoning about visual content. Specifically, CIReVL employs vision-language models like BLIP-2 (Li al. (203) or CoCa Yuu et al. to generate a detailed description of the query image. Subsequently, L.L (s.a. Llama (Touvron et al. 2023) GPT (Brown et al.) 2020

---

Published as a conference paper at ICLR 2024
crafts caption for the desired target image, using both the generated description and the respective textual query. Finally, a VLM like CLIP (Radford et al., (021) retrieves the image. CIReVL ab- stracts away the compositional nature of the problem into the language domain, converting ZS-CIR into an inherently modular task comprising captioning, reasoning and cross-modal retrieval.
This opens up various benefits not available in traditional, trained ZS-CIR approaches. Beyond not requiring additional adaptation resources, the training-free and modular nature offers the flex- ibility for simple model changes and replacements, allowing us to scale up CIReVL using freely available models. Consequently, while CIReVL already matches and even outperforms trained methods on the common ZS-CIR benchmarks CIRCO (Baldrati et al., 2023) and CIRR (Liu al. 2021) using comparable model architectures, simple buug-and-paay of large retrieval models raises improvements significantly, in parts more than doubling previous results. In addition, CIReVI. is modular and operates primarily in the language domain, as the outputs of the captioning module and the L.MM-generated modifications are textual, offering a degree of understanding over the com- positional retrieval process to humans. This is further reflected in the ability for possible human intervention on the retrieval process to fix or post-hoc improve results (Fig Finally, we show the generality of CIReVL on domain conversion (Saito et al. 2023) and conditional image similarity Vaze et al. 2023). and ablation studies elucidate the role of each pipeline component.
Overall, our contributions are: 1) We explore training-free free zero-shot compositional image retrieval, proposing CTRVVI, a new approach that matches or outperforms existing training-basee methods on four CIR benchmarks while only relying on off-the-shelf available pre-trained models. 2) We show how the inherent modularity of CIReVL and its reasoning over the textual query in the language domain facilitates a degree of human understanding over the compositional retrieval process, even allowing for user-level intervention. 3) We conduct multiple additional studies to ablate pipeline components and point to the importance of language-level reasoning over the textual query, while highlighting the simple scalability of CIReVL through its modular, training-free free nature.
2 RELATED WORK
Compositional Image Retrieval. The task of Compositional Image Retrieval has found significant application in conditional search (Wu et al. 2021 Han et al., 2017: Vo et al. 2019), where users perform interactive dialogue to refine a given query image toward retrieving specific items. Classical techniques often employ custom models that project text-image pairs into common embedding
space (Vo et al.] [2019] Baldrati et al. 2022 Chen et al. 2020 Chen & Bazzani 2020 Lee et al,
2021 Anwaar al. 2021) using contrastive objectives (Sohn 2016 Radford et al. [021] Roth et al. 2022b) or cross modal attention (Delmas et al. (022) and is closely related to compositional
learning (Misra al.) 2017; Mancini al.] 202 Karthik et al.] (002)). With the advent of vision-
language foundation models (Bommasani et al. 2021 Radford et al. |021 Jia et al.) 2021). interest
in CIR has surged, especially in zero-shot settings without task task specifie models. Two prominent directions exist: one using pseudo-tokens to represent reference images, which are then concatenated with the reference caption (Saito et al., 2023 Baldrati et al. 2023 bai et al., 2024); the other trains
foundation models on curated triplets tailored for CIR (Liu et al. 2023 Gu et al., 2023] Ventura
jt al., 2024| Levy et al., 2023b). We explore a different pipeline by coupling VLMS with L.MSs to address CIR without any specialized training, in an effective and interpretable manner.
Vision-Language Models. Models like CLIP (Radford et al. 2021 and ALIGN (Jia et al. 2021 have been trained on expansive datasets such as LAION-400MA
(Schuhmann et al. 2021 2022
enabling them map images and text into shared embedding space. These models have seen wide- ranging usage, from generative tasks (Rombach et al. 2022] Ramesh et al. 2022 Gafni et al. 2022
Liu et al. 2022 Chefer et al.) 2023 Karthik et al. 2023) to open-vocabulary classification Radford
et al.)
et al.) Menon & Vondrick] [023] 1 Pratt et al. 2023] Udandarao et al.) 2023
Roth et al. 2023) and (cross-modal) retrieval (Bogolin et al. 2022, Bain et al.] 2022 Roth et al.
2022a] Wu et al. 2023). Further advancements include models like BLIP (Li et al.) 2022 2023)
and Flava (Singh et 2022), which extend beyond shared space projection to address other vision-
language tasks like captioning (Vinyals et al. 2016) and visual question answering (Antol et al.)
2015). While these models have been indirectly applied to CIR through specialized modules (Vo
et al. 2019 Baldrati et al. 2022 Delmas et al. 2022) and with fine-tuning (Gu et al. 2023). work demonstrates that vision- language models alone, when partnered with an LLM, can suffice for effective CIR without additional training.
2

---

Published as a conference paper at ICLR 2024
input image (Q)
caption
Visual encoder V I
Image captioner
Cg V c
Image Database
There teddy sitting on the sidewolk
P
Prompt (p): Following modify Co
"One teddy bear, bigger con on the sidewalk."
Text encoder L
"shows only of them, which is bigger and is trash
trosh
LLM V R
= (p o target caption
retrieved image
text modifier (t)
Compositional Image Retrieval through image and text modifier, use an off-the-shelf vision-language model caption the image. A LLM processes the generated caption and the text modifier to generate a description of the desired target image. To obtain the final image, we use -anguane model and perform text-to-image retrieval. CIReVL is modular, training-free free and human- understandable in natural language.
Figure I:
(CIReVL). Given an input
3 CIREVL METHODOLOGY
This section first details the specifics of the ZS-CIR task in $3.1 before presenting our proposed approach, Compositional Image Retrieval through Vision-by-Language (CIReVL), $3.2
3.1 PRELIMINARIES
Let us define as I and T the image and text space, respectively. For compositional image retrieval (CIR), text modifier t E T describes hypothetical semantic changes a query image Q E I, whose closest realization I' E 'D from some image database D = {I, This inherently multi-moda, task can be defined as scoring task + I T x D - IR. While in stan- dard CIR & is learned via supervised training, common zero-shot CIR (ZS-CIR) approaches sidestep this need by tuning specific modules to invert the query image into associated text. Specifically, they learn an inversion function d; I + Z mapping a given query image to a pre-defined text- token embedding space Z. Practically, de trained over intermediate image representation of specific image encoder 1 (Saito et al. 2023 Baldrati et al. 2023), often part of large-scale pre-trained Lasoon-anguuage representation system such as CLIP (Radford et al.) (02T). Template filling around the text modifier over the corresponding inverted embedding inv, then used to aggregate the information into one target caption (e.g. "a photo of {inv,} that
In should be retrieved.
di (Q)))(Q)
(I).). This target caption is then used for target image retrieval by VLMs like CLIP, encoding using the associated pre-trained text encoder V that projects the target caption and candidate images D into shared, searchable embedding space. The respective matching score is then cos sim( V (T), (rr(inv))) with cosine similarity cos sim.
While promising, such pipeline exhibits certain shorteomings, in that one i) needs to train specific inversion module Φi dependent on the chosen VLM and a separate image-caption dataset, ii) text embedding vectors can be ensured be human-understandable and cannot be verified correct description of the image, and iii) using rigid template filling does not allow for free-form textual representations and semantically flexible target captions.
3.2
COMPOSITIONAL IMAGE RETRIEVAL THROUGH
We can alleviate all the aforementioned shortcomings through Compositional Image Retrieval through Wision-by-Languall available pre-trained VLMs and L.LMs. Similar existing ZS-CIR methods (Saito al., 2023 Baldrati et al. 2023). we build on CLIP our retrieval system, however in a fashion that operates entirely independent on the particular CLIP model choice. We also assume access to traieed
(CIReVL) a simple approach that recombines existing and publicly
captioning models, s.a. readily available BLIP (Li et al.) 2022 2023) or CoCa (Yu et al. 2022
to provide a textual caption for a given image. Finally, we leverage a LLM for textual reason-
ing (Huang & Chang (023). available e.g. through Llama (Touvron et al. 2023), Vicuna (Chiang et al. (023) or the GPI-framework (Brown et al. 2020). We visualize our framework in Fig.
3

---

Published as a conference paper at ICLR 2024
From text embeddings to captions. Textual inversion has two main issues. First, it relies specifically trained by pre-defined, pre-trained encoder (e.g. CLIP). Second, predicted inversion tokens have guar- antee to be human-understandabl. Both shortcomings are created by the learned token-generation module, and can thus be tackled by replacing it with an alternative system. As traditional token inver- sion methods rely on the existence of large-scale pre-trained models, a natural alternative is to find solutions from the larger corpus of large-scale pre-trained models. Specifically, by replacing trained textual inversion functions with an available image captioning system &e c through pre-trained gen- erative VLMs such as BLIP, a simple solution for both problems can be found. Specifically, given query Q, we obtain its textual representation language text domain T, it remains possible for humans to reason over allowing the user clearer insights into the retrieval process and allowing for possible intervention, we explore in $4.3
mapping, tailored to image representations produced
c (C()) T. As Cy lives in the natural
From templates to reasoning targets. Using directly image captions Gg is not sufficient as it does not incorporate the essential context provided by the text modifier t. While simple pre-defined template may recombine these in specific, fixed ways, they have no flexibility to account for different forms of textual modifiers and caption- most suitable to a particular task at hand. To address this issue, we make use of the reasoning capabilities of existing LLMs. Rather than combining Cy and E in a fixed way using template filling, our goal is to obtain unified target caption that models the hypothetical effect of on the query image Q as a change in the thus resulting image caption Cy. This can be done in various ways, but we found simple prompts p to encode sufficient problem context already (see SA for more details), In particular, given a LLM of choice V R. we generate an cntrruction-ateeraae image caption as og = V ne(p o cg ot), which queries the LLM with a concatenation of the base prompt the generated image caption by (prepended with " "emage Content: : "), and the instruction t (prepended with " "Inst cnnstruction ion: "Σ Example queries and outputs are provided in $4.3 Note that the construction of the prompt a one-time process requiring minimal effort and annotation, which we found to translate well across all problems. In addition, as the pnrruction-conditionnn happens entirely in the language domain, a human-in-the-loop can fully reason about the impact of the instruction on the retrieval process by comparing C2 and cy w.r.t. t.
Compositional image retrieval. Given the adapted caption cz.. CIReVL encodes the image- search database DD alongside of using a VLM (e.g. CLIP). The retrieved target I is thus given
I argmax IED
(1)
where the final selected target image is the one most similar to the generated target caption. As the image retrieval system is only introduced and utilized after the combination of query image and instruction, is entirely detached and modular. Consequently, can be easily exchanged with other ones depending efficacy. Overall, this results in a ZS-CIR pipeline in which compositions are human understandable as it operates entirely in the language domain, and the retrieval process exists a clearly separated module, without requiring training of any mapping function on top of it.
practical requirements and the desired trade-off between efficiency and
4 EXPERIMENTS
We first provide the experimental details in $4.1 before showcasing the results of our CIReVL in four different ZS-CIR tasks in $4.2 Finally, we provide an in-depth analysis of our method in $4.3 highlighting it's capacity as well as the impact of the various components.
4.1 IMPLEMENTATION DETAILS
For our experiments we use PyTorch (Paszee et al. 2019), extending the public codebase of Bal- drati al. (2023), and using clusters of NVIDIA V 100 and A A100s. We experiment with different ViT-variants (Dosovitskiy et al. 2021) of CLIP, with weights taken from the official implementation in (Radford et al. 2021). We use the OpenCLIP (Tharco et al.) models for the analysis on scal-
ing laws. As captioner, we leverage the open-source BLIP-2 (Li et al. 2023) with a F5XX-
For possihle adaptation without training, in-context learning (see e.g. Dong et al. 2023 be leveraged.
4

---

Published as a conference paper at ICLR 2024
Table 1: Comparison on CIRCO and CIRR Test Data. On CIRCO, CIReVL significantly out- performs even adaptive methods across retrieval models, while it achieves competitive results on CIRR despite the noise in the benchmark. Its modularity allows for simple further scalability for additional gains. (*) ViT-G/14 uses OpenCLIP weights (Ilbarco et a..))
CIRCO CIRR
CIRCO
CIRR
Metric
mAP@
Recall
R. R.K k=2
Arch
Method
k=5
k=10
k=25
k=
k=1
k=5
k=10
k=50
k=1
k=3
Imags-only
1.34
1.60
2.12
2.41
6.89
22.99
33.68
59.23
21.04
41.04
60.31
Text-only
2.56
2.67
2.98
3.18
21.81
45.22
57.42
81.01
62.24
81.13
90.70
VIT-B/32
Image Text
2.65
3.25
4.14
4.54
35.06
48.94
77.49
32.77
74.96
PALAVRA
4.61
5.32
6.33
6.80
16.62
43.49
58.51
83.95
41.61
65.30
80.94
SEARLE
9,94
11.13
11.84
24.00
53.42
66.82
89.78
54,89
CIReVL
14.94
15.42
17.00
17.82
23,94
52.51
66.0
86.95
60.17
80.05
90.19
Pic2Word
8.72
9.51
10.64
11.29
23.90
51.70
65.30
87,80
VIF-L/14
SEARLE
11.68
12.73
14.33
15.12
24.24
52.48
66,29
88.84
53.76
75.01
88.19
CIReVL
18:57
19.01
20.89
21.80
24.55
52.31
64.92
86.34
59.54
79.88
89,69
VIT-G/14
CIReVL
26.77
27.59
29.96
31.03
34.65
64.29
75,06
91.66
67.95
84.87
93.21
Table 2: Comparison on FashionIQ Test Data. CIReVLis able to significantly outperform adap- tive methods across all Fashion-IQ bubbbbenchmaks, with its inherent modularity allowing for further simply scaling to achieve additional large gains. (*) OpenCLIP weights (Ilharco et al.)
Fashion-IQ
+
Shirt
Dress
Toptee
Average
Backbone
Method
R@10
R@
R@ 10
R@50
R@10
R@50
R@ 10
R@ 50
Image-only
6.92
14.23
4.46
12.19
6.32
13.77
5.90
13.37
Text-only
19.87
34,99
15.42
35.05
20.81
40.49
18.70
36.84
ViT-B/32
Image Text
13.44
26.25
13.83
30.88
17.08
31.67
14.78
29.60
PALAVRA
21.49
37.05
17.25
35.94
20.55
38.76
19.76
37.25
SEARLE
24.44
41.61
18.54
39.51
25.70
46.46
22.89
42.53
CIReVL
28.36
47.84
25.
46.36
31.21
53.85
28.29
49.35
Pic2Word
26.20
43.60
20.00
40.20
27.90
47.40
24.70
43.70
VIT-L/14
SEARLE
26,89
45.58
20.48
43.13
29.32
49.97
25.56
46.23
CIReVL
29.49
47.40
24.79
44.76
31.36
53.65
28.55
48.57
ViT-G/14"
CIReVL
33.71
51.42
27.07
49.53
35.80
56.14
32.19
52,36
language model (Chung et al.
2022).
Ablations consider also BLIP
(Li et al. (022) and CoCa Yuu
jtt al., 2022). As LLM we use gpt-3.5-turbo Brown et al. (2020), but we experiment also with Vi- cunal 3B (Chiang et al., 2023 Llama2-70B (Touvron et al. 2023
and GPT-4 (OpenAl 2023).
Datasets and Baselines. We use the CIRR (Liu et al. 2021). CIRCO (Baldrati al. 2023) FashionIQ-(WW et al., 202I) and GeneCIS (Vaze et al., 2023) datasets which have all been used for CIR. CIRR, the first natural image dataset for CIR, suffers from false negatives (Baldrati al., 2023). since has only a single target image annotated. The recently introduced CIRCO dataset amchioraes this by having multiple positive images for each query. The GeneCIS dataset (sourced from MS-COCO (Lin et al.] 2014) and Visual Attributes in the Wild (Pham et al./ (2021)) introduces four task variations, retrieving or changing a specific attribute or object. Due space constraints, we report results on the ImageNet Domain benchmark (Saito et al. 2023) in the appendix. Fashion-IQ benchmark that is focused around retrieval in fashion settings. Following the original bench- marks, we use Recall@k k as the metric on the CIRR, GeneCIS, and Fashion-IQ. On the CIRCO dataset, since there are multiple positives, we use the mean average precision (mAP@k). We use the 'image-only', 'text-only' and 'image+text" to denote directly performing retrieval with CLIP using only the reference image, modifying instruction, as well as averaging the embeddings for the reference image and modifying text. PALAVRA (Cohen al. 2022). Pic2Word (Saito al. 2023), SEARLE (Baldrati et al. 2023) are textual inversion methods either designed or adapted for ZS-CIR.
4.2 ZS-CIR BENCHMARK COMPARISONS
CIRCO. Our results in Tab. list the performance the hidden test set of CIRCO, accessible through the submission server provided Baldrati et al. (2023). As can be seen, using the default VIT- B/32 and ViT-L/14 CLIP variants, our approach significantly outperforms methods like Pic2Word, SEARLE and PALAVRA (Cohen et al. 2022). For instance, on ViT-L/14, we achieve a mAP@5 of 18.57% notably improving over the 11. 11.68% by the best performing trained alternative SEARLE,
5

---

Published as a conference paper at ICLR 2024
Table Comparison on GeneCIS Test Data. CIReVLis able significantly outperform adaptive methods across all Fashion-IQ uub-benchmarks, with its inherent modularity allowing for further simply scaling to achieve additional large gains. (*) OpenCLIP weights (Ihhrrco al.
GenoCISS
Altibttc
Change Altiiuttc
Facus Chhject
Clange Obyect
Ascrage
Rackhone
Image Ouly
20.3
Teat Onlyy
Imuge
12.6
12.6
Combiner (CIRRR)
15.1
27.7
39.3
12.1
22.8
31.8
13.5
25.4
28.0
39.6
16.0
Combiner (OCSM)
19.0
31.0
41.5
16.6
27.5
36.5
14.7
20.
59.7
16.3
CIRaVL
19.1
31.3
41.5
14.7
26.3
27.0
13.2
23.4
32.7
16.3
28.7
37.9
15.8
VIT DV32
CIRaVL.
SEARLE
17.9
18.9
29.4
30.6
41.3
40.4
13.0
14.8
25.8
23.8
33.7
25.8
12.2
14.0
23.0
24.3
33.3
16.1
23.8
27.8
37.6
14.4
15.9
VTT L/14
CIRaVL
SEARLE
19.5
17.1
31.8
29.6
40.7
42.0
14.4
163
26.0
25.3
34.
25.2
12.3
12.0
22.3
21.8
30.5
17.2
28.9
37.6
15.9
14.4
VIT43114
CIRaVL
20.5
34.00
44.5
16.1
28.6
294
14.7
25.2
33.0
18.1
31.2
41.0
17.4
Viexpoinn no. Color
Count, Zoom
Count, Dontent
Dount, Backgr.
Content
Vieep., Content
"Io abot fron front the
*as only che wearing
"Has two of then instead
"Rae baxes and geass"
"Has chopped casrota instead
"Is taken fron intorior of
grayanale"
Figure 2: Examples from the CIRCO validation set where our method retrieves the desired image. We see that our method is able to perform this task for a wide variety modifier texts.
and more than doubling the performance of Pic2Word, which achieves 8.72%. These are strongly indicative results, as CIRCO constitutes the dataset with the cleanest annotations and unlike other datasets in this field the inclusion of multiple positives for an inherently ambiguous problem, as textual modifications provide key evidence of the efficacy of our freen, vision-by-language approach for ZS-CIR.
an image do not only have a single solution. These strong results thus
For the hidden CIRR test set (accessible using a test server, see e.g. Liu al. (021 we provide results in Tab. [ As previously noted, this dataset is very noisy, with results primarily dependent on the modifying instruction, and the actual reference image having much less relation to the target image (Saito et al. 2023 Baldrati et al. 2023). Still, even this benchmark, we are able to match the performance of prior ZS-CIR methods (e.g. R@l 24.55% for our method versus 24.24% for SEARLE).. without requiring any form of problem-specific training. The CIRR benchmark also provides another evaluation where the correct image has to be retrieved from curated samples. In this evaluation, our results surpass prior work by significant margin (R, @1 60.17% versus 54.89% for SEARLE), underscoring the versatility of our method.
CIRR.
Fashion-IQ. We provide the results on the validation set the Fashion-IQ benchmark Tab. We see that CIReVL is able to outperform prior zero-shot methods by significant margins (average RÔ 10 of 28.55% versus 25.56% for SEARLE). Additionally, we also see the benefits scaling the model size, with the average R@10 improving 19%. This also underscores how approach is able to adapt to diverse domains with minimal changes.
GeneCIS. The versatility of CIReVL is further underlined when transferring it to the GeneCIS benchmark. Unlike CIRCO and CIRR, modifiers consists of a single word that has different interpretation for each task, e.g. focusing/changing particular attribute object. In this case, without re- re-training any method, simple prompt modifications allow for convincing performance. In particular, for the "focus" -tasks, instruction. For "change" ttakks, we simply ask it to replace the corresponding object in the caption. As shown in Tab. CTReVI, nearly matches the performance of Combiner trained large filtered
simply task the LLM to retain the attribute/object listed in the

---

Published as a conference paper at ICLR 2024
Seceence Retriaved
Sample Target
Retriaved
Sample Target
Wxong L.M Mods can
Description
Description
"A person on Poofoorrd riding eove in the ocean" Instraction
*A sroup of people on skio on sapwy nountain."
"Is shirtiezt, the photo has dackez background and is abot Econ closer distancs."
Instriction
"Ace exting and the shoto shows Eever people."
Medifled
"A shertless person riding eave on puc in the
Modified
ocean, with darter background and closer shat."
"Some people are skisng on anovy nombain,"
Sample Target
Target
1500 Secooon Bu
THU get
Pean Mkong
bonnh 'bildding."
"Ioe
unbrella
Modified
Modified
*A building
with
"250 holding tecnsparent wibeella
setting."
Figure 3: We analyze failure cases of our approach. Due to the interpretable nature, we can easily attribute enrors captioning, reasoning or the text-image retrieval. We also find examples where the model is penalized despite a plausible retrieval due to insufficient annotations in the dataset.
Reference
Instiml
Redecence
Instial
Iatervened
TANE PISC
Instructions:
"has the wonen and the ran with the roles extt wwitched..
Instructions: "io shot with the saxe style and nost of the bescher enpry."
Initial Caption: "a zon adjasting a soran's tie.*
Initial Caption: "pigeons efectig on a beach front of bet tdt
Initial Moditicat:on: "a soman adyusting a man's tieu
initial Medificationn to building with ziquesa iordy with only for banches occapied."
Intervened Caption: "a soman adyusting mant tie.
Intervened Caption: "a grayeeate phats or people aiching on pencher in park in trent at butidings."
Intervened Modification: "a man acquacing vonsnta cim."
Intervened Modiescatico: "A black and vaste phoso et expty penchea in pack with buildings in the backepound."
Figure 4:
We demonstrate the possibility of user interventions method. For instance, by fixing the mistakes in the generated caption, we are able to correctly retrieve the desired image without having to make any other changes.
enhance the performance of
version of the CC3M dataset (Sharma et al. 2018) (16.8% Average R@1 versus 15.9%), and surpass the Combiner model finetuned on the CIRR dataset well other baselines. This is notable given the diversity, but also specificity of the tasks, ranging from settings resembling standard image retrieval (e.g. "focus attribute"") to traditional CIR tasks instead (e.g. "change object").
4.3 ABLATION STUDY AND PERFORMANCE ANALYSIS
In this section, we conduct large number of analyses to provide better understanding of our pro- posed method through ablations and qualitative examples, and provide insights into performance bottlenecks. This also includes scalability studies, and insights into the benefits of human under- standability via natural language operations, further enabling the possibility to perform interventions on the retrieval process in order to tackle failure cases.
On the importance of textual reasoning. Since our proposed method heavily relies on the LLM to convert textual inputs to target captions, we study several LLMs to understand the effect they play on the final performance. The results on the CIRCO validation set in Table 4 illustrate that the reasoning is critical to the overall performance. Most notably, we perform a sanity-check where the generated caption is used to fill a template as done prior works (Saito et al. 2023 Baldrati et al.) 2023). This method (' ("Captioning') aims to test the necessity of using LLM for textual reasoning on this task. We see that this method is outperformed by all the LLMs that we tested (mAP@@ of 9.22%). This highlights the necessity of performing textual reasoning to generate the final target caption, and the limits of static templates. For instance, both GPT-3.5-turbo and GPT-4 achieve strong results (mAP@5 of 13.33% and 15.63% respectively), with the improved capabilities of GPT-

---

Published as a conference paper at ICLR 2024
Table 4: Analysis of the impact of LLM and captioner choice. Comparison between different LLMs on CIRCO validation reveals positive relation between performance and reasoning capaci- ties. For captioning models, most recent off-the- ffe-hhe-sheff models achieve strong results.
Arch
Captioner
LLM
mAP@10 mAP@5
mAP@25
mAP@50
9.22
10,05
11.44
12.08
LLama2-70B
10.22
10,60
11.80
12.39
VIT-B/32
BLIP-2
Vicuna- 13B
12.65
13.17
14.48
15.20
GPT-3.5- Turbo
13.33
14.16
15.74
16.35
GPT4
15.63
16.31
17.02
18.12
BLIP
13.1
13.33
15.08
15.92
VTT-B/32
BLIP-2
GPT-3.5-Turoo
13.33
14.16
15.74
16.35
CoCa
13.64
13.37
15.10
15,90
4 offering significant performance gains top. When moving to publicly accessible LLMs such as Llama2-70B and Vicuna-13B (10.22% and 12.65%). we find a fair drop in performance. For cases where access closed-source APIs such GPT-3.5-turb GPT-4 is restricted, alternative usage of public LLMs can thus still offer significant benefits, particularly when evaluating the differences between Vicuna-13B 13B and GPT-3.5-turbo.
On the importance of captioning quality. As our CIR framework builds on the generated reference image cap- tion, we also ablate the importance of the correct caption- ing model choice in Tab. 4 which reports results the CIRCO validation split for faster turnaround. Interest- ingly, find that most state-of-the-art public captioning models perform similarly well, with minor differences be- tween BLIP versions, and slightly improved performance of CoCa. To retain the generality of our approach, we stick with BLIP-2 with Flan-TSs language model since the usage of LLM decoder makes it a more generic tool usable across a variety of domains.
TIFA Alignment Studies
Versus
Versus
Target Image
LLM Caption
1.00
0.75
TIRA Score
0.50
0.25
0.00
LLM
Ref. Cuption.
Target Image
Retr.
Deser.
Image
Reasoning Sanity Check by Measuring Text-Image Alignment. We conduct quantitative evaluation of our reasoning component by assessing the text-image align- ment in the LLM-modified descriptions using TIFA (Hu et al. 2023). Unlike simpler metrics such as CLIP score, TIFA offers more accurate measure of cross- modal alignment by converting it into series of question- answering tasks. Results are presented in Fig- 5 Com- paring the alignment between our generated modified de- scriptions and the generated plain image captions, we clearly see much higher and more consistent average alignment through our modification. In ad- dition, when looking at the alignment the actually retrieved image by CLIP (ViT-B/32) in Fig. see that the alignment of the retrieved image is actually notably lower than that of the ground truth image. Thus, while the CLIP similarity higher for the retrieved image, its actual alignment with the modified caption is lower. This means that the standard CL.IP backbone, even the modified caption actually aligns well with the ground truth image. can often fail retrieve it. This indicates that the CLIP retrieval is a severe bottle- neck (Thrush et al. 2022 Hsich et al.) 2023 Kamath et al. 2023), as the captioning and reasoning together generate valid target captions that sub-optimal retrieval system cannot match.
Figure 5: We use TIFA Hu et al. 2023 to compare alignment between LLM de- scr. & base caption and target image, and LLM descr. to target- and CLIP- retrieved image. It shows the impact of L.MM reasoning over base captions, and CLIP retrieval essential bottleneck.
our modified caption between the ground truth image and
Investigating Scaling Laws for ZS-CIR. A critical benefit of the modular nature of our method is that we can replace and scale each component, without re-training. In particular, we can simply uti- lize our base model used for our benchmark comparisons in $4.2 and utilize different CLIP model for the final retrieval process. This allows us to investigate scaling laws as studied and investigated in e.g. KKallan al. (2020); (aballloo et al. (2023); Cherti et al. (2023) for the particular problem of CIR, and investigate if scale can address the retrieval bottleneck. For this, we leverage CLIP variants provided through the OpenCLIP project (Ilharco al.), with models ranging from around 150M to-
8

---

Published as a conference paper at ICLR 2024
CIRCO
CIRR
Subset-CIRR
28
35
26
34
68
5
@
24
22
33
67
20
32
66
E 18
CLIP Variants
31
64
Log-linear fit
30
16
64
10a
10a
10a
10
10
10
Parameters
Parameters
Parameters
Figure 6: Leveraging our modular CIReVL. to study scaling laws. For model availability and con- sistency, we use LAION-2B pretrained CLIP models provided through OpenCIIP (Ilhrrco et al.).
tal parameters to around 2.5B. On both CIRCO and CIRR, we see clear log-linear relationship between the model capacity and the performance as a result of simply upgrading our retrieval model in size. This clearly highlights the impact of scale even to the complex setting of ZS-CIR, and allows us to partly break the retrieval bottleneck shown above. As we can easily plug-and-play different retrieval models, it also allows us to scale our overall pipeline, allowing us to boost the overall CIR scores on the CIRR dataset up to R@l 34.64% (c.f. 24.55% for the highest reported SEARLE score), and up to mAP = 26.77% on CIRCO (c.f. 11.68% for the highest reported SEARLE score). These results provide clear evidence of our method allowing for a simple transition between efficiency (with a smaller retrieval model) and efficacy with a larger one when needed. We also find that our insights partly contrast those in Vaze et al. (202)), where only minimal gains from scaling can be observed. Our results show that by incorporating conditioning through textual reasoning, CTReVL. better leverages the benefits of large retrieval models without expensive re-training.
Qualitative Examples. Fig. 2 visualizes successful Cl-retrievals with instructions impacting dif- ferent semantic elements of the reference image such as viewpoint, color, object counts, background changes, object insertion, object adaptation or picture manipulations such as zoom. This provides further indication about the diverse applicability of our setup. Following that, Fig. 3 visualizes ex- emplary failure cases. As our CIR process operates primarily in the language domain, it becomes easy to understand the particular failure cases. For instance, we see some cases where the generated Caption either does not correctly describe the image, or does not focus on the relevant aspect in the image (e.g. ignoring the grayscale aspect of the image and focusing on the pigeons). We also observe cases where the LLM does not generate an accurate description of the target image. Additionally, there are also cases where despite the captioning and reasoning working correctly, the CLIP model does not retrieve an accurate image. Finally, we also see ambiguous failure cases where the retrieved target is debatable correct, but not labeled as such due to incomplete annotations (Ambiguity).
Performing User Interventions. Fortunately, by being able break down the retrieval process, we can easily perform user interventions when needed. Simple example scenarios are shown in Fig. where the default modified caption results in incorrect retrievals. As a user, intervention can happen at different stages of the retrieval process, modifying e.g. generated captions, utilized instructions or the final LLL--generatdd caption. As a proof of concept, we showease how inter- vention on the base caption level can already result in correct search queries on the image database. Simple human intervention can easily determine a generated caption incorrect, and simply re- place it with alternative. The rest of the pipeline can simply continue operating on top of this intervention. As can be seen in Fig. 4 intervening on the caption-level can already rectify various end-to-end pipeline
errors. This once again sharply contrasts with existing work that relies with a rigid query template, where there is no possibility to alter the retrieval process.
5 CONCLUSION
In this work, we present novel, training-free approach for Zero-Shot Compositional Image Re- trieval (CIR). Utilizing off-the-shelf pre-trained models, our method not only achieves strong perfor- mance across multiple CIR benchmarks but also, in some cases, doubles the performance of existing state-of-thhr the methods. The method's 's inherent interpretability allows for user intervention, adding an extra layer of flexibility. We also explore the impact of scaling laws our method, revealing that scaling the text-image retrieval component can substantially boost task performance. Collectively, these contributions set the stage for future research in training-free Compositional Image Retrieval.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by DFG project number 276693517, by BMBF FKZ: 1ISS18 by the ERC (853489- DEXIM), by EXC number 2064/1 - project number 390727645, and by the MUR PNRR project FAIR Future Al Research (PE00000013) funded by the NextGenerationEU. Shyamgopal Karthik and Karsten Roth thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support. Karsten Roth would also like to thank the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. The authors also thank Vishaal Udandarao for valubale feedback and Thomas Hummel for help with visuals.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Muhammad Umer Anwaar, Egor Labintcev, and Martin Kleinsteuber. Compositional learning of image-text query for image retrieval. In WACV, 2021.
Yang bai, Xinxing Xu, Yong Liu, Salman Khan, Fahad Khan, Wangmeng Zuo, Rick Siow Mong Goh, and Chun-Mei Feng. Sentence-level prompts benefit composed image retrieval. In ICLR, 2024.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. A clip-hitchhiker's guide long video retrieval. arXiv preprint arXiv:2205.088: 0,8000 2022.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In CVPR Workshops, 2022.
Alberto Baldrati, Lorenzo Agnolucei, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In ICCV, 2023.
William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh. Towards language models that can see: Computer vision through the lens of natural language. arXiv preprint arXiv:2306. 2tt.::336...44100, 2023.
Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation. In CVPR, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In ICLR, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023.
Yanbei Chen and Loris Bazzani. Learning joint visual semantic matching embeddings for language- guided retrieval. In ECCV, 2020.
Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learning. CVPR, 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.
10

---

Published as a conference paper at ICLR 2024
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL https://
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi language mod-
Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling els. arXiv preprint WxXv:2210.114166 2022.
Niv Cohen, Rinon Gal, Eli A. Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-language representations. In ECCV, 2022.
Ginger Delmas, Rafael S. Rezende, Gabriela Csurka, and Diane Larlus. ARTEMIS: Attention-based
retrieval with text-explicit matching and implicit similarity. In ICLR, 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. ECCV, 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun. Compodiff: Versatile composed image retrieval with latent diffusion. arXiv preprint arXiv:2303.11916, 2023.
Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Automatic spatially-aware fashion concept discovery. In ICCV, 2017.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of--istribution generalization. In ICCV, 2021.
Cheng-Yu Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. In NeurlPS, 2023.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In ICCV, 2023.
Jie Huang and Kevin Chen-Chuar Chang. Towards reasoning in large language models: A survey. In ACL Findings, 2023.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. URL https: /ci. org/10.gg/0000 5281 zenodo.
5143773
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021.
Amita Kamath, Jack Hessel, and Kai- Wei Chang. Text encoders are performance bottlenecks in contrastive vision-language models. In EMNLP. 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2rX.v:2001.08361, 2020.

---

Published as a conference paper at ICLR 2024
Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata. Kg-sp: Knowledge guided simple primitives for open world compositional zero-shot learning. In CVPR, 2022.
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. If at first you don't succeed, try, try again: Faithful diffusion-based texl-to-image generation by selection. arXiv preprint arXiv:2305. 2023.
Seungmin Lee, Dongwan Kim, and Bohyung Han. Cosmo: Content-style modulation for image retrieval with text feedback. In CVPR, 2021.
Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Chatting makes perfect-chat chat-based image retrieval. arXiv preprint arXiv:2305.20065 2023a.
Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Data roaming and early fusion for composed image retrieval. arXiv preprint arXiv:2303.09422 2023b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision-language understanding and generation. In ICML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022.
Yikun Liu, Jiangchao Yao, Ya Zhang, Yanfeng Wang, and Weidi Xie. Zero-shot composed text- image retrieval. In BMVC, 2023.
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-langun lan-uuue models. In ICCV, 2021.
Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In CVPR, 2021.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In CVPR, 2017.
OpenAl. GPT-4 Technical Report. arXiv, abs/2303.08774, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, reg--performanc deep learning library. In NeurIPS. 2019.
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivas- tava. Learning to predict visual attributes in the wild. In CVPR, 2021.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.00 06125, 2022.
12

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Integrating language guidance into vision-based deep metric learning. In CVPR, 2022a.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. In CVPR, 2022b.
Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In ICCV, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. I/CV, 2015.
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In CVPR, 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.022 14, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objec- tive. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.). Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL htttps://prceeddd hww.dddi.nnnnnnennsnoere.... ctteeurpsscccppppeeeeeeee ww/paper.p......... lorr_fflesss wiless/appe///016// File/ wttp://00aaabbeeeaaaaaa2aaaaaaae Paee-aaappppp...
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In CVPR, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wrXiv:2307.09288 2023.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training- free name-only transfer of vasion-language models. In ICCV, 2023.
Sagar Vaze, Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional image similarity. In CVPR, 2023.
Lucas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. Covr: Learning composed video retrieval from web video captions. In AAAI, 2024.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE T-PAMI, 2016.
Nam Vo. Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In CVPR, 2019.
13

---

Published as a conference paper at ICLR 2024
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback. CVPR. 2021.
Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In CVPR, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, al. Socratic models: Com- posing zero-shot multimodal reasoning with language. In ICLR, 2023.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint (xXx:233330669/ 2023.
14

---

Published as a conference paper at ICLR 2024
A ADDITIONAL IMPLEMENTATION DETAILS
The prompt that we use draws upon the one used by Liu al. 2023) for curating a dataset curation. Here, we write a similar prompt for the LLM generate the edited description. We list the full prompt below:
"I have an image. Given an instruction to edit the image, carefully generate a description of the edited image. will put my image content beginning with 'Image Content:' The instruction provide will begin with ' "Intrruction'' The edited description you generate should begin with 'Edited Description:'. Each time generate one instruction and one edited description only."
Table 5: Comparison with supervised baselines on CIRR and FashionlQ validation sets. Combiner- FIQ and Combiner-CIRR denote the models from Baldrati al. (2022 trained FashionIQ and CIRR, respectively. We see that zero-shot methods generalize better across multiple datasets.
CIRR
Fashion
Method
R@1
R@5
R@10
R@50
R@10
R@50
Combiner- FIQ Baldrati et (2022
19.88
48.05
61.11
85.51
32.96
54.55
Combiner-CIRA Baldrati et al. (002
32.24
65.46
78.21
95.19
20.91
40.40
CIReVL
29.76
59.69
71.39
90.34
28.29
49.35
B COMPARISON WITH SUPERVISED BASELINES
We provide comparison to supervised CIR methods in Tab. 5 In particular, compare our results against the Combiner architecture (Baldrati et al. 2022). We train the Combiner model on both the Fashion-IQ and CIRR datasets. We see that while the Combiner model achieves strong results when it has been fine-tuned on a particular dataset, it performs poorly on the other benchmarks. In contrast, reoo methods are able achieve strong performance across benchmarks, highlighting the generalization capabilities of these models.
C EXPERIMENTS ON IMAGENET DOMAIN CONVERSION
We also test CIReVL on the ImageNet domain conversion experiment proposed in (Saito et al., 2023). Here we use images from 200 classes of the original ImageNet dataset (Russakovsky al,, 2015) query. and for retrieval images of the same object but in the specified domain from ImageNet-R (Hendryeks et al. 202T) Unlike the previous benchmarks, the task is to simply re- trieve image of the approprite domain for the same semantic object category (i.e cartoon of a goldfish, with a natural goldfish reference image and the modifier "cartoon This requires reasoning over image semanties, the modifier affects independent domain change, with significant improvements over Pic2Word or Combiner already be achieved by leveraging the final description "a domain of capt caption" as in Tab. 6 Our model in parts more than dou- ble the performance of Pic2Word on this task (e.g. R@1 = 19.2% versus 8.0% for conversion to the cartoon domain). These findings mainly reiterate that combining off- the-shelf pre- pre-trained mod- els here CLIP and BLIP-2 can be more effective than customized models operating on top of pre-trained models.
15

---

Published as a conference paper at ICLR 2024
Table 6: Evaluation ImageNet Domain Conversion experiment proposed in Saito al.
2023). The goal is to retrieve an appropriate domain of the object specified in the query image.
Cartoon
Tuy
Origani
Sculpture
R@50
R@50
R@50
R@10
Image-only
0.3
4,5
0.2
1.8
0.6
5,7
0.3
4,0
Text-only
1.1
3.7
ImagetText
2,2
13.3
2.0
10.3
1.2
9.7
1.6
11.6
Combiner (CIRR) (Baldrati et al 2022
6.1
14.8
10.5
21.3
7.0
17.7
8.5
20.4
Pic2Ward Saito et ala 2013
21.9
13.5
25.6
8.7
21.6
10.0
23.8
CIReVL
19.2
42.8
30.2
41.3
22.2
43.1
234
45.0
16


Reviews:
[{"review_id": "xHnx7C2jfO", "reviewer": "Reviewer_W9gL", "summary": "The paper proposes a training-free solution for addressing the compositional image retrieval problem. The idea is quite simple: i) take the pre-trained VLM model to describe what the input image contains, ii) use an LLM to combine the description/caption of the input image with the desired modifications as described by the user query, finally, iii) use the resulting text to generate VLM (i.e., CLIP) representations and search the target image database via knn search. The solution is evaluated against multiple baselines (PALAVRA, Pic2Word, SEARLE) and datasets (CIRCO, CIRR, GeneCIS) and provides better results.", "questions": "- A short discussion about extensibility / applicability of the proposed approach could be beneficial for the research community. \n- FashionIQ is yet another dataset which other papers use for CIR evaluations, it might be worth considering it in the future. \n- Is there a way to detect when the LLM fails to provide a meaningful / valid re-composed text ? How much room left to improve this step? What is the future research direction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is technically sound and widely applicable not only for text-image use case, but also text-video and probably other problem domains. \n- The paper investigates three different datasets against multiple baselines and various additional ablations make the paper and results more appealing. Limitations of the proposed method also gets discussed with examples. \n- The paper is easy to follow / well-written.", "weaknesses": "- [Novelty & Literature review] There are a wider number of papers in the field since last year and some of those papers requires mention (at least the ones before the ICLR submission deadline). The most similar paper ((https://arxiv.org/pdf/2310.05473.pdf) proposes the same approach but with an additional training method for merging the automatically generated image caption and the desired modification's text description. This paper contains additional papers/baselines to cite/include. It would be great if there is a way to understand whether the proposed training-free method would perform better or worse compared to the paper mentioned above.\n\n- [Experiments] Baseline PALAVRA attack different set of problems, uses different datasets and metrics. It does not mention CIR task in the text. Not clear if PALAVRA serves as a baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kKiMeQX6wj", "reviewer": "Reviewer_QaMw", "summary": "This paper aims to achieve training-free zero-shot compositional image retrieval by using off-the-shelf models pretrained on large-scale datasets. The authors use language as an abstraction layer for reasoning about visual content. Particularly, it uses off-the-shelf vision-language models like BLIP-2 or CoCa to generate a detailed description of the query image. Next, an LLM or GPT-like model combines the input image description (from BLIP-2) and an input textual query (e.g., \"shows only one of them, which is bigger and is next to a trash can\") to generate a caption for the desired target image. Finally, a vision language model like CLIP performs text-to-image retrieval using the generated caption from LLM and images from an image database. This modular approach ushers a few benefits like: (i) the resulting approach is training-free and being modular, we can flexibly plug-and-play individual components like replace the LLM with either GPT or Llama, replace the VLM with BLIP-2 or CoCa; (ii) the composition happens in the human interpretable language domain offering human interpretability; and (iii) it is possible for humans to intervene on the retrieval process to fix or post-hoc improvements in retrieval results.", "questions": "I am confused, how is the proposed method different from VisProg (https://arxiv.org/pdf/2211.11559.pdf)?\nIt feels I can get the same benefits of the proposed method by using the VisProg paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "While there are a lot of works on compositional image retrieval, previous works typically require training several components like textual inversion and lack human interpretability. The method proposed in this paper avoids all these problems by simply composing image and textual query in the language domain.\n\nOn the surface this is ingenious because you do it in a modular way and each module is a highly-generalisable large-scale pre-trained model. For example, when training a textual inversion, we do not really guarantee it scales to open-set setups (given we train them in limited data and compute). We just use a small-scale network and hope the rest of powerful models take care of it. The proposed method works zero-shot and you know it works for open-set setups.\n\nAdditionally, unlike prior trainable compositional image retrievals, there is a lot of effort that goes into interpretability -- \"how was the image and query text was composed\". Since the proposed method does this composition entirely in the language space, you know exactly what information was extracted from the image and you can see the generated caption from LLM for the desired target image. This not only makes the retrieval process highly transparent, but also allows post-hoc edits -- you can literally make changes to adjust your retrieval results.\n\nI encourage more works that reuses as much as large-scale models and combines them as modules with minimal or training-free way.", "weaknesses": "Despite its appeal, there are a few important drawbacks. This entire process sticks on the underlying assumption that our image captioning module (e.g., BLIP-2 or CoCa) can provide a \"detailed caption\" that captures all information.\n\nThis, in my opinion, is a strong assumption. A lot depends on your captioning module. While the captioning module may be super accurate, it can miss some \"less important\" details that I want to change. For example, given a photo, the sky was orange and my textual query is \"make the colour of the sky darker\". If the image captioning module omits the colour of sky (i.e., orange) and focuses on the foreground (e.g., a person holding a flower, sitting in a bench near a park where kids are playing) -- the rest of the modules have no way of combining my textual query \"make the colour of sky darker\".\n\nThis is an example, where the entire pipeline fails -- due to no fault of the image captioning module. It provided a detailed image description, but it is not possible to describe every little \"unimportant things\".\n\nLooking at the same problem from a different direction, is the information bandwidth of an image the same as language? (an image can be worth a thousand words)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3DhoH5bBg", "reviewer": "Reviewer_JtMi", "summary": "This paper introduces CIReVL, a training-free pipeline for zero-shot compositional image retrieval by combining existing off-the-shelf foundation models, e.g., BLIP-2 for image captioning, GPT for text editing and CLIP for image retrieval. In addition, the specific and explicit description with captions, instead of text embeddings, facilitates human understanding over the retrieval process. Extensive experiments are conducted to validate the proposed method.", "questions": "1.\tThis paper seems to be a technical report with the application of current state-of-the-art foundation models. Although directly using existing models is convenient, it inevitably introduces errors in each processing step. How to address the cumulative errors resulting from these multiple models?\n2.\tCould the authors provide experimental results of CIR works with a supervised training pattern? Based on the current results in this paper, the overall zero-shot retrieval performance appears to be poor. Do those supervised methods also yield unsatisfactory performance? If so, it seems that more significant efforts are needed to address the fundamental issues in CIR task; If not, does the zero-shot approach still have its value?\n3.\tHow you get the 6 curated samples for the Rs @K metric in Table 1?\n4.\tFor reasoning, the experiments show LLM models obviously affect the performance. I wonder whether the same is true for the prompt template. Except for the prompt mentioned in Appendix A, were there any alternative prompts explored during the experimentation?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper effectively integrates the VLM and LLM for ZS-CIR, offering a flexible and intervenable CIR system.\n2.\tThe authors have conducted thorough experiments to validate the effectiveness of the proposed method, making the evaluation rather reliable.\n3.\tThe figures in this paper are clear, effectively conveying the processing pipeline.", "weaknesses": "1.\tLimited contribution: The proposed method appears to be a combination of several foundation models and simply employs the basic modelling capacity of these models (e.g., BLIP-2 for image captioning, GPT for text editing and CLIP for image retrieval), presenting a naïve and straightforward solution for CIR task. As a result, it is challenging to identify insightful and significant contributions to this field. The authors should conduct further in-depth research to enhance their contributions.\n2.\tIncorrect experimental results: The Recall@10 of CIReVL (ViT-G/14∗) on CIRR dataset in Table 1 is obviously incorrect. \n3.\tInconsistent formatting of table data: In some cases, the data is presented with two decimal places, while in others, only one decimal place is used. Furthermore, there is a case with a mix of formatting in Table 3. The authors may revise the paper more carefully.\n4.\tThe paper effectively combines the existing powerful VLM and LLM models. However, it would be better to provide a more insightful analysis of the caption and reason processes. For example, from Table 3, compared to utilizing different LLM models, the use of various state-of-the-art (SOTA) captioning models has a relatively minor impact on retrieval. Are there any potential explanations for this phenomenon? Does this mean the choice of caption model is not strict, as long as the model can catch the main object or attribute of images? I think it would be better to provide more analysis. \n5.\tThe description about other works in Section 3.1 is a bit obscure for me who is not so familiar with the task but proficient other related tasks.\n6.\tMissing references, such as in ‘Similar to existing ZS-CIR methods’ in the first paragraph of Section 3.2.\n7.\tExperiments: (a) In Table 1, the author only presents the experimental results of ‘image only’ and ‘image+text’ methods for reference while missing that of the ‘text only’ method under the ViT-B/32 setting. However, the authors mention that the results on CIRR benchmark are primarily dependent on the modifying instruction while the actual reference image has much less relation to the target image, which indicates that the ‘text-only’ method can be an important reference for measuring the performance of the proposed method. Better to present the result of ‘text-only’ method and make a fair comparison and discussion. (b) In Table 1, the authors miss the results of ‘image only’, ‘text only’ and ‘image+text’ method for ViT-L/14 and ViT-G/14 settings. Better to include them for reference. (c). When evaluating on the GeneCIS benchmark, the authors do not specify the architecture of the vision backbone adopted in the experiment. Better to specify it clearly for reference.\n8.\tDiscussions and evaluation on the potential limitations: The paper shows that the proposed method has several merits including free of training, good flexibility and scalability. However, it may also have some potential limitations. For example, (a) Inference costs and efficiency: The proposed method utilizes large VLMs and LLMs to conduct the image captioning, language reasoning and cross-modal retrieval during inference. Will it take more computational costs and have longer inference time compared with the previous methods? (b) Limitations of each module: Since the proposed method is composed of three different modules, the effect of each module plays an important role on the final retrieval results. For example, if the image caption module generates partial or false descriptions for the given images, the reasoning and retrieval process will be misled. Thus, it is necessary to analyze the potential negative impact brought by each module in an in-depth manner and quantify them if possible (which factor contributes more to the failure cases). (c) Compatibility of different modules: Since the proposed method conducts cross-modal retrieval by cascading three separate modules, the compatibility of different modules seems to be an important factor. For example, if the captions generated by LLMs have different styles with pretraining data of the VLMs, the VLMs used for cross-modal retrieval may produce some bias, which may hinder the final performance. Overall, it will be appreciated if the authors can present more in-depth discussions and evaluation on the potential limitations to fully demonstrate the properties of the proposed method.\n9.\tWriting: Some parts of the paper don’t flow well. The overall writing requires further improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jh3IV7b9O", "reviewer": "Reviewer_Tg9t", "summary": "This paper proposes a training-free framework for zero-shot image retrieval, by leveraging the pretrained vision-language model and large language model. VLM is used to obtain image caption for query image, and the caption together with the text modifier is fed into LLM to get the target caption; then the target caption is used to do cross-modal retrieval with the help of VLM in database images. This framework also allows scalability and human intervention to improve the retrieval performance due to its modularity and human-understandability.", "questions": "The same as weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The proposed framework is training-free and achieves promising retrieval results.\n2. The retrieval accuracy could be improved by adopting better off-the-shelf VLM and LLM, due to the modularity of the framework.\n3. Since the query is mainly processed in language domain, the framework is human-interpretable and could also improve the retrieval accuracy by involving human intervention.", "weaknesses": "1.\tThe method itself is very intuitive and not quite novel, which is more like an engineering extension of existing models like VLM and LLM. Additionally, the proposed method does not achieve better results on all evaluation datasets. In Table 1, CIReVL underperforms SEARLE on CIRR dataset in terms of CIRR, which shows the unstability. \n2.\tThis work only adopts datasets of everyday life and natural scenes (CIRR, CIRCO, GeneCIS) as evaluation. It is necessary to include datasets of various domains, such as fashion domain datasets like fashioniq and fashion200k to evaluate the generalization ability. Furthermore, for GeneCIS dataset, the comparison against other ZSCIR methods such as Pic2word and SEARLE should be included for complete comparison. \n3.\tThe proposed framework is not very time-efficient since for each query, there will be two auto-regressive processes: query image captioning and target caption generation, which are very time-consuming. Both processes involve large models like VLM and LLM, which may further lead to low efficiency for image search. Therefore, it is necessary to compare the retrieval efficiency (time) with the previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)