Semantic Editing Increment Benefits Zero-Shot Composed Image Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 06:41:19

Abstract: Zero-Shot Composed Image Retrieval (ZS-CIR) has attracted more attention in recent years, focusing on retrieving a specific image based on a query composed of a reference image and a relative text without training samples. Specifically, the relative text describes the differences between the two images. Prevailing ZS-CIR methods employ image-to-text (I2T) models to convert the query image into a single caption, which is further merged with the relative text by text-fusion approaches to form a composed text for retrieval. However, these methods neglect the fact that ZS-CIR entails considering not only the final similarity between the composed text and retrieved images but also the semantic increment during the compositional editing process. To address this limitation, this paper proposes a training-free method called Semantic Editing Increment for ZS-CIR (SEIZE) to retrieve the target image based on the query image and text without training. Firstly, we employ a pre-trained captioning model to generate diverse captions for the reference image and prompt Large Language Models (LLMs) to perform breadth compositional reasoning based on these captions and relative text, thereby covering the potential semantics of the target image. Then, we design a semantic editing search to incorporate the semantic editing increment contributed by the relative text into the retrieval process. Concretely, we comprehensively consider relative semantic increment and absolute similarity as the final retrieval score, which is subsequently utilized to retrieve the target image in the CLIP feature space. Extensive experiments on three public datasets demonstrate that our proposed SEIZE achieves the new state-of-the-art performance. The code is publicly available at https://anonymous.4open.science/r/SEIZE-11BC.

Main paper: 
 introduction: 
 Composed Image Retrieval (CIR) [6, 49, 51] expands upon traditional image retrieval tasks by incorporating natural language descriptions into the image retrieval process. It enables users to search for images based on visual attributes and specify particular alterations to the query image using textual descriptions. The task involves a complex blend of visual content and text modifications to retrieve a novel image that accurately reflects the query's specifics. Previous studies necessitate carefully curated triplets of a reference image, relative text, and a target image, used as training data for a specialized CIR model. However, annotating these triplets is both labor-intensive and challenging. To address this, the recent Zero-Shot Composed Image Retrieval (ZS-CIR) task [4, 41, 43] emphasizes the generalization of CIR models, without requiring annotated triplets, while still focusing on retrieval accuracy.To carry out ZS-CIR, most works leverage the cross-modal alignment capabilities of large-scale pre-trained Vision-Language Models (VLMs) (e.g., CLIP [40] ) to extract aligned features from images and text for matching. Several textual inversion methods [4, 15, 20, 41] have focused on mapping images into pseudo text tokens using image-caption pairs. Then, a static template merges the tokens and the relative text to form target captions. Nonetheless, the reliance of these methods on training images could limit their realworld adaptability and performance. Moreover, the pseudo tokens are often laden with an abundance of visual features, leading to an excessively detailed caption, filled with content, texture, and style specific to the reference image. As illustrated in Figure 1 , the target image does not necessarily align strictly with the reference image, and this overemphasis on visual consistency negatively affects retrieval performance. Unlike textual inversion techniques, recent research has ventured into using Large Language Models (LLMs) for combination. Karthik et al. [27] introduce a simple method involving using a pre-trained image captioning model to caption the reference image. This caption is then recomposed based on the relative text by an LLM to enhance the subsequent retrieval process. In conclusion, the existing methods can be divided into three steps: First, I2T models convert the reference image into a single content text. Second, text-fusion approaches combine the content text with the relative text. Third, the composed caption is projected into the CLIP feature space for target image retrieval.Previous methods presume that the target image is the one most similar to the composed text in CLIP space, relying heavily on the robustness of I2T models and text-fusion approaches. However, this assumption might be flawed in practical applications due to the inherent limitations of these techniques. Given that: (1) The information provided by the reference image often contains noise, which could misguide the retrieval process by introducing irrelevant details, and (2) The relative text carries high-quality, dense information, all of which is relevant and beneficial to the retrieval process. Prior methods often neglect a critical aspect: Despite the target image not being highly similar to the reference image or its text conversion, compositional editing of the reference image and the relative text significantly improves their similarity or ranking. As exemplified in Figure 1 , the ground truth in green does not achieve the highest similarity when compared with the combined text of the reference image and the relative text. This is due to the absence of the concept of 'dock' within the ground truth. However, the similarity increment before and after the combination with the relative text is the highest. Inspired by the above observation, we aim to address Challenge 1: How to represent and incorporate the semantic editing increment contributed by the relative text into the retrieval process? Furthermore, most existing methods overlook that ZS-CIR is essentially a fuzzy matching [9, 12, 39] task, where the semantics of the target image are not strictly determined by the reference image and relative text. This ambiguity stems from the diverse nature of the multi-modal query in ZS-CIR. The relative text aims to modify the semantics of the reference image, but the query does not explicitly specify which visual objects/attributes to modify, retain, or omit. Therefore, as shown in Figure 1 , this semantic ambiguity can result in a wide range of possible semantics of the retrieval target. For example, the relative text does not clarify whether the person in the target image should be on the dock or whether he should be talking on the phone. However, LLM may describe a person on the dock who is not talking on the phone in the target image, which introduces semantic bias. Hence, generating just one edited caption may fail to adequately capture the diversity of potential composed results, leading to suboptimal retrieval performance for ZS-CIR. From the preceding discussion, we have to tackle Challenge 2: How to effectively generate a variety of edited captions that encompass diverse potential semantics of the composed results?To deal with the above challenges, we propose a novel Semantic Editing Increment for ZEro-shot composed image retrieval (SEIZE) method, retrieving a composed image based on a reference image and a relative text without training. For Challenge 1, we propose a semantic editing search to comprehensively consider both the increment of semantic editing and the final similarity. Our approach draws inspiration from formative assessment [7, 8, 14] extensively studied in education, which refers to an assessment process, emphasizing not only the end result but also the improvement facilitated by the teaching process. Therefore, we separately calculate the cosine similarity between the composed text and the content text with the retrieval images. We compute the semantic editing increment as the difference in similarity or ranking, add it to the absolute similarity for the final adjusted score, and then retrieve the composed image based on this score in the CLIP feature space. For Challenge 2, we propose a breadth compositional reasoning to generate diverse edited captions, covering possible semantics of the composed results for fuzzy retrieval. To conduct it, we generate diverse captions for the reference image focusing on different semantic perspectives. Then, we prompt an LLM to infer breadth edited captions based on the relative text, describing possible composed images of diverse semantics. Extensive experiments on three benchmark datasets for ZS-CIR indicate significant performance improvements in our proposed SEIZE compared with the state-of-the-art methods.In summary, our contributions can be summarized as follows:  
related work 2.1 composed image retrieval: 
 Composed Image Retrieval (CIR) task [11, 18, 29, 47] , focusing on retrieving a target image based on a query of a reference image and a relative text, has gained significant attention in recent years. CIR combines compositional learning [25, 28] with image retrieval, forming a distinctive and demanding task, which has been widely applied in conditional search [6] and fashion styling [50] . For instance, Text-Image Residual Gating [47] , leverages ResNet [22] for image feature extraction and LSTM [23] for text feature extraction. Then, a residual module combines these multi-modal query features to produce the composed feature. An extension [11] introduces an attention module to blend hierarchical reference features with relative text features. However, those CIR methods rely on annotated data, which is complicated and requires extensive labor. Therefore, Zero-Shot CIR (ZS-CIR) [4, 41, 43] has recently garnered significant interest, aiming to design generalized CIR models without annotated data. Existing ZS-CIR methods commonly transform the image modality into a text modality using methods like a captioning model or textual inversion. Some existing methods [4, 41] use image-caption pairs to train textual inversions mapping images to text tokens. A static template merges tokens and relative text to obtain target captions, performing CIR without explicit supervision based on CLIP [40] . Recently, Karthik et al. [27] propose a training-free method that captions the reference image using a pre-trained VLM and employs an LLM to recompose the caption based on the relative text for retrieval. However, they solely consider the final similarity between the composed text and retrieved images for matching, overlooking the fact that the target image is not necessarily the most similar to the composed text in CLIP space. To overcome this problem, we propose a plug-and-play semantic editing search to incorporate the semantic editing increment contributed by the relative text into the retrieval process.The popularity of the pre-trained BERT [17] model has sparked interest in developing pre-trained Vision-Language Models (VLMs), including [13, 32, 33, 37, 42] , aiming to create Transformer-based [45] models trained on large-scale image-text triplets to produce vision-and-language representations. For CIR, to map images and text into a shared embedding space, many methods harness large pre-trained multi-modal models, such as CLIP [40] , as the backbone for feature extraction. These models [5, 21, 27] have recently gained popularity due to their exceptional ability to handle multi-modal data. For example, Baldrati et al. [5] utilize CLIP to extract both image and text features and then employ a combiner module to merge the multi-modal query features, achieving excellent retrieval performance. Recently, Han et al. [21] further advance this approach by designing a unified visual-language model capable of managing multiple multi-modal learning tasks, including CIR. They use various cross-attention adaptors and achieve leading performance in CIR by leveraging the large model and multi-task learning. Recent progress has been made with models such as BLIP [31] and OFA [48] , which move beyond shared space projection to address various vision-language tasks, such as captioning [46] and visual question answering [3] . While these models have relied on these models indirectly for CIR via specialized modules [6, 16, 47] and fine-tuning [19] , our research shows that when vision-language models are combined with an LLM, they can effectively perform CIR without requiring additional training. 
problem statement: 
 Zero-Shot Composed Image Retrieval (ZS-CIR). Composed Image Retrieval (CIR) can be defined as a multi-modal retrieval problem. Given a reference image 𝐼 𝑟 and relative text 𝑇 , the objective is to retrieve the target image 𝐼 𝑡 from an image database B that aligns with the relative text while preserving the underlying semantic content of the image that has not been explicitly mentioned. Zero-shot CIR further entails the absence of training samples, which should be conducted with off-the-shelf tools. 
method: 
 As stated in the problem statement, the input of CIR is a multi-modal query 𝑞 = {𝐼 𝑟 ,𝑇 }, where 𝐼 𝑟 and 𝑇 denote the reference image and the relative text, respectively. We leverage a combination of visual and textual information for retrieval. The reference image serves as the visual feature, while the relative text provides additional context or constraints for the retrieval process. Our objective is to find the target image that not only matches the relative text but also captures the essence of the reference image. To this end, we propose a method called Semantic Editing Increment for ZS-CIR (SEIZE), which consists of breadth compositional reasoning and semantic editing search, as depicted in Figure 2 .To capture comprehensive coverage of potential semantics in the composed results, we introduce a method based on LLMs, which facilitates broad-scope compositional reasoning to generate diverse edited captions as shown in Figure 3 . The method comprises a multi-caption generator for crafting multiple reference captions and a multi-prompt editing reasoner for reasoning and editing.4.1.1 Multi-caption Generator. Unlike earlier methods [4, 20, 41] that directly extract visual features from reference images, our approach for CIR uses a language-for-vision strategy. Rather than relying on visual features, we harness a pre-trained captioning model 𝜙 (•), such as OFA [48] , Flamingo [2] or BLIP-2 [30] , to generate natural language descriptions of the reference image. The method for generating multiple captions can be described as follows:EQUATIONwhere 𝑇 𝐶 𝑖 represents the 𝑖-th caption generated for the reference image, and 𝑁 is the total count of captions produced. To ensure the generation of diverse captions and prevent repetition, we employ nucleus sampling [24] during the captioning process, improving the diversity of the generated captions.Reasoner. Relying solely on either relative text or reference captions alone fails to provide complete context and modification details. While predefined templates like "a photo of {REF} that {REL}", can fixedly combine them, they lack the adaptability to accommodate diverse forms of relative text and determine the most suitable caption format for a specific query. To tackle this, we leverage the reasoning capabilities of existing LLMs. Instead of merging the reference caption and relative text by a static template, we derive cohesive, unified, and breadth captions edited by LLMs.Formally, given the diverse captions T 𝐶 of the reference image and relative text 𝑇 , we design a simple prompt template 𝑓 (•, •) inspired by [35] , combining the reference captions and relative text to create the full prompts P for LLM:EQUATIONwhere 𝑝 𝑖 represents the prompt that combines the reference image caption and relative text. The image caption serves as a content prompt prepended with "Image Content:", and the relative text serves as a modification instruction prepended with "Instruction:", as shown in Figure 2 . We fill these two parts into the template to get the full prompt. Then, we input the prompts into the LLM for reasoning and obtain the generated edited captions T 𝐸 :EQUATIONWe include only one example in each LLM query for compositional reasoning to generate a variety of breadth-edited captions efficiently and cost-effectively. Refer to the Supplementary for a detailed description of prompt template 𝑓 (•, •).To effectively incorporate the semantic increment contributed by the relative text into the retrieval process, we propose a semantic editing search to comprehensively consider relative semantic increment and absolute similarity as the final retrieval score.To retrieve the target image from the image database using edited captions, we need to extract features from both modalities and align them in a common feature space. For this purpose, we use the image and text encoders from large-scale VLMs (e.g. CLIP) to process the image database and edited captions, respectively.Visual Feature Extractor: Given the image-search database B, for each image 𝐼 𝑖 ∈ R 𝐻 ×𝑊 ×3 in the database, we use the CLIP image encoder 𝜓 𝐼 (•) to extract the image feature and normalize it to obtain 𝑓 𝐼 𝑖 ∈ R 𝑑 , where 𝑑 is the dimension of the joint feature space. Formally, we extract the visual features F 𝐼 ∈ R 𝑀 ×𝑑 of the entire image database as follows:EQUATIONwhere 𝑀 = |B| is the number of candidate images in the database.Textual Feature Extractor: Given the edited captions T 𝐸 and CLIP text encoder𝜓 𝑇 (•), we extract the textual features F 𝐸 ∈ R 𝑁 ×𝑑 of edited captions as follows:EQUATIONThen, we aggregate the textual features of edited captions into a unified feature as 𝑓 𝐸 = Agg(F 𝐸 ). Similarly, we extract the textual features F 𝐶 of diverse captions T 𝐶 , and the corresponding unified feature 𝑓 𝐶 for subsequent calculations.To quantify the significance of semantic improvement in the similarity between text and images after the compositional editing of the reference image and the relative text, we employ a measure of semantic increment before and after editing as the scoring metric for images in the database. In Figure 4 , the relative text guides the reference image toward the target image to form an edited reference caption. However, the image most similar to the edited caption is not the target image, but the hard negative image. In this case, where traditional retrieval methods fail, the semantic editing increment is denoted by a purple bidirectional arrow. Leveraging the principle that in a triangle, the difference between any two sides is less than the length of the third, it becomes evident that the reference image yields the maximum semantic editing increment to the target image among all retrieval images. Note that for the sake of a more intuitive example, we employ distance metrics. However, the same concept can be seamlessly applied to similarity metrics as well.Formally, given the aggregated feature 𝑓 𝐶 of reference captions and the feature 𝑓 𝐸 of their corresponding edited captions, we independently calculate their similarity to the visual features F 𝐼 as follows:EQUATIONwhere 𝑆 𝐶 , 𝑆 𝐸 ∈ R 𝑀 represent the cosine similarities between the reference and edited captions, respectively, and all 𝑀 retrieval images. Here, 𝑆 𝑖 ∈ [-1, 1] since all visual and text features are preconditioned by 𝐿 2 normalization in the CLIP space. Then, we calculate their difference Δ𝑆 = 𝑆 𝐸 -𝑆 𝐶 as the semantic editing increment. Finally, we calculate the adjusted similarity 𝑆 𝐴 as follows:EQUATIONwhere 𝛼 represents the weight factor for the positive semantic editing increment, and 𝛽 represents the weight factor for the negative semantic editing increment. Indeed, we can also use ranking changes as the semantic editing increment to calculate the adjusted similarity, with comparison results as shown in Section 5.4.Based on the adjusted similarity between the image features in the database and the feature of edited captions, we make the following calculation and search:EQUATIONwhere 𝑘 is the index of the target image in the database and we get the final target retrieved image 𝐼 𝑡 = B (𝑘). 
experiments: 
 We conduct extensive experiments to investigate the effectiveness of our proposed method for composed image retrieval.5.1.1 Datasets. We compare SEIZE with state-of-the-art baselines on three public datasets: CIRCO [4] , CIRR [36] , and FashionIQ [50] , which have been widely used for CIR. CIRR is the first natural image dataset designed specifically for CIR. There are 36,554 queries, each containing a single target image. CIRCO is based on realworld images from the COCO 2017 unlabeled set [34] and provides multiple ground truths. It provides 123,403 images for retrieval and 4.53 ground truths per query on average, which offers a more robust evaluation for CIR models. FashionIQ focuses on the fashion domain and is divided into three subcategories: Dress, Shirt, and Toptee. It comprises 30,135 triplets for query and 77,683 images for retrieval. In particular, we employ the test sets of CIRR and CIRCO and three categories of FashionIQ validation split for ZS-CIR.5.1.2 Evaluation Metrics. On CIRCO, given that each query has multiple target images, we utilize mean Average Precision (mAP), a more fine-grained metric to consider the rank of retrieval results. Specifically, we apply mAP@k for our evaluation, where 𝑘 ∈ {5, 10, 25, 50} represents the number of top-ranked retrieval results under consideration. On CIRR, we use Recall@k (𝑘 ∈ {1, 5, 10}) as the main metric, which denotes the percentage of target images included in the top-k list. Besides, we also evaluate with the subset setting, where there are only 6 most similar images (one ground truth) within the subset of the query in the database, denoted as Recall Subset k (k ∈ {1, 2, 3}). On FashionIQ, we also employ Recall@k (𝑘 ∈ {10, 50}) as the main metric. We separately compute the Recalls for three categories (Dress, Shirt, and Toptee) from the FashionIQ validation split, and then calculate the average of them.For the captioning model, we use the pre-trained BLIP-2 [30] with a large language model OPT-6.7b. We employ nucleus sampling [24] during the caption generation process to generate diverse captions, and we set the number 𝑁 of captions to 15. For VLM, we use the ViT-B/32 and ViT-L/14 CLIP [40] from OpenAI, as well as the ViT-H/14 and ViT-G/14 CLIP from OpenCLIP [26] . For LLM, we use gpt-3.5-turbo [10] by default, but we also conduct extensive experiments with Llama2-70B [44] and GPT-4 [1] . The weight factors 𝛼 and 𝛽 in the SES module are set to 0.13 and 2.1. The feature space dimensions are as follows: 512 for B/32, 768 for L/14, and 1024 for H/14 and G/14. The whole model is implemented by Pytorch [38] with one NVIDIA A100 GPU.We select the following state-of-the-art baselines for ZS-CIR to conduct a comprehensive comparison. Image-only: Only the features of reference images, which are extracted by the CLIP image encoder, are used to compute similarity for retrieval; Text-only:Only the relative text features, extracted by the CLIP text encoder, are utilized as retrieval features to calculate similarity; Captioning:We employ BLIP-2 to generate captions of reference images for retrieval; PALAVRA [15] : A two-stage approach based textual inversion with a mapping function and a subsequent optimization of the pseudo-word token; Pic2Word [41] : A training-dependent method employs a textual inversion network optimized by contrastive loss to capture the pseudo-word token for retrieval; SEARLE [4] : A training-dependent method where pseudo-word tokens of images are generated with a textual inversion and then distill their knowledge to a textual inversion network; SEARLE-OTI [4] : A variant of SEARLE without the distillation network; LinCIR [20] : A selfsupervision method is used for training, which projects text embeddings into the token space for retrieval; CIReVL [27] : A training-free method using a generative VLM and asking an LLM to recompose the caption based on the textual modification for retrieval. (3) SEIZE consistently outperforms all baselines across all metrics and CLIP backbones. For instance, using ViT-L/14, SEIZE outperforms the second-best CIReVL by 34.52% in mAP@5, 35.82% in mAP@10, 35.18% in mAP@25, and 34.63% in mAP@50, proving the effectiveness of SEIZE for CIR. CIRR: The right section of Table 1 presents CIRR test results. From these results, key observations include: (1) Text-only performs significantly better than Image-only and Captioning, indicating a minimal correlation between reference and target images due to the noisy dataset and the lesser information provided by reference images. (2) Despite the noise, SEIZE consistently outperforms all baselines across all CLIP backbones. This highlights the robustness and adaptability of SEIZE even in noisy data and diverse scenarios.(3) Using ViT-L/14 CLIP, SEIZE surpasses the second-best method by 14.42% in Recall@1, 7.34% in Recall@5, and 3.82% in Recall@10, emphasizing the effectiveness of SEIZE.FashionIQ: Table 2 shows the results from the FashionIQ validation set. Key observations include: (1) Text-only methods perform better than Image-only methods, but not significantly better than the Captioning method. This suggests that while reference images lack key information, converting them into captions is empirically a better strategy, possibly due to the subtle differences in style, color, or pattern in fashion images that are hard to capture through visual features directly. ( 2) SEIZE outperforms all baselines in all metrics with ViT-B/32 and L/14 CLIP. When using L/14, SEIZE relatively outperforms the second-best CIReVL by 16.22% in average Recall@10 and 11.61% in average Recall@50. However, LinCIR achieves the best average R@10 on ViT-G/14, likely due to superior supervised optimization of training-based methods on larger VLMs. Despite this, SEIZE remains the top-performing training-free method.In this section, we conduct ablation studies to evaluate the individual contributions of components in our method. To avoid confusion, we assess the three main components separately by comparing SEIZE variants. The following three categories of SEIZE variants are designed for comparative analysis:• Breadth Compositional Reasoning (BCR) module: (1) A variant of SEIZE without LLMs and only using a simple template "a photo of {caption} that {relative text}", (2) a variant without Training-free Shirt Dress Toptee Average R@10 R@50 R@10 R@50 R@10 R@50 R@10 R@50ViT-B/32Image-only LLMs or captioning models and only using the relative text for retrieval, and (3) a variant without multiple diverse captions and only using a single caption.• Aggregate function: A variant of SEIZE with the maximum feature (the maximum similarity feature for each image), and a variant with the optimal feature (the feature of a single edited caption with the best retrieval result) instead of the average feature of edited captions. • Semantic Editing Search (SES) module: A variant of SEIZE without the SES module, and a variant with ranking increment (integer, multiplied by a weight factor of 0.3) instead of similarity.The ablation study results are shown in Table 3 • Effects of the SES module: The results of the SES column show that: SEIZE outperforms the variants with ranking scores and without SES, highlighting the effectiveness of the semantic editing increment for additional consideration of improvements by compositional reasoning. For more details on the impact of SES on various ZS-CIR methods, refer to Section 5.6.To analyze the sensitivity of the hyperparameter in SEIZE, we conduct experiments with controlled variables on the CIRCO test set. We use pre-trained BLIP-2 for captioning, ViT-L/14 CLIP for image and text feature extraction, and gpt-3.5-turbo for reasoning.Number 𝑁 in the Multi-caption Generator: To examine the impact of the number 𝑁 of samples for the captioning model, we vary the value of 𝑁 from 1 to 20 and report the results on the CIRCO test set in Figure 5 . The results across the four metrics show a pattern: an initial sharp increase followed by relative stability as 𝑁 increases. The model achieves a good balance between computational cost and effect when 𝑁 is between 13 and 16.Weight factors 𝛼 and 𝛽 in SES module: We vary 𝛼 from 0.05 to 0.25 and 𝛽 from 0.5 to 2.5 to analyze the influence of weight factors. As 𝛼 increases, results first rise and then fall, while as 𝛽 increases, results initially rise and then stabilize. From Figure 5 , we can observe that the model achieves the best performance with 𝛼 = 0.13 and 𝛽 = 2.1. The negative semantic editing increment is more critical due to its larger weight factor, which maintains fine results as it increases. However, SES is sensitive to the positive semantic editing increment, and increasing its weight leads to poor performance. This aligns with intuition: images matching the semantics of relative text may not necessarily be the target image, but those that deviate from it are, with high probability, not the target image.Considering that current ZS-CIR methods utilize I2T models to transform the query image into a caption, which is then combined with the relative text via text-fusion approaches to generate a final text for retrieval, it is plausible to integrate the Semantic Editing Search (SES) module into various ZS-CIR methods that rely on the language-for-vision strategy. The experimental results and improvements before and after using the SES module on ZS-CIR methods with ViT-L/14 CLIP are shown in Table 4 . It can be observed that the SES module has improved all methods, especially the textual inversion methods, with an average improvement of 21.66%. This proves that our SES module is plug-and-play, simple yet effective, allowing for seamless integration with various methods for ZS-CIR.Since our proposed SEIZE relies heavily on VLMs for retrieval and LLMs for reasoning, we study several VLMs and LLMs to understand their impact on performance. VLM: Table 5 shows that CLIP encoders with larger parameters perform significantly better, suggesting that they can capture more detailed features, leading to performance improvement. When the LLM is set to GPT-3.5-Turbo, larger parameter CLIP models, specifically ViT-H/14 and G/14, show considerable improvements over the smaller parameter model, B/32, with performance increases of 70.52% and 71.95% respectively. LLM: We conduct experiments on CIRCO using various configurations listed in Table 5 : no LLM (with fixed templates), an open-source LLM (LLama2-70B), and closed-source LLMs (GPT-3.5-Turbo and GPT-4). The results show that the LLM-based method significantly outperforms the no LLM approach, highlighting the power of dynamic textual reasoning in generating target captions. Furthermore, GPT-based methods perform better than the opensource LLM. Specifically, GPT variants, GPT-3.5-Turbo and GPT-4, achieve 12.12% and 12.75% relative improvements in mAP@5 compared to LLama2-70B, respectively. The plug-and-play design of SEIZE enables the seamless integration of various retrieval and reasoning models, providing flexibility to scale our pipeline as needed and consider the trade-offs between cost and effectiveness, which is crucial when choosing the best LLM and VLM for applications. Thus, we can customize SEIZE to fit any specific retrieval scenario. 
conclusions: 
 In this paper, we propose a training-free method called Semantic Editing Increment for ZEro-shot composed image retrieval (SEIZE), which can utilize off-the-shelf tools to accurately retrieve a specific image based on a reference image and a relative text without training. Our method can consider not only the final similarity between the composed text and retrieved images but also the semantic increment during the compositional editing process. To cover the possible semantics of the composed results, we propose a breadth compositional reasoning to generate diverse edited captions from different semantic perspectives of the reference image. Extensive experiments on three public datasets demonstrate that the proposed SEIZE significantly outperforms the state-of-the-art methods for ZS-CIR. In the future, we will explore semantic editing increment based on LLMs and VLMs in more multi-modal reasoning tasks, such as visual reasoning and conditional generation.  
 
 Appendix:  


Decision: This paper addresses the challenging task of zero-shot composed image retrieval (ZS-CIR) through the SEIZE method, leveraging semantic editing search and breadth compositional reasoning. The proposed approach aims to enhance retrieval accuracy by incorporating semantic increment and diverse caption generation using large language models (LLMs). While the paper demonstrates clear research motivation and achieves non-trivial improvements over existing methods on multiple datasets, there are notable areas requiring improvement: 1) the lack of in-depth theoretical explanation or analysis of the incorporation of semantic editing increment, or any in-depth theoretical analysis of the experimental results (for example, why did it not perform as well on the fashion dataset?), 2) the lack of detailed ablation study, and 3) language and presentation clarity issues.

Reviews: review: paper_summary: paper_summary: The submission describes an approach to image retrieval heavily depended on large neural networks, that combines text and images in a novel way, so that (less?) training is needed.
strength_weakness: strength_weakness: The paper has a lot of references and tables and the source code of the approach is available. I'm sure another reviewer can make use of it, but I'm too far off this specific field, to judge it.
limitations: limitations: From my point of view - the uninformed reader - the paper is really hard to understand. The interested reader needs to bring along a lot of knowledge in the domain of different ML-based methods to understand the actual point of the paper. Just for the abstract one has to have knowledge about Composed Image Retrieval, Zero-Shot Composed Image Retrieval, image captioning with image2text models, LLMs, and CLIP. Considering this and the list of references, I fear that while the approach is multi-modal, the paper would well fit in ICCV, PAMI, or ICMR rather than ACM MM. 

From my perspective I would have a lot more questions, like:
Why would you define similarity? It all sounds so general, but at the end of the day, it's just defined by the test data sets for evaluation, isn't it? Then, the follow up question would be, what is the hypothesis begind the research and the ultimate goal, or how does it translate to applications or further research. I'm missing a graspable und understandable insight. 

However, I'm definitely the wrong reviewer for this submission.
paper_summary: paper_summary: In this paper, the authors studied the zero-shot composed image retrieval task in the field of fashion and in open domain, aiming to explore the semantic edited increment after the composition process based on LLMs, and improve the recall of the task. To address this issue, this paper proposes a semantic editing increment model leveraging multi captions, which first prompts LLMs to generate semantically interactive captions, and then processes semantic incremental correction to change the similarity matrix for retrieval. The experiments show that compared with existing methods, the proposed model can significantly improve the retrieval accuracy. This paper has clear research motivation. However, there are also some problems in writing, which need to be strengthened.
strength_weakness: strength_weakness: a.This paper found that in the process of zero-shot composed image retrieval, the semantic editing increment process has a positive effect on image retrieval, which is the first work to study the zero-shot composed image retrieval task based on utilizing the semantic increment. 
b.This paper utilizes LLMs to integrate the multi image captions obtained by pre-trained captioning model and relative captions for composition and employs a measure of semantic increment for improving the retrieval accuracy, which takes advantages of the capability of LLMs and mines the metric relationship in the feature space.
c.Extensive experiments are are carried out on three public data sets, and the experiments showed that the proposed method can improve the performance.
limitations: limitations: a.This paper did not explain the main idea clearly especially for this is a training free process. For example, for the challenge of incorporating the semantic editing increment into the retrieval process, although this paper demonstrated the effectiveness of adding increment before retrieval through certain experiments, it did not clearly explain how the process works from a theoretical perspective, especially when this task did not involve an optimization process to mine such increment.
b.Since this paper proved the effectiveness of each module in CIRCO dataset, but did not provided the ablation experiment details in FashionIQ or CIRR dataset, it is recommended to provide a more detailed ablation experiment in more datasets.
c.There are some colloquial problems in the language of this paper, and some expressions need to be improved. It is suggested to polish the language.
paper_summary: paper_summary: In this paper, the authors focused on the zero-shot composed image retrieval. The authors analyzed that the previous studies rely heavily on the robustness of I2T models and text-fusion approaches, which cannot be generalized well in practical applications. The authors proposed a SEIZE method that incorporates a semantic editing search and a breadth compositional reasoning to achieve ZS-CIR. Extensive experiments on three benchmark datasets prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: There are three strong points:
1.	The authors designed a plug-and-play semantic editing search in ZS-CIR, which is novel and effective.
2.	The designed LLM-based breadth compositional reasoning can generate diverse edited captions to improve the model’s robustness capabilities.
3.	Extensive experiments on three benchmark datasets validate the effectiveness of the proposed method.
limitations: limitations: There are also some weak points:
1.	The proposed method involves using MLLM for multiple captions and LLM for reasoning. The introduction of these large models poses challenges for model efficiency. In fact, retrieval tasks demand high responsiveness, so the design of this method may limit its practical application scenarios.
2.	From the experimental results, the performance improvement of the method on the CIRCO and CIRR datasets is quite evident, but the performance improvement on FashionIQ is limited. This may be because the designed method is more inclined towards addressing datasets dominated by modified text or performs poorly on specific domains such as Fashion datasets. The paper lacks further in-depth analysis and explanation of the differences in model performance on different types of datasets.
3.	The authors only conducted ablation study experiments on one backbone and one dataset, lacking persuasive evidence of the effectiveness of different components across various backbones and datasets.