Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-08 03:56:32

Abstract: The Composed Image Retrieval (CIR) task aims to retrieve target images using a composed query consisting of a reference image and a modified text. Advanced methods often utilize contrastive learning as the optimization objective, which benefits from adequate positive and negative examples. However, the triplet for CIR incurs high manual annotation costs, resulting in limited positive examples. Furthermore, existing methods commonly use in-batch negative sampling, which reduces the negative number available for the model. To address the problem of lack of positives, we propose a data generation method by leveraging a multi-modal large language model to construct triplets for CIR. To introduce more negatives during fine-tuning, we design a two-stage fine-tuning framework for CIR, whose second stage introduces plenty of static representations of negatives to optimize the representation space rapidly. The above two improvements can be effectively stacked and designed to be plug-and-play, easily applied to existing CIR models without changing their original architectures. Extensive experiments and ablation analysis demonstrate that our method effectively scales positives and negatives and achieves state-of-the-art results on both FashionIQ and CIRR datasets. In addition, our methods also perform well in zero-shot composed image retrieval, providing a new CIR solution for the low-resources scenario. The code is released at https://anonymous.4open.science/r/45F4 and will be publicly available upon acceptance.

Main paper: 
 related work: 
 Composed Image Retrieval. The recent paradigm in CIR [4, 33, 40, 41] consists of three main steps: (1) extracting the representation of both images and sentences; (2) fusing the representations of sentences and reference images to obtain query representations;(3) aligning the representations of queries and target images with similar semantics. For the first step, early models in CIR utilize two separate encoders [11, 14, 33, 38] while recent CIR models [4, 21, 35, 41] exploit pre-trained vision-language encoders [25, 28] as the backbone. Some works [4, 41] simply use the global representations extracted from these pre-trained encoders while other works [35, 38, 40] integrate local and global representations. For the second step, some works [4, 33, 34, 41] leverage weights or gating mechanisms, while other works [38, 40] design combining modules like cross-modal transformer. For the third step, the most commonly used loss functions in CIR are triplet loss [8, 30, 40] , contrastive learning [4, 15, 31, [33] [34] [35] 41] . Recent advanced methods in the CIR predominantly employ a combination of dual encoders and contrastive learning with in-batch negative sampling. We treat the models obtained from these methods as the first-stage models and continue to train them in the second stage to improve CIR performance further.Data Generation for CIR. InstructPix2Pix [6] first uses GPT-3 to generate modified text for captions and then utilizes a diffusion model to generate images for these texts. COVR [32] mine similar captioned videos from a large database and use a language model to generate modified text that describes the differences between the videos, resulting in the WebVid-CoVR dataset with 1.6 million triplets. CASE [16] uses a data roaming approach that rephrases labels from a large-scale VQA dataset into a form suitable for composed image retrieval. CompDiff [12] constructs triplets for CIR datasets by automatically generating modified texts and corresponding images using large language models and diffusion models. Unlike these works that often require generating images or well-labeled datasets, our method is built on a real image collection without the need for any additional manual annotation and leverages the capabilities of MLLM to construct triplets.Negative Sampling in Contrastive Learning. In the realm of contrastive learning, negative sampling techniques have evolved to enhance model performance: the in-batch negative sampling from SimCLR [7] selects negative examples from the same batch, while the memory bank approach in Bank [37] utilizes a stored set of past instances for more diverse negatives. Additionally, MoCo [13] employs a moving average of representations to create dynamic negatives, contributing to robust representation learning. Compared to Memory Bank and MoCo, our method does not dynamically update the negatives with the aid of additional queues or momentum encoders; instead, it fine-tunes the model for the second stage by introducing a large number of static negative samples at once. 
method 3.1 preliminary: 
 Suppose a CIR dataset consists of 𝑁 annotated triplets, where the 𝑖 th triplet 𝑥 𝑖 is denoted as𝑥 𝑖 = (𝑟 𝑖 , 𝑚 𝑖 , 𝑡 𝑖 ), 𝑟 𝑖 , 𝑡 𝑖 ∈ Ω, 𝑚 𝑖 ∈ 𝑇 (1)where 𝑟 𝑖 , 𝑚 𝑖 , and 𝑡 𝑖 represent the reference image, the modified text 1 , and the target image of the 𝑖 th example, respectively, while Ω is the candidate image set containing all reference and target images of the triplets and 𝑇 is the text set containing all modified texts. The CIR task aims to use the reference image 𝑟 𝑖 and the modified text 𝑚 𝑖 to compose a query 𝑞 𝑖 , and retrieve the target image 𝑡 𝑖 from the candidate set Ω with 𝑞 𝑖 . Then, we describe the classical paradigm of CIR. Multiple annotated triplets are combined into a mini-batch, and the reference images and modified texts in the same batch are then encoded using a query encoder 𝐹 (•) query representations. The target images are encoded using an image encoder 𝐺 (•) to obtain target image representations. For simplicity, we rewrite the representations for the triplet (𝑟 𝑖 , 𝑚 𝑖 , 𝑡 𝑖 ) as q 𝑖 = 𝐹 (𝑟 𝑖 , 𝑚 𝑖 ) and t 𝑖 = 𝐺 (𝑡 𝑖 ), respectively. The cosine similarity 𝑓 (•, •) is then adopted to calculate the similarity between the query and target image representations. EQUATION)where 𝐵 is the batch size and 𝜏 is a temperature hyper-parameter. Despite the good results achieved with this current paradigm, the lack of negative and positive examples still severely limits the performance of contrastive learning. To address these problems, we first propose a method of scaling positive examples using a multi-modal large language model (MLLM). Then, we investigate the impact of different types of negative examples on CIR performance and find that using negative examples obtained by replacing the target image is simple and most effective. Therefore, we propose a two-stage fine-tuning strategy, scaling negative examples using a caching technique based on existing models.Due to the high cost of manually labeling triplets, we propose a simple but effective method with a multi-modal Large Language Model (MLLM) to construct the triplets for CIR. As shown in Fig, 2 , given an image dataset2 𝐷 = {𝐼 1 , 𝐼 2 , ..., 𝐼 𝑀 } with size 𝑀, our method consists of four steps: (1) Generating a suitable caption for each image to obtain the image-text pairs; (2) Constructing M (reference image, target image) pairs; (3) Generating modified texts for image pairs using the captions; (4) Combining the modified texts and image pairs to form triplets.Caption Generation. We introduce a MLLM 𝑔 mllm (•, •) to generate a corresponding caption for each image in the dataset. Specifically, we design a prompt template 𝑃 cap (𝑡𝑦𝑝𝑒, 𝑘) to guide the MLLM to obtain a brief caption for each image under constrained conditions, where 𝑡𝑦𝑝𝑒 and 𝑘 are two dataset-specific parameters to simulate the type and length of modified text in the real dataset. For an image 𝐼 𝑖 in the candidate image set, we input 𝐼 𝑖 and 𝑃 cap together into the MLLM to obtain the corresponding caption 𝐶 𝑖 :𝐶 𝑖 = 𝑔 mllm (𝐼 𝑖 , 𝑃 cap (𝑡𝑦𝑝𝑒, 𝑘)).(EQUATIONThen we can obtain M image-text pairs {(𝐼 1 , 𝐶 1 ), ..., (𝐼 𝑀 , 𝐶 𝑀 )}. In practice, the 𝑃 cap used in this work is written as follows:Please briefly describe the {type} in {k} words.Image Pair Match. After obtaining the image-text pair, we need to match two image-text pairs to generate a quadruplet. Regarding the image in an image-text pair as the reference image, the naive method randomly chooses the image from another imagetext pair as the target image. However, a randomly selected target image may be too similar to the reference image to construct precise modified text or too dissimilar to help models improve performance. Therefore, we introduce a uni-modal image encoder 𝑔 img (•) to get the representation of every image and calculate the pairwise similarity between two different images 𝐼 𝑖 and 𝐼 𝑗 :𝑠𝑖𝑚 𝑖 𝑗 = 𝑓 (𝑔 img (𝐼 𝑖 ), 𝑔 img (𝐼 𝑗 )) (1 ≤ 𝑖, 𝑗 ≤ 𝑀, 𝑖 ≠ 𝑗) (4)Then we can rank the similarities related to 𝐼 𝑖 in descending order. Only one image whose similarity rank is between [𝑐 0 , 𝑐 1 ) (𝑐 0 < 𝑐 1 ) will be chosen as the target image, where 𝑐 0 and 𝑐 1 are two hyperparameters. In practice, we regard each image in the dataset 𝐷 as the reference image and sample a target image for each image. We denote the target image for image 𝐼 𝑖 as 𝐼 t 𝑖 , therefore, we can get 𝑀 (reference image, target image) pairs {(𝐼 1 , 𝐼 t 1 ), ..., (𝐼 𝑀 , 𝐼 t 𝑀 )}. We combine these image pairs with their corresponding captions to form M quadruplets: {(𝐼 1 , 𝐶 1 , 𝐼 t 1 , 𝐶EQUATIONIn this work, we consider three types of templates below.𝑃 temp 0 : {𝐶 t 𝑖 } instead of {𝐶 𝑖 } 𝑃 temp 1 : Unlike {𝐶 𝑖 }, I want {𝐶 t 𝑖 } 𝑃 temp 2 : {𝐶 t 𝑖 }Note that we attempt to use LLM to post-process the generated modified text. The first method involves using LLM to make the Step 1: Caption Generation A white bookshelf with books and a plant in a room, instead of A living room with a blue couch, green pillowsStep 3: Modified Text GenerationStep 2: Image Pair Match modified text more diverse and fluent. The second method uses in-context learning to make LLM mimic the modified text in annotated datasets. However, based on our experiments, neither of these methods surpasses the prompt template method. Specific results can be found in supplementary materials.Fashion GeneralPositive Example Construction. Finally, we could combine image pairs from the second step with the modified texts obtained in the third step to get new M triplets {(𝐼 𝑖 , 𝑚 temp 𝑖 , 𝐼 t 𝑖 )}. So, we can obtain an expanded dataset that is comparable in size to the original dataset. We could use these new examples as a complement to the annotated dataset. We could also use these examples to train a model from scratch, thus allowing for fully automated training of a CIR model without human involvement.Recent works in visual contrastive representation learning [13, 37] have shown that scaling negative numbers can effectively improve performance. However, existing works in CIR employ in-batch negative sampling strategies, restricting the model from seeing enough negatives. Furthermore, recalling that the labeled data in CIR is a triplet, it is theoretically possible to construct negative examples by replacing any element in the triplet. Most works [2, 5, 33, 35] only use the "replace the target image" strategy to construct negative samples without additional interpretation. Therefore, we first explore the performance impact of different methods of constructing negative examples and find that "Replacing the target image" leads to more true and hard negatives than other methods with the popular CIR datasets. After determining the method of negative example construction, we propose a two-stage fine-tuning strategy for CIR that leverages a two-stage framework to scale negative examples during fine-tuning.Constructing Negative Examples. Considering the annotated data in CIR are triplets, for triplet (𝑟 𝑖 , 𝑚 𝑖 , 𝑡 𝑖 ), there are four methods Most previous works use only the third method [2, 5, 9, 23, 33, 35] , and Wang et al. [34] uses the first three methods jointly. However, none of the existing works have explored all four methods completely. To this end, we compare these four negative construction methods while other settings remain the same. As shown in Fig. 3 , we find that constructing the negative examples by replacing target images works best. Based on the examples in Fig. 2 , we can observe that the other three methods easily generate relatively simple or false negatives. For example, since some modified texts (e.g., "a dog") only describe the target image, replacing the reference image with another image can lead to false negatives. Similarly, if the reference image is very similar to the target images, this type of data leads the model to directly use the reference image to retrieve the target image, making it easy to generate false negatives when replacing the modified text (e.g., "dog" and "lying"). Lastly, replacing the whole query pair leads to the simple negatives as the reference image and modified text significantly differ from those in the positive example (e.g., "sofa+shelf" and "llama+dog"). Compared with the other three methods, "replacing the target image" is inherently aligned with the final application scenario, and the probability of generating false negatives is relatively low. In the supplementary materials, we report the performance of every combination of four types of negative examples. The experimental results suggest that incorporating other types of negative examples may lead to increased overhead and potentially compromise model performance.For this reason, we keep consistent with previous work and only consider the negative example type of replacing the target image.Two-Stage Fine-tuning. Previous work on extended negative examples, such as Memory Bank [37] and MoCo [13] , has focussed on visual representation learning, whose models typically follow a simple Siamese network architecture. However, CIR tasks require an information fusion of visual and language, and different methods follow different backbones, such as CLIP [28] , BLIP [18] , and BLIP-2 [17] . Therefore, we propose a more general two-stage framework to ensure fast adaptation of different models in CIR (Right half of Fig. 2 ). Specifically, in the first stage, we fine-tune both the query encoder and the target encoder with in-batch negative sampling as in Eqn.2 following previous works [2, 5, 23] ; while in the second stage, we freeze the target encoder and only fine-tune the query encoder. Therefore, all candidate images, i.e., the entire Ω, can advance past the frozen target image encoder, cached before the second fine-tuning stage. Finally, for triplet (𝑚 𝑖 , 𝑟 𝑖 , 𝑡 𝑖 ), we utilize all non-target images from the candidate set, i.e. Ω -{𝑡 𝑖 }, to form negative examples. The contrastive loss for the second stage can be expressed as EQUATION 
baselines.: 
 To evaluate the superiority of our method, we conduct experiments on four advanced models in CIR: TG-CIR [35] , CLIP4CIR [5] , BLIP4CIR [23] and SPRC [2] . TG-CIR [35] uses CLIP ViT-B/16 as the backbone, which exploits the global and local attribute representations and information from the target image to guide both query fusion and metric learning. CLIP4CIR [5] uses CLIP ResNet50x4 as the backbone, which simply regards element-wise sum as a fusion approach.BLIP4CIR [23] uses BLIP base as the backbone, which adopts the fusion encoder of BLIP to fuse the reference image tokens and modified text tokens. We do not include an extra re-ranker to ensure the evaluation protocols are consistent. SPRC [2] uses BLIP-2 pretrained-vitl as the backbone, which exploits the QFormer [17] as an encoder for query and target image sharing.Protocol. We directly use checkpoints released by baseline works as the first stage models to avoid retraining. For the second stage, we calculate image representations for all images before training and only finetune the query encoder using scaled positives and negatives for 5 epochs. For the main results in section 4.2, we only use images in Ω, so it can be fairly compared to any model that uses the original dataset.  and 4 ,148 for testing. In addition, the images in this dataset are divided into several semantically similar groups to evaluate 𝑅 subset @𝐾 metric (see below).Metrics. 𝑅𝑒𝑐𝑎𝑙𝑙@𝐾 (𝑅@𝐾) is the proportion of queries for which the retrieved top K images include the correct target image. 𝑅𝑒𝑐𝑎𝑙𝑙 subset @𝐾 (𝑅 subset @𝐾) is nearly the same as 𝑅@𝐾 but the model only retrieves inside the semantically similar group of the reference image. For the FashionIQ dataset, following previous works [2, 4, 23] , we evaluate our model through 𝑅@𝐾 (𝐾 = 10, 50) on the original protocol. As in [41] , we also report the mean of all 𝑅@𝐾 scores as Rmean. For the CIRR dataset, following previous works [4, 22] , we evaluate our model through 𝑅@𝐾 (𝐾 = 1, 5, 10, 50) and 𝑅 subset @𝐾 (𝐾 = 1, 2, 3). As in [2] , we also report (𝑅@5+𝑅 subset @1)/2 as Rmean.We use LLaVA-v1 [20] as a multimodal large language model for caption generation. We use the advanced model unicom ViT-L/14 [1] for the unimodal image encoder. As used by CLIP4CIR [5] , we leverage AdamW [24] optimizer. All experiments are conducted on a single Tesla V100 GPU. R@10 R@50 R@10 R@50 R@10 R@50 R@10 R@50 Rmean CIRPLANT [21] w We manually tune 𝜏 ∈ {0.01, 0.02, 0.03, 0.05} and 𝑙𝑒𝑎𝑟𝑛𝑖𝑛𝑔_𝑟𝑎𝑡𝑒 ∈ {2𝑒 -6, 5𝑒 -6, 6𝑒 -6, 1𝑒 -5, 2𝑒 -5}. Detailed hyper-parameters are reported in the supplementary materials. We analyze the modified text in the two datasets using LLAVA tokenizer [20] and count the average annotated token length in Table 1 . For FashionIQ, we set 𝑡𝑦𝑝𝑒 to the name of the split, i.e., dress/shirt/top tee, and 𝑘 to 5. For CIRR and Conceptual Caption, we set 𝑡𝑦𝑝𝑒 to "image" and 𝑘 to 10 in the image captioning template. The detailed data statistics for both the generated and annotated triplets are provided in Table 1 . For both datasets, we set 𝑐 0 to 10000. We set 𝑐 1 to 20000 for FashionIQ and 15000 For CIRR.We compare our method against the following baseline methods: CIRPLANT [21] , ARTEMIS [9] , ComqueryFormer [39] , PL4CIR [41] , TG-CIR [35] , CLIP4CIR [4] , BLIP4CIR [23] , SPRC [2] . Details about these models can be found in supplementary materials. We abbreviate the method of scaling positive examples as SP, the method of scaling negative examples as SN, and the superposition of the two methods as SPN.Results on FashionIQ. Table 2 illustrates the comparison between our model and other recent studies on FashionIQ. It demonstrates that our plug-and-play approach improves the effectiveness of all four baseline models with different architectures. SPN boosts the R@10 metric for TG-CIR by 3.8%, CLIP4CIR by 4.1%, BLIP4CIR by 1.5%, and SPRC by 2.6%. SPN enhances the R@50 of TG-CIR by 3%, CLIP4CIR by 2%, BLIP4CIR by 1%, and SPRC by 2%. This mainly benefits from more negative and positive examples in contrastive learning, which allows the model to learn a better representation.Results on CIRR. Table 3 illustrates the comparison between our model and other recent studies on CIRR. It shows that SPN also improves the performance of all four baseline models. SPN increases the R@1 of TG-CIR by 4.5%, CLIP4CIR by 5.9%, BLIP4CIR by 3.7%, and SPRC by 6%. SPN improves the R@5 of TG-CIR by 1%, CLIP4CIR by 2.9%, BLIP4CIR by 1.4%, and SPRC by 2.1%. This proves our method works well for images in both general and fashion scenes. SPN promotes the R subset @1 of TG-CIR by 3.5%, CLIP4CIR by 5.6%, BLIP4CIR by 1%, and SPRC by 1.1%. The objective of the subset test is to justify whether the model can distinguish between hard negative examples [21] . Such a boost indicates that our model can learn more fine-grained representations than the base model, thus distinguishing harder negative examples. Contribution of SP and SN. To evaluate the effectiveness of SP and SN, we train CLIP4CIR using several variants of our method and test on the validation set of CIRR and FashionIQ. SP variant conducts contrastive learning with in-batch negative sampling on the scaled positive examples. SN variant only scales negative examples without exploiting new positive examples. The results illustrate that removing either SN or SP significantly decreases performance. SP and SN can improve the baseline model by 1.3% to 3.1% on the two datasets, respectively. SN is more effective for the CIRR dataset. SP is more useful for the FashionIQ dataset. We attribute this phenomenon to the fact that the modified texts are more complex in the CIRR dataset and that contrast learning is more lacking in negative than positive examples. While the modified texts are simple in FashionIQ, the situation is exactly the opposite.Discussion on 𝑘. Since the LLAVA tokenizer utilizes the BPE tokenization method, which typically results in a word count to token count ratio of 1:2. So the corresponding word counts for FashionIQ are around 4, and for CIRR, they are around 8. Therefore, we experiment with 𝑘 values that approximate the word count of the modified text in the annotated triplets. As shown in Fig. 4 (a), we find that slightly exceeding the annotated word count yields better results, as lower or higher values lead to performance degradation.Discussion on MLLM Model. The MLLM we use can be replaced with any model that can generate captions for images, so we try three representative models LLAVA [20] , BLIP [18] , and BLIP-2 [17] in Table 4(b) . We find that LLAVA, with great instruction fine-tuning, works best among the three models. But surprisingly, BLIP works better than BLIP-2. This suggests that BLIP-2's ability to follow image captioning instructions is not very good. At the same time, using different MLLMs consistently yields better results than w/o SP, indicating that our method is insensitive to different MLLMs.Examples. SP allows for constructing many triplets based on images, so we consider exactly how many additional triplets on top of the existing ones work best. As shown in Fig 4(c) , as the number of positive examples rises, the effect of the model increases and then decreases, with the best results when increasing nearly 60% of the number of original triplets, that is 12k for FashionIQ and 16k for CIRR.Discussion on Image Pair Match. In Fig. 4 (d), we explore four methods for constructing image pairs. The first method involves selecting target images with the highest similarity to the reference image. The second method entails choosing target images with moderate similarity to the reference image. The third method focuses on selecting target images with the lowest similarity to the reference image, while the fourth method involves selecting target images randomly from the entire set. Our findings indicate that the second method consistently produces superior results across both datasets. FashionIQ CIRR R@10 R@50 R@1 R@5 R subset @1 This observation can be attributed to the higher quality of triplets generated through this selection approach compared to the others.Discussion on Prompt Templates. We can combine three prompt templates in 7 ways. For two or more combinations of templates, we obtain a corresponding number of modified texts for each triplet and randomly select one during training. As shown in Fig. 4 (e), we find that for CIRR, a mixture of the first two works best. For Fash-ionIQ, only the third works best. This indicates that in FashionIQ, more modified texts directly describe the target image. Zero-shot CIR is aimed at building a CIR model without requiring human-labeled triplets for training [29] . For comparison under the zero-shot setting, we introduce two advanced baselines for zero-shot CIR, PIC2WORD [29] and SEARLE [3] . Following these two baselines, we use CLIP ViT-L/14 as the backbone. Before training, we use the method described in Section 3.2 to generate a CIR dataset from an image dataset. Then contrastive learning with inbatch negative sampling is used for first-stage fine-tuning and the method described in Section 3.3 is used to scale negative examples for second-stage fine-tuning.Neither of these baselines uses an in-domain image dataset for training. Therefor, we also utilize images from the out-of-domain dataset Conceptual Caption (CC3M), comprised of 3.3 million imagecaption pairs from the Internet, to generate positive examples for a fair comparison with PIC2WORD. Specifically, we randomly select the 50k images in CC3M to construct the CIR dataset due to computing resource limitations. The number of 50k images is equal to 1.7% of that PIC2WORD used and 50% of that SEARLE used. We abbreviate our model trained with this setting as SPN-CC. As shown in Table 5 , SPN-CC gets the best results while using the least amount of images. This suggests that, given a collection of images out of the domain, our method can automatically construct appropriate triplets and train an acceptable model in the zero-shot setting. We also explore the setting of in-domain images, i.e., those in FashionIQ and CIRR, and abbreviate the model trained with this setting as SPN-IN. Under this setting, SPN-IN yields better results than SPN-CC using out-of-domain data. This suggests that if in the future we need to do a composed image retrieval task for a new scene but with few labeling costs, an accepted solution is to use our SPN method to automatically construct the positive examples within this scene and train a model from scratch.Fig. 5 indicates the retrieval cases of CLIP4CIR w/o and w SPN. The first example is selected from CIRR, and the second one is from FashionIQ. For both examples, we can find that using SPN allows us to learn more of the rarer concepts (e.g. "llama", "logo"), thus enhancing the base model. In the meantime, we can find that the base model has difficulty in retrieving the correct image when the reference and target images are very different (e.g., "panda" and "llama", "dress" and "tee"), and SPN narrows this gap. More examples can be found in supplementary materials. 
conclusion: 
 The Composed Image Retrieval (CIR) task uses a composed query to retrieve target images. While existing methods have achieved impressive results, limited labeled data and contrastive learning with in-batch negative sampling limit the performance of their methods. To address these problems, we first propose a data generation method using a multi-modal large language model to scale positives. We then propose a two-stage fine-tuning framework to scale negatives, introducing static representations of negatives in the second stage. These improvements are plug-and-play, enhancing existing CIR models without architecture changes. Extensive experiments show that we obtain state-of-the-art results on the FashionIQ and CIRR datasets. Moreover, our method could be applied to zero-shot composed image retrieval, offering a novel solution for unannotated CIR scenarios.  
query encoder: 
 current text-to-image retrieval and thus has become a popular research task in the multi-modal field. Previous research on CIR typically involves model architecture [5, 9, 33] and optimization objectives [2, 5, 23, 35] . The methods for the model architecture focus on better representation and fusion methods for texts and images.The contribution of the works in this aspect includes (1) introducing vision-language pre-trained models, like CLIP [28] , BLIP [18] , as the backbone [2, 5, 23, 35] and (2) designing the novel late fusion [5, 33, 35] or early fusion [2, 16, 23] modules to fuse the reference image and the modified text to obtain the single query representation. Therefore, the popular model architecture in CIR can be illustrated in Fig. 1 (a), which consists of a query encoder and a target image encoder. In practice, a collection of image candidates is first converted into image representations by the image encoder for rapid indexing. When the user gives a reference image and a modified text, they are forwarded to the query encoder to fusion and obtain the query representation. Finally, the query representation is computed with the dot product or cosine similarity with all representations of candidate images, and the image with the top similarity is considered the target image.The works for optimization objective [2, 35] focus on aligning the query representation with the target image representations. Advanced methods use contrastive learning [31] to align the representations of queries and images. The key to contrastive learning is to select correct and sufficient positives and negatives. The annotated triplets in the dataset, in the form of (reference image, modified text, target image), usually are regarded as positive examples, while negative examples are generated by replacing the target images with other ones in the mini-batch. However, as shown in Fig. 1(b) , there are two challenges with these works: (1) The number of manually annotated triplets (20K) is deficient, leading to a lack of sufficient positive examples for the model. As a comparison, in other tasks using contrastive learning like visual representation learning [10] , image-text retrieval [19, 27] , and image retrieval [26] , the number of positive examples is at least 60k; (2) Previous CIR tasks typically use in-batch negative sampling, with around 128 negative examples, while many successful works in contrastive learning use over 4k negative examples [7, 13, 18] . Existing works ignore these two problems at the data level, resulting in the inability of contrastive learning to fulfill its capabilities.Therefore, this work is based on a universal and simple motivation: to scale the number of positive and negative samples of the CIR task to the same scale as other tasks with contrastive learning. To construct more positives for CIR, we propose a novel data generation method based on the multi-modal Large Language Model (MLLM). Specifically, we design a four-step pipeline to automatically construct positive samples, which includes (1) caption generation with MLLM; (2) reference-target image pair matching; (3) modified text generation based on templates; and (4) positive example construction. With the help of our method, plenty of acceptable positive examples can be generated without any manual annotation, scaling the triplet number from 20k to 100k without the use of external datasets (Fig. 1(b) ). To introduce more negatives for CIR, we design a two-stage fine-tuning framework. Specifically, in the first stage, we follow previous works [2, 4, 23, 35] and use in-batch negative sampling to enable the model to learn initial representation space for CIR; while in the second stage, we initialize the model trained in the first stage and freeze the target image encoder, only fine-tuning the query encoder. The frozen target image encoder introduces a large number of static representations of negatives at once (Fig. 1(b) ), guiding query encoders to optimize representation space rapidly. Note that the second stage has only about 1/20 of the time overhead of the first stage and can be easily superimposed on existing advanced models in CIR.To verify the effectiveness of our method, we experiment extensively with both the full-supervised and zero-shot settings. For the full-supervised setting, we adopt our method in four advanced models in CIR with different backbones, achieving a 1%-6% performance improvement on the popular FashionIQ and CIRR datasets, reaching a new state-of-the-art. For the zero-shot setting, the model needs to be built without requiring human-labeled triplets for training. We apply our method to in-domain and out-of-domain image datasets to construct sufficient positives and negatives for CIR. With fewer image scales than the baselines, the superior performance of our method demonstrates the ease with which our method can be applied to low-resource scenarios.The contributions of our paper can be summarized as follows: • Extensive experiments and analysis under the full-supervised and zero-shot setting demonstrate the effectiveness and superiority of our proposed method, which achieves state-ofthe-art performance on both FashionIQ and CIRR datasets. 
 
 Appendix: In this work, we refer to the text in the CIR triplet as a "modified text", which is also referred to as a "modification sentence" or "modification text" in other works.Image dataset here could be Ω in the CIR dataset or any image dataset. 


Decision: The article is dedicated to Composed Image Retrieval (CIR), it explores the potential of generating more positives and negatives examples to improve CIR within a contrastive learning framework. Globally, the 4 reviewers opinions revolve around weak/border acceptance or rejection, because they oscillate between the fact that the main idea, augmenting data to improve the framework, is not original but that the experiments prove the effectiveness of the solution, that the plug-and-play module is relevant (making the proposal applicable to other state-of-the-art techniques without structural modification), as well as the study on complexity which is a strength of the proposal. Consequently, this work has no place in an oral presentation, but it has no major drawbacks and could be presented as a poster in order to discuss with the ACM community on its contributions to current CIR architectures with low complexity.

Reviews: review: paper_summary: paper_summary: This paper analyzes the origin of the retrieval performance of composed image retrieval and explores the potential of generating more positives and negatives for the existing CIR methods. To address the problem of lack of positives, the authors propose a data generation method to construct more triplets for CIR. To address the problem of lack of negatives, the authors design a plug-and-play two-stage fine-tuning method for CIR. The experiments on the general domain and fashion domain prove the effectiveness of the proposed methods.
strength_weakness: strength_weakness: + The idea of constructing more positives and negatives for CIR is reasonable, and has been validated by the ablation study.
+ The analysis of negative construction is interesting, and has been little noticed in existing work.
+ The extensive experiments prove the effectiveness of the proposed method.
limitations: limitations: -	The novelty is not enough for ACM MM. Although the idea of constructing more positives and negatives is reasonable, the proposed scaling positives and scaling negatives methods are the common methods for multimodal learning, and lack novelty.
-	It seems the size of generated positives and negatives are restricted by the existing dataset. The number of generated positives equals M, while the number of negatives is restricted by the size of the candidate set.
-	At the end of sec. 3.3, the authors just discussed the detailed time of training, while does not introduce the detailed machine configuration and training set. It is meaningless to discuss time cost without a detailed experimental setup. 
-	The new noun in the axis should be introduced in the caption, such as ‘SP’ and ‘SN’ in Fig. 1(b).
-	Some typos: in line 283, Fig, 2 -> Fig.~2; there is an unexpected line break in line 299. The left double quotation mark should be `` in latex.
paper_summary: paper_summary: This paper examines the challenge of composed image retrieval in the context of limited positive examples in existing datasets, which is often attributed to the high cost of annotation. To address this issue, this paper presents a data generation method for constructing CIR triples using a multimodal large language model. The method functions as a plug-and-play plug-in, enabling the incorporation of CIR triples into existing methods without necessitating any alterations to their original architectures. The experiments demonstrate that the proposed plug-in can effectively enhance the retrieval accuracy of existing models. Furthermore, the method demonstrates efficacy in composed image retrieval with zero samples. Nevertheless, there are still some shortcomings in the description of the method and the analysis of the experimental results.
strength_weakness: strength_weakness: a.This paper proposes a method for generating CIR positive examples data based on multimodal large language models. This method generates high-quality positive examples based on image datasets only, effectively expanding the model training data.
b.This paper proposes a two-stage plug-and-play module that can be applied to existing CIR models with minimal additional overheads, without necessitating any changes to the existing models' architectures.
c.This paper investigates the effectiveness of the proposed method for CIR tasks in two scenarios: fully supervised and zero-sample. Extensive experiments are conducted on several existing CIR models.
limitations: limitations: a.As this paper proposes a plug-in that is orthogonal to the existing methods, it should be explained in detail how the method proposed in this paper is applied to the existing CIR model. Currently, there is less description of this part in the text, which should be rectified. It is recommended that the experimental section be expanded to include more detailed implementation details.
b.With regard to Table 5, the authors present the experimental results for in-domain and out-of-domain data, but do not provide a comprehensive analysis of these results. A similar issue is evident in Section 4.3, where the majority of the author's space is dedicated to merely describing the outcomes of the experiments, with comparatively less attention devoted to analysing the underlying causes of the observed experimental phenomena.
paper_summary: paper_summary: The article is situated within the domain of CIR (Composed Image Retrieval), which focuses on retrieving target images using a query composed of both a reference image and accompanying text. 
The article addresses two challenges within the framework of contrastive learning. The lack of annotated triplets leads to fewer positive examples for the model, resulting in decreased performance. Additionally, the lack of negative examples for model training is also a problem. Their hypothesis is that best performance are obtained by methods using many counterexamples,
This papers proposes three Contributions:
    1. A data generation method to construct more positive examples based on images in the dataset.
    2. A two-step method to construct more negative examples. The first step updates the weights of both encoders (query and target encoders) to improve the model's ability to find target images, while the second step focuses on improving the representation of queries while keeping the representation of target images unchanged. 
    3. The article also presents a comprehensive ablation study where performances are evaluated across various hyperparameters. The best results match those of the state-of-the-art.
strength_weakness: strength_weakness: 1. The article is well-written, providing a clear and comprehensive overview of the current state-of-the-art to support their proposals. 
    2. The authors clearly identify their contributions, which are of significant interest to the scientific community. 
    3. The proposed ablation study is very interesting, highlighting the challenge of adjusting all hyperparameters based on the database context.
    4. The exhaustive experiment results effectively highlight the importance of the contributions made in the paper. 
    5. Complexity of the approach is also a strength of the paper. It reduces significantly the  computation time from 20 hours to 30 minutes.
limitations: limitations: The figures and tables references should be improved, and the terms presented in Figure 4 should be redefined to assist the reader.
paper_summary: paper_summary: In this paper, the authors focused on the task of composed image retrieval (CIR), where the query consists of a reference image and a modified text. The authors found that the existing CIR datasets lack positive examples and negative examples. Toward this end, the authors introduced a MLLM-based positive examples generation method and a two-stage fine-tuning strategy. The proposed framework can be plug-and-play to the existing CIR methods.
strength_weakness: strength_weakness: There are three strong points:
1.	The authors proposed a data generation method with MLLM to augment the CIR datasets.
2.	The designed two-stage plug-and-play framework can scale up negative examples efficiently.
3.	Experiments prove the effectiveness of the proposed method.
limitations: limitations: There are also some weak points:
1.	The primary contribution appears to be centered solely on expanding the scale of the data, which may be considered somewhat lacking.
2.	The modified texts within the generated triplets represent descriptions of the target image, potentially deviating from the original data format. Furthermore, other related works, such as LIMN+[1] and BLIP4CIR[2], aim to address the data scarcity issue. However, the manuscript lacks an analysis of the advantages of the proposed method compared to other data generation approaches.
[1] Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval. TPAMI 2024.
[2] Bi-directional Training for Composed Image Retrieval via Text Prompt Learning. WACV 2024.
3.	The strategy of scaling positive examples may inadvertently include more instances of the same target images, while the scaling negative strategy treats all other images in the dataset as negatives, potentially exacerbating false-negative issues.
4.	There are some typos:
a) There is a mistake on line 299.
b) The use of " " is not formal in line 555.