CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion
Authors: Geonmo Gu
Sanghyuk Chun
Wonjae Kim
HeeJae Jun
Yoohoon Kang
Sangdoo Yun
Venue: Accepted by TMLR ()

Abstract: This paper proposes a novel diffusion-based model, CompoDiff, for solving zero-shot Composed Image Retrieval (ZS-CIR) with latent diffusion. This paper also introduces a new synthetic dataset, named SynthTriplets18M, with 18.8 million reference images, conditions, and corresponding target image triplets to train CIR models. CompoDiff and SynthTriplets18M tackle the shortages of the previous CIR approaches, such as poor generalizability due to the small dataset scale and the limited types of conditions. CompoDiff not only achieves a new state-of-the-art on four ZS-CIR benchmarks, including FashionIQ, CIRR, CIRCO, and GeneCIS, but also enables a more versatile and controllable CIR by accepting various conditions, such as negative text, and image mask conditions. CompoDiff also shows the controllability of the condition strength between text and image queries and the trade-off between inference speed and performance, which are unavailable with existing CIR methods. The code and dataset samples are available at https://github.com/navervision/CompoDiff.

Published in Transactions on Machine Learning Research (07/2024)
CompoDiff: Versatile Composed Image Retrieval With Latent Diffusion
Geonmo Gu'
Sanghyuk Chun"
Wonjae Kim?, HeeJae Jun', Yoohoon Kang', Sangdoo Yun?
'NAVER Vision
NAVER AI Lab
Eqqual contribution
Reviewed on OpenReview:
Abstract
This paper proposes novel diffusion-based model, CompoDiff, for solving zero-shot Com- posed Image Retrieval (ZS-CIR) with latent diffusion. This paper also introduces new syn- thetic dataset, named Synth 18iM,ets18M, with 18.8 million reference images, conditions, and corresponding target image triplets to train CIR models. CompoDiff and Triplets81 tackle the shortages of the previous CIR approaches, such as poor generalizability due to the small dataset scale and the limited types of conditions. CompoDiff not only achieves new state-of-the-art on four ZS-CIR benchmarks, including FashionIQ,, CIRR, CIRCO, and GeneCIS, but also enables more versatile and controllable CIR by accepting various conditions, such as negative texl, and image mask conditions. CompoDiff also shows the controllability of the condition strength between text and image queries and the trade-off between inference speed and performance, which are unavailable with existing CIR mothods. The code and dataset are available at https: //gtthub.t//t////////g//ttebhh wwwaaaaaweeinnnonoooooaoaooieeoe
1 Introduction
Imagine a customer seeking a captivating cloth serendipitously found on social media but not the most appcaling materials and colors. In this scenario, the customer needs search engine that can process composed queries, e.g., the reference garment image along with text specifying the preferred material and color. This task has been recently formulated as Composed Image Retrieval (CIR). CIR systems offer the benefits of searching for visually similar items while providing a high degree of freedom to depict text queries as teett-t--maage retrieval. CIR can also improve the search quality by iteratively taking user feedback.
The existing CIR mothods address the problem by combining image and text fcatures using additional fusion models, e.g., 2; fusion iosioo(2z; (zig, ze) where zi; Zey 2ig are the larget image, conditioning text, and reference image features, respectively'. Although the fusion methods have shown great success, they have fundamental limitations. First, the fusion module is not flexible; it cannot handle versatile conditions beyond a limited ual one. For instance, a user might want to include a negative text that is not desired for the search (xer-) (e-g., an image + "with cherry blossom" "France", as in Fig. 1 (b)), indicate where (Feyz) the condition is applied (e.g., an image + "balloon" + indicator, as in Fig. 1 (c)), or construct a complex condition with a mixture of them. Furthermore, once the fusion model is trained, it will always produce the same 2, for the given zig and z, users. However, practical retrieval system needs to control the strength of conditions by its applications or control the level of serendipity. Second, they need pre-collected human- human-verified verified dataset of triplets Te, x.) consisting of a reference image (Iig), a text condition (æx), and the corresponding target image (æz). However, obtaining such triplets is costly and sometimes impossible; therefore, the existing CIR datasets are small-scale (e.g., 30K triplets for Fashion-TQ (Wu et al., 2021) and 36K triplets for CTRR (Tiu et al., 2021)), resulting in a lack of generalizability to other datasets.
Throughont this paper, we will use denote raw data and denote vector encoded from 2:

---

Published in Transactions on Machine Learning Research (07 (07/2024) /024)
A
A
Input x
Input x
" "With cherry blossom" "France"
"With cherry blossom" "
Text cond. Cr
Mixed text cond. Cy Cy- (b) Our CIR with mixed text conditions
(a) Standard Composed Image Retrieval (CIR)
Input Mask cond. Cur
Input
Mask cond. Car
"Balloon"
"Balloon" t
"Multtilee Balloon"
Text cond.
Mixed text cond. Cr-
(c) Our CIR with mask and text conditions
(d) Our CIR with mask and mixed text conditions
Figure 1:
Composed Image Retrieval (CIR) scenarios. (a) A standard CIR scenario. (b-d) Our versatile CIR scenarios with mixed conditions (c.g., negative text and mask). Results by CompoDiff on LAION-2B.
We aim to achieve a generalizable CIR model with diverse and versatile conditions by using latent diffusion. We treat the CTR task as a conditional image editing task on the latent space, i.e., = Edit(zig| le........ Our diffusion- based CIR model, named CompoDiff, can easily deal with versatile and complex conditions, benefiting from the flexibility of the latent diffusion model (Rombach et al., 2022) and the classifier-free guidance (Ho & Salimans, 2022). Note that the purpose of image editing tasks and CIR tasks are different; image editing model aims find any image satísfying the changes due to the instruction, while CIR model aims to find generalized concept of the given composed queries. For example, assume picture of dog on a grass field and an instruction "change the dog to cat". Here, even if we assume a perfect image generator can generate an image of a cat on a grass field, it could not achieve a good retrieval performance because it cannot generate all possible cat images in the database. In other words, the purpose of image editing "precision", while the purpose of image retrieval "recall"; it makes the difference between image editing and retrieval. In our experiments, we support this claim by showing directly using an editing model performs much worse than other CIR methods. Therefore, we need a specialized model for CIR. rather than directly using the image editing model. In this paper, tackle the problem by using latent diffusion model instead of an leaee-level diffusion model and introducing various retrieval-specific conditions, such as mask conditions. We train a latent diffusion model that translates the embedding of the reference image (Zin) into the embedding of the target image (2:) guided by the embedding of the given text condition (2z). As shown in Fig. 1, CofpoDiff can handle various conditions, which not possible with the standard CTR scenario with the limited text condition Tey- Although our method has an advantage over the existing fusion-basod CIR methods in terms of versatility, CompoDiff also needs to be trained with triplet dalasets where the scale of the existing CIR datasets is extremely small.
We address the datasct scale issue by synthesizing vast set of high-quality 18.8M triplets of (Kin, Ze, TT). Our approach is fully automated without human verification; hence, it is scalable even to 18.8M. We follow InstructPix2Pix (IP2P) (Brooks et al., 2022) for synthesizing triplets, while our dataset contains x4 more triplets and x12.5 more keywords (e.g., objects, background details, or textures) than IP2P. Our massive dataset, named SynthTriplets8MM. is over 500 times larger than existing CIR datasets and covers a diverse and extensive range of conditioning cases, resulting in notable performance improvement for any
2

---

Published in Transactions on Machine Learning Research (07/2024)
CIR model. For example, ARTEMIS (Delmas et al., 2022) trained exclusively with Syuth Tripless11 shows outperforming zero-shot performance even than its Fashion]Q-trainedl counterpart (40.6 vs. 38.2).
To show the generalizability of the models, we evaluate the models on the "zero-shot" " (ZS) CIR scenario using four CIR benchmarks: FashionIQ (Wu et al., 2021), CIRR (Lin et al., 2021), CIRCO (Baldrati et al., 2023), and GeneCTS (Vaze et al., 2023); i.e., we report the retrieval results by the models trained on our SynthTrripeesSSMM and a large-scale image-text paired dataset without access to the target triplet datasets. In all experiments, CompoDiff achieves the best zero-shot performances with significant gaps (See Table 3). Moreover, we observe that the based approaches solely trained on Synth Triplesti1 (e.g., Combiner (Baldrati et al., 2022)) show comparable or outperforming zero-shot CIR performances compared to the previous SOTA zero-shot CIR methods, e.9,, Pic2Word (Saito et al., 2023), SEARLE (Baldrati et al., 2023) and LinCIR (Gu et al., 2024). Furthermore, we qualitatively observe that the retrieval results of CompoDiff are semantically better than previous zero-shot CTR methods, such as Pic2Word, on a large-scale image database, e.g., LAION-2B (Schuhmann al., 2022a).
Another notable advantage of CompoDiff is the controllability of various conditions during inference, which is inherited from the nature of diffusson models. Users can adjust the weight of conditions to make the model focus on their preference. USeTS can also manipulate randomness to vary the degree of serendipity. In addition, CompoDiff can control the speed of inference with minimal sacrifice in retrieval performance, accomplished by adjusting the number of sampling steps in the diffusson model. As a result, CompoDiff can be deployed in various scenarios with different computational budgets. All of these controllability features are achievable by controlling the inference parameters of classifier free guidance without any model training.
Our contributions are first to show the effectiveness of diffusion models non-generative tasks, such multi- modal retrieval and data generation. More specifically: (1) We propose diffusion--asse CIR method, named CompoDiff. CompoDiff can handle various conditions, such as the negative text condition, while the previous methods cannot. We also can control the strength of conditions, e.g., more focusing on the reference image while the changes by the text should be small. (2) We generate Synth 1ri, a diverse and massive synthetic dataset of 18M triplets that can make CTR models achieve zero-shot (ZS) generalizability. Our data generation process is fully automated and easily scalable. (3) Our experiments support the effectiveness of CompoDiff quantitatively (significant ZS- CIR performances on FashionIQ, CIRR, CIRCO, and GeneCIS datasets) and qualitatively (ZS-CIR on billion-scale DB, or image decoding using unCLIP generator).
2 Related Works
Composed image retrieval. The mainstream CIR models have focused on multi-modal fusion methods, which combine image and text features extracted from separate visual and text encoders, as in Fig. 2 (a). For example, Vo et al. (2019) and Yu et al. (2020) used CNN and RNN, and Chen et al. (2022) and Anwaar et al. (2021) used CNN and Transformer. More recent methods address CIR tasks by leveraging external knowledge from models pre-trained on large-scale datasets such as CLIP (Radford et al., 2021). For example, Combiner (Baldrati et al., 2022) fine-tunes the CLIP text encoder the triplet datasct to satisfy the relationship of 2; zig + ze, and then trains Combiner module on top of the encoders. However, these fusion hot still require expensive pre-collected and human-verified triplets of the target domain.
To solve this problem, recent studies aim to solve CIR tasks in a zero-shot manner. Inversion-based zero- shot CTR methods, such as Pic2Word (Saito et al., 2023), SEARLE (Baldrati et al., 2023) and LinCIR (Gu et al., 2024), tackle the problem through text-to-image retrioval tasks where the input image is projected into the condition lext- Fig. 2 (b). Note that our zero-shol CIR scenario is slightly different from theirs; our zero-shot CIR. denotes that the models are not trained on the target triplet datasets, but trained on our synthetic datasot, SynthTriplets18M. and image-text paired dataset, e.g., LATON-2B (Schuhmann et al., 2022a). Meanwhile, Saito et al. (2023), Baldrati et al. (2023) and Gu et al. (2024) use the term zero-shot when the CTR models are trained without a triplet dataset.
All the existing CIR models only focus on text conditions (zo7) (e.g., Fig. (a)) and have difficulties in handling versatile scenarios (e.g., Fig. 1 (b-d)) with a lack of the controllability. On the other hand, our method enables multiple various conditions and controllabilities with strong zero-shot performances by
3

---

Published in Transactions on Machine Learning Research (07/2024)
Diffusion process
Minimize
Fusion module
Projection module
with denoising Transformer
Minimize
Zip.
Zin
(composed)
Zz
Zin
CFG
Image Encoder
Image Encoder
Text Encoder
Image Encoder
Text Encoder
Image Encoder
mask condition
Text Encoder
Minimize
"With cherry blossom"
"A photo of ( with cherry blossom"
7,
"With
cherry
blossom"
Target
Reference Image
Text Condition
Reference
Text Condition
Reference
Text
Image
Image
Image
Condition
(a) Fusion-based approaches
(b) Inversion-based approaches
(c) Our editing-based approach
Figure 2: Comparisons of CIR methods. (a) Fusson-based methods (e-g., ARTEMIS (Delmas et al., 2022) and
Combiner (Baldrati et al., 2022)) make a fused feature from image feature Zi and text feature Ze. (b) Inversion- based methods (e.g., Pic2Word (Saito et al., 2023), SEARLE (Baldrati al., 2023) and LinCIR (Gu al., 2024)) project 4 into the text space, then perform text-to-imag, to-igag retrieval.
(c) We apply diffusion process to a with classifier-free guidance by additional conditions. (b) and (c) use frozen encoders, and (a) usually tunes the encoders.
employing (1) a latent diffusion model (Rombach et al., 2022) with classifter- guidance (Ho & Salimans, 2022) and (2) a massive high-quality synthetic dataset, SyuthTTriplets//
Concurrent with work, CoVR (Ventura et al., 2024), VDG (Jang al., 2024), and MagicLens (Zhang et al., 2024) consider constructing training triplets from the existing image datasets, namely, collecting image pairs and synthesizing the modification texts using LLMss Although they showed empirically good perfor- mances, their approaches need real images where it is often very difficult to generate plausible modification instructions between two real images. Meanwhile, our synthetic dataset more controllable than their datasets. Namely, we can exactly guide the visual difference between two images with the given instructions.
Dataset creation with diff'usio models. A conventional data collection process for (Eig Ter vi) is two- staged: collecting candidate reference-taree target image pairs and manually annotating the modification sentences by human annotators. For example, FashionIQ (Wu et al., 2021) collects the candidate pairs from the same item category (e.g., shirt, dress, and top) and manually annotates the relative captions by crowd workers. CIRR (Liu et al., 2021) gathers the candidate pairs from real-life images from NLVR² (Suhr et al., 2018). The triplet collection process inevitably becomes expensive, making it difficult to scale CIR datasets. We mitigate this problem by gencrating a massive synthetic dataset instead of relying on human annotators.
Recently, there have been altempts to generale synthetic data to improve model performance (Brooks el al., 2022; Nair et al., 2022; Shipard et al., 2023) by exploiting the powerful generation capabilities of diffusion models. In particular, Brooks et al. (2022) proposes generation process for (Iin Te; 2x)) to train an image editing model. We scale up the dataset synthesis process of Brooks et al. (2022) from 1M triplets to 18.8M. Our generation process contains more diverse triplets by applying the object-level modification process, resulting in better CIR performances on the same 1M scale and easily scalable to larger number of samples.
3 CompoDiff: Composed Image Retrieval with Latent Diffusion
This section introduces Composed Image Retrieval with Latent Diffusion (CompoDifI) which employs a diffusion process in the frozen CLIP latent feature space with classifier-free guidance. Unlike previous latent diffusion models, we use a Transformer-basod denoiser. We train our model with various tasks, such as text- text-to-image (T2I) generation, masked T2I and rrpelet-based generation to handle various conditions (c.g., negative text mask), while the previous CIR methods only limit beyond the positive text instruction.
4

---

Published in Transactions on Machine Learning Research (07/2024)
Training: Stage (with LAION-2B)
Training: Stage 2 (with LAION-2B and Syath Triplet18M)
CLIP Img Enc
Dittusion Process (aucligg noise T times)
CP Img. Enc
Diffasion Process (adiling noise I times)
Input Image
Minimize of z, by Eq (1)
Repeat T times
Input Image
CLIP Img Enc
Minimize il 5 z,ll by (2) (3)
Ropest T times
100
Denoising Transformer Es
CLIP Txt Eine
Denoising Transformer
Ref. Image
AAcriiude shring
4,4%
by
F Mask cond. Cv
MLP
Concat
Text condition
"Cheser the chictaine Brd iouites
Txt Eme
CLIPP,
Figure 3: Training overview. Stage 1 trained on LAION-2B with Eq. (1). For stage 2, we alternatively update Denoising Transformer eæ on LAION-2B with Eq. (1) and 2 and TynthTripless@MM with Eq. (3).
3.1 Preliminary: Diffusion model
Given a data sample from the distribution (x ~ p(z)) diffusion model (DM) defines a Markov chain of latent variables 21,.... 2T as: ((=(==-1) = N(= Vo,2t-1, is approximated to Gaussian distribution with a diagonal covariance, and (2) if the size of chain T is sulficiently large, then æ7 will be approximated to NN /). Namely, DMs are general probabilistic method that maps an input distribution p(z) to a normal distribution I), without any assumption on the input data x. From this observation, Rombach et al. (2022) proposed to learn DMs on a latent space (i.e., latent diffusion), which brings a huge improvement in efficiency. In practice, we train a denoising module c0(2,, I) (d.e,, 2-1 Eo Eo(2,))) t)) where the timestamp t conditioned by employing time embeddings et.
og) It is proven that (1) the posterior Ia-als
The recent DMs classifier-free guidance (CFG) (Ho & Salimans, 2022) for training conditional DMs without pre-trained classifier (Dhariwal & Nichol, 2021). In this paper, we provide various conditions via CFG to take two advantages. First, we can easily control the intensity of the condition. Second, we can extend the conditions beyond a single text condition, e.g., negative text condition or mask condition.
CompoDiff has two distinct differences from the previous latent diffusion approaches, such as StableDiffusion (SD) (Rombach et al., 2022). First,, SD performs the diffusion process on 64 x 64-dimensional VQ-GAN (Esser et al., 2021) latent space. Meanwhile, the diffusion process of CompoDiff performed on the CLIP latent embedding space. Therefore, the edited features by CompoDiff can be directly used for retrieval on the CLIP space, while SD cannot. The Dall-e2 prior (Ramesh et al., 2022) performs the diffussin process on the CLIP embedding space, but CompoDiff takes inputs and conditions differently, as described below.
As the second contribution, CompoDiff uses a different architecture for the de-noising module. While SD and other DM models are based on U-Net structure (Ronneberger et al., 2015), CompoDiff uses a simple Transformer (Vaswani al., 2017). Morcover, CompoDiff is designed handle multiple conditions, such as masked conditions, and a triplet relationship. SD cannot handle the localized condition and is designed for pairwise relationship (e.g., text-to-image generation). CompoDiff also handles the condition different from Dalle-2 prior. Dalle-2 prior handles conditions as the input of the diffusion model, but our CompoDiff diffusion Transformer takes the conditions via the cross-attention mechanism. This design choice makes the inference speed of CompoDiff faster. If the conditions are concatenated to the input tokens, the inference speed will be highly degenerated (Song et al., 2022). Table 8 shows that the structure taking all conditions as the concatenated input (e.g., Dallo-2 prior-like) is three times slower than our cross-attention approach.
3.2 Training
CompoDiff uses a two-stage training strategy as illustrated Fig. 3. In stage 1, we train a text-to-image latent diffusion model on LAION-2B. In stage 2, we fine-tune the model on our synthetic triplet dataset, SynthTTriplets1MMM MM, and LAION-2B. Below, we describe the details of each stage.
5

---

Published in Transactions on Machine Learning Research (07/2024)
In stage 1, we train a transformer decoder to convert CLIP textual embeddings into CLIP visual embeddings. This stage is similar to training the Dalle-2 prior, but our model takes only two tokens; a noised CLIP image embedding and a diffusion timestep embedding, The Dalle-2 prior model is computationally inefficient because it also takes 77 encoded CLIP text embeddings as an input. However, CompoDiff uses the encoded text embeddings as conditions through cross-allention mechanismss, which speeds up the process by a factor of three while maintaining similar performance (See Section 5.7). Instead of using the noise prediction of Ho et al. (2020), we train the transformer decoder to predict the denoised z; directly due to the stability.
Now, we introduce the objective of the first stage with CLIP image embeddings of an input image 2,, encoded CLIP text embeddings for text condition Ber: and the denoising Transformer co:
Latagel = E, - ,tEc_))
(1)
During training, we randomly drop the text condition by replacing &er with a null text embedding Der in order to induce CFG. We use the empty text CL.IP embedding (**) for the null embedding.
In stage 2, we incorporate condition embeddings, injected by cross-attention, into CLIP text embeddings, along with CLIP reference image visual embeddings and mask embeddings (See Fig. 3). We fine-tune the model with three different tasks: a conversion task that converts textual embeddings into visual embeddings (Eq. (1)), mask-based conversion task (Eq. (2)), and the triplet-based CIR task (Eq. (3)). The first two tasks are trained on LAION-2B and the last one is trained on
The mask-based conversion task learns a diffusion process that recovers the full image embedding from a masked image embedding. As we do not have mask annotations, we extract masks using a zero-shot text-conditioned segmentation model, CL.IPSeg (Tüddecke & Ecker, 2022). We use the nouns of the given caption for the CLIPSeg conditions. Then, we add a Gaussian random noise to the mask region of the image and extract 2i.masked. We also introduce mask embedding zeu by projecting a 64x64 resized mask to the CLIP embedding dimension using a MLP. Now, the mask-based conversion task is defined as follows:
c Lstagr2_maskee masi_maased______ conversion E, eØ ma U/zez: Ri.masked; Fexu )
(2)
Finally, we introduce the triplet-based training objective to solve CIR tasks on Tyith 18M as follows:
(1) Lstage2 triple = 1x-11, - -e c0(z s t/ Zins Rext )l
(3)
where zig is a reference image feature and Ziz is a modified target image feature.
We update the model by randomly using one of Eq. (1), Eq. (2), and Eq. (3) with the proportions 30,, 30%, 40%. As stage 1, the stage 2 conditions are randomly dropped except for the mask conditions. We use an all-zero mask condition for the tasks that do not use a mask condition. When we drop the image condition of Eq. (3), we replace 2ig with the mull image feature, all zero vector.
Our two-staged training strategy is not mandatory procedure, but it brings better performances. Con- ceptually, stage 1 is a pre-training of diffusion model (DM) with pairwise image-text relationships. The stage 2 is for a finc-tuning process using triplet relationships. Lastly, the alternative optimization strategy for stage 2 for helping optimization, not for resolving the instability. As shown in Table 8, CompoDiff can be trained only with Eq. (3), but adding more objective functions makes the final performances stronger. The two-staged training strategy is also widely used for the other CIR methods, such as SEARLE (Baldrati et al., 2023) or Combiner (Baldrati et al., 2022). In terms of diffusion model fine-tuning, we argue that our method is not specifically complex than other fine-tuning methods, such as ControlNet (Zhang el al., 2023).
3.3 Inference
Fig. 4 shows the overview. Given a reference image feature zige text condition feature Bey; and mask embedding Zears we apply a denoising diffusion process based on CFG (Ho & Salimans, 2022) as follows:
(o(2)²) ,1/zor Piry Foxe) = ca(2)))
(t) , Me,or Dins Fess ) + wr (1) (( Pia Rexr) cg(2(²) (1) ,'eerr Dini Fex)))
+ +wr(e0(z)") (1) ./zerr Zego Rese) - co(2!)) (1)) ,110 cr Zoge Fext ))
6

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Repeat times
CLIP Img Enc
Denoising Transformer &
Retricval DB (CLIP visual features)
Inputxx
Txt Enc (CLIP, T5,
CFG null condition by Eq (4)
"Balloon" "
Concut
Text cond.
Txt Enc
MLP
(CLIP, TS....
(if mask is not given. all zero vector is used)
"Multiple Balloon" (if specified, "*)
Mask cond.
Negative Text cond. 6t
Retrieved images
Figure 4: Inference overview. Using the denoising transformer Eg trained by Stage and 2 (Fig. 3), we perform composed image retrieval (CIR). We use the classifier-free guidance by Eq. (4) transform the input reference image to the target image feature, and perform image-to to- io-image retrieval on the retrieval DB.
where 2 denotes null embeddings, í.e., the empty text ("") CLIP textual embedding for the text null embedding and an all-zero vector for the image mull embedding. wy and Wr weights for controlling the effect of image query or text query, respectively. We can control the degree of the visual similarity (wy) the text instruction (wr) by simply adjusting the weight values without any training. One of the advantages of Eq. (4) is the ability to handle various conditions at the same time. When using negative text, we simply replace 2er with the CLIP text embeddings Cr- for the negative text. These advantages are not available by the previous CTR methods without additional training.
Tenk waggtt
Another advantage of CFG the controllability of the queries without training, e.g., it allows to control the degree of focus image features to preserve the visual similarity with the reference by simply adjusting the weights the image and text weights wy and wr from LAION-2B in Fig. 5. By increasing wy, CompoDiff behaves more like an mmage-to-image retrieval model. Increasing WT, on the other hand, makes CompoDiff focus more on the "pencil sketch" text condition. We use (wy, wr) (1.5, 7.5) for our experiments. The full retrieval performance by varying wy and wr is shown in Fig. 10.
10
wr. We show the top-1 retrieved item by varying
200
As our model is based on a diffusion process, we can easily control the balance between the inference time and the retrieval quality of the mod- ified feature by varying step size. In practice, we set the step size to or 10, which shows the best trade-off.
Query:
Figure 5: Inference condition con- trol by varying wr, ur in Eq. (1).
4 SynthTriplets18MM Massive High-Qualit, Synthesized Dataset
CIR requires a dataset of triplets (Eig Tey ar) of a reference image (Iin), a condition (zz), and the cor- responding target image (a:). Instead of collecting dataset by humans, we propose automatically generate massive triplets by using generative mod- els. We follow the main idea of Instuct Pix2Pix (IP2P) (Brooks et al., 2022). First, we generate (Eegy Te, 2t) where Ftg is a reference caption, & is a modification instruction text, and o, is the caption modified by Te. We use two strategies to generate (Zin Ze, 2t): (1) We collect massive captions from the existing caption datasets and generate the mod-
IP2P SyuthiTripletsIMM
(T Tee Ze) (before filtering)
452k
500M
(Ei Bay (after filtering)
313k
60M
Unique object terms
47,345
586,369
(Zing Te, (Keyword-based)
11.4M
(Nin, Te, 21) (LLM-based)
1M
7.4M
(Kin, Re, Weywi) (Total)
1M
18.8M
Dataset statistics. (Fe Te, 22) denotes the triplet of captions, i.e., (original caption, instruction, and modified caption), and (Zig, Te, 21) denotes the C/R triplet of (original image, instruction, and modified image).
Table 1:

---

Published in Transactions on Machine Learning Research (07 (024))
Keyword-based diverse caption generation (Sec A.D)
LLM-based caption generation (See A.2)
( Extract nouns from the captions using POS lagger
(ii) Make the "Keyword" list
"Ippar suteroslar of you pen!
from extracted nouns
Issrsencr lage grizaly
after frequency filtering
Editd: watorestor of huge grizil
OPT-6.78
POS Tagger
Exangle ception fro
(I) Fine-tune LLM on Instract Pix 450k Captions
Caption datasets
Pre-extracted keywords
(1) Sample caption from caption datasets
"Aiper: irillass loke black and videe festnctionn Add moos
OPT-6. 7B
(2) Detect a keyword from the caption ("strawberry")
Fidted: ontliwn vvir with suse Mark willv
Replace strawberry pal choi based on their embedding similarity
Generating triplct of N, using the fine-tuned LLM
"Add moon"
Modification instruction
"aistrawberrylee Jort is on top of pastry"
"alpuk choi jart is top of pustry"
"wrilium lake black and white"
"triltium lake with a moon black and white" Modified Caption
Orignal Caption X,
Modified Caption X,
Orignal Caption x,
(SD 1.5, SD 2.0, S.I Anime) Prompt2Prompt
Text-to-image generation using captions (Sec A.3)
Pre defined keyword templates
"convert strawberry pak choi"
"Add moon" "
Image query Ri Modified image
Modification instruction
Image query x.,
Modified image x,
Modification inst.
Figure 6: Overview of the generation process for SynthTriplets1SM. (Zig Ze, X.) from (Ieg Bey VT).
ified captions by replacing the keywords the reference caption (Section 4.1). (2) We fine-tune a large language model, OPT-6.7B (Zhang al., 2022), on the generated caption triplets from Brooks al. (2022) (Section 4.2). After generating massive triplets of (ee we generate images from the caption triplets using StableDiffusion (SD) and Prompt-to-Prompe Hertz al. (2022) following IP2P (Section 4.3). We employ based filtering to ensure high-quality triplets (Section 4.4). The entire generation process is illustrated in Fig. 6.
Compared to manual dataset collections (Wu et al., 2021; Liu et al., 2021), our approach can easily generate more diverse triplets even if a triplet rarely occurs reality (See the examples in Fig. 6). For example, Fash- ionIQ only contains text instructions for fashion items (e.g., "is blue and has stripes"), but SynthTriplets18M contains more generic and various instructions. Furthermore, manually annotated instructions can be in- consistent and noisy due to the inherent noisiness of crowdsourcing. For example, CIRR has instructions with no information (e.g., "same environment different species"), or instructions ions which ignore the original information (e.g., "the target photo is of a lighter brown dog walking in white gravel along a wire and wooden fence"). On the other hand, Synth Trippets1MM instructions only have modification information. Hence, a model trained on our CIR triplets can learn meaningful and generalizable representations, showing great ZS-CIR performances. Compared to the synthetic dataset of TP2P, our generation process is more scalable due the keyword-base based diverse caption generation process: Our caption triplets are synthesized based on keywords, SynthTriplets
covers more diverse keywords than IP2P (47k vs. 586k as shown in Table 1). As result, Synth Triplets 8M contains more massive triplets (1M vs. 18M), and CIR models trained on SynthTripletsi 8M achieve better CIR performances even in the same dataset scale (1M).
4.1 Keyword-base based diverse caption generation
As the first approach generating caption triplcts, we collect captions from the existing caption datascts and modify the captions by replacing the object terms in the captions, e.g., ("a strawberry tart ...", "covert strawberry to pak choi", "a pak choi tart is ...") in Fig. 6. For the caption dataset, We the captions from
8

---

Published in Transactions on Machine Learning Research (07/2024)
Templates to change 8(source) to S(target)
"replace $(source) with $(target)"
"suhstitute #(target] $(sourca)"
"change ((sourea) to 8(target) "atee $(source) to #(target)" "smap $(souren) for 8(targat)"
"B(target)"
"((sounce) is removed and #(target) takes siss place" " "modify ((soune)) become #[target)"
"apply #(hargat)"
"convert 5(source) to $(target?"
"customise M(source) to become #(target)"
"redesign #(soucce? as #(target)" "turn $(source) into f(target)" "adapt $(sounce) to fit ((target)" "choose $(target) instend"
"replace S(source) with 5(sarget)"
"change $(souree) to match $(target)"
"apdate ((zource) to #(target)"
"S(target) is introduced after S(ssunce) removed" "B(target) is added in place of $(sourca)" option after
"substitute 8[target] for $(sourca) "alter ((ooroee match 8(target)" "upgrade Hssorce) to $(target)"
"#(target) is introduced as the "f(source) is removed and #(target) is added" "ffocuree] is remooed and #[target) is introduund" "g(target) is added "S(target) is the new option avatlable"
"S(tanget) is the "Stturget) is the new selection" "#ftarget) is new option" "use S(target) from now on"
choice"
"amend 8(sourea] to fit 8(targes)"
"opt for $(larget)" S(source) is removed"
replacement for S(svurce)"
"remodel $(source) into $(target)"
"add $(target)"
"S(target) is added after S(sourea) is removed" "Øftarget) is introduced after 8(sorre)) is retired"
"reeamp #(source) into $(target)"
"I it is #[target)" "S(target) is the updated option" "S(target) is the updated choice"
"enchange 8(acuree] with 8(tergat)" "yoneform $(avurce) into $(target)" "S(target) is the updated version'
"tweak #(source) become #(target)" "S(source) is replaced with S(target)"
Table 2: The full 48 keyword converting templates.
Ratio of samples having tokens less than
Cumulative ve Population of Instructions
Porcomage of cumuiative camples 1N1
100
Porcentage of cumn samples
90.21e
80
80
60
54.7%
40
40
Ours
Fashion
20-
20-
CRCO
GeneCS
0
10
15
20
25
30
35
40
10
20
30
40
50
60
Number of tokens
Number of tokers
Figure 7: Statistics of SynthTTrplets18MM instruc-
Figure 8: Statistics of instructions of the CIR datasets. We show the population of instruction cap- tions (e.g., "change A to B") by the number of tokens. We include captions having larger than 60 tokens in "60".
tions. We show the population of our instruction cap- tions (Rey) by the number of tokens per caption. We include captions having larger than 40 tokens in "40+".
CΟYO 700M (Byeon et al., 2022), StableDiffusion Prompts? of StableDiffusion better), T.TON-22Ben--asthheiie (a subset of LATON-5B (Schuhmann et al., 2022a)) and LAION-COCO datascts (Schuhmann et al., 2022b) (synthetic captions for LAION-5B subsets with COCO style captions (Chen et al., 2015). LAION-COCO less uses proper nouns than the real web texts).
prompts that make the quality
We extract the object terms from the captions using the part-of-speech (POS) tagger provided by Spacy 3. After frequency filtering, we have 586k unique object terms (Table 1). To make the caption triplet Fey we replace the object term of each caption with other similar keywords by using the CLIP similarity score. More specifically, we extract the textual feature of keywords using the CLIP ViT-L/ 14 text encoder (Radford et al., 2021), and we choose an alternative keyword from keywords with a CLIP similarity between 0.3 and 0.7. By converting the original object to a similar object, we have caption pairs of (Itn zi).
Using the caption pair (T), we generate the modification instruction text ズer based on a randomly chosen template from 48 pre-defined templates shown in Table 2. After this process, we have the triplet of (Ziny Ze. 21). We generate x30M caption triplets by the keyword- method.
4.2 Amplifying InstructPix2Pix (IP2P) triplets by LLM
We also re-use the generated ( .x)) by IP2P. We amplify the number of IP2P triplets by applying the efficient LoRA fine-tuning (Hu et al., 2021) to OPT-6.7B (Zhang et al., 2022) on the generated 452k caption triplets provided by Brooks et al. (2022). Using the fine-tuned OPT, we generate =30M caption triplets.
9

---

Published in Transactions on Machine Learning Research (07/2024)
4.3 Triplet generation from caption triplets
We generate 60M caption triplets (ZiR: Te: zt) by the keyword- based generation process (Section 4.1) and the LLM-based generation process (Section 4.2) (See Section 4.5 for the statistics of the captions). We generate images for Tta (original caption) and 2t (modified caption) using state-of-the-art text-to-image generation models, such as StableDiiftssin (SD) 1.5, 2.0, 2.1, and SD Anime. Following Brooks et al. (2022), we apply Prompt-to-Prompl (Hertz et al., 2022), which aims to generate similar images while keeping the identity of the original image (e.g., the examples in Fig. 6). As a result, and are generated by 2ig and 2t, respectively). While IP2P generates the samples only using SD 1.5, our generation process uses multiple DMs, for more diverse images not biased towards specific model.
generate 60M (Fāng Te; (Zer is given; Zin
4.4 CLiP-based filtering
Our generation process can include low-quality triplets, e.g., broken images or non-related image-text pairs. To prevent the issue, we apply filtering process following Brooks et al. (2022) to remove the low-quality (Diny Te, 24).. First, we filter the generated images for image-to-image CLIP threshold of 0.70 (between Fig and v;) to ensure that the images are not too different, an image-caption CLIP threshold of 0.2 to ensure that the images correspond to their captions (f.e., between Ita and Zin, and between It and æ;), and a directional CLIP similarity (Gal et al., 2022) of 0.2 (Ldirection := 1 sim( z;) sim(x 2x), where sim(-) is the CLIP similarity) to ensure that the change in before/after captions correspond with the change in before/ refore/after after images. For keyword-basee based data generation, we filter out for a keyword-image CLIP threshold of 0.20 to ensure that images contain the keyword (e.g., image-text CLIP similarity between the strawberry tart image and the keyword "strawberry" in Fig. 6). For instruction- based data generation, we filter out for instruction-modifee image CLIP threshold of 0.20 to ensure consistency with the given instructions.
The filtering process prevents low-quality triplets, such as broken images. For example, if a modified caption does not make sense to generate a corresponding image (e-g., changing "apples are on the tree" to "apples are on the thunderstorm "), then StableDiffusion and Prompt2Prompt will not be able to generate proper images for the modified caption. These images will be filtered out by CLIP similarity because we measure the similarity between the generated images is sufficiently high, and the broken images will have low similarities with clean images. Similarly, if the original caption is not suitable for generating corresponding image (e.g., "invisible strawberry"), then it will fail to pass the keyword-inage CLIP filtering.
After the filtering, we have 11.4M (Mir) Te, from the keyword-based generated captions and 7.4M (Figy Te, Te, mi) from the L.M--based generated captions. It implies that the fidelity of our keyword-based method higher than OPT fine-tuning in terms of T21 generation. As a result, Synth contains 18.8M synthetic (oin Te; 24). Examples of our dataset are shown in Fig. 9.
4.5 Dataset Statistics
We show the statistics of our generated caption dataset (i.e., before T2I generation, Iin and 22). We use the CLIP tokenizer to measure the statistics of the captions. Fig. 7 shows the cumulative ratio of captions with tokens less than X. About half of the captions have less than 13 tokens, and 90% of the captions have less than 20 tokens. Only 0.8% of the captions have more than 40 tokens.
We also compare SynthTTriplts1SMM. FashionlQ, CIRR, CIRCO, and GeneCIS in the token statistics of instructions (í.., xz). Fig. 8 shows that the instruction statistics vary across different datasets. We presume that this is why the ZS-CIR still has difficulty purpernrgi the task-specific supervised CIR methods.
httppp//s//uggggrrrerrsscssassas 9nttps://spacy.iio
10

---

Published in Transactions on Machine Learning Research (07 (07/2024)
"make the landscape a cityscape"
"make the garden swamp"
NUTTER UNIRU Endtavinn Drandia SONITTTAARA AA ABBIE Kiags, Arttorond
poll
"change the summer"
"make spring"
Tophongrac EOUN randE thehe mud "kaaciarrrrrdee"
'ORETN CONS NERNOLL OF NO
"4k high resolution images is the new choice"
"make it watercolor"
"vinter ramber by sengey rychionn"
"wetser thanderstoru by rychiin
"prrson waterccngh sear waterfal"
"rlng on watermaff noar waterfol"
"make tlunderstorm"
"have the person be dog"
Figure 9: Examples of Tynthtripeess88N We show examples (Zans Ke, xd), i.e., foriginal image, modification instruction, and modified image), as well the generation prompt for Fig and xt.
11

---

Published in Transactions on Machine Learning Research (07/ (2242
5 Experiments
5.1 Implementation details
Encoders. We use three different CLIP models for image encoder (Fig. 4 "CLIP Ing Enc"), the official CLIP ResNet-50 and ViT-I/14 (Radford et al., 2021), and CLIP ViT-G/14 by OpenCIII (Tharco et al., 2021), whose feature dimensions are 768, 1024, and 1280, respectively. Beyond the backbone size, we observe the choice of the text condition encoder is also important (Fig. 4 "Txt Enc"). As shown Balaji et al. (2022), using a text-oriented model such as T5 (Raffel et al., 2020) in addition to the CLIP textual encoder results in improved performance of text-to-imanee image generation models. Motivated by this observation, we also use both the CLIP (exlual encoder and the language-oriented encoder for small image encoder (i.e., CLIP ViT-L/14), We also observed the positive effect of the text-oriented model and experiment results showed that T5-XL, which has 3B parameters, could improve the performance by a large margin in the overall cvaluation metrics.
Denoiser. We use simple Transformer architecture for the denoising procedure, instead of the denoising U-Net (Rombach et al., 2022). We empirically observe that our transformer architecture performs slightly better than the U-Net architecture, but is much simpler. We use the multi-head self-attention blocks as the original Transformer (Vaswani et al., 2017). We set the depth, the number of heads, and the dimensionality of each head to 12, 16, and 64, respectively. The hidden dimension of the Transformer is set to 768 and 1280 for ViT-L and ViT-G, respectively. The denoising Transformer takes two inputs: a noisy visual embedding and a time-step embedding. The conditions (e.g., text, mask and image conditions) are applied only to the cross-attention layer; thereby it is computationally efficient even using many conditions. CompoDiff is similar to the "DiT with cross-attention" by Peebles & Xie (2022), but handles more various conditions.
Training details. For the efficient training, all visual features are pre-extracted and frozen. All training text embeddings are extracted at every iteration. To improve computational efficiency, we reduced the number of input tokens of the T5 models to 77, as in CLIP. A single-layer percoptron was employed to align the dimension of text embeddings extracted from T5-XL with that of CLIP ViT-L/114.
5.2 Experiment settings
All models were trained using AdamW (Loshchilov & Hutter, 2017). We used DDIM (Song et al., 2020) for the sampling variance method. We did not apply any image augmentation but used pre-extracted CLIP image features for computational efficiency; text features were extracted on the fly as text conditions can vary in SynthTTiipltsl1M. We report the detailed hyperparameters in Table A.1.
We evaluate the zero-shot (ZS) capability of CompoDill on four CIR benchmarks, including FashionIQ (Wu et al., 2021), CIRR (Liu et al., 2021), CIRCO (Baldrati et al., 2023) and GeneCTS (Vaze et al., 2023). We compare CompoDiff' to the recent ZS CIR methods, including Pic2Word (Saito et al., 2023) and SEARLE (Baldrati et al., 2023). We also reproduce the fusion-based methods, such as ARTEMIS (Delmas et al., 2022) and Combiner (Baldrati al., 2022), on SyuthTriplets18M and report their 7S performances. Note that the current CIR benchmarks are somewhat insufficient to evaluate the effectiveness of CompoDiff, particularly considering real-world CIR queries. More details are found in Section 5.4. Our work is the first study that shows the impact of the dataset scale and the zero-shot CIR performances with various methods, such our method, ARTEMIS and Combiner. Due to the page limit, the details of cach task and method are in Section 5.3 and 5.4. We also compare our training strategy with the other CIR methods in terms of the complexity in Section 5.3. In summary, we argue that our method is not specifically complex compared to the other methods. The fine-tuned performances on FashionlQ and CIRR are in Appendix B.
5.3 Comparison methods.
We compare CompoDiff with four state-of-the-art CIR methods as follows:
Combiner (Baldrati et al., 2022) involves a two-stage training process. First, the CLIP text encoder is fine- tuned by contrastive learning of z and Zin + ze in the CLIP embedding space. The second stage replaces Zig 2. to the learnable Combiner module. Only the Combiner module is trained during the second stage.
12

---

Published in Transactions on Machine Learning Research (07/2024)
ARTEMIS (Delmas et al., 2022) optimizes two similarities simultaneously. The implicit similarity is com- puted between the combined feature of 2; and Ee, and the combined one of zig and ze. The explicit matching is computed between 2; and Ze. ARTEMIS suffers from the same drawback as previous CIR methods, e.g. TIRG (Vo et al., 2019): As it should compute combined feature of 2; and 2c, it is not feasible to use an approximale nearest neighbor search algorithm, such as FATSS (Johnson et al., 2019). This is not big problem in a small dataset like FashionIQ, but it makes ARTEMIS infeasible in real-world scenarios, c.g., when the entire LAION-2B is the target database. As Combiner and ARTEMIS are trained based on CLIP ResNet-50 (RN50), we also train CompoDiff based on the same RN50 features for fair comparison.
Pic2Word (Saito et al., 2023) projects a visual feature into text embedding space, instcad of combining them. Pic2Word performs a text-to-image retrieval by using the concatenated feature as the input of the CLIP textual encoder. As the projection module is solely trained on cheap paired datasets without expensive triplet datasets, it able to solve CTR a zero-shot manner.
SEARLE (Baldrati al., 2023) is a similar to Pic2Word, while SEARLE employes text-inversion lask instead of projection. Baldrati et al. (2023) also proposed technique, named Optimization-based Textual Inversion (OTI), a GPT--poerred regularization method. In this paper, we compare SEARLE-XL and SEARLE-XI-OTI, which use the ViT-L/14 CLIP backbone for a fair comparison.
We also compare a naive editing-based retrieval method: editing an image using TP2P with an instruction and retrieving images using the CLIP image encoder. Note that Pic2Word and SEARLE are trained on the image-only datasets; therefore, it is impossible to train them on SynthTriplets and ARTEMTS only take triplets for training; hence, they cannot be trained with image-caption datasets, such as Conceptual Captions (Sharma et al., 2018) LAION (Schuhmann et al., 2022n). Due to this reason, only train the previous fusion-based methods, ARTEMIS and Combiner on SynthTriplets18M for comparison. Note that Combiner needs a pre-trained CLIP model trained on a large-scale image-caption dataset. In this point of view, we can argue that Combiner and CompoDiff are comparable in terms of the dataset scale; both methods use a large-scale image-caption dataset and synthetic triplets in our experiment.
Similarly, Combiner
5.4 CIR Datasets.
FashionIQ (Wu et al., 2021) has (46.6k / 15.5k / 15.5k) (training / validation / test) images with three fashion categories: Shirt, Dress, and Toptee. Each category has 18k training triplets and 12k evaluation triplets of (Zin 2t). Examples of xe looks like: "is more strappy and emerald", "is brighter" or "is the same". The main drawback of FashionlQ is that the dataset is limited to the specific subsets of fashion domains, hence it is not possible to evaluate whether the methods are truly useful for real-world CIR tasks.
CIRR (Liu et al., 2021) contains more generic images than FashionIQ. CIRR uses the images from NLVR? (Suhr et al., 2018) with more complex and long descriptions. CIRR has 36k open-domain triplets divided into the train, validation, and test sets in 8:1:1 split. The examples of xe are "remove all but one dog and add a woman hugging it", "It's a full pizza unlike the other with the addition of spice bottles", or "Paint the counter behind the brown table white". As the example captions show, one of the main drawbacks of CIRR is that the text queries are not realistic compared to the real user scenario; we may need more shorter and direct instructions. As the text instruction distribution of CIRR is distinct from others (Fig. 8), we observe that sometimes CIRR results are a not reliable measure of open-world CIR tasks (e.g., retrieval on LAION). Baldrati et al. (2023) also observe another main drawback of CIRR. CIRR contains a lot of false negatives (FNs) due to the nature of the image-text. dataset (Chun et al., 2021; 2022; Chun, 2023)
CIRCO. To tackle the FN issue of CIRR, Baldrati et al. (2023) introduces the CIRCO dataset. This dataset comprises of 1020 queries, where 220 and 800 of them are used for validation and test, respectively. The images are from the COCO dataset (Lin et al., 2014), where the size of the image database is 120K COCO images. Example Ie of CIRCO includes "has two children instead of cats" or "is on a track and has the front wheel in the air". We use mAP scores following Baldrati et al. (2023) which is known to be a robust retrieval metric (Musgrave et al., 2020; Chun et al., 2022).
GeneCIS (Vaze et al., 2023) is a dataset to evaluate different conditional similarities of four categories: (1) focus on an attribute, (2) change an attribute, (3) focus on an object, and (4) change an object. The first
13

---

Published in Transactions on Machine Learning Research (07/2024)
Fashion IQ (Avg)
CTRR
CTRCO
GeneCTS
Method
Arch
R@10
R@50
R@1
Rg01
mAP
mAP010
mAP925
R@1
CLIP + 1221t
VIT-L
7.01
12.33
4.07
6.11
1.83
2.10
2.37
2.44
Previous zero-shot methods (without SynthTripletes Tripletsspp]
Pic2Word REARLE-OTI
VIT-L
21.70
43.70
23.90
53.76
8.72
9.51
10.65
11.16
VIT-L
27.51
47.90
24.87
53.80
10.18
11.03
12.72
SEARLE!
VIT-L
25.56
46.23
24.24
53.76
11.68
12.73
14.33
12.31
Zero-shot results with the models trained with SynthTripletsSMM
ARTEMIS
RN50
33.24
47.99
12.75
21.95
9.35
11.41
13.01
13.52
Combiner
RN50
34.30
49.38
12.82
24.12
9.77
12.08
13.58
14.93
CompoDiff
RN50
35.62
48.45
18.02
57.16
12.01
13.28
15.41
14.65
CompoDiff
VIT-L
36.02
48.64
18.24
57.42
12.55
13.36
15.83
14.88
CifpoDiff
VIT-L & T5-XL
37.36
50.85
19.37
59.13
12.31
13.51
15.67
15.11
CompoDiff
VYT-G
39.02
51.71
26.71
64.54
15.33
17.71
19.45
15.48
Table 3: Zero-shot CIR comparisons. denotes the results by the official model weight, otherwise, models are trained on Syntherripltss/l and LAION-2B (ARTEMIS and Combiner are trained solely on SynthTriplets18M, while CompoDiff is trained on both). $ refers to the use of both the CLIP textual encoder and T5-XL as the encoder for the text condition. The full results for each dataset are shown Appendix B.
two categories are based on the VAW dataset (Pham et al., 2021), which contains massive visual attributes in the wild. The other categories are sampled from COCO.
The limitations of the current CIR benchmarks. One of the significant drawbacks of the existing CIR benchmarks, such FashionIQ (Wu et al., 2021) and CIRR (Liu et al., 2021), is the domain-specific charactcristics that cannot be solved without training the datasets. For example, the real-world queries that we examine are mainly focused on small editing, addition, deletion, replacement. However, because the datasets are constructed by letting human annotators write modification caption for a given two images, the text conditions are somewhat diffcrent from the real-world CIR quories. For example, CIRR dev set has some text conditions like: "show three bottles of soft drink" (different from the common real-world CIR text conditions), "same environment different species" (ambiguous condition). "Change the type of dog and have it walking to the left in dirt with a leash." (multiple conditions at the same time). These types of text conditions extremely difficult to solve in zero-shot manner, but need access to the CIRR training dataset. Instead, we believe that CIRCO is better benchmark than FashionTQ and CTRR. It is because CIRCO aims to resolve the false negatives of FashionlQ and CIRR, focusing on the retrieval evaluation.
Furthermore, the existing benchmarks (even more recent ones, such as CIRCO and GeneCIS) cannot evaluate the negative and mask conditions. Tn practice, when we perform a qualitative study on large image index (i.e,, LAION-2B image index), we observe that CompoDiff outperforms previous methods, such as Pic2Word, in terms of the retrieval quality (See Fig. 11 and Fig. C.1 for examples). Also, we observe that even if the CIR scores similar (e.g., 10M and 18.8M in Table 6), in the LAION-2B index, the retrieval quality be significantly better. Unfortunately, it is impossible to evaluate the quality of retrieval results on the LAION-2B, because a quantitative study requires expensive and infeasible human verification.
5.5 Qualitative comparisons on four Zero-shot CIR (ZS-CIR) benchmarks
Table 3 shows the overview of ZS-CIR comparison results. CLIP IP2P denotes the naive editing-basee approach by editing the reference image with the text condition using IP2P and performing image-to-imane retricval using CLIP VIT-L. In the table, CompoDiff outperforms all the existing methods with significant gaps. The lable shows the elfectiveness both of our diffusion-based CIR approach and our massive hetthetic dataset. In the SanthhTrppltssSM-trriinii methods with large gap, especially on CIRR and CIRCO, which focus on real-life images and complex descriptions. Our improvement is not main due to the architecture, as CompoDiff already outperforms the fusion methods in RN50. We also can observe that the SynthhriiletsslMM4triii group also enables the
group, CompoDiff outperforms previous SOTA Iusion-based CIR
14

---

Published in Transactions on Machine Learning Research (07/2024)
fusion-based methods to have the zero-shot capability competitive to the SOTA zero-shot CIR methods, Pic2Word and SEARLE. More interestingly, ARTEMIS even outperforms its supervised counterpart on FashionIQ with a large gap (40.6 vs. 38.2 in average recall). We provide the full results of Table 3, the supervised results of ARTEMIS, Combiner, and CompoDiff trained on SynthTriplets18M in Appendix B.
Compared to the previous 7S-CIR methods (Pic2Word and SEARLE), CompoDiff achieves remarkable improvements on the same architecture scale (í.e., ViT-L), except on CIRR. We argue that it is due to the noiseness of the CIRR dataset. Instead, CompoDiff outperforms the other methods on FashionIQ, CIRCO and GeneCIS with a significant gap. We believe that it is because CompoDiff explicitly utilizes the diverse and massive synthetic triplets, while Pic2Word and SEARLE only employ images and the "a photo of" caption during training, resulting in a lack of diversity and generalizability.
5.6 Inference time
One possible drawback of CompoDiff is relatively slow inference compared to encoder-only methods. However, we confirm that CompoDiff is practically useful with high throughput (120 ~ 230ms) for a single image (Table 4). The table reports throuputs of the comparison methods on ViT-L/14 backbone (Pic2Word, SEARLE, CompoDiff) or RN50 backbone (ARTEMIS, Combiner). One of the advantages of CompoDiff is that we can control the trade-off between retrieval performance and inference time. Note that impossible for the other methods. If we need a faster inference time, even with a worse retricval performance, we can reduce the number of diffusion steps as shown in Table 4. Even with only 4 or iterations, our model can produce competitive results. If we use 100 steps, we have slightly better performance (42.65 vs. 42.17), but much slower inference time (2.02 sec vs. 0.12 sec). In the experiments, we set the step size to 10.
Step
2
3
4
5
10
50
100
Pir2Word SEARLE
ARTEMIS Combiner
Avg(R I0, R
21.52
29.22
37.63
41.08
42.17
42.33
42.45
42.65
34.20
35.90
40.61
41.84
Time (sec)
0.02
0.05
0.07
0.09
0.12
0.23
1.08
2.02
0.02
0.02
0.005
0.006
GPU Memory (bs
2.60
GB
0.97 GB
1.32 GB
0.26 GB
0.54 GB
GPU Memory (bs 256)
5.80 GB
2.99 GB
3.24 GB
2.82 GB
1.95 GB
Table 4: Performances vs. inference time by varying the number of denoising steps. Numbers are measured on the FashionlQ validation split. CompoDiff (ViT-L) in Table 3 is equivalent to 10 steps, but using 5 single A100 GPU with batch size of 1. CPU
steps is a practical alternative. The inference time was measured memories are the pick-allocated memory measured on batch sizes of 1 and 64, respectively. Note that ARTEMIS cannol support efficient batch operation for inference because its forward path needs triplet information, not image-instruction pair; here, we report ARTEMIS information with training triplets.
Our another contribution is a novel and efficient conditioning for the diffusion model. Instead of using a concatenated vector of all conditions and inputs as the input of the diffusion model, we use the cross-attention mechanism for conditions and leave the input size the same as the original size. As shown in the next section (Table 5), our design choice is three times faster than the naive implementation.
We also remark that recently, there have been huge advances in boosting diffusion model inference speed, such the Consistency Model (Song et al., 2023) or distillation (Meng et al., 2023). Using more efficient diffusion model variants for the CompoDiff model will also greatly improve the inference speed. However, these approaches will need a completely different training framework (Song et al., 2023) or a complicated two-staged distillation process (Meng et al., 2023). While we believe that applying these techniques to CompoDiff will bring a huge benefit, we leave further improvements for future work.
5.7 Analysis
Denoising Transformer ablation. CompoDiff does not lake lextual embeddings žep as concatenated input tokens of the denoising Transformer but as a condition of cross-attention for the efficient inference. We compare the impact of different design choices for handling textual embeddings. First, we evaluate the Dall-e2 "Prior" model (Ramesh et al., 2022) which converts CLIP textual embeddings into CLIP visual embeddings (we use public community model (Tee et al., 2022) because the official model is not yet
15

---

Published in Transactions on Machine Learning Research (07/2024)
COCO 5k
CxC
ECCV Caption mAP@R
Thronghput
Method
R@1
R@5
R@10
R@1
R@5
R@10
R-P
RØl
images/sec
Image to text retrieval
Prior (Lee et al., 2022)
32.04
56.84
67.68
33.76
60.50
71.32
14.19
23.37
46.47
497.28
Prior-like Stagel
34.32
58.40
69.52
35.01
62.35
74.21
16.35
25.20
19.01
497.28
Ours (Stage only)
35.13
59.46
70.26
35.30
62.62
74.02
16.01
25.44
19.64
1475.52
Ours (Stage + Stage2)
33.20
58.00
68.94
34.78
61.68
72.96
13.07
24.39
47.03
1473.92
Text to image retrieval
Prior (Lee al., 2022)
17.05
34.25
43.16
18.62
37.43
47.07
18.10
26.46
46.40
497.28
Prior-like Stagel
22.62
38.31
48.11
21.42
41.42
51.79
20.70
29.80
51.50
497.28
Ours (Stage only)
22.47
39.18
49.08
22.51
42.40
52.77
21.46
30.30
53.75
1475.52
Ours (Stage + Stage2)
20.00
38.63
48.25
21.57
41.71
51.99
20.82
29.84
51.65
1473.92
Table 5: Comparisons of design choices for handling textual embeddings on cross-modal retrieval benchmarks. Throughput was measured A100 GPU with batch size of 32. For all metrics, higher is better. Here, we use the community version of the prior model by Lee et al. (2022) because the official prior model (Ramesh et al., 2022) not public.
publically available). Second, we test the Pike" model by using the denoising Transformer, but taking text guidance as input tokens instead of cross-attention. We also test two more CompoDiff models from two-stage training strategy.
As the Prior models are not designed for handling CIR triplets, we measure their ability on image-to-text (I2T) and text-to-image (T2I) cross-modal retricval benchmarks on the MS-COCO Caption dataset (Chen et al., 2015). We also evaluate them on the exlension of COCO Caption to mitigate the false negative problem of COCO, CxC (Parekh et al., 2020) and FCCV Caption (Chun et al., 2022). Table 5 shows the average 12T and T21 metrics of each benchmark. In the table, we first observe that our design choice is three times faster than the "Prior-ish" counterparts by handling textual embeddings with coss-attention. Second, we observe that Stage only CompoDiff shows a better understanding of T2T and T2T tasks. We speculate that this is because Ours (Stage 1 only) is directly optimized by the image-to-text (TTM) matching style dataset, while Ours (Stage 1 + Slage 2) is also trained with other types of conditions (e.g., masks, negative texts, image conditions). In summary, our design choice shows x faster inference time than the prior model (Ramosh et al., 2022) but better cross-modal retrieval performances on COCO and its extensions.
Impact of dataset scale. Table 6 shows the impact of the dataset scale by SyitheTripees18MM on ARTEMIS, Combiner and CofpoDiff. First, at a scale of 1M, mod- cls trained on our 1M subset significantly outperformed the IP2P triplets. This result indicates that our dalaset has a more diverse representation capability. As the size of our dataset increases, the performance gradually im- proves. Notably, TynthTriplots1MM shows consistent performance improvements from 1M to 18.8M, where manually collecting triplets in this scale is infeasible and nontrivial. Thanks to our diversification strategy, particularly keyword- based generation, we can scale up the triplet to 18.8M without manual human labor.
IP2P(IM)
1M
5M
10M
18.8M
FashionIQ Avg(R@10, R@50)
ARTEMIS
26.03
27.44
36.17
41.35
40.62
Combiner CompoDiff
29.83
29.64
35.23
41.81
41.84
27.24
31.91
38.11
42,41
42,33
CIRR Avg(R@1,
R. @1)
ARTEMIS
14.91
15.12
15.84
17.56
17.35
Combiner
16.50
16.88
17.21
18.77
18.47
CompoDiff 27.42
28.32
31.50
37.25
37.83
Table 6: Impact of dataset scale. IP2P denotes the
public 1M synthetic dataset by (Brooks et al., 2022).
Table shows that the massive data points are not the necessary condition for training CompoDiff, but all methods are consistently improved by scaling up the data points. It is also worth noting that although the FashionIQ and CIRR scores look somewhat saturated after 10M, these scores cannot represent authentic CIR performances due to the limitations of the datasets as discussed in Section 5.4. As far as we know, this is the first study shows the impact of the dataset scale to the zero-shot CIR performances.
16

---

Published in Transactions on Machine Learning Research (07/2024)
Condition text encoder. As observed by Balaji et al. (2022) and Byun et al. (2024), the power of the CLIP textual encoder affects a lot to the performance of image-text tasks. Motivated by this observation, we also use both the CLIP tex- tual encoder and the language-oriented encoder for extracting the text features of Eq. (3). Table 7 shows the choice of the text encoder affects a lot to the performance on ViT-L/14 CLIP backbone. When the CLIP textual encoder and the T5-XL were used together, the results improved significantly. We suspect that this is because the strong T5 encoder can help the CLIP text encoder to better understand given captions. Interestingly, we observe that using T5 alone degrades the performance even compared to using the CLIP textual encoder alone. We presume this is because T5-XL is specified for long lext sequences (e.g., larger than 100 tokens) and text-only data. Meanwhile, SynthTriplets1SM has a short average length (See Fig. 7 and 8), which is not specialized by T5. Also, our dataset based on captions, paired with an image; we also need to consider image information to understand the given caption, but we cannot handle image information alone with T5.
Text Enc
T5-XL
CLIP
CLIP + T5-XL
FashionIQ CTRR
38.20
42.33
44.11
29.19
37.83
39.25
Table 7: Impact of text encoder.
Therefore, use both T5-XL encoder (Raffel et al., 2020) and CLIP text encoder for the VIT-L model. For the ViT-G model, we use the CLIP text encoder only because we empirically observe that the textual representation power of the ViT-G is much more powerful than the ViT-L. Exploring the power of textual representations, such as learnable prompts to the CFG guidance, could be an interesting future work.
Stage ablation. As we described in Section 3.2, we alternatively update the model using three different objectives in Stage 2. Here, we conduct ablation study of our design choice in Table 8. Our alternative learning strategy improves the overall performance. It is because although SynthIripletsL8M is a vast and diverse dataset, its diversity is weaker than LAION. While we train CompoDiff to handle triplets by training on TynthTripeets using Eq. (3), we employ additional tasks using LAION, í.e., Eq. (1) and Eq. (2) for achieving better generalizabilit..
only Eq. (3) Proposed
FIQ CIRR
41.15
42.33
35.25
37.83
Table Stage 2 ablation. Com- pared by FashionlQ Av R@50) and CIRR Avg(R@) R. @1).
The choice of wy and wy. We include the retrieval performances by varying conditions in Fig. 10. As wy increases, the generated image em- beddings become more dependent on the reference image, while increasing Wr results in a greater influence of the text guidance (See Fig. 5). How- ever, large wy and wr are not always beneficial. If wy or wr is too large, it can lead to unexpected results. To find a harmonious combination of wy and WT, we performed sweeping process as shown in Fig. 10. We use ti as 1.5 and wr as 7.5 considering the best content-condition trade-off.
12-
3511
K051
6.1
11.20
35.1
16-
17.75
11:22
6.41
6.35
8.11
3621
20
3627
01.03
0155
41.2
x511
3551
Figure 10: wi and wy Fash- ionIQ ZS-CIR performance.
5.8 Qualitative examples
We qualitatively show the versatility of CompoDiff for handling various conditions. For example, CompoDiff not only can handle a text condition, but it can also handle negalive text condition (e.g., removing specific objects or patterns in the retrieval results), masked text condition (e.g., specifying the area for applying the text condition). CofpoDiff even can handle all conditions simultaneously (e.g., handling positive and negative text conditions with a partly masked reference image at the same time). To show the quality of the retrieval results, we conduct a zero-shot CTR on entire LAION dataset (Schuhmann et al., 2022a) using FAISS (Johnson et al., 2019) for simulating billion-scale CIR scenarios.
Fig. 11 shows qualitative comparsions of zero-shot CIR results by Pic2Word and CompoDiff. CompoDiff results in semantically high-quality retrieval results (e.g., understanding the "crowdedness" of the query image and the meaning of the query text at the same time). However, Pic2Word shows poor understanding of the given queries, resulting in unfortunate retrieval results (e.g., ignoring "grown up" of text query, or the "crowdedness" of the query image). More examples are in Fig. C.I.
Finally, it is worhth noting that CompoDiff generates a feature belonging to the CLIP visual latent space. Namely, CnCLIP (Ramesh et al., 2022), which decodes a CLIP image feature to an image, can be applied
17

---

Published in Transactions on Machine Learning Research (07 (07/202) /224)
Pic2Word: "A photo of up"
Ours:
Grrwn up
Query Image
Pic2Word:
"A photo of from minecrafi"
"From minecraft"
Figure 11: Qualitative comparison of zero-shot CIR for Pic2Word and CompoDiff. We conduct CTR
LATON. As Pic2Word cannot take simple instruction, we made simple modification for the given instruction.
Reference Image
Conditions
unCLIP Generated
LAION Top-1 Reference Conditions
unCLIP Generated
LAION Top-1
image"
"mettoo"
image" "pink vubbit"
"zoom oue"
"mahe the dog a
"Muke long
Generated and retrieved images by CompoDiff. Images generated by unCLIP decoder and Figure 12:
retrieved from LAION using transformed features by CompoDiff. More examples are shown in Appendix C.
our composed features. We compare the top-1 retrieval results from LAION and the generated images in Fig. 12 and Appendix C. We use the Karlo unCLIP decoder (Lee et al., 2022), by replacing the original Prior manipulate the given input reflecting the
module to CompoDiff. As shown in the figures, CompoDiff given conditions. We believe that incorporating unCLIP into the real-world search engine could potentially improve the user experience by generating images when the desired search results are not available.
6 Conclusion
We have introduced CofpoDiff, novel bassiioo-basdd method for solving complex CIR tasks. We have created a large and diverse datasct named SynthTriplet1 consisting of 18.8M triplets of images, modifi- cation texts, and modified images. CompoDiff has demonstrated impressive ZS-CIR capabilities, as well as remarkable versatility in handling diverse conditions, such as negative text or image masks, and the control- lability enhance user experience, such as adjusting image text query weights. Furthermore, by training the existing CIR methods on Synth Triplets11l the models became comparable ZS predictors to the ZS- ZS-CIR methods. We strongly encourage future researchers to leverage dataset to advance the field of CIR.
18

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Societal Impact
Our work is primarily focused solving complex composed image retrieval (CIR) challenges and is not aware that with the use of additional public resources,
designed for image editing purposes. However, such as the community version of the unCLIP feature decoder (Ramesh et al., 2022), our method can potentially be utilized as an image editing method. We would like to emphasize that this unintended application not the primary objective of our research, and we cannot guaranlee the effectiveness or safely of our method in this context. is important to note that we have taken steps to mitigate potential risks associated with the unintended use of our method for image editing. For instance, we applied NSFW filters filter out potentially malicious samples during the creation of SynthTriplets the need for continued research into the ethical and societal implications of AT technologies and pledge to remain vigilant about potential unintended consequences of our work.
Nevertheless, recognize
References
Muhammad Umer Anwaar, Egor Labintcev, and Martin Kleinsteuber. Compositional learning image-text query for image retrieval. (4rXiv:000..1149), May 2021. URL [cs]. 3
11149. arXiv:2006.11149
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi Text-to- toxt-to- diffusion models with an ensemble of expert denoisers. arXi preprint arXis:2211.0193 2022. 12, 17
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clipd features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1959 1968, 2022. 3, 4, 6, 12
Alberto Baldrati, Lorenzo Agnolucei, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 3, 4, 6, 12, 13
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint or orXi::2211.09800 2022. 4, 7, 9, 10, 16
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sachoon Kim. Coyo-700m:
Image-text pair dataset. httpp::/ggichu..com/aaaaaee hotp://///...aaaaaa//oooooooooo. catprrrraannoooodattttteee 2022.
Jaeseok Byun, Seokhyeon Jeong, Wonjae Kim, Sanghyuk Chun, and Taesup Moon. Reducing task discrepancy of text encoders for zero-shot composed image retrieval. arXiv preprinl wrXiv22406.0166. 2024. 17
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Cupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ar preprint xrxi::1504.0000 00322, 2015. 9, 16
Yiyang Chen, Zhedong Zheng, Wei Ji, Leigang Qu, and Tat-Seng Chua. Composed image retrieval with text feedback via multi-grained uncertainty regularization. arXiv preprint or arXim:2211.0780 2022. 3
Sanghyuk Chun. Improved probabilistic image-text representations. arXiv preprint arXim:2905.1811 2023. 13
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Conference on Computer Vision and Pattern Recognétion (CVPR), 2021. 13
Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang Chang, and Seong Joon Oh. Ecev caption: Correcting false negatives by collecting Conference on Computer Vision (ECCV), 2022. 13, 16
image-caption associations for ms-coco. In European
Ginger Delmas, Rafael S Rezende, Gabriela Csurka, and Diane Larlus. Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. In International Conference on Learning Representations, 2022. 3, 4, 12, 13
19

---

Published in Transactions on Machine Learning Research (07/2024)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural infor- mation processing systems, 34:8780- 8794, 2021. 5
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CT /CE//VVFRO conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Rinon Gal, Patashník, Haggai Maron, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(1):1 13, 2022. 10
Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun. Language-only efficient training of zero-shot composed image retrieval. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
3, 1
Amir Hertz, Rou Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-pronpy image editing with cross attention control. arXiv preprint orXiv:2208.0100 2022. 8, 10
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. orXiv preprint ar orXiv2207.1225% 2022. 2, 4, 5,
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020. 6
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint S11::1106006633 2021. 9
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL attps:///ol. org/10. atgg//..581.... 5281/2enodo.514333 below. 12
If you use this software, please cite it as
Young Kyun Jang, Donghyun Kim, Zihang Meng, Dat Huynh, and Ser-Nam Lim. Visual delta generator with large multi-modal models for semi-supervised composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16805 16814, 2024. 4
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEFE Tronsactions on Big Data, 7(3):535-547, 2019. 13, 17
Donghoon Lee, Jiseob Kim, Jisu Choi, Jongmin Kim, Minwoo Byeon, Woonhyuk Back, and Saehoon Kim. Karlo- v1.0.alpha on 10yo-100m and cc15m. https://githen com./kakaabaainnkaall... 2022. 15, 16, 18
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp- 740-755. Springer, 2014. 13
Zheyuan Liu, Cristian Rodriguez- Opazo, Damien Teney, and Stephen Gould. Image retrieval real-life images with pre-trained laion-and-languuge models. In Proceedings of the IEEE/CVE /CVF International Conference on Computer Vision, pp. 2125 2134, 2021. 1, 3, 8, 12, 13, 14
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0511 2017. 12
Timo Lüddeeke and Alexander Eeker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7086-7096, 2022,
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the TEEE/CVF Conference Computer Vision and
Pattern Recognition, pp. 14297-14306, 2023. 15
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In Compuler Vision Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23- 23-28, 2020, Proceedings, Part XXV 16, pp. 681-699. Springer, 2020. 13
ECCV
Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, and Vishal M. Patel. Unite and conquer: Cross dataset multimodal synthesis using diffusion models. (urXiv:2212.000793) Dec 2022. URL http:://axii....t/////// art.v.orvvv..o.//////// 2212. 2212.00793. arXiv:2212.00 2.0079
(os).
20

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscrossed captions: Extended intramodal and internodaa semantic similarity judgments for ms-coco. arXiv preprinl 2rXie:2004. 15020, 2020. 16
William Peebles and Saining Xie. Scalable diffusion models with transformers. ar preprinl urXi::2212.09748. 2022. 12
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recogrition (CVPR), pp. 13018 13028, June 2021. 14
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine leorning, pp. 8748-8763. PMLR, 2021. 9, 12
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020. 12, 17
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-condittonal image generation with clip latents. Xive preprint orXiv:2204.061 2022. 5, 15, 16, 17,
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022. 2, 4, 5, 12
Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image seg- mentation. In Medical Image Computing and Compuler-Assisted Intervention MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234 241. Springer, 2015. 5
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures to words for zero-shot composed image retrieval. arXiv preprint 2to::3300000.. 2023. 3, 4, 12, 13
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXis:2210.088/ 2022a. 3, 9, 13, 17
Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beaumont. Laion coco: 600m coco, 2022b. 9
synthetic captions from laion2b-en. https://huggingfaee http:///wggggggeeecoooo////////a
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annuol Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565, 2018. 13
Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes. Boosting zero-shot classifi- org/
cation with synthetic data diversity via stable diffusion. (a2X1v:2002.03299), Feb 2023. URL
abs/2302. 03298.. arXiv:2302.0322 [cs). 4
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, and Ming-Hsuan Yang. ViDT: An efficient and effective fully transformer-based object detector. In International Conference on Learning Representations (ICLR), 2022. 5
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Demoising diffusion implicit models.
ar preprint
arXtt:2010.02560 2020. 12
Yang Song, Prafulla Dhariwal, Mark Chen, and Tlya Sutskever. Consistency models. arXiv preprint or Xin:22303.01446 2023. 15
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographis orXiv preprint arXiv:1811.00491, 2018, 4, 13
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Mllia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 5, 12
21

---

Published in Transactions on Machine Learning Research (07/2024)
Sagar Vaze, Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional image similarity. In Proceedings of the IEEE/CVE /CVF Conference on Computer Vision and Pallern Recognition, 2023. 3, 12, 13
Lucas Ventura, Antoine Yang, Cordelia Schmid, and Gül Varol. Covr: Learning composed video retrieval from web video captions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 5270-5279, 2024. 4
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4439-6448, 2019. 3, 13
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Renmie, Kristen Grauman, and Rogerio Feris. Fashion
iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recoguition, pp. 11307 11317, 2021. 1, 3, 12, 13, 14
Youngjae Yu, Seunghwan Lee, Yuncheol Choi, and Gunhee Kim. Curlingnet: Compositional learning between images and text for fashion iq data. erXiv preprint arXiv:2003.1220 12299, 2020. 3
Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siynan Qiao, Wenhu Chen, Yu Su, and Ming- Wei Chang. Magiclens: Self-supervised image retrieval with open-ended instructions. arXiv preprint arXin:240.. 19651, 2024.
Lvmin Zhang, Auyi Rao, and Maneesh Agrawala. Adding conditional control to Lext-Lo-image diffusion models. In Proceedings of the IEEE/CWE /CVF International Conference Computer Vision, pp. 3836-383 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 010008 2022. 8,
22

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Appendix
In this additional document, we provide the detailed training hyperparameter settings (Appendix A), the full experimental results (Appendix B) and more qualitative examples (Appendix C).
A Hyperparameter details
Table A.1 shows the detailed training hyperparameters of CompoDiff.
Stagel
Stage2
Fiue-tuning
Diffusion steps
1000
1000
1000
Noise schedule
cosine
cosine
cosine
Sampling steps
10
10
10
Sampling variance method
DDMM
DDIM
DDMM
Dropout
0.1
0.1
0.1
Weight decay
6.0e-2
6.0e-2
6.0e-2
Batch size
4096
2048
2048
Iterations
1M
200K
50K
Learning rate
le-4
1e-5
le-5
Optimizer
AdamW
AdamW
AdamW
EMA decay
0.9999
0.9999
0.9999
Input tokens
2
(0) t
7
Conditions
Zer
Zey: Fige Rexr
Eep Fig
Training dataset
LAION-2B
LAION-2B, SyithTripletsInn
FashionIQ or CIRR trainset
Image encoder
CLIP-L/14
CLIP-L/14
CLIP-L/14
Text encoder
CLIP-L/14 14
CLIP- 14
CLIP-L/ 14
Denoiser depth
12
Denoiser heads
16
16
16
Denoiser head channels
64
64
Table A.1: Hyperparameters. A model trained by Stage and Stage 2 is equivalent "Zero-shot" in the main table. A "supervised model" is the same as the fine-tumed version.
B Full experiment results of Table
In this subsection, we report the full experiment results of Table 3. For FashionIQ and CIRR datasets, we also report additional fine-tuning results of SynhhTriplesss CIR methods (i.e., directly fine-Lune the "zero-shot" methods on the target training dataset).
FashionIQ. Table B.1 shows the comparison of CompoDiff with baselines the FashionIQ dataset. Fol- lowing the standard choice, we use recall@K the evaluation metric. "Zero-shot" means that the models are not trained on FashionlQ (i.e., same as Table 3). ARTEMIS and Combiner were originally designed for the supervised setting, but, we trained them on Tripless for a fair comparison with our method. Namely, we solely train them on SynthTripletsSM for the zero-shot benchmark and fine-tume mee-tune the zero-shot weights on the FashionTQ training set for the supervised benchmark.
CIRR. Table B.2 shows the CIRR results and all experimental settings were identical to PashionIQ. Similar to FashionIQ, CompoDiff also achieves new state-of-the-e CIRR zero-shot performance. It is noteworthy that Combiner performs great in the supervised setting but performs worse than CompoDiff in the zero-shot setting. We presume that the fine-tuned Combiner text encoder is overfitted to long-tailed CIRR captions. Ii is ialt supported by our additional experiments on text. encoder in Section 5.7; a better understanding of complex texts provides better performances.
For both datasets, we achieve the best scores by fine-tuning the Combiner model trained on SynthTriplets&8M to the target dataset. It shows the benefits of our dataset. compared to the limited CIR triplet datasets.
23

---

Published in Transactions on Machine Learning Research (07/2024)
Shirt
Dress
Toptee
Average R@50
Method
R@10
R@50
R@10
R@50
R@10
R@50
R@10
Avg.
Previous zero-shot methods (without Synth Triplets 18)))
CLIP + InstructPix2PP.
7.24
12.71
6.31
10.42
7.49
13.85
7.01
12.33
9.67
Pic2Word
26.20
43.60
20.00
40.20
27.90
17.40
24.70
43.70
34.20
SEARLE-XI-OTI
30.37
47.49
21.57
44.77
30.90
51.76
27.61
47.90
SEARLE-XL
26.89
45.58
20.48
43.13
29.32
49.97
25.56
46.23
Zero-shot results with the models trained with SyithTriplets/
ARTEMIS
30.70
50.43
33.52
46.54
35.49
47.01
33.24
47.99
40.62
CLIP4Cir
32.32
51.65
34.92
48.38
35.65
48.10
34.30
49.38
41.84
CifpoDiff (ViT-L))
37.69
49.08
32.24
16.27
38.12
50.57
36.02
48.64
42.33
CompoDiff (VIT-L)
38.10
52.48
33.91
47.85
40.07
52.22
37.36
50.85
44.11
CompoDiff (VIT-G)
41.31
55.17
37.78
49.10
44.26
56.41
39.02
51.71
46.85
Supervised
JVSM
12.00
27.10
10.70
25.90
13.00
26.90
11.90
26.60
19.25
CIRPLANT w/ OSCAR
17.53
38.81
17.45
40.41
21.64
45.38
18.87
41.53
30.20
TRACE W/ BERT
20.80
40.80
22.70
44.91
24.22
49.80
22.57
46.19
34.38
VAL w/ GloVe
22.38
44.15
22.53
44.00
27.53
51.68
24.15
46.61
35.38
MAAF
21.30
44.20
23.80
48
27.90
53.60
24.30
48.80
36.55
ARTEMIS
21.78
43.64
27.16
52.40
29.20
54.83
26.05
50.29
38.17
CurlingNet.
21.45
44.56
26.15
53.24
30.12
55.23
25.90
51.01
38.46
CoSMo
24.90
49.18
25.64
50.30
29.21
57.46
26.58
52.31
39.45
RTIC-GCN w/ GloVe
23.79
47.25
29.15
54.04
31.61
57.98
28.18
53.09
40.64
DCNet
23.95
47.30
28.95
56.07
30.44
58.29
27.78
53.89
40.84
AACL
24.82
48.85
29.89
55.85
30.88
56.85
28.53
53.85
41.19
SAC w/ BERT
28.02
51.86
26.52
51.01
32.70
61.23
29.08
54.70
41.89
MUR
30.60
57.46
31.54
58.29
37.37
68.41
33.17
61.39
47.28
Combiner
39.99
60.45
33.81
59.40
41.41
65.37
38.32
61.74
50.03
Pre-training with Synth Triplets 1MM
Fine-tuning with Fashion
ARTEMIS
32.17
53.32
34.80
48.10
36.58
47.63
34.52
19.68
42.10
Combiner
37.21
60.71
42.75
60.50
42.98
65.49
40.98
62.23
51.61
CompoDiff (VIT-L)"
40.88
53.06
35.53
49.56
41.15
54.12
39.05
52.34
46.31
CompoDiff (VIT-L + T
40.65
57.14
36.87
57.39
43.93
61.17
40.48
58.57
49.53
CompoDiff (VIT-G)*
41.68
56.02
38.39
51.03
45.70
57.32
39.81
51.90
47.73
Table B.1: Comparisons FashionIQ. The "Zero-shot" scenario performs CIR using model not trained the FashionIQ dataset, while models are trained on FashionIQ for the "Supervised" scenario. The last group denotes that model trained Triphettipl f denotes that the models newly trained by
The detailed CIRCO results are shown Table B.3. In the table, we can observe that in all metrics, CompoDiff achieves the best performances among the zero-shot CIR methods. This result supports the effectiveness of CompoDiff and TynthTriplets1MM in real-world CIR tasks.
CIRCO.
GeneCIS. Finally, we report detailed GeneCIS in Table B.4. In the table, CompoDiff shows the best performance in the average recall. Our method especially out outperforms the other methods in "Change Attribute", "Focus Object" and "Change Object". On the other hand, CompoDiff is less effective than Pic2Word and SEARLE "Focus attribute". We presume that is because the instruction distribution of "Focus Attribute" differs lot from the instruction of SynthTripletsSM. Among other CIR methods in the SynthTripletsl MM- MM-tnieeed group, CompoDiff shows the best performances.
24

---

Published in Transactions on Machine Learning Research (07/2024)
Method
R@1
R@5
R@00
R
Rg 301
Rg 82
Rg .3
Avg(R Rg @1)
Previous zero-shot methods (without Synth Synth'Tripletsi Triplets
CLIP + InstructPix2Pix
4.07
8.41
11.2
15.38
6.11
10.05
13.33
5.09
Pic2Word
23.90
51.70
65.30
87.80
53.76
74.46
87.08
38.83
SEARLE-XL-OTI
21.87
52.31
66.29
88.58
53.80
74.31
86.94
39.34
SEARLE-XL
24.24
52.48
66.29
88.84
53.76
75.01
88.19
39.00
Zero-shot results with the models trained with SynthTriplets 18M
ARTEMIS
12.75
33.84
47.75
80.20
21.95
43.88
62.06
17.35
CLIP4Cir!
12.82
36.83
48.19
81.91
24.12
46.47
63.07
18.47
CiffoDiff (VIT-L)*
18.24
53.14
70.82
90.25
57.42
77.10
87.90
37.83
CofpoDiff (ViT-L))
19.37
53.81
72.02
90.85
59.13
78.81
89.33
39.25
CompoDiff (VIT-G)+
26.71
55.14
74.52
92.01
64.54
82.39
91.81
45.63
Supervised
TIRG
14.61
48.37
64.08
90.03
22.67
44.97
65.14
18.64
TIRG + Cast
11.04
35.68
51.27
83.29
23.82
45.65
64.55
17.43
MAAF
10.31
33.03
48.30
80.06
21.05
41.81
61.60
15.68
MAAF + BERT
10.12
33.10
48.01
80.57
22.04
42.41
62.14
16.08
MAAF-IT
9.90
32.86
48.83
80.27
21.17
42.04
60.91
15.54
MAAF-RP
10.22
33.32
48.68
81.84
21.41
42.17
61.60
15.82
CIRPLANT
15.18
43.36
60.48
87.64
33.81
56.99
75.40
24.50
CIRPLANT w/ OSCAR
19.55
52.55
68.39
92.38
39.20
63.03
79.49
29.38
ARTEMIS
16.96
46.10
61.31
87.73
39.99
62.20
75.67
28.48
Combiner
38.53
69.98
81.86
95.93
68.19
85.64
94.17
53.36
Pre-training with Synth Triplets18M
& Fine-tuning with FashionIQ
ARTEMIS
18.85
51.44
68.01
91.93
38.85
62.00
77.68
28.85
Combiner
39.99
73.63
86.77
96.55
68.41
86.12
94.80
54.20
CompoDiff (VIT-L.)
21.30
55.01
72.62
91.49
58.82
77.60
88.37
40.06
CofpoDiff (VIT-L + T5-XL)
22.35
54.36
73.41
91.77
62.55
81.44
90.21
42.45
CompoDiff (ViT-G)*
32.39
57.61
77.25
94.61
67.88
85.29
94.07
50.14
Table B.2: Compasions on CIRR Test set. Details are the same as Table B.1.
Method
mAP@5
mAP010
mAP@25
mAP@50
Previous zero-shot methods (without Typpes(iip)
CLIP InstructPix2Pix
1.83
2.10
2.37
2.44
Pic2Word
8.72
9.51
10.65
11.29
SEARLE-XL
11.68
12.73
14.33
15.12
Zero-shot results with the models trained with SynthTrippees18MM.
ARTEMIS
9.35
11.41
13.01
Combiner
9.77
12.08
13.58
14.11
CompoDiff (ViT-L)
12.55
13.36
15.83
16.43
CompoDiff (VIT-L + T5-XI)
12.31
13.51
15.67
16.15
CompoDiff (ViT-G)
15.33
17.71
19.45
21.01
Table B.3: Compasions CIRCO Test set.. Details the as Table B.1.
C More qualitative examples
Open world zero-shot CIR comparisons with Pic2Word. We illustrate further comparisons with Pic2Word in Fig. C.1. Here, we can draw the same conclusions as in the main text: Pic2Word often cannot understand images instructions (e.g., ignores the "crowdedness" of the images, retrieves irrelevant
25

---

Published in Transactions on Machine Learning Research (07/2024)
Method
Focus Atcribute
Change Astribute
Focus Object
Change Objeet R
Avg
R@1
Ri22
R133
R@1
Ro22
R@3
R@1
R122
R@3
R@1
R
R@1
Previous wero-shot methods (without SynthTTiilecsskMMM
Pic2Word
15.65
28.16
38 65
13.87
24.67
33.05
8.42
18.01
25.77
6.68
15.05
24.03
11.16
SEARLE-XL
17.00
29.65
40.70
16.38
25.28
34.14
7.96
16.94
25.61
7.91
16.79
24.80
12.31
Zero-shot results with the models trained with TyiplessieS
A ARTEMIS
11.76
21.97
25.44
15.41
25.14
33.10
8.08
16.77
24.70
18.84
30.53
39.98
13.52
Combiner
14.11
21.08
34.12
18.39
28.22
37.13
8.19
16.70
25.21
18.72
30.92
10.11
14.99
CompoDif' (VTT-L)
13.50
24.32
36.11
19.20
28.64
37.20
8.11
16.39
25.08
18.71
31.69
40.55
14.88
Cit (VIT-L)
13.01
25.01
36.75
19.88
28.61
37.18
8.60
16,
25.85
18.94
31.80
40.58
15.11
CompoDifl (VIT-G)
14.32
26.70
38.41
19.72
28.78
37.39
9.18
19.11
23.77
18.71
31.71
00.22
15.48
Table B.4: Compasions on GeneCIS Test set.
Details are the same as Table B.1.
Pic2Word:
a photo of a wearing a school uniform"
Ours:
"Waar school uniform"
Query Image
Pic2Word:
"A photo of dynamic"
Ours: " "More dynomic"
Pic2Word:
"a photo of * with red flowers"
Ours: "as red flowers"
Query Image
Figure C.1: More qualitative comparison of zero-shot CIR for Pic2Word and CompoDiff.
images such as images with woman in the last example). All retrieved results in our paper were obtained using Pic2Word trained on the LAION 2B datasct.
More versatile CIR examples LAION. We illustrate more qualitative examples of the composed features by CompoDiff, such as generated images by unCLIP retricval results, in Fig. C.2, Fig. C.3, Fig. C.4. and Fig. C.5.
26

---

Published in Transactions on Machine Learning Research (07 (07/2024)
"As ascii artworks" "
"As cubism" "
"With
"Make it long sleeve" "
(a) Reference Image
(b) Conditions
(c) Generated by unCLIP (d) Top-1 item from LAION
Figure C.2: Generated vs. retrieved images by CompoDiff. Using the trausformed image feature by CompoDiff, Generated images using unCLIP and top-1 retrieved image from LAION.
27

---

Published in Transactions on Machine Learning Research (07/2022 2024)
"Brown" "
"As pencil sketch"
"soldier"
"tiger"
"As lego"
"As patch sticker"
(a) Reference Image
(b) Conditions
(c) Generated by unCLIP (d) Top-1 item from LAION
Figure C.3: Generated vs. retrieved images by CompoDiff (Continue).
28

---

Published in Transactions on Machine Learning Research (07 (07/2024) /204)
"Change her to spongehob" "
"Change her to spongebob" " "surprised"
"meteor" "
"meteor" "
"zoom out"
"As 4k image"
"As 4k image" "pink rabbit"
(a) Reference Image
(b) Conditions
(c) Generated by unCLIP (d) Top-1 item from LAION
Figure C.4: Generated vs. retrieved images by CompoDiff (Continue).
29

---

Published in Transactions on Machine Learning Research (07 (07/2022 2224)
"Made of ice"
"Made of ice" "Skill of artist"
DDrawn by famous arist from artstation"
"Drawn by famous arist from artstation" "Black and white"
"Add a lighthouse in the background"
"make the dog a cat"
(a) Reference Image
(b) Conditions
(c) Generated by unCLIP (d) Top-1 item from LAION
Figure C.5: Generated vs. retrieved images by CompoDiff (Continue).
30


Reviews:
[{"review_id": "VF7AQmhLxn", "reviewer": "Reviewer_X9BB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe motivation and clarification are clear, and I can understand the details following the descriptions. The proposed CompoDiff, a diffusion-based model with latent diffusion, is effective for solving zero-shot Composed Image Retrieval (ZS-CIR). The constructed dataset, named SynthTriplets18M, is valuable for the community in image retrieval. The experiments are sufficient to show the effectiveness of the proposed method and datasets. \n\n\n**Weaknesses:**\n\n1.I am not sure whether the contribution is significant, due to the image edit tasks has achieved great success. Based on my understanding, the task of image retrieval is easier than the task of image generation. When there are so many great works in image generation (e.g., controllable edit and generation), I am not sure whether the progress this paper contribute in ZS-CIR is significant, especially it is based on the techniques of image generation tasks. \n\n2.This paper points out that the drawbacks of the existing CIR benchmarks are not scalable for “real-world queries”. However, I find the images of the constructed SynthTriplets18M datasets are also not “real-world images”, which are generated by generative models. This leaves the debate that whether the SynthTriplets18M can represent “real-world queries”.\n\nOther minors:  \nIn Eqn.4, the weight $w_I$ and $w_T$ should be illustrated in the following description, rather than putting in another paragraph. When I take took the first look, I thought $w_I$/$w_t$ was a function.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates an interesting task in image retrieval called Composed Image Retrieval (CIR), where the customer needs a search engine that can process composed queries. This paper treats the CIR task as a conditional image editing task on the latent space. It proposes a diffusion-based model with latent diffusion, called CompoDiff, for solving zero-shot Composed Image Retrieval (ZS-CIR). It also creates a large and diverse dataset named SynthTriplets18M, consisting of 18.8M triplets of images, modification texts, and modified images. The experiments are well designed to show the effectiveness of the proposed method and datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.the description of  $w_I$ and $w_T$ following Eqn.4  \n2.Amplificaiton the importances of the Composed Image Retrieval (CIR), and shows the unique difficulties of CIR, compare to image edit/generation.", "broader_impact_concerns": "I believe this paper addressed the Broader Impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "exAbouo6XY", "reviewer": "Reviewer_5Q5f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n- The proposed CompoDiff can handle negative text conditions and mask conditions.\n- The paper is well-written and addresses its relation to many state-of-the-art works.\n- SynthTriplet18M dataset contains many triplets that are automatically generated.\n- The experiment is extensive and promising.\n    - CompoDiff outperforms state-of-the-art methods on this problem.\n     - There are several ablation studies and detailed analysis \n     - Abundant example images are shown.\n\nWeakness.\n- The proposed method is a combination of existing methods.\n- The comparison with ARTEMIS and Combiner seems unfair because they did not utilize the LAION-2B datasets for training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes CompoDiff, a diffusion-based model for solving zero-shot Composed Image Retrieval (ZS-CIR).\n- CompoDiff can handle various conditions, such as negative text and image mask conditions.\n- CompoDiff can control the strength of conditions and the trade-off between inference speed and performance.\n\nThis paper also proposes SynthTriplets18M, which are synthesized triplets for training.\n\nThe experiments support the effectiveness of CompoDIff.\n- Achieved significant performance on FashionIQ, CIR, CIRCO, and GeneCIS datasets.\n- Qualitative performance on ZS-CIR on a billion-scale database, or image decoding using the unCLIP generator.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Justify the comparison with ARTEMIS and Combiner which do not use LIAON-2B datasets.\n\n- In Fig.3, bottom of Stage 1, there are two text conditions.　In my understanding, the change condition, such as \"Change the chickadee to a rooster,\" is not utilized in Stage 1. Could you explain how these two text conditions are employed in Stage 1?\n\n- Add CIRCO and GenCIS datasets to Fig. 8.\n\n- Table3 with Arch RN50, the performance of CompoDIff is lower than that of Combiner on GeneCIS.\nCould you explain this reason?\n\n- In Sec.5.7, Dall-e2 Prior model is referred as (Ramesh et al., 2022). However, Table 5 refers (Lee et al., 2022) which is the reference of footnote 4. To prevent confusion, add the citation ( Lee et al., 2022) in the main text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZmIjgSTgH0", "reviewer": "Reviewer_SMao", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This work enables flexible zero-shot Composed Image Retrieval during inference, thanks to the LDM framework. The input conditions could include the reference image, positive prompt, and negative prompt. The users can also re-weight different conditions to emphasize their true intentions. \n- To better fit the CIR task, the denoiser is designed as a transformer-based model, and achieves good results in 5-10 denoising steps. The condition is integrated via a cross-attention mechanism to improve the inference speed.\n- The constructed large-scale synthetic dataset and the corresponding automatic data generation techniques (e.g., triplet generation) could benefit the Composed Image Retrieval field or other related tasks. The experiment shows that Combiner trained with SynthTriplets18M may be boosted in the zero-shot setting. \n- The dataset statistics analysis in Fig. 7 & Fig. 8 are good for showing the domain gap between different datasets. The dataset scale experiment in Table 6 validates the scalability of the proposed method to some extent.\n\nWeaknesses\n- The inference cost is the biggest limitation. Although it is claimed that the test speed can be improved via a trade-off between inference speed and performance, the current speed is still 10x slower than the encoder-only methods. For example, in Table 4, Combiner can achieve comparable performance with x20 test speed. For the Image Retrieval task, test speed is very important for practical usage.\n- The GPU memory cost analysis is also important for practical usage, which is missing in the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to address zero-shot Composed Image Retrieval (ZS-CIR) task with latent representation via latent diffusion model (LDM). Prior methods usually adopt the encoder technique to encode the query image and other conditions (e.g., text) to a latent embedding for similarity retrieval, which may be limited to training conditions and not flexible during inference. Instead, this paper borrows ideas from the LDM-based image generation framework and classifier-free guidance technique, which enable flexible conditions and better performance. Besides, a large-scale synthetic dataset has been constructed based on various pretrained models and techniques (e.g., Prompt2Prompt). Abundant experiments have been done to show the effectiveness and superiority of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- There have been several techniques (e.g., distillation, samplers, consistency model) proposed to speed up the diffusion process. I recommend the authors to further alleviate inference costs to make the method more practical.\n- GPU memory cost analysis is necessary to make this work more complete.", "broader_impact_concerns": "The societal impact has been claimed in the paper. The potentially malicious samples have been filtered out during the creation of the proposed SynthTriplets18M dataset.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers have acknowledged the contribution of this paper, as well as the revision made in response to the reviews. All reviewers recommended to accept the paper. This paper proposes CompoDiff, a latent diffusion based method for solving composed image retrieval in the zero-shot scenario. It also introduces SynthTriplets18M, a new synthetic dataset with 18.8 million reference images, conditions, and
corresponding target image triplets. These two contributions are good addition to the literature, expanding the scope of generative models to retrieval tasks.
Decision: Accept (poster)