FactEmbed: Factorized MLLM Embeddings for Composed Image Retrieval

Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized. This stems from directly mapping composed queries into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. In this work, we introduce Factorized MLLM Embeddings (FactEmbed), which decouples compositional query comprehension from building discriminative embeddings. Rather than directly projecting the composed query into a single embedding without internal reasoning, FactEmbed employs a set of factorized latents that simulate the intended visual modifications via an intermediate target imagination objective. This process encourages the latents to ground into target semantics before forming discriminative representations for retrieval. Subsequently, to propagate these factorized representations throughout the model for retrieval, we optimize a factor-level contrastive loss while distilling the decomposed representations into a deeper attention layer of the MLLM. Finally, for efficient inference, we retain the [EOS] token to establish a global-level embedding on top of these factorized latents, while we exploit the mean-pooled factorized latents to rerank a short candidate list at negligible cost, as both are produced by the same forward pass. Extensive evaluations demonstrate that FactEmbed outperforms prior works across CIR benchmarks.

Checkpoints

All checkpoints are trained on top of Qwen2.5-VL-7B-Instruct. Each folder holds the trained weights of that stage together with the processor and tokenizer configuration used in training.

Folder	Benchmark	Stage
fashioniq/stage1	FashionIQ	Target-imagination warm-up
fashioniq/stage2	FashionIQ	Joint training (retrieval + imagination + distillation)
fashioniq/stage3	FashionIQ	Post-training with augmented triplets (final model reported in the paper)
cirr/stage1	CIRR	Target-imagination warm-up
cirr/stage2	CIRR	Joint training (retrieval + imagination + distillation)
cirr/stage3	CIRR	Post-training with augmented triplets (final model reported in the paper)

Stage 3 is the model to use for retrieval; stages 1 and 2 are released for reproducibility.

Usage

Load a checkpoint with the release code, e.g. for FashionIQ:

bash scripts/eval/eval_fashioniq.sh <path to fashioniq/stage3>
bash scripts/eval/eval_cirr.sh --model_path <path to cirr/stage3>

A browsable demo of the retrieval results on the FashionIQ and CIRR validation sets: WJ0830/factembed (Space).