Pseudo-Label-based Zero-Shot Composed Image Retrieval with Multimodal Large Language Models
Authors: Yuchen Yang, Yu Wang, Yanfeng Wang


Reviewer paper summaries: This paper addresses the challenges of Composed Image Retrieval (CIR), which involves combining a reference image with modification text to retrieve target images. The main issues in CIR are data scarcity and modal information disparity. The paper proposes a pseudo-label-based zero-shot composed image retrieval method using Multimodal Large Language Models (MLLMs). The method introduces pseudo-label-based data generation (PDE) and pseudo-label-based multi-scale reranking (PMR). PDE uses MLLMs to generate bounding boxes and captions for unlabeled images, creating training triplets to tackle data scarcity. PMR aligns text and images at multiple scales to address modal information disparity. Experiments demonstrate that this method outperforms previous zero-shot CIR approaches.
Summary of strengths: S1: The paper introduces a novel pseudo-label-based approach to address the challenges in Composed Image Retrieval (CIR). The pseudo-label-based data generation (PDE) and multi-scale reranking (PMR) methodologies are innovative solutions that effectively mitigate the issues of data scarcity and modal information disparity.

S2: The pseudo-label-based data generation allows for scalable and flexible data augmentation without the need for labor-intensive human annotation.
Summary of weaknesses: W1: The proposed method's accuracy and robustness depend on the quality of the pseudo-labels generated during the data preprocessing stage. Any noise or inaccuracies in these pseudo-labels could potentially degrade the performance of the retrieval system.

W2: The reliance on powerful Multimodal Large Language Models (MLLMs) for pseudo-labeling and retrieval tasks necessitates significant computational resources, especially for the success of PMR and PDE. Extracting the bounding box of a specific object is not a very easy task and requires powerful VLM.
Comments, suggestions and typos: See weakness, please.
Overall assessment scores: 3