MAI: A Multi-turn Aggregation-Iteration Model for Composed Image Retrieval

Authors: Yanzhe Chen, Zhiwen Yang, Jinglin Xu, Yuxin Peng
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multi-modal, Multi-turn retrieval, Retrospective-based, Composed image retrieval

Abstract:
Multi-Turn Composed Image Retrieval (MTCIR) addresses a real-world scenario where users iteratively refine retrieval results by providing additional information until a target meeting all their requirements is found. Existing methods primarily achieve MTCIR through a "multiple single-turn" paradigm, wherein methods incorrectly converge on shortcuts that only utilize the most recent turn's image, ignoring attributes from historical turns. Consequently, retrieval failures occur when modification requests involve historical information. We argue that explicitly incorporating historical information into the modified text is crucial to addressing this issue. To this end, we build a new retrospective-based MTCIR dataset, **FashionMT**, wherein modification demands are highly associated with historical turns. We also propose a Multi-turn Aggregation-Iteration (**MAI**) model, emphasizing efficient aggregation of multimodal semantics and optimization of information propagation in multi-turn retrieval. Specifically, we propose a new Two-stage Semantic Aggregation (TSA) paradigm coupled with a Cyclic Combination Loss (CCL), achieving improved semantic consistency and modality alignment by progressively interacting the reference image with its caption and the modified text. In addition, we design a Multi-turn Iterative Optimization (MIO) mechanism that dynamically selects representative tokens and reduces redundancy during multi-turn iterations. Extensive experiments demonstrate that the proposed MAI model achieves substantial improvements over state-of-the-art methods.

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730094852839

thread_type:
official_review

<thread_object>
creator: reviewer_gvmc

<content>
summary:
The paper introduces FashionMT and the MAI model. While the method shows strong results, its complexity and potential scalability challenges could limit its immediate applicability in more generalized settings. Further testing on diverse datasets and longer interaction sequences, along with analysis of the model's efficiency, would enhance the practical relevance of this work.

strengths:
- FashionMT is a significant contribution. The dataset provides richer and more realistic interactions with over 1 million images and 95 categories. This dataset fills a critical gap in the field where existing datasets fail to incorporate historical context across multiple turns.
The model’s approach to maintaining and optimizing key tokens during multiple retrieval iterations is highly relevant to real-world e-commerce scenarios, where users iteratively refine their search queries. The MAI method effectively addresses the shortcomings of the “multiple single-turn” paradigm that fails to leverage historical turn information.
- Extensive experiments show that MAI achieves significant improvements over existing methods in both the combination and rollback settings.

weaknesses:
- While the model is innovative, it introduces considerable complexity with multiple components (TSA, CCL,). The addition of multiple layers, clustering mechanisms, and token filtering might make the model difficult to implement and optimize in real-world settings where computational efficiency is key.
- While the results on FashionMT are strong, the paper does not provide comparisons on non-fashion datasets. The model's application to a more diverse range of image retrieval tasks, such as general object retrieval or scene retrieval, would provide a stronger claim to its versatility.

questions:
- Can the MAI model be adapted to other MTCIR applications outside fashion, such as general e-commerce, furniture, or other products?

- It would be beneficial to know how this impacts performance when dealing with significantly larger datasets. Is the trade-off between memory efficiency and retrieval performance consistent across different dataset sizes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582905959

thread_type:
official_review

<thread_object>
creator: reviewer_jtgq

<content>
summary:
This paper introduces MAI (Multi-turn Aggregation-Iteration), a model for multi-turn composed image retrieval (MTCIR), along with FashionMT, a new large-scale dataset specifically designed for MTCIR. The key innovation is a two-stage semantic aggregation approach that uses image captions as a bridge between visual and textual modalities, plus a memory-efficient mechanism for retaining historical information across multiple turns.

strengths:
1. Creation of a large-scale, diverse dataset (FashionMT) that better reflects real-world scenarios
2. Memory-efficient design through the Multi-turn Iterative Optimization mechanism

weaknesses:
1. Limited evaluation of existing datasets (mostly focused on their new dataset), how is the model's performance on existing benchmark like CIRR, FashionIQ and CIRCO. 
2. The fixed number of turns (4) in the dataset may not reflect varying real-world scenarios, does the author have plan to extend the dataset with a different number of turns? 
3. More visualization of the dataset is expected: I'd like to know (1)sample image sequences across multiple turns: namely the quality of the dataset, the relation of images in different turns, and whether the modified text can describe the relationship between images. (2) visualizations of the distribution of different types of modifications: such as Rollback operations and combination operations.

questions:
1. Statistics for Modification text types: is possible to supply a category of the modified text type, so that the performance can be compared according to different categories? such as Rollback operations and combination operations.
2. Will the dataset be open-source? 
3. How does the model perform with varying numbers of turns beyond the fixed 4-turn setup?
4. Could the approach be extended to other domains beyond fashion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620887691

thread_type:
official_review

<thread_object>
creator: reviewer_wpsv

<content>
summary:
The paper proposes a new dataset, FashionMT, specifically designed for multi-turn composed image retrieval (MTCIR) tasks. FashionMT is characterized by its retrospective-based nature, where the modified text in each new turn may involve information from historical reference images, and its massive diversity, containing 14 times more fashion images and 30 times more categories than previous MTCIR datasets. The authors also introduce a new Multi-turn Aggregation-Iteration (MAI) model that focuses on efficient aggregation and iterative optimization of multimodal semantics in MTCIR. The MAI model includes a Two-stage Semantic Aggregation (TSA) paradigm and a Cyclic Combination Loss (CCL) to enhance semantic consistency and modality alignment, as well as a Multi-turn Iterative Optimization (MIO) mechanism to dynamically select representative tokens and reduce redundancy during multi-turn iterations.

strengths:
a.	The paper proposes a new dataset, FashionMT, for multi-turn composed image retrieval (MTCIR) tasks
b.	FashionMT is characterized by its retrospective-based nature and massive diversity, containing 14 times more fashion images and 30 times more categories than previous MTCIR datasets
c.	The authors introduce a new Multi-turn Aggregation-Iteration (MAI) model that focuses on efficient aggregation and iterative optimization of multimodal semantics in MTCIR
d.	The MAI model includes a Two-stage Semantic Aggregation (TSA) paradigm and a Cyclic Combination Loss (CCL) to enhance semantic consistency and modality alignment
e.	The MAI model also includes a Multi-turn Iterative Optimization (MIO) mechanism to dynamically select representative tokens and reduce redundancy during multi-turn iterations

weaknesses:
a.	How to avoid false negative samples during FashionMT dataset construction? The existing single-turn dataset already has the presence of false negative samples, will it be more obvious with multiple rounds?.
b.	For the ZS-CIR method, how was it trained and tested on FashionMT? They are designed for zero-shot, is the adaptation process also zero-shot paradigm?
c.	How does MAI perform for the single-turn dataset? The authors should add the results of MAI for datasets such as FashionIQ, CIRR, etc. to confirm the superiority of MAI even with turn=1.
d.	The clustering used in MIO is affected by the number of cluster centers, denoted as k, and the authors do not mention in the implementation details the value of k, and the number of iterations used when performing k-means, which affects the accuracy of the clustering. And does clustering increase the time overhead of the whole model? Authors need to increase the comparison of model training and inference time.
e.	The Q-former used in MAI is initialized by BLIP-2 with the Flan-t5-xxl language model, while the version of BLIP-2 used in SPRC is blip2-pretrain, right? If MAI uses the same version of BLIP-2 as SPRC to initialize the Q-former parameters, how does the result behave? The authors need to add this experiment to further demonstrate the superiority of MAI?
f.Typos and minors. There are non-standard punctuation marks, such as Line 431, P8, "thanks to TSA", etc. It is recommended to check the whole paper.

questions:
refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708705464

thread_type:
official_review

<thread_object>
creator: reviewer_tgqm

<content>
summary:
Existing multi-turn composed image retrieval (MTCIR) methods adopt the “multiple single-turn” paradigm and neglect the historical correlation information in multi-turn interactions. To address this problem, this paper first builds a new retrospective-based MTCIR dataset, wherein modification demands are highly associated with historical turns. Then, the authors develop a new Multi-turn Aggregation-Iteration (MAI) model, which contains a two-stage semantic aggregation paradigm coupled with a cyclic combination loss. Besides, a multi-turn iterative optimization mechanism is designed to select representative tokens and reduce redundancy dynamically. Experimental results demonstrate the effectiveness of the proposed method.

strengths:
1)	This paper focuses on an interesting multi-turn composed image retrieval task, and points out the critical “multiple single-turn” problem in the existing research field. This research is very valuable and provides significant insights for further research.
2)	The authors have constructed a new dataset for the multi-turn composed image retrieval task, which is more similar to real-world scenarios, and is more massive and diverse. This is an important contribution if the dataset is made publicly available.
3)	A multi-turn key information-aware approach, named the Multi-turn Aggregation model, is proposed to achieve multimodal semantics aggregation and multi-turn information optimization. The proposed method is reasonable and the parameter-free multi-turn iterative optimization (MIO) mechanism is interesting.
4)	The effectiveness of the proposed method has been demonstrated by extensive experimental results.
5)	The writing of this paper is good and easy to follow.

weaknesses:
1)	Will the collected dataset be made publicly available? Some key characteristics of the dataset are still not clear, such as the average and variance of turn number for each query, the average length for modification text, etc. The dataset will be more comprehensible as more detailed statistics are provided.
2)	It seems MIO is executed for each input, and it contains a DPC-kNN for clustering and density computation. The complexity of this module should be analyzed.
3)	In Table 6, what does the w/o CCL mean? It seems w/o CCL has outperformed most of the compared methods. Are all the methods evaluated on the same backbone? If they are not, the comparison results may not be fair.
4)	Except for the proposed dataset, the proposed method should be also evaluated on existing multi-turn composed image retrieval datasets to further demonstrate the effectiveness.

questions:
please try to address the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732505015009

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# Global Author Rebuttal

We would like to express our sincere gratitude for the insightful and helpful feedback provided by the reviewers, especially the encouraging comments: **valuable dataset (Reviewer TgqM, WPSv, JTgq, GvMC)**, **interesting and novel approach (TgqM, GvMC)**, **valuable insights (TgqM)**, **significant improvements (GvMC)**, and **easy to understand (TgqM)**.

In the following separate response section, we will address each reviewer's comments in detail and provide responses to enhance the quality of our work.

Before addressing all questions, please allow us to highlight two key aspects of interest: **(1) evaluation on additional datasets** and **(2) dataset availability**, to further clarify and emphasize the contributions of this study.

1. **Evaluation on Additional Datasets**

    **(1) Multi-turn Composed Retrieval.**
    Despite limitations in existing multi-turn composed retrieval datasets, we add mean recall comparisons on the existing benchmarks MT FIQ and MT Shoes. The table below shows that our approach achieves the **best performance** in all settings due to its fine-grained semantic capture and efficient modality alignment.

    |Methods|MT FashionIQ|MT Shoes|FashionMT (ours)|Mean Recall|
    |-|:-:|:-:|:-:|:-:|
    |FashionNTM|48.1|31.2|36.26|38.52|
    |DQU-CIR|55.9|46.2|39.88|47.33|
    |Context-I2W|49.1|36.5|45.89|43.83|
    |Image2Sentence|51.1|47.0|49.15|49.08|
    |SPRC|57.4|52.7|58.01|56.04|
    |**MAI (ours)**|62.2|56.7|66.64|**61.85**|

    **(2) Single-turn Composed Retrieval.** We follow the standard single-turn composed retrieval settings. For CIRR and FashionIQ, models are independently trained on their respective datasets and evaluated on the test sets for average recall, denoted as Avg. For CIRCO, due to the absence of a training set, we adopt the approach from CoVR2 [a], training on CC-CoIR and testing on the CIRCO test set. By effectively aggregating fine-grained multimodal semantics in composed retrieval scenarios, our approach demonstrates competitive performance even in the single-turn setting.
    * **CIRR and FashionIQ:**

        |Methods|CIRR|FashionIQ|Mean Recall|
        |-|:-:|:-:|:-:|
        |CoVR2 [a]|78.92|60.57|69.75|
        |CALA [b]|78.74|57.96|68.35|
        |ECDE [c]|78.09|60.44|69.27|
        |FashionERN|74.85|62.62|68.74|
        |DQU-CIR|74.55|65.13|69.84|
        |SPRC|82.66|64.85|73.76|
        |**MAI (ours)**|86.44|68.12|**77.28**|

        [a] Ventura et al. Covr-2: Automatic data construction for composed video retrieval, TPAMI 2024.  
        [b] Jiang et al. Cala: Complementary association learning for augmenting comoposed image retrieval, SIGIR 2024.  
        [c] Thawakar et al. Composed video retrieval via enriched context and discriminative embeddings, CVPR 2024.  

    * **CIRCO:**

        |Methods|mAP@5|mAP@10|mAP@25|Mean Recall|
        |-|:-:|:-:|:-:|:-:|
        |Pic2word|8.72|9.51|10.46|9.56|
        |LinCIR [d]|12.59|13.58|15.00|13.72|
        |Image2Sentence|13.19|13.83|15.20|14.07|
        |CoVR [e]|21.43|22.33|24.47|22.74|
        |CoVR2|23.50|24.66|27.05|25.07|
        |**MAI (ours)**|25.18|25.91|27.22|**26.10**|

        [d] Gu et al. Language-only training of zero-shot composed image retrieval, CVPR 2024.  
        [e] Ventura et al. CoVR: Learning composed video retrieval from web video captions, AAAI 2024.

    **(3) Variable Turn Numbers and General Domain.** To evaluate multi-turn composed retrieval in the general domain, we employ the proposed Modification Generation Framework (MGF) to transform the CIRR dataset, which includes general objects and scenes, into CIRR-MT. This dataset consists of 30,000 transactions with varying turn counts (3-turn, 6-turn, and 8-turn), exceeding the scale of existing multi-turn datasets. The results are presented in the table below.

    |Methods|3-turn Recall|6-turn Recall|8-turn Recall|Mean Recall|
    |-|:-:|:-:|:-:|:-:|
    |FashionNTM|54.27|51.52|50.88|52.22|
    |DQU-CIR|57.95|55.13|54.11|55.73|
    |Context-I2W|56.37|53.47|52.31|54.05|
    |Image2Sentence|59.42|58.77|57.98|58.72|
    |SPRC|61.39|59.00|58.02|59.47|
    |**MAI (ours)**|65.07|64.04|62.31|**63.81**|

2. **Dataset Availability**

    We will release our dataset through a dedicated website upon acceptance of the paper. The website will include:
    * **Image Download Links**. Providing access to all images in the FashionMT dataset.
    * **Related Text**. Including the modified text and captions for the images.
    * **Documentation**. Covering data format, usage guidelines, and annotations.
    * **Terms of Use**. Outlining usage policies to ensure ethical and appropriate use of the dataset.
    * **Modification Generation Framework (MGF)**. We plan to release the proposed MGF, compatible with various LLMs (e.g., Llama-3.1-8B-Instruct and Qwen2.5-7B), facilitating the creation of tailored multi-turn scenarios.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper was reviewed by four experts in the field. The final ratings are 8,6,5,6. This paper studies the problem of Multi-Turn Composed Image Retrieval (MTCIR). It proposes a new dataset with the aim of addressing the limitation of existing methods which does not incorporate historical information. It also proposes a Multi-turn Aggregation-Iteration based on the new dataset.

Reviewers agree that this paper addresses an interesting problem. The proposed dataset is likely to be a valuable contribution. The proposed MAI model successfully addresses the limitation of previous methods. Overall, this paper is acceptable to ICLR.