Title: Towards Robust Uncertainty Calibration for Composed Image Retrieval
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Search

Composed Image Retrieval

Uncertainty Estimation
Abstract: The interactive task of composed image retrieval aims to retrieve the most relevant images with the bi-modal query, consisting of a reference image and a modification sentence. Despite significant efforts to bridge the heterogeneous gap within the bi-modal query and leverage contrastive learning to reduce the disparity between positive and negative triplets, prior methods often fail to ensure reliable matching due to aleatoric and epistemic uncertainty. Specifically, the aleatoric uncertainty stems from underlying semantic correlations within candidate instances and annotation noise, and the epistemic uncertainty is usually caused by overconfidence in dominant semantic categories. In this paper, we propose Robust UNcertainty Calibration (RUNC) to quantify the uncertainty and calibrate the imbalanced semantic distribution. To mitigate semantic ambiguity in similarity distribution between fusion queries and targets, RUNC maximizes the matching evidence by utilizing a high-order conjugate prior distribution to fit the semantic covariances in candidate samples. With the estimated uncertainty coefficient of each candidate, the target distribution is calibrated to encourage balanced semantic alignment. Additionally, we minimize the ambiguity in the fusion evidence when forming the unified query by incorporating orthogonal constraints on explicit textual embeddings and implicit queries, to reduce the representation redundancy. Extensive experiments and ablation analysis on benchmark datasets FashionIQ and CIRR verify the robustness of RUNC in predicting reliable retrieval results from a large image gallery.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to adopt uncertainty estimation into Composed Image Retrieval problems to address the problem of ambiguous and imbalanced semantics of the bi-modal queries. High-order evidential priors are employed for semantic distribution adjustment. Orthogonal independent constraints are applied to two branches of queries for disambiguation. Experiments on common datasets show its effectiveness under multiple metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n - The method was rigorously designed and theoretically analyzed to solve the significant uncertainty problem in CIR with great novelty. \n - The Experiments section have compared the proposed method with numerous methods on various metrics, while presenting ablations and discussions on both the proposed components and uncertainty estimation.\n\nWeaknesses:\n1. Clarity and motivation: \n - Some parts lack detailed mathematical definitions (assumption at Line 161-162), motivation statements, and citations of the core method used (Section 3.2), making the method section hard to read. \n - Temperature $\\tau$ is often writen as divided rather than multiplied due to it's meaning. In the paper it is written as multiplied.\n2. There is no subscript for variables in Eq.(12). Meanwhile, it is recommended to explain the meaning and effect of Eq.(12) as it's good for the clarity of the proposed Implicit Concept Matching.\n\nQUESTIONS:\n1. Why is it necessary to adopt an uncertainty estimation method for the problem?\n2. Why does the learnable prompts have unclear semantic concepts (Sec. 3.3)? Is there any evidence?\n3. What are the differences between the two branches of frozen and trainable text encoders? What are the differences between Eq.(1) and Eq.(11)? The variables are matched and using one single branch may also work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of semantic ambiguity and uncertainty in composed image retrieval (CIR), where a retrieval query consists of a reference image and a modification text. The authors identify that prior contrastive learning-based methods often fail to ensure reliable matching due to both aleatoric and epistemic uncertainty arising from implicit semantic overlaps and dataset biases. To tackle this, they propose RUNC (Robust UNcertainty Calibration), a novel framework that estimates and calibrates uncertainty using evidential priors, such as Normal-Inverse Gamma distributions, to balance semantic alignment. RUNC also introduces orthogonal constraints between explicit textual embeddings and implicit query signals to reduce redundancy and ambiguity in the fused query representation. Experiments on benchmark datasets like FashionIQ and CIRR demonstrate that RUNC achieves more robust and reliable retrieval rankings.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper proposes a novel uncertainty-aware composed image retrieval framework that integrates Gaussian regression modeling with implicit query supervision. The method demonstrates solid technical quality, with well-structured theoretical derivations and comprehensive ablation studies. The writing is generally clear, though the explanation of some component abbreviations (e.g., UE, UGC) could be improved.  The proposed approach addresses practical challenges in multi-modal retrieval, such as semantic ambiguity and label imbalance, and shows competitive performance on two benchmarks. The use of Normal Inverse Gamma distribution for uncertainty modeling and virtual guidance for implicit query alignment are relatively novel, although further clarification of the difference from prior contrastive methods would enhance the contribution. Overall, the paper is technically sound and presents meaningful and original contributions to the field.\n\nQUESTIONS:\n1.Is the Gaussian and i.i.d. assumption on semantic matching scores reasonable? The method assumes that semantic similarities follow an independent Gaussian distribution, which may not hold in real-world multimodal data with contextual and correlated semantics. Have the authors tested the robustness of the model when this assumption is relaxed?\n2.How essential is Q-Former to the overall performance? The Q-Former is reused from BLIP-2 with limited adaptation. Could the authors provide a baseline using simpler fusion (e.g., concatenation or attention) to clarify its actual impact?\n3.Are the learnable prompts in Q-Former interpretable? Given the role of learnable prompts in query construction, could the authors provide attention maps or qualitative analyses to show what these prompts are attending to?\n4.Could a progressive ablation be added to show module synergy? Instead of only showing removal of modules, a progressive build-up of the model (e.g., Base → +UE → +UGC → Full) would help demonstrate how each component contributes incrementally.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA novel method named RUNC (Robust Uncertainty Calibration) is proposed in this paper. The core idea of RUNC is to achieve more robust semantic matching by quantifying and calibrating uncertainty. This approach specifically addresses the unreliable matching results found in existing methods for Composed Image Retrieval tasks, a limitation primarily stemming from aleatoric and epistemic uncertainty. Aleatoric uncertainty stems from the data itself, while epistemic uncertainty is typically caused by overconfidence in dominant semantic categories. RUNC introduces Uncertainty Quantification and Semantic Calibration, alongside an Implicit Concept Matching module, to address these challenges.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. The RUNC framework, based on high-order Normal Inverse Gamma distributions for quantifying aleatoric uncertainty and epistemic uncertainty, is built upon a solid theoretical foundation.\n2. The ablation studies are very thorough, clearly demonstrating the positive contributions of RUNC's various components: Uncertainty Estimation, Uncertainty-Guided Semantic Calibration, Implicit Concept Matching, and Semantic Covariance, to the overall performance.\n3. The mathematical principles and workflow of RUNC's various modules – Uncertainty Perceptron, Uncertainty-Guided Semantic Calibration, Penalty on Misleading Evidence, and Implicit Concept Matching – are described in detail, complemented by an intuitive framework diagram (Figure 2).\n4. The paper innovatively introduces the NIG distribution, utilized for uncertainty quantification in evidential deep learning, into the specific task of CIR, skillfully integrating it with CIR task characteristics for handling semantic covariances and imbalanced semantics.\n\nWeaknesses:\n1. The theoretical analysis of \"high-order conjugate prior distributions\" could be more rigorously elaborated in the main text or provided as a more comprehensive overview in the supplementary material.\n2. A more comprehensive comparison with some of the latest SOTA methods is needed to more fully demonstrate the competitiveness of the proposed method.\n3. The specific network structure of the \"Uncertainty Perceptron\" module, including its number of layers, dimensions, and activation functions, is not sufficiently detailed in the main text. The explanation of the implicit query's generation process could be more concrete, including whether it is learnable, how it is initialized, and how its effectiveness in distilling visual semantics without redundant textual information is ensured.\n\nQUESTIONS:\n1. The specific network structure of the \"Uncertainty Perceptron\" module, including its number of layers, dimensions, and activation functions, is not sufficiently detailed in the main text. Could the authors provide a more precise architectural description of this component?\n2. Given that the paper's core is \"Robust Uncertainty Calibration,\" how robust is RUNC to input noise (e.g., slight corruption in reference images, typos in modification texts) or even adversarial attacks, beyond handling inherent data uncertainty? Does the model's uncertainty estimation remain accurate under these conditions?\nIf the authors can provide convincing experimental results demonstrating RUNC's robustness to input noise and adversarial attacks and accuracy of uncertainty estimation under these conditions, my Significance scores will increase.\n3. Considering that RUNC introduces additional modules, could the authors provide a more detailed analysis of its computational efficiency, including comparisons of training and inference times against key baselines? This would strengthen the argument for its practical applicability.\n4. The explanation of the implicit query's generation process, \"virtual guidance,\" could be more concrete. Specifically, could the authors clarify whether this virtual guidance is a learnable parameter, how it is initialized, and what mechanisms ensure its effectiveness in distilling visual semantics without redundant textual information? A pseudo-code or more detailed algorithmic steps would be highly beneficial.\nIf the authors can provide a clear and precise architectural description for the \"Uncertainty Perceptron\" module and offer a concrete explanation of the implicit query's generation process and mechanisms, my Clarity score will increase.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Robust Uncertainty Calibration (RUNC), a novel approach designed to improve composed image retrieval (CIR) by explicitly modeling semantic uncertainty in matching bi-modal queries to target images. RUNC quantifies uncertainty by using a Normal-Inverse Gamma (NIG) prior distribution to model the semantic covariances among candidate images. This estimated uncertainty is then used to calibrate the training objective, promoting a more balanced semantic alignment by focusing on ambiguous and rare concepts. To further enhance the query representation, RUNC incorporates an Implicit Concept Matching (ICM) module that reduces redundancy by enforcing orthogonality between textual and implicit visual components. Experiments conducted on the benchmark FashionIQ and CIRR datasets confirm that RUNC achieves robust and reliable performance, outperforming previous state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tThe paper introduces a novel framework for Composed Image Retrieval (CIR) that explicitly quantifies both aleatoric (data noise) and epistemic (model overconfidence) uncertainty using a Normal-Inverse Gamma (NIG) distribution to model semantic correlations.\n2.\tThe method uses the estimated uncertainty to create a balanced contrastive loss, which adaptively increases the penalty on ambiguous or rare samples, mitigating the influence of dominant semantics and improving learning from difficult examples.\n3.\tRUNC achieves robust and competitive results on two widely-used benchmark datasets, FashionIQ and CIRR, outperforming previous advanced methods.\nWeaknesses\n1. Implicit Concept Matching (ICM) appears only loosely connected to the paper’s central theme of uncertainty modeling. Although the authors claim it alleviates fusion ambiguity, it is unclear whether this ambiguity truly maps onto the aleatoric or epistemic uncertainty defined in the paper. ICM seems better characterized as an independent module for refining fusion representations rather than as an integral part of the uncertainty‑calibration pipeline. While beneficial, its role within the Uncertainty Calibration should be articulated more explicitly and positioned more cautiously.\n2. The calculation of the semantic covariance matrix requires computing pairwise cosine similarities between the features of all candidate samples within each batch. This operation could become computationally expensive and may impact training efficiency, particularly when using a large batch size.\n3. The clarity of the framework diagram in Figure 2 could be improved. Several key symbols, such as p, p*, and q* may be confusing for the reader. The inclusion of a formal legend or more descriptive labels to define these variables would significantly enhance the diagram's readability.\n4. Figure 4 only shows one successful retrieval case to intuitively demonstrate the uncertainty quantification of the model. Analyzing the instances where the model fails and their corresponding uncertainty scores will provide a deeper understanding of the limitations of the model and help verify whether the quantified uncertainty is truly effective in challenging scenarios.\n\nQUESTIONS:\n1. The connection between the Implicit Concept Matching (ICM) module and the paper's central theme of uncertainty should be clarified. \n2. To allow for a complete assessment of the method's practical trade-offs, the paper should include a quantitative analysis of the comprehensive computational overhead. A table comparing training time or GFLOPs against multiple baselines would enable a better evaluation of the model's efficiency.\n3. The qualitative analysis requires expansion beyond the single success case shown in Figure 4.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}