Title: QuARI: Query Adaptive Retrieval Improvement
Venue: NeurIPS 2025
Accepted: yes
Keywords: image retrieval

text image retrieval

hypernetworks

multimodal models

instance retrieval
Abstract: Massive-scale pretraining has made vision-language models increasingly popular for image-to-image and text-to-image retrieval across a broad collection of domains. However, these models do not perform well when used for challenging retrieval tasks, such as instance retrieval in very large-scale image collections. Recent work has shown that linear transformations of VLM features trained for instance retrieval can improve performance by emphasizing subspaces that relate to the domain of interest. In this paper, we explore a more extreme version of this specialization by learning to map a given query to a query-specific feature space transformation.  Because this transformation is linear, it can be applied with minimal computational cost to millions of image embeddings, making it effective for large-scale retrieval or re-ranking.  Results show that this method consistently outperforms state-of-the-art alternatives, including those that require many orders of magnitude more computation at query time.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Methods summary\n\nThe paper tackles the problem of missing fine-grain capabilities when using global embeddings with their proposed approach, QuARI (Query Adaptive Retrieval Improvement). The key idea in QuARI is to adjust the embedding dynamically based on the input per query. This is achieved by using Hypernetworks H - the H model is trained to take the query embedding and produce a modified embedding q' and a transformation function T that is applied to all the gallery embeddings, and the similarliy function is applied to retrieve the closest matches. The transformation function T is a matrix learnt using low-rank decomposition, and the overall objective is realized through contrastive learning. The  CL formulation here is a semi-positive scenario where the top 2 nearest neighbours are considered semi-positive.\n\n# Experiment setup\n\nThe paper uses 2 benchmarks to evaluate on - ILIAS and INQUIRE. ILIAS evals include images from YFCC100M for distraction. For training, the paper claims any text-image dataset can be used and use MSCOCO, Conceptual Captions and BioTrove (images captioned by Qwen2.5-VL-7B-Instruct)\n\n# Results\n\nQuARI is shown to clearly improve upon the baselines in retrieval and re-ranking on both the eval sets. The ablations on each of the components of the QuARI neatly show the importance. Fig 4 also shows the promise of the method wrt time and performance.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n1. The paper is well written with clearly presented details on contribution, implementation, and ablations highlighting the how different components contribute to the performance.\n2. The authors show strong improvement in numbers and closely match closed-source VLMs.\n\n# Weakness\n\n1. The authors could've provided analysis on the gap/regret to best possible re-rank and to the closed-source VLMs on where improvements could be gained. While the numbers between QuARI + SigLIP2 are close to closed-source, there is still a relatively large gap to the ideal re-rank. Some analysis on why that still happens could be really helpful to the readers.\n\n# Comments\n1. I hope the authors diligently release the code fi accepted as promised.\n2. Some papers like [1] should probably be cited if the authors find them relevant in the background section regarding adaptive retrieval and [2] for the semi-positive samples.\n3. The authors should also probably clarify who this is potentially useful for and limitations as to how \"fine-grained\" can it really be. \n\n[1] Vaibhav Balloli, Sara Beery, and Elizabeth Bondi-Kelly. 2024. Are they the same picture? adapting concept bottleneck models for human-AI collaboration in image retrieval. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI '24). Article 866, 7824–7832. https://doi.org/10.24963/ijcai.2024/866\n[2] Dwibedi, Debidatta, et al. \"With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\nQUESTIONS:\n1. Can the authors add more details on the impact of the number of training data samples? The training data looks like a lot, and I wonder what the impact on performance is of dropping 1 or 2 of them? (and potentially adding these results)\n\nLIMITATIONS:\nYes, the authors have addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces QuARI (Query Adaptive Retrieval Improvement), a query-specific retrieval framework that enhances image-to-image and text-to-image retrieval performance. The key idea is to use a transformer-based hypernetwork to predict query-specific linear transformations that adapt pre-computed global embeddings from frozen vision-language models (e.g., CLIP, SigLIP) for each individual query. The method employs iterative refinement with low-rank matrix factorization (rank 64) to generate transformation matrices efficiently. The authors evaluate their approach on two retrieval benchmarks (ILIAS and INQUIRE) and demonstrate significant improvements over baselines while maintaining computational efficiency compared to traditional re-ranking methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Novel approach to retrieval: The paper presents an interesting idea of using query-specific transformations rather than static domain adaptation, which is conceptually appealing and shows strong empirical results.\n- Computational efficiency: The method is significantly more efficient than VLM-based re-ranking approaches while achieving competitive or better performance, making it practical for large-scale deployment.\n- Comprehensive evaluation: The paper includes comparisons with various baselines including domain-specific adaptations, local feature-based re-ranking methods, and VLM re-rankers.\n- Good ablation study: The authors provide ablation studies showing the importance of iterative generation, semi-positive samples, and noise injection.\n\nWeaknesses:\n- Questionable benchmark selection: The paper relies heavily on ILIAS and INQUIRE datasets, which appear to be very recent (2024-2025) with minimal citations (~1 citation each). This raises concerns about the reliability and representativeness of the evaluation.\n- Missing critical ablations:\n  - No justification or ablation for the choice of rank 64 for low-rank factorization\n  - No experiments with different numbers of semi-positive samples (fixed at 2)\n  - Limited analysis on why iterative refinement is necessary beyond showing performance drops\n- Insufficient implementation details: Critical details like MLP dimensions are missing, hindering reproducibility. The authors mention this will be released upon acceptance, but key architectural details should be in the paper.\n- Limited theoretical justification: The paper lacks theoretical analysis for design choices:\n  - Why is low-rank factorization necessary/beneficial?\n  - What is the theoretical basis for the iterative refinement process?\n- Narrow evaluation scope: Evaluation is limited to two non-standard benchmarks. Results on widely-used retrieval benchmarks (e.g., MS-COCO retrieval, Flickr30K) would strengthen the claims.\n\nQUESTIONS:\n- Rank selection: Why was rank 64 chosen for the low-rank factorization? What is the performance-efficiency trade-off for different rank values (e.g., 32, 128, 256)?\n- Semi-positive samples: How sensitive is the method to the number of semi-positive samples? What happens with 1, 5, or 10 semi-positives?\n- Standard benchmarks: Can you provide results on standard retrieval benchmarks like MS-COCO or Flickr30K to validate the generalizability?\n- Theoretical analysis: Can you provide theoretical justification for why low-rank transformations are sufficient? How does this relate to the manifold structure of embeddings?\n\nLIMITATIONS:\nThe authors adequately discuss computational limitations (linear transformations only) and the top-k retrieval constraint. However, they should also acknowledge the limitations of evaluating on non-standard benchmarks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces QuARI, a novel approach for improving retrieval performance by dynamically adapting embedding spaces per query. QuARI learns to map each query to a query-specific feature space transformation that emphasizes relevant features for that particular query. The authors demonstrate that this linear transformation can be applied efficiently to large-scale image collections, achieving state-of-the-art performance on challenging retrieval benchmarks while requiring significantly less computational overhead compared to existing re-ranking methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. Novel and Intuitive Approach: The paper presents an innovative approach for addressing limitations in vision-language models for retrieval tasks, offering a compelling middle ground between general-purpose and domain-specific retrieval methods.\n\n2. Good Performance: The proposed method demonstrates consistent and substantial improvements across various backbone models on different retrieval tasks.\n\n3. Computational Efficiency: The method achieves superior performance with minimal computational overhead compared to alternative re-ranking approaches, making it practical for real-world applications with large-scale image collections.\n\n4. Clear Presentation: The paper is well-written with clear mathematical formulations and is easy to understand.\n\n## Weaknesses\n\n1. Limitations of Linear Transformations: While the paper acknowledges this limitation, there could be more discussion about scenarios where linear transformations might be insufficient for capturing complex relationships in information-rich images.\n\n2. Ablation Study for Hyperparameter Selection: The choice of certain hyperparameters seems arbitrary and lacks empirical justification. For instance, the selection of top-2 image embeddings as semi-positive samples and the number of iterations (L) for the transformer refinement process.\n\nQUESTIONS:\nAn interesting alternative approach worth exploring would be a training-free method using multimodal large language models (MLLMs) to encode query+image pairs for all images in the pool, highlighting query-relevant features. While this would be computationally intensive during inference (requiring MLLM application to all images for each query), it would eliminate training costs. A comparative analysis between this approach and QuARI could provide valuable insights into the trade-offs between training efficiency and inference efficiency in adaptive retrieval systems.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes QuARI, a query-specific retrieval framework that introduces query-adaptive linear transformations to enhance visual-language model (VLM) performance on retrieval tasks. While the empirical results show improvements over baselines, several critical issues undermine the novelty, and practical significance of the work.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. **Comprehensive Evaluation**: The paper conducts thorough evaluations of QuARI across multiple challenging benchmarks (ILIAS and INQUIRE) and compares it against a diverse set of baselines, including recent vision-language models (VLMs) and task-adapted models. This breadth of assessment helps contextualize the method’s performance across different retrieval scenarios.\n2. **Insightful Ablation Studies**: The ablation experiments effectively validate key design choices, such as iterative refinement, semi-positive sample mining, and noise injection. By quantifying the impact of each component, the studies strengthen the rationale for QuARI’s architecture.\n\n### Weaknesses\n1. **Unclear Motivation** for Query-Specific Adaptation: The paper does not sufficiently articulate why query-specific adaptation is necessary beyond domain-level adaptation. It fails to clarify scenarios where per-query adjustments outperform static domain-specific transformations, leaving the core motivation underdeveloped.\n2. **Limited Conceptual Novelty**: The core idea of using query-specific linear projections to adapt embedding spaces lacks originality. While the paper frames QuARI as an \"extreme version\" of prior linear transformation methods, this distinction is superficial. The addition of a transformer network to leverage in-domain data aligns more with engineering-level optimizations than impactful scientific innovation.\n3. **Trivial Design Choices**: The proposed transformation matrix for document embeddings is under-motivated, particularly in the context of cosine similarity-based retrieval. Linear transformations in this space do not address fundamental limitations and is equivalent to linear transformations on query embedding.\n4. **Unfair Baseline Comparisons**: The comparisons to baselines often omit critical controls. For example, while Table 1 contrasts QuARI with \"static task adaptation (TA)\", the performance gains reported in Table 1(b) do not align with the trends in Table 1(a), raising questions about consistency. Additionally, the baselines do not adequately incorporate in-domain data, making it difficult to isolate whether improvements stem from QuARI’s design or superior use of training data.\n\nQUESTIONS:\nsee weaknesses above\n\nLIMITATIONS:\nsee weaknesses above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}