QuARI: Query Adaptive Retrieval Improvement
Authors: Eric Xing
Abby Stylianou
Robert Pless
Nathan Jacobs
Venue: NeurIPS 2025 poster ()
Keywords: image retrieval
text image retrieval
hypernetworks
multimodal models
instance retrieval
Primary area: deep_learning
TL;DR: Hypernetwork-based framework for dynamic adaption of precomputed database features.

Abstract: Massive-scale pretraining has made vision-language models increasingly popular for image-to-image and text-to-image retrieval across a broad collection of domains. However, these models do not perform well when used for challenging retrieval tasks, such as instance retrieval in very large-scale image collections. Recent work has shown that linear transformations of VLM features trained for instance retrieval can improve performance by emphasizing subspaces that relate to the domain of interest. In this paper, we explore a more extreme version of this specialization by learning to map a given query to a query-specific feature space transformation.  Because this transformation is linear, it can be applied with minimal computational cost to millions of image embeddings, making it effective for large-scale retrieval or re-ranking.  Results show that this method consistently outperforms state-of-the-art alternatives, including those that require many orders of magnitude more computation at query time.

QuARI: Query Adaptive Retrieval Improvement
Eric Xing'
Abby Stylianou
Robert Pless
Nathan Jacobs'
Washington University in St. Louis Saint Louis University
George Washington University
Abstract
Massive-scale pretraining has made vision- language models increasingly popular for umage-to-image and text-to-image retrieval across broad collection of domains. However, these models do not perform well when used for challenging retrieval tasks, such as instance retrieval in very large-scale image collections. Recent work has shown that linear transformations of VLM features trained for instance retrieval can improve performance by emphasizing subspaces that relate the domain of interest. In this paper, explore more extreme version of this specialization by learning to map a given query to a query-specific feature space transformation. Because this transformation is linear, it can be applied with minimal computational cost to millions of image embeddings, making it effective for large- scale retrieval re-ranking. Results show that this method consistently outperforms state-of-ther ttthhhhrtt art alternatives, including those that require many orders of magnitude more computation query time. Code and pre-trained models available https::/ bettss///gihhn gigthub....
1 Introduction
Recent advances in language-image pretraining have significantly improved performance in various vision-language tasks, including amage-to-image and text-to-image retrieval. These models learn align images and their corresponding textual descriptions in shared embedding space large-scale datasets. Yet, despite their success, they show notable limitations in retrieval scenarios.
Pretrained models often rely on global image features that encapsulate the overall content of an image. While effective for general classification tasks, these global representations may not capture fine-grained details essential for distinguishing between visually similar images based on specific textual queries [5T]. This limits retrieval performance when nuanced differences are critical.
To address the shortcomings of global feature representations, re-ranking methods have been proposed, which involve a secondary, more expensive analysis of the top retrieval candidates. Techniques such as two-stage retrieval pipelines [29] and re-ranking transformers (38) aim to refine initial retrieval results. However, these approaches often entail substantial computational overhead, making them impractical for real-time applications or large-scale deployments.
In this work, we propose novel approach that integrates query- query-specific embedding projections into the retrieval process. By dynamically adjusting the embedding space based on the input, our method prioritizes the most relevant fine-grained semantic alignments between text and image features for each specific query. Figure contrasts our query-specific retrieval approach with non-specific retrieval using a foundational vision- vision- language model like CLIP [30 and Nomainn-specife retrieval that has been adapted for a given dataset or task. Our framework, Query Adaptive Retrieval Improvement (QuARI), enhances retrieval accuracy without incurring the high computational costs associated with traditional re-ranking methods. Our specific contributions are as follows:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Domain Data
Query:
"aliforrtt Condor tagged with green 26*
Domain Adaptation Network
Query Encoder
Database Encoder
Query Encoder
Database Encoder
Query Encoder
Database
Query Adaptation Network
Non- Non-Specific (e-gr, CLIP)
Speaii-Specific
QuerffSpecific
We propose a new query-specitts approach to retrieval, QuARI. QuARI dynamically adapts embeddings per-query to significantly improve retrieval performance compared to non-specific retrieval with general-purpose embedding features like CLIP, and domain-specific retrieval with transformations learned for specific domain, with little computational overhead. Figure shows details of the Query Adaptation Network.
Figure 1:
We identify and articulate the limitations of contrastively pretrained models in capturing fine-grained details necessary for accurate retrieval, and analyze the inefficiencies of existing re- re-ranking methods and their impact on retrieval performance.
We introduce QuARI, an embedding projection framework that adapts global representations to individual queries, improving accuracy while maintaining computational efficiency.
We demonstrate that QuARI yields large improvements in retrieval accuracy multiple extremely challenging retrieval tasks.
2
Related Work
Text-Image Pretraining and Global Embeddings. Foundation models based language- pretraining, such as CLIP 30 and the SigLIP family models [0], have achieved remarkable success in aligning visual and textual modalities. These models learn global representations by maximizing the similarity between paired image-text inputs. However, the reliance on global features be detrimental in retrieval tasks that require fine-grained discrimination. Methods like RegionCLIPP [51] attempt to address this by incorporating region-level features, but challenges remain in capturing nuanced details essential for accurate retrieval. FILIP introduces token-wise late interaction to capture patch-word similarities [47), while SPARC sparsifies such interactions for efficiency in large-scale pre-training (5). Region-centric pre-training further bridges image-level and region-level semantics for aoce-voccbblary detection [15..
These methods highlight growing recognition of the limitations of global embeddings and attempt to address them through increasingly sophisticated alignment mechanisms. However, they still largely rely single static representation per image. In contrast, method is dynamic, modifying the representation space per query to prioritize the learned aspects that are relevant for the query while de-emphasizing irrelevant aspects.
Efficient Late-Interaation and Re-ranking Paradigms. Late- interaction retrievers (e.g., Col- BERT [4) decouple encoder computation from expensive vross-attentions, enabling scalable yet fine-grained passage search. In vision, there have been variety of approaches to re-ranking, includ- ing local feature based geometric re-ranking approaches (28 33 R4 37), and the recent Re-ranking Transformers (38), which refines top-k candidates with lightweight self-attention.

---

Methods like two-stage retrieval pipelines [29] and re-ranking transformers [88 refine initial retrieval results by re-evaluating top candidates with more sophisticated models. While effective, these approaches introduce significant computational overhead, making them unsuitable for real-time applications large-scale systems. Our approach instead shifts the complexity into a lightweight projection computed once per query. The approach is computationally lightweight enough that it can be used to re-rank very large numbers of candidates, or even entire datasets.
Hypernetworks. Hypernetworks, or networks that predict the weights of other networks, were first introduced by Ha et al. to generate recurrent neural network parameters on the fly [12] Recent ad- vancements have explored the use of hypernetworks for personalization in generative and task-specific models. HyperDreamBooth [32] introduces hypernetwork that generates customized weights for text-to-image diffusion models, enabling the synthesis of subject-specific images with minimal data. HyperCLIP [1] employs a hypernetorrk to generate the weights of task-specific image encoder. These approaches demonstrate the potential of hypernetworks in capturing customized semantics efficiently. However, reliance on task-level customization of entire encoders, in HyperCLIP, yields only modest performance gains on challenging retrieval tasks. Generating complete sets of encoder weights is both computationally expensive and difficult to optimize. In contrast, show that using hypernetworks to adapt aff-the-shelf features with lightweight, query-specific transformations can achieve strong performance without significant computational overhead.
Transformers as Hypernetworks. The concept of using transformers hypernetworks has been explored various domains. For example, Transformers as Meta-Leerrners 77] leverage transformers to predict the weights of implicit neural representations, showcasing their capability in dynamic weight generation. We draw inspiration from these works, employing transformer to predict query- specific projection matrices. In the following section, discuss how we tailor this idea to the retrieval setting.
3 Methodology
Our approach introduces a transformer-based hypernetwork for customized text-to-image retrieval. Unlike traditional methods relying on static embeddings. QuARI predicts customized linear projec- tions conditioned on each query embedding to adapt database features for each query.
3.1 Hypernetwork-Augmeein Retrieval
We begin by extracting global embeddings for queries and candidate images using pretrained vision-language model, such as CLIP [30] or SigLIP [50]. Formally, for query q and set of
"gallery" images G nn = 1, 2, ............... N). we obtain embeddings for query q. Enc(q) and database of gallery embeddings D (d1, d2, ..., dN} (Enc(J1), Enc(T2), Enc(((T))) We learn hypernetoork Hg as follows:
(q's
Ho(q:),
Which outputs transformed query representation q', and a transformation function T, Retrieval is then performed by transforming the database:
D', d''., d'x}
(2)
The transformed database of embeddings D', may then be used to perform retrieval by selecting database indices maximizing some embedding similarity between the query sim(q', (q', d';).
3.2 Hypernetwork Architecture
Our query adaptation network, shown in Figure is hypernetwork which we will denote as Hg. It
a transformer-based module that conditions on the query embedding q RR and predicts both a customized query embedding q' R and transformation matrix T E ExE to adapt image embeddings of dimension E. To constrain computation and promote generalization, we parameterize T' as low-rank matrix of rank 64.
3

---

Query Adaptation Network
Updated Query
Decoder
Query Token
begged
Query
Transformer Bookbone
Factors
Decoder
Output Transfomation
Figure 2: An overview of our query adaptation network. zero initialization of the transformation matrix is tokenized by columns and passed to transformer backbone with conditioning token to obtain refined columns. This process is repeated times, refining the transformation.
Tokenization of the Projection Matrix. low-rank decomposition:
We define the transformation matrix T via learned
T= Σ 4
where each pair (uj, V2) E I x E represents learned rank-one component of the projection matrix. These pairs are generated from a shared set of column-wise tokens, which are iteratively refined via transformer encoder. Intuitively, each token pair (u,, v,) defines single direction in the transformed feature space, contributing one semantic feature to the customized representation.
Initialization and Conditioning. We initialize a bank of 2r tokens per sample in the batch: U-tokens for projecting into the output space and r V-tokens for selecting features from the input space. The initial token representations are cero-initialized and refined over L denoising steps. To condition the generation process the query embedding, we first encode q using two-layer MI.P and add a learned timestep embedding to capture iterative update dynamics.
Iterative Refinement with Transformers. At each step E {1, L), concatenate the query- conditioned control token with the current U/V token sequence and apply shared transformer encoder. We also apply sinusoidal positional encodings [42 the token sequence. The output of the transformer updates the tokens via residual addition:
(++1)
u
+ Au
(5)
(++)
(1)
+ Av (4)
(6)
Decoding to Projection Matrices. Once the tokens are fully refined, we decode each U-token and V-token using separate MLP decoders produce vectors in RE The final transformation matrix is then constructed as:
T Σ (L; MV).
The customized query representation q' is decoded from the control token by another MLP.
3.3 Training
We train the model using a symmetric contrastive loss over the transformed query and image embeddings. Given minibatch of B text-image embeddings d first add noise to every query embedding to help bridge the text-image modality gap. We borrow this formulation from LinCIR [11). This acts regularizer that also allows high performance with image queries, training only on text-image datasets.
qi + q: + +( 1] \
(8)
Then, we compute the customized query text embedding q', and the personalization matrix T, using the hypernetwork Hg.

---

Semi-Positive Sample Mining, In the standard contrastive formulation, there is only one positive sample for each negative sample. This leads the learned transformations overfit to the query. This results in samples that are completely different from the target and samples which share some, but not all, attributes with the target having near-equivalent similarities to the transformed query.
In order to address this, we also compute a set of "semi-positive" samples P. for each target image. For every target image compute set of 100 nearest-neighbors using precomputed backbone embeddings and select the top 2 as "semi-positives." We apply a softmax over the distribution of these 100 cosine similarities and use their logits as target similarity values.
To train QuARI, we optimize a symmetric contrastive objective, as is standard for retrieval with semi- semi-positive samples:
Pij exp(Sr.))
(Sp())
q: dj
43.i
Sig
(9)
Σ
exp(Si,e)
exp(Sk..)
if (positive),
B
B Σ
B
C
Σ 2B
Os,j log Pi.j
-
Σ 1=1
ags log 93s
ai =
Wa,j if E P: (semi (semi-positive),
2=1 1=1
otherwise,
(10)
Where de is the i-th target image embedding transformed by the j-th personalization transformation T3, and is a temperature parameter. We only consider similarities where the embedding similarity of transformed query q' is only computed with target image embeddings transformed by T.
The incorporation of semi- cemi-positive samples discourages the behavior of overfitting to training query- target pairs, and encourages lower ranked images that seem visually similar to the target to be returned with higher similarity than images that lack visual similarity.
Implementation Details We use the Adam optimizer 233 with a cosine annealed learning rate eycling between 1e-5 and 2 and weight decay of 1e-2. Experiments were conducted NVIDIA H100 with 80GB of VRAM. At inference, retrieval is performed by computing cosine similarity scores between the L2-normalized adapted embeddings of a query and database images.
4 Evaluation
Evaluation Datasets. We focus our evaluation on two challenging benchmarks: ILIAS and INQUIRE. ILIAS (Instance-Level Image retrieval At Scale) large-scale dataset designed assess instance-level image retrieval capabilities [16]. has 1,000 object instances, each represented by query and positive images, totaling 5,947 manually collected images. To evaluate retrieval perfor- mance under large-scale settings, ILIAS includes 100 million distractor images from YFCC100M [40]. also includes retrieval task over curated of 5M distractor images, and an image-to-image retrieval re- re-ranking task.
INQUIRE [43 is text-to-imagee retrieval benchmark tailored for expert-level ecological queries. It built on the iNaturalist 2024 dataset, containing five million natural world images across 10,000 species. The benchmark features 250 expert-crafted queries spanning categories such species identification, context, behavior, and appearance. INQUIRE evaluates two retrieval tasks: INQUIRE- Fullrank, requiring models to perform retrieval over the entire dataset, and INQUIRE-Rerank, focusing on refining initial retrieval results.
We also provide results popular text-to-image and image-to-image retrieval benchmarks COCO [19] Flickr30k 48 FORB and TexICaps 35).
Training Datasets. Our framework can be trained on any paired text-image dataset. We utilize Mi- crosoft Common Objects Context (MS COCO) [19], Conceptual Captions 12M, and synthetically augmented Trove [46] to train QuARI MS COCO includes over 330,000 images annotated with
5

---

model
backbone
resolution
121 @ 100M
T21 @ 5M
T21 @ 100M
model
121 @ 100
CAI
VIT-B
224
4.2
2.7
1.6
CLIP TA
OAI CLIP QUARI
VII-B
224
(+4.2)
11.5 (+8.8)
(+7.6)
OAI CLIP QUARI
21.0
SigLIP
VIT-B
512
16.6
14.6
SigLIP
23.0
JP QUARI
VIT.B
512
24.6 (+8.0)
28.9 (+14.3)
25.6 (+14.5)
SigLIP QuARI
42.3 (+19.3)
SigLIP2
VII-B
SigLIP2 +TA
23.5
SigLIP2 QUARI
VIT-B
512
(+1002))
30.7 (+16.1)
27.2 (+16.8)
SigLIP2 QUARI
43.4 (+19.9)
OpenCTIP
7.0
OpenCTIP
13.7
OpenCLIP QUARL
VIT-L
15.6 (+6.2)
20.9 (+11.5)
18.7 (+11.7)
OpenCLIP QuARI
28.4
OAI CLIP
VII-L
336
9.4
8.4
5.8
CLIP -I OAI CLIP QuARI
15.2
OAI CLIP QUARI
VII-L
336
15.9 (+6.5)
20.6 (+12.2)
18.9 (+13.1)
31.1 (+15.9)
SigLIP
VIT-L
384
19.6
22.2
18.1
SigLIP + TA
28.9
SigLIP QUARI
VIT-1.
384
30.9 (+11.3)
36.5 (+14.3)
34.2 (+16.1)
SigL + QuARI
45.8 (+16.9)
SigLIP2
VIT-L
512
20.8
24.7
19.8
SigLIP2 TA
31.3
SigLIP2 + QuARI
VIT-L
512
36.2 (+15.4)
41.2 (+16.5)
38.7 (+18.9)
SigLIP2 QuARI
50.4 (+19.1)
(a) Comparison basclines.
(b) Comparison to static task adaptation (TA).
Table I: We show mAP@ @ Ik for image-to-image (12I) and text-to-image (T2I) retrieval on ILIAS.
model
COCO R@1
Flickr30k
FORB mAP@5
FORB I-mAP@s
TextCaps T21 R@1
SigLIP2
55.2
85,3
93.74
69.24
44.6
SigLIP2
55.0
85.5
92.89
70.03
45.2
SigLIP2+QUARI
77.4
92.9
95.67
78.53
55.8
Table 2: Comparison on other benchmarks with SigLIP2 ViT-L @ 512.
five human- humtn--ritten captions per image. Conceptual Captions 12M 6 is collection of approximately 12 million image-text pairs harvested from the web. Broeeove 46 contains 161.9 million images across approximately 366,600 species, each annotated with taxonomic data. We extract a random subset of 5M samples from BioTrove for training. Since the Brove dataset only includes taxonomic and common- name level annotations, we use Qwen2. Sien2.5-Vi.. 5-VL- 7B-L-BB-ii 77B--Instruc to caption a 500K subset of Trove, providing natural- languugg captions that include visual information beyond species identity (the prompt for constructing these captions is in the supplemental materials).
Metrics. On ILIAS, we measure mean Average Precision @ 1k (mAP@) 1k) across both the full 100M distractor set (@ (0M) and the 5M mini distractor set (@5M). On INQUIRE, we measure mean Average Precision @ 50 (mAP@ 50), Normalized Discounted Cumulative Gain @ 50 (nDCG@50), and Mean Recall Rank (MRR).
Baseline Models. We use popular contrastively pretrained backbone models CLIP 30 SigLIP LIP (50
OpenCLIP [33, and SigLIP2 41 backbone feature encoders. We replicate the baselines published with ILIAS for image-to-image re-ranking built on local feature descriptors from DINOv2 [27]. These include query expansion-basee methods like aQE S local feature and geometric-
matching- based methods like Chamfer Similarity (CS) and Spatial Verification (SP) [45). and transformer-based methods like AMES [37]. aQE-k refers to query expansion with & neiihbs-. We also replicate baselines from INQUIRE using vision-language models (VLMs) as re-rankers. including open-source VLMs LLaVA 22), VILA [18), PaliGemma [4), InstructBLIP [9), and BLIP2 [17].
Results
5.1 Embedding-based Retrieval
ILIAS. In Table Ia] we report the retrieval performance using mAP@ on the image-to-image and text-to-image tasks ILIAS. For each baseline row, there is corresponding row building QuARI

---

model
backbone
resolution
mAP@50
mDCG@50
MRR
VIT-B
224
10.4
OAI
QuARI
VIT-B
224
15.7 (+5.3)
25.4 (+4.5)
0.44 (+0.04)
OAI CLIP
VIT-L
336
23.4
37.7
0.59
OAI CLIP QuARI
VIT-L
336
32.5 (+9.1)
43.6 (+5.9)
0.64 (+0.05)
SigLIP
VIT-L
384
31.1
0.68
SigLIP+ QUARI
VIT-L
384
41.3 (+10.2)
54.7 (+8.1)
0.72 (+0.04)
SigLIP
SoViT-400m
384
34.2
49.1
0.69
SigLIP + QUARI
SoViT-400m
384
45.4 (+11.2)
56.8 (+7.7)
0.74 (+0.05)
Table 3: Comparison of QuARI and baselines on INQUIRE
Re-ranking Method
mAP@50
nDCG@50
MRR
Initial ranking
33.3
48.8
0.69
possihle re-rank
65.6
0.96
Re-ranking Method
mAP@ lk
Open-source VLMs
Initial Ranking
19.6
BLIP-2 FLAN-TS-XXL
31.2
46.5
0.58
BommucBBLPP-TS-XXI
33.0
0.64
aQEI
22.1
PaliGemma-3B-mm ---ix-444-
35.6
50.6
0.68
oQE2
20.4
L.AVA-1.5-13B
32.2
47.9
0.64
aQF5
14.3
LLaVA-v1.6-7B
32.3
47.9
0.62
CS (Chamfer Similarity)
22.9
L.IVA-v1.6-34
35.7
51.2
0.69
SP (Spatial Verification)
21.8
VILA-13B
35.7
50.8
0.65
AMES
26.4
VII.A-40B
40.2
$4.6
0.72
QuARI
29.1
Closed Source VLMs
(a) ILIAS Top-lk Re-ranking
GPT-4V
36.5
0.72
GPT-4o
43.7
57.9
0.78
SigLIP2 QuARI
45.7
55.2
0.76
(b) INQUIRE Top-100 Re-ranking
Table 4: Comparison Re-ranking Methods on ILIAS and INQUIRE
on the same frozen baseline as backbone feature encoder. Across all backbones, QuARI provides strong performance boost from off-the-shelf encoders commonly used for retrieval.
Insufficiency of Static Task Adaptation. A common method to improve the performance of large pretrained encoders is to adapt their features with a simple projection operation learned using dataset that is relevant the specific task [24 31). While this allows models to be adapted for that general task, we show that learning a task adaptation performs significantly worse than QuARI's query-specific adaptations. The authors of ILIAS show that task adaptation trained sample of IM images from the Universal Embeddings dataset 49] improves image-to-image instance retrieval in ILIAS 100M by between 3.7 and 10.5 mAP@ kk. In Table lb compare this task adaptation approach (TA) with a version of QuARI that was also trained on a IM image sample the Universal Embeddings dataset. QuARI shows significant improvements, between 13.1 and 19.7 mAP@ 1k, on top of task adaptation improvements.
INQUIRE. Table 3 shows retrieval performance across baseline model and backbone encoder sizes, along with corresponding QuARI models built upon their frozen features. In all cases, QuARI provides a significant performance improvement over general-purpose global features.
provides results on other popular text-to-image and image-to-image retrieval benchmarks. QuARI demonstrates strong performance improvements over both the off-the- shelf and fine-tuned SigLIP2 backbone model.
Other benchmarks. Table 2

---

ILIAS
INQUIRE
method
121 @ 100M
T2I @ 5M
mAP@50
nDCG
SigLIP2
20.8
24.7
37.2
52.3
Fine-tuned SigLIP2
21.1
25.1
38.9
53.1
QuARI w/o Itcrative Generation
28.5
32.1
43.8
54.0
QuARI w/o Semi-Positives
30.3
35.6
45.9
53.8
QuART w/o Noise
20.0
32,4
44.1
56.7
QuARI (query transformation only)
23.2
26.8
40.1
54.9
QuARI (database transformation only)
33.4
38.2
48.4
57.4
QuARI (query database transformations)
36.2
41.2
50.9
58.4
Table 5: Algorithmic ablation studies QuARI.
5.2 Re-ranking
In Table 4 compare QuARI to other approaches for image-to-image re-ranking. We replicate the baselines used by ILIAS [16], and pre-compute 100 local feature descriptors from DINOv2 [27] for each gallery image, and 600 local feature descriptors for each query image. We evaluate these methods on the image-to-image re-ranking task over set of t0p-1000 initial retrievals.
We demonstrate higher performance than baseline methods, most of which require computing and storing many local feature descriptors per query, while we only use global features that would typically already be stored in database index.
We also evaluate QuARI text-to-image re-ranking in INQUIRE against large VLMs. Table 4b shows that QuARI outperforms all open-source VLMs on text-to-image re-ranking on this task while using precomputed global features and much lower computational overhead (explored further in Section 5.5). QuARI is also competitive with closed-source VLMs GPT-4V and GPT-4o [25 26
5.3
Ablation Studies
We present ablations on both algorithmic design choices and hyperparameter selection.
5.3.1 Algorithmic Ablations
We use the SigLIP2 backbone as the frozen feature extractor and evaluate both image-to mmage-to-image and text-to-image tasks on ILIAS and INQUIRE. First, we compare the performance of the pretrained SigLIP2 model with models that are fine-tuned on the same datasets on which we train QuARI. This shows that fine-tuning alone, even on data from relevant domains, only provides small improvement. We also present ablations demonstrating that it is not sufficient to learn a model learning only query transformations, and that it relatively much more important learn query-speciff transformations of the database embeddings.
We then consider the performance of QuARI when remove different components of the algorithm. First, we remove the iterative generation process and use a one-step generation process instead, resulting 6.8 decrease mAP@ @ Ik on image-to-image retrieval @ 100M, and an 8.5 decrease in mAP@ 1k in text-to-image retrieval @ 5M on ILIAS, and a 6.9 decrease in mAP@ 50 on INQUIRE. Next, remove the consideration of semi-positive samples used during training. This results in degradation of 5.0 mAP@ Ik on both retrieval tasks on ILIAS and 4.8 mAP@ 50 on INQUIRE. Finally, we consider the case where noise is not added to the query representation during training to bridge the modality gap. This has the most significant impact performance, with a drop of 15.3 amage-to-image mAP@ 1K on ILIAS 100M, 8.2 text-to-image mAP@ on ILIAS 5M, and 6.6 mAP@ 50 on INQUIRE. Notably, on the image-to-imane task, QuARI without adding noise to the query representation during training does worse than the baseline SigL.PP model, indicating that with only text-image data, this method could be prone to over-fitting without the additional regularization.

---

rank
121 @ 100M
T21 @ 5M
semi-positives
121 @ 100M
T21 @ 5M
16
23.5
40.6
0
30.3
35.6
32
30.2
32.8
1
33.1
37.2
64
35.3
40.6
2
35.3
40.6
128*
33.6
38.9
3
35.1
39.8
256*
29.4
32.6
(b)
(a)
Table 6: Hyperparameter ablation studies on the rank of the QuARI projection (left) and the number of semi-positive examples considered during training (right). For some experiments, changed hyperparameters necessitated a decrease in batch size-we note those cases with an asterisk. We report all results using SigLIP2 ViT-L backbone.
5.3.2 Hyperparameter Ablations
Table 6 provides ablation studies on the key parameters r, the rank of the learned transformation, and the number of semi-positive samples used during training. We show that both increasing rank and increasing the number of semi-positive samples is beneficial until the batch size has to be reduced to train on single NVIDIA H100 GPU.
5.4 Embedding Visualizations
In Figure 3 explore feature transformations for two queries-an image query from ILIAS and text query from INQUIRE. To visualize the original embedding space, the middle panel of each row shows the SigLIP2 embeddings for a collection composed of the two queries, their corresponding ground -round-truth images, and 5,000 distractor images (2,500 sampled at random from each of the ILIAS and INQUIRE datasets). each row, the query embedding highlighted blue and the ground truth responses in red. The right panel shows the t-SNE embedding of the same set after query-specinee adaptation; here, the ground-truth responses are mapped much closer their query embedding.
5.5 Computational Efficiency
One of the primary strengths of QuARI is that adapts features from a precomputed database of off-the-shelf features. Figure 4 shows the time of the method. QuARI not only achieves state-of-the-ar performance, but is also very lightweight. On the ILIAS image-to-image re-ranking task, QuARI achieves around 3% improvement highest accuracy re-ranking approach in over two orders of magnitude less time, and over 6% better than approaches that have similar speeds without the use of auxiliary local feature descriptors. On the INQUIRE text-to-image re-ranking task, QuARI is almost 10% better than the best-performing vision-language model, and is orders of magnitude faster.
fixed-size query versus the performance
the
Limitations
An inherent limitation of our method is that it applies linear transformation to the retrieved results. While this design choice enables fast inference, restricts the expressiveness of the adaptation. If the original representation space lacks relevant features for the retrieval, a linear transformation of it will be insufficient to improve results. Additionally, like most re-ranking approaches, our method is constrained by the set of top-k results initially retrieved: it cannot recover relevant items that are excluded from this initial set. However, because QuARI extremely efficient, this top-k restriction is not limiting in practice: we can afford to deploy QuARI over very large initial set. Exploring non-linear transformation strategies to overcome representational limitations while maintaining computational tractability is promising avenue for future research.
7 Conclusions
In this work, we introduced a query-specific retrieval framework, QuART, that significantly outper- forms strong baselines, including large vision-language models and models with learned domain-

---

Query
Original SigLIP2 Features
QuARI Adapted Features
Datractors Ground Truths Query
"moray cel with open mouth poking head out of burrows or crevices"
Figure 3: t-SNE visualizations comparing original features and QuARI features.
28
ours
AMES
ours
47.
24
mAP.8
aQE1
SP
mAP@50
42.5
VILA-40B
22
aQE2
LLa VA.v1.1..334
37.5
VILA 138
18
PaliGemma-33
35.0
LLaVA 1.5 135
aQE5
325
InstructBLIP BLIP-2
14
LLaVA-v1.6-BB
30.0
101
10a
201
102
10
10f
15F
Time (ms) 1k image pairs
(ms)
text-image pairs
Figure 4: Comparison of re-ranking performance and inference cost for image-to-image retrieval on the ILIAS dataset (left) and text-to-image retrieval on the INQUIRE dataset (right).
specific adaptations, challenging retrieval benchmarks. By learning to predict transformations tailored to each query, our method significantly improves image-to-image and text-to-image retrieval performance without incurring substantial computational overhead. Unlike traditional re-ranking pipelines that rely on expensive local descriptors or multi-stage processing, our approach operates directly global embeddings and scales efficiently for searching large databases of images. Our results demonstrate that retrieval performance can meaningfully improved not by making the underlying encoders larger more specialized, but by learning lightweight, query encyy-conditioned adaptations of their features.
8 Acknowledgments
This material is based upon work supported in whole or in part with funding from the National Science Foundation (DGE-2112677, IIS- (IS-2441774) and the Department of Defense (DoD). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DOD/NSF and/or any agency or entity of the United States Government.
10

---

References
[1] Victor Akinwande, Mohammad Sadegh Norouzzadch, Devin Willmott, Anna Bair, Madan Ravi Ganesh, and J. Zico Kolter. Hyperclip: Adapting vision-language models with hypemetworks. 2024. URL https: http:::/aarxiv.ooo http:vv..rrr/bb////111/ abs/2412. 16777
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxis org/ www.abbs..a.../ 06450
[3] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[4] Lucas Beyer, Andreas Steiner, André S. Pinto, Alexander Kolesnikov, Xiao Wang, et al. Paligemma: A versatile 3b vision language model for transfer. Technical report, Google Research, 2024.
[5] Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, and Jovana Mitrović. Improving fine-grained understanding in image-text pre-training, 2024. URL https
//arxiv. www./rg///////eo/e///.../e..ee// abwpaa//s.... 2401. 09865
[6] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[7] Yinbo Chen and Xiaolong Wang. Transformers as meta-lcarners for implicit neural representa- tions. In European Conference on Computer Vision, 2022.
[8] Ondřej Chum, James Philbin, Josef Sivic, Michael Isard, and Andrew Zisserman. Total recall: Automatic query expansion with generative feature model for object retrieval. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-8, 2007. URL https: ht http:///appssemaniccccccolaalaee org/ (orpusID::5057:0
[9] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony M. H. Tiong, Junqi Zhao, et al. Instructblip: ceneral-purpose Towards general purpose vision- lasion-language models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS, pages 17246-17262 2023.
[10] Vinh Dang, Thanh-Son Nguyen, Minh-Triet Tran, and Duc-Tien Dang-Nguyen. Detecting mis- information in photos utilizing reverse image search. In Proceedings of the 2024 International Conference Multimedia Retrieval, pages 1321-1323, 2024.
[11] Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun. Language-onl training of zero-shot composed image retrieval. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[12] David Ha, Andrew Dai, and Quoe V. Le. Hypereetorkss. arXiv preprint 1609v:160.009100 2016.
[13] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP. https://doi.org/1055522 0200288///2ooooo5555577333 July 2021.
[14] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextu- alized late interaction over bert. In SIGIR, 2020.
[15] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-centri image- language pretraining for open-vocabulary detection. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
[16] Giorgos Kordopatis-Zilos, Vladan Stojnié, Anna Manko, Pavel Šuma, Dikolaos-Antonion Ypsilantis, Nikos Efthymiadis, Zakaria Laskar, Jiff Matas, Ondřej Chum, and Giorgos Tolias. Ilias: Instance-level image retrieval at scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
11

---

[17] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping language- -mage pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML), volume 202, pages 10439-10460, 2023.
[18] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 23178-23189, 2024.
[19] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pages 740-755. Springer, 2014.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), pages 16112-16128, 2023.
[21] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22926-22933 2024.
[22] Haotian Liu, Chunyuan Li, and the LLaVA Team. Llava-1.6: Release notes and model card. https://llaaa-vll.giie http:a....//alg//00000---------0 14 May 2025.
2025. Accessed
[23] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2017.
[24] Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe. Mini-model adaptation: Efficiently extending pretrained models new languages via aligned shallow training. In Findings of the Association for Computational Linguistics: ACL 2023, pages
2023.
[25] OpenAl. Gpt-4 technical report. Technical report, OpenAL, 2023. URL abs/2303. 08777 Describes the multimodal GPT-4V variant.
[26] OpenAI.
Introducing gpt-4o.
gpt- 40- 4o---d------------------------ more- tot--oot-tttt-eeeeeee 2025. Official launch post.
[27] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. 204v02300..07133, 2023.
[28] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In 2007 IEEE conference on computer vision and pattern recognition, pages 1-8. IEEE, 2007.
[29] Pinecone. Rerankers and two-stage retrieval. https://wwi http:///wwwiinecoe....m /i/ecoon..///aaam/serrsss//// rrag/rerankers/ 2023.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, and al. Learning transferable visual models from natural language supervision. In ICMI,, 2021.
[31] Elan Rosenfeld, Preetum Nakkiran, Hadi Pouransari, Oncel Tuzel, and Fartash Faghri. Ape: Aligning pretrained encoders to quickly learn aligned multimodal representations. In Advances in Neural Information Processing Systems (NeurIPS), 2022. htRt.tttps:///att org/ abs/
2210. 03927]
[32] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, and et al. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
12

---

[33] Shihao Shao, Kaifeng Chen, Arjun Karpur, Qinghua Cui, André Araujo, and Bingyi Cao. Global features are all you need for image retrieval and reranking. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11066-1146, 2023.
[34] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the- uff-the-shelf:: an astounding baseline for recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 806-813, 2014.
[35] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioningwith reading comprehension. 2020.
[36] Abby Stylianou, Hong Xuan, Maya Shende, Jonathan Brandt, Richard Souvenir, and Robert Pless. Hotels-50k: A global hotel recognition dataset. In The AAAI Conference Artificial Intelligence (AAAT), January 2019.
[37] Pavel Suma, Giorgos Kordopatis-Zilos, Ahmet Iscen, and Giorgos Tolias. Ames: Asymmetric and memory-efficient similarity estimation for instance-level retrieval. In European Conference on Computer Vision (ECCV), 2024.
[38] Fuwen Tan, Jiangbo Yuan, and Vicente Ordonez. Instance-level image retrieval using reranking transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2021.
[39] Owen Team. Qwen2.5-vl, January 2025. URL httpittts/:/qeennn.giieee bttichuu.oobblo///g/gg// quen2
5-v1/
[40] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 100m: the new data multimedia research. Commun. 59(2):64-77 Jamuary 2016. ISSN 0001-0782. doi: 10.1145/281 1802.. URL URL.https
//doi.org/10, 114100.11114288880000
[41] Michael Tschanne,, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding. localization, and dense features, 2025. URL (tRLshttss//// //tpxvv.//xxv.a org/eagg/ abs///aaa.//.///////////// 14786
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 17, page 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[43] Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel Brostow, Kate Jones, Oisin Mac Aodha, Sara Beery, and Grant Van Horn. Inquire: A natural world text-to-image retrieval benchmark. Advances in Neural Information Processing Systems, 37:1 167100000-116514, 2024.
[44] Pengxiang Wu, Siman Wang, Kevin Dela Rosa, and Derek Hu. Forb: a flat object retrieval benchmark for universal image embedding. Advances Neural Information Processing Systems, 36:25448 25460, 2023.
[45] Tong Wu, Liang Pan, Junzhe Zhang, Tai WANG, Ziwei Liu, and Dahua Lin. Density-aware chamfer distance as comprehensive metric for point cloud completion. In In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[46] Chih-Hsuan Yang, Benjamin Feuer, Talukder Jubery, Zi Deng, Andre Nakkab, Md Zahid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh Singh, et al. Biotrove: A large curated image dataset enabling ai for biodiversity. Advances Neural Information Processing Systems, 37:102101-102122 2024.
[47] Lewei Yao, Runhui Huang, Lu Hou, and al. Filip: Fine-grained interactive language-image pre-training. In ICLR, 2022.
13

---

[48] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[49] Nikolaos-Antonios Ypsilantis, Kaifeng Chen, Bingyi Cao, Mário Lipovský, Pelin Dogan- Schönberger, Grzegorz Makosa, Boris Bluntschli, Mojtaba Seyedhosseini, Ondiej Chum, and André Araujo. Towards universal image embeddings: A large-scale dataset and challenge for generic image representations. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11290-11300 11301, 2023.
[50] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975-11986, October 2023.
[51] Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region- language-image pretraining. Advances in Neural Information Processing Systems (NeurIPS), 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the main contributions of the paper: the introduction of QuARI, a query-specific hypernetwork for retrieval; its lightweight computational design; and its strong empirical performance on large-scale datasets like ILIAS and INQUIRE. These contributions are all supported by experimental results presented in Sections and 5. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 6 ("Limitations") explicitly discusses the primary limitations of the method, including the reliance on linear transformations which can restrict expressive power and the dependence on an initial shortlist from the base retriever. These reflections are honest and appropriately scoped given the claims made.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper includes comprehensive implementation details in Section 3.3 ("Training"), including model architecture, loss functions, optimization parameters, datasets used, and evaluation metrics. Additional configuration information such as token initial- ization, noise injection, and ablation variants are described in detail. Additionally, code to reproduce all experimental results will be released with the camera ready submission.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All code and datasets will be released upon acceptance of this paper, but not included with this submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
poblic/guides// CodeSubmissionooooo......
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://uuudeee/eeSSbiiiiiiii www.oodeuumiiisoonnee..
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: Section 3.3 and Section 4 provide details about training and evaluation setups, including hyperparameters (optimizer, learning rate, weight decay), backbone models used, dataset splits, and evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments conducted are too computationally expensive to run multiple trials. Following conventions for large-scale datasets, we report experimental results based single trial.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the most important details in the main paper, with more detailed section on hyperparameters the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///eeeeeeessccucuuttttt/tta
Answer: [Yes]
Justification:
The research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [Yes]
Justification:
We include section on Broader Impacts in the Supplemental Materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deg.oymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The models and datasets used in this work are not high-risk for misuse and involve sensitive restricted assets. Therefore, no special safeguards were necessary implemented.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors
should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models used (e.g., MS COCO, Conceptual Captions, BioTrove, CLIP, LigL are credited and cited with references in the bibliography. Their licenses (e.g.. open access research use) are respected.
Guidelines:
19

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaassttssppersseaae paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We primarily leverage existing assets (datasets and models) that are publicly available. The methods for generating the synthetic BioTrove data are included in the Supplemental Materials. We will release trained model weights upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or human subject studies.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer:
[NA]
Justification:
This paper does not involve any crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were only used for minor editing purposes and were not involved in the core methodology formulating ideas.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanntaddard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Implementation Details
QuARI's transformer backbone is randomly initialized with 4-8 transformer layers depending on the size of the backbone encoder. The query encoder and both the query and column decoders are two-layer MLPs with GeLU activation functions and layer normalization [ We train with a batch size of 320 and contrastive temperature of 0.07. All backbone model embeddings are precomputed before training.
B Data Generation Prompt
For all datasets other than BioTrove [6]. the provided natural language annotations as the text label. However, BioTrove does not provide natural language annotations outside taxonomic and common-name identities. Therefore, provide the species annotation along with the image to Quen2. 1.on2.5-VLL.. 5-VL--B-Innt-eee
( with the following instruction:
"For the image shown, write one plain, human- numan--ounning sentence that someone might type into an image search system to find this exact picture of a (species_ species_name). Mention the main objects, their key attributes, and any distinctive action setting. Keep it brief and objective, avoiding flowery descriptors unless they are essential to identify the scene. Output only this sentence."
We collect these annotations on 500K images sampled from BioTrove to augment our training dataset with natural language descriptions of Hodiversity-domal imagery.
C Broader Impacts
Improving retrieval systems to be both more accurate and more computationally efficient has broad positive implications, especially in domains where real-time or large-scale search critical - such as recognizing where victims of human trafficking are photographed [4]. monitoring biodiversity using camera trap images in ccological surveys (2). or identifying the spread of disinformation through manipulated visual media (3). QuARI enables high-quality retrieval even with limited resources, making advanced search capabilities more accessible in a wider range of applications. We do not foresee unique negative societal impacts associated with QuARI beyond those that already exist with general-purpont image retrieval systems. Nevertheless, the broader implications of visual search technologies swanloogges-includinnn potential misuse in surveillance or disinformation rremain important areas for ongoing community oversight and ethical consideration.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv. org/ abs/ /eees/167...0 0307.06450
[2] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[3] Vinh Dang, Thanh-Son Nguyen, Minh-Triee Tran, and Duc-Tien Dang-Nguyen. Detecting misinformation in photos utilizing reverse image search. In Proceedings of the 2024 International Conference on Multimedia Retrieval, pages 1321-1323, 2024.
[4] Abby Stylianou, Hong Xuan, Maya Shende, Jonathan Brandt, Richard Souvenir, and Robert Pless. Hotels-50k: A global hotel recognition dataset. The AAAI Conference on Artificial Intelligence (AAAI), January 2019.
[5] Qwen Team. Qwen2.5-vl, January 2025. URL htllitts://geeellgitthieie www.lliiggtuuuu..oooeeeb qwen2 5-v1/
[6] Chih-Hsuan Yang, Benjamin Feuer, Talukder Jubery, Zi Deng, Andre Nakkab, Md Zahid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh Singh, al. Biotrove: A large curated image dataset enabling ai for biodiversity. Advances in Neural Information Processing Systems, 37:102101-102120 2024.
22


Reviews:
[{"review_id": "YhsZSAeqap", "reviewer": "Reviewer_SPft", "summary": "The paper proposes QuARI, a query-specific retrieval framework that introduces query-adaptive linear transformations to enhance visual-language model (VLM) performance on retrieval tasks. While the empirical results show improvements over baselines, several critical issues undermine the novelty, and practical significance of the work.", "questions": "see weaknesses above", "limitations": "see weaknesses above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. **Comprehensive Evaluation**: The paper conducts thorough evaluations of QuARI across multiple challenging benchmarks (ILIAS and INQUIRE) and compares it against a diverse set of baselines, including recent vision-language models (VLMs) and task-adapted models. This breadth of assessment helps contextualize the method’s performance across different retrieval scenarios.\n2. **Insightful Ablation Studies**: The ablation experiments effectively validate key design choices, such as iterative refinement, semi-positive sample mining, and noise injection. By quantifying the impact of each component, the studies strengthen the rationale for QuARI’s architecture.\n\n### Weaknesses\n1. **Unclear Motivation** for Query-Specific Adaptation: The paper does not sufficiently articulate why query-specific adaptation is necessary beyond domain-level adaptation. It fails to clarify scenarios where per-query adjustments outperform static domain-specific transformations, leaving the core motivation underdeveloped.\n2. **Limited Conceptual Novelty**: The core idea of using query-specific linear projections to adapt embedding spaces lacks originality. While the paper frames QuARI as an \"extreme version\" of prior linear transformation methods, this distinction is superficial. The addition of a transformer network to leverage in-domain data aligns more with engineering-level optimizations than impactful scientific innovation.\n3. **Trivial Design Choices**: The proposed transformation matrix for document embeddings is under-motivated, particularly in the context of cosine similarity-based retrieval. Linear transformations in this space do not address fundamental limitations and is equivalent to linear transformations on query embedding.\n4. **Unfair Baseline Comparisons**: The comparisons to baselines often omit critical controls. For example, while Table 1 contrasts QuARI with \"static task adaptation (TA)\", the performance gains reported in Table 1(b) do not align with the trends in Table 1(a), raising questions about consistency. Additionally, the baselines do not adequately incorporate in-domain data, making it difficult to isolate whether improvements stem from QuARI’s design or superior use of training data.", "was_revised": true, "final_justification": "Given clarifications, particularly regarding the hypernetwork motivation, I revise my rating to positive", "rebuttal": "Thank you for your comments! We appreciate your acknowledgement of our comprehensive evaluation and effective ablation studies. We respond to your concerns below:\n\n> 1.  Unclear Motivation for Query-Specific Adaptation: The paper does not sufficiently articulate why query-specific adaptation is necessary beyond domain-level adaptation. It fails to clarify scenarios where per-query adjustments outperform static domain-specific transformations, leaving the core motivation underdeveloped.\n\nThe motivation for query-specific adaptation is that even within a well-defined domain, individual queries often highlight very different semantic aspects of the data. A single domain-level projection must generalize across all of them, often diluting features that may be crucial for a particular query. In contrast, a lightweight query-specific transformation allows the retrieval system to adapt its representation space dynamically to emphasize what matters most for that query.\n\nAs an example that might help understand our intuition in crafting QuARI: in instance-level image retrieval like in ILIAS—say, trying to find a specific red backpack—the query might depict it lying on a white floor, worn by a person, or placed in a cluttered room. While all are “backpacks,” the relevance of background features will vary by query. A query-specific transformation can downweight irrelevant elements (like clutter), or upweight contextual ones (like co-occurring objects or textures), depending on what distinguishes the instance in that scene.\n\nSimilarly, in the INQUIRE benchmark, queries span a wide range of fine-grained distinctions in the natural world. One user might ask for “a frog with bright blue legs,” while another requests “a frog camouflaged in brown leaves.” Although both refer to frogs, the first emphasizes color and appearance, while the second emphasizes environment and texture. A domain-level projection trained across many such examples may settle on an average representation that captures neither distinction well. A query-specific projection, by contrast, can prioritize the particular attributes mentioned in the query, whether that means focusing on visual details of the object itself or cues from the surrounding context.\n\nSetting aside the intuitional motivation, the results speak for themselves. Query-specific adaptations outperform both fine-tuning and learning of task-level projections by significant margins (as you can see in the original paper’s Table 4 for fine-tuning and in going from the baseline results in Table 1A to the Task Adaptation results in Table 1B for task adaptation). The two approaches are also not mutually exclusive. While we don't explore it in this paper, the potential combination of both domain-level adaptation and query-level adaptation is an interesting direction for follow-up work.\n\n\n> 2. Limited Conceptual Novelty: The core idea of using query-specific linear projections to adapt embedding spaces lacks originality. While the paper frames QuARI as an \"extreme version\" of prior linear transformation methods, this distinction is superficial. The addition of a transformer network to leverage in-domain data aligns more with engineering-level optimizations than impactful scientific innovation.\n\nWe acknowledge that learning linear transformations to adapt pre-trained embedding spaces is not a novel concept, and cite related work in that space in our paper. The novel aspect of QuARI, however, is the learning of **query-specific** database transformations. We respectfully disagree that this distinction is superficial and are unaware of any prior work that negates the novelty of QuARI. We show that it is both a highly performant and computationally efficient approach to improving retrieval performance, especially for extremely challenging retrieval domains. The core contribution of our paper is the thorough demonstration that learning of query-specific transformations is feasible with a transformer hypernetwork, and that this setup results in strong practical performance improvements without excessive computational cost. We are also excited to have expanded on these results (in response to the concerns of other reviewers), showing that the query-specific adaptation of QuARI yields significant retrieval improvement on a number of additional benchmarks.\n\n> 3. Trivial Design Choices: The proposed transformation matrix for document embeddings is under-motivated, particularly in the context of cosine similarity-based retrieval. Linear transformations in this space do not address fundamental limitations and is equivalent to linear transformations on query embedding.\n\nWe aren't performing a single linear transformation of the entire query space, but exploring the hypothesis that we can learn effectively a linear transformation of the embedding of the gallery images (and the query) based on the query itself.  In the context of the entire image retrieval problem, this is a highly non-linear transformation, but can be implemented very efficiently at run time because it is linear for each query.  Our results suggest that this approach is very effective.  Therefore, we argue that query-specific linear transformations do address the fundamental limitation that a pre-trained embedding space may not align with the semantic properties of a downstream retrieval task. General domain pre-training results in strong learned features that are useful for a diverse set of downstream domains, but the learning of features relevant to one task may result in the inclusion of features spurious to another task [1,2].  QuARI is effective because it learns to suppress these features that are irrelevant to the goal of the query. \n\n> 4. Unfair Baseline Comparisons: The comparisons to baselines often omit critical controls. For example, while Table 1 contrasts QuARI with \"static task adaptation (TA)\", the performance gains reported in Table 1(b) do not align with the trends in Table 1(a), raising questions about consistency. Additionally, the baselines do not adequately incorporate in-domain data, making it difficult to isolate whether improvements stem from QuARI’s design or superior use of training data.\n\nThank you for pointing out a source of confusion in the current version of the paper. “Backbone + QuARI” rows in Table 1(b) actually do not refer to the same model as the corresponding row in Table 1(a). Table 1(b) aims to compare the gap between linear adaptation of features on the task-level and the linear adaptation of features on the query-level (QuARI). So, QuARI in Table 1(b) refers to QuARI fine-tuned on a 1M subset of Universal Embeddings, while QuARI in Table 1(a) is trained without “in-domain” instance-level examples. We will clarify this distinction more carefully in the final version of our manuscript. \n\n---\n\n[1] Li, Weiwei, et al. \"Let Samples Speak: Mitigating Spurious Correlation by Exploiting the Clusterness of Samples.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025.\n\n[2] An, Bang, et al. \"More context, less distraction: Improving zero-shot inference of CLIP by inferring and describing spurious features.\" Workshop on Efficient Systems for Foundation Models, ICML. 2023."}, {"review_id": "juMPpJNluD", "reviewer": "Reviewer_nZKs", "summary": "This paper introduces QuARI, a novel approach for improving retrieval performance by dynamically adapting embedding spaces per query. QuARI learns to map each query to a query-specific feature space transformation that emphasizes relevant features for that particular query. The authors demonstrate that this linear transformation can be applied efficiently to large-scale image collections, achieving state-of-the-art performance on challenging retrieval benchmarks while requiring significantly less computational overhead compared to existing re-ranking methods.", "questions": "An interesting alternative approach worth exploring would be a training-free method using multimodal large language models (MLLMs) to encode query+image pairs for all images in the pool, highlighting query-relevant features. While this would be computationally intensive during inference (requiring MLLM application to all images for each query), it would eliminate training costs. A comparative analysis between this approach and QuARI could provide valuable insights into the trade-offs between training efficiency and inference efficiency in adaptive retrieval systems.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. Novel and Intuitive Approach: The paper presents an innovative approach for addressing limitations in vision-language models for retrieval tasks, offering a compelling middle ground between general-purpose and domain-specific retrieval methods.\n\n2. Good Performance: The proposed method demonstrates consistent and substantial improvements across various backbone models on different retrieval tasks.\n\n3. Computational Efficiency: The method achieves superior performance with minimal computational overhead compared to alternative re-ranking approaches, making it practical for real-world applications with large-scale image collections.\n\n4. Clear Presentation: The paper is well-written with clear mathematical formulations and is easy to understand.\n\n## Weaknesses\n\n1. Limitations of Linear Transformations: While the paper acknowledges this limitation, there could be more discussion about scenarios where linear transformations might be insufficient for capturing complex relationships in information-rich images.\n\n2. Ablation Study for Hyperparameter Selection: The choice of certain hyperparameters seems arbitrary and lacks empirical justification. For instance, the selection of top-2 image embeddings as semi-positive samples and the number of iterations (L) for the transformer refinement process.", "was_revised": true, "final_justification": "I thank the authors for the rebuttal and for making the effort to conduct additional experiments. It addresses my concern about the hyperparameter selection. I will maintain my positive rating.", "rebuttal": "Thank you for your comments! We are glad you found our approach innovative, strong in performance/efficiency, and clearly presented. We address your concerns as follows:\n\n> 1. Limitations of Linear Transformations: While the paper acknowledges this limitation, there could be more discussion about scenarios where linear transformations might be insufficient for capturing complex relationships in information-rich images.\n\nLearning nonlinear transformations would certainly allow for greater expressive potential in adapting features from pre-trained embedding spaces that have significant misalignment with a downstream retrieval task. Our work focuses on showing that even within the constraints of lightweight, linear database transformations, it is possible to achieve substantial performance gains through query-specific adaptation, making QuARI practical for large-scale deployment and low-latency retrieval. We see this efficiency as a key strength of our approach. That said, exploring more expressive nonlinear database transformation strategies, building on the nonlinear transform for the query already implemented, is an exciting direction for future work and one that may offer additional flexibility in more semantically complex settings.\n\n> 2. Ablation Study for Hyperparameter Selection: The choice of certain hyperparameters seems arbitrary and lacks empirical justification. For instance, the selection of top-2 image embeddings as semi-positive samples and the number of iterations (L) for the transformer refinement process.\n    \nThank you for pointing out a limitation of our evaluation. For the semi-positives, we compare the results without any semi-positives, and with one, two, and three semi-positive examples, showing that two semi-positives are significantly better than zero or one, and a little bit better than three. For some experiments, changed hyperparameters necessitated a decrease in batch size—we note those cases with an asterisk. We report all results using a SigLIP2 (ViT-L) backbone.\n\n\n| number semi-positives | I2I @ 100M | T2I @ 5M |\n| --------------------- | ---------- | -------- |\n| 0                     | 30.3       | 35.6     |\n| 1                     | 33.1       | 37.2     |\n| 2 (ours)              | **35.3**   | **40.6** |\n| 3*                    | 35.1       | 39.8     |\n\nWe will incorporate this ablation into the final paper. Regarding the refinement, we adopted an iterative refinement strategy based on recommendations from previous work [1,2].\n\n> - An interesting alternative approach worth exploring would be a training-free method using multimodal large language models (MLLMs) to encode query+image pairs for all images in the pool, highlighting query-relevant features. While this would be computationally intensive during inference (requiring MLLM application to all images for each query), it would eliminate training costs. A comparative analysis between this approach and QuARI could provide valuable insights into the trade-offs between training efficiency and inference efficiency in adaptive retrieval systems.\n\nThank you for the suggestion! The suggested approach would certainly allow the usage of MLLMs trained on larger datasets, but conducting inference over very large galleries (5M from INQUIRE, and 100M from ILIAS) becomes computationally infeasible. While this would serve as a potentially strong baseline as a reranking method, we leave the exploration of using MLLMs as retrieval rerankers to future work. \n\n---\n\n[1] Ruiz, Nataniel, et al. \"Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Alaluf, Yuval, et al. \"Hyperstyle: Stylegan inversion with hypernetworks for real image editing.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022."}, {"review_id": "fW4i3CwEGq", "reviewer": "Reviewer_MTco", "summary": "This paper introduces QuARI (Query Adaptive Retrieval Improvement), a query-specific retrieval framework that enhances image-to-image and text-to-image retrieval performance. The key idea is to use a transformer-based hypernetwork to predict query-specific linear transformations that adapt pre-computed global embeddings from frozen vision-language models (e.g., CLIP, SigLIP) for each individual query. The method employs iterative refinement with low-rank matrix factorization (rank 64) to generate transformation matrices efficiently. The authors evaluate their approach on two retrieval benchmarks (ILIAS and INQUIRE) and demonstrate significant improvements over baselines while maintaining computational efficiency compared to traditional re-ranking methods.", "questions": "- Rank selection: Why was rank 64 chosen for the low-rank factorization? What is the performance-efficiency trade-off for different rank values (e.g., 32, 128, 256)?\n- Semi-positive samples: How sensitive is the method to the number of semi-positive samples? What happens with 1, 5, or 10 semi-positives?\n- Standard benchmarks: Can you provide results on standard retrieval benchmarks like MS-COCO or Flickr30K to validate the generalizability?\n- Theoretical analysis: Can you provide theoretical justification for why low-rank transformations are sufficient? How does this relate to the manifold structure of embeddings?", "limitations": "The authors adequately discuss computational limitations (linear transformations only) and the top-k retrieval constraint. However, they should also acknowledge the limitations of evaluating on non-standard benchmarks.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Novel approach to retrieval: The paper presents an interesting idea of using query-specific transformations rather than static domain adaptation, which is conceptually appealing and shows strong empirical results.\n- Computational efficiency: The method is significantly more efficient than VLM-based re-ranking approaches while achieving competitive or better performance, making it practical for large-scale deployment.\n- Comprehensive evaluation: The paper includes comparisons with various baselines including domain-specific adaptations, local feature-based re-ranking methods, and VLM re-rankers.\n- Good ablation study: The authors provide ablation studies showing the importance of iterative generation, semi-positive samples, and noise injection.\n\nWeaknesses:\n- Questionable benchmark selection: The paper relies heavily on ILIAS and INQUIRE datasets, which appear to be very recent (2024-2025) with minimal citations (~1 citation each). This raises concerns about the reliability and representativeness of the evaluation.\n- Missing critical ablations:\n  - No justification or ablation for the choice of rank 64 for low-rank factorization\n  - No experiments with different numbers of semi-positive samples (fixed at 2)\n  - Limited analysis on why iterative refinement is necessary beyond showing performance drops\n- Insufficient implementation details: Critical details like MLP dimensions are missing, hindering reproducibility. The authors mention this will be released upon acceptance, but key architectural details should be in the paper.\n- Limited theoretical justification: The paper lacks theoretical analysis for design choices:\n  - Why is low-rank factorization necessary/beneficial?\n  - What is the theoretical basis for the iterative refinement process?\n- Narrow evaluation scope: Evaluation is limited to two non-standard benchmarks. Results on widely-used retrieval benchmarks (e.g., MS-COCO retrieval, Flickr30K) would strengthen the claims.", "was_revised": true, "final_justification": "The rebuttal addresses all my major concerns. The additional experiments on standard benchmarks show strong generalization, the ablation studies justify the design choices, and the theoretical explanations are satisfactory. The consistent improvements across diverse retrieval tasks demonstrate the practical value of QuARI.\n\nGiven these substantial improvements and clarifications, I am updating my rating from 3 (Borderline reject) to 5 (Accept). The paper now presents a well-validated approach with clear practical benefits and sufficient experimental rigor.\n\nI strongly encourage the authors to include all these additional results and explanations in the final paper, as they significantly strengthen the contribution.", "rebuttal": "Thank you for your review! We appreciate your recognition of our novel approach, strong results, and computational efficiency. We address your comments below:\n\n> - Questionable benchmark selection: The paper relies heavily on ILIAS and INQUIRE datasets, which appear to be very recent (2024-2025) with minimal citations (~1 citation each). This raises concerns about the reliability and representativeness of the evaluation.\n> - Narrow evaluation scope: Evaluation is limited to two non-standard benchmarks. Results on widely-used retrieval benchmarks (e.g., MS-COCO retrieval, Flickr30K) would strengthen the claims.\n\nWe appreciate the concern raised regarding the emphasis on the ILIAS and INQUIRE benchmarks. We selected ILIAS and INQUIRE explicitly because they are *extremely hard*. Pre-trained embedding models produce strong performance in general domain retrieval problems, with state-of-the-art models like SigLIP2 achieving 55.2 R@1 on COCO’s and 84.5 R@1 on Flickr30k on text-to-image retrieval with just a ViT-B backbone. \n\nBy comparison, ILIAS is so challenging that the retrieval results are reported as mean average precision ***at 1000***, with a pre-trained SigLIP2 model achieving only 19.8 mAP@1K on the text-to-image retrieval task. mAP@1 results are not reported, because they would be essentially 0. On INQUIRE, a pre-trained SigLIP model achieves just 34.2 mAP@50.\n\nSuch challenging real-world retrieval tasks are extremely compelling compared to benchmarks where the retrieval results are already quite good with pre-trained models, or with simple fine-tuning or task-specific learned adaptations. This motivated our design of QuARI. Dataset-level finetuning or task-specific adaptation on their own only provide a small boost (as you can see in the original paper’s Table 4 for fine-tuning, and in going from the baseline results in Table 1A to the Task Adaptation results in Table 1B for task adaptation). We designed QuARI’s query-level retrieval adaptation specifically to improve retrieval on such incredibly challenging problem domains.\n\nWe acknowledge, though, that reporting results on the best-known datasets helps the broader research community understand the contribution, so we have added evaluations on the following datasets:\n\n1. Requested Benchmarks:\n* Flickr30k\n* MS COCO — COCO was part of the training data for QuARI so we remade the training dataset without COCO and show evaluation results on that data (we leave the COCO data in the training set for all other benchmarks).\n2. Additional Benchmarks:\n* Flat Objects Retrieval Benchmark (FORB)\n* TextCaps\n\nAcross all of these new evaluation benchmarks, QuARI, built on SigLIP2 (ViT-L) features, yields significant improvements. We also report the performance of a fine-tuned ViT-L SigLIP2 model on the same training data as QuARI for fairness:\n\n| Model         | COCO T2I R@1 | Flickr30k T2I R@1 | FORB mAP@5 | FORB t-mAP@5 | TextCaps T2I R@1 |\n| :------------ | -----------: | ----------------: | ---------: | -----------: | ---------------: |\n| SigLIP2       |         55.2 |              85.3 |      93.74 |        69.24 |             44.6 |\n| FT SigLIP2    |         55.0 |              85.5 |      92.89 |        70.03 |             45.2 |\n| SigLIP2+QuARI |     **77.4** |          **92.9** |  **95.67** |    **78.53** |         **55.8** |\n\nAs a reminder, all of the evaluations use the exact same QuARI model (except for the COCO evaluation, which was re-trained to remove the COCO dataset from the training data—no additional training data was added, and it was trained in the exact same way as the model that included COCO data). This further demonstrates that our approach generalizes across a wide variety of domains without retraining. We really appreciate the suggestion for evaluation on COCO and Flickr30k and are excited to include these additional results in the final paper.\n\n> - No justification or ablation for the choice of rank 64 for low-rank factorization\n> - No experiments with different numbers of semi-positive samples (fixed at 2)\n> - Rank selection: Why was rank 64 chosen for the low-rank factorization? What is the performance-efficiency trade-off for different rank values (e.g., 32, 128, 256)?\n\nThe reviewer is correct that experimental evidence supporting the selection of these hyperparameters would strengthen the paper—thank you for pointing this out! We note that we constrain all experiments to fit on a single 80GB NVIDIA H100 GPU. For some experiments, changing hyperparameters necessitated a decrease in batch size—we note these cases with an asterisk. We report all results using a SigLIP2 (ViT-L) backbone.\n\nFor the choice of rank of the projection, we compare ranks of 16, 32, 64 (the rank included in the original paper), 128 and 256. The rank-64 projections yield the best performance:\n\n| rank | I2I @ 100M | T2I @ 5M |\n| ---- | ---------- | -------- |\n| 16   | 23.5       | 26.4     |\n| 32   | 30.2       | 32.8     |\n| 64   | **35.3**   | **40.6** |\n| 128* | 33.6       | 38.9     |\n| 256* | 29.4       | 32.6     |\n\nFor the semi-positives, we compare the results without any semi-positives, and with one, two and three semi-positive examples, showing that two semi-positives is significantly better than zero or one, and a tiny bit better than three:\n\n| number semi-positives | I2I @ 100M | T2I @ 5M |\n| --------------------- | ---------- | -------- |\n| 0                     | 30.3       | 35.6     |\n| 1                     | 33.1       | 37.2     |\n| 2 (ours)              | **35.3**   | **40.6** |\n| 3*                    | 35.1       | 39.8     |\n\nWe will incorporate these ablations into the final paper.\n\n\n> - Limited analysis on why iterative refinement is necessary beyond showing performance drops\n> - What is the theoretical basis for the iterative refinement process?\n\nOur use of iterative refinement is motivated both by practical gains and by prior work showing that complex structured prediction tasks can benefit from a progressive optimization process [1,2].\n\nIn our setting, the model predicts a 128-dimensional transformation basis from a query. This prediction is inherently difficult, as it must adapt a global embedding space to reflect fine-grained, query-specific semantics. As shown in prior hypernetwork-based work like HyperStyle [1] and HyperDreamBooth [2], iterative refinement allows the model to correct coarse or partially aligned initial predictions by re-evaluating and adjusting the predicted transformation in the context of its own outputs. \n\nTheoretically, iterative refinement aligns with the concept of fixed-point iteration or recurrent function application, where successive outputs of a function (or network) approach a stable solution. In practice, this leads to more accurate predictions in tasks where the output lies in a complex, high-dimensional space that cannot be captured effectively in a single forward pass. As seen in [1,2], a small number of refinement steps (<5 iterations) is sufficient to yield large improvements, balancing accuracy and efficiency.\n\nWe follow this paradigm and find that QuARI similarly benefits from refinement in both performance and stability.\n\n> - Insufficient implementation details: Critical details like MLP dimensions are missing, hindering reproducibility. The authors mention this will be released upon acceptance, but key architectural details should be in the paper.\n\nWe use a two-layer MLP with a hidden dimension of 512 and a GeLU activation function. We will add these details, along with sharing all code upon acceptance. \n    \n> - Why is low-rank factorization necessary/beneficial?\n> - Theoretical analysis: Can you provide theoretical justification for why low-rank transformations are sufficient? How does this relate to the manifold structure of embeddings?\n\nThank you for pointing out the limitations of the discussion in our paper. Low-rank factorization serves as both a regularizer and practical design choice for computational efficiency. As each u-v token pair defines a rank-1 component of the output transformation, learning a full-rank transformation would require prohibitively large amounts of GPU memory and inference time. \n\nLow-rank transformations are often sufficient in practice because learned vision-language embedding spaces tend to lie on low-dimensional manifolds within the high-dimensional ambient space. This observation has been made in general [3], in language modeling [4], and in visual representation learning [5, 6], where most of the task-relevant variation can be captured by a small number of dominant directions.\n\nBy constraining QuARI's transformation to be low-rank, we effectively bias the model to operate on these dominant directions relevant to the query while avoiding overfitting to spurious or noisy components of the embedding space.\n\n---\n\n[1] Ruiz, Nataniel, et al. \"Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Alaluf, Yuval, et al. \"Hyperstyle: Stylegan inversion with hypernetworks for real image editing.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Feng, Ruili, et al. \"Rank diminishing in deep neural networks.\" Advances in Neural Information Processing Systems. 2022.\n\n[4] Aghajanyan, Armen, et al. “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021.\n\n[5] Jaderberg, Max, et al. \"Speeding up Convolutional Neural Networks with Low Rank Expansions.\" British Machine Vision Conference. 2014.\n\n[6] Dong, Wei, et al. \"Low-rank rescaled vision transformer fine-tuning: A residual design approach.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024."}, {"review_id": "Yy2leTS8Mz", "reviewer": "Reviewer_UAyH", "summary": "# Methods summary\n\nThe paper tackles the problem of missing fine-grain capabilities when using global embeddings with their proposed approach, QuARI (Query Adaptive Retrieval Improvement). The key idea in QuARI is to adjust the embedding dynamically based on the input per query. This is achieved by using Hypernetworks H - the H model is trained to take the query embedding and produce a modified embedding q' and a transformation function T that is applied to all the gallery embeddings, and the similarliy function is applied to retrieve the closest matches. The transformation function T is a matrix learnt using low-rank decomposition, and the overall objective is realized through contrastive learning. The  CL formulation here is a semi-positive scenario where the top 2 nearest neighbours are considered semi-positive.\n\n# Experiment setup\n\nThe paper uses 2 benchmarks to evaluate on - ILIAS and INQUIRE. ILIAS evals include images from YFCC100M for distraction. For training, the paper claims any text-image dataset can be used and use MSCOCO, Conceptual Captions and BioTrove (images captioned by Qwen2.5-VL-7B-Instruct)\n\n# Results\n\nQuARI is shown to clearly improve upon the baselines in retrieval and re-ranking on both the eval sets. The ablations on each of the components of the QuARI neatly show the importance. Fig 4 also shows the promise of the method wrt time and performance.", "questions": "1. Can the authors add more details on the impact of the number of training data samples? The training data looks like a lot, and I wonder what the impact on performance is of dropping 1 or 2 of them? (and potentially adding these results)", "limitations": "Yes, the authors have addressed limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n\n1. The paper is well written with clearly presented details on contribution, implementation, and ablations highlighting the how different components contribute to the performance.\n2. The authors show strong improvement in numbers and closely match closed-source VLMs.\n\n# Weakness\n\n1. The authors could've provided analysis on the gap/regret to best possible re-rank and to the closed-source VLMs on where improvements could be gained. While the numbers between QuARI + SigLIP2 are close to closed-source, there is still a relatively large gap to the ideal re-rank. Some analysis on why that still happens could be really helpful to the readers.\n\n# Comments\n1. I hope the authors diligently release the code fi accepted as promised.\n2. Some papers like [1] should probably be cited if the authors find them relevant in the background section regarding adaptive retrieval and [2] for the semi-positive samples.\n3. The authors should also probably clarify who this is potentially useful for and limitations as to how \"fine-grained\" can it really be. \n\n[1] Vaibhav Balloli, Sara Beery, and Elizabeth Bondi-Kelly. 2024. Are they the same picture? adapting concept bottleneck models for human-AI collaboration in image retrieval. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI '24). Article 866, 7824–7832. https://doi.org/10.24963/ijcai.2024/866\n[2] Dwibedi, Debidatta, et al. \"With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "was_revised": true, "final_justification": "> The queries were addressed and maintaining my score", "rebuttal": "Thank you for your review! We appreciate your recognition of our strong improvements and clear presentation and evaluation. We address your comments below:\n\n> The authors could've provided analysis on the gap/regret to best possible re-rank and to the closed-source VLMs on where improvements could be gained. While the numbers between QuARI + SigLIP2 are close to closed-source, there is still a relatively large gap to the ideal re-rank. Some analysis on why that still happens could be really helpful to the readers.\n\nUnderstanding the gap between QuARI and the possible performance of an oracle is certainly interesting. We reviewed example queries with the largest gaps in performance between our results and the optimal results, and observe that they are simply quite difficult queries: images that include occluded subjects, poor lighting/angles, etc., and query texts that require domain knowledge, have complicated adjective lists, etc. We will share representative images that remain unretrieved by either QuARI in either the final paper or supplemental materials.\n\n> I hope the authors diligently release the code fi accepted as promised.\n\nWe will absolutely share our code and pre-trained models!\n\n> Some papers like [1] should probably be cited if the authors find them relevant in the background section regarding adaptive retrieval and [2] for the semi-positive samples.\n\nThese are excellent references and we will include them in our related work—thanks!\n\n> The authors should also probably clarify who this is potentially useful for and limitations as to how \"fine-grained\" can it really be.\n\nWe think this model is useful for anyone to perform image-to-image or text-to-image retrieval. We believe the model generalizes across many domains because we demonstrate strong performance in our paper, and are excited to present results on additional benchmarks in response to concerns raised by reviewers.  The following benchmarks are evaluated on:\n* Flickr30k\n* MS COCO — COCO was part of the training data for QuARI, so we remade the training dataset without COCO and show evaluation results on that data (we leave the COCO data in the training set for all other benchmarks).\n* Flat Object Retrieval Benchmark (FORB)\n* TextCaps\n\nAcross all of these new evaluation benchmarks, QuARI, built on SigLIP2 (ViT-L) features, yields significant improvements:\n\n| Model         | COCO T2I R@1 | Flickr30k T2I R@1 | FORB mAP@5 | FORB t-mAP@5 | TextCaps T2I R@1 |\n| :------------ | -----------: | ----------------: | ---------: | -----------: | ---------------: |\n| SigLIP2       |         55.2 |              85.3 |      93.74 |        69.24 |             44.6 |\n| FT SigLIP2    |         55.0 |              85.5 |      92.89 |        70.03 |             45.2 |\n| SigLIP2+QuARI |     **77.4** |          **92.9** |  **95.67** |    **78.53** |         **55.8** |\n\nNotably, there is no retraining for new domains—we use exactly the same model to translate the query into a projection matrix that refines the retrieval results (except for the model trained without COCO data in order to evaluate on COCO).\n\nWhile we have not yet conducted a fine-grained analysis of how “specific” the learned projections are, our experiments demonstrate consistent performance improvements across diverse datasets and query types. This suggests that the model learns broadly useful transformations that are self-adapting to varying retrieval goals. We agree that a deeper analysis of projection specificity would provide valuable insights, and we view this as an exciting direction for future work.\n\n> Can the authors add more details on the impact of the number of training data samples? The training data looks like a lot, and I wonder what the impact on performance is of dropping 1 or 2 of them? (and potentially adding these results)\n\nThank you for pointing out a very interesting dimension of evaluation! We explore this by investigating the performance of QuARI built on SigLIP2 (ViT-L) across random samples of 25%, 50%, 75%, and 100% of the training data. We summarize these results in the table below:\n\n| data sample | ILIAS I2I@100M (mAP@1k) | ILIAS T2I@5M (mAP@1k) | INQUIRE mAP@50 | INQUIRE nDCG@50 |\n| ----------- | ----------------------: | --------------------: | -------------: | --------------: |\n| 25%         |                    31.7 |                  36.6 |           47.4 |            56.7 |\n| 50%         |                    32.4 |                  37.5 |           48.2 |            57.1 |\n| 75%         |                    34.1 |                  39.2 |           49.5 |            57.8 |\n| 100%        |                    35.3 |                  40.6 |           50.7 |            58.3 |"}]

Author rebuttal:

Meta-review: This paper proposes QuARI, a framework that learns query-specific linear transformations of embedding spaces for image/text retrieval. The approach is lightweight, efficient, and shows strong improvements across both challenging new benchmarks and standard datasets. Reviewers initially raised concerns about novelty, reliance on new benchmarks, and missing ablations. After rebuttal, all reviewers acknowledged the clarifications, with one upgrading to positive and others maintaining acceptance.

Overall, QuARI offers a practical and well-validated contribution to adaptive retrieval. I recommend acceptance.
Decision: Accept (poster)