Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization
Authors: Omri Uzan
Asaf Yehudai
roi pony
Eyal Shnarch
Ariel Gera
Venue: ICLR 2026 Poster (2026)
Keywords: Visual Document Retrieval
Test Time
Hybrid Retrieval
multimodal
RAG
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Testing hybrid retrieval for multimodal retrieval, achieving SOTA through test-time query optimization approach

Abstract: Multimodal encoders have pushed the boundaries of visual document retrieval, matching textual tokens directly to image patches and achieving state-of-the-art performance on challenging benchmarks. Recent models relying on this paradigm have massively scaled the dimensionality of their query and document representations, presenting obstacles to deployment and scalability in real-world pipelines.
Furthermore, purely vision-centric approaches may be constrained by the inherent modality gap still exhibited by modern vision-language models. In this work, we connect these challenges to the paradigm of hybrid retrieval, investigating whether a lightweight dense text retriever can enhance a stronger vision-centric model. Existing hybrid methods, which rely on coarse-grained fusion of ranks or scores,
fail to exploit the rich interactions within each model’s representation space. To address this, we introduce Guided Query Refinement (GQR), a novel test-time optimization method that refines a primary retriever’s query embedding using guidance from a complementary retriever’s scores. Through extensive experiments on visual document retrieval benchmarks, we demonstrate that GQR allows ColPali-based models to match the performance of models with significantly larger representations, while being up to 14x faster and requiring 54x less memory. Our findings show that GQR effectively pushes the Pareto frontier for performance and efficiency in multimodal retrieval. We release our code at https://github.com/IBM/test-time-hybrid-retrieval

GUIDED QUERY REFINEMENT: MULTIMODAL HYBRID RETRIEVAL WITH TEST-TIME OPTIMIZATION
Omri Uzan'* Asaf Yehudai22
Roi Pony?
Eyal Shnarch Ariel Gera²
'Stanford University 'IBM Research
'The Hebrew University of Jerusalem
uzan@stanford.edu
ABSTRACT
Multimodal encoders have pushed the boundaries of visual document retrieval, matching textual query tokens directly to image patches and achieving state- of-the-art performance public benchmarks. Recent models relying this paradigm have massively scaled the sizes of their query and document representa- tions, presenting obstacles deployment and scalability in real-world pipelines. Furthermore, purely vision-centrie approaches may be constrained by the inher- ent modality gap still exhibited by modern Laniunn models. In this work. we connect these challenges to the paradigm of hybrid retrieval, investigating whether a lightweight dense text retriever can enhance a stronger vision-centric model. Existing hybrid methods, which rely coarse-grained fusion of ranks or scores, fail to exploit the rich interactions within each model's representation space. To address this, we introduce Guided Query Refinement (GQR), a novel test-time optimization method that refines a primary retriever's s query embedding using guidance from complementary retriever's scores. Through extensive periments on visual document retrieval benchmarks, we demonstrate that GQR allows vision-centric models match the performance of models with signifi- cantly larger representations, while being up to 14x faster and requiring 54x less memory. Our findings show that GQR effectively pushes the Pareto frontier for performance and efficiency in multimodal retrieval. We release our code here
INTRODUCTION
Visual document retrieval is the task of returning relevant documents -typically PDFs containing figures, tables, and other visual elements- in response to textual query (Mathew et al. 202 a Li ct al. 2024 Zhu et al. 2022 Faysse et al. 2025). To tackle this task, neural retrieval pipelines offen Tollow texi-centric approach, relying OCR or vision-language models convert source docu- ments into textual chunks. and then constructing an index using semantic text encoders (Karpukhin al. 2020 Tanaka et al., 202I). An alternative, vision-centrie, approach relies instead on mul- timodal encoder models. Building on the ColBERT (Khattab & Zaharia (020) late-interaction approach, ColPali-basee (Faysse al. 2025) encoders operate directly on image patches, and yield multi-vector embedding representations of images and queries.
While this approach achieves state-of-the-art results on public benchmarks of visual document re- trieva] (Macé et al. 2025). open challenges within this paradigm remain. First, to pursue state-of- the-art performance, recent late-interaction multimodal retrievers massively scale the length and dimensionality of query and document representations. This incur substantial latency and storage overhead, hindering the ability to provide an efficient and scalable solution. For example, EEMMRRTRIEEERROOO
represents each document page with 10 MB of mem- ory (Xu et al. 2025), three orders magnitude more than single-vector dense retrievers (Table 4). Secondly, a vision-centric approach for matching textual queries textually rich documents may
TTis work conducted at IBM Research.
https:// http:///tgggggt.o...ooooo co/b) og/mmm//vvidoraev/ "Hencefort,, use "retrievers" and "encoders" interchangeably. For additional background on these emerging retrieval paradigms, refer to Appendix A

---

Rankings
Ranking- mvrl
Scores
0.78
0.34
0.69
Scare-l ewel
0.62
0.68
0.87
Guided Query Refinement
Representations
{ }
Represert Representation Level
d1
d2
d3
9
d2
d2
Retri ever 2
Retriever
Whizt was the Q2
growth?
Figure 1: Hybrid retrieval methods. Aggregating the outputs of two retrievers is typically done the level of ranks (22. or scores ($2.2). Utilizing the information of both representations effectively and efficiently difficult achieve in practice. Here we propose novel approach of Guided Query Refinement (GQR), using similarity scores from an complementary retriever (left) at test time, to inform the query representation of primary retriever (right).
be limited by the substantial modality gap CClavié & Brand 2025 Li et al. 2025 Role et al. 2025) exhibited by modern vision-language models. These gaps motivate exploring complementary ap- proaches for improving the performance of multimodal encoders.
An early concept the application of neural retrievers has been that hybrid retrieval (dos Santos itt al. 2015; Kuzi et al. (2020), where the outputs of different retrievers are aggregated at the level of ranks or query-document similarity scores (see Figure I obtain the final list of retrieved doc- uments. Hybrid retrieval (or hybrid search) most commonly refers to the combination of neural semantic text retriever with sparse lexical representation (e.g., BM25). More broadly, it reflects the notion that models relying on different types of representations can capture complementary aspects of the data, which can in turn leveraged to boost the overall performance on the task.
In this work, seek to connect these two threads, testing whether the paradigm of hybrid retrieval can complement modern multimodal encoders. Dense text retrievers typically have low online la- tencies and incur small storage footprint, and they provide uni-modal signal between the query and the document representation. Thus, hybrid retrieval between text and image encoders emerges natural candidate for enhancing the performance of multimodal retrieval system. However, in this work we argue that standard hybrid retrieval methods rely on a rather coarse-grained view of the perspective of each retriever they cannot utilize the rich query query-document interactions within the model representation space FFguure 1).
Aiming to harness this untapped potential of hybrid retrieval, in this work we propose Guided Query Refinement (GQR) (Figure 2). a novel approach for aggregating retriever outputs. Given a query at test time, GQR iteratively optimizes the query representations of primary retriever with gradient descent, using similarity scores from a complementary retriever. The refined query representation softly incorporates the complementary retriever's signal, remaining subject the query-documen interactions in the primary retriever space. This updated query embedding is then used to score the documents and return an updated document list. Notably, GQR architecture-agnosti and can be used to optimize both single- and multi vector query embeddings.
We conduct extensive experiments on established visual retrieval benchmarks, evaluating nine pairs of state-of-the-an vision and text retrievers and comparing GQR to standard hybrid retrieval ap- proaches. Our results (3.2) demonstrate consistent gains for models using GQR over base models and other hybrid baselines. Despite the fact that text-centric models achieve lower performance on the task, we find that the complementary signal they provide through GQR proves useful for ColPali-based models. On ViDoRe 2, WAW.OOI--EEEE-MMMMTMMMMMM
with GQR is nearly
2

---

while being x14 faster and requiring x54 less memory, and outperforms while being = x7 faster and requiring = x24 less memory. Our results and analysis establish that ColPali- ColPali-based methods using GQR are the latency and memory Pareto-fronts on the task of visual document retrieval.
on par with LLAMA-NEMORETRIETE DEELOOEEEREEEERRCCLLEEEEEEE
METHODS
Hybrid retrieval variants can be organized into three conceptual levels, reflecting the granularity in which test-time aggregation is performed (Figure the level of document rankings, the level of query-document similarity scores, or the level of embedding representations. The earlier the aggre- gation, the more information is available, and the more informative the exchange between models can be; however, richer information also increases the burden of normalization and geometrical alignment across spaces.
We explain that while early representation-llve aggregation could be desired due to its richnes,, it is difficult to achieve in practice. We then present our method, Guided Query Refinement (GQR), which lies between the levels of scores and of representations 2.3) and provide the motivation for it ($2.4).
We begin this section by outlining prominent methods at each level 22.1 $2.2).
Notations. Given a query and retriever m, we denote the representation of q by m as e Sim- ilarly, given set of documents D {d.} we have en for all dg E D. Document relevance to the query is estimated using similarity score 8m (g. d;) between the representations e% and e typically via cosine similarity (Equation 3) MaxSim (Equation 4).
Tm (q) denotes the list of the documents returned by retriever for query 9. K is the length of the retrieved list of documents, and we assume is constant across retrievers. rank (d) is the l-indexed position of Tm (q) after sorting nm (q) by the scores 8m (g, in descending order. If dl & (m) then rankm (g, d) = K11.
Finally, while the formulation general and applies to any number of retrievers M, in this work we focus on the case of M 2.
2.1 RANKING-LEVE AGGREGATION
Ranking-level aggregation is the simplest form of information exchange between retrievers: query and document pair is reduced to single integer rank. While limited in its expressivity, it re- quires no extra normalizations or alignments, and is therefore widely used in production pipeline
each
combines ranked lists by weighting each item based on the reciprocal of its rank. The RRF constant x > 0 dampens the impact very high ranks and controls how much credit is given to mid-list occurrences.
Reciprocal Rank Fusion (RRF). RRF (Cormack et al.
2009)
M
RRF(d)
Σ
(I)
K + rankm (d)
m=1
We also consider Average Ranking, which directly averages ranks across retrievers; see Equation for the formal definition.
2.2 SCORE-LEVEL AGGREGATION
Score-level aggregation operates one step deeper than ranking aggregation, operating on the real- valued similarity scores sm (g, d,) between the query and documents. To ensure that the scores of different retrievers are in the same scale and range, the common practice (Bruch et al., 2023) is to first apply a normalization function N,, yielding š. (g. d,) and then aggregate across retrievers:
1
8m (m. d.) - 11.mmpm (g. d.))
Score(g, d,)
Σ
sm (g. d:).
(2)
M
m=1
Milvus does; Flasticscarch does
3

---

In this work, we evaluate two variants with different normalizations, Score Aggregation (Min-Max) and Score Aggregation (SoftMax). See Appendix C for details.
Tuned Variants. More generally, the above methods can be viewed weighted aggregation, where the examples above are the uniform case, with each retriever assigned a weight a = 1 1/M (for two retrievers, a 0.5 for each). Given development set, these weights can be fit (Bruch et al.) 2023). Here, for M = 2, the two retrievers are assigned relative weights a and 1- o, yielding the parameterized variants Average Ranking Tuned, RRF- Tuned, Score Aggregation (Min-Max) Tuned, and Score Aggregation (SoftMax) Tuned. Details are in Appendix
2.3 GUIDED QUERY REFINEMENT
Representation-lewe information carries the richest potential for effective aggregation. Embedding- level projections that align representational spaces are used extensively in modern systems to combine visual and textual inputs (Radford et al. [021] Tia et al.) 2021 Li et al., [021] 2022). At test time, however, operating directly on representations is hindered by heterogeneity: encoders may use a single vector or many vectors per document and query, and they operate within differing dimensionalities and scales. Thus, with strict latency and memory budgets and without access to supervision, aggregation at this level is not trivial
Our goal in this work is to exploit the rich information in query and document representations while remaining architecture-agnese Query Refinement (GQR), a novel method for combining the outputs of two retrievers primary retriever my and a complementary retriever m2 (Algorithm Figure 2). GQR refines my 's query representation based on the signal of m2 's scores.
lightweight, and practical. To this end, we propose Guided
At inference time, given a user query an index search is run with each retriever to obtain its top- document list (g). The union of these lists, C(q) = U (g).). serves as the candidate pool of documents. For each retriever my E {m,, m2), we define distribution over C(q) via a Softmax:
exp (s,(g= d;))
py(de ee))
for i 1,, 1,... IC(g)).
C(o) exp (s,(4, dx))
We denote the initial query embedding of ma by 20) ef (GQR is applicable to both single and multi-vector embeddings), and we update it at each step t, 2 for T steps. At step t, the consensus distribution of my and m2 is defined as
1 (d) = 2 (pr (d (d) + p2(d eg)).
such that only p1 depends on & through (1) and p2 is fixed by eg.
We minimize
c KL(ph) (d) ll p 20)).
Here, KL is the Kullback-Leible divergence (Equation 12).
We apply a gradient step on the query representation with step size
((+1) 2 =v)
where T and a are hyperparameters.
We then compute the final scores from retriever m.,
$ (q. = (2( d) for E C(q).
Finally, we produce the list of retrieval results by sorting C(g) in descending order of s (q. (g,d), returning the first K elements.
'Concatenating singlc-vector embeddings is feasible, yet under dot-product scoring this reduces to an normalized sum of separate scores, and does not enable interaction between the spaces. We define GQR with gradient descent for simplicity, but in practice we found Adam (Kingma & Ba 2015 perform hetter and it the optimizer.
4

---

Output: Ranked list of most relevant documents based iteretively refined query
Final Ranking R(g) cups (gm)
{dg, dey dz)
Final Primary Scores - (20),) forde
Create Candidate Pool
(kg)=Iinion
Afor I lteaiion
Refined query
id 2
Ed dg
Compute update
Retriever Indexx
Retriever 2 Index
(Pos (Ct 20) (Cte)) x11) g000 a g -v prker
mb Consensus distribution
(Ct)) 2! (px 20) (C 9
hdl
d+b
Reiriever
Reil lever
Primary distribution (C(o): 20?)
Flored
What was the Q2 revenue growth?
0 cf
Aux distribution (((Q)))))
Stage 1: Candidate Pool Creation
Stage 2: Query Refinement
Figure 2: Guided Query Refinement (GQR). Stage 1: Two retrievers independently encode the query and retrieve top-K documents, forming a candidate pool. Stage 2: The primary query em- bedding is iteratively refined (z(*) over 7 iterations, by minimizing the KL divergence between consensus distribution and the primary distribution.
2.4 GQR MOTIVATION AND RATIONALE
Our approach is inspired by test time optimization methods that rely on pseudo-relevance feedback from stronger cross-encoder Yu et al. 2021: Sung et al. 2023 Gangi Reddy et al.) (025). Min- imizing KL pushes P1(- 2(4) to place higher probability where the other distribution does, and reduce probability where the other distribution low. However, instead relying on a heavy
cross-encoder, here we utilize feedback from a lightweight bi-encoder whose performance can be on par or even weaker than the primary encoder, and thus the consensus distribution set as Pavg
Compared to simple weighted average of scores, GQR operates by updating the query representa- (t) tion of m rather than defining new scoring rule. The complementary distribution Pavg is used as
learning signal, and the gradient of g(t) is computed through $1 (z, d), so any change in the ranking must be achievable by moving the query within the primary model's embedding space. Intuitively, this allows feedback from m2 to influence the query in a way that constrained by mi notion of similarity, since documents contribute to the update through their embeddings and their relative probabilities. In score-level fusion, each document's probability pulled toward the secondary re- triever's distribution by the same fixed amount, so the update is always uniform weighted average. In GQR, different documents can shift by different magnitudes along a non-linear trajectory dictated by the geometry of that space. In cases where my is weaker misaligned, this setup help in- corporate its signal more softly, since it is filtered through m2 's representation rather than directly overriding scores.
3 EXPERIMENTS
3.1 SETUP
Task. Visual document retrieval (Mathew al. 202.b Li al. 2024 Zhu et al. 2022 Faysse jet al. 2025) assumes a corpus documents, that contain visual elements such as charts, images, and tables, and set of document-grounnde textual queries. The goal is to retrieve the most relevant
5

---

Table 1: NDCG@5 over ViDoRe by primary and complementary models. Columns show scores by subset and the overall average. Deltas are absolute changes vs. the No refinement row within the same base.
Primary Model
GQR complementary model
Avg
Biomed Lectures
Economics
ESG Human
ESG Full
val
A
val
A
val
A
val
A
val
A
Jina (text)
53.4
0.0
48.6
0.0
51.4
0.1
59.5
0.0
54.1
0.0
Ling-Embed
55.3
0.0
58.0
00
32.0
0.11
58.8
0.0
52.4
0.0
Qwen3
46.8
64.0
00
44.6
0.0
50.2
0.0
38.3
0.0
Colnomic-7B
No refinement
60.3
0.0
61.3
0.0
31.4
0.0
68.2
0.0
51.1
0.0
Jina (text)
63.1
[-28
64.7
140.
57.0
142.6
70.3
1-2.1
60.2
146.
Ling-Emmbed
62.8
65.4
56.7
1+2.3
67.7
61.2
--7.1
Qwen3
1-0.7
J0.1
7+2.0
Jina (vision)
No refinement
57.2
61.6
33.5
61.7
52.0
Jina
60.7
1+0.1
35.3
1+18
66.9
0-5.2
58.8
Linq-Embed
61.2
7-4.0
64.7
1+3.1
1+3.7
65.7
0-4.0
9-5.1
Qwen3
59.8
1-2.6
63.2
141.6
53.6
140.1
67.8
146.1
54.4
1404
Jama-Nemo
0.0
74.5
0.0
Jina (text)
61.2
f+1.2
64.5
1+08
57.6
1+08
74.2
J03
60.4
143.5
Ling-Embeed
65.2
fx22
66.4
1427
56.8
0.01
74.6
1+0.1
62.8
-45.9
Qwen3
63.3
1-0.3
65.0
1+1.
56.4
74.1
J0.4
58.7
<+1.8
documents for each query. We conduct experiments on ViDoRe (Faysse al. 2025). ViDoRe (Macé et al. (2025) and the ViDoRe 3 benchmarks, which are established benchmarks for this task. Corpus documents are embedded by encoder models, either directly from page images, or following ingestion of document pages into text (see Apeendixxx D).
Models.
We evaluate a diverse pool of multimodal and textual state-of faaee-of-the oof-the-an art retrieval mod- els. The Colpali-bassed set includes three encoders:
(Team 2025b), JINA-EMBEDI UNN--EMEEDDNNGSYY (Günther et al., 2025). ) and EMBED-MISTRAL
COLEMBED-38 (Xu et al. 2025). The set text models includes LINQ-EMBED EMBED- MISTRAL (Choi ct al. 2024) and in its multi-vector textual configuration. This yields 3 text-based models and image- mod- els in total. See Table 4 for details the models. For the ViDoRe benchmark, excluded LLAMA-NEMORETRREVVER MAAARREERRBEEERCOLLEEEDDDD COLEMBED-3B evaluation impractical this larger benchmark.
(Zhang et al. 2025), as well as JINA- EMBEDDINGSS---- -V
due to its high computational overhead, which makes
Metrics and Evaluation. We NDCG@5 as the primary metric for evaluations (we also report @5 in Appendix F) For each ColPali-basee vision-centric model, we test each of the text-centric models the complementary retriever used for GQR. This yields 9 GQR pairs in total for ViDoRe and ViDoRe 2 and for ViDoRe 3, 3 per ColPali-based model. We also evaluate different hybrid methods for each vision-text model pair. See Sce/Appendix for technical details about tuning the hyperparameters for ViDoRe and ViDoRe 2.
To simulate real production use cases, we evaluate both tuned and zero-shot settings. We begin with the tuned setting on ViDoRe 1 and ViDoRe 2, and then use the findings from ViDoRe 2 to guide zero-shot evaluation on ViDoRe
3.2 COMPARISON TO SINGLE-RETRIEVE PIPELINES
Results with Hyperparameter Tuning. Table reports GQR against the corresponding models on ViDoRe 2. The first block lists the text-only models, which average between 46.8 for Qwen and 55.3 for Linq. each subsequent block, a ColPali-bas retriever is fixed, and we show its score alongside the GQR variants, with deltas computed relative the primary retriever. For Colnomic-

---

Table N over ViDoRe by primary and complementary models. Columns show subset scores, average, and deltas relative to the no-refinement baseline within each primary model.
Prinsry Medd
GOR compknneittry
AVR
CIXTR
fia on
Ge.Si
he
industial
phanma
plasics
model
vil
val
A.
Jin (lest)
$1.01
80
05.5
08
D4.2
20
64.0
ALLL
LO
54.5
422
23
65.3
00
40.5
6.3
Ling Eathod
462
64.5
48.G
20
40.3
28.7
42.5
50.1
07.7
41.4
Qwes3
07.6
50.1
$7.3
36.1
Calamio-78
74.0
20
49.5
02.0
Jinn (les))
57.
1407
74.5
-408
64.0
14.5
44.6
1439
59.4
1-23
SULS
1423
02.5
1433
44.6
1464
Ling Emrbed
57.1
140.
75.1
7+13
63.4
14.1
56.9
42.0
142.2
G5.3
50.8
63.4
1414
45.7
Jim (vilio)
refinement
33-4
81
69.2
000
59.8
50
52.4
38.8
100
17.8
61.2
-00
430
Jis (tss))
15.1
7667
70.2
5110
61.5
1121
26.5
42.6
142.4
57.1
1/04
06.1
110
61.6
1-34
6333
7165
Ling- Eimbee
55.4
72.0
122
6122
64.9
41.5
1407
68.81
6388
65.1
Table 3: Percentage gain in NDCG@5 of hybrid retrieval over the primary retriever on ViDoRe 2. Each cell depicts the average gain over 9 retriever pairs (3 multimodal base retrievers and 3 text retrievers).
Per-Subset Gain
Method
Avg
Sid
Biomed Lectures
Beonomics
Human
ESG Full
Average Ranking
4-3.0%
2.5
4-6.6%
f+0.5%
1-6.4%
1+0.4%
RRF
1-2.8%
2.6
1-6.5%
140.3%
1-6.2%
f+1.3%
Score Aggregation (Min-Max)
140.4%
2.6
1-3.0%
f+1.7%
1-1.8%
144.7%
Score Aggregation (Softmux)
f+1.55
1.9
1-0,9%
140.8%
f+0.6%
145,4%
Average Ranking (Tuned)
1-0.3%
1.8
1-2.9%
f+1.3%
1-2.8%
f43.2%
RRF (Tuned)
J-0.1%
1.9
L-2.9%
f+1.2%
1-4.0%
145.5%
Score Aggregation (Min-Max, Tuned)
1+3.45
2.1
f+1.2%
1+2.9%
1+2,9%
1+6.7%
Score Aggregation (Softmax, Tuned)
1+2.6%
2.3
1+0,3%
1+0.9%
1+2.2%
1+7.1%
Guided Query Refinement (GQR)
143.9%
1.9
1+1.5%
142.00%
143.3%
148.7%
7B, the average score rises from 60.3 to 63.1 with query refinement from Jina (text) (+2.8) and to 62.8 with LingeEmbed (+2.5). For Llama-Nemo, the strongest model ViDoRe date, GQR improves the average from 63.0 to 65.2 with Linq-Embed (+2.2). Notably, the text models clearly underperform the ColPali-based retrievers. yet with GQR the complementary signal they provide boosts performance. This is clearest in the Llama-Nemo versus Qwen3 setting, where GQR delivers small gain of 0-3.3, despite 16.2 point gap in base NDCG@5 (Llama-Nemo 63.0 vs. Qwen 46.8). Subsets where GQR harms performance are rare for Colnomic-7B and Llama-Nemo, absent for Jina (vision), and on average across the benchmark GQR consistently improves retrieval quality. We observe similar patterns for the Recall@5 metric (Table 12). On ViDoRe (Table 7 in the Appendix) GQR is generally on par with the base models, yet the benchmark clearly suffers from saturation, with many subset scores reaching 90 or higher.
Zero Shot. For ViDoRe 3, we use a single hyperparameter configuration across all sub- sets and model pairs, a choice informed by hyperparameter investigation top of ViDoRe 2 in ssubsection 4.1 Table 2) shows that GQR improves performance across all model pairs and bench- mark subsets, consistently outperforming the no-refinement baseline. For example, on Colnomic-7BB it raises the average NDCG@5 from 55.7 to 57.4 with Jina as the complementary model (a gain of +1.7), and delivers per-subset gains as high as +4.44 on fin.en. Similar improvements appear for Jina (vision), where GQR increases the average from 53.4 to 55.4 with Ling- (+2.0).
Results
3.3 COMPARISON TO HYBRID RETRIEVAL PIPELINES
Table depicts the aggregated performance of the different hybrid retrieval methods on ViDoRe 2, presented the average percentage gain relative to the base retriever over all pairs (Table 6 in the appendix lists the average absolute values). The Std column captures the standard deviation of performance across the pairs for each method. It shows that the ranking aggregation methods (RRF,
7

---

0.65
Base Model
Colnomic-7B
Liama Nemo
GoR (lna (acct)
0.64
(Owend)
ndcg@i5
0.63
60K (Ling Enteed)
0.62
CURRENT SOTA
0.61
G2R (Qwand))
100
200
300
500
1,000
2,000
4,000
Milliseconds per Query
Figure 3: cidency-quality tradeoff online querying. The x axis is runtime in milliseconds for single query, on log scale, and the y axis the average evaluation score (NDCG@5). Empty squares indicating the primary retriever alone (without applying GQR).
Average Ranking) generally lead to deterioration in performance. All aggregation methods benefit from parameter tuning, with tuned score aggregation methods achieving consistent gains over the base retriever. Table 5 in Appendix F shows the results in zero-shot settings on ViDoRe 3 (see Figure 7 for how the baseline zero-shot configuration is chosen).
Across the benchmarks, GQR outperforms all other hybrid retrieval variants, with the largest average gains over the base multimodal retriever, while also emerging as the most stable positive method with low standard deviation across pairs.
3.4 RESULTS- EFFICIENCY
Online Querying Latency. To characterize the test-time costs of GQR and ColPali-based retriev- ers, ran latency measurements randomly sampled queries from each ViDoRe subset (averaging across 100 runs). Figure 3/depicts the quality-latenge trade-offss of the different retrievers, where each point on the graph represents ei- ther a base retriever or retriever with GQR (Appendix Fishows full values). The plot illustrates the substantial costs of the strongest base model Llama-Nemo (orange square, right) which attains NDCG@5 62.9 at cost of 2,591 ms per query. Notably, our Colnomic GQR hybrid, with Linq as the refinement model (green diamond, left), reaches NDCG@5 62.7 at 181 ms ( 14x faster), and the Colnomic hybrid with Jina (green circle) attains NDCG@5 = 63.0 at 350 ms 7x faster), surpassing Nemo, Across base models, applying GQR increases latency by small relative measures with large gains in performance, shifting the Pareto frontier left and upwards. Appendix F shows that GQR is also on the index-storage Pareto frontier.
single NVIDIA A A100 GPU, measuring document retrieval for
ANALYSIS
4.1 HYPERPARAMETER INVESTIGATION
GQR relies on two hyperparameters the learning rate a and the number of optimization steps T. We evaluate performance over the Cartesian product {1 10-5, 5 x 10-5, 10 10 10-3} x (5, 10, 15,.... 90, 95, 100} of a and T'. We run this analysis over model pairs, excluding LLAMA-NEMORETERR CEREERREEEEE--CLEBEE.... COLEMBED-33 3B due to its computational overhead.
5 x 10-4, 10-3 x
Figure 4/depicts GQR performance on ViDoRe 2 for each combination of a and T, averaged over the 6 model pairs. It shows that high learning rates (10-3 3 5 10 are compatible with small T' values (10-25), where beyond that performance starts decreasing. Medium learning rates (10- 4, 5 x 10-5), show consistent improvement and stable optimization, yet require more steps (75-100) to reach
8

---

Reformance laverane)
T-22
150m
0.003
0.600
0.585
0.500
56.05
T-50
0.585
1e:03
Labencc
0.500
Minary Indeo search
Conglamentary
(nummer steps]
Figure 4: Hyperparameter sweep over GQR's learning rate a and optimization steps T', aver- aged over six model pairs on ViDoRe 2.
Figure Online latency breakdown of GQR for T 25 and T = 50.
higher performance. The smallest (10 5) and largest (5 x 10 learning rates are suboptimal, where the latter even results performance degradation relative to the primary retriever.
The results capture tradeoff between latency and stability. Higher learning rates provide performance boost faster, but might deteriorate quickly past a certain T value, while medium learn- ing rates more stable yet require more online optimization steps. Overall, three configurations emerge as the most promising for GQR: (1)
10-3 with 5 < T < 15, (2) a 5 x 10 with 15 < T' 30, and (3) 10-4 with 50 T < 75. We leave it future work to explore methods for optimizing this tradeoff, potentially by incorporating learning-ras scheduling procedures.
4.2 ONLINE LATENCY BREAKDOWN
Figure 5)
describes the online latency breakdown of GQR, for T = 25 and T = 50. The primary model used is WEEVOMIC-EMBBED-E Www-MMULTOOO-LPPA and the complementary is LINQ-EMS MANN-EMBED- MISTRAL. Latencies are collected with an A100 gpu over the ViDoRe 2 benchmark.
The test-time optimization accounts for 50 of 192 total latency for T 25 and 99 ms of 241 ms total for 7 = 50, averaging roughly 2 ms per optimization step. The procedure is shown here in sequential form, although with sufficient GPU memory the query encoding and index search for the two retrievers can be parallelized, reducing overall latency.
4.3 RERANKER COMPARISON
Cross-encoder rerankers are widely used in retrieval pipelines, applying full query to document interaction via self-attention the top-K documents. Similarly to GQR, rerankers operate at test time on a candidate pool returned by a bi-encoder, and thus provide a natural point of comparison. We evaluate GQR against Tiglttonai/MonnQWen22 --tweenn--V--O..... VL-v0.1, open-weights multimodal reranker (Chafiin & Lac 2024). Figure 6) iillustrates the antency-performanc characteristies of GQR against reranking the top-5 (on the left) and the top-10 (on the right) candidates returned by each retriever. GQR is run with a default K 10 configuration and with Linq-Embor as the refinement model. The left plot illustrates that GQR outperforms a top-5 reranking pipeline on both the latency and performance axes. Against the top-10 reranking pipelines (right), GQR achieves close performance while being 21x faster for Colnomic, 16x faster for Jina (vision), and 2x faster for Nemo. Across both comparisons, GQR remains on the Pareto front and indicates attractive latency performance trade-offs. We report the full results Table
4.4 GQR DESIGN CHOICES
Extra search stage. Prior works test-time query optimization often run a second index search with the optimized query 2(T), aiming retrieve documents beyond the initial top-K and increase recall (Sung et al. 2023 Gangi Reddy et al. 2025). This extra pass adds latency as it repeats index traversal and candidate generation. We test this modification GQR where perform an

---

additional search with the optimized query the full index, and observe improvement in performance (Table 16). This suggests that the effects of GQR in our setting are largely confined to the original pool C(q) of candidate documents.
Choice of objective. The GQR optimization process minimizes the KL-divergence between the distribution of the primary retriever and consensus distribution of the two retrievers, KT(pavg pt)- We additionally test two other loss functions: The ennen-SSannon divergence, JS(p2l p1) [KL(p2l Pavg) + { || Pavg) and KL with the target distribution p2. i.e. KL(p2ll pt). We find
Table 18) that these GQR variants generally perform similarly well, indicating that GQR applies across different loss formulations of the two distributions.
We conduct additional experiments on GQR design choices in Appendix E
RELATED WORKS
Multimodal Retrieval. Recent advances in visual document retrieval have been dominated by late-interaction, multi-vector architectures. This paradigm, first introduced to the multimodal do- main by ColPali (Faysse et al.) 2025), adapts the ColBERT framework (Khattab & Zaharia (020) by treating image patches visual tokens that interact with textual query tokens via MaxSim op- erations. Subsequent models built on this foundation such as Maaaa--eemoRerrivee--CCffff (Xu al., (025) and
(Team] 20256) have established state-of-the- art performance by capturing fine-grained interactions that are lost in single-vector representations. However, this performance comes at a significant cost: to achieve their results, these models rely on massively scaled representations, leading to substantial latency and storage overheads that can hinder practical deployment. This pressing trade-off between performance and efficiency motivates our work. Instead of pursuing ever-larger monolithic models, we investigate an alternative direc- tion: enhancing these powerful vision-centric retrievers by fusing their signal with a complementary lightweight text-based encoder at test time.
Hybrid Search. Numerous works apply hybrid retrieval in the context of combining dense seman- tic text retrieval with sparse lexical representations (Karpukhin et al. 2020 Kuzi et al. 20200] Luan al. 2021] Chen et al. (022).. Bruch et al. (2023) conduct theoretical and empirical analysis of the different ways to perform such dense-sparse fusions. Specifically, they compare RRF (Cormack) al. 2009) score-based fusion, and analyze the sensitivity to the choice of tuned weights and normalizations. [Hsu & Tzeng (2025) propose to set the score-fusion weights dynamically for each query at test time, based on (costly) feedback from an LLM judge. In contrast, GQR departs from these approaches by operating at test time on the representation level of the primary retriever.
Test-time query refinement. Prior work optimizes query representations during inference in text- only setups using pseudo-relevance feedback, often distilling from a cross- cross-encoder re-ra re-ranker to -nnge- dense retriever (Yu et al., 2021] Sung et al., [2023] Gangi Reddy et al., 2025). Cross- encoders provide rich interactions, but incur substantial test-time cost. Here we replace the cross- encoder with a complementary bi-encoder (possibly of a different modality), which preserves low latency while still providing a strong guidance signal.
6 CONCLUSION
In this work, we introduced Guided Query Refinement (GQR), a novel test-time hybrid retrieval method that refines the query representations of a primary retriever using signals from a comple- mentary one. Unlike traditional hybrid techniques that operate on rankings scores, GQR leverages representation-level interactions while maintaining efficiency and modularity.
Through extensive experiments on the ViDoRe benchmarks, we demonstrated that GQR consistently improves retrieval performance across diverse model pairs, pushing ColPali-based retrievers to the eeeency-meoryy Pareto frontier. Our findings highlight that even weaker retrievers can provide valuable complementary guidance, underscoring the potential for resource- efficiennt retrieval systems in multimodal large-scale settings.
10

---

REFERENCES
Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vage- nas, Cesar Berrospi Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, et al. Docling technical report. arXiv preprint arXiv:2408. 47X0::2408.09869, 2024. URL https://aaxiii www. abs/
2408 09869
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. PaliGemma: versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726. 2024. URL
wttp:/vv...g//a/2220..0000 2tt.a//224..7777 https: //wpiv///xivvva/aaa org/
Sebastian Bruch, Siyu Gai, and Amir Ingber. An analysis of fusion functions for hybrid retrieval.
ACM Transactions Information Systems, 42(1), August 2023. URL http:://pp:: .rrg/
10.1145/359511
Antoine Chaffin and Aurélien Lac. Monogwen: Visual document reranking, 2024. URL https:
//hugg ing.gg..gfac...o cog.gghhan.clligttcccnniiaaaaaaa hatp://aaaammoooooeeee------
Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, and Mare Najork. Out-of-domain se- mantics to the rescue! Zero-shot hybrid retrieval models. In Advances in Information Retrieval;
44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedings, Part I, pp. 95-1 110, Berlin, Heidelberg. 2022. Springer- 99735-9. URL btLpstps:: /oi. org/0/..rrg100. 1007 00000099777----99733333 99736-699976---
ISBN 978-3-030-
Chanyeol Choi, Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, and Jy-yong Sohn. Linq embed-mmaaa technical report. arXiv:2412.03223,, 2024. URL
kttpps:/ee wttp:///aaivvvv/a org/ abs / 03223
Benjamin Clavié and Florian Brand. Readbench: Measuring the dense text visual reading ability of vision-language models, 2025.
https/////aavvv..ccaaa////////// 19091
Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. Reciprocal rank fusion outper- forms condorcet and individual rank learning methods. SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pp- 758-759, New York, NY, USA, 2009. URL http: /ooi acm. www..aac..r//11...... o0t.....g//0..1115111111110 1572114
Cícero dos Santos, Luciano Barbosa, Dasha Bogdanova, and Bianca Zadrozny. Learning hybrid rep- resentations to retrieve semantically equivalent questions. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Pa- pers), pp. 694 699, Beijing, China, July 2015. Association for Computational Linguistics. URL www.eoooyoorrr//// Btrgg/P15. P15-2114/
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Celine Hudelot, and Pierre Colombo. ColPali: Efficient document retrieval with vision language models. In The Thirteenth International Conference Learning Representations, 2025. htthttps:: ptttpp:://peneee //ppnrevvew.
net:/forum Btt.:mmm22do00gpp23555333
Revanth Gangi Reddy, Pradeep Dasigi, Md Arafat Sultan, Arman Cohan, Avirup Sil, Heng Ji, and Hannanch Hajishirzi. A large-scale study of reranker relevance feedback at inference. In Proceed- ings of the 48th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, SIGIR '25, pp. 3010-3011 New York, NY, USA, 2025. Association for Com- puting Machinery. ISBN 9798400715921. URL
37301 60
Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureamu, Bo Wang, Sedigheh Eslami. Scott Martens, Maximilian Werk. Nan Wang, et al. jina-embeddings-- v4: Universal embeddings for multimodal multilingual retrieval. arXiv:2506. 18902, 2025. URL
https://erxxie wttp://rgg/bb//.oooo 10002289022

---

Hsin-Ling Hsu and Jengnan Tzeng. DAT: Dynamic alpha tuning for hybrid retrieval in retrieval-
augmented generation. arXiv:2503.23013, 2025. URL httph:tts//////t http://xxivv.rroo. org./.rrg/ooo abs//bb3/.//
23013
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint 1rXiv:21/2.09111 2021. bttp::tt:://artt orgg http//bb//112/// /t1.///./221111111
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learn- ing with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of ICML, pp. 4904-4916. PMLR, 2021. URL https://
proceedings mlr. pzees//139/jiallb...
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769 6781, Online, Novem- ber 2020. Association for Computational Linguistics. URL ht ht aatppp:s//cllaarooloooog.oo/ org/ 2020 wm-wmmnaalmmmm.mmmmm 5507
Omar Khattab and Matei Zaharia. COlBERT: Efficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 39-48, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015. URL
6980
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snyder Kaifeng Chen, Sham Kakade, Prateck Jain, et al. Ma- tryoshka representation learning. In Advances in Neural Information Processing Systems, Decem- ber 2022. URL httpa//ctppee/ppooreeeeesssseeee neera httpa//ddeeeeee.eere.roo.o..eeee httppaaarrrrrlleeeeaeeee///////e wawapeerr-ooeeeeeeeeeeee
Saar Kuzi, Mingyang Zhang, Cheng Li, Michael Bendersky, and Mare Najork. Leveraging semantic and lexical matching to improve the recall of document retrieval systems: A hybrid approach.
2rX1v:2010.01195, 2020. httts::::///axtt
wttpooor///aboaaaoa /tttpp/////aii...oorar/////0... 01195
Binxu Li, Yuhui Zhang, Xiaohan Wang, Weixin Liang, Ludwig Schmidt, and Serena Yeung-Levy. Closing the modality gap for mixed modality search, 2025. URL https::/axxiv.. org/
abs/2507.1905
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong. and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems, NeurlPS, 2021. URL https::/proceeeeeeeee
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888 -2900. PMLR, 2022.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal ArXiv: A dataset for improving scientific comprehension of large vision-language models. In Len-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14369- 14387, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL
https: http:://alaanthhhc watcchoooy.... ogy. org/ 2024 acl- 775
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329-345, 2021. SRI.http:: //acl wttthooloy..rgg//i/
tacl-all...... 120
12

---

Quentin Macé, António Loison, and Manuel Faysse. ViDoRe benchmark v2: Raising the bar for visual retrieval. arXiv:2505.17766, 2025. URL hRLlhtttss///aii..oomm
www.aa....22.... 17166
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Emnest Valveny, and C. V Jawahar. InfographicVOA, 2021a. URL wwtpthtpt//a//t httll//tpss/aa.....oo///aaaa//a
12756
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. DocVQA: A dataset for VQA docu- ment images, 2021b. URL httttttt/ttxiiiiiie http:///..ogggaa/////////// 60398
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint arXh::1910.14444, 2019. ULL/ttpp://axvvv..ooo//bb//
14424
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive pre- dictive coding. arXiv preprint 10XYv:807..03748. 2018. URL https://azxiv..ott wttp:iiivorr/aa... org/abs////aoooooooo....o......a
18C7.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2010.08191, 2020. URL https:
/ www.ii.org.abs./..oooooo. http:/ab////.....00 08191
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of /CML, pp. 8748-8763. PMLR, 2021. URL httpp:::/ppoceeeeeeeeeeee
mlr. press.prrs/e/.oo.....ee...e.. v139/ wwwforddfoooaaaaaaa htmmmammaa...mlmmmmmmmmmmmmmm
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982 3992, Hong Kong, China, November 2019, Association for Computational Linguistics. URL /tttps://aclatteen logy. org/D199-111 1410/
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Haacock-Beaullng Mike Gatford, et al. Okapi at TREC-3. British Library Research and Development Department, 1995.
François Role, Sébastien Meyer, and Victor Amblard. Fill the gap: Quantifying and reducing the modality gap in image-text representation learning, 2025. URL
2555. 03703
Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513-523 1988.
Keshav Santhanam, Omar Khattab. Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint
arXiv:2112.01488, 2021. Uttpstts:///axxv.. wttp://.gggabbb//.222../ abs/aals..nn/////// 01488
Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. PLAID: an efficient engine for late interaction retrieval. In Proceedings of the I ACM International Conference on Information & Knowledge Management, pp. 1747-1756, 2022.
Mujeen Sung, Jungsoo Park, Jaewoo Kang, Danqi Chen, and Jinhyuk Lee. Optimizing test-time query representations for dense retrieval. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings the Association for Computational Linguistics: ACL 2023, pp. 5731- 5746, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https:
findinngss findingss-cce 354/
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension document images. In AAAI, 2021.
Deep Search Team. Docling technical report. Technical report, BM Research, 8 2024. URL https: www.xwwwwrgaabs//22//.
13

---

IBM Research Team.
2025a. URL https:/tp//cc..cccc httttps:/huugingffaeee em.edingmb
20/ibm- /ttpa//wg..att///aaaaateeaaaaaaa
Nomic Team. Nomic embed multimodal: Interleaved text, image, and screenshots
for visual document retrieval, 2025b.
URL
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.121999 2024. URL URL https://arxiv org/ abs 12199119991111111
Navve Wasserman, Oliver Heinimann, Yuval Golbari, Tal Zimbalist, Eli Schwartz, and Michal Irani. Docrerank: Single-page hard negative query generation for training multi-modal rag rerankers. arXiv preprini arXiv:250..22544 2025. BRRLhttts://axiiii
http://bba22555...0 22505225544424 22584
Lee Xiong, Chenyan Xiong, Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Armold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint MxXiv:2007.00800 2020. BRLhhtps::/ http://axvvv.co org hbsg//bb//200
org
00808
Mengyao Xu, Gabriel Moreira, Ronay Ak, Radek Osmulski, Yauhen Babakhin, Zhiding Yu, Benedikt Schifferer, and Even Oldridge. Llama nemoretriever colembed: Top-performing text- image retrieval model. arXiv:2507.05513, 2025. URL httpppp://axxii.or//abbbb//////s
05513
HongChien Yu, Chenyan Xiong, and Jamie Callan, Improving query representations for dense retrieval with pseudo relevance feedback, 2021. URL
13454
Chengxiang Zhai and John Lafferty. A study of smoothing methods for language models applied to ad hoe information retrieval. In Acm sigir forum, volume 51, pp. 268-276. ACM New York, NY, USA, 2017.
Yanzhao Zhang, Mingxin Li, Dingkun Long. Xin Zhang. Huan Lin, Baosong Yang, Pengjun Xie, An Yang. Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Ad- vancing text embedding and reranking through foundation models. arXiv:2506.05/77 2025. URL htt.hhtttt///aatt
org/ abs/ 2506 05176
Fenghin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. To- wards complex document understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference Multimedia, pp. 4857-4866, 2022.
A BACKGROUND
Neural Information Retrieval represents a fundamental shift from traditional lexical matching meth- ods like BM25 (Robertson al.) 1995), TF-IDF (Salton Buckley) 1988), and other term-based approaches (Zhai & Lafferty (017). Unlike these sparse retrieval methods that rely on exact term matches and statistical properties, neural approaches learn dense semantic representations that cap- ture conceptual similarity, enabling retrieval based meaning rather than just shared vocabulary.
DENSE ENCODERS have transformed information retrieval by learning semantic representations of queries and passages shared embedding space. These models typically produce a single dense vector representation for each passage and query, enabling efficient similarity computation through operations like cosine similarity inner product. Early work by Karpukhin al. (2020) introduced Dense Passage Retrieval (DPR), demonstrating that dense representations could outperform tradi- tional sparse methods like BM25 for open-domain question answering. For query, q and passage p. The similarity score S is defined as:
9.
S(g.p)
(3)
|alpp]
14

---

The BERT-based bi-encoder architecture, where queries and passages independently encoded using separate or shared BERT models, became the foundational paradigm for neural retrieval. This framework has shaped the field, with numerous dense retrieval models building upon it (Reimers & Gurevych 2019 Xiong et al. 2020, Qu et al., 2020; Izacard et al. 2021). The bi-encoder design enables pre-computation of passage embeddings, making real-time retrieval feasible at scale. Re- cently, advanced models such as Lung-Embed-Ming
(Choi et al.) (022)) and Qwen3-Embedding
(Zhang et al. (0255) extend this paradigm by leveraging larger language models as the underlying encoder to create more powerful dense representations.
LATE-INTERACTION MODELS compute query-document similarity through more sinengranned in- teraction mechanisms. Rather than compressing all information into single vector, these models preserve individual token embeddings for both queries and passages. ColBERT (Khattab & Zaharia 2020 pioneered this approach with its MaxSim operation, which computes the maximum similar- ity between each query token and all passage tokens, then aggregates these scores. The MaxSim equation, as defined in ColBERT (see eq. 44) finds for query token embedding qi the maximum similarity (dot product) with any passage token embedding Pj. These maximum scores are then summed up to get the final relevance score.
S(q.p) = Σ
Pl max qi Pj j=1
(4)
This approach balances effectiveness and efficiency. as passage representations can still be pre- computed and indexed while the token-level matching captures finer details than single-vector ap- proaches. Subsequent work improved efficiency and retrieval quality through various optimizations (Santhanam et al. 2021 2022).
CONTRASTIVE TRAINING forms the backbone of modern retrieval model optimization. These methods learn representations by pulling positive query-passage pairs closer while pushing nega-
tive pairs apart in the embedding space. The InfoNCE loss (Oord et al., (2088), which maximizes the similarity of positive pairs relative to negative ones through a softmax-like normalization, has become the dominant training objective across retrieval architectures.
"xp(sim(q.p*)//
C log exp(sim(q, p+)//) Σ exp(sim((, pT
(5)
CROSS-ENCODER RE- RANKERS represent a different paradigm where query and passage are are jointly encoded, enabling deep self-attention between their representations. Unlike bi-encoders that in- dependently encode queries and passages, cross-encoders process query-passage pairs through single model, allowing full attention across all tokens. This joint encoding is computationally ex- pensive, making impractical for first-stage retrieval over large corpora. However, cross-encoders excel as re-rankers, refining the top-K results from efficient first-stage retrievers. Examples include MonoBERT (Nogueira al. (2019) for text retrieval and Chaffin Lac (0244) Wasserman et al. 2025) for visual document retrieval.
A.1 VISUAL DOCUMENT RETRIEVAL
VERBALIZATION--BAE METHODS were the dominant approach before the advent of end-to-end vision models. These pipelines convert visual documents into text through various techniques: tra- ditional Optical Character Recognition (OCR) tools like docling (Auer et al. 2024) extract printed text, while Vision-Languaa Models (VLMs) can generate textual descriptions of visual elements such charts, diagrams, and infographies. After verbalization. these methods apply standard text retrieval techniques to the extracted content. While verbalization- approaches can leverage powerful text-only retrieval models, they inherently lose spatial relationships and visual context during the text extraction process.
COLPALI ARCHITECTURES introduced a new approach to visual document retrieval by directly encoding document images without intermediate text extraction. These Viee-based embedding
15

---

Table 4: Different retrieval models evaluated in our work. The modality column describes the representation used for the documents page-level text, page-level images.
Vectors per Page
Taken Dim
Floats Page
Storage
Model
Size
1M Does (GB)
Page Modality
LINO-EMBED-MISTKE
7.1B
4096
4096
7.63
Text
TWEN3-EMBEDDING...
4B
2560
2560
4.77
Text
JINA-EMBEDDDINGS-VE (Text)
3.8B
128
Text
JINA-EMBEDDINGS--V (Image)
3.8B
767
128
98176
182.87
Image
COLLNOMIC-EMBBU--MUUMMMNON
767
98176
182.87
Image
MLAM--NaaoRErRuvvu=--LLEEEEEEEE 3B
4.4B
1802
3072
5535744
10311.13
Image
models transform pre-trained generative Vision-Language Models (such PaliGemma, Beyer et al. and Qwen- VL, Wang et al. 2024) into multi-vector embedding models optimized for retrieval.
the first model to introduce this approach, building upon PaliGemma (Beyer et al., 2024) and adapting the late-interaction framework to vision-language models by treating image patches visual tokens that interact with textual query tokens through MaxSim operations. ColPali provides native text-query support due to its VLM-based design. Queries remain text, are encoded by the model's language tower, and are matched directly against visual page/passage tokens, eliminating any OCR at query time. Training is contrastive, typically InfoNCE-style with in-batch/hard negatives to align query tokens with relevant visual tokens. This architecture preserves spatial layout information and visual features that verbalization-base ods discard. Following ColPali's success, subsequent models have adopted and extended this ColPali paradigm by leveraging different base VLMs: JJama-NemoRetriiver Reeeerieeve--CoEEmeeee
ColPali Faasse al., 2025)
meth-
ColEmbed (Xu et al.) 2025)) (Team 20256), Granite- Vision- Eebondnn (Team 2025a) and Jina-
Bestonic-Embed--Muue Embeddings-v4 (Günther et al. 2025
B MODEL INFORMATION
Table depicts the details of the models evaluated in our work. Storage assumes 16-bit represen- tation and follows official reports (Xu et al.] 2025). JINA-EMBEDDINGS-VA supports both text and image document representations, single-vector and multi-vector retrieval, and flexible embedding sizes via a Matryoshka scheme (Kusupati et al. (2022). In its multi-vector textual configuration, the number vectors per page for Jina varies between pages.
C ADDITIONAL DEFINITIONS
Average Ranking Computes the average rank across retricvers.
M AvgRank(d) Σ rankn (d)
(6)
Min-Max normalization
sm(g, -minn, mind,enm(()) (g. dj)
Sm (g. dz)
max,, m (n) Sm sm(g.dd (g) min (n) 8m (q, d,) + e²
with small for numerical stability.
Softmax normalization
exp(sm(g,d)) Ed, exp(sm(g. d,))
3m (g)
(8)
If d2 & mm (q) we set 5.m (n. d:)
16

---

Average Ranking- Tuned (for M=2)
AvgRank(d) a rankim, (d) (1 -la0) a) rank (d)
RRF- Tuned (for M=2)
2a
+ 2(1 (11-a) a)) K + rankma (d)
(RF(d) K. rankmi (d)
(10)
Score aggregation Tuned (for M=2)
Score(g, d4) a 8m (4, di) - a) 8mz (g, di)
(11)
KL divergence.
P(d) log P(d)
KT(PIIQ) E
(12)
Q(d)
dEC(2)
where and Q are distributions over C(g).
D IMPLEMENTATION DETAILS
Offiine indexing. For each model and dataset, we construct an offline index of page-level docu- ment representations. For ColPali-based models, multi-page documents are rendered page images and encoded directly. For text-only embedding models, for the ViDoRe and ViDoRe 2 datasets an ingestion pipeline converts each page text. We use Docling T Team 2024) ingest the images. Docling is an open library providing OCR capabilities combined with document layout analysis, allowing recover page content via simple function calls. The resulting text is stored alongside the page images without any chunking, ensuring consistent alignment hetween visual and textual page representations across the datasets. We run the document converter from Docling v2.34 using default parameters. For ViDoRe 3, we use the official markdown of the benchmark as the text input for the embedding models.
For SIN--EMBEDDINGS-Và (Günther et al., 2025), a single model accepts both image and text input documents. We thus use in both a vision configuration and in a text configuration, denoted Jina (Vision) and Jina (Text). both running in multi-vector setting. Jina (Text) is thus used to test the applicability of GQR where a multi-vector architecture used as the the GQR complementary encoder.
Hyperparameters For RRF we set x = 60, a common default (Chen et al. 2022 Cormack al.
2009). For tuning the weighted hybrid baselines and GQR, we follow previous works (Sung et al.
2023 Gangi Reddy et al. 2025; Bruch et al., (023) and rely on an in-domain development set of queries. We reserve 10% of each subset and tune the hyperparameters, T, and for each, select- ing by development set NDCG@5 performance. The splits are fixed for all experiments. For the weighted hybrid variants, the weight is tuned over {0.1, 0.2, 0.9) for each subset. For GQR, the learning rates are (1 x 10-5, 5 x 10-5, 10-4, 4 5 x 10-4 10 10-3 5 x 10-3), and we consider T {10, 25, 50). We report the selected hyperparameters for all methods in Tables 25 We opted for Adam (Kingma & Ba 2015) as the optimizer for GQR, based on preliminary experiments comparing different optimizer choices. We use K 10 across all retrievers and methods.
E ADDITIONAL DESIGN CHOICES FOR GQR
Candidate pool policy. In implementation the pool C(g) of candidate documents used for query refinement is a union of the top-K documents from the primary and complementary retrievers. One alternative is opt for reranker-like setup, where the only documents considered are those initially returned in the top-K of the primary retriever. Our analysis shows (Table 17) that this modi- fication does not have consistent positive negative effect compared
union pool of candidates. While the complementary dense retriever's 's index search is relatively quick, this configuration can be adopted sensitive deployments of GQR for additional latency gains.
17

---

k=
P Primary distribution on C(y) Consensus (average) distribution Compute the loss k 10
0.66
(shape)
(Shape)
0.62
0.00
0.38
100
200
300
500
1,000 2,000
4,000
100
200
300
500
1,000
2,000
4,000
(ybllon)
Liama-Ninee Nama-
Figure 6: Latency-quality tradeoff in online querying. The axis is runtime in milliseconds for single query, on a log scale, and the y axis the average evaluation score (NDCG@5). Marker color encodes the primary model; marker shape encodes the usage of GQR reranking with top 5 (left) or top 10 (right) candidates. The dashed lines represent the Pareto frontier.
Algorithm Guided Query Refinement (GQR)
Require: Query primary encoder mi, complementary encoder m2, iterations T, step size o,
top-K value K
1: 2(0) of
Initialize the primary encoder's query embedding & Union of per-encoder top-K lists
2: C(g) + CANDIDATEPOOL(G m, m2, K)
3: (12(g,d))
+ ScoREm, (q, C(q))
> Fixed guidance scores
4: p2(d, |cg) softmax (s2(g, d for d, E C(q)
Normalize m2 scores over C(g)
5: for 00 T do
Pi(d () t softmax (z(0) d4)) for di E C(q)
6:
7: Pavg(d, (d: 20())) + (p) (d. (0)) + p2(ds (e())
8: CKL t (Pav (d, 22)) p2(d: 2(1))
9:
20++) + 21) 2200 LKL
Gradient step on the query representation
10: end for
11: $1 (d.) + $1 (d; =1()) for d, C(a)
b Final primary scores after refinement
12: R(g)= t topk dec()) s/" (q,
Return ordered top- by score
13: return R(g)
Primary and complementary roles We evaluate text-centric and vision-centric retrievers both primary and complementary retrievers within GQR, reporting results in Tabll 19 Across model pairs, both role assignments improve over the base retrievers. The alternative role configuration yields larger gains relative to the primary encoder alone, while the default GQR attains the strongest absolute score on ViDoRe 2.
F ADDITIONAL RESULTS
Tabset/s the results on ViDoRe is noticeable that this benchmark suffers from saturation, with many subset release of ViDoRe 2, Maaé et al. 2025). Nevertheless, our method does not harm performance, in contrast other hybrid retrieval methods, seen Tables and
reaching 90 higher (and indeed this was the direct motivation for the
Storage. Figure in the Appendix shows that the added storage in GQR is modest. As in the latency plot, GQR dominates the strongest base model by a wide margin. The Llama-Nemo index represents each document with 10.6 MB of memory, whereas the Colnomie hybrid, using Ling as
18

---

Method family RAF
simscore simscore, softmax
a average vs main model baseline
-10
-12
0.0
0.2
0.4
0.6
0.8
1.0
weight (alpha)
Figure 7: Baseline results on ViDoRe 2 across different values of the weight a, averaged over model pairs. shows that the optimal weight for each method varies. Generally the softmax variant emerges as the most stable, while the min-max variant reaches the best performance if tuned correctly. The values for the zero-shot configuration for each bascline in Table were selected based on this plot.
Table 5: Percentage gain in NDCG@5 of hybrid retrieval over the primary retriever on ViDoRe 3. Each cell reports the average gain and standard deviation over 6 retriever pairs (2 multimodal base retrievers and 3 text retrievers). The Hyperparameters column indicates which hyperparameters were selected for zero-shot evaluation.
Method
Avg
Std
Hyperparameters
Average Ranking (Tuned)
++0.28%
1.0
0.9
RRF (Tuned)
1+0.13%
0.57
a = 0.5
Score Aggregation (Min-Max, Tuned)
f+2.66%
0.91
a 0.7
Score Aggregation (Softmax, Tuned)
1+1.53%
1.05
0.4
Guided Query Resinement (GQR)
+42.89%
0.67
Ir = 10 10-4 4 T = 50
the refinement model, nearly matches quality with 0.20 MB per document (+s 54x less). Using Jina, GQR surpasses Nemo while requiring 0.37 MB per document (*x 28x less).
G ANALYZING PER-QUERY DYNAMICS
As we explain in subsection 2.4 the dynamics in GQR differ from simple weighted average of scores - the distribution changes are constrained by the geometry of the primary model's embedding space, and different documents can shift by different magnitudes. To demonstrate this more directly, inluur we visualize the changes in document ranks for given query as a function of the number of GQR steps and function of the weight of the Score Aggregation (Min-Max) baseline.
19

---

Table 6: NDCG@5 for hybrid retrieval on ViDoRe 2. Results are averaged across 9 model pairs.
Method
Avg
Biomed Lectures
Economics
Human
ESG Full
Average Ranking
58.1
59.0
55.1
63.6
54.5
RRF
58.2
59.1
55.0
63.8
55.0
Score Aggregation (Min-Max)
60.2
61.3
55.8
66.7
56.8
Score Aggregation (Softmax)
60.9
62.6
55.3
68.4
57.2
Average Ranking Tuned
59.8
61.3
55.6
66.2
56.0
RRF Tuned
59.9
61.4
55.5
65.3
57.3
Score Aggregation (Min-Max)- Tuned
62.1
63.9
56.5
70.0
58.0
Score Aggregation (SoftMax)- Tuned
61.6
63.4
55,4
69.5
58.2
Guided Query Refinement (GQR)
62.3
64.2
56.0
70.2
59.0
Table NDCG@5 over ViDoRe 1, by primary and complementary models. Deltas are absolute changes the No refinement within the same base.
CUR condooenaaar
BANOA
TARQUAD
BAT
SydDags Famp
Spaloog Dur Sps Hoaks
Hem)) Linq-libeed
112
97.6
R24
73.8
65.3
32.8
75.2
945
42.5
756
80,0
58.
BL.3
Qvend
66.2
359
97.2
No Jalaccc
86.5
612
92A
9AT
F2.
BKT
906
553
95.1
98.1
Jan (esl)
8011
424
Jiw Holen
Lig Kiabed
50.5
Oveci
56.
$6.4
334
930
T6.4
11.
08.3
96.1
96.6
Jou Here)
910
868
998
368
555
982
Lin Timbed
110
100
968
034
104
91.
959
142
K29
201
M1T
567
901
500
993
Ovei
912
148
632
163
$40
910
145
62.8
268
7000
99.2
As can be seen in the figure, for some queries the dynamics of GQR and score aggregation are relatively similar, whereas for others the effects and dynamics starkly different. Notably, while score aggregation operates across all candidate documents, the effects of GQR are often focused on specific subset of documents.
20

---

DCR SANO
Wiight
OGR Step
Wwight
Drammed Park
0GR
wweght
Harka
Ramro
Ranto
Rarro
Figure 8: Query-level dynamics of GQR versus of how the document ranks change as a function of either the GQR optimization step & (center left panels), the weight of linear score aggregation baseline (Score Aggregation Min-MMx, center right panels). For both methods we see gradual progression from the ranks of the primary retriever mi (left) towards those of the complementary retriever m2 (right). However, the dynamics differ across queries, as well as across documents within a query. Examples are taken from the ESG subset of ViDoRe 2, for the model pair COL EMBED- -ULTTMOAAA--..BLLIMMM. EMBED-MISTRR (using a GQR learning rate of 10-1).
aggregation. The heat maps depict examples
21

---

Table 8: NDCG@5 for hybrid retrieval on ViDoRe 1. Results are averaged across 9 retriever pairs.
Method
arXiQAA
DocVQA
InfoVGA
TabeQuAD
IAIDOA
ShiftProj
SyathA
Synthbiergy
SynthGio
SyanHHealth
Average Ranking
55.6
44.0
76.2
93.4
78.0
53.8
44.1
76.5
93.5
93.3
Score Aggregation (Min-Max)
84.4
74.4
53.0
85.5
95.4
09.4
83.2
97.9
94.0
95.4
95.9
Scure Aggregation (Sefimax)
88.6
83.3
57.7
91.0
96.5
78.3
88.2
99.6
96.0
96.6
98.6
Average Kanking Tamed
85.5
79.5
57.3
87.0
95.5
74.4
36.0
97.8
88.5
92.3
96.9
RRF Tined
84.5
76.6
55.8
85.9
95.7
73.4
$4.4
97.6
88.0
92.3
95.2
Score Aggregation (Min-Max) Timed
88.5
88.0
62.2
92.6
96.1
79.6
88.4
97.7
88.9
93.9
98.0
Score Aggregation (SoftMax)
89.4
87.7
62.4
92.
96.5
80.2
86.9
99.1
95.3
96.0
97.6
Timed
Guided Query Refinement
90.1
88.1
62.5
93.0
96.6
80.2
90.0
99.6
96.1
96.5
98.8
(GQR)
Table 9: Percentage gain, in NDCG@5, of hybrid retrieval the primary retriever for ViDoRe 1. Each cell depicts average gain over 9 retriever pairs (3 multimodal base retrievers x 3 text retrievers).
Ang
xA
DaeVQA
IndoVQA
TaNQQuAD
TATDQA
Shillhuj
SystAA
SynthEnergy
SynthGrr
SynthHicals
Ascragg Banking
J-84.7%
1-31.15
[-18115
1-3.35
1-13.95
1-3.5%
[-5.5%
RRF
1-14.5%
3-39.1%
J-299%
[-12.95
1-3.25
1-23.15
[-12.25
1-3.5%
L-6.0%
1-5.258
1-5.79
Scom Aggresstion (Min Marc)
--703%
L-15.8%
J-15.5%
1-815%
[-1.25
[-13.9%
1-769%
[-t.7%
L-2.200
[-1.156
[-5.15
Score Appegation (ScCbmax)
2213
1-5.858
1.8.155
1.22%
0.0
12.3%
1/2075
0.0
L-0.13
110.2%
1-0.4%
Avergs Banking Tated
J4.55
L-10.15
1-8.9%
16.65
1-1.85
1-7.9%
144.5%
1-1.8%
L-8 198
[-4955
1-2.15
1-13.4%
1-7.%
1.09%
14.55
L.8.35
144,5%
(Min-Mac)-
1-0.45
L-7.8%
1-2.7%
Scoce Asgrezation (SelMau))
1.0.55
1.0.55
3.0.%%
4-0.1%
0.0
1.0.5%
L-0.55
4-0.45
Query Refnement (OQR)
0013
1-0.5%
1.0755
30.1%
140.0%
140.155
4-0.1%
0.0
140.1%
1-0.35
Table 10: Performance, latency (ms per query), and memory (MB per document) by primary and complementary models.
Primary Model
Complementary Model
Performance
Latency
Latency Diff
Memory (MB)
Colnomic-70 7b
No refinement Linq-Embor
60.25
115.98
0.20
Colnomic- 7b
62.75
181.21
65.23
0.20
Colnomic-76
Qwen3
60.98
196.16
80.15
0,20
Colnomic-7b
Jina (text)
63.05
350.13
194.15
0.39
Jina (vision)
No refinement
57.20
153.45
0,20
Jina (vision)
Line-Embed
61.18
213.97
60.5
0.20
Jina (vision)
Qwen3
59.75
233.06
79.61
0.20
Jina (vision)
(text)
60.68
394.64
241.19
0.39
Llama-Nemo
No refinement
62.98
2591. 14
11.07
Llama-Nemo
Linq-Embed
65.15
2674.84
83.7
11.08
Llama-Nema
Qwen3
63.30
2712.12
120.98
11.08
Llama-Nemo
Jina (text)
64.18
2934.61
343.47
11.27
22

---

0.65
Model (color)
Colnomic-78
(vision)
0.64
JJam-Nema
0.63
0.62
ndcg@5
0.61
Complementary Model (shape)
0.59
Qwen3
Ling-Embed
0.58
Jina (text)
No feedback
0.57
10
MB per Document
Figure 9: Storage-quality tradeoff. The axis is memory in MB, log scale, and the axis the average evaluation score (NDCG@S). Marker color encodes the primary retriever; marker shape encodes the GQR complementary retriever, with squares indicating the primary retriever alone (without applying GQR).
Performance and end-to-end latency of reranking pipelines against GQR. Rows grouped by reranker candidate size A dedicated block reports GQR.
Table 11:
Reranking Retriever
Latency
NDCG@5
Recall@5
Colnomic-7B (multi)
11.5.98
60.25
57.32
No-reranking
Jina (vision, multi)
153.45
57.20
56.17
Llama Nemo 3B (multi)
2591.14
62.97
59.75
Colnomic- 7B (multi)
181.21
62.75
58.0
Jina (vision, multi)
213.97
61.0
57.6
GQR
Llama Nemo 3B (multi)
2674.83
65.15
60.1
Colnomic-7B (multi)
1823.03
62.12
57.32
5 Jina (vision, multi)
1860.714
60.70
56.17
Ilama Nemo 3B (multi)
4332.55
64.72
59.75
Colnomic-7B 7B (multi)
3586.809
64.37
59.92
10 Jina (vision, multi) Ilama Nemo (multi)
3585.946
63.27
58.8
6027.078
10
61.8
Colnomie-7B (multi)
7035.953
65.07
60.27
Jina (vision, multi)
7251.81
64.10
59.6
Llama Nemo 3B (multi)
9470.134
65.77
61.02
23

---

Table 12: Recallã5 on ViDoRe 2, by primary and complementary models.
Avg
Biomed Lectures
Economics
ESG Human
ESG Full
Primary Model
Complementury Model
Jina-Embeddings (Text)
50.2
50.7
26.3
68.1
55.5
Ling-Embed
50.1
60.3
25.7
62.7
51.8
Owen3-Embeddingg
44.6
57.5
25.5
53.6
41.9
Colnomic-Ember
57.3
66.9
30.9
74.2
57.3
Jina- Ena--Eneedinngs (Text)
58.7
67.1
30.1
74.9
62.7
Linq-Embed
Embed
58.0
29.7
61.5
Owen3-Embeddiig
58.7
65.1
30.8
77.3
61.4
Jina- Jina-Embeuddings
56.2
64.2
29.6
71.8
59.1
Embeddined (Text)
57.5
27.7
75.9
62.1
Linq-Embed
Embed
57.6
66.9
29.4
72.3
61.7
Owen3--mbeedee
56.9
66.2
29.4
71.7
60.2
30.7
80.1
61.7
Jina-Embos Embeddddddd (Text)
60.2
65.9
30.5
80.1
64.1
Linq-Embed
Embed
60.1
68.4
28.4
79.5
64.1
Dwen3-Embbdding
59.9
67.1
30.4
79.3
62.7
Table 13: Recall@5 for hybrid retrieval on ViDoRe 2. Results averaged across 9 retriever pairs.
Method
Avg
Biomed Lectures
Economics
ESG Human
ESG Full
Average Ranking
54.3
61.4
29.1
RRF
54.6
61.5
29.1
69.7
58.0
Score Aggregation (Min-Max)
56.7
64.7
29.7
72.3
59.9
Score Aggregation (Softmax)
65.3
29.9
73.7
60.4
Average Ranking- Tuned
56.4
64.9
29.5
71.5
59.7
RRF Tuned
56.3
64.6
29.5
Score Aggregation (Min-Max) Tuned
58.5
66.9
30.3
75.5
61.4
Score Aggregation (SoftMax) Tuned
57.9
65.8
29.6
74.5
61.6
Guided Query Refinement (GQR)
58.6
66.6
29.6
76.0
62.3
Table 14: Recall@5 on ViDoRe 1, by primary and complementary models.
arKivQA DocVQA
TaMRQAAD TATDQA StinProj SymtAl SyathEnergy
SyathGov SyntHHeath
Model
Complementary Model
Jin-FEnboldings
43.9
$0.9
43.7
69.2
85.6
100.0
94.4
98.9
94.4
97.6
82.4
(Text)
Linq-Fmmbod
73,8
360
78.2
75.6
Embedding
Coinomic Embeed
93.1
66.2
95.6
100.0
100.0
100.0
Jina Embeddings (Tes)
90.0
93.4
67.2
95.5
95.6
100.0
97.8
1000
100.0
98.
89.4
Linq Emnbed
93.1
93.7
66.5
95.3
95.6
100.0
97.8
100.0
1000
98.3
89.7
Quent-Embeekinng
66.3
95.6
Jis--Enbolddnnss
Jin-EEneedding-- (Tesi)
90.9
94.0
71.0
95.4
97.8
100.0
97.8
100.0
100.0
98.8
88.5
Linq-Embed Embed
92.0
94.0
69.6
95.5
97.8
100.0
97.8
100.0
100.0
98.8
88.7
Owend Embedding
92.2
93.9
68.7
95.4
97.8
100.0
97.8
100.0
100.0
98.9
88.5
Liama Nemotetriver
73.0
98.9
Jina Embeidings
92.4
94.6
72.7
97.8
100.0
100.0
100.0
99.6
Linq-Eubedd
92.4
94.8
72.4
97.6
98.9
100.0
97.8
100.0
100.0
99.6
88.9
Owent Embedding
91.6
94.5
21.5
98.2
98.9
100.0
96.7
100.0
100.0
98.8
88.8
24

---

Table 15: R5call@5 for hybrid retrieval on ViDoRe 1, Results are averaged across 9 retriever pairs.
Method
Avg
arXivQA
InfoVQA
TabEQuAD
TATDQA
ShiftProj
SyntAA
SynthEnengy
SynthGov
Symthleaatth
Average Ranking
72.4
Scure Aggregation (Min Max)
88.4
63.5
93.1
98.5
83.0
100.0
Scure Aggregation (Solmmax)
92.5
87.4
63.5
94.5
99.1
87.0
96,1
100.0
97.3
1000
100.0
Average Ranking Timeed
92.9
92.2
68.7
95.6
98.8
87.6
95.2
93.3
98.2
100.0
RRF R Tourd
91.5
90.9
66.2
93.5
98.8
85.0
92.4
99.5
92.8
98.2
97.2
Scure Aggregation (Min-Max) Tuned
93.5
92.3
69.3
96.3
99.0
88.7
96.6
99.5
93.8
99.1
100.0
Score Aggregation (SottMax)
83.5
91.4
69.4
95.8
99.0
88.9
94.7
99.9
97.1
99.6
98.8
Tuned
Gunded Query Refinemnn
92.0
69.5
96.3
99.0
89.0
97.3
100.0
97.6
1000
1000
Table 16: Effect of extra index search on GQR NDCG@5 performance, over ViDoRe 2.
Avg
Blomed Lectures
Economics
ESG Human
ESG Fall
Model
Complementary Model
Variant
Colnomic- Emmee
Jina Eimhedldings
GQR
63.0
64.7
57.0
70.3
60.2
GQR Search
63.1
64.7
57.1
70.3
60.2
Linq-Eimeed
GQR
61.2
GQR Search
62.7
63.4
56.5
67.9
61.0
Owens-Embeekiing
GQR
61.0
61.9
54.3
70.2
57.5
GQR Search
61.0
61.7
54.3
70.6
57.5
Eina-Embeddinng
Eeae-mmeediings
GQR
60.7
61.7
55.3
66.9
58.8
GQR Search
60.7
61.7
55.2
66.9
58.9
LinqeEEbee
GQR
GQR Search
61.0
64.7
57.2
65.0
57.2
Qwen Beenn--Emeeklingg
GQR
59.8
63.2
51.6
67.8
54.4
GQR Search
59.8
63.2
$3.6
67.8
54.4
Llama- Nemeretriever
Jina-Eibeet Embeeddings
GQR
64.2
64.5
57.6
74.2
60.4
GQR
Ling-Fimeed
GQR
65.1
66.4
56.8
74.6
62.8
GQR Search
65.3
66.5
57.2
74.6
62.8
Qwens--mbeckiing
GQR
63.3
63.0
33.4
74.1
38.7
GQR Search
63.3
64.8
55.4
74.1
58.7
Table 17: Effect of candidate pool GQR NDCG@5 performance, ViDoRe 2.
Avg
Blomed Lectures
Economics
Human
ESG
Model
Complementary Model
Variant
Colnomic Embed
Jina Embeddings
GQR
63.0
64.7
$7.0
70.3
60.2
GQR (Top-K only)
62.9
64.4
56.3
71.2
39.6
Ling-Embet
GQR
62.8
65.4
56.7
67.7
61.2
(Top-K only)
62.4
60.1
GQR
61.0
61.9
54.3
702
57.5
GQR (Top-K only)
61.5
63.4
54.3
71.6
56.7
Embeddings
fin-Emmeeddings
GQR
GQR (Top-K K only)
$9.7
61.7
56.8
Ling Embed
GQR
61.2
64.7
57.2
65.7
57.1
GQR (Top- only)
61.0
64.7
56.9
66.2
56.2
Owen3-Embeldinng
GQR (Top-K only)
59.0
63.5
53.3
65.0
$4.0
Jina- Enneednns
GQR
64.2
64.5
57.6
74.2
60.4
GQR (Top-K only)
74.1
60.3
Lino--Embed
62.8
GQR (Top- K only)
64.7
65.4
$7.7
74.8
60.8
Qwen3- Eamedd--meeinngg
GQR
63.3
65.0
55.4
74.1
58.7
GOR (Top-K only)
25

---

Table 18: Effect of loss function on GQR NDCG@5 performance, over ViDoRe 2.
Avg
Biomed Lectures
Economics
ESG Human
ESG Full
Model
Complementary Model
Loss Variant
Cohnomic-rithed
Ingg-onnheddings
Jensen-Shumrinn Shumnon
62.7
$6.6
Kullbuck-Leibler (Consensus)
63.0
64.7
57.0
70.3
60.2
Kullbbckk Leibler (Target)
Liny- Embed
Jomen- Shumoo
63.3
65.3
54.9
71.3
61.9
Leibler (Consensus)
Kullback-Leibler (Target)
63.8
64.9
57.3
71.3
61.7
Quent-Finkttiing
Jensen- Shunnon
61.4
Kullbuck Kullbuck-Leblerr (Consemuus)
61.0
61.9
54.3
70.2
$7.5
Kullbock-1 eibler (Target)
61.3
54.1
Fine- Enn-Embookings
Jine Emiocddings
Jensn- Shumoon
60.3
61.7
56.1
67.1
$6.5
Kullback Leibler (Consensus)
60.7
61.7
55.3
66.9
58.8
Kullbackk Laahback-Leilerrree (Target)
60.9
55.3
Linq-
Consen-Shunooe
62.5
63.7
58.7
69.8
57.8
KullBack LLibler (Consemsus)
Kullback-1 (Target)
61.3
64.7
55.5
68.9
57.1
Qubell-mn
Shumnon
59.0
Kullbuck Leibler (Consensus)
99.8
63.2
53.6
67.8
$1.4
Kullback- Leibter (Target)
Laama-Nemoretn Nemorettieeee
Iine Eminedding
lensen-Shanion Shanoon
64.0
64.1
57.3
74,3
60.4
(Consensus)
57.6
Kullback-Leibler (Target)
64.2
64.6
52.2
74.3
60.6
Ling-Embel
Jensen Shumion
65.0
56.3
Kullbuck Leibler (Consensus)
65.1
66.4
56.8
74,6
62.8
Kullbock Leibler (Target)
64.7
66.2
56.7
24.3
61.5
Dweed-Embedding
Jensen- Shanoon
63.4
03.1
58.6
Kullback: Leibler (Consensus)
63.3
55.4
74.1
58.7
Kullback-Lehhie (Target)
63.3
55.4
Table 19: Swapping primary and complementary roles GQR across model pairs. The first two columns specify the role of each encoder. For each setting report the absolute score ViDoRe 2 and the absolute gain relative to the primary encoder alone.
Primary model
Complementary model
NDCG@5
Gain
Colnomic-7B
Jina (text)
63.05
2.8
Jina (text)
Colnomic-7B
62.22
8.82
Colnomic-7B
Linq-Embed Colnomic-7B
62.75
2.5
Line-Embed
61.3
Colnomic-7B
Qwen
60.97
0.7
Qwen3
Colnomic-7B
54.4
7.6
Jina (vision)
Jina (text)
60.67
3.5
Jina (text)
Jina (vision)
59.25
5.85
Jina (vision)
Linq-Embed
61.17
4.0
Linq-Embed
Jina (vision)
61.37
6.07
Jina (vision)
Qwen3
61.17
2.6
Qwen3
Jina (vision)
52.05
5.25
Llama-Nemo
Jina (text)
64.17
1.2
Jina (text)
Llama-Nemo
59.3
5.9
Llama-Nemo
Linq-Embed
65.15
2.2
Linq-Embed
Llama-Nemo
60.27
4.97
Llama-Nemo
Qwen3
63.3
0.3
Qwen3
Llama-Nemo
52.8
26

---

Table 20: Tuned values for the GQR step size parameter a (learning rate) over ViDoRe 2.
Selected a value
Primary model
Complementary model
Biomedical
Economics
ESG Human
Colnomie-mmbed- Embed Multimodal-77
Jina Eada--dddddg-- V4 Text
x 10
x 10
1x 10
5 x 10
Ban--Emeed-ao Mistral
x 10
5 10
1x 10-3
5 10-2
Qwen Wven2-Embedding-BB
1 10-3
1 10-5
1 10-3
1 10
Jina- Jina-Embeediing-VA
Jina- Mna-Embeeddings-V V4. Text
1x 10
1x 10
1x 10
5 10
Ling-Embed- Embed--isttal
5x 10
1 10
1x 10
1 10
Qwen Fmeen-nnnndd
5
1
5
1
Jina- Min.-mmbeddinn----- Text
5x 10
1
1x
10
Enbbeeeed-Mistall
1 10-4
5 10-5
1x 1x-1
10-3
Qwen3 Emhedding-444
10-4
10-5
1x
1
Table 21: Tuned values for the number of GQR optimization steps parameter T' over ViDoRe 2.
Selected 7 value Economics
Primary model
Complementary model
Biomedical
ESG
ESG Human
Colnomic-Embed Mumibeedd-Mttindaa--
Jina- man--Embedding--V V4 Text
25
25
25
Ling-Embed-Ne Misttrall
25
10
Owen3-Fmbeddan 4B
10
50
Jina- Fima-eddddiin--V V4
Jina- Embeeddnndd V4 Text
10
50
25
25
Ling-Embee--e Mistral
Eisted-Mistaal
25
25
50
25
Qwen3- Fmbeedding-41
10
50
50
10
Jina- Ton--Embedding--Ve V4 Text
25
50
50
10
Linq- Embra-Mistral
10
10
10
50
10
Table Tuned values for the weight parameter a of the Average Ranking baseline over ViDoRe
2.
Selected a value
Primary model
Complementary model
Biomedical
Economics
ESG
ESG Human
Minomii-Embed-Muttindddee
Min-EEmbedding--V V4 14-10 Text
0.7
0.6
0.8
Ling-Ember Mistral
Embed-Mistral
0.5
0.7
0.8
-BB
0.9
0.9
Jina- Tha--Embeddings-V V4
Jina-Embeda Ena--Embeddings- V4 Text
0.9
0.5
0.5
0.8
Ling-Ember Mistral
0.7
0.3
0.9
0.2
Qwen3 Baeenn---meeddin-----
0,9
0.9
0.8
0.5
Jina- Ton--Emeedding--Ve V4 Text
0.9
0.9
0.9
Ling Hing-Embed-Minn Mistral
0.5
0.9
0.9
Qwen3- Febedddinbeddnn-41
0.9
0.8
0.9
Table 23: Tuned values for the weight parameter a the RRF baseline over ViDoRe
Selected value Economics
Primary model
Complementary model
Biomedical
ESG
ESG Human
Sehomii-Embee--Mltiinee
You--Embedding--Ve
V4 V4 Text
0.9
0.7
0.6
0.9
Ling-Embed-- Misttral
0.5
0.5
0.9
0.7
Qwen3-Em Boend--Emeedding--
0.9
0.9
0.9
0.6
Jina- Saaa-Emboddings--4
Ein-Emmbedding- V4 Text
0.5
0.5
0.8
Linq-Embed- Mistral
0.3
0.8
0.6
Qwen3- Eanadd--mbddi-gg---ee 4B
0.9
0.8
0.8
Lina-Embeddings V4 Text
0.8
0.9
0.9
0.9
Lined- Mistral
0.9
0.4
0.9
0.9
Qwen3-Fnne Been--Embedding-18
0,9
0.9
0.9
0.9
27

---

Table 24: Tuned values for the weight parameter of the Score Aggregation (Min-Max) baseline over ViDoRe 2.
Selected value
Primary model
Complementary model
Biomedical
Economics
ESG
ESG Human
Colommi-Embed-
Man--Embeddings- V4 Text
0.8
0.4
0.7
0.8
Ling-Embed- Mistral
0.6
0.5
0.6
0.8
Swen3-Embeddiing
0,7
0.8
0.9
0.7
Jina-Embedding--V
Ein--Embeddings- V4 Text
0.7
0.6
0.8
Linq-Ember Misttall
0.5
0.8
Swean--Emeedding-t-
0.9
0.8
0.6
Jina- Embaedddddd V4 Text
0.8
0.7
0.8
0.9
Ling-Embed--e Embed-mimtttt Mistral
0.7
0.8
0.7
0.9
Qwen3- Fmbedding-44
0.8
0.9
0.9
0.9
Table 25: Tuned values for the weight parameter of the Score Aggregation (Softmax) baseline over ViDoRe 2.
Selected value
Primary model
Complementary model
Biomedical
Economics
ESG
ESG Human
Colomii-Embedd-Mliinne
Jina-Embeddings- V4 V4 Text
0.6
0.1
0.5
0.7
Embed-Mistral Mistral Ling Ling-Embed--
0.1
0.2
0.1
0.1
Doen3--Ebeedding41
0.1
0.2
0.4
0.1
Jina- Mma-EEmbodding--4
Souu-Embeddings-Ve V4 Text
0.5
0.6
0.6
Linq--mbed- Mistral
Embee-Misttal
0.1
0.5
0.9
0.1
Jina-Emben Eanaa-ddeddiig---t V4 Text
0.8
0.8
0.9
0.9
Ling Bined- Mistral
EimbedMMittal
0.1
0.1
0.1
0.9
Qwen3 Boend--mbedding-11
0.2
0.6
0.8
0.9
28


Reviews:
[{"review_id": "OokZoLwacd", "reviewer": "Reviewer_StJj", "summary": "This paper proposes Guided Query Refinement (GQR), a novel test-time hybrid retrieval method. Specifically, GQR iteratively optimizes the representation of a primary retriever using a complementary retriever as a guidance signal. The optimization is performed via gradient descent to minimize the KL divergence between the primary retriever’s distribution and a consensus distribution (initialized as the average of the primary and complementary retrievers). Experimental results show that the proposed method achieves performance comparable to SOTA models while being significantly more efficient.", "questions": "1. Could the authors clarify each method listed in Table 2?\n2. How does the step T influence the performance of the proposed method?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed GQR method is an effective approach to achieving an impressive performance–efficiency trade-off.\n2. The experiments effectively validate the proposed method’s advantages compared to both base retrievers and hybrid baselines.\n3. The finding that a weaker retriever can still provide a complementary signal to improve a stronger retriever is interesting.", "weaknesses": "1. The authors should clarify the “Score Aggregation – Tuned” results in Table 2. From my current understanding, the Score Aggregation method simply merges the scores from different retrievers. If this is correct, the Score Aggregation (Min-Max) – Tuned method (with just an arithmetic weighting) only drops 0.5% in performance but is much more efficient than GQR, which requires gradient optimization.\n2. The hyperparameters are selected based on the task, raising concerns about generalization.\n3. There is no ablation study on the impact of the step T on performance.\n4. There is a lack of exploration regarding how to choose the consensus distribution and how to define the distance between the consensus and the primary retriever distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZQvTmumJV", "reviewer": "Reviewer_LwFe", "summary": "This paper focuses on the problem of efficient visual document retrieval and proposes Guided Query Refinement (GQR), a test-time optimization framework that refines a primary retriever’s query embedding using guidance from a complementary retriever. By integrating rich cross-model interactions often missed by standard hybrid methods, GQR enables ColPali-based multimodal retrievers to achieve state-of-the-art performance. Extensive experiments demonstrate that GQR effectively advances the Pareto frontier of performance and efficiency in multimodal document retrieval.", "questions": "1.\tThe paper does not include experiments on the number of iterations T. How does this hyperparameter affect performance?\n2.\tIn the introduction, the authors claim that the hybrid retrieval approach aims to leverage the advantages of dense text retrievers in terms of lower time and storage costs. However, the main model proposed in this paper is a multimodal, vision-centric encoder, and the dense text retriever merely provides additional supervision. So where exactly does the efficiency advantage of the hybrid retrieval paradigm manifest?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper innovatively adopts a vision-centric aggregation paradigm, supplemented by a text-centric approach, to address the issues of excessive modality gaps and high computational and storage costs in visual document retrieval tasks.\n2.\tThe arguments in this paper are well-substantiated, and the experimental section is thorough and comprehensive.\n3.\tThe description of the GQR algorithm in this paper is clear and easy to understand; even though there are many formulas, they are still easy to follow.", "weaknesses": "1.\tThe figures in this paper are somewhat rough, and Figure 4, which illustrates the core GQR architecture, should not have been placed in the appendix.\n2.\tThe introduction does not provide sufficient background information and lacks an explanation of the necessity of applying hybrid retrieval methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JAqAuUs582", "reviewer": "Reviewer_NbzC", "summary": "This paper introduces Guided Query Refinement (GQR), a test-time optimization technique designed to improve visual document retrieval, by iteratively refining the query embedding of a primary retriever using guidance from a complementary retriever. Instead of aggregating outputs at the ranking or score level, GQR operates at the representation level. The method is evaluated on ViDoRe 1 and ViDoRe 2.", "questions": "The authors need to address the above weaknesses. In addition, I have a few further questions and suggestions.\n\n1. If I want to use multiple complementary retrievers in parallel, how should the algorithm be modified specifically? Have the authors explored this direction? I am also curious whether the differences in query representations extracted by multiple complementary retrievers would have an impact on the final results.\n\n2. Figure 1 does not provide sufficient information. I suggest that the authors integrate Figure 4 into the main text.\n\n3. The visuals in Figure 2 and Figure 3 appear rather sparse and monotonous. It would be better if these figures were made more visually appealing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The core idea and methodology of the paper are concise, and the inclusion of algorithm pseudocode makes it easy to understand and follow.\n2. The authors evaluated nine retrieval pairs on standard retrieval visual document benchmarks, demonstrating the efficiency of the proposed method in most cases.\n3. The code has been made publicly available, but there appear to be some minor shortcomings.", "weaknesses": "1. The paper is titled Multimodal Hybrid Retrieval; however, the authors focus solely on a single visual document retrieval task. In fact, multimodal hybrid retrieval covers a much broader scope — it may also include tasks such as video-text retrieval and composed image retrieval. Therefore, the paper does not fully demonstrate that the proposed method is applicable to the broader multimodal hybrid retrieval setting.\n\n2. The algorithm relies on tuning hyperparameters such as $\\alpha$ and $T$, yet the authors do not present any analysis of the model’s sensitivity to these hyperparameters, and the paper lacks corresponding ablation studies.\n\n3. The proposed approach is neither plug-and-play nor zero-shot. As I understand it, it requires manually collecting a sub-dataset before fine-tuning can begin. Such a high “startup cost” could hinder its scalability and practical adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4a31149Sia", "reviewer": "Reviewer_EqPn", "summary": "Guided Query Refinement is a test-time retrieval ensembling method that iteratively updates the primary retriever’s query representation using gradient descent to minimize the KL divergence of the (query, top k document) probability distribution to the secondary retriever's distribution. On the ViDoRe 2 benchmark,  it outperforms other ensembling methods, as well as the primary retriever when used on its own.", "questions": "1.  (See *Weakness Objective function simplification*) Is there any reason to justify the additional complexity of the \"averaging step\" in the objective function that seems redundant ? \n\n2. You state that hyperparameters are chosen on a dev split of 10% of the data. To my knowledge, no dev split exists for Vidore V2. Do you thus split the test split and report results on 90% of the remaining docs ? If this is the case, is there any overlap between docs in the \"test\" and your current \"dev\" split here ? \n\n3. How do you explain GQR does not improve results on ViDoRe 1 ?\n\n4. Under the \"nudging towards p2\" hypothesis, GQR would be ~roughly~ equivalent to a \"tuned\" normalized score averaging technique in which the tuning is done through gradient descent. In your results, the best baselines in practice are also \"tuned\" normalized score averaging, but the alpha parameter that can be optimized is discrete (steps of 0.1). Could the performance gap be closed if alpha becomes a learnable parameter here (or at least is optimized at a more granular level) ? Could you at least report alpha per subset ?\n\n**Suggestions**:\nSeems figures are big enough to integrate text from the legends directly in the plot. This may make it a bit easier to understand the plots in one glance. This is very minor of course.\nAdd x-axis label in figure 3.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper provides an interesting and novel framework to refine query representations at test time through gradient descent on the representation itself, in order to leverage information from a complementary retriever. The method used enables optimizing multi and single vector models, and yields strong results on Vidore V2.\n\nThe paper is mostly well written and relatively easy to understand.", "weaknesses": "**Objective function simplification**: The objective function that is minimized here admits a \"trivial\" minimum when p1(t) = p2. While this may be hard to reach in practice, the role of the \"extra-step\" of averaging p1(t) and p2 is unclear to me: since p_avg(t) depends on p1(t), it just seems that by minimizing KL(p_avg(t) || p1(t)), we are in practice (mostly) minimimizing KL(p2||p1(t)), and thus nudging p1(t) to match the p2 distribution. The ablations that are run l421 mostly confirm this is the case.  Under this light, it is not clear what the additional complexity of the \"averaging step\" brings.\n\n**Objective function rationale**: In all cases, it seems that the effect of GQR is to modify the query representation(s) to more closely match the probability distribution yielded by a weaker secondary retriever. I personally find hard to gain intuitions on why this would help in practice - and how this differs from some sort of tuned probability distribution averaging method.\n\nI would find interesting to understand how scores evolve through training steps: intuitively, at t=0, we match the primary model nDCG, then (according to your results), the ranking is slightly improved, and I would imagine that eventually, as more steps are made and p1(t) too closely matches p2, performance drops and starts matching those of the secondary models ? Understanding these dynamics could be key here in explaining why this method works. It would also be interesting to have some plots of the score distributions before/after GQR to better understand what is going on. Finally, a measure of how much p1(0) and p2 impact the final distribution (ratio of KL divs for instance) could be given over the different data subsets on which hyperparam optimization is run to gain a sense of how data dependant GQR is. \n\n**Hyperparameter sensitivity**:  From my understanding, the dynamic described above would indicate that there is a bounded ranges of optimization step that can be made before collapse, and thus that GQR is sensitive to hyperparameter choice. It also seems you independently optimize parameters for each data subset in Vidore 2. Would it be possible to show results where hyperparameters are not optimized on the same data distribution (ie. optimize on Vidore train set, test on the full Vidore v2 with the same parameters across each split ?). \n\n*I believe it is key to add the final hyperparameter values (for the method as well as the baselines) used in the appendix. Not having them makes assessing the work much more difficult.*\n\n**Missing files**: The anonymous4science link works (and is appreciated) but most files are missing. I can only see the Readme and the latency benchmarking codefiles, not the rest. I also don't see the chosen hyperparams.\n\n**Clarity**: In this paper, the query embeddings that are optimized are multi-vector embeddings (in most cases). While the query refinement technique applies directly here, this could be more clearly explicited - I feel this \"form agnosticity\" is one of the strenghts of this method.\n\n**Overall**: I am aware I suggest many things here, the overall point is that the rationale behind \"why\" GQR works remains unclear, and understanding the method's sensitivity to hyperparameters would be key to inform practical uses of this method beyond benchmark maximisation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)