Visual Abstraction: A Plug-and-Play Approach for Text-Visual Retrieval
Authors: Guofeng Ding
Yiding Lu
Peng Hu
Mouxing Yang
Yijie Lin
Xi Peng
Venue: ICML 2025 poster (2025)
Keywords: text-to-image retrieval
text-to-video retrieval
plug-and-play
Primary area: applications->computer_vision

Abstract: Text-to-visual retrieval often struggles with semantic redundancy and granularity mismatches between textual queries and visual content. Unlike existing methods that address these challenges during training, we propose VISual Abstraction (VISA), a test-time approach that enhances retrieval by transforming visual content into textual descriptions using off-the-shelf large models. The generated text descriptions, with their dense semantics, naturally filter out low-level redundant visual information. To further address granularity issues, VISA incorporates a question-answering process, enhancing the text description with the specific granularity information requested by the user. Extensive experiments demonstrate that VISA brings substantial improvements in text-to-image and text-to-video retrieval for both short- and long-context queries, offering a plug-and-play enhancement to existing retrieval systems.

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Guofeng Ding'
Yiding Lu
Peng Hu
Mouxing Yang'
Yijie Lin'! Xi Peng'
com
ICML-VISA
Abstract
cess, we observe that traditional VLMs struggle to satisfy the precise and diverse retrieval demands users, due to limitations in both training strategies and data. Specifically,
Text-to-visual retrieval often struggles with se- mantic redundancy and granularity mismatches between textual queries and visual content. Un- like existing methods that address these chal- lenges during training, we propose VISual Ab- straction (VISA), a test-time approach that en- hances retrieval by transforming visual content into textual descriptions using off-the-shelf large models. The generated text descriptions, with their dense semantics. naturally filter out low- level redundant visual information. To further address granularity issues, VISA incorporates a question-answering process, enhancing the text description with the specific granularity infor- mation requested by the user. Extensive exper- iments demonstrate that VISA brings substantial improvements in text-to-image and text-to- ro-video retrieval for both short- and long-context queries, offering plug-and-pe and-play enhancement existing retrieval systems.
i) Training Strategy Aspect. Retrieval tasks require the model to focus on core semantic elements of an image based the query while disregarding superfluous details. How- ever, VLMs like CLIP typically contrast text and images at a global level to emphasize overall relationships. This global strategy inevitably incorporates low-level visual el- ements such as snow textures or tree branches, which are irrelevant to high-level semanties. These redundant features can hinder the model's ability to understand key concepts, causing it to incorrectly attend to unimportant patches, such as the tire tracks (see red rectangle in Query I of Fig. (())).
ii) Training Data Aspect. As the saying goes, "an image is worth a thousand words". While images inherently exhibit unlimited granularity (Tang et al., 2023), the web-crawled image descriptions are often brief and coarse (Xiao et al., 2024), failing capture the multi-granular concepts in im- ages. However, retrieval tasks require models to dynami- cally adjust their focus based on the granularity of the user query. This mismatch prevents the model from effectively matching user queries of varying granularities with the cor- responding image content. As shown in Queries 2 and with different granularities in Fig. (a)), VLM predominantly focuses on the main action, skiin, but fails to capture fine- grained details such the hat and jacket.
1. Introduction
The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise. Edsger Wybe Dijkstra
To address these challenges, early works (Li et al., 2020; Chen et al., 2020; Lee et al., 2018) utilize object detection techniques (Ren et al., 2016) to extract key visual concepts, filtering out low-level details to cover the essential seman- ties of an image. However, these methods are inherently limited by a finite number of objects (Li et al., 2022a) and cannot explicitly represent relationships, actions, scenes. Recent works (Zheng et al., 2024) attempt to address the varying granularity demands by leveraging large multimodal models (Liu et al., 2024a; Hurst et al., 2024) to generate fine- grained captions, enriching text data with multi- calti-granullrrty information (Cao et al., 2024).
Text-to-visual retrieval (TVR) attempts to identify relevant images videos based on textual query. In this field, lasion-language pre-training models (VLMs) (Radford et al., 2021; Jia et al., 2021) have made significant progress by employing contrastive learning on large-scale paired data to construct a cross- cross- joint space. Despite their suc-
School of Computer Science, Sichuan University, Chengdu, China National Key Laboratory of Fundamental Algorithms and Models for Engineering Numerical Simulation, Sichuan University, China. "Correspondence to: Yijie Lin <linyijic. gm @ gmail.com> Xi Peng <pengx. gm @ gomi..com>.
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
However, the above methods require training VLMs from scratch, which is complex and non- con-trivial endeavor. Un-

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
image avnny some,
popple
farcameand
Image
wisse cletting, including tur, their hitden from view
drill
bandled
Baseline
VISA
secord child, seessed in biue stowoutt and yellow skis looking sealty, walking wits sis poies The beckemmnnd shoon a man weatnng cann jaccet and vinse las, cleasig the smow off parked cur, These are also nxn-etroreel cas lind up along tuch dries the smoet Tall hare tow with veew thiir bearches liu sacd. Thenky everest with ans cloak, and geseral stmooptace feoh quict chamsonett typical
Language
COCO
Description
ShareGPTTy
Flickr20K
Query
1. The cars are covered snow.
2. Two people are skilng street.
The wortnn skzing is wearing a hat and green jacket.
55.6
86
Urhankk
$1.1
MSR-VTT
-4.
Query-Image Attention
52.0
54.0
98.2
I1W
62.5
LSMDC
background thows man wearing dark jacket and a
This image snowy
the fonegroundl wommm dressed bright ski pants
scene with several people and cass covered snow. In the dressed bright ski pants a jacket red hat skiing.. A second
61.2
Query-Text Attention
white hul, cleaming
a jocket and red hat is skilnng comyying a child back
off parked car There are
foreground
snow covered cars linel up
DC1
60.2
DiDeMo
along both of the street
child, dressed blue snowsut
and yellow akis skiing scarby
MSVD
(a) Our Key Observation
(b) Performance Comparison
Figure 1. (a) Our key observation. We visualize the attention maps of vision-language pre-training model LoTLIP (Wu et al., 2024) and text retriever gemma2 (Chen al., 2024a) given diverse text queries. For clarity, only a subset the query-text attention is shown here; full attention details are available in Fig. 6. (b) Performance comparison (R@) @ 1) across ten image and video datasets.
like previous works, we explore a test-time solution that addresses these two challenges and enhances the retrieval capabilities of VLMs. The core idea is to transform the retrieval process from a modal space to text-only space, inspired by our observation in the last row of Fig. I(a). Specifically, we observe that text descriptions of images, with dense semanties, naturally omit the low-level redundant information that often present in visual signals and enable sharper focus the core concepts (Query 1). Moreover, when textual descriptions encompass comprehensive con- ceptual details, they can seamlessly match user queries at varying granularities (Queries 2 and 3), effectively satisfy- ing diverse retrieval demands.
ing the visual concepts into natural language through a question-answering process, possibly providing new perspective towards advancing visual retrieval during test- time.
By employing off-the-shelf large models, our method works in a plug-and-play manner, achieving significant improvements for both short- and long-context queries.
2. Related Work
2.1. Vision-language Pre-training Model in Tixt-visual Retrieval
Inspired by this observation, we propose VISual Abstrac- tion (VISA), a plug-and-play approach that abstracts sparse visual concepts in images into dense natural language hance text-to-visual retrieval. Specifically, VISA leverages sff-th large multimodal model (LMM) and text re- trievers to transform visual signals into textual descriptions and perform similarity measurements within the textual space. By abstracting visual content into textual represen- tations. VISA filters out irrelevant low-level details and focuses the core concepts. To match the required gran- ularity, VISA employs a question-am -anwwerinn strategy that decomposes the user query into targeted questions and uses off-the-shelf' LMMs to answer these for each candidate. The resulting answers better align the visual abstraction with the user query. The contribution of this paper is summarized as,
Vision-language pre- pre-training models (Radford et al., 2021; Lin et al., 2024; Huang et al., 2024) have emerged as the cor- nerstone for text-visual retrieval in recent years. Based on the differences in architectural design, existing approaches can be roughly categorized into three groups: i) Single- stream architectures like UNITER (Chen et al., 2020) and OSCAR (Li et al., 2020) use object detectors for seman- tic feature extraction and fine-grained fusion of visual and textual data. However, the high computational overhead and limited concepts of object detector hinder their scala- bility to large-scale datasets. ii) Dual-stream architectures, such as CLIP (Radford et al., 2021) and Frozen (Bain et al., 2021), employ separated encoders for each modality, achiev- ing high-level semantic alignment but struggling with fine- grained cross-modal interactions. iii) Hybrid architectures, such as ALBEF (Li et al., 2021) and Intern Video (Wang et al., 2022; 2024b), take the best of both worlds to balance
We propose to enhance text-to- -osuual retrieval by abstract-
2

---

Visual Abstraction: A Plug-and-Plaa Approach for Text- Visual Retrieval
VISA Text Query
TTxt Query: Several men waarring athletic wear are planing soccer on the playground) One of candidutes
Iidge Gallery
Target
General Description: The image captures soccer game between two teums. Severul men are actively engaged in the gume, with one player in black jersey intently watching the ball. Two other players, wearing gra jerseys, positioned nearby, appeuring ready to intercepe or respond ball. All players are dressed in typical seceet attire. including shorts, long sooks, and their body language reflects focus and anticipation. The atmosphere suggestsi a competitvve yet casual match. In the background, trees with auturnn folinge in shades of orange, and green; provide a colorful scasonal setting. while now of parked cars in vartous colors lines the odge of the field....
Text Retriever
Text Query
Key Phruses: 1. Seeerall men athletic wear 3. playing soccer playgground... Questions: 1. What are the men wearing while playing soccer Where is soccer game taking place?...
LLMq
Top Candidate
General Description
Key Phrases
LMMcD
Answers: 1. The men playing soccer are weuring jerseys, athletic shorts, long socks with some in gray jerseys, 2. The soceer gume taking place on grussy playground surrounded by trees with autumn folinge, parked cars.
Questions
LMMA
Answers
Figure 2. The pipeline of our method VISA. (Left) VISA operates as plug-and-play enhancement for existing VLMs, transforming visual content into text descriptions improve text-to-visual retrieval within the text space. The visual abstraction consists of general description and the answers from QA-hased description refinement. (Right) An illustrated example where query-related information are highlighted red. Additional examples are provided in Appendix F.
fine-grained alignment with computational efficiency.
stantial computational resources to retrain or fine-tune mod- els. contrast, this work introduces a test-time computing approach to enhance the retrieval performance of existing VLMs. Specifically, our method leverages off-the- LMMs and text retriever to transform visual content into abstract textual descriptions, providing a plug-and- and-play solu- tion that improves retrieval performance without additional training. Furthermore, by utilizing LLM- based text retriever, our approach benefits from their advanced contextual under- standing to handle long-context queries.
Although existing VLMs achieve significant processes in TVR, they often fail to capture multi-granularity seman- ties inherent in the visual modality, resulting in suboptimal retrieval performance. To address these limitations, we ab- stract visual content into textual descriptions to provide comprehensive semantic coverage across different levels of granularity. Notably, the proposed VISA framework is orthogonal to existing VL.M-based methods and serves as alug-and-play module, boosting retrieval performance across various architectures demonstrated in experi- ments.
3. Method
In this section, we introduce VISA, a plug-and-play ap- proach designed to enhance text-to-visual retrieval during test-time. First, we explain how VISA integrates seamlessly into existing VLMs for text-to-visual retrieval in Sec. 3.1. Then, detail the process of abstracting visual content into dense natural language in Sec. 3.2.
2.2. Large Models in Text-to-visual Retrieval
In very recent, large language models (LLMs) and LMMs have opened new avenues for cross modal retrieval due to their strong understanding capabilities. These approaches can be categorized into three groups: i) Augmentation with LMM-generated captions (Zheng et al., 2024; Xiao et al., 2024; Shi et al., 2024; Lu et al., 2025), which enhance exist- ing VLMs by leveraging enriched datasets containing fine- grained captions generated by LMMs. ii) Augmentation with frozen LLMs (Cao et al., 2024), which extends the in- put length and contextual understanding of L.MSs to VL.Ms, improving retrieval performance for complex queries or long text inputs. iii) Augmentation by fine-tuning LMMs (Chen et al., 2024c; Liu et al., 2024c), which transforms LMMs into sophisticated langua-- re-rankers, leveraging their advanced cross-modal reasoning capabilities refine retrieval results.
3.1. Enhancing Text-to-visual Retrieval with VISA
Text-to-visual retrieval aims to identify the most relevant im- age video based on textual user query. Formally, given user query q and a visual gallery G = (71, I2...., J,), where represents an image or video, VLMs typically en- code both modalities into a shared cross-modal space. This space is designed to capture the semantic content of both the textual query and the candidate visual samples. The matching score for the i-th sample is computed as:
s( (T(gg) = normalization CooooonVV.... (VLM (T,,
(1)
The aforementioned methods rely on eesoucce-intnnsiv training to improve retrieval performance, requiring sub-
where VLM q) denotes the similarity between candidate I; and query g computed by a specific VLM. The function
3

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
normalization rescales similarity scores to [0, 1] using min- max scaling based the scores among all candidates.
This step offers two key advantages: i) Eliminating low- level redundancies. Unlike raw visual signals, textual de- scriptions naturally filter out irrelevant details such as back- ground noise and texture redundancies, leading to a more semantically compact representation. ii) Aligning visual representations with text queries. By converting sparse vi- sual signals into dense language descriptions, we create representations that are more directly comparable to textual queries, enhancing retrieval consistency within a unified text space.
Despite the effectiveness of VLMs in text-to-visual retrieval, they often struggle with granularity mismatch and seman- tic inconsistency between text and visual content. In brief, the general embeddings learned by VLMs may overlook fine-grained details, leading to suboptimal ranking, partic- ularly when user queries require various granularity rea- soning beyond the high-level semantic alignment captured by VLMs. To address this limitation, we introduce VISA, a plug-and-play approach that enhances text-to-visual re- trieval by reranking the top-k retrieved candidates -op-k-- Specifically, VISA abstracts visual content into textual de- scriptions and performs retrieval within the text space, al- lowing for better alignment with user queries.
3.2.2. QA- QA-BASED DESCRIPTION REFINEMENT
While the general description effectively removes low-level redundancies, it may lack the necessary granularity to fully align with the user query. To address this, we introduce a question-ans answering process that refines the description by extracting query-relevant details, ensuring a more precise alignment between the visual abstraction and the user query.
Let Ti represent the visual abstraction of 4;- The matching score of the i-th sample in the text space is computed as:
* (T:lg) = (Teenalliiaiiioeext-t-e Retri (T:
(2)
This process mitigates the semantic gramularity discrepancy by ensuring that descriptions focus on the most relevant aspects of the query. To achieve this, we employ a chain- of thought approach, where an LLM first analyzes the user query to extract key phrases and generates targeted questions based on these phrases. An LMM then examines each visual candidate and provides context-aware answers, determining whether the extracted key phrases are present in the image or video.
where Text-Retri(T.uu q) denotes the similarity score be- tween the query 9 and the candidate text Tü, calculated by an off-the-shelf text retriever within a unified text space. The final retrieval result I' is obtained by combining the matching scores from both the VLM and the text retriever:
1' = argmaxx (s (Tilg) + (TTil))),
(3)
Question generation with key phrases. As shown in chain- of-thought prompt Prompto in Table 10, we first identify and extract key elements from the user query, focusing objects, attributes, actions, locations, and interactions to accurately capture the core visual conceptes. Once these key phrases are identified, the system generates query-specific questions based on them:
3.2. Visual Abstraction
As illustrated in Fig. 2, VISA consists of two key processes: i) General Description Generation: we employ off-the-shelf LMMs to generate long-form descriptions for each of the top-k retrieved candidates. This process filters out low- level redundancies in visual signals, producing concise yet information-rich descriptions that better align with textual queries. ii) DA-Based Description Refinement: while gen- eral descriptions provide a broad contextual overview, they may lack the specific granularity details required by the query. To refine these descriptions, we introduce a question- answering (QA) process guided by a chain-of-thought (CoT) strategy (Wei al., 2023).
Questions = LLMM LLMo (Prompty. key-phrases, 4),
(5)
where the Prompt, is carefully designed to ensure the ques- tions meet the following criteria:
Clearly Answerable. The generated questions are de- signed so that the LMM can provide definitive answers based on the visual content. This ensures that the pres- ence or absence of queried objects, attributes, or actions can be explicitly confirmed or denied, reducing ambiguity and preventing speculative or overly broad responses.
3.2.1. GENERAL DESCRIPTION GENERATION
To construct semantically rich textual representation of each candidate, we employ off-the-shelf LMMs to gener- ate a long-form general description of the visual content. Formally, the general description T is derived as:
Consistent Granularity. The questions are strictly derived from the key phrases extracted from the user query, ensur- ing that each question targets specific semantic aspect of the visual content. By instructing LLM to avoid direct repetition or close paraphrasing of the original query in Promptg. the questions guides the answering process to- ward distinct details, enabling the description of visual
T.E0 = LMMap (Prompten: It), VI, E Gwp-
(4)
where the prompt for LMM is provided in Table 8, utiliz- ing the default prompt settings of these models to generate detailed descriptions.

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Table Image retrieval results on COCO and FIickr30K, indicates results are re-evaluated using official checkpoints from HuggingFace.
COCO
Flickr30K
R@1 1
R@5
R@ 10
R@1
R@5
R@ 10
CoCa (Yu et al., 2022)
51.2
74.2
82.0
80.4
95.7
97.7
FLAME (Cao et al., 2024)
43.9
70.4
79.7
73.3
91.7
95.5
DreamL.P (Zheng et al., 2024)
67.0
76.6
88.3
93.3
FLAIR (Xiao et al., 2024)
53.3
77.5
81.1
94.9
UMG-CLIP (Shi et al., 2024)
57.5
80.4
87.3
81.2
95.7
97.7
RAGVL (Chen et al., 2024c)
84.4
95.2
96.3
IntemVL-G (Chen et al., 2024b)
81.3
88.0
85.0
97.0
98.6
SigLIP" EVVA-LLIP
56.0
79.1
85.9
84.1
96.7
98.3
BLIP-2 ct al., 2023)
89.7
98.1
98.9
SigLIP" (Zhai et al., 2023)
54.2
76.8
84.2
83.0
96.1
98.0
SigLIP* + VISA (Ours)
57.2
80.3
86.9
85,1
97.1
98.6
A
+3.0
+3.5
+2.7
42.1
+1.0
+0.6
EVA-CLIP" (Sun et al.. 2023)
55.6
77.9
85,2
83.1
95.8
97.9
EVA-CLIP* VISA (Ours)
59.5
81.2
87.5
86.1
97.3
98.6
A
+3.9
+3.3
+2.3
+3.0
+1.5
+0.7
content across varying levels of granularity, from high- level scenes to lineggriine attributes.
4. Experiments
In this section, first describe the implementation details of VISA. Next, we integrate VISA into various VLMs and compare its performance with state-of-the-ard approaches across ten datasets. Finally, we conduct detailed analyses and ablation studies to evaluate the effectiveness and robust- ness of our method. Due to space limitations, additional experimental details and results, including visualization ex- amples, are provided in Appendices A to F.
Answering based on key phrases. To supplement the details required by the user, we use an LMM to answer questions for each top-k candidate sample. Formally, the corresponding answer is obtained as:
T:a LMMA (Prompla. key-phrases, questions, 4,), (6)
where the key phrases are also provided to guide the LMM in generating context-aware answers. As shown in Table 9, the prompt Prompt, is carefully designed to ensure:
4.1. Experimental Settings
Unless stated otherwise, we utilize LMMs LLaVA-vI.8 34B (Liu et al., 2024a) and LLaVA-Video--33 (Zhang al., 2024b) obtain the general descriptions of images and videos, respectively. For the guestion-answering process, we employ LLM Qwen2.5-32D (Team, 2024) as the ques- tion generator, LMM Swee2-VL-777 (Wang et al., 2024a) as the answer generator, and gemma2 (Chen et al., 2024a) as the text retriever. For video data, the frames per second (FPS) input to LMMs is gallery (k) is set to 20 and the number of questions is 3 for all datasets. For all evaluated models listed in Appendix E, we use the default
Encouraging Detailed Responses. The model explicitly instructed to avoid short or binary answers (e.g.. "Yes" or "No"") and instead provide complete, context-rich re- sponses that enhance text retrieval.
Ensuring Relevance. The model is guided to consider the relationship between the candidate and key phrases, avoiding out-of-context or irrelevant answers.
3. The size of the reranking
Handling Uncertainty. If visual content does not provide a clear answer, the model is explicitly instructed to respond with "Uncertain" and such responses are subsequently discarded. This is crucial in preventing hallucinated misleading information, ensuring responses remain the actual content rather than introducing irrelevant details.
provided on Hugging- Face. All experiments are conducted on Ubuntu 20.04 with NVIDIA 4090 GPUs.
4.2. Main Results
By employing this CoT-based QA approach, VISA ensures that varying granularity details effectively captured. The final visual abstraction T; is constructed by concatenating the general description 7, with the QA-generated answers TAA for the i-th candidate, ensuring query-consistent granu- larity in retrieval. The final retrieval results are then obtained through Eqs. and 3.
We evaluate the effectiveness of our proposed method, VISA, on both short- and long-context retrieval tasks for images and videos.
5

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Table 2. Video retrieval results CER-VTT, DiDeMo and MSVD.
MSR-VTT
LSMDC
DiDeMo
MSVD
R@1 1
R@5
R@10
R@1
R@5
R@
10
R@)
R@5
R@ 10
R@1
R@5 66.9
R@ 10
CLIP4Clip (Luo et al., 2021)
32.0
57.0
66.9
15.1
28.5
36.4
38.5
76.8
Video (Wang et al., 2022)
40.7
17.6
32.4
40.2
31.5
57.6
68.2
43.4
BT-Adapter (Liu et al., 2024b)
40,9
64.7
73.5
19.5
35.9
45.0
35.6
61.9
72.6
vid-TLDR (Choi et al., 2024)
42.1
63.9
72.4
52.0
74,0
81.0
50.0
77.6
85.5
UMT-L (Li et al., 2024)
42.6
64.4
73.1
25.2
43.0
50.5
48.6
72.9
79.0
49.0
76.9
84.7
LanguageBind (Zhu et al., 2024)
42.8
67.5
76.0
39.7
65.5
73.8
54.1
81.1
88.1
mPLUG-2 (Xu ct al., 2023)
47.1
69.7
79.0
24.1
43.8
52.0
45.7
71.1
79.2
VAST (Chen et al., 2023b)
49.3
68.3
73.9
55.5
74.3
79.6
InterVideo2-C" (Wang et al., 2024h)
46.0
70.3
79.3
24.3
42.8
50.5
45.9
71.8
79.8
55.1
81.2
87.9
InternVideo2-C* VISA (Ours)
48.8
73.7
80.9
28.3
47.3
55.1
54.8
78.8
84.0
57.8
83.2
89.1
A
+2.8
43.4
+1.6
+4.0
+4.5
+4.6
4-8.9
+7.0
44.2
42.7
42.0
+1.2
Intern Viden2-9 (Wang et al., 2024b)
52.0
74.6
81.8
32.0
52.4
59.4
61.2
82,4
87.3
58.9
83.0
88.7
Intern Video-iid VISA (Ours)
54.4
75.3
82,9
35.0
53.1
61.0
63.7
86.0
89.7
60.2
84.5
89.7
A
+2,4
+0.7
+1.1
+3.0
+0.7
+1.6
+2.5
+3.6
+2,4
+1.3
+1.5
+1.0
4.2.1. 1. SHORT-CONTEXX IMAGE RETRIEVAL
perform methods like DreamLIP and FLAIR, which gen- erate long textual descriptions of images to enrich seman- tics. Additionally, our method surpasses reranking approach RAGVL (Chen et al., 2024c) that fine-tune LMMs, as well as FLAME (Cao et al., 2024) that distill knowledge from frozen LLMs. This illustrates that employing large models at test time effectively enhance retrieval performance without requiring extensive retraining fine-tumin tne-tunnig.
We evaluate approach on the MS-COCO (Chen et al., 2015) and Flickr30K (Plummer et al., 2015) datasets. MS- COCO includes 5,000 test images, each annotated with five manual annotated captions describing a diverse range of objects. Flickr30K emphasizes real-world scenarios with complex interactions, providing five detailed annotations for each of its 1,000 test images, capturing object relationships and actions.
4.2.2. SHORT-CONTEXT VIDEO RETRIEVAL.
We first evaluate the effectiveness of VISA based on widely- used VL.Ms SigL.P (Zhai et al., 2023) and EVA-CLIP (Sun et al., 2023). As shown in Table 1, our approach achieves substantial improvements in image retrieval, particularly in terms R@I For instance, applying VISA SigLIP (approximately 1B parameters) boosts R 1 from 54.2% to 57.2% on MS-COCO and from 83.0% to 85.1% Flickr30K. Similarly, integrating VISA with the much larger 18B- -aB-parameter EVA-CLIP raises R @ from 55.6% to 59.5% on MS-COCO and from 83.1% to 86.1% on Flickr30K. Note that BLIP-2 is finetuned on COCO and zero-shot transferred Flickr30K, contributing to the enhanced zero-shot perfor- mance on Flickr30K (Chen al., 2024b).
For the video retrieval task, we evaluate our approach on four widely-used datasets: MSR-VTV (Jun et al., 2016), DiDeMo (Hendricks et al., 2017), LSMDC (Rohrbach et al., 2015). and MSVD (Chen Dolan, 2011). MSR-VTT spans broad range of scenarios, including sports, music, cooking, and social activities. DiDeMo provides sentence-level anno- tations describing different time segments of video, which are merged into a single query for video- paragraph retrieval. LSMDC consists clips from movies across various gen- res, annotated by a concise description of key events and interactions. MSVD includes YouTube videos capturing ev- eryday activities and events. These datasets contain diverse scenarios and annotation styles, offering comprehensive benchmark for evaluating video retrieval tasks.
Notably, even combining SigLIP and EVA-CLIP, which together contain a massive number of parameters, under- performs compared to our approach. This suggests that scrger-scale VLMs do not inherently guarantee better perfor- mance, highlighting the efficacy of our carefully designed visual abstraction strategy.
We employ Intern Video2-C and InternVideo2-C (Wang et al., 2024b) as the base VLM model. Intern Video2-C is a dual-stream architecture, while InternVideo2-Gi is a single-dual hybrid model built upon InternVideo2-C. As shown in Table our method consistently achieves higher R@ I scores across all datasets compared competing ap- proaches. For the dual-stream Intern Video2-C, our method improves R@ from 46.0% to 48.8% on MSR-VTT and
Our approach also achieves superior performance com- pared to existing methods that leverage large models for retrieval (as discussed in Sec. 2.2). Specifically, VISA out-
6

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Table 3. Long-context retrieval results DCI, II, Urban Ik and ShareGPT4v.
DCI
I1W
Urbanlk
ShareGPT4v
@@1
R@5
R@ 10
R@1
R@5
R@ 10
R@1 1
R@5
R 10
R@1
R@5
R@ 10
CLIP (Radford et al., 2021)
46.0
67.8
75.5
91.3
98.5
99.3
62.1
82.7
89.2
82.2
96.2
98.3
LiT (Fan al., 2023)
40.9
82.7
80.0
ALIGN (Jia et al., 2021)
57.4
90.7
85.3
SigLIP" (Zhai et al., 2023)
61.3
79.0
84.3
93.3
99.5
99.7
74.8
88.8
92.8
88.9
98.2
99.2
FLAME (Cao et al., 2024)
87.9
93.2
FLAIR (Xiao et al., 2024)
66.2
87.7
98.0
Long-CLIP (Zhang et al., 2024a)
61.0
79.6
85.1
94.6
99.3
99.5
80.5
95.4
96.8
95.4
99.9
100.0
LoTLIP" (Wu et al., 2024)
62.5
82.9
88.1
94.0
99.0
100.0
85.9
97.1
98.4
96.1
99.5
99.8
LoTLIP" + VISA (Ours)
74.6
88.7
91.1
98.2
100.0
100.0
94.6
99.4
99.8
97.8
99.8
99.9
A
+12.1
+5.8
43.0
+4.2
+1.0
+0.0
+8.7
+2.3
+1.4
+1.7
+0.3
40.1
24.3% to 28.3% on LSMDC. Notably, it achieves a sig- nificant R@l gain on DiDeMo, increasing from 45.9% to 54.8%, a remarkable improvement of 8.9%. Similarly, VISA benefits the single-stream Video2-G signifi-
2024a; Xiao et al., 2024; Cao et al., 2024), our method con- sistently outperforms them, demonstrating its effectiveness in handling long-context retrieval tasks.
cantly, with R@1 improvements from 52.0% 54.4% on MSR-VTT, 32.0% to 35.0% LSMDC, and 61.2% to 63.7% DiDeMo. These results demonstrate that VISA could boost retrieval across various architectures, showcas- ing its versatility and robustness.
4.3. Analysis Experiment
To evaluate the robustness of our method, we conduct a series of analytical experiments.
General Description. Table 4(a) presents the results of our method using different LMMs to generate long textual descriptions. Here the results obtained without applying the suestionn-nsswene refinement. As shown, small mod- els such LLaVA-7B and Qwen2VL-2BB already achieve significant improvements. Additionally, as the model size increases, the quality of generated captions improves ac- cordingly, leading to better retrieval performance.
4.2.3. LONG-CONTEXT RETRIRVAL.
The ability to effectively process and retrieve long-context texts plays critical role in meeting fine-grained and multi-granularity retrieval demands. To evaluate our ap- proach, we utilize four datasets: DCI (Urbanek et al., 2024), IIW (Garg et al., 2024), Urban-1k (Zhang et al., 2024a), and ShareGPT4v (Chen et al., 2023a). DCI and Urban-lk both include urban scenes, with DCI featuring descriptions averaging over 170 tokens, and Urban-1k k highlighting object types and spatial relation- ships. IIW provides multi-granularity annotations capturing spatial arrangements and nuanced attributes. ShareGPT4v emphasizes detailed visual information.
Number of Questions. We analyze the impact of varying the number of questions on retrieval performance. As shown in Table 4(b), increasing the number of questions generally improves performance, with 3-5 questions providing an op- timal balance between effectiveness and computational cost. However, using more than five questions Ieads to perfor- mance drop on MSR-VTT, likely because the user queries in this dataset are relatively short, resulting in redundant questions that repetitively asks the common concepts across different candidates.
We use LoTLIP (Wu et al., 2024) as the base text-to-image model. As shown in Table 3, our method achieves signifi- cant improvements across all datasets, with consistent gains across all metrics. Specifically, our approach boosts R@1 on DCI from 62.5% to 74.6%, remarkable gain of 12.1 1%, demonstrating its capability to handle complex fine-grained details. Similarly, R@I increases by 8.7% on Urban Ik, 4.2% on IIW. and 1. 1.7% on ShareGPT4v, further highlight- ing the robustness and generalizability of our method across diverse datasets.
Answer Generator. We evaluate different answer genera- tors in Table 4(c). The results demonstrate that large-scale answer generators significantly enhance performance. Inter- estingly, the Qwen 7B model achieves performance compa- rable the much larger LLaVA-34B model. Based on our observations of the generated answers, we attribute Qwen's 's superior capabilities to its excellence in both answering questions and rejecting irrelevant ones (i.e., response with "Uncertain" as discussed in Sec. 3.2.2).
Notably, the substantial improvements on DCI and Ur- banlk emphasize VISA's ability to process dense and multi- granularity user queries. Compared to existing methods specifically designed for long-context queries (Zhang et al.,
Text Retriever. We evaluate different text retriever Ta- ble 4(d). Our findings indicate that while single-stream
7

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Table Analysis experiments different modules of VISA framework. We report R@ and R@5 on Flickr30K (based EVA-CL.I)) MSR-VTT (based on Intern Video2-G) G) and Urbanlk (based on LoTLIP). Default setting are marked in gray
(a) General Description (GD). Higher quality descriptions lead better retrieval performance.
(b) Number of Questions (NQ). A medium number of questions balances effectiveness and computational cost.
FOickr30K
AISR-TT1
Urban k
Flickr30K
MSR-VTT
Urban I
GD
NQ
R@1
R@5
R@1
R@5
R@1
R@5
R @
R@5
R@
R@5
R@1
R@5
w/o GD
83.1
95.8
52.0
74.6
85.9
97.1
85.6
96.8
53.3
75.0
94.2
99.2
LLaVA-7B
84.8
96.8
52.4
73.6
93.3
99.4
85.7
97.0
53.9
74.3
94.3
99.1
LLa VA-3B
85.6
53.3
75.0
94.2
99.2
86.1
97.3
54.4
75.3
99.4
Qwen2VL-2B
84.5
96.6
52.5
73.9
95.1
99.5
86.2
97.1
54.6
75.8
94.6
99.6
Qwen2VL-7B
85.1
96,9
53.5
74.5
95.6
99.6
7
86.1
97.1
53.8
75.2
94.6
99.4
Qwen2VI.-72B
85.4
96.8
53.6
74.9
95.9
99.7
86.2
97.2
53.9
75.3
94.9
99.4
(c) Answer Generator (AG). Stronger answer generators lead to improved retrieval performance.
(d) Text Retriever (TR). Whether single- dual-stream, or lightweight models can effectively enhance retrieval performance.
AG
Flickr30K
MSR-VTT
Urban 1k
TR
type
Flickr30K
MSR-
VTT
Urbanlk
R@
R@5
R@D
R@5
R@1
R@5
R@1 1
R@5
R@1
R@5
R@1
R@5
w/o TR
83.1
95.8
52.0
74.6
85.9
97.1
AG
85.6
96.8
53.3
94.2
99.2
stella-435M
dual
85.2
97.2
53.4
75.0
93.3
98.9
LLaVA-7B
83.4
96.4
52.6
73.7
92.1
99.2
gte-Qwen2-11.m
1.5BB
dual
85.3
97.0
53.4
75.3
93.0
99.0
LLaVA-34B
86.4
97.2
54.2
75.4
93.9
99.3
bge-en-iel-7B
dual
85.9
97.2
53.6
75.4
94.1
99.2
reranker-568M
single
85.5
97.4
53.5
74.0
91.0
98.7
Qwen2VI.-2B
85.5
96.7
53.2
75.0
94.2
99.3
layerwise-2.7B
single
86.2
97.4
74.8
94.1
Qwen2VL-7B
86.1
97.3
54.4
75.3
94.6
99.4
gemma2-9B
single
86.1
97.3
75.3
94.6
99.4
providing comprehensive visual abstraction for capturing the image's semantics. Similarly, removing the question- answering process (w/o QA) also degrades performance, although to a lesser extent. Notably, eliminating both com- ponents together (w/o GD&QA) results in the largest de- cline, highlighting their complementary roles retrieval process. Additionally, we observe consistent performance decline when the key phrases in chain-of-thought is removed (w/o CoT), confirming the critical role of deep reasoning in generating questions. These findings strongly support our hypothesis that granularly consistent visual abstraction significantly enhances visual retrieval performance.
Table Ablation study. Here, "GD", "QA" and "CoT" refer to general description, question-answering. and chain-of-thought, respectively, Default setting are marked gray.
Flickr30K
MSR-VTT
Urban1k
method
R@1
R@5
R@1 1
R@5
R@1
R@5
VISA (Ours)
86.1
97.3
54.4
75.3
94.6
99.4
w/o GD
84.4
96.9
53.2
74.1
90.3
98.9
w/o QA
85.6
96.8
53.3
75.0
94.2
99.2
w/o GD&QA
83.1
95.8
52.0
74.6
85.9
97.1
w/o CoT
85.6
97.1
54.0
73.0
94.2
99.4
models exhibit explicit sentence fusion and achieve better performance than dual-stream models, dual-stream mod- els still demonstrate substantial performance improvements. Moreover, performance improvements are consistently ob- served across models of varying sizes, including both large- scale and lightweight retrievers.
5. Conclusion
In this paper, explore new perspective for eext-to-visual retrieval by transforming visual content into textual repre- sentations and conducting retrieval within unified text space. To achieve this, we propose VISA, a plug-and-play approach that seamlessly integrates with existing VLMs, leveraging off-the-shelf large models to generate concise and query-aligned descriptions, thereby enhancing retrieval performance test time. By reformulating traditional cross- modal tasks into pure text space, visual abstraction has the potential to benefit other multi-modal applications, such as video grounding and composed image retrieval, provid-
4.4. A Ablation Study
In this section, we conduct ablation studies to evaluate the impact of different modules in VISA. As shown in Table 5, omitting the general descriptions (w/o GD) results a sig- nificant performance drop, highlighting the importance of
8

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
ing foundational framework for improving cross-modal alignment.
Data collection and evaluation server. arXiv preprint arXiv:/504.000 2015.
Chen, Y.-C., Li, L.,, Yu, L., EI Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., and Liu, J. Uniter: Universal image-text repre- sentation learning. In European conference on computer vision, pp. 104-120. Springer, 2020.
Impact Statement
VISA inherits potential challenges associated with LMM- generated descriptions, such as biases and hallucinations. Since deseriptions are automatically generated, the retrieval process might be influenced by inaccurate biased rep- resentations, leading unintended consequences, particu- larly in sensitive domains like news media analysis or legal Decision-making Additionally, transforming images into textual descriptions could raise privacy concerns, especially when applied to personal or surveillance-related datasets.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks, 2024b. URL https: hatss://axxvv.vr abs 2tt./212.1111 02331.112336
Chen, Z., Xu, C., Qi, Y., and Guo, J. Mllm is strong reranker: Advancing multimodal retrieval-augmente generation via knowledge-enhancee reranking and noise- injected training, 2024c. URL
Acknowledgments
This work was supported in part by NSFC under Grant 62176171, 624B2099, 62472295, U24B20174; in part by the Fundamental Research Funds for the Central Universi- ties under Grant CJ202303, CJ202403; in part by Sichuan Science and Technology Planning Project under Grant 24NSFTD0130; and part by Baidu Scholarship.
/arxiv.
org/ abs/ / 2407 21439.
Choi, J., Lee, S., Chu, J.. Choi, M. and Kim, H. J. vid-tldr: Training free token merging for light-weight video trans- former, 2024. URL https: http://axvvvv//// /abs/ 2403 13347.
References
Fan, L., Krishnan, D., Isola. P., Katabi, D., and Tian, Y. Improving clip training with language rewrites, 2023. URL https: //arxiv/aiiivii///a//eaeeererai org/oogg///.....oooo..oo.....o.e httpa/ab/2222222//
Bain, M., Nagrani, A., Varol, G., and Zisserman, Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
Garg, R., Burns, A., Ayan, B. K., Bitton, Y., Montgomery, C., Onoe, Y., Bunner, A., Krishna, R., Baldridge, J., and Soricut, R. Imageinwords: Unlocking hyper-detailed image descriptions, 2024.
2021.
Cao, A.,
Wei, X., and Ma, Z. Flame: Frozen large lan- guage models enable data-efficient language- aangugge-image pre- training, 2024. URL https: https: /ttp:://axxvvvvo org. /bs/
Hendricks, L. A., Wang, O., Shechtman, E., Sivie, J., Dar- rell, T., and Russell, B. Localizing moments in video with natural language. In 2017 IEEE International Con- ference on Computer Vision (ICCV), Oct 2017. doi: 10. 109//ccv.001..6.8.. URL http:// //dttp:/dx..... http::://xx.....ooo/o www.dooioooo.o.ooooo org/ 10. 12/109/1can 2017 2017.618 618.
Chen, D. and Dolan, W. Collecting highly parallel data for paraphrase evaluation. Meeting of the Association for Computational Linguisties, Jun 2011.
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embedding: Multi-lingual, multi- functionality, multi-granularity text embeddings through self-knowledge distillation, 2024a.
Huang. Z., Yang. M., Xiao, X., Hu, P., and Peng. X. Noise- robust vision-language pre-training with positive-negative learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi- modal models with better captions, 2023a. URL https: //arxiv. org/ abs 2aobs/23// 2311 111311111111 12793.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint 2rXiv:2410.2233 2024.
Chen, S., Li, H., Wang. Q., Zhao, Z., Sun, M., Zhu, X., and omni-modality
Liu, J. Vast: A vision-audio-suutiil--- foundation model and dataset, 2023b. URL https: //arxiv. org/ abs/ 2305. 18500.
Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. Scaling up visual and laniunne representation learning with noisy text supervision, 2021. URL https:////// //arxiv.. org/abs/215 019188188.........
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P. and Zitnick, C. L. Microsoft coco captions:

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Jun, X., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. IEEF Conference Proceedings, IEEE Conference Proceedings, Jan 2016.
Liu, Z., Sun, W. Teney, D., and Gould, S. Candidate set re-ranking for composed image retrieval with dual multi-modal encoder. Transactions on Machine Learn- ing Research, 2024c. ISSN 2835-8856. URL https: / / //openrevi ew net. forum? f d=weAvec
Lee, K.-H., Chen, X., Hua, G., Hu, H.,, and He, X. Stacked cross attention for image-text matching. In Proceedings of the European conference computer vision (ECCV), pp. 201-216, 2018.
Lu, Y., Yang, M., Peng, D., Hu, P., Lin, Y., and Peng, X. Llava-reid: Selective multi-image questioner for in- teractive person re-identification, 2025. URL https: // wlaaaiivorgg/aaaee.aa. 2504. 10174.
Li, D., Li, J., Li, H., Niebles, J. C., and Hoi, S. C. Align and prompt: aideo-and-language pre-training with entity prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4953- 4963, 2022a.
Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. Clip4clip: An empirical study of clip for end to end video clip retrieval, 2021. URL httts:://aaatt org/ / abs 21 04.00.... 08860.
Li, J., Selvaraju, R. R.,, Gotmare, A. D., Joty, S., Xiong, C., and Hoi, S. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021.
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockemmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. CoRR, abs/1505.00870 2015. URL http: //rxivi/// .rgg abs /1055 04870.
Li, J.. Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-iman image pre-training for unified vision-language understanding and generation, 2022b. URL https:// arxiv. org/ abs 2201. 12086.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. URL https: arxiv org/ abs /2103. 200200000..0.....
Li, J. Li, D., Savarese, S., and Hoi, S. Blip-2: Boot- pre-training with frozen im-
strapping language-ing age encoders and large language models, 2023. URL https: arxiv. org/ abs/ abs/aa01...... 22011 12597.
Ren, S., He, K., Girshick, R., and Sun, Faster r-cmn: Towards real-time object detection with region proposal networks. IEEE transactions pattern analysis and machine intelligence, 39(6):137--1199 2016.
Li, K., Wang, Y., Li, Y., Wang, Y., He, Y., Wang, L., and Qiao, Y. Unmasked teacher: Towards training- efficient video foundation models, 2024. URL https: //arxiv. org/abs///////oooo.o....o.....ee abaaaaa3..3./. 16058
Rohrbach, A., Rohrbach, M., Tandon, N., and Schiele, B. A dataset for movie description. arXiv: Computer Vision arapiitionn,xxiv.: Pattern Recognition, Jan 2015.
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Object- semantics aligned pre-training for vision-language tasks. In Computer Essio-EECCY 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137. Springer, 2020.
and Pattern Recognition, aivi Computer Vision and
Shi, B., Zhao, P., Wang, Z., Zhang, Y., Wang, Y., Li, J., Dai, W., Zou, J., Xiong, H., Tian, Q. and Zhang, X. Umg-clip: A unified multi uuatiigranuulrity vision generalist for open-world understanding. 2024. URL https:// arxiv. orooorgg///.... /4001.41111 06397.
Lin, Y., Zhang. J., Huang, Z., Liu, J., Wen, Z., and Peng, X. Multi-granularity correspondence learning from long-term noisy videos, 2024. URL https::///// /arxiv... org/ abs//bb///// 240111116770
Sun, Q., Wang, J., Yu, Q., Cui, Y., Zhang, F., Zhang, X., and Wang, X. Eva-clip- Scaling clip to 18 billion 2023.
Liu, H. Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL
parameters. arXiv preprint
Tang, C., Xie, L., Zhang, X., Hu, X., and Tian, Q Visual recognition by request. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15265-15274, 2023.
tig.....tt.... io/blog/ 2024-00 01 30 20-vaava next/.
Liu, R., Li, C., Ge, Y., Shan, Y., Li, T. H., and Li, G. Bt- adapter: Video conversation is feasible without video in- struction tuning, 2024b. httss: /ttpass///xxvv..o/e //aaxiivooorg...... org/ abs/2309 15785.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https:/ qwenlm. github. io/ blog gwwng.en2552
10

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Urbanek, J., Bordes, F., Astolfi, P., Williamson, M., Sharma, V., and Romero-Soriano, A. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 86700-26709, June 2024.
Zhang, B., Zhang, P., Dong, X., Zang, Y., and Wang. J.
Long-clip: Unlocking the long-text capability of clip, 2024a. URL https: /ttp/axxivvv../// org abg//bbbb/433 15378.
Zhang, D., Li, J., Zeng, Z., and Wang, F. Jasper and stella: distillation of sota embedding models, 2025. URL https: / /awpiv.iiiiv.///.a///aaa/aaaaava org/ hhtp//b///20 /ttpa//b///......
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K. Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint 12211 2024a.
Zhang, Y., Li, B., Liu, h., Lee, Y. j.. Gui, L.., Fu, D., Feng, J., Liu, Z., and Li, C. Llava-next: A strong zero-shot video understanding model, April 2024b. URL https: attp:::/Laav--vuggihh.... blog/ 2222-04-30--1aaa--
Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang. Z., Xing, S., Chen, G., Pan, J., Yu, J., Wang, Y., Wang. I.,, and Qiao, Y. Internvideo: General video foundation models via gen- erative and discriminative learning, 2022. URL https: //arxiv. org/ abs/ 2212.2222222 03191.
Zheng, K., Zhang, Y., Wu, W. Lu, F. Ma. S., Jin, X., Chen, W., and Shen, Y. Dreamlip: Language-image pre-training with long captions, 2024. URL https: / f/arxiv.oo orge abs 240.. 17007.
Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang, Y., Jiang, W., Zhang, J., Li, Z., Zhang, W., Li, Z., Liu, W., and Yuan, L. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment, 2024. URL https: / /arxivvv.v. org /abs/ 2310. 01852.
Wang, Y., Li, K., Li, X., Yu, J.. He, Y., Wang, C., Chen, G., Pei, B., Yan, Z., Zheng, R., Xu, J., Wang, Z., Shi, Y., Jiang, T., Li, S., Zhang, H., Huang, Y., Qiao, Y., Wang, Y., and Wang, L. Internvide02: Scaling foundation models for multimodal video understanding, 2024b. URL https: / arxiv. org/ /trg/bb//// 2403. 15377.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q.. and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models, 2023. URL https: /tap:://arxiii /ttp//axxvvv///////v//.///..aavl orgoorggaaooo / 2201.11903 11903.
Wu, W., Zheng. K.. Ma, S., Lu, F., Guo, Y., Zhang, Y.. Chen, W., Guo, Q. Shen, Y., and Zha, 7.-J. Lotlip: Improving language-image pre-training for long text understand- ing, 2024. URL https: //tp://////v..///a/eeaeeeeeaaave wrtpb/s.r///////o 05249.
Xiao, R., Kim, S., Georgescu, M.-I., Akata, Z., and Alaniz, S. Flair: Vlm with fine-grained hanguage-informee image representations, 2024. URL https: / //tp://aavvvv///e///a//eeeeaawva org/ abs/ 011.//222...3.6.
Xu, H., Ye, Q., Yan, M., Shi, Y., J., Y., Li, C., Bi, B., Qian, Q. Wang, W., Xu, G., Zhang, J., Huang, S., Huang, F., and Zhou. J. mplug-2: A modularized multi-modal foundation model across text, image and video, 2023. URL https:tp//// www.a.avvv.... org/ abs 2002022222 00402.
Yu, J., Wang, Z., Vasudevan, V., Yeung, L.. Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models, 2022. URL htthht//p///aaa
hrg//bs/// 22... 01917.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training, 2023.

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Appendix Overview
This supplementary document is organized as follows:
Sec. A shows the performance of VISA based on BLIP-2.
Sec. B shows the FLOPs and lateney of VISA.
Sec. C places the prompts for large models used in VISA.
Sec. D shows the details of the evaluation datasets.
Sec. E shows the details of the evaluation models
Sec. F presents visualization results.
A. VISA based on BLIP-2
We compare our method with BLIP-2, referencing Table 5 from the original BLIP-2 paper (Li et al., 2023). As shown in Table 6, VISA consistently improves performance when applied top of BLIP-2. This demonstrates the compatibility and effectiveness of our method even when applied to finetuned VLMs.
Table 6. Results of VISA based on BLIP-2.
COCO Fine-tuned
Flickr30K Zero-shot
Method
R@D
R@S
R@ 10
R@ 1
R@s
R@ 10
ALBEF (Li et al., 2021)
60.7
84.3
90.5
82.8
96.3
98.1
BLIP (Li et al., 2022b)
65.1
86.3
91.8
86.7
97.3
98.7
BLIP-2 ViT-L (Li et al., 2023)
66.3
86.5
91.8
88.6
97.6
98.9
BLIP-2 VTT-g (Li et al., 2023)
68.3
87.7
92.6
89.7
98.1
98.9
BLIP-2 ViT-g + VISA (Ours)
68.9
88.0
92.9
90.2
98.4
99.2
A
+0.6
+0.3
+0.3
40.5
40.3
+0.3
B. Efficiency of VISA
In this section, we analyze the efficiency of VISA on the Flickr30K dataset. When deployed in practice, the VISA-based retrieval process can be divided into two stages:
Offline stage precomputes visual features (via VLM) and generates general descriptions for all gallery candidates.
Online stage performs the text encoding (via VLM), QA-based description refinement and text-level re-ranking during inference.
Table FLOPs latency VISA.
Type
Model
GFLOPs
Latency (second dataset)
VLM (SigLIP/ EVA CLIP BLIP-2)
4560
4/ 437.5
offline
General Description (LLaVA-v1.6-34p)
138000
Type
Model
GFLOPs
Latency (second per query)
VLM (SigLIP/ / EVA-CLIP BLIP-2)
26 / 1460
0.0002 / 0.0029 0.0421
Question Generator (Qwen2.5-32B)
26870
0.02
online
Answer Generator (Qwen2VLL-7B)
15450
1.00
Text Retriever (gemma-9B stella-435M)
4160 1.6
0.13 0,0005
We report the FLOPs and latency of VISA in Table 7. The offline stage of VISA includes generating general descriptions for gallery candidates using a large multi-modal modal model (e.g., ALLVVA-v166-34B) with a total computational cost of 138K
12

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
GFLOPs and 437.5 seconds the Flickr30K dataset. This stage is executed only once and does affect online inference. For fair comparison, report only the dual-stream latency of BLIP-2 in the offline stage, given its hybrid-stream architecture.
The online stage includes query encoding, QA-based refinement, and text-level reranking with an average latency of approximately second per query. This stage supports multiple configurations that enable trade-offs between performance and efficiency: i) QA-based refinement can be treated as an optional step. Using only general descriptions still yields strong performance, as shown in Table 4(a); ii) the default text retriever can replaced with a lightweight model such as stella-435M (Zhang al., 2025), reducing latency to 0.0005 seconds per query while still providing substantial performance gains (as demonstrated in Table 4(d)). These configurable options allow VISA adapt flexibly to different deployment scenarios, balancing retrieval quality with computational efficiency based on application demands.
C. Prompts of Large Models
The prompts utilized in VISA are provided in Tables 10.
Table The prompt of general deseription generation.
prompt image data=
f**"
Please generate detailed descriptions of the given image.
prompt of video data-
f"nn
Please provide detailed description of the video, focusing on the main subjects, their actions, the background scenes.
Table 9, The prompt of answer generator.
prompt "*"
You need to answer the following question about the given i image/ /vi deo in English: {question).
Avoid giving answers are just 'Yes', 'No', or a single word. Each response should provide complete sentence.
If you cannot determine the answer or there are no objects that are asked by the question, just answer single word 'Uncertain'.
Note that if the given video unrelated or only loosely related these phrases: phrases), just response single word 'Uncertain',
13

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Table 10. The prompt for the question generator, including an example specify the output format which is used for all datasets.
prompt "*""
You are observing a managevvides A single sentence is provided to describe what you are seeing in the image / video.
Based on this sentence, perform the following tasks:
Extract Key Phrases: Identify and list key phrases from the sentence that represent the main elements of the scene. Facus on capturing: object types, object attributes, object actions, object locations, interactions between objects or other dynamic elements.
2. Generate Three Questions: Based the extracted key phrases, create three natural language questions that visual Al assistant might ask about the image/ /vvido. The questions should:
2.1 Be specific and focused on the visual details described in the sentence.
2.2 Ensure that the answer to each question can be determined confidently:
Either the content is present and can be answered confidently from the image hogge/viide viico.
content absent, and its non- con-preeence can confidently verified.
2.3 Only ask questions about quantities (e.g.. "How many...") the description explicitly mentions number or quantity
(c.g., "three," "several").
2.4 Avoid directly repeating closely paraphrasing most of the sentence.
2.5 Ensure the answers to the questions can be provided using the extracted key phrases.
Output Requirements:
Key phrases: Provide numbered list of key phrases extracted from the input sentence.
Questions: Provide numbered list of three relevant questions based on the key phrases.
Example:
Input:
A green and yellow tennis sweater is hanging the back of the sofa.
Output:
key phrases:
1. green and yellow
2. tennis sweater
green and yellow tennis sweater hanging on
5. the back of
6. sofa
7. the back of the sofa
hanging on the back of the sofa
questions:
1. What colors are present on the tennis sweater?
2. Where is the tennis sweater located?
What type of object is the sweater hanging on?
Now, perform these tasks for the following input: {query}
14

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
D. Evaluation Datasets
In this section, we place more details about the evaluation datasets involving text-lo-image retrieval (COCO and Flickr30k), text-to-video (MST-VTT, LSMDC, DiDeMo, and MSVD), and long- long-context retrieval (DCI, IIW, Urban Ik and ShareGPT4v).
COCO (Chen al., 2015) includes images featuring wide variety of objects such people, animals, vehicles, and household items. The test set of COCO contains 5,000 images with 25,010 manual annotations.
Flickr30k (Plummer al., 2015) focuses on real-world scenes with complex interactions and relationships. Each image is annotated with five descriptive sentences, capturing detailed information about objects, actions, and their interactions. The test set of Flickr30k contains 1,000 images with 5,000 manual annotations.
MSR- VTT (Jun et al., 2016) spans a broad spectrum of everyday scenarios and events, including sports, music performances, cooking demonstrations, gaming sessions, and various social and cultural activities. The test set of MSR-VTT contains 1000 videos.
DiDeMo (Hendricks et al., 2017) encompasses a wide variety of real-world scenarios, including cooking, sports, travel, and general daily activities. The test set DiDeMo contains 1,003 videos, each trimmed to a maximum 30 seconds and paired with several descriptive sentences that highlight distinet moments. We perform video-paragraph retrieval by merging all sentence descriptions of video into a single query.
LSMDC (Rohrbach et al., 2015) contains short video clips from movies, spanning various genres such as drama, comedy, and action. The test set LSMDC contains 1,000 clips and each clip is annotated with one concise English descriptions focusing on key events, dialogues, and character interactions.
MSVD (Chen & Dolan, 2011) covers short YouTube videos capturing diverse everyday activities, objects, and events. The test set of MSVD contains 670 videos, each annotated with around 40 English descriptions that highlight various actions and contexts. We perform video aadoo-paragrah retrieval by merging all sentence descriptions of a video into a single query.
DCI (Urbanek et al., 2024) includes a diverse range of real-world scenes, covering urban landscapes, architectural structures, natural environments, and everyday objects. It consists of 7,805 natural images, each accompanied by descriptions averaging over 170 tokens per image.
detailed
IIW (Garg et al., 2024) includes object-level annotations and image-level descriptions that capture spatial arrangements, interactions, and nuanced attributes. It contains 612 images with hyper-detailed descriptions, averaging 217.2 tokens per image.
Urban-Ik Ik (Zhang al., 2024a) consists of 1,000 urban images sourced from the Visual Genome dataset, paired with captions generated by GPT-4V. Each caption averages approximately 107 words and offers detailed descriptions, including object types, colors, and spatial relationships.
ShareGPT4v (Chen et al., 2023a) emphasize a wide range of visual information, including object properties, spa- tial relationships. aesthetic evaluations, and contextual knowledge. It contains 1.000 images with 1,000 detailed descriptions.
15

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
E. Details of Evaluated Models
Table 11 provides references for all the models used our paper.
Table Model links, The table is categorized based on model types, including VLMs, LLMs, LMMs, and text retrievers. "HF" is the
abbreviation of HageingFace.
Type
Model Name
Official Name
Model Link
SigLIP
SigLIP
EVA-CLIP
CVA-CLIP-18B
HE:BAAAVEVA-CLI--I
VLM
LoTLIP
LoTLIP
Intern Vidco2
Intern InternVidco2
Github:OOeenGllllb//mmmmWiiiiii
LLM
Qwen2.5-32B
Dwen2.5-32B-Insttee
LLaVA-7B
(ora-vl_6-mistrall
LLaVA-v1.6-34B
Ilava-v1.6344)
HFllubaoiia//llvv--1....33
LLaVA-Video-320
LLaVA-Ne
DIVA--eXT-Viido-3BBB
HF Saaus-llbbLLLaA--NeXX--eee-----
LMM
Qwen2-VL.-BB
Qwen2-VL-22
HI: 1
Qwen2-VL-7B
Qwen2-VL-7B-Instte
H VF---waaiOOwee2---V--------aa
Qwen2-VL-72B
Qwen2-VL-72B-Immtrrucl
HF-OyraiiOvaa22---222---nnnn
stella-435M
stella-en-400M-v55
IE-NonSSeacchkkllllee-----002222
gte-Qwen2-I.5B
See-Qween2-1.B--nnsttcc
bge-en-icl-7B
bge-en-icl
HF:BAAAl/hge-en-iccl
Text Retriever
reranker-568M
hge-reranker-v2-mtt
IIF:BAAAThgg--rraanee-----11
Alyerwise2.7BB
bge-reranker-v 2- chinicpm-layerwee
:B::BAA//gge AFFFBAAA//beeerraakerr reaeeerreanee--e/ v2- saniiieer----miippmmm layerwise
gemma2-98
bge-reranker-v22. -...mammm222-mmmm gemma2-lightweight lightweight
HF:BAAAlbge--eraan www.:/bge-rrraane---2e-2 v2.5 -25--gemma2 gemma2--ightweieee lightweight
F. Visualization Results
The visualization results of method presented in Fig. 3-5. These examples highlight how VISA enhances retrieval by refining visual abstraction through question-answering
In Fig. 3, our method generates questions related to clothing and an apron, refining fine-grained details. Specifically, in the top case, the general description incorrectly identifies black shirt, whereas the person is actually wearing a black jacket. The question-answering process corrects this mistake, improving the accuracy the visual abstraction. In the bottom case, our method correctly identifies blue shirt and an apron, ensuring better alignment with the user query.
In Fig. 4, the QA process successfully incorporates the countdown timer, aligning the retrieved content more closely with the user query.
In Fig. 5, the question generator identifies that "windows" in the query refers to the operating system and formulates relevant questions accordingly. Although the query includes "man" in the first video sample, does not contain the "Windows" concept. Intern Video fails to capture this, whereas VISA successfully recognizes and incorporates the missing context, leading more accurate retrieval. This improvement is attributed to the contextual understanding capabilities of the questioner LLM, which enables more precise alignment between the query and the retrieved content.
16

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Questions:
Text Query:
what is the man wearing on top?
A man in blue shirt and apron prepaers food.
2. What is the man doing in the image?
3. Is the man wearing apron while preparing food?
General Description:
the image. a man is standing in kitchen, engrossed in the of cooking, He is dressed in black shirt and is holding a weeden spoon, stiriing something in black frying pan that's placed on white stave. The stave situated on countertop, which is adorned with various
Answers:
1. The man is wearing a black jacket.
2.The man in the image is preparing food.
3. Uncertain
kitchen items. The countertop hub of activity, with white cutting hoard, the culinary scene. The cutting board and the howl are
bowl, and red pot adding
EVA-CLIP score:0.1935 VISA score:0.6729
empty, perhops waiting to used the next step of
cooking process. The red pot, the other hand, is filled with something, its contents hidden from view. Above the blackbourd hangs the wall, hears the inseription I'MISSING ITS MATE", perhaps indicating a ingredient. The overall secne
Confusable Candidate
EVA-CLIP rank:
VISA rank:5
scarch for missing suggests homey and comfortable kitchen environment, filled the warmth of home cooking.
Answers:
General Description:
1. The man is wearing blue shirt on top.
The image shows person in the process of preparing food. The individual uppears be wearing dark blue upron and is focused on the tusk at hand, They are using a knife cut into green fruit, which could be melon similar type of fruit, and there are pieces of the fruit already cut and
2. The man in the image is preparing food.
Specifically cutting or handing fish, as he is wcaring apron and appears be in kitchen or market setting.
placed weoden cutting board.
cutting
placed surface that looks like motal tray or pan, In the background, there are other people who seem to be engaged in similar food preparation activitics, supgesting that this could be a commercial kitchen or food stall. various
3. Yes, the man wearing apron while preparing food.
counter, including appears be a bowl and some other kitchen tools. The environmennt looks busy and focused on food preparation. The image candid quality, capturing moment in the daily work of the persen preparing the food. The lighting is natural, and overall stmossphere suggests bustling. active workspace.
EVA-CIIP sc85c:0.1857 VISA score:0.9291
Ground Truth
EVA-CLIP rank:2 VISA rank:1
Figure 3. A visualization example from Flickr30K with two candidate images.
17

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Text Query:
This image captures an urban street scene dominated by a traffic signal pole at the center. Affixed to the pole are various signs: digital crosswalk signal indicating red hand with countdown of
Questions:
1"31" seconds, a street name sign for ""2nd Avel" with green background and white lettering, and ""ONE WAY!" sign with a directional arrow pointing to the right. In the background, there's blue and yellow sign that advertises "BBreakfast All Day!" above a restaurant entrance. In the foreground, pedestrians appear be walking along the sidewalk, and a set of traffic lights is partially visible at the top right corner. The location looks like bustling downtown area with ample sunshine. in
1. What is the countdown number displayed on the digital crosswalk signal?
2. What is the name of the street indicated by the sign with
green background and white lettering? 3. What type of sign visible above restaurant entrance, and what does it advertise?
General Description:
Answers:
The image captures vibrant city sceee an intersection. Dominating the frame traffic light, its light glowing brightly, signaling vehicles
1. Uncertain
light
2. The name of the street indicated by the sign with green background and white lettering is F.D.R. Drive. Uncertain
affixed to sturdy metal pole, which also supports street sign. The sign, pointed uniking combination green and white, reads F.D.R. Drive"", indicating the name of the street. lust below the itaffic light, a blue and white marking
LAT
sign announess location of the interectionn. The signs are olbar and casily visible, guiding both pedestraass and drivers, In the adomed with a sign that reads '"Innovate", perhups indicating the the business the spirii the neighborhood. The building's presence
LoTLIP score:0.2713 VISA score:0.5778
background, huilding
Confusable Candidate
sense depts image. The image is stapshot of urban life, capturing the thythm the city its everyday lustle bustle.
LoTLIP rank:1 VISA rank:2
Answers:
The countdown number displayed digital crosswalk signal is 3.
General Description:
Breakfast All
The image captures vibeant city scene the comnrr of 2nd Avenne The street is bustling with life with people
2. The name of street indicated by the sign with a green background and white lettering is 2nd Avenue.
driving by: buildings
yellow and blue sign restaurant named ' B'rekkkaa Averne! prominenly displayed, indicating the location. A black and white "'nne Way!" is also visible, guiding the flow sky above clear suggesting sunny day. The image snapshoc of urban life. capturing the life variety husinesses, underly flow traffic. momeni frozen time, slice lnfe that's always
lining the street nr adorned with various signs, including
WAY
3. The visible sign above the restaurant entrance adverises that the breakfast is available all day.
Day!?, green street sign reading
LoTLIP score:0.2530
Ground Truth
hustle busile
VISA score:0.91
LoTLIP rank:2 VISA rank:1
Figure 4. A visualization example from Urban k two candidate images.
18

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
Questions:
Text Query:
1. What is the man demonstrating in the vidco?
man describing how to do something in windows.
2. In which software or operating system is the man giving instructions?
3. Is the man providing verbal instructions in the video?
in the out
General Description:
Answers:
video begins with person standing roum, wearing
V-neck shirt holding a blue
1.Uncertain
backpack. The bockground features map of the United States on the wall. a clock showing 10:10, and several
2. Uncertain
hats henging the left side, A sign on the wall reads belleve can. Thee per son hand gestures. The
explaining
something,
table filled with various neaily folded elothes, shoos, and other travel eosentalss The parsens is soen packing these items into the blue backpack. The video then returns to the instial scene with the person continuing esplain, still holding the backpack. The final frame shows collage everlaid indacating a passible theme
close
including
3. Yes, the man is providing verbal instructions in the video.
the cut urban explorers expluration. The continues same person standing in the same room, still wearing the dark bluz V-teck shirt and holding the blue backpuok. The background romains convistent with the map the United States, the cleck showing 10:10 and the hats also visible. person
related with
Intern Video2-G score:0.0200 VISA score:0.8534
Intern Video2-G rank: Confusable Candidate VISA rank:2
hanging the side. The sign something, msing hand gestures, simiher to the precious sornes. The scene maintains focus un the person and their actions, with signfficntt changes in the background or additional elements introduced.
theliese you
explain
General Deseription:
Answers:
The video begins with view a computer desitop festuring the Windows logo and various icons on the left side. The Remote Deskop Connection window upen, displaying the
The man in the video is demonstrating how to use the Remote
Desktop Connection feature in Windows.
Remone Desktop Connection dizlog box with fields for entering the name of the computer to
The types contoso'
'Copuuter field and clicks 'Comnett bution. A
2. The man is giving instructions in the Windows operating
new dialog appears, prompting the user to enter acsdestials. loonooul Adminiematoo the User name field and clicks the 'Eneer button. The user than enters
user eaters
system.
password in the Paaswoord field and clicks the OK' button. The user confinus the possward by button in Remote Desktop Connection' dinlog The scene transitions to a close up view of the 'Remote Desktop Connection' dialog bax, where the user has entered bootosoo Adminittato the "Uer name field and is about click the OK" button. The background shows Windows desktop with Windows logo and vanous clicks 'OK' button, and 'Remote Desktop Connection' window appears, showing a progress bar that fills up from 0% to 100% The user clicks the Show Options' button, revealing additional options such "Connect maother button, progress starts filling up again, indicating connection process vidco contines with a close-up view of the Remote Desktop Counection" window, showing the progress har fillng from 0% to 10%% The background remains the same, with the Windows? 7 deskiop Windows logo visible. The
3. Yes, the man providing verbal instructions in the video.
clicking
Intern Video2-G score:0,020 VISA 00099:0.9693
Ground Truth
desktop and 'Save The clicks the "onnzer"
Intern Video2-G rank:2 VISA rank:1
clicks the 'Show Options button, revealing
additional options such 'Commaci to another remote desking' and 'Svve Asi The user clicks the Connect button, and the peugress bar start filling again, indicating the coneecton process. The video concludes with the progress still filling up, suggestnng the conncction being established.
Figure 5, A visualization example from MSR-VTT with two candidate videos.
19

---

Visual Abstraction: A Plug-and-Play Approach for Text- Visual Retrieval
The
cars are
covered
in snow
This image shows snowy street scene
with several people and woman dressed red hat is The child she including blue snow suit and yellow skis A second
cars covered
snow
the foreground
in bright ski pants child in warm winter clothing hidden from view
jacket and
skiing carrying is
her back
carrying is
bundled
hat
and their face
child dressed
skiing nearby wearing snow off
walking with
dark jacket and poles white hat
The background shows
cleaning the
also snow along both sides of the street bare trees with snow on their branches line the street sky overcast with gray clouds quiet and cold
parked car
There
covered
cars lined
Tall
The
characteristic of and the generaktmosphere feels typical winter scene
Two
people are skiing on the street
This
image shows covered in snow
with several people and woman dressed red hat is The child she warm winter clothing and their face is hidden from view
snowy street scene In the foreground jacket and child on her back bundled up
cars
bright ski pants carrying carrying
skiing
including second The background shows is dark jacket and a white hat parked car
hat
child dressed in blue snow suit and yellow skis
skiing nearby
walking with ski poles
man wearing
cleaning the covered
snow off
cars lined up along both sides There are also snow the street their branches line the street
trees with snow sky overcast with gray clouds quiet and cold
bare
The feels
characteristic of and the generaktmosphere typical winter scene
The
green jacket hat and with several people and woman dressed red hat The child she including child dressed in blue snow suit and yellow skis A second
nomanskking is wearing
This image shows
snowy street scene foreground jacket and her back
cars covered bright skiing carrying is
snow
pants child in warm winter clothing
carrying is bundled
hat
and their face is hidden from view
skiing nearby man wearing
The background shows white hat also snow lined up along both sides of the street
walking with ski poles
dark jacket and There parked
cleaning the covered
snow off
cars
Tall The
bare
trees with snow on their branches line the street
sky
overcast with gray clouds
and the
generabtmosphere feels
quiet and cold
characteristic of
typical
winter
scene
Figure Visualization attention heatmaps Gemma2 (Chen et al., 2024a) given text queries and descriptions from Fig.
20


Reviews:
[{"review_id": "EDLa8cnNUI", "reviewer": "Reviewer_1FJx", "summary": "The paper studies the problem of text-to-visual retrieval, which involves both text-to-image retrieval and text-to-video retrieval. The authors propose a framework to enhance the retrieval via converting the visual content to the text domain, and then do the retrieval. Experiments show improvement of the proposed method over several previous models.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:  Efficiency Concerns (FLOPs and latency)**\n\nThank you for the valuable suggestions. We include the FLOPs and latency comparisions on the Flickr dataset below. For clarity, we divide the retrieval process into two stages:  \n\n- Offline stage precomputes visual features (via VLM) and generates general descriptions (via VISA) for the gallery candidates.\n- Online stage performs the text encoding (via VLM), QA-based description refinement and text-level re-ranking (both via VISA) during inference.\n\n| Type     | Module                               | GFLOPs     | Latency (second/dataset)       |\n| -------- | ------------------------------------ | ---------- | ------------------------------ |\n| offline  | VLM (SigLIP/EVA-CLIP)                | 335/4560   | 4/18                           |\n| offline  | General Description (LLaVA-v1.6-34B) | 138000     | 437.5                          |\n| **Type** | **Module**                           | **GFLOPs** | **Latency (second/per query)** |\n| online   | VLM (SigLIP/EVA-CLIP)                | 26/9       | 0.0002/0.0029                  |\n| online   | Question Generator (Qwen2.5-32B)     | 26870      | 0.02                           |\n| online   | Answer Generator (Qwen2VL-7B)        | 15450      | 1.00                           |\n| online   | Text Retriever (gemma-9B)            | 4160       | 0.13                           |\n\nIn the offline stage, generating general descriptions using large multimodal models (LMMs) indeed incurs high FLOPs (138,000 vs. 335 compared with SigLIP, approximately 400×). However, this process could be executed ahead of time and does not influence the real-time inference latency. In the online stage, inference overhead is significantly reduced through techniques like KV cache and parallel execution, supported by efficient frameworks such as SGLang. In practice, this results in only ~1 second of additional latency on an NVIDIA 4090 server compared to the original SigLIP pipeline.\n\nWe acknowledge that VISA introduces a trade-off between efficiency and performance, and it may not be ideal for strict low-latency applications. However, we would like to emphasize two key points:\n\n- Test-Time Augmentation Paradigm. To our knowledge, VISA might be the first work to systematically explore **test-time augmentation** for VLM-based retrieval. This paradigm aligns with trends in LLM research (e.g., test-time scaling), where huge additional compute yields significant performance gains. Specifically, VISA achieves average R@1 improvements of **+2.3% (video)** and **+6.7% (long-context image retrieval)**. We also include BLIP-2 comparisons below, showing VISA improves over BLIP-2 by **+0.6% (COCO)** and **+0.5% (Flickr30K)** in R@1.\n- Flexible Trade-Offs. Even using only the General Description module (without QA) yields strong performance boosts of **+2.5%, +1.3%, and +8.3%** R@1 across three tasks (see Table 6(a)), offering a **lighter-weight deployment option** (0.13 second) when needed.\n\n> **Q2: Comparison to BLIP-2**\n\nThanks for highlighting this important baseline. We initially did not compare against BLIP-2 because our evaluation follows the widely adopted zero-shot retrieval protocol in recent VLM research. In contrast, **BLIP-2 is finetuned on COCO and then evaluated on COCO and Flickr**, making it less directly comparable to the evaluation setup in Table 1.\n\nHere we include BLIP-2 in our evaluation (referencing Table 5 from the [BLIP-2 paper](https://arxiv.org/pdf/2301.12597)). As shown below, VISA continues to yield improvements when applied on top of BLIP-2. This demonstrates the compatibility and effectiveness of our method even when applied to strong VLMs like BLIP-2.\n\n| Method                | COCO (R@1｜R@5｜R@10) | Flickr (R@1｜R@5｜R@10) |\n| --------------------- | --------------------- | ----------------------- |\n| BLIP                  | 65.1｜86.3｜91.8      | 86.7｜97.3｜98.7        |\n| BLIP-2                | 68.3｜87.7｜92.6      | 89.7｜98.1｜98.9        |\n| BLIP-2 + VISA（Ours） | 68.9｜88.0｜92.9      | 90.2｜98.4｜99.2        |\n\nMore broadly, while BLIP-2 is highly competitive for short-query retrieval tasks such as COCO and Flickr30K, our method demonstrates notable improvements on long-text queries and video retrieval tasks as well, across **10 widely-used datasets**. This highlights the generality of VISA as a plug-and-play enhancement applicable to a range of backbone models.", "claims_and_evidence": "The authors claim they achieve state-of-the-art performance, while their numbers for text-to-image retrieval is lower than the actual state-of-the-art BLIP-2.", "methods_and_evaluation_criteria": "The proposed method uses LLM for better text-to-image retrieval. One biggest advantage of using CLIP/SigLIP for text-to-image retrieval is that it is efficient. But with LLM introduced and runned for multiple times as in the proposed method, the FLOPS may be significant increased so it is against the motivation of developing an efficient retrieval system.\n\nAlso, at the evaluation side, the authors are not comparing with state-of-the-art architectures, such as BLIP-2.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The authors are not comparing with state-of-the-art architectures, such as BLIP-2, while they claim they achieve the state-of-the-art results. It is a significant problem of doing this. Actually their numbers for text-to-image retrieval is lower than BLIP-2.", "relation_to_broader_scientific_literature": "Text-to-visual retrieval is an important task, and effective innovations for the task will be of a wide interest. But my concern is the authors have not honestly compared with the well-known state-of-the-art models while making such a claim. Besides, they have not considered about the efficiency of retrieval, which is very important factor in the community.", "essential_references_not_discussed": "The authors have not properly compared their methods with state-of-the-art BLIP-2.", "other_strengths_and_weaknesses": "I think the major weakness of the paper is the evaluation and method design.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Is it possible to include the FLOPS comparison of the proposed method and the original model?\n2. Is it possible to include BLIP-2 results for text-to-image retrieval and compare with it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9yfXcm0QNN", "reviewer": "Reviewer_1ri6", "summary": "This paper proposes Visual Abstraction (VISA), a plug-and-play approach designed to enhance text-to-visual retrieval. Unlike traditional retrieval methods that operate in a cross-modal embedding space, VISA transforms visual content into textual descriptions using off-the-shelf large models. This transformation filters out redundant low-level visual details. Additionally, VISA incorporates a question-answering mechanism to refine descriptions based on user-specified granularity. Extensive experiments demonstrate that VISA significantly improves retrieval performance across text-to-image and text-to-video tasks, outperforming state-of-the-art models on both short- and long-context queries. The approach requires no additional training and can be seamlessly integrated into existing retrieval systems.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:Efficiency Concerns**\n\nThanks for the suggestion regarding the efficiency of using large text retrievers. Importantly, VISA does not require LLMs with over 7B parameters to work effectively. To demonstrate this, we conducted experiments using smaller text retrievers with **400M and 1.5B parameters**, and the results are summarized below:\n\n| Text Retriever | Flickr30K **(R@1｜R@5)** | MSR-VTT **(R@1｜R@5)** | Urban1k **(R@1｜R@5)** |\n| -------------- | ------------------------ | ---------------------- | ---------------------- |\n| None           | 83.1｜95.8               | 52.0｜74.6             | 85.9｜97.1             |\n| stella-400M    | 85.2｜97.2               | 53.4｜75.0             | 93.3｜98.9             |\n| stella-1.5B    | 85.4｜97.1               | 53.6｜75.4             | 93.4 \\| 98.9           |\n| gemma2-9B      | 86.1｜97.3               | 54.4｜75.3             | 94.6｜99.4             |\n\nAs shown, **even relatively small models such as Stella-400M yield significant performance gains** over the baseline without a text retriever. This demonstrates that the performance gains from VISA are not solely due to model size, but rather arise from the integration of visual abstraction into the retrieval pipeline. These findings reinforce the generalizability and flexibility of VISA. In scenarios where inference efficiency is a priority, smaller and more efficient models can be adopted without substantial performance loss. For additional discussion on latency and FLOPs, please refer to our response to **Q1 of Reviewer 1FJx**.\n\n> **Q2:Impact of abstraction length** \n\nWe explored how varying the length of the generated general descriptions affects retrieval performance. Specifically, we modified the prompt for the captioning model to: *“Please generate descriptions of the given image in approximately {num} words.”*  The corresponding results are available at the anonymous URL: **[https://imgur.com/a/amTI9n5](https://imgur.com/a/amTI9n5)**.\n\nOur results show that **longer visual captions generally improve retrieval performance**, as they encode richer semantic information such as object attributes, spatial relationships, and contextual details. However, we also observe a clear **performance saturation** beyond a certain caption length. That is, once the caption adequately captures the essential visual content, adding more tokens results in **diminishing returns**. Importantly, the optimal caption length depends on the **complexity of the query.** For short-context queries (e.g., Flickr30K), moderately long captions are sufficient to achieve strong performance. In contrast, long-context queries(e.g., Urban1K) benefit from longer, more detailed captions that better capture fine-grained visual elements.\n\n> **Q3:Using VISA on other cross-modal tasks**\n\nWe believe that the **visual abstraction mechanism** is broadly applicable to other cross-modal tasks. For example:\n\n- Video Moment Retrieval: By segmenting a video into temporal clips and generating visual abstractions for each segment, one can match a textual query to the semantic descriptions of these segments, thereby localizing relevant time intervals.\n\n- Text-based Re-identification and Composed Image Retrieval (CIR): This can benefit from generating abstractions for both **gallery items** and **query items** (such as image queries in CIR) and comparing them in the language space.\n\nWe plan to include this discussion in the next version.\n\n> **Q4:Image-to-text (I2T) retrieval**\n\nWe initially did not include text-to-image (T2I) experiments because our primary focus is on supporting diverse user retrieval demands with varying levels of query granularity—a scenario more naturally aligned with T2I settings.\n\nTo address this concern, we conducted image-to-text (I2T) retrieval experiments on Flickr30K, MSR-VTT, and Urban1K, using the same retrieval setup and backbones as in the T2I setting. Specifically, for each query image, we first retrieve the top-20 candidate text descriptions. For each candidate, we generate three questions and append the corresponding answers derived from the image. Finally, we re-rank the candidates using text-level retrieval.\n\n| Method                | Flikcr (R@1｜R@5｜R@10)      |\n| --------------------- | ---------------------------- |\n| SigLIP                | 94.4｜99.7｜99.8             |\n| SigLIP + VISA         | 95.0｜99.8｜100.0            |\n| BLIP-2                | 97.6｜100.0｜100.0           |\n| BLIP-2 + VISA         | 97.9｜100.0｜100.0           |\n|                       | **MSR-VTT (R@1｜R@5｜R@10)** |\n| InternVideo2-G        | 49.6｜73.4｜81.0             |\n| InternVideo2-G + VISA | 54.4｜78.0｜84.8             |\n|                       | **Urban1k (R@1｜R@5｜R@10**) |\n| LoTLIP                | 89.6｜97.8｜98.9             |\n| LoTLIP + VISA         | 93.7｜99.0｜99.5             |\n\nThese consistent improvements across datasets and backbones confirm that VISA not only enhances T2I retrieval, but also generalizes effectively to I2T retrieval.", "claims_and_evidence": "Claim 1: VISA enhances retrieval by converting visual content into text, filtering out redundant details, and improving alignment with textual queries.\n\nEvidence: Experiments show that VISA improves recall@1 across multiple datasets, demonstrating better retrieval performance.\n\nClaim 2: The question-answering process helps refine descriptions, ensuring a more precise match to user queries.\n Evidence: Ablation studies indicate that removing the QA process reduces retrieval accuracy, confirming its contribution to performance.", "methods_and_evaluation_criteria": "After carefully checking the manuscript, the proposed methods and evaluation criteria make sense for text-to-visual retrieval.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "Experimental evaluations are comprehensive.", "relation_to_broader_scientific_literature": "VISA relates to prior vision-language pretraining models such as CLIP, EVA-CLIP, and FLAME, but differs by transforming retrieval into a text-enhance problem", "essential_references_not_discussed": "The paper sufficiently covers relevant references.", "other_strengths_and_weaknesses": "strengths:\n1.  VISA introduces a plug-and-play test-time retrieval enhancement, avoiding the need for expensive retraining.\n2.  VISA is model-agnostic and can enhance retrieval performance without modifying existing models.\n3.  Experimental results show consistent recall@1,5,10 gains across multiple datasets.\n\nweaknesses:\n1. Although VISA is plug-and-play, text-based retrieval process introduces additional computational overhead. Does text retrieval necessarily require using LLMs with over 7B parameters? This parameter scale is significantly larger than most vision-language models (VLMs), which may raise concerns about efficiency and deployment feasibility.\n2. What is the impact of abstraction length on retrieval performance? Does generating longer textual descriptions always improve accuracy, or is there an optimal level of abstraction that balances efficiency and effectiveness?\n3. Can VISA be extended beyond text-to-visual retrieval? Could this approach be generalized to other cross-modal tasks while maintaining its plug-and-play nature?\n4. The paper lacks image-to-text (I2T) retrieval experiments. While the work focuses on text-to-image retrieval from a user-driven perspective, cross-modal retrieval generally includes both T2I and I2T tasks. It would be valuable to see how VISA performs in the reverse retrieval direction.", "other_comments_or_suggestions": "The paper lacks citations for InternVideo in Table 4. Please ensure that all referenced methods are properly cited to maintain clarity.", "questions_for_authors": "Please refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}, {"review_id": "Uvu5XL4SSm", "reviewer": "Reviewer_dHkt", "summary": "This  paper introduces Visual Abstraction (VISA), a novel test-time approach for enhancing text-to-visual retrieval by converting visual content into textual descriptions using large pre-trained models. VISA utilizes a question-answering mechanism to refine these descriptions to match the granularity of user queries accurately. The approach demonstrates superior performance over existing methods across several benchmark datasets for both image and video retrieval tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:Integrating VLMs**\n\nThank you for this insightful suggestion. To explore this idea, we integrate SigLIP (as the base model) with EVA-CLIP (18B parameters), which has a significantly higher parameter count compared to the text retrieval model gemma-9B. This hybrid setup yields notable performance gains over SigLIP alone:\n\n| Method             | COCO (R@1｜R@5｜R@10) | Flickr (R@1｜R@5｜R@10) |\n| ------------------ | --------------------- | ----------------------- |\n| SigLIP             | 54.2｜76.8｜84.2      | 83.0｜96.1｜98.0        |\n| SigLIP+EVA-CLIP    | 56.0｜79.1｜85.9      | 84.1｜96.7｜98.3        |\n| SigLIP+VISA (Ours) | 57.2｜80.3｜86.9      | 85.1｜97.1｜98.6        |\n\nNotably, our proposed method achieves even greater improvements on both datasets. This suggests that generative visual abstraction offers more effective enhancement than simply fusing large VLMs. Importantly, VISA maintains its plug-and-play design, requiring no model retraining or architecture modification. Additionally, as detailed in our response to **Reviewer 1ri6 (Q1)**, our method is compatible with smaller, more efficient text retrievers (e.g., 400M and 1.5B parameters), which also achieves strong performance.\n\n\n> **Q2:The explanation of 'No' and 'Uncertain' responses**\n\nOur core motivation is to abstract visual content into textual descriptions, enabling more effective reasoning and relationship modeling in the text space. In the question-answering module, we distinguish between two types of negative responses: \"No\" and \"Uncertain\", each serving a different purpose:\n\n- **\"No\"** is used when the question refers to a subject that is **not present in the image**, or when a specific condition, action, or attribute **does not apply**. For example, if the image shows a person in red shirt playing basketball and the question asks, \"Is the person in red shirt playing football?\", the correct response would be \"No\". This type of response provides **explicit negative evidence** which helps refine semantic alignment.\n- **\"Uncertain\"** is used when the **question cannot be reliably answered** based on the visual content, e.g., asking about the shirt color of someone playing football when no such activity is depicted. In these cases, the response is considered **uninformative**, and the corresponding QA pair is **discarded** from downstream processing.\n\n> **Q3:How VISA manages ambiguous queries where multiple visual interpretations are possible.**\n\nVISA incorporates LLMs to manage query ambiguities by leveraging **context** to disambiguate meanings. For example, in a query like \"How to do something in windows?\" (as shown in Figure 5 of the main paper), the surrounding context helps the model infer that \"windows\" refers to the **operating system Windows** instead of **glass windows**. In the future, we plan to further enhance this capability with reflection mechanism to check the visual interpretations.", "claims_and_evidence": "The paper's claims about the effectiveness in improving text-to-visual retrieval are well-supported by extensive experimental results. The authors present a comprehensive set of experiments and comparisons that demonstrate the approach's superiority over existing state-of-the-art methods across various datasets and query types.", "methods_and_evaluation_criteria": "The proposed methods are logically sound and well-aligned with the problems at hand. The evaluation is conducted using standard benchmark datasets like COCO, Flickr30K, and others, which are appropriate for the task. The use of large pre-trained models and a novel question-answering refinement process adds to the methodological robustness.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental evaluations are thorough, with detailed ablation studies and comparisons to baseline and state-of-the-art models. The authors effectively demonstrate the benefits of their approach using both quantitative metrics and qualitative examples.", "relation_to_broader_scientific_literature": "The paper is related to vision-language pre-training and retrieval tasks.", "essential_references_not_discussed": "I do not identify any critical references or related works that are missing from the discussion.", "other_strengths_and_weaknesses": "This paper introduces a unique test-time enhancement for text-to-visual retrieval by converting visual content into textual descriptions. This method diverges from traditional training-time modifications, offering a fresh perspective on improving retrieval effectiveness. The plug-and-play nature of VISA allows it to be seamlessly integrated with existing visual-language models. This compatibility is a significant advantage for enhancing the recommended system without extensive redevelopment. \n\nHowever, the paper has several limitations:\n\n1. The paper primarily focuses on integrating text-based retrieval enhancements into existing cross-modal frameworks but does not explore the potential of integrating two multimodal models.\n2. The differences between 'No' and 'Uncertain' responses in the QA process are not well-explained. It is unclear whether these responses are treated differently in the retrieval ranking process.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. This work integrates a text model into existing cross-modal frameworks. Could you explore the performance outcomes of integrating two multimodal models (VLMs) of similar parameter scales?\n2. In the question-answering process, how does the system distinguish between definitive 'No' and 'Uncertain' responses? What implications do these responses have on the retrieval process?\n3. How does VISA handle queries that could be interpreted in multiple valid ways visually? Are there specific mechanisms or algorithms within VISA that help resolve or manage such ambiguities? The paper does not fully address how VISA manages ambiguous queries where multiple visual interpretations are possible.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTUHeZMo1q", "reviewer": "Reviewer_nqY2", "summary": "The paper introduces a test‐time, plug‐and‐play approach VISA for text-to-visual retrieval. VISA converts visual content into dense, natural language descriptions using off‐the-shelf large multimodal models (LMMs). It then refines these descriptions via a question-answering module that leverages chain-of-thought prompting to capture query-specific details. The approach addresses two main challenges: (1) filtering out low-level, redundant visual details and (2) addressing granularity mismatches between textual queries and visual content. Extensive experiments on both image (e.g., MS-COCO, Urban1k) and video (e.g., MSR-VTT, LSMDC) datasets demonstrate that VISA improves retrieval performance over several state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:Concern of computational latency**\n\nWe would like to clarify that our method is **compatible with smaller, more efficient retrievers**, and we have observed that models with significantly fewer parameters (e.g., 400M or 1.5B) still achieve **comparable performance** (see our response to Reviewer 1ri6, Q1).\n\nIn addition, the **general description module** can be executed entirely **offline**, prior to deployment. This design ensures that it incurs **no cost during online inference**. As noted in our response to Reviewer 1FJx (Q1), even when the **QA component is omitted**, VISA still delivers **competitive performance** relative to existing VLMs. In such case, only the **text retriever** operates in real-time, and its computational overhead can be adjusted by choosing a model of a suitable size.\n\nTogether, the use of lightweight retrievers and offline gallery processing make VISA practically deployable and efficient, even in resource-constrained settings.\n\n> **Q2:Image-to-text (I2T) retrieval experiments**\n\nWe initially did not include text-to-image (T2I) experiments because our primary focus is on supporting diverse user retrieval demands with varying levels of query granularity—a scenario more naturally aligned with T2I settings.\n\nTo address this concern, we conducted image-to-text (I2T) retrieval experiments on Flickr30K, MSR-VTT, and Urban1K, using the same retrieval setup and backbones as in the T2I setting. Specifically, for each query image, we first retrieve the top-20 candidate text descriptions. For each candidate, we generate three questions and append the corresponding answers derived from the image. Finally, we re-rank the candidates using text-level retrieval.\n\n| Method                | Flikcr (R@1｜R@5｜R@10)      |\n| --------------------- | ---------------------------- |\n| SigLIP                | 94.4｜99.7｜99.8             |\n| SigLIP + VISA         | 95.0｜99.8｜100.0            |\n| BLIP-2                | 97.6｜100.0｜100.0           |\n| BLIP-2 + VISA         | 97.9｜100.0｜100.0           |\n|                       | **MSR-VTT (R@1｜R@5｜R@10)** |\n| InternVideo2-G        | 49.6｜73.4｜81.0             |\n| InternVideo2-G + VISA | 54.4｜78.0｜84.8             |\n|                       | **Urban1k (R@1｜R@5｜R@10**) |\n| LoTLIP                | 89.6｜97.8｜98.9             |\n| LoTLIP + VISA         | 93.7｜99.0｜99.5             |\n\nThese consistent improvements across datasets and backbones confirm that VISA not only enhances T2I retrieval, but also generalizes effectively to I2T retrieval.\n\n> **Q3:Compared with multimodal retrieval methods that use LMMs**\n\nAs there are only a few works directly built on LMMs, we have compared our method with RAGVL in Table 1. On Flickr30k, our approach achieves higher performance with R@1 = 85.1 compared to RAGVL’s 84.4 and further improves to 86.1 when using EVA-CLIP. The plug-and-play design does not require end-to-end fine-tuning of large models.", "claims_and_evidence": "The paper claims that converting images and videos into rich textual representations can boost retrieval performance. This claim is supported by comprehensive experimental results, e.g., consistent improvements in recall metrics across diverse datasets and detailed ablation studies that highlight the contribution of each component (general description, QA refinement, chain-of-thought). Overall, the evidence is clear and convincing.", "methods_and_evaluation_criteria": "The proposed method is well-motivated and leverages the strengths of existing LMMs and LLMs without requiring additional training. The experimental design is thorough, employing both short-context and long-context retrieval tasks with appropriate benchmarks and evaluation metrics. This makes the method not only conceptually sound but also practical for real-world applications.", "theoretical_claims": "There are no theoretical claims in this work, which is consistent with the paper’s focus on a practical, empirically driven solution.", "experimental_designs_or_analyses": "The authors compare VISA against multiple baselines on a range of datasets and perform extensive ablation studies that demonstrate the robustness and effectiveness of the proposed components. The analysis including comparisons of inference time provides a well-rounded view of the method.", "relation_to_broader_scientific_literature": "The paper discusses the current literature on vision-language models, such as CLIP, Frozen, and recent works on fine-grained retrieval enhancement. It builds on prior findings by addressing the inherent limitations of training-based approaches and provides a novel test-time alternative.", "essential_references_not_discussed": "While the reference list appears comprehensive, a discussion of some very recent works on multimodal retrieval methods that use LMMs might further strengthen the literature context. However, no critical prior work seems to be omitted.", "other_strengths_and_weaknesses": "*Strengths:*\n\n– The paper presents an innovative plug-and-play framework that sidesteps the need for expensive retraining. It integrates with existing vision-language models without additional training and leverages pre-trained models at test time, significantly reducing computational costs.\n\n– The paper features extensive experimental validation and rigorous ablation studies. The method is evaluated on multiple benchmark datasets, and the ablation studies clearly demonstrate the contribution of each module.\n\n– The method achieves clear improvements over established baselines across diverse datasets. Empirical results show consistent gains in key metrics on both image and video retrieval tasks, highlighting its effectiveness in addressing semantic redundancy and granularity mismatches.\n\n*Weaknesses:*\n\n– The reliance on large off-the-shelf models at test time could raise practical deployment concerns regarding computational latency.\n\n– The paper lacks experiments evaluating the method from an image-to-text perspective. Including standard I2T benchmarks would provide additional insights into the robustness and generality of the approach.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. How about your method compared with multimodal retrieval methods that use LMMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel method named Visual Abstraction (VISA), which is a plug-and-play approach designed to enhance text-to-visual retrieval. In particular, this approach transforms visual content into textual descriptions using off-the-shelf large models. Then, the generated text descriptions with their dense semantics would naturally filter out low-level redundant visual information. Finally, this approach adopts a question-answering process to enhancing the text description with the specific granularity information requested by the user. 

Reviewers agreed that this paper presents a novel approach for test-time text-visual retrieval. The paper is well written and easy to follow. Reviewers raised some questions about baselines, image-to-text experiments, and efficiency. Most of them have been well addressed by the detailed responses from the authors. One reviewer still concerned on the efficiency and the comparison with BLIP-2. However, efficiency is not claimed as a contribution of this work, and there is often a trade-off between inference-time cost and performance. 

The authors are strongly encouraged to incorporate the new results and discussions to their camera-ready version. For instance, the authors may consider adding the results of BLIP-2 to the main table in shaded gray (similar to the InternVL paper that also points out the improved performance of BLIP-2 on Flickr30K due to finetuning on COCO).
Decision: Accept (poster)