Efficient Discriminative Joint Encoders for Large Scale Vision-Language Reranking
Authors: Mitchell Keren Taraday
Shahaf Wagner
Chaim Baskin
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Retrieval
Vision–Language Models
Joint Encoding
Efficient Re-ranking
Token Compression
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose EDJE, an efficient vision–language joint encoder with token-compression that enables fast multimodal re-ranking, achieving up to 53× higher throughput while matching the accuracy of prior joint encoders.

Abstract: Multimodal retrieval still leans on embedding-based models like CLIP for fast
vector search over pre-computed image embeddings. Yet, unlike text retrieval
where joint-encoder rerankers are standard, comparable vision–language rerankers
are largely absent. We find that seminal joint encoders such as BLIP are severely
bottlenecked by an expensive visual feature-extraction stage, preventing practical deployment at scale.
Motivated by this bottleneck, we introduce EDJE , an
Efficient Discriminative Joint Encoder that precomputes vision tokens offline and
compresses them via a lightweight attention-based adapter, so online inference runs
only a compact joint encoder over a small set of visual tokens plus the text. EDJE
preserves strong retrieval performance while drastically reducing storage and online
compute, enabling high-throughput inference. Specifically, EDJE processes 50k
image–text pairs/second while requiring 49kB of disk storage per image, matching
prior art on Flickr (zero-shot) and COCO (fine-tuned) retrieval.

Published as a conference paper at ICLR 2026
EFFICIENT DISCRIMINATIVE JOINT ENCODERS FOR LARGE SCALE VISION-LANUUAGE RE-RANKING
Mitchell Keren Taraday Shahaf Wagner' Chaim Baskin'
INSIGHT Lab, Ben-Gurion University of the Negev, Israel
ABSTRACT
Multimodal retrieval still leans on embedding- based models like CLIP for fast vec- search over pre-computed image embeddings. Yet, unlike text retrieval where joint-encoder re-rankers are standard, comparable vision-language re-rankers are largely absent. We find that seminal joint encoders such as BLIP are severely bottlenecked by an expensive visual reature-extractio cal deployment scale. Motivated by this bottleneck, we introduce EDJE, an Efficient Diseriminative Joint Encoder that precomputes vision tokens offline and compresses them via lightweight attention-basee adapter, so online inference runs only compact joint encoder over small set of visual tokens plus the text. EDJE preserves strong retrieval performance while drastically reducing storage and online compute, enabling tigh-throughput inference. Specifically, EDJE processes 50k mmate-text pairs/second while requiring 49kB of disk storage per p image, matching prior art Flickr (zero-shot) and COCO (fine-tuned) retrieval. |
stage, preventing practi-
INTRODUCTION
Large-scale multimodal retrieval finding the most relevant images for text query, retrieving descriptive text given image is fundamental challenge in Aasion-language modeling. Its importance spans a wide range of applications, including web-scale image search, multimodal dataset curation, content moderation, and serrivaal-augmente generation. Because such applications often involve searching across millions of candidates, retrieval systems must be both efficient and accurate.
A major breakthrough came with the emergence of models that align visual and textual modalities within a shared embedding space, such as CLIP RRaffrdd et al. (202T) By enabling efficient similarity search through simple vector comparisons, these models made content-based large-scale retrieval feasible. Beyond retrieval, they have also shown strong generalization to tasks such as zero-shot
"qqual contribution.
'The paper's website, including code, is publicly available at hthttpss/Shhhhhiiaaaiuubbbbbbbbb
Inference part:
Joint encoding
EDJE (Ours)
Flickr image Retrieval R@1
ALBEF
VIT-B
90
RUP-2 VITs
Method
BUP
VIT-B
88
BOU NTTLL
x53
RJP
a6
BUP
RUP WT
84
BUP-2
VIT-L
250
500
750
1000
1250
1500
5
30
20
50
100
Inference Time (ms)
Joint Encoding Inference Time (ms)
(a) Inference time breakdown
(b) Retrieval encoding time
Figure 1: Inference efficiency and retrieval performance. (a) Methods with strong discriminative capabilities are dominated by costly ViT feature extraction, prohibiting their practical use for re-ranking. (b) EDJE achieves competitive zcro-shot retrieval performance with up 53x faster inference. Its token compression makes storing visual features practical, enabling large-scale retrieval.

---

Published as a conference paper at ICLR 2026
image classification, inspiring rapid improvements of this paradigm (Wang et al. 2022 Zhai et al. 2023 Cherti et al. 2023 Fang et al. 2023a T'scaanen et al. 2025).
In parallel, the remarkable success of large language models (LLMs) has driven efforts to integrate vision, enabling cesruuctionfollowinng over multimodal inputs. Early approaches (Chen ct al. 2020
Li al. 2021 2022 2023) aimed to build foundation vision-language models (VLMs) capable of both generative and discriminative tasks. However, the research community has shifted its interest towards generative-only models, typically by coupling a vision encoder with a pre-trained LLM (Liu et al. 2023] 2024; Wang et al. [2024] Gemma 2025). This shift has effectively divided the research community into two main directions: (1) advancing embedding- models for vision- language alignment, and (2) improving text generation over multimodal inputs - leaving the discriminative potential of joint encoders largely underexplored.
Unlike embedding- models. joint encoders process both modalities together, allowing richer cross-modal interactions. Prior work (Li et al., 2021 2022 2023) has shown that such models can significantly improve cross-modal retrieval performance by re-ranking the top-k candidates retrieved by an embedding model. However, their adoption in practical retrieval pipelines has remained limited; each candidate pair must be evaluated independently, and existing architectures are slow. In particular, these models rely on heavy, tigh-resolution CLIP-style vision backbones to extract highly expressive image features that poses severe efficiency bottleneck (Figure Ta). This raises a central question:
Can we harness the benefits of joint modeling while achieving the efficiency required for large-scale retrieval?
To this end, we introduce EDJE, an efficient discriminative joint encoder that allows fine-grained cross-modal interactions without requiring online visual feature extraction. The core idea is to shift visual feature extraction offline: images are encoded once and stored on disk; at query time a compact encoder-only language model jointly processes these with text tokens to produce re-ranking score. We further improve scalability by introducing lightweight token-comppe compressio adapter that reduces the number of cached vision tokens. Instead of storing the full sequence produced by the vision backbone, the adapter utilizes a small set of learnable queries that aggregates the most relevant information through gross-attentin and projects them to the embedding space of the joint encoder. This compressed representation substantially lowers storage requirements and decreases the number of tokens the joint encoder must process at query time.
Empirically, EDJE consistently improves zero-shot retrieval when paired with a variety of embedding- based models, spanning multiple visual backbones and input resolutions. This demonstrates its modularity as a drop-in re-ranker that can enhance retrieval pipelines regardless of the underlying embedding model. Moreover, when equipped with a strong visual backbone such as SigLIP2 (Tschannen et al. 2025), EDJE surpasses or matches the retrieval performance of prior joint encoders on standard benchmarks (Flickr30k zero-shot; MS-COCO fine-tuned) (Plummer et al.) 2015, Lin et al. 2014), while operating with substantially greater efficiency (Figure lbp. Finally, we evaluate the robustness of EDJE under compression, quantifying the trade-off between retrieval performance and storage cost as the number of compressed tokens is reduced, and conducting further ablations on re-ranking pool size and training objectives.
Contributions. In this work, we address the challenge of bringing the benefits of joint vision- language modeling to large-scale retrieval while maintaining efficiency. Our main contributions are as follows:
1. We introduce EDJE, an efficient discriminative joint encoder that performs fine-grained cross-modal re-ranking while shifting heavy vision precomputation offline. We further propose a lightweight cokpressomeesiionn adapter that condenses vision features into a compact representation, substantially reducing storage and computation.
2. Empirically, EDJE demonstrates consistent gains over a variety of umbedding-basee models. With strong visual backbone, EDJE achieves performance competitive with sfate-of-the-an joint encoders on standard benchmarks while operating with substantially greater efficiency.
3. We conduct comprehensive analyses of scalability and robustness, quantifying trade-offs between retrieval performance, storage costs, re-ranking pool size, and training objectives.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
The success of CLIP (Radford et al. 2021 and ALIGN (Jia et al. 2021) in aligning vision and language modalities within shared embedding space marked breakthrough in vision-language modeling. By scaling contrastive learning to large architectures and massive tmaae-ccption datasets, these models enabled efficient vector similarity search and inspired abundance of follow-up work.
Subsequent research has been directed towards reproducing and extending this paradigm in several directions. For example, LA1ON-400M (Schuhmann et al. 2021 released an open dataset of paired image-caption training data. Other efforts scaled model size and data (Xu et al. 2024] Fang et al. 2023b), filtered noisy captions (Fang et al., 2023a Gadre et al. 2023), or generated synthetic ones (Li
2023 kokitsi Maninis et al. 2025). Additional work explored alternative loss functions (Zhai et al.
al. 2022 Nguyen al.
2023) or auxiliary objectives to enrich localization (Naeem et al., 2024
kokitsi Maninis et al. 2025) and language generation capabilities (Wan et al. [2024] Tschannen et al.
2025). Despite their efficiency and scalability, embedding- baadd approaches compress modalities independently (late interaction), limiting fine-grained cross-modal interactions.
Parallel to contrastive approaches, researchers have pursued models that process modalities jointly. Early systems such LXMERT and UNITER (Tan and Bansal [2099] Chen et al., 2020) relied on region features from R-CNN (Ren et al., 2015) combined with text embeddings. LightningDOT (Sun 202T relies on these methods to perform re-ranking with pre-computed region-level representations to enable feasible storage. However, because each region is collapsed into a single vector, such re-ranker behaves much closer an embedding model rather than a true joint encoder that sees the full image. In practice, this leads to performance that now lags behind modern embedding models such as SigI.IP2 (T'schannen et al. 2025).
The emergence of vision transformers made combining vision and language modalities more straight- forward as both modalities are represented as a sequence of tokens. Consequently, some works aimed at creating transformer models capable of processing both images and texts jointly (Wang) jet al, 202T; 2022). Such models require heavy pre-training by masking either text or both text and image tokens. Another line of work introduced cross-atee attention architectures such ALBER, BLIP, BLIP-2, and CoCa (Li et al. [021]] [222] 2023) Yu et al.) 2022). which fuse pretrained vision encoders and language models through cross-attention layers. These joint encoders not only unify discriminative and generative modeling, but also consistently outperform emhedding- based models on discriminative tasks. In particular, multimodal retrieval performance can be significantly enhanced
by re-ranking embedding- based results with a joint encoder (Li et al. 2021 2022, (023), echoing
common practices in text retrieval where cross-encoders are widely used (Chen et 2024 Zhang
et al. 2024).
more recent trend is the integration of pretrained vision encoders into large language models (LLMs), yielding generative Assion-languee models (VLMs). Methods such as LLaVA (Liu et al.)
2023) introduce a lightweight projection that maps vision tokens into the L.LM embedding space,
Cross Attention Models
Fully Joint Models
Large Language Models
Croes Afteemon
Muttmedal Transformer
large Larguage Midil
Velen Eicodrr
Salf Altindon
Viatn Encoder
"Sclentst lng yollow samales"
"Sc endist handling weslligyllwooomooo
Ing
Figure 2: Taxonomy of lasion-l joint encoders. Left: Cross-attention models integrate modalities through cross-attention layers interleaved with textual self-attention (Li et al. 2021 2022 2023). Middle: Joint foundation models such as BEÎT-3 (Wang et al.) (022) employ unified self- attention over native visual and textual tokens, enabling full cross-modal interaction. Right: Modern generative VLMs (Liu et al. 2023) combine a pretrained vision encoder with a large language model, tuning the latter to process projected vision tokens if they originated from text.
3

---

Published as a conference paper at ICLR 2026
followed by fine-tuning curated captioning datasets. Variants extend this with parameter-efficien fine-tuning (Hu et al. 2022) and supervised fine-tuning (Liu et al. 2024; Abdin et al. 2024 Zohar
et al. 2025 Microsoft et al. 2025). While this approach makes simpler to integrate varios
modalities into highly optimized LLMs (Zohar et al, 2025 Microsoft et al. 2025). it often emphasizes
instruction-followinn and generation the expense of discriminative power.
taxonomy contemporary vision-language joint encoders is provided in Figure
3 TOWARDS EFFICIENT JOINT ENCODERS
We now build up the design of our efficient discriminative joint encoder (EDJE) step by step. First, we examine why existing multimodal joint encoders remain impractical for retrieval, pinpointing the vision backbone as the critical bottleneck (Section B.I). J. Next, show how precomputing vision features offers an appealing solution, while also introducing new challenge: the considerable cost of storing tokens (Section 3.2). Next, we discuss an efficient integration of vision and language modalities through compact joint encoder (Section 3.3 Finally, present compr-cooooession adapter that resolves the storage challenge by compressing long sequences vision tokens into a compact set of expressive tokens (Section 3.4
3.1 ON THE ABSENCE MULTIMODAL RE-RANKERS
2023) achieve strong performance Existing joint encoders such BLIP and BLIP-2 (Li al. 2022
but rely on visual feature extraction through large backbones like ViT-B/16 (384) and ViT-L/16 (384). This reliance introduces a severe bottleneck: encoding batch of 64 images requires about 400 ms with ViT-B and nearly 1,400 ms with ViT-L on an A6000 GPU before any cross cross-modal interaction even occurs. Specifically. for the BLIP family, the visual feature extraction alone consumes 83% of inference time in the ViT-B case and 93% with ViT-L. In practice, such inference times make infeasible use these models for retrieval, where thousands of candidates must be re-ranked per query. In comparison, the most downloaded text re-ranker model in HuggingFace is based on the MiniLM architecture (Wang et al. 2020), has just 22M parameters and processes a similar batch of full full-context sequences in under 60 order of magnitude faster. This gap explains the absence multimodal re-rankers real-world systems: the cost of extracting visual features alone is prohibitive.
3.2 PARADIGM SHIFT: VISION PRECOMPUTATION
With the vision backbone identified as the bottleneck, we next ask: must vision features always be extracted online? Cross-attention-base models and VLMs suggest otherwise: since the vision encoder operates purely on images, output can be cached and reused. Thus, we propose treating the vision encoder as a preprocessing stage, with vision tokens computed and stored to disk offline.
For standard ViT-B (Dosovitskiy et al. 2021) projecting each 16 16 patch to d84 embedding stored in FP16 occupies the same space the uncompressed 8-bit RGB image These token represen- tations can reside on disk rather than memory, as in late-interaction models like ColBERT (Khattab) and Zaharia 2020) and ColPali (Faysse et al. 2025). Under fixed token dimensionality, scaling the vision encoder improves representation quality while leaving per-image storage unchanged shiftinn heavy computation offline without increasing online cost. However, storing all tokens is intractable at scale: raw image size is typically too large, amounting to terabytes for web-scale databases. This problem motivates the development of strategies to compress the visual features.
Key takeaway: Precomputing vision tokens moves expensive computation of- fline, enabling powerful vision encoders without slowing inference. However, it comes at large storage costs, motivating methods to compress the visual features.
https: hatess://huggggigte co/ cross- wacpoasseeeoodd//n---eeeee wwwp://sssaacaaaiiaaaaa L-M 1 v2 T6 3 x bits per patch vs. 384 16 bits per token.
4

---

Published as a conference paper at ICLR 2026
Onliine
Linear
Small Lar Languege Model
Adapter
Laye/Norm & MLP
handing
Vision Fnooder
Cruss Attention
a
(a) A high-level view EDIE
(b) Token-compression adapter
Figure 3: EDJE architecture overview and adapter. (a) Offline stage (left): images are encoded by vision encoder and projected by the adapter into a compact set tokens compatible with language model. Online stage (right): the small language model consumes the compressed tokens together with text. (b) compeesoonressionn adapter: cross-attention utilizes & universal query tokens that act as feature extractors acting the visual tokens. The ML.P projects the extracted features to the embedding space of the language model.
3.3 INTEGRATING THE VISION MODALITY
Once visual tokens are computed, the question becomes how best to integrate them with text. Considering how the vision tokens are integrated into "cross-attention" models versus how they are integrated into large attention" models vision tokens are considered in the cross attention layers, large VLMs instead project vision tokens into the language embedding space and concatenate them with text tokens; this allows the cross-modal interaction to be handled entirely by self-altention layers. In our setting, the large language model can be replaced with compact, efficient language model to meet throughput targets. This yields an architecture with many benefits: (i) Fast: the language model can as small Minil (Wang al., 2020) any other efficient language model. (ii) Modular: any ViT-based vision encoder can be paired with any pre-trained language model via a lightweight adapter as a bridge between modalities. (iii) Expressive: modern vision encoders produce highly expressive tokens that capture both semantics and local spatial structure (Tschannen et al. (2025). (iv) Data
laan--aannggudoeeeeee models, we make an interesting observation: while in "cross
efficient: only the adapter has be trained from scratch. In the VLM literature has been observed that the language model and vision encoder require minimal tuning (Liu et al. 2023 2024 Abdin
al. 2024). high-level view of EDJE given in Figure
Key takeaway: Replacing the LLM in a typical VLM with a small, efficient language model yields joint-encoder architecture well suited for discriminative modeling: fast, expressive, modular and data efficient.
3.4 VISION-LANGUAGE ADAPTER LAYER
The adapter projects cached vision tokens into the language embedding space. It has been demon- strated the VLM literature that even very simple adapters linear layers (Liu et al.) 2023) or multi-layer perceptrons (MLPs) (Liu al. 2024) surprisingly effective despite their parameter count. However, local adapters map each vision token to one language token, limiting flexibility: vision encoders with larger context improve expressiveness but proportionally inflate storage.
To address this, we propose an attention-basee token compression layer that compresses vision token sequences into compact set of tokens. Specifically, we introduce of m learnable universal
5

---

Published as a conference paper at ICLR 2026
query tokens Q ........ 9ml that attend over the vision encoder tokens X [01, [1,..... Zn,]:
K XWK, == XWy
H = MultiHeedAAteention([( K.V)
(2)
The output H //i,...,ha, hm] is composed of m tokens that share their dimensionality with the vision encoder, h; Rd It is useful to regard the query tokens universal feature extractors, that softly select visual features most relevant for semantic matching. These states are than passed through standard residual block and projected into the language model embedding space Rdingun through simple linear projection:
0 H + -aaPP(aayeNNor((E)
Y OWpro proj
peoj
Risis Xchasguge
This mechanism provides a more flexible way of integrating visual information. Note that it gen- eralizes attention pooling strategies used in embedding models (Zhai et al., (2023) and has some
connection to the Q-Former layer (Li et 2023). The token compression layer depieted in
Figure 3b
We refer EDJE with simple MLP adapter layer as the "local" variant vs. when equipped with token-compression adapter which we refer to as the "loken-compreseed" variant.
The proposed token compression layer substantially decreases storage costs, seamlessly enabling vision encoders with longer context, higher input resolution and capacity.
Key takeaway:
4 EFFECTIVE DISCRIMINATIVE TRAINING
To obtain a joint encoder with strong discriminative performance, natural choice is to optimize for image-text matching. This involves determining the correspondence fe(t, v) of given image textual description f, where each positive pair must be contrasted against non-matching samples. Directly training the encoder in this way poses several challenges:
Negative pairs. While obtaining positive pairs from paired ward, selecting negative pairs is considerably more challenging. Random negatives are typically too easy, failing to distinguish fine-grained matches from loosely related examples. Conversely, mining negatives with another model introduces an inevitable dependence on that model's quality. To address this, we adopt in-batch hard-negative mining strategy utilizing an embedding model (matching to the vision encoder). For each mini-batch B, we compute pairwise similarities between all texts and images using the embedding model, obtaining weak similarity matrix S.y. For every anchor pair, we then select the top-k most similar (non-anchor) images and texts according to S.z as negatives. This approach effectively exposes the joint encoder the most confusable candidates without requiring full pairwise late interaction. Although this procedure may introduce occasional false negatives, in practice the abundance of informative negatives improves discriminative performance.
datasets is straightfor-
Vision-language alignment. While image-text matching is the central task of interest, it only provides limited signal for aligning langunne features and learning meaningful global representation. Large lanionnl models like I.L.VA (Liu et al., 2023] 2024) achieve this aligmment through a pre-training phase in which the model is encouraged to reproduce the caption matching certain image. Inspired by the pre-training phase of such models and to exploit the bidirectionality of our joint encoder, we employ masked language modeling with aggressive text-only masking. To strengthen the dependence of the [CLS] token on textual inputs, we introduce a projection layer top of the [CLS] representation and encourage recover the text embedding of the underlying embedding model when provided with text-only inputs.
Thus, pre-training strategy jointly optimizes three heads top of shared backbone:

---

Published as a conference paper at ICLR 2026
1. Image-text matching: binary classification matched image-caption pairs mined in-batch hard negatives.
2. Masked language modeling: mask 50% of caption tokens and predict the masked tokens given visual tokens and unmasked text.
3. Text-embedding recovery: we opt for recovering the embeddings of the text encoder g paired with the vision encoder using a cosine objective (f) cos (fa(t) (t), g(t)
Local-to-compresse distillation. To further enhance the performance of the token-compressed models we perform logit-level knowledge distillation using the local-adapter model variant as a teacher. Specifically, we encourage the liken-oompresee model (student) to imitate the image-text matching logits of the full-adapter joint encoder (teacher). For each positive, negative-image and negative-text pair consider the binary cross-entropy between student and teacher predictions:
Liiistit = - [yv loogoog (.) ) + +1 - -y)+) log(1 - -ms)]
with y/: = = (t, v)) and U. = (Sstudene (t, v)) where a is the sigmoidal function and s(t, v)
denotes the similarity logit that corresponds to and
EXPERIMENTS
To goal of this section is extensively investigate the empirical benefits of integrating EDJE into large scale retrieval pipelines. Specifically, we aim to address the following questions:
(Q1) Can EDJE, as a minimal-scale joint encoder, beat highly-optimized embedding models?
(Q2) How EDJE compares with existing joint encoders in terms of performance and efficiency?
(Q3) What is the significance of each component constituting EDJE?
5.1 EXPERIMENTAL SETUP
We train EDJE using& a two-phase protocol consisting of pre-training and fine-tuning phases as described in Section During both phases we freeze the vision encoder and train only the adapter of interest and the language model to process both modalities. We experiment with a variety of vision encoder families at multiple scales and input resolutions, including CLIP (Radford et al.) (021).
DEN (Fang et al. 2023a). MetaCLIP (Xu et al. 2024) and SigLIP2 (Tschannen al. [2025). Except for SigLIP2, we use the penultimate-layer hidden states as the vision-encoder output. The language model is fixed to be MiniL MomiLM--L2-uuncasee 1nnccaed (Wang et al. 2020) all experiments. To ensure fair comparison, use the smaller dataset mixture of BLIP for training; the pre-training data is composed of CC12M (Changpinyo et al.] 202T), CC3M (Sharma et al. 2018), SBU (Ordonez et al. (001)). Visual Genome (Krishna al. 2016), and COCO (Lin al. (0144,, totaling 14M sraage-caption pairs while fine-tuning only utilizes COCO. Full training hyperparameters are provided in Appendix AA]
For evaluation, we follow two-stage retrieval pipeline: for each query, we first retrieve the top-k candidates using embedding-basee retrieval with a CLIP-like model. These candidates are then re-ranked by EDJE, which jointly processes image token embeddings and captions. Unless otherwise stated, we fix the pool-size to & = 10. We report both text-to-image (T2I) and image-to-text (12T) performance under &ecall 1, 5. 10). consistent with prior foundation model benchmarks (Li et al.) 2021; 2022 2023 Wang et al. (002).. Since most embedding- based models report other or non- retrieval metrics, we reproduce them using the OpenCLIP framework (Ilharco et al.) 2021). verifying agreement with their reported numbers before presenting the aforementioned metrics. We evaluate on Flickr30k (Plummer et al. 2015) for zero-shot retrieval and on COCO (Lin et al., 2014) for fine-tuned retrieval. Following standard practice, we adopt the Karpathy split for COCO and the standard test split of 1,000 images for Flickr30k, each paired with five captions. We additionally evaluate EDJE under a more challenging retrieval setup in which all training images and captions inserted into the candidate pool, following LightningDOT (Sun et al., 2021). This makes the task considerably more challenging and resemble real-world scenarios. We provide the full details and results in Appendix F
7

---

Published as a conference paper at ICLR 2026
Table Zero-shot retrieval results on Flickr30K. We report Recall @ 1/5/10 for text-to-image and image-to- tasks across four backbones (CLIP, DFN, MetaCLIP and SigLIP 2) using various ViT scales and resolutions. Rows marked red represent EDIE with the corresponding ViT backbone.
Text- Text-To-Image
Image- TexTex
Model
ViT variant
Res.
@
R@5
R@ 10
R@
R R@5
@
10
62.1
85.6
91.8
81.3
96.1
98.3
ViT-B/16
224
76.8
90.7
91.7
91.1
98.2
98.4
CLIP
65.2
87
92.1
85.1
97.1
98.9
ViT-L/14
224
80.6
91.4
92.2
92.8
98.5
98.9
67.7
88.8
93.3
86.7
98.2
99
ViT-L/14
336
81.9
92.8
93.3
93.8
98.6
99.9
75.1
92.7
96
90
98.6
99.4
DFN
ViT-L/14
224
77.5
94
96.1
91.1
98.4
99.4
76.3
93.6
96.3
90.6
98.5
99.5
MetaCLIP
ViT-L/14
224
79.2
94.5
96.3
91.9
99.0
99.5
82.1
95.5
97.9
93.8
99.3
99.9
VIT-B/16
3842
84.3
96.6
97.9
94.3
99.9
99.9
SigLIP 2
82.3
96
98
94.8
99.6
99.9
ViT-L/16
3842
87.8
97.3
98
96.5
99.8
99.9
5.2
MAIN RESULTS
We first examine whether EDJE, when considered as a lightweight joint encoder with minimal capacity, can substantially improve retrieval performance over embedding based pipelines. To this end, we deploy the local variant as top-k re-ranker: for each embedding model tested, EDJE reuses its vision backbone and pairs it with MiniL. as the shared language encoder. evaluate zero-shot retrieval performance on Flickr30k with standard text-to-image and image-to-text retrieval tasks. The results are summarized in Table
EDJE boosts retrieval performance across all tested embedding models, emphasizing the potential of integrating re-rankers to existing retrieval pipelines. Specifically, we observe massive gains for the original CLIP (Radford et al.) 2021) model, with Recall@ improvements of up to 15% for image retrieval and 10% for text retrieval. Noticeable gains are also obtained for the SigLIP2 backbone, despite being highly optimized state-of-the-art embedding model. The improvements for DFN and MetaCLIP are less noticeable; however, DFN relies on filtering network fine-tuned on Flickr.
We next assess EDJE when considered a practical alternative to prior joint encoders (Li et al. 2021 2022. 2023), fixing the visual backbone to SigLIP 2 with resolution of 384² to match their setup. To ensure fairness, we evaluate both local and loken-compressed variants under cached-vision regime. Namely, we assume that the visual features are precomputed, so that only the online joint-encoding part considered allowing us compare against previous methods. Under this setup we compare methods along several axes: retrieval accuracy (Recall @ on Flickr, zero-shot, and COCO, fine-tuned, for both image-to-text and text-to-image), per-image storage (kilobytes), joint-encoder parameter count, online inference time (milliscconds for batch of 64 training data used. The results are summarized Table
A6000 GPU), and the amount of
EDJE achieves favorable accuracy-efficiency profile relative to existing joint encoders. The local variant matches prior work on Flickr (zero-shot) and remains competitive on COCO (fine-tuned), while using much smaller joint-encoder (tens of millions of parameters rather than hundreds) and substantially lower online latency. Crucially, these gains come with affordable storage costs: even in its uncompressed form it may be suitable for some use-cases, and the token-compressed variant
8

---

Published as a conference paper at ICLR 2026
Table Comparison prior art. We compare EDJE its Local and token-compressed (Com- pressed) variants (highlighted in red) against ALBEF, BLIP, and BLIP-2 Li et al. 2021 2022 2023) in both base and large configurations. The table reports retrieval performance: text-to-image and image-fo-text R@l on Flickr (zero-shot) and COCO (fine-tuned). We also report the amount of training data used. Finally, we include per-image storage, joint-encoder parameter count, and inference time for a batch of 64 samples.
Training data
Flickr-ZS
COCO-FT
Storage per image
Joint encoding parameters
Inference time (ms)
Method
T21
12T
T2I
ALBEF
12M
82.8
94.1
60.7
77.6
1,769 kB
147M
45.92
BLIP VEB8/16
12M
84.9
94.8
63.1
80.6
1,769 kB
139M
83.27
BLIP VTFL/16
129M
86.7
96.7
65.1
82,4
2,359 kB
139M
101.61
BLIP-2 VIT-216
400M
88.6
96.9
66.3
83.5
2,359 kB
167M
98.64
Local WTB/16
12M
84.3
94.3
60.9
76.1
442kB
33M
2.86
Local WTL/16
12M
87.8
96.5
64.9
81
442kB
33M
4.14
Compressed- 128 VIT-/16
12M
96.3
64.6
98kB
33M
2.04
Compressed-66 WTT-116
12M
86.9
96.4
64.6
80.9
49kB
33M
1.91
has minimal storage costs while preserving most of the retrieval accuracy. Interestingly, we find out that quantizing the compressed tokens before storing them, then de-quantizing upon inference yields minimal performance degradation and can further improve storage- perrgee-erffrmancce tradeoff. We refer the reader Appendix H for more details.
5.3 ABLATION STUDIES
We conduct series of ablation experiments assess the robustness of EDJE, isolating the contribu- tions of different design choices.
We begin by analyzing how varying the number of compressed tokens affects retrieval performance. Specifically, we evaluate Flickr30k zero-shot image retrieval using {32, 64, 128, 256} target tokens in the coke--compression adapter (Figure As expected, increasing the number of tokens yields better performance, with clear gap between the heavily compressed 32-token variant and the uncompressed "local" variant (576 tokens). Notably, 64 tokens strike an attractive balance between efficiency and retrieval quality. We additionally compare against simpler token-reduction strategies, providing alternative baselines to the proposed adapter; see Appendix G for details.
Next, we examine the sensitivity of EDJE to the size of the re-ranked pool k. Larger pools increase the likelihood of including relevant candidates but also introduce more distractors. therefore
TC Rdi
TC Rgg00 Local Rg1
Local RgS
TC Rgg
Local Rg10
Image Retrieval R@1
Flickr Zare-Sho Shat Image R
88
Flickr Zero- Zero-Shot Shot
0.98
0.96
87
0.94
0.90
0.88
128
Number tokens
20
60
80
100
120
140
Reranking size
Figure 4: Retrieval performance number of tokens. Flickr image retrieval for varying token counts, illustrating the comprssionn-perfomaacc tradeoff.
Figure Retrieval performance vs. re- ranking pool size. Robustness of local and 64-token variants under different pool sizes Flickr.

---

Published as a conference paper at ICLR 2026
important to test the robustness of the re-ranker to different pool sizes and evaluate its tolerance to noise. We measure zero-shot retrieval under @ecall@ {1, 5, 10} across varying pool sizes for both the local and 64-token variants (Figure[)) ( Results remain stable: while individual metries fluctuate slightly, overall retrieval performance is consistent.
Finally, ablate the pre-training objectives introduced in Section For the local variant, compare: (i) optimizing image-text text matching (TTM) alone, (ii) ITM combined with masked language modeling (MLM), and (iii) the full objective that further adds text-embedding recovery. Each auxiliary loss contributes positively, with the full objective delivering the strongest results. We further evaluate the impact of local-to-comen co-oopeesse knowledge distillation, which provides further gains for compressed variants by transferring discriminative capacity from the local model. We also investigate cross-model negative selection understand how EDIE behaves under different vision-encoder geometries. We refer the reader to Appendix B and Appendix E for more details.
5.4 INTERPRETABLE SEMANTICS COMPRESSED VISION TOKENS
To better understand the information preserved by the compressed visual tokens produced by EDJE, we aim to analyze their semantic structure regardless of any caption that may be equipped with the original image. We achieve this by inspecting the projection of each visual token into the language- model embedding space and assigning its nearest textual token from the LM vocabulary. Collecting these nearest neighbors (one per compressed tokens) reveals which language tokens the model most frequently associates with visual tokens, as depicted in Figure
reindeer
geologist
tunne
execut Lves teenagers
archangel Limestone
Light
t
Teapt
LO
Lovvee
S
sphinx
taliban
Sanditsknell
Tanponer
ora
lar
lare
brotherhged
veatonery
e
boulde
cavern
miners
to
Writiing
dridddsshiss
S
scr ecrambiin Lnn
quar
oEto
wa radia
untraaat
Figure 6: Semantic structure the 64-token compressed representation. Left: example image from the Flickr-30k test set. Right: nearest- nearest-neighbor textual tokens assigned to compressed vision tokens; word size reflects frequency the token distribution.
The compressed tokens map meaningful object and scene descriptors such as boulders, caves, glare, or trio, indicating that the adapter preserves important semantic information. In contrast, we find that interestingly many of the uncompressed 576-token representation map meaningless special tokens (unused80) 0), suggesting that large portion of native ViT tokens carry redundant content for retrieval. We refer the reader Appendix for further explanations and experiments.
6 DISCUSSION
We studied how to make joint vision language re-rankers practical at scale. We recognize visual feature extraction as the key bottleneck existing joint encoders. We tackle this problem by introducing EDJE, The approach of EDJE is precompute the vision tokens and compress them with lightweight adapter in an offline manner, in addition to a compact joint encoder that can deliver high-throughput inference while retaining high performance.
Limitations and future work. We think of this paper as proof of concept that may inspire follow-up work; for instance, we did not cover datullingullmuuitmmoooo retrieval, which has drawn attention recently (Thapliyal et al. (022)) or other modalities such audio video. More broadly, we believe joint encoders are largely underexplored; putting effort into improving them can benefit a variety of applications including zero-shot classification and filtering large paired datasets.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint
arXiv:2404. 14219, 2024. BRtt:::s: wtgp/.twwi....ooaaaaoooooooooora awtaaass/////
wttp:/rrg.bbb.//../////// 14219
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In CVPR, 2021.
Jianlv Chen, Shitao Xiao, Peitian Zhang. Kun Luo, Defu Lian, and Zheng Liu, Bge m3-embedding: multi uulti-granularity text embeddings through self-knowledge
Multi- Multi-lingual, distillation, 2024.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning, In European conference on computer vision, pages 104-120. Springer, 2020.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutamova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associa- tion for Computatioonal Linguistics, 2019. URL https: http://aa....mmaa.ccc.c.ccccc.ee CorpusTD:529617
org/
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. htthts: 11929
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 17425, 2023a.
Yuxin Fang, Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19358 19369, June 2023b.
Manuel Faysse Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https
/oopenevvew http://oooommidoooooaaa2aa
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis. Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song. Hannanch Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: in search of the next generation of multimodal datasets. Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Team Gemma.
Gemma 3 technical report, 2025. URL.
19786
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https::////.......ttt.ttt http:::///peneeew..eee.. het.:ewwwwwweeeeeeeeeeeeeeeeeeee
id=nZZeWeeeFFF)
11

---

Published as a conference paper at ICLR 2026
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL httpt: http://oi..oo oro.0.00000..200 If you use this software, please cite it as below.
zenodo. 5143773
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- vision-language representation learning with noisy text supervision. In International conference machine learning, pages 4904-4916. PMLR, 2021.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39-48, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164, doi: 10.1145/3397271. 3401075.
3401077
Kevis kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Arjun Karpur, Koert Chen, Ye Xia, Bingyi Cao. Daniel Salz, Guangxing Han, Jan Dlabal, Dan Gnanapragasam, Mojtaba Seyedhosseini, Howard Zhou, and Andre Araujo. TIPS: Text-image pretraining with spatial awareness. In The Thirteenth International Conference on Learning Representations, 2025. URL https
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei- Fei. Visual genome: Connecting language and vision using crowdsourced dense image an- notations. International Journal of Computer Vision, 123:32 73, 2016. URL https
//api. www.seematiiccccccccoeeeeeccccce
4492210
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-ing image pre- training for unified vision-language understanding and generation. In International conference machine learning, pages 12888-1222 -19800 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023. URL https: / /api. semant icscholar org/ CorpusiDp
256390509
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023. URL
08485
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296-263066 June 2024.
Muhammad Maaz, H. Rasheed, Salman H. Khan, and F. Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Annual Meeting of the Association for Com- putational Linguistics, 2023. URL https:: //api. api. remanticschol.. wwtp:orposoooooooooooooooo
259108333
Microsoft, :, AbdeIrahman Abouelenin, Atabak Ashfaq. Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang
12

---

Published as a conference paper at ICLR 2026
Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Tetyana Sych, Pranectha Vaddamanu, Shuohang Wang. Yiming Wang, Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang, Ziyi Yang, Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. Phi-4-mini technical report: Compact yet powerful multimodal language /a0pa//55033 01743
models via mixture-of-loras, 2025. URL /RLthttss:///xxii>
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari. SILC: Improving vision language pretraining with seff-distillation, 2024. URL https //openreview wwt.:/eooomiiidddedoeeeee
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: opeerevieew hett/Forun2id=VIReeer
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 1m2text: Describing images using million captioned photographs. In Neural Information Processing Systems (NIPS), 2011.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-piie to-phass correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-877 8763. PMLR, 2021.
H. Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Khan. Fine-tuned clip models are efficient video learners. 2023 JEEE/CVF Conference Computer Vision and Pat- tern Recognition (CVPR), pages 6545-6554, 2022. httts:t: semanticschols
lorg/CorpusTd:25440 :d:
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL 2015
newrips ca/pper files/paper/
Cristoph Schuhmano, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. In Proceedings of Neurips Data- Centa-Ceetricea Al Workshop, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. Proceedings of ACI. 2018.
Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, and Jingjing Liu. Lightningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval. In NAACL-HLT, 2021.
Hao Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Conference on Empirical Methods in Natural Language Processing, 2019. URL
https: www.ntiisscoooo...om..
wttg//orppaaoooooooo Corpus ID: 201103729
Ashish Thapliyal, Jordi Pont-Tuset Xi Chen, and Radu Soricut. Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset. In EMNLP, 2022.
13

---

Published as a conference paper at ICLR 2026
Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Nacem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual risionllanguag encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502. /4786, 2025.
Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic, Ibrahim Alabdulmohsin, Xiao Wang. André Susano Pinto. Andreas Peter Steiner, Lucas Beyer, and Xiaohua Zhai. Locca: Visual pretraining with location-aware captioners. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:///ppervvviieee ww. wwt.wt/wfrrnttttett//// d=
Jiahui Wang, Yutong Bai, Vijay Vasudevan, Dit- Yan Yeung, C Lawrence Zitmick, and Zhuowen Tu. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprini arXiv:2108. 10904, 2021.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution, 2024. URL http:: //apxivv///axv...o/ wwtp://wwwww//a////////// 12191
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
Hu Xu, Saining Xie, Xiaoqing Tan, Yo- Huang, Russell Howes, Vasu Sharma, Shang- When Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024. URL https:
/openreview. net fttp::/eeoooommm111--
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 00X1v72205.01917. 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11975-11986. 2023.
Xin Zhang, Yanzhao Zhang, Dingkum Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:
Industry Track, pages 1393-1412, 2024.
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng
Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, and Xide Xia. Apollo: An exploration of video understanding in large multimodal models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18891-18901, June 2025.
14

---

Published as a conference paper at ICLR 2026
A TRAINING HYPERPARAMETERS
Table A summarizes the main hyperparameters used throughout pre-training and fine-tuning. We fix the language model MiniLMM-L12-HHS&--unaaa and freeze the vision encoder in all runs. Unless otherwise stated, all experiments use in-batch negative mining with three negatives per sample, and re-ranking pool size of 10. These settings were chosen to balance training efficiency and retrieval quality and remained consistent across all backbones.
Setting
Value
Language model
MiniLMMLL12-H388mummmaa
Adapter hidden dim
8192
Re-ranking pool size k
10
Negatives per sample
3
Negative mining
In-batch, softmax-weighted top-k
Hard negatives
None
Distillation
Sigmoid-BB -BCE on pos/neg logits
Masking target
Caption tokens only 0.5
MI.M masking probability
Mask excludes
Special tokens, image tokens
Truncation policy
only_fin firt.
Text max length
64
Batch size (evaluation)
32
Optimizer
AdamW
Weight decay
0.05
Pre-train LR
3c-4
Fine-tune LR
2c-5
Warmup steps
100
Warmup LR
le-6
Min LR
le-6
LR decay rate
0.9
Input resolution
384
Table Key hyperparameters, masking strategy, and negative sampling settings used in our experi- ments.
15

---

Published as a conference paper at ICLR 2026
B ABLATION STUDY: TRAINING OBJECTIVES
We provide additional ablations on the training objectives. shows the incremental effect of adding masked language modeling (MLM) and text-image contrastive learning (ITC) top of the ITM baseline. Each component contributes positively to retrieval accuracy on Flickr30k with the Siggee 384² backbone, with ITM+MLM+ITC yielding the strongest results.
Table focuses on token compression and highlights the effect of applying distillation. Here, knowledge distillation provides additional improvements when compressing visual tokens. This section analyzes the interpretability properties of the compressed visual tokens produced by EDJE. Although the token compression module discards large portion of the original vision encoder sequence, we find that the resulting compact representation retains coherent semantic structure. We study two aspects: (1) the emergence high-level concepts within individual compressed tokens, and (2) the alignment between compressed visual tokens and text features used during retrieval.
Table 4: Ablation training objectives. We evaluate the effect of adding MLM and ITC on top of ITM. All configurations are evaluated on SigLIP2 Large 384 backbone. Results are reported in terms of R @ on Flickr30k.
ITM
MLMM
ITC
R@1
82.3
85.5
87.8
Table 5: Effeet of distillation with token compression. We report R @ Flickr30k using SigLIP2 Large 384 with 64 tokens compression.
Distillation
R @@1 1
X
83.8
86.9
16

---

Published as a conference paper at ICLR 2026
C USING EDJE IN PRACTICE
We provide a pseudo-code for using EDJE in practical retrieval pipelines, combining a clip-like embedding model, vector store and EDJE an efficient re-ranking model. A pseudo-code for the indexing stage is given in Algorithm
T while the retrieval stage is summarized in Algorithm 2
Algorithm Indexing (offline)
Input: Images (ema, vector store (store), CLIP-like vision encoder (vi siio__nnodddeeeeeeeeeee finetuned EDJE token adapter (adapter)
1: image_loader
DataLoader (images)
2: for image_batch in image_looee loader do
3: Extract both the usual embeddings and patch features in one pass
4: features, embeddings
(image_batch)
5: // Apply the token adapter on the encoders' features
6: features adapter (features)
7: // Store embeddings on RAM and features on disk
8:
store. insert
9:
egbeeddins==meeete
10:
extra_dar - "eatures"; features)
11:
12: end for
Algorithm 2 Retrieval
Input:
Query text (query), vector store (store), CLIP-like text encoder (text_encoder),
finetuned EDJE re-ranker (re- ree-ranke)), re-ranking pool size (k)
Output: re-ranked retrieval results
1: // Compute the usual query text-embedding
2: text_ember cext_embedding
text _ee_deeccderrrree (query)
3: // Retrieve candidates from vector store
4: candidated
store. knn_query (text_embeddiing k=k)
5: // Compute re-ranking scores with EDJE
6: features torch. cat ( [[aandidate] ("features") for candidate in candidates), dim=0)
7: scores re-ranker (query, features)
8: results candidates [scores argsort (descending=Truee)
return
results
C.1 TOKEN-FETCH I/O CONSIDERATIONS
Beyond compute, practical deployment of EDJE must also account for the cost of fetching precomputed vision tokens from storage. In our experiments, each image is represented by 64 compressed tokens in BF16, corresponding roughly 49 kB per image. For re-ranking pool of 50k candidates, this amounts to reading approximately 2.46 GB of data per query.
To estimate the expected I/O overhead, we benchmark two storage layouts a PCIe 4.0 NVMe SSD: (i) a single contiguous NumPy array storing image representations, and (ii) a more realistic memory mapped array accessed via random indices. In the contiguous case, loading the full 2.46 GB block takes around 0.39 0.003 s over 10 runs (approximately 6.3 GB/s), which is in line with the advertised bandwidth of the SSD. With random access
100k-image memory-mapped index (50k random entries), the same amount of data is loaded in 0.59 0.04
These measurements indicate that, modern local SSDs, the I/O cost of fetching compressed tokens on the same order as the compute cost of the joint encoder and still allows end-to-end processing of tens of thousands of pairs per second. Note however that networked storage can exhibit substantially lower throughput, so we do not recommend it for EDJE deployments.
17

---

Published as a conference paper at ICLR 2026
D VISUAL TOKENS INTERPRETABILITY
D.1 EMERGING VISION-TOKEN SEMANTICS
We next study the semantic content of the visual tokens produced by EDJE in inriiinnindependeen manner. The goal to qualitatively understand what information the compressed visual tokens carry, regardless of any ground-truth caption that may may-not be provided.
Given an image, let )= denote either the compressed visual tokens (e.g. 64 tokens from the loken-compression adapter) the full ViT sequence (e.g 576 tokens, transformed locally). We first project each into the language-model embedding space using the same projection as in the joint encoder. For every visual token Us, then retrieve nearest language token wy from the LM vocabulary (using cosine similarity). Collecting these nearest neighbors {w; [w over the Flickr30k test allows to analyze which language-token tokens the model most frequently associates with visual tokens.
This analysis has two desirable properties: (i) it evaluates the semantics of visual tokens purely through the geometry of the joint embedding space, without using the paired captions for that image, and (ii) yields interpretable text tokens that can be visualized as through frequency histograms. We perform this analysis for two concrete example images for the compressed tokens, as illustrated in Figure
secluded
canine
ravine
grolling
tongue
ue
footsteps
ries
lipstick
collar
woodlard
trail
dog
bracelet
girlfriends
grasses
deciduous
shadows
boulder
appalachzan
C
erettt tory
hased
oathwa
y
s
forest
shrubs
ts
reddish
rhodesiahusky
reindeer
geologist
tunne
executives
lves
teenagers
archangel limestone
t
sunlight
Teapt
10
da
icness
coveer
s
sphinx
commensi
prece Pecespeat tal
panditskmn
Liba
Hanpower
sknelt
ona
lare
brotherhood
restonary
e
cavern
You
boulde
JolLette
iLl
Brriendships
S
thrill
climb
scramb arbbm ling
quarr
oEEo
wa waterfal Wi
I
ring
tiig
urdraaht
Figure 7: Emerging token semantics for the 64-token compressed representation. Left (per row): example images from Flickr test set. Right (per row): textual tokens word cloud, size indicates frequency according to vision token nearest-neighbors histogram.
18

---

Published as a conference paper at ICLR 2026
The compressed tokens exhibit stable and highly interpretable semantics. Tokens frequently corre spond to concrete object categories (e.g., dog, collar, boulders, trio) and scene attributes (e.g., glare, shadows). Despite the large reduction in sequence length, the model preserves rich vocabulary of
visual cues. We than repeated the analysis using the "local" EDJE variant, as illustrated in Figure g
lino
vegetation
upland
oux
kinng
Horest
Untwigs
*
er
y
bur
chihuahua
gundy
wae
nassif
land
ravine
ra
grasses
taa
blackanad
shadows
bry
rney
unuu
Suppy
in
sea
iae
panzer
Spp
passagmay
Lachaac
g
iferous
treees
aras
CLS
treacherous
Srushes
parpathiaa
mammal trait
auus
shrubs
Sam
hike
boulders S Canineoider
jaceron
on
tongue
tze
pendletoon, shropshire iness
cho bsarpss brace
mounds
isabella
nighttine
lino
crouched
cap
Ve
r n
KLLS CLL.canyon
sands spectacular
Staring
o
nonaster.. les
sedowel
vine
The
oou
stalt
d
ders
preechhrous
S
Oides
rrlyy
radiating
Polunteered
avin
a
irinttatin
carpathian
sparkle
scrambling
dints
acher
he mee
-
sunlight flashlight.
pundy traniylvania
barreBine
Climb
grating
uman
unused8e
passagelayuuue
protaggnnes
Vo
demetrius
sliveryillume thoee
Figure 8: Token semantics for the 576-local EDJE representation. Left (per row): example images from Flickr test set. Right (per row): textual tokens word cloud, size indicates frequency according to vision token nearest-neighhors histogram.
In this case, however, large fraction of tokens map to special vocabulary item, unused80, or to scattered wwwffrequncy words, resulting in much less concentrated distributions. This suggests that many of the original ViT tokens have low semantic content in the joint space and largely redundant for downstream retrieval.
19

---

Published as a conference paper at ICLR 2026
D.2 AMAGE-CAPTINE TOKEN ALIGNMENT
We complement the auption-independern analysis above with quantitative study of how well the compressed visual tokens align with caption tokens. The goal is measure what extent cross- modal alignment is preserved when reducing the number of visual tokens from the full ViT sequence to the compressed EDJE representation.
For each image-caption pair, we first preprocess the caption by lowercasing all words and removing punetuation and standard English stopwords (e.g.. "A child playing in the ocean." "child playing ocean").
Let (1,}ا= denote the remaining textual tokens and {u:]==1 the set visual tokens for the corre- sponding image, where {u;) is either the full (locally transformed) ViT sequence (e.g., n 576) or the compressed set produced by EDJE (e.g., 64).
We embed both text tokens and visual tokens into the joint embedding space using the same projections as in the joint encoder, and compute ColBERT-like alignment score (Khattab and Zaharia 2020 Faysse et al. 2025). For each textual token Ły, we compute its maximum cosine similarity to any visual token,
$j
max Isign
where f(.) and g()) are the corresponding text and vision projections. We then define the alignment score for the image- caption pair as the average of these maxima,
1 m
S
85-
Intuitively, measures how well each caption word can "find" least one strongly related visual token in the image. Note that we didn't directly optimize S as scoring criterion.
We compute the distribution alignment scores S over all Flickr30k pairs for both the full 576-token representation and the 64-token compressed representation as depicted in Figure 9
Alignment scores: full token-compressed
175
Full
Noker-Compeessed
125
Count
75
50
25
Alignment score
Figure 9: Distribution of alignment scores for full token sequences and their token compressed counterparts. The histogram shows that compression preserves the overall alignment structure while slightly shifting the distribution toward lower but still tightly concentrated scores. This supports the claim that the compressed representations retain most of the semantic signal needed for retrieval.
The token-compressed representation achieves average alignment score of 0.2516 = 0.0491, while the full-sequence representation achieves 0.2404 + 0.0405. The corresponding histograms overlap and exhibit very similar shapes, with the compressed representation showing slightly higher mean. two-sided paired t-test on the per-pair scores indicates that this difference is statistically significant (p 3.0 10-153) with mmll-to-mmderrat effect size (d 0.39). Overall, these results show that
20

---

Published as a conference paper at ICLR 2026
EDJE 's 64-token representation preserves, and slightly improves, image-caption alignment relative to the full 576-token ViT sequence, despite using 9x fewer visual tokens.
21

---

Published as a conference paper at ICLR 2026
E CROSS MODEL NEGATIVE MINING
In the main experiments, hard negatives for EDJE are mined using the same embedding model family the underlying retriever (e.g., SigLIP2 for a SigLIP2-based EDJE). To assess the impact of this design choice, we additionally study cross-model negative mining, where the miner and the backbone come from different embedding models.
Concretely, we consider two settings: (i) using SigLIP2 as negative miner for CLiP-based EDJE, and (ii) using CLIP negative miner for SigLIP2-basee EDJE. both cases, the joint encoder and token-compression components remain fixed; only the model used to select hard negatives during fine-tuning changed.
Table Using SigLIP2 as a negative miner for CLIP based EDJE. Mining with SigLIP2 does not substantially affect CLIP performance.
Model
Flickr Zero-Shot
COCO Finetuned
Text-to-Image
Image-to-Text
Text-to-Image
Image-to-Tens
CLIP ViT L/14@336
81.9
93.8
54.6
70.7
+SigLIP2 L miner
81.8
93.5
54.6
70.8
Table Using CLIP as a negative miner for SigLIP2 based EDJE, Mining with CLIP significantly degrades SigLIP2 performance.
Model
Flickr Zero-Shot
COCO Finetuned
Image-to-Text Texae
Text-to-Image
Image-to-Text
SigLIP2 VIT L/16@336 384
87.8
96.5
64.9
81.0
+CLIP L miner
81.5
93.0
58.9
74.48
The results Tables and 7 reveal interesting pattern. When SigLIP2 used a negative miner for CLIP-based EDTE, performance changes only marginally: Flickr30k zero-shot R@ remains essentially unchanged, and COCO fine-tuned scores are within a similar range. This suggests that in this setting, overall performance is largely limited by the vision backbone rather than by the precise geometry of the negative miner. In contrast, when CLIP used as a miner a SigL.I2--based EDJE, performance drops substantially image-to-tex retrieval. This degradation emphasizes the importance of selecting sufficiently hard negatives. Overall, these experiments support the practical guideline that hard-negative mining should be performed with a model that is at least as strong as, and well aligned with, the underlying vision-encoder used in EDJE.
both Flickr30k and COCO, for both text-to-image and
22

---

Published as a conference paper at ICLR 2026
F FULL DATASET RETRIEVAL EVALUATION
To provide a more comprehensive evaluation, we adopt the full-dataset retrieval protocol used in LightningDOT (Sun et al. 202I). where retrieval is performed against all images and captions in the dataset, including the train/validation splits. This setting is considerably more challenging and better reflects real- world retrieval scenarios. We follow LightningOOT's setup for both Flickr Full and COCO Full, and scale the re- re-ranking pool size to 100 in all experiments.
Table 8: Flickr Full retrieval results under the LightningDOT setup. Retrieval is performed against all dataset's images/captions as retrieved instances. EDJE evaluated in a zero-shot setting and substantially outperforms LightningDOT in both text-to-image and image-to-text retrieval.
Model
Text-to-Image
Image-to-Tens
R@5
R@ 10
R@20
R@5
R@ 10
R@20
LightningDOOT
60.1
69.5
78.3
75.1
83.9
90.5
EDIE
78.32
84.54
89.58
92.4
95.9
97.7
Table 9: COCO Full retrieval results under the LightningDOT setup. Retrieval is performed against all dataset's 's images/captions as retrieved instances. EDJE significantly outperforms LightningDOT across all recall levels and directions.
Model
Text-to-Image
Image-to- Text
R@5
@
R R@20
R@5
R@ 10
R@20
LightningDOT
37.3
46.8
56.4
48.0
59.0
68.9
EDIE
52.23
60.55
68.08
69.86
76.96
82.64
As shown in Tables 8 and 9 EDJE substantially improves full-dataset retrieval performance over LightningDOT on both Flickr Full and COCO Full. On Flickr Full (zero-shot), EDJE yields large gains in both directions and at all recall levels (e.g., text-to-image R@5 improves from 60.1 to 78.32, and image-to-text R@5 from 75.1 to 92.40). On COCO Full (fine-tuned), EDJE again surpasses Lig by wide margin (e.g.. text-to-image R R@5 from 37.3 to 52.23, image-to- text R@5 from 48.0 to 69.86). These results confirm that EDJE remains effective more realistic large-candidate retrieval scenarios.
23

---

Published as a conference paper at ICLR 2026
G TOKEN COMPRESSION BASELINES
In this section we compare the proposed token-compression adapter in EDJE to several alternative ways of reducing the number of visual tokens produced by the SigLIP2 ViT-L encoder. All methods start from the same 576-token ViT sequence x resolution) and compress it to 64 tokens per image. We evaluate on Flickr30k (zero-shot) and COCO (fine-tuned).
We consider the following loken-compression strategies:
Striding. A simple subsampling baseline that keeps every 9th token from the 576-token VIT sequence, yielding 64 tokens in total (576/9 / 64).
2. Token clustering. We run k-means++ over the 576 visual tokens for each image to obtain clusters, and use the cluster centroids as compressed tokens.
3. Attention pruning. We compute the attention scores of each token in the last ViT layer and keep the 64 most attended tokens (where attendance is averaged across heads).
For fairness, each configuration is pretrained and fine-tuned end-to-end with the same protocol as our conepssoonressoonn model. Particularly, we distill from the uncompressed (576-token) teacher in the main EDJE experiments.
Comparison of cokee-compeessionn strategies applied to the SigLIP2 ViT-L image encoder. We report Recall @ I on Flickr30k (zero-shot) and COCO (fine-tuned), SigL/P2 (Baseline) denotes the original embedding-om model. All other methods compress the 576 ViT tokens to 64 tokens per image and use the same joint encoder architecture.
Table 10:
Model
Flickr Zero-Shot
COCO Finetuned
Text-to-lmage
Image-to-Text
Text-to-Image
Image-to-Texi
SigLIP2 (Baseline)
82.3
94.8
Striding
83.24
94.1
60.09
77.2
Clustering
85.66
96.1
63.31
79.76
Attention Pruning
82.4
93.7
58.6
76.4
EDJE
86.9
96.4
64.6
80.9
We observe that EDJE's token-compression adapter consistently outperforms striding, clustering, and attention pruning across both datasets and directions (text-to-image and image-to-text). This emphasizes the superiority of the learned query-based adapter over generic pooling or pruning schemes.
24

---

Published as a conference paper at ICLR 2026
H QUANTIZING THE VISION TOKENS
In all main experiments, the precomputed vision tokens are stored in BF16 (or FP16) format. Because EDJE is designed for farge-scale retrieval with potentially billions of stored image representations, is interesting to examine how far the storage footprint can be reduced via quantizing the numerical precision of the vision tokens.
We perform post- post-training quantization only on the precomputed vision tokens, while keeping the joint encoder itself in BF16. For each quantization type, the cached tokens are quantized on disk and de-quantized immediately before being fed to the joint encoder at inference time, with no additional finetuning.
We consider two representation families: (i) the full 576-token ViT sequence, and (ii) the 64-token compressed representation. For each family we evaluate three numeric formats: (a) BF16 (no quantization), (b) FP8-E4M3, and (c) FP4-E2M1. We measure Flickr30k zero-shot image retrieval performance @R@ 1) and compute the average storage per image in kilobytes, taking into account the number of tokens, hidden dimension, and numeric precision. This allows experimentation with wide range of formats including FP4 and 1-bit signed vectors.
Retrieval Performance (Flckrt0k) vs Represenattin Storage
TE84 199
11. Image 140/oz 161
0.25
0.33
30 50
102
200 500
Agprosarna On Stat300 145)
Figure 10: Retrieval performance (Flickr30k zeoo image retrieval Recall@ 1) versus image representation storage size (in kilo-bytes, log scaled). We compare the full 576-token ViT representa- tion and the 64-token compressed representation under BF16, FP8-E4M3, and FP4-E2M1 formats. EDJE compressed tokens retain strong performance even under aggressive quantization, while substantially reducing storage.
Figure 10 summarizes the rrfde-off between retrieval performance and representation storage. The plot shows that the compe-compoession is remarkably robust to aggressive quantization: moving from BF16 to FP8 and even FP4 leads to only minor changes in R@ 1, while substantially reducing storage. Moreover, combining FP8 quantization with 64-token compression cuts the per-image storage in half beyond the already compact 49 kB, with negligible loss in retrieval quality, pushing the storage-performance tradeoff even further.
25

---

Published as a conference paper at ICLR 2026
I ON EXTENDING EDJE TO VIDEO RETRIEVAL
Although video retrieval lies outside the scope of this work, EDJE naturally admits several extensions the video domain. A video may be represented sequence of frames ff, ff,...., fr). each processed independently by the vision encoder to produce X; = (o(ft) E Rnnx Concatenating per-frame features yields high-dimensional tensor
X
X
Xr] E R Txuxd
corresponding T visual tokens. The central challenge is therefore compress hundreds of frames efficiently while preserving temporal structure and producing a compact representation suitable for re-ranking,
Below, we outline two concrete strategies for adapting EDJE to video-text retrieval.
1.
EDJE's adapter temporal compressor. A direct extension is to repurpose the EDJE token-compression adapter to operate across time. Each frame is first mapped to a single visual token using any image embedding model, yielding temporal sequence of T tokens. A temporal adapter can then be trained to compress these tokens into a small set of temporally aggregated tokens. This design captures long-range semantics across many frames and improves upon simple temporal pooling strategies such as averaging frame embeddings (Rasheed al. 2022 Maaz et al., 2023). Temporal positional encodings can be incorporated by adding a learned position vector to each frame token before passing it to the adapter.
2. Spatial compression followed by temporal compression. An improvement over the afore- mentioned idea is to enhance single frame representations by replacing naive embeddings by compressing frames via a pretrained version EDJE's token-compression adapter, applied each frame to better compress information spatially. Then, similar poken-compressee adapter is learned to temporally aggregate all of the compressed tokens from each frame in efficient manner (e.g 576 64 per frame).
26

---

Published as a conference paper at ICLR 2026
J IMPACT OF TEXT-ENCODER CHOICE ON RETRIEVAL
The joint encoder EDJE uses an extremely lightweight MiniLMM-L12-H384 text model for joint encoding, A natural question is whether replacing MiniLM with larger, more expressive language encoder such as BERT-Base could further improve retrieval performance. Prior work in vision lan- guage modeling suggests that stronger language encoders sometimes offer marginal gains in caption understanding, but they also introduce greater computational cost and may provide diminishing returns retrieval settings where the visual features dominate (Devlin et al. 2019 Radford et al. 2021 Jia et al. 2021 Li et al. 2022).
To study this trade-off, we replace MiniLM with BERT-Bas (uncased) while keeping all other components fixed. In particular, we use the same SigLIP2 Base visual encoder, the same token- compression module, and the same training protocol used in our main experiments. The joint encoder therefore differs only in the text backbone. We evaluate this configuration on Flickr30k using the standard zero-shot retrieval protocol.
Table 11:
Flickr30k zero-shot retrieval results when substituting MiniLM with BERT-Base as the text encoder in the joint module. The visual encoder is SigL.P2-Base in all configurations. EDJE (MiniLM) denotes the baseline from the main paper.
Text Encoder
Text-to-Image
Image-to-Texx
R@D 1
R@5
R@ 10
R@1 1
R@5
R
10
MinILM with Siglip Base
84.3
96.6
97.9
94.3
99.9
99.9
BERT-Base with Siglip Base
84.52
96.58
97.9
93.9
99.9
99.9
As shown in Table substituting MiniLM with BERT-Base produces mixed but notable effects. BERT-Base yields slight improvements in the text-to-image setting, particularly on R@) I and R@ 10 (e.g., R@ increases from 84.3 to 84.52, and @ 10 from 97.9 97.98). In contrast, MiniLM remains marginally stronger for image-to-texx text retrieval (e.g.. R@ improves from 93.9 with BERT to 94.3 with MiniLM), while both models at higher recall levels.
Although these results do not indicate that larger language encoders provide substantial gains in our setup, it important to note that our model trained on only 12M images, considered small-scale relative to typical size, does not appear to bottleneck the coken-compressed retrieval pipeline.
pretraining regimes. Nevertheless, MiniLM, despite its compact
Overall, this experiment shows that BERT-Bac can slightly improve specific aspects of retrieval, but lightweight, well-distilled text encoders like MiniLM remain highly competitive and more efficient for EDJE's joint-encoder design.
27


Reviews:
[{"review_id": "DEpTGDZKZs", "reviewer": "Reviewer_kbcQ", "summary": "The paper introduces EDJE, a system that makes image–text search faster and more accurate by running a small joint model on precomputed and compressed image features. Unlike existing models that are too slow to use in large-scale systems, EDJE speeds up inference by more than 50× while keeping similar accuracy. It improves search results across various models and datasets without requiring extra fine-tuning.", "questions": "- As I pointed above, I would suggest the authors to expand the experiments to more realistic evaluation settings. \n- For token compression, what about the following baselines?\n    - Simple pooling (e.g., mean pooling)\n    - Direct token pruning techniques (like token clustering, attention pruning) on the vision tokens, similar to methods used in ViT efficiency papers, as an alternative to query-based compression.\n    - What if we adopt the pre-trained Q-former from BLIP and keep it frozen?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a practical method EDJE for image-text reranking by combining precomputed vision features with an cross-attention-based token compression adapter. The token compression design is well-motivated. \n- Empirical validation are conducted across multiple vision backbones and datasets. The proposed method achieves competitive or superior retrieval accuracy with drastically improved efficiency.", "weaknesses": "Major weakness:\n- Although the performance is strong, the retrieval benchmarks are quite outdated. To mimc closer to real-world retrieval setting, I would suggest the authors to follow\n\n    [1] Sun, Siqi, et al. \"Lightningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n    This paper introduces retrieval across the full COCO/FLICKR dataset to better mimc real-world retrieval setting. \n\n    [2] Jiang, Ziyan, et al. \"Vlm2vec: Training vision-language models for massive multimodal embedding tasks.\" arXiv preprint arXiv:2410.05160 (2024).\n\n    This paper introduces a more realistic benchmark MMEB where the retrieval query and target are both multimodal.\n\n- In addition, the reranking system conceptually is similar to Lightningdot, although the implementation details are quite different and now inspired by latest advanced in VLMs. \n\n- EDJE drastically improves in zero-shot retrieval. However, on the fine-tuned COCO retrieval task, EDJE is only on par with prior models. One could argue that a larger-capacity joint encoder with more parameters might still have an advantage in the fine-tuned regime.The accuracy vs. model size trade-off is not fully explored in this paper. It would be helpful if the paper explore using a larger language model or more parameters in EDJE. \n\nMinor weakness:\n- All citations should use \\citep{} instead of \\cite{}\n- The term “local variant” vs. “token-compressed variant” is used to distinguish the full-token vs. compressed EDJE, but the term “local” isn’t very self-explanatory. If I understand correctly, it means the MLP adapter outputs the full local tokens without compression.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXP1uGChjk", "reviewer": "Reviewer_tFqx", "summary": "The authors propose an Efficient Discriminative Joint Encoder (EDJE) to address the deployment challenges of vision-language reranking models in large-scale scenarios due to computational constraints. Multimodal retrieval typically incurs higher inference costs than text retrieval due to visual computation, making efficient interaction between textual and visual information a key factor in determining the practicality of multimodal retrieval models. The authors alleviate the aforementioned issue to some extent by further compressing visual information into a more compact token sequence, which proves effective according to the experimental results. However, the novelty of this work and the interpretability of the model require further clarification.", "questions": "1. How does the token-compression adapter preserve or improve multimodal alignment between the compressed visual tokens and textual representations?\n\n2. If a different vision encoder is used, must the adapter be retrained and all previously stored tokens regenerated? Could the authors comment on the practical implications of this design choice?\n\n3. How is the semantic quality of the compressed visual features assessed? Would visualizations or qualitative analysis of the compressed tokens help to evaluate information retention?\n\n4. After compression by the adapter, how interpretable are the visual tokens? Are there methods to analyze or explain what visual information each token represents?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The research focuses on the demand for efficient retrieval in the current multimodal retrieval field, which has strong practical significance. It proposes a vision-language reranking model that addresses the deployment challenges of multimodal retrieval joint encoders and provides new insights for multimodal retrieval methods. Experimentally, EDJE demonstrates competitive performance while significantly reducing storage requirements and online computational costs, achieving high-throughput inference. The core concepts of the paper are clearly defined, with no ambiguous expressions or logical contradictions.", "weaknesses": "The proposed framework essentially follows the well-established two-stage retrieval paradigm, where visual features are pre-extracted offline and a lightweight joint model is used for re-ranking. While this design improves efficiency, it does not introduce a fundamentally new retrieval or representation mechanism. Similar strategies have been adopted in prior works such as LightningDOT [1] and VISTA [2]. The token-compression adapter in EDJE, as described, functions primarily to further compress visual encoder outputs into a compact set of tokens for offline storage and later consumption by the language model; this operational choice—offline compression of visual features into a small set of learned tokens—is conceptually similar to prior approaches such as TokenLearner [3] and the Q-Former used in BLIP-2 [4]. Consequently, the methodological novelty appears limited: EDJE mainly integrates established components (offline encoding, attention-based token compression, and a lightweight re-ranker) for an engineering trade-off between storage and online compute. \n\n[1] Sun, Siqi, et al. \"Lightningdot: Pre-training visual-semantic embeddings for real-time image-text retrieval.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n[2] Zhou, Junjie, et al. \"Vista: Visualized text embedding for universal multi-modal retrieval.\" arXiv preprint arXiv:2406.04292 (2024).\n\n[3]Ryoo, Michael, et al. \"Tokenlearner: Adaptive space-time tokenization for videos.\" Advances in neural information processing systems 34 (2021): 12786-12797.\n\n[4] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" International conference on machine learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WsDVxP29PG", "reviewer": "Reviewer_bquk", "summary": "The paper introduces EDJE, an efficient discriminative joint encoder for large‑scale vision–language reranking. EDJE precomputes vision tokens offline, compresses them via a lightweight attention‑based adapter into a small set of tokens, and feeds these with text into a compact language model at query time. On Flickr30k (zero‑shot) and COCO (fine‑tuned), EDJE matches or approaches prior joint encoders while drastically reducing online latency and storage (e.g., 1.9-4.1 ms per 64-sample batch; 49 kB/image with 64 tokens). The paper includes ablations on token count, pool size, and training objectives.", "questions": "- Can you report end‑to‑end throughput/latency (including token fetch from SSD/NVMe, k‑reranking, and aggregation) to contextualize the 50k pairs/s claim?\n\n- How sensitive is performance to storage format/hardware (e.g., memory‑mapped arrays vs individual files; SATA vs NVMe vs network storage)? Any prefetching?\n\n- What is the effect of quantizing compressed tokens (FP8/INT8) on R@k and latency? Can training be made quantization‑aware?\n\n- Negative mining currently uses the same embedding model family; how does EDJE fare with cross‑model negatives to reduce bias toward the retriever geometry?\n\n- Beyond varying token count, can you vary adapter capacity (depth/width) to separate capacity vs token‑count effects at 32–64 tokens?\n\n- Any preliminary multilingual results (e.g., Crossmodal-3600) to support future-work claims?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Moves ViT compute offline; integrates vision via a compact joint encoder, addressing the main blocker for joint reranking.\n\n- Universal‑query adapter reduces hundreds of ViT tokens to tens while preserving accuracy; 64 tokens ($\\sim$49 kB/image) is a strong trade‑off.\n\n- EDJE consistently improves Flickr30k ZS R@1, e.g., CLIP ViT‑L/14@336: 67.7->81.9 (T2I).\n\n- With SigLIP2‑L/16@384, EDJE‑Local gets 87.8/96.5 (Flickr T2I/I2T R@1) at 4.14 ms (64‑batch), vs BLIP‑2's 88.6/96.9 at ~99 ms.\n\n- Objectives (ITM+MLM+ITC) and distillation help; pool‑size sensitivity is stable.", "weaknesses": "- Latency excludes disk I/O/feature fetch; the 50k pairs/s figure from the abstract lacks a pipeline breakdown.\n\n- Only Flickr30k ZS and COCO FT; no tests under domain shift, web‑scale noise, or multilingual queries despite claims of modularity.\n\n- EDJE is competitive but not strictly superior to BLIP‑2 on R@1; the paper could more precisely state where it leads/trails.\n\n- Lacks comparisons to alternative compression/pooling (e.g., Perceiver‑style latents, strided token dropping) to justify the chosen adapter beyond results shown.\n\n- Storage numbers imply FP16 tokens; quantization (FP8/INT8) and its impact aren’t analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lthq5CCeid", "reviewer": "Reviewer_EbYo", "summary": "The paper studies the problem of visual-language retrieval. More specifically, the paper focuses on the re-ranking stage of visual-language retrieval. To this end, the authors propose a new architecture, i.e., efficient discriminative joint encoder, to do the re-ranking efficiently. Besides, the authors propose a lightweight token-compression adapter that condenses vision features to reduce storage and computation. Experiments show the re-ranking method can consistently achieve a gain for the text-image retrieval performance. Analysis demonstrates the efficiency of the proposed method.", "questions": "Could the method also be applied to video-text retrieval and have the same conclusion as it is for text-image retrieval? This is very important concerning the scope and significance of the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I like the idea of the paper. The design of the architecture enhances the efficiency of re-ranking in a smart way.\n\n2. Performance is good with efficiency. The experiments show the proposed method achieves a consistent performance gain in re-ranking, while being highly efficient.\n\n3. Clear presentation of the paper. The paper is clearly presented. The teaser figure is good, together with several highlighted 'key takeaways', making the important ideas of the paper very clear.", "weaknesses": "1. Scope of the paper. Although the title of the paper is 'vision-language', the paper only studies text-image retrieval, and the video-text retrieval, which is a very important and more challenging task, has not been touched. It would be better if the paper can also include the results for video-text retrieval so that it is more general and will have a wider audience for the ICLR conference.\n\n2. Small issues: 'rerank' should be 're-rank' through out the entire paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)