Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignment
Authors: Dahun Kim
Anelia Angelova
Venue: COLM 2025 (2025)
Keywords: CLIP
contrastive learning
LLM embedding
TL;DR: We propose a method that uses multiple context-adaptive prompts with a pretrained LLM to generate diverse text embeddings for contrastive vision-language learning.

Abstract: We propose Context-Adaptive Multi-Prompt Embedding, a novel approach to enrich semantic representations in vision-language contrastive learning. Unlike standard CLIP-style models that rely on a single text embedding, our method introduces multiple structured prompts, each containing a distinct adaptive token that captures diverse semantic aspects of the input text. We leverage a pretrained LLM as the text encoder within the CLIP framework, processing all prompts jointly in a single forward pass. The resulting prompt embeddings are combined into a unified text representation, enabling semantically richer alignment with visual features. To further promote semantic diversity and representation quality, we incorporate a diversity regularization loss and a negation-aware loss, encouraging specialization across prompts and improving contrastive discrimination. Our method achieves consistent improvements on both image-text and video-text retrieval benchmarks.

Published as a conference paper at COLM 2025
Context-Adaptive Multi-Prompt Embedding with Large Lan- guage Models for Vision-Language Alignment
Dahun Kim
Anelia Angelova
Google DeepMind
Google DeepMind
Abstract
We propose Context-Adaptive Multi-Prompt Embedding, novel approach to enrich semantic representations in vision-language contrastive learning. Unlike standard CLiP-style models that rely on single text embedding, our method introduces multiple structured prompts, each containing a distinct adaptive token that captures diverse semantic aspects of the input text. We leverage a pretrained LLM as the text encoder within the CLIP framework, processing all prompts jointly in a single forward pass. The resulting prompt embeddings are combined into unified text represen- tation, enabling semantically richer alignment with visual features. To further promote semantic diversity and representation quality, incorpo- rate a diversity regularization loss and a negation-aware loss, encouraging specialization across prompts and improving contrastive discrimination. Our method achieves consistent improvements on both image-text and video-text retrieval benchmarks.
1 Introduction
Contrastive vision-language models, such as CLIP (Radford et al., 2021), have become foundational for zero-shot image-text and video-text retrieval. These models align visual and textual representations by maximizing similarity between paired modalities. However, most approaches rely on single text embedding per input, which can limit the ability to capture the full range of semantic cues natural language descriptions.
Effective alignment between text and image or video often requires nuanced, multi-aspect matching. caption may describe the main subject, relevant objects, or background context, highlighting the need for expressive and semantically rich text embeddings. Pretrained Large Language Models (LLMs), especially decoder-only architectures like GPT (Achiam et al., 2023; Dubey et al., 2024; Team et al., 2024), offer strong semantic reasoning capabilities, making them promising candidates for text encoders in this setting.
However, using LLMs for CLIP requires careful adaptation. The causal structure of decoder- only LLMs limits their ability to summarize entire sequences using standard pooling strate- gies like first-token or mean pooling. To overcome this, recent methods have proposed prompt-based last-token pooling (Jiang et al., 2023a; Lei et al., 2024a), which guide the model to produce more meaningful representations by prompting it to summarize the input at the final token position. While these methods improve representation quality, they typically use single prompt or fixed prompt templates, limiting adaptability. This restricts their ability to capture diverse semantic views aligned with visual inputs. Meanwhile, retrieval tasks often require recognizing multiple aspects of visual-text pair, such as subjects, objects, or scene-level cues.
To address this, we propose Context-Adapticc Multi-Prompt Embedding, a method that introduces multiple structured prompts, each with a distinct adaptive token trained to specialize in a unique semantic aspect of the input. These prompts are processed by pretrained LLM, and their embeddings are combined into unified representation aligned with the visual embedding. To further enrich alignment, we introduce two additional learning objectives: a diversity regularization loss that encourages semantic diversity across

---

Published as a conference paper at COLM 2025
prompt embeddings, and negation-aware loss that introduces additional contrastive signals using negated prompt variants. Our method does not require additional text-only pretraining (e.g. SimCSE (Gao et al., 2021)) and generalizes well to both image-text and video-text retrieval. Extensive experiments on standard retrieval benchmarks demonstrate consistent performance gains over CLIP-style baselines and ablation variants, confirming the benefit of semantically diverse alignment in vision-language contrastive learning.
2 Related Work
2.1 Vision-Language Contrastive Learning
Contrastive learning has become a foundational approach for aligning visual and textual modalities, driven by the success of CLIP (Radford et al., 2021) and its variants. These models optimize contrastive losses over large-scale image-text or video-text pairs, enabling robust retrieval performance across domains. Subsequent works have explored architectural and training improvements (Li et al., 2021; Jia et al., 2021; Yu et al., 2022), as well as extensions to the video domain by incorporating temporal components into the dual- encoder architecture (Xu et al., 2021; Luo et al., 2022; Wang et al., 2022; Kim et al., 2025). While these methods are effective, they typically rely on simple text encoders. Our work enhances this framework by leveraging rich language representations from pretrained LLMs through multiple context-adaptive prompt embeddings, optimized within vision-language contrastive setting.
2.2 Decoder-Only LLMs for Text Embeddings
Decoder-only large language models (LLMs) such as GPT (Brown et al., 2020), LLaMA (Tou- vron et al., 2023), Gemini (Comanici et al., 2025) and Gemma (Team et al., 2024) have demonstrated impressive capabilities in generative tasks. However, leveraging them as effective encoders for text representation remains challenging due to their unidirectional causal attention mechanism. A common approach is to use the hidden state of the final token a as a sentence embedding (Neelakantan et al., 2022; Ma et al., 2023; Wang al., 2023), though this can yield suboptimal semantic summarization. Several studies have proposed archi- tectural adjustments to address this issue, such as relaxing the attention mask or enabling partial bidirectional attention in later layers (Li et al., 2023; Dukić & Snajder, 2024). Other works explore prompt-based strategies (Jiang et al., 2023b; Lei et al., 2024b; Zhuang et al., 2024; Zhang et al., 2024) to better guide the final token's 's representation. PromptFOL. (Jiang et al., 2023b) proposes summarization prompts such as "This image means just in one word:" to steer the final token representation toward semantic abstraction. MetaEOL (Lei et al., 2024b) expands this idea using diverse set of fixed task-oriented prompts. Additional strategies include Echo (Springer et al., 2024), which duplicates the input to simulate bidirec- tional context, and LLM2Vec (BehnamGhader et al., 2024), which enhances representations through hybrid attention during supervised contrastive learning. While these methods are designed for text-only embedding tasks, we extend this direction to vision-language contrastive learning by introducing multiple context-adaptive prompts that are learned to enhance semantic diversity and improve alignment with visual content.
2.3
LLMs in Vision-Language Contrastive Models
Recent efforts explore the integration of LLMs as text encoders in vision-language contrastive models. JinaCLIP (Xiao et al., 2024) and LLM2CLIP (Wu et al.) replace the CLIP text encoder with off-the-shelf LLMss such as Jina-v2 or OPT (Zhang al., 2022). These methods typically apply additional text-text contrastive learning such as SimCSE (Gao et al., 2021) before adapting to vision-language training. E5-V (Jiang et al., 2024b) uses a fixed prompt to extract LLM embeddings and fuses them with frozen vision features at the Multimodal LLM layer. While these works demonstrate that LLMs can be effective text encoders for CLIP-style tasks, most approaches rely on single fixed prompts require pretraining on additional text-text contrastive tasks. Our approach directly learns multiple context- adaptive prompt embeddings within vision-language contrastive learning. We further
2

---

Published as a conference paper at COLM 2025
Vision-text contrastivee learning
Prompt diversity regularization
Aggation-angmmenteddoneee learning
caption text
LLM
pj
p?
pe
Q
+
+
VIT
Lcon
Lpiv
LNeg
p
concat
Unfrozen last layer
LLM (causal prompt-wise masked attention)
caption-text
prompt-1
prompt-2
prompt-
negated-1
negated-2
negated-K
caption-text young boy looking two little dogs sicting in large sneakers.
prompt-i:: : the [APT-i] of this image means:
negated-i: the [APT-i] of this image does NOT
mean:
Figure 1: Overview of our method. Given input text, construct multiple structured prompts, each with a distinct adaptive prompt token [APT-i] (Sec. 3.2). The entire sequence processed jointly in single LLM forward pass via prompt-wise attention masking (Sec. 3.3). This yields embeddings of size D/K K, concatenated into a D-dimensional text embedding. A ViT encoder with attention pooling (Sec. 3.1) produces matching image or video embedding. Training combines contrastive loss (Lcon) with a prompt diversity regularization loss (Laing Sec. 3.4) and negation-awan loss (Lneg, Sec. 3.5) to encourage semantic variation and incorporate contrastive signals from negated prompts.
enhance this framework with prompt diversity regularization and negation-aware prompt embeddings, leading to more discriminative representations for vision-language alignment.
Method
We introduce Context-Adaptive Multi-Prommpt Embedding for vision-language alignment, novel approach designed to enhance semantic richness in vision-language contrastive learning frameworks such as CLIP. Our method leverages pretrained Large Language Models (LLMs) to extract multiple diverse semantic embeddings for each textual input, significantly improving semantic alignment with visual information.
3.1 Leveraging Pretrained LLMs for Text Embeddings
Standard CLIP models typically employ Transformer architectures with bidirectional at- tention (similar to BERT (Devlin et al., 2018)) for text encoding. These models prepend special token, such as [CLS), to the input text and utilize the hidden state of this token (first- token pooling) as the text embedding. In contrast, recent large language models (LLMs), such GPT variants, adopt decoder-only architectures with causal attention. Due to the unidirectional nature of causal attention, first-token pooling is ineffective as the embedding of the first token cannot summarize subsequent text.
To obtain high-quality embeddings from decoder-only LLMs, alternative pooling strategies are required. Last-token pooling is intuitively suitable for such models, as only the last position potentially contains information about the entire text. However, decoder-only models trained via next-token prediction inherently align the embedding at the last position primarily toward predicting subsequent tokens, rather than summarizing previous semantic content. To mitigate this, prampt-based last pooling methods, such PromptEOL, (Jiang
3

---

Published as a conference paper at COLM 2025
et al., 2023a), introduce explicitly structured prompts to guide the model toward semantic summarization at the last position. PromptËOL employs prompts such as: "[input_text]] This image means just in one word:", thereby instructing the model to succinctly sum- marize input text semantics. Similarly, MetaFOL. (Lei et al., 2024b) introduces multiple manually designed prompts to generate richer semantic embeddings.
Adopting pretrained LLMs as textual encoders in vision-language frameworks such as CLIP provides significant advantages. These LLMs have been extensively pretrained on vast textual datasets, equipping them with rich semantic knowledge and strong generalization capabilities, By leveraging such pretrained LLMs, our model can effectively transfer their deep semantic understanding to vision-text alignment tasks.
3.2 Context-Adaptive Multi-Prompt Embedding
Existing prompt-based embedding methods, such PromptËOL (Jiang et al., 2023a) and MetaFOL. (Lei et al., 2024a), rely on fixed, manually designed prompts to guide repre- sentation learning. While effective in text-only settings, these approaches limit flexibility when extended to vision-language contrastive learning. In vision-language tasks, textual embeddings are directly trained to align with visual features. This setup presents new opportunity: prompts can be learned and adapted during training to better reflect the semantic alignment between text and image or video content.
To leverage this, we introduce context-adaptiv multi-prompt embedding strategy that dynamically learns K distinct prompts during vision-text contrastive training. Each prompt contains an Adaptive Prompt Token [APT-iJ, special token whose representation is learned to capture distinct semantic aspects of the input text. These tokens are trained in the context of aligning textual and visual representations, enabling the prompts to specialize in diverse and complementary interpretations. The structured prompt follows the format:
" "[input_text]. The [APT-i] of this image means:"
Each prompt is processed by the pretrained LLM,, and we apply last-token pooling to extract single embedding per prompt. This embedding is then passed through projection layer to produce an embedding of size D/K, where D is the target embedding size that matches the CLIP visual embedding. The K projected embeddings are concatenated along the channel dimension to form a single text embedding of size D. This concatenation preserves the distinct semantics of each prompt while enabling specialized alignment with the visual representation. Since contrastive learning operates through element-wise dot product between text and visual embeddings, each prompt embedding is matched to specific channel segment of the visual embedding. This design encourages each [APT-i] to specialize and align with distinct visual-semantic concepts, as demonstrated later in Sec. 4.5.
Both text and vision embeddings are L2 normalized resulting in and respectively. The cosine similarity of the embeddings in batch B, scaled by a learnable temperature forms the input to the InfoNCE loss (Oord et al., 2018; Radford et al., 2021), defined as:
1 B
exp(piq/TT)
)
1
explaxP1/T) (xp1pp///T7/)))) (1)
LT21
log
,12T
log 1=1
B =
exp(pigj/z)
Σ expp(gip;/7)
The final contrastive loss is averaged as Leon (L + L/2
3.3 Efficient Prompt-Wise Attention Masking
Computing embeddings from multiple distinct prompts conventionally requires multiple forward passes through the LLM, which is computationally inefficient. To address this, we introduce prompt-wise attention masking, enabling efficient computation of all embeddings within a single forward pass. Specifically, we concatenate multiple prompts into one sequence. A representative concatenated prompt is:
[input text]. The [APT-1] of this image means:" [APT-2] of this image means:" [APT-K] of this image means:"
4

---

Published as a conference paper at COLM 2025
In this concatenated prompt, the initial prefix tokens (linput.text The) are globally acces- sible and shared among all prompt segments. However, the subsequent prompt-specific segments ([APT-i] of this image means:") are masked from attending to tokens in other segments [[APT-j] of this image means:"). Then, each prompt's embedding is independently computed from the last token (") position of each segment.
To further optimize training efficiency and model size, we freeze most layers of the pre- trained LLM, only unfreezing its last transformer layers.
3.4 Encouraging Prompt Diversity with Regularization
To explicitly promote semantic diversification among the prompt embeddings, we introduce a Diversity Loss Ldiv- Given K embeddings from distinct prompts, we compute pairwise cosine similarities among these embeddings. Specifically, we calculate K(K-1) similarity scores, averaging them to yield a single scalar representing the mean similarity. By mini- mizing this similarity, encourage each prompt embedding to capture unique semantic aspects effectively. The diversity loss is thus: Ldie K(K-1) Σizj CosSim( Emb;).
3.5 Negation-Aware Prompt Embedding
Vision-language models often struggle with distinguishing fine-grained semantic differences or explicitly negated scenarios, as typical training does not encourage embeddings to clearly represent these contrasts. Without explicitly guiding the model to differentiate between what the image does and does not represent, embeddings may become overly generalized or ambiguous. To enhance the discriminative capabilities of vision-language embeddings, we propose a Negation Prompt Embedding strategy. Our approach explicitly generates embeddings representing semantic concepts excluded or negated by the original prompt. Specifically, we construct structured negation prompts such as:
[input text]. The [APT-i] of this image does NOT mean:"
This explicit negation approach aims to encourage embeddings that explicitly capture semantics irrelevant to diverging from the original input. We create negation prompts corresponding directly to each of the K original prompts, resulting in K additional negation embeddings [n:}. Each negation embedding undergoes dimensional projection D/K and are then concatenated, forming single negation embedding of size D.
Specifically, for each vision embedding 9i, we contrast both the original text embeddings Pi, and their corresponding negation embeddings n1, This effectively doubles the number of text embeddings in the contrastive training. The negation-aware contrastive loss Lneg
(image-to-text only) is computed as: Lneg = E log (expian///)teppiiaao.//.
exp(gips/z)
Consequently, this encourages the vision-language model to differentiate between accurate and negated semantic content, improving semantic discrimination and retrieval accuracy.
3.6 Training Objective
Our total loss function integrates the standard CLIP contrastive loss with the diversity loss and negation-aware loss: Lota! Leon + Lgic + B Lneg- We set the coefficients a and / to 0.1. Our experiments show this combined objective further improves semantic alignment.
3.7 Video-Text Contrastive Learning
To extend our approach to video-text alignment, we adapt the vision encoder to handle video inputs while keeping the text encoder unchanged. Each video frame is independently processed by the ViT encoder, yielding token embeddings of shape T is the number of frames and N is the number of tokens per frame. To incorporate temporal information, we add learnable temporal positional encodings of shape 1 x 1 before flattening the frame-wise tokens into sequence of TN D. These then
N x D, where
5

---

Published as a conference paper at COLM 2025
passed through the same attention pooling layer used in the image encoder, resulting in a single video-level embedding of dimension D. The text encoder continues to leverage our multi-prompt embedding mechanism, allowing it to produce rich, context-adaptive representations aligned with the video content.
Experiments
We evaluate our context-adaptive multi-prompp embedding method on image-text and video-text retrieval tasks. Our experiments include comparisons with CLIP baselines and ablation variants to demonstrate the contribution of our approach.
4.1 Implementation Details
We train contrastive image-text models from scratch using ViT-B/16 as the image encoder (D 768) and LAION (Schuhmann et al., 2022) as the training dataset. Images are resized to x24x224, with visual embeddings obtained via attention pooling. The vanilla CLIP uses a 12-layer Transformer text encoder, following standard practices of Radford et al. (2021); Yu al. (2022). In our method, we replace the text encoder with the pretrained Gemma 2B LLM (Team et al., 2024), freezing all layers except the last L Transformer layers. We use L=2 for Gemma 2B backbone. Due to the model's causal attention, we apply last-token pooling with structured prompts. The output of the LLM text encoder is first passed through linear projection layer to reduce its feature dimension
D/K, where is the embedding dimension of the VIT visual encoder and K is the number of adaptive prompts. We use A with a learning rate of 5e-4, linear warmup for 10k steps, training for 500k iterations. We use batch size 1024 unless otherwise noted. For video-text training, we initialize from the pretrained image-text CLIP and finetune on VideoCC3M (Nagrani et al., 2022) for 50k steps with learning rate of 1e-5, batch size 128, and 16 uniformly sampled frames per video. We evaluate under zero-shot settings on Flickr30K (Plummer et al., 2015) and MSCOCO (Chen et al., 2015) for image-text retrieval, and MSR-VTT (Jun Xu & Rui, 2016) for video-text retrieval.
Ablation Studies
We perform several ablation experiments to demonstrate the contribution of our context- adaptive prompting strategy. We use Gemma-2B text encoder and batch size 1024 unless otherwise noted.
Baseline Comparisons. As shown in Table 1, replacing the vanilla CLIP text encoder with fully frozen pretrained LLM leads to a substantial drop in retrieval performance, likely due to insufficient adaptation to the vision-text alignment objective. Allowing partial adaptation by unfreezing the last transformer layers of the LLM helps reducing this gap. Additional gains are observed by applying prompt-based last-token pooling using the template [input_text]. This image means: Our method also builds on this prompt- based last pooling approach, extending it with multiple context-adaptive prompts.
Number Prompts (K). Introducing our context-adaptive multi-prompt embedding leads to clear performance improvements. Our method uses structured prompts of the form "[input_text]. The [APT-i] of this image means: where each adaptive prompt token ([APT-i]] dynamically captures distinct semantic aspects. As shown in Table 2, retrieval accuracy significantly increases as the number of prompts grows to 6. We select K=6, since further increases yield minimal additional benefit.
Prompt Design and Text Embedding Construction. Table 3 presents ablations on prompt structure and embedding construction strategies. All variants use K=6. Using single shared [APT] token across all K prompts leads to redundant representations and yields limited performance similar to using single prompt (K=1). Employing K manually crafted fixed prompts (e.g. [input text]. The main category of the image means:", " or [input text]. The primary object in the image means: ") improves performance, but remains suboptimal

---

Published as a conference paper at COLM 2025
Flickr R@1
MSCOCO R@1
backbone
pooling
img-to-txt
txt-to-img
img-to-txt
txt-to-img
Vanilla CLIP
mean
61.4
43.7
38.3
24.0
LLM-frozen
last
41.4
28.6
21.6
13.4
LLM-unfrozen last layers
60.6
43.4
36.6
23.4
LLM-unfrozen last layers
last
54.0
37.8
32.6
20.3
LLM-unfrozen last layers
prompt-last
54.5
38.0
32.8
20.8
Table 1: Baseline image-text retrieval performance. We compare vanilla CLIP and CLIP variants with pretrained LLM text encoder.
Flickr R@1
MSCOCO R@1
Prompts (K)
img-to-txt
txt-to-img
img-to-txt
txt-to-img
54.7
38.1
32.8
20.9
60.3
42.1
36.7
23.0
66.0
47.1
41.0
25.2
66.1
47.1
40.8
25.3
Table 2: Effect of the number adaptive prompts (K).
Flickr R@1
MSCOCO R@1
method
ing-to-txt
txt-to-img
img-to-ixt
bxt-t--im
Shared [APT] token
54.6
38.3
33.0
20,8
Fixed prompts
64.1
44.7
39.7
24.3
Minimal prompts
63.0
43.5
38.9
24.1
Averaged K embeddings
56.8
39.2
35.1
22.3
Context-ade adaatiev prompts (Ours)
66.0
47.1
41.0
25.2
Table 3: Ablation prompt design and embedding construction. All methods K=6.
compared to our adaptive prompts. We further evaluate minimal variant that removes contextual phrasing, using simplified format: [input_text]. [APT-i]:". Although this setup performs competitively, it underperforms the full structured version, highlighting the utility of explicitly guiding the LLM to extract "[APT-i] of the image". We also study variant where K adaptive prompt embeddings (each of size instead of D/K) are averaged element-wise instead of concatenated. While this approach brings some improvement over single prompt, it still underperforms our method. In contrast, our channel-wise concatenation of K adaptive prompt embeddings achieves the best performance. This design allows each [APT-i] to specialize and align with distinct channel segment the visual embedding, encouraging semantically diverse alignment further shown in Sec. 4.5.
Prompt Diversity Regularization. We study the effect of our diversity regularization loss Ldir in Table 4. Introducing moderate diversity regularization improves retrieval performance, demonstrating the benefit of explicit encouraging diversity among adaptive prompts. We set the regularization weight 0.1, higher values do not yield additional improvements.
Negation-Aware Embedding and Loss Combination. In Table 5, we evaluate the impact of incorporating negation-aware embeddings and the loss Lneg. Adding this negation embedding improves retrieval performance, showing that explicitly modeling semantic negation improves semantic discrimination. Combining both negation embedding and diversity regularization further boosts performance, achieving our best overall results.
LLM Text Encoder Size. Table 6 presents the effect increasing the size of the LLM backbone from Gemma-2B to Gemma-9B. Using larger pretrained language model clearly improves retrieval results.
Contrastive Batch Size. Table 7 studies how method scales by varying contrastive batch sizes. our method consistently outperforms the vanilla CLIP baseline across different batch sizes (1024 and 4096). Increasing the batch size to 4096 clearly improves retrieval
7

---

Published as a conference paper at COLM 2025
Flickr R@1
MSCOCO R@1
method
img-to-txt
txt-to-img
img-to-txt
txt-to-img
(w/o Ldio)
66.0
47.1
41.0
25.2
66.8
48.2
41.7
25.9
a 1.0
65.8
47.3
41.2
25.6
Table 4: Token diversity regularization loss (Laiz).
Flickr R@1
MSCOCO R@1
method
img-to-txt
bxt-to-img
img-to-txt
txt-to-img
Toon Leon & Lneg I Leon & did & Lueg
66.0
47.1
41.0
25.2
67.2
48.0
41.8
26.0
68.3
48.6
42.3
26.4
Table 5: Negation-aware embedding (Lueg) and loss combination.
Flickr R@1
MSCOCO R@1
backbone
img-to-txt
txt-to-img
img-to-txt
txt-to-img
Gemma-2B
68.3
48.6
42.3
26.4
Gemma-9B
70.3
52.7
44.8
27.6
Table 6: Effect of text encoder
performance, highlighting our method's capability to effectively leverage large batch size in contrastive learning.
Number of Trainable LLM Layers. Table 7 also examines the effect of the number of unfrozen LLM layers (L), with and without the learnable vocabularies (tokenizer weights). On top of unfreezing few last layers, making the vocabulary learnable provides additional improvements. This shows that allowing more of the LLM to adapt to the contrastive objective is beneficial, though balance must be struck with computational cost.
4.3 Image-Text Retrieval Results
We further evaluate the scalability of our method on image-text retrieval by increasing the contrastive batch size to 16384. For reference, OpenAl CLIP (Radford et al., 2021) leverages an even larger batch size of 32768. As shown in Table 8, our method continues to benefit from this scaling, achieving stronger performance than baseline models. We also include comparisons with other ViT-B-based CLIP methods, highlighting the effectiveness of our approach to capture diverse semantic signals for vision-language alignment.
4.4 Video-Text Retrieval
We extend our approach to video-text retrieval by initializing from the image-text pretrained model (with Gemma-2B as text encoder) and further training on the VideoCC3M (Nagrani et al., 2022) dataset. We evaluate a zero-shot setting. As shown in Table 9, our method brings substantial improvements compared the vanilla CLIP baseline, demonstrating its effectiveness in aligning video and text through semantically diverse prompt embeddings.
the MSR-VTT (Jun Xu & Rui, 2016) benchmark under
4.5 Attention Visualization
Our text embedding is constructed by concatenating K adaptive prompt embeddings along the channel dimension. During contrastive training, the textual and visual embeddings are matched through element-wise dot product, aligning each prompt embedding with corresponding channel segment of the visual embedding, To better understand how these adaptive prompts evolve, visualize attention patterns from the final attention pooling
8

---

Published as a conference paper at COLM 2025
batch size
unfrozen layers (L)
Learn. vocab.
Flickr R@1
MSCOCO R@1
ZS INet
img-to-txt
txt-to-img
img-to-txt
Ext-to-img
Top-1 acc.
1024
Vanilla CLIP
61.4
43.7
38.3
24.0
53.1
1024
N
46.5
32.2
27.6
17.7
44.2
1024
68.3
48.6
42.3
26.4
52.4
1024
Y
65.1
46.2
39.1
24.5
52.3
1024
Y
66.6
47.9
42.5
26.3
53.5
1024
69.1
49.6
43.5
27.1
54.0
1024
75.4
53.4
48.3
29.6
55.9
4096
Vanilla CLIP
81.0
60.2
51.5
34.8
67.3
4096
Y
81.3
63.9
54.6
36.1
67.7
4096
84.3
66.2
60.4
39.4
68.1
Table 7: Effect of contrastive batch size, number of unfrozen last layers (L), and learnable vocab- ularies. K= 6 used.
Flickr R@1
MSCOCO R@1
method
img-to-txt txt-to-img
img-to-txt txt-t--img
OpenAI CLIP-B (Radford et al., 2021)
81.9
62.1
52.4
33.1
LongCLIP-B (Xiao et al., 2024)
85.8
70.6
56.9
40.9
E5-V (liang et al., 2024a)
79.5
67.8
51.6
41.2
JinaCLIP-B (Xiao al., 2024)
80.6
67.4
55.6
41.1
Ours
84.7
68.7
58.5
41.4
Table 8: Zero-shot image-text retrieval and classification. Batch size 16k, K=6, L=2 is used.
MSR-VTT R@1
method
text-to-video
video-to-text
OpenAl CLIP-B B (Radford et al., 2021)
23.3
43.3
SocraticModell-B (Zeng et al., 2022)
46.9
CLIP4Clip (Luo et al., 2022)
32.0
VideoCoCa-B (Yan et al., 2022)
31.2
Vanilla CLIP (Sec. 3.7)
31.6
45.1
Ours (Sec. 3.7)
35.8
48.7
Table 9; Zero-shot video-text retrieval.
layer of the VIT encoder. By dividing the visual embedding channels into K segments, obtain K attention maps, each averaged over all attention heads its segment. We use 384x388 384 input images x24 resolution attention maps) for visualization. We observe that different adaptive prompts focus on different semantic areas: some prompts emphasize subjects (e.g. [AP'I-1]) others highlight relevant objects (e.g. [APT-2[]), and some capture broader contextual background elements.
Conclusion
We presented Context-Adaptive Multi-Prompt Embedding, a novel approach for enhancing textual representations in vision-languape contrastive learning. Our method leverages pretrained LLMs to generate multiple prompt-guided embeddings that dynamically adapt during training, allowing for richer semantic alignment with visual content. By introduc- ing context-adaptive prompt tokens, prompt diversity regularization, and negation-aware embeddings, improve the model's ability to capture discriminative semantics. Ex- tensive experiments on image-text and video-text retrieval benchmarks demonstrate the effectiveness of our approach.

---

Published as a conference paper at COLM 2025
Image
[APT-1]
[APT-2]
[APT-3]
[APT-4]
[APT-S]
[APT-6]
Figure 2: Attention maps from the visual encoder corresponding to each adaptive prompt token [APT-i]. For each input image (row), we visualize the atfention from the ViT attention pooling layer segmented by the K=6 prompt-specific embedding channels (columns). Each map shows how a specific APTTi attends to different spatial regions such as objects, subjects, or background, reflecting diverse visual-textual alignments.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPTT-4 technical report. arXiv preprint ar arXio:2303.08774 2023.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. Xiv preprint ar arXiv:2404.05961, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Jared Kaplan Melanie Subbiah, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vorr Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Christopher Hesse Clemens Winter, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. In https://arrieeorra////5554. 000225, 2015.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXin:2507.06261 2025.
10

---

Published as a conference paper at COLM 2025
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXio e-prints, art. arXiv:1810.04005, October 2018. doi: 10.48550/ arXiv.1810.04805.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. Xiv preprint srXiv:2407.21783, 2024.
David Dukić and Jan Snajder. Looking right is sometimes right: Investigating the capabilities of decoder-only Ilms for sequence labeling. arXiv preprint, 2024. URL.https:/arxiw.orrg abs/2401. 4556.
T Gao, Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In EMNLP 2021-2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp- 4904-4916. PMLR, 2021.
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. 2023a. URL https://arxiii http://aaxxvvv..raabbbba org/abs/ 2307. 16645.
Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. Scaling sentence embeddings with large language models. arXiv preprint ar Ntf::2307.1664555 2023b.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. ArXiv, abs/ 22//2240772220 2024a. URL https: //api. semanticscholar.
org/CorpusID::71222 271244554
Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arviv preprint ar xxXi:2247.12580, 2024b.
Ting Yao Jun Xu, Tao Mei and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
Dahun Kim, AJ Piergiovanni, Ganesh Mallya, and Anelia Angelova. Videocomp: Advancing fine-grained compositional and temporal alignment in video-text models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29060-29070, 2025.
Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. Meta-task prompting elicits embeddings from large language models. 2024a. URL https://arxivv orgg/bs//44e 18458.
Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. Meta-task prompting elicits embeddings from large language models. arXiv preprint arXio:2402.18458, 2024b.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.
Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu lee Wang, Qing Li, and Xiaoqin Zhong. Label supervised 1lama finetuning. arXia preprint, 2023. URL https: //arxiv.org/ abs/2310 01208.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocom- puting, 508:293-300 2022.
11

---

Published as a conference paper at COLM 2025
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning LLaMA for multi-stage text retrieval. arv preprint, 2023. URL /tars://arxiv... httpi//..rrgab///////// http//abb//11.0/////3 08319.
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. In ECCV, pp. 407-426. Springer, 2022.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training. arviv preprint, 2022. URL https:/ htaps://arxiii /ttp//aii...ggbb///////////aaaat 2201. 10005.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ariiv preprint ar Sttt:1807.037445 2018.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting eegion-to-praass correspondences models. In ICCV, pp. 2641-2649, 2015.
for richer
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25291 2022.
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. ar arviv preprint ar Xiv:22402. 15449, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar 2rXi:23302.33971, 2023.
Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video- language pre-training arXiv preprint ar XrXi:2203.07303 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint, 2023, URL
http:://aaxiv..or/abb////ss
http:/abb/221..00330 00368.
Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Chunyu Wang, Liang Hu, Xiyang Dai, Dongdong Chen, Chong Luo, Lili Qiu, et al. Llm2clip: Powerful language model unlock richer visual representation. In NeurIPS 2024 Workshop: Self-Supervised Learning-Theor and Practice.
Han Xiao, Georgios Mastrapas, and Bo Wang. Jina clip: Your clip model is also your text retriever. In Multi-modal Foundation Model meets Embodied AI Workshop@ ICML,2024, 2024.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. ar Xiv preprint arXi::2200911 1084,, 2021.
12

---

Published as a conference paper at COLM 2025
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners. arvio preprint arXi::2212.0499,, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022. URL
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint (rXio:2204.00598, 2022.
Bowen Zhang, Kehua Chang, and Chumping Li. Simple techniques for enhancing sentence embeddings in generative language models. In International Conference on Intelligent Computing, pp. 52-64. Springer, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. Xiv preprint arXto:2205.01068, 2022.
Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Promptreps: Prompting large language models to generate dense and sparse representa- tions for zero-shot document retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4375 4391, 2024.
13


Reviews:
[{"review_id": "VAQm7r8fVn", "reviewer": "Reviewer_TY9F", "summary": "This paper introduces \"Context-Adaptive Multi-Prompt Embedding\" to align visual data with textual descriptions. Unlike existing methods like CLIP, the method introduces multiple structured prompts, where a distinct adaptive token captures diverse semantic aspects of the input text. A pretrained LLM is used to process all the prompts all at a single pass. With the additional regularization loss for diversity and a negation-aware loss, the proposed method achieves consistent improvements on image-text and video-text retrieval tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of using LLM to process both positive and negative prompts is interesting.\n2. The paper is overall well presented and easy to follow.", "reasons_to_reject": "1. The major of concern of the reviewer is on lack of proper baseline comparisons. There has been highly relevant work [a,b] on either formulating the decomposition of a prompt into fine-grained embeddings like [a] or leverage LLM to improve vision-language model [b] that needs to be more discussed and compared. For example, baselines like multiple fine-grained embeddings could be constructed in [a]'s manner through text prompts; baselines like using LLM to possibly generate more positive/negative descriptions can be provided to validate that current approach is indeed an ideal way of using LLM in vision-language representation learning.\n\n2. Currently vanilla CLIP has a much smaller text encoder compared to the proposed one. What if use the same Gemma model as initialization and perform PEFT for it in CLIP's setup?\n\n3. Could the proposed design bring additional different usage in inference time? This could have been explored more.\n\nPost-rebuttal:\nThe reviewer increases the score as the new baselines comparisons addressed the concerns.\n\n\n[a] Learning to Decompose Visual Features with Latent Textual Prompts\n\n[b] Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts", "questions_to_authors": "Please check the concerns above. The reviewer is open to increase score if the concerns can be addressed.", "extra_scores": {}}, {"review_id": "GM7VBRa0Na", "reviewer": "Reviewer_L9Jh", "summary": "In the context of image-text retrieval, this paper proposes a text embedding method using multiple prompts containing special tokens that are combined to produce a text embedding (e.g., \"the [SPECIAL TOKEN] of this prompt means:\"). In contrast to prior methods that use static prompts, this approach allows a significant part of each prompt to be learned. The approach is trained using contrastive learning that includes negated embeddings (\"... does NOT mean:\") and combined with a diversity loss based on the pairwise cosine similarity between embeddings. Results on common image-caption retrieval benchmarks indicate that the proposed approach outperforms comparable CLIP baselines, despite being trained with a much smaller batch size.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an interesting approach that addresses a shortcoming in prior work (i.e., prompts are static). Experiments demonstrate that this approach is effective for image-caption retrieval. The paper is generally easy to follow and likely could be reproduced without too much effort, though this would be easier if code and weights were made available.", "reasons_to_reject": "While the proposed approach does outperform CLIP, the results are not particularly strong compared to SigLIP (or SigLIP2) and it is not clear why the approach was applied to image-text retrieval specifically, given that the prior work used as motivation considered text only. While this is a challenge setting, the motivation of moving from static to dynamic prompts would also apply to text.", "questions_to_authors": "- How does the performance of prompt-wise attention masking compare to batching when using prefix caching? Is there some factor that should make the masking more efficient?", "extra_scores": {}}, {"review_id": "N4D8TUGHim", "reviewer": "Reviewer_JaD2", "summary": "This paper introduces Context-Adaptive Multi-Prompt Embedding, a new method for enhancing semantic representations in vision-language contrastive learning. Departing from traditional CLIP-style models that rely on a single text embedding, the proposed approach uses multiple structured prompts, each with an adaptive token designed to capture distinct semantic aspects of the text. These prompts are processed in a single forward pass using a pretrained LLM via prompt-wise masked attention. The resulting embeddings are combined into a unified text representation, allowing for more semantically rich alignment with visual inputs. To further enhance diversity and representation quality, the method incorporates a diversity regularization loss and a negation-aware loss with negated prompts, which encourage semantic specialization among prompts and improve contrastive learning performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces a multi-prompt framework with adaptive tokens, enabling finer-grained semantic modeling than single-prompt approaches, which leverages prompt-wise masked attention in a single forward pass, making the method efficient while maintaining expressive power.\n\n2. The proposed method incorporates both diversity regularization and negation-aware loss, boosting semantic discrimination and contrastive learning effectiveness.", "reasons_to_reject": "1. The proposed method introduces additional computational cost due to the forward pass through the LLM, which is significantly heavier than CLIP. This overhead should be quantified and compared to baseline models to assess its feasibility in practical settings.\n\n2. The current evaluation is limited to a single LLM variant (Gemma). To demonstrate generalizability and robustness, the method should be tested with a wider range of base models, such as LLaMA, Qwen, and other widely used LLMs.", "questions_to_authors": "Could you provide a comparison of the computational costs (e.g., memory and FLOPs) between your proposed method and CLIP-based methods?\n\nCould you evaluate your method using other base models, such as LLaMA or similar LLMs, to demonstrate its robustness and generalizability across different architectures?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The core idea of the paper is to move beyond single-text embeddings (like those in standard CLIP models) by using multiple structured prompts, each with a distinct adaptive token. These tokens are designed to capture diverse semantic aspects of the input text. The inclusion of both diversity regularization loss and negation-aware loss effectively encourages specialization across prompts and improves contrastive discrimination, leading to richer and more robust representations.

 The authors provided thorough rebuttals that addressed the initial concerns regarding baseline comparisons, computational cost, and generalizability to different LLMs. These updates should be included in the revised paper.
Decision: Accept