Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement
Authors: Yu-Che Tsai
Kuan-Yu Chen
Yuan-Chi Li
Yuan-Hao Chen
Ching-Yu Tsai
Shou-De Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Text embedding
LLM
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Existing large language model (LLM)-based embeddings typically adopt an encoder-only
paradigm, treating LLMs as static feature extractors and overlooking their core gener-
ative strengths. We introduce GIRCSE (Generative Iterative Refinement for Contrastive
Sentence Embeddings), a novel framework that leverages autoregressive generation to iter-
atively refine semantic representations. By producing sequences of soft tokens optimized
under a contrastive objective, GIRCSE captures latent concepts and implicit semantics
that encoder-only methods often miss. To guide this process, we propose an Iterative
Contrastive Refinement (ICR) objective that encourages each refinement step to yield bet-
ter representations. Extensive experiments show that GIRCSE outperforms strong LLM-
based embedding baselines on the MTEB embedding benchmark. Moreover, GIRCSE ex-
hibits an emergent test-time scaling property: generating more tokens at inference steadily
improves embedding quality. Our results establish generative iterative refinement as a new
paradigm for representation learning.

Published as conference paper ICLR 2026
LET LLMs SPEAK EMBEDDING LANGUAGES: GENERATIVE TEXT EMBEDDINGS VIA ITERATIVE CONTRASTIVE REFINEMENT
Yu-Che Tsai
Kuan-Yu Chen
Yuan-Chi Li Yuan-Hao Chen' Ching-Yu Tsai
Shou-De Lin
DDppartment of Computer Science and Information Engineering, National Taiwan University
2 Taipei, Taiwan
National Taiwan University AI Center of Research Excellence
{ (f09922081, d13922034,44 (001992200400992110000 b09902110, 01119921110 r192000 sdl ntw.eieen........ .ed..tw
ABSTRACT
Existing large language model (LLM)-based embeddings typically adopt an encoder-only paradigm, treating LLMs as static feature extractors and overlooking their core gener- ative strengths. We introduce GIRCSE (Generative Iterative Refinement for Contrastive Sentence Embeddings), novel framework that leverages autoregressive generation to iter- atively refine semantic representations. By producing sequences of soft tokens optimized under contrastive objective, GIRCSE captures latent concepts and implicit semantics that encoder-only methods often miss. To guide this process, we propose an Iterative Con- trastive Refinement (ICR) objective that encourages each refinement step to yield better representations. Extensive experiments show that GIRCSE outperforms strong L.M-- based embedding baselines the MTEB benchmark and follloii tasks. Moreover, GIRCSE exhibits an emergent test-time scaling property: generating more to- kens at inference steadily improves embedding quality. Our results establish generative iterative refinement as a new paradigm for representation learning. Our code and pre- trained models are available at athttps: bortps://githus .tt.//oyyaaa22///IRC...
INTRODUCTION
Text embeddings are fundamental to a wide range natural language processing (NLP) applications, in- cluding information retrieval, semantic search, clustering, and recommendation (Krrpkkhnn al. 2020 Liu jet al. 2024 Xiong et al., 2021). With the rise large language models (LLMs), representation learning has advanced further: fine-tuning LLMs large corpora now yields superior performance several embedding benchmarks (Tao et al. 2024).
However, current LLM-based embeddings typically operate single-pass feature extractors: embeddings are extracted in a single forward pass with contrastive learning objectives, without leveraging the gener- ative capacity of LLMs. This overlooks key strength of pretrained LLMs- their ability to reason and iteratively refine through autoregressive generation (Wei et al. 2022] Muennighoff et al. 2025). This raises a fundamental question: Can LLM-based embedding models also benefit from iterative generation? We hypothesize that generation enables iterative refinement of embeddings, allowing models to progressively consolidate semantics over multiple steps rather than encoding all semantics single pass.
"Equal contribution.

---

Published as a conference paper ICLR 2026
Performance improves through longer reasoning
?
Represent the emotion of this text.
(CATT
The emotiio
this
test
Is
Static and unexpandable
Why hard track down this card?
Em beddingg LLM
.000 Hidden States
C Text Embedding
Embedding Quality benefit from iterative generation
Ostpels somanck reprosentatio
GIRCSE: Generative Embedding
Soft Token Generation
Iterative Controstive Refinement (ICR)
Expands beyond surfoce-level diccmmiinaeive cmb
taken semantic information and retnin grodient flow
Quality Imorovez with
4
L
L
Saft Tokken Diamrbuttonn
maggle
LM
frustvated
Generative Pooling
Why
Token Emb. Toble
Ieport Taken
Saft Token
M.S. isput Token
LLM
LUM
LLM
of soft Token
00
Fest Ereöueddigg
Weighted sum
Seft Tokeo Gun.
LLM
Shared LLM dnc.
Figure 1: Top: Comparison between embedding LLMs that extract static representations and generative LLMs that can iteratively refine through reasoning. Bottom: Overview of GIRCSE. Our framework com- bines Soft Token Generation and Iterative Contrastive Refinement to enable end-to-end generative training.
Challenges. Designing effective generative embeddings presents several challenges. First, naive genera- tion degrades embedding quality since pretrained LLMs are optimized for fluent text, not tokens aligned with semantic similarity (see (Section 5.1p. Second, unlike traditional language modeling, there is no clear generation target: it is unclear what content the model should generate to obtain universally useful embed- dings. Third, existing embedding learning frameworks do not accommodate multi-step generative refine- ment. Therefore, it is necessary to develop new training paradigms that provide meaningful supervision for generative embeddings.
Motivation. We argue that LLMs should learn to speak an embedding language: generating tokens not constrained by human readability but optimized for semantic representation. Crucially, these tokens should be discovered through end-to-end training jointly with contrastive objectives, enabling the model to generate semantically meaningful tokens for iterative embedding refinement.
Building on this motivation, we propose tence Embeddings-a novel framework that bridges this gap between generative LLM capabilities and em- bedding optimization. GIRCSE consists of two major innovations: (1) Soft Token Generation preserves differentiability for end-to-end contrastive training and captures richer semantics by retaining the diversity of the full probability distribution. (2) Iterative Contrastive Refinement (ICR) provides contrastive super- vision at every generation step, forcing the early generated tokens to capture useful semantics while later tokens progressively refine representations. As illustrated in Fig. I and detailed in Section 5.2) this paradigm enables GIRCSE to generate instruction-an awaaee refinement tokens (e.g.." "frustrated" and x "struggle") that cap- ture the implicit emotion beyond the surface text. In summary, make the following contributions:
Iterative Refinement for Contrastive Sen-
Novel embedding framework. We propose GIRCSE, novel end-to-end framework that integrates toregressive generation with contrastive learning. Unlike prior methods, GIRCSE generates soft refinement tokens without explicit targets, progressively distilling semantics into high-qualit, embeddings.
2

---

Published as conference paper ICLR 2026
Superior performance. We compare GIRCSE with 18 state-of-the-art embedding models. By generating only up to 5-20 additional tokens, GIRCSE ranks within top 5-6 on MTEB and top 2-3 on instruction following, leading to the best overall ranking across benchmarks. Meanwhile, GIRCSE consistently shows stable improvements over reproduced fair baselines on different backbone and varying data scales.
Test-time scaling ability for text embedding. We demonstrate that GIRCSE exhibits consistent embed- ding quality improvements with increased refinement steps inference time, representing a novel scaling paradigm for embedding models analogous to test-time compute scaling in reasoning LLMs.
2 RELATED WORK
Early Embedding Models. Text embedding methods have evolved from traditional word-level repre- sentations to sophisticated neural approaches. Early methods like Word2Vec (Mikolov et al.) 2013) and GloVe (Pennington et al., 2014) captured basic semantic relationships but lacked contextual understand- ing. The introduction of transformer- based models marked significant advancement, with BERT-based approaches like Centence-BERY) (Reimers & Gurevych 2019) and SimCSE (Gao et al. 2021 establishing contrastive learning as the dominant paradigm for learning sentence representations.
LLM-based Embedding models. Recent works have successfully adapted LLMs for representation learn- ing through various architectural and training modifications. E5-mistral (Wang et al., (024) is one of the early works that demonstrated that fine-tuning I.I.M could significantly outperform early stage meth- ods. Recognizing that the unidirectional attention mechanism in LLMs may limit text embedding quality, LLM2Vec (BehnamGhader et al.) (224) introduces a bidirectional attention mechanism combined with av- erage pooling enhance embedding quality. NV-Embed (Lee et al. 2024) further improves the pooling method by incorporating an additional Latent Attention Layer and implements a two-stage training strategy address the challenge of false negatives in non-retrieval tasks. BGE-en-icl (Li al., (0255) suggests that retaining the original framework of LLMs and leveraging in-context learning is the optimal approach for generating text embeddings.
Towards Generative Text Embedding. A smaller line of research has explored generative approaches for text embeddings. For example, Inbedder (Peng al. 2024) combines instruction finetuning with token generation, achieving strong performance on instruction- woltooon---oowiinng tasks but showing limited generalization broader tasks (see Table 2). As summarized in Table most existing approaches differ only pooling strategies or auxiliary training techniques, while generative embeddings remain largely underexplored.
Table I:
Comparison of LLM-based embedding models. "Bidir." indicates bidirectional attention, "GP" means generated tokens pooling, "TTS" refers to test-time scaling capability.
Method
Attention
Pooling
LoRA
Generation
TTS
Gen. Token
Training Ohj.
ES-MRRTAAL
EOS
sup.
FER-EMMEDDING
Cansal
EOS
x
X
NIA
sup. CL
BGE-EN-ICL
Causal
EOS
X
NIA
ICL & sup. CL
GRITLM
Bidir.
Avg.
X
NIA
NTP sup.
LLM2VEC
Bidir.
Avg.
X
X
NIA
MLM & sup. CL
GTE-QWEN2
Bidir.
Avg.
X
N/A
unsup. CL & sup. CL
NV-EMBED-V1
Bidir.
LAT
x
X
N/A
Two-stage sup. CL
INBEDDER GIRCSE
Causal
GP
/
Hard Token
Instruction Tuning sup. CL & ICR
Causal
GP
Soft Token
3

---

Published as conference paper ICLR 2026
3 GIRCSE: FROM DISCRIMINATIVE TO GENERATIVE EMBEDDING
first establishes core toregressive embedding generation process, introducing the fundamental concepts and notation, [Section 3.2 then details the soft token generation mechanism that enables differentiable optimization within this frame- work. Finally, Section 3.3 presents our iterative contrastive refinement objective, guiding the model towards progressively higher-quality representations.
We now detail our proposed generative embedding framework. Section 3.1
3.1 GENERATIVE EMBEDDING FRAMEWORK
We consider a pretrained language model with parameters d (E. where E E R| is the token embedding matrix, 9 denotes the parameters of the Transformer decoder, and corresponds to the parameters of the LM head for next token generation. Here d is the embedding dimension and [V] is the vocabulary size. Given an input sequence embeddings X as:
tokens T
from vocabulary V, we first obtain token
X = (E(4), El(2).... E((N]) E RN
Next, our goal is to autoregressively generate a sequence of K auxiliary soft tokens S = (s1. S2. sk) & x|V| that iteratively refines the representation space. Each soft token Sy is generated autoregressively conditioned on the input sequence and previously generated tokens:
K
(S T)
II k=1
(s
T, Scx),,
(2)
where P& represents the generative distribution and S = (si...., sk-1) are the previously generated soft
tokens. The soft tokens are then mapped into embedding space producing D = (di....,dy, dx) E R'd and is concatenated with input embeddings X further feed into the Transformer decoder fo:
H = ([X: = (h,
(L) hn
NNK) (N+K))xd
(3)
where h24)) denotes the hidden state of the i-th token at the final (i.e., I--h) layer. We then extract the representations corresponding to the generated soft tokens, and aggregate them into a single representation z via a pooling operation:
P(G)
(L) g: G (g gg...... gk) (hN4
(2) b Rxd
(4)
where P denotes general pooling function, with pooling default choice.
Computational Considerations. Since our approach involves iterative generation with steps, naturally incurs higher computational cost compared to single-step baselines (Appendix E}. However, find that generating only a small number of tokens (e.g., K = 5 or 10) is sufficient to achieve strong performance (Section 5.1). Moreover, this cost could be largely mitigated via KV caching (Appendix D), where the FLOPs are reduced to nearly the same level as standard embedding models (within 11.0-111x)
3.2 SOFT TOKEN GENERATION
critical challenge our autoregressive framework is to maintain the differentiability throughout the gen- eration process during training. Traditional discrete token sampling would break gradient flow, preventing
We defer the detailed soft token generation mechanism to Section 3.2 while here focus the overall framework
4

---

Published as a conference paper ICLR 2026
Algorithm 1: GIRCSE: Autoregressive Generative Embedding
Input: Input tokens (f1..... it), generation steps K, embedding matrix E, LLM decoder fo, LM head parameters We and by. pooling function P
Output: Final embedding R
X [[tt]...... E(ta])
// Embed the input tokens 11 Initialize the list of generated embeddings
D
for k 1 to K do
H < D)) h f- HIN +k- -]
// Forward with input and generated tokens // Last hidden state for next prediction // Compute soft token distribution
Sk: f- softmax Whhh + bo)
d: EM e
// Compute soft embedding // Append soft embedding for next step
D - D D
H - (x; D)
G - N + H + KJ)
// Collect the last K hidden states // Pool generated representations into a single embedding
2 - P(G) P())
return
end-to-end optimization. We address this through a novel soft token generation mechanism that preserves continuous optimization while capturing rich semantic information.
At each generation step & E (1...., K). the generative distribution ps is instantiated via the LM head ø. Let h h(() h denote the last layer hidden representation produced by the encoder given the input sequence and the previously generated soft tokens up to step k 1, the LM head then produces soft token S₁ E RVV as a probability distribution over the vocabulary:
S = softmax ( (Wah + ba):
(5)
where We E R is the LM head weight matrix and b& is the bias term. Given the soft token Sk: its embedding dg E Rd is obtained by computing convex combination of all token embeddings according their predicted probabilities:
M d2 Σ S 1=1
(6)
where sk, is the i-th component of s and C; is the i-th row of the embedding matrix E. This soft token
generation approach offers two advantages: (1) Differentiability: The weighted combination preserves gradients throughout the generation process, enabling end-to-end training with contrastive objectives. (2) Semantic Richness: Rather than collapse the next-token distribution into single token, soft tokens capture the semantic diversity of the full probability distribution.
3.3 GUIDING GENERATIVE EMBEDDING WITH ITERATIVE CONTRASTIVE REFINEMENT
To guide the generative embedding process towards high-quality representations, we introduce iterative contrastive refinement (ICR) objective that encourages each generation step to yield increasingly refined representations. ICR combines (1) Stepwise Contrastive Loss, which supervises each generation step with contrastive loss, and (2) Iterative Refinement Regularization, which enforces progressive embedding enneresiiveeebbeddnngg qual- ity improvement for each step.
Stepwise Contrastive Loss. In autoregressive soft token generation, supervising only the final embedding (i.e., K-th generation step) might collapse intermediate steps into trivial or noisy representations. We instead
5

---

Published as conference paper ICLR 2026
apply contrastive supervision at every generation step. Concretely, for step we pool the first generated tokens to form an intermediate embedding ( following Eq. (4) Given documett pair (g, d we compute the contrastive loss for all generation steps as:
K
exp 2k )/T)
Conntrast
LK
LE = log
(7)
ELEB exp (2 (22zz()) =T)/T)²
where B denotes the document set (both positive and negative documents), a is the cosine similarity function, and is the temperature hyperparameter. This stepwise supervision ensures all intermediate representations align with the contrastive objective, preventing early steps from drifting and providing richer supervision.
Iterative Refinement Regularization. We empirically observe that simply increasing the number of gen- eration steps does not guarantee improved embedding quality, as LLMs often produce highly similar tokens which leads redundant information in the multi-step process. To address this, introduce regularization term that encourages monotonic improvement across generation steps:
K
Lrgg
max ( Lu+1 - log LRs 0).
(8)
K
This regularization term penalizes cases where later generation steps fail to outperform earlier ones. Finally, the overall fine-tuning objective for generative embeddings combines the two terms: Lroral = Loonrast XCregs where À is hyperparameter that balances contrastive alignment and refinement regularization.
We summarize the overall training procedure in Algorithm [ During each training iteration, we sample a mini-batch of queries and documents. For every input sequence, the model performs autoregressive soft token generation for К steps to obtain the sequence of embeddings. Finally, the model parameters & are updated end- and-to-end via gradient descent to minimize the combined objective Lioal-
4
EXPERIMENT
4.1 EXPERIMENT SETUP
Backbone LLM. Following prior works (Wang et al. 2024 Mucnnighoff et al. 2024). we adopt Mistral-
7B (Jiang al. 2023) as the primary backbone and further validate on Qwen2.5-7B Yang et al.) 2024).
2025), which integrates supervised Training Details. For training data, we use the dataset from (Li et al.
pairs and hard negatives for contrastive learning across diverse tasks. Due to computational limits, we
Li et al. (025). we fine-tune the LLM as an embedding model with LoRA and contrastive loss, applying task-specinc instruction templates. Specifically, for given query g, we format it q= Instruct: (task definition) (aanittoo)"'nuuusee !nQuey: {4). Detailed hyperparameters and instructions are in Appendices Cand
sample 20% (0.2M) data for training. Following (Wang et al. 2024
Evaluation. We evaluate on MTEB (English, v2) (Enevoldsen et al. 2025). covering 41 datasets across 7 task types, reporting official leaderboard scores when available. Following InBedder (Peng et al., 2024). we also evaluate INTENTEMOTION and NYTCLUSTERING to test the instruction-folloiing ability of embedding models. For more extensive comparison, we have also evaluated on TREC datasets used in FollowIR (Weller et al. 2025) and BEIR (Thakur et al. (021). detailed results can be found in Amppendix H
Comparison Methods. We compare GIRCSE against four categories of text embedding models. (1)
Non-LLM methods including encoder-based models such E5-Large (Wang et al. 2022), GTE-
Large (Li et al.) 20235), BGE-Large (Li et al., 2023a), and UAE-Large (Li & Li 2024). (2)
LLM-based methods instruction-tuned LLM embeddings, including LLM2Vec [BehnamGhader et al.
6

---

Published as a conference paper ICLR 2026
2.24 GritLM (Muennighoff' al. 2024), E5-Mistral (Wang et al. 2024), NV-Embed-vi Lee
et al. 2024). SFR-Embedding-22 (Meng* et al. 2024), and gte-Qwen2 (Li et al. 2023b). (3)
Generative embeddings cover (i) two-stage approaches that expand text with an auxiliary LLM before re-encoding (see AApeendi F] for detail) and (ii) the end-to-end generative model Inbedder (Peng et al.,
2024). (4) Fair Baselines are included by re-implementing two paradigms on the same training data for
fair comparison: (i) Causal-EOS (causal attention + EOS pooling) and (ii) Bidirectional-Avg (bidirectional attention + average pooling), equivalent to E5-Mistral and GritLM respectively but trained with less data.
4.2
MAIN RESULTS
Table reports the performance comparison across MTEB tasks and cetructioo-folloinng benchmarks. We highlight the following observations:
Trade-off between generic tasks and instruction following. State-of-thheart non-generative embedding models achieve strong results on generic MTEB tasks but lag behind on instruction-following benchmarks. For example, Ale-QWEN2 performs competitively on MTEB (rank 1) but drops notably on instruction- following tasks (rank 18). Similarly, E5-Mistral ranks 4 on MTEB but falls to 10 on instruction following, contrast, generative embedding approaches such as Inbedder reverse this trend, achieving top instruction- following performance (rank 1). since it is explicitly trained for this setting, but performing poorly MTEB (rank 20). A comparable trade-off is also observed in two-stage generative variants of non- generatieee mod- els. For instance, E5-Mistral (w/ gen) improves on instruction following (rank 10 5) but degrades on MTEB (rank 4 + 12). Similar patterns are also observed for E5-Large (w/ gen) and GritLM (w/ gen).
GIRCSE overcomes trade-off and strikes balanced performance. Unlike prior methods, GIRCSE delivers consistently strong results across both task categories. It not only outperforms fair baselines and competitive embedding models (e.g.. GritLM, LLM2Vec), but also avoids the severe trade-off' observed in existing approaches. Specifically, GIRCSE ranks within the top 5-6 on MTEB and top 2-3 on instruction- following tasks, leading the best overall rankings of 3.5 and 4.5 across benchmarks. Remarkably, while prior SOTA methods rely on multi-miliin--eee performance with only 0.2M training examples. These results highlight GIRCSE as an efficient embedding model that achieves both strong general-purpont performance and robust instruction-folloiie ability.
training datasets, GIRCSE achieves comparable or better
4.3 ABLATION STUDY
To better understand the contributions of different components in GIRCSE, Table represents an ablation study generative embedding, stepwise loss (SL), and iterative refinement (IR). Starting from the variant without generation (i.e., the CCO baseline), we observe a substantial drop in performance across both MTEB and instruction-followwe following tasks. Incorporating generative embedding alone yields consistent improvements across nearly all tasks. Adding SL provides further gains, particularly for classification and summarization, while the combination of SL and IR achieves the strongest overall performance. Overall, these results validate the effectiveness of our design in GIRCSE.
5 ANALYSIS ON GENERATED TOKENS
While|Sections 4.3 highlights the importance of generative embedding, remains unclear how the generation process itself translates to the improved performance. To address this, we present thorough analysis of the generation process. In Sectiionn 5.1] we first discuss how the embedding quality changes by varying the number of generated tokens inference. Next, in /Section 5.2 conduct qualitative analysis to under- stand what tokens are generated and how they evolve under different instructions. This analysis clarifies how iterative generation improves performance and what semantic signals are encoded in the embedding space.
7

---

Published as a conference paper ICLR 2026
Table 2: Performance on MTEB and mnstruction-folloiigg followwig tasks. Results obtained from the official MTEB leaderboard. Causal-EOS: causal attention with EOS pooling: Bidirectional-Arg bidirectional attention with average pooling. Highlighted rows are our reproductions, trained on smaller dataset (0.2M) for fair comparison. Bold better than fair baselines with same backhone; = statistically significant (p < 0.05).
For detailed performance of each MTEB dataset, please refer to Appendix I
MTEB (English, v2)
Instruct Fellowing
Overall Rank
Task
Size
Backhone
Retr. 10
Rerank.
Clust.
PairClass.
Class.
STS
Summ.
Avg. (Rank)
IntEmm
NVT
Avg. (Rank)
distoeets
N.w--.M Methody
B5 Large'
0.38
1B
BERT
49.31
45:72
45.23
86.06
76.44
80.67
32.34
62.79
48.63
50.96
49.80
16.0
GTB Large!
0.30
2B
WERT
53:29
47.84
48.20
85.08
75.47
83.27
32,90
64.77 (14)
52.62
17.52
35.07 (18)
16.0
BCIE-Lizge'
0.38
200M
BERT
35.44
48.25
48.01
87.13
78.34
$2.79
33.13
65.89 (12)
51.66
61.38
56.52
10.5
UAE Large!
0.38
RoBERTa
48.33
47.36
87.25
79.08
50.49
60.54
10.0
F..M-houod: Cossal-ROS
E5 Miseral
7B
1.3M
Mistral
57.62
49.78
51.44
88.42
79.85
84.32
36.57
67.97
48.84
65.06
56.95 (10)
7.0
SFR Eithedding-2" 2
7B
1.7M
Mistral
53.75
48.9
59.39
88.09
90.54
50.86
35.54
69.82: (2)
50.49
60.54
55.52 (I1)
6.5
Owen2"
7B
800M
OWEN2
38.09
50.47
59.97
85.90
38.52
$2.69
35.74
70.72(03
52.62
17.52
35,07 (18)
9.5
Fair Revdlin
7B
0
Midral
55.24
49.21
54.18
85.65
84.36
73.98
96.31
66.32((1))
35.33
58.76
47.05 (15)
12.5
Fair Bescline
7B
0.2M
OWEN2
51.10
47.49
55.26
84.46
80.10
74.71
33.21
64.18.017
66. 14
14.71
40.42 (17)
17.0
Aased: Distrecrional-tygg
7B
51.27
47.74
44.30
87.99
79.74
(1557(15)
56.52
GRILM
7B
251
Misral
54.95
49.59
50.82
87.29
81.25
53.03
35.65
67.07 (7)
39.30
79.25
59.25 (6)
6.5
7B
1.I
Mistral
60.13
40. 16
49.50
87.05
84.11
92.30
31.40
66.32203)
52.61
60.62
56.62.77
5.0
Fair Baseline
7B
0.2M
Mistral
$5.41
63,
54.57
86.34
$6.98
75.87
36.09
66.96.00 (5)
21.45
66.42
63.94 (16)
12.0
Fair Baseline
70
0.2M
QWEN2
$2.99
47.11
54.75
83.31
$2.66
72.81
35.30
(4.97(16)
45:26
65.21
54.24 (13)
14.5
L.M--bused Two-Ntuge Generative Enaledding
0,30
43.87
45.37
72:70
37.35
(9985
51.34
15.5
ES-Mierrl gon.)
7B
$3.00
65.92(13)
Chiil gon))
7.5
LLM -aseddased: EndZEind Generative Embodding
Inbedder
7B
0.2M
LLaMM2
12:50
39.21
51.24
61.17
72.41
34.41
17.24
30.32(20)
89 69
64.65
77.17 (1)
10.5
GIRCSE
7B
0.2M
Mistral QWEN2
57.10
48.85
56.26
86.18
85.33
76.37
33.56
67.83" (5)
$2.19
73.75
62.97 (2)
3.5
GIRCSE
7B
0
55.16
49.28
36.62
85.17
36.69
76.30
35.42
67.67 (6)
64.92
61104
62,48 (3)
4.5
Table 3: Ablation study of GIRCSE with generative embedding (Gen.), stepwise loss (SL), and iterative re- finement (IR). The variant without generation corresponds to the Causal-EOS baseline. Results are reported using the Mistral-7B backbone trained on 50K samples.
MTEB (English, v2)
Instruct Following
Gen.
SL
IR
Retr.
Rerank.
Clust.
PairCls.
Class.
STS
Summ.
Avg.
IntEmo
NYT
Avg.
x
50.55
48.97
49.91
85.27
80.36
75.76
34.02
63.84
33.11
58.76
47.05
53.17
48.32
52.74
84.75
78.34
78.70
33.86
65.21
48.00
64.93
56.47
54.97
48.86
52.07
85,04
78.63
78.87
35.28
65.69
53.88
66.37
60.13
55.53
48.26
53.71
84.87
79.53
78.93
34.19
66.27
62.70
73.75
62.97
5.1 EFFECT OF GENERATION LENGTH AT INFERENCE
We first examine how performance varies with the number of generated tokens K at inference. We evaluate K E (1,3,5 5, 10, 15, 20} and compare against the non-ge non-generative baseline Causal-EOS. Results (Fig. 2 are reported on three LLM backbones and three representative MTEB tasks, with relative improvements measured against K = 1 for GIRCSE and against the no-generation baseline for Causal-EOS. For more comprehensive analysis, we further evaluate GIRCSE trained with two additional backbones: QWEN2.5. 0.5B Yang et al. 2024) and LLaMA3.2-3B (Dubey et al. 2024). We have the following two key findings:
(1) GIRCSE exhibits test-time scaling for embeddings. Increasing K consistently improves performance across diverse tasks (e.g., STS, clustering, classification) and across model sizes. In contrast, the non- generative method (Causal-EOS) does not benefit from additional generation and often degrades in per- formance. This suggests that GIRCSE successfully learns an iterative refinement mechanism that converts
8

---

Published as conference paper ICLR 2026
GIRCSE: STS
GIRCSE: Clustering
GIRCSE: Classirication
0.030
0.0375
0.003
0.0255
OWEN-O.5P (GIRCSE)
0.00
5
10
15
20
0.000
10
15
20
30
15
20
OWEN-O.5 0.58 (Causal-EOO)
Generated Tokens
Generated Tokens
GGreratte Tokens
LLaMA (GRCSE)
Causal-EOS: STS
Causal-FOS: Clustering
Causal-EOS: Classification
LLaM/+30 30 (CausalEEOS)
be impron.
6.0
-0 025
2.000
Mistral-78 (Causal-5O5)
Mistral-78 (GIRCSE)
-0.075
-02
-0.109
10
20
10
20
15
Generated Takens
Generated Takens
Generated Tokens
Figure 2: Effect of generation length at inference. Top: GIRCSE consistently improves with longer gener- ations (10-20 tokens) despite been trained on only 5 tokens. Bottom: Baseline models show degraded or fluctuated performance across generation lengths. Gray area indicates configurations beyond training length.
additional inference computation into stronger semantic representations analooguus to test-time sitetimm compute scaling reasoning L.MMs (Muennighoff al. 2023). but novel in the context of embedding models.
(2) ICR enables GIRCSE to generalize beyond training configurations. Although GIRCSE is trained with K = 5, its performance improves monotonically within the training regime (K 1,3, 5) and continues to improve even beyond it (K = 10, 15, 20). This extrapolation capability suggests that our ICR training objective enables the learned refinement process to generalize beyond the training configuration, allowing GIRCSE to continue improving with additional inference steps. Overall, GIRCSE establishes test-time scaling as a new paradigm for embedding models, enabling controllable and training-free free performance gains through adjustable generation length.
5.2 QUALITATIVE ANALYSIS GENERATED TOKENS
Having shown that generating more tokens improves performance, we next ask: what do these tokens cap- ture? We analyze generations for the sentence "Why is so hard track down this card?" under two prompts: representing intention and emotion. At each generation step k, we collect the top-30 candidates from the soft token distribution 8;, aggregate across steps, and report most frequent tokens in Table 4| alongside results from GIRCSE before contrastive fine-tuning. Before fine-tuning. GIRCSE (before FT) often yields generic or semantically weak tokens. After fine-tuning, we observe progressive semantic re- finement that aligns with the results in /Section 5.1 At early steps (1-5), GIRCSE generates core content words (e.g., why, hard, card). While at later steps, the outputs diverge by different instructions: intention produces tokens such as seek. elusive, inquiry, and emotion yields tokens like frustrating, struggle. This suggests multi-step generation acts as semantic chain of thought, iteratively steering representations toward nuanced, instruction-aligned regions of the embedding space.
6 DISCUSSION OF ROBUSTNESS AND LEARNING EFFICIENCY
To assess the robustness and learning efficiency of our method, conduct comprehensive experiments across varying data scales and backbone architectures. Specifically, we train different models with (50K, 100K, 200K) training samples using three widely adopted open-source LLMS as base models: Qwen-0.5B, Llama-3B, and Mistral-7B. We compare GIRCSE against two fair baselines: Causal-EOS and Bidirectional- Avg. Fig. 3 shows that our method consistently outperforms both baselines across all data scales and model
9

---

Published as conference paper ICLR 2026
Table 4: Qualitative analysis of generated tokens. Gray indicates generic/stopword-ike tokens. Yellow marks core input-related tokens shared across instructions. Instruction-speeific expansions shown in Green (intention) and Red (emotion).
Input Sentence
"Why is it hard track down this card?"
Instruction
"Represent the intention of this text."
"Represent the emotion of this text."
GIRCSE (Before FT)
this, so, do, how, 1, is, it, the, we, what why, is, it, hard, track/trackingg card, this why, is, it, hard, track, tracking. card, this, so, difficult [prev.] frustrating, tough. persistent, Struggle, challenging
why, how, what, this, it, is, you, can
GIRCSE (Step 1-5)
GIRCSE (Step 6-10)
[prev]] sock. clusive, so
GIRCSE (Step 11-20)
[prev.] + question, inquiry
[prev.] + perseverance, Stuck, complicated
sizes. In particular, when trained with only 50K samples, our method improves over Causal-EOS by +5.7 points Qwen-O.5B (61.2% vs. 55.5%) and by +2.8 points on Llama-3B (65.5% vs. 62.7%). Even with stronger backbones such as Mistral-7B, our approach still yields gains of +2.4 points (66.2% vs. 63.8%). The performance gap becomes more pronounced when the training data limited. These findings indicate that our approach not only achieves superior performance across different scales of model size but also learns more effectively under limited training data.
Causal-EOS
Bidirectional-Aug
GIRCSE
Qwen-0.5B
LLaMA-3B
Mistral-7B
Avg. MTEB (96)
70%
61.5
65.5
15. 66.1
65.00.0
27.
65%
65%
65%
63.863.
02.
31.0
60%
60%
60%
5:.
55%
55%
55%
50%
50%
50%
50K
100K
200K
50K
100K
200K
50K
100K
200K
Training Samples
Training Samples
Training Samples
Figure 3: Comparison of average MTEB scores (%) between GIRCSE and two fair baselines across three backbone LLMs and varying training sample sizes. GIRCSE consistently delivers superior performance, especially under limited-data settings.
7 CONCLUSION
We presented GIRCSE, generative embedding framework that leverages autoregressive refinement to move beyond single-pass LLM encoders. By generating soft refinement tokens and training with iterative con- trastive refinement, GIRCSE enables embeddings to progressively distill semantics rather than compressing them in one step. Experiments show that GIRCSE achieves state-of- of-the-art or competitive performance across benchmarks while introducing novel scaling property: embedding quality improves with additional refinement steps at test time. These results highlight autoregressive generation as a powerful mechanism for embedding optimization and open directions for scalable, semantically rich representations.
ACKNOWLEDGMENTS
This material is based upon work supported by National Science and Technology Council, ROC under grant 1..000--34-MM 134-MY3 and by Taiwan Centers of Excellence (TCE)
number 1 14-22211-E-022.
10

---

Published as a conference paper ICLR 2026
REFERENCES
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024.
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective 1.2 norm- based strategy for kv cache compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18476-18499, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, et al. Mmteb: Massive multilingual text em- bedding benchmark. In International Conference on Learning Representations. International Conference on Learning Representations, 2025.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894 6910, 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representationss 2022. /tthhtss:://// httptttss///eeeeeeeeeeeeeeeeeeee
net wtt.rooormnndndeeeeeeeee
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/23 10.06225, 2023. URL https://api.
semanticscholar.orre htt.://a.....roooaaaaa0022333300
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models. In The Thirteenth International Conference Learning Representations, 2024.
Chaofan Li. Minghao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Defu Lian, Yingxia Shao, and Zheng Liu. Making text embedders few-shot learners. In The Thirteenth International Conference on Learning Representations, 2025. URL htttpt::/oppereeeettee
net forum?irummmm www.rrummidwifwiii000 j00u
Xianming Li and Jing Li. Aoe: Angle-optimized embeddings for semantic textual similarity. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1825-1839, 2024.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03288 032881 2023a.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.088 032811 2023b.
11

---

Published as a conference paper ICLR 2026
Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. Once: Boosting content- matentt-based recommendation with both open-and closed-souree large language models. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 452 461, 2024.
Rui Meng*, Ye Liu", Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embedding- 2: Advanced text embedding with stage training, 2024. URL Salesforce SFR- Embedding-2
co/
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 1301.3... 2013.
Niklas Muennighoff, SU Hongjin, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh. and Douwe Kiela. Generative representational instruction tuning. In The Thirteenth International Conference on Learning Representations, 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi. Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettle- moyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. sl: Simple test-time scaling. In Workshop on Reasoning and Planning for Large Language Models, 2025.
Letian Peng, Yuwei Zhang, Zilong Wang, Jayanth Srinivasa, Gaowen Liu, Zihan Wang, and Jingbo Shang. Answer is all you need: Instruction-followin text embedding via answering the question. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 459-477, 2024.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word repre- sentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 2014.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-IJCNLP), pp. 3982-3992, 2019.
Chongyang Tao, Tao Shen, Shen Gao, Junshuo Zhang, Zhen Li, Zhengwei Tao, and Shuai Ma. Lims are also effective embedding models: An in-depth overview. arXiv preprint arXiv:2412.12591, 2024.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: het- erogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httpss://o http:::///ppeneeveee net www.ooomndwwuuuooooo
jej
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by reakly-supervised contrastive pre-training. arXiv preprint 1rXiv:2212.03552. 2022.
Liang Wang, Nan Yang, Xiaolong Huang. Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11897-11916, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. -hain-of-tthough prompting elicits reasoning in large language models. Advances in neural infor- mation processing systems, 25:24824-24837, 2022.
12

---

Published as conference paper ICLR 2026
Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. Followir: Evaluating and teaching information retrieval models to follow instructions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the As- sociation for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), Pp. 11926-11942, 2025.
Lee Xiong, Chenyan Xiong. Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. net/
In International Conference on Learning Representations, 2021. URL
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint ar 1511: 2024.
13

---

Published as conference paper ICLR 2026
A USE-OF-LLMS
In this work, utilized large language models (LLMS)) part of the research methodology. Specifi- cally, we fine-tuned existing open-source L.MMs (e.g., LLaMA-3 and Mistral) to develop embedding models. These pre-trained models served as the foundation for our experiments, and our main contributions build upon their architectures and representations. Additionally, an LLM-based assistant (OpenAI GPT-5) was used for minor writing support, including grammar checking and improving manuscript readability. All de- cisions regarding research design, fine-tuning strategies, experimental setup, and final interpretations were made solely by the authors.
B REPRODUCIBILITY FOR EMBEDDING MODELS
To facilitate reproducibility of our experiments, provide links to all open-soureed embedding models used in this paper in Table 5. These links allow researchers to directly access the exact model checkpoints we relied on.
Table 5: List of models with links for reproducibility.
Model
Link
E5-Large
GTE-Large
Juggunghiee.c/hheelp///ggtgggg
BGE-Large
juggung/acccoo/BAAAllgglgggggge
UAE-Large
VI
E5-Mistral
SFR-Embeddingg-2
gte-Qwen2
LLM2
gitibb.com/MGiiillNP////WVVVV
GritLM
huggunggaac..o//GmLLMMMCMLLMVVV
NV-Embed-vl
nugguggaaee.oowwddaVVVVVVmbb
C HYPERPARAMETER SETTINGS
For both GIRCSE and the baseline models we re-implement for comparison, we use nearly identical fine- tuning hyperparameters across different model sizes (QWEN-0.5BB Llama-3B, and Mistral-7B) and training data scales (50k, 100k, and 200k examples).
We adopt Low-Rank Adaptation (LoRA) (Hu et al. 2022) for efficient fine-tuning, setting the rank to 64 and the scaling factor a to 32. The default learning rate le-5 with a warmup ratio of 0.I. The only exception is Llama-3B, for which we use a learning rate of le-4 to address convergence issues. For other hyperparameters, we set the temperature of the contrastive loss (Eq. (7) to 0.02 across all models, and the weighting coefficient in GIRCSE for balancing contrastive alignment and refinement regularization to Due to limited computational resources, we train with a batch size of 2 and accumulate gradients over 8 steps, resulting in an effective batch size of 16. All models fine-tuned for single epoch.
For GIRCSE, we set the number of generated tokens K to 5 during training to avoid the high computational cost of multiple autoregressive forward passes. During inference, increase K to 20 enable longer
14

---

Published as conference paper ICLR 2026
generations and improved embedding refinement, while mitigating the computational overhead using KV- cache techniques.
D SCALABILITY ANALYSIS
While GIRCSE offers superior representation quality, a natural concern arises regarding its computational efficiency. We acknowledge that GIRCSE introduces additional overhead compared traditional embed- ding models due to its generative process; a detailed analysis of theoretical computation and memory costs relative to the discriminative embedding paradigm is provided in (Appendix Nevertheless, this overhead can be substantially mitigated through the use of KV caching techniques (Devoto et al. 2024). As shown in T GIRCSE without caching requires 2.0-6.0x more FLOPs across different sequence lengths due auto-regressive computation. In contrast, with caching enabled, the FLOPs are effectively reduced to base- line levels (== (=0,,),, while memory consumption remains comparable traditional methods. These results demonstrate that caching not only ensures scalability but also makes our approach practical for real-world deployment.
Table 6: Computational efficiency comparison across sequence lengths (512. 1024, 2048) and generation budgets k. Lower better (4) for both FLOPs and memory. GIRCSE without eaching (red) incurs signif- icant computational overhead due to auto-regressive processing, whereas KV caching (blue) dramatically mitigates this cost, Multipliers parentheses show overhead relative to Causal-EOS method.
FLOPS (T) 4
Memory (GB)
Method
512
1K
2K
512
1K
Causal-EOS
7.33
14.65
29.30
13.89
14.34
15.11
Bidirectional-Ayg
7.33
14.65
29.30
14.03
14.45
15.24
14.67 (2.00)
29.32 (2.000)
58.62 (2,000
13.72
13,90
14.28
GIRCSE (w/o cache)
29.39 (4.00)
58.70 (4.000)
117.31
(4.000
13.74
13.92
14.29
44.17 (600))
88.13
(6.02x)
176.04
(5.01)
13.75
13.94
14.30
7.34 (1,00)
14.67
(1.00k)
29.32
13.73
13.91
14.32
GIRCSE (w/ cache)
7.37 (1,03)
14.70
(1,000)
29.35 (1,000
13.75
13.93
14.34
7.40 (1.030)
14.73
(1.01))
29.38
13.77
13.96
14.35
E THEORETICAL COMPUTATIONAL AND MEMORY COST ANALYSIS
To better analyze the additional training and inference cost introduced by GIRCSE, we compare the compu- tational and memory complexity of the proposed generative embedding framework against the conventional discriminative embedding paradigm. In the baseline discriminative case, the encoder processes an input of length N, leading to a per-layer cost dominated by self-attention of order O(N2d) and memory footprint O(LN2), where d is the embedding dimension and 7. is the number of layers.
the generative framework, K auxiliary soft tokens are generated autoregressively. Each generation step requires an encoder forward pass over N + tokens (j O...., K 1) followed by vocabulary softmax cost Od)))). After generation, a final encoder pass is performed over the extended sequence of length N + K. The total aitention-dominatte computation ratio with respect to the baseline is:
Cgen Obasc
(K + +K + NK(K + 1) K)K+ +11
Recmpuuaaiinn
6
(9)
N2
15

---

Published as a conference paper ICLR 2026
which simplifies to Reomputation K + 1 when K N. The additional softmax operations contribute K O(d)V)) which is typically small compared to the quadratic encoder cost unless N is short or [V] is very large.
In terms of memory, peak training- time activation usage is dominated by the final encoder pass over N + K tokens. Thus, the relative peak memory ratio is:
M gen
(N + K)²
Remmory
(10)
Muuse
N
This indicates that while the generative embedding framework incurs roughly K additional encoder passes in computation, the increase in peak memory is modest, scaling quadratically with the extended sequence length N + K.
DETAIL IMPLEMENTATION OF TWO-STAGE GENERATION EMBEDDING
The two-stage generation embedding approach enhances representation quality by introducing an interme- diate expansion step before re-encoding.
the first stage, an auxiliary large language model (LLM) is prompted to generate a short augmentation of the input, detailed expansion prompt can be found in Table 7| The prompt instructs the model to output only the augmentation, within a fixed token budget, without explanations or additional formatting. This augmentation is designed to highlight or enrich semantic information that may be useful for downstream
tasks.
In the second stage, the original instruction and text is concatenated with the generated augmentation, and the combined sequence is re-encoded into an embedding. This two-step process allows the encoder to capture a more informative and contextually aligned representation than directly embedding the raw text alone.
Table 7: LLM expansion prompt used for two-stage generation methods.
Input:
Given the INSTRUCTION and the TEXT, produce a helpful augmentation that, when concatenated to the original TEXT and embedded, likely to improve embedding quality for the instruction's task. Do not explain yourself output anything other than the augmentation. Your answer must be written within {256) tokens.
INSTRUCTION: (instruction)
TEXT: (text)
G TRAINING STABILITY ANALYSIS
A potential concern when optimizing models that involve the generation of soft tokens the risk of gradient instability. To empirically validate the stability of our proposed model, GIRCSE, monitor its training dynamics. We present the training loss curve in Figure the 1.2 norm of the gradients in Figure
As shown in the figures, the training loss (Figure demonstrates smooth and consistent decrease, indi- cating stable convergence. Furthermore, the gradient norm (Figure 5)) converges normally throughout the training process.
16

---

Published as conference paper ICLR 2026
Training loss (GIRCSE-Missral77--000
17.5
Raw loss
12.5
Train loss
7.5
5.0
0.0
2000
4000
6000
8000
10000
12000
14000
step
Figure 4: Training loss curve of GIRCSE.
Training grad grad_norm (GIRCCE-Mistral77--0000
grad. norm
EMA smoothed grad norm
1750
1500
grad rom
750
500
250
2000
4000
6000
8000
10000
12000
14000
Global step
Figure Training gradient norm (L2) GIRCSE, plotted with the top 2% of outliers removed for clarity.
H MORE EVALUATION ON NANOBEIR AND TREC BENCHMARKS
this section, we provide additional evaluation results on the NanoBEIR and TREC benchmarks. These benchmarks complement the main paper by covering wider range of retrieval tasks and domains. The results, presented in TTalle and Table 9 highlight consistent trends with our main findings, further demon- strating the robustness and generalization ability of the proposed model compared to strong baselines.
17

---

Published as conference paper ICLR 2026
Table 8: Performance comparison on NanoBEIR benchmark.
Dutaset
ES-Mistral
SFR-Embeddding-2
gte-Qwen2
LLM2Vec
GritEM
NV-Embed-ri
GIRCSE
GIRCSE
7B
Vol.
L&M
1.7M
800M QWEN2
1.5M Mistral
2M Mistral
1.I Mistral
0.2M
0.2M
Backbone
Mistral
Mistral
Mistral
QWEN2
ArguAna
60.13
64.33
72.37
54.41
67.72
68.00
70.18
70.88
ClimateFever
42.93
36.83
DBPedia
71.59
73.81
12.14
70.78
68.15
71.44
71.01
69.81
FEVER
95.30
95.18
80,71
96.94
94.38
96.15
94.23
95.41
FQQA
60.50
63.08
68.23
61.46
65.98
68.07
60.79
$5.90
HorporQA
86.16
88.14
91.05
88,10
88.31
MSMARCO
66.58
66.51
70.67
64.30
62.82
67.80
64.03
61.85
NFCorpus
32.36
38,99
37.67
39.19
40.39
38.54
40.75
36.57
NQ
76.64
82.20
80.99
83.81
82.91
87.11
78.57
70.01
Quoraa
96.28
95.48
97.44
95.64
95.76
91.59
92.73
88.76
SCIDOCS
36,00
48.18
50.76
43.72
46.32
38.11
44.27
43.97
SciFact
78.30
89.67
67.58
79.41
81.18
78.95
80.90
81.70
Touche202U
51.72
56.66
39.44
49,46
51.89
58.51
46.09
39.03
65.64
70.14
65.68
66.57
68.16
69.10
66.81
64.14
I FULL MTEB PERFORMANCE
the main paper, we reported averaged performance across task categories for clarity. Here, we provide the full per-dataset results the MTEB benchmark, covering all tasks included in evaluation. The detailed scores in Table allow more granular comparison across individual datasets and complement the averaged results presented in the main section.
18

---

Published as conference paper ICLR 2026
Table 9: Performance comparison on TREC datasets. Results obtained from (Weller et al. results per task (i.e., column) in bold, with second best results are underline.
2025 Best
Robust04 MAP
News21 nDCG
Core17 MAP
Avg.
Model
No-Instruction IR
E5-base-v2
13.4
20.9
14.0
16.1
Contriever
19.7
22.9
15.3
19.3
MonoBERT
21.0
25.1
18.4
21.5
BM25
12.1
19.3
8.1
13.2
MonoTT-base'
15.7
11.0
12.2
13.0
E5-large-v2
17.4
24.3
17.0
19.6
MonoT5-3B
27.3
16.5
18.2
20.7
Instruction-IR
TART-Contriever'
14.3
21.8
13.3
16.5
INSTRUCTOR-bin
17.2
22.1
15.5
18.3
E5-mistral
23.1
27.8
18.3
23.1
BGE-base
16.8
20.0
14.6
17.1
INSTRUCTOR-X
19.7
26.1
16.8
20.9
BGE-large?
17.5
22.3
15.0
18.3
GritLM-7B
28.6
24.4
20.8
24.6
TART-FLAN-TS-XI
24.6
12.8
17.0
18.1
APIs
OpenAI v3 Large*
27.2
27.2
21.6
25.3
Cohere v3 English
22.3
28.3
20.6
23.7
Google Gecko
23.3
29.5
23.2
25.3
Instruct LMs
FLAN-T5-basse
6.4
6.1
6.5
6.3
Llama-2-7B-che
6.3
1.7
5.4
4.5
FLAN-T5-large"
14.7
8.0
11.4
11.4
GritLM-Rerankee
9.7
10.2
9.8
9.9
Mistral-7BBinstruee
23.2
27.2
19.7
23,4
FollowIR-7B
24.8
29.6
20.0
24.8
End2End Generative Embedding
GIRCSE-Mistral-77
27.9
26.8
23.0
25.9
23.9
24.1
21.0
23.0
19

---

Published as conference paper ICLR 2026
Table 10: Full evaluation results across MTEB tasks for GIRCSE, Causal-EOS, and Bidirectional-Ayy with Mistral and Qwen backbones. Best results per task (i.e., row) in bold.
GIRCSE
Causal-EOS
Bidirectional-Avg
Task
Mistral
Qwen
Mistral
Qwen
Mistral
Qwen
MindSmallRerankingg
0.294
0.321
0.307
0.308
0.302
0.311
AskUbunuDuppuussionn
0.684
0.664
0.677
0.642
0.673
0.631
TwitterSemFvvl22155
0.769
0.747
0.760
0.732
0.786
0.707
SuckEEchhngeCCussernngZ222
0.499
0.508
0.457
0,463
0.439
0.479
BiotriCCuuserrngg22PP22
0.487
0.497
0.439
0,431
0,459
0.472
SICK-R
0.727
0.742
0.731
0.741
0.753
0.746
ToxiiCoonveraaionCCaaaiiiiiaii
0.841
0.870
0.781
0.735
0.811
0.760
0.762
0.775
0.730
0.703
0.744
0.723
wventy/Nwwsgrroppcuussteingg.
0.611
0.627
0.598
0.582
0.624
0,569
STS15
0.845
0.833
0.835
0.817
0.839
0.815
MTOOPDomainClasiffcttoo
0.957
0.967
0.958
0.960
0.963
0.944
STSBenchmark
0.832
0.820
0.807
0.814
0.823
0.773
STS17
0.801
0.795
0.774
0.821
0.805
0.806
CinnaceFEYEEREardNNggiiis
0.268
0.261
0.220
0.222
0.227
0.269
HotporQAAHardNeaativvs
0.729
0.737
0.738
0.708
0.735
0.686
FIQA2018
0.552
0,489
0.486
0.480
0,521
0.440
CQADuusticKaaninngeeiievd
0.641
0.656
0.587
0.576
0.615
0.574
SprintDuplicaeeeuussion
0.943
0.949
0.946
0.940
0.939
0,942
ArguAna
0.699
0.692
0.694
0.574
0.703
0.662
MassivelntemmClassticaaiin
0.772
0.781
0.775
0.757
0.772
0.756
SCIDOCS
0.237
0.250
0.234
0.231
0.242
0.231
STS22.v2
0.690
0.665
0.607
0.447
0.645
0.679
STS12
0.730
0.740
0.724
0.731
0.738
0.708
STS13
0.717
0.725
0.704
0.762
0.714
0.581
MedrxiiCluuseeringP22P22
0,429
0.433
0.386
0.338
0.387
0.418
MassiveSeenarroClaasifcciio
0.793
0.802
0.807
0.778
0.798
0.790
STS14
0.746
0.744
0.710
0.739
0.741
0.665
AXKi/HirrarchiiaalClsseenggghg
0.645
0.630
0.639
0.646
0.634
0.639
ImdbClassification
0.960
0.962
0.955
0.906
0.959
0.949
Banking?7CClasificttion
0.855
0.861
0.849
0.838
0.853
0.823
Touchee2020Rerriivall...
0.478
0.391
0.490
0.459
0.452
0.470
SunmmFyaSSummarizationnz
0.336
0.354
0.363
0.332
0.361
0.353
TwitteUURI,Corput
0.873
0.859
0.864
0.862
0.865
0.850
AmazzonCoomteffittlllCCaiicciaaa
0.887
0.918
0.893
0.731
0.896
0.867
Medrxiv
SeddaiivChusteernng225.222
0.423
0.428
0.421
0.409
0.423
0.403
StcckExchaneeCChsstrrnngng.
0.759
0.772
0.758
0.747
0.755
0.742
TEVERiiardNegaative
0,853
0,857
0.786
0.702
0,809
0.835
Unixeeteel
0.522
0.545
0.493
0.455
0.504
0,450
TRECCOVID
0.731
0.640
0.797
0.702
0.733
0.682
BIOSSES
0.786
0.804
0.765
0.851
0.771
0.781
AVKivHlicarchiclllllseeenggg.g
0.648
0.634
0.644
0.645
0.657
J FULL DATASET INSTRUCTIONS
This section provides the complete set of natural language instructions used with the datasets in exper- iments. We include both the instructions for the MTEB benchmark datasets and those for the Instruction
20

---

Published as a conference paper ICLR 2026
Following tasks. These instructions define the intended task for each dataset and serve the input prompts during embedding evaluation. The full text of the instructions is listed in Table 11 and Table 12, respectively.
Table 11:
Instructions for the corresponding datasets in the MTEB benchmark. We mainly follow the instructions from the GritLM paper. Note that for retrieval and reranking datasets, queries (Q) and corpus (C) documents may require different instructions, denoted as {dataset) -Q and { datasets with query instructions only (i.e., [dataset]-Q) Q), no instructions are applied to the corpus.
(dataset)-CC respectively. For
Dataset
Instruction
SummévvaSummarizztion
Given news summary, retrieve other semantically similar summaries. Identify the main and secondary category of Arxiv papers based the titles and abstracts.
ArXoelicaachicalllleeennggg?
ArXiVlieracciicalCuseeeiigg'"
Identify the main and secondary category of Arxiv papers based on the titles.
Touche2002URerriivval.33- CiimaeFEVERRHanNNggatvvs Q
Given question, retrieve passages that answer the question.
Given a claim about climate change, retrieve documents that support or refute the claim.
FEVERRHandNegarivs-- HosponQAAHandNeeaativee
Given a claim, retrieve documents that support or refute the claim. Given multi-hop question, retrieve documents that can help answer the question.
AuazzooCoonteefaauulCCCaaiiaaaii
Classify given Amazon customer review text as either counterfactual or not-counterfactual.
Amaazoaollrityylaasifaaaaaa
Classify Amazon reviews into positive or negative sentiment. Classify the given Amazon review into its appropriate rating category. Given online banking query, find the corresponding intents. Classify the emotion expressed in the given Twitter message into one of the six emotions: anger, fear, joy, love, sadness, and surprise. Classify the sentiment expressed in the given movie review text from the IMDB dataset.
AmazzoneeviossClasiiictttoo
Banking77Classificatinn
EmottinClassification
ImdhClassification
MassivelneennCCasssfictttin
Given user utterance as query. find the user intents. Given a user utterance as query, find the user scenarios. Classify the intent domain of the given utterance in task-oriented con- versation.
MassiveSeenrioCllasificatoo
MTTOPoomainCassiiicatiiiin
MTTOPIntenCClassfccation
Classify the intent of the given utterance in task-oriented conversation. Classify the given comments as either toxic or not toxic. Classify the sentiment of given tweet as either positive, negative, or neutral.
ToxicCoomrrsattooCCCaaiiiiciii
ArxivClusteringP2P
Identify the main and secondary category of Arxiv papers based the titles and abstracts. Identify the main and secondary category of Arxiv papers based on the titles.
ArxivClluseeings25
BiorxieChusteringPPP
Identify the main category of Biorxiv papers based on the titles and ab- stracts.
BiorxisChnsteringS2S
Identify the main category of Biorxiv papers based on titles.
MedrxivClustering222
Identify the main category of Medrxiv papers based on the titles and abstracts.
MedrxivClluterrngg55
Identify the main category of Medrxiv papers based the titles.
RedditClustering
Identify the topic theme of Reddit posts based on the titles.
RedditClussterngg22P
Identify the topic or theme of Reddit posts based on the titles and posts.
StackExchangeeClusterng
Identify the topic or theme of StackExchange posts based on the titles. Identify the topic theme of StackExchange posts based on the given paragraphs.
StacKExchaneCCuustrrnngg?
TventyNewwrruussChustrrngggg
Identify the topic or theme the given news articles. Retrieve duplicate questions from Sprint forum. Continued on next page
Sprintuuplicateeuusstions
21

---

Published as conference paper ICLR 2026
Table continued from previous page
Dataset
Instruction
TwitterSemEva12015
Retrieve tweets that semantically similar the given tweet.
PanterrRRRCorpus
Retricve tweets that are semantically similar the given tweet.
CuuuuntuuupQuustioon Q
Retrieve duplicate questions from UskUbuntu forum.
AkUUbntuDupQQussionn
Retrieve duplicate questions from Askutuntu forum.
MindSmallRerankingg Q
Retrieve relevant news articles based on user browsing history. articles based Given a title of scientific paper, retrieve the titles of other relevant papers.
MindSmallReranking.
Retrieve relevant
browsing history.
SciDocsRR-Q
SciDocsRR-C
Given a title of scientific paper. retrieve the titles of other relevant papers.
TackOOverfoow DDupqeesttooo TaackOverffow DDuQeesstioi
Retrieve duplicate questions from StackOverflow forum.
Retrieve duplicate questions from StackOverflow forum. Given claim, find documents that refute the claim. Given claim about climate change, retrieve documents that support or refule the claim.
ArguAna-Q ClimateEVER--Q
CQADupstackRetrival-
Given a question, retrieve detailed question descriptions from Stackex- change that are duplicates to the given question. Given query, retrieve relevant entity descriptions from DBPedia. Given claim, retrieve documents that support refute the claim. Given financial question, retrieve question.
DBPedia- Q
FEVER Q
FiQA2018-Q
replies that best
the
HotpotQA-Q
Given multi-hop question, retrieve documents that can help answer the question.
MSMARCO-Q
Given a web search query, retrieve relevant passages that answer the
query.
NFCorpus-Q
Given question, retrieve relevant documents that best answer the ques- tion.
NQ-Q QuoraRetrieval-Q
Given question, retrieve Wikipedia passages that answer the question.
Given question, retrieve questions that are semantically equivalent the given question.
SCIDOCS-Q
Given a scientific paper title, retrieve paper abstracts that are cited by the given paper.
SciFact-
Given scientific claim, retrieve documents that support refute the claim.
Touche2020-Q
Given question, retrieve detailed and persuasive arguments that answer the question.
TRECCOVID-Q
Given query COVID-19, 19, retrieve documents that answer the query. Retrieve semantically similar text.
STS12
STS13
Retrieve semantically similar text.
STS14
Retrieve semantically similar text.
STS15
Retrieve semantically similar text.
STS16
Retrieve semantically similar text.
STS17
Retrieve semantically similar text.
STS22
Retrieve semantically similar text.
BIOSSES
Retrieve semantically similar text.
SICK-R
Retrieve semantically similar text.
STSBenchmark
Retrieve semantically similar text.
SummEval
Given news summary, retrieve other semantically similar summaries. Represent the title of a user question to find duplicate user question title with body from the Tex StackFxchange forum.
CQADuustackTeaRRttivvvl.
CQADupptacKTesReerivva
Represent the question title with body posted by user to find duplicate user question title from the Tex StackExchange forum.
Continued on next page
22

---

Published as a conference paper ICLR 2026
Table continued from previous page
Dataset QQDDuppscccebbmsseeeeerriiiug
Instruction
Represent the title of a user question to find duplicate user question title with body from the Webmasters StackExchange forum.
Represent the question title with body posted by user to find duplicate user question title from the Webmasters StackExchange forum.
C
Represent the title of a user question to find duplicate user question title with body from the English StackExchange forum.
COADupsaackinglisieettivvva
Represent the question title with body posted by user to find duplicate user question title from the English StackExchange forum.
(OADDupsaccCGaningeeettvvval
Represent the title of a user question to find duplicate user question title with body from the Gaming StackExchange forum.
COAADupstacKGaminnkeettrrvall
Represent the question title with body posted by user find duplicate user question title from the Gaming StackExchange forum.
COADDupttacKGisReeriivaaa-
Represent the title of a user question to find duplicate user question title with body from the Gis StackExchange forum.
CQAADupstccCiisRetrivvaaal
Represent the question title with body posted by user to find duplicate user question title from the StackExchange forum.
CQADDupsackUniixRettivvvaa
Represent the title of a user question to find a duplicate user question title with body from the Unix StackExchange forum.
CQANDussaccUnnsRRtttivaa-l-
Represent the question title with body posted by user to find duplicate user question title from the Unix StackExchange forum.
Represent the title of a user question to find duplicate user question title with body from the Mathematica StackExchange forum.
CQADipptaacaateemiiiseeettttll
Represent the question title with body posted by user to find duplicate user question title from the Mathematica StackExchange forum.
CQADuustackSaasRettivvaaaa
Represent the title a user question to find duplicate user question title with body from the Stats Stack Exchange forum.
COADunstackStasserriival.l
Represent the question title with body posted by user to find duplicate user question title from the Stats StackExe forum.
Represent the title of user question find duplicate
question
title with body from the Physics StackExchange forum.
COADDusstacdPhysiiseerrivvaaC
Represent the question title with body posted by user to find a duplicate user question title from the Physics StackExchange forum.
Represent the title of a user question find duplicate title with body from the Programmers StackExchange forum. Represent the question title with body posted by user to find duplicate user question title from the Programmers Stack Exchange forum. Represent the title of a user question to find duplicate user question title with body from the Android StackExchange forum.
question
CQADuussaatAnaddddRRRrriialll
CQADDupsteCAaaaboddReetivvaaCC
Represent the question title with body posted by user to find duplicate user question title from the Android StackExchange forum.
Represent the title of a user question to find duplicate user question title with body from the Wordpress StackExchange forum.
COADupstakKoorbbessRttrivvvl
Represent the question title with body posted by user find duplicate user question title from the Wordpress StackExchange forum.
Table 12: Instructions for two Instruction Following datasets used in Inbedder paper.
Dataset
Instruction
IntentEmotion (Intent) IntentEmotion (Emotion)
Represent the intent of this text. Represent the emotion of this text.
NYTClustering (Location)
Represent the text based on where the news happen.
Continued on next page
23

---

Published as a conference paper ICLR 2026
Table continued from previous page
Dataset
Instruction
NYTClustering (Topic)
Represent the text based the main news category.
24


Reviews:
[{"review_id": "W9U0sSt3bM", "reviewer": "Reviewer_HLGU", "summary": "The paper \"Let LLMs Speak Embedding Languages...\" aims to address the problem that algorithms generate representations from text using LLMs are usually static, in that they just make one fixed embedding from e.g. the last layer, last token embeddings. To address this they propose a method to iteratively refine the embeddings over several steps. The algorithm auto-regressively generates so-called \"soft-tokens\" which should give the model a richer ability to refine the representations. This has the drawback of requiring a fine-tuning of the language model but they show that you only need to use a limited amount of data to still achieve decent performance improvements. The primary empirical results show for popular models, (mistral, qwen) that this algorithm can reliably improve the learned representations.", "questions": "1. Is the fine-tuning stage of this algorithm more computationally expensive then regular SFT on the same .2B tokens?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "I think the primary strengths of this work are as follows:\n1. Originality. The iterative algorithm and soft-token idea is well founded and intuitive, and to the best of my knowledge is novel at least in this use case. \n2. Quality. This paper uses a very reasonable suite of evaluations and tests appropriate models that people are typically using in practice.\n3. Clarity. I think the empirical results are presented well.\n4. Significance. This algorithm is general, and I think it can be applied to many different architectures. The ability to scale this method in terms of the number of time steps in a test-time compute fashion is quite novel to the field of representation learning, and to my knowledge has not been as successful as this paper demonstrates in the past.", "weaknesses": "I would happily raise my score on the \"presentation\" category if these are addressed:\n1. I thought that the presentation of the GIRCSE algorithm was not clear in the main text in Section 3.2 / 3.3. I did however find that the \"Algorithm 1\" pseudo-code description made this much more obvious what is going on. I would recommend moving this to the main text if space permits.\n2. Figure 1 is not very clear, it looks more like what would be expected for a poster than a figure introducing the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yed3C5FGg6", "reviewer": "Reviewer_gt8m", "summary": "This paper presents a novel framework to train that enables generative embedding models. Unlike naive next token prediction which leads to nondifferentiability, it incorporates soft-token generation which can be seen as the weighted average of possible next tokens. Furthermore, these soft-tokens are trained with contrastive objective which is regularized by the their new objective which ensures that newly generated soft-tokens lead to better performance. Overall, the framework adapts existing decoder models to generative text embedding models which eventually could lead to test-time scaling behaviour observed in recent reasoning models.", "questions": "1) What is the performance of standart embeddings after your training methodology?\n2) Why do you only train with soft tokens? Can't you use the standart embeddings as an additional objective?\n3) You train with query-answer pair, have you tried using wikipedia 1 million datasets from SimCSE to train SimCSE/LLM2Vec style where we rely on augmentations?\n4) What is the number of non-soft tokens, because even generating 20 soft tokens is small given the large document or query lengths?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The framework is novel, and idea of test-time scaling is relevant and interesting given the recent inference-time scaling efforts.\n* Performance is robust across different datasets and models.\n* Paper is well-written and easy to follow.", "weaknesses": "* A significant test-time scaling [methodology](https://jina.ai/news/scaling-test-time-compute-for-embedding-models/) for embedding model is not cited.\n* Test-time scaling claims are ambitious, gains are marginal or non-existent. Even though methodology is interesting, I dont think generating more tokens leads to similar observations made in o1/s1 style reasoning works. Scores barely change, and decrease in the causal-eos is expected since they are not trained to generate new tokens. Comparison with a model like GritLM would be more accurate.\n\n* 5-20 tokens are really low for test-time scaling, generating hundreds of tokens would be more interesting to draw any conclusions. Moreover, model is limited to 5 token generation during training. This is quite conservative and reduces the scope of paper.\n\n*  Instruction following setup is not ideal. They are often used to measure models' generative performance. Popular benchmarks are MMLU, GSM8K, Alpaca, Human eval etc. Current evaluation setup is not diverse enough to capture instruction following or any other generative task performance. GritLM's setup is much more preferred and should be used in my opinion.\n\n\n\n\n\n\n\n\n\n* There is a typo in line 153, it should be decoder instead of encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YWib6I5DMi", "reviewer": "Reviewer_VYNC", "summary": "This paper proposes a generative framework that lets LLMs “speak embedding languages” instead of acting as static encoders. The key idea is to iteratively generate soft tokens that refine sentence representations under a new Iterative Contrastive Refinement (ICR) objective.", "questions": "1. Given that the number of refinement steps (K) is fixed, have the authors considered adaptive stopping criteria based on semantic saturation?\n2. What evidence supports the claim that soft tokens capture latent semantics beyond surface-level similarity? Could the authors visualize or cluster the soft-token space?\n3. Is gradient flow stable through the soft token generation? Since each soft token is a weighted sum over all vocabulary embeddings, gradients must pass through the softmax distribution. In multi-step generation, could this cause gradients to become unstable or too weak (for example, when the softmax becomes too flat or too sharp)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The core idea is creative and well-motivated\n2. The paper is clearly written and supported by both quantitative and qualitative analyses.\n3. provides a neat way to connect generation and contrastive learning.", "weaknesses": "1. The framework fixes the number of refinement steps (K) during training and inference; an adaptive stopping criterion based on semantic convergence could make the process more efficient and principled.\n2. While the paper claims that soft tokens capture latent semantics, there is no direct evidence or visualization showing how these tokens differ structurally from standard hidden states.\n3. It is unclear whether gradient flow through the softmax-weighted soft tokens remains stable in multi-step generation. Flat or peaked distributions could cause weak or unstable gradients, potentially limiting refinement quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "haP2aqsQOP", "reviewer": "Reviewer_Ja2S", "summary": "This paper raises a problem of existing LLM-based embeddings: their overlooking of LLMs' generalization capabilities in embedding learning. To address this issue, this paper proposes a novel framework that employs autoregressive generation to iteratively refine semantic representations, thereby enhancing embedding learning. Extensive experimental results demonstrate the effectiveness of the proposed approach.", "questions": "**Questions:**\n\n- Could prompt engineering help steer LLMs to generate more useful tokens and potentially reduce the required number of iteration steps? It would be valuable to investigate whether task-specific or constrained prompts that guide the topic or style of generated tokens could improve both efficiency and effectiveness.\n- How to set the soft token number properly? especially when using it in real-world applications.\n\n\n**Suggestions:**\n- The claim that \"LLM-based embeddings typically adopt an encoder-only paradigm\" may cause confusion, as many LLM-based embedding methods still employ LLMs in an autoregressive manner, thus maintaining a decoder-only architecture.\n- Correct quotation marks to proper LaTeX formatting: \"GP\" → ``GP'', \"TTS\" → ``TTS''", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed approach is novel and interesting. It effectively exploits the generative capacity of LLMs for embedding learning.\n- The soft token generation mechanism is particularly elegant, enabling the iterative refinement process to remain differentiable while capturing richer semantic information from generated tokens.\n- The experimental results demonstrate competitive performance compared to strong baselines across diverse evaluation tasks, validating the effectiveness of the proposed approach.\n- The analysis of iteration steps (number of generated tokens) provides valuable insights into the trade-off between computational cost and performance, offering practical guidance.", "weaknesses": "- While the method incurs higher computational costs than existing LLM embeddings, the KV cache can alleviate the efficiency issue. \n- The paper does not provide code\n- It would be better to also discuss the limitations\n- The ablation study could be improved: While soft token generation is interesting and beneficial for optimization, the paper lacks a comparison between (a) directly encoding the concatenation of input and generated text, and (c) the proposed soft token approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)