GRACE: Generative Representation Learning via Contrastive Policy Optimization
Authors: Jiashuo Sun
Shixuan Liu
Zhaochen Su
Xianrui Zhong
Pengcheng Jiang
Bowen Jin
Peiran Li
Weijia Shi
Jiawei Han
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Text Representation
Reinforcement Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: GRACE reimagines contrastive learning as reward-guided generative reasoning, turning LLMs into interpretable embedders that generate explicit rationale traces. It boosts MTEB performance by up to 11.5% while preserving general capabilities.

Abstract: Prevailing methods for training Large Language Models (LLMs) as text encoders rely on contrastive losses that treat the model as a black-box function, discarding its generative and reasoning capabilities in favor of static embeddings. We introduce \GRACE{} (Generative Representation Learning via Contrastive Policy Optimization), a novel framework that reimagines contrastive signals not as losses to be minimized, but as rewards that guide a generative policy. In GRACE, the LLM acts as a policy $\pi_\theta$ that produces explicit, human-interpretable rationales—structured natural language explanations of its semantic understanding. These rationales are then encoded into high-quality embeddings via mean pooling. Using policy gradient optimization, we train the model with a multi-component reward function that maximizes similarity between query--positive pairs and minimizes similarity with negatives. This transforms the LLM from an opaque encoder into an interpretable agent whose reasoning process is transparent and inspectable. On MTEB benchmark, GRACE yields broad cross-category gains: averaged over four backbones, the supervised setting improves overall score by 11.5\% over base models, and the unsupervised variant adds 6.9\%, while preserving general capabilities. This work treats contrastive objectives as rewards over rationales, unifying representation learning with generation to produce stronger embeddings and transparent decision traces.

Published as a conference paper at ICLR 2026
GRACE: GENERATIVE REPRESENTATION LEARNING VIA CONTRASTIVE POLICY OPTIMIZATION
Jiashuo Sun'
Shixuan Liu²
Zhaochen Su Xianrui Zhong'
Pengcheng Jiang'
Bowen Jin'
Peiran Li
Weijia Shi Jiawei Han'
University Illinois Urbana- Champaign 2 AAustrllian National University "Hong Kong University of Science and Technology *University of Wisconsin-Madin 6 University of Washington
ABSTRACT
Prevailing methods for training Large Language Models (LLMs) text encoders rely on contrastive losses that treat the model as a black-box function, discarding its generative and reasoning capabilities in favor of static embeddings. We introduce GRACE (Generative Representation Learning via Contrastive Policy Optimization), novel framework that reimagines contrastive signals not as losses to be mini- mized, but as rewards that guide generative policy. In GRACE, the LLM acts as a policy Ty that produces explicit, human-interpretabbe rationales-Struutuut language explanations of its semantic understanding. These rationales are then encoded into high-quality embeddings via mean pooling. Using policy gradient optimization, we train the model with multi-componett reward function that max- imizes similarity between query-positive pairs and minimizes similarity with nega- tives. This transforms the LLM from opaque encoder into interpretable agent whose reasoning process is transparent and inspectable. On MTEB benchmark, GRACE yields broad cross-category gains: averaged over four backbones, the super- vised setting improves overall score by 1 11.5% over base models, and the unsuper- vised variant adds 6.9%, while preserving general capabilities. This work treats trastive objectives as rewards over rationales, unifying representation learning with generation to produce stronger embeddings and transparent decision traces. Our code is publicly available at lhttp::/ http//gttttt fiighub...... com/ com//Gaas1s1L53333//2222
natural
70
65
e Mila LLM2vec
Embedding Performance
60
sest-embedding--
GRACE-IB
55
GRACE-3B
50
GRACE-1.5B
Gemini
Quen-3-4B-iIntreact
40
0
Grek
GPT
LLaMA 3.2.D Instruct
DeepSeek
Gemini
35
Qwen-2.5-1.58 Instract
30
20
50 Generative Performance
70
Figure 1: Joint comparison generative and embedding performance across existing basclines and our GRACE models. Embedding models cluster at the top-left (strong embedding, weak generation). while generative models occupy the bottom-right (strong generation, weak embeddings). Trained without additional generative supervision, GRACE models shift instruction- -uned bases upward, simultaneously improving embedding performance while retaining generative competence.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The advent of Large Language Models (LL.Ms) has marked a paradigm shift in the field of Natural Language Processing (NLP) (OpenAI, 2023] Yang et al., 2024 2025; Dubey et al., 2024). Owing to their vast parameter scale and pre-training on massive text corpora, these models have demon- strated remarkable capabilities in language understanding, reasoning, and generation, leading to breakthroughs in nearly all NLP tasks. One of the most critical application areas is the use of LLMs as universal text encoders to provide high-quality semantic representations--rr or text embeddings- downstream tasks such as semantic retrieval, text clustering, and recommendation systems Wang et al. (2024) BehnamGhaaerr al. (2024); Lee al. (2025 |Springer et al. (2025). From BERT[Devlin et al. 2019) to contemporary instruction-tune LLMs Ouyang et al. (2022), , the research community has persistently explored methods more effectively leverage the knowledge within these models to construct more accurate and robust representation spaces.
However, the prevailing paradigm for training LLMs as representation models harbors a profound inherent contradiction. Current approaches predominantly rely on contrastive learning frameworks that optimize discriminative objectives such as InfoNCE loss van den Oord et al. (2018), treating the L.M merely as parameterized encoder function So X + Iे.. This paradigm forces these inherently generative models to produce static embedding vectors through simple pooling mechanisms, fundamentally suppressing their capacity for structured reasoning and natural language generation. The model learns to minimize distances between positive pairs while maximizing distances from negatives, but this process occurs entirely within opaque latent space. Consequently, we lose the interpretability that makes LLMs valuable-the ability to understand and articulate their reasoning process. When an embedding model determines that two texts are similar, we cannot inspect why made that judgment or which semantic features it prioritized.
This fundamental limitation motivates to reconceptualize the role of contrastive signals in represen- tation learning. Rather than treating contrastive objectives as loss functions to be minimized through gradient descent, we view them as reward signals that guide a generative policy. This perspective naturally leads to a reinforcement learning framework in which the LLM acts as policy To that generates interpretable understandings of input texts. These understandings serve dual purpose: they provide human-readable explanations of the model's semantic reasoning and are simultaneously encoded into high-quality representations of the inputs. By formulating representation learning as sequential decision-make making problem, we leverage the full generative capacity of LLMs, yielding interpretable reasoning and effective text representations.
To realize this vision, present GRACE (Generative Representation Learning via Contrastive Policy Optimization), a framework that turns LLMs into interpretable representation learners using volicy-gradien optimization. The model first produces explicit rationales r that analysizes and reasoning the input. From r we derive the final embedding h via mean pooling over hidden states. We recast contrastive learning signals as rewards that increase query-positive similarity and decrease query-negative similarity. Optimizing this reward with standard volicy-gradiens methods teaches the model generate faithful rationales while simultaneously learning effective text representations.
The main contributions of this work can be summarized as follows. First, we present the first empirical evidence that rewards derived from contrastive learning can be leveraged to train policy models, resulting in improved representational capabilities. Second, we propose a novel methodology that enables the transformation of existing LLMs into powerful representation models while preserving capabilities without performance degradation, as shown in Figure | Third,
their general-purpol this work represents a substantial advancement in text representation interpretability. as the model's reasoning can be directly inspected through its textual outputs. Fourth, our method yields a significant performance gain of avg 11.5 % over baseline models when evaluated on the MTEB benchmark. Finally, to facilitate reproducibility and advance future research in this domain, we will make all models, datasets, and code publicly available.
GENERATIVE REPRESENTATION LEARNING VIA CONTRASTIVE POLICY OPTIMIZATION
We propose to fundamentally reimagine the role of contrastive signals in representation learning. Rather than treating them loss functions be minimized, we reconceptualize them as reward
2

---

Published as a conference paper at ICLR 2026
Representation Model training with Contrastive Learning
D+ positive
/ max
Query
Representation
D-
negative
Model
min
D+
D
Ours
Rationale for q
Reward
Generate
Concat
Query
Policy Model
/ max
Rationale for D+
D+
min
Rationale for D
RI.
Comparison of standard contrastive learning (top) and our RL-based method (bottom). Given query with positive (D+) and negative (D-) documents, our policy model generates rationales for q, Dt and D concatenates them obtain the final representation, and is optimized with rewards that increase similarity between the and D while decreasing similarity between the g and
Figure 2:
signals that guide a generative policy. Our framework transforms the L.L from a passive encoder that outputs static embeddings into active agent that generates interpretable rationale of the input text.
2.1 RATIONALB-GGNERRATIN GENERATING POLICY
To enable the LLM to perform structured reasoning for similarity judgments, adopt policy that generates an explicit reasoning trace (rationale) for each input. Let P(-) denote the prompting function that prepends the representation instruction the input text. For any input {q. d+ d }. the policy Te produces a structured rationale:
f- P(x)).
(1)
The rationale identifies salient semantic features, key concepts, and potential relations grounded in domain knowledge, providing a transparent trace that supports downstream similarity assessment.
2.2 FROM RATIONALE TO REPRESENTATION
Given the generated rationale for input F, obtain contextualized hidden states by conditioning on both the caaructioo-augmeent input and the rationale:
E no (P(x) @ r) IR Lxd
where L is the sequence length, d the hidden dimension, and @ denotes concatenation. To focus on semantic content while excluding instructional artifacts, we apply masked mean pooling over the last-layer hidden states:
Σ E, = {t: Lays < <t L, mask, 1}.
M
where Rd is the final representation and Lays denotes the length of system prompt tokens to be excluded. Anchoring representation extraction to an explicit rationale yields semantically rich, interpretable embeddings and enables more reliable similarity judgments.
2.3
CONTRASTIVE REWARDS POLICY GUIDANCE
Given the recent advances of outcome-reward and policy optimization methods in LLMs, and the practical difficulties and inefficiencies of training value models, our algorithm is correspondingly designed around based optimization approach. For exposition, adopt Group Relative Policy Optimization (GRPO) Shao et al. (2024) to walk through the procedure, though our framework agnostic to the specific policy-gradient algorithm.

---

Published as a conference paper at ICLR 2026
2.3.1 ROLLOUT STRATEGY
To balance exploration diversity with computational efficiency, we adopt an asymmetric rollout strategy. For each training instance (g:, df. d,), we employ different generation strategies based on text type:
yq. (-(1((())) 92, (.(((())))
(4)
For positive documents d we perform K stochastic rollouts generate diverse rationales, enabling the model to explore different interpretational perspectives of the same content. In contrast, queries qa and negative documents d, undergo single-sample generation produce reference representations. which serves as fixed anchors for reward computation. The computed similarity- -based contrastive rewards facilitate advantage estimation within the GRPO framework, driving policy updates that enhance both the quality and diversity of text understanding across all input types.
2.3.2 REWARD DESIGN
We design a composite reward function that translates contrastive learning objectives into actionable policy guidance. For each positive document rollout Va we compute four synergistic reward components that collectively shape the learning dynamics.
Let Dj- {d m,m m=1 be the of negatives for query q, The foundation the contrastive learning reward RCL. which encourages semantic alignment between queries and relevant documents while penalizing spurious correlations with irrelevant ones:
M Σ
(c.M) sim b. h R
sim h h hd
(5)
To ensure semantic coherence across multiple interpretations of the same document, we introduce the consistency reward, which encourages similar representations among concurrent rollouts:
K sim
R (3, consist
5 3 47 5
K
Most critically, we incorporate hard negative mining inspired by in-batch negative sampling strategics. For each query, we identify the most challenging distractors (positives from other training instances that exhibit spuriously high similarity). Let B denote the batch size and E (1...., K} index rollouts. We select, for each other instance j. the maximum similarity across its rollouts obtain the hardest distractor for the current query:
R
hard
1
sim Linhhe h h n.)
B
IIKK
The composite reward integrates these complementary objectives:
R (i.k)
toral
RaL
(i.k)
>a R2 (i.k)
consist + X R(2
hard
(8)
where A and A hyperparameters that balance the relative contributions of consistency preserva- tion and hard negative discrimination.
To sharpen the reward distribution and stabilize training dynamics, apply temperature scaling to the composite rewards:
R (i. total
(1.
Total
where is the reward temperature parameter that controls the sharpness of the advantage distribution.
4

---

Published as a conference paper at ICLR 2026
2.4 POLICY OPTIMIZATION OBJECTIVE
Following the GRPO framework, we compute advantages relative to the group baseline but remove standard deviation:
(
(1,A) Rinal
R
Final
(3,1)
(10)
K
The policy is then optimized to maximize the advantage-weighres likelihood:
A log TO ( (P((t))(
Lioial - [gg ,d- JoD
(11)
Since our optimization on-policy, we omit importance sampling here.
2.5 UNSUPERVISED LEARNING EXTENSION
Inspired by SimCSE's Gao al. 202 unsupervised paradigm, we extend our framework to settings where only raw text available without explicit query-document pairs. The key insight that different interpretations of the same text should maintain semantic coherence while being distinguishable from interpretations of different texts.
Given batch of texts B {x, Me perform asymmetric rollouts for each text: anchor
~ To( (P()))(x)),,
(vA)) k== To(-1P(xx))
(12)
Here the anchor serves as the positive counterpart for its own rollouts, directly analogous SimCSE's same sentence positive constructed via independent noise, while the rollouts probe diverse yet semantically consistent interpretations of 2;.
We instantiate the unsupervised reward with the self-alignment term between the anchor interpretation and each rollout of the same text. Given the anchor representation fannchor and rollout h the reward is
R(i
sim (annhaaanee Ln
Jannhhor
(k) P
(13)
The remaining terms, within-text consistency across rollouts of the same F, and in-batch hard- negativee mining against other texts in the batch- are identical to their definitions in Sec. [23.2] For brevity, we omit their formulas here. When enabled, the overall unsupervised objective is the same weighted combination in Sec. 2.3.2
3
EXPERIMENT
3.1 DATASETS AND EVALUATION METRICS
We conduct comprehensive evaluations using the Massive Text Embedding Benchmark (MTEB) Muennighoff et al. (2023), standardized framework that covers 7 task categories and 56 datasets, spanning retrieval (Retr.), reranking (Rerank.), clustering (Clust.), pair classification (PairClass.), classification (Class.), semantic textual similarity (STS), and summarization (Summ.). For repre- sentation aggregation, we adopt mean pooling Reimers & Gurevych (2019), explicitly excluding instruction tokens to avoid instruction-speeifie artifacts.
3.2 BASELINES
Supervised Baselines We compare four variants in the supervised setting: (1) Base (No Training) uses the pre-trained instruction-tuned model without any representation-specffc optimization and serves as the starting point; (2) Base Reasoning applies our reasoning prompt to the same model but performs further training, isolating the effect of reasoning-style outputs on embeddings; (3)
Contrastive Learning (CL) fine-tunes the base model with a standard InfoNCE objective using in-batch negatives Chen et al. (2020), representing the predominant contrastive approach: and (4)
GRACE introduces reward- reward-guided policy optimization that explicitly aligns generative reasoning with representation quality.

---

Published as a conference paper at ICLR 2026
Categories # of datasets +
Retr. 15
Rerank. 4
Clust. 11
PairClass. 3
Class. 12
STS 10
Summ. 1
Avg.
56
Qwen2 5-1. 5B-Instruct
Base
22.15
29.32
25.44
36.18
35.77
44.11
26.32
30.33
w/ reasoning
24.83
32.45
27.88
39.20
39.42
47.26
26.78
32.92
w/ CL training
38.95
43.88
36.21
52.02
53.87
56.39
28.43
43.21
GRACE
40.44
46.95
39.55
54.84
55.36
59.42
30.41
45.48
LLaMA-3
- rust
Base
31.28
38.16
32.05
48.12
47.36
59.25
27.78
39.34
w/ reasoning
33.21
40.44
34.28
51.27
50.89
61.78
28.14
41.54
w/ CL training
42.42
47.35
39.92
58.66
58.15
65.55
28.63
47.39
GRACE
44.01
49.12
41.30
60.44
60.72
64.02
29.10
48.49
Qwen2 1B-3BInnstrutt
Base
37.38
44.16
36.85
53.72
53.36
66.15
26.26
44.12
w/ reasoning
39.42
46.55
38.82
56.61
57.23
68.22
28.55
46.59
w/ CL training
45.90
52.87
43.26
74.08
65.94
70.05
29.68
52.10
GRACE
49.42
54.85
44.73
79.64
68.25
74.65
30.10
54.74
Qwen3 A-B-Instrcct 2507
Base
37.42
48.16
38.55
55.33
54.87
66.02
29.44
45.49
w/ reasoning
38.91
49.72
40.76
57.20
55,41
68.35
29.62
46.87
w/ CL training
48.66
53.38
43.02
78.81
69.94
74.12
29.91
54.34
GRACE
52.11
55.85
45.24
82.94
71.02
77.38
30.46
56.64
Table 1: Supervised results on MTEB.
Unsupervised Baselines In the unsupervised setting, we report: (1) Other Open Models, including representative encoder baselines (e.g., BERT Devlin et al. (2019), RoBERTa Liu et al. (2019) and recent LLM-embedding methods (e.g., LLM2Vec BBenamGhader
(2024), Echo SSringer et al. 2025 (2) Base (No Training), the same instruction-tune tuned model used zero-shot without representation-speeiin training; (3) SimCSE, which fine-tunes with the unsupervised SimCSE
objective based on dropout-induced positives Gao et al. (202T); and (4) GRACE, which applies the same soward-guided policy optimization to align generative reasoning with embedding quality in an unsupervised regime.
Implementation Details We conduct experiments with four decoder-only language models: Qwen2.5-1.55 1.en2.5-1.5B/3B-. Instruct Yang et al.] (2024), Qwen3-4B Yang et al. (2025). and LLaMA-3.2. 3B-Instruct et al. (2024). For supervised training, we use replication of the public portion of the ES dataset Wang et al. (2024), which contains 1. 1.5M samples following BehnamGhader et al. (2024). For unsupervised training we follow the SimCSE Gao al. (202I) setup without collecting any additional unlabeled corpus. All experiments are run on single node with 4x NVIDIA H100 GPUs (94GB each). Training is performed for epochs with batch size of 64. We the maximum prompt length to 1024 tokens and the maximum response length to 2048 tokens, applying right-side truncation when sequences exceed these limits.
3.3
MAIN RESULTS
Supervised Results Table reports supervised MTEB results across seven task families for four dncoder-only backbones and four training settings. Averaged across the four backbones, our method improves the MTEB average by 11.52% over the Base model, indicating consistent improvements across model sizes and families. The gains are broad based, with especially strong uplift retrieval and pair classification, while classification, clustering, STS, and summarization also improve. The intermediate variants also contribute: explicit reasoning provides modest but reliable increase, and contrastive training further enlarges margins; the final method delivers the most balanced cross-task performance.

---

Published as a conference paper at ICLR 2026
Categories > # of datasets >
Retr. 15
Rerank.
Clust. 11
PairClass.
Class. 12
STS
Summ.
Avg.
10
56
Other Open Models
BERT 10.59
43.44
30.12
56.33
61.66
54.36
29.82
38.33
RoBERTa
62.63
29.05
56.95
41.92
8.62
55.24
28.64
37.86
LLM2VeLLLMA-3AR
24.75
49.20
39.74
65.91
69.00
67.85
25.59
48.84
EchoMistnd-78
71.63
33.51
72.31
47.43
22.85
73.64
31.02
49.02
Qwen2.
5-1. IB-Instruct
Base
22.15
29.32
25.44
36.18
35.77
44.11
26.32
30.33
w/ SimCSE
31.28
38.94
33.12
50.67
47.53
58.21
28.34
39.65
GRACE
34.57
41.86
35,49
51.12
50.78
56.44
29.07
41.45
LLaMA-3 3. 2-3B- Instruct
Base
31.28
38.16
32.05
48.12
47.36
59.25
27.78
39.34
w/ SimCSE
35.42
41.27
34.96
52.88
50.73
65.44
29.24
43,00
GRACE
36.55
43.15
36.88
55.27
53.62
63.18
29.52
44.04
Qwen2.
5-3B--3mm- IBB-Instuct
Base
37.38
44.16
36.85
53.72
53.36
66.15
26.26
44.12
w/ SimCSE
42.25
48.33
39.87
68.22
60.15
70.84
29.56
49.17
GRACE
43.15
49.72
41.58
70.44
62.91
69.38
29.63
50.15
Qwen
4B- 10-IInsrruut22507
Base SimCSE
37.42
16.16
38.55
55.33
54.87
66.02
29.44
45.49
42.18
50.72
41.63
69.14
61.25
72.48
29.62
50.11
GRACE
43.67
52.34
42.87
70.05
62.73
71.66
30.16
51.03
Table 2: UnSupervised results on MTEB.
Unsupervised Results Table summarizes unsupervised MTEB performance and shows consis- tent stepwise improvement from Base, SimCSE and GR ACE across all four backbones. Averaged across backbones, our unsupervised method improves the MTEB average by 6.85% over the Base model. Relative to widely used open baselines, our best unsupervised results (c.g.. Qwen3-4B and Qwen2.5-3B) past LLM2Vec and Echo, while comfortably exceeding encoder-only BERT/ROBERTa averages. Improvements are broad based, with retrieval and pair classification benefiting most, and STS remaining strong without sacrificing performance on classification or clustering. These trends suggest that even without supervised signals, injecting reasoning- contrastive objectives yields robust, transferable enhancements over both naive LLM pooling and standard SimCSE-style tuning.
3.4 ABLATION STUDIES
The MTEB benchmark covers a wide range of embedding tasks across diverse types. domains, and difficulty levels. For computational efficiency and comparability, following BBchnamhhader al.) (202)). we evaluate representative 16-task subset for ablations and analysis (Table A.5.1).
3.4.1 ABLATION ANALYSIS OF REWARD FUNCTION DESIGN
We conduct a comprehensive ablation study on the two key hyperparameters our reward function: consistency (Az) and hard negative mining (A2), evaluating a 5 x 5 grid of configurations with five discrete values for each parameter (0.0, 0.3, 0.5, 0.7, 1.0). Figure presents the performance landscape for GRACE-3B under both supervised and unsupervised paradigms, revealing several critical insights. The results demonstrate that removing all reward constraints (A₃ = 0, X2 = 0) yields poor performance for supervised and unsupervised training respectively, confirming the necessity of these structured reward signals. When only one reward component is active, pure hard negative mining
(A1 0, λ2 1) reaches 50.2 % and 45.2 %, while pure consistency weighting (A₃ 1, X2 0) achieves 49.3 and 45.0 %, indicating the former provides a better baseline. Notably, the model exhibits significantly higher sensitivity to the hard negative mining weight (A2) compared to the

---

Published as a conference paper at ICLR 2026
1.0
50.2
51.0
50.8
50.1
50.3
1.0
45.2
46.6
46.0
45.4
45.7
56
53.4
53.2
0.7
47.1
48.1
47.5
54
A2
55.4
55.9
56.0
55.9
55.1
A2
0.5
49.0
49.7
49.9
50.5
49.2
0.3
52.5
53.1
53.2
52.6
52.8
0.3
51.2
50.5
51.3
51.5
51.0
0.0
49.5
49.8
50.0
49.7
49.3
0.0
44.9
45.3
45.6
46.1
45.0
44
0.0
0.3
0.5
0.7
1.0
0.0
0.3
0.7
1.0
Figure 3: Reward function ablation study for GRACE-3B showing performance across different combinations of the consistency weight (A1) and the hard negative mining weight (A2). Left: supervised training; Right: unsupervised training. The heat intensity indicates performance levels, with darker red representing higher scores.
consistency weight (A1), suggesting that hard negative discrimination plays a more critical role in determining overall performance.
3.4.2 COMPARISON WITH ALTERNATIVE RL ALGORITHMS
To verify that approach is not tied specific optimization scheme, also applied it different RL algorithms. As shown Table 3.4.2 our method consistently improves performance across all three algorithms, demonstrating both its portability and generalizability. Among them, GRPO remains the most effective in our setting, while REINFORCE++ ReMax and DAPO bring smaller gains, likely because their design focuses issues (e.g., reward hacking, weak credit assignment and long-CoT instability) that are less critical in our tasks. These findings confirm that our framework can be readily integrated with diverse RL optimizers, ensuring wide applicability.
Algorithm
Retr.
Rerank.
Clust.
ClisClass.
Class.
STS
Summ,
Avg.
datasets
3
GRACE-3B
ReMax et al. 2024
42.63
61.24
35.61
62.3
70.8
29.04
53.36
REINFORCEF:
2025
43.10
64.9
37.67
68.0
63.18
71.60
29.91
54.64
DAPO Yu al. 2025
44.58
65.62
39.24
70.67
63.58
72.8
30.04
55.78
w/ GRPO Staa el al.
2024
44.72
65.71
38.99
70.87
64.35
72.53
30.09
55.89
Table 3: Comparison of our supervised method with different RL algorithms on subsets of MTEB.
3.4.3 GENERALIZATION GENERAL DOMAIN TASKS
Table evalluats whether embedding- oriented fine-tuning affects broad capabilities across mathemat- ics (GSM&K Cobbe et al. (2021)), knowledge and reasoning (MMLU Hendrycks et al. (2021). BBH Suzgun et al. (2023), TriviaQA Joshi et al. (201)), FEVER Thorne el al. 2018), and code generation (HumanEvall Chen al. (2021). Across four backbones. our method preserves general performance:
both supervised and unsupervised settings yield near-zero average shifts relative to the instruction- tuned base. In sharp contrast, the CL fine-tuning baseline suffers severe deterioration, indicating that naive contrastive objectives can substantially erode general-domain competence. We attribute the stability of our approach the way the contrastive signal integrated into learning. Rather than minimizing token-agnostic loss, we optimize contrastive reward within an RI. framework, which (1) aligns updates with the generative policy, preserving instruction following and problem solving while shaping the embedding geometry; (2) uses relative, ddlaanaage-weighie updates that resist representation collapse and the rescaling/drift common direct Info training; and (3) keeps reasoning generation intact the policy continues practice skills needed for general tasks. As result, representations improve for retrieval objectives without sacrificing the general capabilities conferred by pretraining and instruction tuning.
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance General Domain Tasks
Dataset +
GSM8K EM
MMLU
TriviaQA EM
FEVER
BBH
HumanEval Pass
Avg.
A
Metric >
EM
Acc
EM
Qwen-2 2.--- 5B- Instruct
Base
32.06
18.35
66.91
25.25
46.95
40.74
Bsse w/ CL training
0.0
0.0
0.0
50.28
0.0
0.0
8,38
-32.36
GRACE (Supervised)
32.54
55.12
18.10
67.43
25.01
47.30
41.08
+0.34
GRACE (Unsupervised)
32.21
54.81
18.29
66.75
25.34
47.05
40.88
+0.14
LLaMA-3.1
2-3---3-B-- -nstuutnt
Base
16.75
14.74
29.21
64.52
9.72
38.41
28.89
Bsse CI. training
50.01
0.0
8.33
-00.56
GRACE (Supervised)
17.02
15.01
29.05
65.10
9.61
38.90
29.27
+0.38
GRACE (Unsupervised)
16.81
14.69
29.18
64.40
9.80
38.55
28.91
+0.02
Qwen 2 5-33--B---- Instruct
Basc
57.90
62.60
28.80
71,50
35.00
52.80
51.40
Bsse w/ CL training
0.0
0.0
0.0
50.13
0.0
0.0
8,35
-43.05
GRACE (Supervised)
59.10
61.20
27.60
72.80
34.30
53.10
51.50
+0.10
GRACE (Unsupervised)
58.00
28.30
71.20
34.80
52.90
51.10
-0.30
Qwen 3-4B 40--BB-aa Instructttttttte -2507
Base
75.96
69.45
31,55
83.53
35.01
68.90
60.73
Base w/ CL training
0.0
0.0
0.0
51.07
0.0
0.0
8.51
-52.22
GRACE (Supervised)
76.42
69.71
31.40
84.02
34.88
69.35
61.13
+0.40
GRACE (Unsupervised)
76.05
69.38
31.52
83.40
35.09
69.01
60.74
+0.01
RELATED WORK
LLM as Embedding In recent years, the rise of large language models (LLMs) has sparked growing interest in using them directly text embedding models Zhang al. (2025): Yan al. 2025);Ji et al. (2025). Research in this area has generally followed two paths: tuning-free free approaches Li Zhou (2025p: Springer et al. (2025), which study the impact of instructions on embedding quality, and tuning-bait based approaches Muennighoff et al. (2025); BehnamGhader et al. (2024). which adapt models for improved performance. While tuning-free methods offer simplicity, state-of-the-an results increasingly rely on tuning-based strategies. Notable examples include the BGE Xiao et al. (2024) and GTE et al. (2023) series, which strengthen semantic representations through contrastive learning large-scale text pairs. Building this line of work, reinterpret contrastive learning as a reward signal guiding a generative policy, turning LLMs into interpretable representation models.
Reinforcement Learning in Reasoning Reinforcement learning has been central to advancing reasoning language models, exemplified by DeepSeek-FR DeepSeek-AT et al. (2025). which achieved major breakthroughs with Reinforcement Learning with Verifiable Rewards (RLVR), RL formulates reasoning as a policy optimization, improving capabilities by maximizing expected rewards. Algo- rithms such as PPO Schulman al. (2017) and GRPO Shao et al. (2024) dominate, with GRPO boosting efficiency by removing the critic model and adopting group- wise reward normalization. Extensions including DAPO Yu et al. (2025), ReMax [i et al. (2024), and REINFORCE++ Hu et al.) (2025) further refine this paradigm. Inspired by these advances, we reinterpret contrastive learning as reward signal that guides a generative policy, thereby turning LLMs into representation models.
CONCLUSION
We propose GRACE (Generative Representation Learning via Contrastive Policy Optimization), a novel framework that reframes contrastive signals as rewards for a generative policy, turning LLMs from opaque encoders into interpretable representation leamers. Optimized with standard policy-gradient gradient updates, GRACE directly shapes the reasoning that gives rise to embeddings, yield- ing representations whose semantics inspectable through the rationales. Empirically, GRACE delivers consistent, cross-cate cross-category gains on MTEB across multiple backbones in both supervised and unsupervised settings, while preserving general capabilities non-embedding tasks.

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
This work adheres to the ICLR Code Ethics. All datasets used in our experiments, including the public portions of E5 and the MTEB benchmark, publicly available and widely adopted in the community. No personally identifiable, sensitive, or proprictary data were collected or used. Our study focuses on methodological contributions representation learning, without direct experimentation on human subjects or deployment in safety-critical applications. While we do not anticipate immediate societal harms, we acknowledge that improved embedding models could inadvertently amplify biases present in training data. We encourage future work to systematically evaluate fairness, bias, and robustness in broader contexts. The authors declare that there are no conflicts of interest related to this submission.
7 REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of our work. A detailed description of our framework, including the reasoning policy architecture, rollout strategies, and reward design, is provided in Sections 2 and Comprehensive implementation details, such model backbones, datasets, training configurations, and evaluation metries, are reported in Section B and further expanded in the Appendix. To facilitate replication, we followed standardized benchmarks (MTEB) with publicly available datasets and describe all preprocessing steps the supplementary materials. Our ablation studies 3.4 further validate the robustness of our design choices. We commit to releasing the codebase, trained model chockpoints, and scripts for data processing and evaluation in open-source repository soon as possible, so that all reported results can be independently verified.
8 ACKNOWLEDGEMMET
Research was supported in part by the AI Institute for Molecular Discovery, Synthetic Strategy, and Manufacturing: Molecule Maker Lab Institute (MMLI), funded by U.S. National Science Foundation under Award 2505932, NSF IIS 25-37827, and the Institute for Geospatial Understanding through Integrative Discovery Enviromment (I-GUIDE) by NSF under Award No. 2118329. The research has used the Delta/DeltaAl advanced computing and data resource, supported in part by the University of Illinois TirbnnaChhappice and through allocation #250851 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants OAC 2320345, #2138259, #2138286, #2138307, #2137603, and #2138296.
REFERENCES
Anthropic. Claude 3.5 sonnet model card addendum, 6 2024. URL https: www cdn anthropic com/
Bard_Claude 3_ddddd_dddmmmmm_ pdf Addendum to the Claude 3 model card.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vee: Large language models are secretly powerful text encoders. CoRR. abs/2404.05961, 2024. doi: 10-48550ARXXI.24400000
URL
org/10
48550/arXiv 2404 05961
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating
10

---

Published as a conference paper at ICLR 2026
large language models trained on code. CoRR, abs/2107.03374, 2021. URL https:
org/abs/2 aws/abc7.............. 03374
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 1 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020. URL
w.r.ppees/vill httSsss/v19///oe 22tess//119/hhee22).hhaa wttp//h.nnnnnam.em
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, 1is/2110.14168, 2021. URL
https://as wwerxiv..orgaaee alos/ 2110 14168
DeepSeck-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li. J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, J. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. L.i. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. CoRR, abs/2501. 2025. URL
https://dot org/oogggoooooooooooooooo 10. 48550/ arXiv. 2501. 12948
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/V/N19--4225, URL
https://dol: 1ttp:///....111//////11111111111
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Ilama herd of models. CoRR. abs/2447.221788 2024. doi: 10.48550/RXXV.2400.2222 17783. URL https: 240.7.....4......0 227833
//doi.org//1 000.l.oorg/1..4555//aaaaa 48550/
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih
11

---

Published as a conference paper at ICLR 2026
(eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6894 6910. Association for Computational Linguistics, 2021. doi: 10.18653/V
552. htt.hhtts:////. //ooi .eloooo..rr//// www//o................ 8653 2016653311222 http////11112 . emn
552
Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning and long-context multi- modality, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 9th International Conference on Learning Representations, ICLR 2021, Viriual Event Austria, May 3-7, 2021. OpenReview.net, 2021.
net/ wttp://ooummmmoo////
jmI 3GmQ
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025.
Yifan Ji, Zhipeng Xu, Zhenghao Liu, Yukun Yan, Shi Yu, Yishan Li, Zhiyuan Liu, Yu Gu, Ge Yu, effective representations for dense retrieval through deliberate
and Maosong Sun. Learning thinking before search. CoRR, abw/2552..129744.. 2025. doi: 1044555/ARXIV2550.2222 httpss://oli org/10..rgg11... 48540 /a./S0/arXiv/// 2502 12974
URL
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.). Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume I: Long Pupers, pp. 1601-1611. Association for Computational Linguistics, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pp. 6769-6781, Association for Computational Linguistics, 2020. doi: 0016665/1122220.MMPMMMMM.M ponlp-maiin 550
URL https: org/ 181..11112112222
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training iims as generalist embedding models. In The Thirteenth International Conference on Learning Representattions ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL 1gsyLSsDRel
net forumm
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. CoRR, doi: 0046550/ARXXV22200.0322 URL Ittt::://od..or//11..e 0155://....555//aa// 03281
2023,
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL httpt::t////// http:s////ppereviee...oo net forum?id 5tr8hXkpe
Ziyue Li and Tianyi Zhou. Your mixture-of-experts LLM is secretly an embedding model for free. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL htw.t:ttp////cccc http:////ppeeeeeeeeeeeeeeeeeeeee net.://wweeeeeeeeee.eeee.ee www.foruttettet.oooo eGG977
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 11tp/1977.1199.... 2019. htt 20000//////ppp/////rrtt/////////
12

---

Published as a conference paper at ICLR 2026
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, EACI. 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 2006-2029. Association for Computational Linguisties, 2023. doi: 10.18653/V1/2023, FAC.AA/IIA...AAAAAAAAAAA FACL-MAIN.145 URL. https
//doi. wwt../11661vv21 201.5//////222..co wwwaaaalmaanna..a.aaa main. 148
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In The Thirteenth International Con- ference on Learning Representatiinss ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. httppppse/peeeeeeee.eeeeetetrere
IIvE22v
OpenAL GPT-4 technical reportt CoRR, wbtp//aa../3/370/4..... doi: 1048550ARXXIV2303.0777 URL
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.). Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: http:://apaprr.ee nips oC papprappppp____ www.fiiiaappppppp.eappaaaapaeeae 22tp://22pppe222/////22222 201///22///2222 2022/hashh http://ewss.nntosocpppppeeeeeeee
html
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,
2019, Hong Kong. China, November 3-7, 2019, pp. 3980-399999 3990. Association for Computational Linguistics, 2019. doi: 10.18653VV//D1-- 1410. htt.https://do/ wrw.d/oooorgg/... 00w....666///111111
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, fass/777.006344 2017. URL htttp://axxiv.o http//axxxvvo..gabbb. /
1707. 0634
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingehuan Zhang, K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300 2024. doi: 10.4550/ARXXI..402..0332 https: //doi wrg.1o.rg11.... 0855500.4485000 48550/5550000 wttp://rrivvi...... 2402 03300
URL
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. CoRR, abs/2502.02013, 2025.
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28. 2025. OpenReview.net, 2025. URL .et. httpt//oormmidddddm http:/oormmm--hhh2h..2..-
https::/cpenreet
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether of -fouthtttt can solve them. In Anna Rogers, Jordan L. Bryd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational I Linguistics: ACL. 2023,
13

---

Published as a conference paper at ICLR 2026
Toronto, Canada, July 9-14, 2023, pp. 2023.
Association for Computational Linguisties,
James Thome, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and verification. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies. NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pp. 809-819. Association for Computational Linguisties, 2018.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/ 1807 01088807774400 03748, 2018 URL httt:://axxx: org
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 1897-11916. Association for Computational LinggtttiiiiiiLLiiiiiiiiiiii Linguistics, 2024, doi: 11L.1666/11222AAALLLLLLL
URL
https://doii http://....2211.111555////31 2024 acl ong. 642
Ronald J Williams.
Simple statistical gradient- alliingggggooo algorithms for connectionist reinforcemnnt learning. Machine learning, 8(3):229-2) 256, 1992.
XAI. Grok 4 model card, 8 2025.
URL
https://Gata.x.c
1025-08-20-gro-44mooeee card pdf Model card.
Shitao Xiao, Zheng Liu, Peitian Zhang. Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C-pack: Packed resources for general chinese embeddings. In Grace Hui Yang. Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pp. 641-649, ACM, 2024.
Ruiran Yan, Zheng Liu, and Defu Lian. O1 embedder: Let retrievers think before action. CORR, abs/2502.07555,, 2025. doi: 1044850/ARXIV.5002.0007
URL htttps://di..orr dtg//ooo.ooogoo/oo
18550/arXiv. 0755557555
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui. Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412. 2024. doi: 21.S50/ARRII2221215555111 URL /ttpps://dol.
15115
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, https://doi: org/ 48550/555440 a 2505. 09388
2025. doi: 0.44550/ARXXV.2555.08222 URL
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingiun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Guangming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai. Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an ooun-source LLM reinforcement learning system scale. CoRR, abs/2503. 14476, 2025.
14

---

Published as a conference paper at ICLR 2026
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding; Advanc- ing text embedding and reranking through foundation models. CoRR, abs/2506.05176, 2025.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow., (6(12):3488-388 2023.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS
In preparing this paper, used LLMs solely auxiliary tools refine the clarity and style of our writing. Specifically, LLMs were employed to help polish phrasing and improve readability of paragraphs. All research ideas, experimental designs, analyses, and conclusions were conceived and executed by the authors. We take full responsibility for the content of this work.
A.2 PRELIMINARIES
A.2.1 CONTRASTIVE LEARNING FOR TEXT EMBEDDINGS
Traditional contrastive learning for text representation follows discriminative paradigm. Given a dataset D = {(g d," D, } where 9: denotes a query, d, denotes a relevant document, and D. {dr,, } denotes irreeevan documents, an encoder fo X - Rd is trained to minimize the InfoNCE loss:
exp(sim(fo(()) fo fo(d, ))/r) eD; (oppsim(((())
LInfONCE log
(14)
where sim(::-) denotes cosine similarity and is temperature parameter. In practice, in-batch negatives Karpukhin et al. (2020) are commonly employed for computational efficiency:
B Σ log
exp(sim q )/F)
Loatch
(15)
j=1 exp(sim(q)) q. d //7)
13
where is the batch size. The effectiveness this approach critically depends on hard negative mining-ident mining--denttifing the most challenging negative samples that lie close the decision boundary.
A.2.2 POLICY GRADIENT OPTIMIZATION
Policy gradient methods optimize a parametrized policy The by directly maximizing the expected reward over generated trajectories. Given prompt r and policy ne, the probability of generating a sequence y
Te (y)lz) x) = II Tee x, yee).
(16)
With reward function r(z, y) that evaluates the quality of response y to prompt the optimization objective is the expected reward:
( Eoy) [r(x, w)].
(17)
By applying the policy gradient theorem, the gradient of the objective can be expressed as
( (yy- [r y) o log me ()(y)>)),
(18)
which provides unbiased estimator of the true gradient Williams 1992); Sutton et al, (1999). To
reduce the variance of policy gradient estimates, it is common to subtract a baseline ((x) that does not depend on the sampled action. The gradient then becomes:
(mo( = (y v) M(x) Vo log log
(19)
16

---

Published as a conference paper at ICLR 2026
A.2.3 GROUP RELATIVE POLICY OPTIMIZATION
Group Relative Policy Optimization (GRPO) treats model training as a reinforcement learning problem Shao et al. (2024). Given a policy me. GRPO samples a group of G responses {VI,..., yc} ~ Te (-Ix> from the old policy Bol for each input 2, and computes the advantage of each response relative to the group:
A; r(z, vi) - σ r r(2,8s)
(20)
std std
where r(x, Us) is the reward of ye, and the token-level advantage A., = A. The importance ratio is defined as (0) (Ms cm. The objective without KL divergence is:
(Mt. (a.)..e))
G
1- (o)+))
JGRPO (0) =E
min
(21)
14
A.3 TRAINING ALGORITIM
A.4
THEORETICAL PERSPECTIVE ANALYSIS
Our framework establishes principled connection between contrastive learning and reinforcement learning, leveraging their shared ability to learn from feedback rather than absolute ground truth labels. This fundamental similarity enables natural integration of their objectives.
A.4.1 UNIFIED LEARNING WITHOUT GROUND TRUTH
Both contrastive learning and reinforcement learning operate on comparative signals: contrastive learning distinguishes positive from negative samples, while reinforcement learning optimizes based on rewards signals. This parallel allows us to reformulate the contrastive objective as reward signal:
(sp(sim(q, ))
LaL -lo EvED exp(sim(q,d'))
R sim(q, (t
Σ sim d")
(22)
d-
This transformation enables policy gradient optimization without requiring explicit labels, only relative preferences encoded in the contrastive structure.
A.4.2 CONNECTION TO INFONCE
Our framework be understood implicitly optimizing a generative variant of the InfoNCE objective. Consider the expected reward under our multi-faceted design:
E [ (1
E
R
(i.R)
+ R1 (i.k) cousist X Rard
(6)
(23)
Expanding the contrastive learning component:
ply (A)
d 9h T8)
R (2.k) log
(24)
p(U, 1ii, T8)
With hard negative mining across the batch:
R10 = log
Σ
max p(u lai TT)
(25)
jyt
Combining these terms, the expected total reward approximates:
p(d |g, me)
E[[mmai] x log
A A(oossiiitne)
(26)
p(d 1g, T8) 11iye maxt p(u nis TO) T0)
17

---

Published as a conference paper at ICLR 2026
Algorithm 1 GRACE: Generative Representation Learning via Contrastive Policy Optimization
Require: Training data D = {(g;, di di ) Initial policy
Require: Hyperparameters: rollouts K, batch size B, coefficients A1, A2
Ensure: Fine-tuned policy mo with enhanced representation capabilities
1: for epoch to Nepuehs do
2: for batch B = {(g; di. d= ~ D do
3: // Phase 1: Generate interpretable rationale via policy
4: for to B in parallel do
5:
Generate query rationale: Vq: =r~ To T-(-1((9:))
6:
Generate negative rationale: Vaz To( TT(-1P(d7))
(1) 1K Sample positive rationale: { Mk=1 ~ To ( (-(()()))) end for
7:
8:
9:
// Phase 2: Extract semantic representations via masked pooling
10:
Encode all rationale: = (Pr() + yy for all (x, y) pairs
11:
Apply masked mean pooling (Eq. 3)
h hd [h
(k) lk=1 < MaskedPool E)
13:
// Phase 3: Compute multi-dimensional contrastive rewards
14:
for i to B do
Identify hard negatives: H. = ii + i: maxy sim (b, 47 }
for h to K do (1,k) = sim h
17:
R CL
(k) dt
Em M, sim hgg harmm
18:
R (i.k)
consist
K-1
1. Σ j
sim (h
4.
.hh
19:
R (i)
hard
B-T
1 Σ j= max ISISK sim sim(har)
20:
R R(i,k)
R
+ A1 (t.t)
(i)
X (i.b)
rotal
consist
21:
R gotal
R R(ikk) /T
22:
end for
end for
24:
// Phuse 4: Optimize policy via GRPO
Compute group baseline: b, k >K K
R(
(i.k)
25:
26:
Compute advantages: (ti,k) R (i,k)
b;
27:
Update policy: 8 + 8 a CoROO where
28:
LGRPO Σi, A log To (1) (P(d)))
(1))
29:
end for
30:
end for
31:
return Optimized policy n
A.4.3 CONVERGENCE ANALYSIS
The optimization landscape of our framework benefits from the variance reduction properties of both contrastive learning and policy gradient optimization. The consistency reward acts regularizer, bounding the policy updates:
[V)(( S C. E [[.al + An Reomsiall ||
Xa + RaaaRaa
(27)
where C is constant dependent on the policy parameterization. The consistency term À Reonsist (i) < e for small e, preventing divergent interpretations and ensuring stable ensures that Ilv Dd! convergence.
This theoretical foundation reveals that our approach maintains the discriminative power of contrastive learning while unleashing the generative capabilities of LLMs, enabling the policy to actively interpret and understand text rather than merely discriminate between samples.
18

---

Published as a conference paper at ICLR 2026
A.5
FURTHER ANALYSIS
A.5.1 SUBSET OF MTEB BENCHMARK
To enable eompute-efficient ablations while preserving coverage across the MTEB taxonomy, we evaluate on a compact, representative subset of 16 datasets spanning seven task families [TalleA.551] Retrieval (3), Reranking (2), Clustering (3), Pair Classification (1), Classification (3), STS (3), and Summarization (1). We run all ablation studies on this 16-task subset in order to systematically isolate component contributions, probe hyperparameter sensitivity, and compare training strategies under a fixed compute budget, while maintaining fidelity to the full benchmark. The selection balances domain diversity (biomedical, news, open-domain, intent), supervision formats (point- wise, pair- pair-wise, list-wise), and difficulty, yielding stable signals for both representation quality and downstream utility. For each task we report the official metric defined by MTEB (e.g., nDCG @ 10 for retrieval, MAP for reranking, V-measure for clustering, accuracy/AP for classification and pair classification, and Spearman correlation for STS; summarization uses the SummEval correlation provided by the harness).
Task
Dataset
SciFact
Retrieval (3)
ArguAna
NFCorpus
aaecOOeerfooxDuuyuustttto
Reranking (2)
SciDocsRR
BiorxivChusteringS22
Clustering (3)
MedrxivChhsserrnng222
Twenty/NewsgroopssCllseerrgg
Pair Classification (1)
SprintDapliicateeuusstioss
Banking77Clasification
Classification (3)
EmotionClassificatonn
MassivelntemCCCassiiittton
STS17
STS (3)
SICK-R
STSBenchmark
SummEval (1)
SummEval
Overall
16 datasets
Table 5: Subset of MTEB tasks used for our ablations and analysis.
A.5.2 EFFICIENCY ANALYSIS
We decompose end-to-end latency into encoding (Tencode). generation (Tgen). and matching (Tiaach). Figure ((a) shows that for generative pipelines (Base and GRACE), Tgen overwhelmingly dominates the budget, whereas Temsode and T Tmuich are comparatively small. As result, encoder-style approaches (BERT and Direct forward method) achieve much lower latency since they avoid generation. In- creasing the decoding budget from G-256 to G-512 further amplifies Tgen with diminishing returns, indicating a practical knee around G=225..
Figure summarizes the quality-latency trade-off. BERT and Direct forward method occupy the ultra-low- region but underperform accuracy, while generative methods deliver higher quality at greater cost. At matched G. GRACE consistently shifts the Pareto frontier upward relative to Base, yielding better accuracy without extra latency and making GRACE G=256 strong default for balanced deployments.
Pragmatically, because Tgen dominates end-to-end latency, the most effective lever is to accelerate generation. Deploying on newer accelerators and using optimized inference stacks with fused attention kernels, paged/continuous batching, and CUDA Graphs materially reduce gen at fixed
19

---

Published as a conference paper at ICLR 2026
GRACE (G-512)
X GRACE (G-512
GRACE (G-256)
GRACE (G-256)
Porformanca
812)
Base (G-
45
Direct forward
Base (G-512)
Direct forward
Base (U-256)
'Tngen
T encode
T_match
500
1000
1500
2000
2500
3000
0
500
1000
1500
2000
2500
3000
E2E Intency (ms)
(a) Latency breakdown across different approaches.
(b) Performance- trade-off.
Figure 4: Efficiency comparison of different embedding approaches.
G. Additional gains come from KV-cache optimizations, speculative/assiited decoding, and low- precision execution (FPS/INTS/INT4) that preserve accuracy under our evaluation budgets. short, improving generation throughput shifts the entire quality-latena curve downward without changing the training procedure, making GRACE at G=256 even more attractive for balanced deployments.
A.6 JOINT ANALYSIS OF GENERATIVE AND EMBEDDING PERFORMANCE
Figure illustrates the positioning of proposed GRACE models in the joint landscape of gener- ative and embedding performance. Existing encoder-only methods such as E5 Wang et al. 2024).
et al. (2024). and (ext-eemeediingg3 aatebbeeddddnnaaaaaaieeeeve achieve strong embedding qua ity but
exhibit minimal generative competence, while instruction-tuned decoders (e.g.. GPT OpenA/ (2023).
Claude Anthropiic (2024). Gemini Gemini Team (2025). Grok xAI (2025), Deepseek DeepSeek-Al
et al. (2025)) demonstrate strong generative capabilities but poor embedding performance. This highlights the standinnn trade-off between the two dimensions.
By contrast, the GRACE family consistently shifts base models upward in embedding quality while largely preserving their generative competence. For example, GRACE-I.SB and GRACE-3B improve the embedding strength of Qwen2.5-1.55 and 1LaMA-3.2233 by more than 15 % on average without sacrificing generative ability. Similarly, GRACE-4B substantially enhances Qwen-3-4B, pushing it into a previously unattained regime of balanced performance. These results underscore the effectiveness of our generative-contrasiis optimization framework bridging the gap between high-quality embeddings and generative reasoning.
A.6.1 TRAINING PROGRESSION ANALYSIS
As shown in Figure 5 both task performance and response length increase steadily with more training steps. The accuracy on subtasks follows a consistent upward trajectory, reflecting that extended training directly strengthens the models' ability for representation. At the same time, the generated responses become progressively longer. Importantly, this lengthening indicates that the models are producing answers with richer information density and more explicit reasoning chains. In earlier stages, responses tend to be short and often incomplete, whereas later stages exhibit structured explanations that combine factual correctness with reasoning depth.
A.6.2 EFFECTS OF VARIOUS REPRESENTATION APPROACHES
We investigate the impact of different token representation methods on model performance across both supervised and unsupervised settings. Figure presents comparative analysis four representation approaches: EOS token, Max Pooling, Mean Pooling from the last layer (LL), and Mean Pooling from the penultimate layer (PL). The results demonstrate that mean pooling approaches consistently outperform both EOS token and max pooling methods across all model variants in both settings. Mean
20

---

Published as a conference paper at ICLR 2026
GRACE-1.88
55
GRACE-JB (Qwen) (LLaMA)
1000
GRACE-3B (Qwwen)
GRACE-3B (LLaMA]
GRACE-4B
GRACE-48
Response Length
Performence
45
400
35
30
800
200
600
800
1000
Training
Training
(a) Accuracy progression across training steps.
(b) Response length progression across training steps.
Training progression ARA models. Left: accuracy on subtasks steadily improves with more training steps. Right: response length also increases, reflecting enhanced information density and richer reasoning chains.
Figure 5:
pooling from the last layer and penultimate layer achieve remarkably similar performance levels and slightly higher in unsupervised models, while EOS token and max pooling show substantially lower performance. The minimal performance difference between LL and PL mean pooling suggests that both layers capture equally effective representations, which aligns with Skean et al. (2023), indicating that comprehensive token aggregation through mean pooling crucial for optimal performance.
EOS Mix Pooling E Moan Pooling (L..) Ei Mom Pooling (PL.
DEOS Max Pooling Mean Pooling (LL) C Mean Pooling (PL)
60
50
30
30
20
20
10
GRACE-L.5B
GRACE-JB
GRACE-SB (LLaMA)
GRACE-IB
GRACE-L.BB
GRACE:3B
GRACE-3B
GRACE44
(Qwen)
(Qwen)
(a) Performance of various representation approaches (b) Performance of various representation approaches unsupervised training.
supervised fine-tuning.
Figure Comparison different token representation methods across GRACE model variants. Mean pooling from both last layer (LL) and penultimate layer (PL) consistently outperform EOS token and max pooling approaches in both supervised and unsupervised settings.
A.6.3 ADDITIONAL TRAINING DETAILS
We incorporate length regularization to prevent degenerate solutions where the model generates excessively long responses without meaningful content:
R
(G.A)
if 1y (A)) dt Linax or y di
(8) [-1] EOS
R
(i.k) Final
(28)
otherwise
where Lmax is the maximum allowed sequence length and y is a penalty coefficient for over-length generations.
A.6.4 ADDITIONAL. IMPLEMENTATION DETAILS
Training uses the Adam W Loshchilov & Hutter 2019 optimizer with a fixed learning rate of 1 x 10-6 and no warmup schedule. The maximum prompt length is set to 1024 tokens and the maximum
21

---

Published as a conference paper at ICLR 2026
GRACE-1.58
GRACE-3B (LLaMA) GRACE-4B
GRACE-L.58
GRACE-3B (LLaMA)
GRACE-3B (Qwen)
(Qwen)
58
55
56
Performance
Performance
50
54
52
45
50
40
48
Batch size (bs)
(rollouts per prompt)
(a) Sensitivity w.r.t. batch size (bs).
(b) Sensitivity w.r.t rollout number (7).
Figure 7: Hyperparameter sensitivity for GRACE models. Both curves show monotonic performance improvements. Left: performance increases monotonically with larger bs. Right: higher yields steady gains with reduced variance at larger values. The best setting is bs 64 and 8, with further improvements possible given more GPU resources.
response length to 2048 tokens, with over-length sequences truncated on the right side. A length penalty 7 1.0 applied when responses reach the maximum length without emitting an EOS token. For GRPO, we generate K = 8 rollouts per positive document using temperature sampling with T= 1.0, and compute advantages relative
mean baseline without normalization. The consistency weight A = 0.2, and the hard negative weight A = 0.2. The scaling for our training is set to 10. Multi-GPU training is managed with Fully Sharded Data Parallel (FSDP) Zhao et (2023), with parameter and optimizer state sharding to alleviate memory constraints. Training- time generation leverages vLLM with tensor-paralle size 2 and 50% GPÚ memory utilization eager mode for stability. Instruction tokens are automatically detected and excluded from pooling operations to prevent contamination in representation extraction. At inference time, we adopt the same generation and pooling configurations as training ensure consistency.
A.6.5 HYPERPARAMETER SENSITIVITY ANALYSIS
We analyze the sensitivity of model performance to two training-time hyperparameters: the batch size bs and the number of rollouts per prompt n. Overall, all models exhibit monotonic improvement with increasing batch size. Figure 7 (right) varies n E {2, 4, 8) and shows consistent, monotonic gains across models, with diminishing variance at higher In practice, we find bs 2 32 and n 2 4 to be good Pareto choice under typical compute budgets, while larger settings such as = 64 deliver the best observed performance. Further improvements are likely possible given more GPU resources.
=
A.6.6 CASE STUDY
Output Evolution During Training Tilusteis how outputs evolve with training. The base model provides only brief topical summary, while step 200 introduces structured listing with concrete figures. By step 600, the output grows substantially longer and begins to integrate historical context, crises, and institutional concepts. step 1000, response is the most comprehensive: it connects anecdotes with broader political and constitutional developments, offering coherent narrative. This progression shows that training not only increases output length but also enhances contextualization and information density, reflecting richer internal representations.
In-depth Analysis of Model Response Patterns We highlight two representative cases that demonstrate the model's response patterns across different input types, shown Table 7 For
https::/gitthbb.... http://..hhbbbccmmmmmmmmmmmmoaro
22

---

Published as a conference paper at ICLR 2026
Original Doc: BBC- History- British History in dopth: Prime Ministers and Politics Timeline On This Day Prime Ministers and Politics Timeline Do you know which prime minister brought 'fallen women' to 10 Downing Street! Or which one lought a duel? Or who was known 'the Goat'? Take political joumey through nearly 300 years high ideals and low cunning. from Gordon Brown the first man hold prime minister...
Base Model (480 tokens)
Training Step 200 (562 to- kens)
Training Step 600 (824 to- keus)
Training Step 1000 (1051 to- keus)
"The text is comprised of an interactive timeline on his- tory of British Prime Minis- ters, concise hiography sev- eral past Prime Ministers, and a brouder overview of the of- fice."
"Provides an overview of sev- eral key British prime minis- ters and the history of the of-
"Informative passage about key British Prime Ministers, such as Margaret Thatcher, James Callaghan, and Harold Wil- son."
"Provides comprehensive un- derstanding of the office the Prime Minister and its histori- context."
"Starts with Margaret Thatcher. Britain's ' first female PM, and also mentions Robert Walpole and Henry Camphell- Bannerman."
"Roots traced back Robert Walpole, later evolution shaped constitutional changes."
"Traces the evolution of the mile from 18th century cubinet system, with Walpolc as carly figure""
"It begins with timed ance- dotes highiighting key ligures and events from 1681 to the present day."
"Mentions modern fig- ures like Thatcher and Tony Blair, situating them in UK's political-ccommiic
"Walpole is seen as the first Bammeell-Bannemmnn the first formally called so."
"Highlights crises like Harold Wilson's industrial dis- pules and economic instabil- ity."
"Discusses powers and stmints of PM, from cabinet appoiatments parliamentary limits."
"Explains parliamentary countability and collective re- sponsibilityy"
"References amecdotes (e. (e.g. 'fallen women') as outduted, while contextualizing gender role shifts."
Table 6: Case study for output evolution. As training progresses, outputs grow from brief topic recognition to structured lists, then detailed coverage, and finally full contextualization.
Doc Case (Belt and Road Initiative)
Query Case (Machu Picchu)
Original Document:
Original Query:
"policy coordination an important guarantee for implementing the Initiative... We should carry forward the spirit of Iriendly coopera-
which country would you find the ruins of the ancient Inca city known Machu Picchu?"
tion of the Silk Road by promoting cultural and academic...
Model Response:
Model Response:
"The Belt and Road Initiative is an extensive cooperation project pro- posed by China... Policy Coordination involves inter- govermeenaal cosperation, expansion of shared interesis, and enhancing mutual trust... Prople-to-Proge Poooplle Ronds promote cultural exchange. volun- teer services, and modia cooperation... The initiative faces chal- lenges such debt sustainsblity gaopolitical risks...
"Machu Picehu is located in Peru... situated in the Andes Mountains, Cusco Region, about 2,430m elevation... rediscovered in 1911 by Hiram Bingham... UNESCO World Herilaaee Site and major tourist destination..."
Table 7: Representative cases of model responses: left column shows document understanding (doc case), right column shows query expansion (query case).
a long document on the "Belt and Road Initiative", the model goes beyond paraphrase to produce a compact outline (policy coordination, infrastructure and finance, people-to-people links, geopolitical constraints), turning an abstract preface into structured analysis. For a minimal factual query (c.g.. "Where is Machu Picchu?"). it answers precisely while adding concise, high-signal context (Peru; Andean setting; Inca and UNESCO notes; basic access considerations), avoiding unnecessary narrative. This adaptive behavior yields information-dense, well-factored texts whose embeddings align with latent topics and relations, improving separability for retrieval, clustering, and pair classification.
23


Reviews:
[{"review_id": "5Xt56dBc0Q", "reviewer": "Reviewer_yvNK", "summary": "This paper introduces GRACE (Generative Representation Learning via Contrastive Policy Optimization) which uses contrastive learning objectives as a reward function to guide a generative policy (i.e. an LLM). The LLM policy generates human-interpretable rationales which are structured natural language explanations of its understanding. These rationales are mean-pooled to obtain embeddings. With this new reward function, existing LLMs can be finetuned with reinforcement learning to improve their embedding performance while retaining most of their generative performance.", "questions": "Please address my comments/questions from the weaknesses section", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is fairly easy to read and follow\n\n* GRACE can improve embedding performance without affecting the generative performance. And using the contrastive loss as a reward for finetuning LLMs with RL seems to be novel.", "weaknesses": "* It is unclear what the rationale output is exactly. It would be good to add some qualitative examples for it. Because the paper claims (in the abstract for instance) that the LLM produces human-interpretable rationales and that the proposed method leads to transparent decision traces. But these claims are not explicitly evaluated. Ideally, there should be a human study to validate improvements in these aspects over the baseline LLM.\n\n* Fig. 3: It seems that performance is quite sensitive to the choice of the hard negative mining weight $\\lambda_2$. Even between the supervised and unsupervised paradigm, a different $\\lambda_2$ is needed for the best performance. Given that training is expensive, it would be good to have some heuristics to select good hyperparameters. Also it would be good to report if the $\\lambda_1, \\lambda_2$ hyperparameters also had to be tuned for the other base LLM experiments with GRACE.\n\n* It is also unclear if there could be any potential reward hacking [W1] with the proposed contrastive reward (e.g. if there are certain text patterns that yield high rewards). There needs to be some discussion on this (the earlier suggested human study could reveal if this problem exists).\n\n### References\n\n[W1] Pang et al., \"Reward Gaming in Conditional Text Generation\", ACL 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vSCpbfjnyE", "reviewer": "Reviewer_Bk7x", "summary": "The paper presents GRACE (Generative Representation Learning via Contrastive Policy Optimization), a framework that reframes contrastive objectives as rewards for large language models. Instead of minimizing contrastive losses, GRACE applies policy-gradient optimization (e.g., GRPO) to maximize semantic alignment between queries and positives while discouraging hard negatives. The LLM generates textual rationales, from which embeddings are derived via mean pooling. Experiments on MTEB benchmarks across several instruction-tuned backbones show 6–11% average improvements in embedding quality, with minimal loss in general generative ability.", "questions": "1.Disentangling RL vs. generative representation: could you implement or report a baseline that uses rationale-conditioned embeddings trained with a standard InfoNCE loss (i.e., CL + rationale) to quantify how much of the improvement is due purely to the RL optimization step?\n\n2.Interpretability validation: the paper already include a few illustrative rationale examples in the appendix, but they remain anecdotal. To substantiate the strong interpretability claim, it would be very helpful to (i) provide more systematic qualitative comparisons across Base / CL / GRACE, and (ii) add a small-scale human or automatic evaluation of rationale quality (e.g., relevance/usefulness to the similarity decision, or faithfulness to the underlying embedding behavior).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Method: Elegant reformulation of contrastive learning into a reinforcement learning (RL) framework using explicit rewards rather than losses.\n\n2. Interpretability: The introduction of “rationales” offers a clear path toward explainable embeddings.\n\n3. Broad empirical validation: Results are reported on both supervised and unsupervised settings with multiple backbones, and the ablations (λ₁, λ₂ grid, alternative RL algorithms) are systematically conducted.\n\n4. Practical impact: Demonstrates that reward-based generative alignment can mitigate the trade-off between embedding quality and generative capability.", "weaknesses": "1.Theoretical contribution is incremental.The paper frames GRACE as a “generative reformulation” of contrastive learning but, in practice, the proposed reward is a simple contrastive score built from similarity differences. The paper provides only a high-level convergence discussion that bounds the gradient norm by the magnitude of its reward terms, without offering non-trivial guarantees or formal comparison to the optimization behavior of standard contrastive learning. Consequently, the claimed “unification of contrastive and RL paradigms” feels more like a notational reinterpretation than a fundamentally new theory.\n\n2.Ablation design does not isolate the effect of RL. The supervised comparison includes four settings (Base, Base w/ reasoning, CL, GRACE), which partially decouple the contribution of reasoning prompts and RL. However, a crucial baseline is missing — Contrastive Learning + Rationale Embeddings trained with the same rationale generation path but optimized via InfoNCE. Without this, it is difficult to cleanly disentangle whether GRACE’s improvements arise from the reinforcement update or simply from the generative representation pathway.\n\n3.The general-capability evaluation in Table 4 maybe unreliable. CL fine-tuning collapses performance to 0.0 on five of six tasks across all backbones (except FEVER), an implausible pattern for standard contrastive tuning. The paper provides no diagnostics or ablations to explain this, yet uses it to claim that “CL severely damages general-domain ability.” Without verification, variance reporting, or analysis of training setup, this conclusion may reflect issues in the CL setup rather than an inherent flaw of contrastive objectives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0heT6L5tMk", "reviewer": "Reviewer_vZAz", "summary": "This paper introduces GRACE, a novel framework that reimagines how large language models (LLMs) can be trained as text encoders.  GRACE treats contrastive signals as rewards that guide a generative policy. The LLM acts as a policy that generates explicit, human-readable rationales for its semantic understanding. These rationales are then pooled into embeddings. Through policy gradient optimization, the model is trained to maximize similarity between query-positive pairs and minimize similarity with negatives. The method is evaluated on the MTEB benchmark and shows significant improvements.", "questions": "- Could you discuss specific application scenarios or use cases where it is particularly advantageous to have a single model that performs both tasks, as opposed to using separate, specialized models for generation and embedding?\n- In Table 3, , what is the specific baseline model to which the \"Grace-3B w/ [RL Algorithm]\" results are compared? Additionally, there appears to be a typo in the text reference on line 402, which reads \"Table 3.4.2\".\n- How is $\\mathcal{R}_{final}^{(i,k)}$ in Equation 10 precisely defined?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Unifies generative reasoning and embedding learning, boosting representation quality while preserving general capabilities.\n- Extensive MTEB evaluation shows consistent gains in both supervised  and unsupervised settings across multiple reinforcement learning algorithms.\n- The appendix provides additional theoretical and empirical analysis. This enhances the paper's reproducibility", "weaknesses": "- The paper would benefit from an evaluation of whether the generated rationales faithfully explain the embedding process. Currently, the examples seem to paraphrase or extend the original content, rather than justifying the model's semantic decisions.\n- High inference latency due to autoregressive generation limits practical deployment. There are existing, highly efficient encoder-only models that provide strong embedding performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GHygIJu3aQ", "reviewer": "Reviewer_uisq", "summary": "The paper introduces a new method that improves embeddings based on generative models. The method is more interpretable than the traditional contrastive learning set up. Authors demonstrate performance gains compared to base models and in general domain tasks.", "questions": "-", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- New method utilizing generative model capabilities to create an interpretable reward signal. The method shows clear improvements over the baselines. \n\n- Authors conduct detailed ablation studies.\n\n- Performance on general domain tasks is preserved.", "weaknesses": "- Lack of statistical significance reporting. To ensure the improvement was not a result of noise, it would be appropriate to report confidence intervals or p-values.\n\n- Authors claim their method is more interpretable but interpretability is not evaluated. Examples of the rationales are also not provided, at least in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3UMEdoE3Sy", "reviewer": "Reviewer_eAv2", "summary": "The work proposes a new framework for training Large Language Models (LLMs) as interpretable text encoders by reinterpreting contrastive objectives as rewards rather than losses. Instead of producing static embeddings, GRACE trains LLMs as policy models that generate explicit rationales before deriving embeddings from those rationales. Using policy-gradient optimization based on GRPO, GRACE aligns query-positive similarity and discourages query-negative similarity through a multi-component reward incorporating contrastive alignment, consistency, and hard negative mining. Experiments on the MTEB benchmark show strong cross-task gains over base models while preserving general reasoning, math, and coding capabilities.", "questions": "- How did the authors define and evaluate “human-readable” rationales? Are they consistent with ground-truth semantics or just stylistically coherent?\n- During training, did the authors observe training instability or reward collapse during GRPO updates? How did the authors tune the reward weights for contrastive, consistency, and negative components?\n- Since GRACE encourages rationale generation, models may produce verbose or generic explanations that sound fluent but carry little semantic value. How did the authors regularize or constrain rationale length and content to ensure concise yet meaningful reasoning?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- GRACE delivers consistent and cross-category improvements on the MTEB benchmark, with a significant gain over baselines.\n- The introduction clearly articulates the fundamental contradiction in existing embedding models and motivates GRACE as a conceptually sound fix.\n- GRACE is architecture-agnostic. It can be applied to any instruction-tuned LLM without modifying backbone structure.", "weaknesses": "- By optimizing for rationale quality and semantic contrast simultaneously, GRACE might bias representations toward linguistic fluency or stylistic patterns rather than true semantic alignment.\n- The framework’s reliance on policy optimization with sampled rationales makes it computationally heavier and more memory-intensive than deterministic encoders.\n- All evaluations are centered on MTEB, which is known to correlate strongly with models optimized for textual similarity tasks. There is no evidence that GRACE embeddings generalize beyond MTEB-style benchmarks. Meanwhile, the reward formulation (contrastive alignment + consistency + hard negative mining) implicitly encodes assumptions about semantic similarity that may not generalize across domains (e.g., legal or medical text). There’s also a risk that reward weights are tuned for specific datasets, leading to domain overfitting or poor generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)