PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similarity
Authors: Zixin Song
Bowen Zhang
Qian-Wen Zhang
di yin
Xing Sun
Chunping Li
Venue: ICLR 2026 Poster (2026)
Keywords: Conditional Semantic Textual Similarity
Reinforcement Learning
Large Language Models
Natural Language Processing
Curriculum Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose PoLi-RL, a two-stage RL framework that enables effective listwise optimization for the C-STS task via a progressive curriculum and a Parallel Slice Ranking Reward mechanism , achieving SOTA performance for the cross encoder architecture.

Abstract: Conditional Semantic Textual Similarity (C-STS) measures the semantic proximity between text segments under a specific condition, thereby overcoming the ambiguity inherent in traditional STS. However, existing methods are largely confined to discriminative models, failing to fully leverage recent breakthroughs in the NLP community involving Large Language Models (LLMs) and Reinforcement Learning (RL). RL is a particularly well-suited paradigm for this task, as it can directly optimize the non-differentiable Spearman ranking metric and guide the reasoning process required by C-STS. Nevertheless, we find that naively applying listwise RL fails to produce meaningful improvements, as the model struggles with complex, coarse-grained reward signals, leading to optimization difficulties. To address this challenge, we introduce PoLi-RL, a novel Point-to-List Reinforcement Learning framework. PoLi-RL employs a two-stage curriculum: it first trains the model with a simple pointwise reward to establish fundamental scoring capabilities, then transitions to a hybrid reward that combines pointwise, pairwise, and listwise objectives to refine the model's ability to discern subtle semantic distinctions. Crucially, we propose an innovative Parallel Slice Ranking Reward (PSRR) mechanism that computes ranking rewards in parallel slices, where each slice consists of completions with the same index from different samples. This provides a precise, differentiated learning signal for each individual completion, enabling granular credit assignment and effective optimization. On the official C-STS benchmark, PoLi-RL achieves a Spearman correlation coefficient of 48.18, establishing a new SOTA for the cross-encoder architecture. Furthermore, PoLi-RL also maintains SOTA status on the re-annotated C-STS dataset, confirming its robust generalization capabilities. As the first work to successfully apply RL to C-STS, our study introduces a powerful paradigm for aligning LLMs for complex, ranking-based conditional judgment tasks. Our code and checkpoints are available at https://github.com/ZBWpro/PoLi-RL.

Published as a conference paper at ICLR 2026
POLI-RL: A POINT-TO-LIST REINFORCEMENT LEARNING FRAMEWORK FOR CONDITIONAL SE- MANTIC TEXTUAL SIMILARITY
Zixin Song", Bowen Zhang"; Qian-Wen Zhang", Yin", Xing Sun", Chunping Li TTinghua University, Teeneet Youtu Lab {songzx24, 3 wttemmml..ssiinsuaaaaaeeeaeeeeee edu, en, cl weiitsinnhuu..... edu.cc en (cowenzhang, endymecyyin, wwnfredsunneteneeeeee com
ABSTRACT
Conditional Semantic Textual Similarity (C-STS) measures the semantic prox- imity between text segments under specific condition, thereby overcoming the ambiguity inherent in traditional STS. However, existing methods are largely con- fined to discriminative models, failing to fully leverage recent breakthroughs the NLP community involving Large Language Models (LLMs) and Reinforce- ment Learning (RL). RL is a particularly well-suited paradigm for this task, as it can directly optimize the non-differentiable Spearman ranking metric and guide the reasoning process required by C-STS. Nevertheless, we find that naively ap- plying listwise RL fails to produce meaningful improvements, the model strug- gles with complex, coarse-grained reward signals, leading to optimization diffi culties. To address this challenge, introduce PoLi-RL, novel Point-to-List Reinforcement Learning framework. PoLi-RL employs two-stage curriculum: it first trains the model with simple pointwise reward to establish fundamen- tal scoring capabilities, then transitions to a hybrid reward that combines point- wise, pairwise, and listwise objectives to refine the model's ability to discern subtle semantic distinctions. Crucially, we propose an innovative Parallel Slice Ranking Reward (PSRR) mechanism that computes ranking rewards in parallel slices, where each slice consists of completions with the same index from different samples. This provides a precise, differentiated learning signal for each individ- ual completion, enabling granular credit assignment and effective optimization. On the official C-STS benchmark, PoLi-RL. achieves a Spearman correlation co- efficient of 48.18, establishing Furthermore, PoLi-RL also maintains SOTA status on the re-annotated C-STS dataset, confirming its robust generalization capabilities. As the first work suc- cessfully apply RL to C-STS, our study introduces a powerful paradigm for align- ing LLMs for complex, ranking-based conditional judgment tasks. Our code and checkpoints are available at httpt::/// http::://iithb.... /ggttub.......///gg com/ /BWp////ooooiiii
SOTA for the cross-encoder architecture.
INTRODUCTION
As a core research area in Computational Linguistics, Semantic Textual Similarity (STS) finds ex- tensive applications across diverse scenarios, including topic modeling, dialogue systems, text sum- marization, and agent memory (Tang et al. 2205) Zhang et al., [2024b] 2025a). However, traditional STS tasks exhibit inherent ambiguity because similarity definitions are often susceptible to subjec- tive bias. To address this limitation, the Conditional Semantic Textual Similarity (C-STS) task was developed (Deshpande al. 2023). By incorporating an explicit natural language condition, C-STS enables more precise and objective similarity judgments, yet simultaneously imposes higher de- mands on model's reasoning capabilities. For instance, consider the following two text fragments: "A player is shooting from beyond the three-point line" and "A player is taking free throw". Under
"qqual contribution. Work done during the internship at Tencent Youtu Lab. Corresponning author.

---

Published as a conference paper at ICLR 2026
the condition "The activity of the player", their similarity is high. However, under the condition "The player's s distance from the basket", their similarity is low.
Research on this nascent task has yielded three primary paradigms: Bi-encoder (Liu et al., 2025), Tri-encoder (Lin et al., 2024), and Cross-encodee (Li et al. (024). The Cross-encoder architecture, which processes the text pair and the guiding condition simultaneously, is inherently compatible with modern generative pre-trained models. Despite this, the integration C-STS with LLMs remains in its early stages. Current LLM applications are limited two main approaches: direct inference via few-shot prompting, where even state-of-the-art models struggle to achieve satisfactory results Deshpande et al. 2023); and utilizing them as feature extractors for generating text embeddings Yamada & Zhang (025). which is an extension of the discriminative paradigm. To the best of our knowledge, no prior work has implemented an end-to-end L.L.- cross-encoder for the C-STS task, nor has any integrated it with advanced training techniques like reinforcement learning (RL), leaving significant research gap.
This paper aims to fill this gap. We argue that incorporating RL into an LLM-based cross-encoder paradigm natural fit. This is reflected in two key aspects: First, C-STS requires sophisticated, scenario-based reasoning. For example, in the basketball case described earlier, to correctly assess similarity under the "distance" condition, the model must transcend surface-level semantics to iden- tify the underlying spatial relationship between 'beyond the three-point line" and 'at the free throw line', a process demanding strong abstraction and inference. RL, through its explicit reward signals, can more effectively guide the reasoning process of LLMs (Guo al., 2025). Second, from an opti- mization standpoint, RL aligns closely with the task's ' evaluation criteria. The Spearman correlation coefficient (Zar) 2005), a core evaluation metric of C-STS, is a non-differentiable measure of rank- ing quality. Traditional SFT methods can only indirectly and approximately optimize this objective through loss functions like Mean Squared Error (MSE) (Zhang & Li 2024b). In contrast, RL allows for the direct optimization of ranking- based reward functions designed to correlate strongly with the final Spearman metric, aligning the training objective more closely with the evaluation criteria.
However, naive application of RL to this task presents significant challenges. As illustrated in Figure our preliminary experiments indicate that directly applying a listwise ranking reward (e.g., Spearman's 's correlation coefficient) across entire batch of completions does not show any advan- tages compared to the few-shot method. This approach suffers from two fundamental problems. First, the ranking objective is too complex for model that has not yet grasped the task's funda- mental scoring rules, often leading to training collapse. Second, a single reward computed across the entire batch is too coarse provide precise credit assignment, few poor completions can unfairly penalize other good ones.
50.0
Spearman
48.18
48.27
47.5
Pearson
45.0
44.77
42.5
40.0
(37.90
38.54
[38.19
37.5
35,0
32.5
30,0
Few-shot Inference
Naive RL (Listwise)
PoLi-RL Stage 1
PoI.-RLL Stage 2
Figure 1: Performance comparison of different strategies on the C-STS task. Directly applying listwise ranking reward for RI. does not significantly outperform the few-shot baseline. In contrast, both stages of our method (Pol.i-RL) achieve substantial improvements, validating its effectiveness.
To address these challenges, propose PoLi-RL, two-stage Point-to-List Reinforcement Learning framework. PoLi-RL features a two-stage curriculum to manage the complexity of the learning task. In the first stage, simple pointwise reward to ground the model in the basic
2

---

Published as a conference paper at ICLR 2026
scoring rules of the task. Building on this foundation, the second stage introduces richer, hybrid reward signal that combines a robust pointwise anchor with more nuanced pairwise and listwise ranking rewards, enabling the model discern subtle semantic differences.
Furthermore, to resolve the problem coarse-grainee reward signals that arises from ranking all completions in batch together, we innovatively introduce Parallel Slice Ranking Reward (PSRR) mechanism, which employs a two-level decomposition. First, for batch N input samples, the model generates G completions for each. We then form G "parallel slices", where the i-th slice is composed of the i-th completion from every sample. Second, and more importantly, within each slice, rather than assigning a single reward, we compute the rank difference for each individual completion against its ideal rank. This two-level decomposition allows each of the N G comple- tions to receive unique and precise reward that reflects its quality, thereby enabling granular credit assignment and more effective optimization.
The primary contributions of this paper are outlined as follows:
To the best of our knowledge, this the first work to propose an end-to-end, LLM-bbasd cross-encoder for the C-STS task, and also represents the first application of reinforcement learning in this domain.
We design and implement PoLi-RL, novel two-stage training curriculum that overcomes the optimization difficulties of direct rank-based learning by progressing from a basic point- wise objective a comprehensive hybrid reward.
We propose the Parallel Slice Ranking Reward (PSRR) mechanism, which delivers pre- cise and differentiated learning signals by computing ranking rewards within independent 'parallel slices.' This mechanism offers generalizable strategy for other ranking tasks involving multiple candidate generations.
On the official C-STS benchmark, PoLi-RL achieves Spearman's 's correlation coefficient of 48.18 establishing a new SOTA for the cross-encoder architecture and surpassing strong proprietary models, including GPT-4o (44.23) and Deepseek-R1 (42.85). Furthermore, achieving SOTA on the re-annotated dataset and our qualitative analysis jointly confirm the model's superior capability in nuanced conditional reasoning.
jointly
METHODOLOGY
This section details proposed strategy. We begin by formulating the C-STS task within end- to-end, LLM-based cross-encoder paradigm in subsection 2.T Then, subsection 2.2 we map this task onto the mathematical framework of Reinforcement Learning and specify its optimiza- tion objective. Finally, in subsection [2.3] we provide a comprehensive description our PoLi-RL framework, including its two-stage design and the innovative PSRR mechanism.
2.1 PROBLEM FORMULATION
The core objective of C-STS to learn a scoring function that accurately reflects the semantic similarity between two text segments under a specific condition. Formally, each C-STS data sample defined as pair (Ir y). where the input
(4,, (1,42,c) c) consists of two text segments and a natural language condition, and the label y E [1, 5] is the scale (Likeri 1932). Specifically, the label y corresponds to a fine- Ane-grained set of semantic criteria. According to the C-STS annotation guidelines (Deshpande et al. 2023), the scores represent: (1) Completely dissimilar; (2) Thematically related but dissimilar; (3) Roughly equivalent, but with some important information difffrrncces (4) Mostly equivalent, differing only in unimportant details; (5) Completely equivalent. This level of granularity demands that the model perform fine- grained reasoning beyond surface-level semantics, posing significant challenge to its capabilities.
similarity judgment on the Likert
distinctive feature of the C-STS dataset its paired structure: samples are organized adjacent pairs that share the same text segments (f1, but feature different conditions and maintain strict ordinal relationship between their labels, i.e., Yhigh 2 Mow- This structure provides a solid foundation
for our pairwise reward design, detailed in subsection 2.3
3

---

Published as a conference paper at ICLR 2026
Our task is to train scoring model Te. parameterized by 8. For each sample, the model takes unified prompt p = [ z (detailed in Appendix (A.2) as input, consisting of the instruction I. K few-shot demonstrations & {zzg, yv ) k== and the query input to generate output sequence = No (p). From this sequence, we parse the final predicted score, ñ = Parse(o). The overall training objective is maximize the ranking consistency between the set of predicted scores (ūs ) E and the ground-truth scores (0 by optimizing the policy ne. process primarily quantified by Spearman's 's correlation coefficient. Since this metric is rank-based and non-differentiable, RL. emerges more promising optimization paradigm than traditional supervised fine-tuning.
2.2 REINFORCEMENT LEARNING FOR C-STS
We formulate the C-STS task as a Markov Decision Process (MDP), defined by tuple M = (S, A, T, R),, where the agent is the LLM policy mg. The generation process is formulated as a sequential decision-making process, where each step involves generating single token. A state 8 E S at timestep t the sequence of tokens generated so far, conditioned on the initial prompt, i.e., (p, (pt)) An action 0r E A is the selection of the next token from the model's vocabulary, governed by the policy To [o(ar[s)). which provides probability distribution over all possible tokens. The transition function T is deterministic, where the next state 81+1 is formed by appending the selected token ar to St- We employ terminal reward setting, where a reward RT = R1x y, o) is given only after the entire sequence has been generated. Finally, the discount factor y is set to to ensure that the terminal reward is backpropagated without decay to all actions that contributed to the final output. Based on this framework, the objective is to find the optimal parameters 6 that maximize the expected reward over the data distribution D:
g" arg
(1)
To optimize this objective, we employ Decoupled Clip and Dynamic Sampling Policy Optimization
(DAPO) (Yu et al. 2025). an extension of GRPO (Shao et al. 2024) that introduces several key techniques for effective RL. For each sample z, the policy model generates set of G' completions {o,) A scalar reward r; R(x, y, o;) is computed for each completion. The advantage Å, for each completion is then calculated by normalizing its reward against the statistics of the group rewards via Z-score normalization:
A = TV - -mean( (r {t)((
Since we adopt an on-policy training approach, completions are sampled directly from the current policy me being optimized. The objective function for updating the model parameters 0 is:
G lo.l
To(oi, or. [To(Or, oi, [nogrdd
(!) = E (2.g)) (8 (m)()))))
A,
(3)
Σ
lo.
where Hnograd stops gradient flow through the denominator, ensuring only the numerator is updated.
2.3 POLI-RL: A TWO-STAGE REINFORCEMENT LEARNING FRAMEWORK
To optimize effectively, we propose PoLi-RL, a framework built on a two-stage progressive reward curriculum. This subsection elaborates our pipeline and its reward mechanisms (Figure 2
Stage I: Foundational Skill Acquisition. The goal of Stage I ground the model in the fun- damental scoring rules of the C-STS task. For each input sample, the LLM policy generates G completions, from which we parse is a weighted sum of three components:
predicted scores {ū,}. The total reward for Stage I, R91,
Rs1 = X Rpointiss + Rpinary X Rioomat
(4)
The Pointwise Accuracy Reward (Rnointwise) is the primary component in Stage I. measures the normalized distance between the predicted score } and the ground-trute score yy:
|0; - ツ
R Rpointwise max(Y) min(Y)
(5)
4

---

Published as a conference paper at ICLR 2026
Prompt
Stage
Stage II
Sentence I:A man cooks in the kitchen.
Format Reward
Poini-nii -ise Accuracy Reward
riding bike Condition: The number of people.
Pair-wise Ranking Reward
Answer F Format: Binary similarity judgement with fine-grained label, eg yes(4), no(1)
List-wise Ronking Reward
advantago computation axis
Binary Judgement Reward
Com Sonupnn rank differenic Sammin smath lahalk
A₁
A
Ac
Answer Evaluztion
Policy Model
Bi
Bg
B
Point-wise
BOLZO
Accuracy Reward
C2
Co
prediction - Tabel
DAPO
mosr(baall) minllabee
Figure 2: An overview of the PoLi-RL framework. It employs a two-stage curriculum, progressing from Stage I, where the model learns foundational scoring rules, Stage II, which refines the model's 's ability to discern fine-gra tne-grained semantic differences. The core of our method is the PSRR mechanism in Stage II, where pairwise and listwise ranking rewards are computed vertically within slices of the same-indexed completions to provide precise, differentiated learning signals.
where max(YY and min(Y) = are the bounds the label space.
To mitigate the model's tendency to converge to safe, intermediate scores, we introduce a Binary Judgment Reward (Riimury) According the C-STS guideline that scores 2 indicate similarity, while scores <. 2 indicate dissimilarity, this reward encourages the model to first master this basic binary classification:
{)
if (ū, 2334 3) (ū) < <) otherwise
Ruinary
(6)
Finally, simple Format Reward (Rrormat) ensures the output adheres to the required structure, which consists of a binary judgment ("yes" "no"") followed by the numerical score in parentheses.
Stage II: Fine- Grained Semantic Distinction. Building on the foundational scoring abilities from Stage Stage refines the model's capacity for nuanced semantic distinctions. This is achieved through hybrid reward signal that incorporates both pairwise and listwise ranking objec- tives, both of which are enabled by our proposed Parallel Slice Ranking Reward (PSRR) mechanism.
PARALLEL SLICE RANKING REWARD MECHANISM. A primary challenge in directly optimizing ranking metries is that a single, batch-level reward signal too coarse to assign credit precisely. To address this, propose the Parallel Slice Ranking Reward (PSRR) mechanism. The core idea generated outputs to enable more granular reward computation. For batch of N samples, begin by having the policy generate G completions {o,,1..... OG,G } for each sample. From each completion ot we then parse the predicted score Ut.j- Instead treating the completions slices". Each slice, denoted Y yired is defined as the collection of the j-th predicted score from all N samples in the batch: y pred ( .2.j..... HN.J.. J. where j {l,..., G). This slicing architecture forms the foundation of our ranking rewards, ensuring each completion receives specific learning signal based on relative performance within its independent slice.
of PSRR is to restructure
single flat list, we organize these N x G predicted scores into G "parallel
A sufficiently large batch size N is crucial for stable and meaningful ranking signal. To make this computationally feasible with limited GPU memory, we employ gradient accumulation strat- egy. Specifically, we first generate the full set of N x G' completions and organize them into parallel slices to compute rewards and advantages globally. Subsequently, we process smaller sub-batches sequentially to compute losses and accumulate gradients ecuting single optimizer step. This strategy enables the model to learn from a rich ranking signal without incurring prohibitive memory costs.
multiple backward passes before ex-
5

---

Published as a conference paper at ICLR 2026
PAIRWISE RANKING REWARD. Computed within each parallel slice Y prod this reward leverages the paired structure of the C-STS dataset to provide local ranking signal. It is applied only to adjacent input samples that form a valid pair. We define the predicted difference as Apred = yo. 9i+1, and the true difference as Arme yi 3++1- To accommodate cases where paired samples share identical ground-truth scores (Airue = 0), we distinguish them from strictly ordered pairs. The reward R is formulated as:
1Apool
if Auue
Dema
p Ri
Pease + Ribase): (1
Ppea Amee
if sgn( Aprdd) = sgn (Ame)
Dmax
otherwise
where D max represents the maximum possible score difference (i.e., 4 for the 1-5 scale). For strictly ordered pairs (Ame * 0), the model receives a base reward Rhase for correct ordinality, with an additional penalty for prediction error magnitude. For tied pairs (Ame 0), the reward simply minimizes the absolute prediction difference.
LISTWISE RANKING REWARD. While the pairwise reward focuses on local comparisons, the list- wise reward provides more global ranking perspective within each slice. It is calculated as the normalized difference between a completion's predicted rank and its ground-truth rank:
Rankk spica) Rank(ys, Yuue))
plstwise
N
where Yume = fy...... yN) is the set of ground-truv labels for the current batch, the function Rank(v, returns the rank of element within the S (from I to N), and the division by N normalizes the rank error to the range [0. 1].
The final reward for Stage II, Rs2, combines the robust Pointwise Reward from Stage as an anchor with the ranking-bass based rewards enabled by PSRR. The total reward is a weighted combination of these three components:
Rs2 = l1 Rpointwies + Uug Rpirwise + Mg Rlissiis
(9)
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
We build Pol.--R upon the Qwen3 Yang al. 2025) model family (0.6B, 4B, and 8B) using the ms-swift framework (Zhao al. 2024) for RL training. Detailed hyperparameter configurations are provided in Appendix A All experiments are conducted on the official C-STS dataset (Deshpande ct al. 2023). Following prior work, we use Spearman correlation as the primary metric and Pearson as the secondary. We compare our method against three baseline categories: First, discriminative models in the cross-encoder setting, such as SimCSE (Gao et al., 2022). Second, state-of-the-ant art proprietary and open-source reasoning models, including Deepseek-Ri (Guo al. [2025). GPT-4 Achiam et al. 2023), GPT-4o, Flan-T5 (Chung et al., 2024). Tk-Instruct (Wang et al. (022). etc. Finally, our SFT and few-shot implementations Qwen3 backbones for direct comparison.
3.2 MAIN RESULTS
Table summarizes the performance of PoLi-RI.. Our framework establishes new state-of-the-arr (SOTA) for the cross-encoder architecture, with the 8B model achieving Spearman correlation of 48.18. The significance of this achievement underscored by several key observations:
Superiority over Strong Baselines. PoLi-RL significantly outperforms the previous cross-encoder SOTA, SEAVER [Li et al., 2024), by a margin of 4.35 points. Compared to standard SFT, our method yields consistent gains across all model sizes. Notably, Qwen3-0.6B
yields substantial absolute improvements of 19.09 points over few-shot inference and 8.75 points over standard SFT, showcasing the substantial benefits of our progressive, multi-component reward optimization.

---

Published as a conference paper at ICLR 2026
Efficacy Across Model Scales. A striking finding is the efficacy of PoLi-RL on smaller architec- tures. As shown in the results for Qwen3-0.6B and 4B, our framework enables compact models 0.6B model (44.34) not only outper-
to perform significantly beyond their scale. Remarkably, forms the proprietary giant GPT-4 (43.6) but also surpasses the previous SOTA SEAVER (43.83). This demonstrates that the performance gains stem from aligning the reasoning process with ranking objectives, rather than relying solely on parameter scaling.
Ad Advantage over Proprietary Models. Benchmarking against state-of-the-art reasoning models, our 8B model outperforms GPT-4o by 3.95 points and DeepSeek-RI by 5.33 points. This suggests that while proprietary models possess strong general reasoning capabilities. they struggle strictly align with the fine-gratn iine-grained quantization standards of C-STS in few-shot setting, PoLi-RL bridges this gap by explicitly optimizing this alignment via RI., proving that a specialized, smaller model can surpass general-purpose giants on complex conditional ranking tasks.
Beyond standard benchmarks, we conduct rigorous validation to ensure the robustness of our claims. Comparisons with differentiable ranking objectives (Appendix (A.4) confirm the superiority of RL over regression-basee approaches. Furthermore, evaluation on the re-annotated C-STS dataset (Ap- pendix A.5) validates PoLi-RL's SOTA performance, demonstrating that its gains stem from genuine reasoning improvements rather than overfitting to label noise.
Table Main results on the official C-STS benchmark. All scores are reported as Spearman/Pearson correlation coefficients multiplied by 100. Results marked with are obtained from (Deshpande ctt al. 2023), while I denotes results from (Li et al.) (202)).
Methods
Training Paradigm Parameters
Spearman
Pearson t
Discriminative Model Baselines (Cross-Encoder Architecture)
RoBERTaLARGE SimCSELARGE
SFT
355M
40.7
40.8
SFT
355M
43.2
43.2
SEAVER SimCSELARGE
SFT
355M
43.83
43.81
Generative Large Language Model Baselines
Tan-T5xx f
Few-shot
11B
30.6
Plan-UL2"
Few-shot
20B
23.5
Tk-Instruct
Few-shot
11B
17.8
GPT-3.5t
Few-shot
175B
15.3
DeepSeek-R1
Few-shot
42.85
42.36
GPT-4
Few-shot
43.6
GPT-4o
Few-shot
44.23
44.07
Our Implementation
Qwen3-0.6B
Qwen3-0.6B
Few-shot
0.6B
25.25
25.19
Qwen3-0.6B
SFT
0.6B
35.59
36.83
PoLi-RL (Qwen3-0.6B)
RL
0.6B
44.34
44.36
Our Implementation Qwen3-4B
Qwen3-4B
Few-shot
4B
37.97
38.48
Qwen3-4B
SFT
4B
38.41
39.45
PoLi-RL (Qwen3-4B)
RL
4B
46.23
46.19
Our Implementation on Qwen3-8B
Qwen3-8B
Few-shot
8B
37.9
38.54
Qwen3-8B
SFT
8B
40.42
40.83
PoLi-RL (Qwen3-8B)
RL
8B
48.18
48.27
3.3 ABLATION STUDIES
Effectiveness of the Two-Stage Curriculum and Reward Components. Table dissects the effectiveness of our progressive training schedule. We observe that a Naive RL approach (Row 2), which relies solely on batch-level listwise objective, yields negligible improvement over the few- shot baseline (Row 1), highlighting the necessity of the progressive curriculum. Our PoL.-RL Stage

---

Published as a conference paper at ICLR 2026
I (Row 3) addresses this by building robust foundation, substantially outperforming the few-shot baseline by 6.87 points. Within this stage, ablating the binary reward (Row 4) leads to a discemmible dip in performance, validating its role in grounding the model in the task's ss'ss basic binary judgment.
Building upon this, the full PoLi-RL model (Row 5) further boosts performance by another 3,41 points (Row 5). Deconstructing the success of this final stage reveals that both ranking signals vital: removing the listwise reward (Row 6) incurs the most significant penalty, while removing the pairwise reward (Row 7) also hinders performance. These findings confirm that both the two-stage curriculum and each of its constituent reward signals are essential for achieving optimal results, with the listwise signal being the most critical component in the final refinement stage.
Table 2: Ablation study on PoLi-RL's two-stage training design and its reward components. The A column shows the absolute improvement in Spearman correlation over the indicated baseline.
Reward
Method
Spearman
Pearson 1
A (Spearman)
Component(s)
(I) Few-shot Inference
37.9
38.54
(2) Naive RL
Listwise
38.19
38.34
+0.29 (I)
PoLi-RL (Stage I)
Pointwise + Binary
44.77
44.45
+6.87 vs. (1)
w/o Binary
Pointwise
44.19
43.54
-0.58 vs. (3)
(5) PoLi-RL (Full)
Pointwise + Pairwise
48.18
48.27
+3.41 (3)
Listwise
6 w/o Listwise
Pointwise + Pairwise
46.71
46.37
-1.47 vs. (5)
77) w/o Pairwise
Pointwise + Listwise
47.6
47.59
-0.58 vs. (5)
Sensitivity to Hyperparameters. We analyze the framework's robustness to key hyperparame- ters across both training stages. Regarding reward weights, Table shows that Stage II is robust to weight variations, with peak performance achieved by moderately increasing the pairwise weight
1.5. The model performs strongly even with the default equidistant weights (1:1:1), and shows considerable tolerance to pairwise signal, as halving its weight to 0.5 results in only a marginal performance drop 47.77. Similarly, for the pointwise and listwise weights, deviations from their baseline of 1.0 result in only minor fluctuations. Crucially, the framework exhibits stable conver- gence across all configurations without training collapse. This confirms that our hybrid reward design effectively resolves the optimization difficulties encountered by naive listwise approaches.
Table 3: Ablation study on the reward weights (H1, (2,, (a) in PoLi-RL's Stage II.
Method μn (Pointwise)
p2 (Pairwise)
μa (Listwise)
Spearman
Pearson
1.0
1.0
1.0
47.83
47.83
1.0
1.5
1.0
48.18
48.27
PoLi-RL (Stage II)
1.5
1.0
1.0
47.3
47.23
1.0
1.0
1.5
47.46
47.48
1.0
0.5
1.0
47.77
47.31
0.5
1.0
1.0
47.36
47.18
0.5
47,39
47.27
To determine the optimal configuration for our PSRR mechanism, we study the impact the slice size N, with results presented in Table 4 The empirical results reveal a clear trend: performance peaks at an intermediate slice size of This suggests that optimal balance is required for the ranking signal. A slice that is too small may provide less stable ranking signal, while one that is too large makes the ranking task overly complex for the model to learn effectively. This finding validates the design principle behind our PSRR mechanism: carefully-size
24 and degrades as the size deviates in either direction.
localized ranking signal is more effective than a purely global overly-restricted one. Further sensitivity analyses the binary reward weights, pairwise base reward ( (Rpsse). ) and generation multiplicity (G) are provided in Appendix A.3 consistently validating the robustness of our default configurations.
8

---

Published as a conference paper at ICLR 2026
Table 4: Analysis the impact of the parallel slice size (N) in Stage II. N represents the number of samples used for listwise ranking reward computation in each slice.
Method
N (Slice Size)
Spearman f
Pearson f
16
47.16
46.96
48.18
48.27
PoLi-RL(Stage II)
32
47.44
47.19
40
47.18
47.32
48
46.78
46.84
3.4 GENERALIZABILIT PSRR MECHANISM
To empirically validate the generalizability of our framework beyond the C-STS domain, we extend
our evaluation of PoLi-RL to the WMT-QE 2020 task (Fomicheva et al. 2020). While this task shares the overarching goal of optimizing global Spearman correlation, it diverges fundamentally from C-STS across three dimensions: (1) Domain: multilingual translation quality estimation, (2)
Label Scale: continuous scores from 0 to 100, and (3) Data Structure: independent samples lacking deterministic paired ordinality of C-STS.
For this experiment, we employ minimalist configuration of PoLi-RL by disabling the C-STS- specific pairwise and binary rewards. We relied solely on the pointwise reward (Stage and the PSRR-based listwise ranking reward (Stage II), with prompt settings adapted from Sato al. 2024)
As shown in Table 5 even with this simplified setup, PoLi-RL achieves Spearman correlation of 54.33, yielding substantial 3.43 point gain the progression from Stage (51.72) to Stage II explicitly highlights the added value of the listwise ranking objective. These results confirm that the PSRR mechanism timization for C-STS, but transferable solution for aligning LLMs with listwise ranking objectives across domains and label scales.
the strong SFT baseline (50.90). Furthermore,
merely task-specific op-
Table 5:
Generalization performance on WMT-QE 2020 task (en-zh subset) using only the core PSRR mechanism, excluding C-STS-specific reward components.
Methods
Training Paradigm
Spearman
Pearson 1
Qwen3-88
Fhw-shot
45.03
44.18
Qwen3-8B
SFT
50.90
51.09
PoLi-RL (Stage I)
RL
51.72
50.58
PoLi-RL (Stage II)
RI.
54.33
51.09
4 ANALYSIS
4.1 ANALYSIS OF PREDICTION ERROR DISTRIBUTION
Figure visualizes the distribution of absolute prediction errors on the C-STS validation set. The plot reveals several key insights into the models' behaviors. While error of is the most frequent outcome for all models, likely reflecting the inherent nuances of the C-STS scale, a clear progression of improvement evident. Compared the few-shot and SFT model, PoLi-RL demonstrates more favorable етror distribution. It not only achieves the highest density of perfect predictions (error=0) but, more importantly, effectively suppresses the probability of high-error predictions (errors 2 2). This indicates that our method significantly reduces the frequency of large, unreliable deviations, yielding model that more precise against severe misjudgments.
4.2 QUALITATIVE ANALYSIS: CASE STUDY ON NUANCED REASONING
To qualitatively analyze framework's nuanced reasoning ability. present study in Figure X on the challenging condition "The person's connection with the ground". The baseline models show clear deficiencies: the few-shot model exhibits brittle, literal interpretation, fixating

---

Published as a conference paper at ICLR 2026
1.0
Raw Model
0.8
Model
PoLi-RI
0.6
0.2
0.0
2
Figure 3: Distribution of absolute prediction emror for the raw model, SFT model, and PoLi--I. The r-axis represents the absolute error (Prredicted True) ), and the y-axis is the probability density.
on the preposition 'on' while ignoring the physical context, leading to incorrect prediction. The SFT Baseline, while closer to the target, lacks the reasoning depth to make the necessary fine-grained distinction. In contrast, PoLi-RL applies world knowledge infer that both "jumping" and "doing a flip" result in the same "not connected" state, arriving at the accurate score of 4 by identifying the differing actions leading the state unimportant details. This case study demonstrates that PoLi-RL cultivates the contextual inference ability required for complex conditional judgments.
Case Study: Model Comparison Reasoning Sample
Sentence 1: person riding a snowboard jumps high over the snowy hill.
Sentence 2: A man in blue snowsuit does flip huge snowy mountain. Condition: The person's connection with the ground.
Gold Label: 4 (Mostly equivalent)
Few-shot Inference Prediction: no(1)
Reasoning: In Sentence I... the key action is jumping, which implies the person is airborne and not in contact with the ground. In Sentence 2... the preposition 'on' suggest that he's on the mountain' surface. So. he's connected the ground. Therefore, the sentences have different connections with the ground. Hence, the answer 'no(I)'.
SFT Baseline Prediction: yes(3)
PoLi-RL Prediction: yes(4)
Sentence describes... So he's airborne during the jump, meaning he 's's not connected the ground. Sentence 2 talks about infer that this action, similar to jump trick in snow- sports context, also means he the air. Therefore, they similar because both have the person connected to ground. But final score should because action (jumpin/flipping) leading to this state reprexent minor, differing details.
Reasoning:
Figure 4: A detailed case study comparing the reasoning processes and final outputs of the Few-shot, SFT baselines, and PoLi-RL C-STS sample that requires nuanced reasoning.
CONCLUSION
We introduce PoLi-RL, two-stage reinforcement learning framework that resolves the coarse- grained credit assignment problem associated with listwise objectives C-STS through a progres- sive curriculum and an innovative Parallei-Slice Ranking Reward (PSRR) mechanism. Our method establishes a new tatee--f-hhe--ee art for the cross-encoded architecture, significantly outperforming SFT baselines and proprietary reasoning models like GPT-4o. As the first successful application of reinforcement learning to this task, our study validates powerful paradigm for aligning LLMs with complex ranking objectives, demonstrating the framework's potential to other ranking- tasks.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint htt::2303.087744 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetune lan- guage models. Journal of Machine Learning Research, 25(70): 53, 2024.
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Ra- jpurohit, Ashwin Kalyan, Danqi Chen, and Karthik Narasimhan. C-STS: Conditional seman- tic textual similarity. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. tional Linguisties. doi: 10. 11116//////12222.mmmmmmmaaaaa org 22232003... emn 1-mnnp-mall 345/
Singapore, December 2023. Association for Computa-
URL
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8: 539-555, 2020.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings, 2022. URL
08821
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Baixuan Li, Yunlong Fan, and Zhiqiang Gao. SEAVER: Attention reallocation for mitigating distrac- tions in language models for conditional semantic textual similarity measurement. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 78-95, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/1/2024 5/
findings-emnlpp URL htt.pp:://cclnnthooogyo.... 2024. 2ttp://222......nnnsnn-...
Xianming Li and Jing Li. Bellm: Backward dependency enhanced large language model for sentence embeddings. arXiv preprint arXiv:2311.05296, 2023.
Rensis Likert. A technique for the measurement of attitudes. Archives of psychology, 1932.
Ziyong Lin, Quansen Wang, Zixia Jia, and Zilong Zheng. Varying sentence representations via condition-specifed routers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17390-174011 2024.
Xinyue Liu, Zeyang Qin, Zeyu Wang, Wenxin Liang, Linlin Zong, and Bo Xu. Conditional semantic textual similarity via conditional contrastive learning. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 4548-4560. 2025.
Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wil- son. SemEval-2013 task 2: Sentiment analysis in Twitter. In Suresh Manandhar and Deniz Yuret (eds.), Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop Semantie Evaluation (SemEval 2013), pp. 312-320, Atlanta, Georgia, USA, June 2013. Association for Computational Linguistics. URL htttps::/aclannhholooyg www.ooy..oor.ogg.. S13 2052/
Ayako Sato, Kyotaro Nakajima, Hwichan Kim, Zhousi Chen, and Mamoru Komachi, Tmu-hit's submission for the wmt24 quality estimation shared task: gpt-4 a good evaluator for machine translation? In Proceedings of the Ninth Conference on Machine Translation, pp. 529-534, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1tt.:::77700344777 2017.

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, et al. Agent kb: Leveraging cross-domane experience for agentic problem solving. arXiv preprint arXiv:2507.06229, 2025.
Jingxuan Tu, Keer Xu, Liulu Yue, Bingyang Ye, Kyeongmin Rim, and James Pustejovsky. Linguis- tically conditioned semantic textual similarity. arXiv preprint urXiv:2406.03673, 2024.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaci, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Supper-naturalinsstuuttonss Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07700. 2022.
Kosuke Yamada and Peinan Zhang. Out-of-the-box conditional text embeddings from large lan- guage models. arXiv preprint arXiv:2504.16411 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.0991 0..088,,,,,, 2025.
Young Hyun Yoo, Jii Cha, Changhyeon Kim, and Taeuk Kim. Hyper-cl: Conditioning sentence representations with hypernetworks. arXiv preprint arXiv:2403.09490, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14477 2025.
Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatisties, 7, 2005.
Bowen Zhang and Chunping Li. Pee-tuning: Breaking the contrastive learning ceiling in
mantic textual similarity. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14290- 14302, Miami, Florida, USA, November 2024a. Association for Computational Linguis- tics. doi: 10. 1/22/4////02 m91. TRLhttps:: seattps://alaantclloyyy org/20224 emm 1 p-ma
ma i m.. 791/
Bowen Zhang and Chunping Li. Advancing semantic textual similarity modeling: A regression framework with translated ReLU and smooth k2 loss. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11882-11893, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.3//44/1// emnlp-main m63..663.. URL https: ptttts:://ccaatholooy org/ 2rgg/000 22224
653/
Bowen Zhang, Kehua Chang, and Chunping Li. Cot-bert: Enhancing unsupervised sentence repre- sentation through chain-of-thoughi In International Conference on Artificial Neural Networks, pp. 1448-163. Springer, 2024a.
Bowen Zhang, Kehua Chang, and Chunping Li. Simple techniques for enhancing sentence embed- dings in generative language models. International Conference on Intelligent Computing, pp. 52 64. Springer, 2024b.
Bowen Zhang, Zixin Song, Chunquan Chen, Qian- Qian-Wen Zhang, Di Yin, and Xing Sun. Codiemb: A collaborative yet distinct framework for unified represennation learning in information retrieval and semantic textual similarity. arXiv preprint arXiv:2508.1 1442, 2025a.
Bowen Zhang, Zixin Song, and Chumping Li. Csc-sfp: Enabling unsupervised sentence represen- tation learning via a single forward pass. In Proceedings the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1402-1411 1412, 2025b.
Gaifan Zhang, Yi Zhou, and Danushka Bollegala. Annotating training data for conditional seman- tic textual similarity measurement using large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 27003-27015, 2025c.
12

---

Published as a conference paper at ICLR 2026
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swiftr scalable lightweight infrastructure for fine-tuning, 2024. URL // /arxiv...... org/ www.aabs....e. 05517
A APPENDIX
A.1 RELATED WORK
Conditional Semantic Textual Similarity. C-STS is recent advancement over traditional STS (Nkovv et al., 2013] Zhang et al., 2024a 2025b) that introduces a natural language condition to disambiguate the measurement of similarity between two texts. Research in this area has primarily established three mainstream paradigms: Bi-encoder, Tri-encoder, and Cross-encoder. Given two texts and condition, the bi-encoder architecture typically uses a Siamese network to encode two text-condition pairs, while the tri-encoder architecture encodes each text and the condition separately before aggregation step. A prevalent optimization strategy for these paradigms is contrastive learning. For instance, (Liu et al. 2025) propose a conditional contrastive learning framework that pulls representations of the same text pair under high-similarity condition closer, while pushing them apart under low-similarity one. Extending this, Hyper-CL (Yoo et al., 2024) utilizes a tri- encoder setup where hypernetwork generates condition-specific projectors to dynamically adapt sentence representations within a contrastive objective. More recently, CSR introduced a parameter- free router for the tri-encoder, using the condition to re-weight sentence tokens to amplify relevant information without increasing model size.
In contrast, the cross-encoder architecture processes the concatenated texts and condition as a sin- gle input, enabling deep, token-level interaction. However, this theoretical advantage did not con- sistently translate to superior performance in earlier discriminative models. The state-of-the-an method in this setting, SEAVER, addressed this discrepancy by identifying that such models can be distracted by attention reallocation mechanism that optimizes the model by re-weighting internal attention scores during fine-tuning, forcing a focus on the most salient information.
tokens. To resolve this, SEAVER (Li et al. (2224) introduces an
The advent of LLMs has introduced as feature extractors (Yamada & Zhang 2025) (Li & Lil 2023). However, all these prior works, regardless of architecture, Our work is the first to apply reinforcement learning to this task.
approaches, mainly few-shot inference and using LLMs
confined to supervised paradigms like SFT contrastive learning.
Reinforcement Learning for Large Language Models. Reinforcement Learning (RL) is pivotal for aligning and enhancing Large Language Models (LLMs), with algorithms like Proximal Policy Optimization (PPO) (Schulman al., 2017) widely used to optimize non-differentiable objectives in tasks such as reasoning and code generation. To address PPO's 's limitations, such as high variance in long- sequence tasks, advanced variants have emerged.
Group Relative Policy Optimization (GRPO) (Guo et al. 2025) addresses these limitations by intro- ducing group-relative advantage estimation, which eliminates the need for a separate value function through Z-score normalization of rewards within sample groups. This approach simplifies training and enhances sample efficiency, as demonstrated in models like DeepSeek-RI. Building on this, Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) (Yu et al. 2025) provides an open-source, scalable RL system tailored for LLMs. DAPO incorporates key improvements, includ- ing dynamic sampling to adaptively adjust the number of generated completions based on reward variance, making it particularly effective for long-horizon reasoning tasks.
Leveraging DAPO's powerful optimization engine, our work, PoLi-RL, marks significant depar- ture by being the first to introduce a reinforcement learning framework to the C-STS task, thereby establishing new optimization paradigm.
A.2 PROMPT TEMPLATE FOR POLI-RL
Below is the detailed few-shot prompt used for both the few-shot inference baseline and the training, evaluation process of PoLi-RL.
13

---

Published as a conference paper at ICLR 2026
Prompt for C-STS Task
Judge the semantic similarity between Sentence 1 and Sentence 2 based completely on the given Condition. The final output must be exactly in this format: the similarity judgment ('yes' 'no") followed by score parentheses, wrapped haaswer><<answee tags. Examples: <anawer>yes (4) </answer>, Kanswer>no (1) </answer>, Include no other text, tags, or explanations.
To arrive this output, follow these two steps:
Step 1: Binary Judgment. Determine if sentences are 'similar' ('yes') or 'not similar' ('no").
'similar': The sentences roughly, mostly, or completely equivalent under the con-
dition.
"not similar"; The sentences dissimilar under the condition.
Step 2: Fine-grained Score, Assign an integer score based on Step I:
For 'yes' judgment:
5: The two sentences completely equivalent as they mean the same thing with respect the condition.
4: The two sentences are mostly equivalent, but some unimportant details differ with respect to the condition.
*
The two sentences are roughly equivalent, some important information dif- fers or is missing with respect to the condition.
For 'no' judgment:
The two sentences are dissimilar, are on similar topic with respect to the condition or shares a close semantic relationship. This applies when items are clearly different, but not direct opposites.
The two sentences are dissimilar with respect to condition, representing a direct opposition or a clear, unrelated difference. (e.g.. 'man' vs. 'woman').
Here some examples:
Example 1:
<Sentencel>; A girl is cooking a kitchen and man standing next her. <Sentence2>; A man sitting with a pizza his hand in front of pizza on the table.
<Condition>: The number of people.
<answer>no (1) < answer>>>>
Explanation: The first sentence mentions two people, while the second sentence mentions only person.
Example 2:
<Sentencel>; A wood table sitting by wood framed bed with lamp on it.
<Sentence2>: A microwave, refrigerator, television, and wooden drawers sit in the corner of
bedroom.
<Condition>: The room type. <answer>ycs (5) </answer>
Explanation: We can infer from the two sentences that the room rype are both bedroom.
Example 3:
<Seatencel>; A small crowd gathered around the injured person. <Sentence2>: A crowd jumps up and down the tunes played by artist.
<Coadition>: number of people
<answer>yes (3) </answer>
Explanation: While both sentences mention crowds, is important and unclear how many people there
Now, apply these steps to the following sentences:
<Sentencel>; (sentencel
<Sentence2>: [sentence2] <Conditionx: [condition]
14

---

Published as a conference paper at ICLR 2026
A.3 HYPERPARAMETER CONFIGURATIONS AND OTHER ABLATION STUDY
This section details the hyperparameter settings and provides empirical justification for selections covered in the main paper. The default configurations used for main results are summarized in T
Table 6: Default hyperparameter configurations for the main results.
HyperParameter Default Value
\ 22 As
1:0.25:1
[4 : μ2 Ms
1:1.5:1
Rbase
0.5
G
8
N
24
addition ablation studies presented main paper, we conducted sensitivity analyses on three key hyperparameters: the binary reward weights in Stage I, generation multiplicity (G) and the pairwise base reward (Rease).
Sensitivity to Reward Weights in PoLi-RL Stage In Stage I, the Binary Judgment Reward serves an auxiliary signal to prevent the model from converging to "safe" median scores. We test the sensitivity the weight of this component (A2). Results in Table 7 show that PoLi-RL is highly robust to variations in X2, consistently establishing strong foundation for Stage II training.
Table 7: Sensitivity to Binary Reward Weight (A2) in PoLi-RL's Stage I.
Method
\ (Pointwise)
A (Binary)
À (Format)
Spearman
Pearson
1.0
0.1
0.1
44.94
44.93
PoLi-RL 1.0
0.25
0.1
44.77
44.45
(Stage I)
1.0
0.5
0.1
44.67
44,42
1.0
1.0
0.1
44.76
44.92
Impact of Generation Multiplicity. We investigate the orade-off between exploration diversity and computational overhead by varying G {4, 8, 12). As shown in Table 8 while increasing G theoretically aids RI. exploration, empirical results indicate that G = 4 offers insufficient explo- ration coverage: while the restricted search space may suffice for basic scoring, it fails to gener- ate the diverse candidates necessary for learning the fine-grainee semantic distinctions in Stage II. Conversely, scaling to G = 12 leads to performance saturation and incurs higher computational overhead. These results validate selection of G 8 as the optimal configuration for balancing performance and efficiency.
Table 8: Ablation on generation multiplicity (G) in PoLi-RL both stages.
Method
Spearman
Pearson t
4
PoLaeRL(Stage I)
44.55
44.49
4
PoLi-RL(Stage II)
47.47
47.45
8
PoLi-RL(Stage I)
44.77
44.45
8
PoLi-RL(Stage II)
48.18
48.27
12
PoL-i-RL(Stage I)
45.51
45.61
12
PStaee IT)
47.98
48.06
Rationale for Pairwise Base Reward. The pairwise reward (Eq. incorporates constant base reward Rbase to ensure a guaranteed positive signal when the model correctly predicts the ranking direction (ordinality), even if the exact score gap (cardinality) is imprecise. We compare the default
15

---

Published as a conference paper at ICLR 2026
Roase 0.5 against removing it (0.0) overweighting it (0.75). Table demonstrates that 0.5 yields the best performance, justifying the need for balanced reward structure.
Table 9: Sensitivity analysis the pairwise base reward (Rhase)
Method
Rbase
Spearman
Pearson 1
47.90
47.59
0.25
47.78
47.81
PoLi-RL (Stage II)
0,5
48.18
48.27
0.75
47.54
47.33
A.4 COMPARISON WITH DIFFERENTIABLE RANKING OBJECTIVES
In this section, we explicitly compare PoLi-RL against a strong cross-encoder regression baseline trained with differentiable ranking objectives. The goal is to determine whether optimizing a surro- gate loss is sufficient to capture the rank-based nuances of C-STS. Specifically, we compare PoLi-RL against a baseline trained with the Pearson Correlation Coefficient (Pec) Loss (Zhang & Li 2024a), a state-of-the-art differentiable proxy for Spearman metric, Pec loss defined as:
C (X,(X,
(10)
Ox
where X represents the predicted scores and Y is the ground-truth labels. Table 10 shows that while the regression baseline outperforms standard SFT, PoLi-RL still maintains a clear advantage (1.74). This superiority stems from the paradigm shift: Regression treats the model "black box" that maps embeddings to a scalar. In contrast, PoLi-RL optimizes the reasoning process itself. By aligning the token-level generation probability with the non-differentiable ranking metric, the model learns how to reason towards the correct score, enabling better generalization in complex conditional scenarios.
Table 10: Comparison between PoLi-RI. and SFT with differentiable Pearson Correlation Coeffi-
cient (Poc) Loss regression.
Methods
Training Paradigm
Spearman f
Pearson 1
Qwen3-8B SFT (Auto-regressive)
Few-shot
37.90
38.54
Qwen3-8B
40.42
40.83
Qwen3-8B
SFT (Regression)
46.44
46.59
PoLi-RL (Qwen3-8B)
RL
48.18
48.27
A.5 PERFORMANCE ON RE-ANNOTATED C-STS DATASET
Recent work by Tu et 2024) identified potential label noise the original C-STS dataset and released re-annotated validation set. Subsequently, Zhang et al. (2025c) further refined the dataset by correcting the condition descriptions and utilizing LLMs to clean the training labels. To ensure the robustness of method against data quality issues, we re-evaluate PoLi-RL the refined dataset provided by Zhang et al. (202cc).
As detailed in Table 11. PoLi-RL maintains its significant performance advantage the annotated data, achieving a state-of-the-an Spearman correlation of 76.08 and outperforming the SFT baseline by nearly points. This confirms that our reported improvements robust and valid, rather than an artifact of overfitting to label noise.
A.6 LLM USAGE STATEMENT
The large language model (LLM) was utilized during the preparation this manuscript. The use of this technology strictly confined the role of writing assistant for the sole purpose of
16

---

Published as a conference paper at ICLR 2026
Table 11: Performance the Re-annotated C-STS Dataset. Evaluation performed on the 30% hold- out split of the validation set following the protocol of Tu et al. (2024) and Zhang et al. (2025c Results marked with are obtained from (Tu et al. 2024).
Methods
Training Paradigm
Parameters
Spearman t
Pearson
QA-SimCSELARGE
SFT
355M
72.9
72.3
GPT-3.5t
Few-shot
66.1
64.8
GPT-4t
Few-shot
64.4
60.2
Qwen3-8B
Few-shot
8B
64.42
64.5
Qwen3-8B
SFT
8B
72.09
70.41
PoLi-RL (Qwen3-8B)
RL
8B
76.08
74.16
improving the linguistic quality of the text. Specifically, the LLM was employed for tasks related to grammar. syntax, phrasing, and overall readability. Its function was exclusively to perform surface- level linguistic refinements text already written by the human authors. Crucially, the LLM did not contribute to any substantive or intellectual aspects the research. The conceptualization of the study, the design of the methodology, the execution of experiments, the interpretation of results, and the formulation of conclusions were all executed by the human authors.
17


Reviews:
[{"review_id": "qiCDMuKRuh", "reviewer": "Reviewer_fkXh", "summary": "This paper introduces PoLi-RL, a novel Point-to-List reinforcement learning framework for Conditional Semantic Textual Similarity (C-STS). The work addresses the limitations of existing discriminative models by leveraging large language models (LLMs) and RL. To overcome the significant challenge of reward sparsity in this generative task, PoLi-RL reformulates the problem from a point-wise generation task into a list-wise ranking task. This approach effectively densifies the reward signal.", "questions": "What is the base reward $R_{base}$ in equation (7)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is built on a clear motivation and is well-written, with clean figures that effectively illustrate the proposed method.\n- The method is a reasonable and well-designed approach that directly addresses the issue of sparse rewards, which is a key challenge for applying naive RL to this task.\n- The experimental results are impressive. The paper provides comprehensive experiments and the corresponding analysis.", "weaknesses": "The authors did not conduct experiments on smaller generative models (e.g., qwen 0.5B) that would be more comparable in parameter count to the discriminative models used as baselines. It is unclear how much of the impressive performance gap is attributable to the method itself versus the model's scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKoU76jQ0f", "reviewer": "Reviewer_PEjZ", "summary": "The method is novel and promising to some extent, with respectable results on C-STS; however, evidence for generalization is nearly absent (the reward and training organization are over-specialized to C-STS). Combined with missing strong reasoning baselines and insufficient stabilization details, the present claims do not extrapolate well.", "questions": "Q1. Please add strong baselines for recent reasoning-style models under few-shot/CoT.\nQ2. The current setup relies heavily on the 1–5 label scale, adjacent pairing, and a “cross-sample slice” list domain. Please provide any empirical evidence for generalizability:\n1. Cross-condition/cross-domain/cross-source evaluation;\n2. Changing label scales and thresholds (non-1–5, non-≥3/≤2);\n3. Replace the list domain from “cross-sample slicing” to “per-input candidate set,” and compare stability/effectiveness;\n4. Robustness under noise and distribution shift.\nIf you do not intend to generalize, explicitly delimit the applicability scope and tone down the claims in the abstract/conclusion.\nQ3. Please provide testable rationale for “why these settings,” plus the full hyperparameter details and reasons. Explain the design motivation for each reward and its structure; for several key hyperparameters, add principled justifications or ablations.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1 Technical fit. Using RL to align with non-differentiable rank-correlation/ranking metrics (e.g., Spearman; more broadly NDCG, etc.) is naturally consistent with how C-STS is evaluated.\nS2 Method design. The two-stage curriculum (Point → List) + PSRR’s “parallel slicing” refines credit assignment; the engineering gives reproducible slicing/normalization and a weighting composition.\nS3 Competitive results. Achieves Spearman comparable to or better than strong existing baselines on the C-STS cross-encoder track, addressing the observation that “even strong models may not break 50” on this task.\nS4 Reproducibility. Anonymous code and checkpoints are provided.", "weaknesses": "W1 Baseline coverage. \nThe paper only reports scores of non-reasoning large models (e.g., GPT-4) on C-STS, while reasoning models have progressed rapidly this year. The paper lacks strong baselines for **reasoning-style** models (e.g., DeepSeek-R1, OpenAI o3) under few-shot/CoT, leaving open the concern that “C-STS may already be easily solved by reasoning models.”\nW2 Severe lack of generalizability.\nThere is essentially no evidence supporting “the method generalizes,” and the reward design is highly specialized to C-STS’s annotation and organization:\n1. Stage-I strongly relies on the 1–5 score scale (Eq. (5) uses min-max normalization) and the ≥3 / ≤2 binary threshold (Eq. (6)).\n2. Stage-II’s pairwise term applies only to “adjacent pairs,” normalized by “maximum difference = 3” (Eq. (7)), which is tightly bound to C-STS’s pairing construction and score gaps.\n3. The listwise reward is built on PSRR’s cross-sample slices (Eq. (8)), rather than the more general definition of “ranking a candidate set per input/query.” This makes the choice of “list domain” heavily affected by batch and slicing, with limited extrapolative interpretability.\n4. There is no validation across conditions/domains/data sources; no changes of label scales (e.g., 0–3, 0–4), no alternative list domains, nor robustness reports under different noise/shift settings.\nTaken together, the current evidence only supports the weak claim that the method is “effective on C-STS.” Without systematic experiments across setups, the paper’s main claims lack external validity.\nW3 Insufficient theoretical and empirical support for hyperparameters and reward design.\nThe paper introduces many reward components and hyperparameters (Stage-I’s R_pointwise, R_binary, R_format and weight (λ); Stage-II’s (R_pairwise, R_listwise), baseline constant (R_base ), slice/batch sizes (N, G), and “maximum difference = 3,” etc.), but offers only intuitive motivation. It lacks (i) testable theoretical grounds/invariance guarantees (why the shaping does not change the optimal policy, or why it aligns with the target metric), and (ii) systematic necessity evidence (component-wise/hyperparameter ablations and significance tests). Current experiments cover only a minority of hyperparameters; several key choices remain unexplained or unvalidated, making it hard to disentangle method gains from empirical tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGxBL1oYnx", "reviewer": "Reviewer_6HRs", "summary": "The paper proposes PoLi-RL, a two-stage point-to-list reinforcement learning framework for conditional STS with a cross-encoder backbone. Stage I teaches the model to “score” with pointwise distance, binary-consistency, and formatting rewards; Stage II adds pairwise and listwise ranking signals computed over parallel slices of completions (PSRR), yielding finer credit assignment aligned with rank-correlation objectives. Experiments on the standard C-STS benchmark (with an 8B-scale backbone) show consistent gains over few-shot prompting and supervised fine-tuning; ablations indicate that both the curriculum and the ranking-based rewards contribute.", "questions": "- How does performance scale with model size (e.g., 3B/14B/32B) and with generation multiplicity G and slice size N?\n- Why RL over differentiable ranking losses? Under equal data/compute, how do listwise/pairwise differentiable surrogates perform relative to PoLi-RL Stage II?\n- What is the exact hyperparameter-tuning protocol? Which hyperparameters were tuned on dev, what ranges were explored, and what metric decided the final selection?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The slice-wise ranking mechanism produces differentiated signals per completion, better aligned to Spearman/Kendall-style goals than batch-level listwise rewards.\n- The paper explains how to make PSRR tractable (generation multiplicity, gradient accumulation), offering a reproducible training recipe.\n- Strong performance with an 8B model suggests good cost-effectiveness.", "weaknesses": "- Comparisons to the latest closed-source systems (e.g., GPT-4o, Claude-3.7 Sonnet) are missing, leaving the method’s headroom to stronger proprietary models underexplored.\n- All results are on Qwen3-8B only. Please add multi-size backbones or report compute-parity/parameter-parity comparisons to strengthen fairness claims.\n- The few-shot template and examples are relegated to the appendix; the main text’s notation does not formalize the few-shot setup, which hinders reproducibility. The authors should surface the key prompt template and define the few-shot variables in the notation section.\n- C-STS has been shown to contain substantial label noise [1], and a recent study has produced a re-annotated version of the data [2]. Training and evaluating on this re-annotated dataset would provide a more accurate assessment of the model’s effectiveness. I understand that [2] was released only a few days before the ICLR deadline; however, given the importance of data quality, I believe that evaluation on the newly annotated data would more clearly demonstrate the actual performance gains.\n\n[1] Linguistically conditioned semantic textual similarity. ACL2024\n\n[2] Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models. EMNLP2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)