Angles Don’t Lie: Unlocking Training‑Efficient RL Through the Model’s Own Signals
Authors: Qinsi Wang
Jinghan Ke
Hancheng Ye
Yueqian Lin
Yuzhe Fu
Jianyi Zhang
Kurt Keutzer
Chenfeng Xu
Yiran Chen
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM Efficiency
Data Optimization
Reinforcement Fine-Tuning
Training‑Efficient RFT
Primary area: deep_learning
TL;DR: We present a model-aware approach that leverages the model’s own signals to dynamically choose training data, markedly boosting both training and data efficiency in RL fine-tuning.

Abstract: Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models (LLMs) suffer from sample inefficiency due to the redundant exposure of identical queries under uniform data sampling. While previous work has explored curriculum learning via heuristic difficulty metrics, these strategies exhibit limitations by neglecting the intrinsic learning signals generated by the model itself, thus leading to suboptimal training regimes. In this paper, we identify a model-inherent signal termed *angle concentration* that effectively reflects an LLM's capacity to learn from specific data. We theoretically and empirically demonstrate a correlation between the angular distribution of token hidden state vectors and the resulting gradient, revealing a learning preference for data exhibiting higher angle concentration. Inspired by this finding, we propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. 
By leveraging the model's intrinsic angle concentration signal, GAIN-RL dynamically selects training data in each epoch, ensuring consistently impactful gradient updates and thus significantly enhancing overall training efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5$\times$ acceleration in training efficiency across diverse mathematical and coding tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient sampling yields data-efficient training, achieving better performance with half the original data compared to vanilla GRPO with full training data.

Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals
Qinsi Wang'
Jinghan Ke²
Hancheng Ye!
Yueqian Lin' Yuzhe Fu'
Jianyi Zhang'
Kurt K
Chenfeng Xu²* Yiran Chen
Duke University
University of California, Berkeley
Abstract
Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models (LLMs) suffer from sample inefficiency due to the redundant exposure of identical queries under uniform data sampling. While previous work has explored curriculum learning via heuristic difficulty metrics, these strategies exhibit limitations by neglecting the intrinsic learning signals generated by the model itself, thus leading to suboptimal training regimes. In this paper, we identify a model-inherent signal termed angle concentrution that effectively reflects an LLM's capacity to learn from specific data. We theoretically and empirically demonstrate a correlation between the angular distribution of token hidden state vectors and the resulting gradient, revealing a learning preference for data exhibiting higher angle concentration. Inspired by this finding. we propose GAIN-RL, a Gradient-driver ddiven Angle-Informer Navigated RL framework. By leveraging the model's intrinsic angle concentration signal, GAIN-RL dynamically selects training data in each iteration, ensuring consistently impactful gradient updates and thus significantly enhancing overall training efficiency. Empirical evaluations show that GAIN-RL. (GRPO) achieves over 2.5x x acceleration in training efficiency across diverse mathematical and coding tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s 's efficient sampling yields data-efficient training, achieving better performance with half the original data compared to vanilla GRPO with full training data. Code is realsed at http:///ww....oooo.ooo/nnn/nnlll
Introdction
Since the emergence of groundbreaking Reinforcement Learning Fine-tuning (RFT) techniques exemplified by Deepseek-RI ( and OpenAl's 2 significant attention has converged on leveraging these approaches to enhance performance, notably in mathematical reasoning 3 4 [5] and code generation tasks [6] (7). This interest has catalyzed the development of numerous algorithmic optinization methods, including GRPO B. ReMax (9) and Reinforce++ [00. Despite such remarkable progress, critical challenges persist: RFT remains hindered by persistent issues of low sample efficiency and prohibitively high computational costs. For instance, the GRPO fine-tuning phase on Qwen 2. (Ray- + vLLM) still consumed roughly 240 GPU hours (16 x H100-80 GB for 1.5h) to complete only 100 steps over 8k samples (II). This low sample efficiency prompts the question: Is it truly necessary to repeatedly expose every data point to the model hundreds of times? Our answer is No. Humans adaptively learn by focusing rather than rote repetition of simple concepts. We extend this insight into reinforcement learning fine-tuning for LLM reasoning models from the perspective of data manipulation.
what they don't ton't yet understand,
Manipulating the data crucial for accelerating data-driven LLM training. Existing data manipulation techniques fall into two main categories: Sample selection methods, such as LIMO [2] and S1 [3]
*Conresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DATA
1. Anglle-based Data Reordering
Sorted DATA
2. Gaussian-based DATA Sampling
Sampling Probabiltty
N (1,
System
Prompt
Token
MODEL
#Prediiliing
High
Cavra
BI FIBE ua DATA index
Angle Signal
+ Caver
+ (acc)) (anglen)
Layer index
Question
Low
MODEL
Deccdding Pes
Token
Cruationt
3. Dynamii Probability Update
Figure 1: Overview of GAIN-RL. GAIN-RI. consists of three steps: (1 (Angll--aasee Data Reorder- ing: Before training. the model pre-fills all data and ranks them by the combined angle concentration signals: Cinter + Cintra- (2)Gaussian- sampling reordered data using Gaussian distribution. (3)Dynamic Probability Update: Epoch-wise accuracy and angle concentration are collected to dynamically update 1t+11 GAIN-RL guides the model focus on high-angle, high-loss data, promoting effective gradients and faster convergence.
Data Sampling: During training. each iteration begins by
have shown that training on a carefully curated subset of high-quality data can improve performance. Separately, data ordering strategies, like ADARFT [14) have demonstrated that dynamically adjusting data difficulty during training can accelerate convergence. However, existing approaches suffer from two fundamental limitations that hinder their effectiveness in RFT. First and foremost, current methods neglect the intrinsic characteristics of the models themselves. They rely on fixed, evaluate data without accounting for how the target model itself perceives the data. We point out that different models interpret the same problem in markedly different ways. As shown in Fig. 2 different models produce diverging accuracy distributions on the same set of data, indicating that mee-siz--iits-ll difficulty measures can lead to suboptimal training outcomes. Second, existing methods suffer from high data preprocessing costs. For instance, $1 and LIMO necessitate running large-scale models (e.g., Owen2.5-Math-- across entire datasets to compute quality and difficulty scores. Similarly, curriculum learning often relies manual annotation or expert-defined difficulty labels. These eesource-intensive preprocessing steps significantly limit the scalability and practical responsiveness of these approaches.
model- aggnosiic criteria- -ucc as difficulty or diversity
We aim to attack the aforementioned challenges and propose model-informen signal that (1) reflects the learning capacity of specific model on particular data, (2) incurs minimal computational costs, and (3) maintains generalizability across diverse models and datasets. Achieving such requirements inherently challenging, as accurately capturing model-data interactions typically needs resource- intensive decoding stage. To overcome this challenge, we explore three key questions in this work:
Which signal should we focus on? By reformulating the gradient expressions, we find that the cosine similarity between token hidden states during inference (hereafter referred to as angle concentration) directly influences gradient norm, underscoring as a critical signal.
2. What are the characteristics of the signal? By tracking the layer-wise evolution, we observe transition from inter-segment to intra-segment angle concentration, which jointly facilitates information flow and constitutes the Layer-wise Angle Concentration Pattern.
3. How these signals accelerate training? By monitoring angle concentration throughout training, we observe continuous convergence of intra- and inter-segment angles over epochs, revealing an Epoch-wise Angle Concentration Pattern. Moreover, the model preferentially learns samples with higher angle concentration before those with lower concentration, indicating Data-wise Angle Concentration Pattern. These patterns highlight the model's data-learnng preferences and can be leveraged to accelerate training.
Building upon these insights, we propose GAIN-RL, Gradient-driver Angle- Informed Navigated Reinforcement Learning Framework, illustrated in Fig. [ GAIN-RL comprises three primary components: Data Reordering, Data Sampling and Probability Update. Before training, we reorder the training data based on model-informed angular concentration to enhance learning efficiency. During training, a dynamic Gaussian probability sampling strategy progressively guides the model towards data with lower angular concentration, with the pace adjusted according to real- cal-time accuracy

---

Correct Num(Max (Max=10) -00)
10.0
Llama-3.1 3.2-1BBInsttee
Chatgpt
7.5
5.0
2.5
0.0
20
40
60
80
100
Question Index
Figure 2: Visualization of models' responses. We evaluate the first 100 GSM8K questions by having Qwen2.5-0.5B- --een.--00.B-lnsttuuc and LLaMA3.22 I---truntt each generate 10 answers per question, recording the number of correct responses. ChatGPT-40 rates question ease on 0-10 scale (10 easiest).
and angular signals. Notably, our data preprocessing involves only the inexpensive pre-filling stage, requiring under 10 minutes for over 7,000 samples. Overall, GAIN-RL provides a dlug-and-playy broadly applicable, and data-efficient reinforcement learning solution. Experiments validate that GAIN-RL accelerates training by over 2.5 and, remarkably, surpasses full-sample performance using only half the data demonstratnn its powerful capability in driving data-efficient RFT.
In summary, our key contributions include:
We propose that the angle concentration serves as a critical signal influencing gradient norm during training, thus predicting the model's ability learn specific data samples.
We demonstrate that angle concentration intrinsically reflects information propagation during inference and learning dynamics during training, and explicitly reveals Layer- wise, Epsch-wise, and Data-wise Angle Concentration Patterns.
We introduce GAIN-RL, a framework that dynamically allocates training data per epoch based angle concentration signals. GAIN-RL is the first framework to utilize intrinsic model signals for data scheduling, providing a novel paradigm for efficient RFT.
2 Model-Informed Data Evaluation Signals
With the goal of identifying signals that can effectively reflect a specific model's 's learning capability for data, in this section, we explore three key questions: (1) Which signal should focus on? (2) What are the characteristics of this signal? (3) How can this signal be leveraged to accelerate training?
2.1 Which Signals Should We Focus On?
As mentioned in Sect. 1. obtaining model feedback via decoding is computationally expensive. In contrast, the pre-filling stage incurs significantly lower costs, as requires only a single forward pass. Intuitively, we hope to identify signals from the pre-filling stage that can inform the training process.
Model training driven by incremental gradient updates. Hence, to investigate how forward signals influence the backward process, we begin by examining and reformulating the gradient representation. Given a weight matrix W Rddx,, the hidden states of input can be denoted as a E Rmmxd, where consists of tokens, and the hidden state of the i-th token is represented activation is denoted a = x xW, where a E mmx Assuming the loss function has a gradient VaL E mm with respect to the activation the gradient of L with respect to W is given by
2; The output
VwC = x vaC= Σ x (Val),
where (VaC); denotes the gradient of the loss L with respect to the i-th activation vector a, To more precisely quantify the magnitude of the gradients, we consider the Frobenius norm of V Vw L. Leveraging the linearity of the Frobenius inner product, w/ L11 can be expanded as:
V Cw =
Es (VaC),
E J=1
2, (vac)))r (2)
= ==1==1
Next, to simplify Eq.
2 we utilize the compatibility between the Frobenius inner product and the matrix outer product. In particular, for any u, w RRd and v, Rt, the following identity holds:
(uv)) w x0 T) tr((uv' " J[ue (w=T)) (u w) (v 2).
3

---

where tr denotes the matrix trace operator. Applying this identity term-wise in Eq. gives
WW [[
Σ |
cos cos ((VaL) (4)
where cos 0,g denotes the cosine similarity of the angle between zy and z. Eq. exxpliitty reveals that, during inference, both the magnitudes of token hidden states and the angles between them directly influence the gradient values computed during backpropagation.
Since the magnitudes of token hidden states are normalized in each layer during inference, they cannot effectively convey useful information. Consequently, in this paper, we specifically focus on exploring the characteristies of relative angles of token hidden states. Furthermore, in the Appendix [B.1] provide proof' demonstrating that the nonlinear transformations in LLM inference anference-including both attention mechanisms and activation functions-arr inherently angle-dependent and continuously modify the angles among token hidden states. We can now answer the first question:
A1. We should focus on the relative angles between token hidden states during inference as fundamentally impacts the gradient Frobenius norm computed during backpropagation. Specifically, the more concentrated the angles between tokens, the larger the gradient norm.
2.2 What Are the Characteristics of This Signal?
In the previous subsection, show that angles between token hidden states directly influence the gradients. To identify the characteristios we should focus on, we explore its layer-wise evolution.
Layer-wise Observation. We conducted experiments the Qwen2.5-00 ---..bb-nnsttcct model to observe the evolution process across different layers. The experimental results, shown as Fig. 3 clearly illustrate that in the initial layers of the model, no distinct pattern emerges, with the angles primarily determined by the input embeddings. As the layer depth increases, angles gradually exhibit segmented structure, whereby the hidden states of tokens within the same segment tend to cluster more closely. Upon further examination, we found these segments correspond precisely to distinct parts of the input sequence: the system prompt, few-shot examples, and the question. Eventually, in the final layers, angles between tokens from different segments begin to converge, reaching the highest degree of concentration. We give more vivid demonstration in Fig.
Based on the above observations, we introduce Layer-wise Angle Concentration Pattern: during inference, the model first induces intra-segment angle concentration and subsequently promotes inter-segment angle concentration. These two forms of clustering collaboratively facilitate informa- tion propagation through the model. A detailed demonstration is provided in Appendix C.T
Therefore, assume the length of the input tokens is m. The first tokens constitute the system prompt and the few-shot examples, which are the same across all data samples. The remaining m tokens represent the specific question be answered. The characteristies we should focus on are:
Cintra
Σ
m
m
cos Bssjs
Cinter
1
m
T
cos B.
(5)
(m n2
(m 12)
1=
1
where Cintra measures angle concentration within the question and Cinter measures angle concentration between question tokens and the system prompt and few-shot tokens. We measure both at the final layer, where inter-segment clustering is maximal. Concentration within the system prompt and few-shot tokens is omitted, as is constant across different questions.
Attention-based Explanation. To better understand the observed pattern, also provide an analytical explanation from attention scores. In general, we find that tokens with higher angle concentration correspond to higher attention scores. Specifically, Cintra represents the strength of attention within the question itself, while Cinter indicates the model's ability to follow instructions. Furthermore, the presence of sink attention encourages intra-segment and inter-segment angle concentration. Details can be found in Appendix B.2 Our answer to the second question is:
A2. The angles between token hidden states show both intra-segment and inter-segment concentrations during inference. In particular, we should pay attention to the final layer as the inter-segment clustering is most pronounced, resulting in the highest overall concentration.

---

Conin Valuee
Layer 0
Layer 5
Layer 15
Layer 23
Figure 3: Visualization of Layer- Concentration Pattern. Experiment is performed on the Qwen2.5-0.3 50.55---..-Insttrucc In each subplot, the pixel at row i, column j represents the cosine similarity of the angle between the i-th and j-th token hidden states of the layer output. Blue, yellow, and gray arrows above the figure represent the tokens of the system prompt, few-shot examples and question, respectively. To better highlight the pattern, values clipped between the 3rd and 97th percentiles.
0.40
5-w.n2.5-0.58 Insis
0.38
0.23
Qleen2, 5 5-0.5B-trrttt
.0.22
Qwen2.5 5 5.5.3 Instr
0.60
Liama 3.2
lnstr
0.35
0.22
Libma 322 LB totte
0.20
0.62
Elama 3.2 19 instr
0.55
0.33
0.21
0.388
0.60
0.50
0.21
0.36
0.45
0.37
0.34
100 Training Iteration
150
250
0.20
50
200
0.32
100
200
250
Training Itoration
Training lteration
Figure 4: Visualization of Epoch-wise Concentration Pattern. (Left) Cintra's (Mid) Cinteri (Right) Cintra + Cunter- We train Swen2.5-0.5B. -.5tttruttttt and LLaMA3.2-1b5 In-Istuccttt on GSM8K using GRPO for 250 iterations. To accelerate observation, we use training batch size of 16, generate responses per question, and set the learning rate to le-5. After each iteration, we perform pre- pre-filling on the entire dataset and record the angle concentration from the hidden states of the final layer output.
2.3 How Can This Signal Be Leveraged to Accelerate Training?
Having established that intra-segment and mete-segmment angular concentrations at the final layer crucial characteristics, we now further examine their evolution during training deepen our understanding of how they reflect and influence training progress.
Epoch-wise Observation. To track how angular concentrations evolve during training, we perform inference on the same dataset on models of different epochs and monitor three signals: (1) Intra- question concentration Cintra. (2) Inter-segment concentration Cinter. (3) Combined signal Cintra - Cinter- As illustrated in Fig. 4 we can observe the Epoch-wise Angle Concentration Pattern: during training, both inter-segment and intra-segment angle concentration increase progressively. Additionally, we note that the intra-question concentration initially decreases before subsequently increasing, which suggests the model prioritizes mastering instruction- catruction-following capabilities before refining its focus internally on individual questions. These observations validate our hypotheses from the previous subsections, reinforcing that angular concentration effectively mirrors training dynamics.
Data- Observation. Furthermore, examine data-wise angle behavior during training, track the model's responses to samples with varying angle concentrations over epochs. As depicted in Fig. 5 surprisingly, the results revealed that during training, the model tends to prioritize learning from higher-angle concentration data before addressing lower-angle concentration data, which we introduce as Data-wise Angle Concentration Pattern. For instance, by iteration 100, questions with maximal angular measurements were almost entirely answered correctly, whereas those with smaller angles remained uncorrected. Note that the angle concentration distribution is measured on the untrained model, indicating that despite its evolution during training, the initial angle concentration of the data provides meaningful guidance for the training process.
To understand these patterns, we provide explanations from two perspectives: gradients and neurons.
Pradient-based Explanation. As shown in Eq. 4 gradients influenced by both angle concentra- tion and loss. Early in training, when losses are relatively uniform across samples, those with higher angle concentration receive stronger gradients and are learned faster. As training continues, angle concentration rises overall: high-angle samples, already mastered, see lower losses; while lowlengle samples gain concentration, inherit larger gradients, and are learned next. This creates natural, angle-driven learning progression. Unlike traditional curriculum learning based on task difficulty, angle concentration offers more intuitive, model-centric training signal.
Neuron-based Explanation. Consider FFN block with weights W., and Wa, and activation function SiLU, Given the input denoted as x, the transformation process be represented as:
5

---

Epoch 0
Epoch 50
Epoch 100
Avg. Correct(Max 10)
Number of Sample
100
50
0.50
0.55
0.60
0.50
0.55
0.60
0.50
0.55
0.60
Cintra + Canter
Cintra + Cinter
Citraa + Gitter
Figure 5: Visualization of Data-wise Concentration Pattern. Experiments conducted on We first performed pre-filling on 1,000 samples of GSM8K using the untrained model to collect Cintra + Cinter of samples, and plotted their statistical distributions (the histogram in the figure). Then we monitored the model responses to these samples at various iterations and recorded the average number of correct responses from samples in each angle concentration interval (brighter colors indicate a higher number of correctly answered samples within the interval).
1ween2.5-0.5t-Instree
Principal Component 2
Epoch
Epoch 50
Epoch 300
Avg. Correct(Maa 10)
200
-200
400
500
500
500
500 -500
D
500
Principal Component
Principal Component
Principal Component
Relationship between neuron activation patterns and accuracy over training. At selected epochs, we collect both activation and answers for the first 1000 GSM8K samples. For each question, we identify the most frequently activated 20% of neurons across all tokens in the final layer, and use their indices construct binary core-ncuron vector. We apply PCA to reduce these vectors to 2D. Each dot represents a sample; brighter colors indicate higher answer accuracy for the sample.
Figure 6:
= æW.. u. A SiLU(z), y AWa where and A denote the pre-activation and activation outputs, respectively, and y represents the output. The gradients of j-th neuron in Wa and Wa are
(Vw.L).s Σ x: ((VA)),) (iLU(()))) (w.,
A (V,C),
where SiLU (Zt,) is the derivative of the SiLU activation with respect to Zi.j. When =.j < 0, have SiLU' x O and SiLU(zz.j) R O. This indicates that the number of gradient components received by a neuron is proportional to the frequency of its activation.
Further analysis shows that tokens with higher angle concentration activate similar neurons due to shared value patterns (see Appendix B.3). These neurons receive stronger cumulative gradients, making them more effectively trained. Fig. further illustrates how neuron activations converge during training, forming a distinet cluster correlated with higher accuracy. Samples activating neurons far from this cluster are harder to learn. Following prior work on neuron specialization, we hypothesize this cluster encodes domain-specae knowledge [5 16 17].
Synthesizing the above analyses, we provide an answer to the third question as follows:
A3. We should follow the model's inherent learning dynamics prioritizing higher-angle concentration data in the early stages and gradually transitioning to lower-angle concentration data. This progression ensures more effective gradient updates and improves training efficiency.
3 GAIN-RL Framework
Based on the conclusions from the Sect. Informed Navigated-data RL framework, a plug-and- training acceleration framework compatible with any model and dataset, incurring negligible costs. GAIN-RL consists of three components:
we introduce GAIN-RL, a Gradient-driven Angle-
Data Reordering Based on Angular Concentration. Guided by our findings in Sect. 2.3 that models preferentially leam from data with higher angular concentration--we order the training data by angular concentration before training to improve efficiency. Given model M and dataset D D= {di, d2..... ... dv), first perform pre-filling on all data samples using M to collect angular information. Subsequently, the data is sorted based on the combined signal at the final layer output,
Cas (d.) Ciatra (d.) C.M (d.), D2 Sort (D;CM (d+), descending)
(7)

---

Table I: Comparison of @ass accuracy on Math benchmarks. We report the accuracy at epoch 200 and the number of epochs needed to match vanilla GRPO's 200-epoch accuracy (Fpoo Same Acc).
ADARFT(GRPO) and GAIN-RL(GRPO) denote GRPO combined with the respective optimization.
p...............................
Experiments Setting
Task Performance (200 lteration)
Hardware Efficiency
GSMSK 3
Math 4
AMC'23
AIME 24
Olympiad Bench 20
"Minerva Math 21
Herr
Speed
Model
Method
18
19
Avg
Same Acc
Up
Qwen 2.5
GRPO
84.15
64,40
38.55
10.00
25.63
13.97
39.95
200
1x
Math
ADARFTIGRPO)
85.52
66.00
40.96
13.33
26.07
14.71
41.09
150
1.33x
GAIN-RL(GRPO)
88.09
67.20
43.37
13.33
27.26
16.54
42.63
2.50
GRPO
74.60
40.20
19.28
6.67
11.70
8.46
26.8
200
1x
LLaMA 3.2 3B Instruct
ADARFT(CRPO)
78.01
39.00
18.07
6.67
12.89
8.56
27.2
140
1.4x
GAIN-RI(GRPO)
76.04
42.00
21.69
6.67
14.22
10.29
28.5
80
2.50x
Qwen 2.5 Math 7B
GRPO
91.96
68.40
40.96
10.00
25.78
20.22
42.89
200
1x
ADARFT(GRPO)
92.65
70.20
42.17
10.00
25.33
21.32
43.61
150
1.33x
GAIP-RL(GRPO)
93.71
72.80
45,78
13,33
26.81
23.53
46.33
70
2.86
Table 2: Comparison of model performance on Code benchmarks. ADARFT is not compared because DeepCoder lacks the difficulty coefficients required by ADARFT.
Experiments Setting
Task Performance (200 Iteration)
Hardware Efficiency
Model
Method
LCB 0
LCB
Codeforces
2*
Codeforces
Humaneval+
23
Avg
Iter@
Speed
Pass
P
Pass G
Pass
Same Ace
Up
Qwen 2.5 Crider
108
21.5
5.15
200
GAIN-LL/GRPO)
12.8
25.1
6.14
18.2
81.5
28.8
110
1.81
the sorted dataset D2 is directly employed in subsequent training. Notably, this sorting process is computationally efficient, it only requires the pre-filling step, which can be efficiently batched. For example, sorting approximately 7000 samples of GSM8K with the 2.5--225-0.5--1 -.5-instuuct model takes less than 10 minutes on a single NVIDIA A100 GPU. In contrast, previous approaches often required manual annotation or generation via large models, typically consuming several days.
Data Sampling Guided by Gaussian Probability. During training, consistently prioritize data with higher angular concentration. At the th training step, we assign sampling probabilities to each data sample in the sorted dataset D, = {di, dā .q.....d d based on Gaussian distribution parameterized by 14 and 0N- A subset ((() of size is then sampled according to,
(i 22) P.(dft) = 7, (-
1
d()) (10~~ Sample (D,; Pr,
exp
20
where Ze is a normalization constant ensuring that probabilities sum to unity. Employing probabilistic sampling instead of strictly sequential sampling enhances the stability and robustness of training.
Probability Update Based on Accuracy and Angular Signals. The Gaussian mean μe sets the peak-sampling region. We initialize pa = 0 to prioritize high-angle concentration data, then gradually increase be as the model masters these samples, shifting focus oweer-nnge concentration ones. At each step, 14 is updated from the batch d(() using its average accuracy and angle concentration,
Acc (0)
1
ACCM, (d."), c(1)_
Σ CM, (d))),
where M, represents the model the th training step, and AceM, (a and C.s. (d( denote accuracy and angular concentration signals. Notably, computing these signals incurs no additional cost model inference is inherently performed during training. The update rule for fe41 1 given by,
1441 f4 + n tanh (a(Acc()) 23) -3) + = tann (x-c()).
(10)
where a adjusts accuracy sensitivity, 33 sets the target accuracy, and y controls angle sensitivity (see Sect. | for guidelines). This update strategy maintains high-gradien training by targeting samples near the desired accuracy while gradually incorporating harder, lower-angle data efficiently.
Appendix experiments reveal that weighting the signal as Ciutra + k CMM can further boost performance. For simplicity and broad applicability, we adopt the unweighted signal here and leave signal optimization to future work. Our main goal is to highlight angle concentration as key signal.
7

---

Accuracy
37
GAIN-RIAN (olur))
20
GAIN- (AIN-RL(Ourr)
25
50
75
100
125
15c
375
200
23
50
75
100
125
150
175
200
Training Iteration
Training Iberation
Figure 7: Learning Dynamics of Different Methods (Left) Qwen2.5-Math-1.. --5tsrrruutt and (Right) LLMMA--32-3-IInsttuee The y-axis shows average accuracy over GSM8K, Math, AMC 23, AIME 24, OlympiadBench, and Minerva Math. Performance is evaluated every epochs.
High AnpConn-ie Sampling
Uniferm Sampling
Low www.AnCCoot-aaaee -biused Sanpling
24
AngiOnn Aigooen--iioeed binse Samplnn
Sumpping
22
Sampling
Accuracy
20
GRPOLPUII
(Ustrainc Mod.ll
1xx E40 [Uniminned Modal)
Cintraited Nodall
25
50 75 100 125 350
Training Iieration
175
200
Figure 8: Data Efficiency Analysis of GAIN-RL. We sampled half of the data from the math dataset using three distinet sampling methods train the Qwwe2.55-0.5b-instmu model using GAIN- RL(GRPO):: (1) High Angle Concentration- biased Sampling, (2) Uniform Sampling, and (3) Low Angle Concentration- biased Sampling. The first three figures illustrate the distribution of the sampled data (highlighted in bright colors) within the overall dataset (grey) under each sampling scenario. The rightmost figure presents the performance models trained on differently distributed data, where GAIN-RL (Full) and GRPO (Full) denote results obtained by training with the complete dataset.
Experiments
To evaluate the effectiveness of GAIN-RL, we conducted a comprehensive experimental study on five levels: (1) Training Efficiency(Sect. 4.1 (2) Data Efficiency(Sect. 4.2h. (3) RL Algorithms Gen-
eralization(Sect. 4.3 (4) Performance on Individual Tasks(Sect, 4.4). and (5) Ablation Studies(Sect..
4.5) For detailed descriptions of the used models and datasets, please refer to the Appendix E
Training and Hyperparameter Settings. We set the target accuracy 8 = 0.5 to maintain strong gradients during training. Sensitivity parameters
2 (for accuracy) and concentration) are tuned on a validation set to ensure stable learning, keeping the tanh function approximately linear over (t) E [o, 1] and c(o) E [,,, Training conducted using GRPO with batch size and sampling number n of 1024, implemented the VerL. framework with NVIDIA A A100 GPUs Additional details are provided in the Appendix
0.5 (for angle
Baseline Settings. We compare GAIN-RL with the vanilla GRPO and ADARFT state-of-the- art dynamic curriculum learning method RL. The training settings are the same for all methods.
4.1 Training Efficiency of GAIN-RI
To evaluate the training efficiency of GAIN-RL, we use the DeepScaleR (24] and DeepCoder [25] datasets to train models for math and code tasks, respectively. They both cover diverse problem types and difficulty levels. Performance is tested on six math and three code benchmarks.
Model Performance. We report the performance of each method at 200 iterations during training. As demonstrated in Tab. and Tab. 2 models trained with GAIN-RL(GRPO) exhibited superior performance across nearly all math and code datasets compared to models trained by vanilla GRPO and ADARFT(GRPO), Notably, with the Qwen2.5-Mathh 1iae-11.BBIInttiii 3B-Insstrct model, GAIN-RI(GRPO) increased average accuracy across six math datasets from 39.95% and 41.09% to 42.63%, representing gains of 2.68% and 1.54%, respectively. Furthermore, the performance improvement observed on

---

51
Acnuracy
Aomuracy
45
39
GRPO
GREO ADARFTIOSPOI
GAIN-RI (ours)
BRIO
28
ADAR TIGRPP
GAIN-RL (eard)
GAIN-RI (ours)
35
100
175
125 17s
Training
Training Iteration
Training
Figure 9: Fine-tuning Performance on Single Tasks. (Left) GSM8k. (Mid) Math. (Right) AMC 23. Models trained the training sets and evaluated their validation sets. ADARFT is excluded from GSM8K due to missing difficulty coefficients.
24
Sort(Angla) Updata((ngll & Anc)
GAIN (1-21
Sort(Angie)+ SaceAAmiio/pUUuaaeAeeeee Update(Acc)
GAIN-RL-im-768
22
SortiAngiee Upsaattnnio
GAUN-RLi==1020)
Sort(Arrr) UpdstelAco
20
GRPON==1024)
Accuracy
20
16
12
0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Training Iteration
Training Iteration
Figure 10: (Left) Ablation Study Module Components. (Right) Small Batch Scalability Test.
Experiments use Qwen2.5-0.5b-Im on Math training set, evaluate on the test set every 20 iterations.
LLaMA3.2-3B-Imn Innstrutt demonstrates that GAIN-RL(GRPO) generalizes across different model families. These results highlight both model-level and task-level generality of method.
Hardware Efficiency. To evaluate hardware efficiency, we report the number of iterations required for each method to reach the performance of vanilla GRPO at 200 iterations. As shown in Tab. and Tab. 2 GAIN-RL(GRPO)) requires approximately only half the iterations across various models and tasks. Specifically, for the --.. RL(GRPO) achieves approximately 2.5 speedup, significantly outperforming ADARFT(GRPO) speedups of 1.33 and 1.43x, respectively. Fig. further visualizes performance trajectories during training, where RAIN-RL(GBE) performance every iterations. As analyzed in Sect. 2 of our method to maintain strong gradient signals throughout training, leading to faster learning.
and
M-MMAA..2-3BIntteeee models, GAIN-
consistently demonstrates faster convergence and superior
the acceleration mainly stems from the ability
4.2 Data Efficiency of GAIN-RL
To further investigate whether GAIN-RL can enhance data effectiveness in RFT, we fine-tuned the Qwen2.5-0.55 -.-.ruuntttt model on the Math dataset. We sampled half of the training dataset using three distinct distribution strategies for training data was randomly selected; (2) High Angular Concentration-
training: (1) Uniform Sampling, where half of Sampling, where data points were sampled based on their angular concentration scores, assigning linear sampling probability with a slope (prioritizing data points with higher angular concentration); and (3) Low Angular Concentration-bian biased Sampling. applying linear sampling probability with slope of -1.
4.3 RL Algorithms Generalization To verify the algorithm generalization of GAIN-RL, we also combine GAIN-RL with PPO. We evaluate the final performance and hardware efficiency. Results in Tab, highllghh substantial gains in both performance and hardware efficiency with GAIN-RL(PPO), achieving an average of 2 training speedup across three mathemat- ical benchmarks. This consistent benefit arises because gradient updates are fundamental across RI. algorithms, validating GAIN-RL's universal acceleration capability.
Table 3: Performance GAIN-RI. com- bined with PPO. -reee..5-0.bb-nmsee is trained and evaluated on three datasets. Im- plementation details
Appendix E
Task Performance
Hardware Efficiency
GAIN-RI.
Here
Dataset
PPO
Speed Up
GSMBK
42.46
45.26
80
2.5x
Math
32.15
34,80
100
2.0%

---

We compared the performance of models trained under these sampling strategies. The experimental results, illustrated in Fig. [ reveal that when using only half of the data, the model trained with High Angular Concentration- Based on the analysis in Sect. [.3] we hypothesize this phenomenon occurs as data points with low angular concentration tend to produce smaller gradient updates and activate dispersed neural regions. Consequently excluding these data points may enhance training efficiency. This insight provides new guidance for data selection: prioritizing high angular concentration data and discarding low angular concentration data can significantly improve data effectiveness of RFT. These findings highlight GAIN-RL's potential and effectiveness in guiding data selection. With uniform sampling, GAIN-RL using half the data performs slightly worse than with full data but remains on par with vanilla GRPO. In contrast, biased sampling toward low-angle concentration leads to unstable and poor results, consistent with our analysis of weaker gradients and dispersed neuron activations.
Sampling outperformed even the model trained with the full dataset.
4.4 Performance on Individual Tasks
To further demonstrate the generality of GAIN-RL(GRPO), we also evaluate its performance on single-task RFT, which demands higher precision in data ordering and selection due to narrower difficulty ranges. Using 1eeen..50..bbInnttrue
we fine-tuned separately on GSM8K, MATH, and AMC training sets and evaluated on their test sets. Results in Fig. iidicate that GAIN-RL(GRPO) consistently yielded higher performance and efficiency. Specifically, on GSM8K, GAIN-RL(GRPO) achieved a 3.33x x training speedup and 4. 4.72% final accuracy improvement. On the more challenging MATH and AMC 23 datasets, In contrast, ADARFT(GRPO) provided less improvement due to its fixed difficulty scoring, which may not align precisely with actual peree--ereeived difficulty. In contrast, by leveraging model- informed angle signals, GAIN-RL(GRPO) can predict learning priorities more accurately.
also achieves 2.5x and 2 speedups, respectively.
4.5 Ablation Studies
Module Ablation. To better understand the contribution of diffcrent components in GAIN-RL, we conduct ablation studies, including (1) Data ordering + Accuracy-based -bsed probability updates, (2) Data ordering + Angle-based probability updates, and (3) Accuracy-only (no ordering, discarding fully correct data to reduce training costs). As shown in Fig. 10 (left), all ablation variants exhibit degraded performance. The Accuracy-onfy group exhibits a marked performance drop, initially improving but later declining due to forgetting from prematurely discarding data. This underscores that data ordering and sampling constitute EAI-ORL(GRPO'' primary performance advantages. Moreover, while both Data ordering + Accuracy-based probability updates and Data ordering + Angle-based probability updates show steady improvements over vanilla GRPO, they do not match the performance of GAIN-RL(GRPO), as each only captures a single aspect of gradient.
Small-Batch Scalability Test To validate the scalability of GAIN-RL under reduced training batch sizes, evaluated the model's performance with varying batch sizes. Specifically, we conducted experiments using the arren..5-0.5b-insttee model on the Math dataset with training batch sizes of 512, 768, and 1024, respectively. The experimental outcomes, depicted in Fig. T0 (right), illustrate consistent and stable performance improvement across iterations for all batch sizes. Remarkably, even with the batch size reduced to half of the original size (n=512), which significantly reduces computational time and memory usage per iteration, GAIN-RL maintained performance comparable vanilla GRPO. These results demonstrate that GAIN-RL effectively scales with smaller batch sizes, offering the flexibility to accelerate training by lowering batch size, with only minor performance degradation, especially beneficial under limited computational or memory resources.
5 Conclusion
We propose GAIN-RL, a novel Reinforcement Learning Fine-tuning framework that leverages the angle concentration signals dynamically allocate training data at each iteration. GAIN-RL leverage the model's 's intrinsic angle concentration signal to dynamically selects training data in each iteration, ensuring consistently impactful gradient updates and thus significantly enhancing overall training efficiency. Furthermore, our empirical results further show that GAIN-RL (GRPO) achieves over a 2.5x acceleration in training efficiency across diverse mathematical and coding tasks and varying model scales. Overall, GAIN-RI. introduces novel paradigm for training-efffcieney RFT, highlighting how model-centric data- pata-procsssing approaches can remedy the sub-optimality of current RFT methods and further elevate their effectiveness.
10

---

Acknowledgements
Qinsi Wang. Jianyi Zhang and Yiran Chen disclose the support from NSF 2112562 and ARO W911NF-23-2-0224 We thank area chair and reviewers for their valuable comments.
References
[1] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.1294: 2025.
[2] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishks Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1rXXv:22422166720 2024.
[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 17Xi:22110.4466. 2021.
[4] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 033444 2021.
[5] Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking. arXiv preprint arXiv:2502.12887 2025.
[6] Naman Jain, King Han, Alex Gu, Wen-Ding MennDing Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando
Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. urXiv preprint arXiv:2403.000:
2024.
[7] Qinsi Wang and Sihai Zhang. Dgl: Device generic latency model for neural architecture search on mobile devices. IEEE Transactions on Mobile Computing, 23(2): 1954-1977, 2023.
[8] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2xxv:2402.003300 2024.
[9] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
[10] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262. 2025.
[1]] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arxiv preprint arXiv:2503.1888 2025.
[12] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint 27X 2025.
[13] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.1930 2025.
[14] Taiwei Shi, Yiyang Wu, Linxin Song. Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint arXiv:2504.05520, 2025.
[15] Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation. arXiv preprint arXiv:2410.18311 2024.

---

[16] Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. arXiv preprint arXiv:2312.12141, 2023.
[17] Chaojun Xiao, Zhengyan Zhang, Chenyang Song, Dazhi Jiang, Feng Yao, Xu Han, Xiaozhi Wang, Shuo Wang. Yufei Huang, Guanyu Lin, et al. Configurable foundation models: Building Ilms from modular perspective. arXiv preprint urXiv:2409.02877 2024.
[18] Proceedings of the ACM Web Conference 2023, Austin, TX, USA, 2023, Association for Com- 1111111.115555555555
puting Machinery. URL https::/://d.... acm. wrg.:eoor/gooooooo htgg///l/prroceedn///o......1
[19]
2024 american invitational mathematics examination (aime). https://www www.maa...........om.. org/
math- wowpetitionnntommmmmmcc 2024. Accessed: 2025-05-14.
[20] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with Hlympiad-leved bilingual multimodal scientific problems. arXiv preprint orXiv:2402.14000 2024.
[21] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil. Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857 2022.
[22] Mikhail Mirzayanov. Open codeforces rating system. httpp://ccodfforesssccm/////// entry/20762 2015. Accessed: 2025-05-14.
[23] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[24] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol o1-preview with 1.5 b model by scaling rl. Notion Blog, 2025.
[25] Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, al. Deepcoder: A fully open-source 14b coder at 03-mini level, 2025.
[26] Qinsi Wang, Hancheng Ye, Ming- Yu Chung, Yudong Liu, Yueqian Lin, Martin Kuo, Mingyuan Ma, Jianyi Zhang, and Yiran Chen. Corematching: A co-adaptive sparse inference framework with token and neuron pruning for comprehensive acceleration of vision-language models. arXiv preprint a5Xi 19235, 2025.
[27] Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Dif- ferentiable svd for Ilm compression and some new perspectives. In The Thirteenth International Conference on Learning Representations.
[28] Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data-centric rlhf: Simple metrics for preference dataset comparison. In NeurlPS, 2024. URL https::/arxii... wwwpxivv.orr/aeo www./tgg////oooooooooooooo...ee/ babbs/2499 09603
[29] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36:49205-49233, 2023.
[30] Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. Advances in Neural Information Processing Systems, 36:25793-2588 2023.
[31] Dante Everaert and Christopher Potts. Gio: Gradient information optimization for training dataset selection. arXiv preprint arXiv:2306. 1670, 2023.
[32] Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXivv2401.12926, 2024.
12

---

[33] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: 1906. / 1829, 2019.
[34] Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming- Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, et al. Kvcomm: Online cross-context kv-cache communication for efficient lim-based multi-agent systems. arXiv preprint wrXiv:2510.12872
2025.
[35] Yueqian Lin, Qinsi Wang. Hancheng Ye, Yuzhe Fu, Hai Li, Yiran Chen, et al. Hippomm: sutpoocampal-nsspired multimodal memory for long audiovisual event understanding. arXiv preprint arXiv: 10739, 2025.
[36] Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, Yueqian Lin, Yiran Chen, Hai Helen Li, Kun Wan, and Wentian Zhao. Vision-zero: Scalable vlm self- self-improvement via strategic gamified self- self-play. arXiv preprint arXivv2509.25541 2025.
[37] Zishan Shao, Yixiao Wang, Qinsi Wang. Ting Jiang, Zhixu Du, Hancheng Ye, Danyang Zhuo, Yiran Chen, and Hai Li. Flashsvd: Memory-efficient inference with streaming for low-rank models. arXiv preprint 2rXiv22508.05000 2025.
[38] Yueqian Lin, Zhengmian Hu, Qinsi Wang, Yudong Liu, Hengfan Zhang, Jayakumar Subra- manian, Nikos Vlassis, Hai Helen Li, and Yiran Chen. Voice evaluation of reasoning ability: Diagnosing the modality-induced performance gap. arXiv preprint arXiv:2509. 26542, 2025.
[39] Hancheng Ye*, Jiakang Yuan*, Renqiu Xia, Xiangchao Yan, Tao Chen, Junchi Yan, Botian Shi, and Bo Zhang. Training-free adaptive diffusion with bounded difference approximation strategy. In The Thirty-Eightn Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[40] Hancheng Ye, Chong Yu, Peng Ye, Renqiu Xia, Yansong Tang, Jiwen Lu, Tao Chen, and Bo Zhang. Once for both: Single stage of importance and sparsity search for vision transformer compression. In JEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2024.
[41] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, (6:53728-53744 2023.
[42] Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint arXiv:2402.10500. 2024.
[43] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[44] Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEEE transactions pattern analysis and machine intelligence, 43(4 2019.
[45] Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 3 1,, 2018
[46] Yudong Liu, Jingwei Sun, Yueqian Lin, Jingyang Zhang, Ming Yin, Qinsi Wang, Jianyi Zhang, Hai Li, and Yiran Chen. Keyframe-oriented vision token pruning: Enhancing efficiency of large vision language models on long-form video processing. arXiv preprint 10744 2025.
[47] Qinsi Wang*, Jinghan Ke*, Zhi Liang, and Sihai Zhang. Mathnas: if blocks have a role in mathematical architecture design. Advances in Neural Information Processing Systems, 36: 2023.
[48] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
13

---

Organization In this Appendix, we provide in-depth descriptions of the materials that are not covered the main paper, and report additional experimental results. The document organized as follows:
Related Work
B
Theoretical Supplement
B.1
Theoretical Justification of Angle-Deepenent Effects in LLMs
B.2
Attention-Based Explanation Layer-wise Angle Concentration Patterns
3.3
Neuron-Based Explanation of Data-wise Angle Concentration Patterns
Visualization Results
C.1
Layer-wise Angie Concentration Patterns
Data-wise Angle Concentration Patterns
D
GAIN-RL Algorithm
Experimental Setup
Additional Experimental Results
Performance of Weighted Signals
Model Performance on Single Task
Discussion and Future Work
A Related Work
Reinforcement Fine-Tunie Tuning (RFT) 10 has demonstrated significant effectiveness in enhancing the reasoning capabilities of large language models [15 26 27). However, despite its promising potential, its low sample efficiency and high computational costs remain critical barriers to the broader adoption of RFT. Recent efforts aimed addressing these efficiency have primarily focused on algorithmic optimizations and data-centric strategies. Algorithmic optimizations, exemplified by methods such as GRPO 8 REINFORCE++ [10), and ReMax [ seek reduce computational complexity by streamlining underlying RL algorithms, Although these methods typically improve efficiency and stability, they often involve inherent trade-offss. For instance, GRPO estimates advantages through relative comparisons within output groups, thereby eliminating the need for value functions. While this approach reduces complexity and the reliance on critics, can introduce instability due to increased noise in advantage estimation, higher variance in updates, and greater sample requirements.
In parallel, data-centric strategies 28
29
30
31
32
33 34
35 36
39 40 have emerged as
provides the first quantitative audit of preference datasets, introducing metrics for scale, noise, and information density that expose quality bottlenecks before any policy update. Direct Preference Optimization (DPO) [4]] simplifies the entire loop by replacing on-policy RL with closed-form classification loss, making the choice of high-value preference data the primary driver of alignment quality. To further reduce annotation cost, Active Preference Optimization 42 casts RLHF as an active-learning bandit, adaptively querying only the prompts expected to maximize reward-mode improvement. The latest data-centric strategies can be categorized based on their approach to data manipulation: data selection and data sequencing. Data selection techniques involve filtering extensive datasets to retain only small subset of high-quality training data based on predefined metrics. Methods such as [1MO and s [ have demonstrated that carefully curated small supervised fine-tuning datasets can achieve robust performance using orders of magnitude less data. On the other hand, data sequencing strategies{43) enhance model learning speed by rearranging the order of training data within existing datasets. Approaches like ADARFT 4 have shown that dynamically selecting data for each iteration can effectively accelerate the training process.
promising alternatives for efficient fine-tuning. For example, 28
Nevertheless, existing data-centric strategies have generally failed account for the unique char- acteristics of different models, applying uniform data handling procedures across diverse model architectures. Such uniformity can lead to suboptimal outcomes because models differ significantly in their sensitivity and response to the same datasets. To overcome this challenge, this paper aims to identify intrinsic signals within models that can reflect their perceptual capabilities toward data, thereby enabling tailored data strategies without incurring substantial additional costs.
14

---

B Theoretical Supplement
In this section, we provide the theoretical explanation supporting the main text in Section 2. Specifi- cally, we elaborate the Angle-Dependee Section 2.1 of the main tex(), the Attention-Bann Basdd Explanation of Layer-wise Angle Concentration Patterns (corresponding Section 2.2 of the main text), and the Neuron-Basse Explanation of Data-wise Angle Concentration Patterns (corresponding to Section 2.3 of the main text).
Effects of Attention and Activation (comrespondigg to
B.1 Theoretical Justification of Angle-Dependent Effects in LLMs
In this section, we demonstrate through derivation of the LLM computational process that the nonlin- ear operations in AL.Ms-namee attention and activation computations- are inherently dependent on the angles between the input hidden states.
To support this claim, we first introduce two empirical assumptions based on observation:
Observation 1. W, and W& are nearly approximately orthogonal to each other, i.c., W, W R 01. O is constant. (Wo, W.,, Wa) are upproximately orthogonal matrices, i.e., W W R AI. X constant.
Observation 2. For activation function output vectors A; and Am, cos( (os(L(A,,,) AM)) is proportional to the number of intersections of activated neurons, ie., [Tx;) O [(TM)
Tay
L2177
LE
Figure 11: Visualization of WD W D.T at different layers in 100.2..--..B-Iniii
LAye
1np117
Lap
Law
Figure 12: Visualization of Wo&W T.T at different layers in Swen2.5-00.5. --IStrIInttr
15

---

Liven
Lecr3
Lewer 10
Lww-33
Lewe
Loye33
Lewer14
Loyer33
Lower
11
Lowr
Lip-11
Lower
Lope23
Iowe13
Lope223
Figure 13. Visualization of Wy T. at different layers in Qwen2.5-0.5B-Imss -nItructt
We provide empirical validation to support both of these assumptions.
For Observation 1, we show Wo @Ww T. Wy@Wy.T, T. Wn@WD.T of all different layers of Qwen2.5-0.5BB Qwven2.5-0.B--Imstuucnn Fig. I and 13 can be seen that different layers have this orthogonal relationship. In fact, the orthogonal relationship of matrices in neural networks has been studied since a long time ago. In particular, [44] proposed new regularization method that encourages the weight matrix of the neural network to maintain orthogonality during training by introducing self-orthogonalityy module. This method helps to improve the training stability and generalization
ability of the model. [45 46 47 explores adding orthogonal regularization to weights during training to improve training stability. The author proposed an orthogonal regularization method for weights, aiming solve the gradient vanishing and explosion problems encountered by deep convolutional neural networks during training. It can be seen that modules with orthogonality are found in various different models improve the training stability and performance of the model. To the best of knowledge, we are the first work to intuitively show this orthogonal performance in LLM, which can be more fully explored in subsequent research.
For Observation 2. this observation is illustrated in Fig. 14. and TM activate more of the same neurons, A, and AM will have more positive values in common positions, making cos(A, Aas) larger.
An intuitive understanding is that if x;
Lover
Layer
Layer
CO-Nec Narrn NAY
2/B HiLun Rum
1P0 Wease Mun
Corac Sauross fure
Coad Waron fanh
1800
ARO ReamS
Au Nive Nev
253 Ne eve Ni
aB0 J88
25g Movwni
Char wearteeeee
100
205 feom Say
oe Nr
128 120 I
Orai Morrs fae
Dial Swess
Casch
1a0g
900 3ia
Figure 14: Visualization of cos(2(A,
AM)) and co-act neurons number at different layers in
Dween.5500.B-Innstuee
Theoretical Insight 1. Within an attention block, the degree of interaction between two tokens is governed by the relative angle of their input hidden states.
16

---

Justification. For a single token, suppose its input to the Attention block is Consider a sequence of tokens [y,, ................................ MMM for token ym its computation in the Attention layer can be expressed as:
Dm = LayerNorm(ym)) Vn = Im WV:
Qim Softmaa ((Ds W₂) (8m Wx)²), )) i< m
(11)
Om 01m V1 a.2m V/ +....
+ ammmm Vm=
where aim is the attention score between the i-th and the m-th token. Om is the output vector of the m--th token. To examine the influence of the i-th token ys on the final output Om of the m-th token, we can consider the following projection value:
||ooio... (aim vi) = I|acm Vill cos (<(V, Om)).
(12)
where [Poooo (asm Vi) is the projection value of d'im Va2 on O, And cos (<(V,, 0m)) is the
cosine value of the angle between the V; and Om vectors. From Eq. Om sum of vectors in different directions. Since the self-attention score Omm is typically much higher than aimm for other
tokens, we can simplify the projection by assuming that On, is primarily determined by am Vm i.e.,
Projoo,. (a Vi)Il ze ||asm Vill cos(<(V,, Vm)),
(13)
where dism Softmaz (0: Wo) (im Wo) T /Va). Since the Softmax function monotonic, that is. Soflmar(z)) x We can have oam (ū: W₂, (fm Ws)". Therefore,
| Vill cos( (V,, Vm))
x (Ur Wa) (Im Ws) [[V&l] cos( (V;, Vm)) (V,, (V,, Vll Il (b. Im W&
(14)
(W, W )9Th (ú. (W, wT (o)))/)))///
Based on the Observation 1, WT x OI, W. WT ~ AI. Therefore, combining Eq. 13 and Eq. 14
||Projo.. (aam Vi) x 0x Dm) (ūs, (mm)/|/Vmml
(15)
Given that ý is the result of y after LayerNorm, ý and y share the same direction, and || 9 ||= holds that (Vi, Vm)) (os(<(1n, Vm))). We can have
|Projo... (04m Vi)ll < cos( cos(/(y); Mm))),
(16)
Eq. 19 shows that the angle between different tokens directly affects their mutual interaction in attention layer. The closer the angles of two tokens, the greater their mutual influence.
We also demonstrate that the computation of activations influenced by the angular relationships between hidden states. This justification is presented as Theoretical Insight 4 in Section B.3
B.2 Attention-Based Explanation of Layer-wise Angle Concentration Patterns
In this section, present an attention- based explanation of Layer-wise angular concentration patterns. Specifically, we theoretically show that: (1) the degree of angular concentration between two hidden states influences the magnitude of their attention scores, and (2) the presence of sink attention structure encourages angular concentration among tokens.
Theoretical Insight 2. The smaller the angle between the input hidden states of two tokens to the attention block, the higher their corresponding attention score.
Justification. For two tokens i and j. let their inputs to an attention block be denoted as x; and ay- respectively. Then, their attention score aij can be expressed as:
Ú: = LayerNorm(yss, úū = LayerNorm(y,),
(17)
aij = Softmar ((), W₂) (ūs Wx)" /Vd),
Based on the Observation 1, .WW.W OT, Therefore,
ag = Softmaz (0., W₂) (ū) W&)" /V)) = Softmar (onnarr((WWW)) (b. 8y Ww//Va/
(18)
Softmar (W,WW 1o /d)) Softmax (0.00)//1
17

---

Furthermore, since LayerNorm preserves the direction of vectors by normalizing only their magnitude, the inner product between normalized outputs satisfies (j.. 0j) cos(<(3: Vj)),
akjj So f ftmar cos( ((v, //3/Vd) x cos (1,, mj))
(19)
Equation indicates that the more aligned the hidden states of two input tokens are (i.e., the smaller the angle between them), the higher their attention score.
Therefore, conjunction with the layer-wise angle concentration pattern discussed in the main text, observe that inter-segment angular concentration reflects the model's degree of attention to internal components the problem, while intra-segment angular concentration captures the level of attention between the question and the system prompt-Serving an indicator of the model's cnaructtoonfollowing capability.
Theoretical Insight3. Presence of sink attention promotes angular concentration among hidden states.
peed
sco Eca
cco
sccar
2313
kas
can
Faoe wen
ccae
ecca
kan
nas
aous
sack 2
nace
Figure 15: Visualization of attention score at different layers in 1eem2..5-0.5-IIntttre
Justification. To understand why token angles exhibitl.aeer-wit angular concentration patterns, we start our analysis from the phenomenon of sink attention, which similarly shows segment-wise characteristics yet remains insufficiently understood. In LLMs, attention scores exhibit segment-wise tendencies, and, apart from self-attention, attention scores typically peak at the first token of each segment. This phenomenon is termed sink attention, as illustrated in Figure
Given an input representation for an attention block in an LLM as E Rmxd where the i-th token is denoted yi, the attention mechanism output is defined O aV, where represents the attention scores, and = LayerNorm y) W. denotes the value vectors. Suppose the attention scores within segment "sink" to the first token vector y, then the outputs for the sink token y; and another token ys within the same segment can be approximated as:
O, V.V. V. O R 0 O.RA VE.
where Osk denotes the attention score between tokens and k. This approximation arises because the sink token's self-attention score significantly surpasses its attention scores to other tokens, while other tokens primarily attend to themselves and the sink token.
Substituting these approximations, the angle between outputs O, and O₃ can be expressed as:
Olkk Vil + Ouk Vall cos( (V,, Vx))
cos (4(00,
V V V
V 20g V V cos( (os( yk))
Furthermore, since W,, is approximately an orthonormal matrix (detailed proof the appendix) and thus preserves angles, we have W, W 3I, where 3 is constant. Combined with the fact that
18

---

LayerNorm scales only magnitudes without altering angles, we get:
<os( V+)) = cos( (yr, 3k)).
Substituting into the earlier expression, we derive:
Olsk + Ork cos ( (1rs yk))
= B
+ a 20. cos( (yy,, yx))
Squaring both sides and simplifying, we arrive at the condition:
cos( (((,,, Ok)) cos(<(y), y:)) if 3(cos (yr y)))) < and B
Since weight parameters in LLMS are typically constrained to values less than 1, both / < and ((cos(/(1); yk)) yx)))² < generally hold. Thus, Equation (7) demonstrates that sink attention inherently promotes angle concentration within segments. detailed derivation is provided in the appendix.
In Figure 15 we present the distribution of attention scores across different layers. In intermediate layers, sink tokens operate primarily within segments to enhance intra-segment angle concentration. In later layers, sink tokens across segments begin to interact, promoting inter-segment concentration. This indicates that, due to the influence of sink tokens, token angles are increasingly concentrated through the forward pass. The final layer's 's angle concentration particularly important as reflects the culmination of this process and directly determines the model's output.
B.3 Neuron-Basse Explanation of Data-wise Angle Concentration Patterns
In Section 2.3 of the main text, we demonstrate that the greater the number of tokens activating the same neuron, the more gradient components that neuron receive. In this section, we further show that number of commonly activated neurons between tokens has direct effect on the angle between their output hidden states at the current layer.
Theoretical Insight 4. Within the FFN block, the extent of overlap activated neurons between two tokens directly affects the angular relationship between their output hidden states.
Justification. To analyze how the activation layer affects cos(<(y;, yM)), first decompose its computation formula. Suppose the activation output of the i-th token is A,, and y; = A, W. Based on Observation I, Wa scalar multiple of a unitary self-orthogonal matrix, applying the same rotation to any input while preserving the inner product and angle between any two input vectors. Thus, can have:
cos(L(y) BN)) = (A,Wa, Aas Wa)/ W (.33)))433/)))) [Illallllllll 7/4,, Asi) /(lla (IlBosll),)
(20)
where 7) constant based on Observation 1. Furthermore, since Wa is orthogonal matrix,
|| 1
|| a. (A.W.)(AAW.))"
(21)
A. AT" "AAAA A.13.
which means ||
/ Substituting this into Eq. 20 we can have
cos cos(<(yi, MM)) ,,,, Am) / /.yllA/ll|Aaa
(22)
= cos( ((ss, An))
This shows that the orthogonal matrix Wa does change the angles between the input vectors. Furthermore, based on Observation 2, cos ((A, AM)) x [((x)) nn [(*M)). we can get
cos(<(u) vM)) [(x)) O [(*M]].
(23)
which is consistent with Insight 2.
Eq. 23 shows that activation layers adjust token angles by controlling the intersections of their activated neurons. More shared activated neurons lead to smaller angles and greater mutual influence.
19

---

C Visualization Results
In the main text, we presented layer-wise wise, spoch-wise. and data-wise patterns of angular concentration. In this section, we provide more comprehensive visualizations further support our conclusions.
C.1 Layer-wise Angle Concentration Patterns
In Fig. [6 17 and 18 present the layer-wise angular concentration patterns across all layers of the Qwen2.5-0.55 InIstructt model for tasks of easy, medium, and high difficulty, respectively. Notably, the model consistently demonstrates first intra-segment angle concentration and subsequently inter- segment angle muncentration-regardde of problem difficulty. This consistency suggests that the observed pattern generalizable property of the model's internal representation dynamics.
Ligar
Liger 2
Lagar
LaperS
Lecer
Lige7
Uoar 3
Liyar's
Layer 20
Lo 21
Laver 12
15
Laver 25
Love
Lpper
Laye
Figure 16: Visualization of Layer-wise Angle Concentration at different layers in 2.5.5.5- Instruct at easy sample (correct num = 10).
Figure 17: Visualization of Layer-wise Angle Concentration at different layers in Qwen2.5-0.55 5-5..BB
Instruct at medium difficulty sample (correct num 5).
20

---

Liyer
Liger
Layer:3
Figure 18: Visualization of Laser-wise Angle Concentration different layers in Qwen2.5-0.5B.
Instruct at difficult sample (correct num = 0).
C.2 Data-wise Angle Concentration Patterns
Fig. 19 provides a more fine-grained view of the data-wise angle-concentration patterns. Consistent with the conclusions in the main text, it reveals curriculum-like trend: the model learns samples exhibiting high angular concentration earlier, followed by samples with lower angular concentration.
Epoch 0
Epoch 25
Epoch 50
Number of SSapls
646
0.0
Castra Cintrr
Catra Cunter
Cutra Gater
Epoch 75
Epoch 100
Number of Sample
1.30
Avg. Correct
0.6
Cytra Cyter
Cytro Ginter
Figure 19: Visualization of Data-wise Angle Concentration at different iterations in Qwen2.5-0.5B.
Instruct at difficult sample. The training setting is consistent with Figure 3 in the main text.
21

---

D GAIN-RL Algorithm
Algorithm 1: GAIN RL (GRPO): Gradient-driven Angle-Informed Navigated RL Frame-
work
Input: Training data D = {d1, .2,.... dx}; model M; training steps T'; batch size n. accuracy sensitivity a; target accuracy 8; angle sensitivity 7, sampling variance Output: Trained model Mr
1 Step Reorder training data by angle signal
2 for i <- 1 to N do
3
Prefill d, with model
Extract angle signal (d, = C.M C (d,) + Cinter (d,)
5 Sort D by (-)-) in descending order to obtain D,
Step Train with dynamic probabilistie sampling
7 Initialize Gaussian distribution Po N(0, 22)
8 for t <- 1 to T do
Sample batch d( ~ Sample(D,; Pt.
for 1 to n do
Let model M, answer df
11
Record Cm, (d(") and accuracy ACCM, (d.")
12
Update M, with accuracy-basee loss Compute mean accuracy (cc E ACcM, (c)
Compute mean angle c(t) 1 Ei=1 Csa. (d
15
16
Update sampling-mes mean mean
n P++1 s4 + (anb(a(Ace!!) -3)) + 2
tanb(> (anb(7c(*)) c(t))
2
E
Experimental Setup
In this section, we describe our experimental setup in detail, covering the models, datasets, and hyperparameters used.
E.1 Model and Dataset
To comprehensively evaluate the effectiveness of GAIN-RL, we conduct experiments across multiple models and datasets. Specifically, we select models varying in size, including Qwen2.5-0.55 Instruct. Swwnn..-MMath-..B-BB. and ILaMA3. Instructt. We primarily focus on two tasks: Math and Code. To evaluate the training efficiency GAIN-RI. (Section 4.1 in the main text), we use the DeepScaleR (24 dataset for mathematical task training and DeepCoder (25) for coding tasks training, each integrating problems from diverse sources and covering a wide range of difficulty levels. For mathematical evaluations, we employed six benchmark datasets of varying difficulty: GSMSK [3) MATH [4, AMC 23 [18), AIME 24 [19]. OlympiadBench [20), and Minerva Math [48). For coding evaluations, we utilized three standard benchmark datasets: LivecodeBench (8/1/24-2/1/25) (6), Codeforces 22] and Humaneval+
23 For other experiments (Section 4.2-Section 4.5), we train model on the training dataset of single tasks including GSM8K, MATH and AMC 23 to facilitate more convenient comparisons.
E.2
Training Configuration
We trained the models using the GRPO algorithm. The training was performed on single node equipped with 8 A100 GPUs. Each model was trained for about 200 steps using the veRL library.
22

---

To evaluate the training efficiency on GRPO-RL. the main training configuration for Qwen2.5- Math-7B- BB-nnsrrcct shown below. For Qwen2.5-Math-l-----me we set max_ meax__esposs__. 1arp:oossseennthh00000000 to accommodate its shorter context window of 4096 to- kens, while keeping other parameters unchanged. For Qwen2.5-0.5H, training, we set max wtt.pppp__eengggaaaaeeeeeeeeeeee 1enggnhhhh1_ while keeping other param- eters unchanged. For Qwenn/Owenn.5--ddd-33BBBB max max_response reepoonselengthhtteeeee wew.enggttne.n...t hrapre/88.rraaaaaaaacceeeeeeeee sizech2s and ppo_minis beatch_sizere due to its higher single sample memory usage.
and L
and single task
we set max_
python3 -m tra..trraeermmain.pp algoritmn.ady..sttmattorrr. data. tra.iinn.filee=sttrrii..iiie data. stttavalllfllee==$$ttttttti. cafa..crainnbbtcchh_cii0011222
Instruct
pac.cpppe..aaacc.............a.r \
bat.h aetor.ocloot.eeeeeeee.eeepp..oep
pradher..cttt:c__aammmp001111111
wraberr.oogger//oooollleevvaav>
Baaelee__m_kkaaah_h_ \
fastrucct_uucttion_m)
trainer.n.ggpu_per.nooee3
trainer.mnodes=11
trainer.save_freque
trainer.
.rainer..testffreqs.
trainer.total trainer.tocal_gpchs=2200 $0
Additional Experimental Results
In this section, we present additional experiments to further validate the effectiveness of GAIN-RL.
Performance Weighted Signals
In the main text, we employed an unweighted angular signal of the form: (dr( = CM (d,) CM (de). Here, we explore weighted variant of the signal: Cas(d, = CM (d:) c C.d (d+),
where is constant weight.
23

---

As shown in Tab. 4 we investigate the final performance of the Qwen2.5-0.5B -nnstrucct model after 200 training iterations on the Math dataset, using different values of in the weighted signal. The results show that setting = 4.0 yields the best performance for this model, suggesting that carefully designed angle-based signals can further improve training effectiveness.
However, selecting the optimal weighting coefficient requires extensive empirical tuning. To ensure scalability and practical applicability, adopt the unweighted version of the signal in this work and leave signal optimization for future research. Our goal is to demonstrate that even without finely tuned weighting, GAIN-RL is still capable of accelerating both training and data efficiency- highlighttng the strong potential of model-signal siaiel--innl--aasee based RLHF strategies.
Table 4: Model performance iteration 200 under different weighting coefficients c.
0.25
0.5
1.0
2.0
4.0
8.0
Accuracy 36.20
37,20
37.40
38.00
38.40
38.20
F.2
Model Performance on Single Task
Fig. 9 in the main text illustrates the training dynamics of datasets. For a more detailed comparison, Tab. reports the final performance iteration 200 and the corresponding training speedup across different training sets and methods. Notably, on the GSM8K dataset, GAIN-RL outperforms the original GRPO by 4.72% in final accuracy and achieves 3.3x improvement in training speed.
on three single-task
These results demonstrate that GAIN-RL can effectively distinguish between samples of varying learnablliyy even in the single-task setting, highlighting its efficiency and general applicability.
Table 5: Fine-tuning performance on single tasks. Models are trained on the training sets and evaluated
on their validation sets. ADARFT is excluded on GSMSK due to missing difficulty coefficients.
Prepare
GSM8K
Math
AMC
Terr@ 200Acc
Speed Up
Ierr@
Speed Up
ACC@
Iter@
Speed
Metric
Time
200ltr
200lter
200Acc
2001ter
200Acc
Up
GRPO ADARFTIGKPO) GAIN-RL/CRPO)
48.43
200
1x
34.80
200
1x
9.64
200
1x
Difficuity
S1 day
35.80
150
1.33x
9.64
160
1.25x
Angle
min
53.15
3,33x
37.40
80
2.50x
12.04
100
2.00x
G Discussion and Future Work
In Section 3, we demonstrate that angles between token hidden states fundamentally mirror and influence both the information propagation during inference and the learning dynamics throughout model training. The proposed angle-based signals can, in fact, be generalized beyond RFT to enhance model-centric effectiveness in various other domains. For instance, during pre-training, monitoring angle signals could enable real-time evaluation of model's learning capacity across different domains, thus allowing adjustments to training data to improve stability and final performance. Furthermore, during inference, tracking changes in angle concentration between layers could provide insights into the model's comprehension of inputs and indicate whether additional test-timo adjustments are necessary to boost output accuracy. In future work, we plan to further investigate how this signal can be leveraged across multiple domains to achieve comprehensive, model-centric optimizations.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The motivations and contributions are well depicted and summarized in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not to penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
The theory assumptions and proofs are described in Section 2 and Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The selected models and benchmarks are clearly and fully presented in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
The code will be provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
ooiiionnnliyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ lttipp.cc/pulli/guuii.e gwtcc//ggge///eee CoodeSumiiiie wwweiissiinnpeeooe ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details are carefully presented in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the accuracy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information of the employed compute resources elaborated in the implementation details of Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: peattp:://neuipptt
Answer: [Yes]
Justification:
We make sure that the research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We discuss in the Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all papers that we used for experiments.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
30

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This paper describe the usage of LLMs in code and math tasks, mentioned in Section 1.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
31


Reviews:
[{"review_id": "TrXQ0HnuaC", "reviewer": "Reviewer_ERbf", "summary": "This paper introduces GAIN-RL, a novel Reinforcement Learning Fine-tuning (RFT) framework designed to address the sample inefficiency and high computational costs in training Large Language Models. The core insight is the identification of \"angle concentration\" – the cosine similarity between token hidden states – as a model-inherent signal that effectively reflects an LLM's learning capacity. GAIN-RL leverages this signal by reordering training data based on initial angle concentration and dynamically sampling data during training using a Gaussian distribution, with the mean updated by real-time accuracy and angle signals. Experimental results demonstrate that GAIN-RL significantly accelerates training and achieves better performance with less data compared to vanilla GRPO.", "questions": "see in previous comment", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1.\tThe paper addresses a critical problem of sample inefficiency and high computational costs in LLM RFT, proposing a novel idea of using the model's inherent \"angle concentration\" signal for data manipulation.\n\n2.\tThe paper provides strong support for angle concentration as a learning signal through both theoretical reformulation of gradients and empirical observations of various angle concentration patterns.\n\n3.\tExperimental results show significant performance and efficiency gains, with GAIN-RL (GRPO) achieving over 2.5x training acceleration and superior performance with half the data.\n\n4.\tThe framework demonstrates good generalizability, presented as a plug-and-play solution compatible with different models and RL algorithms, with minimal preprocessing cost.\n\n5.\tClear ablation studies effectively demonstrate the contribution of each component, and small-batch scalability tests further validate the method's robustness.\nWeaknesses\n1.\tThe claim of \"no additional cost\" for computing accuracy and angle concentration for probability updates requires more detailed justification across various training setups to ensure consistent fairness.\n\n2.\tThe paper's strong results with \"High Angular Concentration-biased Sampling\" (using half the data) need more nuanced discussion regarding the broader implications of potentially excluding certain data points.\n\n3.\tThe comparison with existing curriculum learning methods should be broader and more deeply analyze GAIN-RL's advantages over a wider range of strategies.\n\n4.\tThe paper contains several minor typos and inconsistent formatting details that, while not severely impacting understanding, should be addressed for improved professionalism.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the very positive rating and insightful feedback. We are encouraged to hear that the reviewer finds our discussion of leveraging angle concentration as a model-inherent learning signal to be theoretically sound and empirically compelling, and considers our proposed approaches to be both original and highly significant for improving the efficiency of RL fine-tuning. We hope the following discussions can address the reviewer's remaining questions.\n\n\n\n> ***Weakness1. The claim of \"no additional cost\" for computing accuracy and angle concentration for probability updates requires more detailed justification across various training setups to ensure consistent fairness.***\n\n**Responce**: We sincerely thank the reviewer for the thoughtful comment and fully agree that the claim of “no additional cost” should be carefully justified across different training setups. We appreciate the opportunity to clarify this point.\n\nIn our implementation, both **accuracy** and **angle concentration** are collected without introducing any additional inference steps. Specifically, accuracy is computed as part of the loss evaluation for each training sample, which is already standard in most RL fine-tuning frameworks. Angle concentration, on the other hand, is derived from the logits during the forward pass, which the model performs anyway. Therefore, **both signals are gathered naturally during training without any extra computational overhead.**\n\n**Importantly, this process is independent of model size, architecture, or hyperparameter settings, and applies consistently across different training setup.** We will revise the submission to make this explanation clearer and more explicit.\n\nWe sincerely appreciate the reviewer’s careful reading and valuable feedback, which helps us improve the clarity and fairness of our claims.\n\n\n\n> ***Weakness2. The paper's strong results with \"High Angular Concentration-biased Sampling\" (using half the data) need more nuanced discussion regarding the broader implications of potentially excluding certain data points.***\n\n**Responce**: We sincerely thank the reviewer for this thoughtful and insightful comment. As the reviewer rightly pointed out, Figure 8 deserves a more nuanced explanation. We fully agree and are happy to provide a deeper analysis of this promising result.\n\nFigure 8 illustrates the effect of our proposed High Angular Concentration-biased Sampling method, which selectively excludes samples with low angular concentration. The results suggest that **removing such samples can significantly improve training efficiency while preserving gradient effectiveness and stability.** This observation is supported by several layers of evidence:\n\n1. **Theoretical justification via Equation (4):** Low angular concentration samples reduce the mean and increase the variance of training gradients. As shown in Equation (4), angular concentration among hidden states directly influences the Frobenius norm of the gradient through the sum of cos θ terms. A higher angular concentration yields stronger, more consistent gradient updates, accelerating the learning process and improving efficiency.\n2. **Empirical support from the neuron activation patterns (Figure 6):** As shown in Figure 6, high-angular-concentration samples tend to activate a focused cluster of core neurons, enabling effective gradient accumulation and rapid formation of high-performing neuron clusters. In contrast, low-concentration samples activate more scattered neurons, leading to dispersed gradient directions. Although more samples tend to activate core neurons as training progresses, a subset of low-concentration samples continues to produce noisy and inconsistent activations, thereby reducing overall training stability and efficiency.\n3. **Alignment with prior research findings:** Several prior works [1,2,3] have shown that not all training samples contribute positively to learning, and removing \"negative gain\" samples can lead to faster and more stable optimization. Our method provides a practical and computationally efficient way to filter out such low-value data points based on angular concentration.\n\nWe will expand our discussion of this result in the revised version and plan to explore this direction further in future work.\n\nOnce again, we sincerely appreciate the reviewer’s constructive feedback, which has helped us improve the depth and completeness of our paper.\n\n\n\n> ***Weakness3. The comparison with existing curriculum learning methods should be broader and more deeply analyze GAIN-RL's advantages over a wider range of strategies.***\n\n**Responce**:  We sincerely thank the reviewer for this valuable suggestion. We fully agree that a broader and more in-depth comparison with existing curriculum learning and data manipulation strategies would strengthen our paper. Below, we elaborate on the key advantages of GAIN-RL over a wider range of prior approaches:\n\n1. **Model-aware signal vs. static data assumptions**: A core distinction of GAIN-RL lies in its use of the *model’s own perception* of data difficulty, rather than relying solely on static properties of the data distribution. Prior approaches—such as curriculum learning and expert data selection—typically define difficulty based on external metrics or predefined heuristics, which do not account for how the target model perceives different examples. As shown in Figure 2 of the main paper, we observe that different models often perceive the same data samples with very different levels of difficulty. This highlights a limitation of model-agnostic methods. In contrast, GAIN-RL leverages model-internal feedback to dynamically guide data selection, resulting in more efficient and tailored training.\n2. **High generality and broad applicability**: GAIN-RL is model- and dataset-agnostic, making it easy to integrate into a wide range of training pipelines. Many existing methods are only effective under specific conditions—for instance, expert data filtering (e.g., s1 and limo) requires large, high-quality, and diverse datasets (e.g., s1 collects 59k samples across 16 datasets), while curriculum learning relies on datasets with clear and measurable difficulty variations. These constraints limit their scalability to everyday training scenarios. In contrast, GAIN-RL requires no special assumptions about data quality or model type: it simply utilizes a lightweight pre-filling stage to collect difficulty signals from the model itself, allowing it to be broadly deployed across diverse training settings.\n3. **Minimal computational and human cost**: GAIN-RL can be implemented with only minute-level GPU overhead, whereas many prior methods require substantial resources and human or agent-based evaluation. For example, s1 uses Claude 3.5 to score samples by difficulty and diversity, and curriculum learning strategies often depend on human performance statistics from competitions to assess question difficulty. These approaches are computationally intensive and prohibitively costly for most researchers. GAIN-RL, on the other hand, relies solely on the model's own forward pass, making it far more accessible and scalable for real-world use.\n\nWhile we included several representative baselines in our current experiments, their scope was limited due to the aforementioned challenges of reproducibility and cost. For instance, curriculum-based difficulty metrics are mostly defined on math datasets and do not generalize well to domains like code. Similarly, expert filtering approaches depend on diverse and curated datasets, along with expensive agent assistance, which makes them difficult to reimplement in general-purpose training scenarios.\n\nWe are actively collecting additional baselines and will incorporate broader experimental comparisons and deeper discussions of GAIN-RL's advantages in the revised version. We once again thank the reviewer for this constructive feedback, which will help us significantly improve the completeness and rigor of our work.\n\n\n\n> ***Weakness4. The paper contains several minor typos and inconsistent formatting details that, while not severely impacting understanding, should be addressed for improved professionalism.***\n\n**Responce**:  We sincerely thank the reviewer for carefully reading our paper and pointing out the formatting and typographical issues. We will thoroughly proofread the manuscript and correct these minor errors in the revised version to improve clarity and overall professionalism.\n\n\n\n**References:**\n\n *[1] Schaul T, Quan J, Antonoglou I, et al. Prioritized experience replay[J]. arXiv preprint arXiv:1511.05952, 2015.*\n\n *[2] Pleiss L S, Sutter T, Schiffer M. Reliability-Adjusted Prioritized Experience Replay[J]. arXiv preprint arXiv:2506.18482, 2025.*\n\n *[3] Isele D, Cosgun A. Selective experience replay for lifelong learning[C]//Proceedings of the AAAI conference on artificial intelligence. 2018, 32(1).*"}, {"review_id": "HFcppG6iGm", "reviewer": "Reviewer_ecE4", "summary": "The paper proposes a novel RFT framework for large language models, called GAIN-RL, which enhances training efficiency by leveraging a model-intrinsic signal called *angle concentration*. This metric quantifies the model’s potential to learn from a given prompt. It is both model-based and computationally efficient, requiring no reliance on heuristic measures of question difficulty. Empirical results demonstrate that angle concentration is a measurable and effective signal for guiding reinforcement learning.", "questions": "1. Why is Gaussian probability applied? Is it because Gaussian is very light tailed such that only certain $d_i^s$s will be chosen with high probability? More explanation would be helpful.\n2. What’s $\\sigma_t$ in (8)?\n3. Is this a typo?: the left equation in Eq 8 only includes $i$ instead of $d_i^s$.", "limitations": "The observations regarding angle concentration are primarily based on the 0.5B model. It would strengthen the claims and improve the generalizability of the findings if similar analyses were also conducted on larger models.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper addresses an important and interesting question, i.e., how to improve sample efficiency and accelerate training in reinforcement learning for large language models. It is widely recognized that selecting appropriate prompts at different stages of training plays a critical role in enhancing both the speed and robustness of RL. The authors introduce a novel, model-based metric that avoids expensive data preprocessing and can effectively guide a new training strategy, GAIN-RL. This strategy is shown to outperform vanilla RL methods based on uniform sampling. Sufficient results confirm the effectiveness of GAIN-RL. \n\nWhile the paper is well-written and clearly structured, I still have some suggestions and questions about the motivation part, i.e. section 2. \n1. The role of the gradient norm could be clarified. While I understand that small gradients might suggest proximity to a solution, it is less obvious why large gradients necessarily imply the opposite. As a first-time reader, I found it unclear why the Frobenius norm of the gradient should be the primary focus.\n2. Equation (4) does not clearly isolate the role of the angle in $l_2$ space. It is not fully explained why the norm of the hidden states can be disregarded. Are these norms assumed to be fixed? If not, then both the angle and the magnitudes contribute to the Frobenius norm. Moreover, it seems to me that the angle in equation 4 comes from the Frobenius norm. If another norm were used, then the angle in the $l_2$ norm space doesn’t always show explicitly in the equation.\n\nI suggest the authors revise this section to provide more intuitive explanations and to make the motivation more accessible to a broader audience.", "was_revised": true, "final_justification": "The authors fully addressed my concerns about paper clarity. This is a good paper.", "rebuttal": "We sincerely thank the reviewer for the kind and encouraging feedback. We are honored that the reviewer considers the problem we address to be important and recognizes the effectiveness of our proposed GAIN-RL strategy. We greatly appreciate the positive evaluation of our contributions and the time and effort dedicated to reviewing our work. We hope the following responses provide further clarification and address any remaining concerns.\n\n\n\n> ***Weakness1. The role of the gradient norm could be clarified.***\n\n**Responce**:  We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the opportunity to further clarify the motivation behind focusing on the gradient Frobenius norm in our work.\n\n**1. Why gradient norm reflects learnability:**\n\nThe gradient norm is a fundamental indicator of how far a model is from satisfying the first-order optimality condition. Specifically, if $\\nabla f= 0$, then $x$ is a critical point. Conversely, if $\\|\\|\\nabla f\\|\\|> 0$, it directly quantifies the gap from optimality. In smooth and strongly convex settings, the gradient norm also provides bounds on the distance to the optimum [1,2]. Hence, a large gradient norm indicates that the model still has substantial room to learn, rather than proximity to convergence.\n\nThis perspective is widely adopted in prior work across various learning paradigms. For instance, in multi-task learning, GradNorm [3] adjusts task weights based on gradient magnitudes; in training efficiency, Selective Backprop [4] prioritizes samples with large gradients; and in reinforcement learning, gradient norms serve as reward signals for curriculum generation [5]. These works demonstrate the empirical utility of gradient norms as a measure of learnability.\n\n**2. Why we use the Frobenius norm:**\n\nWe adopt the Frobenius norm because it is theoretically principled and practically efficient. It treats the gradient tensor as a Euclidean vector and is invariant to orthogonal transformations—an important property for deep networks with parameter symmetries. It also aligns naturally with the geometry of first-order optimizers like SGD and Adam. Unlike spectral norms, it has linear-time complexity and avoids the non-differentiability of L1 norms.\n\nIn practice, the Frobenius norm is widely used as the default global gradient clipping metric in frameworks like PyTorch and TensorFlow. It is also used in estimating the gradient noise scale for adaptive batch sizing in large-scale transformer training. These conventions further validate its role as a reliable, well-established metric for capturing how much a model has yet to learn.\n\nWe will improve the explanation in the revised version with clearer intuition and motivation to make it more accessible to first-time readers. \n\n\n\n> ***Weakness2. Equation (4) does not clearly isolate the role of the angle in l2 space.***\n\n**Response**: We thank the reviewer for the insightful question about Equation (4). Below we clarify why the angle term is emphasized and why hidden state norms can be treated as approximately constant.\n\n**1. Why the norms $\\|\\|x_i\\|\\|$ and $\\|\\|x_j\\|\\|$ can be disregarded in Equation (4):**\n\nIn Transformer models like Qwen and LLaMA, hidden vector norms are stabilized by LayerNorm, which normalizes vectors and applies learnable scaling ($\\gamma$) and bias ($\\beta$). The resulting norm, $|y - \\beta|_2 = \\sqrt{\\sum_i \\gamma_i^2}$, is nearly constant across samples and evolves slowly during training. With typical hidden sizes ($d \\approx 10^3$), the relative variation in norms is under 3%, allowing us to approximate them as constant scaling factors when analyzing gradient norms. This simplification is also supported by prior work [6].\n\n**2. Why we analyze angles in ℓ₂ space (Frobenius norm):**\n\nAngles are meaningful only in inner-product spaces, which the Frobenius norm supports. Non-Euclidean norms like ℓ₁ do not preserve inner products and thus lack a coherent notion of angle. Moreover, analyzing angles under the ℓ₂ norm is both standard and appropriate. As discussed in our response to Weakness 1, the Frobenius norm is widely used in practice, aligns with standard optimization assumptions, and corresponds directly to generalization bounds tied to ℓ₂ regularization. Thus, focusing on angle dynamics within the ℓ₂ geometry provides both interpretability and practical relevance.\n\nWe will clarify these points in the revised version and we thank the reviewer for the insightful feedback.\n\n\n\n> ***Question1. Why is Gaussian probability applied?***\n\n**Response**: We sincerely thank the reviewer for the thoughtful question. Our choice of Gaussian is motivated by two key considerations:\n\n1. **Preventing data forgetting by maintaining non-zero sampling probability**:\n    As the reviewer suggested, Gaussian sampling—with its light tails—ensures that all data points retain a non-zero probability of being sampled. This is crucial for preventing knowledge forgetting during training. In our implementation, we set the standard deviation $\\sigma = N/3$ (where $N$ is the total number of training samples). Even in the extreme case where the sampling center μₜ reaches the rightmost end of the data (i.e., $\\mu_t = N$), the earliest 10% of the data (with the highest angular concentration) still retains a 7% probability of being sampled. This ensures that previously seen samples remain accessible throughout training.\n2. **Smooth and controllable transition from high- to low-angle data**:\n   Gaussian sampling enables a dynamic, symmetric distribution centered at $\\mu_t$, assigning the highest probability to samples around $d_{\\mu_t}$ and gradually decreasing for samples further away. This property allows us to focus the training data distribution around a shifting center, ensuring that at each time step, the model is exposed to a narrow band of samples with relatively similar angular concentration. As $\\mu_t$ increases over time, the sampling smoothly transitions from high-concentration to low-concentration samples, which aligns with our intended training curriculum.\n\nOur experiments show that Gaussian sampling consistently accelerates training across models of various sizes, demonstrating both robustness and generalizability. We will further elaborate it in the revised version.  \n\n\n\n> ***Question2.  What is $\\sigma_t$ in eqa. 8?***\n\n**Response**:  We thank the reviewer for the careful reading and for pointing out this oversight. We apologize for the confusion caused by the missing explanation of $\\sigma_t$.\n\nIn Equation (8), the sampling probability in GAIN-RL follows a Gaussian distribution with $\\mu_t$ and $\\sigma_t$ as the mean and standard deviation. While $\\mu_t$ increases over time as defined in Equation (10), guiding the sampling center, we fix $\\sigma_t = N/3$, where N is the total number of training samples.\n\nThis choice ensures that even when $\\mu_t = N$, early samples (top 10% angular concentration) still have a notable sampling probability (~7%), helping mitigate knowledge forgetting. We also performed ablation studies on different σ values (see response to Reviewer LV4y, Q2). We will clarify this in the revised version. Thank you again for your helpful feedback.\n\n \n\n> ***Question3.  Is this a typo?: the left equation in Eq 8 only includes $i$ instead of $d_i^s$.***\n\n**Response**: We sincerely thank the reviewer for the careful reading and for raising this question. This is not a typo, and we are happy to provide a more detailed explanation of Equation (8).\n\nGAIN-RL training involves three stages: data reordering, probabilistic sampling, and probability updating. In the reordering stage, all training samples are sorted based on their angular concentration. That is, in the reordered dataset $D_s =d_1^s, d_1^s,..., d_n^s $, a smaller index $i$ indicates higher angular concentration, while a larger $i$ corresponds to lower concentration.\n\nIn Equation (8), the left-hand side uses $i$ to denote the index of a sample in the reordered set. This formulation expresses the sampling probability at time step $t$ based on the sample’s position in the ordering. The sample with index $i=\\mu_t$ has the highest sampling probability at time $t$, and probabilities decrease symmetrically for samples with indices farther from $\\mu_t$. As  $\\mu_t$ increases over training (as defined in Equation (10)), the sampling center shifts gradually from high- to low-concentration samples, enabling a smooth curriculum from easy to harder data.\n\nWe appreciate the reviewer’s question and will revise the paper to make this interpretation of the index-based sampling clearer in the updated version.\n\n\n> ***Limitation.  Observations on angle concentration should also be conducted on larger models to improve the generalizability of the findings.***\n\n**Response**:  Thank you for the valuable suggestion. We agree that analyzing angular concentration on larger models would enhance the generalizability of our findings. For efficiency, we only presented visualizations on the 0.5B model, but experiments show GAIN-RL consistently accelerates training across models up to 7B parameters, supporting the broader applicability of our insights. We will add visualizations for larger models in the revised version to further support our claims. We appreciate the reviewer’s helpful feedback.\n\n\n\n\n\n\n\n**References:**\n\n[1] Boyd, Stephen P., and Lieven Vandenberghe. *Convex optimization*. Cambridge university press, 2004.\n\n[2] Nesterov, Yurii. *Lectures on convex optimization*. Vol. 137. Berlin: Springer International Publishing, 2018.\n\n[3] Chen, Zhao, et al. \"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.\".\n\n[4] Jiang, Angela H., et al. \"Accelerating deep learning by focusing on the biggest losers.\".\n\n[5] Campbell, Ryan, and Junsang Yoon. \"Automatic curriculum learning with gradient reward signals.\".\n\n[6]Xiong, Ruibin, et al. \"On layer normalization in the transformer architecture.\"."}, {"review_id": "1CFjvaY7Gk", "reviewer": "Reviewer_LV4y", "summary": "Authors present an interesting observation on LLM training dynamics: the Qwen2.5-0.5B-Instruct model prioritizes learning samples with high angle score ($C_{intra}+C_{inter}$) and progressively learns lower-score samples. The angle score has both theoretical and empirical connection to the attention score between question tokens and between question tokens and instruction/example tokens.\n\nWith this observation in hand, the paper proposes a novel framework that skews the sample distribution to favor those with high angle score, updating the distribution to lower scores as training progresses. They were able to achieve significant improvements in sample efficiency with virtually no computation overhead made in the dataset preprocessing (sorting) phase.", "questions": "1. As stated in the weaknesses, \"High AngConc-biased Sampling\" seems to perform better than GAIN-RL and is much simpler (e.g., less hyperparameters). I cannot help but ask: why not just use that?\n\n2. In eq8, how is $\\sigma_t$ tuned?\n\n3. Looking at eq10, it seems like $\\mu_t$ starts from 0 then increases indefinitely. Are there guarantees that $\\mu_t$ doesn't exceed $n$, or is it explicitly forced to be under $n$?\n\n4. As $\\mu_t$ increases, would samples with low index (high angle scores) eventually have zero probability? If so, wouldn't it cause issues for large datasets such as catastrophic forgetting?", "limitations": "Authors do discuss future work in Appendix.G but not specifically their limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The main observation/motivation is interesting, and is supported by theoretical and empirical evidences.\n- The experimental setup is clear and straighforward, and their results support their aforementioned observation across diverse LLM models and sizes.\n\nWeaknesses\n- I would have been nice to see experiment on much larger models (at least on few menchmarks), although I believe 7B meets the standard.\n- Figure8 seems to reveal that angle-biased sampling is a more simple yet superior method to GAIN-RL, which I think needs more explanation than what's said on line 278-280.", "was_revised": true, "final_justification": "The main problem was that the writing was a bit confusing and difficult to read. I think this has been sufficiently addressed.", "rebuttal": "We thank the reviewer for the positive and encouraging feedback. We’re glad that the motivation and supporting evidence were found compelling, and that the experimental setup was seen as clear and effective. We hope the following responses further address any remaining questions.\n\n\n\n> ***Weakness1. I would have been nice to see experiment on much larger models (at least on few menchmarks), although I believe 7B meets the standard.***\n\n**Responce**:  We thank the reviewer for the helpful suggestion. We fully agree that experiments on larger models would further highlight the effectiveness and scalability of our approach. In fact, we have already begun running experiments on larger models and plan to include the results in the discussion section or in an updated version of the paper.\n\n\n\n> ***Weakness2. Figure8 seems to reveal that angle-biased sampling is a more simple yet superior method to GAIN-RL, which I think needs more explanation than what's said on line 278-280.***\n\n**Responce**: We sincerely thank the reviewer for this thoughtful and insightful comment. As the reviewer rightly pointed out, Figure 8 deserves a more nuanced explanation. We fully agree and are happy to provide a deeper analysis of this promising result.\n\nFigure 8 illustrates the effect of our proposed High Angular Concentration-biased Sampling method, which selectively excludes samples with low angular concentration. The results suggest that **removing such samples can significantly improve training efficiency while preserving gradient effectiveness and stability.** This observation is supported by several layers of evidence:\n\n1. **Theoretical justification via Equation (4):** Low angular concentration samples reduce the mean and increase the variance of training gradients. As shown in Equation (4), angular concentration among hidden states directly influences the Frobenius norm of the gradient through the sum of cos θ terms. A higher angular concentration yields stronger, more consistent gradient updates, accelerating the learning process and improving efficiency.\n2. **Empirical support from the neuron activation patterns (Figure 6):** As shown in Figure 6, high-angular-concentration samples tend to activate a focused cluster of core neurons, enabling effective gradient accumulation and rapid formation of high-performing neuron clusters. In contrast, low-concentration samples activate more scattered neurons, leading to dispersed gradient directions. Although more samples tend to activate core neurons as training progresses, a subset of low-concentration samples continues to produce noisy and inconsistent activations, thereby reducing overall training stability and efficiency.\n3. **Alignment with prior research findings:** Several prior works [1,2,3] have shown that not all training samples contribute positively to learning, and removing \"negative gain\" samples can lead to faster and more stable optimization. Our method provides a practical and computationally efficient way to filter out such low-value data points based on angular concentration.\n\nWe will expand our discussion of this result in the revised version and plan to explore this direction further in future work.\n\nOnce again, we sincerely appreciate the reviewer’s constructive feedback, which has helped us improve the depth and completeness of our paper.\n\n\n\n> ***Question 1. As stated in the weaknesses, \"High AngConc-biased Sampling\" seems to perform better than GAIN-RL and is much simpler (e.g., less hyperparameters). I cannot help but ask: why not just use that?***\n\n**Responce**:  We sincerely thank the reviewer for raising this insightful question. We appreciate the recognition of the effectiveness of high angular concentration-based sampling and are happy to clarify why we adopt GAIN-RL over a simpler hard-filtering approach.\n\n1. **Angular concentration is dynamic during training:** As model capacity improves, samples with initially low angular concentration often evolve into high-concentration ones (see Figure 4 in the main text). These examples, though have low angular concentration and weak early on, may contain long-tail semantics or rare states that become high angular concentration and valuable later. Therefore, filtering based on static pre-training concentration may prematurely discard useful data.\n2. **Static thresholds lack generalizability; GAIN-RL offers a dynamic solution:** While Figure 8 shows promising results using a fixed top 50% threshold, this cutoff is empirical and dataset-specific. Angular concentration distributions vary widely across datasets, and different settings may require different thresholds. In contrast, GAIN-RL introduces a dynamic, model-aware sampling strategy that avoids trial-and-error tuning and generalizes better across tasks and architectures.\n3. **Hard data removal can be harmful:** In large-scale RL + LLM training, completely discarding data risks overfitting to high-concentration clusters or triggering training collapse. Prior studies [1,2,3] also suggest that importance-weighted sampling is more robust and generalizable than hard filtering. Even when de-emphasizing low-concentration samples, it is often better to retain them with down-weighted importance than to remove them entirely.\n\nWhile we agree that angular concentration is a powerful signal for data selection, the current static filtering setup (i.e., dropping half the data) requires further validation to assess its generalization. We plan to include these results in the discussion section or an updated version of the paper. In contrast, GAIN-RL has already demonstrated strong generalization across datasets and model sizes, making it a more practical and robust choice.\n\nOnce again, we sincerely thank the reviewer for the insightful comment, which helped us clarify and strengthen our position.\n\n\n\n> ***Question 2. In eq8, how is $\\sigma_t$ tuned?***\n\n**Responce**:   We sincerely thank the reviewer for the careful reading and for raising this important question. We apologize for the omission of the explanation regarding the choice of $\\sigma_t$ in the paper, and we are happy to clarify it here.\n\nIn our implementation, we set $\\sigma_t$ as a fixed value: $\\sigma_t = N/3$ , where $N$ is the total number of training samples. This choice is motivated by both theoretical considerations and empirical ablation studies:\n\n**Theoretical motivation:** To avoid catastrophic forgetting during training, we aim to ensure that all samples maintain a non-zero probability of being selected at any point. For a Gaussian distribution, 99.73% of the mass lies within $[\\mu - 3 \\sigma, \\mu + 3 \\sigma]$. By setting $\\sigma_t = N/3$, even in the most extreme case (when $\\mu_t = N$), the earliest 10% of samples—those with the highest angular concentration—still retain approximately 7% sampling probability. This helps preserve important early-stage knowledge throughout training.\n\n**Empirical validation:** We conducted ablation experiments on the MATH dataset using Qwen2.5-0.5B-Instruct, evaluating model performance after 200 epochs under different values of σ. The results show that: A smaller $\\sigma_t$ (e.g., $N / 5$) leads to sharp forgetting, with validation accuracy rising initially but then degrading due to over-focusing on a narrow sample range. A larger $\\sigma_t$ (e.g., $N / 2$) slows down training by diluting the prioritization of high-gradient samples. $\\sigma_t = N / 3$ strikes a good balance, enabling both efficient training and knowledge retention.\n\n| Sigma Value | N/2   | N/3   | N/4   | N/5   |\n| ----------- | ----- | ----- | ----- | ----- |\n| Accuracy    | 21.2% | 22.6% | 22.5% | 19.3% |\n\nWe will include a detailed explanation and ablation results regarding the choice of σ in the revised version. We sincerely appreciate the reviewer’s helpful comment, which has allowed us to clarify this important design decision.\n\n\n\n\n\n> ***Question 3. Looking at eq10, it seems like $\\mu_t$ starts from 0 then increases indefinitely. Are there guarantees that $\\mu_t$ doesn't exceed , or is it explicitly forced to be under n?***\n\n**Responce**:  We sincerely thank the reviewer for the helpful question. In our implementation, we explicitly enforce a truncation on $\\mu_t$ such that $\\mu_t = min(\\mu_t, N)$, where $N$ is the total number of training samples. Once $\\mu_t$ reaches $N$, the current training round is terminated at the end of that epoch, and a new round is restarted with $\\mu_t$ reset to 0. This design ensures that sampling remains valid and the training process remains effective throughout.\n\nWe will clarify this implementation detail in the revised version. We appreciate the reviewer’s question, which helps improve the completeness of our explanation.\n\n\n\n\n\n> ***Question 4. As $\\mu_t$ increases, would samples with low index (high angle scores) eventually have zero probability? If so, wouldn't it cause issues for large datasets such as catastrophic forgetting?***\n\n**Responce**:  We sincerely thank the reviewer for raising this important concern. As mentioned in our response to Question 2, our $\\sigma_t$ value is set relative to the total number of training samples ($\\sigma_t = N/3$), which ensures that all samples retain a non-zero probability of being selected. This design helps effectively prevent catastrophic forgetting, even as $\\mu_t$ increases during training [4].\n\n\n\n**References:**\n\n*[1] Schaul T, Quan J, Antonoglou I, et al. Prioritized experience replay[J]. arXiv preprint arXiv:1511.05952, 2015.*\n\n*[2] Pleiss L S, Sutter T, Schiffer M. Reliability-Adjusted Prioritized Experience Replay[J]. arXiv preprint arXiv:2506.18482, 2025.*\n\n*[3] Isele D, Cosgun A. Selective experience replay for lifelong learning[C]//Proceedings of the AAAI conference on artificial intelligence. 2018, 32(1).*\n\n*[4] Lopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" *Advances in neural information processing systems* 30 (2017).*"}]

Author rebuttal:

Meta-review: The paper proposes GAIN-RL, a reinforcement learning fine-tuning (RFT) algorithm for LLM  that improves sample efficiency and reduces computational cost. The key idea is to use angle concentration—cosine similarity among hidden states—as a model-intrinsic signal to guide data reordering and dynamic sampling. By leveraging this signal with Gaussian-based sampling updates, GAIN-RL achieves faster training and stronger performance compared to vanilla GRPO and related baselines. 

All three reviewers rated the paper Accept (5), and they agree on the technical soundness, originality, and strong empirical validation. While some questions remain (e.g., broader baselines, clearer theoretical exposition), the consensus is that the contribution is novel and timely. Therefore, I recommend to accept the paper.
Decision: Accept (spotlight)