Title: Angles Don’t Lie: Unlocking Training‑Efficient RL Through the Model’s Own Signals
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Efficiency

Data Optimization

Reinforcement Fine-Tuning

Training‑Efficient RFT
Abstract: Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models (LLMs) suffer from sample inefficiency due to the redundant exposure of identical queries under uniform data sampling. While previous work has explored curriculum learning via heuristic difficulty metrics, these strategies exhibit limitations by neglecting the intrinsic learning signals generated by the model itself, thus leading to suboptimal training regimes. In this paper, we identify a model-inherent signal termed *angle concentration* that effectively reflects an LLM's capacity to learn from specific data. We theoretically and empirically demonstrate a correlation between the angular distribution of token hidden state vectors and the resulting gradient, revealing a learning preference for data exhibiting higher angle concentration. Inspired by this finding, we propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. 
By leveraging the model's intrinsic angle concentration signal, GAIN-RL dynamically selects training data in each epoch, ensuring consistently impactful gradient updates and thus significantly enhancing overall training efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5$\times$ acceleration in training efficiency across diverse mathematical and coding tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient sampling yields data-efficient training, achieving better performance with half the original data compared to vanilla GRPO with full training data.

Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals

Abstract

Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models (LLMs) suffer from sample inefficiency due to the redundant exposure of identical queries under uniform data sampling. While previous work has explored curriculum learning via heuristic difficulty metrics, these strategies exhibit limitations by neglecting the intrinsic learning signals generated by the model itself, thus leading to suboptimal training regimes. In this paper, we identify a model-inherent signal termed angle concentration that effectively reflects an LLM's capacity to learn from specific data. We theoretically and empirically demonstrate a correlation between the angular distribution of token hidden state vectors and the resulting gradient, revealing a learning preference for data exhibiting higher angle concentration. Inspired by this finding, we propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. By leveraging the model's intrinsic angle concentration signal, GAIN-RL dynamically selects training data in each iteration, ensuring consistently impactful gradient updates and thus significantly enhancing overall training efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5 × acceleration in training efficiency across diverse mathematical and coding tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient sampling yields data-efficient training, achieving better performance with half the original data compared to vanilla GRPO with full training data. Code is realsed at https://github.com/wangqinsi1/GAINRL/tree/main.

1 Introdction

Since the emergence of groundbreaking Reinforcement Learning Fine-tuning (RFT) techniques exemplified by Deepseek-R1 [1] and OpenAI's O1 [2], significant attention has converged on leveraging these approaches to enhance performance, notably in mathematical reasoning [3, 4, 5] and code generation tasks [6, 7]. This interest has catalyzed the development of numerous algorithmic optimization methods , including GRPO [8], ReMax [9], and Reinforce++ [10]. Despite such remarkable progress, critical challenges persist: RFT remains hindered by persistent issues of low sample efficiency and prohibitively high computational costs. For instance, the GRPO fine-tuning phase on Qwen 2.5-7B (Ray + vLLM) still consumed roughly 240 GPU hours (16 × H100-80 GB for 15h) to complete only 100 steps over 8k samples [11]. This low sample efficiency prompts the question: Is it truly necessary to repeatedly expose every data point to the model hundreds of times? Our answer is No . Humans adaptively learn by focusing on what they don't yet understand, rather than rote repetition of simple concepts. We extend this insight into reinforcement learning fine-tuning for LLM reasoning models from the perspective of data manipulation.

Manipulating the data is crucial for accelerating data-driven LLM training. Existing data manipulation techniques fall into two main categories: Sample selection methods, such as LIMO [12] and S1 [13],

1

2. Gaussian-based

DATA Sampling

Sorted DATA index

𝜇𝑡+1

𝜇𝑡

𝒩 𝜇,𝜎 2

MODEL

#Decoding

Weight

update

𝜇𝑡+1

= 𝜇𝑡 +ℱ acct

+𝒢 anglet

3. Dynamic Probability Update

System

Prompt

Token

Question

Token

DATA

#Pre-filling

MODEL

Layer index

1. Angle-based Data Reordering

Angle

Signal

intra

C

inter

C

+

Sorted

DATA

Low

High

∝

Gradient

have shown that training on a carefully curated subset of high-quality data can improve performance. Separately, data ordering strategies, like ADARFT [14], have demonstrated that dynamically adjusting data difficulty during training can accelerate convergence. However, existing approaches suffer from two fundamental limitations that hinder their effectiveness in RFT. First and foremost, current methods neglect the intrinsic characteristics of the models themselves. They rely on fixed, model-agnostic criteria-such as difficulty or diversity-to evaluate data without accounting for how the target model itself perceives the data. We point out that different models interpret the same problem in markedly different ways. As shown in Fig. 2, different models produce diverging accuracy distributions on the same set of data, indicating that one-size-fits-all difficulty measures can lead to suboptimal training outcomes. Second, existing methods suffer from high data preprocessing costs. For instance, S1 and LIMO necessitate running large-scale models (e.g., Qwen2.5-Math-7B) across entire datasets to compute quality and difficulty scores. Similarly, curriculum learning often relies on manual annotation or expert-defined difficulty labels. These resource-intensive preprocessing steps significantly limit the scalability and practical responsiveness of these approaches.

We aim to attack the aforementioned challenges and propose a model-informed signal that (1) reflects the learning capacity of a specific model on particular data , (2) incurs minimal computational costs , and (3) maintains generalizability across diverse models and datasets . Achieving such requirements is inherently challenging, as accurately capturing model-data interactions typically needs a resourceintensive decoding stage. To overcome this challenge, we explore three key questions in this work:

Which signal should we focus on? By reformulating the gradient expressions, we find that the cosine similarity between token hidden states during inference (hereafter referred to as angle concentration ) directly influences gradient norm, underscoring it as a critical signal.

What are the characteristics of the signal? By tracking the layer-wise evolution, we observe a transition from inter-segment to intra-segment angle concentration, which jointly facilitates information flow and constitutes the Layer-wise Angle Concentration Pattern.

How can these signals accelerate training? By monitoring angle concentration throughout training, we observe continuous convergence of intra- and inter-segment angles over epochs, revealing an Epoch-wise Angle Concentration Pattern . Moreover, the model preferentially learns samples with higher angle concentration before those with lower concentration, indicating a Data-wise Angle Concentration Pattern . These patterns highlight the model's data-learning preferences and can be leveraged to accelerate training.

Building upon these insights, we propose GAIN-RL , a G radient-driven A ngleI nformed N avigated Reinforcement Learning Framework, illustrated in Fig. 1. GAIN-RL comprises three primary components: Data Reordering, Data Sampling and Probability Update. Before training, we reorder the training data based on model-informed angular concentration to enhance learning efficiency. During training, a dynamic Gaussian probability sampling strategy progressively guides the model towards data with lower angular concentration, with the pace adjusted according to real-time accuracy

0

20

40

60

80

100

Question Index

0.0

2.5

5.0

7.5

10.0

Correct Num(Max=10)

Qwen2.5-0.5B-Instruct

Llama-3.2-1B-Instruct

Chatgpt

and angular signals. Notably, our data preprocessing involves only the inexpensive pre-filling stage, requiring under 10 minutes for over 7,000 samples. Overall, GAIN-RL provides a plug-and-play, broadly applicable, and data-efficient reinforcement learning solution. Experiments validate that GAIN-RL accelerates training by over 2.5 × and, remarkably, surpasses full-sample performance using only half the data-demonstrating its powerful capability in driving data-efficient RFT.

In summary, our key contributions include:

We propose that the angle concentration serves as a critical signal influencing gradient norm during training, thus predicting the model's ability to learn specific data samples.

We demonstrate that angle concentration intrinsically reflects information propagation during inference and learning dynamics during training, and explicitly reveals Layerwise, Epoch-wise, and Data-wise Angle Concentration Patterns.

We introduce GAIN-RL, a framework that dynamically allocates training data per epoch based on angle concentration signals. GAIN-RL is the first framework to utilize intrinsic model signals for data scheduling, providing a novel paradigm for efficient RFT.

2 Model-Informed Data Evaluation Signals

With the goal of identifying signals that can effectively reflect a specific model's learning capability for data, in this section, we explore three key questions: (1) Which signal should we focus on? (2) What are the characteristics of this signal? (3) How can this signal be leveraged to accelerate training?

2.1 Which Signals Should We Focus On?

As mentioned in Sect. 1, obtaining model feedback via decoding is computationally expensive. In contrast, the pre-filling stage incurs significantly lower costs, as it requires only a single forward pass. Intuitively, we hope to identify signals from the pre-filling stage that can inform the training process.

Model training is driven by incremental gradient updates. Hence, to investigate how forward signals influence the backward process, we begin by examining and reformulating the gradient representation. Given a weight matrix W ∈ R d × h , the hidden states of input can be denoted as x ∈ R m × d , where x consists of m tokens, and the hidden state of the i -th token is represented as x i . The output activation is denoted as a = xW , where a ∈ R m × h . Assuming the loss function L has a gradient ∇ a L ∈ R m × h with respect to the activation a , the gradient of L with respect to W is given by m

where ( ∇ a L ) i denotes the gradient of the loss L with respect to the i -th activation vector a i . To more precisely quantify the magnitude of the gradients, we consider the Frobenius norm of ∇ W L . Leveraging the linearity of the Frobenius inner product, ∥∇ W L∥ 2 F can be expanded as:

Next, to simplify Eq. 2, we utilize the compatibility between the Frobenius inner product and the matrix outer product. In particular, for any u , w ∈ R d and v , z ∈ R h , the following identity holds:

where tr denotes the matrix trace operator. Applying this identity term-wise in Eq. 2 gives

where cos θ i,j denotes the cosine similarity of the angle between x i and x j . Eq. 4 explicitly reveals that, during inference, both the magnitudes of token hidden states and the angles between them directly influence the gradient values computed during backpropagation.

Since the magnitudes of token hidden states are normalized in each layer during inference, they cannot effectively convey useful information. Consequently, in this paper, we specifically focus on exploring the characteristics of relative angles of token hidden states. Furthermore, in the Appendix B.1, we provide proofs demonstrating that the nonlinear transformations in LLM inference-including both attention mechanisms and activation functions-are inherently angle-dependent and continuously modify the angles among token hidden states. We can now answer the first question:

A1. We should focus on the relative angles between token hidden states during inference as it fundamentally impacts the gradient Frobenius norm computed during backpropagation. Specifically, the more concentrated the angles between tokens, the larger the gradient norm.

2.2 What Are the Characteristics of This Signal?

In the previous subsection, we show that angles between token hidden states directly influence the gradients. To identify the characteristics we should focus on, we explore its layer-wise evolution.

Layer-wise Observation. We conducted experiments on the Qwen2.5-0.5b-Instruct model to observe the evolution process across different layers. The experimental results, shown as Fig. 3, clearly illustrate that in the initial layers of the model, no distinct pattern emerges, with the angles primarily determined by the input embeddings. As the layer depth increases, the angles gradually exhibit a segmented structure, whereby the hidden states of tokens within the same segment tend to cluster more closely. Upon further examination, we found these segments correspond precisely to distinct parts of the input sequence: the system prompt, few-shot examples, and the question. Eventually, in the final layers, angles between tokens from different segments begin to converge, reaching the highest degree of concentration. We give a more vivid demonstration in Fig. 1.

Based on the above observations, we introduce Layer-wise Angle Concentration Pattern : during inference, the model first induces intra-segment angle concentration and subsequently promotes inter-segment angle concentration. These two forms of clustering collaboratively facilitate information propagation through the model. A detailed demonstration is provided in Appendix C.1.

Therefore, assume the length of the input tokens is m . The first n tokens constitute the system prompt and the few-shot examples, which are the same across all data samples. The remaining m -n tokens represent the specific question to be answered. The characteristics we should focus on are:

where C intra measures angle concentration within the question and C inter measures angle concentration between question tokens and the system prompt and few-shot tokens. We measure both at the final layer, where inter-segment clustering is maximal. Concentration within the system prompt and few-shot tokens is omitted, as it is constant across different questions.

Attention-based Explanation. To better understand the observed pattern, we also provide an analytical explanation from attention scores. In general, we find that tokens with higher angle concentration correspond to higher attention scores. Specifically, C intra represents the strength of attention within the question itself, while C inter indicates the model's ability to follow instructions. Furthermore, the presence of sink attention encourages intra-segment and inter-segment angle concentration. Details can be found in Appendix B.2. Our answer to the second question is:

A2. The angles between token hidden states show both intra-segment and inter-segment concentrations during inference. In particular, we should pay attention to the final layer as the inter-segment clustering is most pronounced, resulting in the highest overall concentration.

0.55

-0.5

0.50

-0.40

0.5

- 0.4

0.45

0.35

CosinValue

0.30

0.40

0.3

0.25

0.35

0.20

0.2

0.30

0.15

0.25

0.10

0.2

0.1

0.20

0.05

0.1

Layer 0

Layer5

Layer15

Layer 23

0

50

100

150

200

250

Training Iteration

0.37

0.38

0.39

0.40

intra

Qwen2.5-0.5B-Instr

Llama-3.2-1B-Instr

0.28

0.30

0.33

0.35

0.38

0

50

100

150

200

250

Training Iteration

0.20

0.21

0.21

0.21

0.22

0.23

inter

Qwen2.5-0.5B-Instr

Llama-3.2-1B-Instr

0.12

0.14

0.16

0.18

0.20

0.22

0

50

100

150

200

250

Training Iteration

0.58

0.60

0.62

intra+

inter

Qwen2.5-0.5B-Instr

Llama-3.2-1B-Instr

0.40

0.45

0.50

0.55

0.60

2.3 How Can This Signal Be Leveraged to Accelerate Training?

Having established that intra-segment and inter-segment angular concentrations at the final layer are crucial characteristics, we now further examine their evolution during training to deepen our understanding of how they reflect and influence the training progress.

Epoch-wise Observation. To track how angular concentrations evolve during training, we perform inference on the same dataset on models of different epochs and monitor three signals: (1) Intraquestion concentration C intra , (2) Inter-segment concentration C inter , (3) Combined signal C intra + C inter . As illustrated in Fig. 4, we can observe the Epoch-wise Angle Concentration Pattern : during training, both inter-segment and intra-segment angle concentration increase progressively. Additionally, we note that the intra-question concentration initially decreases before subsequently increasing, which suggests the model prioritizes mastering instruction-following capabilities before refining its focus internally on individual questions. These observations validate our hypotheses from the previous subsections, reinforcing that angular concentration effectively mirrors training dynamics.

Data-wise Observation. Furthermore, to examine data-wise angle behavior during training, we track the model's responses to samples with varying angle concentrations over epochs. As depicted in Fig. 5, surprisingly, the results revealed that during training, the model tends to prioritize learning from higher-angle concentration data before addressing lower-angle concentration data, which we introduce as Data-wise Angle Concentration Pattern . For instance, by iteration 100, questions with maximal angular measurements were almost entirely answered correctly, whereas those with smaller angles remained uncorrected. Note that the angle concentration distribution is measured on the untrained model, indicating that despite its evolution during training, the initial angle concentration of the data provides meaningful guidance for the training process.

To understand these patterns, we provide explanations from two perspectives: gradients and neurons.

Gradient-based Explanation. As shown in Eq. 4, gradients are influenced by both angle concentration and loss. Early in training, when losses are relatively uniform across samples, those with higher angle concentration receive stronger gradients and are learned faster. As training continues, angle concentration rises overall: high-angle samples, already mastered, see lower losses; while low-angle samples gain concentration, inherit larger gradients, and are learned next. This creates a natural, angle-driven learning progression. Unlike traditional curriculum learning based on task difficulty, angle concentration offers a more intuitive, model-centric training signal.

Neuron-based Explanation. Consider a FFN block with weights W u and W d , and activation function SiLU . Given the input denoted as x , the transformation process can be represented as:

0.50

0.55

0.60

intra+

inter

0

50

100

Number of Sample

0.50

0.55

0.60

intra+

inter

Epoch 50

0.50

0.55

0.60

intra+

inter

Epoch 100

0

2

4

6

8

10

Avg. Correct(Max=10)

500

0

500

Principal Component 1

400

200

0

200

400

Principal Component 2

500

0

500

Principal Component 1

Epoch 50

500

0

500

Principal Component 1

Epoch 300

0

2

4

6

8

10

Avg. Correct(Max=10)

z = xW u , A = SiLU( z ) , y = AW d , where z and A denote the pre-activation and activation outputs, respectively, and y represents the output. The gradients of j -th neuron in W u and W d are

where SiLU ′ ( z i,j ) is the derivative of the SiLU activation with respect to z i,j . When z i,j < 0 , we have SiLU ′ ( z i,j ) ≈ 0 and SiLU( z i,j ) ≈ 0 . This indicates that the number of gradient components received by a neuron is proportional to the frequency of its activation.

Further analysis shows that tokens with higher angle concentration activate similar neurons due to shared value patterns (see Appendix B.3). These neurons receive stronger cumulative gradients, making them more effectively trained. Fig. 6 further illustrates how neuron activations converge during training, forming a distinct cluster correlated with higher accuracy. Samples activating neurons far from this cluster are harder to learn. Following prior work on neuron specialization, we hypothesize this cluster encodes domain-specific knowledge [15, 16, 17].

Synthesizing the above analyses, we provide an answer to the third question as follows:

A3. We should follow the model's inherent learning dynamics - prioritizing higher-angle concentration data in the early stages and gradually transitioning to lower-angle concentration data. This progression ensures more effective gradient updates and improves training efficiency.

3 GAIN-RL Framework

Based on the conclusions from the Sect. 2, we introduce GAIN-RL , a G radient-driven A ngleI nformed N avigated-data RL framework, a plug-and-play training acceleration framework compatible with any model and dataset, incurring negligible costs. GAIN-RL consists of three components:

Data Reordering Based on Angular Concentration. Guided by our findings in Sect. 2.3-that models preferentially learn from data with higher angular concentration-we order the training data by angular concentration before training to improve efficiency. Given a model M and a dataset D = { d 1 , d 2 , . . . , d N } , we first perform pre-filling on all data samples using M to collect angular information. Subsequently, the data is sorted based on the combined signal at the final layer output,

the sorted dataset D s is directly employed in subsequent training. Notably, this sorting process is computationally efficient, as it only requires the pre-filling step, which can be efficiently batched. For example, sorting approximately 7000 samples of GSM8K with the Qwen-2.5-0.5-instruct model takes less than 10 minutes on a single NVIDIA A100 GPU. In contrast, previous approaches often required manual annotation or generation via large models, typically consuming several days.

Data Sampling Guided by Gaussian Probability. During training, we consistently prioritize data with higher angular concentration. At the t th training step, we assign sampling probabilities to each data sample in the sorted dataset D s = { d s 1 , d s 2 , . . . , d s N } based on a Gaussian distribution parameterized by µ t and σ t . A subset d ( t ) of size n is then sampled according to,

where Z t is a normalization constant ensuring that probabilities sum to unity. Employing probabilistic sampling instead of strictly sequential sampling enhances the stability and robustness of training.

Probability Update Based on Accuracy and Angular Signals. The Gaussian mean µ t sets the peak-sampling region. We initialize µ 0 = 0 to prioritize high-angle concentration data, then gradually increase µ t as the model masters these samples, shifting focus to lower-angle concentration ones. At each step, µ t is updated from the batch d ( t ) using its average accuracy and angle concentration,

where M t represents the model at the t th training step, and Acc M t ( d ( t ) i ) and C M t ( d ( t ) i ) denote accuracy and angular concentration signals. Notably, computing these signals incurs no additional cost as model inference is inherently performed during training. The update rule for µ t +1 is given by,

where α adjusts accuracy sensitivity, β sets the target accuracy, and γ controls angle sensitivity (see Sect. 4 for guidelines). This update strategy maintains high-gradient training by targeting samples near the desired accuracy while gradually incorporating harder, lower-angle data efficiently.

Appendix F experiments reveal that weighting the signal as C M intra + k · C M inter can further boost performance. For simplicity and broad applicability, we adopt the unweighted signal here and leave signal optimization to future work. Our main goal is to highlight angle concentration as a key signal.

0

25

50

75

100

125

150

175

200

Training Iteration

37

38

39

40

41

42

GRPO

ADARFT(GRPO)

GAIN-RL(ours)

25

50

75

100

125

150

175

200

Training Iteration

20

22

24

26

28

GRPO

ADARFT(GRPO)

GAIN-RL(ours)

0.40

0.45

0.50

0.55

0.60

0.65

Avg. Angle (Untrained Model)

0

100

200

300

400

500

600

700

800

Number of Sample

High AngConc biased Sampling

0.40

0.45

0.50

0.55

0.60

0.65

Avg. Angle (Untrained Model)

0

100

200

300

400

500

600

700

800

Number of Sample

Uniform Sampling

0.40

0.45

0.50

0.55

0.60

0.65

Avg. Angle (Untrained Model)

0

100

200

300

400

500

600

700

800

Number of Sample

Low AngConc biased Sampling

0

25

50

75

100

125

150

175

200

Training Iteration

12

14

16

18

20

22

24

High AngConc biased Sampling

Uniform Sampling

Low AngConc biased Sampling

GAIN-RL(Full)

GRPO(Full)

4 Experiments

To evaluate the effectiveness of GAIN-RL, we conducted a comprehensive experimental study on five levels: (1) Training Efficiency(Sect. 4.1), (2) Data Efficiency(Sect. 4.2), (3) RL Algorithms Generalization(Sect. 4.3), (4) Performance on Individual Tasks(Sect. 4.4), and (5) Ablation Studies(Sect. 4.5). For detailed descriptions of the used models and datasets, please refer to the Appendix E.

Training and Hyperparameter Settings. We set the target accuracy β = 0 . 5 to maintain strong gradients during training. Sensitivity parameters α = 2 (for accuracy) and γ = 0 . 5 (for angle concentration) are tuned on a validation set to ensure stable learning, keeping the tanh function approximately linear over Acc ( t ) ∈ [0 , 1] and C ( t ) ∈ [ -1 , 1] . Training is conducted using GRPO with a batch size and sampling number n of 1024, implemented on the VerL framework with 8 NVIDIA A100 GPUs. Additional details are provided in the Appendix E.

Baseline Settings. We compare GAIN-RL with the vanilla GRPO and ADARFT[14], a state-of-theart dynamic curriculum learning method in RL. The training settings are the same for all methods.

4.1 Training Efficiency of GAIN-RL

To evaluate the training efficiency of GAIN-RL, we use the DeepScaleR [24] and DeepCoder [25] datasets to train models for math and code tasks, respectively. They both cover diverse problem types and difficulty levels. Performance is tested on six math and three code benchmarks.

Model Performance. We report the performance of each method at 200 iterations during training. As demonstrated in Tab. 1 and Tab. 2, models trained with GAIN-RL(GRPO) exhibited superior performance across nearly all math and code datasets compared to models trained by vanilla GRPO and ADARFT(GRPO). Notably, with the Qwen2.5-Math-1.5B-Instruct model, GAIN-RL(GRPO) increased average accuracy across six math datasets from 39.95% and 41.09% to 42.63%, representing gains of 2.68% and 1.54%, respectively. Furthermore, the performance improvement observed on

0

25

50

75

100

125

150

175

200

Training Iteration

33

36

39

42

45

48

51

GRPO

GAIN-RL(ours)

0

25

50

75

100

125

150

175

200

Training Iteration

28

30

32

34

36

GRPO

ADARFT(GRPO)

GAIN-RL(ours)

0

25

50

75

100

125

150

175

200

Training Iteration

4

5

6

7

8

9

10

11

12

GRPO

ADARFT(GRPO)

GAIN-RL(ours)

0

25

50

75

100

125

150

175

200

Training Iteration

12

14

16

18

20

22

24

Sort(Angle)+Update(Angle & Acc)

Sort(Angle)+Update(Acc)

Sort(Angle)+Update(Angle)

Sort(Acc)+Update(Acc)

GRPO

0

25

50

75

100

125

150

175

200

Training Iteration

12

14

16

18

20

22

GAIN-RL(n=512)

GAIN-RL(n=768)

GAIN-RL(n=1024)

GRPO(n=1024)

LLaMA3.2-3B-Instruct demonstrates that GAIN-RL(GRPO) generalizes across different model families. These results highlight both model-level and task-level generality of our method.

Hardware Efficiency. To evaluate hardware efficiency, we report the number of iterations required for each method to reach the performance of vanilla GRPO at 200 iterations. As shown in Tab. 1 and Tab. 2, GAIN-RL(GRPO) requires approximately only half the iterations across various models and tasks. Specifically, for the Qwen2.5-Math-1.5B-Instruct and LLaMA3.2-3B-Instruct models, GAINRL(GRPO) achieves approximately 2.5 × speedup, significantly outperforming ADARFT(GRPO) speedups of 1.33 × and 1.43 × , respectively. Fig. 7 further visualizes performance trajectories during training, where GAIN-RL(GRPO) consistently demonstrates faster convergence and superior performance at every iterations. As analyzed in Sect. 2, the acceleration mainly stems from the ability of our method to maintain strong gradient signals throughout training, leading to faster learning.

4.2 Data Efficiency of GAIN-RL

To further investigate whether GAIN-RL can enhance data effectiveness in RFT, we fine-tuned the Qwen2.5-0.5b-Instruct model on the Math dataset. We sampled half of the training dataset using three distinct distribution strategies for GAIN-RL(GRPO) training: (1) Uniform Sampling, where half of the training data was randomly selected; (2) High Angular Concentration-biased Sampling, where data points were sampled based on their angular concentration scores, assigning a linear sampling probability with a slope of 1 (prioritizing data points with higher angular concentration); and (3) Low Angular Concentration-biased Sampling, applying a linear sampling probability with a slope of -1.

4.3 RL Algorithms Generalization

To verify the algorithm generalization of GAIN-RL, we also combine GAIN-RL with PPO. We evaluate the final performance and hardware efficiency. Results in Tab. 3 highlight substantial gains in both performance and hardware efficiency with GAIN-RL(PPO), achieving an average of 2.2 × training speedup across three mathematical benchmarks. This consistent benefit arises because gradient updates are fundamental across RL algorithms, validating GAIN-RL's universal acceleration capability.

We compared the performance of models trained under these sampling strategies. The experimental results, illustrated in Fig. 8, reveal that when using only half of the data, the model trained with High Angular Concentration-biased Sampling outperformed even the model trained with the full dataset. Based on the analysis in Sect. 2.3, we hypothesize this phenomenon occurs as data points with low angular concentration tend to produce smaller gradient updates and activate dispersed neural regions. Consequently, excluding these data points may enhance training efficiency. This insight provides new guidance for data selection: prioritizing high angular concentration data and discarding low angular concentration data can significantly improve data effectiveness of RFT. These findings highlight GAIN-RL's potential and effectiveness in guiding data selection. With uniform sampling, GAIN-RL using half the data performs slightly worse than with full data but remains on par with vanilla GRPO. In contrast, biased sampling toward low-angle concentration leads to unstable and poor results, consistent with our analysis of weaker gradients and dispersed neuron activations.

4.4 Performance on Individual Tasks

To further demonstrate the generality of GAIN-RL(GRPO), we also evaluate its performance on single-task RFT, which demands higher precision in data ordering and selection due to narrower difficulty ranges. Using Qwen2.5-0.5b-Instruct, we fine-tuned separately on GSM8K, MATH, and AMC training sets and evaluated on their test sets. Results in Fig. 9 indicate that GAIN-RL(GRPO) consistently yielded higher performance and efficiency. Specifically, on GSM8K, GAIN-RL(GRPO) achieved a 3.33 × training speedup and a 4.72% final accuracy improvement. On the more challenging MATH and AMC 23 datasets, GAIN-RL(GRPO) also achieves 2.5 × and 2 × speedups, respectively. In contrast, ADARFT(GRPO) provided less improvement due to its fixed difficulty scoring, which may not align precisely with actual model-perceived difficulty. In contrast, by leveraging modelinformed angle signals, GAIN-RL(GRPO) can predict learning priorities more accurately.

4.5 Ablation Studies

Module Ablation. To better understand the contribution of different components in GAIN-RL, we conduct ablation studies, including (1) Data ordering + Accuracy-based probability updates, (2) Data ordering + Angle-based probability updates, and (3) Accuracy-only (no ordering, discarding fully correct data to reduce training costs). As shown in Fig. 10 (left), all ablation variants exhibit degraded performance. The Accuracy-only group exhibits a marked performance drop, initially improving but later declining due to forgetting from prematurely discarding data. This underscores that data ordering and sampling constitute GAIN-RL(GRPO)'s primary performance advantages. Moreover, while both Data ordering + Accuracy-based probability updates and Data ordering + Angle-based probability updates show steady improvements over vanilla GRPO, they do not match the performance of GAIN-RL(GRPO), as each only captures a single aspect of gradient.

Small-Batch Scalability Test. To validate the scalability of GAIN-RL under reduced training batch sizes, we evaluated the model's performance with varying batch sizes. Specifically, we conducted experiments using the Qwen2.5-0.5b-instruct model on the Math dataset with training batch sizes of 512, 768, and 1024, respectively. The experimental outcomes, depicted in Fig. 10 (right), illustrate consistent and stable performance improvement across iterations for all batch sizes. Remarkably, even with the batch size reduced to half of the original size (n=512), which significantly reduces computational time and memory usage per iteration, GAIN-RL maintained performance comparable to vanilla GRPO. These results demonstrate that GAIN-RL effectively scales with smaller batch sizes, offering the flexibility to accelerate training by lowering batch size, with only minor performance degradation, especially beneficial under limited computational or memory resources.

5 Conclusion

We propose GAIN-RL, a novel Reinforcement Learning Fine-tuning framework that leverages the angle concentration signals to dynamically allocate training data at each iteration. GAIN-RL leverage the model's intrinsic angle concentration signal to dynamically selects training data in each iteration, ensuring consistently impactful gradient updates and thus significantly enhancing overall training efficiency. Furthermore, our empirical results further show that GAIN-RL (GRPO) achieves over a 2.5 × acceleration in training efficiency across diverse mathematical and coding tasks and varying model scales. Overall, GAIN-RL introduces a novel paradigm for training-efficiency RFT, highlighting how model-centric data-processing approaches can remedy the sub-optimality of current RFT methods and further elevate their effectiveness.

References

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021.

Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking. arXiv preprint arXiv:2502.12893 , 2025.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 , 2024.

Qinsi Wang and Sihai Zhang. Dgl: Device generic latency model for neural architecture search on mobile devices. IEEE Transactions on Mobile Computing , 23(2):1954-1967, 2023.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024.

Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310.10505 , 2023.

Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 , 2025.

Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892 , 2025.

Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387 , 2025.

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 , 2025.

Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint arXiv:2504.05520 , 2025.

Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation. arXiv preprint arXiv:2410.18311 , 2024.

Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. arXiv preprint arXiv:2312.12141 , 2023.

Chaojun Xiao, Zhengyan Zhang, Chenyang Song, Dazhi Jiang, Feng Yao, Xu Han, Xiaozhi Wang, Shuo Wang, Yufei Huang, Guanyu Lin, et al. Configurable foundation models: Building llms from a modular perspective. arXiv preprint arXiv:2409.02877 , 2024.

Proceedings of the ACM Web Conference 2023 , Austin, TX, USA, 2023. Association for Computing Machinery. URL https://dl.acm.org/doi/proceedings/10.1145/3543507 .

2024 american invitational mathematics examination (aime). https://www.maa.org/ math-competitions , 2024. Accessed: 2025-05-14.

Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008 , 2024.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems , 35:3843-3857, 2022.

Mikhail Mirzayanov. Open codeforces rating system. https://codeforces.com/blog/ entry/20762 , 2015. Accessed: 2025-05-14.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl. Notion Blog , 2025.

Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, et al. Deepcoder: A fully open-source 14b coder at o3-mini level, 2025.

Qinsi Wang, Hancheng Ye, Ming-Yu Chung, Yudong Liu, Yueqian Lin, Martin Kuo, Mingyuan Ma, Jianyi Zhang, and Yiran Chen. Corematching: A co-adaptive sparse inference framework with token and neuron pruning for comprehensive acceleration of vision-language models. arXiv preprint arXiv:2505.19235 , 2025.

Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Differentiable svd for llm compression and some new perspectives. In The Thirteenth International Conference on Learning Representations .

Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data-centric rlhf: Simple metrics for preference dataset comparison. In NeurIPS , 2024. URL https://arxiv.org/abs/2409. 09603 .

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems , 36:49205-49233, 2023.

Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. Advances in Neural Information Processing Systems , 36:25793-25818, 2023.

Dante Everaert and Christopher Potts. Gio: Gradient information optimization for training dataset selection. arXiv preprint arXiv:2306.11670 , 2023.

Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926 , 2024.

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829 , 2019.

Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, et al. Kvcomm: Online cross-context kv-cache communication for efficient llm-based multi-agent systems. arXiv preprint arXiv:2510.12872 , 2025.

Yueqian Lin, Qinsi Wang, Hancheng Ye, Yuzhe Fu, Hai Li, Yiran Chen, et al. Hippomm: Hippocampal-inspired multimodal memory for long audiovisual event understanding. arXiv preprint arXiv:2504.10739 , 2025.

Qinsi Wang, Bo Liu, Tianyi Zhou, Jing Shi, Yueqian Lin, Yiran Chen, Hai Helen Li, Kun Wan, and Wentian Zhao. Vision-zero: Scalable vlm self-improvement via strategic gamified self-play. arXiv preprint arXiv:2509.25541 , 2025.

Zishan Shao, Yixiao Wang, Qinsi Wang, Ting Jiang, Zhixu Du, Hancheng Ye, Danyang Zhuo, Yiran Chen, and Hai Li. Flashsvd: Memory-efficient inference with streaming for low-rank models. arXiv preprint arXiv:2508.01506 , 2025.

Yueqian Lin, Zhengmian Hu, Qinsi Wang, Yudong Liu, Hengfan Zhang, Jayakumar Subramanian, Nikos Vlassis, Hai Helen Li, and Yiran Chen. Voice evaluation of reasoning ability: Diagnosing the modality-induced performance gap. arXiv preprint arXiv:2509.26542 , 2025.

Hancheng Ye*, Jiakang Yuan*, Renqiu Xia, Xiangchao Yan, Tao Chen, Junchi Yan, Botian Shi, and Bo Zhang. Training-free adaptive diffusion with bounded difference approximation strategy. In The Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS) , 2024.

Hancheng Ye, Chong Yu, Peng Ye, Renqiu Xia, Yansong Tang, Jiwen Lu, Tao Chen, and Bo Zhang. Once for both: Single stage of importance and sparsity search for vision transformer compression. In IEEE/CVF Computer Vision and Pattern Recognition (CVPR) , 2024.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728-53741, 2023.

Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint arXiv:2402.10500 , 2024.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEEE transactions on pattern analysis and machine intelligence , 43(4):1352-1368, 2019.

Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems , 31, 2018.

Yudong Liu, Jingwei Sun, Yueqian Lin, Jingyang Zhang, Ming Yin, Qinsi Wang, Jianyi Zhang, Hai Li, and Yiran Chen. Keyframe-oriented vision token pruning: Enhancing efficiency of large vision language models on long-form video processing. arXiv preprint arXiv:2503.10742 , 2025.

Qinsi Wang*, Jinghan Ke*, Zhi Liang, and Sihai Zhang. Mathnas: if blocks have a role in mathematical architecture design. Advances in Neural Information Processing Systems , 36: 47475-47486, 2023.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems , 35:3843-3857, 2022.

Organization In this Appendix, we provide in-depth descriptions of the materials that are not covered in the main paper, and report additional experimental results. The document is organized as follows:

A - Related Work

B - Theoretical Supplement

-B.1 Theoretical Justification of Angle-Dependent Effects in LLMs

-B.2 Attention-Based Explanation of Layer-wise Angle Concentration Patterns

-B.3 Neuron-Based Explanation of Data-wise Angle Concentration Patterns

C - Visualization Results

-C.1 Layer-wise Angle Concentration Patterns

-C.2 Data-wise Angle Concentration Patterns

D - GAIN-RL Algorithm

E - Experimental Setup

F - Additional Experimental Results

-F.1 Performance of Weighted Signals

-F.2 Model Performance on Single Task

G - Discussion and Future Work

A Related Work

Reinforcement Fine-Tuning (RFT)[8, 10] has demonstrated significant effectiveness in enhancing the reasoning capabilities of large language models [15, 26, 27]. However, despite its promising potential, its low sample efficiency and high computational costs remain critical barriers to the broader adoption of RFT. Recent efforts aimed at addressing these efficiency have primarily focused on algorithmic optimizations and data-centric strategies. Algorithmic optimizations, exemplified by methods such as GRPO [8], REINFORCE++ [10], and ReMax [9], seek to reduce computational complexity by streamlining underlying RL algorithms. Although these methods typically improve efficiency and stability, they often involve inherent trade-offs. For instance, GRPO estimates advantages through relative comparisons within output groups, thereby eliminating the need for value functions. While this approach reduces complexity and the reliance on critics, it can introduce instability due to increased noise in advantage estimation, higher variance in updates, and greater sample requirements.

In parallel, data-centric strategies [28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40] have emerged as promising alternatives for efficient fine-tuning. For example, [28] provides the first quantitative audit of preference datasets, introducing metrics for scale, noise, and information density that expose quality bottlenecks before any policy update. Direct Preference Optimization (DPO) [41] simplifies the entire loop by replacing on-policy RL with a closed-form classification loss, making the choice of high-value preference data the primary driver of alignment quality. To further reduce annotation cost, Active Preference Optimization [42] casts RLHF as an active-learning bandit, adaptively querying only the prompts expected to maximize reward-model improvement. The latest data-centric strategies can be categorized based on their approach to data manipulation: data selection and data sequencing. Data selection techniques involve filtering extensive datasets to retain only a small subset of high-quality training data based on predefined metrics. Methods such as LIMO[12] and s1[13] have demonstrated that carefully curated small supervised fine-tuning datasets can achieve robust performance using orders of magnitude less data. On the other hand, data sequencing strategies[43] enhance model learning speed by rearranging the order of training data within existing datasets. Approaches like ADARFT[14] have shown that dynamically selecting data for each iteration can effectively accelerate the training process.

Nevertheless, existing data-centric strategies have generally failed to account for the unique characteristics of different models, applying uniform data handling procedures across diverse model architectures. Such uniformity can lead to suboptimal outcomes because models differ significantly in their sensitivity and response to the same datasets. To overcome this challenge, this paper aims to identify intrinsic signals within models that can reflect their perceptual capabilities toward data, thereby enabling tailored data strategies without incurring substantial additional costs.

B Theoretical Supplement

In this section, we provide the theoretical explanation supporting the main text in Section 2. Specifically, we elaborate on the Angle-Dependent Effects of Attention and Activation (corresponding to Section 2.1 of the main text), the Attention-Based Explanation of Layer-wise Angle Concentration Patterns (corresponding to Section 2.2 of the main text), and the Neuron-Based Explanation of Data-wise Angle Concentration Patterns (corresponding to Section 2.3 of the main text).

B.1 Theoretical Justification of Angle-Dependent Effects in LLMs

In this section, we demonstrate through derivation of the LLM computational process that the nonlinear operations in LLMs-namely attention and activation computations-are inherently dependent on the angles between the input hidden states.

To support this claim, we first introduce two empirical assumptions based on observation:

Observation 1. W q and W k are nearly approximately orthogonal to each other, i.e., W q W T k ≈ θI. θ is a constant. ( W o , W u , W d ) are approximately orthogonal matrices, i.e., WW T ≈ λI . λ is a constant.

Observation 2. For activation function output vectors A i and A M , cos ( ∠ ( A i , A M )) is proportional to the number of intersections of activated neurons, i.e., ∣ ∣ Γ( x i ) ∩ Γ( x M ) ∣ ∣ .

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 0

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 1

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 2

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 3

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 4

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 5

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 6

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 7

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 8

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 9

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 10

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 11

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 12

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 13

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 14

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 15

0

8

0

8

16

24

32

40

48

56

Layer 16

0

8

16

24

32

40

48

56

Layer 17

0

8

16

24

32

40

48

56

Layer 18

0

8

16

24

32

40

48

56

Layer 19

0

8

16

24

32

40

48

56

Layer 20

0

8

16

24

32

40

48

56

Layer 21

0

8

16

24

32

40

48

56

Layer 22

0

8

16

24

32

40

48

56

Layer 23

0.0

0.2

0.0

0.2

0.0

0.2

0.0

0.2

0.0

0.2

0.0

0.2

0.0

0.2

0.4

0.0

0.5

0.00

0.25

0.50

0.0

0.2

0.0

0.2

0.4

0.0

0.2

0.4

0.00

0.25

0.50

0.0

0.5

0.00

0.25

0.50

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

16

56

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 0

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 1

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 2

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 3

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 4

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 5

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 6

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 7

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 8

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 9

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 10

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 11

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 12

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 13

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 14

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 15

0

8

16

0

8

16

24

32

40

48

56

Layer 16

0

8

16

24

32

40

48

56

Layer 17

0

8

16

24

32

40

48

56

Layer 18

0

8

16

24

32

40

48

56

Layer 19

0

8

16

24

32

40

48

56

Layer 20

0

8

16

24

32

40

48

56

Layer 21

0

8

16

24

32

40

48

56

Layer 22

56

0

8

16

24

32

40

48

56

Layer 23

0.00

0.25

0.50

0.0

0.2

0.0

0.2

0.0

0.2

0.0

0.1

0.0

0.2

0.4

0.0

0.2

0.4

0.0

0.2

0.0

0.2

0.4

0.0

0.2

0.0

0.2

0.4

0.0

0.2

0.0

0.1

0.2

0.0

0.1

0.0

0.2

0.0

0.5

0.0

0.1

0.2

0.0

0.2

0.0

0.2

0.0

0.2

0.4

0.0

0.5

0.0

0.5

0.0

0.5

0.00

0.25

0.50

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 0

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 1

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 2

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 3

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 4

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 5

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 6

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 7

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 8

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 9

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 10

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 11

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 12

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 13

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 14

0

8

16

24

32

40

48

56

0

8

16

24

32

40

48

56

Layer 15

0

8

0

8

16

24

32

40

48

56

Layer 16

0

8

16

24

32

40

48

56

Layer 17

0

8

16

24

32

40

48

56

Layer 18

0

8

16

24

32

40

48

56

Layer 19

0

8

16

24

32

40

48

56

Layer 20

0

8

16

24

32

40

48

56

Layer 21

0

8

16

24

32

40

48

56

Layer 22

0

8

16

24

32

40

48

56

Layer 23

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

0

1

2

0

1

0

1

0

1

0

1

2

0

1

2

0

1

2

0

1

2

0

1

2

0

1

0

1

2

0

1

2

0

1

2

0

1

2

0

5

0

2

16

24

32

We provide empirical validation to support both of these assumptions.

For Observation 1 , we show W Q @ W K .T , W V @ W V .T , W D @ W D .T of all different layers of Qwen2.5-0.5B-Instruct in Fig. 11, 12, and 13. It can be seen that different layers have this orthogonal relationship. In fact, the orthogonal relationship of matrices in neural networks has been studied since a long time ago. In particular, [44] proposed a new regularization method that encourages the weight matrix of the neural network to maintain orthogonality during training by introducing a self-orthogonality module. This method helps to improve the training stability and generalization ability of the model. [45, 46, 47] explores adding orthogonal regularization to weights during training to improve training stability. The author proposed an orthogonal regularization method for weights, aiming to solve the gradient vanishing and explosion problems encountered by deep convolutional neural networks during training. It can be seen that modules with orthogonality are found in various different models to improve the training stability and performance of the model. To the best of our knowledge, we are the first work to intuitively show this orthogonal performance in LLM, which can be more fully explored in subsequent research.

For Observation 2 , this observation is illustrated in Fig. 14. An intuitive understanding is that if x i and x M activate more of the same neurons, A i and A M will have more positive values in common positions, making cos( A i , A M ) larger.

2700

2800

0.05

0.10

0.15

0.20

Cos(Ai,AM)

Layer 0

2700

2800

0.000

0.025

0.050

0.075

0.100

Cos(Ai,AM)

Layer 1

2700

2800

0.00

0.02

0.04

0.06

Cos(Ai,AM)

Layer 2

2700

2800

0.04

0.02

0.00

0.02

0.04

Cos(Ai,AM)

Layer 3

2700

2800

2900

0.00

0.05

0.10

Cos(Ai,AM)

Layer 4

2800

2900

3000

0.00

0.05

0.10

0.15

Cos(Ai,AM)

Layer 5

2800

3000

0.00

0.05

0.10

0.15

Cos(Ai,AM)

Layer 6

2800

3000

0.0

0.1

0.2

Cos(Ai,AM)

Layer 7

2800

3000

0.0

0.1

0.2

Cos(Ai,AM)

Layer 8

2800

3000

0.0

0.1

0.2

Cos(Ai,AM)

Layer 9

2800

3000

3200

0.0

0.1

0.2

Cos(Ai,AM)

Layer 10

2750

3000

3250

0.0

0.1

0.2

0.3

Cos(Ai,AM)

Layer 11

2800

3000

3200

0.0

0.1

0.2

0.3

0.4

Cos(Ai,AM)

Layer 12

2750

3000

3250

0.0

0.1

0.2

0.3

0.4

Cos(Ai,AM)

Layer 13

2750

3000

3250

0.0

0.1

0.2

0.3

Cos(Ai,AM)

Layer 14

2750

3000

3250

0.0

0.2

0.4

Cos(Ai,AM)

Layer 15

2750

3000

3250

0.0

0.2

0.4

Cos(Ai,AM)

Layer 16

2750

3000

3250

0.0

0.2

0.4

0.6

Cos(Ai,AM)

Layer 17

3000

3500

0.0

0.2

0.4

Cos(Ai,AM)

Layer 18

2750

3000

3250

0.0

0.2

0.4

Cos(Ai,AM)

Layer 19

3000

3500

0.0

0.2

0.4

0.6

Cos(Ai,AM)

Layer 20

2750

3000

3250

0.0

0.2

0.4

Cos(Ai,AM)

Layer 21

3000

3500

0.0

0.2

0.4

0.6

Cos(Ai,AM)

Layer 22

3000

3500

0.0

0.2

0.4

Cos(Ai,AM)

Layer 23

3000

3500

0.0

0.2

0.4

Cos(Ai,AM)

Layer 24

3000

3500

0.0

0.2

0.4

0.6

Cos(Ai,AM)

Layer 25

3000

3500

0.0

0.2

0.4

Cos(Ai,AM)

Layer 26

2750

3000

3250

0.0

0.2

0.4

0.6

Cos(Ai,AM)

Layer 27

3000

3500

0.2

0.4

Cos(Ai,AM)

Layer 28

3000

3500

0.1

0.2

0.3

0.4

0.5

Cos(Ai,AM)

Layer 29

3000

3250

3500

0.2

0.4

0.6

Cos(Ai,AM)

Layer 30

3000

3500

0.2

0.4

0.6

Cos(Ai,AM)

Layer 31

Theoretical Insight 1. Within an attention block, the degree of interaction between two tokens is governed by the relative angle of their input hidden states.

32

40

48

56

Justification. For a single token, suppose its input to the Attention block is y . Consider a sequence of tokens [ y 1 , y 2 , . . . , y M ] , for token y m , its computation in the Attention layer can be expressed as:

where α im is the attention score between the i -th and the m -th token. O m is the output vector of the m -th token. To examine the influence of the i -th token y i on the final output O m of the m -th token, we can consider the following projection value:

where ∥ ∥ Proj O m ( α im V i )∥ ∥ is the projection value of α im V i on O i . And cos ( ∠ ( V i , O m ) ) is the cosine value of the angle between the V i and O m vectors. From Eq. 11, O m is a sum of vectors in different directions. Since the self-attention score α mm is typically much higher than α im for other tokens, we can simplify the projection by assuming that O m is primarily determined by α mm V m , i.e.,

where α im = Softmax ( (ˆ y i W q ) ( ˆ y m W k ) T / √ d ) . Since the Softmax function is monotonic, that is, Softmax ( x ) ∝ x . We can have α im ∝ (ˆ y i W q ) ( ˆ y m W k ) T . Therefore,

Based on the Observation 1, W q W T k ≈ θI , W v W T v ≈ λI . Therefore, combining Eq. 13 and Eq. 14,

Given that ˆ y is the result of y after LayerNorm, ˆ y and y share the same direction, and ∥ ˆ y ∥ = 1 , it holds that ⟨ ˆ y i , ˆ y m ⟩ = cos ( ∠ ( y i , y m )) . We can have

Eq. 19 shows that the angle between different tokens directly affects their mutual interaction in attention layer. The closer the angles of two tokens, the greater their mutual influence.

We also demonstrate that the computation of activations is influenced by the angular relationships between hidden states. This justification is presented as Theoretical Insight 4 in Section B.3.

B.2 Attention-Based Explanation of Layer-wise Angle Concentration Patterns

In this section, we present an attention-based explanation of Layer-wise angular concentration patterns. Specifically, we theoretically show that: (1) the degree of angular concentration between two hidden states influences the magnitude of their attention scores, and (2) the presence of sink attention structure encourages angular concentration among tokens.

Theoretical Insight 2. The smaller the angle between the input hidden states of two tokens to the attention block, the higher their corresponding attention score.

Justification. For two tokens i and j , let their inputs to an attention block be denoted as x i and x j , respectively. Then, their attention score α ij can be expressed as:

Based on the Observation 1, W q W T k ≈ θI , Therefore,

Furthermore, since LayerNorm preserves the direction of vectors by normalizing only their magnitude, the inner product between normalized outputs satisfies ⟨ ˆ y i , ˆ y j ⟩ = cos( ∠ ( y i , y j )) ,

Equation 19 indicates that the more aligned the hidden states of two input tokens are (i.e., the smaller the angle between them), the higher their attention score.

Therefore, in conjunction with the layer-wise angle concentration pattern discussed in the main text, we observe that inter-segment angular concentration reflects the model's degree of attention to internal components of the problem, while intra-segment angular concentration captures the level of attention between the question and the system prompt-serving as an indicator of the model's instruction-following capability.

Theoretical Insight3. Presence of sink attention promotes angular concentration among hidden states.

Layer 0

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

Layer 11

Layer 12

Layer 13

Layer 14

Layer 15

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

Layer 21

Layer 22

Layer 23

0.00

0.01

0.02

0.03

Value

0.000

0.005

0.010

0.015

0.020

0.025

0.030

Value

0.00

0.01

0.02

0.03

Value

0.000

0.005

0.010

0.015

0.020

0.025

Value

0.000

0.005

0.010

0.015

0.020

0.025

Value

0.000

0.005

0.010

0.015

0.020

Value

0.000

0.005

0.010

0.015

0.020

0.025

0.030

Value

0.000

0.005

0.010

0.015

0.020

Value

0.00

0.01

0.02

0.03

Value

0.000

0.005

0.010

0.015

0.020

Value

0.000

0.005

0.010

0.015

0.020

0.025

Value

0.000

0.005

0.010

0.015

Value

0.000

0.005

0.010

0.015

0.020

0.025

0.030

Value

0.000

0.005

0.010

0.015

0.020

0.025

Value

0.000

0.005

0.010

0.015

0.020

0.025

0.030

Value

0.000

0.005

0.010

0.015

0.020

0.025

0.030

Value

0.000

0.002

0.004

0.006

0.008

0.010

Value

0.000

0.005

0.010

0.015

Value

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

0.0150

Value

0.000

0.005

0.010

0.015

Value

0.000

0.005

0.010

0.015

Value

0.0000

0.0025

0.0050

0.0075

0.0100

0.0125

Value

0.00

0.01

0.02

0.03

Value

0.00

0.01

0.02

0.03

Value

■

■

Justification. To understand why token angles exhibitLayer-wise angular concentration patterns, we start our analysis from the phenomenon of sink attention , which similarly shows segment-wise characteristics yet remains insufficiently understood. In LLMs, attention scores exhibit segment-wise tendencies, and, apart from self-attention, attention scores typically peak at the first token of each segment. This phenomenon is termed sink attention , as illustrated in Figure 15.

Given an input representation for an attention block in an LLM as y ∈ R m × d , where the i -th token is denoted as y i , the attention mechanism output is defined as O = α V , where α represents the attention scores, and V = LayerNorm ( y ) W v denotes the value vectors. Suppose the attention scores within a segment 'sink' to the first token vector y i , then the outputs for the sink token y i and another token y k within the same segment can be approximated as:

where α ik denotes the attention score between tokens i and k . This approximation arises because the sink token's self-attention score significantly surpasses its attention scores to other tokens, while other tokens primarily attend to themselves and the sink token.

Substituting these approximations, the angle between outputs O i and O k can be expressed as:

Furthermore, since W v is approximately an orthonormal matrix (detailed proof in the appendix) and thus preserves angles, we have W v W ⊤ v ≈ β I , where β is a constant. Combined with the fact that

LayerNorm scales only magnitudes without altering angles, we get:

Substituting into the earlier expression, we derive:

Squaring both sides and simplifying, we arrive at the condition:

Since weight parameters in LLMs are typically constrained to values less than 1, both β < 1 and β (cos( ∠ ( y i , y k ))) 2 < 1 generally hold. Thus, Equation (7) demonstrates that sink attention inherently promotes angle concentration within segments. A detailed derivation is provided in the appendix.

In Figure 15, we present the distribution of attention scores across different layers. In intermediate layers, sink tokens operate primarily within segments to enhance intra-segment angle concentration. In later layers, sink tokens across segments begin to interact, promoting inter-segment concentration. This indicates that, due to the influence of sink tokens, token angles are increasingly concentrated through the forward pass. The final layer's angle concentration is particularly important as it reflects the culmination of this process and directly determines the model's output.

B.3 Neuron-Based Explanation of Data-wise Angle Concentration Patterns

In Section 2.3 of the main text, we demonstrate that the greater the number of tokens activating the same neuron, the more gradient components that neuron receive. In this section, we further show that the number of commonly activated neurons between tokens has a direct effect on the angle between their output hidden states at the current layer.

Theoretical Insight 4. Within the FFN block, the extent of overlap in activated neurons between two tokens directly affects the angular relationship between their output hidden states.

Justification. To analyze how the activation layer affects cos( ∠ ( y i , y M )) , we first decompose its computation formula. Suppose the activation output of the i -th token is A i , and y i = A i W d . Based on Observation 1, W d is a scalar multiple of a unitary self-orthogonal matrix, applying the same rotation to any input while preserving the inner product and angle between any two input vectors. Thus, we can have:

where η is a constant based on Observation 1. Furthermore, since W d is an orthogonal matrix,

which means ∥ y i ∥ = √ η ∥ A i ∥ . Substituting this into Eq. 20 we can have

This shows that the orthogonal matrix W d does not change the angles between the input vectors. Furthermore, based on Observation 2, cos ( ∠ ( A i , A M )) ∝ ∣ ∣ Γ( x i ) ∩ Γ( x M ) ∣ ∣ , we can get

which is consistent with Insight 2.

Eq. 23 shows that activation layers adjust token angles by controlling the intersections of their activated neurons. More shared activated neurons lead to smaller angles and greater mutual influence.

C Visualization Results

In the main text, we presented layer-wise, epoch-wise, and data-wise patterns of angular concentration. In this section, we provide more comprehensive visualizations to further support our conclusions.

C.1 Layer-wise Angle Concentration Patterns

In Fig. 16, 17 and 18, we present the layer-wise angular concentration patterns across all layers of the Qwen2.5-0.5B-Instruct model for tasks of easy, medium, and high difficulty, respectively. Notably, the model consistently demonstrates first intra-segment angle concentration and subsequently intersegment angle concentration-regardless of problem difficulty. This consistency suggests that the observed pattern is a generalizable property of the model's internal representation dynamics.

Layer 0

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

Layer 11

Layer 12

Layer 13

Layer 14

Layer 15

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

Layer 21

Layer 22

Layer 23

0.4

0.2

0.0

0.2

0.4

0.6

Value

0.0

0.1

0.2

0.3

0.4

Value

0.1

0.2

0.3

0.4

0.5

0.6

Value

0.1

0.2

0.3

0.4

0.5

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.1

0.2

0.3

0.4

Value

11

840.

801

8832

1

■

.0

011

153181

Layer 0

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

Layer 11

Layer 12

Layer 13

Layer 14

Layer 15

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

Layer 21

Layer 22

Layer 23

0.4

0.2

0.0

0.2

0.4

0.6

Value

0.0

0.1

0.2

0.3

0.4

Value

0.1

0.2

0.3

0.4

0.5

0.6

Value

0.1

0.2

0.3

0.4

0.5

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.1

0.2

0.3

0.4

Value

0081

Layer 0

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

Layer 11

Layer 12

Layer 13

Layer 14

Layer 15

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

Layer 21

Layer 22

Layer 23

0.4

0.2

0.0

0.2

0.4

0.6

Value

0.0

0.1

0.2

0.3

0.4

Value

0.1

0.2

0.3

0.4

0.5

0.6

Value

0.1

0.2

0.3

0.4

0.5

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

0.6

Value

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

0.5

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.0

0.1

0.2

0.3

0.4

Value

0.1

0.2

0.3

0.4

Value

1

C.2 Data-wise Angle Concentration Patterns

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

intra+

inter

0

20

40

60

80

100

120

Number of Sample

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

intra+

inter

Epoch 25

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

intra+

inter

Epoch 50

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

intra+

0

20

40

60

80

100

120

Number of Sample

Epoch 75

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

intra+

Epoch 100

0

2

4

6

8

10

Avg. Correct (Max=10)

inter inter

D GAIN-RL Algorithm

Algorithm 1: GAIN-RL (GRPO): Gradient-driven Angle-Informed Navigated RL Framework

E Experimental Setup

In this section, we describe our experimental setup in detail, covering the models, datasets, and hyperparameters used.

E.1 Model and Dataset

To comprehensively evaluate the effectiveness of GAIN-RL, we conduct experiments across multiple models and datasets. Specifically, we select models varying in size, including Qwen2.5-0.5bInstruct, Qwen2.5-Math-1.5B-Instruct, Qwen2.5-Math-7B-Instruct, Qwen2.5-Coder-3B-Instruct, and LLaMA3.2-3B-Instruct. We primarily focus on two tasks: Math and Code. To evaluate the training efficiency of GAIN-RL (Section 4.1 in the main text), we use the DeepScaleR [24] dataset for mathematical task training and DeepCoder [25] for coding tasks training, each integrating problems from diverse sources and covering a wide range of difficulty levels. For mathematical evaluations, we employed six benchmark datasets of varying difficulty: GSM8K [3], MATH [4], AMC 23 [18], AIME 24 [19], OlympiadBench [20], and Minerva Math [48]. For coding evaluations, we utilized three standard benchmark datasets: LivecodeBench (8/1/24-2/1/25) [6], Codeforces [22], and Humaneval+ [23]. For other experiments (Section 4.2-Section 4.5), we train model on the training dataset of single tasks including GSM8K, MATH and AMC 23 to facilitate more convenient comparisons.

E.2 Training Configuration

We trained the models using the GRPO algorithm. The training was performed on a single node equipped with 8 A100 GPUs. Each model was trained for about 200 steps using the veRL library.

To evaluate the training efficiency on GRPO-RL, the main training configuration for Qwen2.5Math-7B-Instruct is shown below. For Qwen2.5-Math-1.5B-Instruct and LLaMA3.2-3B-Instruct, we set max_response_length=3000 to accommodate its shorter context window of 4096 tokens, while keeping all other parameters unchanged. For Qwen2.5-0.5B-Instruct and single task training, we set max_prompt_length=max_response_length=512 , while keeping other parameters unchanged. For Qwen/Qwen2.5-Coder-3B-Instruct, we set max_prompt_length=2048 , max_response_length=16384 , train_batch_size=512 and ppo_mini_batch_size=64 due to its higher single sample memory usage.

F Additional Experimental Results

In this section, we present additional experiments to further validate the effectiveness of GAIN-RL.

F.1 Performance of Weighted Signals

In the main text, we employed an unweighted angular signal of the form: C M ( d i ) = C M intra ( d i ) + C M inter ( d i ) . Here, we explore a weighted variant of the signal: C M ( d i ) = C M intra ( d i ) + c · C M inter ( d i ) , where c is a constant weight.

As shown in Tab. 4, we investigate the final performance of the Qwen2.5-0.5B-Instruct model after 200 training iterations on the Math dataset, using different values of c in the weighted signal. The results show that setting c = 4 . 0 yields the best performance for this model, suggesting that carefully designed angle-based signals can further improve training effectiveness.

However, selecting the optimal weighting coefficient c requires extensive empirical tuning. To ensure scalability and practical applicability, we adopt the unweighted version of the signal in this work and leave signal optimization for future research. Our goal is to demonstrate that even without finely tuned weighting , GAIN-RL is still capable of accelerating both training and data efficiency-highlighting the strong potential of model-signal-based RLHF strategies.

F.2 Model Performance on Single Task

Fig. 9 in the main text illustrates the training dynamics of Qwen-2.5-0.5B-Instruct on three single-task datasets. For a more detailed comparison, Tab. 5 reports the final performance at iteration 200 and the corresponding training speedup across different training sets and methods. Notably, on the GSM8K dataset, GAIN-RL outperforms the original GRPO by 4.72% in final accuracy and achieves a 3.3 × improvement in training speed.

These results demonstrate that GAIN-RL can effectively distinguish between samples of varying learnability even in the single-task setting, highlighting its efficiency and general applicability.

G Discussion and Future Work

In Section 3, we demonstrate that angles between token hidden states fundamentally mirror and influence both the information propagation during inference and the learning dynamics throughout model training. The proposed angle-based signals can, in fact, be generalized beyond RFT to enhance model-centric effectiveness in various other domains. For instance, during pre-training, monitoring angle signals could enable real-time evaluation of a model's learning capacity across different domains, thus allowing adjustments to training data to improve stability and final performance. Furthermore, during inference, tracking changes in angle concentration between layers could provide insights into the model's comprehension of inputs and indicate whether additional test-time adjustments are necessary to boost output accuracy. In future work, we plan to further investigate how this signal can be leveraged across multiple domains to achieve comprehensive, model-centric optimizations.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The motivations and contributions are well depicted and summarized in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not to penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theory assumptions and proofs are described in Section 2 and Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The selected models and benchmarks are clearly and fully presented in Section 4 and Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are carefully presented in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the accuracy.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information of the employed compute resources is elaborated in the implementation details of Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We make sure that the research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We discuss it in the Appendix

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all papers that we used for experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: This paper describe the usage of LLMs in code and math tasks, as mentioned in Section 1.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors present an interesting observation on LLM training dynamics: the Qwen2.5-0.5B-Instruct model prioritizes learning samples with high angle score ($C_{intra}+C_{inter}$) and progressively learns lower-score samples. The angle score has both theoretical and empirical connection to the attention score between question tokens and between question tokens and instruction/example tokens.\n\nWith this observation in hand, the paper proposes a novel framework that skews the sample distribution to favor those with high angle score, updating the distribution to lower scores as training progresses. They were able to achieve significant improvements in sample efficiency with virtually no computation overhead made in the dataset preprocessing (sorting) phase.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The main observation/motivation is interesting, and is supported by theoretical and empirical evidences.\n- The experimental setup is clear and straighforward, and their results support their aforementioned observation across diverse LLM models and sizes.\n\nWeaknesses\n- I would have been nice to see experiment on much larger models (at least on few menchmarks), although I believe 7B meets the standard.\n- Figure8 seems to reveal that angle-biased sampling is a more simple yet superior method to GAIN-RL, which I think needs more explanation than what's said on line 278-280.\n\nQUESTIONS:\n1. As stated in the weaknesses, \"High AngConc-biased Sampling\" seems to perform better than GAIN-RL and is much simpler (e.g., less hyperparameters). I cannot help but ask: why not just use that?\n\n2. In eq8, how is $\\sigma_t$ tuned?\n\n3. Looking at eq10, it seems like $\\mu_t$ starts from 0 then increases indefinitely. Are there guarantees that $\\mu_t$ doesn't exceed $n$, or is it explicitly forced to be under $n$?\n\n4. As $\\mu_t$ increases, would samples with low index (high angle scores) eventually have zero probability? If so, wouldn't it cause issues for large datasets such as catastrophic forgetting?\n\nLIMITATIONS:\nAuthors do discuss future work in Appendix.G but not specifically their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel RFT framework for large language models, called GAIN-RL, which enhances training efficiency by leveraging a model-intrinsic signal called *angle concentration*. This metric quantifies the model’s potential to learn from a given prompt. It is both model-based and computationally efficient, requiring no reliance on heuristic measures of question difficulty. Empirical results demonstrate that angle concentration is a measurable and effective signal for guiding reinforcement learning.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses an important and interesting question, i.e., how to improve sample efficiency and accelerate training in reinforcement learning for large language models. It is widely recognized that selecting appropriate prompts at different stages of training plays a critical role in enhancing both the speed and robustness of RL. The authors introduce a novel, model-based metric that avoids expensive data preprocessing and can effectively guide a new training strategy, GAIN-RL. This strategy is shown to outperform vanilla RL methods based on uniform sampling. Sufficient results confirm the effectiveness of GAIN-RL. \n\nWhile the paper is well-written and clearly structured, I still have some suggestions and questions about the motivation part, i.e. section 2. \n1. The role of the gradient norm could be clarified. While I understand that small gradients might suggest proximity to a solution, it is less obvious why large gradients necessarily imply the opposite. As a first-time reader, I found it unclear why the Frobenius norm of the gradient should be the primary focus.\n2. Equation (4) does not clearly isolate the role of the angle in $l_2$ space. It is not fully explained why the norm of the hidden states can be disregarded. Are these norms assumed to be fixed? If not, then both the angle and the magnitudes contribute to the Frobenius norm. Moreover, it seems to me that the angle in equation 4 comes from the Frobenius norm. If another norm were used, then the angle in the $l_2$ norm space doesn’t always show explicitly in the equation.\n\nI suggest the authors revise this section to provide more intuitive explanations and to make the motivation more accessible to a broader audience.\n\nQUESTIONS:\n1. Why is Gaussian probability applied? Is it because Gaussian is very light tailed such that only certain $d_i^s$s will be chosen with high probability? More explanation would be helpful.\n2. What’s $\\sigma_t$ in (8)?\n3. Is this a typo?: the left equation in Eq 8 only includes $i$ instead of $d_i^s$.\n\nLIMITATIONS:\nThe observations regarding angle concentration are primarily based on the 0.5B model. It would strengthen the claims and improve the generalizability of the findings if similar analyses were also conducted on larger models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GAIN-RL, a novel Reinforcement Learning Fine-tuning (RFT) framework designed to address the sample inefficiency and high computational costs in training Large Language Models. The core insight is the identification of \"angle concentration\" – the cosine similarity between token hidden states – as a model-inherent signal that effectively reflects an LLM's learning capacity. GAIN-RL leverages this signal by reordering training data based on initial angle concentration and dynamically sampling data during training using a Gaussian distribution, with the mean updated by real-time accuracy and angle signals. Experimental results demonstrate that GAIN-RL significantly accelerates training and achieves better performance with less data compared to vanilla GRPO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tThe paper addresses a critical problem of sample inefficiency and high computational costs in LLM RFT, proposing a novel idea of using the model's inherent \"angle concentration\" signal for data manipulation.\n\n2.\tThe paper provides strong support for angle concentration as a learning signal through both theoretical reformulation of gradients and empirical observations of various angle concentration patterns.\n\n3.\tExperimental results show significant performance and efficiency gains, with GAIN-RL (GRPO) achieving over 2.5x training acceleration and superior performance with half the data.\n\n4.\tThe framework demonstrates good generalizability, presented as a plug-and-play solution compatible with different models and RL algorithms, with minimal preprocessing cost.\n\n5.\tClear ablation studies effectively demonstrate the contribution of each component, and small-batch scalability tests further validate the method's robustness.\nWeaknesses\n1.\tThe claim of \"no additional cost\" for computing accuracy and angle concentration for probability updates requires more detailed justification across various training setups to ensure consistent fairness.\n\n2.\tThe paper's strong results with \"High Angular Concentration-biased Sampling\" (using half the data) need more nuanced discussion regarding the broader implications of potentially excluding certain data points.\n\n3.\tThe comparison with existing curriculum learning methods should be broader and more deeply analyze GAIN-RL's advantages over a wider range of strategies.\n\n4.\tThe paper contains several minor typos and inconsistent formatting details that, while not severely impacting understanding, should be addressed for improved professionalism.\n\nQUESTIONS:\nsee in previous comment\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-07"}