In Good GRACES: Principled Teacher Selection for Knowledge Distillation
Authors: Abhishek Panigrahi
Bingbin Liu
Sadhika Malladi
Sham M. Kakade
Surbhi Goel
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge distillation
Directional coverage
Gradient variance
Cross Validation
Best Teacher prediction
Primary area: foundation or frontier models, including LLMs
TL;DR: GRACE is a gradient-based score that efficiently predicts the best teacher for knowledge distillation, without requiring teacher internals or test data

Abstract: Knowledge distillation is an efficient strategy to use data generated by large
“teacher” language models to train smaller capable “student” models, but selecting
the optimal teacher for a specific student-task combination requires expensive
trial-and-error. We propose a lightweight score called GRACE to quantify how
effective a teacher will be for post-training a student model. GRACE measures
distributional properties of the student’s gradients without access to a verifier,
teacher logits, teacher internals, or test data. From an information-theoretic
perspective, GRACE connects to leave-one-out stability of gradient-based
algorithms, which controls the generalization performance of the distilled students.
On GSM8K and MATH, GRACE correlates strongly (up to 86% Spearman
correlation) with the performance of the distilled LLaMA and OLMo students.
In particular, training a student using the GRACE-selected teacher can improve
the performance by up to 7.4% over naively using the best-performing teacher.
Further, GRACE can provide guidance on crucial design choices in distillation,
including (1) the best temperature to use when generating from the teacher, (2)
the best teacher to use given a size constraint, and (3) the best teacher to use within
a specific model family. Altogether, our findings demonstrate that GRACE can
efficiently and effectively identify a strongly compatible teacher for a given student
and provide fine-grained guidance on how to perform distillation.

Published as a conference paper at ICLR 2026
IN GOOD GRACES: PRINCIPLED TEACHER SELECTION FOR KNOWLEDGE DISTILLATION
Abhishek Panigrahi' Bingbin Liu
Sadhika Malladi
Sham Kakade
Surbhi Goel
Princeton Language and Intelligence
" "Kempner Institute, Harvard 'University of Pennsylvania
# Microsoft Research, New York
ABSTRACT
Knowledge distillation is an efficient strategy to use data generated by large "teacher" language models to train smaller capable "student" models, but selecting the optimal teacher for a specific student-task combination requires expensive trial-and-error. We propose a lightweight score called GRACE quantify how effective a teacher will be for post-training a student model. GRACE measures distributional properties of the student's gradients without access to verifier, teacher logits, teacher internals, or test data. From an information-theorele perspective, GRACE connects leave-one-out stability of gradient- based algorithms, which controls the generalization performance of the distilled students. On GSM8K and MATH, GRACE correlates strongly (up to 86% Spearman correlation) with the performance of the distilled LLaMA and OLMo students. In particular, training a student using the GRACE-sclected teacher can improve the performance by up 7.4% naively using the best- sest-peffrming teacher. Further, GRACE can provide guidance on crucial design choices in distillation, including (1) the best temperature to when generating from the teacher, (2) the best teacher to use given a size constraint, and (3) the best teacher to use within specific model family. Altogether, our findings demonstrate that GRACE efficiently and effectively identify a strongly compatible teacher for given student and provide fine-graine guidance on how perform distillation.
INTRODUCTION
Distillation is an efficient and effective method to produce capable small models from existing, powerful teacher models. In this work, we focus on the specific case of training autoregressive language models on text generated by a teacher model. It is difficult to select the right teacher for given student and task: counterintuitive fact is that a better teacher, which has been observed in classic classification/regessss al., 2019; Harutyunyan al., 2023; Panigrahi al., 2025) and more recently in the context of language models (Zhang et al., 2023; 2025a; Peng et al., 2025; Razin et al., 2025). Given the large number of available models as potential teachers, the current approach of guess-and-check costly, because it requires collecting generations from capable teacher and subsequently training a student on those generations. Additionally, the specific hyperparameters used in both phases can dramatically affect the final performance of the student, underscoring the need for careful, repeated testing to select the right teacher. As such, the current work seeks to address the following question:
model is not always settings (Mirzadeh
Given pool of candidates, can efficiently identify the best teacher for given student and task?
We argue that identifying good teacher requires jointly considering the teacher and the student. To this end, we study the teacher's influence on the student's optimization process by analyzing the student's 's gradients. We propose score, GRACE (GRAdient Cross-validation Evaluation), which captures the distributional properties of the student's gradients small set of teacher-generated data, enabling efficient and effective identification of the most compatible teacher (Section 2.2). Inspired by prior data selection and distillation works, GRACE unifies data diversity and teacher-studes
Correspondence to: ap34 noneon. edu, b1 fug. harvard.ee edu. Code is available at https: / thub. com/ abbi shekpanigras grahi 199999//AAC

---

Published as a conference paper at ICLR 2026
1.0
LLaMA-1B training on GSM8K
Spearman Correlation 1
GSMBK
0.76
MATH
0.74
0.86
0.88
Performance Regret (in %)
15
performance
Teacher
14.47
Loss
G-Vendi
0.6
10
GRACE
0.44
0.44
0.4
5
4.22 16,
4.89
4.99
3.65
0.2
9.032.99
0.11
0.27
0.22
0676
0.0
Teacher performance
Loss
G-Vendi (Jung et al.) (Ours)
GRACE
LLaMA
Qwen Teacher Family
OLMo
Figure 1: GRACE correlates with student performance after distillation on math-related reasoning tasks. We evaluate LLaMA-1B 1-B and LLaMA-3B students on GSM8K and MATH respectively with 15 teachers across LLaMA, Gemma, Qwen, OLMo, and Phi families. (Left) GRACE shows the strongesi correlation with student accuracy among four scores performance, student's loss (before training) on the teacher generations, G-Vendi, and GRACE. (Right) GRACE reliably selects near-optimal teachers within euch leacher family, measured by its small teacher-studen regret, which the absolute gap in final performance between the best overall student and that obtained from the teacher chosen by each score. Performance is measured by average accuracy over 16 generations per prompt.
alignment desiderata into a single score. Specifically, GRACE leverages a cross-validation structure, where it takes gradients from one split of data and measures their magnitudes re-weighted under the spectrum of the other split. The gradient spectrum reflects data diversity, while the gradient norm encodes teacher-student alignment. This vooss-validttion structure also allows us draw a natural connection to conditional mutual information- bbadd generalization bounds (Steinke & Zakynthinou, 2020; Rammal et al., 2022), providing insight into why GRACE works (Lemma 1). Importantly, GRACE is efficient to compute and does not rely on access to an external verifier, teacher logits, teacher representations, or test data.
We conduct thorough experiments to verify that the GRACE scores of teachers accurately reflect the student's 's performance when trained on those teachers. We focus on math-related datasets, namely GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), because the broad community interest in mathematical reasoning has led to the development of a large, diverse set of teachers that are readily available and suitable for distillation. We train LLaMA- 1B-BBBB- 1B- Base, OLMo- 1e---B--Baaa Base and Gemma- 2B-Base (for GSM8K) as well as A-LMM-----BB 15 candidate teachers drawn from the LLaMA (Team, 2024c), OLMo (OLMo, 2024), Qwen (Qwen et al., 2025), Gemma (Team, 2024b), and Phi (Abdin et al., 2024) families. Our results show that:
(for MATH) using generations sampled from
GRACE correlates strongly with the student's distillation performance (Figure 1, left), outperform- ing strong baselines such G- Vendi (Jung al., 2025).
GRACE identifies teachers that are strongly compatible to given student. Selecting teachers using GRACE yields 7% and 2% improvement in student performance compared to using the best best-performing teacher, on GSM8K and MATH respectively.
GRACE offers actionable insights to practitioners. It helps identify close to 1) optimal generation temperature for given teacher model, 2) best teacher within size constraint, and 3) best teacher within model family.
Given the limited research on teacher selection in LLM distillation, our findings indicate that GRACE improves performance and offers reliable guidance for effective distillation.
2 GRACE: GRADIENT CROSS- VALIDATION EVALUATION
We consider the case of using distillation to fine-tune pre-trained student model to solve specific downstream tasks. For each of the N prompts x E X. we autoregressively generate M responses ,,,.., PM from teacher distribution #7. This distribution encodes the temperature may be
2

---

gradient information to Published as a conference paper at ICLR 2026
sampled at from the teacher well. We then fine-tune the pre-trained student with the standard autoregressive cross-entropy objective L on dataset Disiill containing N x M teacher generations. In contrast to logit-based based distillation, learning from generations permits distillation across architectures and in cases where the teacher's logits are not available. We measure the performance of students and teachers as the average accuracy of l sampled responses for given prompt (i.e., average-at-k) We will use Ts to denote the pre-trained student, and refer to its parameters as θş E RD when necessary.
2.1 GRADIENT-BAR BASED SCORES
The problem of selecting a teacher for distillation is closely connected to the well-studied field of data selection: choosing the best teacher based its generations can be viewed selecting the best subset from the union of all teachers' generations, with the constraint that each subset must come from single teacher. For language models, many successful data selection methods (Pruthi et al., 2020; Xia et al.. 2024; Engstrom et al., 2024) rely on first- or identify useful data for given task.
In contrast to selecting individual datapoints from dataset, we would like to select data distribution (i.e., corresponding
teacher). As such, instead of quantifying the value of individual datapoints, we turn our attention to gradient-baser based approaches to measure data quality in terms of its distributional features. For a teacher Ty, we assume access to only a subsampled dataset Deyal C D containing prompt-generation pairs, where n,, may be much smaller than N, M. In our experiments (Section 3), n x m is 60x smaller compared to the x M.
In the following, we will first describe two baselines (i.e., G-Vendi (Jung et al., 2025) and G-Norm), then introduce the proposed GRACE score combining the advantages of both. The two baselines measure data diversity (via directional entropy) and teacher-student alignment (via gradient norms). GRACE constructs a single score that balances these two desiderata by penalizing large gradients along low-eigenvalue directions.
Gradients. We establish some useful notation to work with gradients. Let g(x,y) [y [yy)x; Qs) be the student's gradient on the response y conditioned on prompt x, which is averaged over response tokens, following prior works on data selection. Since all gradients are computed with respect to the student model's parameters, we omit the explicit dependency on Oş for notational clarity.
We process the gradient with two steps. First, for computational reasons, we work with a random low-dimensional projection of the gradient, denoted Ilg E Rd with II E {#1/vD) D dx (Park et al., 2023). We also rescale the gradient by log |y| to account for the response length |y This is motivated by the empirical observation that the gradient norm averaged over length-T sequence roughly decreases as 1/ log T (Figure 28), which can cause gradient-bart computations to unduly favor short sequences (Xia et al., 2024).
The processed gradient is denoted h(x. := log(ly)) [lg(xy)) For a dataset 'D of generations, we define the matrix consisting of processed gradients (i.e., h) as G(D) E Ramxd and define the matrix consisting of processed and normalized gradients (i.c., h = h h/|l as G(D) E R R Then, we define the mean and second moment of the unnormalized gradients as:
1 G
1 (D)
= Tum
E(D) : : G(())
nmm
((D)) EE(D) are defined similarly. We additionally use Σ(()) = ε(D) + I with smoothing parameter v > 0 for numerical stability.
Gradient information has been used for data selection, where samples with higher influence to a validation set is preferred (Pruthi et al., 2020; Xia al., 2024). One can adapt these methods for teacher selection. However, we find them to be not strongly predictive empirically; we provide details in Section C. Consequently, we evaluate training data quality by analyzing its distributional properties, without relying on validation set.
G-Vendi (Jung et al., 2025). One natural distributional measure of data quality is diversity. Along these lines, Jung et al. (2025) propose the G-Vendi score, which measures the directional coverage of
3

---

Published as a conference paper at ICLR 2026
D as the entropy of the eigenvalues of E(D) (Equation (1)):
G- G-Vendi(D) Entropy(X(E(D))) (EA(E(D))
A log A,
(2)
where A( A(E(D) denotes the set of eigenvalues of E(D), with (A(E min (nm, d}. A larger G- G-Vendi score is preferred it corresponds better diversity. Jung et (2025) use Genddi to select an optimal subset of training data D from full dataset generated by single teacher. However, using G- Vendi to select teacher out of many candidates may yield suboptimal choices. For example, when the student serves its own teacher, we find that its G-Vendi score (5.93) is higher than that of all other teachers (Figure 2), even though the self-distilled student has a low accuracy of 4%. This occurs because untrained model can produce random responses, which exhibit high gradient entropy and therefore a high G-Vendi score, yet offer no meaningful learning signal for distillation.
G-Norm. The limitation of Vendi leads us to investigate another gradient- based distributional score, namely the gradient-norm score (G-Norm):
G-Norm ()) = Tr (E(D)
Σ (x.))ED
[b(xx)))l
THT
Because gradients become smaller near an optimum, a lower G-Norm indicates that the student requires less parameter updates to reduce loss on the given task. G-Norm therefore serves as proxy for teacher-student alignment. This explains why gegh-performing models may be poor teachers: although Gemma-2 Instruct models achieve strong accuracy GSM8K and MATH, the student model shows high G-Norm on their generated responses suggesting weak teacher-studen alignment and shows weaker post-distillation performance (Figure 2). For students which have been pretrained, G-Norm can additionally reflect the teacher's ' generation quality, as G-Norm is high when a teacher provides uninformative random responses. In particular, using the student itself as the teacher results in the highest G-Norm, showing that its outputs do not provide a meaningful distillation signal.
Nevertheless, G-Norm alone remains insufficient. For instance, GSM8K, find that G-Norm's value largely differentiates different teacher families and does not correlate with the student's perfor- mance when generation temperature from the teacher is varied (Figures 2 and 5). This is because G-Norm captures only the magnitude of gradients, without considering the directional spread of the gradients across dimensions.
G-Norm and G-Vendi capture complementary distributional properties and can sometimes trend in different directions. For instance, we find that increasing the teacher's generation temperature increases G-Norm, consistent with the observation that higher temperatures induce worse data, but the same time increases G- Vendi, indicating higher data diversity (Figure 5). As such, we treat G-Norm and G- Vendi as baselines and propose GRACE as single score that unifies both.
2.2 THE GRACE SCORE
GRAdient Cross-validation Evaluation (GRACE) computes the norm of the gradients weighted under the spectrum of the normalized gradient second- moment matrix. thereby combining the advantages of G- GeVeni and G-Norm. GRACE is computed solely using the student's gradients on the teacher's is gen- erations, and does not rely on a verifier or access to test samples. the following, we will first define the GRACE score, and then describe its connection to leave-one-out conditional mutual information.
GRACE. For a dataset DD of teacher generations containing n x m prompt-generation pairs and choice of hyperparameter C, construct C partitions of the prompts in the dataset D, denoted (D, | each containing n/C' prompts and their generations. Let D_, denote the concatenation of all partitions except the partition D,. Then, GRACE is defined as
GRACE(D) = Σ Tr ((D_,)-'((D)))) = 1 Σ Σ 122x (4) i=1
1
nm
(x,))D,
where E(D-,) = E((D.) I with smoothing parameter > 0 for numerical stability.
4

---

Published as a conference paper at ICLR 2026
GRACE computes the expected squared norm of opectral-weighter gradients. Given random partition (D,, D_1), let {A,, uj he(4 denote the set of eigenvalues and cigenvectors for (D-+i). For this partition, GRACE computes
Σ
y)(x,))T& 22)²
(5)
Di
jeld
(X,))ED,
bias-variance tradcoff. GRACE can be split into two terms:
GRACE(D)
Tr (T((((-)) G.())))
nm ((D) E(D_:)- ((D);
mm
CRRACEVariance(]]
GRACE-BiasD)
where GD) = GDD IT( denotes the centered gradient matrix. The term
GRACE-Variancee Variance(D) captures the variance in gradients under the spectrum of Σ. while GRACE-Bias(D) measures the norm of the mean gradient under the spectrum of Σ.
Repeating our intuitions from G-Norm, GRACE-Bias is highly useful for identifying pathologi- cal teachers: for example, if a teacher provides random responses, GRACE-Bias becomes large, indicating that such data not suitable for distillation.
When we do not encounter such teachers, most of the predictive power comes from GRACE-Variance. In our experiments, GRACE-Variane dominates GRACE-Bise and our conclusions remain the same whether we consider GRACE distillation teacher, as corresponds to a smaller variance of gradients along all eigenvectors of E. Gradient variance in directions with smaller eigenvalues are penalized more heavily, which is desirable as high gradient variance along such directions can more easily induce instability or poor generaliza- tion. We formalize this view in Section 2.2.1. The directional spectrum is taken from the normalized gradients, since the gradient norm is less relevant than the gradient direction with the use of adaptive optimizers and normalization layers (Loshchilov & Hutter, 2019; Ba al., 2016; Li al., 2022).
ACr--aaiee A smaller GRACE- Variiace indicates better
2.2.1 CONNECTING GRACE TO LEAVE-ONE-OUT CMI
GRACE connects naturally to leave-one-out conditional mutual information (CMI), frequently used concept in studying generalization (Xu & Raginsky, 2017: Steinke & Zakynthinou, 2020; Haghifam al., 2022; Rammal et al., 2022). CMI captures how sensitive the learning outcome is the removal of a single sample. A higher sensitivity suggests heavier memorization, which can lead to low generalization to unseen test examples. Under this framework, we show that GRACE successfully unifies G-Norm and G-Vendi.
Formally, we overload (); T (ixyjen((xxx) θ) denote the average gradient on dataset DD To keep our discussion general, we consider the following gradient update that uses gradients and preconditioner matrix M:
O - ()
where 77 denotes the learning rate and e (O, I'I) denotes the gradient noise. Setting M as identity recovers gradient descent, and setting M as a function of gradient second moments recovers various adaptive algorithms in practice.
Let O denote the resulting parameters after a gradient update with D, and ()(()))) denote the parameters from set where all training data connected to uniformly sampled prompt are dropped from the training set CMI measures the mutual information between the parameters (xD((x)))
and the dropped prompt x. We show that CMI can be bounded as follows:
Lemma (Informal; ef Theorem D.2). Define GRACE with = n. For any D', take M(D', A) S(D') 1/2, where then CMI S लम?? PRRACE-Varianne (D) < म. GRACE (D).
Thus, GRACE offers upper bound the student's generalization performance after training with single gradient update under the preconditioner matrix --1/ Intuitively, GRACE captures how
5

---

Published as a conference paper at ICLR 2026
LLaMA
Gemma
Qwen
Qwen-Math
OLMo
Phi
Spearman: -0.53
Spearman: 0.44
Spearman: -0.86
Regret: 10.8%
50
Regret: 14.5%
50
Regret: 0.3%
Average-at 16 (in
45
45
40
log(G-Norm)
G-Vendi
log(GRACE)
Figure 2: GRACE achieves 86% Spearman correlation to LLaMA-1B's post-distillation per- formance on GSM8K, much higher than G-Norm (53%) and G- (44%). When evaluated by teacher-student regret, GRACE selects teacher with regret of 0.3%, outperforming G-Norm and G- G-Vendi, which incur regrets of 10.8% and 14.5%, respectively. Stars denote students trained from the teacher chosen by each score. Gemma teachers are outliers, because they give extremely concise responses to each prompt. More discussion is in Section G.1
uniformly spread the gradients are across examples, which mirrors the core idea behind CMI. Higher gradient uniformity leads on held-out evaluation.
stable optimization, which in can help stronger performance
There several ways extend Lemma 1. First, Lemma is based particular choice of the preconditioner matrix (i.e. M(D', θ) 1//2), which motivated by adaptive optimizers used in practice (Kingma, 2014; Loshchilov & Hutter, 2019; Duchi et al.. 2011). In principle, one could obtain even tighter upper bounds CMI by choosing M optimally. Further, one could improve Lemma by connecting directly to performance metrics other than the loss, and by considering multiple gradient steps. We provide more discussions in Section
EXPERIMENTS
We compare the three scores mentioned in the previous section, G-Norm, G-Vendi, and GRACE, on two common math reasoning datasets, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021). These datasets have a diverse set of strong teacher models readily available, due to the broad community interest in mathematical reasoning. For each prompt-response pair, the model receives binary correctness score, and we quantify its performance by the average accuracy (in percentage) achieved when sampling & responses for each prompt, referred average-at-k:
The student model taken to be LLaMA-1B -B-bse OLMo-1B-baal or Gemma-2BB-aad on GSMSK (Cobbe et al., 2021), and I.L.MA3B--baal
Settings.
MATH (Hendrycks et al., 2021). We compare 3/8/70B Instruct models, Qwen-2.5 1 1.5/3/7/14B Instruct models, Qwen-2.5 Math 1.5/7B Instruct models. Gemma-2 2/9/27B Instruct models, OLMo 7/13B Instruct models, and Phi-4 on both MATH and GSM8K (Dubey et al., 2024; Abdin et al., 2024; Yang et al., 2024; Qwen al., 2025; Team, 2024a). The teacher's generation temperature varied from 0.3 to 1.0 in increments of 0.1. The computational costs of GRACE are detailed in Section E.7. also compare against several common baselines, described in detail in Section C.3. All baselines considered do not require access verifier except for teacher performance.
15 text-only trained teachers: LLaMA-(3.//33
To compute scores, we use subset of n = 512 prompts randomly selected from the training set, with generations per prompt. For GRACE, use 10-way cross validation (i.c. C. 10). The student gradients are randomly projected to dimension = n = 512; we provide ablation results on these hyperparameter choices in Section 3.3.
We use short chain-o6 of thought reasoning models as teachers, given the extensive number of such available models, which also enables extensive ablations. Our experiments focus small-scale students, for which prior work has shown that long chain-of-though teachers do necessarily henefit such models (Li et al., 2025).

---

Published as a conference paper at ICLR 2026
Each distillation training run uses learning rate" 10-5 and 4 epochs over the training set. We use the cosine learning rate schedule with 5% warmup, 0 weight decay, and batch size 64. We generate M = 16 responses per prompt from each teacher and fine-tune the student on all generations without filtering for correctness of the final answer.?
Evaluation. We evaluate the utility of each score with two tests. First, compute the Spearman conrelation of each score with the performance of the trained student models. Second, we measure the teacher-student regret, defined the difference between the final student performance when trained by the actual best teacher, and the final student performance when trained under the best teacher selected by each score. Performances of the best students are reported in Table 3.
We measure performance of the student by average-at-16 performance based on 16 generations temperature 1.0. Average-at-16 is a more stringent evaluation metric that tests consistency of generation from the model by requiring to solve problem correctly across multiple attempts. It has recently been used for assessing robust performance on olympiad-les as a key loss component when training models via reinforcement learning (Guo et al., 2025). We discuss later in Section 3.3 how the results change when we look at other performance metrics.
benchmarks and also serves
Scatter plots. We present our findings in form of scatter plots (e.g. Figures 2 and 10). In all seatter plots, each scatter point represents the student's distillation performance from specific teacher-temperature pair. Different teacher families are shown in distinct colors for clarity, We mark best teacher selected by each score by "star" in its corresponding subplot.
3.1 GRACE CORRELATES WELL WITH STUDENT'S PERFORMANCE AND RETURNS LOW REGRET
Figure 2 shows that for a LLaMA-I IB model trained on GSM8K, GRACE achieves the best Spearman correlation with the student performance (0.86) when compared against G-Norm (0.55) and G-Vendi (0.44). Furthermore, GR ACE returns a lower regret (0.3 %) when selecting the best teacher, compared to G-Norm (4.9 %) and G-Vendi (14.5%) respectively.
Additional experiments with OLMo- 1B and Gemma 2B models trained on GSM8K (Appendix Figures 10 and 11) as well as with a LLaMA-3B model trained on MATH (appendix Figure 14) verify the utility of GRACE. Summary of all our findings is in Table 3 in the appendix. We also compare against other data selection baselines in Figure Across both GSM8K and MATH, GRACE stands out as the only score maintaining strong correlation (> 85%) with student performance, and simultaneously yields the lowest regret, 0.3% on GSM8K and 3.9% on MATH.
Comparisons with teacher performance and student loss. Two intuitive baselines fail to reflect the student's 's performance. The first is the teacher's s own performance, which only shows a weak correlation of 11% for LLaMA- 1B on GSM8K. This agrees with findings in prior works (Mirzadeh et al., 2019; Harutyunyan et al., 2023; Panigrahi et al., 2025; Zhang et al., 2023; 2025a; Peng et al., 2025; Razin et al., 2025). As an example, LLLMMA-0B Instruct has the best performance among all teachers, but a student trained with LLaMA-70B Instruct reaches only 44.5% average-at-Ie perfor- mance, which returns a regret of 7.7% when compared against the best performing student. Similarly, the student's s loss on teacher's s generations, measured before training, is also poorly correlated with the student's performance (44% with LLaMA-1B 1B training on GSM8K) and returns a regret of 5.4%.
3.2 GUIDING DISTILLATION PRACTICE WITH GRACE
GRACE can go beyond identifying the best teacher and inform distillation practices. Below discuss how GRACE provides guidance under common scenarios.
Selecting generation temperature. The temperature used to rescale the teacher's logits when gen- erating responses is known to have a strong influence on student performance after distillation (Zheng & YANG, 2024; Peng et al., 2025). However, there hasn't been principled approach to choose the temperature. We show in Figure 5 that GRACE can identify such good generation temperature for two Qwen teachers: it closely predicts the optimal generation temperature for LLaMA-1B training, which are 0.8 (vs. predicted 0.9) with the 3B teacher and 0.4 (vs. predicted 0.5) with the 1 1,5B
We search over learning rates (5 x 10-5, 10 5 5 x 10-6) and find 10 10-5 to be consistently the best. "Surprisingly, ablations in Section F.1 show that results are not significantly affected if we filter by correctness.
7

---

Published as a conference paper at ICLR 2026
LLaMA-1B on GSMBK
LLaMA-3B on MATH
Gllendi
0.44
BADGE, et
0.28
Gradient cosine
0.56
3.06
Determinant
Grad ent inner
0.00
0.18
G-Nerm
0.53
0.90
0.86
0.85
Average Longth
0.56
0.57
Loss
0.44
Average Probabilities
0.50
0.75
Teacher Performance
Absolute Spearman
Absolute Spearman
Figure 3: GRACE achieves the strongest correlation to student performance, among all scores for LLaMA-1B on GSM8K and LLaMA-3B MATH. Blue bars represent gradient- scores, green bars denote student logit-based scores on the training data, and the gray bar corresponds to teacher performance. Teacher performance and the student's loss teacher generations (Loss) before training show only weak correlations. While G-Norm correlates well with student performance on MATH, it is significantly worse on GSM8K.
LLaMA-1B on GSM8K
LLaMA-3B on MATH
Geeeddi
BADGE, Ash et
14.50
24.13
Gradient cosine
15.60
Determinant
14.50
25.60
Gradient inner prod
14.50
32.40
GRACE
0.30
3.90
Average Length
Loss
5.40
5.60
Average Probabilites
Teacher Performance
7,70
5.90
34
28
Average-at-16 Regret (in %))
Average-at-16 Regret (in 96)
Figure 4: GRACE achieves the minimum regret, among all scores for LLaMA-IB on GSM8K and LLaMA-3B on MATH. Naively selecting the teacher with the best performance shows a regret of least 7.7% and 5.9% in performance of student training on GSM8K and MATH respectively. On the other hand, GRACE achieves regret of 0.3% and 3.9% on GSM8K and MATH respectively. In the left plot, scores that show 14.5 regret select the same teacher, resulting identical regret values.
teacher. In comparison, G-Norm and G-Vendi tend increase monotonically with the temperature, even though the student's performance shows an inverse U-shape in temperature. Aggregating across all teachers in Figure (right), GRACE achieves a 75% correlation with student performance as temperature varies, substantially outperforming G- Gendi (59%). Interestingly, G-Norm is negatively correlated (-53%) with performance in this setting.
Selecting a teacher within a size budget. In practice, one common resource constraint for distillation the compute required to locally host open-source teachers. Motivated by this, we test whether evaluate three scale constraints: (I) 3B and below, (2) 10B and below, and (3) 30B and below. Figure shows that GRACE achieves over 79% correlation with student performance across all scale constraints. Moreover, within each scale constraint, using GRACE to select a teacher yields a regret below 0.3%, compared to at least 9% regret with G-Norm and G-Vendi.
GRACE can be used to select teacher under given size. Specifically,
Selecting teachers within a model family, Another practical limitation the family of models that access, motivating us test GRACE against models within each model family. We split
8

---

Published as a conference paper at ICLR 2026
Qwen2. 5-1.5.--111..tucc
Twen2.5-3B-Instrue
Per teacher corr. with temperature
48
1.00
Normalized scores
0.8
Normalized scores
0.8
52
0.75
average
0,50
0.59
0.55
0.6
44
0.6
Spearman
0.4
0.4
0.00
Student
2-0.5
-0.75
0.3 0.4 0.5 0.6 0.7 0.8 Temperature
1.0
38
0.0
0.3 0.4 0.5 0.6
0.8 0.9 1.0
Temperature
-1.00
-Loss
G-Vendi
---orrm
GRACE
Student average-at-16
Loss
G-Vendit G-Norm GRACE
Figure 5: GRACE identifies good teacher generation temperature. (Left) Results are shown for LLaMA-IB trained with Qwen-2.5-1.5B-IB
and Qwen-2.5 2.5-33-IInstruce teachers on GSM8K. (I) a lower temperature is optimal for Qwen-2.5-1 2.5-1-2..-1..B- ----ntstuct, and (2) higher tem- perature effective for Qwen-2.5-33 2.---------tnsttc In contrast, G-Norm can only identify (1) and G- Vendi can only identify (2). For clarity, all scores normalized [0, 1]. The signs of Loss, G-Norm, and GRACE are inverted so that all scores become higher-is-better for better visualization. (Right) Av- erage correlations of student performance with each score when the generation temperature varied from a teacher. GRACE achieves 75% correlation, higher than G- G-Vendi (59%) and other scores.
GRACE identifies
Absolute Spearman Correlation( 1
Regret in Average-at-16(%]
1.2
G-Norm
Loss
1.0
G-Vendi
GRACE
C-Vendi
GRACE
15
0.8
10.81
10.8
0.6
0.51
0.42
0.4
0.19
0.38
5.38
5.38
5.38
0.2
0.27
0.27
0.27
#38
110B
=30B
$38
$10B
s30B
Teacher Scale Group
Teacher Scale Group
Figure 6: GRACE effectively prediets student performance under scale-constrained teachers. Results are for LLaMA-ID IB on GSMSK. (Left) Across all teacher scale constraints, GRACE has high correlation least 79%) with the student performance after training with the teacher. (Right) Across all teacher scale constraints, GRACE-selects against other scores. In the right plot, G- Vendi and Loss consistently select the same teacher across all scale constraints, resulting in identical regret values in each case.
teacher has minimum regret (< 1%) when compared
the teacher models by model family and consider all generation temperatures. Since some families include only small number of teachers, the Spearman correlations can be unreliable. We hence report the regret of GRACE for each teacher family. As shown in Figure 15 in appendix, GRACE achieves an average regret of just 1% when selecting a teacher within each family, while other metrics yield at least 3% regret on average. Interestingly, the best teacher is not always from the same family as the student. For instance, a LLaMA-1IB student learns better from a Qwen-Instruce teacher than from any LLaMA LLaMA-Instruct teacher.
3.3 ABLATIONS AND ADDITIONAL EXPERIMENTS
Effect of hyperparameters. We test the effect of various hyperparameters used in the GRACE computation. We vary the number prompts (n), the number of generations per prompt (m), and the dimension of the gradient random projection (d). For the LLaMA- student on GSM8K, we find that GRACE is generally robust to these hyperparameter choices, and the default values (m = d = 512, 4) work well (see details in Section F.3). We also vary the number of cross-vall cosssvalidation splits used GRACE. For both GSMSK and MATH, the correlation with student performance remains fairly stable C >= (Figure 27), set C 10 for experiments.

---

Published as a conference paper at ICLR 2026
Robustness of the scores. To test the robustness with respect to teacher selection, evaluate correlations on random subsets of teachers. In addition to the case studies in Section 3.2, we repeatedly compute scores over random subsets of teachers. As shown in Figure 29, GRACE consistently maintains high correlations across these subsets (see details in Section F.5).
Changing Average-at-k to other performance measures. We further examine how correlations change when replacing Average-at-k with other evaluation metrics. For GSM8K, we find that Spearman correlation drops when switching from Average-at-K to either greedy or best-of-k accuracy, but GR GRACE still returns the lowest regret in the corresponding performance measure (< 2.5%) when selecting the best teacher (Tables 5 and in appendix). Greedy reflects performance from a single generation at temperature 0.0, and best-of-k measures whether the student answers correctly at least once over & responses at generation temperature 1 1.0. deeper investigation into the discrepancy between Average-at-k and these discrete performance metrics is left to future work.
dittr--disttibution evaluation: We evaluate LLaMA-IB trained on GSM8K on GSM-Symbolic (Mirzadeh et al., 2025) (Figure 16), and LLaMA-3B trained on MATH on MATH (Shah et al., 2024) and MATH Perturb (Huang et al., 2025b) (Table 4). On GSM8K, student performance is strongly cor- related between GSM8K and GSM-Symbolic, and GRACE is highly predictive under this OOD shift.
For MATH, we observe a nuanced pattern. On the simpler OOD task MATH Perturb (Simple), G-Norm is a stronger predictor of student performance actoss teachers than GRACE. However, the more challenging OOD settings, MATH and MATH Perturb (Hard), GRACE achieves similar or higher spearman correlation and lower regret than G-Norm, indicating that GRACE is a reliable score for identifying teachers for which the trained student generalizes well on difficult OOD tasks.
4 DISCUSSION
This work proposes the GRACE score for identifying the most suitable distillation teacher. Motivated from optimization and connected to CMI, GRACE leverages two distributional properties of the student's 's gradients: the directional coverage of the (normalized) gradients, and the gradient variance. Experiments on GSM8K and MATH establish that GRACE strongly correlates with the student's performance after distillation, and enables principled comparison across teachers and offers actionable insights into practical scenarios. Our results highlight GRACE's potential as practical and general- purpose tool for guiding distillation practices.
Below discuss promising avenues for future work.
First, GRACE is not guaranteed to be robust in adversarial setups. One example is when a teacher repeatedly emits partial reasoning steps redundant text before eventually arriving at the correct answer. Due to the in-context ability of language models, growing the number of repetitions will make the prediction closer to deterministic, and hence having an increasingly small gradient. In this case, the GRACE score may be deceptively low even though the student can perform arbitrarily poorly. Fortunately, such behaviors are rare among well-performing teachers and can be mitigated via post-processing Fully characterizing the failure cases of GRACE remains an open question.
Moreover, while GRACE captures two important distributional properties of the student's gradients, incorporating properties of the teacher and characteristics of the data can be fruitful. For example, although GRACE's design intentionally avoids requiring teacher logits, incorporating logit-level information where available may lead to further performance gains. It is also interesting to investigate GRACE's utility adaptive distillation strategies, where teacher choice may vary dynamically across training stages or subsets of data, possibly in an online fashion akin to reinforcement learning.
Finally, while this work focuses on mathematical reasoning due to the abundance of teacher choices and the ease of verification, GRACE is not defined specifically for math. Extending GRACE to more general domains would be an interesting direction for future work. As a preliminary study, we extend our analysis to the ARC dataset (Clark et al., 2018) in Table 7. GRACE achieves the highest Spearman correlation with student performance among all scores that do not rely on a verifier, and achieves the lowest regret among all scores. Interestingly on this task, observe cases where teachers with extremely high GRACE scores can still yield strong students. Understanding the broader conditions under which GRACE reliably predicts student performance is promising future direction.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Jordan Ash, Alex Damian, Tianyu Gao, Jonathan Geuter, Sara Kangaslahti, Nihal Nayak, Xingyu Zhu, Simon Park, and Noam Razin for helpful discussions and suggestions. Abhishek Panigrahi acknowledges support from Apple AIML and Siebel Scholarships. This work was enabled part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffimann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report. arXiv preprint arXiv: 2412.08905. 2024.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models. Trans. Mach. Learn. Res., 2024, 2024. URL https: http://pppereiiieeeeeeeee/ee net / forum?id- X XFHWcNTSHP.
https::/openzeview
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. Why exposure bias matters: An imitation learning perspective of eπтor accumulation in language generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.). Findings of the Association for Computational Linguistics: ACL 2022, pp. 700-710, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 8653/v fwn.el.iiiiinnaanaa acl.58. URL https: http///aaaaaatlta//a//oo org/ 2022. findings acl. 58/.
Ash, Chicheng Zhang, A. Krishnamurthy, Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. International Conference on Learning Representations, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.06400650 2016.
Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russell Webb. Distillation scaling laws. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 5977-6045. PMLR, 13-19 Jul 2025. URL https: httsss//poccceedeeeeeee m1 r. www.sessseeeo v2 hwt.//////bbr/eee dge25a html.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arX1v:1803.05457vv1 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: 2110.14168. 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv: 2305.07759 2023.

---

Published as a conference paper at ICLR 2026
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: model-aware dataset selection with datamodels. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Lizhe Fang, Yifei Wang. Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? In The Thirteenth International Conference on Learning Representations, 2025. URL https: // /oppereevew.nner wttpr/wwwwweetttooiideeeeee-e
Etash Kumar Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Rea Sprague, Ashima Suvarna, Benjamin Feuer, Leon Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang. Shreyas Pimpalgaonkar, Kartik sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah M Pratt, Vivek Ramanujan, Jon Saad-Falcon, Stutee Acharya, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alex Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models. In The Fourleenth International Conference on Learning Representations, 2026. / wwwppppeeeiieeeeeeeeeeeeeeeeeeee iew. net/ / forum? ? idd JoTuaNMN.
URL hwtps:tpp/////.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforce- ment learning. Nature, 645(808 (0600881)6336638 2025.
Mahdi Haghifam, Shay Moran, Daniel M Roy, and Gintare Karolina Dziugiate. Understanding generalization via leave-one-out conditional mutual information. In 2022 IEEE International Symposium on Information Theory (ISIT), pp. 2487-2492. IEFE, 2022.
Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, and Sanjiv Kumar. Supervision complexity and its role in knowledge distillation. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:://// wptp/eeeeevvieeeeeeeeee net wtstt/forummiii iicummiddSe f 7N7mA.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, D. Song, and Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS Datasets and Benchmarks, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.02531, 2015.
Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J Foster. Is best-of-n the best of them? Coverage, scaling, and optimality in inference-time alignment. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 25075-25126. PMLR, 13-19 Jul 2025a. URL https: http://pooeeeeeeeeee///e//o/oeee mlr wwt.esm..r.p..r..... w267 http////hhaan....eeee/e html.
Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, Yue Wu, Ming Yin, Shange Tang, Yangsibo Huang, Chi Jin, Xinyun Chen, Chiyuan Zhang, and Mengdi Wang. MATH-perturb: Benchmarking LLMs' math reasoning abilities against hard perturbations. In Forty- International Conference on Machine Learning. 2025b. URL https: http:///peeeeeieeeeeeeeeeee net/ /wweteeetttruee forum? i 1d=0oy70Jggxx..
Aref Jafari, Mehdi Rezagholizadehh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distillation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2493-2504, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/ manelmmain221 212. URL https: //l:://aaee /acl hothhologyy .rg/ /tgg//2111222222 2021 eacl -aael-maiie in 212/.
12

---

Published as a conference paper at ICLR 2026
Jachun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, and Yejin Choi. Prismatic synthesis: Gradient- based data diversification boosts generalization in LLM reasoning. In The Thirty- Annual Conference on Neural Information Processing Systems, 2025. URL https: patttps:/oppereeee /opnr/vvee.e.eeeeeememee wet/forunt d=R0dc7xzzbb..
Hoang Anh Just, Myeongseob Ko, and Ruoxi Jia. Distilling reasoning into student IIms: Local naturalness for selecting teacher data. arXiv preprin! arXiv:25/0.03990 2025.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 8110-8118, 2021.
Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint 1412:1412.69900 2014.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv: 0009005463, 2023,
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengging Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. Annual Meeting of the Association for Computational Linguistics, 2025, doi: 10.48550/arxiv.2502. 12143.
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. In International Conference Machine Learning, pp.
PMLR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL https: /openreiiee/eeeeee.eemee new htt.:/umw.eee/ooomitttte. BkgCRYC
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL HuggingFaceew/ wtt.ee/////ewweeedduu
co/ co/dttsets//
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and H. Ghasemzadeh. Improved knowledge distillation via teacher assistant. AAAI Conference on Artificial Intelligence, 2019. doi: 10.1609/AAAL V34104.5933.
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symoolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttt:p:::/oppreee /wtp///eeeeeeeeeee/ee/e/ee n Aarrummidd=AjKkaae
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 9950-
PMLR, 2020.
Team OLMo. 2 olmo 2 furious. arXiv preprint arXiv: 2501.00656, 2024.
Abhishek Panigrahi, Bing Liu, Sadhika Malladi, Andrej Risteski, and Surbhi Goel. Progressive distillation induces an implicit curriculum. International Conference on Learning Representations, 2025.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior at scale. In International Conference on Machine Learning (ICML), 2023.
13

---

Published as a conference paper at ICLR 2026
Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang. Lei Hou, and Juanzi Li. Pre-training distillation for large language models: A design space exploration. In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3003336033333 2025.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:// /ttp//aavvvv/v.////////a./.a.av. owg/ abs / 15115.
Mohamad Rida Rammal, Alessandro Achille, Aditya Golatkar, Suhas Diggavi, and Stefano Soatto. On leave-one-out conditional mutual information for generalization. Advances in Neural Information Processing Systems, 35: 10119990099,, 2022.
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What makes reward model good teacher? an optimization perspective. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025. URL https: /ttp:r/v///peeeiee.e/e net/fozee id= 7pf00oSJc.
Dhruv Rohatgi, Adam Block, Audrey Huang, Akshay Krishnamurthy, and Dylan Foster. Computational-statsttis tradeoffss at the next-token prediction barrier: Autoregressive and imitation learning under misspecification (extended abstract). Nika Haghtalab and Ankur Moitra (eds.), Proceedings of Thirty Eighth Conference Learning Theory, volume 291 of Proceedings of Machine Learning Research, pp. 4831-4837. PMLR, 30 Jun-04 Jul 2025. URL https: http///roceeedeee ngs.
ngs. ml wtt..mml..pprre prwsssrsss/ee. /2 v2 91 rohat 1.2. html.
Yash Savani, Asher Trockman, Zhili Feng, Yixuan Even Xu, Avi Schwarzschild, Alexander Robey, Marc Anton Finzi, and J Zico Kolter. Antidistillation sampling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: oppnneveieee net net/forum: VoVo2UHquu8..
Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.2 2024.
Yuda Song, Gokul Swamy, Aarti Singh, J Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. Advances in Neural Information Processing Systems, 37:12243-12270 2024.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Be- yond neural scaling laws: beating power law scaling via data pruning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022. 2022. URL http: http:::///ppppppee http:/paarrrsss./ee nips. oc/ cc/paper_pppppp/ppppppppeeee.e files paper/2005 hhaah/hah///// 7b75da 61eee 2aaeaae440005555555555555 -atpaa/ooteotaaataaatttt-oooooo - html.
Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference Learning Theory, pp. 3437-3452. PMLR, 2020.
Gemma Team. Gemma 2: Improving open language models at practical size, 2024a. URL https: http://wwwigg///aee..//////o
wrg//rgg/ss.......... alss///////// 2408 00118.
Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv: 2408.00/ 2024b.
The Llama Team. The laaa 3 herd of models. arXiv preprint arXiv: 2407.2/783, 2024c.
14

---

Published as a conference paper at ICLR 2026
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: selecting influential data for targeted instruction tuning. In Forty-first International Conference on Machine Learning. ICML 2024, Viemna, Austria. July 21-27, 2024. OpenReview.net, 2024. URL httpps: wpwnreviiieeeeeeeeeeeeeeeeeeeeee ew net / forum? d=PG5
Aolin Xu and Maxim Raginsky. information-theorethe analysis of generalization capability of learning algorithms. Advances in neural information processing systems, 30, 2017.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Stronger models are not always stronger teachers for instruction tuning. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.). Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4392 Albuquerque, New Mexico, April 2025. Association for Computational doi: 10.18653/v htt.//5///11222.aaaaaaaaaaa
Linguistics. ISBN 87779-888877---8--999 //aclanthoolc wtw.hhooooooooooooo.ooooooo org. www.aaalllloooo 224/.
URL https:
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via seff-improvemmnt. arXiv preprint arXiv:2409. /2 122, 2024.
Chen Zhang, Yang Yang. Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, and Dawei Song. Lifting the curse of capacity gap in distilling language models. Annual Meeting of the Association for Computational Linguistics, 2023. doi: 10, 1385500arXiv.2305..222
Chen Zhang, Qiuchi Li, Dawei Song, Zheyu Ye, Yan Gao, and Yao Hu. Towards the law of capacity gap in distilling language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 22504-22528, 2025a.
Dylan Zhang, Qirun Dai, and Hao Peng. The best instruction-tune tunnon data are those that fit. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025b. URL https: / wpwnrevieveeeeeeeeeeeeeeeeeeeeee net / forum?id=4jFSekaanl
Kaixiang Zheng and EN-HUI YANG. Knowledge distillation based on transformed teacher matching.
In The Twelfth International Conference on Learning Representations, 2024. URL https: / openrevieww . net/ puttfforume forum?
15

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
GRACE: Gradient Crros-Vallddtion Evaluation
2.1
Gradient-Be Based Scores
2.2
The GRACE Score
2.2.1
Connecting GRACE to leave-one-out CMI
3
Experiments
3.1
GRACE correlates well with student's performance and returns low regret
3.2
Guiding distillation practice with GRACE
3.3 Ablations and additional experiments
4
Discussion
10
A Outline of the appendix
17
Related work
17
Discussion the baselines
18
C.1 Using gradient influence for teacher selection
18
C.1.1 Details LESS (Xia et al., 2024)
18
C.2 Issues with influence based metrics for teacher selection
19
C.2.1 Can we use teacher generated validation sets for each teacher?
20
C.3 Descriptions of other baselines
20
D Connecting GRACE to Leave-one-out Conditional Mutual Information (LO0-CMI)
21
D.1
Background on LOO-CMI
D.2
Connecting GRACE to LOO-CMI
22
D.2.1 Proof of Lemma 2
24
D.3 Can we connect GRACE's predictive behavior of student performance to student's validation loss after training?
26
D.3.1
Does validation loss after training correlate to student performance?
26
D.3.2
Does GRACE correlate with validation loss after training?
27
More experimental results and comparisons
28
E.1
Additional results
28
E.2 Mcasuring effectiveness of GRACE within each teacher family
29
E.3
Out-of-distribution evaluation
29
E.4 How does knowing the teacher performance change the behavior of the different scores?
30
E.5 Measuring correlations with other measures of student performance
16

---

Published as a conference paper at ICLR 2026
E.6
Preliminary exploration: General NLP tasks beyond math
33
E.7
Computational complexity
34
Ablations
35
F.1
Filtering v/s No filtering
35
F.2
Ablation on training hyperparameters
................................
F3 Ablations on the parameters of GRACE
36
F.4 Gradient norm's relation length
36
F55 Ablation on robustness of metrics
38
Prompt format
40
G.1 Qualitative analysis of responses from different teachers
41
A OUTLINE OF THE APPENDIX
Below, we outline the structure of the appendix.
Section B provides situates our work among existing literature. Section C details the baseline methods against which we compare GRACE. Section C. introduces gradient influence-based scores and explains why they are ineffective for identifying the best teacher. Additional scoring methods are discussed in Section C.3.
Section D formally defines conditional mutual information (CMI), which provides the theoretical basis for GRACE. Section D.3 highlights key limitations using CMI to explain the success of GRACE and outlines open questions to bridge this gap.
Section E shows additional experimental results: additional models on GSM8K and MATH (Sec- tion E) results within individual teacher families (Section E.2); the effect of teacher performance on GRACE (Section E.4); correlations for metrics beyond Average-at- 16 (Section E.5); evaluation non-math datasets (Section E.6); and computational complexity analysis (Section E.7). Finally, Section F presents ablations studying key design choices GRACE
B RELATED WORK
Knowledge distillation. Knowledge distillation is a classic method used to improve the optimiza- tion and generalization of a small model (Hinton et al., 2015). A counterintuitive finding is that a better-performing model is not necessarily better teacher, which has been observed in both classic classification or regression settings (Mirzadeh et al., 2019; Jafari et al., 2021; Harutyunyan et al., 2023) and more recently in language models (Zhang al., 2025a; 2023; Xu et al., 2025; Panigrahi et al., 2025). For language models, one can distill from either the logits of the teacher or the generated texts. While the former can lead better student performance, is more computationally costly, requires higher access, and is less flexible due to tokenizer choices. We hence focus on distilling from generated texts (Eldan Li, 2023; Li al., 2023: Busbridge et al., 2025). Recent work by Guha et al. (2026) supports our findings: they demonstrate that a weaker teacher can yield stronger distilled model, that distillation benefits from increased sample size, and that filtering has little impact on the resulting student's performance.
Data selection. For text- based distillation, selecting the best teacher can be considered as the problem of choosing the most useful subset of samples from the generations of all teachers. This aligns with the broad task of data selection, which aims to identify subsets of data that maximize certain utility (Sorscher et al., 2022: Albalak et al., 2024). Many approaches leverage gradient
"We consider generations following standard next-token distrihutions, opposed to antidistillation sampling (Sa-
vani et al., 2025).
17

---

Published as a conference paper at ICLR 2026
information (Mirzasoleiman et al., 2020; Killamsetty al., 2021; Pruthi al., 2020; Xia al., 2024), including some that directly rely on notions of coverage (Ash et al., 2019; Jung et al., 2025). Directional coverage also ties to the notion of coverage in reinforcement learning. Specifically, autoregressive training on teacher generations can be viewed as a form behavior cloning, for which increasing the coverage is provably beneficial (Song al., 2024; Huang al., 2025a; Rohatgi et al., 2025). Despite these similarities, distillation differs from standard data selection in that it allows generating new data and offers richer design space (Peng al., 2025). An effective teacher-sclection score should therefore be versatile and broadly applicable across scenarios, property that GRACE demonstrates shown in Section 3.2.
Zhang et al. (2025b) study teacher selection at the granularity of individual prompts, proposing to score a pool of responses from different teachers by the student's loss and train on the best-scoring completion for each prompt. However, as also observed by the contemporary work (Just et al., 2025) and confirmed our experiments, such loss-based scoring is only a coarse proxy for response quality. Just et al. (2025) propose using loss short reasoning segments mitigate this. They show that such scoring can be used select the best global teacher, however their evaluation limited to only three teachers. By contrast, evaluate GRACE across fifteen teachers and eight temperature settings each. Extending GRACE to per-prompt teacher selection is a promising direction for future work.
C DISCUSSION ON THE BASELINES
C.I USING GRADIENT INFLUENCE FOR TEACHER SELECTION
Existing data selection methods estimate the importance of each data point based its influence validation set. Among these, gradient-basee approaches define this influence through the relationship between the gradient of an individual training example and the gradients computed on the validation set.
straightforward extension of these data selection strategies is to measure the quality of the gradient distribution on 'D w.r.t. a heldout validation set D'. Adapting the gradient influence metric from Pruthi al. (2020), we measure the influence the gradients from training set on the validation set D'. Mathematically, this is defined as
cos(h(x, y), ,(xx,x))) ID'
Σ (N.jED (x' E'DD
Σ
Following Xia al. (2024), we can additionally improve this definition by taking Adam optimization into account. This requires an initial warmup training of the model on an independent copy of D; more details are provided in Section C.I.I.
We compare teachers for LLaMA-1B 1B trained on GSM8K, with D' being the validation split consisting of human-written written solutions. also report the loss on the reference set D' the trained models after the warmup phase following Xia et al. (2024), As shown in Figure 7, influence-based metrics show low correlation with the student performance after training.
C.I.1 DETAILS ON LESS (XIA AL., 2024)
For discussion in this section, we will use few additional notations: we will use Oş E RD to generally denote the parameters of the student model. e will denote the parameters the model time t.
Suppose, we train our model with Adam (Kingma, 2014) with hyperparameters 7, 8, 12,32, e, that tracks two gradient statistics: m, and en (initialized Os) at each step of training. The gradient
18

---

Published as a conference paper at ICLR 2026
update step at any step t 0 with gradient vector g E R
is given by
0 t - (
where r(+1(g) = m1+1 O
VV++1 +e²
m141 f 1 B ((1 - B1)m, + +1g)
U141 6 1 B221 ((1 ((1-)2)er
82) u + (2 (g@gg)
where T/ denotes the learning rate at time step f, that can be a function of 1j and time-step & depending on the learning rate schedule. Here I' denotes gradient preciiniiiinnnng function that uses the gradient statistics m and v.
LESS (Xia al., 2024) considers the gradient update under Adam to compute the influence of the gradients from training dataset DD on validation set 'D'. Suppose in addition to D, we have warmup dataset D Dwarmup- Then, the model trained for steps (4 epochs) on the warmup dataset DD Dwarmup-
Let Ty,,, Tyy2, T7/ and FT be the pre-conditioning functions at step T/4, T/2, 3T/4, and T respectively. Furthermore, let firy4 denote the average learning rate between timestep 0 and T/4 (similarly, define //r/2 as the average learning rate hetween timestep T //4 and 7/2, and so on). Then, the influence of each input (x, y) E DD denoted by Inf(x. is given by
Inf
Σ ("(')") i=1
Me / (T (Er7:(002x.)).ggxxxx"
(6)
For computational efficiency, random projection II:
Σ Inf(x,y) = A/D' (''Y')DD'
Cosinee (IT.))l((xx)))
In LESS, Inf(x, y) was used to pick the most influential data points in 'D. We report the per- formance LESS for teacher selection by using an average influence function the dataset D to measure its overall quality:
GRACE
Norm
Inf = D
Σ ynf (x,))D
(jung et all
Vendi
Lass reference sat (after warmup training)
LESS (Xla et al
C.2 ISSUES WITH INFLUENCE BASED METRICS FOR TEACHER SELECTION
Gradient influence (Pruthi at a
154
We hypothesize that key limitation of these influence-basee metrics is the choice of D': it should both reflect a common ground truth and effectively differentiate models without bias to- ward any particular one, which can challeng- ing to construct.
Absolute Spearman 7
Figure 7: Influence- measures that rely on a validation set are ineffective for selecting good teachers. We report LLaMA-IB performance on GSM8K across different teacher choices at a generation temperature of 0.6. The validation split from Cobbe et al. (2021) serves as the held-out set for computing Gradient Influence and LESS (Pruthi et al., 2020; Xia et al., 2024).
Gradient influence metrics like LESS approxi- mate the decrease in validation loss when tak- ing a gradient optimization step. On an update e (t+1) from @ using gradient vector with
the corresponding student given by F (f+1) and
(t) n respectively, we can show with first order taylor expansion:
19

---

Published as a conference paper at ICLR 2026
E yo) (<(y'ix''' ( L(y') x)) o E (xx (o) - 0 3(r'xx)) = =B(x") (g)+ x
However, there are two primary components in the solution y': chain-of-thougle and the final answer. we denote each response y' as composition of (cot', a'), then the above formulation can be re-formulated as
"'e's")DD''''ooo (E(cot', a ['(x'''// ns ( [(cot', a a'''' x
For a given prompt x', the final answer will be the same across different responses but the chain-of-thought cot' can differ. For pool of teachers (Ti, Tp) with training generations Dr, for each teacher, the thain-of-though patterns differ across the teachers. Therefore, the validation D' should ideally be constructed such that the student's loss or gradient metries evaluated on it can effectively distinguish between the diverse shain-of-thougee behaviors across all the teachers' training generations.
In Figure 7, we show that using the validation set from the GSM8k dataset, which has human- -witten answers to each question, as D' doesn't help effective teacher selection. In fact, the loss on the set D itself doesn't 't help to effectively distinguish between different teachers. We keep principled selection of a common validation set D' for selecting the right teacher to future work.
C.2.1 CAN WE USE TEACHER GENERATED VALIDATION SETS FOR EACH TEACHER?
An alternative is to use separate validation sets for different teachers, each containing responses generated by a specific teacher on shared set of questions. That is, on given validation set of prompts D', we simply collect the response of a teacher T to each prompt x' E D' and create validation set D'.. Then, for each teacher T, we modify the LESS influence metric on an example yx E Dy from Equation (6) as
Inf(x,y)
Σ (X'Y")ED,
Σ
TsT / (III (wr [gg(x,, y
4
where II denotes random projection matrix.
We report the performance of gradient influence metrics under this setup in Figure 8. We find that the performance of the gradient influence metrics for teacher selection do not substantially change under this setup, compared to using common validation set in Figure 7.
One primary concern is that this approach breaks comparability across students trained on different teachers, as gradient estimates from different validation sets cannot be used to compare across them. That is, if (DT, D/,) ) and (DT,, D',)) represent the training and validation sets from two teachers, influence estimate scores with (DT, Dr and (DT,, Dyy)) can't be used to directly differentiate between the generations of the teachers Ti and Tz-
C.3 DESCRIPTIONS OF OTHER BASELINES
We consider the following additional baselines:
1. Student Loss on the teacher's generations: This is defined by the average cross entropy loss of the student model on the teacher's generations before training.
2. G-Norm: Defined Equation (3), this measures the variance of the gradients w.r.t. the mean gradient.
3. G-Vendi: Defined in Equation (2), this measures the directional entropy in the gradients.
4. Determinant: This defined by the determinant of the gradient matrix G(D) and serves as notion of diversity of the gradients.
20

---

Published as a conference paper at ICLR 2026
Liama
gemma
Qwen
Qwen-Math
OLMo
phi
0.52
Spearman: -0, -0.19.55
Spearman: -0.30
Spearman: -0.55
Regret: 0.1%
Regret: 0.1%
Regret: 0.0%
0.50
Average-at-15
0,48
0.48
0.48
0.46
0.46
0.44
0.44
0.44
0.42
0.42
0.40
0.12
0.36
0.20
0.24
2750 2500.275.00 Gradient Influence (Pruthi et al.)
"6110001550220022550 2500
3250 350
0.36
0.38
0.40
0.42
0.44 0.46
LESS
1
validation loss
after warmup training &
Figure 8: We repeat experiments from Figure 7. But instead of using a common validation set with human written answers from Cobbe al. (2021) measure the gradient influence metrics across the teachers, we use separate validation set for each teacher, where we use the respective teacher's response on the validation on GSM8K across different teacher choices at generation temperature of 0.6. We find that the teacher-selection performance of the influence-basee metries does not improve under this setup. For comparison, refer to Figure 21, where gradient-bare based metrics achieve notably higher Spearman correlation with student performance.
questions. We then report the performance of LLaMA-
5. Determinant X gradient norm, corresponding to BADGE (Ash et al., 2019): This is defined by the determinant of the gradient matrix G(D) and serves notion of diversity of the gradients, that also takes their norm into account.
6. Gradient cosine, which is another way to capture gradient diversity: Given gradients from the training set we compute pairwise inner product between the normalized gradients of generations for the same prompt:
Ex (x.))))))))
g1
82
|gil12
|g 2
where gi = h(x,y1),
g2 h2x,
7. Gradient inner product: we compute pairwise inner product between the gradients of generations from the same prompt.
ExB(ox.i))oxaa)- g1 g2:
where gi = h(x,
g2 h(x,y2).
This can be considered as considering gradient magnitude in addition to the gradient cosine. 8. Average Probabilities (per token): this computes the average probability per token of the student on the teacher's generations, averaged over all generations and all prompts.
Average length: this computes the average length of the teacher's generations.
As mentioned in Section 3, naive metrics not useful for identifying the best teachers.
D CONNECTING GRACE TO LEAVE-ONE-OUT CONDITIONAL MUTUAL INFORMATION (LO0-CMI)
D.1 BACKGROUND ON LO0-CMI
We start with general description of the LOO-CMI framework from Steinke Zakynthinou (2020); Rammal et al. (2022). In this setup, have dataset D {7,, Zz, 7n,) of i.i.d. samples
21

---

Published as a conference paper at ICLR 2026
drawn from the parent distribution P, where Z refers to general datapoint containing the data-label pair (x,y).
Consider training algorithm A: D θ that takes training set D and returns weight parameter O. Let u be an index uniformly drawn from {1...., n), and denote by D-, == {Z,} the size-(n 1) dataset with the u-th sample removed. Denote the output of the training algorithm D_u by e.. = A(D_w). Then, the leave-one-out conditional mutual information (LOO-CMI) of A defined as
I:o0(A: :1(0))))((000:1 1) = Bu--Uniffi] [KL(Pe_..u..DIPee....))
(8)
where [(::- denotes the conditional mutual information, KL(-I|- the Kullback- divergence and Pe denotes the distribution over parameters O, where the randomness comes from gradient noise, The LOO-CMI quantifies how much information the trained model parameters @ reveal about which sample index U was removed when training on the remaining data. A small value of (oo D) indicates that the algorithm's output is insensitive to the removal of single training example, which corresponds to higher algorithmic stability and better generalization.
Generalization bound. With slight abuse of notation, let f θ x D + [0, 1] denote the loss on a dataset D, and let f: θ > [0, 1] denote the population loss, i.e. taking expectation over The generalization gap of a randomized algorithm A that has trained on dataset 'D is measured as
gen(A) |EA.=P, (( (<((D))
((A(D), D)][
Rammal al. (2022) show that the expected generalization gap of A satisfies
(An <
V2(n 18(n-1 ED
n
(9)
D.2 CONNECTING GRACE TO LOO-CMI
Our discussions below focus on the original gradients without pre-processing.
Notations. Let Os E IR denote the parameters of the student model before training. For the theoretical analysis, we use m = 1 teacher response per prompt on the n prompts in the training set D, though our results can be generalized to random prompt-generation pair yx.y) E DD in place of the scale random variable u ~ Unif((n) where (D) n. For example, will use D D_ -xx.y) denote the dataset with (x, removed from D. For simplicity, we will drop Oş from all our notations involving M and g.
1. From
we will use the uniformly
We will use È to denote the empirical expectation over D. For each yx, we perform a single gradient update with preconditioner matrix M that can depend the training D-(xy))
(x,y + θs - 17 E) (M(D((x))) (g(,)) +
(10)
where e <~ N((0, o² I)) denotes a Gaussian noise, which represents the randomness our training algorithm.
For any dataset D. we define the following notations:
((D)= È (xx.) y)), Σ(D) G(D)TG(())
E(D) E(D) + I,,
D
where > 0 is added for numerical stability.
We will further use the following shorthands:
(_(x,)) == (D (D-(x))), M (x.y) M M(D-(x.))
Then, the CMI bounds for the preconditioned update step in Equation (10) are given by:
22

---

Published as a conference paper at ICLR 2026
Lemma (Bounds for Pre-conditioned Gradient Descent). Suppose 38, > such that [ MM (x,9)) I E for all (x, (x. y) E D, and llg(x, y)ll S B for all (x, y) E D. Under the one-step update rule on the parameter θ with learning rate TJ (Equation (10) algorithm A, the LOO-CMI can be bounded as
3n²²
[Lo0(A; <
B(xy)ED
o² 1)2
3n1²
MM (M-(x.)) BIXSSEDM MDD) O(3/nt), -0(B7n) (11)
+
B
B(xxjjDD
where By (x.)) = -g (DD).
There are two notable special cases: the first is when M = I, corresponding to update step in the standard gradient descent update algorithm. The second is when M = É-1/2, which denotes the preconditioner matrix used define GRACE
Special cases:
Corollary D.1 (Connecting CMI G-Norm). When M set identity matrix I for all (x,y) D, then LOO-CMI for the one-step update rule on the parameter θ with learning rate given by
3n
(o: S 0²(n 1)2 R (XY))D
(x,
where gu (x, y) = g(x,y) - (D)
This connects CMI the G-Norm (Equation (3)) metric.
Corollary D.2 (Connecting CMI to GRACE). M(D) is set as Ê( for any training set D. Denote E [x((xx)) == E((--(xx)) for any (x, y) pair for simplicity. Suppose there exists a constant 0 < a < s.t. for any (x, y) E D.
( (xx)) (x,)
1/2 - (x $ 1/2 (x.3) p (D)
a n2 | (x.
(12)
Then LOO-CMI for the one-step update rule on the parameter θ with learning rate 1j is bounded as
3(1 + a)n?
Looo(A < 0² - 1)2
E(xy)ED
| (x. (0(B/))
3
aa
GRACE-Variance(D + ((3/n²)
0² d2 (n 1)2
31 + +a)
(
( 1)2
where g, (x, g(x,y) (DD)
This connects CMI to the GRACE (Equation (4)) score with n-fold cross-validation.
Theorem D.2 indicates that GRACE evaluates the stability of one-step gradient update when few prompts are removed from the batch. Importantly, the outcome of this update depends on the optimization method, since gradient descent and preconditioned updates can behave differently. In our setting, the preconditioner matrix is closely related to the one used in AdaGrad (Duchi et al., 2011). Since adaptive optimizers the facto choice for training language models, essential to incorporate this preconditioning effect in our analysis.
However, the pre-conditioner matrix that we use doesn't 't exactly match the optimization states of the Adam optimization algorithm that we use in practice, leaving space for further improving the definition of GRACE. This might require short warm-up training phase of the student model and setting M as function of the optimizer states during the warm-up training, akin to Xia et al. (2024). leave more thorough exploration of this direction to future work.
23

---

Published as a conference paper at ICLR 2026
Mean rañio 0.442 Std-dev 0.130
10
Term 1
10-7
(tatio 1)
10
10 6
Term
Figure 9: Validating assumption Equation (12) in Theorem D.2 using teachers on LLaMA-1B
and GSM8K using = 512. Here Term I is given by n² - (x.) (x y) Term 2 is given
by (>>-/>)) - ((x..) ((D) |
is defined as Term 2 Term 1. Main takeaway:
Term dominates Term across all teachers, with mean = 0.442, validating the assumption in Theorem D.2.
Note on theoretical limitations: Our current analysis establishes a connection between GRACE and an upper bound on LOO-CMI, which has been shown to upper bound the generalization gap in terms of the loss (Rammal et al., 2022). However, Lemma 2 is stated only for single gradient step, while in practice we compare the student's performance across different teachers after extensive training. Moreover, the loss may not tightly reflect the performance metric of interest. In particular, the student model's test loss after training does not correlate with the student performance when compared across different teachers, as shown by our empirical results. Such theory-practee gap highlights the need for further theoretical understanding of GRACE, which we leave to future work.
Validating assumption in Equation
(12): For simplicity, denote the two terms of interest
Term
1 8-1/2 1/2 (x.y) (x,y) Bu (x, y) n²
Term 2: - (x.y) ( -1/22
Expectation of Term gives GRACE with C = n and is the main term of interest. Term 2 measures the stability of the pre-conditioning matrix Σ when prompt is dropped. The ratio of Term I and Term 2 measures how dominant is Term compared to Term in Equation (11), formally referred to in Equation (12). If a 1, this implies the bound the conditional mutual information is primarily driven by our formulation GRACE. Empirically, on LLaMA-1B IB training on GSM8K, show that a has average value between 0.442 across all teachers in Figure 9.
D.2.1 PROOF OF LEMMA 2
Proof. For any y) pair, denote the mean parameter update on the training set D_(xxy) as (_(x)) = (xx) Starting from student parameters Os, the update rule for any set D ((x,y)) given by
-_xxy) + Os + + +
where e ~ No0 oT)) is Gaussian noise and captures the randomness in the algorithm. Hence (xxx) follows the distribution
+ - ~ N (Os + 6_(x,)),
WWe will extensively discuss the comparison of GRACE, and loss in Section D.3.
24

---

Published as a conference paper at ICLR 2026
Then, using the properties of Gaussian distribution:
(1o((A:)) B(xYtEDDKI Pe ( ((sssepPo..s))
BE.. (0 (.)) (108( - 11l--- (1l-- log BeSSED - -1s-- (x 11/200
(13)
(-N (log ge - - E log ge - -o-11----cs- (x 11 l2//2²)) ))
$
(14)
1 ( (-11s-d R ||s 5
20t
B.x.)EED (-M(b (6-ons) (- ||5 - -- + ||c - 5-(2.))11))
20
1
(R (60 d (x,y) l
(15)
2g
B (x.iiED (-(x))) B(s.))Edd ||
(16)
In the second step (Equation (13)), we simply use the CDF formulation of gaussian distribution, where Z = (2xe)-D D_ The third step applies jensen's 's inequality (Equation (14)). In Equation (15), we use for any two vectors a, b:
E(-M(o,0T) (-IKs- a12 + +lls- +lk--bli) 21 b) ||all2 + ||b12
2(a - b) T a ||a|| 2 |bb|
|b all²
Finally, Equation (16) follows because both 6_(x,y) and 6_ come from the identical set. Using the definition of we have from Equation (16):
1
M M-xxx))(((xx) M I
(( D) <
Warmup: When the pre-conditioner is identity matrix Then for any (x.y) pair, we have M. (x.y) =I I. Then, the formulation simplifies to
E (ml(((())))() I
2
p-(xx) E[SS)EDM- ! (xx))D
E
112
1 2
E
(xy))D
((S)ED (--""(D) n (1(1) n n B(xx)) ---B(x.y) -
(1(1 n 8(x
σ²
(17)
n
(xyiED g((xy) ((,,))DD((x ?
02 1)2
1/ 0²(n - 1)2
172
BxDY)D (lg(x.)) - (D) = d²(n - 1)2 E
(x,)eD l (x,
The first step (Equation (17)) follows from the fact that m(D)
((
1 E g(x,)) y) and (x(xx)) - E(x s g(x,y'
General pre-conditioner M: We follow similar steps as above:
E Enn. (D)
5 G2
BisseED MM(xxx) (x.8) - BIs. (2.8114 (2.)0MM88)))((8)1
25

---

Published as a conference paper at ICLR 2026
n M (xy))(D)
0² xyYED
n n-1 M_ (x,))((x,))
n-1
B(DD))DD
M (xs))(D n-1
n M (2,)
1
n
(similar to Equation (17))
1
n
(M_(x.)) -(sssteMM ((D)
(X.jED
1
M M (x.)))(()) (X,)ED (M- (x.)))8(8 l
n
(xyjED
72
M (x,y) - BL.sjED ((D)
M (x,y) y)( (B(b)))B((.)JDBBEEE ((
(xSSEE (M- M M 8(
31/²
BIx ||
- ((.) M(D)||²
()D
- B(
0²
11
3//²
1
| (x M M (x,))) g(x, 8) /|:
+
E (xx))
(18)
(n 1)
37²²
1
BixDDED M (g(x,)) (x 112
+ σ²
(n - 1)2
3
E
(xx))D
(n-1)
BIXDEEDD | (M (x,) (M
- B M(D)
372²
| BIXSEED M (g(x,y) - B 8) + 00 10
1
+ 9² (n- 1)2
(19)
Equation (19) uses assumptions of bounded gradient norms and stability of the pre-conditioner matrix to bound the term in Equation (18).
D.3 CAN WE CONNECT GRACE'S S PREDICTIVE BEHAVIOR OF STUDENT PERFORMANCE TO STUDENT'S S VALIDATION LOSS AFTER TRAINING?
In Lemma 1, we establish connection between leave-one-out conditional mutual information (LOO- CMI) and GRACE. Because LOO-CMI has been used to derive generalization bounds for training algorithms, primarily measured via the model's 's loss after training, we ask whether GRACE correlates with the student's 's loss after being trained on different teachers. To do so, we investigate two key questions: (a) does validation loss of the student after training correlate with student performance? and (b) does GRACE correlate with the validation loss of the student after training?
D.3.1 DOES VALIDATION LOSS AFTER TRAINING CORRELATE TO STUDENT PERFORMANCE?
For each teacher, create validation using the teacher's responses on the validation set of prompts. We use the following loss functions of the student after training on teacher's 's responses on a training set:
Validation loss is computed as the cross entropy loss of the student on the teacher's 's responses on the validation set.
Validation Train loss is computed as the difference in the cross entropy loss of the student on the teacher's responses on the training set and the validation set.
Forward KL computed as the difference in the cross entropy loss of the student and the teacher on the teacher's 's responses on the training set.
Backward KL is computed as the difference in the cross entropy loss of the teacher and the student the student's responses on the training set.
26

---

Published as a conference paper at ICLR 2026
Loss pre-training is computed the cross entropy loss of the student the pre-training data. Because we don't 't have access to pre-training data of LLaMA- IB, we use examples from fineweb-edu as surrogate (Lozhkov et al., 2024).
We measure correlations between each loss score and the student average-at-10 performance after training across all teachers.
Performance score (after training)
Absolute Spearman (1)
Regret (J)
Validation Loss
0.15
4.1 %
Validation Forward KI.
0.44
4.6 %
Validation Backward KL
0.14
12.9
Validation Train Loss
0.17
4.1 %
Loss on pre-training
0.07
6.5 %
Table 1: Setting: LLaMA-1B training GSM8K. Observation: None of the loss variants of the student after training correlate with the student performance. The performance scores have been computed the student after training on each teacher generations.
In Table we observe that none of the loss-based scores correlate well with student performance. Several factors may explain this. First, the validation sets differ across teachers due to variations in their of-thuhht reasoning styles, making the resulting losses across teachers non-comparable. More critically, there exists an inherent mismatch between the cross-entropy loss and the auto- regressive generation objective: cross-entropy loss measures the average token-level prediction error on fixed reference set, whereas auto-regressive evaluation tests whether the model can generate the correct solution through sequential token generation. Thus, a student can have low loss on the teacher's is generations after training, but struggle with getting correct auto-regressive generations. This discrepancy has been extensively discussed in prior works (Arora et al., 2022; Fang et al., 2025).
D.3.2 DOES GRACE CORRELATE VALIDATION LOSS AFTER TRAINING?
Here, we study whether GRACE correlates with validation loss of the student after training. To do so, we consider the correlations between the loss-based scores of the student after training, described in the previous section, and the scores GRACE, G- Vendii, and G-Norm computed on the gradients of student before training.
Performance score (after training)
G-Norm
G-Vendi
GRACE
Validation Loss
0.34
0.27
0.09
Validation Forward KL
0.10
0.28
0.15
Validation Backward KL
0.09
0.40
0.01
Validation Train Loss
0.0
0.08
0.23
Loss on pre-training data
0.03
-0.07
0.11
Table 2: Setting: LLaMA-1B 1B training on GSM8K. Here, we measure absolute spearman correla- tions between each loss-based scores with each of G-Vendi, GRACE, and G-Norm. Observation: GRACE doesn't correlate well with any of the loss variants of the student after training. GRACE, G-Vendi, G-Norm have been computed using the gradients of the student before training, and the performance scores have been computed on the student after training.
As shown in Table 2, GRACE exhibits only a weak correlation with the student's loss across teachers. Interestingly, G-Vendi and G-Norm show even stronger correlation with the student's loss-based scores. This suggests that the predictive effecttivnness of GRACE cannot be directly attributed to the student's validation loss after training.
Together, these gaps weaken the theoretical link between LOO-CMI and GRACE, leaving open the question of why GRACE correlates more strongly with student performance. Furthermore, since LOO- CMI only offers upper bound on the generalization error, it cannot serve direct comparative metric. A deeper theoretical understanding of how GRACE relates to student performance after training remains important avenue for future investigation.
27

---

Published as a conference paper at ICLR 2026
LLaMA
Gemma
Qwen
Qwen-Math
OLMo
Phi
Spearman: -0.41 Regret: 9.1%
Spearman: 0.48
Spearman: -0.74
Regret: 8.2%
Regret: 0.1%
Average-at 16 (in
36
36
36
32
32
log(G-Norm)
G-Vendi
log(GRACE)
Figure 10: GRACE achieves 74% Spearman correlation to OLMo-1B's post-distillation per- formance on GSM8K, significantly outperforming G-Norm (41%) and G- Vendi (48%). When evaluated by teacher-student regret, GRACE selects teacher with regret of 0.1%, outperforming G-Norm and G- Vendi, which incur regrets 9.1% and 8.2%, respectively. Stars denote students trained from the teacher chosen by each score. Similar observations hold for Gemma-2B student (Figure 11).
E
MORE EXPERIMENTAL RESULTS AND COMPARISONS
Select by GRACE
Select by Teacher Performance
Student
Student's Performance
Student's Performance
Teacher chosen
Teacher chosen
LLaMA-1B
Owen2.5-3B-Instrucc (T = 0.9)
52.2
I.LaMA.3-77BB-nmmne Instructt
44.8
(T = 0.8)
GSM8K
OLMo-IB
2wen2.5-3B-Instteee
46.0
LLaMA3.3-70B Instruct (T = 0.8)
36.1
(T = 0.9)
Gemma-2B
Qwenn2.53BB-Intrrueee (T 0.6)
69.0
LLaMA3.3-70B-IIstme.
62.4
(T 0.8)
MATH
LLaMA-3B
Phi-4
36.0
Qwen2.5-7B-Inss ----tutttttt
34.0
(7 0.8)
(r 0.4)
Table 3: Summary of student performance (Average-at-16) under teachers selected with GRACE and teacher performance from Figures 2, 10, 11 and 14.
E. ADDITIONAL RESULTS
First, show the behavior of Gemma-2B, when trained with different teachers in Figure 11. We also observe that GRACE has higher Spearman correlation to the student performance, compared to G-Norm and G-Vendi.
GSM8K:
We also report the performance when we allow more computation for the computation of GRACE. In Figures 12 and 13, show that the final Spearman correlation improves by atleast 7% and 4% for OLMo-IB IB and LLaMA- 1B respectively on GSM8K, when we increase the projected dimension of gradients from 512 1024.
In Figure 14, we show the scatter plot of the performance of the model when trained with different teachers. We show that G-Norm and GRACE achieve 89% and 88% Spearman correlation to the student performance post distillation training, while G-Ven Vendi only achieves 74% Spearman correlation.
MATH:
28

---

Published as a conference paper at ICLR 2026
Liama
gemma
Qwen
Qwen-MMath
OLMo
phi
Spearman: -0.69
Spearman: 0.53 Regret: 13.4%
Spearman: -0.89
Regret: 5.0%
Regret: 0.1%
Average-
80
60
-3.6
log(G-Norm)
-10 -24
4.5
G-Vendi
5.0
55
4.0
log(GRACE)
56
Figure 11: GRACE achieves 89% correlation to Gemma-2B performance after training GSM8K, across all teacher, generation temperature combinations. G-Norm and G- G-Vendi can achieve 69% and 53% correlation respectively. Here, n 512, d 512 used compute all scores. When evaluated by regret, GRACE selects a teacher that trains the student to within 0. of the best achievable performance, outperforming G-Norm and G- G-Vendi, which incur regrets of 5% and 13.4%, respectively.
LLaMA
Gemma
Qwen
Qwen-Math
OLMo
Spearman: -0.69
Spearman: 0.53
Spearman: -0.89 Regret:
Regret:
Regret: 13.4%
Average-ate %6)
85
65
55
-30
5.0 G-Vendi
4.0
5.4
log(G-Norm) 1
log(GRACE)
Figure 12: Repeated experiment from Figure 2 but with 1024. GRACE achieves 90% corre- lation to LLaMA-1B performance after training GSM8K, across all teacher, generation temperature combinations. G-Norm and G- Vendi can only achieve 53% and 47% correlation respectively. When evaluated by regret, GRACE selects teacher that trains the student to within 0.3% of the best achievable performance, outperforming G-Norm and G- Vendi, which incur regrets of 10.8% and 14.5%, respectively.
E.2 MEASURING EFFECTIVENESS OF GRACE WITHIN EACH TEACHER FAMILY
We measure the regret of teacher selection using each within individual teacher familics. For a given family, the regret of a score defined as the performance gap between (i) the best student performance achieved after distilling from all teachers in that family and (ii) the student performance obtained when the teacher is chosen according to that score. On average, GRACE yields only 1.04% regret, while other metrics incur least 3%. See Figure 15 for details.
E.3 OUT-OF-DISTRIBUTION EVALUATION
One question whether GRACE can also predict the performance the trained teacher in out-of- distribution settings. To check, we measure the performance of the trained LLaMA-1 models (trained on GSM8K) on GSM-Symbolic (Mirzadeh et al., 2025) and measure the Spearman correlations and regrets of G-Norm, G-Vendi, and GRACE with respect the OOD performance (Figure 16). Similarly, we perform OOD evaluation of LLaMA-3B models (trained on MATH) on MATH (Shah et al., 2024) and MATH Perturb (Simple & Hard) (Huang al., 2025b), and measure the Spearman correlations and regrets of G-Norm, G- Vendi, and GRACE with respect to the OOD performance (TTble 4). We observe the following:
29

---

Published as a conference paper at ICLR 2026
LLaMA
Gemma
Qwen
Owen-Math
OLMo
Phi
Spearman: -0.40 Regret: 9.1%
Spearman: 0.50 Regret: 8.2%
Spearman: -0.81
Regret: 0.1%
Average-at 16 (in
36
36
36
32
32
log(G-Norm)
G-Vendi
log(GRACE)
Figure 13: Repeated experiment from Figure 10 but with d 1024. GRACE achieves 81% correlation to OLMo-1B performance after training on GSM8K, across all teacher, generation temperature combinations. G-Norm and G-Vendi only achieve 40% and 50% correlation respectively. When evaluated by regret, GRACE selects a teacher that trains the student to within 0.1% of the best achievable performance, outperforming G-Norm and G-Vendi, which incur regrets of 9.1 % and 8.2%, respectively.
LLaMA
Gemma
Owen
Qwen- Math
OLMo
Spearman:
Spearman:
Spearman: -0.88
Regret:
Regret:
Regret:
Averege-at-16
68
5.5 5.6
4.8
5.7
log(GRÅCE) 2.4
log(G-Norm)
G-Vendi T
Figure 14: GRACE achieves 88% correlation to LLaMA-3B performance after training on MATH, across all teacher, generation temperature combinations. G-Norm and G-Vendi can achieve 90% and 74% correlation respectively. Here, n = 512, d = 512 are used to compute all metrics. When evaluated by regret, GRACE selects teacher that trains the student within 3.9% of the best achievable performance (similar to G-Norm), outperforming G- G-Vendi which incurs regret of 12.4%.
On ILaMA-IB, GRACE has both high Spearman correlation (78%) and low regret (0.8%) compared to G-Norm and G-Vendi. Thus, good in-distribution performance of the student also translates to good OOD performance.
On LLaMA-3B, we observe a more nuanced pattern. G-Norm attains higher Spearman correlation and lower regret than GRACE on MATH Perturb (Simple), which constitutes relatively mild out-of-distribution shift. contrast, GRACE matches or surpasses G-Norm the more challenging MATH and MATH Perturb (Hard) settings. These results sug- gest that GRACE provides a reliable indicator student performance under substantial distribution shifts.
E.4 How DOES KNOWING THE TEACHER PERFORMANCE CHANGE THE BEHAVIOR OF THE DIFFERENT SCORES?
Our main results focus the setting where no verifier is used to pre-filter the teachers. A key question is whether the teacher's 's own performance influences the correlation between GRACE and student performance, and if teacher performance predictive power of various scores. To examine this, we analyze how the behavior of GRACE varies when teachers grouped into different performance bins.
known, whether it could further improve the
30

---

Published as a conference paper at ICLR 2026
Regret in Average-at-16(%9
15
Metric
Avg Regrets Across all ramilies:
G-Norm: 3.02
G-Norm
G-Vendi: 3.25
G-Vendi
10
GRACE: 1.04
Loss
Loss: 4.05
GRACE
5
0
LLaMA
Qwen
Qwen-Math
Gemma
OLMo
Phi
Teacher Family
Figure 15: Here, we measure the regret incurred when a score is used to select a teacher within each teacher family, On average, GRACE identifies teacher that enables the student achieve performance within 1.04% of the best performance possible from distillation with teacher in that family, whereas other scores incur an average regret of least 3%.
LLaMA
Gemma
Qwen
Owen-Math
OLMo
Phi
Spearman: -0.44
25.0
Spearman: 0.29
Spearman: -0.78
Regret: 7.8%
Regret: 8.5%
Regret: 0.8%
Average-at-11 (in
22.5
22.5
22.5
20.0
20.0
20.0
17.5
17.5
-5.6
-5.2
4.4
4.8
5.2
1.0
1.5
2.0
2.5
log(G-Norm)
G-Vendi
log(GRACE)
Figure 16: Out-of-distribution results show consistent trends on GSM8K. We further evaluate the LLaMA-1B students (trained on GSM8K) on the SyMbolic dataset (Mirzadeh al., 2025), an out-of-distribution benchmark relative to GSM8K. GRACE achieves Spearman correlation of 78% with student performance, outperforming both G-Norm and G- G-Vendi. As expected, this slightly lower than the 86% correlation observed in-distribution (Figure 2). In terms of regret, GRACE achieves the lowest regret (0.8%), outperforming both G-Norm and G- Vendi in this OOD setting.
To verify, we divide all our teachers into three equal bins based on their own performance. In each bin, we compute the Spearman correlation of each metric to the student performance post distillation training. We also report the regret of a score within each bin as the performance gap between (i) the best student performance achieved after distilling from all teachers in that performance bin and (ii) the student performance obtained when the teacher is chosen according to that score.
In Figure 17, we show that GRACE shows better Spearman correlation and also lower regret, across all the performance bins for LLaMA-1B training on GSMSK. Advantage GRACE are particularly more pronounced for distinguishing high performing teachers.
In Figure 18, for LLaMA-3B training MATH, GRACE maintains stronger Spearman correlation with student performance across all bins. However, G- Vendi shows lower regret teacher bin.
than GRACE in the
E.5 MEASURING CORRELATIONS WITH OTHER MEASURES OF STUDENT PERFORMANCE
We primarily measured the correlation of the metrics to Average-at- 16 performance of the student post distillation, when generate responses at temperature 1.0. A key question is: how do the correlations change when the performance metric of the student is changed? To verify, we use two other commonly used performance metrics greedy student performance and Best-of- 16 performance.
31

---

Published as a conference paper at ICLR 2026
Evaluation dataset
Best Student Average-at-Ie
Score
Absolute Spearman (1)
Regret (4)
G-Norm G-Vendi
0.89
2.8 %
MATH Perturb (Simple)
31.32%
0.59
16.7%
GRACE
0.76
2.8 %
G-Norm 0.87
0.1 %
MATH
9.36%
G- G-Vendi
0.66
3.9%
GRACE
0.88
0.1%
G-Norm 0.80
2.1 %
MATH Perturb (Hard)
13.72%
GeVendi 0.75
5%
GRACE 0.88
2.1%
Table 4:
results show consistent trends on MATH. We further evaluate the LLaMA-3B trained students (on MATH) on MATH' (Shah et al., 2024) and MATH Perturb (Simple & Hard) (Huang et al., 2025b), out-of-distribution benchmarks relative MATH. While G-Norm yields similar or higher Spearman correlation and lower regret than GRACE on MATH Perturb (Simple). mild OOD benchmark, GRACE outperforms G-Norm both MATH and MATH Perturb (Hard), which represent substantially more challenging OOD shifts.
Correlations per Teacher-Perf. Range(
Regret in ( J
G-Norm
0.99
G-Var
Absolute Spearman
G-Vendi
12.5
G-Vendi
0.8
Loss
GRACE
10.0
GRACE
0.6
0.58 0.50
7.5
0.4
0.4
5.0
4.98
0.28
4.03
0.2
2.5
2.38
0.0
0.0
0.00
0.00
2.01
0.01
(56.96,
83.57]
(83.57, 91. 26]
191.26, 95.95]
(56. 96. 83.57]
(83.57, 91 26]
(91.26, 95.95]
Teacher Average-at-16 (in %)
Teacher Average-at-16 (in %)
Figure 17: Setting: LLaMA-IB training on GSM8K. When divide teachers into three equal bins based on their own performance, we compare both Spearman correlation and Regret. Across all bins, GRACE shows stronger Spearman correlation with student performance after distillation and consistently shows minimum regret. Its advantage is more pronounced when distinguishing among high-performing (top performance bin) teachers.
Greedy performance measures correctness of single student's generation response per prompt at temperature 0.0. Best-of-16 performance measures whether the student gets atleast one correct response, when generating 16 responses per prompt at temperature 1.0. In Table 5, we observe that the Spearman correlation of GRACE to the student performance drops, when switch performance measure to greedy accuracy or Best-of-16 performance. However, GRACE still achieves small regret with the new performance metrics, compared G-Norm and G-Vendi.
Metric
Average-at- 1.0)
Greedy accuracy
Best-of-16 (T 1.0)
Abs. Spearman (t)
Regret (L)
Abs. Spearman (t)
Regret (L)
Abs. Spearman
Regret
(t)
(L)
G- Vendi
0.44
14.5 %
0.65
4.5%
0.68
4.5%
G-Norm
0.55
4.9 %
0.33
13.5%
0.30
7.5%
GRACE 0.86
0.3%
0.70
2.4%
0.64
0%
Table 5: GRACE leads to stronger students in terms of various performance measures. We compare G-Vendi, G-Norm, and GRACE using average-at-Ils 16, greedy, and best-of-16 performance of the post-distillation students. Results are on LLaMA-IB training on GSM8K (setting of Figure 2).
GRACE achieves the strongest close stronger Spearman correlation, and consistently leads the lowest regret compared to G-Norm and G- Vendi
32

---

Published as a conference paper at ICLR 2026
Correlations per Teacher-Perf. Range( 1
Regret in Average-at-16 (%)
Absolute Spearman
Vondi
GRACE
Regret: (absolute 3k)
Vendi
GRACE
0:84
3.625
2,412
0.39
0.575
0.575
(48-75)
(75-85)
(48-75)
(75-85)
Figure 18: Setting: LLaMA-3B training on MATH. We split teachers into two equal bins according their own performance and evaluate each bin using both Spearman correlation and regret. Across bins, GRACE maintains a stronger correlation with the student's post-distillation performance. However, the top-performing bin, G- Vendn achieves lower regret than GRACE
Absolute Spearman (1)
Regret (L)
Student Temperature
G-Norm
G-Vendi
GRACE
G-Norm
G-Vendi
GRACE
0.0
0.34
0.63
0.70
13.5
4.5
2.4
0.3
0.42
0.65
0.75
9.2
4.1
0.2
0.6
0.51
0.62
0.83
9.5
8.5
1.2
0.8
0.58
0.57
0.87
8.0
8.3
0.3
1.0
0.44
0.56
0.86
10.8
14.5
0.3
Table GRACE can guide temperature choice. We compare G-Norm, G-Vendi, GRACE in terms
of the student's s average-at-le accuracy at various temperatures. Results are on LLaMA-1B training on GSM8K (setting Figure 2). GRACE achieves the highest Spearman correlation (0.7 and above) and the lowest regret across all temperatures, demonstrating robustness to the temperature choice.
E.6 PRELIMINARY EXPLORATION: GENERAL NLP TASKS BEYOND MATH
One question is whether the predictive nature of GRACE holds beyond math based tasks. We study ARC (Clark al., 2018). ARC a general reasoning task, where the model is given question and choices to choose from to answer. An example of question in ARC is as follows:
A fold observed in layers of sedimentary rock most likely resulted from the:
(a) cooling of flowing magma.
(b) converging of crustal plates.
(c) deposition of river sediments.
(d) solution of carbonate minerals.
Experiment Setting: We train a LLaMA-IB model and use the same set of teachers as in our GSM8K and MATH experiments, considering subset of generation temperatures 0.4, 0.6, 0.8, 1.0. Each teacher is prompted to provide the correct answer explicitly, rather than just the choice number. Nonetheless, some teachers still respond with the choice number only. To account for this, evaluation accepts either the choice number or the full answer. For choice numbers, we apply exact match, while for textual answers, we compute the edit distance each of the four options to determine correctness. We sample 16 responses per prompt from each teacher to construct the training dataset and evaluate performance on the ARC validation split.
We observe that the teacher's performance itself is strong indicator of the student performance after training. However, among all scores that do not assume access to such verifier based filtering on the teacher, GRACE returns the best correlation to the student performance (however, lower than Spearman correlation could achieve on math based tasks). Among all scores (teacher performance included), GRACE returns lower regret in selecting the best teacher for training the student.
Primary observations:
These observations indicate that GRACE is not perfectly predictive measure of student performance after training. However, it remains highly effective for selecting strong teachers, consistently yielding lower regret compared to other metrics. For instance, in our experiments. Gemma-2B teachers exhibited much higher (worse) GRACE scores (10-3) than Qwen-Math models (10-4) yet students distilled from Gemma-28 teachers (56.4%) outperformed those trained from Qwen-Math (41.0%).
33

---

Published as a conference paper at ICLR 2026
LLaMA
Gemma
Qwen
Qwen-Math
OLMo
Phi
Spearman: -0.31
Spearman: 0.62 Regret: 4.5%
Spearman: -0.69
Regret: 13.5%
Regret: 2.4%
Greedy accurage
log(G-Norm) 1 -18
3.5 20 log(GRACE)
G-Vendi
Figure 19: Repeated experiment from Figure 2 but greedy performance of trained student model. GRACE achieves only 70% correlation to LLaMA-1B performance after training on GSM8K, across all teacher, generation temperature combinations. This is sharp reduction from 86% correlation to Average-at-16. However, GRACE still predicts close to the optimal teacher.
LLaMA
Gemma
Qwen
Qwen-Math
OLMo
Phi
Spearman: -0.28
Spearman: 0.68
Spearman: -0.64
Regret: 7.5%
Regret: 4.0%
Regret: 0.0%
Best-of-18
-5.
log(G-Norm) 1
G-Vendi
log(GRACE)
Figure 20: Repeated experiment from Figure but best-of-16 performance of trained student model. GRACE achieves only 65% correlation to LLaMA-1B performance after training on GSM8K, across all teacher, generation temperature combinations. This is sharp reduction from 86% correlation to Average-at-16, However, GRACE still predicts the optimal teacher.
Conversely, directly choosing the highest-performiin teacher (Llama-700) Monnaa-7-Biinstttee led to student with an average-at-le score of 58.6%, whereas selecting based on GRACE produced a student reaching 59.5%.
This pattern stands in contrast to findings math-related tasks, where GRACE highly predictive student performance after distillation and also could reliably identify close-to-optimal teacher. We keep extensive search beyond math-related tasks to future work.
Overall, GRACE remains a reliable score teacher selection, even though teachers with low GRACE values may yield strong students lendents-leading to slightly weaker correlations. Identifying the conditions under which GRACE serves as a near-perfect predictor of student performance remains an important open question for future work.
E.7 COMPUTATIONAL COMPLEXITY
GRACE is computationally inexpensive to compute. As shown in Table 8, for n = 512 and 4, the gradients for each model takes around 10 minutes compute and around 4.3MB to store. Computing the GRACE score given the gradients of the student w.r.t. teacher inexpensive and takes roughly 10 seconds.
34

---

Published as a conference paper at ICLR 2026
Score
Absolute Spearman (1)
Regret (4)
(Best Average-at-16
student: 61 61%)
G-Vendi
0.11
19.1 %
G-Norm
0.43
2.4
GRACE
0.56
1.5%
Determinant
0.09
28.9 %
BADGE
0.19
28.9 %
Gradient inner product
0.09
6.4 %
Gradient cosine
0.19
28.9 %
Student's s loss on teacher's generations
0.14
12.9%
Average (token) probabilities
0.09
11.5%
Average length
0.06
6.4%
Teacher average-at-Ie 16 performance
0.82
2.4 %
Table 7: LLaMA-IB training ARC. Among all scores that do not rely on verifier for teacher filtering, GRACE achieves the best Spearman correlation and regret identifying the optimal teacher. While teacher performance, that assumes access verifier, shows much stronger correlation with student performance after training, GRACE still yields lower regret and remains the more effective criterion for teacher selection.
Gradient Features Computation
Metrie Computation
Computation complexity Running time
On m P-d) ~ 10 minutes
O(n m d2 + d < 10 seconds
Storage Complexity Actual storage
On m d)
4.3 MB
Table Time complexity to compute GRACE. The running time and the actual storage have been computed on ñ 512, 4, d 512 for LLaMA- IB training GSM8K, and have been reported as a rough average across all settings. Wall-clock time has been reported on a single H100 (80 GB) GPU. For gradient computation, P denotes the number of parameters in the model.
32 parallel CPU threads following Park et al. (2023). Here,
F ABLATIONS
F.1 FILTERING v/s NO FILTERING
In our experiments in the main paper, we perform no filtering of the responses from the teacher. Here, we compare to the case when we filter the teacher's responses by correctness. We sample 16 responses from each teacher and remove the incorrect responses. Then, we sample with repetition to get a set of 16 responses to train the model.
First, we find that the student gets worse performance with filtering correct responses from the teacher (Figure 22). However, we find that when we compare our metrics to the student performance after training, we find that our metries have slightly higher spearman correlation with the student performance when we train with filtering teacher responses, compared to student trained with no filtering on the teacher responses (Figure 23).
F.2 ABLATION ON TRAINING HYPERPARAMETER.
We observe that Llama- 1B model trained generations of Llama-70B Instruct models and Gemma- 2-27B Instruct models perform badly. We train with learning 1e0-5 on the 16 generations per prompt of the teacher for 4 epochs. One primary question is whether the small model is over-optimizing on the teacher's generations. To check this, we track the train and test performance of the trained model with varying number of generations (Figure 24) and epochs of training (Figure 25). We observe that the performance of the trained student model improves with increasing number of epochs and number of generations, implying over-optimization in our training setting.
35

---

Published as a conference paper at ICLR 2026
Liama
gemma
Qwen
Owen-Math
OLMo
phi
Spearmen: Regret: 6.4%
Speerman: 0.47
Speermen: - Regret: 0.0%
Regret: 2.8%
Average-ar- (in
45
8
-3.2 log(G-Norm)
G-Vendi
6.8
14 log(GRACE)
Figure 21: Qualitative study: GRACE effectively correlate with student performance when compared across different teacher choices. Here, we report LLaMA-1B performance on GSM8K across different teacher choices at generation temperature 0.6. G-Norm capture differences between Llama and Qwen teacher families, however can 't differentiate between Qwen, Qwen-Math and Phi teacher families. G- G-Vendi provides better separation among these families, but assigns higher scores to sub-optimal teachers. GRACE achieves 90% correlation with student performance after training, while also predicting Qwen-3B Swwen33-IInstrue be the optimal teacher.
Average-ar- aereear--6/6 Filtering
50
48
46
44
42
40
40
45
50
Iverage-at-16(%) No filtering
Figure 22: Comparing teachers, when we filter correct responses from the teacher v/s when we don't filter correct responses from the teacher. Here, train LLaMA-IB on GSM8K with 15 teachers and generation temperatures 0.4, 0.6, 0.8, 1.0. We compare students trained from teacher without filtering (x-axis) with students trained from teacher with correct answer filtering (y-axis). We find that students trained with no filtering outperforms models trained with filtering.
F.3 ABLATIONS ON THE PARAMETERS OF GRACE
In Figure 26, we use LLaMA-I1 IB trained on GSM8K as a case study and ablate two key factors affecting GRACE: the gradient projection dimension (d) and the number of prompts (n) used to compute the score. With a fixed n, the correlation between GRACE and student performance consistently improves as d increases. Conversely, when holding d fixed, correlation increases with n, peaking when n d/2.
We further vary the number of generations per prompt (m) while keeping n and fixed. In our base configuration, using m = 2, 4, 8 yields correlations of 0.70, 0.86, and 0.87, respectively, suggesting that 4 already provides sufficiently strong estimate of GRACE.
We additionally vary the number of cross-validation splits used in GRACE. As shown in Figure 27, the correlations the student performance do vary much for both GSM8k and MATH for more than C' 6 splits. We take C' 10 as the default.
F.4 GRADIENT NORM'S ' RELATION TO LENGTH
Figure 28 shows that the norm of the gradient (averaged over tokens) on a generation decreases as the generation length grows, roughly following trend of 1/ log T for length-T generations, consistent
36

---

Published as a conference paper at ICLR 2026
LLaMA
Gemma
Owen
Qwen-MMth
OLMo
Phi
Spearman:
Spearman:
Spearman:
Regret:
Regret:
Regret:
40
Averagee-ttia
-5.6
log(G-Norm) 1
-52
-18
4.4
4.8 G-Vendi
5.2
log(GRACE)
15
2.0
25
LLaMA
Gemma
Qwen
Qwen-Math
OLMo
Spearman: -0.64
Spearman: 0.45
Spearman: -0.87
Regret: 6.6%
Regret: 14.5%
Regret: 0.0%
80
45
Aver Average-en 16
40
log(G-Norm)
G-Vendi
(og(CE))E)
Figure 23: Comparisons between the metrics and the student performance when we filter responses v/s we don't filter correct responses from the teacher. Here, we train LLaMA-IB on GSM8K with 15 teachers and generation temperatures 0.4, 0.6, 0.8, 1.0. We find that our metrics have slightly higher spearman correlation to the student performance when we filter correct responses from the teacher and train only on them.
gemma-2-22 2-27b-it
30
Average at-16 %6
16 (In %6)
22
Test 0.6)
Average 16
Train (r-0.6)
Test (r-0.8)
Train 0.8)
2.0
25
1.0
4.
1.0
2,2
Epochs
Epochs
Figure 24: LLaMA-1B training GSM8K with 16 responses per prompt of gemma-276 maauma-77b-isstuue
and 1lama-70b instruct model. We vary the number of epochs and observe that both train and test performance improves with more epochs of training.
gemma-2-27b-4
Llama-3.
Average 15 (in %))
Average 16 (In %)
Test (T=0.6)
Test (T=0.6)
22-
Train (T=D.6)
(T=0.6)
82
Test (r=0.8)
Test (T=0.8)
Train (r=0.8)
Train (r=0.8)
32
Number generations:
Number of generations: k
Figure 25; LLaMA-1B training on GSM8K with varying number of responses per prompt gemma- 27b-instruct and llama-700 instruct model. We observe that both train and test performance improves with more training samples from the teacher.
37

---

Published as a conference paper at ICLR 2026
Correlation vs Gradient Dimension
Correlation vs Number of prompts
Correlation
Correlatior:
0.3
0.00
0.75
0.81
0.65
0.80
3x
mension (d)
Number of prompts (n)
Figure 26: Varying hyperparameters for GRACE on LLaMA-IB IB training on GSM8K. We use the base setup 512, and 512. We vary one of them, while fixing the other. Main takeaway: (a) GRACE improves with increasing gradient dimension, (b) GRACE generally increases with number of prompts that we consider but shows a small dip as we increase further. n d/2 gives the best results for spearman correlation.
correction
0.895
0.866 0.857 0.855 0.858 0.858 0.856 0.855 0.851
Abselutee Sww.uucceeeeeeee mIME correctton
0.854 0.858
0.872 0.867 0.869
0.0
0.8
0.737
0.6
0.4
0.4
0.2
0.2
0.0
10
20
splits
Figure 27: Varying number of cross-validation splits on GSMSK (left) and MATH (right).
with observations in Xia et al. (2024). Intuitively, this is likely because longer generations tend to contain larger fraction less important tokens that do contribute much to the overall gradient. This observation motivates the log T scaling in Section 2.
F.5 ABLATION ROBUSTNESS OF METRICS
We check the robustness of each metric by reporting the distributions of the metric values computed over random subsets teachers. Specifically, we use 100 random draws of subsets consisting of 60% of teachers.
0.06
(x.y) = (seq length, grad norm)
0.05
1/log T
Gradient norm
0.04
0.03
0.02
0.01
0.00
250
500
750
1000
1250
1500
1750
2000
Sequence length
Figure 28: Gradient norm decreases inversely with log T, where T is the sequence length. This motivates the gradient scaling in Section
38

---

Published as a conference paper at ICLR 2026
We compare GRACE against the baselines listed in Section C.3, with an additional score of the average probabilities of the correct responses. Among all candidate metrics, GRACE is the only one showing consistently strong correlations.
0.538)
Sear: pari performence (median 0.118)
Loss Imadiare .0.455)
GoVendi 0.033)
820
Combees
Camtateer 0.539)
Comeleton
Conelater
G-Nom
(medaa::
Grad (median:: 0.564)
20
10
-5
410--75--55-15
Cemei
Conmaber
cameacan
Conmater
only)
-9
-9 -50--475-039
-.00-
Comsator
Comiidann
Cinsater
Figure 29: Robustness of scores: We report the distribution of scores for LLaMA-IB on GSM8K. computed over 200 random subsets of teacher-temperature combinations, where each subset is 60% of the full set. The dashed lines and numbers in parentheses mark the median. The proposed GRACE score consistently shows stronger correlations over all other metrics.
39

---

Published as a conference paper at ICLR 2026
G PROMPT FORMAT
We train the student model with the following format across distillation from all teachers:
# #### # Problem: {input}
### # Solution: {output}
One example is follows:
Chandra has four bowls. Each is different color (red, blue, yellow, green). She also has exactly one glass the same color each bowl. If she chooses bowl and glass from the cupboard, how many pairings are possible? One such pairing is blue bowl and yellow glass. ### Solution:
### Problem:
determine the total number of possible pairings between bowls and glasses,
we need to consider the number choices available for each item and then multiply these choices together.
Chandra has 4 bowls, and each bowl can be paired with any of the 4 glasses.
Therefore, the number of possible pairings is calculated by maltiplying the number of choices for the bowl by the number of choices for the glass.
This can be expressed as:
AM 4 \ (choices for bowls) Wtimes 4 Altext (choices for glasses)) 16 WV
Thus, the total number of possible pairings is (bxleexxe \endiverbatim
We use the following formats to get responses from each teacher family, which have been taken from their model-cards huggingface. We use vLLM 0.7.0 ((Kwon al., 2023)) to get responses from all models.
Prompt to LLaMA teacher models
<lstart_headess [
>system<|ees lnd hed_eleeee________e__eeeeee Seader_id|>Yee are a helpful
<lstart__eader_id] >user<| end_header_id
Please reason step by step, and put your final answer within \bboxed(11)
(input)
<leot_id|>
ilstart_headess id) >assistant hee i
40

---

Published as a conference paper at ICLR 2026
Prompt to Qwen teacher models
< <lim sim_ssarrrll_____ -system)nYoo are a helpful assistant. ii__end]
< <lim_start >user
Please reason step by step, and put your final answer within (bboxed(()):
(input)
< <lim_end|>
< <lim_staal start >assistant
Prompt to OLMo teacher models
<endoftext >>< user
Please reason step by step, and put your final answer within (bcxed(()).
(input) <lassistant >>
Prompt to Phi teacher models
< <lim_start /
You are a medieval knight and must provide explanations to modern people.
< [lim_end]>
< tim_starr/>>sserreee im_sep|>>>>>>>>>>>
Please reason step by step, and put your final answer within (1b
{input)
< <lim_s [im__nd|>
< lii_sstart baattt/sssssaattttt jmm i
Prompt to Gemma teacher models
<bos><start of httssttart_oo_tuuuusssstsee turn>uueee
You are a helpful assistant. Please reason step by step, and put your final answer within (1)
(input)
<end of turn estart_of_turm>mmd
G.1 QUALITATIVE ANALYSIS OF RESPONSES FROM DIFFERENT TEACHERS
Here, we give qualitative analysis of responses from each teacher. In the table below, we show the average length of responses from different teachers. We observe that Gemma models give very concise responses, which leads very high G-Norm and GRACE (in Figure 2). Response length can differentiate between the responses different teacher families, however, it's not a perfect predictor (Figures and 4).
We give examples responses from Qwen-2.5 3B Instruct, LLaMA-3.3 70B Instruct, and Gemma-2 9B Instruct teachers to question. We find that even though both Qwen-2.5 3B Instruct and LLaMA- 3.3 70B Instruct qualitatively give similar amount of details in each step in their responses, Qwen teachers provide more details in their numerical computations.
41

---

Published as a conference paper at ICLR 2026
Teacher model
Average response length (across all temperatures)
LLaMA
1LaMA-3.1-8B Instruct
212.083
LLaMA-3.2-3B Instruct
224.492
LLMMA-333-70B Instruct
222.314
Qwen
Qwen2.5-1.55 Instruct
259.411
Qwen2.5-3B Instruct
289.952
Qwen2.5-7B Instruct
271.666
Qwen2.5-14B Instruct
287.611
Qwen-Math
0wen2.5-Math-1. Instruct
272.988
Qwen2.5-Math-78 Instruct
332.1 119
Gemma-2
Gemma-2-2b-4
166.888
Commma-2-9bit
138. 4622
Gemma-2-27b-it
140.294
OLMo-2
OLMo-2-77 M-Moo-2-BB Instruct
286.604
OLMo-2-13B Instruct
252.683
Phi
Phi-4
293.160
Table 9: Average response lengths (in terms of number of tokens under the tokenizer of LLaMA-1B) across all teachers on GSM8K on responses to subsampled set of 512 prompts from the training set.
Question
Janet's s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
Response from Gemma-2-9B Instruct (temperature=0.6)
Here's how to solve the problem step-by-step:
**gggs used:** Janet uses eggs for breakfast. 4 eggs for muffins = eggs per day.
2. **Eggs remaining:** She has 16 eggs 7 eggs 9 eggs left for the market.
3. **Earnings:** She sells 9 eggs $2 per egg $18.
(boxed($18) $18)
42

---

Published as a conference paper at ICLR 2026
Teacher model
Average response length (across all temperatures)
LLaMA
1-aMA-3.1-8B Instruct
519.514
LLaMA-3.2-3B Instruct
537.69
LLMMA-333-70B Instruct
529.951
Qwen
Qwen2.5-1.55 Instruct
572.480
Qwen2.5-3B Instruct
581.836
Qwen2.5-7B Instruct
561.407
Qwen2.5-14B Instruct
587.693
Qwen-Math
Owen2.5-Math-1.. Instruct
531.055
Qwen2.5-Math-78 Instruct
784.821
Gemma
Gemma-2-20B Instruct
357.610
Gemma-2-9B Instruct
341.838
Gemma-2-27B Instruct
355,279
OLMo
OLMo-2-75 Instruct
632.502
OLMo-2-13B Instruct
589.932
Phi
Phi-4
666.602
Table 10: Average response lengths (in terms of number of tokens under the tokenizer of LLaMA-3B) across all teachers on MATH on responses to subsampled set of 512 prompts from the training set.
Response from LLaMMA-33--70 Instruet (temperature=0.6)
To find out how much Janet makes every day at the farmers' market first need determine how many eggs she sells there. Here's the step-by-step process:
1. ++ooaal eggs laid per day**: Janet's ducks lay 16 eggs day.
2. *ggg eaten for breakfast**: She eats 3 eggs for breakfast every morning.
3. **Eggs used for baking muffins**: She uses eggs to bake muffins for her friends every day.
4. **ootal eggs used**: The total number of eggs she uses (eats and bakes) is 3 (for breakfast) + (for muffins) eggs.
++gg sold at the farmers' market**: The number of eggs she sells at the market is the total number of eggs laid minus the eggs she uses. So, 16 (total eggs) (eggs used)
9 eggs sold.
6. Money made from selling eggs**: She sells each egg for $2.
Therefore, the amount of money she makes from selling 9 eggs is 9 eggs $2/egg = $18.
So, Janet makes $18 every day at the farmers' market. The final answer is: $\ $V\boxe[[18B$

---

dollars every day at Published as a conference paper at ICLR 2026
Response from Qwen-2.5-3B Instruct (lemperature=0.6)
To determine how much Janet makes every day the farmers' market, we need to follow these steps:
1. Calculate the total number of eggs laid by the ducks per day.
2. Subtract the number of eggs Janet eats for breakfast.
3. Subtract the number of eggs used for baking muffins.
4. The remainder will be the number of eggs sold
at the farmers' market.
5. Multiply the number of eggs sold by the price per egg to find
out how much she makes each day.
Let's go through each step in detail:
The ducks lay 16 eggs per day.
2. Janet eats 3 eggs for breakfast. Janet uses eggs for baking muffins.
First, we calculate the number eggs remaining after Janet's s consumption and baking:
AM
16 \ (total eggs) - 3 Wtext( (eaten for breakfast) \ltext (used for baking) } Altext (remaining eggs) W
Next, Janet sells the remaining eggs at the farmers' market for $2 per egg. So, we calculate her earnings:
WM
9 Atext (remaining eggs) Wtimes 2 \ltett (dellars per egg) \ltext { (dollars)
Thus, Janet makes
the farmers' market.
44


Reviews:
[{"review_id": "VNFatMnAVw", "reviewer": "Reviewer_EhFJ", "summary": "The paper introduces GRACE, a lightweight score designed to optimize teacher selection for knowledge distillation in math problem-solving tasks. \nGRACE quantifies the effectiveness of a teacher model by measuring the distributional properties of student gradients, without needing access to teacher logits, internals, or test data. \nIt is linked to generalization performance and correlates strongly with the performance of distilled student models (up to 86% on GSM8K and MATH datasets). \nGRACE improves student performance by at least 6% over traditional methods that use the best-performing teacher. \nAdditionally, GRACE provides valuable insights on key distillation decisions, such as selecting the optimal teacher, generation temperature, and model size. \nOverall, GRACE offers a more efficient and effective approach for teacher selection in distillation.", "questions": "- Is there any possibility of generalizing to other tasks or any insights benefiting other tasks?\n\n- Would it be better to incorporate comprehensive information?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- GRACE offers an efficient method to identify the most suitable teacher for distillation, improving the distillation process without requiring extensive trial and error.\n\n- The proposed GRACE strongly correlates with student performance, as demonstrated by its high correlation (up to 92%) with post-distillation results, out-performing traditional metrics like G-Vendi and G-Var.\n\n- It provides actionable insights to practitioners, including optimal generation temperature and teacher selection under size or model family constraints.", "weaknesses": "- While effective, GRACE's correlations with student performance are not perfect, indicating that further refinement and additional explanatory factors might improve its accuracy. \n\n- The score’s performance can be sensitive to hyperparameters, such as the number of prompts and generation dimension, which can complicate its application.\n\n- The current task is limited to Math problem-solving, however, this is a small task within the NLP and AI domain. Is there any possibility of generalizing to other tasks or any insights benefiting other tasks?\n\n- GRACE primarily relies on gradient information, which may not fully capture other important aspects of the distillation process, such as task-specific features. While not needing teacher logits is an advantage, in some cases, having access to them could potentially further improve the performance of GRACE. Would it be better to incorporate comprehensive information?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kB36wfGSLj", "reviewer": "Reviewer_miyx", "summary": "This paper addresses the problem of selecting the optimal teacher model for knowledge distillation, particularly in the context of large language models. The authors propose a score called GRACE that predicts a teacher's effectiveness for a given student and task. The score is computed using student gradients on a small set of teacher-generated data, without requiring access to teacher internals or test data. The authors provide a theoretical motivation for GRACE by connecting it to leave-one-out conditional mutual information and demonstrate empirically that it correlates strongly with final student performance.", "questions": "1. In Fig 6, could you please clarify what the right figure is showing and how it should be compared to the other two figures? It seems that the results of the proposed metric does not match the right figure.\n\n2. The authors nicely study the effect of their score on the correctness of the answer. How do you expect GRACE to perform on tasks where generation diversity is more important than correctness, such as dialogue or story generation?\n\n3. Is there a principled way of choosing $C$ in your method? It seems that it is costly to run the trail and error.\n\n4. The paper finds that filtering teacher generations for correctness is not beneficial for student performance (Fig 15). This is an interesting result. Do you have a hypothesis for why this happens? Could this perhaps suggest that the style or reasoning process is more important to distill than the final correct answer?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-motivated. The trial-and-error approach for teacher selection is computationally costly, and a principled, efficient method is a valuable contribution.\n\n2. The connection of the proposed score to CMI for generalization bound is clever.\n\n3. A key strength is that GRACE is lightweight and does not require access to teacher logits or internal states. This makes the method broadly applicable, even in black-box or API-based distillation scenarios.", "weaknesses": "1. The theoretical motivation (Lemma 2.1 and Appendix A) connects GRACE to the generalization gap in terms of loss. However, the empirical experiments measure correlation with final test accuracy (Average-at-k). The authors also note in the appendix (lines 752-755) that \"GRACE serves as a reliable predictor of student performance, even though it fails to correlate with loss-based quantities\". This is a disconnect that undermines the theoretical argument and should be discussed more prominently.\n\n2. The paper's writing could improve. Particularly, Lemma 2.1 does not explain what are the variables in the conditional MI and which variable is the condition. This lemma needs to become well-defined.\n\n3. There are a few claims in the paper that either I misunderstood or they are wrong. For example, lines 190-193 connect the small eigenvalues to high-signal direction. This is quite contrary to what eigenvalues express. Another example, In Fig 6, final sentence of the caption is not correct, it does not match the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yYpbmF3W1a", "reviewer": "Reviewer_vXv9", "summary": "This paper proposes a metric for evaluating teacher LLMs for synthetic data generation, for training student models. They propose a per-sample student-model gradient based metric GRACE for evaluating the teacher models. The proposed metric is well correlated with student model performance, and often currently identifies the best/ideal teacher for training the student. The metric is evaluated across a range of teacher model families, and multiple student models and datasets.", "questions": "1. Regarding Figure 21, does this show the gradient norms for gsm or for math? How does the gradient norm distribution vary across the two datasets?\n1. Did the authors try other fits for the gradient norm, for example power law $G = A + B*L^C$ ?\n1. How sensitive is the performance of the GRACE metric to the choice of this gradient norm scaling? Do other scalings significantly affect the correlation/best teacher selection?\n1. G-vendi seems to perform very well and similar to the author's method (Figure 13), except for the Qwen family. Do the authors have any conjecture as to why that may be?\n1. Line 878 says \"we use d=1024\". Are these the results in Figure 8?\n1. Does \"GradCV\" in Figure22/23 refer to GRACE?\n1. The proposed method will penalize gradient variance - eg if a teacher were to generate \"diverse\" generations in some sense. In multi-task settings, or mixture of datasets (as is common in llm finetuning), will this penalty be counterproductive in selecting the right teacher, selecting teachers with less \"coverage\"?\n1. (minor) Legend colors are missing in Figure 13.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 4, "strengths": "1. The proposed metric is well motivated, and reasonably extends G-Vendi, and is easy to compute cheaply.\n1. The proposed method achieves good predictive correlation and succesffully predicts the best teacher model across multiple datasets and student models.\n1. The proposed method correctly identifies/predicts generation temperature across different teacher model sizes, unlike some prior baselines which monotonically increase with temperature.\n\nWith synthetic LLM-generated data and seqKD being used ever more often, this paper explores a useful direction.", "weaknesses": "1. Most of the experiments in the paper (except figure 11) use temperature 1. Most llms however are typically used a much lower temperatures. The performance correlation of GRACE drops sharply for greedy sampling, which implies the method is indeed affected by temperature. This significantly weakens the impact of most of the analysis/experiments in this paper.\n1. The proposed method uses an ad-hoc gradient scaling of $1/log(L)$, motivated by a decrease in gradient norm with increasing sequence length, and the authors show the proposed scaling fitting their dataset. But Xia et al 2024 (which the authors refer to) shows in Figure 3 that this gradient shape varies somewhat across datasets, and is difficult to model. In my own experiments, modelling token gradients correlated random variables along the sequence gives good fits ($1/sqrt(L)$) at predicting grad norm, and in some other domains, losses from longer sequences are sometimes normalized with $1/L$. It is not immidiately obvious why this particular proposed gradient scaling would be ideal. This same scaling seems to have also been used for the baselines.\n1. While the paper presents a very detailed analysis and results, the paper layout is somewhat confusing. The appendix in particular seems not well laid-out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)