Efficient Data Selection at Scale via Influence Distillation
Authors: Mahdi Nikdan
Vincent Cohen-Addad
Dan Alistarh
Vahab Mirrokni
Venue: NeurIPS 2025 poster ()
Keywords: Data Selection
Data Weighting
Influence Distillation
Fine Tuning
Efficient Training
Large Language Models
Primary area: deep_learning
TL;DR: Influence Distillation is a mathematically justified data selection method for LLM fine-tuning that assigns optimal weights to training samples, achieving performance on par with or better than state-of-the-art while being substantially faster.

Abstract: Effective data selection is critical for efficient training of modern Large Language Models (LLMs). This paper introduces Influence Distillation, a novel, mathematically-justified framework for data selection that employs second-order information to optimally weight training samples. By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data for LLM fine-tuning, guiding it toward strong performance on the target domain. We derive these optimal weights for both Gradient Descent and Adam optimizers. To ensure scalability and reduce computational cost, we propose a $\textit{landmark-based approximation}$: influence is precisely computed for a small subset of "landmark" samples and then efficiently propagated to all other samples to determine their weights. We validate Influence Distillation by applying it to instruction tuning on the Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU, across several models from the Llama and Qwen families. Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5\times$ faster selection.

Efficient Data Selection at Scale via Influence Distillation
Mahdi Nikdan* ISTA & Google Research
Vincent Cohen-Addad
Google Research
Dan Alistarh ISTA & Red Hat Al
Vahab Mirrokni Google Research
Abstract
Effective data selection is critical for efficient training of modern Large Lan- guage Models (LLMs). This paper introduces Influence Distillation, novel, mathematicalllyjusttffi framework for data selection that employs second-order information to optimally weight training samples. By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data for LLM fine-tuning, guiding it toward strong per- formance on the target domain. We derive these optimal weights for both Gradient Descent and Adam optimizers. To ensure scalability and reduce computational cost, approximation: influence is precisely computed for
we propose a landmark-buse small subset of "landmark" samples and then efficiently propagated to all other samples to determine their weights. We validate Influence Distillation by applying it to instruction tuning on the Tulu V2 dataset, targeting range of tasks including GSMSk, SQuAD, and MMLU, across several models from the Llama and Qwen families. Experiments show that Influence Distillation matches or outperforms state-of-th the-ttt- art performance while achieving up to 3.5x faster selection.
1 Introduction
The rise Large Language Models (LLMs) has driven significant advances in natural language processing; yet, training and fine-tuning these models requires massive computational resources and carefully-curated datasets. One key direction towards improved training efficiency has been via data
selection and data weighting methods [Xia et al.) 2024 Yin and Rush 2024 Antonello et al. 2020
Marion et al.) 2023 Annkne et al. 20224]] Li et al.] 2023a a Ivison et al. [025]] Axiotis et al. 2024
Xie et al. 2023a Engstrom al.] 2024a Huang al. 2024 Feng al. 2024], which aim curate
training subsets that maximize a model's effectiveness, often with respect to particular target data
distribution or downstream task. However, existing approaches typically rely on heuristics such as perplexity-basee filtering-on require expensive proxy model training or expensive embedding functions generate data representations.
More precisely, existing methods face several limitations. First, many existing methods utilize fixed, model-agnostic features representations (c.g., static embeddings) that may not capture the full relationship between training samples and the target distribution (Yin and Rush 2024 Antonello et al. 2020 Marion et al. 2023] Ankner al.) 2024). Second, methods that update weights during training lack theoretical justification and can be unstable [Xie et af.) 2023a Huang et al.) 2024). Lastly, approaches that rely on reference model training or costly embeddings computationally
"Work done while intern at Google Research. "Corespondence to mnikdan @ista.ac.at and cohenaddad @ google.com.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

et intensive and often challenging to scale et al., 2023a Xia et al. 2024 Ivison et al. 2025 Yu et al.
2024| Feng et al., 2024). Thus, there remains a clear need for a taaeeemmaiialllggouuddee efficient, and scalable framework for data selection that directly optimizes for performance on specific target distribution.
Contribution. We introduce Inftuence Distilla- tion, a novel framework for data selection that addresses these challenges. Given pre-trained model and target task (represented by a small target dataset), Influence Distillation formulates the influence of training samples on the target distribution's 's loss via a second-orde -odder approxi- mation. Influence Distillation directly optimizes sample weights by analyzing how each train- ing sample, if included in gradient step, is the tar-
Meart Ace Impr: over Uniterm 10/20kk
2.5
2.0
RDS+
1.0
Unifom
Phe paskoally
Distil lattonn
Porets
0.0
-0.3
2500 Ing [TemFLLPP)
20000
expected to affect model performance get data. This formulation leads to a quadratic optimization objective for the sample weights, which we demonstrate can be solved efficiently. We provide derivations for these optimal weights under both standard Gradient Descent and the adaptive Adam optimizer, backed by theoretical justifications.
Figure 1: Average improvement over uniform sam- pling across six tasks vs. runtime. The model used is Llama2-7B [Touvron et al. 2023). and the training dataset is Tulu V2 [Ivison et al. [002)). The annotation "M/N" indicates that the method selected M samples from a pool of size N. Further details are provided in Section 5
To ensure scalability to large datasets, we further introduce an efficient landmark-bun approximation. This approach first involves selecting a small subset of "landmark" samples and precisely computing their influence. The influence for all other samples is then efficiently approximated by transferring the computed influence from these landmarks. This transfer mechanism is guided by novel and computationally inexpensive embedding space derived from Jacobian-ven vector Products. This significantly reduces the computational overhead of gradient computations for the entire dataset.
We validate Influence Distillation via comprehensive instruction tuning experiments using standard open LLMs (from the Llama (Touvron et al. 2023 Grattafiori et al.) 2024] and Qwen [Team] 2024
families) on the Tulu V2 [Ivison et al. tasks like MMLU (Hendrycks et al. 202 [ 2023
training dataset, while targeting advanced downstream
mathematics and code. Our results demonstrate that Influence Distillation not only substantially outperforms uniform random selection but also in most cases matches or exceeds the performance of oftte-ff-the-art data selection methods, while offering significant computational speedups on the same selection problem- to 3.5x in embedding + selection runtime. This positions Influence Distillation strong method on the Pareto frontier of overall embedding, selection and training cost versus downstream task accuracy (see Figure II).
2 Related Work
Data selection ('pruning') and weighting methods have become increasingly important in the context of efficient LLM training. In a celebrated paper. [Sorscher al. [2022] al. showed that (model- agnostic) data pruning, and in particular deduplication, helps go beyond scaling laws for LLMs. This
was later further improved by AAbbas et al. [2023..
Early work on model-dependent data pruning focused on heuristics like perplexity-based filtering and confidence-based selection: Marion al. (2023) found that selecting examples with moderate perplexity scores often outperforms training on the full dataset or examples selected by other metrics. Do and Gaspers 2019 introduced DSIR, which uses importance resampling based on n-gram features to select relevant training examples, with promising results on mathematical reasoning and clinical text summarization. Similarly, Xie al. [2023b] proposed clustering loss trajectories to identify representative training examples, though their approach focused more on general domain adaptation rather than specific target distributions. Another approach, so-called Classifier, was introduced by Brown et al. 2020 and has been employed in subsequent work (Gao et al. (2020), Chowwheer et al. [2023]. Du et al. 2022]. Other strategies include selecting examples that maximize the loss difference between LMs trained on candidate and reference datasets (Moore and Lewis

---

2010] Axelrod 2017], Feng et al. [2022] Another direction is employing large, well-trained language models to select informative samples [Chen et al.] 2023a Pang et al, 2024). Simpler, yet common, techniques involve filtering documents based on length or the presence of excessive special characters (Raffel et al. (2020). Xie al. (023b]). related, though distinct, task in the LM domain is optimizing the weights for sampling from mixed data sources (Chen et al. (2024). Albalak et al. (2023]. Recently, Ivison et al. (2025) proposed RDS+, which uses similarity between model-dependent embeddings computed by Dnsition-weighte mean pool of the last hidden layer
states.
Recent work has also highlighted the importance of considering the training dynamics when selecting data. Zhou al. [2023a] proposed measuring "learnability" based on loss changes during training, while Shale/Swayamdiita et al. 2020) introduced "dataset cartography" to analyze training dynamics across examples. These methods provide useful signals about which examples are most valuable for training: at the same time, they require training reference models which can be computationally expensive. For large-scale applications, |Bhatt et al. [2024] evaluated various data selection approaches for L.LM fine-tuning, and found that Pecility-locaaiinn
selection based on hidden representations was particularly effective. However, Tirumala et al. [2023) observed that generating these representations for large datasets remains computationally challenging. More recently, [ngstrom et al. [2024b] framed the data selection problem as an optimization problem: Given the learning algorithm, find the subset of the data that maximizes the performance of the trained model. To obtain an efficient solution, they design a model that given a subset of the training data S and a target example t, predicts the loss of the model trained on S on t. |xiiotis et al. [2024] recently use coreset-related ideas to propose computationally efficient way of sampling an unbiased estimator of the model loss from the training data so to train smaller input.
While previous methods like DSIR and facility location selection rely on fixed features or representa- tions, our method directly optimizes sample weights based their influence on the target distribution through second-order approximation. Importantly, this does not require training proxy model to predict the value of the elements and is computed directly from the input, model and learning algorithm. Unlike curriculum learning or confidence- based approaches that update weights during training, we derive optimal weights analytically for both SGD and Adam optimizers. In contrast to methods that require training reference models, our landmark- weight computation without extensive pre-training.
approximation allows efficient
There is large body of work on data selection methods for other learning tasks and mode, and is beyond the scope of this paper to provide detailed overview. We refer the reader to Kaushal et al. [2019). Killamsetty et al. [202T], Wei et al. (2015), Chen et al. [2023b], Cao et al. [023). Sener and Savarese 2017] and references therein.
3 Method
3.1 Problem and Notation
Let 0 E Rd be the model parameters. For any dataset D of size n and any vector of sample weights w = =[ wy, wn]²
denote L(0; D, w) n Σr we E(0; D:) as the weighted average of the model loss f on the samples of dataset D at point O. Additionally, define M(0: D, w) as a training mechanism that returns the parameters after being trained on dataset weighted by w. Unless otherwise stated, we will assume M is simply one step of (full) gradient descent.
Let S and T represent the training (source) and downstream (target) distributions, respectively. Assume we have access to a dataset S sampled from S and a small representative dataset from T. Our high-level goal will be to determine sample weights w" such that:
w = arg min S,
where E R represents the all-ones vector. In words, we wish to find sample weights w for instances within the source dataset S, such that training on S using these weights results in minimal loss on the target dataset T. Notably, this notation also allows for the special case of S T, where our method would find weights that maximize in-distribution loss improvement.
3

---

150
100
Defaul
Unconstrained
125
80
120
Count
100
Count
60
Velcaton
105
Rubusl (A 0.03)
75
90
40
50
60
20
45
30
-1000
-500
0
500 1000
3
20 Iteraion
30
Value
Value
Figure 2: (Left) Distribution unconstrained weights, (Middle) Distribution of robust weights for À = 0.02, and (Right) validation loss during training with different variants in the running experiment setting. Robust weights are found by minimizing Objective using the SLSQP algorithm [Kraft] 1988
implemented in the SciPy library [Virtanen et al. 2020
3.2 A Running Example
Throughout this section, we utilize toy training setting to illustrate variants of our method. Specifically, we consider a linear regression model parameterized by with the loss function ((0; x, v) (0' x y)² for any 0. E Rd y {0, 13. For the source dataset, random instances from the first two classes of the CIFAR-10 dataset [Krizhessky] 2009] and combine them with 256 synthetic samples generated from a Gaussian distribution with the same mean and standard deviation as the real samples. The target dataset consists of another set of 256 samples from CIFAR-10 10. We use gradient descent with learning rate of 10-3 as the optimizer. Finally, the loss values are reported on validation dataset of size 256, also sampled from CIFAR-10.
sample 256
3.3 Influence Distillation
Case 1: Unconstrained Weights. Let g0) VeC(0:T, and (T V2C(O:T, denote the gradient vector and Hcssian matrix of the loss with respect to the model parameters on the target dataset. Construct G0 E Rdliixd by stacking the gradients of the loss with respect to O across samples of S. As mentioned before, assume M is one step of gradient descent, iee. M(0; D, w) = 0 v)VeL(0; D, w) = [s] G ())w,, where 7) denotes the learning rate. We estimate Objective by:
" = arg min L(M(O; 1)
w
" (f)w: T,1)
= arg min C(e
[
of " (O( argg (0) + s " T
2 arg min [C,
S 8T (0) (0) (
"
2 ss
(
(2)
where the approximation comes from second-order Taylor expansion, i.e., L(O + 5; T, L(0; T, gf (0)5 o (O)ō where ó is replaced with - G)
Next, we define two key objects, p E RISI and Q E RIS|IIS| as follows:
S, = Gs(())TT()))
(3)
Q0; = |S| ())()),,)0G
(4)
For brevity, unless stated otherwise, will omit S and T from the arguments of and Q. Let
fww; 0) = - w
" Two" Q(0)w.
(5)
Then, the objective in Equation becomes
w = arg min f (w; 0).
(6)
w

---

In words, f represents scaled approximation of the change in loss on T when the model at point O trained on S with weights w. It is quadratic function in w, as p and Q do not depend on w. This objective can be minimized in closed form as w² (()))
While simple, the proposed solution has several crucial limitations: (a) it may produce negative highly irregular sample weights, such as excessively large values, which lack intuitive interpretation, (b) the weights may overfit to the current set of parameters 8, and (c) the weights may also overfit to the target dataset. The first two issues can be easily observed in our running experiment. The irregularity of the weights is illustrated in Figure (left). Furthermore, Figure 2 (right) demonstrates that unconstrained weights become invalidated after just one step of training, suggesting that the weights "overfit" to the current model parameters O. We note that, since the model in our running example is linear, the second-order approximation is seact-thus, the first update step reaches the optimum on T. However, this behavior does not generalize to non-linear models.
Discussion.
Case 2: Robust Weights. We modify Objective 6o address the above limitations. First, we restrict the weights to non-negative values, i.e., V < i S |S]: w; 2 O. Second, we require the weights to sum to the size of the source dataset, w = S). This prevents weights from becoming excessively large and ensures that rescaling the weights does not change the effective step size: using ar with learning rate 7 is equivalent to using w with learning rate an..
To mitigate "overfitting", a standard approach is to add regularization term. Indeed, Appendix derives such a term for linear models. In the general case, we employ a simple 1.2 regularization term.
The Robust Influence Objective. Hence, we define the robust Influence Distillation objective:
w 2 0
w * arg min fww; 0)
2 w
s.t.
T |S|
Refer to Section 4.4 for a discussion on how we tune À in practice.
We compute the robust weights with À E {0.01, 0.02, 0.03) in the context of our running example. Figure (right) highlights the effectiveness of these robust weights, showing that all three config- urations outperform the default weights while remaining stable throughout training. Additionally, Figure 2 (middle) depicts the distribution of weights for À = 0.02.
Adam Optimizer. The Adam optimizer [Kingma| 2014 is the default choice for fine-tuning LLMs. Therefore, we tailor our method for Adam optimizers. To this end. we employ greedy approach. where we assume the first- and second-order momentums (m and v, respectively) are fixed after warm-up. In this case, the Q Adam and p Adam objects are calculated as follows:
p Adam (0) = G
Adam (0)(gr(()
1/ (r)b),,,
(8)
S
1 Adam (O) Aylm (0)²
Adam 0
S
where b
9,m
and G Adam (0) is constructed by element-wise multiplying every row
(1 (1- 1-37
of (o) by a (1-81) 1a +r) Additionally, is the number of warm-up steps, and (81, B2, e)
Adam hyperparameters. See Appendix for more details.
Handling Variable Lebel Lengths. A common practice in data selection is to normalize the gradients prior to measuring similarities (Xia et al. [024]. This is motivated by the observation that the norm of a sample's 'sss gradient tends to correlate negatively with the number of label tokens, thereby biasing unnormalized gradient-based methods toward shorter samples. Normalizing the gradients mitigates this issue and shifts the similarity measure from a dot product to cosine similarity. In our approach, we adopt this normalization well. See Appendix A for study on this correlation.
Per-target Importance. The formulation above assigns weights to training samples based on their average influence over the target set. However, recent work [Xia et al. 2024 Ivison et al.) 2025) has shown that selecting training samples based on per-targel influence can yield better performance; that is, iterating over individual target samples repeatedly and selecting one top-scoring training sample each time. We adopt this approach, noting that influence scores for each target can be computed by running Influence Distillation [T'] times- once per target sample.
5

---

Efficient Influence Distillation
In this section, we tackle several challenges regarding the implementation of Influence Distillation.
Cost of Hessian. While Q(0) can be calculated exactly using Hessian- Products (HVPs), these HVPs require storing the backward graph, which can incur extra memory costs practice.
Cost of Gs. Constructing the matrix Gs requires computing the gradient of the model with respect to each individual sample in the training set. This process is computationally expensive, as it incurs a cost similar to one full epoch of training on the entire dataset S. Furthermore, storing the matrix Gs requires memory proportional to |S] times the size of the model, which is intractable in practice.
Regularization Coefficient The solution to Equation is sensitive to the choice of regularization strength X. A key challenge, therefore, is determining how select in a practical and effective way.
4.1 First-order Influence Distillation
Recall the definition of f(@; w) from Equation 5 where the second-order term scaled by the learning rate n). In Appendix H
observe that in settings, 1) is small enough that the second- order term becomes negligible. As a result, computing Q can be avoided with little to no loss in performance. This first-order approximation aligns with prior work gradient-based influence estimation, such as the methods proposed by Xia et al. 22024
4.2 Gradient Projection
To reduce the cost of storing the gradients, we take a similar approach to Xia et al. (0224) and project
each gradient vector upon calculation into a k-dimensional space, where k < d. As opposed to the mentioned work, which uses random projections sampled from the Rademacher distribution (#11 with equal probability), we find that projection using a Randomized Hadamard Transform is faster in practice. For more details on the projections, see Appendix !|
4.3 Bordmark-Basee Gradient Approximation
To cireumvent the need to compute gradients for every training sample, we introduce a landmark- based approach. At a high level, this method provides an efficient low-rank approximation of the gradient matrix Gs. given by G. CG1, where G2 E géxd contains the gradients of l < |S| selected landmark samples. The matrix C E RISSIE holds the coefficients that express each sample's gradient as a linear combination of the landmark gradients.
Specifically, let L S S denote a set of é landmark samples (c.g., selected at random), and suppose we have access to low-cost per-sample embeddings, represented by Es E R S| As before, we assume that all embedding and gradient vectors normalized. To compute the coefficient matrix C, we minimize the objective mincer** [Es CE,,1 where E,, E Réxe contains the embeddings of linear
the landmark samples. In words, this procedure approximates each sample's embedding combination of landmark embeddings. We then estimate the i-th row of the gradient matrix, gi by g: G. c, where c; is the i-th row of C. This approximation implicitly assumes that the linear relationships learned in the embedding space transfer to the gradient space.
Theoretical Justification. Although this approximation is not expected to recover the true gradients with high accuracy, the key intuition that, as long as it is unbiased, even a weak recovery can yield similar per-sample weights in dimenssonnii spaces. Theorem 4.1 demonstaatss this property for the specific case of the first-order variant of Influence Distillation.
Theorem 4.1. (Informal version of Theorem D.3 and Corollary D.4 tailored to landmark-based approximation - see Appendix Consider the special case of first-order Inftuence Distillation. Let gi and g; denote the true and the landmark based approximated gradients for sample i, and assume Influence Distillation with Gg and Gg results in sample weights of w and i.. Further assume:
Unbiased: Vi E
L[g
i.e., the approximation is unbiased.
* Bounded Low-rank MSE: Let 63 [[
g: 11,, and for some A 0: - A2 5

---

Then E[[| rn)]
with X being the regularization coefficient in Influence Distillation. x
This theorem relates the accuracy of the weights to the low-rank approximation error A?, given the training set size [S), dimension d, and regularization parameter X. If the approximations are unbiased, high dimension d, suffices to reasonably control A in order to recover the correct weights.
Integration with Influence Distillation. Given a low-rank approximation of the gradient matrix G5 R G3 LGL. one can define approximations to objects p and Q as below:
p(O,S,T,A) - CG)(0) C ((0:LTT)
(10)
LL] [S] L,) CT
CG, Q(O:S.T.L) [
(11)
where C mingeRexd E 1 is the coefficient matrix, defined above. As shown in Equations I0 and 11 the landmark-ban based Influence Distillation computes p and Q only landmark points, and then propagates them to the remaining samples.
JVP Embeddings. We observe that existing embedding methods perform poorly in this setting, exhibiting weak correlation with the true gradients (see Appendix [Fo a detailed empirical analysis). To address this issue, we introduce Jacobian-vector Product (JVP) Embeddings.
Given a sample x E S, we define its JVP embedding as:
ONE(x)
(uvr(o; = V
Σ VEV
(12)
00,
where N is the model being trained, NN(-) represents the logits of the next predicted token after processing æ through the first & layers (or transformer blocks, in case of LLMs), and 0g are the parameters of these l layers. The set V contains random Gaussian vectors matching the shape of Or, and the term @Nr(2) is the Jacobian of (x)) with respect to 0g. In words, JVP embeddings project 86, the Jacobian of an intermediate model output onto set of random directions parameter space.
4.4 Tuning the Regularization Coefficient.
Finally, describe our approach for selecting the regularization coefficient À in Equation 7 As detailed in Appendix G when is small enough for the recond-ordet term to be negligible and A = 0, the solution assigns all the weight to single sample. As increases, the solution becomes progressively less sparse, distributing weight across more samples. In the limit À - co, the solution becomes fully dense, assigning equal weight all samples. In practice, given budget of samples to select for training, we tune À via binary search to achieve a target sparsity level of |S|-k thereby ensuring that exactly It samples receive non-zero weight, which we will pick for training.
[9]
5 Experiments
In this section, we evaluate Influence Distillation across several challenging tasks. We start by detailing the datasets, models, and hyperparameters used in our experiments. Then we present our main results and ablations. Further studies are included in Appendix.
5.1 Setting
We largely follow the experimental setup Ivison al. [025 and reuse their code.
Training Dataset. We use Tulu V2 [Ivison et al. 2023 a combination of 9 instruction-tumn
datasets containing approximately 5.8 million samples. Detailed descriptions of each component
dataset are provided in Appendix(E [ Unless stated otherwise, we randomly sample 200k examples
from Tulu V2, and then use sampling methods pick subset of 10k samples from this pool.
Target Datasets. We evaluate on six target datasets: MMLU (Hendrycks et al. 2021a b). GSM8k
[Cobee et al. 2021 BBH (Suzgun et al. 2022 TyDIQA Clark et al. 20200. Codex Chen et al.
7

---

Table 1: Accuracy (+t standard deviation) and estimated embedding and selection cost (Embd+Sel, in TF, TeraFLOPs) of various methods across tasks and models. For each dadel- pair, 10k training samples are selected from pool of size 200k from the Tulu V2 dataset [vissn et al., 2023 We additionally report average improvement over the Uniform baseline (Avg. A w/ Uniform). Top performing selection methods, as well Full training numbers in bold.
Modd
Method
MMLUU
GSMER
BRH
TyDIQA
CODRX
SQuAD
Avg. A w/ Uniform
Fmmhd-Sel Cost
Uniferm
45.5 4: 0.43
17.5
41.8 0.20
51.6 0.38
27.0
0.60
80.8 1.05
1ama2-7B
Mid-PPt.
0.86
15.0
0.23
1.56
-0.65
2800
46.3
20:2
42.7 0.61
50.5=
30.4
0.96
2500
InfDiat
48.3 :: 0.21
20.3
1,65
43.2
0.67
53.6
0.34
29.5
3.14
83.2
1.02
+2.30
872 TF
Full
48.8 1: 0.08
21.2
1: 0.85
43.9 1: 0.24
51.3
0.18
29.3
= 3,72
83.6
=: 0.30
+2.30
Uniform
53.9 + 0.53
34.6 + 1.22
48.9 0.67
63.
0.36
56.1
1.35
80. 0.51
0.00
PPL
54.0
1: 0.27
29.5
1: 0.12
48.3 1: 0.44
65.9
L: 0.66
55.9
1. 4.40
80.9
1: 0.18
-0.42
1200 TF
KDS-
53.1
+ 0.58
38.4 0.58
49.6 0.45
61.0= 0.35
6886
1.77
84.2 0.47
+1.65
1300 TF
InfDist
54,0
1: 0.94
35.7
1.2 37,0
48.6 0.27
64.6
1.29 62.5
$5.4
1.10
83.3 1 1.97
+0.77
417 TF
Full
$2.9
0.14
2.81
83.9
+0,98
Uniform
58.8:::0.11
0.38
55.0
4.65
Qwen2.5-1.. 1.5
RDS-
Mid-PPE.
58.3
58.0 0.17
0.07
63.2-+
60.1 0.13
44.3 0.31
44.2
0.24
$3.0
0.38
72.3
70.6
0.00
1.61
22.3
46.0
3.12
+4.28
+0.90
500 TI'
Infiiist
$9.4 =: 0.12
62.0 0.46
44.1
# 0.55
57.6
0.32
69.8
1.15
54.4
13:13
+6.52
208 TF
Full
59.4 +: 0.113
60.3
0.38
44.0
+: 0.22
50.3
1.21
73.0
= 1,79
63.2
6.05
+7.00
Uniferm
63.7 0.27
08.7 2: 1.57
54.9 +: 0.24
65.6
0.16
83.1
1.35
84.5 =: 0.54
(1.00
U
Qwen2.5:3B
Mid-PPr.
63.7
+-118
70.8 0.84
55.1
0.31
65.3
641
X3.3
3,14
79.9
2.61
-0.40
1200 TF
RDS+
63.6
0.19
67.4 0.68
54,0
0.63
65.1
0.33
62.4
1.46
86.3
0.33
-0.28
1200 TF
Infiiit
0.39
0.36
66.9
0:22
0.06
71.0
0.32
1.41
0.42
2021], and SQuAD [Rajpurkar et al. 2016]. For each, we assume access to 8-500 examples from their train, dev, or eval splits Details are in Appendix
Model. We mainly consider fine-tuning the LlaMA-2 7B model [Tuvron et al., 2023 consistent
with the Tulu V2 paper [Ivison al. [2023] and the experiments of Ivison et al. [2025] We also
experiment with Llama-3.. 3.2 3B Grattatiori et al. 2024] and Qwen 2.5 T.5/3B TTamm 2024].
Baselines. We consider four main baselines: (1) Random Uniform selection, which picks samples uniformly at random, (2) The state-of-the-ar RDS+ [Ivison et al.] 2025) embedding-basse method, where the embeddings
computed by position-weightee mean pool of the last hidden layer states, (3) Mid-PPL. [Yin and Rush 2024). where samples are sorted by their perplexity, and the middle ones are selected, and (4) Full, where we do not perform any sampling and train the full dataset. Additionally, we also include a comparison with LESS [Xia et al. (2024). which is similar to first-order Influence Distillation but requires exact gradients for each sample.
Hyperparameters. We use the AdamW optimizer with a learning rate of 2 x 10-5 and a linear schedule for 2 epochs. The sequence length is fixed at 2048, and we gradient accumulation over 128 steps. All experiments are conducted on a single H100 GPU, and each are repeated with 3 random seeds, including the selection of 200k samples from Tulu V2.
microbatch size of 1 with
By default, we use first-order Influence Distillation with 4096 landmarks. We select the landmarks uniformly at random, as we find this performs comparably to more complex methods such as leverage score sampling (see Appendix K.4). Linear coefficients are computed via Kernel Ridge Regression (KRR) with an RBF kernel and dampening of 0.01. JVP embeddings are obtained from the first four transformer blocks using two random vectors (e = 4. |V = 2), following a brief warm-up on 10k random samples. The model is then reset and trained on the selected subset. This warm-up is needed to stabilize gradients (see Appendix AA). Gradients are projected to 131072 dimensions via Hadamard projections; we use the largest dimension that fits in GPU memory, projection cost does not depend on the dimension (Appendix]]) [].. After selection, we do not incorporate the sample weights during training. experiments in Appendix Soppendxxssugeee suggest this does not improve performance.
5.2 Results
Main Experiments. Table summarizes our main experimental results. In each case, a subset of size 10k is selected from a pool of 200k Tulu V2 [Ivison et al. [023) samples. On average, Influence Distillation achieves higher performance compared to other more expensive selection baselines in three out of four models and remains competitive in the fourth, while enabling 2.9-3.5x faster
Following Ivison et al.
2025 except omit AlpacaEval Li 2023b as it requires paid API access.
8

---

selection. Notably, for two models, it matches or surpasses training on the full dataset. These results clearly showcase the effectiveness and efficiency of Influence Distillation.
Selection Runtime Estimation. The table also reports the approximate FLOPPs required for sample selection. Following the estimation from Kaplan et al. 22020] each forward pass is assumed to cost 2d FLOPs and each backward pass 4d FLOPs, where d is the number of model parameters. Mid-PPL and RDS+ require one forward pass per sample. Influence Distillation requires computing JVP embedding for each sample, along with full forward and backward passes for the 4096 selected landmarks. We estimate the cost of a JVP as 2 that of partial forward pass over the same number of blocks, following Cobb et al. (2024).
Pareto Superiority. We repeat the above experiments the Llama2-7B model using pool sizes 50k, 100k, 150k, and 200k. For each pool size, we select 2048, 4096, 6144, and 8192 landmarks, respec- tively, maintaining fixed nool-to-landmark ratio. As shown in Figure [ all points corresponding to Influence Distillation lie on the Pareto frontier, matching or surpassing the performance of RDS+ while requiring lower overall cost of embedding, sampling, and training.
Effeet of Number of Landmarks. To evaluate how the number of landmarks impacts performance, we repeat the experiments on Llama2-7B and report average improvement over the Uniform baseline across the six tasks in Figure 3 (Left). As shown, Influence Distillation improves with more landmarks, surpassing RDS+ beyond 2048 landmarks.
Effeec of Pool Size and Number of Selected Samples. Figure 3 (Right) presents a heatmap of
MMLJ accuracy on Llama2-7B across different combinations of pool size (up to 200k) and number of selected samples. For each pool size, we use the same number of landmarks as the Pareto experiment. As expected, accuracy improves with larger pools and more selected samples, highlighting the scalability and robustness of Influence Distillation.
Comparison with LESS. We ran the 10k/200k selection experiment on Llama2-7B [Tovrron et al. 2023] using the LESS Xia et al. 2024] method (with one seed due to its high computational cost). In this setup, gradients are computed after a 10k-step warmup, projected down to 8192 dimensions [Park] al., 2023). The top 10k examples are then selected based on similarity with the target gradients. following the original LESS recipe. Table 2 presents a comparison between Uniform, Influence Distillation with JVP embeddings, LESS, and Full.
Table 2: Comparison with LESS Xia al. 2024
Model
Method
MMLLU
GSMSK
BBH
TyDIQA
CODEX
SQUAD
Arg. A w/ Unifform
Embd Sel Cost
Uniform
15.6 0.43
17.5 1 1.08
41.8 0.20
51.6 0.38
27. 1 0.60
80.8 1.05
0.00
LJuma2-7R
InfDist
48.3
20.3 # 1,6
432
0.34
29.5 = 314
83.2 1.02
+2,30
22,7
43.4
21.2 0.S
43.9
51.3 0.18
29,3 3.72
0.30
+2.30
While LESS achieves higher average accuracy, it is roughly ten times more computationally expensive than our method, requiring about 8400 TeraFLOPS (TFs) compared to our 872 TFs for embedding and selection. Its computational cost is prohibitively high in realistic settings, as it requires computing and projecting gradients for every sample in the pool, followed by fine-tuning on the selected subset. Even when disregarding the significant cost of gradient projection, the overall computation still exceeds that of training on the full dataset for one epoch. A similar observation was made by [vison al.) [0255, who did not compare with LESS in large-scal scale experiments due to its high computational cost. This limitation underscores that LESS is not practically comparable to Influence Distillation, which achieves efficiency through landmark- approximation.
Additional Studies. We present comprehensive set of additional empirical studies in Appendix K These studies investigate topics such as weight transferability across models and tasks, the impact of the target set size, and various landmark selection methods, among others.
6 Limitations and Future Work
Below, we outline three main limitations of our work, along with corresponding directions for future research.
No Target Distribution. While we demonstrate that Influence Distillation is highly effective for targeted instruction tuning across range of models and tasks, it does not directly extend to general

---

2.30
2500
45.02026
46.420-4
46,720.19
47,0m3.99
48.0
2:38
Neen Arc dooee inniremmen
2.0
46.12013
47.0e0
46.9
1.8
1.93
47.0
10000
46 46.220.6
47.520-25
47.620.12
46.2m2.05
46.5
1.2
1.35
20000
4622:0.6
47.1m544
47.9c6.14
48.1=3:1
46.0
RDG+
1.0
512
1024
2048
4096
8192
16388
150k
Landimerks
Size
Figure 3: (Left) Effect of the number of landmarks on the performance of Influence Distillation across six tasks using Llama2-7B. (Right) MMLU accuracy of Influence Distillation on Llama2-7B across different pool sizes and number of selected samples.
data selection scenarios where no target dataset is available. In such cases, one could define the target distribution as small high-quality examples or representative subset of the training corpus. Investigating how to construct and utilize such proxy targets is important direction for future work.
Pre-training. Extending Influence Distillation to the pre-training setting presents unique challenges. In particular, the considerably longer duration of pre-training implies that gradients may shift substantially over time, likely making single static selection insufficient. This suggests the need for muultippasse selection strategy, such as periodic re-sampling. We leave the exploration of such dynamic approaches to future work.
Warm-up Cost. We exclude the cost of the warm-up phase from our runtime measurements for Two reasons: (1) as the training pool grows, the cost of brief warm-up small random subset becomes negligible compared to embedding the full dataset; and (2) the warm-up can shortened (as shown in Appendix A) or compressed-for example, via Low-Rank Adaptation [Hu et al./ 2022).
as in Xia et al. 2024
We leave rigorous investigation of warm-up optimization to future work.
Acknowledgments
We would like thank ISTA SciComp for providing access their GPU resources. We also wish thank Baharan Mirzasoleiman for their insightful feedback and helpful discussions.
References
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333. 2024.
Junjie Oscar Yin and Alexander M Rush. Compute-constrained data selection. arXiv preprint arXiv:2410.16208 2024.
Richard Antonello, Nicole Beckage, Javier Turek, and Alexander Huth. Selecting informative contexts improves language model finetuning, arXiv preprint 4rXiv:2005.00/75 2020.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining 1lms at scale. arXiv preprint Ntt::2309.04644 2023.
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew I. Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541 2024.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng. Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction tuning, arXiv preprint 20Xiv:2308.1203. 2023a.
10

---

Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scate data selection for instruction tuning. arXiv preprint orXiv:2503.0100) 2025.
Kyriakos Axiotis, Vincent Cohen-Addad, Monika Henzinger, Sammy Jerome, Vahab Mirrokni, David Saulpie, David P. Woodruff, and Michael Wunder. Data-efficient learning via clustering-based sensitivity sampling: Foundation models and beyond. In Forty-firsi International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
https://oppeneviies net http:/ooummii-wuUAAaaa..
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances Neural Information Processing Systems, 36: 34201-34227, 2023a.
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024a.
Wei Huang, Yunxiao Zhang. Shangmin Guo, Yuming Shang, and Xiangling Fu. Dynimpt: A dynamic data selection method for improving model training efficiency. IEEE Transactions on Knowledge and Data Engineering, 2024.
Lan Feng, Fan Nie, Yuejiang Liu, and Alexandre Alahi. Tarot: Targeted data selection via optimal transport. arXiv preprint arXiv:2412.0044 2024.
Zichun Yu, Spandan Das, and Chenyan Xiong, Mates: Model-aware data selection for efficient
pretraining with data influence models. Advances in Neural Information Processing Systems, 37: 108735-108759 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Hamish Ivison, Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing im adaptation with tulu 2. arXiv preprint arXiv:2311.10700 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv.2407.21783, 2024.
Qwen Team. Qwen2.5; A party of foundation models, September 2024. URL htttt::///wennnn github.io/blog/ggee.... /we....5////e2/////
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523-1996 2022.
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data- efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540. 2023.
Quynh Do and Judith Gaspers. Cross-lingual transfer learning with data selection for large-scale spoken language understanding. EMNLP, 2019.
Sang Michael Xie et al. Smalltolarge (s21): Scalable data selection for fine-tuning large language models by summarizing training loss trajectories of small models. arXiv preprint, 2023b.

---

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Voss, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint urXiv:2101.0002 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240)1--13,, 2023.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-eppers. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
Robert C Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACL 2010 conference short papers, pages 220-224, 2010.
Amittai Axelrod. Cynical selection of language model training data, arXiv preprint arXiv:1709.02277 2017.
Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc. Automatic document selection for efficient encoder pretraining. arXiv preprint arXiv:2210. 10951, 2022.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint arXiv:2307.087000 10000000000.1.111 2023a.
Jinlong Pang. Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang, Chen Qian, Yang Liu, Yujia Bao, and Wei Wei. Improving data efficiency via curating lIm-driven rating systems. arXiv preprint arXiv:2410. /0877, 2024.
Colin Raffll, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (1440)):161 2020.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang. Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024.
Alon Albalak, Lianggnngg Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023.
Haotian Zhou et al. Lobass: Gauging Icarnability in supervised fine-tuning data. arXiv preprint arXiv:2310. 13008, 2023a.
Swabha Swayamdipta. Roy Schwartz, Nicholas Lourie, Yizhong Wang. Hannanch Hajishirzi, Noah Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. EMNLP. 2020.
Gantavya Bhatt al. An experimental design framework for label-efficient supervised finetuning of large language models. arXiv preprint arXiv:2401.06692 2024.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: improving LLM pretraining via document de-duplication and diversification. Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Ad- vances Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::htttppp//oe nips wwwpapprrpp..e http:ppppprfeeeeeeeee wttp///fllepeee////eaeaa papp:/pp2//ereeettt.eee http:/e/eeeebbbbo1-babbreeeeeee Datanets
Alice Oh, Tristan
and_ lee___nnd_____eeee Benchmarks html
12

---

Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. BtI.https:/ htttts:://ppereeeeteeee /owpr//eeeeeeeeeeeeeeeee wttprrevvievene//fururu.. Id=GCSHkKeHBB
Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: unified data subset selection and active learning framework for computer vision. In 20/9 FEEE Winter Conference on Applications of Computer Vision (WACV), pages 1289-1299. IEEE, 2019.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh lyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in neural information processing systems, 34:14488- 14501, 2021.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pages 1954-1963. PMLR, 2015.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint arXiv:2307.08701, 2023b.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.06290, 2023.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint AXXiv:708.004400 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Dieter Kraft. software package for sequential quadratic programming. Forschungsbericht Deutsche Forschungs- und Versuchsansialt fur Luft- und Raumfahrt, 1988.
P Virtanen, R Gommers, TE Oliphant, M Haberland, T Reddy, D Cournapeau, E Burovski, P Peterson, W Weckesser, Bright, et al. Fundamental algorithms for scientific computing in python and scipy 1.0 contributors. scipy 1.0. Nat. Methods, 17:261-272, 2020.
Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 11X.v:2110.14168. 2021.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thouge can solve them. arXiv preprint arXiv:2210.0920 2022.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information -eeeaatiiooneeiinnggg question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint wrXiv:2107.03372 2021.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference Empiricul Methods in Natural Language Processing, pages 2383 2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 18 D16- 1264. URL https: http:tttts///alanne httpa/////lllaahooo/o// hwt.h/oooooyy.og/11...... 1264
13

---

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of cestructoonfoolowing models, 2023b.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wwX.:v2001.083600 2020.
Adam D Cobb, Atilmm Güneş Baydin, Barak A Pearlmutter, and Susmit Jha. Second-ordes forward- mode automatic differentiation for optimization. arXiv preprint arXiv:2408.10411 2024.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303. 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020..
Alexandra Peste, Adrian Vladu, Eldar Kurtic, Christoph H Lampert, and Dan Alistarh. Cram: A compression-aware minimizer. arXiv preprint arXiv: 14200, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling models. Journal of Machine Learning Research, 25(70):1-53 53, 2024.
language
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant couversafoons-eemorrttiinng large language model alignment. Advances in Neural Information Processing Systems, 36:47669-47681, 2023.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned Ilm, 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Sahil Chaudhary. Code alpaca: An instruction-follovinn Ilama model for code generation, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55023 2023b.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1222 2023.
Wing Lian, Bleys Goodson. Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces, 2023.
David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, et al. Sciriff: A resource to enhance language model Eestucciionfollowiinn over scientific literature. arXiv preprint S.Xi:22006.07335, 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Chankyu Lee, Rajarshi Roy. Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428 2024.
14

---

Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Abrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming Wei Chang, al. Large dual encoders are generalizable retrievers. arXiv preprint 1rXiv:2112.07855 2021.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
Krish Agarwal, Rishi Astra, Adnan Hoque, Mudhakar Srivatsa, Raghu Ganti, Less Wright, and Sijia Chen. Hadacore: Tensor core accelerated hadamard transform kernel. arXiv preprint
wrXiv:2412.08832, 2024.
Tri Dao. Fast hadamard transform in cuda, with pytorch interface, 2023. URL https://githubb com Dao- AlLab/fast hadamard- trrnnforranorr
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Every claim made in the abstract and introduction is addressed, including the main weighting method and the landmark- approach in Sections and as well as the experimental results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section for discussion on limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: Theorems 4.1 in the main text, and the results in Appendix D state the assumptions clearly. All derivations are correct, the best of our knowledge.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Every detail and hyperparameter is shared in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code will be uploaded zip file in supplementary material, along with instructions and commands for reproducing the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc)
aablic/guudees/Coeeebbbmsss wtt.eeeeuuuiiisioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuaaaeeeae Ctteeee//ooSSSeeeSe..ee.a www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The full experimental setting is described in detail in Section with the exception of dataset descriptions and splits, which are deferred to Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Every experiment was repeated on three seeds. In Table and Figure 3 the standard deviations are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: As described in Section 5.1 all experiments were conducted on single H100 GPU. Additionally, the compute cost for most experiments is calculated and reported in terms of FL.OPs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that our paper complies with all guidelines. Anonymity is maintained throughout both the paper and the code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not pose such risks. The paper focuses on fine-tuning existing open models using publicly available open datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used are properly cited, and the licenses for models and datasets listed in Appendix
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not publish any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This research does not involve any crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This research does not involve any crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: This paper involves LLM fine-tuning, though the core idea is general and may extend other model types avenue did not explore
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) httt::://eerrpp.ccc//oofeeeeeeer / for what should or should not be described.
22

---

A Gradient Analysis During Training
In this section, we analyze the behavior of gradients throughout training. We fine-tune LLaMA2-7B
model [Touvron et al. 2023] on 10000 randomly selected samples from Tulu V2 [Ivison et al. 2023
for 2 epochs, saving model checkpoints every 10 steps.
For each
the initial and final models we compute the gradients of 1000
held-out samples from Tulu V2, as well samples from the target dataset BBH [Suzgun et al. 2022]
and project them into an 8192-dimensional space using random Rademacher matrices, following
the efficient GPU implementation of Park et al. [203], also adopted in Xia al. [024]] For each
dataset, we compute the average gradient cosine similarity across checkpoints. As shown in Figure 4 while the gradient directions can change substantially in the early steps, they stabilize quickly during training. This observation justifies the use of a short warm-up phase as both necessary and sufficient. Similar plots for GSM8k [Cobbe et al. 202T] and SQuAD [Rajpurkar et al. 2016 are provided later in the Appendix (Figure 9
Additionally, for each dataset and checkpoint, measure the Pearson product-moment correlation between gradient norms and the number of label tokens per sample. As shown in Figure [ we observe consistent negative correlation, which supports decision to normalize gradients prior distillation.
B Linear Model Study
In this section, we show that a regularization term can be effective in robustifying Objective 6 to small changes in the model parameters 0, when the model is linear and the loss is quadratic.
For fixed 0, define objective below:
w" = arg min max |88]5e
f(w; 0 + +),
(13)
w
minimizing the maximum value of ff around a point a neighborhood of radius e. This ensures the weights are stable as long as 0 is in this neighborhood. To solve for w, we employ Lemma B.l below:
Lemma B.1. Assume L(0: D. w) = E! w( 0) - vP, where D =
{xx vP (x))(())) (x. y (xTb) y } is a dataset. For datasets and T. ler H = V2C(O,T,1),
gI = VoL(O:T,1), H. VOL(0: S, w), and gu Vol(0; S, w). Define aw and B as
below:
aw HogT HTgw + HHH
(14)
B = Hr 7 H Hr
(15)
In the setting above (linear model with quadratic loss), the function has the property that v 0, 5
Rd, w e R":
f(w; 0 + 8) = f(w; 0) a + 8 B
(16)
Proof. First notice that, for simplicity, the loss here is defined as the sum (as opposed to the average) of per-sample losses, which drops the [S] terms in the loss, gradient, Hessian, and Q objects.
Recalling the definition of f from 5 we can write [ww; 0 Ô) pp0 + 1))++))=0m w 2 " Q(0 1 8) w.. Since the loss is quadratic in 0, the Hessian is independent of 0, and the derivatives above the second order are zero. Hence, defining g (0) and H." as the gradient and Hessian of the sample in D, we can write:
g. p 8) g (0) HP8
(17)
for any 8 with the same dimension as 0. Setting D T and summing across samples, write:
gr ( + 8) = gr( + H
(18)
Additionally, setting D and taking weighted sum we can write:
Gs + 8)w Gs(0) + H
(19)
23

---

1.0
1.00
0.40
0.39
0.49
0.39
0.38
0.38
0.38
0.37
0.36
0.36
0.35
0.45
0.35
0.45
0.45
0:45
10 0:40
1.00
0 87
0.82
0.81
0.78
0.78
0.76
0.76
0.74
0.74
0.73
0.73
0.73
0.73
0.73
0.73
20 0:39
0.87
1.00
0.94
0.91
0.89
88
0.86
0
86
0.84
0.84
0.83
0.83
0.83
0.9
30 0.39
0 94
1.00
0.92
0.91
0.90
0.89
40
50
0..8
60 0.38
0.78
0 88
0.93
0.95
0.97
1.00
0.98
0.97
0.95
0.95
0.95
0.95
0.95
0.95
0.95
0:95
Step
70 0.39
0.76
0 86
0.92
0.94
0.96
0.98/
1.00
0.97
0.97
0.96
0.96
0.96
0.96
0.96
0.96
0:96
0.7
80
0.37
0.76
0. 86
0.92
0.94
0.96
0.97
0.97
1.00
0.98
0.98
0.97
0.97
0.97
0.97
0.97
0:97
90
0.92
0.97
1.00
0.98
0:98
100
0:32
0.96
1.00
0.6
110
120
0:35
0.73
0.91
0:93
0.96
0.99
1.00
0.99
130
0.35
0.73
0.83
0.89
0.91
0.93
0.95
0.96
0.97
0.98
0.98
0.99
0.99
0.00
0.99
0.99
0:99
0.5
140
0.35
0.73
0 83
0.90
0.92
0.94
0.95
0.96
0.97
0.98
0.98
0 99
0.99
0. 99
1 1.00
0.99
0.99
150
0.35
0.73
0. 83
0.89
0.91
0.94
0.95
0.96
0.97
0.98
0.98
0.99
0.99
99
0.99
1.00
0.99
0.4
152
0.98
1.00
10
2
M
40
50
60
70
80
90
100
110
120
130
140
150
152
Step
0 1.00
0.61
0.67
0.67
0.69
0.69
0.70
0.70
0171
0.70
0.70
0.69
0.69
0.69
0.69
0.69
0.69
1.0
10 0.61
1.00
0.06
0.84
0.82
0.81
0.79
0.78
0.78
0.78
0.78
0.78
0.78
0.78
0.78
0.78
0.78
20 0.67
0.85
1.00
0.97
0.95
0.94
0.93
0.92
0.92
0.91
0.91
0.91
0.90
0.90
0.90
0.90
0:90
30
0.67
0.84
0.97
1.00
0.98
0.97
0.95
0.94
0.93
0.92
0.93
0.93
0.92
0
0.92
0.92
0:92
40
0.95
0.9
50
60
Step
70 0.70
0.78
0. 92
0.94
0.95
0.97
0.98
1.00
0.99
0.98
0.98
0.98
0.98
0.97
0.97
0.97
0.98
80 0.71
0,78
0. 92
0.93
0.94
0.96
0.97
0.99
1.00
0.99
0.99
0.99
0.98
0.98
0.98
0.98
0:98
0.8
90 0.70
0.78
0.91
0.92
0.93
0.95
0.97
0.98
0.99
1.00
0.99
0.99
0.99
0.99
0.99
0.99
0:99
100 0.70
0.78
0. 91
0.93
0.
0.96
0.97
0.98
0.99
0.99
1.00
0.99
0.99
99
0.99
0.98
0:99
110
120
130
0.96
1.00
0.7
140 0.69
0.78
0.90
0.92
0.93
0.95
0.96
0.97
98
0.99
0.99
1. 1.00
1.00
1.00
1 1.00
1.00
1.00
150 0.69
0.78
0.90
0.92
0.93
0.95
0.97
0.97
0.98
0.99
0.99
0.99
1.00
0.99
1.00
1.00
0.99
152
0.69
0.78
0.9
0.92
0.93
0.95
0.96
0.98
0.98
0.99
0.99
0.99
1.00
1.00
1.00
0.99
1.00
10
20
30
40
50
60
70
80
9,
100
110
120
130
140
150
152
Step
Figure 4: Average gradient cosine similarity on unseen samples from Tulu V2 (top) and BBH (bottom) across checkpoints.
Next, we see that,
p(0 + ())
(gr(0))* 5" 6 Hr) H = =p(0)Tw (gr(0) H (0) 8" Hr
(20)
And,
w 0(Q ") 8
= Gw + 8d H (G H. =w"Q(O) + 2g. (0) H + 8 HH H.
(21)

---

-0.3
00.4
-0.5
Correlation
-0.6
Tulu
-0.7
GSMBK
BBH
-0.8
SQUAD
-0.9
10
100
110
120
130
140
150
152
Step
Figure 5: Correlation between gradient norm and number label tokens, across checkpoints on four datasets.
Putting them together:
flw: 0 + = -p + 6) w +
11 w Q + 66w
2
=f(w; 0) ((g (0)
H gw (0) 5 H"H Hw
!
(2g (0) H o 6 8 H. H
2
f(w; 0) + (-gr(() H w gw
(0) (0) H 7 (0) H
- H H H.)
f(w; 0) + a O
which concludes the proof.
Substituting the result of the B.1 into Objective 13
write
w" arg min
max [8lsse
f(w: 0) + a + o
arg min
[(w; 0) + max
( 8"h O Bw [8]se
(22)
We maximize r(6) = 2 + 8 O in the sphere with radius approximately by taking a single
gradient direction, i.e., 8" sharpness-aware optimization literature [Foret et al.
step of size
r'(0)
This approximation is standard in the
2020 Peste et al. 2022]. which addresses a
similar min-max objective search for flat minima. Note that r (0) ((( = aw and
a.
B + B 6,, hence
a B'B B
max To + 5 o,. Rs e ||a ++?
(23)
|aw
Substituting into Equation 22
we get the following objective:
w
~ arg min w
[(w; 0) + lewllawll ||+
a Bwa | 1
(24)
This suggests that the robustness of the weights can be controlled via the hyperparameter c, which determines the strength of the regularization.
We apply this regularization to the running example introduced Section 3.2 shown in Figure 6 using the tuned value e 10-1 yields better performance than the default weights. However, due to the high computational cost of this regularization term, we use standard 1.2 regularization for general non-linear models.
25

---

C Adam Optimizer
Here we derive Equations & and which adapt the vector p and the matrix Q to the case of the Adam optimizer. Assume that after warm-up phase, the first- and second-moment estimates of Adam are m and v, respectively. For new gradient g, the Adam update rule can be written as:
m
AO
11
(25)
Vv
where is the learning rate, and m' estimates, with (31, (2) being the Adam beta values for first- and second-order estimate updates, and being the number steps the optimizer has already been trained for.
81 m (1-81)11 11 and v' = (1-+(1-82) are the updated moment 1-
1-35
For single update, we note that 82v + (1 f2)gg² R v. That because (1) the value 32 typically very close to 1, e.g., 0.995 or 0.999, and (2) due to the warm-up, is stabilized and is not expected to change much. This allows us ignore the dependence of v' on g, i.e., v' 1-33 simplifying the computations.
Enabled by this, we revisit the Taylor expansion in Equation
arg min C(M (0: S, 1.
9im + (1 GGs (G 10w
17
arg min C(O
[S]
v
+
1-82
B m
(1 ((O)
arg min L(O w
(26)
[S]
(1 89)
I-33 + t)
(1 - 39) 1- +
Let a = (1-9
(1-9g) - te)
and b = (1-89)
8im
Construct G. (0) by element-wise
- +e)
multiplying each column of Gs( by a. We can now continue Equation 26 by:
w" arg min L(O 17 (b
G
Adam (0) (0)w)
2 arg min [C(O;T,1)
T Bi
(0)
+ G Adam (0)w)+
n (b"
w
G
Adam S
0 T H ()) (b Gādam ()w)
2 S
arg min
- (0)
b
H
Adam
(0)w
S
" w G
Adam (0) (0)TH H (0) G Adam (0)w
2 S
arg min Adam (0)TT T
2 w T Q Adam 0) TTw
(27)
Where p Adam and Q Adam are defined in Equations
and respectively.
D Proof of Theorem 4.1
We begin by noting a property of the landmark-based approximation introduced Section 4.3 it exhibits rotational equivariance. That is, if all source and target gradients are rotated by an orthonormal matrix, the resulting landmark- gradient approximations will also be simply rotated by the same matrix.
In the remainder of this section, we prove two useful lemmas Lemmas-Lemma D.I and Lemma D.2 We then state and prove Theorem D.3, which bounds the error in the vector for any unbiased approximation that satisfies rotational equivariance. Finally, Corollary [D44 bounds the difference in the resulting sample weights, thereby completing the proof of Theorem
26

---

8
Default
Unconstrained
300
Robust Linear (E= 1e 3)
120
Robust Linear (E= 4)
105
Count
200
Validation
90
75
100
60
45
0
30
5 Value
10
0
10
20
30
40
Iteration
Figure 6: (Left) Distribution theoretical robust weights for the linear case with c = 10 10-4, and (Right) validation loss during training with different variants in the running experiment setting.
Lemma D.I. Given unit vectors g, tER assume g- noisy approximation to g. Additionally, assume Rdd a zero-mean, isotropic random vector, i.e., [[e 0 and Cov(e) = La for some
0. Let S t). Then E[S] [g, t), and Var(S) El 8
Proof. The expectation of S follows directly from zero-mean property e. bound its variance, let E denote the covariance matrix of e. Since is isotropic, E I for some a. We can write:
Var(S) Vår t)
Var((g. t) (c, t))
Var((e, t\)
r
of11t112
o²
(28)
Also,
Elllg =|| ||||e||1
tr())
do²
(29)
Hence ²² Elllg-g111 which concludes the proof.
Lemma D.2. Assume E Rd is random vector from arbitrary distribution. For any random orthonormal matrix of the form R PD, where
P a random permutation matrix
D is diagonal matrix with i.i.d. Rademacher signs (+1)
the random vector RRx is isotropic, i.e., Cov(y) = La some real value
Proof. We can write:
Cov(Rx) [ R²]
=E PDxx DP
= [x PDxx DP Xx]]
(EE D |P" |x]
(30)
27

---

Now note that ED Dxx D x = diag(x²). Substituting into the expectation over P, we need to compute Ep (x pp However, since random permutation, off-diagonal elements are 7ero and for the diagonal elements, any element of x² can be picked with equal probability. Hence, the expectation over P equals 2||x||
Putting it all together in Equation 30 wc get
||| d
Cov(Rx):
Id.
(31)
which concludes the proof.
Theorem D.3. Let G E RRadd and t E RR. with and each row of G having unit lengths. Let gi denote the i i'th row in G. Additionally, assume access to (randomized) mapping function h: (g.,g,..... g2...., gn} + Rd, and let Vi (1,2..... n) gi = h(g G). Additionally, assume h(.)
satisfies:
Unbiased: Vi E (1,2.....
n : E gi ie., h(.) is unbiased.
2. Bounded Average Mean Squared Error: Ler 83 Elllg: g:11 Then:
n
6 Σ A
for some 2 2
3. Rotation Equivariance: For any orthonormal rotation matrix R Raxd and Vi (1,2,..., n h(Rg; GR) = Rg.
Construct the vector p= 1 P2............................ ...... Pr) IT such that p. P:=(g;; t). Similarly define p P2, 12,........... ........ pnlT.
where P: = (8: t). Then
nA2 d
Flll - 2] <
(32)
Proof. For all i, let e; = g: gi denote the error. By the Unibased assumption, [[[e:] = 0.
Without loss of generality, we can assume that for all i, the vector is isotropic, i.e., Cov(e; ) is scalar multiple of the identity matrix. If this is not the case, we take advantage of Lemma D.2 and apply change of variables: G + GR and t < Rt, where R PD, P is a permutation matrix, and is diagonal matrix with entries chosen uniformly at random from {+1]. Note that by the Rotation Equivariance assumption, this transformation implies g: +- Rg. Under this transformation, the error vectors e; are mapped into space where they become isotropie, and the pairwise dot products and distances remain unchanged as R is orthonormal.
Now can directly apply Lemma D.1 for each coordinate i: Elp.] =p; and Var(p.) Ellls - This means:
Ellpp
pl
E((p: P.)
Var(p.)
E[ll@ - -
d
A
d
where the last inequality comes from the Bounded Average Mean Squared Error assumption.
Corollary D.4. In the setting of Theorem D.3 if define:
[ x x20
w(p) arg min -p x +
| s.t.
1x'1
(33)
28

---

then
n X²d
E[][(()) wp())
(34)
Proof. Let F2(x)
-ppx
2 1xx[| and C fx R x 0, x,x21 n). Note that the objective
above has unique solution since Fp is A-strongly convex and C' is convex set independent of
By strong convexity, Vr, E
A 2/1y x|1
Fş(y) 2 Fx(x) V,FF(x) (y x)
(35)
Set x w == w() and y = i == w()). Since w minimizes Fp over C, VxF2(x) w) 2 0.
Hence:
x 20
Fp(w)) 2 Fp(w)) +
w
(36)
2
Swapping and it,
FØ(w) 2 Fp(io) +
1 - أ0
(37)
2
Adding the two equations above:
(p- - p) (w wه 2 A w - 1013
(38)
Applying Cauchy-Sehwand on the left hand side, we get
||p pll-1|w w |/ 211W0 uه
(39)
Hence
w w S p pll
(40)
Combining with the result of Theorem D.3
w 2 nd²
(41)
X2d
E
Dataset and Model Details
This section provides details on the datasets and models used throughout the paper.
E.1
Datasets
For the datasets, we largely follow the setup of Ivison al. [2025]
Tulu V2 (ODC-BY License). The Tulu V2 dataset [vison et al.
2023 also known as the Tulu
V2 SFT Mixture, is comprehensive instruction-tuniin dataset. Following Ivison al.) [2025).
consider the unfiltered version with 5.8M samples, consisting of 961,322 samples from FLAN v2
[Chung al. |024 , 398,439 samples from FLAN CoT [Chung al. 2024 , 7,707 samples from
Open Assistant KKöpf et al. 2023 15,007 from Dolly [Conover et al. 2023 ), 52,002 from GPT-4
Alpaca (Peng et al. 2023). J. 20,022 from Code Alpaca [Chaudhary 2023 100,054 from ShareGPT,
1,030 from L.MA Zhou et al. 2023b) 142,802 from Wizard Evol-unstruct V2 (Xu et al. 2023)
4.111,858 from Open Orca [Lian et al., 2023 7,535 from SCiRIFF (Wadden et al. 2024 and 14 from Hardcoded. For more information, we refer the reader to Ivison et al. (002)]
MMLU (MIT License). The Massive Multitask Language Understanding (MMLU) dataset Hendrycks et al. 2021a b) consists of challenging multiple-choice questions from 57 topics, such as abstract algebra, astronomy, machine learning, and more. It includes 5 development samples per category and a total of 14,042 test samples. We use the development samples as our target set and evaluate the final model zero-shot the test set.
GSM8K (MIT License). This dataset comprises grade school math questions, with 7.47k training and 1.32k test samples [Cobbe et al. 2021 We evaluate the models on the test set using 8 examples
29

---

in the context (8-shot evaluation) and use the same individual samples as the target set. As is standard, only the final answer to each question considered.
Hig-Bench--arre (MIT License). This dataset includes questions from 27 challenging tasks, such as causal judgment, multi-step arithmetic, and logic. Following Suzgun et al. [002]. we perform 3-shot
evaluations using the same samples per category (a total of 81) as the target set.
TyDIQA (Apache-2.0 License). TyDIQA is dataset of 204k
samples across 11
languages [Clark et al. 2020]. For evaluation, we follow Ivison et al. [025]] which turn follows Anil et al. 2023 using 1-shot prompting. We select 9 samples per language for the target set.
Codex (MIT License). This dataset contains 164 Python programming questions [Chen et al. 2021] of which 16 are used as the target set and the remaining as the test set. See[vison additional evaluation details.
[vison et al. 2025 for
SQuAD (CC BY-SA 4.0 License). The Stanford Question Answering Dataset (SQuAD) RRajpurkrr al. 2016] contains reading comprehension questions based on Wikipedia articles. We use 500 random samples from the training split as the target set. We perform 3-shot evaluations with three samples randomly selected from the training set.
E.2 Model Licenses
In this paper, we utilize LLaMA 2 Touvron et al. 2023 LLaMA 3.2 3B [Grattafiori et al. 2024),
Qwen 2.5 1.5B [Tem 2024], and Qwen 2.5 3B [Tem 2024 models. These models are distributed under the LLaMA 2 Community License, LLaMA 3.2 Community License, Apache-2.0 License, and Qwen Research License, respectively.
F Embeddings Study
In Section we noted that existing embedding functions are insufficient for our landmark-based gradient approximations and introduced the JVP embeddings as an alternative. In this section, we compare different embedding functions in two settings. all the experiments, the model we consider is Llama-2 7B Touvron et al. 2023].
Gradient Recovery. First, we randomly take 200k samples from Tulu V2 Ivison al. 2023 and embed them using various embedding functions. We then use a small number of landmark gradient samples (selected uniformly random) approximate the gradients for all data points, following the method described in Section 4.3 This process is repeated for different numbers landmarks evaluate how performance varies with landmark count. We report the average cosine similarity between the approximated gradients and the true gradients (projected into 8192-dimensional space
using Rademacher-bane projections [Ivison al. 2025 Park et al., 2023) for each case.
We evaluate several embedding functions: the RDS+ embeddings from Ivison et al. 22025, NVIDIA's NV-Embed-v2 [Lee et al. 2024], GTR-base [Ni al. 22021) and our proposed JVP- approach
using two random vectors and four transformer blocks.
As lower bound, we also include a Trivial embedding: here, we assume that the gradients for the landmark samples are perfectly recovered, while the gradients for all other samples are treated as completely random.
Figure 7 (Left) presents comparison of these embedding functions. Our JVP embeddings outperform all other methods, including the more computationally intensive RDS+ and NV-Fmbed-v2.
Finally, compute upper bound by using the true projected gradients as the embedding function and repeating the same experiment. As shown Figure 7 (Right), this idcalized setting quickly achieves high accuracy in gradient approximation- surrttiiinnn-uusssiinnnnna 0.9 cosine similarity with just over 4096 landmarks. This suggests that the gradients are approximately low-rank, a known phenomenon in LLMs (uu al. 2022 Zhao et al., 2024).
End-to-end Selection and Training. We repeat the selection and fine-tuning experiments from Table
I this time replacing the JVP embeddings with either GTR- -bae or true gradient embeddings. Table 3 reports the resulting accuracy for each task. Due to the high computational cost of obtaining true gradients, we include only a single random seed for this setting.
30

---

Cosine Similrity
Average Cosine Similerity
1.0
0.25
0.8
0.20
JUP
GTR-base
0.8
0.7
0.6
0.05
0.00
255
532
1924
2048
4095
6192
18384
32786
236
512
1024
2240
4096
8192
18:38
32758
Number of Landmarks flog scale)
Number Landmarks flog scall)
Figure 7: (Left) Gradient direction recovery vs number of landmarks, when different proxy embdeding functions are used, and (Right) gradient direction recovery when the actual gradients used as an ideal embedding.
Table 3: Accuracy (# standard deviation) of Llama2-7B across six tasks when using Influence Distillation with different embeddings to select 10k samples from a pool of 200k in the Tulu V2 dataset [Ivison et al. 2023 The number of landmarks is fixed at 4096.
Model
Embedding
MMLU
GSMSk
BBH
TyDIQA
CODEX
SQUAD
Avg. A w/ Uniform
45.3
Llama2-7B
JVP
48.3 0.21
20.3 à 1.65
0.67 43.2
53.5 L:
0.34 $1.7
2 27.7
83.2 4: 1.02
46.4
Grad
48.3
20.2
43.7
84.5
46.0
We fix the number of landmarks 4096 across all experiments. The results show that while GTR-base consistently underperforms, the JVP and true gradient embeddings yield comparable securacy-falling within each other's standard deviation in most cases. This indicates that the gradient approximations provided by JVP embeddings are sufficiently accurate for end-to-ens training.
Finally, we note that since Figure 7 (Right) demonstrates near-perfect gradient recovery using the Grad embedding, the corresponding row in Table closely mirrors the performance the LESS method Xia al. 2024
G An Active-Set Solution
In this appendix we derive the solution to the Influence Distillation objective under the assumption that 1 + Al is positive definite (PD). This setting includes the special first-order case used in the main body of the paper, where + O. Concretely, we solve
arg min -D" w "
1 Qw + 2 TTT w, s.l.
w 2 w
(42)
where denotes the dimension of and n/Q AI > 0.
Introduce the Lagrange multipliers E for the equality constraint and a E R for the non- negativity constraints. The Lagrangian is
L(w, r, a) -p" "w Tu zw Tw w -(1 w n) w.
(43)
Differentiating L with respect to w and setting it equal to zero yields
n w Aw p
(44)
Let R = 1/Q + XI 0. Then
Ray p
a 0.
(45)
By complementary slackness, Vi: w.a, = 0. Let A {i Wi = 0} be the active set and B its complement. Restricting (45) the free indices gives
B = Pæ + +1g.
(46)
Because Rgg is principal sub-matris of the PD matrix R, it is itself PD. Hence
w w B)
(p rlg).
(47)
31

---

Enforcing 1'w = n determines T:
T B BB (D + +rl) = n,
(48)
and therefore
7' n 1 R BBPB
(49)
1/ RBBBB
Substituting T" back into w(r; B) gives us the weights on B:
R
wi = R Rgg Ps +
1BB
(50)
1 R R
For indices the active set A we have " = 0. giving the final candidate solution w" (wo a w (z).
Optimality requires that the remaining Karush Kuhn-Tucker (KKT) conditions hold, namely vi e B, wi 2 0 (primal feasibilityy and Vj E A, a, 2 (dual feasibility). Because the objective is
convex (R > 0), any partition A, B satisfying these conditions is the global optimum.
Examining the coordinates in A in 455 gives
0 (RABwB) - PA -
(51)
Problems of this type are typically solved with primal-dual active-set algorithm. We start from the feasible point w = (so A = Ø, B (I,...,n)
n)) and repeat:
1. Solve for w via 50
2. If any component of wi is negative, move its index to A.
3. Compute a' if any component is negative, move its index back to B.
Each move strictly decreases the objective, and with only finitely many index sets the algorithm terminates once all components of wB and a are non non-negative.
The Special Case of 1 + O. This setting corresponds to the first-order Influence Distillation variant used throughout the main body of the paper. this case, we demonstrate that À increases, the solution w" becomes denser-thes
is, it contains more non-zero elements. This observation is leveraged in Section 4.4 for tuning the parameter
When 17 0, we can write R AI. which implies R RBB I and RAB 0. Substituting these into Equations
50 and 51 we obtain:
T" nd - 1PB
(52)
B
1 (p + +
(53)
244 (
(54)
Since both w and a must be non-negative, the last two equations imply that the active set B must satisfy B = {i p: 2 --* i... B is necessarily a set of top-k elements from for some k.
Consider two values A1 < d2, and let B1 and Bg denote their optimal supports with sizes h and k2, and (1) and (2) their respective optimal weight vectors; similarly, let (1) and (2) denote their associated dual variables. Suppose for contradiction that k2 < kn. Note that B, consists of the indices of the top h elements in p. while B3 B1 includes the top h2 elements of Let sk, and Sky represent the sums of the top k and K2 elements in p. respectively. Define j as the index of the k -th largest element in p. Since j B1, have w.ll 0, and since j B2. it follows that
32

---

(2) 2 0. Therefore,
(1)
w 2 0
nd1
8k1
7 Pj +
h1
7 nn 2 8k1
-kip = Σ
(p: -Py)
iEB:
and
(2) 2 0 0
nd Sk22 <0
Dj +
K2
7 n Ska
k2pj
Σ iE
(P)
Observe that E.e Pp) < Σ B, (P: Pj) by definition of Py. leading to the inequality nd I nd1, which contradicts our initial assumption that \ X2-
This contradiction confirms that as the regularization parameter À increases, the solution becomes progressively denser. Specifically, À O, the solution concentrates all weight on the largest element of to minimize the objective, whereas in the limit as À - oo, the regularization dominates, resulting in w 1.
H
First- vs Second-Order Influence Distillation
Recall the robust Objective
0 |
w" arg min f(w; 0) 2 11 12 s.t.
(55)
ww""
where,
f(w: 0) = - w +
7 w k(0))
(56)
2
In this section, we compare the first-order term Ti = (0) w with the second-order term T, " ww"" Q()w.. To do so, we sample 128 random examples from the Tulu V2 dataset [Ivison et al.
2023) the source dataset, and 4 examples from either GSM8K [Cobbe al. 2021 or MMLU Hendrycks et al. 2021a as the target dataset.
We compute the vectors and the matrices Q exactly for the Qwen-2.5 1.5B model [Team 2024], using Hessian-vector products to obtain Q. We then evaluate both Ti and T2 using default weights 1 and range of learning rates. To measure the relative contribution of the second-order term, we report the ratio
Ta
As shown in Figure 8) the second-order term is generally negligible for practical learning rates (n) < 10-4), indicating that the first-order approximation is sufficient in this setting.
I Projection Details
While in some of our lower-cost experiments we employ Rademacher-based projections-including projecting JVP embeddings a 4096-dimensional space using this method, as supported on GPUs by Park et al. [2023) we find that projecting the landmark gradients with Rademacher projections becomes computational bottleneck. To address this. we instead use combination of pre-masking and Randomized Hadamard Transform-baud based projections, as described below.
reddamard-based Projection. Given high-dimensional gradient vector we first pad it with zeros the nearest power of two, 2 Then, we apply a random sign (+1) each element. The
signed vector is reshaped into matrix X of dimensions = 2 and = 2 2l. We then apply
33

---

2nd- to 1tt-order Ratio
1.25
GSM8K
1.00
MMLU
0.75
0.50
0.00
10-6
10"5
10-4
10
Learning Rate (n)
Figure 8: Ratio of second- to first-order terms for Qwen-2.5 1.5B across learning rates on two target datasets.
Table 4: Accuracy (# standard deviation) of Llama2-7B across six tasks when using Influncc Distillation select 10k samples from a pool of 200k in the Tulu V2 dataset [Ivison al., 2023). with and without loss weighting during training. The number of landmarks is fixed at 8192.
Model
Embedding
MMLU
GSM8k
BBH
TyDIQA
CODEX
SQuAD
Arg. w/ Uniform
Llama2-7B
Weighted
47.8
0.26
270
Weighted
0.79
1.67
29.3
0.86
Hadamard transforms from both sides: H. XH,, The resulting matrix flattened, and a random subset of its entries is selected the projected vector.
Importantly, both the random sign patterns and the final index subset generated once and reused across all projected vectors. This ensures consistency and enables meaningful comparison. The left and right Hadamard transforms are highly efficient and provide strong mixing across rows and columns.
Pre-masking, Although efficient GPU implementations of the Hadamard transform exist [Agarwal et al. 2024 Dao, 2023). they support transforms to dimension 2 32,765, This allows us to efficiently project vectors of up 230 1,073, 741,824 elements-jus over one billion. However, the full gradients of large language models (LLMs) can exceed this size.
To address this, apply pre-masking: randomly select one billion elements from the gradient vector before projection. For LLaMA-2 7B [Touvron et al. 2023). we select these clements from the down_proj matrices, which find to represent the overall gradients well. For smaller models, randomly sample one billion elements from the entire gradient vector.
Weighted Training Loss
In this section, we investigate the effect of incorporating the weights derived by Influence Distillation into the training loss. Specifically, we conduct an experiment using LLaMA-2 7B (Touvron et al. 2023), with pool size of 200k and 8192 landmarks sampled from Tulu V2 [Ivison et al. [023 During training, we scale the loss of each selected sample by its corresponding weight.
Table compares this weighted training setup with baseline where the weights of the selected samples are ignored. The results show that incorporating weights during training does not improve performance and in some cases, it may even degrade it. This may be due to some samples having near-zero weights, effectively pruning them from the training process.
K
Additional Studies
K.1
Analysis of Training Progression on MMLU
For more granular analysis, we repeated the Llama2-7B experiment on MMLU to compare the accuracies of Influence Distillation, Uniform sampling, and RDS+ throughout training. In this
34

---

experiment (using a single seed), model checkpoints were saved every 25 steps to track performance over time. The MMLU accuracies these intervals are reported in Table
Table 5: MMLU Accuracy Progression for Llama2-7B by Method.
Method
Step 25
Step 50
Step 75
Step 100
Step 125
Step 150
Uniform 44.7
45.4
45.8
46.2
46.2
46.1
RDS+
45.1
45.3
46.0
45.9
46.2
46.2
InfDist
45.2
46.1
47.9
48.3
48.4
48.5
We also conducted similar experiment using full training on the entire dataset. Our findings indicate that full training requires around 1 100 steps to reach an accuracy of 48.5. In contrast, Influence Distillation achieves this same accuracy at step 150, representing a more than 7x speedup training efficiency.
K.2
Investigation of Weight Transferability.
To investigate the transferability of weights across tasks, we compute sample weights for different target sets as described in the paper and subsequently analyze their pairwise Pearson correlations. This analysis utilizes the Llama2-7B [Touvron et al. 2023 model with 1024 landmarks and data pool of 200k examples.
The the computed weights for four of the six tasks, presented in Table 6 exhibit high correlation
(0.5 - 0.8). This suggests that sample weights computed for one task may transfer effectively to others.
Table 6: Pairwise Pearson correlations of sample weights across tasks.
MMLU
GSM8K
BBH
TyDiQA
Codex
SQuAD
MMLU
+1.0000
-0.0430
-0.5430
+0.6953
+0.7422
+0.7734
GSM8k
+0.9961
+1.0000
+0.2520
+0.2188
+0.2637
-0.0938
BBH
-0.5430
+0.2520
+1.0000
-0.2021
-0.2295
-0.6719
TyDiQA
+0.6953
+0.2188
-0.2021
+1.0000
+0.5938
+0.5312
Codex
+0.7422
+0.2637
-0.2295
+0.5938
+1.0000
+0.5547
SQUAD
+0.7734
-0.0938
-0.6719
+0.5312
+0.5547
+1.0000
To further assess cross-task generalization, we use the sample weights computed for MMLU to select a 10k subset from the training pool. We then train model on this subset and evaluate it on each of the remaining tasks. The results, comparing Influence Distillation and uniform sampling (averaged over three seeds), are presented in Table
Table 7: Cross-task generalization performance. Samples are selected using MMLU weights.
Method
MMLU
GSMSk
BBH
TyDiQA
Codex
SQuAD
Uniform 45.6
17.5
41.8
51.6
27.0
80.8
InfDist 48.2
16.8
42.6
50.6
28.2
82.1
These results show that, even though the samples were selected based on MMLU, Influence Distilla- tion still outperforms uniform selection in most tasks when evaluated for three seeds.
K.3 Ablation Study on Target Set Size
To study the effect of the target set size, we vary the number of samples per category in the MMLU target distribution. MMLU contains 57 categories. While the main paper uses samples per category (285 samples in total) as the target set, this ablation repeats the Llama2-7B [Touvron et al. 2023 experiment using 1, 2, 3, and samples per category. This results in target set sizes of 57, 114, 171, and 228, respectively.
35

---

The MMLU accuracies for each target set size are presented in Table
Table 8: Effect of Target Set Size on MMLU Accuracy.
Target Size MMLU Accuracy
57
48.1
114
48.6
171
48.2
228
48.1
285
48.5
These results suggest that Influence Distillation is relatively robust to the size of the target dataset in this setting, demonstrating stable performance across the tested range of target set sizes.
K.4 Impact of Landmark Selection
We repeat the experiments from Appendix Figure (Left) to compare different landmark selection methods. Specifically, we fix the number of landmarks to 4096 and compute the average cosine similarity of the landmark-based approximated gradients with true gradients when using the following landmark selection methods:
1. Uniformly at random
2. Leverage score-based sampling (in the JVP embeddings space)
3. Longest samples
4. Shortest samples
The results for the Llama2-7B (Touvron et al. 2023 model are presented in Table 9
Table 9: Gradient recovery by landmark selection method.
Uniform
Leverage
Shortest
Longest
Gradient recovery
0.105
0.105
0.035
0.046
These results show that deliberately choosing the longest or shortest samples significantly degrades gradient approximation quality. On the other hand, choosing landmarks uniformly random is par with leverage score-based sampling, as mentioned in the Hyperparameters paragraph in Section
5.1
K.S
Transferability of JVP Embeddings Between Models
To investigate JVP embedding transferability, we conducted an experiment to study how the average cosine similarity between approximated and true gradients of Llama2-7B [Touvron et al.) 2023 is affected by the choice of embedding model.
Specifically, we compare the performance of the following embeddings, fixing the number of landmarks at 4096:
JVP from Llama2-7B (same the target model)
JVP from Llama3.2-33 [Grattafiori et al. 2024
JVP from Qwen2.5-1.5B [Team] 2024
JVP from Qwen2.5-3B
TTam 2024
The results are summarized in Table 10 and compared against the recovery achieved by RDS+ Ivison
et al. 20255.
Comparing these results with the recovery achieved by RDS+, we conclude that JVP embeddings are highly transferable across models. Notably, JVP embeddings computed from Qwen2.5-3B approximate the gradients Llama2-7B with similar quality those obtained directly from Ilama2-
7B itself.
36

---

Table 10: Gradient recovery using JVP embeddings from different models.
JVP (Llama2-78)
JVP (1.lam33.2338
JVP (Qwen2.5-1.55)
JVP (Qwen2.533)
RDS+ (Llama2-78)
Gradient recovery
0.105
0.098
0.093
0.103
0.073
1.0
1.00
0.60
0 69
0.68
0.68
0.67
0.65
0.65
0.64
0.64
0.63
0.62
0.62
0.62
0.61
0.62
0.62
10 0.60
1.00
0.85
0.80
0.81
0.74
0.71
0.70
0.69
0.70
0.70
0.69
0.69
0.68
0.68
0.88
20 0:69
0.85
1. 1.00
0.97
0.
0.93
0.89
0.88
0.87
0.87
0.87
0.85
0.85
0.84
0.85
0.85
30
40
0.99
50 0.67 0.74
0:97
0:96
60 0.65
0.71
0.89
0.94
0.94
0.98
1.00
0.99
0.97
0.98
0.97
0.95
0.96
0.
0.95
0.96
0:96
Step
70 0.65
0.70
0.08
0.94
0.93
0.98
0.99
1.00
0:99
0.99
0.98
0.99
0.98
0.98
0:97
0.97
0:97
80 0.64
0.69
0.87
0.92
0.92
9.96
0.97
0.99
1.00
0.99
0.98
0.99
0.98
9.98
0.98
0.97
0:98
0.8
90 0.64
0.70
0.93
0.
0.97
9899
0.99
1.00
0.99
0.98
0.99
0.98
0:98
100
0.98
1.00
0:98
110
120
1.00
1.00
0.7
130 0.62
0.68
0.85
0.91
0.91
0.95
0.96
0.98
98
0.99
1.00
D. 8.88
1.00
1.00
0.58
1.00
1.00
140 0.61
0.68
0.84
0.90
0.90
0.94
0.95
0.97
0.98
0.98
0.99
1.00
0.99
0.98
1.00
0.98
0:98
150
0.62
0.68
0. 85
0.90
0.90
0.95
96
0.97
0.97
0.98
0.99
O. 98
0.99
1:00
O 9898
1.00
1.00
1 122
0.62
0.68
0.90
0.90
0.95
96
0.97
0.98
0.98
0.99
0. 0.98
1.00
1.00
1.00
0.6
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150
152
Step
1.0
0.19
10
20 0.17
0.74
00
0.85
0.65
0.59
0.56
0.56
0.57
30 0.14
0.66
0 89
1.00
0.94
0.87
79
0.78
0.74
0.70
0.71
0.69
0.70
0.68
0.68
0.69
0.69
0.8
40
0.15
0.67
0. 85
0.94
1.00
0.87
0.79
0.76
0.72
0.71
0.71
0.68
0.68
0.67
0.67
0.67
0.68
50 0.14
0.59
0.74
0.87
0.87
1.00
9494
0.93
0.90
0.87
0.87
0.87
0.87
0.06
0.85
0.86
0.87
60
Step
70
0.6
80 0.13
0.51
0.61
0.74
0.72
0.97
0.95
0.95
0.96
90 0.12
0.51
58
0.70
0.71
0.87
0.
0.93
D 96
1.00
0.98
0.97
0.96
0.95
0.96
0.96
100 0.12
0.51
5.59
0.71
0.71
0:87
90
0.92
0. 95
0.98
1.00
0.99
0.96
9.97
0.97
0.97
0.97
110
0.11
0.48
0.56
0.69
0.68
0.87
90
0.93
0.95
0.97
0.99
1.00
0.99
0.99
0.98
0.99
0:99
0.4
120
0.48
0.94
0.98
130
0.93
0.97
0.99
140
0.67
150
0.48
0.56
0.67
0.94
0.97
0.99
1.00
1.00
0.2
152
0.12
0.48
0.57
0.69
0.68
0.87
0.90
0.94
0.96
0.96
0.97
0.99
0.99
0.99
0.99
1.00
1.00
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150
152
Step
Figure 9; Average gradient cosine similarity on unseen samples from GSM8k (top) and SQuAD (bottom) across checkpoints.
L Differed Figures
Figure is included in this section, having been moved from Appendix A due to its size to improve readability.
37


Reviews:
[{"review_id": "kN0QojEi0h", "reviewer": "Reviewer_nWgK", "summary": "The paper introduces the Influence Distillation framework, which selects a subset of data from a large training corpus to improve both the performance of LLMs trained on the selected data and the overall training efficiency. The core idea of Influence Distillation is to assign weights to training samples based on their estimated influence on downstream tasks. This influence is computed using a second-order Taylor approximation of the loss on the target dataset, following a gradient step on the training data. Experimental results show that the Influence Distillation framework outperforms several existing data selection methods, while also reducing computational cost and achieving data selection up to 3.5 times faster.", "questions": "Please move the proofs of the formulas from the main text to the appendix.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "> **Strengths**\n\n1. The proposed Influence Distillation framework outperforms several strong baselines, including Mid-PPL and RDS+, in terms of model performance as well as the time and computational cost of data selection.\n2. The paper introduces landmark-based gradient approximation and JVP embeddings to improve the efficiency of gradient computation, thereby reducing the computational cost of the Influence Distillation framework.\n\n> **Weaknesses**\n\n1. The paper does not compare its method with the LESS approach [1] for data selection, which is similar to the Influence Distillation framework. Both methods select training examples based on their influence on the validation set, with influence calculated using model gradients.\n2. I wonder whether it is necessary to select as many as 10k examples for instruction tuning. When training on the full dataset, LLM performance may converge after only a few thousand examples. It might be more informative to report the performance of LLMs at various training steps on the selected datasets obtained through different data selection strategies.\n\n**References**\n\n[1] LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024)", "was_revised": true, "final_justification": "4: Borderline accept", "rebuttal": "We thank the reviewer for their insightful feedback. Here are our answers to their questions.\n\n> 1. Comparison with LESS [1]\n\nWe thank the reviewer for this valuable comment. As noted in the last paragraph of Appendix F (also mentioned in Section 5.1, Baselines), we have included a brief discussion of the LESS method. Specifically, as an ablation, we replaced JVP embeddings with actual projected gradients. Figure 7 (right) in the appendix shows that this yields a gradient recovery of 0.9 (in terms of average cosine similarity) with 4096 landmarks, making the approach, in this case, very similar to LESS which uses exact (projected) gradients. The last row of Table 2 shows that this variant performs on par with Influence Distillation using JVP embeddings.\n\nTo provide a more complete comparison, we also ran the 10k/200k selection experiment on Llama2-7B using the LESS method (with one seed due to its high computational cost). In this setup, gradients are computed after a 10k-step warmup, projected down to 8192 dimensions, and compared to the target gradients. The top 10k examples are then selected based on similarity—following the original LESS procedure. Below we present a comparison between uniform sampling, InfDist with JVP embeddings, and LESS.\n\n|Method|MMLU|GSM8k|BBH|TyDIQA|CODEX|SQuAD|Avg|\n|------|----|-----|---|------|-----|-----|----|\n|Uniform|45.6±0.43|17.5±1.08|41.8±0.20|51.6±0.38|27.0±0.60|80.8±1.05|44.1|\n|InfDist+JVP Embedding (ours)|48.3±0.21|20.3±1.65|43.2±0.67|53.6±0.34|29.5±3.14|83.2±1.02|46.4|\n|LESS|48.2|22.7|43.4|55.8|29.1|86.0|47.5|\n\nWhile LESS achieves higher accuracy on average, it is significantly more computationally expensive. Our method requires approximately 872 TeraFLOPs (TFs) for embedding and selection, whereas LESS requires around 8400 TFs—approximately 10x more.\n\nWe thank the reviewer again for this insightful suggestion and will include these results in the main body of the next revision.\n\n[1] https://arxiv.org/pdf/2402.04333\n\n> 2. Accuracy throughout training and the size of the target set\n\nWe would like to note that our 10k example selection setting follows the setting introduced in [2], where 10k examples were selected from the same dataset.\n\nTo be complete, we repeated the Llama2-7B experiment on MMLU using Influence Distillation, Uniform sampling, and RDS+. In this run (with a single seed), we saved model checkpoints every 25 steps to better track performance over time. Below, we report MMLU accuracies throughout training:\n\n|Method |Step 25|Step 50|Step 75|Step 100|Step 125|Step 150|\n|-------|--------|--------|--------|--------|--------|--------|\n|Uniform|44.7|45.4|45.8|46.2|46.2|46.1|\n|RDS+|45.1|45.3|46.0|45.9|46.2|46.2|\n|InfDist|45.2|46.1|47.9|48.3|48.4|48.5|\n\nWe also repeated the experiment using full training on the entire dataset. Our results show that full training reaches an accuracy of 48.5 only around step 1100. In contrast, Influence Distillation achieves the same accuracy at step 150 – over 7x faster.\n\nWe thank the reviewer for this insightful comment. We will include these results in the next revision.\n\n[2] https://arxiv.org/pdf/2503.01807\n\n> 3. Placement of proofs in the main body\n\nThank you for the suggestion regarding the placement of proofs. Upon review, we note that the main text does not currently include full proofs; the closest instance is the four-line Taylor expansion leading to Equation 2. We agree that this would be better suited to the appendix and will make this adjustment in the next revision."}, {"review_id": "h4Zp9GVsXc", "reviewer": "Reviewer_yYDh", "summary": "This paper introduces a learned data‐sampling strategy for instruction‐tuning large language models. At its core, the method maintains a “target task” small dataset T (e.g., a benchmark’s train/dev split) and a large generic pool S. It repeatedly estimates, via an influence‐distillation objective, how much each sample in S would improve performance on T if added to the fine‐tuning batch. Sampling weights are updated to favor high‐influence examples, and the model is fine‐tuned on these examples in turn. To keep this process scalable, the authors propose a landmark‐based approximation that reduces the cost of influence estimation. They evaluate on six downstream benchmarks, showing consistent gains over uniform or heuristic sampling with only modest extra computation.", "questions": "1. Have you tried transferring a sampler trained on one benchmark’s T to fine‐tuning on a different downstream task without retraining the sampling weights? If so, how does performance compare? Demonstrating such transfer would greatly strengthen claims of broad utility.\n    \n2. In settings with no small held‐out dev for T, can your method bootstrap itself—e.g., by iterative self‐training or clustering‐based pseudo-targets? If not, how critical is the quality/size of T for gains?\n    \n3. Do you foresee any modular extension of your framework to unsupervised pretraining data selection or reinforcement-learning fine-tuning? Even a pilot experiment or discussion would clarify the method’s broader applicability.", "limitations": "The authors note reliance on a representative target dataset and mention scalability trade-offs. However, they do not explore cases where no such dataset exists or where target and source distributions diverge drastically. I recommend adding a discussion or small experiment on cold-start scenarios and potential mitigation strategies (e.g., self-supervised proxy tasks).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Quality (Strengths)**\n\n- Rigorous formulation of the sampling objective via influence functions, with clear derivations.\n    \n- Includes end‐to‐end cost accounting (fine‐tuning + embedding + sampling), giving a realistic sense of overhead.\n    \n- Empirical gains are consistent across six diverse instruction‐tuning benchmarks, demonstrating robustness.\n    \n\n**Quality (Weaknesses)**\n\n- Limitation to instruction‐tuning stage only; does not address data selection during pretraining or reinforcement‐learning fine‐tuning, where large‐scale unsupervised data could benefit most.\n    \n- All evaluations tie the sampling policy to *each* target benchmark’s dev set; no experiments test transfer of a learned sampler to a truly unseen task.\n    \n\n**Clarity (Strengths)**\n\n- Well‐structured presentation: algorithm pseudocode, clear notation for S, T, influence scores.\n    \n- Landmark approximation is motivated and explained intuitively before formalizing.\n    \n\n**Significance (Strengths)**\n\n- Addresses an important bottleneck: sampling the “right” examples during instruction tuning to maximize downstream performance per compute.\n    \n- Opens a new direction for learning to sample data based on target‐task feedback, which could complement or replace heuristic curricula.\n    \n\n**Significance (Weaknesses)**\n\n- Impact is limited if users lack access to any representative target dev set (cold‐start scenario).\n    \n- Would be more significant if shown to carry over across tasks or to larger unsupervised pools in pretraining.\n    \n\n**Originality (Strengths)**\n\n- Novel combination of influence estimation with a learnable sampling distribution in the instruction‐tuning context.\n    \n- Landmark‐based approximation appears to be a fresh scalability trick.", "was_revised": true, "final_justification": "The authors have replied to my questions. I'll keep the original score 4 due to the limited scope of the work.", "rebuttal": "We would like to thank the reviewer for their valuable comments. Please find the answers to your concerns below.\n\n> 1. Extension to pre-training and the no-target case\n\nOur current focus is indeed on the supervised fine-tuning phase, rather than pre-training or reinforcement learning. This choice reflects our goal of improving training efficiency in realistic fine-tuning settings, where large amounts of instruction-tuning data are available and training cost is a bottleneck.\n\nThat said, we believe that several components of our method are applicable to pre-training scenarios as well:\n\nGradient approximation via JVP embeddings remains effective in pre-training after a short warmup period, as the gradients have been shown to exhibit low-rank structure after this initial phase [3]. This suggests that the embedding-based similarity approach could be extended to the pre-training stage with minimal overhead.\n\nIn cases where there is no explicit target distribution, there are two possible adaptations worth exploring:\n\n(i) One could set the source and target sets to be the same (i.e., $S=T$). In this case, both sets share a common gradient matrix $G$, which can be written as $G=CL$ (similar to the paper). This allows for efficient computation of pairwise gradient similarities as $G^TG=L^TC^TCL$. This structure could enable selection of influential samples using, for example, leverage score-based sampling.\n\n(ii) Alternatively, one could define a small, high-quality subset of examples as the target distribution and select pre-training data based on their influence relative to that set.\n\nWhile these directions are promising, they would require additional experiments to fully validate. We view this as an exciting avenue for future work and thank the reviewer for this suggestion.\n\n[3] https://arxiv.org/abs/2403.03507\n\n\n> 2. Transferability of weights\n\nWe agree that this is an interesting experiment. To investigate this, we compute sample weights for different target sets as described in the paper, and analyze their pairwise Pearson correlations. For this analysis, we use the Llama2-7B model with 1024 landmarks and a data pool of 200k examples. As shown in the table below, the computed weights for four of the six tasks exhibit high correlation (0.5–0.8), suggesting that sample weights computed for one task may transfer well to others.\n\n|                | MMLU | GSM8k | BBH | TyDiQA | Codex  | SQuAD  |\n|----------------|------------|-------------|-----------|--------|--------|--------|\n| MMLU     | 0.9961     | -0.0430     | -0.5430   | 0.6953 | 0.7422 | 0.7734 |\n| GSM8k    | -0.0430    | 0.9961      | 0.2520    | 0.2188 | 0.2637 | -0.0938|\n| BBH      | -0.5430    | 0.2520      | 1.0000    | -0.2021| -0.2295| -0.6719|\n| TyDiQA         | 0.6953     | 0.2188      | -0.2021   | 1.0000 | 0.5938 | 0.5312 |\n| Codex          | 0.7422     | 0.2637      | -0.2295   | 0.5938 | 1.0000 | 0.5547 |\n| SQuAD          | 0.7734     | -0.0938     | -0.6719   | 0.5312 | 0.5547 | 1.0000 |\n\n\nTo further assess cross-task generalization, we use the sample weights computed for MMLU to select a 10k subset from the training pool. We then train a model on this subset and evaluate it on each of the remaining tasks. The results are presented in the table below, comparing Influence Distillation and uniform sampling (the results are averaged over three seeds):\n\n| Method  | MMLU | GSM8k | BBH  | TyDIQA | CODEX | SQuAD |\n|---------|------|-------|------|--------|-------|-------|\n| Uniform | 45.6 | 17.5  | 41.8 | 51.6   | 27.0  | 80.8  |\n| InfDist | 48.2 | 16.8  | 42.6 | 50.6   | 28.2  | 82.1  |\n\nThese results show that, even though the samples were selected for MMLU, Influence Distillation still outperforms uniform selection in most tasks when evaluated for three seeds.\n\n> 3. Study on size of T\n\nTo study the effect of the target set size, we vary the number of samples per category in the MMLU target distribution. MMLU contains 57 categories, and in the main paper we use 5 samples per category (285 samples in total) as the target set. For this ablation, we repeat the Llama2-7B experiment using 1, 2, 3, and 4 samples per category, resulting in target set sizes of 57, 114, 171, and 228, respectively.\n\n| Target Size |57|114|171|228|285|\n|-|------|------|------|------|------|\n|MMLU accuracy|48.1|48.6|48.2|48.1|48.5|\n\nThese results suggest that Influence Distillation is relatively robust to the size of the target dataset in this setting.\nWe thank the reviewer for this valuable comment and will include these findings—along with similar analyses for other target datasets—in the next revision of the paper.\n\n> 4. Divergence of Target and Source\n\nIn this case, it intuitively would be hard to perform well on the target distribution for which we don’t have any data. We believe that this would be a limitation of all data selection methods."}, {"review_id": "LXQ3qBnc1X", "reviewer": "Reviewer_29so", "summary": "This paper propose a influence-based distillation method to optimal weight or select data samples. It estimates each sample's influence on a downstream target task using a second-order approximation of the loss, and accelerates computation through a landmark-based approximation combined with Jacobian-vector product (JVP) embeddings. The method supports both SGD and Adam optimizers, and introduces regularization for robust weight estimation.", "questions": "- What is the impact of the chosen landmark samples? Do differences in sample quality or token length affect the final performance of Influence Distillation?\n\n- Could the authors explain why performance drops significantly when the number of landmark samples reaches 8192? Intuitively, increasing the number of landmarks should enhance performance rather than degrade it.\n\n- Could using a different embedding model—with a potentially different embedding space—affect performance or violate the assumption made in Line 235?\n\n- The reviewer wonders why sample weights are not used during training (Lines 309–310) and whether this is due to the approximations in Section 4. Further explanation would be appreciated.\n\n- The reviewer recommends including the performance of the raw base model for reference.\n\n- Typo: \" fine-tuneing\" in Line 287", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n-  This paper is straightforward and  easy to follow.\n- Derives optimal sample weights for both SGD and Adam.\n- Provide Landmark + JVP-based approximations reduce computational cost heavily, which is novel and interesting.\n\n**Weaknesses**\n\n- The method shows limited gains over baselines, especially Full training, and lacks comparisons to stronger recent methods like prompt-based scoring [1-2] or LESS, weakening its empirical support.\n\n- The paper introduces multiple approximations (e.g., gradient projection, landmark-based gradient estimation, first-order influence), but does not analyze their joint effect on the robust Influence Distillation objective. For instance, in the experiments, the authors claim that the sample weights no longer contribute to performance, which raises concerns about the overall impact of these approximations.\n Besides, it remains unclear whether the combined approximations still preserve the intended influence estimation accuracy. The reviewer recommends including a small ablation study to analyze the correlation between the approximated influence scores and the true (non-approximated) influence scores. \n\n\n\n[1] AlpaGasus: Training A Better Alpaca with Fewer Data, ICLR 2024.\n\n[2] Improving Data Efficiency via Curating LLM-Driven Rating Systems, ICLR 2025.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. Therefore, I lean positive towards this work and raise my rating score to 4.", "rebuttal": "We thank the reviewer for their constructive feedback. Below we address the raised concerns:\n\n> 1. Discussion on the baselines and related work\n\nWe would like to emphasize that, compared to training on the full dataset, Influence Distillation offers significant computational savings. For instance, in the Llama2-7B experiments, full training on 200k examples requires approximately 8400 TeraFLOPs (TF), whereas Influence Distillation with 4096 landmarks requires only 1292 TF for embedding, selection, and training on the 10k subset. This results in a 6.5x speedup while recovering the accuracy of full training across the average of six tasks (see Table 1).\n\nWhen compared to other baselines included in our paper, we argue that Influence Distillation is Pareto-superior—that is, it is both faster and at least as accurate on average across all tasks. This trade-off is illustrated in Figure 1.\n\nRegarding a comparison with the LESS method, please see the next point below\n\nWe appreciate the reviewer’s suggestion and thank them for highlighting these related works. The two additional methods (AlpaGasus and DS2) involve evaluating each sample in the pool using one or more large-scale language models, such as ChatGPT, which results in substantially higher computational cost due to the size of the models involved. Moreover, these methods are generally model-agnostic and do not offer the same level of theoretical grounding as our approach.\n\nFinally, we thank the reviewer for this valuable comment. We will incorporate the new results and include a discussion of AlpaGasus and DS2 in the next revision of the paper.\n\n> 2. Comparison with LESS [1]\n\nWe thank the reviewer for this valuable comment. As noted in the last paragraph of Appendix F (also mentioned in Section 5.1, Baselines), we have included a brief discussion of the LESS method. Specifically, as an ablation, we replaced JVP embeddings with actual projected gradients. Figure 7 (right) in the appendix shows that this yields a gradient recovery of 0.9 (in terms of average cosine similarity) with 4096 landmarks, making the approach, in this case, very similar to LESS which uses exact (projected) gradients. The last row of Table 2 shows that this variant performs on par with Influence Distillation using JVP embeddings.\n\nTo provide a more complete comparison, we also ran the 10k/200k selection experiment on Llama2-7B using the LESS method (with one seed due to its high computational cost). In this setup, gradients are computed after a 10k-step warmup, projected down to 8192 dimensions, and compared to the target gradients. The top 10k examples are then selected based on similarity—following the original LESS procedure. Below we present a comparison between uniform sampling, InfDist with JVP embeddings, and LESS.\n\n|Method|MMLU|GSM8k|BBH|TyDIQA|CODEX|SQuAD|Avg|\n|------|----|-----|---|------|-----|-----|----|\n|Uniform|45.6±0.43|17.5±1.08|41.8±0.20|51.6±0.38|27.0±0.60|80.8±1.05|44.1|\n|InfDist+JVP Embedding (ours)|48.3±0.21|20.3±1.65|43.2±0.67|53.6±0.34|29.5±3.14|83.2±1.02|46.4|\n|LESS|48.2|22.7|43.4|55.8|29.1|86.0|47.5|\n\nWhile LESS achieves higher accuracy on average, it is significantly more computationally expensive. Our method requires approximately 872 TeraFLOPs (TFs) for embedding and selection, whereas LESS requires around 8400 TFs—approximately 10x more.\n\nWe thank the reviewer again for this insightful suggestion and will include these results in the main body of the next revision.\n\n[1] https://arxiv.org/pdf/2402.04333\n\n> 3. Discussion on approximations\n\nWe would like to thank the reviewer for this insightful comment. As suggested, we first include an ablation measuring the similarity between approximated and non-approximated influence weights.\n\nWe note that computing the similarity between all pairs of source and target gradients without projection is computationally intractable for our data scale. Specifically, it requires $O(|S| \\times |T|)$ gradient computations, as we can only store a constant number of gradients in memory. Additionally, given the high dimensionality, the source data pool must be large to yield meaningful similarities. Therefore, even for the non-approximated weights, we apply random projections to reduce the gradient dimensionality to 8192, using Rademacher projections (as in LESS).\n\nFor this experiment, we consider the Llama2-7B model, the 200k Tulu V2 training dataset, 32 random MMLU target samples, and 4096 landmarks. We compute both the approximate Influence Distillation weights and the \"true\" second-order influence weights (with both gradients and Hessian-vector products projected to 8192 dimensions), with learning rate 2e-5, which we used for our fine-tuning experiments. We find that the two sets of weights have a Pearson correlation of around 0.7, which aligns with the theory presented in Theorem 1: while the low-rank gradient approximation may yield a weak recovery of individual gradients, their similarities with target gradients are preserved due to the high dimensional structure.\n\nAs for the observation that using weights during training does not improve performance, we believe this is largely due to the regularization tuning procedure. As mentioned in the paper, we choose the regularization strength $\\lambda$ such that the final solution has exactly 10k non-zero weights. In practice, many of these weights are small. In the experiment described above, approximately 20% of selected samples had weights less than 5, while another 20% had weights larger than 30, with some samples with weights as high as 160. When training with these weights, the low-weight samples contribute very little and are effectively ignored, reducing the advantage of weighting. A potential alternative would be to allow a larger number of non-zero weights and then select the top-k most influential samples, but we leave this to future work.\n\nWe thank the reviewer again for raising this important point and will include these new findings and discussions in the next revision of the paper.\n\n> 4. Impact of landmark selection\n\nWe repeat the experiments from the appendix Figure 7 (Left) with different landmark selection methods. Specifically, we compute the average cosine similarity of the landmark-based approximated gradients with true gradients when using JVP embeddings. We fix the number of landmarks to 4096, and compare the following landmark selection methods: (1) uniformly at random, (2) leverage score-based sampling in the JVP embeddings space, (3) longest samples, and (4) shortest samples. The results are presented in the following table (the model is Llama2-7B):\n\n| |Uniform|Leverage|Shortest|Longest|\n|--|--|--|--|--|\n|Gradient recovery|0.105|0.105|0.035|0.046|\n\nThese results show that deliberately choosing longest or shortest samples significantly degrades gradient approximation quality. On the other hand, choosing landmarks uniformly at random is on par with leverage score-based sampling, as mentioned in the Hyperparameters paragraph in Section 5.1.\n\nWe thank the reviewer for this feedback, and will include this study in the next revision.\n\n> 5. Performance degradation when #landmarks=8192\n\nWe acknowledge that the reviewer’s observation is correct: as the number of landmarks increases, the quality of gradient recovery should improve, and consequently, the performance of Influence Distillation is expected to increase. This trend is indeed reflected in the $\\textit{general}$ behavior shown in Figure 3.\n\nHowever, as noted in Table 1, certain tasks—such as GSM8k and Codex—show a high variability and are particularly noisy. We believe the slight performance drop observed at 8192 landmarks is likely due to this noise.\n\n> 6. Transferability of embeddings between models\n\nTo investigate this point, we conducted an experiment to study how the average cosine similarity between approximated and true gradients of Llama2-7B is affected by the choice of embedding model. Specifically, we compare the performance of the following embeddings: JVP from Llama2-7B (same as the target model), JVP from Llama3.2-3B, JVP from Qwen2.5-1.5B, and JVP from Qwen2.5-3B. We fix the number of landmarks at 4096. The results are summarized in the following table:\n\n| |JVP (Llama2-7B)|JVP (Llama3.2-3B)|JVP (Qwen2.5-1.5B)|JVP (Qwen2.5-3B)|RDS+ (Llama2-7B)|\n|--|--|--|--|--|--|\n|gradient recovery|0.105|0.098|0.093|0.103|0.073|\n\nComparing these results with the recovery achieved by RDS+, we conclude that JVP embeddings are highly transferable across models. Notably, JVP embeddings computed from Qwen2.5-3B approximate the gradients of Llama2-7B with similar quality to those obtained directly from Llama2-7B itself.\n\nWe appreciate the reviewer’s constructive comment and will include these results and a discussion in the next revision of the paper.\n\n> 7. Performance of raw base models\n\nWe thank the reviewer for the suggestion and will include these results for all models in the next revision. For the purpose of this rebuttal, we have computed the results for Llama2-7B, which are presented in the table below:\n\n| |MMLU|GSM8k|BBH|TyDiQA|Codex|SQuAD|\n|-|----|---|---|------|-----|-----|\n|Base Llama2-7B|39.9|3.1|39.8|17.3|18.9|10.3|\n|InfDist|48.3±0.21|20.3±1.65|43.2±0.67|53.6±0.34|29.5±3.14|83.2±1.02|\n\nAs expected, without instruction tuning, the base model performs poorly on all tasks.\n\n> 8. Typo: fine-tuning\n\nWe appreciate the reviewer’s attention to detail and will correct this typo in the next revision."}, {"review_id": "Gz67LfCuCu", "reviewer": "Reviewer_MsKh", "summary": "This paper introduces Influence Distillation, a new data selection framework for efficiently training Large Language Models (LLMs). The method uses second-order information to assign model-specific weights to training samples, which helps guide LLM fine-tuning towards strong performance on a target domain. To achieve scalability and reduce computational costs, Influence Distillation employs a landmark-based approximation where influence is calculated for a small subset of \"landmark\" samples and then efficiently extended to other samples. Experiments on the Tulu V2 dataset, covering tasks like GSM8k, SQUAD, and MMLU, demonstrate that Influence Distillation performs comparably to or better than state-of-the-art methods while accelerating selection by up to 3.5 times. The framework has been validated across various models from the Llama and Qwen families.", "questions": "N/A", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The method is computationally efficient and scalable, especially for large datasets, by using a landmark-based approximation. This allows influence to be precisely computed for a small subset of \"landmark\" samples and then efficiently propagated to all other samples.\n\n2. Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to 3.5x faster selection. It also achieves higher performance compared to other more expensive selection baselines in three out of four models and remains competitive in the fourth.\n\n3. It is a mathematically justified framework for data selection.\n\n\nWeaknesses:\n\n1. The unconstrained version of the proposed solution may produce negative or highly irregular sample weights, including excessively large values, which lack intuitive interpretation. And the unconstrained weights may overfit to the current set of model parameters and the target dataset.\n\n2. The cost of the warm-up phase is excluded from runtime measurements, with the justification that it becomes negligible for larger training pools or can be shortened. However, this is an exclusion from the reported runtime.\n\n3. Calculating the Hessian (Q(θ)) exactly requires storing the backward graph, which can incur extra memory costs. Constructing the G_S matrix requires computing gradients for each individual training sample, which is computationally expensive and memory-intensive", "was_revised": true, "final_justification": "The author provide sufficient explanation to my questions. I will uphold my score.", "rebuttal": "We thank the reviewer for their comments. Below please find our response to the issues raised:\n\n> 1. Unconstrained weights being irregular\n\nThe reviewer’s observation is indeed correct, and we also observe this in the “Discussion” paragraph around line 157 of the paper. In response to this issue, we have introduced a regularization scheme for the influence weights that (1) enforces the weights to sum to the size of the source dataset, (2) constrains them to be non-negative, and (3) applies L2 regularization to prevent overly large values. This leads to what we refer to as “Robust Weights” (introduced around line 166), which effectively address the concerns raised by the reviewer. An example of the resulting weight distribution is shown in Figure 2 (middle).\n\n> 2. The cost of warm-up\n\nWe acknowledge this limitation, as noted in the Limitations section of our paper. However, we believe it is not a major concern for two reasons: (1) as the training pool grows, the cost of a brief warm-up on a small random subset becomes negligible compared to the cost of embedding the full dataset; and (2) the warm-up phase can be shortened (as demonstrated in Appendix A) or further compressed—for example, using Low-Rank Adaptation (LoRA) [1], as done in the LESS method [2]. A more thorough exploration of warm-up optimization is left for future work.\n\nTo provide a clearer picture of the current runtime, we estimate the cost of our (unnecessarily long) warm-up phase. In the setting of Table 1, the warm-up cost for Llama2-7B is approximately 420 TeraFLOPs (TF). Including this, the total runtime of Influence Distillation with 4096 landmarks is about 1292 TF, which is still 2.2x faster than the RDS+ baseline.\n\n[1] https://arxiv.org/pdf/2106.09685\n\n[2] https://arxiv.org/pdf/2402.04333\n\n> 3. The cost of gradients and Hessian\n\nWe agree that the naive version of Influence Distillation would be prohibitively expensive due to the need to compute and store full gradients and Hessians. This is why Section 4 of our submission is dedicated to addressing exactly these challenges. Specifically: (1) we approximate gradients using a theoretically grounded landmark-based method, which enables efficient and accurate estimation of gradient similarities between source and target samples; and (2) we argue that explicit computation of Hessians is unnecessary, as the second-order term becomes negligible under practical learning rates. This significantly reduces the computational burden while maintaining performance, shifting the pareto frontier of runtime vs accuracy as shown in Figure 1."}, {"review_id": "zbuI7Pqkeu", "reviewer": "Reviewer_a4sj", "summary": "This paper introduces Influence Distillation, a novel mathematically-justified framework for efficient data selection in large language model (LLM) training. By leveraging second-order information (gradients and Hessians), the method optimally weights training samples based on their influence on a target distribution, with derivations provided for both Gradient Descent and Adam optimizers. To address scalability, a landmark-based approximation is proposed, where influence is computed for a small subset of landmark samples and propagated to all others using Jacobian-vector product embeddings, significantly reducing computational costs. Experiments on the Tulu V2 dataset with Llama and Qwen family models across tasks like MMLU, GSM8k, and SQuAD show that Influence Distillation matches or outperforms state-of-the-art methods while achieving up to 3.5× faster selection, positioning it as an efficient solution on the Pareto frontier of cost versus accuracy.", "questions": "1. The paper lacks comparisons with the following baselines: LESS[1], DSIR[2], QUAD[3], TAROT[4] and IFD[5]. Specifically, LESS replaces the influence function with gradient similarity to approximate data utility. QUAD employs the K-FAC method and data clustering to accelerate the computation of inverse Hessian-vector products (iHVP).  Superfiltering performs data selection based solely on loss metrics such as perplexity (PPL) or IFD. Please compare these methods in the experiments and explain the results in terms of both effectiveness and efficiency.\n\n[1] LESS: Selecting influential data for targeted instruction tuning.\n\n[2] Data selection for language models via importance resampling.\n\n[3] Harnessing Diversity for Important Data Selection in Pretraining Large Language Models.\n\n[4] TAROT: Targeted Data Selection via Optimal Transport.\n\n[5] Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.\n\n2. The paper lacks an ablation study on critical hyperparameters, such as the optimal size of the \"small subset of landmarks\" during LLM instruction tuning—specifically, how many samples comprise this subset. This omission leaves users uncertain about efficiently determining this value when applying the method, as well as whether the optimal number remains consistent across different models and datasets.\n\n3. To the best of my knowledge, there are several existing baselines that are closely related to your method, including:\n\ni) IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment\n\nii) Harnessing Diversity for Important Data Selection in Pretraining Large Language Models\n\niii) MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models\n\niv) TAROT: Targeted Data Selection via Optimal Transport\n\nSpecifically, LESS improves effectiveness by computing gradient similarity at each training iteration. QUAD and IDEAL employs the K-FAC method to accelerate the computation of inverse Hessian-vector products (iHVP). TAROT leverages optimal transport in the gradient space to measure distances between data points. Please provide a detailed comparison(Only a theoretical explanation is needed) of your method against these approaches, highlighting both advantages and limitations.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a mathematically justified framework for data selection using second-order information (gradients and Hessians), offering closed-form solutions for optimal sample weights under both Gradient Descent and Adam optimizers. This theoretical foundation distinguishes it from heuristic-based methods .\n\n2. The landmark-based approximation significantly reduces computational overhead by leveraging a small subset of \"landmark\" samples and propagating their influence via Jacobian-vector product (JVP) embeddings. This makes the method feasible for large datasets.\n\n3. Extensive experiments on multiple LLMs (Llama, Qwen) and diverse tasks (MMLU, GSM8k, SQuAD) demonstrate that Influence Distillation outperforms or matches state-of-the-art methods.\n\nWeaknesses:\n\n1. The experimental section includes relatively few baselines and lacks an ablation study on key parameters (see question 1 and 2).\n\n2. There is a lack of discussion on other influence function-based methods for data selection in LLMs (see question 3).", "was_revised": true, "final_justification": "The rebuttal has largely addressed my concerns, and I raise my score to 4.", "rebuttal": "We thank the reviewer for sharing their valuable insights. Below find our response to the questions:\n\n> 1. The choice of number of landmarks\n\nWe appreciate the reviewer’s comment. As shown in Figure 3 (left), we have empirically studied the effect of the number of landmarks on the performance of Llama2-7B. The results indicate that performance stabilizes around 4096 landmarks. For other models, we did not perform any tuning and simply used the same value (4096) across the board.\n\nDetermining the optimal number of landmarks is indeed an important and interesting question. One potential approach is to incrementally add landmarks and, at each step, approximate the gradient of each landmark using the other ones. Once the approximation quality reaches a certain threshold, the process can be stopped and that number can be used as the chosen landmark count.\n\nWe explored this idea empirically by implementing a procedure to adaptively select landmarks for Llama2-7B. The method proceeds as follows:\n\n1. Begin with a small set of landmarks L, initialized with 32 randomly selected samples. Compute and project their gradients using a Rademacher-based projection (similar to LESS [1]).\n\n\n2. For each sample in L, approximate its gradient as a linear combination of the other samples (**excluding itself**). We use kernel ridge regression (KRR) with an RBF kernel to perform this approximation. To ensure that each sample is not used in approximating itself, we set the diagonal of the Gram matrix to zero.\n\n\n3. Compute and report the average cosine similarity between the true and approximated gradients.\n\n\n4. If the size of L is less than 8192, add one new random sample to L, compute and project its gradient, and repeat from step 2. Otherwise, stop the procedure.\n\n\nRunning this experiment, we observe that the average cosine similarity improves smoothly as the number of landmarks increases. The table below presents the average cosine similarity values for several different sizes of the landmark set L:\n| Size of L|32|64|128|256|512|1024|2048|4096|8192|\n|-|--|--|---|---|---|----|----|----|----|\n|Gradient recovery|0.1106|0.2317|0.3416|0.4534|0.5586|0.6797|0.7958|0.8894|0.9480|\n\nThese results suggest that setting a threshold of 0.8-0.9 for approximation quality would naturally lead to selecting a number of landmarks consistent with what we used in the paper.\n\nImportantly, this procedure does not incur additional computational cost, as the gradients of the landmarks must be computed anyway for Influence Distillation.\n\nWe thank the reviewer for this insightful and valuable suggestion. While we believe that this method needs further investigation to establish its effectiveness, we agree that including this or a similar approach for selecting the number of landmarks would strengthen the paper.\n\n[1] https://arxiv.org/pdf/2402.04333\n\n> 2. Discussion on other baselines\n\nWe thank the reviewer for their constructive feedback and for suggesting several relevant baselines for comparison and discussion. This will help better position our work and clarify its contributions. We provide a point-by-point response below. We will include these results and discussions in the next revision of the paper.\n\n- IFD is a heuristic-based filter that relies on metrics like perplexity and is independent of the target model's training dynamics. Previous work (e.g., the RDS+ paper [6]) has already shown that such heuristics are generally outperformed by more sophisticated data selection methods like RDS+ with a large margin. Given our focus on model-aware influence and our competitiveness with state-of-the-art data selection methods, we believe a direct experimental comparison would be less informative for demonstrating the specific advantages of our approach.\n\n- DSIR is a model-agnostic method that uses importance resampling and does not leverage the target model’s parameters for selection. This places it in a different category from model-aware influence function methods like ours. This distinction is already discussed in the Related Work section.\n\n- MATES trains a small, separate model to predict the influence of data points during the main model's training. A key limitation is the substantial cost and complexity of training this auxiliary model, which must be done repeatedly. By contrast, our Influence Distillation technique is a far more streamlined framework that requires no auxiliary models.\n\n\n- IDEAL is an interesting concurrent work (published after the NeurIPS deadline), which _addresses a different problem_: Its goal is to find the optimal mixing of data from different domains or sources. Our method, in contrast, selects the most influential individual data points from a given data pool.\n\n- QUAD & TAROT & LESS: We agree that QUAD, TAROT, and LESS are relevant baselines. TAROT and LESS, however, require computing the gradients for each sample in the pool, which means they are significantly slower than Influence Distillation due to our efficient landmark-based gradient approximation. QUAD, on the other hand, though closely related, focuses on pre-training settings, which differs from our focus on supervised fine-tuning. Due to the significant computational resources and time required to implement and run these methods fairly, we only have included a comparison with LESS in the next point below. We will include comparisons with QUAD and TAROT as well in the next revision of our paper.\n\nOverall, this detailed examination of prior/concurrent work suggests that our original submission already compares against the most relevant baselines, and does not impact our claims of novelty or performance. We thank the reviewer again for their detailed additional references, which we will incorporate into the next revision of our work. \n\n[6] https://arxiv.org/pdf/2503.01807\n\n> 3. Comparison with LESS [1]\n\nWe thank the reviewer for this valuable comment. As noted in the last paragraph of Appendix F (also mentioned in Section 5.1, Baselines), we have included a brief discussion of the LESS method. Specifically, as an ablation, we replaced JVP embeddings with actual projected gradients. Figure 7 (right) in the appendix shows that this yields a gradient recovery of 0.9 (in terms of average cosine similarity) with 4096 landmarks, making the approach, in this case, very similar to LESS which uses exact (projected) gradients. The last row of Table 2 shows that this variant performs on par with Influence Distillation using JVP embeddings.\n\nTo provide a more complete comparison, we also ran the 10k/200k selection experiment on Llama2-7B using the LESS method (with one seed due to its high computational cost). In this setup, gradients are computed after a 10k-step warmup, projected down to 8192 dimensions, and compared to the target gradients. The top 10k examples are then selected based on similarity—following the original LESS procedure. Below we present a comparison between uniform sampling, InfDist with JVP embeddings, and LESS.\n\n|Method|MMLU|GSM8k|BBH|TyDIQA|CODEX|SQuAD|Avg|\n|------|----|-----|---|------|-----|-----|----|\n|Uniform|45.6±0.43|17.5±1.08|41.8±0.20|51.6±0.38|27.0±0.60|80.8±1.05|44.1|\n|InfDist+JVP Embedding (ours)|48.3±0.21|20.3±1.65|43.2±0.67|53.6±0.34|29.5±3.14|83.2±1.02|46.4|\n|LESS|48.2|22.7|43.4|55.8|29.1|86.0|47.5|\n\nWhile LESS achieves higher accuracy on average, it is significantly more computationally expensive. Our method requires approximately 872 TeraFLOPs (TFs) for embedding and selection, whereas LESS requires around 8400 TFs—approximately 10x more.\n\nWe thank the reviewer again for this insightful suggestion and will include these results in the main body of the next revision.\n\n[1] https://arxiv.org/pdf/2402.04333"}]

Author rebuttal:

Meta-review: This paper studies the problem of data selection for efficient model training. The authors introduces Influence Distillation, a data selection framework that uses second-order information to assign weights to training samples. The authors also propose a landmark-based approximation where influence is compute on a small landmark subset and propagated to the remaining samples. The proposed method achieves good empirical performance compared to several baselines. Reviewers raised concerns regarding comparisons to the LESS approach [1]. In the rebuttal, the authors showed that while LESS achieves higher performance, the proposed approach is much more efficient. After the rebuttal, all reviewers leaned toward acceptance.

[1] LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024)
Decision: Accept (poster)