FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Authors: Rohan Deb
Kiran Koshy Thekumparampil
Kousha Kalantari
Gaurush Hiranandani
Shoham Sabach
Branislav Kveton
Venue: ICML 2025 poster (2025)
Keywords: active learning
optimal design
supervised fine-tuning
large language models
Primary area: general_machine_learning->online_learning_active_learning_and_bandits

Abstract: Supervised fine-tuning (SFT) is the most common way of adapting large language models (LLMs) to a new domain. In this paper, we improve the efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we select those that maximize information gain, as measured by the Fisher information matrix of the SFT objective. We approximate it efficiently by linearization at the last layer of the LLM. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, with both quantitative results and LLM-as-a-judge evaluations.

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Rohan Deb Kiran Thekumparampil
Kousha Kalantari? Gaurush Hiranandani Shoham Sabach 24 Branislav Kveton
Abstract
2022). The key idea in LoRA is to add low-rank matrices to the matrices in the transformer layers and only optimize those during fine-tuning. The computational cost of LoRA and the quality of its approximations increase with the rank of the low-rank matrices. Therefore, the rank is a natural tunable parameter. The simplicity of LoRA made it popular in practice and thousands of different adapters have been trained (Mangrulkar et al., 2022). We propose a comple- mentary approach that selects subset of most informative training examples for fine-tuning. The computational cost of fine-tuning is linear in the size of the chosen subset. There- fore, as in LoRA, the number of chosen examples naturally trades off the computational cost of fine-tuning for quality.
Supervised fine-tuning (SFT) the most common way of adapting large language models (LLMs) new domain. In this paper, we improve the effi-
ciency of SFT by selecting an informative subset of training examples. Specifically, for a fixed bud- get of training examples, which determines the computational cost of fine-tuning, we select those that maximize information gain, as measured by the Fisher information matrix of the SFT objec- tive. We approximate it efficiently by lincariza- tion at the last layer of the LLM. Our approach is computationally efficient, analyzable, and per- forms well empirically. We demonstrate this on several problems, with both quantitative results and auudee evaluations.
The idea of selecting better training examples for SFT is not new and has been explored extensively before. Coverage- based approaches (Phillips, 2017: Tukan al., 2021) select diverse examples to form coresets. Quality-baset sampling (Wenzek et al., 2019; Muenchigoff et al., 2023) weeds out low-value or unhelpful examples. In ASK-LLM (Sachdeva et al., 2024), proxy LLM is prompted with potential train- ing examples and asked they should be used for training. We review these approaches in detail in Appendix A. The main difference in our work that we choose training ex- amples using their information value, measured by the Fisher information matrix of the SFT objective.
1. Introduction
Large language models (LLMs) (Bommasani et al., 2021) have emerged as general purpose tools that can solve natural language tasks in both zero-shot and few-shot settings (Rad- ford et al., 2019; Brown et al., 2020). LLMs are typically trained in three stages (Ouyang et al., 2022): pre-training on large corpus of diverse text, supervised fine-tuning in the domain of interest (Wei et al., 2022), and alignment to human preferences (Ouyang et al., 2022; Rafailov et al., 2023). The main challenge in all stages is the sheer scale of LLMs, which increased by four orders of magnitude in just four years: from 117 million parameters in GPT-2 (2019) to 1.76 trillion parameters in GPT-4 (2023).
Without loss of generality, we view training examples in fine-tuning as sentences, each being a sequence of tokens. We want to select the most informative n sentences, which determines the computational cost of fine-tuning. The key insight our work that the SFT objective is sum of next token oog-prbbabilitie,, each represented by multinomial logistic regression model (Bishop, 2006) that depends on all previous tokens. Therefore, the problem of selecting the most informative sentences for fine-tuning is a variant of an optimal design (Pukelsheim, 2006; Stufken & Yang, 2012) for multinomial logistic regression, where the information gain of sentence depends on all tokens in it. We derive an efficient approximation the Hessian of the SFT objective, which measures how informative set of sentences is, and then optimize its lower bound to find the most informative sentences.
In this paper, we focus making supervised fine-tuning (SFT) (Wei et al., 2022) more efficient. A standard approach in SFT is to optimize a low-rank adapter (LoRA) (Hu et al.,
University of Illinois, Oabana-Champaign. The work - was done during an internship at Amazon AAmazon "Typeface "Technion * 'Adobe Research. Correspondence to: Rohan Deb <id22@iliinois.du.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
We make the following contributions:
sentence by sequence of its tokens,
(1) We establish connection between the SFT objective of LLMs and product of multinomial logistic regression models in Section 2.
y/ (Vi,1, ...,,,...., M.,
To model the evolution of the sentence token-by-token, we define a vector x E R- that captures the relevant history up to position j in sentence In large language models, x,, is the output of the pre-logit layer that encodes contextual information about tokens yi,1,---1 Vi.j-1 and prompt 25-
(2) We propose our method in Section 3. Our main contri- bution is a computationally-efiiii
approximation to the oog-determinant of the Hessian of the SFT log-likeliile likelihood. Specifically, all matrices in this approximation are d x d, where d is the size of transformer embeddings, as opposing to dT, x dL, where I. is the number of distinct tokens. Since the log-determinant is both monotone and submodular, we maximize it greedily (Nemhauser et al., 1978). We call our algorithm FisherSFT because greedily selects sentences with jointly most informative tokens. This in stark con- trast to representing sentences by embeddings (Das al., 2024; Mukherjee et al., 2024; Thekumparampil al., 2024; Liu et al., 2024; Scheid et al., 2024), which we compare to in Section 5.
L be matrix of parameters in the last logit layer of the LLM and 07 RRd be the parameter vector corresponding to token & E [4]. Then the probability of token & at position j in sentence i is
Let O (07)ett)) E R
p(l x +*) exp(0; X1.))
(2)
Σk exp(O% x:.)
Using this notation, the SFT objective in (1) can be equiva- lently expressed as
(3) We analyze FisherSFT in Section 4 and show that its prediction error decreases at rate Ó(T./ /n), where n is the number of chosen sentences. The dependence on n is similar other recent results (Zhu et al., 2023; Mukherjee
1 ( - N
N Σ ==1
M; log log((nn
Xi θ)
(3)
Our objective to select a subset of n sentences out of N for fine-tuning. We denote by c [NN] and assume that |S| = The negative log- oog-liielhhoo on S, which helps us to choose the sentences, is defined as
et al., 2024; Thekumparampil et al., 2024).
(4) We evaluate FisherSFT empirically in Section 5. Our experiments on synthetic problems show that FisherSFT yields a lower prediction error than the baselines. We also fine-tune GPT-2 models and evaluate them using an LLM- as-a-judge (Zheng et al., 2023). The judge prefers the text generated by FisherSFT models by large margin.
Mi logp(un. x)
Cs(0) = 71 iES
(4)
j=1
Let
2. Problem Formulation
- argmin (s(
(5)
9
Supervised fine-tuning (SFT) (Mangrulkar et al., 2022; Hu et al., 2022) a direct application of supervised learning to LLMs. The objective of SFT is to minimize the negative log-
be the maximam likelihood estimate (MLE) of @* under the negative log- log-likelihood in (4).
likelihood of LLM responses given prompts. Specifically, let yi be response to prompt 2, and D = {(2,, yı) [ be a dataset N such prompt-response pairs. Then the SFT objective is to minimize
3. Algorithm
The Hessian of (4), which is also known as the Fisher infor- mation matrix (Fisher, 1922), is
Σ logr(y: 2:)
Mi
N
V L2
-
- iES 1=1
v2 logg(uis
Xiy; 9).
(6)
72
where p(y: 2:) denotes the probability of y/ given 2; under the LLM. In this work, we select the most informative data points for optimizing (1) by linearizing it, and building on existing results for active learning in linear and generalized linear models (Pukelsheim, 2006; Stufken & Yang, 2012).
The inverse of this matrix is the covariance matrix of the MLE in (5). Therefore, v²Ls(O*) can be used to quantify the uncertainty of Ô and gather training examples for im- proving our estimate of O' (Pukelsheim, 2006; Stufken & Yang. 2012; Lattimore & Szepesvari, 2019).
The lincarization is done as follows. We call yı a sentence. Let sentence yi consist of M, tokens indexed by j E [M,]. Let yr. E [L] be the token at position j in sentence i. The tokens belong to a vocabulary of size L.. We represent the
In this paper, we optimize L²Ls(0") by maximizing its eigenvalues with respect to S, and approximate this as max- imizing log det Vet((V(., LO()))) This optimization problem
2

---

FisherSFT: Data-Efficieet Supervised Fine-Tuning of Language Models Using Information Gain
Algorithm 1 Greedy optimal design for language models.
Algorithm 2 FisherSFT: Fast Implementation of Algo-
1: Input: Embeddings {(x),)))= J
rithm 1.
2: Design matrix V +- --004, J
1: Input: Embeddings {(xi,)))=1 ==)==1 batch size B
3: Selected sentences 8 5 0
Design matrix V + oold
4: for = 1...., n do
3: Selected sentences S t- /
Me
4: Cached information gains g 00N
5: I < argmax log det iE[M]
V+ Σ Xis
5: for /. = 1,..., do
j=1
6: Jmax t 0
6:
S S + {
7: for b = 1, ,,...,N/BI. do
M
B 4- {(b 1)B + 1,............ bB}
7:
V+
Xk.j
for all E B do
10:
if gi Imax then
8: Output: S
Mr
11:
9i t log det
V + Σ ==1
difficult for three reasons. First, V Ls is a dL x dL matrix. For practical values of d = 10 and 10 is computationally infeasible optimize dL x dL matrices. Second, V (e*) cannot be optimized directly because O" is not known. To address these two issues, we derive lower bound on log det(V2 (V)) that only involves d x d matrices and does not depend on O We present the lower bound in the following lemma.
log (V)
12:
Jumx max (19)) {9mmat)
13:
k + E 9:
14:
S+ S + {k}
M
15:
V+
16: Output: S
Lemma 3.1. Consider the loss function given in (4) and let p(- [xt,; θ) be the probability vector whose l-th compo- nent is p([|x ②). Then the Hessian of the loss is
3.1. Greedy Optimal Design
Our algorithm is presented in Algorithm 1 and we explain it next. We call the Hessian of the SFT objective design matrix because we use it to design the set of chosen sen- tences. The design matrix is initialized as V = oola (line 2), where 2 0 is the strength of regularization. The set of chosen sentences initialized as S = 0 (line 3). In step E [n], we select a sentence from the remaining sentences [N] 3 S that maximizes log det of the design matrix of the previously chosen sentences (line 5). This sentence has the highest information gain. Intuitively, it contains the most diverse embeddings X;, since log det(()) is the logarithm of the volume of the ellipsoid represented by V, and this is maximized when the lengths of all its axes increase equally (Lattimore & Szepesvari, 2019). After the sentence is cho- sen, we add it to the current subset of sentences S (line 6) and Σ X added to the design matrix V (line 7).
M
Ls))
I
Σ
diag(p( |Xi_i 0))
11
VES
- -p( | 0)p( [)
& x
where & is the tensor product and diag(v) is a matrix with diagonal v. Moreover, if
diag(p( 0)) - p( [xi.i O)p( O)
holds for some 7 > 0, then
M, ΣΣ
log det((V22eeeee ( (s)))) 2 d'log det
( iES
x
Algorithm 1 selects sentence per step (line 5). In each step, we compute log det for all remaining sentences. This is clearly not practical. In Section 3.2, we present a faster algorithm that leverages the submodularity of log det and parallelism to produce the same set of sentences S.
Proof. The lemma is proved in Section 3.4.
Therefore, instead of maximizing log det (O))). we maximize log det(Eies Σド xx,x),)) and this is the key idea in our algorithm FisherSFT. The last challenge is that we have a combinatorial optimization problem, choose a subset of n sentences out of N. Since log det is a mono- tone submodular function, we solve this problem greedily (Nemhauser et al., 1978).
3.2. Algorithm FisherSPT
Now we present a more computationaall--ffiiiie variant of Algorithm that exploits the submodularity of log det and parallelism (Algorithm 2). Simply put, we implement line
3

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
in Algorithm more efficiently. This corresponds line 13 in Algorithm 2.
is a training point in SFT, for the conditional probability of token Vi.j given the history embedding Xi.- Therefore, longer sentences may have higher information gains. Nev- ertheless, note that the sentences in FisherSFT are chosen based on their total information gain and not just length.
The key idea is to cache information gains, where 9: is the cached information gain for sentence i E [.V] The gains are initialized as 9: = oc (line 4), updated in line 1 11, and we act greedily with respect to them in line 13. If the gains were always updated, line 13 would be clearly identical to line 5 in Algorithm because the maximized values in line 13 are only offset by constant log det(V).
The key idea in our work can be viewed as using the last- layer embedding in LLMs as a featurizer (Xu et al., 2022). This is known yield robust performance in active learning (Riquelme et al., 2018). Other popular active learning tech- niques utilize gradients (Ash et al., 2020) or are based on bandit exploration (Zhou et al., 2020; Zhang et al., 2021). These methods are computationally prohibitive in modern neural networks because of the sheer size of their parameter spaces.
The key insight to efficient updates is that log det is a mono- tone submodular function. Therefore, the gains cannot in- crease as V is updated and hence do not have to be recom- puted when they are smaller than the highest gain gmax at step & [n]] thus far. We exploit this structure in line 10 and update gmax in line 12. Finally, we update gi in batches of size B (line 9) in parallel. This results in additional (BB) speedup. We use this implementation in our experiments and call it FisherSFT.
3.4. Proof of Lemma 3.1
In Proposition B.1, we show that
Mr
V 1 IES
ΣΣ Σ
(diag(p( (xx,;:)))
3.3. Discussion
We consider two variants of FisherSFT in this work. In Section 4, we analyze an idealized variant where the pre- logit layer of the LLM is treated as a fixed feature vector. After n sentences are chosen by FisherSFT, we estimate the model parameter O" by maximizing the log-likelihood in (5). We argue that Ô approaches O* as the sample size n increases in Section 4. We experiment with this setting in Sections 5.1 and 5.2.
- p( |xs.ji O)p( (x(s; @ x
Now suppose that
diag(r( [x.j; θ)) - -p( [x.;; O)p( (x,j; =y
holds for some > 0. Then using Theorem 4.2.12 of Horn & Johnson (1991), we get
We fine-tune LLMs in Section 5.3. The main difference in this setting is that (s(e) is only used to select a subset of n sentences but we fine-tune the LLM instead of solving (5). The structure of the LLM is taken into account using embeddings Xt.j At a high level, FisherSFT chooses di- verse sentences ensure more uniform coverage of x,,-- This reduces the original dL x dL Fisher information ma- trix optimization into a tractable d x d problem, enabling efficient optimization. We discuss several notable aspects of our approximation next.
M &
V L'L)
ΣΣ r
x
TI iES
where IT. is an L-dimensional identity matrix. In turn,
Mi X
det(v>L(((O) 2 det (Ir.
2 11
xT.s))
VE
y=1
Now using the fact that (c & B) det (A) det(B)) for A E RexP and B Raxq (Bersttii, 2009, Proposition 7.1.11), we have
The Fisher information matrix in FisherSFT is derived in Lemma 3.1. The algebraic form of this matrix, the outer product of feature vectors, would be the true Fisher infor- mation matrix in linear models. Optimization of this matrix leads to choosing feature vectors that uniformly cover all directions spanned by x.j- Therefore, FisherSFT controls the worst-case prediction error over Xi.j and hence ensures robustness. We observe this in our experiments. This pro- vides different perspective than Sorscher et al. (2022) who show that it is possible to break beyond power law scaling by intelligently choosing easy and hard training examples.
M. M
det V2 Ls())) 2 det(IL)* det (2
Σ
X x
iES j jml
M.
det
2
ΣΣ
X
7 iES
Taking the logarithm of both sides completes the proof.
4. Error Bound
FisherSFT is naturally biased towards selecting longer sen- tences. This bias arises because each token in a sentence
Our main Theorem 4.3 provides a O(1/vn) bound on the maximum prediction error of the estimated parameter é
4

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
from the sentences collected by FisherSFT. The maximum prediction error is given by
where C >> 0 is some global constant.
We sketch the proof next. Let S be the set of n sentences generated by FisherSFT. For any positive-definite matrix
[8X ΣE=1 110* X1,j -Txx,,, X1,
A Rdxd and vector E R we define || Vx Av.. With é (Φe) and O (e())) we can bound the maximum prediction error as
Note that -1- ll2 is the sum of prediction errors over vocab- ulary of size L. We make the following assumption on the feature vectors and unknown parameter O*
Assumption 4.1. Let ||x S for any i E [N] and j E [M.]. Moreover, assume that the true model parameter satisfies O* B, where
M
6 Xz, - 0¹ xx,5112
max LE[N]
B { θ = (0))RE)2) O2 E R. .10ell2 S 1,
M.
S max
1 - -07) 'x,,
d
sE[N]
02x - 0, 0, 1
j=1
M,
s max sE|M
110e 07 l11ss | 1ss.
For any subset& S S [ we define the regularized design
t=1
matrix for S as
Mr max KE(N
110g
07 12ssss)
Xx llsg'.
(8)
M x x
1=1
oo + iES j=1
(7)
for do 0. FisherSFT iteratively computes this matrix as new sentences are added to the subset S (line 15).
Term I represents a self-normalized error between the true parameter @* and its MI.E é. Term II measures the infor- mation value of sentences S. We start with bounding term II. Under Assumption 4.2, we get the following bound.
Next we make a diversity assumption on our dataset. Assumption 4.2. Let S be the set of chosen sentences up step / in FisherSFT and I, be the index of the chosen sentence at step t. We assume that there exists constant 2 such that for any step & [r|] and sentence i E [N].
Lemma 4.4. Suppose that Assumption 4.2 holds. Let be the set of n sentences generated by FisherSFT and (7) be their design matrix. Let M mmaxeel]] Mj. Then
log [La +
Σ
M,
1
1/2
x 11
1
M max 8E [
Xx,
0 2oo (1 +
2o 3nM
d
RdM
/=1
log(1 + t00
n
Mi.
S #elogdet(La +
Σ
2-11
1 1/2
X Xl.d
x 1T1
T
Proof of Lemma 4.4. See Appendix C.1
where ET = Ds....
Using the Cauchy- Schwar-Schwarz. inequality, we get a corollary
The assumption says that the information gain of the chosen sentence (line 13 of FisherSFT) an approximate upper bound, up to multiplicative x 2 1, on the information gain of any other sentence, including those previously chosen that cannot be chosen again.
S M oa log (1 + To 2nM
M; Σ
rd
IDAX N 1=1
log(1 + 20)
n
We bound term I in (8) by relating it to the difference be- tween the loss and its first order approximation,
Now we state our main result.
Theorem 4.3. Suppose that Assumptions 4.1 and 4.2 hold, Then for any 8 O, the maximum prediction error of θ is bounded with probability 1 - å as
Ls(0) Ls(()) - (VLs(0"), -
(4) - -VCss(())
M
Xi.j
θ T xixii
VeLs(O)) ((ee -7)
N
To 2 log
(1+ 55 353nM
I
S SCMPLA
d
dk(ld + log(L/0) (L/))
Σ ViLs |عاا 110e 0fl15s 99)
log(1 + o.
n
5

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
where the dot product between matrices A and B is defined as (A, B) En, Ai, B... Inequality (a) is consequence of (() < Ls(())), because Ô is the minimizer in (5). Inequality (6) follows from the Gauchy-Schward inequality. Finally, we bound the left-hand side of (9) from below using the fact that the loss is strongly convex.
5. Experiments
We empirically evaluate FisherSFT on a variety of prob- lems. We experiment with a synthetic autoregressive predic- tion task in Section 5.1, with pre-trained word embeddings in Section 5.2, and with GPT-2 models in Section 5.3. Abla- tion studies are conducted in Section 5.4. Our implementa- tion is available at github.
Lemma 4.5. Suppose that Assumption 4.1 holds and Ô is the MLE in (5) such that Ô E B. Then there exists some a < 1 such that
5.1. Synthetic Experiments
Ls(Ø) (s(())
(VLs(0"))
[L] We start with a simplified setup where each token & E is associated with a vector sampled from a standard normal distribution, X₃ ~ NNO,, Ta). The number of tokens is I, = 20 and d = 10. All entries of O, are sampled i.i.d. from (0, 1). The first token in sentences is sampled uniformly at random from [4]. The next tokens are sampled from the softmax model in (2), where x is the token embedding at position j 1 in sentence i.
e 2a
1
2
2
Σ
110g - 8j 8il1sss
L
Proof of Lemma 4.5. See Appendix C.2
Using Lemma 4.5 and (9), we have
2cx
L
100g 07 ||5s))
We consider several baselines. Uniform selects sentences uniformly at random. SentenceOD selects sentences greed- ily by maximizing the log-determinant of a sentence-level Fisher information matrix. We construct sentence embed- dings by summing all token embeddings in that sentence. Specifically, x, = =ンك Xi,j is the embedding of sentence i E [N]. Densi ComnstySSmpiiin (Sachdeva et al., 2024) selects sentences based on inverse propensity scores estimated by a kernel density estimator. ClusteredSampll
L
1=1
VeLs( "V -0il0- 07llss
M,
VeLs )
Σ 110e 07 11s
sup <E|4)
1=1
(Axiotis et al., 2024) clusters sentence embeddings using k-means and then samples them proportionally to their distance to the closest mean plus its loss. The baselines are described in detail in Appendix A.
and therefore
110e 07llss S 2n2 L sup
VeLs 11)
[E[L]
S e² sup L)
VeLs(0)) بلاا" (10)
All methods choose n sentences and learn multinomial logistic regression model by solving (5). We evaluate the methods by two metrics: maximum prediction error
The next lemma bounds suptat4] VeLss ".z
1|
Lemma 4.6. With probability 1 - 8, the gradient of the loss satisfies
M
Emax (1))
max [E[N
10 Xi.j xxxille
[Vees((")lls) SCVd+ (l))), EEIL]
sup
and mean prediction error
where C > 0 is some global constant.
M.
Proof of Lemma 4.6. See Appendix C.3
Emean (n)
Σ (E[N] ;=1
Σ
xi - - .Tx,,ll2. 10
NN
Now we combine (8), the corollary of Lemma 4.4, (10), and Lemma 4.6, and get that
The maximum error measures the performance on the most challenging sentence, while the mean error measures the average performance on all sentences. We bound the maxi- mum prediction error of FisherSFT in Theorem 4.3.
Mi
10 Xi.j - Ô @Txisll2
MAX N
The prediction errors of all methods reported in Fig- ure 1. We observe that FisherSFT performs better than all baselines in both metrics. In fact, it is much more sample efficient than the best baseline. As an example, the lowest maximum prediction error of the best baseline, attained at holds with probability 1- 6, where C > is some constant. This completes the proof.
T 2 log (1 (1+ aznM d
dr((d + log(L
S LCME2L
log(1 1 + ob)
n = 2 000, is attained by Fi FisherSFT at n = 000.
6

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Synthetic
Synthetic
FisherSFT
FisherSFT
Uniform
Uniform
SertenceOO
Mean prediction error
SentenceeD
Masimum prediction
Surppnng
ClusterddSamplingg
DensitySampiing
DensitySemping
100
2003
1000
2000
Sample size
Sample size n
Figure 1. Comparison of maximum and mean prediction errors on synthetic tokens. The x axis shows the number of selected sentences for training the model. The y axis shows the averaged 20 runs.
word2vec
word2vec
Maximum prediction error
60
Uniform
FisherSFT
20.0
Uniform
FisherSFT
SentenoeOD
Mean prediction arror
SentonceOD
ClusteredSamplingg
CuusteredSamping
60
DenisitSSapling
DenistySampling
16.0
40
1000
1250
1500
1750
2000
1020
1250
1500
1750
2000
Sample size
Sample
Figure Comparison of maximum and mean prediction word2vec tokens. The axis shows the number of selected sentences for training the model. The axis shows the errors averaged over 20 runs.
Uniform Sampling
FisherSFT
First Citizen:
The gods grant that! Might but thee Ishall lose that crown,
My life and services
First Citizen: The wars this matter, and more are open'd, More are open and more ungovern't 'd are people. mean drop
Will thereto witness ss good Rome and that thou
In token my deeds: both your grace
theirs and, but when they do, They will bring in for consul; and therefore, in time, For certain care wheth er they will or no respect. You sigh when you see them
ylelded.
life and all that lost
them
life, which never
My life and services
At any time when
saw My life; which never
My life,
never life, which
BAGILIA: So must fell, know, and love you
life, which never
CORIOLANUS: Hail, soldiers!
My life, which never
My life shall have seen; being now dishonour'd d by my life,
MENENIUS: You weary of the tribunes' suit: Why, patience! what care they of?
And howled thet life, but beheld In doing the thing said honoursbly;
Being criminal, which
Are bound and to blame Your igurignorance, which finds not till you have The sta mp of that banishment.
COMINIUS:
The gods begin mock you,
My life, which never
My life did incur the life-service,
My life which never
My life service the life; giving Life to both parts
Figure 3. Text generated by fine-tuned GPT-2 models on sentences chosen by Uniform and FisherSPT. The latter is more coherent.

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
5.2. Word2vec Embeddings
Method
Number of selected sentences
This experiment is similar to Section 5.1. The difference is that we use pre-trained word2vec embeddings (Mikolov et al., 2013) of dimension 300. We randomly select L = 20 words from the word2vec vocabulary and project their embeddings randomly to d = 10 dimensions. The vector associated with token E [L] is X The prediction errors of all methods are reported in Figure 2. We observe again that FisherSFT outperforms all baselines in both metrics. As an example, the lowest mean prediction error of the best baseline, attained at 2 000, is attained by FisherSFT at = 1 250.
FisherSFT vs
100
200
500
1000
2000
5000
Uniform
0.80 0.56
0.60
0.59
0.64
0.74
DensitySampling
0.61
0.66
0.68
0.62
0.54
0.84
AsKLLM
0.59 0.52 0.68
0.59
0.68
0.74
Table I. Win rates of FisherSFT with respect to three baselines on the Shakespeare corpus. A higher win rate than 0.5 means that FisherSFT is preferred.
Method
Number of selected sentences
FisherSFT vs
100
200
500
1000
2000
5000
Uniform
0,84
0.81
0.75
0.64
0.65
0.93
5.3. Experiments with GPT-2
DenaitySampling
0.68
0.75
0.69
0.74
0.65
0.88
AsKLLM
0.74
0,58 0.65
0.60
0.61
0.89
We experiment with GPT-2 models (Radford et al., 2019) next. All methods select subset of sentences and then fine-tune the GPT-2 model on Hugging Face (Wolf et al., 2020). The vector Xi,j is the output of the pre-logit layer at position j in sentence i and Bi, is the token at that po- sition. We consider two corpora: tiny Shakespeare corpus (Karpathy, 2015) and Sherlock Holmes corpus (Doyle). We subsample 000 sentences from both corpora and experi- ment with learning from E [100, 000] sentences.
Table 2. Win rates of FisherSFT with respect to three baselines on the Sherlock Holmes corpus. A higher win rate than 0.5 means that Fi FisherSFT is preferred.
gets the expected benefit of our method, that it yields more fluent, coherent, and natural text. The text generated by the compared methods is randomized: one randomly-chosen method replaces textl and the other text2. We use the same initial phrase to generate text and text2. The LLM judge chooses the first position with probability 0.54, which is sufficiently close to a completely unbiased 0.5. Examples of textl and text 2 are shown in Figure The model trained on uniformly selected sentences generates worse text, which is repetitive. In contrast, the FisherSFT text is more coherent and similar to the Shakespeare corpus.
We choose two baselines from the previous experiments: Uniform and VensitySampling chosen because it tends to outperform other baselines on language model fine-tuning tasks (Sachdeva et al., 2024). We also experiment with AskLLM (Sachdeva et al., 2024), where proxy LLM is prompted with potential training examples and asked if they should be used for training. The baselines are described in detail in Appendix A.
ConsitySamp ing is
We evaluate FisherSFT by its win rate, the fraction of time that its responses are judged as more fluent than the baseline. The win rate is estimated from 100 runs. A higher win rate than 0.5 means that FisherSFT outperforms the baseline. We report the results on Shakespeare and Sherlock Holmes corpora in Tables and 2, respectively. Note that all win rates are higher than 0.5. Notably, all win rates except one in Table 2 are at least 0.6. Therefore, FisherSFT performs better than the baselines by a large margin.
Unlike in Sections 5.1 and 5.2, the true model parameter is unknown, and hence the prediction errors cannot be com- puted. To address this, we evaluate the fine-tuned models based on their generated text. Specifically, we prompt them with few words from the original text corpus (represented by tokens yi. Di.p) and the model completes the sen- tence by generating pi 1: Mipt2,.... 3i, 1024. We evaluate the quality of the completed sentences using a larger GPT- 4o model, which serves as an I.L.Massa--udge (Zheng et al., 2023). For the Shakespeare corpus, we use prompt
5.4. Ablation Studies
You are Judge of Shakespeare text. <tagl>texti</tag>> <tag2>text2<<tagp>
Convergence rate. We validate the empirical convergence rate of FisherSFT in Figure 4a. Specifically, we take the synthetic problem Section 5.1 and plot the logarithm of the error rate as a function of the logarithm of the sample size. We observe slope of -0.3, which validates that the error rate is O(1P). We believe that this is sufficiently close to the expected p -0.5, especially since other factors may have played a role at our small sample sizes.
Respond 2 if the text inside <tag?> is more fluent shakespeare text than the text inside <tagl>, Respond otherwise.
For the Sherlock Holmes corpus, we replace "Shakespeare" with "Sherlock".
Computation time. We report the computation times of the slow and fast implementations of FisherSFT in Algo-
The prompt does not name the compared methods and tar-
8

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Synthetic
Snakespeare
FisherSFT
Computation Time (minutes)
FisherSFT (Algorithm-
Maximum prediction error
FisherSFT (Algorithm 2)
400
300
200
100
10a
10a
1000
Sample size
Sample szee
Figure 4, a: A log-log plot of the error rate of FisherSFT as a function of the sample size n. b: Computation times of the slow and fast implementations of FisherSFT in Algorithms and 2, respectively.
Synthetic (L 50)
Synthetic (d = 20)
0.7
FisherSFT
FisherSFT
Maximum prediction error
Uniforrm
Maximum prediction error
Uniform
0.6
SentenceOD
SentenceOD
DensitySarppling
DensitySampling
0.5
ClusterdSSampling
0.4
ClusteredSampling
0.4
0.3
0.2
10
15
20
30
35
40
20
25
30
35
40
50
Figure 5, Error rates of FisherSFT and all baselines from Section 5.1 as functions of the embedding dimension d and vocabulary size L.
rithms and 2. respectively, in Figure 4b. The experiment is done 5 000 sentences from the Shakespeare dataset in Section 5.3. We plot the computation times for various sample sizes and observe that the fast implementation is about 4 times faster.
Our work can be extended in several directions. First, we only experiment with small GPT-2 model. Second, fine-tuned models are evaluated by an LLM-as-a-judge but not humans. Finally, note that the optimal design could be computed using different embedding than that from the fine-tuned L.M. This could lead to computational savings if the embedding had much lower dimensionality.
Trends in and L. We plot the error rates of FisherSFT and all baselines from Section 5.1 functions of the em- bedding dimension d and vocabulary size in Figure We observe that FisherSFT has the lowest error in all set- tings, indicating robustness to changes in the embedding dimension and vocabulary size.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
6. Conclusions
We improve the efficiency of supervised fine-tuning by se- lecting the most informative training examples. The exam- ples maximize novel approximation to the information gain of the SFT objective. The approximation is computa- tionally efficient and motivates our algorithm FisherSFT. We analyze FisherSFT, and evaluate it empirically on both synthetic problems and fine-tuning GPT-2 models. We ob- serve that FisherSFT has lower prediction error than the baselines and its fine-tuned models preferred over the baselines in judee evaluations.
References
Abbas, A., Tirmala, K., Simig, D., Ganguli, S., and Mor- cos, A. S. Semdedup: Data-efficient learning at web- scale through semantic deduplication. arXiv preprint arXiv:2303.0955. 2023.
Ash, J., Zhang, C., Krishnamurthy, A., Langford, J., and Agarwal, A. Deep batch active learning by diverse, un- certain gradient lower bounds. Proceedings of the 8th

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
International Conference on Learning Representations, 2020.
Doyle, A. C. Sherlock holmes collection. https: /twww/wwwwww..
kaggle come sherlock Seerlocc-hoones--- coteesstoolciiionn
/tw.etttts/aaaaaaaaaaaaa r0925/
Axiotis, K., Cohen-Addad, V., Henzinger, M., Jerome, S., Mirrokni, V., Saulpic, D., Woodruff, D. P., and Wunder, sensitivity
Feldman, V. and Zhang, C. and why: discovering the long tail via influence esti- mation. Advances Neural Information Processing Systems, volume 33, pp. 2881-2891, 2020.
What neural networks memorize
M. Data-efficient learning via clustering-bue sampling: Foundation models and beyond. Proceed- ings of the s1 International Conference on Machine Learning. PMLR, 2024.
Fisher, R. On the mathematical foundations theoretical statistics. Philosophical Transactions of the Royal Society of London: Series A, 222:309-368, 1922.
Bernstein, D. S. Matrix Mathematics: Theory, Facts, and Formulas with Application to Linear Systems Theory. Princeton University Press, Princeton, NJ, 2nd edition, 2009. ISBN 978-0691118028.
Hajek, B., Oh, S., and Xu, J. Minimax-optimal inference from partial rankings. arXiv preprint 140.:v:1406.66330 2014. URL attps://arxivv
Bishop, C. Puttern Recognition and Machine Learning. Springer, New York, NY, 2006.
abo/1406. 1406.
5638.
Bommasani, R. al. On the opportunities and risks of foundation models. CoRR, abs/2108.0720 072558, 2021. URL 07258.
Hastings, W. K. Monte carlo sampling methods using
markov chains and their applications. Biometrika, 57 (1):97-109, 1970.
https: /ttp:///aavvvv/iia http://xxxxv......aaa/// abs /200///2111 220. 21758
Horn, R. and Johnson, C. R. Topics Matrix Analysis. Cambridge University Press, 1991.
Borsos, Z., Mutny, M., and Krause, A. Coresets via bilevel optimization for continual learning and streaming. In Advances in Neural Information Processing Systems, vol- ume 33, pp. 14879-14890, 2020.
Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation large language models. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, 2020.
Indyk, P., Mahabadi, S., Mahdian, M., and Mirrokni, V. S. Composable core-sets for diversity and coverage maxi- mization. In Proceedings of the 33rd ACM SIGMOD- SIGACT-SIGART/ TeARRTsyyppan symposium on Principles of database systems, pp. 100-108, 2014.
Chen, Y., Welling, M., and Smola, A. Super-samples from kernel herding. arXiv preprint arXiv: 1203. 2012.
Chitta, K., Álvarez, J. M., Haussmann, E., and Fardet, E. Training data subset search with ensemble active learning. IEEE Transactions on Intelligent Transportation Systems, 23(9):14741-14722 2021.
Karnin, Z. and Liberty, E. Discrepancy, coreset, and sketches in machine learning. In Conference on Learning Theory, pp. 1975-1993. PMLR, 2019.
Coleman, B. and Shrivastava, A. Sub-linear race sketches for approximate kernel density estimation on streaming data. In Proceedings of The Web Conference 2020, WWW '20, pp. 1739-1749, New York, NY, USA, 2020. Associa- tion for Computing Machinery, ISBN 9781450370233. doi: 10. 00011445/336642333388222 org /tg/10. 14111145/3330 00555/3366633333 3380244,
Karpathy, A. char-mnn. https: https://gittbb.comm
karpathy hta//chaarrrraa 2015.
Lattimore, and Szepesvari, C. Bandit Algorithms. Cam- bridge University Press, 2019.
URL https: / //doi.
Lee, A., Miranda, B., and Koyejo, S. Beyond scale: The diversity coefficient as a data quality metric demonstrates lims are pre-trained on formally diverse data. arXiv preprint arXiv:2306. 13840, 2023.
Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Selec- tion via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020.
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison- Burch, C., and Carlini, N. Deduplicating train- ing data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 8424- 8445, 2022.
Das, N., Chakraborty, S., Pacchiano, A., and Chowdhury, S. R. Active preference optimization for sample efficient RLHE. CoRR, abs/2402.105000, 2024. URL https: //arxiv. hww.vorg/aabo. 2202. 10500.
10

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Liu, P., Shi, C., and Sun, W. W. Dual active learning for reinforcement learning from human feedback. CORR, abs/2410.02504, 2024. URL abs/2410. 02504.
Pukelsheim, Optimal Design of Experiments, volume 50 of Classics in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2006. ISBN 0898716047.
Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-ar parameter- efficient fine-tuning methods. https: /etg//ggttu/// ccm/hugge wiggiingfan face/ peft, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models supervised multitask learners. OpenAl Technical Report, 2019.
batteer-Langgggg--mddeaaaaaaa sabel__ee atnelll_are_ussuprrrieedded httpperrviieed_uuitttte learners.pde
Meding, R., Buschtoff, L. M. S., Geirhos, R., and Wich- mann, F. A. Trivial or impossible diie-tocchhoous data diffi- culty makes model differences (on imagenet and beyond). arXiv preprint arXiv:21 2021.
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, C., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36, 2023.
Mikolov, T., Chen, K., Corrado, G. S., and Dean, J. Efficient estimation of word representations in vector space. In International Conference on Learning Representations, 2013. URL https: / https///.a.....tcsssessseeeee www.mnticsccolll....... org Ttg/CorpusiIn 5959482.
Riquelme, C., Tucker, G., and Snock,
Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling, In International Conference Learning Representations, 2018. URL https: httpa::://pppeeeeeeeeeee net /ort/ffrmm i id=SyYe6k-CM,
Mindermann, S., Brauner, J., Razzak, M., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A., Morisot, A., Farquhar, S., et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In In- ternational Conference on Machine Learning, pp. 15630- 15649. PMLR, 2022.
Sachdeva, N., Wu, C.-J,, and McAuley, J. SVP-CF: Se- lection via proxy for collaborative filtering data. arXiv preprint arXiv:2
Muenchigoff, M., Rush, A. M., Barak, B., Seao, T. L.., Piktus, T., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models. arXiv preprint arXiv:2305. 10623, 2023.
2021.
Sachdeva, N., Coleman, B., Kang, W.-C., Ni, J., Hong, L., Chi, E. H., Caverlee, J., and Cheng, D. Z. How to train data-efficient Ilms. arXiv preprint arXiv:2402.09668, 2024.
Mukherjee, S., Lalitha, A., Kalantari, K., Deshmukh, A., Liu, G., Ma, Y., and Kveton, B. Optimal design for human preference elicitation. Advances in Neural Information Processing Systems 37, 2024.
Scheid, A., Boursier, E., Durmus, A., Jordan, M., Menard, P., Moulines, E., and Valko, M. Optimal design for reward modeling in RLHF. CoRR, abs/2410. 2024. URL https: /tapp:://axxi....// wwwpwv.wrrg/bb///10..111
Nemhauser, G. L., Wolsey, L. A., and Fisher, M. L. An analysis of approximations for maximizing submodular functions I. Mathematical Programming, 14(1): 265-294, 1978.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. Beyond neural scaling laws: Beating power law scaling via data pruning. In Advances in Neural Information Processing Systems 35, 2022.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, 2022.
Stufken, J. and Yang, M. Optimal designs for generalized linear models. In Design and Analysis of Experiments, pp. 137-164. John Wiley & Sons, 2012.
Thekumparampil. K., Hiranandani. G., Kalantari, K., Sabach, S., and Kveton, B. Comparing few to rank many: Active human preference learning using random- ized Frank Wolfe. CORR, abx/2412.19999 2024. URL https: /tlss///axxivi wttp://ww...rr//aa///2211113339.
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning data diet: Finding important examples early in training. In Advances in Neural Information Processing Systems, volume 34, pp. 2960-2971, 2021.
Tirmala, K., Simig, D., Aghajanyan, A., and Morcos, A. S. D4: Improving Im pre-training via docu- ment de-uuplicttion and diversification. arXiv preprint arXiv:2308, /2284, 2023.
Phillips, M. Coresets and sketches. In Handbook of discrete and computational geometry, pp. 1269-1288. Chapman and Hall/CRC, 2017.

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Tukan, M., Baykal, C., Feldman, D., and Rus, D. On core- sets for support vector machines. Theoretical Computer Science, 890:171-191, 2021.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, W., Lester, B., Du, N., Dai, A., and Le, Q. Finetuned language models zero-shot learners. In Proceedings of the 10th International Conference Learning Representations, 2022.
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv: /91 1.00359, 2019.
Wolf, T., Debut, L., Sanh, V., Chaumond, J.. Delangue, C.,, Moi, A., Cistac, Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L.,, Gugger, S., Drame, M., Lhoest, Q. and Rush, A. M. Huggingface's 's transformers: State-of-the the-art art natural language processing, 2020. URL https: / arxiv. org/ abs/ 1910..99... 03771.
Xu, P., Wen, Z., Zhao, H., and Gu, Q. Neural contextual bandits with deep representation and shallow exploration. In International Conference on Learning Representations (ICLR), 2022. URL https:/ /ttps///ppeeeveieeeeeeeeeee net./ (errum??d=BbbYWHFFPP
Poster.
Zhang, W. Zhou, D., Li, L., and Gu, Q. Neural thomp- son sampling. In International Conference on Learning Representation (ICLR), 20.1.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z.,, Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J., and Stoica, I. Judging LLM--s-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36, 2023.
Zhou, D., Li, L., and Gu, Q. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning, pp. 11492-1 11022. PMLR, 2020.
Zhu, B., Jiao, J., and Jordan, M. Principled reinforce- ment learning with human feedback from pairwise or K-wise comparisons. CoRR, abs/2301.1127771 2023. URL https:
230taaas////310 11270.
12

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
A. Related Works
Algorithm 3 Inverse Propensity Sampling (IPS) via Kernel Density Estimation (KDE) (Sachdeva et al., 2021)
1: Dataset D {I1, 22, ..... N } of embeddings, sample size k. kernel with corresponding LSH family H (Coleman Shrivastava, 2020), hash range B, rows R. random seed
2: Ensure subset of 'D of size k sampled with probability p (see line 14).
3: Initialize KDE sketch S <- O
4: Generate It independent hash functions h hR from H. with range B and random seed s.
5: for n < to N do
6: for r + to R do
7:
(n,n,(u.)) f Sn, Nr (x.) + 1
8: Initialize a list of scores + [].
9: for n + to N do
10: score 6 0
11: for r 1 to R do
12:
score f- score + Str, h (mm))
13: Append 40grE S. H
14:
Output: Select elements from D with probability p (sampled without replacement).
Coverage-oriented approaches center on ensuring that a training set reflects the entire input distribution as broadly as possible. One common strategy is cluster sampling (Lee et al., 2023), which embeds data points in a metric space (often via learned representations) and selects mutually distant examples to form "coresets" (Phillips, 2017; Tukan et al., 2021). Related methods include prototype- -ased sampling for vision (Sorscher et al., 2022) and deduplication algorithms (Abbas et al., 2023; Lee et al., 2022; Tirmala et al., 2023) that remove near-duplicates or redundancies. More sophisticated procedures-sucl as submodular optimization (Chen et al., 2012; Indyk et al., 2014; Borsos et al., 2020) and discrepancy minimization (Karnin & Liberty, 2019)further refine coverage by balancing representation across diverse data regions.
Quality-based sampling, in contrast, prioritizes weeding out low-value or unhelpful examples. prominent technique is perplexity filtering (Wenzek et al., 2019; Muenchigoff et al., 2023), which prefers samples with higher likelihood under pretrained model, though this can inadvertently discard valuable but rare text. Other approaches compute "uncertainty scores" via ensemble disagreement (Chitta et al., 2021; Meding et al., 2021) or examine whether examples are memorized (Feldman & Zhang, 2020) or unlearnable (Mindermann et al., 2022). The SVP algorithm (Coleman et al., 2020; Sachdeva et al., 2021) estimates each sample's 's importance by its validation-lose variance, while EL2N scores (Paul et al., 2021) track a model's 's difficulty in learning particular data points. These methods fit into a "score-and-sample framework (Hastings, 1970), where the final selection depends on the magnitude of each item's quality score.
For more detailed deseription see (Sachdeva et al., 2024). Below we describe the two algorithms proposed in (Sachdeva et al., 2024) and used as benchmarks in Section 5.
ASK-LLM: In ASK-LLM (Sachdeva et al., 2024), a proxy LLM is prompted with a potential training example and asked whether the example should be used for training. More specifically, the proxy LLM is provided the training example followed by the prompt "Does the previous paragraph contain informative signal for fine-tuning large-language model? An informative datapoint should be well-formatted, contain some usable knowledge of the world, and strictly NOT have any harmful, racist, sexist, etc. content. OPTIONS: yes, no". It then takes the softmax probability of the token "yes" as the estimated data-quality score and sorts according score to pick Top data points.
Density sampling: (Sachdeva et al., 2024) assumes access to embeddings from a pre-trained LLM. Given a dataset D it uses a kernel k(x,, y), to estimate the density using the following score.
score(y)) = Σ kx(a,)),
ZED
where À is a smoothing parameter and controls the scale of the data points' effects. Density Sampling then uses Inverse propensity sampling (IPS) to select items proportional to their re- re-weighted and normalized inverse score. The algorithm as provided in (Sachdeva et al., 2024) is summarized below.
13

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Clustering Based Sensitivity Sampling: (Axiotis et al., 2024) The method uses k-means clustering and sensitivity sampling using the embedding representation of the data with respect which the model loss is measured and ensures that the sampled elements' average loss corresponds to the average loss of the whole dataset. The algorithm as presented in (Axiotis et al., 2024) is summarized below.
Algorithm 4 Clustering Based Sensitivity Sampling (D, k, C) (Axiotis et al., 2024)
1: Input: a dataset D partitioned into clusters (Ct...., C₂) with centers ci...., cj. and a k-tuple of parameters .1....,Ag
2: for e C₁ do
3: Define ((e) == f(ci) and v(e) lle c/1
[(e)+A 4: s (2 + 23/3 For C; define == Σ, A, E.ED f(x) and w(e) = = ppp
5: Compute a sample S of s points, picked independently following the distribution Pe.
6: Output: the set S with weights w.
B. Gradient and Hessian of the Loss
Proposition B.1. Consider the Loss function defined in (4) and suppose assumption 4.1 holds. Then the gradient and Hessian of Ls are respectively given by
1 Σ Σ
VLs(O)
vec Xis @ (pl [x,j; 1(yi)))
T. BES JE(MG)
L=Ls(0)-
1 - VES JE[M,]
Σ
diag(p( |xi,; 0)) p( (xi.;()p' (xis; () 60 X
Proof. Recall that the loss function is given by
Cs(0) = Σ
log P( = f|x,,: 0)5(v)) =
iES JE[M.] CE[L)
exp ((0Tx1))
- log
6(y6.5 = =))
TE iES JE[M.]
(E[L]
Σ exp ((0Tx)))e))
Now the loss can be re-written as
L
Cs(9) =
-1 Σ
@ Di.j
log 2 exp((ffxx xm.)
n
iES jElM,
1=1
Now note that
0 TT
x1,j 8(yk, = =)xx,)
00, U2.3
and that,
i
L
E/. exp(0Xixx,) x 8(y1.5 E)xis
log
002
Ele exp xi.i)
#=1
p(yi, E' x' ()8(Ues E)x1,5
8=1
p(vis E(xios θ )xxi
14

---

FisherSFT: Data- Supervised Fine-Tuning of Language Models Using Information Gain
Combining both get
0 -1
(s(
-
Σ iES JE[M]
(o(nn. = plus.s = E|xi,j 0) Xis
00
Therefore the gradient of the loss (() with respect to θ is given by
VLs(O)
- Σ Σ ni
vec (x.. @ (1(y))) - ply.. x (x)))(0)))
(11)
iES JE[M,]
where (vis) E &R is a one-hot vector with the yi,--th entry as and @ is the Kronecker product.
Next we compute the Hessian. Note that
0² -1 Σ Cs(0)
Σ
(biis = p(ui.s = E|x,,s: 0)) X
00,0g
n 00;
iES jE(M,
Σ
p ply..j = e|xis; 3) x3.
①
n JE(M.]
Σ
p(u.. = (xx,;:)) () (6((= - -p(u.. = f(xx;;:) )));; x
n EES ME(M
and therefore, the Hessian of the loss is given by
L²L)())= = - Σ Σ
(diag((( |xi,; 9)) pl |x,y; O)p( (x,;; () @ Xx
(12)
iES JE[M,]
Now note that pl |xi,ji 2 e- where supe.aj Therefore
x S a.
V²Ls = -
Σ
- - - @ xxixxT
(13)
iES JE[M,]
Assume (e-2a1,, - --44111 2 I for some
> 0. Then we have
(2(s = 1 Σ Σ
714 & x
(14)
iES JE
C. Proof of Error Bound
C.1. Proof of Lemma 4.4
Lemma 4,4. Suppose that Assumption 4.2 holds. Let S be the set of n sentences generated by FisherSFT and (7) be their design matrix. Let M maxie[N] M,. Then
00 2 log (1 3o 3nd
M.
kdM
max N
X
2
log(1 2)
n
Proof. We derive an upper bound on ||x,)|ls,,, where X, E Rdd is feature vector and Σn E Idddd is a design matrix obtained by greedy log-determinant maximization. Let D (xi, : [N].j E [M]]] be a dataset of N data points such
15

---

FisherSFT: Data-Efffcient Supervised Fine-Tuning of Language Models Using Information Gain
that و| I 1. Let I, E [N] be the index of the I-th chosen feature vector and S ((6)/=1 be the first chosen feature vectors. For simplicity use Es and E,,, interchangeably. Let
M; ΣΣ
x
iES
where do 0 is constant that guarantees that Eo is well defined.
The t-th feature vector is chosen as
Mr Σ + Σ
4. == argmax log det xe[N],51-1
(15)
j=1
Lemma C.1. For any E [N] and E [n].
Mi
Mi S
1.
X,
b,3 2, xz.j
j=1
j=1
Proof. Define the matrix
X
X
X
X,
so that each X, is a column of X. Then can write
Mr
Σ
X x
=
XXT.
Hence want to find the inverse of
+
Using Cherrann-Morrson-Woodde identity, which states that for an invertible matrix and any matrices U, C, V of compatible dimensions (with C also invertible), one has
(A (A+UCV)-L = A- A
In our case, we set
A Σ,1 U=X, X, C -=IMe V=XT.
where IM, is the M, x M, identity matrix. Then
A+UCV = - XT = -
By applying the identity, we get
(E + 1-1
5 X (TM. + X" STE11 x 5- "-1
which implies
Σイ
Therefore we get v -E < u -T,,00 for any vector E Rd which concludes the proof.
Lemma C.1 implies that
Mi
Mi
X
2n
Xi.j <
X
D
Xi.j
1=1
1=1 ==1
16

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
holds for any i [N]. This allows us attribute the quality of the solution to individual greedy steps in (15).
scope of the maximization was E [N]] the inequality شززك - Xa.i S Σn=1 x I 1 X would hold for
any E [NT]. Since the scope is [N] St-1, we make Assumption 4.2.
We also the following logarithmic transformation.
Lemma C.2. For any i E [N] and E [n]]
00 log(1 (1 + +x, - 1 (xis)
x 2 وー Xij
log(1 + )
Proof. We start with an upper bound on Σ/=11 T x Xz.j. By Weyl's inequalities, we have 1-j-
(1 A S 27 (oğTa) =00².
Therefore, under the assumption that ||x;j 112 S 1, we have E/e1 xT X S 0 M. Now note that for any E [0, u).
log(1 + +x) <
0E log(1 x) log(1 + x) log(1 u)
max
log(1 (1 + x)
log(1 + x)
Finally, we set x = Ex= xT Xt, and u = oo 2 2M, and get our claim.
Assumption C.3.
There exists constant 2 1 such that
Mr.
log det +
1-- 1/2 1-1 x
5-112)
<llog det +
- 1/2
X1.
X x,.
T
E-1/2
holds for any i E S,-1 and e [n].
Now we apply Assumption C.3 and Lemma C.2, use the telescoping property of the sum, and M max,e[N] M, get
17

---

FisherSFT: Data- Supervised Fine-Tuning of Language Models Using Information Gain
M
(a)
M
2
log [1 + xx - (()
log + % 2)
j=1
t=1 j=1
Mi
0
log det (La + 5T xi. x -/-
log(1 00 2)
==1
00 M
n
log La +
1
Σ
M,
5.
1/1
Xi x 1-,13
log(1 + +00 2)
Mi
j=1
0 2
Σ log det +
Σ
1/2 T 1/2
x-1
Xi,
log(1 + 0
5=1
00 2 M
M4.
log det +
5-1/2 1/2
Xl.,) x
[c.j
T 5-12
log(1 + 00"
==1
Mr.
(c)= ROn MM
log det 11 +
Σ
log det(Ec,1)
log(1 00"
==1
KOy" 2 M
(f)
log det(E.) - llgg del((e))))
log(1 + 00²)
KOO 2
(log det(En) log det (Eo()
log(1 + % 3)
KO 2
(log del - d dllog(on))
log(1 + 0
Here (a) follows by Lemma C.2, (b) follows by log(1 (1+xxx++xxx (x)xxx S log det (I ( x1/2xx" xxA/2xxTAA/3 and (c) follows by Jensen's 'ss inequality. Further (d) follows by Assumption 4.2, (e) follows by log det (T A-12//A--1A// log det 4 + B) log (c)( (f) follows by the definition of E, and finally (g) follows by telescoping.
Furthermore,
Mi.
{ f tr (Em) dlog
log det(Ex)) s d dlog
1+ Σ tr 1=1 I <dlog
X1. x
M
nM
= dog
ogla +
og +
d
tm jml
Finally, we combine all claims and get
M,
0 2 2M log(1 + -2
M
0 log
1 +
vo 3
d
2nM
kd
max
2 Xi.
<
log del( -tr(
Xi x(,) d dlog(on))
[
log (1 + +000
7
j=1
This concludes the proof.
18

---

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
C.2. Proof of Lemma 4.5
Lemma 4.5. Suppose that Assumption 4.1 holds and θ is the MLE in (5) such that Ô B. Then there exists some a < 1 such that
(s( - Ls( - -VVLs(0"), 0*)
e
2a
Σ
|00 8i 1111ss51 1is
2
L
1=1
Proof. Using Taylor's expansion
(s(()) + 00 -0) Ls
The Hessian is given by
M,
V²Cs(0)
Σ
Σ iES
diag(Pi.)) Pr.sPl.)) @ (x,,)x1
71
where Pi. = p(: |xi.j 0). Now using Claim 1 from (Hajek al., 2014) have
1
2s (diag(Pa.)) - Pa.sP
T, IL
11
T
where max,, y X I 1. Therefore have
M. diag(P Pi.pPes) @ (x1,)x]
1
V2Cs(()
Ti
iES
M,
2n
e 20
IXLL
11
& (x))x())
L
L
iES
j=1
Now consider (Ó - or, V²Ls( 0*). By defining AO = Ô O³, we can express this as follows:
M. iES
(Viiag(p..)AAa.. ) (xxs) wi
00)
diag(p.))AO..t
R.K'
(AO A Pig.
AO.. (xx,)
Tr V diag(pi.,) (xم,) v diag(pr.))
12
FES j=1
- -Tr(pT, AO 10x,,xx1 x AO A0PI.)
Tr
T x
A0
(dagg(p..) - pi.p...) AOTX xis)
IES j=1
> Σ Mr
(e
(e-2a
222
Tr xT A0
IIxL
I 11 AOTx,.))
iES 1=
19

---

FisherSFT: Data-Efffcient Supervised Fine-Tuning of Language Models Using Information Gain
Now observe that A01 0 follows from Assumption 4.1 and solution Ô. Therefore,
20x
(é - --.'),,((
6*)
Tr(AO Tr(AOXx,,xX xx,x.,
nL
iES
2x
Tr MIAATESAG
2a
Tr TTAATV VESV Es
e2aa
| }
2a
2
Σ 100 02115,s
1.2
C.3. Proof of Lemma 4.6
Lemma 4.6. With probability 1 1- 6, the gradient of the loss satisfies
sup L]
||ecs((0)) 15 S
where C > 0 is some global constant.
Proof.
First observe that [VLLs = n VeL VeL where Es Ess. Next recall that the gradient is given by
VLs(0) = Σ Σ
vec (x.s @ (pl 1x:,; - -(),)))
iES jE(MM]
Therefore
VeLs(9)
- Σ
x x.s = ((xi,;; I(yiis = E)).
TI iES JE(M,
Define X E xd as the matrix whose rows are X E S.j E [Ma), and v be the n.M, dimensional vector whose entries are p(y..j f|x; O) 1(y.. f), i.e.
VH exp(O. x
I(yr. e).
Et=res exp((
Note that E[V*] = and [V]]
2, which implies V is 4 nub-Gaussian. Therefore
VeLs(0) 1 n² (V') XE x T Vd <
1 |V
Using Bernstein's 's inequality, with probability ō, for some constant C > 0
[V)Cs c (d log log(1/6))
Taking union bound over all e E [L] we have with probability - 8, for some constant C
|[>[_(())|E_ c (d + log(L/)))
sup EE[L]
which implies we have with probability 1 8, for some constant C' > 0
sup []
VeLs( S /dd+log(L/))
20


Reviews:
[{"review_id": "NrOGT22gOu", "reviewer": "Reviewer_v3ni", "summary": "This paper casts the problem of data pruning for fine-tuning an LLM via SFT as one of optimal experiment design. In optimal experiment design, one wants to \"probe\" a system in a way that the combination of the probes you use are most effective in allowing you to extract the designed information from the system. By recognizing that the structure of modern LLMs has a penultimate layer that feeds into a linear layer that generates token logits, they treat the outputs of that layer as the relevant features for an input token. At that point, they invoke the idea of the Fisher information matrix of the loss to quantify quality. But this is intractable, and so by making a leap-of-faith technical assumption, they are able to get a lower bound on it by looking at something much much simpler --- just the sum of outer-products of the feature embeddings. Since maximizing log-det for such a sum has very nice submodularity properties, they are able to turn this into a pretty efficient greedy algorithm to select the \"most informative\" pieces of text to fine-tune on.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wanted to thank the reviewer for positive evaluation and clearly summarizing the main technical contributions of our work. Our rebuttal is below. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.\n\n### **LLM as a Judge**\n\nWe completely reworked the LLM-as-a-judge evaluation. The new prompt is\n\n> You are a judge of Shakespeare text. \\\n><tag1>text1</tag1> \\\n><tag2>text2</tag2> \\\n> Respond 2 if the text inside <tag2> is more fluent Shakespeare text than the text inside <tag1>. Respond 1 otherwise.\n\nThe prompt got simplified, does not name the methods, and targets our perceived benefit (improved language). We use a state-of-the-art LLM GPT-4o to judge. The text generated by the compared methods is randomized: one randomly-chosen method replaces text1 and the other text2. We tested the LLM judge and it chooses the first position with probability 0.54, which is slightly higher than 0.5 for a position-unbiased judge.\n\nIn addition to improving evaluation, we added a new baseline Ask-LLM (Sachdeva et al., 2024). See Appendix A of our paper for more details on this method.\n\nWe report the win rates of TokenOD on the Shakespeare dataset, as a function of sample size $n$, below:\n\n| TokenOD versus  | 100  |  200 |  500 | 1000 | 2000 | 5000 |\n|-----------------|------|------|------|------|------|------|\n| Uniform         | 0.80 | 0.56 | 0.60 | 0.59 | 0.64 | 0.74 |\n| DensitySampling | 0.61 | 0.66 | 0.68 | 0.62 | 0.54 | 0.84 |\n| Ask-LLM         | 0.59 | 0.52 | 0.68 | 0.59 | 0.68 | 0.74 |\n\nWe observe that TokenOD consistently outperforms all baselines.\n\nWe also added a new experiment on the Sherlock dataset from [Sherlock Holmes Next Word Prediction Corpus](https://www.kaggle.com/datasets/muhammadbilalhaneef/sherlock-holmes-next-word-prediction-corpus). The evaluation protocol is the same as in the Shakespeare dataset, except that \"Shakespeare\" is replaced with \"Sherlock\". The win rates of TokenOD on the Sherlock dataset are:\n\n| TokenOD versus  | 100  |  200 |  500 | 1000 | 2000 | 5000 |\n|-----------------|------|------|------|------|------|------|\n| Uniform         | 0.84 | 0.81 | 0.75 | 0.64 | 0.65 | 0.93 |\n| DensitySampling | 0.68 | 0.75 | 0.69 | 0.74 | 0.65 | 0.88 |\n| Ask-LLM         | 0.74 | 0.58 | 0.65 | 0.60 | 0.61 | 0.89 |\n\nWe observe again that TokenOD consistently outperforms all baselines.\n\n### **How Is the Fine-Tuned LLM Used in Data Selection**\n\nOur data selection procedure depends on the LLM through embeddings $x_{i, j}$. They arise in both the log-likelihood in (2) and algorithm TokenOD. Simply put, TokenOD selects diverse sentences, which cover the embeddings $x_{i, j}$ more uniformly. As you pointed out, we reduce the original $d L \\times d L$ Fisher information matrix into a $d \\times d$ matrix, which can be optimized efficiently. While this neglects token-level prediction accuracy, it incorporates the LLM through the embeddings. A more direct optimization of the original matrix is an interesting direction that would require addressing the computational challenge.\n\n### **Different Weaker LLM for Optimal Design**\n\nWe agree that this is feasible when the embeddings of the weaker and stronger LLMs can be related. We have not done this in our work because we wanted to start with a simpler problem, the same LLM is used for both the optimal design and fine-tuning. We will discuss this option in the paper.\n\n### **Missing Reference**\n\nThank you for the reference. We will include it in the next version of the paper.", "claims_and_evidence": "The fact that these simplifying assumptions lead to an efficient algorithm is undisputed. However, when one steps back, the radical boldness of what they are claiming is shocking and the evidence for that is weaker. Why? Because their method seems to never take into account in any way whether the LLM is already any good at producing the relevant sentences. Extremely low perplexity sentences and very high perplexity sentences are overtly treated the same!!! \n\nBut the proof of the pudding is in the eating, and they do experiments. However, there is a question I have regarding the use of LLM-as-judge here.", "methods_and_evaluation_criteria": "At one level, this all feels very reasonable. They're hand-picked example is striking, but they didn't tell us how that was picked so we can't trust that as an evaluation.\n\nBut there is a bigger question: their LLM-as-judge prompt literally labels one of the methods \"Optimal Design\" and asks the LLM to compare it against the other. Unless they repeated this evaluation with the labels flipped to check consistency, this violates one of the most basic principles of using any kind of evaluation --- you never prejudge the streams by giving them names that have any subjective valence or positivity/negativity. It's like asking a person whether they prefer the described actions of \"Hero\" vs \"Villain.\"", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes, see above.", "relation_to_broader_scientific_literature": "This is pretty decent", "essential_references_not_discussed": "I was surprised to see the following paper not cited:\n\nhttps://proceedings.neurips.cc/paper_files/paper/2022/hash/7b75da9b61eda40fa35453ee5d077df6-Abstract-Conference.html\nBeyond neural scaling laws: beating power law scaling via data pruning\n\nThis takes a nuanced perspective on easy vs hard examples. \n\nThere is a sense in which the proposed approach in this paper is fundamentally about a different axis: picking a collectively loud set of examples vis-a-vis LLM feature space.", "other_strengths_and_weaknesses": "I understand that one can always ask for more examples, but I am very surprised that the authors didn't anticipate the natural question: what if you use an LLM other than the one being fine-tuned to do the design? This transferability might be key in many practical scenarios where one might be using a black-box-API for doing the fine-tuning for a very strong model but have access to a weaker open-weights model to help with the choice of examples.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Please report the results of what happens in the LLM-as-judge comparisons when you flip the labels? As well as when you call the two things \"method A\" and \"method B\"...", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YVNCMv4cAP", "reviewer": "Reviewer_5LdG", "summary": "The paper studies the problem of data selection for the training (in particular, fine-tuning) of autoregressive language model. The data selection regime is based on an efficient estimation of a lower bound of the empirical fisher matrix. This estimation is then combined with a greedy optimal design algorithm for selecting data point for training: In particular, for each data point, the algorithm first computes the sum of the outer product the data point's embedding across all token positions, then the algorithm picks the data point that increases the volume of the design matrix most, and finally include this data point into the design matrix.\nThen the authors further propose an accelerated version of the algorithm that avoids enumeration over all data points in the dataset.\n\nThe author then performed the evaluation on GPT2: Compared with past approaches that use the embedding of the sentences, the proposed approach shows a smaller error for all sizes of training dataset considered.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wanted to thank the reviewer for positive evaluation, and recognizing the practicality of our solution as well as the importance of the solved problem. Our rebuttal is below. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.\n\n### **Active Learning Using the Last-Layer Embedding**\n\nModel gradients are often used in active learning and bandit exploration ([Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://arxiv.org/pdf/1906.03671), [Neural Contextual Bandits with UCB-based Exploration](https://arxiv.org/pdf/1911.04462)). We particularly note that such methods are computationally intensive since the parameter space of modern neural networks is huge in comparison to the last-layer embedding. Our method uses the last-layer embedding as a featurizer and is similar in spirit to [Neural Contextual Bandits with Deep Representation and Shallow Exploration](https://arxiv.org/pdf/2012.01780). This approach is known to be robust and has a much lower empirical regret than other uncertainty representation techniques ([Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://arxiv.org/abs/1802.09127)).\n\n### **Empirical Convergence Rate of TokenOD**\n\nWe plot the error rate of TokenOD on the synthetic problem in Section 5.1 at [anonymized link](https://imgur.com/a/4HWu2oP). Specifically, we take the logarithm of the error rate and plot it as a function of the logarithm of the sample size. We observe a slope of $-0.3$, which means that the error rate is $O(n^p)$. We believe that this is close enough to the expected $p = -0.5$, especially since other factors may have played a role at our sample sizes.\n\n### **More Extensive Empirical Evaluation**\n\nOur computational resources are limited and therefore we did not experiment beyond GPT-2. To partially address your concern, we further expanded our evaluation as follows. In addition to improving the LLM-as-a-judge evaluation, which other reviewers asked for, we added a new baseline Ask-LLM (Sachdeva et al., 2024). See Appendix A of our paper for more details on this method.\n\nWe report the win rates of TokenOD on the Shakespeare dataset, as a function of sample size $n$, below:\n\n| TokenOD versus  | 100  |  200 |  500 | 1000 | 2000 | 5000 |\n|-----------------|------|------|------|------|------|------|\n| Uniform         | 0.80 | 0.56 | 0.60 | 0.59 | 0.64 | 0.74 |\n| DensitySampling | 0.61 | 0.66 | 0.68 | 0.62 | 0.54 | 0.84 |\n| Ask-LLM         | 0.59 | 0.52 | 0.68 | 0.59 | 0.68 | 0.74 |\n\nWe observe that TokenOD consistently outperforms all baselines.\n\nWe also added a new experiment on the Sherlock dataset from [Sherlock Holmes Next Word Prediction Corpus](https://www.kaggle.com/datasets/muhammadbilalhaneef/sherlock-holmes-next-word-prediction-corpus). The evaluation protocol is the same as in the Shakespeare dataset, except that \"Shakespeare\" is replaced with \"Sherlock\". The win rates of TokenOD on the Sherlock dataset are:\n\n| TokenOD versus  | 100  |  200 |  500 | 1000 | 2000 | 5000 |\n|-----------------|------|------|------|------|------|------|\n| Uniform         | 0.84 | 0.81 | 0.75 | 0.64 | 0.65 | 0.93 |\n| DensitySampling | 0.68 | 0.75 | 0.69 | 0.74 | 0.65 | 0.88 |\n| Ask-LLM         | 0.74 | 0.58 | 0.65 | 0.60 | 0.61 | 0.89 |\n\nWe observe again that TokenOD consistently outperforms all baselines.\n\n### **Bias Towards Longer Sentences**\n\nTokenOD is biased towards selecting longer sentences. This bias naturally arises because each token in a sentence is a training point in supervised fine-tuning, for the conditional probability of token $y_{i, j}$ given the history embedding $x_{i, j}$. In a sense, longer sentences can contribute more to information gain. However, the selection of the sentence depends on its total information gain in comparison to other available sentences. We also observe a similar bias towards longer sentences in the [GPT-2 code on Hugging Face](https://huggingface.co/transformers/v4.3.3/_modules/transformers/models/gpt2/modeling_gpt2.html#:~:text=loss_fct%20%3D%20CrossEntropyLoss,1)", "claims_and_evidence": "The paper claims that the proposed approach is efficient and a principled approach for dataset selection, and indeed the proposed algorithm and the empirical results provided confirm the claims.", "methods_and_evaluation_criteria": "The proposed method intuitively makes a lot of sense and the evaluation setting: Supervised fine-tuning is a very practical and common problem/setting.\n\nHowever, the fact that the algorithm does not rely on any gradient information is a little bit confusing. I did not check the derivation in detail and I believe in the correctness, but intuitively, you can use a quantity that only relies on the last layer embedding vectors as proxy for the fisher matrix, which depends on the gradient, is suprising. \n\nThe model considered is a little bit too small scale if I have to be picky: It would be better if the authors can demonstrate that the method works for e.g. LoRA fine-tuning of a 1M/3M model.", "theoretical_claims": "The paper provides a validation for Algorithm. 2 and the proposed lower bound of the fisher matrix in Section 3. and theoretical guarantee for the prediction error under the proposed data selection regime in Section 4. I did not check the proof details.\n\nHowever it would be nice if the author could provide an empirical demonstration of the 1 / sqrt(n) convergence rate in the experiments.", "experimental_designs_or_analyses": "The experiment design and analyses looks sensible to me.", "relation_to_broader_scientific_literature": "The problem studied is of great interest, e.g. supervised fine-tuning is used in lots of RLHF pipeline and in LLM adaptation. The proposed method can be beneficial for many LLM practioner.", "essential_references_not_discussed": "I am not familiar with the literature. However, considering that the model only needs the pre read-out layer embedding of the sentence, I don't think there are lots of methods along this line, since most of them would require e.g. gradient information or an external model.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "In line 112, input of Algorithm 1, the x denotes the sentence embedding rather than the \"Sentences\" themselves right? \n\nEq.4 should correspond to empirical fisher rather than the fisher matrix? Since there is no expectation with respect to all y values.", "questions_for_authors": "Would the data selection algorithm be biased by the length of the training sample? In line 5 of algorithm 1, the summation would contribute more as M_i , i.e. the number of tokens, increases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZyckXhRF79", "reviewer": "Reviewer_M5jD", "summary": "The paper presents a statistical approach to enhancing supervised fine-tuning efficiency through strategic training example selection, which stands in contrast to conventional random sampling methods. By conceptualizing the example selection problem as an optimal design task that maximizes the Hessian of the LLM's log-likelihood, the authors establish a theoretical framework that prioritizes examples with greater information value. The technical innovation lies in their efficient approximation of the LLM at the last layer using multinomial logistic regression models, which enables the development of TokenOD—a greedy algorithm that exploits log determinant submodularity to select sentences containing jointly informative tokens.\n\nTheir rigorous theoretical analysis demonstrates that the prediction error decreases at rate O(dL/√n), while extensive empirical evaluation across synthetic data, word embeddings, and GPT-2 fine-tuning on Shakespeare corpus consistently shows TokenOD outperforming baseline methods including uniform sampling and density-based approaches. The practical significance of this contribution becomes evident in the marked improvement in sample efficiency, with TokenOD frequently achieving with 1,000 examples what baseline methods require 2,000 examples to accomplish, thereby making the fine-tuning process substantially more resource-efficient while maintaining or improving the quality of generated text as evaluated by larger language models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wanted to thank the reviewer for positive evaluation, recognizing our contributions, and bringing up the prompt bias issue to our attention. Our rebuttal is below. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.\n\n### **LLM as a Judge**\n\nWe completely reworked the LLM-as-a-judge evaluation. The new prompt is\n\n> You are a judge of Shakespeare text. \\\n><tag1>text1</tag1> \\\n><tag2>text2</tag2> \\\n> Respond 2 if the text inside <tag2> is more fluent Shakespeare text than the text inside <tag1>. Respond 1 otherwise.\n\nThe prompt got simplified, does not name the methods, and targets our perceived benefit (improved language). We use a state-of-the-art LLM GPT-4o to judge. The text generated by the compared methods is randomized: one randomly-chosen method replaces text1 and the other text2. We tested the LLM judge and it chooses the first position with probability 0.54, which is slightly higher than 0.5 for a position-unbiased judge.\n\nIn addition to improving evaluation, we added a new baseline Ask-LLM (Sachdeva et al., 2024). See Appendix A of our paper for more details on this method.\n\nWe report the win rates of TokenOD on the Shakespeare dataset, as a function of sample size $n$, below:\n\n| TokenOD versus  | 100  |  200 |  500 | 1000 | 2000 | 5000 |\n|-----------------|------|------|------|------|------|------|\n| Uniform         | 0.80 | 0.56 | 0.60 | 0.59 | 0.64 | 0.74 |\n| DensitySampling | 0.61 | 0.66 | 0.68 | 0.62 | 0.54 | 0.84 |\n| Ask-LLM         | 0.59 | 0.52 | 0.68 | 0.59 | 0.68 | 0.74 |\n\nWe observe that TokenOD consistently outperforms all baselines.\n\nWe also added a new experiment on the Sherlock dataset from [Sherlock Holmes Next Word Prediction Corpus](https://www.kaggle.com/datasets/muhammadbilalhaneef/sherlock-holmes-next-word-prediction-corpus). The evaluation protocol is the same as in the Shakespeare dataset, except that \"Shakespeare\" is replaced with \"Sherlock\". The win rates of TokenOD on the Sherlock dataset are:\n\n| TokenOD versus  | 100  |  200 |  500 | 1000 | 2000 | 5000 |\n|-----------------|------|------|------|------|------|------|\n| Uniform         | 0.84 | 0.81 | 0.75 | 0.64 | 0.65 | 0.93 |\n| DensitySampling | 0.68 | 0.75 | 0.69 | 0.74 | 0.65 | 0.88 |\n| Ask-LLM         | 0.74 | 0.58 | 0.65 | 0.60 | 0.61 | 0.89 |\n\nWe observe again that TokenOD consistently outperforms all baselines.", "claims_and_evidence": "The claims in the paper \"Autoregressive Optimal Design for Language Models\" are generally supported by convincing evidence, though there are areas where the evidence could be strengthened.\n\nThe core claim that TokenOD improves statistical efficiency for supervised fine-tuning is well-supported through both theoretical analysis (Section 4's error bound theorem) and empirical results across multiple experimental settings. The synthetic experiments in Section 5.1 and word embedding experiments in Section 5.2 provide quantitative evidence showing clear improvements in both maximum and mean prediction errors compared to multiple baselines. These results consistently demonstrate TokenOD's superiority across different sample sizes.\n\nThe GPT-2 fine-tuning experiments provide additional evidence through LLM-based evaluation, showing that text generated by models trained on TokenOD-selected examples is preferred 56.5-77% of the time over baseline methods. While this evaluation approach using Claude 3 Sonnet as a judge is reasonable, it does introduce some subjectivity that could be acknowledged as a limitation.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed in \"Autoregressive Optimal Design for Language Models\" are generally appropriate and well-designed for addressing the problem of improving supervised fine-tuning efficiency in LLMs.", "theoretical_claims": "Overall, the theoretical claims and proofs (Lemma 3.1, Theorem 4) are sound and follow established techniques from matrix analysis and statistical learning theory.", "experimental_designs_or_analyses": "- in using LLM-as-a-judge, it would be better practice to hide the names of the two methods and to consider the effects of both orders. I don't think this would have a major effect but should be done in a revision. \"better\" is also ambiguously defined to the LLM", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "- The paper would benefit from including discussion of limitations.", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zkduwxCbxT", "reviewer": "Reviewer_yVhv", "summary": "This paper presents an active learning methods to select the most informative examples in the training samples within a fixed budget, with the idea of consider greedily optimise the fisher information over the Hessian matrix with respect to the LLMs. To solve the issue under high-dimensional and large sample size, an efficient method is proposed by linking the setting to a multinomial logistic regression model. The authors performs some empirical experiments to verify their proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wanted to thank the reviewer for positive evaluation and appreciating that we bring classic ideas from statistical learning theory to LLMs. Our rebuttal is below. We focus on major issues and will incorporate all comments of the reviewer in the next version of our paper. If you have additional concerns, please reach out to us to discuss them.\n\n### **Other Values of $d$ and $L$ in Synthetic Experiments**\n\nWe plot the error curves for the synthetic problem in Section 5.1, and various $d \\in \\\\{5, 10, 20, 30, 40\\\\}$ and $L \\in \\\\{20, 30, 40, 50\\\\}$, at [anonymized link](https://imgur.com/a/9WvjPYu). TokenOD consistently performs better. We also show the trends for $L$ ([anonymized link](https://imgur.com/a/CL19WtK)) and $d$ ([anonymized link](https://imgur.com/a/kNKjio7)) when fixing the number of samples at $1000$.\n\n### **Computational Gains in Fast Implementation of TokenOD**\n\nWe compare the computation times of the fast and slow implementations of TokenOD on the problem in Section 5.3 at [anonymized link](https://imgur.com/a/J3RaNuH). The number of sentences is $5000$ and we plot the computation time for various values of $n$. The fast implementation of TokenOD is about $4$ times faster.\n \n### **More Details on Experimental Setup in Section 5.3**\n\nThe experiment can be described in our notation as follows. $x_{i,j}$ is the transformer embedding at the $j$-th token of the $i$-th sentence in the text corpus and $y_{i, j}$ is the identity of that token. For each compared method, we fine-tune a GPT-2 model on sentences collected by that method. At test time, we prompt the fine-tuned model with a few words from the original text corpus (represented by tokens $y_{i, 1}, y_{i, 2}, \\dots, y_{i, p}$) and the model completes the sentence by generating $y_{i, p + 1}, y_{i, p + 2}, \\dots, y_{i, 1024}$. Finally, we compare the quality of the completed sentences generated by the various models using an LLM as a judge.\n\n### **Fisher Information**\n\nThe Fisher information matrix in TokenOD is an approximation derived in Lemma 3.1. This algebraic form, an outer product of feature vectors, is the true Fisher information matrix for least squares. How is it useful beyond least squares? Simply put, optimization of this matrix leads to choosing feature vectors that cover all directions in training data uniformly. Because of that, it is possible to bound a worst-case prediction error over all feature vectors and be robust.\n\n### **LLM as a Judge**\n\nWe improved the evaluation, including reporting biases. See the rebuttal for [Reviewer M5jD](https://openreview.net/forum?id=e02oLEbehE&noteId=OaYaGwaft7) for details.", "claims_and_evidence": "The claims are in general well supported by evidence, although I am not able to check section 4 (error bound). For experiments, since the initial claim is on high-dimensional and long sequence, can I ask the authors to provide synthetic examples other than L =20 and d = 10? and also it will be beneficial to show if we scale up the length or dimension, the proposed method is consistent better? Although it will be nice to show the difference between the computational gain comparing between algorithm 3.1 and 3.2. Same applies to experiment 4.2. Also, it is good to see experiment 4.3 on real-world data but then the setup is not very clear to me, perhaps it is worthwhile to map with one of the synthetic/semi-synthetic setup (or have this as an additional setup)?", "methods_and_evaluation_criteria": "The general methods seems to be reasonable with some comments regarding experiment setup (see **Claims and Evidence**). Also, using LLMs as judge directly is not preferred and not convincing enough without any comparing with human experts (but this is less of a problem in my opinion).", "theoretical_claims": "I am not able to check the details, but the general logic seems to be reasonable.", "experimental_designs_or_analyses": "Please refers to my comments earlier.", "relation_to_broader_scientific_literature": "This finding is interesting with direct links to classical statistical learning theory concepts (e.g. Fisher information and Bayesian modelling for multivariate Gaussian distribution) and provide an interesting view on selecting best training examples in high-dimensional space.", "essential_references_not_discussed": "I can not comments on this since I am not an expert.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Please see my comments earlier.\n\n## update after rebuttal\n\nI would like to thank all the authors for the further clarifications. I am in favour of accepting this paper, and my score remains unchanged.", "questions_for_authors": "Q 1: Regarding experiments setup and evaluation, detailed in previous comments?\n\nQ2: This method is essentially utilising the Fisher information which is the biggest variance among the training samples. However, this most informative may not equivalent to the best training examples for particular type of problem (if the training samples are not well selected or if the problem is ill-defined). Could you comments on this and consider adding this as a part of the discussion?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a method for selecting the most informative training examples for fine-tuning language models, using an efficient approximation of the Fisher information matrix to optimize the selection process. The reviewers praised the paper for its well-written presentation and interesting method, which demonstrates improved sample efficiency in fine-tuning language models. The reviewers' main criticisms centered around the experimental setup, the need for more details on the method's limitations, and the potential bias of the LLM-as-a-judge evaluation, with some reviewers also questioning the scalability of the approach and the lack of comparison to other methods.

The authors responded by providing additional experimental results, clarifying the technical details of their method, and addressing the concerns about the evaluation protocol. The reviewers indicated that the authors' rebuttal adequately addressed most of the criticisms, particularly those related to the evaluation protocol and the comparison to other methods. Therefore, the reviewers unanimously agreed to accept the paper. We would still recommend that the authors take the reviewers' feedback into account when preparing the camera-ready version.
Decision: Accept (poster)