Quadratic Coreset Selection: Certifying and Reconciling Sequence and Token Mining for Efficient Instruction Tuning
Authors: Ziliang Chen
Yongsen Zheng
Zhao-Rong Lai
Zhanfu Yang
Cuixi Li
Yang Liu
Liang Lin
Venue: NeurIPS 2025 poster ()
Keywords: instruction tuning;principled data selection for LLM
Primary area: deep_learning

Abstract: Instruction-Tuning (IT) was recently found the impressive data efficiency in post-training large language models (LLMs). While the pursuit of efficiency predominantly focuses on sequence-level curation, often overlooking the nuanced impact of critical tokens and the inherent risks of token noise and biases. Drawing inspiration from bi-level coreset selection, our work provides the principled view of the motivation behind selecting instructions' responses. It leads to our approach Quadratic Coreset Selection (QCS) that reconciles sequence-level and token-level influence contributions, deriving more expressive LLMs with established theoretical result. Despite the original QCS framework challenged by prohibitive computation from inverted LLM-scale Hessian matrices, we overcome this barrier by proposing a novel QCS probabilistic variant, which relaxes the original formulation through re-parameterized densities. This innovative solver is efficiently learned using hierarchical policy gradients without requiring back-propagation, achieving provable convergence and certified asymptotic equivalence to the original objective. Our experiments demonstrate QCS's superior sequence-level data efficiency and reveal how strategically leveraging token-level influence elevates the performance ceiling of data-efficient IT. Furthermore, QCS's adaptability is showcased through its successes in regular IT and challenging targeted IT scenarios, particularly in the cases of free-form complex instruction-following and CoT reasoning. They underscore QCS's potential for a wide array of versatile post-training applications.

Quadratic Coreset Selection: Certifying and Reconciling Sequence and Token Mining for Efficient Instruction Tuning
Ziliang Chen', Yongsen Zheng", Zhao-rong Lai", Zhanfu Yang', Cuixi Li", Yang Liu", Liang Lin !Pegg Cheng Laboratory, "Nanyang Technological University, 3 "Jinan University, *Rutgers University, SUNSW, Sun Yat-sen University
Abstract
cestruction-Tuning (IT) was recently found the impressive data efficiency in post- training large language models (LLMs). While the pursuit of efficiency predomi- nantly focuses on sequence-level curation, often overlooking the nuanced impact critical tokens and the inherent risks of token noise and biases. Drawing inspiration from bi-level coreset selection, our work provides the principled view of the moti- vation behind selecting instructions' responses. It leads our approach Quadratic Coreset Selection (QCS) that reconciles sequence-leve and token-level influence contributions, deriving more expressive LLMs with established theoretical result. Despite the original QCS framework challenged by prohibitive computation from inverted LLM-scale Hessian matrices, we overcome this barrier by proposing a novel QCS probabilistic variant, which relaxes the original formulation through re-pa re-aaramteerized densities. This innovative solver is efficiently learned using hier- archical policy gradients without requiring back- poeck--opaggtion, achieving provable convergence and certified asymptotic equivalence the original objective. Our experiments demonstrate QCS's superior sequence-level data efficiency and reveal how strategically leveraging token-level influence elevates the performance ceiling of data-efficient IT. Furthermore, QCS's adaptability is showcased through its suc- cesses in regular IT and challenging targeted IT scenarios, particularly in the cases of free-form complex instruction-follwiin and CoT reasoning. They underscore QCS's potential for wide array of versatile post-training applications.
Introduction
Scaling up the sizes of architecture and datasets behind large language models (LLMs) consistently yields the improvement of next-token prediction, elevating the achievement of compressing the broad world knowledge into their parameters [3 3. After unsupervised pre-training, task- instruiitnut tuning (IT) [57]. preference modeling approaches such as reinforeement learning from human feedback (RLHF) 4 and direct preference optimization (DPO) 447, can follow to capture the human preference with supervision. Nevertheless, distinct from RLHF and DPO that demand considerable human-preference data to approximate a trustful Bradley-Tee -Terry model 77). IT shows otherwise in existing research [68 18] where curated instruction-responee dyads extracted from instruction tuning datasets facilitate comparable instruction following capabilities in LLMs. Such trained model even capably surpasses the performance of LLMs fine-tuned on the complete of instances. With this regards, efficiently curating diverse and high-quality instruction-respone data for post-training LLMs gradually becomes trend in the community.
Cooresspndiigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Leading approaches to curate high-quality data mostly select the influential instance as a sequence composed of instruction and response. The conventional next-token prediction loss fairly treats each token in the response, overshadowing the potential profits of critical tokens 22 in the sequence, and the risks in token noises [36] [ and biases [31] in Zipf"s law. contrast, token-level information recently grasps more attention to enhance the L.L.-baat -based fine-tuning performances [5] 66, 67].
Despite so, the existing algorithms are rarely investigated to curate tokens along with sequence subset selection due to the prohibitive computational burden.
To rescue IT from the data-efficient dilemma, we dived into the principled analysis between sequence and token selection derived from the spirit of bi-level coreset selection [6]. The methodology inspired from bi-level optimization, yields the objective with small subset to train a model consistent with that one trained with the entire set. It is observed that the common goals of most existing sequence selection and token selection methods can be formally interpreted by the bi-level coreset selection problem with sequences or tokens as their selected instances. With such regards, we propose quadratic coreset selection (QCS) to optimize the subset weights jointly determined by the sequence-level and the token-level influence contributions. It can be proved that the solution set of QCS incorporates all LLMs elicited by the sequence-level and token-level selection, verifying the technical benefit to reconcile sequence and token mining for IT.
Despite achieving the best of both worlds, QCS is hampered by the costly token-level computation from the traditional solver using the inverses of Hessian matrix with the dimensions in the parameter size of LLMs. To this end, our work circumvents the unaffordable computation by a newly proposed QCS probabilistic solver, relaxing the original QCS formulation through re- parameterized density to assign the quadratic coreset weights across tokens and sequences with the sampling policies on their own. We certify the asymptotic equivalence between the next-token prediction empirical risk trained with the subsets drawn from QCS and its probabilistic parameterized variant and more importantly, the solver can be learned by hierarchical policy gradients without propppropa,ttion served by the provable convergence under mild conditions.
Our experimental results demonstrate the efficacy of our QCS algorithm from two perspectives: (1). we evaluate its cntrucctoonfollowinnn ability with the sequence-level data efficiency, showcasing the generalization with limited data; (2) we analyze the token-level benefits and inner-loop adjustments within QCS, revealing the cause of its success. QCS is further evidenced through its reconfiguration for targeted IT ((60), a challenging transfer learning setup for data-efficient training for LLMs; and its replay-based variant to overcome the catastrophic-
challenges in continual IT (Appendix.D) They jointly justify its potential to suit more versatile scenarios of post-training in practice.
2 Related Work
Data efficiency in instruction tuning (IT). Using high-calibrated IT data has been demonstrated to significantly enhance the performance of LLM. IT datasets can be mostly categorized into two threads,
i.e., task-oriented datasets derived from established NLP tasks [56] 48 58 and open-ended instruction
following datasets that encompass a broad spectrum of subject matter [5
16
35|
62
43 17 Among
recent research, the priority of data quality and diversity instead of quantity, enables more efficacious
cultivation of instruction-follon foollwinn ability 13 12 19 38 10
II
30
37 69]. More recently,
[60 20 proposed using efficient estimation 44 25 of influence functions 46 to select the instruction
sequences with key effects in IT. They are technically connected with our methodology.
Curating tokens to learn Miveerse trends of LLMM-based training involve exploration in token-
level knowledge, including pre-training [29 39 and fnc-tuning [65] 66. 67]. Different types of
tokens, as investigated in previous studics [39 24 36 53
32
34
2 may hold diverse influences
to LLMs. Their evidences imply the potential improvement of IT if sequence-level IT data selection can be merged with token-level knowledge.
Coreset-based data subset selection. The process of curating high high-quality data can be reframed as a coreset selection problem [6] The methodology endeavors to identify a weighted subset of instances whose objective function, when utilized, yields the consistent performance to that trained through the usage of the entire datasets. Early efforts investigate the solver of specific learning algorithms, e.g., k-means (22 and GMMs 42), etc, hardly combined with arbitrary models. Recently, [6] unifies black-box parameter learning with coreset selection into a discrete bi-level optimization

---

paradigm:
1 a Σ (o) s.t. (0( Earg min EEC 12
Σ Us v,,,)(())
min-
(I)
BEG K
where L; denotes a training error of the i-th sample, v E indicates the binary weights of training set valued in feasible region C=(p :2=0/1, ||o|lo with integer K'00 to control the coreset size. The coreset v searched by T enables the integration with many applications. However, its greedy algorithm based cone constrained optimization [40] needs compute Hessian inverse with the parameter dimension, poorly scaling to big models. Two threads alternatives refer to matrix approximation 45 probabilistic derivation 70),, respectively. The first is closely relevant with influential functions [46] employed by LESS and the second resorts to a Bernoulli density to re- re-parameterize I laying the background of our algorithm.
3 Sequence Token Mining for Instruction Tuning: A Coreset Viewpoint
Let first review the IT procedure as the next-token prediction with cross-entropy losses as follows:
CIT(O)
log P(ri (11
(0)
(2)
where we represent Lij = log P(ri.j [i z;; 0), then (2) holds the equivalent form Li)(O)
M M N. ( M indicates the number of instruction-responss pairs, then given the instance LLMs with the pre-trained parameter 9 receives the instruction z; along with the observed response content Zi,zý, to predict the j-th token z (2) is simple yet powerful paradigm for the first step most existing post-training, while more evidences in leading research demonstrated that does not require the whole dataset: post-training with high-quality instruction responses the emphasis of critical tokens might elicit the consistent, even more powerful models than the original.
As discussed previously, suppose that some data data-efficient IT algorithm aims to select K instruction- response pairs (K <M) to chase for the optima with regards to (2). Hence given the selected instances denoted by the indicator set lies in Cseq 0;-0/1, ||o||o S KJ. the motivation behind refers to bi-level objective
M
N (0" (v))), (sequence level coreset)
min DCCr
M
N₂
1=1
(3)
M
1 K
Σ s.t., 0* (v) arg min BEB
Σ
(O(o))
N
where (v) represents the LLM trained with instances identified by v, seeking to achieve the original goal of LiT (0). obviously, (2) exactly refers the bi-level coreset selection in (1) with regards to instructiin--rssonne sequences selected instances.
Resembling the similar analysis, token-selection algorithms that attempt maintain the consistent performance with the original IT goal, also elicits the token-level corset selection
N;
min eCCan
C,, (u))), (token level coreset)
M
N
(4)
N
s.t., (v) E arg min BEB M
(..Fa,,(000))
Kg
with CcknnC, and Cie|,M 0. [K,) (K, indicates the cap of tokens selected
in the sequence i).
Motivation:
the dilemma between sequence-level and token-level coreset selection. The pursuit
of data efficiency in instruction tuning through coreset selection, whether at the sequence level ( (3 or token level ( 4), reveals an inherent tension in methodology. Specifically,

---

Sequence-level coreset overlooks token information. encourages LLMs to follow task instructions with the instruction-responss coreset selected IT datasets. It consistent with other data-curating methods, treats tokens equally per sequence. Yet it easily suffers from token noises and bias since critical tokens with factuality and others are fair a sequence.
Token-level coreset is computationally expensive and memory-hungry. The optimal token-level coreset in (4) serves a lower bound of (3) in the same feasible regions. Despite this superiority, { is hardly achieved in practice due to its prohibitive computation with the token amount quadratically larger than sequential instances. Moreover, the data efficiency by token selection is a mirage: suppose the j-th token æi,j is pruned by (4). The token Vi,j remains required as a part of the conditioned response context Li, <itk k in P(zi.++k) Di, 24; 8) E N4). The retention for contexts negates practical memory saving.
4 Proposed Approach: Quadratic Coreset Selection
In the previous section, we note that common goals of massive sequence or token selection algorithms typically refer to 3 that remain the conflict. While their disadvantages are precisely the advantages of each other, suggesting that the limitations can be effectively mitigated by their synergistic integra- tion. To this end, we unify them to derive the novel quadratic coreset selection (QCS) paradigm to obtain the best of both worlds:
N Ci (0,,0 v))),
1 M
min PE
L(0 v))
N
(5)
K
N; Σ j=1
s.t., (v) E arg min [(O(w, v)), L(B(w, v))= leB
we
K
K.
(w, v) indicates the parameter optimized with regards to w to select the instruction-respone sequences and v to select the tokens in each sequence. It leads to the optimization objective with wr and v before using them in IT.
Theorem 1. (Expressiveness of QCS) Suppose that (2(Sseq). 1(0rok) denotes the solution sets 9seq 8ook in the sequence-leve coreset selection (3) and token-level coreset selection (4) respectively; and
Ω2 (00css) denotes the solution sets of 9qcs in It holds (2(qcs) 52(Oseq) n 1 ((
The aforementioned theory demonstrates that, fine-tuning LLMs with instructions with either the response sequences selected by (3) the next-token prediction losses screened by 44, potentially under-ped perforrmproommmmmm due to resulting LLMs with the lower expressiveness. Observed that most existing sequence or token selection nails down the common goal in (3), the theoretical result verifies the technical merit of the marriage between sequence and token mining for IT.. In our Appendix.A. using the typical algorithm for original bi-level coreset selection, we further unveils that the next-token prediction losses in QCS are simultaneously associated with sequence-level empirical influence and token-level empirical inftuence. It demonstrates the deep connection between QCS and LESS (60).
Despite reflecting the compound influence effect, traditional algorithms for bi-level coreset are hardly
transferred into QCS due to their need of the inverse of Hessian matrix [a(0(w" (owwww"w")) ,v')) which in 88
reality in colossal size that grow with the number LLM's parameters, As an inner-loop stage of bi-level optimization objective, computing the inverse is too compromising to suit the regime. Some alternative refers to find its approximation, which have been employed by influential based approaches, e.g., LESS (60). In contrast, propose variant objective relaxed by continual value with probability.
Probabilistic Re-parameterized Solver for Quadratic Coreset
5.1 Probabilistic Re-parameterized Solver with Asymptotic Equivalence
More specifically, we re-conceptualize (5) by the probabilistic reparameterization, where the original discrete coreset weights and were turned into binary random variables that underlie Bernoulli distribution family, i.e.,
Vi E [K] Bern(w:); Vj E [K,], Ui.j Bern(w....)).
(6)

---

where vi E [K], w, [0. 1] and Vj [K.], w,,,, [0, 1] indicate the hyper- parameters to develop the Bernoulli distribution family. Assume that the sequence variables underlying Bern(w;) (vi K) are independent, then the distribution function of sequence-level coreset w could be achieved by p(w(w) T (wc))((( w,) (1-w), Similarly, suppose that vi E [K], we got their discrete weights w drawn from pl u w),, then the distribution of the token-level coreset v, i.e., p( w O v) can be formulated as
K
K
) (1 - Ui.)) (1-noss)))
Note that p(v) w v) depends on the coreset w drawn from p(w ww) It aligned with the fact that the token-level coreset selection with regards to is only available in the chosen sequences with regards to w.
With the coreset densities p(w) w) and p(o/10 @ v), the coreset sizes defined by Czeq and Ctkn can be controlled via their probability masses 1 w and 1 v because of (w--p(o)ww) 10-lll= W and Hemp(( v wov) 1volo=>> i=1 Bespte/wsw) co, 11lb- Ej=r ||va,lla- It turns the feasible regions Creq. Cukn into their continuous counterparts
Cseq {w 0 = w = 1, l||l S K GHtC = C e [K],
(8)
C [v 0 1 le. 3 1, K,),
Hence the original QCS objective can be eventually transformed into
min WEGng,
(w,, v)
E. Pp(ww/)ppr//ww// [(0 (w, v)), s.t. (m) E arg min (0 v)) BE0
The probabilistic reparameterization for QCS offers many technical benefits: first, encourages the coreset optimized based on the learnable parameters of the Bernoulii distribution family efficiently by gradients (as demonstrated in the next section); second, the stochastic sampling provides the lower bound of the original, leading to a better optimization objective; third, the sparsity constraints on and v make their optimal value in either theorem essentially supports the feasibility to replace (5) by (9) given training data sufficiently large.
1, i.e., approach the solution of (5). Our following
Theorem 2. (Asympiotic equivalence between QCS and its probabilistic re- re-parameterized solver) Under the assumptions in Appendix.A, as n oc the difference between the deterministic objective and the probabilistic (expected) objective converges at the standard Monte Carlo rate. More precisely,
Vn (C (w0, v)) - v)w 4 (0,
(10)
for some o² > 0. Equivalently,
[C(0 v)) - t6 v)] Op
Vn
where Op(-) denotes convergence in probability.
5.2 Solver Learning by Hierarchical Policy Gradients
Distinct from (5) directly chasing for the optimal coresets w" and v*, the probabilistic QCS solver achieves IT with the coresets obtained by drawing the sequences w and tokens v from the Bernoulli distribution family (6) with the optimal hyper-parameter w' and v". The deterministic coreset optimization with w and v is non-trivial because their gradient estimators contain their implicit dif- ferentiation. Instead, its probabilistic solver transfer the problems into learning the hyper-parameters and If we consider the samplers p(w|w and p(v) wu @ v) through lens of sampling policy, their optimization can be achieved by our hierarchical policy gradient estimators (HPGEs), i.e.
V. v) Vn (f C(O (w, v))p(w w)p(u/w @ v)dwdo)
/ 2(0 (w. v)) Vwp(w)/) p(wp v)dwdo
(11)
p(wlw)
= Bp(ww))p(hheecv [[('' (wo,)))
5

---

and
V.D(W.V) 2 [.))av([[) v. [100(ws (l v)dwdd)
(f V.p(ulw @v)
- [(00 1(00(m (w, w) w))p(w/w
p(alur & v)dundr
(12)
p( @.
(ptent)sp [(0 (w, v) In [ [n
Note that C( 2)) V ( ( and C( (w, V In p v) unbiased stochastic estimators of (ww v) and v. v). It encourages the forward gradient calculation to estimate (w v), V. V.D(w, v) without the extra backward operation. Moreover, Vw v) and V. v) can be independently estimated despite the dependency from sequence-level policy to token-level policy. They sufficiently ease the calculation of the coreset selection process.
Projected hierarchical policy gradients. In terms the constraints refined to QCS probabilistic solver in (8), we may simply apply them in HPGE by projected stochastic gradient decent, in order to constrain the haper-parameters optimized in the proper range:
w f- Pc.. w 7/2 ((0" (w, v))) V Inp(w//)
(13)
V < Pe.. (v 72 (20((0(w b))) V. (n O.).
With this observation, we can directly obtain the close-form solution to the projection, which yields very efficient process to update and for coreset sampler (see our Appendix.A). More importantly, further prove that if the inner-loop optimization L(O(w, v)) with the sampled coresets w, converges, the outer-loop algorithm for our hierarchical coreset selection policy also converge as the projected stochastic gradient decent (23]:
Proposition 3. [Convergence of hierarchica-poolie coreset sampler (Informal) Under the mild conditions of (ww, v) and the step sizes Ti, T/2, the average of the expectation of the norms for sequence-level and token level projected gradients, i.e.
w. - Perc (w,- - 1/1 V (w:, ve)) 771
1 Vc PCac (ve - 172 VVl,(,, (14)
1/2
both converge T oc.
6 Algorithms
6.1 Practical Implementation of Inner-Loop Update
Whatever variants of QCS require the optimization process in the inner-outer loop alternation, where the inner loop achieves with coreset samples. In contrast with the efficiency in the outer loop, learning in the inner-loop conveys the heavy computation because 0 denotes the parameter of LLMs. To this end, we further discuss the possible improvement to the inner loop.
Elastic token coreset. In the QCS inner loop, [(O(w, vi) is optimized via the sampled binary weights of w.v. Distinct from sequences, the token selection does not earn any memory efficiency as we previously discussed, on the contrary, the sparsity of o may cause unpredictable variance to minimize the token losses. In this regards, after the optimized syper-parameterd select the coresets w.v we re- re-weight each value of token-level coreset v from {0, 1} into {c, - -c,, where c E 1) denotes the elastic parameter close O.
Parameter-efffcient fine-tuning (PEFT). During the alternative optimization, the process the inner loop actually serves for learning the solver parameters w. With this finding, is unnecessary to fine-tune the full parameters in the LLM until v, w converge. In lieu of full fall-parameter IT, we employ low-rank adaptation (LoRA) (28 to facilitate the IT process in the inner loop. The LoRA adapter is trained with the full set of IT data. The process is executed along with the update of v. w.
6.2 Transfer instruction-tuning with quadratic coresets
QCS implemented by the gradient update just provide the pre-selection probability of Derain while does not incorporate any information from Dvat- To enable the task transfer of training sequence-token selection from Dtrain to Dvat. encourage the instance-level Bernoulli hyper paramters w; in

---

and Vi,j in v generated by language model itself. Specifically, we augment the pre-trained LLM with frozen parameter with different LoRA modules with regards to w and then produce binary classifier heads to generate w; and Va.j. respectively, as follows:
LLM (I,, Vs.j
Vs.j = L (z zi), S.t.V zi) Drrain.
(15)
These uncoupled sub-nets and vi are trained with Derain using the selection labels indicated by the optimal wţ and vi, obtained by the gradients (TT.
12), then simple entropy-based test-time adaptation (54 would be employed to fast update their serection probability in Dall-
The vanilla pipeline of QCS and its transfer learning variant have been shown in Appendix.A.
7
Experiments
In this section, we evaluate QCS based targeted IT and regular IT. Appendix. D also illustrate the empirical study on continual IT. They verify the versatility of QCS.
7.1
Targeted Instruction Following
7.1.1 Experimental Setup
Targeted IT denotes the problem that based on handful samples Dj for each target IT task t, we accordingly select small proportion of source data to adapt the task. The experimental setup is coordinated with the setup of LESS, where 5% training sequences are selected to update the LLMs for each task. QCS follows its transfer learning setup. More details found in Appendix.C.
Training Datasets. Our fine-tuming resembles the setup in LESS [60]. Specifically, we adopt the IT datasets outlined by (55] that includes two branches. (1) the datasets derived from existing
collections, such as FLAN V2 [4] and COT [59], and (2) open-ended generation datasets featuring
human-authored responses, including DOLLY 16 and OPEN ASSISTANT 35 corpus includes 270K sequences in wide range of formats and reasoning tasks.
The diverse
Evaluation Datasets. Our evaluation leverages four comprehensive benchmarks: MMLU, TY- DIQA, BBH [4)]. and GSM8K [15) Each evaluation set has multiple tasks while samples in each task split into Dval and Prest where Dval only contains few-shot demonstration sequences while we need to use Dval to select the influential samples in the training datasets, specifically available to this task, then use them training LLM in order to well perform in Drest.
Base LLMs for Data Selection and Training. Before introducing baselines, we specify the LLM used to evaluate the baselines for comparison. Concretely, we employ the popular LLAMA-2 series 522 to evaluate the main results across evaluated baselines for MMLU, TYDIQA, BBH; yet select LLAMA-2-7B and L.LAMA--8BB the base LLMs for GSM8K. Finally, we employ PYTHIA 5 small LLM series to evaluate the adaptability of algorithm under scaling law.
Baselines. The motivation of QCS rises from the data efficiency of IT. To this, beyond the original IT baseline FULL using full IT data, we also employ another two data selection approaches invented to choose key sequences for IT. Specifically, we adopt RAND the simplest baseline to randomly choose some proportion of sequences for IT, and LESS [60), to the best of our knowledge, the most competitive baseline in targeted IT problem.
7.1.2 Main Results
We present main results of different basclines via diverse LLAMA's family models in Figure [ and a thorough evaluation between LESS and QCS across Pythia series in 1,2
QCS v.s. LESS: the response size matters. The results of four baselines across four benchmarks using different foundation models are observed in [ Though targeted data selection only exceeds FULL in 1 of benchmarks, they significantly outperform Rand in all cases. In MMLU and TYDIQA. QCS is very competitive yet remains slightly inferior to LESS. It is possibly due to the short response size of evaluation set: LLM only need to select the options in the context.
BBH and GSM- 8K, with their answers in a free-form long responses, QCS outperform LESS significantly. In the modern I.L.MA-3-8B, even outperforms the LESS baseline by over 12 points (56.7). , proving its sophisticated ability to prioritize the most impactful training instances.
7

---

Figure 1: Results of LESS (selected with target model's gradient store) and QCS (pre-trained to obtain, then fined-tuned with ) on LLAMA-2-7B and LLAMA-2-13B. Full denote full dataset, and otherwise we select 5% of the data with random selection (Rand), LESS and QCS. Bold numbers denotes the best performing selected subset. Underlined numbers denote that the selected subset outperforms the full dataset. Numbers in the parentheses are standard deviations.
Table 2: Performance comparison of using differ- ent variants of Pythia models for data-efficient targeted IT.
Table Performance comparison of using different variants of Pythia models for data- efficient targeted IT.
RAND
LESS Soqerece invll
QCS Juiees level
FUL.L
Raselinss
Wanmp tnining
Adiptive
selection
PT
PATRIA-
5%
11475
<I
PyTHIA 410m
PNTBIA-th
3.8
7.8
0.4
5.0
33
8.5
Menary uni
LESS QCS
176.9
17.0
B 1769M
PYTRIA- 126
17.
15.3
37.3
14.9
32.64
S776
29.8
52.3
33.2
35.3
Avenge
955
12.4
15.0
15.6
18.2
18.2
Rivaling LESS with less computation. The empirical results presented in Table ] demonstrate the computation--conomicll superiority sequence subset selection methodology compared with LESS. As we know, instead of LESS using feature gradient to select sequences, QCS only uses Bernoulli distribution to sample the coreset weights. It saves the computation and memory cost to achieve sequence -okenn joint selection, while reserve the more impressive performance than LESS.
Exceeding LESS under token-level computation burden. The data percentages in Table are calculated at the sequence level, because other baselines are sequence-base data selection approaches. As QCS set 000.5, it just softly select half of tokens in the selected sequences. In this regards, Paaiin
we turn to set oK 5% rather than K 55% as another QCS baseline (token-level) with 6=0
to achieve full-parameter IT. thus, with the consistent burden of token computation. The empirical study is presented in Table where Pythia model series and LLAMA-2-7B are evaluated with TYDIQA. As observed, using small LLMS potentially expands the performance gap between LFSS and QCS and more importantly, most of the cases in Pythia models fail to recover the full-data-I' performance. In particular, the performance of QCS exhibits more pronounced underperformanee relative to LESS. This observed degradation can elucidated through the lens of Eq. (5). implies that IT by sequence-level data selection heavily depends on the capability of base models. While interesting, under the consistent token computation, QCS baselines with
ak 55% significantly
D D
improve the original performances: not only exceed the performance of LESS but also recover the LLM performance trained with full IT datasets.
7.2 Analysis of Token-level Selection
Here we further verify QCS ability to capture token-level information through some verification, where K =5% is promised the token selection happen after the sequence selection.
Paain
Elastic ratio e of tokens. We first consider how the variation e causes the vibration of target IT. In particular, given the corresponding LLM performance trained by QCS. As observed in Fig. (a)). QCS results with c=0.1 and c=0.5 trivially distinct, whereas the performances significantly drop with a larger value. As c=0, the performance are inferior than c-0.1.0.5.
0.5, we change the value of in the range of {0. 0.1, 0.5, 1}. then testify
Selection percentage of tokens. We then turn to evaluate the change of proportion about tokens selected a single sequence completion from (0.2, 0.5, 1) when 1. As illustrated in 22 (b)), the performance of QCS exhibits notable sensitivity to the value of a. Results obtained with values

---

Elstiic
roñoo sequence
offical lokens
eniiica
Tusken selection rañlo sequence
Elastis token ratio
Token notem ratio
Tokeen noise ratio
(b)
(c)
(4)
Figure 2: Parameter analysis of token-level selection.
of 0.5 and 1 demonstrate comparable efficacy, while settings where a = 0.2 yield substantially
inferior outcomes relative to other configurations.
Token-level robustness. We inject random noises into the tokens 339 then further observe the performance change given the previous and value. As shown in Fig. 2). we found that sufficiently large and a could lead to more fragile IT models, with significant performance drop with 10% token noises injection. While small value may produce the better performance than the noiseless cases. However, as the noise ratio increases, critical tokens are more possibly contaminated, thus, all the performances drop in this scenario.
Critical Tokens. Critical tokens in sequences play crucial role of efficiently following instructions with few demonstrations. In order to verify QCS's ' ability to select critical tokens, we consider the identification approach critical tokens [32], since there are no annotation critical tokens in our evaluation benchmarks. To this, we evaluate how many tokens selected by QCS are categorized into the critical tokens identified by 32. The results are shown (c). As we see in Fig. (ef), the percentage of critical tokens selected by QCS is significantly more than random selection, which demonstrates the remarkablity QCS of sensing critical tokens while the sequence-toker selection.
In Fig. (2 d), we observe the variation of E with tiny effect to the number of critical tokens selected.
7.3 Regular Efficient Instruction-E Followiiinn
Experimental Setup. We turn justify the QCS's feasibility in the regular data-efficient tasks using LLAMA-2-7B and LLAMA-2-13B. We consider the vanilla QCS algorithm to obtain the optimal and with regards the training dataset Dirain defined by the self-instruct paradigm. It is derived from two setups: (1) we consider the regular instruction following setup in (38) then evaluate the baselines on 3 common instruction-following benchmarks
MMLU in Hugging-fae face Open LLM Leaderboard-v If evaluation results are presented in Table. in D)p (2) we consider AlpaEva to evaluate LLMs trained by selecting data on FLAN V2, COT, DOLLY, and OPEN ASSISTANT 1, justiffiig the free-form response generation capability. In the first setup, we compared QCS with FULL, Rand, and two data-efficient IT methods, i.e., Instruction- Following Difficulty (IFD) (38 and Dataset Quantization (DQ) [69]. In the second setup, QCS compared with those except for DQ. since almost fails to other methods.
LLAMA-2-7B
LL
0C5 weiu
Pementage
Figure The AlpaEval comparison on the win rates for QCS against different other baselines.
Results AlpaEval. QCS achieves a dominant win rate over Random and IDS across open- ended, instruction-followingg prompts. The plot shows QCS winning the large majority of pairwise
http:/gggw..cooooe////eeeerrrrrn hat.ro/arrttttoeelllmeeeeeeerrrr leaderboadb "ightuu..oo besthub.com/aatuuulaaaaaaaa lablaatsasaaaaaaaaaaaaaaaaaaaaea

---

Table 3: The LLAMA-2 family trained by Data-efficient IT with (10%)
instruction-respone se-
quences, then evaluated in Huggingface Open LLM Leaderboard (ARC, TruthfulQA, MMLU)
5%
LLAMA-2-7B
LLAMA-2-13B
Average
ARC
MMLU
TruthfulQA
Average
ARC
MMLU
TruthfulQA
wo sequence selection
47.45
54.35
47.02
40.98
51.04
57.59
54.05
41.49
DQ
42.32
47.01
42.34
40.12
48.72
50.22
51.91
40.81
IFD
48.28
57.94
44.19
40.62
53.40
62.37
55.65
42.82
QCS (token-selected 20%)
48.86
58.45
45.54
41.67
52.43
61.04
54.46
42.54
QCS (token--clected 50%)
49.46
58.51
47.13
42.36
53.33
62.67
54.56
43.17
10%
LLAMA-2-7B
LLAMA-2-13B
Average
ARC
MMLU
TruthfulQA
Average
ARC
MMLU TruthfulQA
wo sequence selection
47.45
54,35
47.02
40.98
51.04
57.59
54.05
41.49
DQ
44.38
48.17
44.55
40.42
48.72
52.07
52.89
41.22
IFD
48.28
58,02
46.64
40.18
53.40
62.97
55.29
41,93
QCS (token-selected 20%)
48.86
58.78
46.34
41.47
53.34
62.57
55.20
42.24
QCS (token-sclected 50%)
49.46
59.60
46.53
42.26
54.03
63.16
55.86
43.07
15%
LLAMA-2-7B
LLAMA-2-13B
Average
ARC
MMLU
TruthfulQA
Average
ARC
MMLU TruthfulQA
wo sequence selection
47.45
54.35
47.02
40.98
51.04
57.59
54.05
41.49
DQ
43.88
48.27
44.79
40.01
48.22
51.12
52.29
41.30
IFD
48.28
57.42
46.40
40.95
53.40
62.37
55.56
43.42
QCS (token-selected 20%)
48.90
58.58
46.61
41.87
53.04
62.23
55.25
41.81
QCS (token-sclected 50%)
49.16
59.51
46.93
42.56
54.33
61.62
55.68
43.19
comparisons (e.g., 511 wins vs. 168 losses against Random), indicating that its bi-level selection reliably surfaces higher-value examples for general instruction following. While does not surpass the model trained on the full dataset, this is expected; the key finding is that with only small selected subset, QCS matches or approaches the full-data model and clearly beats other selection heuristics. The advantage is most evident in long-form responses, where token-level prioritization highlights factual terms and connective logic, yielding more informative updates than sequence-only curation. Consistency across both modern (LLAMA-3) and established (LLAMA-2) bases suggests QCS is robust to the backbone's capability and provides principled, data-efficient alternative for regular IT.
Results on includes two QCS baselines by ranging sequence-selection ratio is ranged from 5%, 10%, 15%, respectively; and our QCS is implemented with 20%, 50% token ratios to derive two variants. The results are observed in Table Under the regular efficient IT setup, QCS with token selection (especially 50%) delivers the best or near-best averages LLAMA-2-7B/13B outperforming IFD and DQ and often exceeding "no selection" at the same budget. Gains come from emphasizing salient tokens within selected sequences, not just choosing sequences. However, improvements are smaller than on AlpacaEval: these benchmarks mostly require short outputs (single-token or brief spans), limiting token-level leverage; hence QCS's benefit over strong sequence-only methods is modest but consistent. Overall, QCS remains competitive across all three leaderboards and confirms that its largest margin emerges tasks with longer, content-rich responses.
MMLU. Distinct from ApacEval, the experimental evaluation also
8 Conclusion
We introduced Quadratic Coreset Selection (QCS), a principled framework that reconciles sequence- level and token-level mining for data-efficient instruction tuning. By casting both selections into unified bi-level coreset objective and relaxing it via a probabilistic re-parameterization, QCS avoids Hessian inverses, learns with hierarchical policy gradients, and enjoys provable convergence with asymptotic equivalence the original objective. Empirically, QCS delivers superior or competitive performance under stringent data budgets, particularly on long-form, reasoning- haasvy tasks, and adapts well to targeted and continual IT. Analyses reveal token-level benefits-robussee robustness to noise, emphasis on critical tokens, and sensitivity to elastic weighting-clarffyingg when and why QCS outperforms sequence-only selection. QCS thus offers scalable, theoretically grounded path to efficient post-training.
10

---

Acknowledgement
The research was supported in part by Guangdong S&T Programme (Grant No. 202440101010003) in part by Open research fund of Pengcheng Laboratory under Grant 2025KFIB00R in part by The Major Key Project of PCL (No. PCL2024A04, PCL2025A02); in part by National Natural Sci- ence Foundation of China (NSFC) under Grant No.622061 10, 62176103, 22377208,62572499 and 622761 14; in part by the Science and Technology Planning Project of Guangzhou under grants 2024A04J989,, 2025A03/3565 National Natural Science Foundation of China under Grant No. 62572498
References
[1] Cute: Measuring Ilms' understanding of their tokens.
[2] Unlocking tokens as data points for generalization bounds on larger language models.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2XXi::204.00862 2022.
[5] Stella Biderman, Hailey Schoelkopf, Quemtin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages
PMLR, 2023.
[6] Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in neural information processing systems, 33:14879- 14890, 2020.
[7] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-34) 1952.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] Alexander Bukharin and Tuo Zhao. Data diversity matters for robust instruction tuning. arXiv preprint arXiv:2311. 14736, 2023.
[10] Hongyi Cai, Yuqian Fu, Hongming Fu, and Bo Zhao. Mergeit: From selection to merging for efficient instruction tuming. arXiv preprint arXiv:2503.00034, 2025.
[11] Hongyi Cai, Jie Li, and Wenzhen Dong. Low-confidence gold: Refining low-confidence samples for efficient instruction tuning. arXiv preprint 18899 2025.
[12] Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selection for large language models. arXiv preprint arXiv:2307.062990 2023.
[13] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint axXiv:2307.0877
2023.
[14] Wei-Lin Chiang, Zhuohan Li. Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2(3);6, 2023.

---

[15] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[16] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's ' first truly open instruction-tume Ilm. Company Blog of Databricks, 2023.
[17] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint 2r..v:2305...... /4233, 2023.
[18) Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang. Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition.
[19] Qianlong Du, Chengqing Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311. 15653, 2023.
[20] Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024.
[21] Hugging Face. Open rl: A fully open reproduction of deepscek-rl, january 2025. URL https:///gtthu.. comdhhugiingfac//open-/l page 9.
[22] Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pages 569-578, 2011.
[23] Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approxima- tion methods for nonconvex stochastic composite optimization. Mathematical Programming, 155(1))267-305 2016.
[24] Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. closer look at the limitations of instruction tuning. In Forty-first International Conference on Machine Learning.
[25] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 03296, 2023.
[26] Jie Hao, Kaiyi Ji, and Mingrui Liu. Bilevel coreset selection in continual learning: A new formulation and algorithm. Advances in Neural Information Processing Systems, 36, 2024.
[27] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv. org/abs/2103.000 033744 2, 2024.
(28) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. JCLR, 1(2):3, 2022.
[29] Qian Huang, Eric Zelikman, Sarah Chen, Yuhuai Wu, Gregory Valiant, and Percy S Liang. Lexinvariant language models. Advances in Neural Information Processing Systems, 36, 2024.
[30] Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.0180) 2025.
[31] Bowen Jiang. Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo Taylor, and Dan Roth. A peck into token bias: Large language models are not yet genuine reasoners. arXiv preprint arXiv:2406. 10550, 2024.
[32] Lifeng Jin, Baolin Peng, Linfeng Song, Haitao Mi, Ye Tian, and Dong Yu. Collaborative decoding of critical tokens for boosting factuality of large language models, arXiv preprint 200. 2024.
12

---

[33] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001008611, 2020.
[34] Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784-794, 2022.
[35] Andreas Köpf. Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis. Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations- demorrriiinnn large language model alignment. Advances Neural Information Processing Systems, 36, 2024.
[36] Ankit Kumar, Piyush Makhija, and Anuj Gupta. Noisy text data: Achilles' heel of bert. arXiv preprint arXiv:2003. 12932, 2020.
[37] Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak to-strong data filtering for fast instruction-tuning arXiv preprint arXiv:2402.005330 2024.
[38] Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self- guided data selection for instruction tuning. arXiv preprint ar3iv:2308. /0022, 2023.
[39] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint wwXiv:2404.07965 2024.
[40] Francesco Locatello, Michael Tschannen, Gunnar Rätsch, and Martin Jaggi. Greedy algorithms for cone constrained optimization with convergence guarantees. Advances in Neural Information Processing Systems, 30, 2017.
[41] Shayne Longpre, Le Hou, To Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages PMLR, 2023.
[42] Mario Lucic, Matthew Faulkner, Andreas Krause, and Dan Feldman. Training gaussian mixture models at scale via coresets. Journal of Machine Learning Research, 18(160):1-25, 2018
[43] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar. Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.0m 02707, 2023.
[44] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023.
[45] Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International conference on machine learning, pages 737-746. PMLR, 2016.
[46] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
[47] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[48] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv: 20000220 2021.
13

---

[49] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann. Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi. Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint urXiv2210.099200 2022.
[50] Taori, I Gulrajani. Zhangg Y Dubois, Li, C Guestrin, P Liingg, and TB Hashimoto Stanford Alpaca. An instruction-following llama model, 2023.
[51] Csaba D Toth, Joseph O' Rourke, and Jacob E Goodman. Handbook of discrete and computa- tional geometry. CRC press, 2017.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models (2023), arXiv preprint arXiv:2302.13977 2023.
[53] Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In International Conference on Machine Learning, pages 35413-35425. PMLR, 2023.
[54] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations.
[55] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764- 74786, 2023.
[56] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Supermaturalinstructioos generalization via declarative instructions 1600+ nlp tasks. arXiv preprint XrXiv 2204.07705, 2022.
[57] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
(58) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint 100.0220, 2021.
[59] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of- thouhtt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[60] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint 203.43340..0433. 2024.
[61] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022.
[62] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 1..3304.2244 /2244, 2023.
[63] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement, arXiv preprint arXiv:2409. /2/22, 2024.
[64] Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Xiaojun Wu, Honghao Liu, Hui Xiong, and Jian Guo. Select2reason: Efficient cenrucctiiottunnnn data selection for long-cot reasoning. arXiv preprint 1126: 2025.
14

---

[65] Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou, sefference-grounder token-level guidance for language model fine-tuning, Advances in Neural Information Processing Systems, 36, 2024.
[66] Yongcheng Zeng, Guoqing Liu, Weiyu Ma. Ning Yang, Haifeng Zhang, and Jun Wang. Token level direct preference optimization. arXiv preprint arXiv 2404 11999, 2024.
[67]
Yuanhao Zeng, Min Wang, Yihang Wang, and Yingxia Shao. Token-efficient leverage learning in large language models. arXiv preprint Ttt1:2404.009144 2024.
[68] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
[69] Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17205-17216, 2023.
[70] Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Prob- abilistic bilevel coreset selection. In International Conference Machine Learning, pages 27287-27302. PMLR, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check the abstract and introduction in details.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer to Section 2.2 and our discussion in the experiments.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please check Section 3,4, and Appendix.A.B.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please check Section. 5.6, and Appendix. C.D.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: The work is involved with some private project in Pengcheng Lab, and we are going release the code when the private period is ended.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please check Section.5 5,6, and Appendix.C.D.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Refer the experiment.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The training and inference stages are conveyed by a single A-100x8 SXM server to achieve experiment across all experiments.
9. Code Of Ethics
16

---

Question:
Does the research conducted in the paper conform, in every respect. with the
NeurIPS Code of Ethics Stt.gh/tpppe..eoorrr.../oo//o//a
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is required.
Answer: [Yes].
Justification: The paper's writing was significantly polished via LLM tools. The usage of LLM strictly follows the rules in http::///earrrieecc//ooooeerrres
17

---

Appendix.A: Technical Discussions
In this section, unveil the connection between QCS and empirical influence functions, and provide the back-propagation free solution of w and V.
The Interpretability of QCS via Empirical Influence
To be selected by the original QCS objective (5), the token in the position (i,j), i.e.. at the j-th order of the i-th sequence, have to satisfy the following two condition: (1). provided the optimal ut Czeq. W," 1; (2). provided the optimal vt Ctkns Vi.j" 1. The solutions are derived from the formulations
M
1 M
min (wr 0))= WECmm
Lig (0 v))),, s.t., (v) E arg min ((0(w, v));
N₂
8EB
i=1
(16)
K
N₂
1
L( u))
Us.j i (0(2, v)))
K
K;
1=1
==1
and
1 M
min C( (w. u))
Cis (o (w, v))) s.t., st.,0"()) arg [(0( v)),
vECrin
M
BEG
i=1
==1
(17)
N
((O(w, v))
Σ
1
Σ
Ut.j Ut. Ci, 0)))
K;
j=1
According to the approximation results derived from cone constrained optimization (6. we got the greedy search algorithm with the selection rule
Of(0(w s/n Ust2)) ) 8 de
(vo(c10))
token (i" (i',j") E {ir :f arg max Vow, CTT, jE[N,-]
0000
s.t. W6-
wish
Of(O(w SCN Ust2) ))
(vocc(o))). (18)
token (i',j") ( :j
; E arg max VOL. JEINJ
0000 0
0S(0(W sth stz 1)
s.t. ie r :1 E aarg max
Σ VOULLLiN)
No
TBT A
SE[N]
AEINd
w.r.t. Us.k E
where Sbeg. Stn denote a set sequence atoms of size t1 (t1 I K) and a set token atoms of size t2(t2 & K2) have been selected, respectively. Note that, A. 18 is closely related with the token-level
empirical influence
39" JEsto
Of((0 .x"') V VoCi, and the sequence-level empirical VoLr,((0)) In particular, if the greedy search k=
B
Sean
influence 89" Ueeen
#f(0(w" 00
11
No
results u s4y = and et s/2 = A.
holds
token (i .) E 1"=5""
:
arg arg max jE(N.)
00
(Vo(C(OT))).
Dēthn
emc==U
s.t. iE =0 E arg max- - sE[N]
00*
I(o*)) (Vo(C(
DEsea
=0
1 EKENN) Vo.. (eR..e)
w.r.t. (v) -0 (m + n);
Em)(v)
(m n))
(1, EnE(NN) Von C...)
(19)
where Ve., L indicates the m-th element of vector VoL.
18

---

The Coreset Sampler Update without Back-propagation
We consider the projection w <- Pen. (w 7n [(0 (2f((0 w))) v In p(v) w v) be solved accordingly. Specifically, assume the projected result is w, then the origin denotes Zw. solve refers to
(1000(m (w, w))) V In p( w)), then v f- Per. (v
Pem (w m m12(00 (we u))) V (nw(()))
min
W Zw| Zw|12. 12, s.t. 1 S K. 0 S
(20)
WERN
refers solving the problem with Lagrangian multipliers
min
||w
zwl1 + A1 (1 Twww -)
WERN
(21)
N x2 s.t. A.t. S.
min
||W
(Zw 21)11 A1 K)
/
WER
To minimize the formulation with respect to w, we have
1z- -1111 + (Zw 11)12>m-1>00
(22)
So given 2 0, we have
1 -11l2 - (A + 1)1 + A(1 zu w) ((A) = 2 ا1| 12w 11]-11++
N
X2
2
(23)
(A) = 1 min(1, max(0, Zw A1)) K
Since g'(A) is monotone decreasing with respect to À so that we can solve g'()) = with solution
AG where the maximum of g(A) is achieved at 0 if A < 0 and X if X5 > 0. Set X max Ai}. holds
= + (Zw max (0. 12=
(24)
min(1, m0 Zw max(0, 1))
Similarly, Yw ~ p( w) we have
v = -w@ 10x. + (w @ z, max 11 -00.:00mmm0000/1100 (25)
min(1, max(0, z, m0
where A5 2 indicates the maximum of I(A):
N
M(A) - 1| ④ x1]-112 + ||[ ② z, - (A + (x+1)1)++ + w ② 2, v)
2
K
+h((A) 1 mix((,,axx00 wz A1))
Σ K₁
(26)
Appendix.B: Theoretical Justification
In this section, we provide the proofs of our theoretical results in the paper.
Proof of Theorem.
Proof. Let (. (w) = {810(w) e arg mig & EM wi w EN (r,( s.t., w
(0, 10.1)MM |wo| S {K] and Sin (K).))))2)))))) (v) = {010(0) arg ming 1 EM ( Σnnn Us Ci
s.t,, vs E N), |usl s K,, Vi E [M.] Given this, The solution sets of (3) and (4) can be rewritten as
Sunq { (w, (w, 0) arg min
Las (0))
WECm
BES ol0 (o.K
(w) M
i=1
(27)
N (0))
Sun (0,00] (v, arg min
vECn. dE DE (0.(K))
(v)
19

---

Let Sqcs (v, (u, w) = (o)((w))) v) arg Ωेr Eie w/ (N. شラ= (. s.t.
(0,1) |20| S k; E (0,1)% lodl s K₁ Vi [M]. The solution set of QCS can be rewritten into
M
Sycs = (100 (,,, (w, 0, arg min vECn, siee (w0o) M
1
L (0)) (28)
N
i=1
Observe that if we set K = M and 1 the equívalence holds
S(ø) qcs (1 MM v)
N Ui (r) v, {0, (0,1)%% s K,,ii [M}
={010(1 M .v) arg min
1 K
Σ
o'ee
N
-s(o.,Kk) (v).
tkn
(29)
Similarly, Vi E [M] w.r.t = 1, U4, L, j [N,]. We set 0 = falu, = 1 M iff w; 1}. then
w. 0) S 5
N
=000 0) E arg min
1
Σ
wi
1
(0') s.t., w E (0, 10. |w| s KK
(30)
W'E
K
N
j=1
-sw)
Therefore, we have
M
N.
Seea = ww, (ww]), 0) arg min
(0) }
wCC BES (w) M
N
1=1
i=1
(31)
M
=((w, e, ([w,, 0) arg min
1
Σ
(o)
WEC 00 (m0,6)
M
N
and
M
N
San (v, (1, E arg min
(
Des (v) M
j=l
={(1M ,<( N,M,,,0)(1*'
v, arg min WEC (1 .e)
1
M
(o)
M
(32)
Observe that è C Ctkn and S (w, 0) Ssed (w,v). v). v0- E ((0ggs)) = (0,, u, E Sqes) and V tkn E (2(et))) = (f) 0) Stn). it holds
M
N C.d 10 (0am))
M
N
S
Cns (02ta))
(33)
M
M
N 1=1
1=1
implies Vo E (00ss)), 02es [((m))). Since 41 Csoq and Seed (1 M v) C Seed (w, v).
((d))
(
v0 E (00ges) (w v, 0) E ) and V0 ((Ogseg) (w, 0) E Sseq) also result in
1 M
N: L.s (OGss)
M
N
1
S
Lig (Osea))
(34)
M
N
M
1=1
It implies v0- E R(Dges), Oges ((0seg). Summarize Ω (((tn)).
33
34
and we have 2(0gcs) C 2(0sea)
20

---

Proofs of Theorem.2
There are some assumptions required to justify the asymptotic approximation between the original QCS and its probabilistic variant (Theorem.2).
Assumption 4. (Lippschitz Continuity)) The next-token prediction loss Li.j is I-Lipschitz with respect to the selection weights, thus, vv, E Ctkn and w.w' E Czeq. Le. (f(,,)) Cr (0(w', v')ll s L(ll - w'll + vv'l
Assumption 5. (Tokenwise Boundedness) The per-token loss (t) are uniformly bounded (or, more generally, sub-Caussian).
Assumption 6. (Large Sample Size) We assume that the effective number of independent selections, denoted by n (which could be the total number of tokens or sequences), tends infinity.
The validity of assumptions. In general, the boundedness and Lipschitz continuity conditions for the next-token prediction loss with respect to the selection weights serves for two conditions, thus, the loss serves Lipschitz continuity to the model parameter; and the model parameter satisfies the I Lipschitz continuity to the selection weights. The first case is practical. Specifically, the tokenwise prediction loss Lig- = log P(ri.j (ūj :2; 2;;0) () is often defined as the cross-entropy loss or negative log-likelihten oog-likllhhood so that C.,, E [0, +00). though in practical implementations, Lig is commonly capped due to numerical stability (e.g., Li,j I log where > is the smallest representable probability), in order to avoid issues with extremely small probabilities. In the second condition, thus, the model parameter 0 satisfies the Lipschitz continuity to the selection weights and w. Observe that the gradients of & with respect to v and w is bounded with respect to the policy gradients in (11,12), and the model parameter holds the Lipschitz continuity to 4 in terms of its the composite of Li, in probability. So the second condition also holds.
Li. is also tokenwise bounded due to the smoothness of the softmax function (the token-prediction head) and bounded gradients with respect to the model parameters. Assumption.3 implies the deviations induced by the Bernoulli sampling can be controlled via concentration inequalities. Note that it actually holds since given the optimized v, the selection of Us in each token position is totally random, so do w.
Proof. Consider w and v obtained by optimization. Let (w", u²) be one realization of the binary selections that
w= p(w/w), 8" ~ p(0/20 @ v)
(35)
Since the probabilistic objective is the expectation of L(0(w, v)) with w, drawn from the optimal and v, thus
F v) (pp(w//p(() wov) C(0" (w" o²))-
((f)
(36)
By construction, w and hold E w) (we [m=]] and M..j = Exp (w))(( wov) [v[,] due to
the definition of Bernoulli distributions. is observed that the objective with respect to w and satisfy the Lipschitz continuity, thus, we have for any realization
||0(0 (ww", x²)) -(( (w. v))l
lim
|| (C((""w',, (w',v')) L(0 v))ll
w w .
(37)
lim
L(llV [oll--vll+ 1|w' w(])
= =[(|Vv - o² || + || |ww>
Since the deviations ||w- w01 and ||v w'll are sums of independent bounded (or sub- Gaussian) random variables, the central limit theorem applies. More specifically, wn and vf,5 are binary and lie in {0, }. their differences from the continuous weights are bounded. By applying Hoeffding's inequality to these independent binary variables, we can assert that with probability at least 1
v w|| + || - w/l s CV 1og(1/5)
(38)
for some constant C > 0 and effective number of independent samples n. It allows that
log(1/8)
|| [[ (w0", v²)) - L(0 (w, v))l & V
(39)
n
21

---

Taking the expectation over the binary selections yields
||<(00(w", ,")",")))))""""""")""")"")) ((w,, w)ll S STL
(40)
7
Proof of Proposition.3
Let first provide the formal convergence result derived from
Proposition Suppose that ((w. v) is L-smooth with respect to w,v, respectively; and the policy gradient variance ((C( V In p(w/l"') Vw I o², and E(C(0 (w, V.) Imp(v/w 2 v²) V.D(w', "olww'v')) v')) S o². Let m. m2 I E then holds
Σ
E (IGH [1()))) s 22
1 mn lm/22)
(41)
T
where Gür G denote the gradient mappings with respect to v at the t-th iteration, i.e.
g _(m,- PCcny(N (w' 7/1 (w g (ve Pera (v' 7/2 V, v'))).
71
(42)
Note that Proposition.3 can be proved when the proposition above is satisfied. prove the proposition above, we first present the lemmas derived from (23]:
Lemma 8. Given a compact convex set C. Id and let Pe be the projection operator on C, then for any u, V Rdd we have
[[P(()) - - & (u u - Pe(u))
(43)
Lemma 9. Given compact convex set C Rd and let Pe be the projection operator C, then for u, U E Rd and any, E C we have
||Pe(v + -Pe(u c)ll s ||u - v/l
(44)
Given this, we have the following proof of Proposition.3
Proof. Consider the update
w Pen, (w' - m)gi), gi L(0 v) V Inp(w/w")
(45)
v++ Pe (v' - m22gh), 9 L(0 (w, v) wn pn
Since the stochastic gradient mappings Gt, J and deterministic gradient mappings denote as
G (w: Wo41), 0i - -(v, ve+1):
711
71
gt (w: Pen. Pena(w' DWD vV)),, gt - (v. Pean (v' 1/2 (w v()))
(46)
22

---

Therefore
B(w' v ((ww, v-) + (Vw v²), we+1 we)
+ (V. v²), ve+1 vz) + 2 || - -wal? + L
L
=l11111 vell
2
LuB
B(ww, vv) mm -m11@ 1| 1/2 ||
+
19u112
+
||@011
2
2
+ n (8), (. + 162 96) (Lemma8)
S ((ww, vt) - (n1 Lmj )1|@/11 2 -(12 Ln2 |1ś311 + n (51 (W) - -22 ot)
2
+ mm111691 =m11691111s - لWll + 122168111 Gill
- D(w', v²) - (011 Tnf 1112 - -nn2 - Tug ||
2
2
+ m (81, (h) + (02. (.) + =m1loil1 + 121112111 (Lemma9)
(47)
So obtain
Lidi 11 + (12 2
(12
||(!111 - @(w', v2) ((w't1, v
2
5m (0), gum + m2 (82, (b) + 7 11 + n2110311.
Laf
+ (12 1 Inp اااا s Ø(w! v\) D( v2+1
(n
)11@111
(48)
2
Σ m (6) (.h + 12 9 + m1119111 7/ 16211- t=1
Since E88, ( 0 and E(52 9
0. Beyond this, we also have E|00
Blggi
V 1 (w", )|l < d² and B = E 92 V. v')ll < 0². Combine them and we obtain
((w' v') 4-
o²
Σ ((
(49)
min maa (1 Ln/2yT minge m2) (1 Lm
Then finally, we can bound the expectation of the deterministic gradient norms:
E 1 < 2E 21 + 2Bll Vwb(w', v¹)l < 2 2 + 20
(50)
< 2B11 + Eg VoR(w', vi)l 2 + 20²
So
T
T Σ
E
|| + =1
T
Σ
E
(11@ 112 + 11 + =20²)
T
(51)
t=1
((w! vl) - x
minne (m. (1 Lm/2)T +
Ln/ + +20²)
min (m m2) (1/Ln/2)
220 min (1
Appendix.C: C: Implementation
The pipelines of our QCS and its transfer learning variant for targeted IT, are illustrated in Algorithm U 2. The code would be released.
23

---

Algorithm Quadratic Coreset Selection (QCS)
Input: A pre-trained LLM with parameter 0, IT datasets D.
Parameter: 7/, T)2, C, a,
Output: QCS solver with w" .v", LLM with parameter D*
Initialize QCS by w;= [D] Vie [K]; vr, Vjj [K:].
2: Initialize LoRA parameter O' with respect to 0.
3: while w.v not converge do
4: Draw a stochastic batch 'D C D.
5: Sample sequence-level and token-level coresets of D3 via w--((w)w), vmp(pll @ v).
6: Ví E [K].j E [K Vs. -- Ui.j + (-1)"
7: Perform LoRA tuning update 0' via w,
8: Update w, v via 33 344 with D,.
9: end while
10:
Obtain -'
Sample K sequences by w' as w" construct stochastic training batches;
12:
while each training batch not converge do
13:
Sample tokens by v* with respect to the sequences in the training batch, update vi &
Ui, + (-1)
14: Update 9
15: end while
16: return w' .v, -0
Algorithm 2 QCS transfer learning variant
Input: A pre-trained LLM with parameter 0, IT source datasets Derain target datasets D Parameter: 71. 7)2. K
Output: QCS solver with w" w"'"", LLM with parameter 0*
Use Algorithm[] obtain w".v" by Derain:
Initialize separate LoRA module 6,t;; For Dirain. generate the stochastic label w for each sequence, generate the stochastic label vi for each token;
For Derain, update á to predict wt for each sentence, update v to predict wiy for each token; For Dval. apply test-time training to update update á and update ti; Sample K sequences by ø as w" to construct stochastic training batches; while each training batch not converge do
Sample tokens by vi with respect to the sequences in the training batch, update Uz.j + Vij +
(-1)
Update 0 end while return 0*+0..
The experimental setup of regular IT refers to 388 The experimental setup of targeted IT is followed.
Training and evaluation setups In the first experiment,
follow the training and evaluation settings in [60] and for the second experiment, we adopt the continual IT setup derived from (56) the task
orders (1): QA QG - SA Sum. -> Trans.; (2): Trans. SA QA Sum. QG; (3): Sum. -> QG > Trans. + QA + SA. The continual learning order is as follows: (Warmup: Alpaca52k >> task order. The experiment conducted with 1% 0.1% of the training data from SuperNI tasks and Alpaca-52k as demonstrations, respectively.
Dataset
Shot
Tasks
[Dval
|resst]
Answer Type
MMLU
5
57
285
18,721
Letter options
TYDIQA
9
9
1,713
Span
BBH
3
23
69
920
COT and answer
GSM-8K
8
10
1319
COT and answer
Table 4: The statistic of evaluation set.
24

---

Training. We implemented our QCS optimization strategy with LoRA incorporated learning rate scheduler featuring linear warm-up up and cosine decay, with the learning rate peaking We standardized the training process across all selected datasets, utilizing a batch size of 128 and conducting 4 epochs to obtain the QCS parameters and the optimal coreset distribution parameters v, w. In the adaptation phase, we fix the LoRA parameters then fine-tune w with one epoch. The setup has been applied targeted IT and continual IT experiments.
10
Training setup of targeted IT. The sizes revealed that extending the epoch count for smaller datasets did not yield significant performance enhancements. Consequently, we adopted uniform epoch count across all experimental conditions. The LoRA module was configured with rank of 128, an parameter of 512, and dropout rate of 0.1. We applied LoRA matrices to all attention matrices within the model architecture. This configuration resulted in trainable parameter counts of 135 million (1.95% of total parameters) for LLAAMA-277B and 209 million (1.59%) for LLAMA-2-13B, To ensure robustness, each experimental condition was replicated thrice using distinct random seeds. For random selection methodologies, this entailed generating three unique subsets from the training corpus.
Training setup of regular IT. In TruthfulQAATuuttuuUQAA, MMLU, following 38 we use the instruction prompts in Vicuna [4] construction, then obtain the self-generated responses from the pre- trained LLAMA-2 family models. We evaluate the base models trained with the instruction instruction-response response pairs through the three common instruction-followe trained with data selected by the mixture of FLAN V2, COT, DOLLY, and OPEN ASSISTANT (More refers to the setup in targeted ITT).
benchmarks. In AlpaEval experiment, LLM is
Appendix.D: D: Complementary Experiments
QCS v.s. LESS: more analysis on computation and memory cost
With regards to 1a I, the training time for LoRA in Algo. is almost consistent with the warmup stage in LESS, yet we need a labeling process (IO process) to prepare Algo.2 (it leads over a half' of computation cost in the warm-up stage in our implementation) and another LoRA in Algo.2. Note that the labeling procedure can be replaced by a teacher network that dynamically offers the soft label in Algo.2. This memory- time trade-oft off may further cuts off a significant amout of computation in GPU hours.
As for the convergence behavior, The charts demonstrate a clear and consistent convergence pattern for Algo2, where the validation loss in BBH exhibits a distinct U-shaped curve, reaching its minimum around epoch 1-2 before starting to increase due to overfitting. This convergence behavior reveals a crucial practical efficiency gain: optimal validation performance is achieved very early in training. This allows for early stopping, saving significant training time and computational resources beyond the per-step metrics in Table 2, thereby confirming the methods' practical efficiency.
Efficient instruction-tuninn for mathematical reasoning
Note that QCS is particularly impressive when the response is long in the selection data. is supposed to be superior in mathematical reasoning tasks, so we provide a more challenging evaluation that focus on this problem.
Setup. We adopt a large reasoning model (LRM) as the backbone, specifically 5-Math-7B-Innn ( Instrrucct (63. Fine-tuning is performed with 10% long CoT reasoning responses selected from OpenR1-Math 220k [21] using our QCS sampler (e = 0.1, a = 0.2). For comparisons under the regular instruction-tunin 2 5-Mthh- (Base LRM), (ii) the model fine-tuned on 100% of OpenR1- -Math 2200k (Full), (iii) the model fine-tuned with 10% data randomly sampled from OpenR1 -Math- (Random), and (iv) the model fine-tuned with 10% IDS-selected data (IDS).
(IT) setup,
include: (i) the original
Training Details. We use the system prompt "Please reason step by step, and put your final answer within \boxed(()." Decoding is greedy with temperature set to and top_p=l, producing a single output. Evaluation follows the rule-based criteria of [44] checking both numbers and formulas. We report Pass I (P@1 1), following (63. All runs are conducted on a server with 4x A100-SXX448800 GPUs. We set the sequence length to 16,384 via RoPE scaling, batch size 1, gradient accumulation
25

---

4. and learning rate 5e-5 with warmup ratio 0.1 and cosine decay. Training lasts 3 epochs. For the judge model, we apply LoRA with rank 16, a 32, and dropout 0.1 for epoch. After training, we evaluate on MATH 27 and GaoKao-MATH 63
Table 5:
Pass@1
(P@1) on
MATH
and
GaoKao-MATHE
Backbone:
Qwen-2. 1--5--M----- 7B- Instruct.
-ath-77-Innsrruc.
Benchmarks/ Baselines
Base LRM
Full
Random
IDS
QCS
MATH MATH
0.842 0.781
0.894
0.878
0.889
0.901
0.783
0.789
0.863
0.878
Results. Table reports P@ 1. Using only 10% of OpenR1-Math-225 QCS surpasses Full on MATH (0.901 vs. 0.894) and shows clear improvements over Random and IDS. On GaoKao-MATH, QCS achieves 0.878, outperforming Full (0.783), Random (0.789), and IDS (0.863), demonstrating strong sample efficiency for complex, long-response reasoning.
Case Analysis of Selected Instances
We illustrate some instances selected by our algorithm along with the tokens that were identified as "critical" during post-training.
Case Study 1: Mathematical Reasoning (GSM8K)
Instance A: A High- Value, Complex Problem (Selected and Prioritized by QCS)
Problem:
"A garden has 10 yellow flowers. There are 80% more purple flowers than yellow ones, and the number green flowers 25% of the yellow and purple flowers combined. How many flowers are in the garden?"
Sequence-
QCS flags this as high-value because it requires multiple dependent steps and understanding percentages, a complex reasoning skill.
Level
Analysis:
Token-
Within the solution, QCS identifies and up- up-weights critical tokens that form the logical backbone of the reasoning process.
Level
Analysis
(QCS Ad-
vantage):
Solution with Criti-
"First, calculate 80% of the 10 yellow flowers: 0.80 x = 8. The number of
purple flowers 10 8 18. Next, the total of yellow and purple 10 18 28. Then, calculate 25% of that total: 0.25 x 28 = 7. Finally, the total number of flowers
cal Tokens Empha- sized:
18 35."
Comparison:
A sequence-level method would treat all tokens equally. QCS focuses training on crucial numbers (10, 18, 28, 35), operators (x, +), and percentages (0.80, 0.25), helping the model learn the mathematical procedure, not just the narrative.
Case Study 2: Mathematical Reasoning (GSM8K)
Instance B: A Simpler, Less Influential Problem (Diwnn-eiighed by QCS)
Problem:
"A bakery sold 15 cakes on Monday. On Tuesday, it sold 12 more cakes than on Monday. How many cakes were sold in total?"
Sequence-
A standard method might select this for its basic arithmetic. QCS ranks it lower because the reasoning path is simpler and less informative for an already capable L.M.
Level
Analysis:
Token-
The critical tokens basic (15, +, 12, =, 27). While correct, they do not introduce the same level of complexity as Instance A; QCS recognizes lower informational density.
Level
Analysis:
26

---

Case Study 3: Complex Instruction Following (AlpacaEval /OpenAssstaan)))
Instance C: A High-Value, Complex Problem (Selected and Prioritized by QCS)
Instruction:
"Explain the main differences between nuclear fission and nuclear fusion, and list one practical application for each."
Sequence-
QCS identifies this as a high-quality instruction because it requires factual accuracy, comparison, and structured knowledge.
Level
Analysis:
Token-
QCS prioritizes key concepts, relationships, and entities that must appear a correct response.
Level
Analysis
(QCS Ad-
vantage):
Solution
"Nuclear fission is the splitting of a heavy, unstable nucleus into two lighter nuclei. A practical application is in nuclear power plants. contrast, nuclear fusion is the process where two light atomic nuclei combine to form a heavier nucleus, releasing energy. practical application is the energy source of stars like our Sun."
with Criti-
cal Tokens
Empha-
sized:
Comparison:
A sequence-level method may not differentiate between key terms and connecting language. QCS focuses the model on core vocabulary (fission, fusion, splitting, combine, heavy, light) and crucial entities (nuclear power plants, stars), ensuring factual and relational knowledge learned correctly..
Limitation and Future Work
QCS is sensitive to hyperparameters and stochasticity. Its performance hinges on budgets, token ratio, elastic weight, and policy learning rates; coupled with Bernoulli sampling, this can introduce high variance and unstable convergence-espeinng selection offers limited practical memory or speed gains because pruned tokens often remain in context for next-token prediction; without architectural support (e.g., chunked attention or selective forwarding), compute savings are modest. Also, theoretical guarantees depend on smoothness, boundedness, and large-sample assumptions that can be violated in real L.M training (optimizer momentum, mixed precision), reducing the tightness the bounds.
on smaller or less-stable backbones. The token-level
Beyond these, the transfer selectors (sequence/token heads) can be brittle under domain shift, mis- calibrating selection probabilities across styles, lengths, or reasoning types; additional adaptation is often necessary. Engineering overhead remains nontrivial: although QCS avoids Hessian inverses, the hierarchical optimization, projections, selector training, and repeated sampling introduce IO and scheduling costs at scale. Together, these constraints highlight the need for automated hyperparameter tuning, variance reduction, architectural co-design for token pruning, stronger comain-adaptive selectors, and streamlined pipelines.
Overall, QCS pave the principled way understand and investigate token-level information while selecting long-response instance for ciaructioo-following However, the hyper-parameter tuning and the complex training pipeline make it difficult to extend and further improvement. Some simple-yet-effective approaches inspired from Eq.5 can be more promising and expected.
27


Reviews:
[{"review_id": "j2Dmw9TWwF", "reviewer": "Reviewer_beGB", "summary": "This paper introduces Quadratic Coreset Selection (QCS), a novel framework for improving the data efficiency of instruction tuning for large language models. The central idea is to reconcile the two approaches: sequence-level data curation and token-level selection. The authors formalize both approaches from a bi-level coreset selection perspective and propose a unified QCS objective that jointly optimizes for selecting influential instruction-response sequences and critical tokens within them. The experimental results demonstrate the efficacy of our QCS algorithm from two perspectives: (1) its instruction-following ability with the sequence-level data efficiency, showcasing the generalization with limited data; (2) analyze the token-level benefits and inner-loop adjustments within QCS, revealing the cause of its success.", "questions": "As pointed out in my review, I want to know why old checkpoints like Llama 2 are used for experiments but not the newer models.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The paper addresses the crucial challenge of data efficiency in instruction tuning. The contribution is both novel and important by unifying sequence and token-level selection into a single principled framework. \n\n2. Experimental results showcase that the proposed method can achieve performance close to the full set while using only 5% data, which is also better than the compared baselines.\n\n\nWeakness:\n\n1. The paper is conducted on Llama 2 base models, which I think is quite outdated. I strongly encourage the authors to perform experiment on newly-realeased foundation checkpoints like Llama 3 or Llama 3.1, or Qwen 3 family models, since the conclusions on such checkpoints may make more sense and be more insightful. I am looking forward to new results on newer model checkpoints in the rebuttal.\n\n2. The result section in the work is somehow weak. I think case studies are always desired for this topic since readers want to understand from specific cases how this method works. I also look forward to more example cases in the rebuttal.", "was_revised": true, "final_justification": "The authors provided detailed explanations in the rebuttal which addresses most of my concerns. So I decide to raise my score a bit.", "rebuttal": "Thanks for your constructive comments and suggestions.\n\n**Q1: The paper is conducted on Llama 2 base models, which I think is quite outdated. why old checkpoints like Llama 2 are used for experiments but not the newer models? I strongly encourage the authors to perform experiment on newly-realeased foundation checkpoints like Llama 3 or Llama 3.1, or Qwen 3 family models, since the conclusions on such checkpoints may make more sense and be more insightful.** \n\n**A1**: We provide the respose from the two aspects:\n\n**(1), The reason to use the old checkpoints instead of newer models**: Most baselines in our experiments were directly derived from their reported results in the original paper by following their experimental setup. It saves our time but we commited that such setting may not fully reflect the strenght of QCS (refer to our discussion in the rebuttal response A2 of the reviewer Mibm). With this regards, we tend to provide the experiments on the benchmarks GSM-8K and AlpacaEval, which are recommended by the reviewer zsf7 to further demonstrate the generalizability of QCS.   \n\n**(2), The new experiments with newly-realeased foundation checkpoints:** Specifically, we consider llama-3 as our base model, then conduct the target instruction-tuning (T-IT) setup in GSM-8K in order to compare with the competitive baseline LESS. Beyond this, we also consider llama-3 as our base model for the benchmark in the regular instruction-tuning (R-IT) setup, due to no availabel validation set in this benchmark. The both setups adopt the the pre-training dataset collected from FLAN V2, COT, DOLLY, and OPEN ASSISTANT 1, and for all the benchmarks, we compare our QCS with the final model trained with full data and randomly selected data (5% selected from the validation set in GSM-8K; 6% selected from the for AlpacaEval). \n\n|llama-3 8b| GSM-8K | \n|----------|----------|\n|No Fine-Tuning| 34.3 |\n|Full| 91.7 |\n|Random| 43.6 | \n|LESS| 56.7 | \n|QCS (ours)| 68.8 |  \n\n **AlpacaEval** (win;tie;lose): \n\nQCS vs Full: 303:32:465\n\nQCS vs Random: 511:121:168\n\nQCS vs IDS: 466:56:278\n\nwhere the evaluation of GSM-8K uses the 8-shot setting with CoT with respect to (Wang Y, Ivison H, Dasigi P, et al). Similar with BBH, 10 examples from GSM-8K are drawn to construct the validation set, which guides the targeted IT select the training subset in GSM-8K. As the same in BBH, QCS employs the constructed validation set to setup the early stopping checkpoint in Algo.2, which produce the model to predict $v_{i,j}$ and $w_i$ for targeted IT. The results are found above. The results clearly demonstrate that the QCS algorithm is highly effective for data-efficient instruction tuning. Specifically:\n\nOn the challenging GSM-8K mathematical reasoning benchmark, QCS achieves a score of 68.8 when using a Llama-3 8B base model. This performance significantly surpasses other data selection methods that use the same small fraction of training data. Specifically, QCS outperforms the competitive LESS baseline (56.7) by over 12 points and leaves Random selection (43.6) far behind, proving its sophisticated ability to identify and prioritize the most impactful training instances. While the model trained on the full dataset understandably sets the performance ceiling at 91.7, QCS achieves over 75% of this peak performance while using only 5% of the data. This highlights an exceptional trade-off between computational cost and model capability.\n\nThe AlpacaEval results further reinforce this conclusion. QCS demonstrates a dominant win rate against both Random selection (511 wins to 168 losses) and IDS, confirming its ability to select a more potent data subset for general instruction-following tasks. Although it loses to the model trained on full data, this is an expected outcome. The key insight is that QCS’s method of reconciling both sequence-level and token-level information allows it to create a coreset that is substantially more valuable than those produced by other data-curation techniques, making it a state-of-the-art approach for targeted, efficient fine-tuning.\n\n**Q2: The result section in the work is somehow weak. I think case studies are always desired for this topic since readers want to understand from specific cases how this method works. I also look forward to more example cases in the rebuttal.**\n\n**A2: ** We thank the reviewer for this insightful suggestion. Providing specific case studies is an excellent way to build intuition for our method. To that end, we present two comparative examples below—one from the GSM-8K benchmark selected in the targeted IT setup for mathematical reasoning; and the other from an Open Assistant-style instruction drawn from Open Assistant 1 to follow the regular IT setup in AlpacaEval. These cases illustrate how Quadratic Coreset Selection (QCS) moves beyond simple sequence-level selection to achieve superior results.\n\n**Case Study 1: Mathematical Reasoning (GSM-8K)**\nConsider two instances from GSM-8K. A standard sequence-level selection by LESS evaluate them based on overall difficulty or gradient influence, but QCS performs a deeper, two-level analysis.\n\n**Instance A: A High-Value, Complex Problem (Selected and Prioritized by QCS)**\n\n*Problem*: \"A garden has 10 yellow flowers. There are 80% more purple flowers than yellow ones, and the number of green flowers is 25% of the yellow and purple flowers combined. How many flowers are in the garden?\"\n\n*Sequence-Level Analysis*: QCS identifies this as a high-value sequence because it involves multiple, dependent steps and requires understanding percentages—a complex reasoning skill.\n\n*Token-Level Analysis* (**The QCS Advantage**): Within the solution, QCS identifies and up-weights the critical tokens that form the logical backbone of the reasoning process.\n\n*Solution with Critical Tokens Emphasized by QCS*: \"First, calculate 80% of the **10** yellow flowers: **0.80** * 10 = 8. The number of purple flowers is **10 +** 8 = **18**. Next, the total of yellow and purple is **10 +** 18 = **28**. Then, calculate 25% of that total: **0.25** * 28 = **7**. Finally, the total number of flowers is 10 + 18 + 7 = 35.\"\n\n*Comparison*: A sequence-level method would treat all tokens in this solution equally. QCS, however, focuses the model's training on the crucial numbers (**10, 18, 28, 35**), operators **(*, +)**, and percentages (**0.80, 0.25**), ensuring the model learns the mathematical procedure, not just the narrative text.\n\n**Instance B: A Simpler, Less Influential Problem (Down-weighted by QCS)**\n\n*Problem:* \"A bakery sold 15 cakes on Monday. On Tuesday, it sold 12 more cakes than on Monday. How many cakes were sold in total?\"\n\n*Sequence-Level Analysis*: A standard method might select this for its basic arithmetic structure. QCS would rank it lower because its reasoning path is simpler and less \"influential\" for teaching an already capable LLM.\n\n*Token-Level Analysis*: The critical tokens are basic (**15, +, 12, =, 27**). While correct, they don't introduce the same level of complexity as Instance A. QCS recognizes that the \"informational density\" of critical tokens is lower here.\n\nBy selecting Instance A and then focusing on its critical tokens, QCS uses its budget to teach the model complex, multi-step reasoning involving percentages, which is far more efficient than re-teaching it simple addition.\n\nCase Study 2: Complex Instruction Following (AlpacaEval / Open Assistant)\nThis advantage extends beyond math problems to general instruction following.\n\n**Instance C: A High-Value, Factual Instruction (Selected and Prioritized by QCS)**\n\n*Instruction*: \"Explain the main differences between nuclear fission and nuclear fusion, and list one practical application for each.\"\nSequence-Level Analysis: QCS identifies this as a high-quality instruction because it demands factual accuracy, comparison, and structured knowledge.\n\n*Token-Level Analysis (The QCS Advantage)*: QCS would prioritize the key concepts, relationships, and entities within a correct response.\n\n*Solution with Critical Tokens Emphasized by QCS:*\n\"Nuclear **fission** is the **splitting** of a **heavy**, unstable nucleus into **two lighter nuclei**. A practical application is in **nuclear power plants**. In contrast, nuclear fusion is the process where two light atomic nuclei combine to form a heavier nucleus, releasing energy. A practical application is the energy source of stars like our Sun.\"\n\n*Comparison*: A sequence-level method would not differentiate between the key terms and the connecting language. QCS focuses the model on the core vocabulary (**fission, fusion, splitting, combine, heavy, light**) and the crucial entities (**nuclear power plants, stars**), ensuring it learns the factual and relational knowledge correctly.\n\n[0*] Wang Y, Ivison H, Dasigi P, et al. How far can camels go? exploring the state of instruction tuning on open resources[J]. Advances in Neural Information Processing Systems, 2023, 36: 74764-74786."}, {"review_id": "OMtqbAtZiw", "reviewer": "Reviewer_Mibm", "summary": "This paper introduces Quadratic Coreset Selection (QCS), a novel approach designed to improve the efficiency of instruction-tuning (IT) for large language models (LLMs). QCS uniquely combines sequence-level and token-level data selection, addressing the limitations of methods that focus solely on one or the other. The paper proposes a probabilistic solver for QCS that bypasses computationally intensive traditional methods, which ensures provable convergence and asymptotic equivalence to the original objective. Experimental results demonstrate QCS's superior data efficiency in various IT scenarios, including targeted and continual IT, which leads to more expressive and better-performing LLMs with reduced data requirements.", "questions": "Please see the concerns noted in the paper's weaknesses.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper introduces a principled approach that jointly optimizes sequence- and token-level data selection, which addresses limitations of treating them separately.\n\n- By reparameterizing coreset weights as Bernoulli distributions and applying hierarchical policy gradients, the method avoids costly gradient computations and scales to large models.\n\n- QCS achieves strong results using only a small amount of training data, often matching or surpassing full-data baselines across multiple instruction-following benchmarks.\n\n- The method is evaluated in regular, targeted, and continual instruction tuning settings, which shows consistent benefits and adaptability.\n\nWeaknesses\n\n- While QCS combines sequence- and token-level selection, it builds heavily on existing coreset and influence-function frameworks, which offers incremental rather than fundamental innovation.\n\n- The paper does not thoroughly investigate or explain scenarios where QCS underperforms (in Table 1, QCS underperforms compared to Full and LESS on MMLU and BBH across both LLAMA-2-7B and 13B).\n\n- Section 7.2 shows that QCS performance is highly sensitive to the elastic parameter ϵ and token selection ratio α; for instance, α = 0.2 yields substantially worse results than α = 0.5 or 1, which indicates fragility.\n\n- Most experiments focus on instruction-following tasks; the generality of QCS to other post-training settings remains untested.\n\n- While the paper claims that QCS is more efficient than LESS, the memory and time comparisons in Table 2 are minimal and lack full context (e.g., training time for LoRA or convergence behavior), which makes it hard to assess practical efficiency gains.", "was_revised": true, "final_justification": "Most of my questions have been addressed by the authors. They have also expressed appreciation for my suggestions and committed to including them in the next version of the paper. I will keep my original positive score because I believe it reflects the paper's current state.", "rebuttal": "Thanks for your concerns and kindly suggestions.\n\n**Q1: While QCS combines sequence- and token-level selection, it builds heavily on existing coreset and influence-function frameworks, which offers incremental rather than fundamental innovation.**\n\n**A1**: We argue for our methodological novelty from three points of view:\n       \n**(1)**, coreset selection frameworks refer to a technical thread in the field of data selection, which is aligned with the case that diffusion-based models refer to the technical thread in the field of deep generative models. We consider the novelty of the techiniques in a method itself rather than the framework behind.\n       \n**(2)**, QCS particularlly differs from existing coreset frameworks from three aspects: **first**, its hierarchical data selection scheme that unifies token and sequence selection in a single framework is original and seamlessly suits our motivation; **second** we provide the new theory to certify its superiority (Theorem.1) compared with any sequence/token coreset selection frameworks designed for IT, which is totally original; **third**, existing coreset selection algorithms can not adapt to QCS due to their heavy computation cost in the LLM background, so that we derive the feasible algorithm for QCS that holds the BP-free property (see Eq.7, Eq.10 in our Appendix.A) and convergence guarantee (Theorem.2,3). \n       \n**(3)**, **QCS holds a nuanced connection with, But Does Not Depend On Influence Function frameworks**. In fact, any bi-level coreset frameworks can be related with influence function since they can be observed from a view of cone constrained optimization problem related with influence functions [6], but they don't depend on (do not have to be solved) as a influence function. \n\n\n**Q2: The paper does not thoroughly investigate or explain scenarios where QCS underperforms (in Table 1, QCS underperforms compared to Full and LESS on MMLU and BBH across both LLAMA-2-7B and 13B).**\n\n**A2**: It is a very good question, which we also concern about. In the experiment of targeted IT, we exactly follow the benchmark setups in LESS, but **their setup can not truly reflect our strength**: QCS's superiority is built on the concern of selecting critical tokens in response, however, the responses in the validation sets (used for targetted transfer tuning) of MMLU and TYDI QA only contain a single token, and BBH's responses are relatively short compared with the pre-training datasets, i.e.   \nFLAN V2 (31.2), COT (53.2), DOLLY (91.3), and OPEN ASSISTANT 1 (212.5), with the content in the brackets indicate the average number of response tokens in the datasets, respectively. Given this observation, in the rebuttal responses A1 for the reviewer beGB and A1 for the reviewer zsf7,  we further provided the experiments with a focus on long response benchmarks, including tasks of open-ended language generation and long CoT reasoning. The results sufficiently verified the QCS's superiority compared with LESS.  \n\n**Q3: Section 7.2 shows that QCS performance is highly sensitive to the elastic parameter $\\epsilon$ and token selection ratio $\\alpha$; for instance, $\\alpha$ = 0.2 yields substantially worse results than $\\alpha$ = 0.5 or 1, which indicates fragility.**\n\n**A3**: Thanks for your concern yet we respectfully do not agree with the ``fragility'' judgement. **The reviewer misunderstood the interpretation in the analysis of $\\alpha$ and $\\epsilon$**, and we argue from three aspects:\n\n**(1), The performance variation of $\\alpha$ in Fig.1 (b) should be large, which exactly verifies the motivation behind our paper**: When $\\alpha$$\\rightarrow1$; QCS will collapse into a pure sequence-level coreset selection formula in Eq3. In this case, if the QCS performs equally with the sequence-level selection in Eq3, the token-level granularity would be meaningless, which is conflicted with our motivation and our theoretical result in Theorem.1. When $\\alpha$$\\rightarrow0$, it means that the selected tokens are sparse so that most of tokens are weighted  $\\rightarrow0$, which can not provide sufficient feedbacks to support intruction tuning.\n \n**(2)**,  **According to our QCS motivation and Theorem.1, the performances when $\\epsilon=1$, $\\epsilon=0.5$ should also fall behind the cases $\\epsilon=0$, $\\epsilon=0.1$.** Specifically, when $\\epsilon=0.5$, it implies all token losses are equally weighted by $0.5$ whatever the tokens were selected or not by QCS; when $\\epsilon=1$, it implies the tokens abandoned by QCS will join the training while the tokens selected by QCS will be abandoned, totally opposed with the QCS's choices. **If the performances in the cases $\\epsilon=0.5,1$ are consistent with $\\epsilon=0,0.1$, it implies that QCS contributes nothing to instruction tuning. It violates our paper's motivation and Theorem.1.**\n\n**(3)** , the performance difference between the cases $\\epsilon=0$ and $\\epsilon=0.1$ is subtle, and $\\epsilon=0.1$ slicely outperform $\\epsilon=0$. It is probably because the soft weights can provide more training feedback compared with the sparse weights. Such evidence supports our strategy in lines 218-223.  \n\n**Q4: Most experiments focus on instruction-following tasks; the generality of QCS to other post-training settings remains untested.**\n\n**A4**: Our QCS in Eq.5 can exactly refers to SFT. SFT is the basic post-training requirement when the other post-training frameworks, e.g., RLHF, RLVR, DPO, etc. It is not only applied for instruction-following tasks but can also influence a wide range of post-training settings as SFT.\n\nBeyond this, QCS in Eq.5 can be also applied in other post-training tasks, e.g., post-training a Multimodal LLM. To verify our claim, we apply a new experiment by post-training a multimodal LLM using QCS. Specifically, we derive the baseline by using the pre-trained Vicuna-7B backbone, then post-training the checkpoints with QCS to select the coreset in the geometric-problem solving data in (Gao J, Pi R, Zhang J, et al.). We setup the sequence selection ratio with 10% and the token selection ration with $alpha=0.5$ and $\\epsilon=0.1$. Then we compare the QCS-derived model with the full-data baseline and the baseline model trained with random selection with the same selection proportion in QCS. Their results refer to Full (100%): 53.4; Random (10%): 35.3; QCS (10%):49.6, such that QCS obtained impressive data efficiency in other post-training tasks.   \n\n**Q5: While the paper claims that QCS is more efficient than LESS, the memory and time comparisons in Table 2 are minimal and lack full context (e.g., training time for LoRA or convergence behavior), which makes it hard to assess practical efficiency gains.**\n\n**A5**: With regards to Table.2, the training time for LoRA in Algo.1 (see the page 7 in our Appendix) is almost consistent with the warmup stage in LESS, yet we need a labeling process (IO process) to prepare Algo.2 (it leads to over a half of computation cost in the warm-up stage in our implementation) and another LoRA in Algo.2 (see the page 7 in our Appendix). Note that the labeling procedure can be replaced by a teacher network that dynamically offers the soft label in Algo.2. This memory-time trade-off may further cuts off a significant amout of computation in GPU hours. \n\nAs for the convergence behavior, since the rebuttal format does not support figures, we verbally discuss them in the rebuttal while add the convergence plots to our next version. Specifically, The charts demonstrate a clear and consistent convergence pattern for Algo2 (the page 7 in our Appendix), where the validation loss in BBH exhibits a distinct U-shaped curve, reaching its minimum around epoch 1-2 before starting to increase due to overfitting. This convergence behavior reveals a crucial practical efficiency gain: optimal validation performance is achieved very early in training. This allows for early stopping, saving significant training time and computational resources beyond the per-step metrics in Table 2, thereby confirming the methods' practical efficiency.\n\n[1*] Gao J, Pi R, Zhang J, et al. G-llava: Solving geometric problem with multi-modal large language model[J]. arXiv preprint arXiv:2312.11370, 2023."}, {"review_id": "x1J01UVJKP", "reviewer": "Reviewer_73Re", "summary": "This paper proposes Quadratic Coreset Selection (QCS) , a novel framework for efficient LLM instruction tuning that simultaneously optimizes sequence-level and token-level data selection. By reconciling both selection granularities through a bi-level optimization objective, QCS achieves superior data efficiency while mitigating token noise and bias. To overcome computational barriers, the authors develop a probabilistic re-parameterized solver using hierarchical policy gradients that avoids prohibitive Hessian inversions. The authors conduct extensive experiments to demonstrate the effectiveness of QCS.", "questions": "1. It seems like the selected strategy requires iterative updates. If the number of iterations is large, I’m concerned whether the computation time will significantly exceed the time for training with Full data using LoRA. Is there any direct data on this?\n2. QCS’s performance is sensitive to the token selection ratio. Is there any plan to design an adaptive strategy?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. The proposed method is quite innovative. QCS innovatively reconciles sequence-level and token-level data selection within a unified bi-level framework.\n2. The probabilistic re-parameterization and hierarchical policy gradients elegantly bypass prohibitive Hessian inversions, which is a clever approach.\n\nWeaknesses:  \nSee \"Questions\" below.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your concerns and suggestions.\n\n**Q1:It seems like the selected strategy requires iterative updates. If the number of iterations is large, I’m concerned whether the computation time will significantly exceed the time for training with Full data using LoRA. Is there any direct data on this?**\n\n**A1**: We discuss this from three aspects:\n\n**(1)**, Despite our formulation (Eq.5) refers to a bi-level problem including inner-loop and outer-loop optimization, **the implementation by our HPGE actually does not require any iterations of inner-loop optimization with respect to $v_{i,j}$, $w_i$** in Algo.1,2. More specifically, our theoretical results in Eq.13 in the paper and Eq.9, Eq.10 in our Appendix.A (lines 28,29) implies the closed-form solution of $v_{i,j}$, $w_i$ can be directly estimated when the losses are given. In our implementation of Algo.1,2, we directly infer and estiamte the value of  $v_{i,j}$, $w_i$ (the first the 'while' loop in Algo.2) along with the LoRA update (the second 'while' loop in Algo.1).\n\n**(2)**, As observed by the discussion in the rebuttal response A5 of the reviewer Mibm, similar with the convergence achieved by LESS, the direct digit of convergence behaviors showed that the early stopping with only 2-3 epoches are sufficient to the LoRA update. \n\n**(3), Most importantly**, in targeted instruction tuning setup, for each given target dataset (e.g., BBH and GSM-8K in A1 of the reviewer beGB and the reviewer zsf7), we don't need to train a LoRA model to select their training subset but just like LESS, we can directly apply the data selection model pre-trained by Algo.2 to select the influential sequences and tokens (<10%) to train the LLM with a downstream-task-specific LoRA layer. Compared with the downstream-task-specific LoRA layer trained with 100% data for each downstream task, it significantly improve the data efficiency.\n\n**Q2: QCS’s performance is sensitive to the token selection ratio. Is there any plan to design an adaptive strategy?**\n\n**A2**: Thanks for your concern yet distinct from your concern, **QCS is actually stable instead of sensitive with regards to $\\alpha$ and $\\epsilon$ since you may misunderstand the interpretation in the analysis of $\\alpha$ and $\\epsilon$**, and we further explain it from five aspects:\n\n**(1), The performance variation of $\\alpha$ in Fig.1 (b) should be large, which exactly verifies the motivation behind our paper**: When $\\alpha$$\\rightarrow1$; QCS will collapse into a pure sequence-level coreset selection formula in Eq3. In this case, if the QCS performs equally with the sequence-level selection in Eq3, the token-level granularity would be meaningless, which is conflicted with our motivation and our theoretical result in Theorem.1. When $\\alpha$$\\rightarrow0$, it means that the selected tokens are sparse so that most of tokens are weighted  $\\rightarrow0$, which can not provide sufficient feedbacks to support intruction tuning.\n \n**(2)**,  **According to our QCS motivation and Theorem.1, the performances when $\\epsilon=1$, $\\epsilon=0.5$ should also fall behind the cases $\\epsilon=0$, $\\epsilon=0.1$.** Specifically, when $\\epsilon=0.5$, it implies all token losses are equally weighted by $0.5$ whatever the tokens were selected or not by QCS; when $\\epsilon=1$, it implies the tokens abandoned by QCS will join the training while the tokens selected by QCS will be abandoned, totally opposed with the QCS's choices. **If the performances in the cases $\\epsilon=0.5,1$ are consistent with $\\epsilon=0,0.1$, it implies that QCS contributes nothing to instruction tuning. It violates our paper's motivation and Theorem.1.**\n\n**(3)** , The performance difference between the cases $\\epsilon=0$ and $\\epsilon=0.1$ is subtle, and $\\epsilon=0.1$ slicely outperform $\\epsilon=0$. It is probably because the soft weights can provide more training feedback compared with the sparse weights. Such evidence supports our strategy in lines 218-223.  \n\n**(4)**, We further provide the experimental results with $\\alpha=0.3, 0.7$. In the cases without token noise injection, when $\\alpha > 0$, the performance variation remains stable (with 0.531 $\\alpha=0.3$ and 0.527 with $\\alpha=0.7$); In the cases with token noise injection (10%), when $\\alpha \\leq 0.5$, the performance variation remains stable (with 0.529 $\\alpha=0.3$ and 0.521 with $\\alpha=0.7$).  It implies that setting $\\alpha \\leq 0.5$ can yield stable and token-noise robust performances.\n\n**(5)**, It is possible to design token-specific adaptation strategies to the hyper-parameters $\\alpha, \\epsilon$, e.g. curriculum learning (Jiang L, Meng D, Zhao Q, et al) or online data reweighting (Sow D, Woisetschläger H, et al). The ideas are considered as our future research direction.\n\n[2*] Jiang L, Meng D, Zhao Q, et al. Self-paced curriculum learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2015, 29(1).\n\n[3*] Sow D, Woisetschläger H, Bulusu S, et al. Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining[C]//The Thirteenth International Conference on Learning Representations."}, {"review_id": "M8ybsB9Id6", "reviewer": "Reviewer_Voi8", "summary": "The paper studies the problem of coreset selection for LLM instruction tuning, both at the sequence and the token levels. The authors propose an optimization problem that optimizes both sequence selection and token selection within the selection sequence. To overcome the prohibitively high computation cost of solving the original optimization problem, the authors proposed a probabilistic variant, which assigns probabilistic densities to the training set. In addition, the authors further extend the proposed method for transfer instruction tuning,", "questions": "- I suggest adding an illustration of the end-to-end algorithm with examples.\n- For the results in Table 4, could you also provide the selection and training costs? If the proposed method consumes significantly more resources in the selection stage than the baselines, it may be unfair to compare under the same selection ratio.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- To the best of my knowledge, this work is the first that focuses on both sequence-level and token-level coreset selection, providing a more fine-grained approach to instruction selection for LLM tuning.\n- The paper demonstrates strong technical depth. The relaxation of the original optimization problem is innovative and reasonable, with a detailed analysis.\n- The extension to transfer instruction tuning shows the broader applicability of the proposed method.\n\nWeakness:\n- While the paper provides extensive technical detail, it lacks sufficient high-level intuition and explanation, which makes it difficult to grasp the main idea. For example, the absence of a step-by-step algorithm and a concrete example makes the method harder to follow and implement.\n- The proposed method does not significantly outperform the baseline method in the transfer tuning setting. The results in Table 1 show that the performance of the proposed method is similar to or even worse than that of LESS.", "was_revised": true, "final_justification": "This paper is technically strong, with solid methodology and contributions that I find meaningful and relevant. The presentation could be improved, particularly in clarity. I will keep my current score.", "rebuttal": "Thank you for your concerns and suggestions.\n\n**Q1: While the paper provides extensive technical detail, it lacks sufficient high-level intuition and explanation, which makes it difficult to grasp the main idea. I suggest adding an illustration of the end-to-end algorithm with examples.**\n\n**A1:** . We have provided the step-by-step algorithms elaborated in the page 7 in our Appendix. Due to the complexity of our piplines, we provide a more illustrative example to demonstrate the main idea of our appraoch. Specifically, let's consider the following two instances in GSM-8K:\n\n**Instance 1 (81 response tokens):**\n \n**Question:**\nAlbert is wondering how much pizza he can eat in one day. He buys 2 large pizzas and 2 small pizzas. A large pizza has 16 slices and a small pizza has 8 slices. If he eats it all, how many pieces does he eat that day? \n\n**Solution:**\nFirst, calculate the total number of slices from the large pizzas: **2** large pizzas ***** **16** slices/large pizza = 32 slices.\nNext, calculate the total number of slices from the small pizzas: 2 small pizzas * **8** slices/small pizza = 16 slices.\nFinally, add the slices from both pizza sizes to find the total number of pieces Albert eats: 32 slices **+** 16 slices = **48** slices.\n\n**Instance 2 (138 response tokens):**\n\n**Question:**\nMark has a garden with flowers. He planted plants of three different colors in it. Ten of them are yellow, and there are 80% more of those in purple. There are only 25% as many green flowers as there are yellow and purple flowers. How many flowers does Mark have in his garden? \n\n**Solution:**\nFirst, determine the number of purple flowers. There are **80%** more purple flowers than yellow ones, so calculate 80% of the **10** yellow flowers: 0.80 ***** 10 = 8. The number of purple flowers is 10 + 8 = **18**.\nNext, find the total number of yellow and purple flowers: 10 yellow + 18 purple = **28** flowers.\nThen, calculate the number of green flowers, which is **25%** of the combined number of yellow and purple flowers: 0.25 ***** 28 = 7 green flowers.\nFinally, to find the total number of flowers in the garden, add the quantities of all three colors: 10 yellow **+** 18 purple + 7 green = 35 flowers.\n\nwhere we highlight the critical response token with the top-10% rating of $v_{i,j}$, which are produced by model trained via Algo2 (i.e., 2, *, 16, 8, +, 48 are the critical tokens in the first response and 80%, 10, *, +, 18, 28, 25%, *, +, 35 are the critical tokens in the second response).  \n\n**High-Level Intuition of QCS:**\n\n Before diving into the example, let's establish the core idea. Instruction Tuning (IT) an LLM with a massive dataset is computationally expensive, and not all data provides equal value. Some examples are simple and redundant, while others are rich and complex. Even within a good example, some parts (like key numbers and operations) are more important than others (like filler words). **QCS** is a principled method to tackle this by acting as a sophisticated, two-level data filter: **Sequence-Level Selection**: First, it scans the entire dataset and selects a small subset of the most \"influential\" or \"high-quality\" instruction-response pairs. This is like a chef choosing only the best dishes to put on a refined menu. **Token-Level Selection**: Then, for each sequence that was selected, it further identifies and prioritizes the \"critical tokens\" within the response. This is like the chef identifying the key ingredients in a dish that truly define its flavor. By combining these two steps, QCS ensures the LLM is trained on a small, highly-concentrated set of the most valuable information, leading to greater efficiency and potentially better performance.\n\n**Illustration with GSM-8K Examples**:\n\nLet's walk through the QCS process, assuming our initial training dataset contains both Instance 1 (the simple pizza problem) and Instance 2 (the complex flowers problem), among thousands of others.\n\n**Step 1: The Initial State - A Large Dataset of Instruction-Response Pairs**\n\nImagine a massive dataset, $D_train$, filled with math problems. Our two examples are present:\n\n**Instance 1 (Pizza)**: A simple, two-step arithmetic problem.\n\n**Instance 2 (Flowers)**: A more complex, multi-step problem involving percentages and dependencies.\n\nThe goal of QCS is to select a small, powerful subset from $D_train$ to fine-tune an LLM without using the whole dataset.\n\n**Step 2: Sequence-Level Coreset Selection (Choosing the Best Problems)**\n\nQCS first analyzes every instruction-response pair in the dataset to determine its value for improving the LLM's overall reasoning ability. It assigns a binary selection weight, w_i, to each instance i. If an instance is valuable, w_i = 1 (select); otherwise, w_i = 0 (abandon).\n\n**Instance 1 (Pizza Problem) is Analyzed:**\n\n**QCS Assessment**: The algorithm evaluates this problem. It recognizes that the reasoning is straightforward (two multiplications and one addition). In a large dataset, there are likely many similar or even simpler problems. The \"learning value\" it provides is relatively low compared to more complex examples.\n\n**Decision**: QCS determines this instance is redundant or not influential enough for efficient tuning. It assigns a sequence weight w_pizza = 0.\n\n**Outcome**: The entire pizza problem is abandoned. It will not be used in the training process.\n\n**Instance 2 (Flowers Problem) is Analyzed:**\n\n**QCS Assessment**: The algorithm evaluates this problem and identifies it as high-quality. It involves percentages (80%, 25%), dependent calculations (the number of green flowers depends on the sum of yellow and purple), and multiple logical steps. This is an \"influential\" example that can effectively teach the model complex reasoning.\n\n**Decision**: QCS deems this instance highly valuable. It assigns a sequence weight w_flowers = 1.\n\n**Outcome**: The flowers problem is selected and proceeds to the next stage of the QCS process.\n\nAt the end of this step, we have filtered our massive dataset down to a small coreset of the most promising sequences. The flowers problem is in this coreset; the pizza problem is not.\n\n**Step 3: Token-Level Coreset Selection (Focusing on Critical Information)**\n\nNow, QCS only looks at the sequences that were selected in Step 2 (i.e., those with w_i = 1). For each of these, it analyzes the solution to identify the most critical tokens. It assigns a binary weight, v_i,j, to each token j in the solution of sequence i.\n\n**Instance 1 (Pizza Problem)**: This instance is ignored in this step because it was already abandoned (w_pizza = 0).\n\n**Instance 2 (Flowers Problem)** is Analyzed:\n\n**QCS Assessment**: The algorithm examines the solution text provided. The goal is to pinpoint the tokens that are essential for the mathematical logic, as opposed to the narrative text.\n\n**Decision & Outcome**: It assigns token weights v_flowers,j = 1 to the critical tokens and v_flowers,j = 0 (or a very small \"elastic\" value $\\epsilon$ as mentioned in the paper) to the non-critical ones. This corresponds exactly to the highlighting you requested:\n\n*First, determine the number of purple flowers. There are 80% more purple flowers than yellow ones, so calculate 80% of the 10 yellow flowers: 0.80 * 10 = 8. The number of purple flowers is 10 + 8 = 18. Next, find the total number of yellow and purple flowers: 10 yellow + 18 purple = 28 flowers. Then, calculate the number of green flowers, which is 25% of the combined number of yellow and purple flowers: 0.25 * 28 = 7 green flowers. Finally, to find the total number of flowers in the garden, add the quantities of all three colors: 10 yellow + 18 purple + 7 green = 35 flowers.*\n\n**Step 4: Efficient Instruction Tuning with the QCS Coreset**\n\nFinally, the LLM is fine-tuned. However, instead of learning from all tokens in all examples, the training process is now focused by the QCS weights.\n\n\n**Q2: For the results in Table 4, could you also provide the selection and training costs?**\n\n**A2:** Our paper directly cite the results of the other baselines ( i.e., IDS and ) reported in their papers, which did not provide the digits about the selection and training costs explicitly. Beyond this, since our data selection and training are coupled with each other (the rebuttal response A1 of the Reviewer 73Re), it is inappropriate to compare them directly. However, we can provide the analysis about the comparison between their selection and training costs and our QCS, with regards to their experimental details. Specifically,\n\n*(1) The comparison of selection cost*: \n\nFirst, it is difficult to compare other baselines with DQ since according to the section 4.1 in their original paper, for the LLM task, the authors use OpenAI's Embedding API as the feature extractor. It is completely blackbox to their inference system and also leads to an unfair comparison with the other baselines.\n\nSecond, for IDS, it only requires two forward passes for each instance to compute the Conditioned Answer Score and Direct Answer Score, which jointly lead to the data selection result. The total cost requires around 104,000 forward passes (52,000 samples × 2 passes) to scan the dataset.\n\nThird, for QCS, it also requires two forward passes for each instance i to compute w_i and v_i. In the regular IT setup, this process is aligned with the LoRA training in Algo.1, and the final model is obtained without Algo.2. We need 4 epoches to finish the LoRA tuning so that also require (52,000 samples × 2 passes × 4) to scan the dataset.\n\n*(2) The comparison of training cost*: In Table.4, the most difference between QCS and others is QCS using LoRA yet IDS and DQ employ full-parameter tuning to achieve the results. It leads to the significant cost of IDS and DQ since they have to achieve the data selection before full-parameter tuning. LoRA tuning is over 20 times more efficient in GPU-hours for one epoch compared to full-parameter fine-tuning, which makes QCS more efficient in practice."}, {"review_id": "1laNrGDceW", "reviewer": "Reviewer_zsf7", "summary": "Quadratic Coreset Selection (QCS) addresses the data efficiency bottleneck in instruction-tuning large language models by jointly leveraging information at both the sequence and token levels. The authors first present a coreset perspective on next-token prediction, demonstrating that existing sequence-level subset selection and emerging token-level mining methods can both be formulated as distinct bi-level optimization problems. They then introduce QCS as a unified objective whose solution set strictly subsumes those of the individual problems—guaranteeing that a model trained on a QCS-selected subset is at least as expressive as one trained using either sequence-only or token-only selections.\n\nSolving the original QCS objective requires inverting Hessians at the scale of the model, which is computationally infeasible. To overcome this, the authors propose a probabilistic relaxation: inclusion indicators for sequences and tokens are modeled as Bernoulli random variables with learnable parameters. These parameters are optimized using a hierarchical policy gradient estimator that avoids backpropagating through the language model. A projection step enforces subset size constraints. The authors prove that this relaxed formulation is asymptotically equivalent to the deterministic QCS objective, and that the projected updates converge under mild conditions.\n\nEmpirical results show that QCS achieves comparable or superior performance to the targeted instruction-tuning baseline LESS using only 5% of the data, while requiring significantly less memory and computation. When the token-level component is enabled, QCS even matches or surpasses full-data performance on benchmarks such as TYDI-QA and BBH. Further experiments demonstrate QCS’s advantages in capturing critical tokens, handling noisy labels, and supporting continual and transfer instruction tuning—highlighting the value of integrating sequence- and token-level influence in a single coreset-based framework.", "questions": "Generalizability beyond current benchmarks\n- You report results on MMLU, TYDI-QA, and BBH. Could you evaluate QCS on at least one additional instruction-tuning benchmark with a distinct task profile—such as GSM8K (math reasoning) or AlpacaEval (open-ended generation)—to test its generalizability? If performance drops in these settings, could you provide an error analysis to help understand where the quadratic coreset formulation may be falling short?\n\nWall-clock efficiency and scaling behavior\n- While Table 3 details memory savings, the runtime benefits remain unclear. You mention that the QCS outer-loop takes approximately 30 minutes, compared to ~6 minutes for LESS warm-up. Could you provide a comprehensive report of end-to-end GPU hours and wall-clock time for QCS, LESS, and full-data fine-tuning at both the 7B and 13B model scales? Additionally, would you consider including a plot of accuracy versus training time across data budgets (e.g., 1% to 10%) to better illustrate the cost-efficiency trade-offs?\n\nTypos and formatting issues\n- There are several textual inconsistencies that may impact readability. Would you be able to fix the following?\n\nLines 61–65: \"OCS is further evidenced...\" → should be \"QCS\".\n\nLines 280–282: \"Benorlli distribution\" → should be \"Bernoulli distribution\".\n\nTable 1 caption: \"fined-tune\" → should be \"fine-tuned\".\n\nLines 269, 280, etc.: Standardize model name as either \"LLaMA-2\" or \"LLAMA-2\" consistently.\n\nLine 220: \"as we previously discuss\" → should be \"as we previously discussed\".", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The authors propose a quadratic coreset objective that unifies sequence- and token-level subset selection. The resulting solution set is provably at least as expressive as either granularity alone, representing a clear conceptual advance for data-efficient instruction tuning.\n\n- A Bernoulli relaxation with a hierarchical policy-gradient estimator is introduced to avoid intractable Hessian inversions while preserving asymptotic equivalence to the original QCS formulation. This makes the method practically feasible on standard hardware.\n\n- The method achieves full-data or better accuracy on TYDI-QA and strong parity on MMLU and BBH using only 5% of the training sequences, reducing memory from approximately 17.7 GB to 176 MB and eliminating tens of GPU-hours typically required for gradient-based data mining—demonstrating strong practical value.\n\nWeaknesses\n\n- The experimental scope is limited, with results reported on only three benchmarks. QCS also does not outperform LESS on TYDIQA or BBH, suggesting that broader validation is needed to establish generality.\n\n- The wall-clock efficiency and latency benefits remain unclear. While memory and compute savings are reported, the outer-loop optimization takes considerably longer than LESS warm-up, making the real-world efficiency somewhat speculative. Due to the introduced quadratic coreset selection scheme, it is not surprised to see the performance may outperform LESS. However, with a more complex scheme, the stability may be worse.", "was_revised": true, "final_justification": "My previous concerns have been addressed by the authors' response and I recommend an accept.", "rebuttal": "Thanks for your comments and valuable suggestions.\n\n**Q1: There are several textual inconsistencies that may impact readability. Would you be able to fix the following?**\n\nLines 61–65: \"OCS is further evidenced...\" → should be \"QCS\".\n\nLines 280–282: \"Benorlli distribution\" → should be \"Bernoulli distribution\".\n\nTable 1 caption: \"fined-tune\" → should be \"fine-tuned\".\n\nLines 269, 280, etc.: Standardize model name as either \"LLaMA-2\" or \"LLAMA-2\" consistently.\n\nLine 220: \"as we previously discuss\" → should be \"as we previously discussed\".**\n\n**A1**: The typos and textual inconsistency would be fixied and the paper would be proofread to ensure the readability in our next manuscript.\n\n**Q2: You report results on MMLU, TYDI-QA, and BBH. Could you evaluate QCS on at least one additional instruction-tuning benchmark with a distinct task profile—such as GSM8K (math reasoning) or AlpacaEval (open-ended generation)—to test its generalizability? If performance drops in these settings, could you provide an error analysis to help understand where the quadratic coreset formulation may be falling short?**\n\n**A2**: In the rebuttal response A1 of the Reviewer beGB, we have provided the experiments on GSM-8K and AlpacaEval using llama-3 as the backbone, which significantly demonstrates its superiority compared with LESS in the targeted IT setup and IDS in the regular IT setup. Here we also provided the experiments on GSM-8K and AlpacaEval with llama-2 7b. The implementation details are consistent with  the rebuttal response A1 of the Reviewer beGB.\n\n|llama-2 7b| GSM-8K | \n|----------|----------|\n|No Fine-Tuning| 14.0 |\n|Full| 30.5 |\n|Random| 17.0 | \n|LESS| 21.0 | \n|QCS (ours)| 25.3 |  \n\n **AlpacaEval** (win;tie;lose): \n\nQCS vs Full: 378:65:357\n\nQCS vs Random: 578:47:175\n\nQCS vs IDS: 429:103:268\n\nObserve the experimental results in A1 of the Reviewer beGB using Llama-3 8b as the backbone, and our results using Llama-2 as the backbone above. They strongly reinforce the conclusions drawn from the Llama-3 experiments, demonstrating the robustness and versatility of our QCS algorithm across different model architectures. While the absolute scores are naturally lower due to the less capable base model, the relative performance gains of QCS remain remarkably consistent.\n\nOn the GSM-8K benchmark, QCS achieves a score of 25.3, which is a significant improvement over both the competitive LESS baseline (21.0) and Random selection (17.0). Impressively, this score represents over 80% of the performance achieved by fine-tuning on the full dataset (30.5), showcasing exceptional data efficiency. This mirrors the trend observed with Llama-3, where QCS also substantially outperformed other data selection methods. The key insight is that the performance gap between QCS and other methods holds, proving that QCS’s ability to select a superior data coreset is a fundamental advantage of the algorithm itself, not just a byproduct of a powerful base model.\n\nThe AlpacaEval results further confirm this. QCS maintains a dominant win rate against both Random selection and IDS, and remains highly competitive even against the model trained on the full dataset. This consistency across both a cutting-edge model like Llama-3 and an established one like Llama-2 validates that QCS provides a fundamentally better approach to data curation for targeted instruction tuning, delivering superior performance regardless of the underlying model's initial capabilities.\n\nThe reviewers may be curious about the different behaviors of QCS in GSM-8K (math reasoning), and AlpacaEval (open-ended generation). The reasons why QCS significantly outperform LESS in these dataset, probably due to their data composed of long responses, such that QCS can yield the significant token-level data selection. Instead, MMLU and TYDI QA almost contain a single token in the response of their data, and BBH is relatively short and contains less information in the responses of their data. They are not so suitable as a targeted dataset to justify the superiority behind our methodology (it can also explain some weak improvement with regards to the results of MMLU in Table.4). Despite so, the performances of QCS in these benchmarks still rival the results in LESS   \n\n**Q3: While Table 3 details memory savings, the runtime benefits remain unclear. You mention that the QCS outer-loop takes approximately 30 minutes, compared to ~6 minutes for LESS warm-up. Could you provide a comprehensive report of end-to-end GPU hours and wall-clock time for QCS, LESS, and full-data fine-tuning at both the 7B and 13B model scales? Additionally, would you consider including a plot of accuracy versus training time across data budgets (e.g., 1% to 10%) to better illustrate the cost-efficiency trade-offs?**\n\n**A3:** Table 3 refers to the targeted instruction tuning setup that consists of three stages. The warm-up stage, which refers to the computation caused by Algo.1 to obtain the discrete value assignment to $v_{i,j}$ and $w_{i}$, which can not be directly applied to select the data subset in the target benchmarks, e.g., GSM-8K, BBH, etc. To achieve the transfer, we need to train a ``labeling machine'' that learns to predict the soft value between 0,1 for $v_{i,j}$ and $w_{i}$, and such *labeling machine* is trained with hard labels either 0 or 1 obtained in Algo.1. The Algo.1 and Algo.2 cost around 8 and 6 GPU hours, respectively; while the labeling process require around 16 hours to prepare for Algo.2. If we use the model yielded by Algo.1 as the teacher model, we can skip the labeling process, so that the warm-up time can be reduced to ~14 GPU hours compared with 6 GPU hours required for LESS in its warm-up stage. The benefit of QCS is the stage 2, which fast adapt to the few-shot instances in the validation set then directly predict the soft-value assignment of $v_{i,j}$ and $w_{i}$ through the *labeling machine* obtained by Algo.2 in our warm-up stage. It significantly outperforms the treatment requiring gradient feature computation in LESS. In the final stage, we rank the target dataset by the soft value of $w_{i}$ to obtain the top-5%, then assign them with $v_{i,j}$ to achieve the IT process in the targeted IT setting.\n\nWe will further clarify this process and its comparison with LESS in our next version.\n\nBeyond this, we would and tend to include a plot of accuracy versus training time across data budgets (e.g., 1% to 10%) in our next version, because the rebuttal format is hard to achieve the proper presentation."}]

Author rebuttal:

Meta-review: This paper studies data selection for efficient LLM instruction tuning. The authors propose an approach that combines both sequence-level and token-level data selection. The problem is formalized as a bi-level optimization problem, and the authors introduce a probabilistic re-parameterization solver to efficient solve it. Experimental results demonstrate the effectiveness of the proposed approach. After the rebuttal, all reviewers leaned toward acceptance.
Decision: Accept (poster)