SPICE: Submodular Penalized Information–Conflict Selection for Efficient Large Language Model Training
Authors: Powei Chang
Jinpeng Zhang
Bowen Chen
Chenyu Wang
Chenlu Guo
Yixing Zhang
Yukang Gao
JianXiang Xiang
Yue Gao
Chaoqun Sun
Yiyi Chen
Dongying kong
Venue: ICLR 2026 Poster (2026)
Keywords: Data selection; Submodular; Log-determinant Fisher information; Instruction tuning
Primary area: foundation or frontier models, including LLMs
TL;DR: We prove gradient conflicts accelerate marginal log-det FIM decay through ε-analysis , and introduce SPICE—an adaptive conflict-penalized greedy selector that matches full-data results with 10% data.

Abstract: Information-based data selection for instruction tuning is compelling: maximizing the log-determinant of the Fisher information yields a monotone submodular objective, enabling greedy algorithms to achieve a $(1-1/e)$ approximation under a cardinality budget. In practice, however, we identify alleviating gradient conflicts, misalignment between per-sample gradients, is a key factor that slows down the decay of marginal log-determinant information gains, thereby preventing significant loss of information. We formalize this via an $\varepsilon$-decomposition that quantifies the deviation from ideal submodularity as a function of conflict statistics, yielding data-dependent approximation factors that tighten as conflicts diminish. Guided by this analysis, we propose SPICE, a conflict-aware selector that maximizes information while penalizing misalignment, and that supports early stopping and proxy models for efficiency. Empirically, SPICE selects subsets with higher log-determinant information than original criteria, and these informational gains translate into performance improvements: across 8 benchmarks with LLaMA2-7B and Qwen2-7B, SPICE uses only 10% of the data, yet matches or exceeds 6 methods including full-data tuning. This achieves performance improvements with substantially lower training cost.
Code is available at https://github.com/Chang-pw/SPICE#.

Published as a conference paper at ICLR 2026
SPICE: SUBMODULAR PENALIZED INFORMA- TION-CONFLICT SELECTION FOR EFFICIENT LARGE
LANGUAGE MODEL TRAINING
Powei Chang
Jinpeng Zhang"
Bowen Chen
Chenyu Wang
Cheniu Guo Yixing Zhang
Yukang Gao
JianXiang Xiang
Yue Gao
Chaoqun Sun
Yiyi Chen
Dongying Kong'
Bilibili Inc.
Equal contribution
i Corresponding author
Correspondence to:
.ooggoogggn..ggg @ @iliibili.cm
ABSTRACT
Information-based data selection for instruction tuning is compelling; maximizing the log-determinant of the Fisher information yields monotone submodular ob- jective, enabling greedy algorithms to achieve (1 - 1/e) approximation under a cardinality budget. In practice, however, we identify alleviating gradient conflicts, misalignment between per-sample gradients, is a key factor that slows down the decay of marginal log-determinant information gains, thereby preventing signifi- cant loss of information. We formalize this via an e-decomposition that quantifies the deviation from ideal submodularity as a function of conflict statistics, yiclding data-dependent approximation factors that tighten conflicts diminish. Guided by this analysis, we propose SPICE, a conflict-awar selector that maximizes in- formation while penalizing misalignment, and that supports early stopping and proxy models for efficiency. Empirically, SPICE selects subsets with higher log- determinant information than original criteria, and these informational gains trans- late into performance improvements: across 8 benchmarks with ILaMA2-7B and Qwen2-7B, SPICE uses only 10% of the data, yet matches or exceeds 6 methods including full-data tuning. This achieves performance improvements with sub- stantially lower training cost. Code is available code.
Concept: Decay marginal
Empirical: Decay marginal gains
1.-
Slow decay
low-conffict high-conflict
Fast decay
Slower decay + larger total info
2.5-
Lower conflict
Marginal gain a.t
Marginal gain a,t
larger total info
36-
6.6
step
step
Figure 1: (a) Concept. At step f, the marginal information gain A is the incremental increase of Fisher Information utility when adding one sample under the current set S. Slower decay of A, yields larger cumulative information under the same budget k. (b) Empirical. A low-conflict selection (conflict measured by negative cosine alignment to the mean gradient) exhibits slower decay and thus higher information utility at equal budgets.
INTRODUCTION
The remarkable capabilities of large language models (LLMs) have fundamentally transformed nat- ural language processing. However, their performance critically depends on the quality and compo- data (Albalak al., 2024). Instruction tuning has emerged
sition of instruction-tumi paradigm for improving both performance and alignment by fine-tuning with instruction- respooss pairs (Chang et al., 2023). Surprisingly, increasing the amount of training data does not always yield
effective

---

Published as a conference paper at ICLR 2026
better performance (Albalak et al., 2024; Zhou al., 2023b; Li et al., 2024b). Recent empirical studies reveal a striking phenomenon: in instruction tuning, training on only 10-20% of the total data can outperform using the full dataset (Xia et al., 2024b). This observation raises fundamental question: how can we match or surpass full-data tuning performance with less data?
promising solution to the high cost of large-scale training. Recent advances demonstrate notable gains: LESS (Xia et al., 2024a) attains 90% of full-data performance with only 5% of training samples, while SelectIT (Liu al., 2025) achieves comparable results with 10-20%. Among these approaches, gradient-based selection methods show great promise. In particular, approaches based on the -og-determinant of the Fisher Information Matrix (log-det FIM) offer principled formulation and certain theoretical guarantees via submodularity (Fisher. 1922; Deb et al., 2025). However, these methods suffer from heavy computational overhead (e.g., some- times exceeding 100 GPU-hours) and offer only partial theoretical explanations for their empirical behavior, leaving a noticeable gap between theory and practice.
Data selection has emerged
In information-based selection, this gap is particularly evident in greedy selection with FIM as the objective (Deb et al., 2025). While theoretical guarantees suggest consistently near-optimal perfor- mance, practitioners observe that greedy selection often deteriorates far more rapidly than predicted. In fact, some selections perform substantially worse than the bound implics, and the discrepancy be- comes particularly pronounced once the selected subset exceeds certain threshold. At this point, marginal (information) gain, i.e., the incremental increase in the log-det FIM when adding one sample under current set, collapse rapidly despite theoretical assurances of gradual diminishing returns. When marginal information gains diminish more slowly, they yield larger total information under cardinality budget. Such observations naturally prompt central question: what causes this theory-practice gap, and how can it be bridged?
Our key insight addresses the missing factor: the decay rate of marginal information gains. The log- det FIM is indeed submodular (Lovász, 1983), ensuring diminishing returns. However, its marginal information gains do not decrease uniformly. Instead, the decay rate varies substantially depend- ing on interactions among sample gradients. We demonstrate that this variability is governed by gradient conflicts. These occur when gradients of different samples are poorly aligned (Liu al., 2024a). Importantly, we go beyond prior gradient alignment work by establishing direct connec- tion between conflicts and submodular data selection theory, rather than focusing solely on training dynamics.
We provide a novel theoretical framework that quantitatively links gradient conflicts to marginal information gains decay. Our analysis splits marginal information gains into submodular bascline and a perturbation term. We prove that the magnitude of this perturbation, which determines how quickly the decay of marginal information gains, is bounded by the sum of squared gradient inner products. When gradients are highly misaligned, this perturbation grows, accelerating information gains decay. This result explains why greedy approximation guarantees weaken in practice: con- flicts accelerate the erosion of marginal information gains. Importantly, this does not challenge the submodularity of the log-det FIM, but rather provides deeper understanding of its practical behavior.
Building on this theoretical framework, we design conflict-aware greedy selection algorithm: SPICE (Submodular Penalized Information Confflic sEllction). SPICE maximizes the log-det FIM objective while adaptively penalizing gradient conflicts. It incorporates data-driven early stopping rule that selects an appropriate subset size marginal information gains decay. This de- sign preserves computational efficiency while yielding substantially higher-quality subsets. Finally, we conduct extensive experiments to validate the effectiveness of our approach.
Our contributions can be summarized as follows:
(1) Theoretical Framework. To our knowledge, we establish the first quantitative e-analysis link- ing gradient conflicts to the decay marginal FIM gains. We provide formal bounds suggesting that the effective submodularity curvature, which governs greedy approximation quality, is directly con- trolled by gradient conflict intensity, bridging the incory-practice gap for subsequent optimization. These theoretical insights are also validated empirically, demonstrating their practical effectiveness.
(2) Cowflict-Aware Algorithm. Guided by this theory, we propose SPICE, a greedy selection method that adaptively penalizes conflicts rather than discarding informative samples. SPICE
2

---

Published as a conference paper at ICLR 2026
achieves efficient selection complexity of D[d) and demonstrates consistent improvements state-of-thee -frt--rf- baselines across 8 benchmarks while training on only 10% of the data with a total cost of 20 GPU-hours. Overall, demonstrate empirically that such larger information translates into practical gains on downstream tasks.
Beyond practical gains, work highlights central insight: gradient conflicts underlying the accelerated diminishing returns in data selection. By making the decay rate of marginal information gains explicit and tractable, we offer both sharper theoretical understanding and practical tool for constructing compact, high-quality training sets. This framework opens new directions for understanding and improving data efficiency large-scale training, even billion-scal LLMs, with potential applications spanning computer vision, reinforcement learning, and multi- modal learning.
key factor
THEORETICAL GUARANTEES
Gradient-basee data selection methods frequently fall short of their theoretical guarantees. Existing methods like FisherSFT (Deb et al., 2025) rely on greedy maximization the FIM, which enjoys strong submodularity guarantees. Yet practitioners observe performance degradation that deviates from the smooth diminishing returns that theory suggests. Our key insight is that sample interactions lead non-uniform decay in marginal gains, accelerating performance degradation beyond classical bounds. We formalize this through a novel e-decomposition that explains the theory-practice gap and provides data-dependent bounds for better selection algorithms.
2.1 PRELIMINARIES
1:) instruction-responss pairs, aiming to find subset DD with |S] S & that maximizes learning efficiency (e.g., downstream performance).
We consider data selection from D = {(x,,)))
Definition 1 (Fisher Information Utility). For each sample let 9; = Vol(( U:); 0) denote the gradient vector. The empirical Fisher information matrix (FIM) is Fs = Lies 9:9; and the utility function is:
F(S) = log det(I + aFs),
where a > O ensures positive definiteness.
For computational efficiency in large models, also consider the AdaFisher variant (Gomes al., 2025) that reduces complexity from (dd) to Old) (details in Appendix A), while preserving the same utility structure.
Definition 2 (Submodular Function). A set function F: 2 - IR. is submodular if all A C B C
D and x E D \ B, the marginal gain (S) & F(S U [x}) - F(S) satisfies:
(A 2 (B))
(2)
This diminishing returns property ensures that greedy algorithms achieve a (1 11-12 1/e) approximation the optimal solution Theorem 2 (Nemhauser et al., 1978).
2.2 3-DECOMPOSITOON BEYOND STANDARD SUBMODULARITY
While the Fisher utility F(S) = log (I (I aF s) is indeed submodular (proof in Appendix B), this standard result alone cannot explain why greedy selection often deteriorates so rapidly in practice. The limitation is that submodularity only guarantees diminishing returns, but says nothing about the rate of this decay. Our key insight involves decomposing each marginal gain into two compo- nents: modular baseline that captures the intrinsic value of each sample, and perturbation term that quantifies how sample interactions affect marginal utility.
Definition 3 (base e-decomposition). For any E D and S D, we define the modular baseline
base, A log 1l)g=11²),
and the perturbation as
1 + a g + aFs) 9z + |2
EA(S) (S) base, log
(4)
3

---

Published as a conference paper at ICLR 2026
Theorem 1 (e-submodularity) The E-decomposition reveals that submodularity of F is entirely driven by the perturbation terms: for any sets A - B D and element x: E D B,
- -A)
|base. basez]
=x
BBB200 0. (5)
cuptures diminishing returns
Since (S) S O and is non-increasing as |S| grows, its value on large set S summarizes how much the marginal gain of x has decayed from its baseline at S = 23. Indeed, along any chain So C C ST we have
(0) - (S) --(0))-)())-((A((5))) ((A,(S-)))
- -E(a((s))) en (S.) - ES), (6)
80 e (S)) is exactly this cumulative decay of the marginal gain of x.
2.3 IMPROVED APPROXIMATION VIA E-CONTROL.
The e-decomposition allows improvement beyond the standard (1-1/6 1/e) approximation guarantee. The key insight that controlling the perturbation magnitude |e=(5)) improves the approximation quality through the submodular curvature.
Theorem 2 (Classical greedy guarantee). Let F be normalized, monotone submodular function and let Sgredd be the set returned by the greedy algorithm under a cardinality constraint k. Then
F(Sgrendy) 2 (1- F(S*),
where S an optimal k-subset. The full proof is provided in Appendix C.
This provides as the baseline approximation factor. However, tighter guarantees are possible when the function exhibits small curvature, which motivates our key theoretical contribution.
Theorem 3 Survature-dependee guarantee). Let F be monotone submodular and normalized.
Define the total curvature as:
A, (D [0,
L min ZED
(8)
A,
then the greedy solution S Sgrendy under budget is satisfies:
F(Sgreedy) 2 1 F(S*).
(9)
When e this recovers the classical (1 (1-18 (1-1/e) bound; as - the factor approaches 1.
Intuitively, measures how much marginal gains can decrease due element interactions. The proof uses multilinear extension techniques (Feige & Vondrák, 2010; Conforti & Cormuéjols. 1984)
and is detailed in Appendix D).
Our main result links the e-decomposition to control of the submodular curvature. Lemma 1 (Curvature upper bound via E). For the Fisher utility with the decomposition A, = basc + (S) and basc, (2)),, we assume basc, for all x and & S. Then:
c= min base, + 62 (D) \ (x)) =-min min Ex Ez (x))
1e Smax
(x))]
(10)
basc,
base,
base,
In particular, the total curvature is governed by the worst-case normalized perturbation (Ex(()
(z) //ass,,,,,,, so larger perturbations on the full set D \ {x) correspond to larger curvature and hence weaker greedy guarantees.
This establishes that controlling lez (S)) can improve approximation guarantees by reducing the curvature. We next bound (S') in terms of gradient inner products: Theorem 4 (Perturbation bound via gradient alignment). standard assumptions', the pertur- bution term satisfies:
a² EvEs (ices(oT9p 92)²
[xx( S C.
with & S,
(11)
1+a ++aalgell'
where C' is a problem-depentee constant and the proof is provided in Appendix E.
For the assumption, it is used only in App. E for the Neumann series closed- form bound: it is not required for our submodularity e-decomposition. The boundary holds in 90% of steps and the breaking may still arise in practice (10%), see App. E.4.1 for detailed analysis.
4

---

Published as a conference paper at ICLR 2026
Gumulative marginal ubilit
between corfict and manginal
Lanvcanfidd
-038
Nomm DLMNC
50000
-082
poieflcct
-028
Mean 6H-A7
Seat
Relative
Spnammn cometation confict and varpp
0.21
projection
008
8.6-
6.4
004
0.2
0.8
200.6
Information
Data Gradient Direction
10
20
42
Average Gradient Step Cirectinn
Step1
(b1
[6)
Figure 2: (a) Gradient Visualization: The direction and information distribution of two represen- tative data points in gradient 2D and 3D spaces, indicating that there are always gradient conflicts with high information in the data; (b) Decay of the high/low conflict subsets: Low conflicts can bring greater information in a limited number of steps; (c) Spearman correlation analysis: The over- all conflict is negatively correlated with marginal gain A, and positively correlated with
Corollary (Data-dependent approximation guarantee). Combining the above results, greedy se- lection achieves:
F(Sgredy)) 1-6- FF(s)))
(12)
where è x max (y 9,) Smaller inner products yield better approximation guarantees.
Our e-decomposition framework reveals that sample interactions, quantified by gradient inner prod- ucts [S) x Eyes(3 (9. my)?, are major driver performance degradation in greedy selection. Smaller gradient inner products lead to smaller perturbations, lower curvature, and stronger approximation guarantees. This provides principled foundation for designing selection algo- rithms that control sample interactions to achieve better performance.
EMPIRICAL ANALYSIS
Our theoretical analysis predicts that gradient inner products (g 92 upper-bound and are strongly associated with the perturbation magnitude |e (S) and influence approximation quality via curva- ture. We now empirically access these predictions by measuring gradient conflicts and their impact on data selection performance.
To access the theory, we quantify gradient conflict via cosine alignment against selection-adaptive reference that capture the intuition behind gradient inner products, At iteration t, for gradients m from the current candidate set (size m). with mean Ø: |s. Exes.,, Ix, we define:
Definition (Gradient Conflict), Alignment is measured as cosine similarity, and conftict quantifies the degree of opposition to the mean direction: Confltic(g.)) = max(00 -Align(gi)),
9. 9
Align(g:)
(13)
[9. Illll +
where 10 ensures numerical stability (Yu et al.. 2020; Liu et al., 2024b).
Figure (a) reveals that conflicting gradients prevalent and often carry substantial Fisher infor- mation, necessitating a principled trade-off between information gain and conflict minimization.
Conflict-driven utility decay. Splitting samples by conflict level (top/bottom 20%), observe marked differences (Appendix F.2) in sacherr-reedy performance. Low-conflict groups achieve sig- nificantly higher cumulative utility and slower marginal decay, while high-conflict groups exhibit rapid diminishing returns (Figure (b)). Notably, marginal gains reach half-life (50% of the first- step A) within 10-30 steps, suggesting early stopping strategies for practical efficiency.
5

---

Published as a conference paper at ICLR 2026
Quantitative correlation analysis. Direct correlation analysis validates core theoretical pre- dictions. Step-wise Spearman correlations reveal strong negative correlation between conflict and marginal gain (p -0.792) and strong positive correlation between conflict and perturbation mag- nitude [s)] (p = 0.901), as shown in Figure 2 (c). These results provide direct support for Corollary and demonstrate consistency across multiple datasets.
These empirical results strongly validate our e-decomposition framework: gradient conflicts sys- tematically drive perturbation growth and accelerate marginal utility decay. Building on this validated understanding, we next develop SPICE, a conflict-aware selection algorithm that explic- itly balances information gain with conflict minimization. To more comprehensively demonstrate the validity of our theoretical analysis. we conducted additional detailed experiments (Appendix F).
4 METHODOLOGY
We propose a practical pipeline that opera- tionalizes the theoretical insights in Section 2 into a conflict-aware data selection method. The core idea is to perform conflict-aware greedy selection that balances log-det Fisher in- formation gain and gradient alignment. In ad- dition to fixed greedy budget k, we also em- ploy adaptive early-stopping criteria (Algo- rithm 1).
Algorithm SPICE Data Sclection
Require: Dataset D, conflict penalty À 0,
budget k:
Ensure: Selected subset S
1: Initialize: S f- 0
2: Compute gradients {9:} using proxy model
3: for 1= to h: do
4:
for each x E D \ do
A, + FIM FIM.MARGINAL(T, S) 5: conflict
4.1 CONFLICT-AWARE GREEDY SCORE
max (0, coosin similarity(0z: [s)] score(x) + A, conflict
7:
At iteration with current set S and
8:
end for
per-sample gradients {92} computed at fixed checkpoint, flict penalty increases when the sample points against D: A simple and effective choice to apply the hinge function to the nega- tive alignment (Def. 4): conflict(: S,-1) - max (0. align(x 1.-1)). Let (S be the Fisher marginal (Def. 2). We score each candidate x by
define that the con-
9:
x² +- arg max, score(x)
10:
if A, reaches the early stopping criteria
then
11:
break
12:
end
13:
SUSU
14: end for
return S
score(#| (S.1) (S A conflict( (S-1),
220 0.
(14)
Intuitively, the first term favors high-informann samples, while the second discourages strong di- rectional conflicts with the current update. Crucially, we do not discard high-value but conflicting samples: as long as A₂ (S,-1) is sufficiently large, the conffict-aware greedy score can remain com- petitive and select them. This mirrors our theory: penalizing conflicts tends to reduce perturbation |e] and curvature (Theorem Corollary 1) without eliminating informative conflict-direction data.
In Figure 6, we provide a more intuitive visualization. When training with standard greedy search (A=0) that targets only high information gain, the step gradient directions often conflict, which not indicative of stable convergence or performance improvement.
4.2 GREEDY STOPPING CRITERIA
Let 2t arg max score(æ) S.-1) and m, A,, (S,-1) be its marginal gain A. Our method supports two stopping criteria to accommodate different experimental requirements: an absolute threshold on the best marginal A,, or adaptive rule with E 10,
Adaptive Early Stopping (Data-driven). Instead of fixing the greedy budget is, we stop adap- tively once diminishing returns are observed. Let Ar; (So) be the marginal gain of the first selected sample 1T. We stop at the first step tstp where the current marginal gain satisfies
Stop
min L: An, (St-1) S w. An, (5o)
(15)

---

Published as a conference paper at ICLR 2026
Le., once the marginal contribution of adding a new sample falls below w times the first-step gain. This ensures that selection terminates when additional samples contribute little new information, so the effective budget herr
Esop $ k.
Fixed Budget Stopping (Budget-controllee). For fair comparison with baselines, we also sup- port fixed-budget mode: selection continues until exactly k samples are chosen, maximizing in- formation utility under the budget. This ensures consistent dataset sizes across different selection strategies while still benefiting from our conflict-aware scoring mechanism.
We report fixed-budget (equal-k) results in the main experiment Section 5.2 for fairness; the adaptive rule SPICE+ (with 0.5) and its equal-budget reference appear in Appendix H.3.
4.3 EFFICIENT IMPLEMENTATION SELECTION
Due to computational constraints of performing gradient- based selection directly with large-scale models during training, we adopt a proxy-based selection approach. Specifically:
Proxy Model Selection. We employ smaller model (e.g., a 0.5B model) with the same architec- ture target models (LLaMA2-7B (Touvron al., 2023) and Qwen2-7B (Yang et al., 2024a)) to perform the conflict-aware greedy selection. This approach leverages the observation that gradient- based selection patterns transfer effectively across model scales (Xia al., 2024a; Liu al., 2025).
Selection Schedule. The data selection process operates periodic schedule: data selection occurs continuously but training 7 steps are performed periodically. Within one candidate pool, at each greedy iteration, we select one sample using our conflict-aaaee greedy algorithm and add it to an accumulating subset. Every T E (1, 10, 50) iterations, we perform a training step on the accu- mulated subset, then reset the selection buffer for the next cycle maintain training efficiency (Lin et al., 2025).
This design improves the efficiency of data selection while maintaining downstream performance. We present detailed ablation studies in Section 5.4 to validate the effectiveness of this approach.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
To evaluate SPICE's effectiveness, we compare against state-of-the-art data selection methods across diverse instruction-tuning scenarios. Details appear in Appendix G.
Data and Evaluation. We construct 97.5K training corpus spanning mathematical reasoning (GSM8K (Cobbe al., 2021)), code generation (Alpaca Code (Chaudhary, 2023)). and general knowledge (ShareGPT (Lu et al., 2023), Alpaca (Taori et al., 2023)). Evaluation covers 8 bench- marks across reasoning (GSM8K (Cobhe et al., 2021), BBH (Suzgun et al., 2022), ARC (Clark al., 2018)), knowledge (MMLU (Hendrycks et al., 2021), TruthfulQA (Lin et al., 2022), IFEval (Zhou et al., 2023c)), and coding (HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021)).
Baselines. We select several representative data selection methods as baselines: Full data, Random Sampling (Xia et al., 2024b), Instruction- Folllooinn Difficulty (IFD) (Li et al., 2024b), Fisher (Deb et al., 2025), LESS (Xia et al., 2024a), SelectIT (Liu et al., 2025) DPP (Zhang et al., 2023), TSDS (Liu et al., 2024d) and LEAD (Lin et al., 2025). In the subsequent settings, will fix the data number and record the time for selection to compare with our SPICE. In addition, we also evaluated the of SPICE+ (adaptive stopping with w=0.5; Section 4.2). Because its selected-set size may differ from 10%, we report SPICE+ results separately in Appendix H.3 for completeness.
Settings. To ensure fair comparisons, each baseline is trained using 10% of the data selected by its own criterion. For SPICE, we also target 10%: from each 120-sample candidate pool we select k=12 samples per cycle (default T=10, A= Section 4.2). For reproducibility. detailed training configurations and random seeds. All experiments are repeated three times, and the reported results averaged to ensure reliability. After selecting data, to prevent data leakage,
will also release the

---

Published as a conference paper at ICLR 2026
have chosen the previously widely-used LLaMA2-7B (Touvron et al., 2023) and Qwen2-7B (Yang et al., 2024a) as our base models, and perform tinettuning using LoRA (with settings: r=16, a = 32, and dropout = 0.05) H20 GPUs.
5.2
MAIN RESULT
Methods (Qwen2-7B)
Bench. Sample
Full 100%
Random 10%
IFD 10%
Fisher 10%
LESS
SeleetIT
DPP
TSDS 10%
LEAD 10%
SPICE
10%
10%
10%
10%
Reasoning Ability
GSM8K
84.2
84.9
85.8
86.5
82.3
83.6
86.5
86.0
84.5
86.7
+2.5
BBH
61.3
60.8
60.2
60.8
61.0
60.9
61.0
60.0
61.0
61.0
-0.3
Generul Multi-task Knowledge
MMLU
65.7
63,4
63.8
65.2
66.1
64.7
66.0
63.7
66.0
67.1
+1.4
ARC-C
50.5
49.6
50.1
50.3
49.5
50.3
$1.0
50.4
50.8
51.8
+1.3
TruthfulQA
54.8
54.8
55.2
55.0
54.3
54.4
55.0
55.0
54,9
55.5
+0.7
[FEval
33.5
28.3
27.4
30.6
26.0
30.8
35.4
28.0
33.0
38.6
+5.1
Code Generation
HvaanEval 45.7
45.7
46.3
44.5
46.3
46.3
45.0
46.2
46,1
47.1
+1.4
MBPP
55.2
56.1
54.2
54.6
56.2
55.2
55.7
54.5
55.6
56.2
+1.0
Average
56.4
55.5
55.4
55.9
55.2
55.8
57.0
55.5
56.5
58.0
+1.6
Table Evaluation data selection methods on Qwen2-7B. Bold indicates the best result, underline the second best. A denotes the gain of SPICE over full-data tuning. Overall, SPICE (Ours) achieves superior performance on reasoning, knowledge, and code generation tasks with lower time cost.
Following the settings described in Section 5.1, we conduct data selection using various methods and then perform distributed training on LLaMA2-7B and Qwen2-7B using LlamaFactory (Zheng et al., 2024). Finally, we evaluate the models on benchmarks with results of Qwen2-7B shown in Table and LLaMA2-7B shown Table
For Qwen2-7B, our method achieves the best overall performance with an average score of 58.0, outperforming full-data training (56.4) and all baseline methods. Notably, SPICE uses only 10% of the training data while delivering consistent improvements across diverse tasks. The method demon- strates particular strength in reasoning and knowledge-intensiee tasks, with 7 out of 8 benchmarks showing best performance, especially IFEval with +5.1, and the remaining achieving second-best result. For aMA--BB,, it still maintains good performance while using less data. can be seen from this that I. LaMM2 has lower overall score compared to Qwen2, but our data selection method still maintains performance, especially on General Multi task Knowledge. In addition, it still yields strong performance when fine-tuning 70B+ models (see Section H.4). Beyond the strong results at the 10% budget, SPICE also performs well in tiny data even with only 1%/5% the data (see Section H.6). But for data selection method, not only does it need to maintain performance, but it also requires less cost. Next, we will conduct a cost analysis.
5.3 COST ANALYSIS
For data selection problems, we must focus reducing computational costs while matching or improving full-data training performance. Therefore, we recorded the normalized time costs of selection, and compared the differences between the two stopping criteria (SPICE vs SPICE+).
As shown in Figure (a), SPICE and SPICE+ can still achieve good performance (surpassing Full) while maintaining low time costs in data selection. SPICE+ yields slightly better accuracy than SPICE, at the expense of higher selection time due to adaptive stopping. Moreover, in terms of both overall selection and training time, total time is lower than that of full-data. In terms of computational complexity, the selection procedure requires O D)d) time with O ((dd) peak memory. The specific time cost and computation complexity provided in Appendix H.7 and H.8.
8

---

Published as a conference paper at ICLR 2026
Methods (LLaMA2-7B)
Bench. Sample
Full 100%
Random 10%
IFD 10%
Fisher 10%
LESS
Select[T
DPP
TSDS 10%
LEAD
SPICE
10%
10%
10%
10%
10%
Reasoning Ability
GSM8K
13.6
12.9
12.7
13.6
13.6
12.5
13.6
12.8
13.4
13.8
+0.2
BBH
41.3
39.6
39.9
40.5
40.0
39.7
40.2
40.0
40.8
40.8
-0.5
General Mulli-task Knowledge
MMLU
40.8
41.5
40.7
41.5
40.8
41.5
41.0
40.4
40.8
41.9
+1.1
ARC-C
46.1
45.7
45.8
46.7
45.7
45.7
47.0
45.3
46.4
47.7
+1.6
TruthfulQA
43.5
40.0
40.4
39.3
40.2
40.8
40.0
40.5
40.5
40.3
-3.2
IFEval
19.4
20.7
20.1
22.0
20.8
21.9
22.2
20.5
21.8
22.6
+3.2
Code Generation
HumanEval 16.5
16.4
16.5
15.2
15.6
15,9
15.9
16.7
16.5
16.7
+0.2
MRPP
25.2
23.0
23.6
23.2
24.6
23.0
23.5
23.5
23.8
24.6
-0.8
Average 30.8
30.1
30.0
30.3
30.2
30.1
30.4
30.0
30.5
31.1
+1.8
Table 2: Evaluation of data selection methods on LLaMA2-7B. Bold indicates the best result, un- derline the second best. denotes the gain of SPICE over full-data tuning. Overall, SPICE achieves superior performance reasoning, knowledge, and code generation tasks.
Performance Analysis
(b) Sensitivity analysis of conflict penalty parameter
58.5
Average Performance
58.0
SPICE
Random
TSDS
Nuffrmanca
68.0
MMLU
87.0
48.0
HumanEval
SPICE+
IFD
LEAD
Ub.5
46
06.0
57.0
LESS
SPICE+
86.0
0.0
44.0
LEAD
SelectIT
Penalty
Perally
Penelty
56.5
(c) Impact of Proxy Model Size and Step Intervals
Fisher
Intervals T-10
Random
IFD
LESS
SelectIT
67
TSDS
55.0
0.2
0.4
0.6
0.58
1.56 Prosy Model Size
78
Normalized Time Cost
Figure 3: (a) Cost-accuracy trade-off. Average performance versus normalized time cost. SPICE/SPICE+ attain higher accuracy than Full with lower cost; SPICE+ is slightly more curate but slower due to adaptive stopping. (b) Penalty parameter sensitivity. Varying E (0, 0.2, 0.5, 1.0), performance degrades at À without penalty (non-confict-Awarl greedy selection), while À E 00.10.5, performs well. (c) Efficient selection. Impact of efficient selection: smaller proxy models and larger step intervals preserve performance while reducing cost.
5.4 ABLATION STUDY
Sensitivity analysis of conflict penalty parameter X. To provide a more comprehensive analysis {0,
of the conflict penalty == {0, 0.1, 0.2, 0.5, 1.0), we tested the impact on both performance and the characteristics of the selected dataset. In Figure (b), we observe that the performance degrades significantly when À (no penalty), whereas values of À [0.1, 0.5] yield consistently stronger results, demonstrating the effectiveness of incorporating the conflict penalty.
Impact of Efficient Selection. As mentioned in Section 4.3 of the methodology, We will compare the performance differences across proxy model sizes {0.5B, 1.5B, 7B) and step intervals T = {1, 10, 50) respectively in average score of three benchmarks (MMLU, GSKK and HvmanEval), The result in Figure (c) shows that smaller proxies and larger intervals can maintain good results while bringing lower costs. Among them, under the combination of (T 10, 1.5BB and (T' S 7B), the performance reaches the optimal score. Overall, within a reasonable range, the method is relatively insensitive to conflict penalty, proxy model size and step intervals.
Appendix H.4 further compares proxy architectures. Using Qwen2 proxies of different sizes (0.5B/1.5B/7B) yields similar accuracy, with 7B improving the average by only 0.1 point while incurring higher cost. In contrast, using LLaMA2-style proxy performs noticeably worse when se- lecting data for Qwen2-7B, indicating limited cross-architecture transfer (Coleman et al., 2020;

---

Published as a conference paper at ICLR 2026
Yang al., 2024b; Xia et al., 2024a; Liu et al., 2025). In addition, we provide more detailed analysis and case study the selected data in Appendix H.5.
5.3 DIVERSITY ANALYSIS
Intuitively, the log-det Fisher tends to cover a broader set of non-redundant gradient directions, while our conflict term suppresses only destructive opposition rather than indis- criminately penalizing similarity. To substan- tiate this, we evaluate subset diversity along two methods domain coverage and non- redundancy: (i) NovelSum and LDD (Yang et al., 2025; Zhang et al., 2023) as set-level di- versity metries, and (ii) the proportions of do- main coverage across code math-reasoning general to assess balance.
Method (10%)
Novelt Sum
Domain Coveraget
LDD1
Code
Math-R
General
Random
-9.5
30.3
10%
10%
Fisher
19.4
40.3
12%
2%
9%
LESS
4.9
38.7
1%
12%
11%
Select|T
17.6
39.0
8%
13%
9%
TSDS
-1.8
34.8
4%
5%
10%
DPP
31,1
42.5
7%
7%
9%
LEAD
9.8
37.5
2%
6%
11%
SPICE
22.0
41.3
10%
9%
As shown in Table 3, on Qwen2-7B with a 10% budget, SPICE achieves NovelSum/L.DD scores that clearly exceed Random and most baselines and are comparable DPP: mean- while, its domain coverage closely matches the full corpus, indicating that SPICE preserves broad and balanced semantic coverage even under aggressive compression.
Table 3: Comparison of diversity metrics (LDD, NovelSum) and domain coverage for 10% subsets on Qwen2-7B across SPICE and baselines.
6 RELATED WORK
Data selection for instruction tuning has emerged central research topic and methods are com- monly grouped into model-agnostic and model-aware categories (Albalak et al., 2024).
Model-agnostic Methods. Model-agnostic methods include rule-based filtering (Cao et al., 2024), quality metrics such as perplexity (Li et al., 2024a), and Instruction-Follloving Difficulty (IFD) (Li et al., 2024b). Gradient-basee and coermatiionthheoreee approaches (Chen et al., 2025; Wang et al., 2025: Ivison et al., 2025) have also gained attention, such FisherSFT (Deb et al., 2025), which leverages Fisher information of gradients for efficient SFT. In addition, diversity-orienter selection via Determinantal Point Processes (DPP) (Zhang et al., 2023) and cask-conditioned TSDS (Liu al., 2024d) have been explored to promote coverage and task alignment. But these methods fail to adapt to model-specific learning dynamics.
Model-aware methods tailor data selection to the target model's evolving state. Non-iterative approaches, like LESS (Xia et al., 2024a), perform a one-time selection, while iterative methods adapt dynamically during training like LEAD (Lin et al., 2025). Recent work, such as SelectIT (Liu et al., 2025), uses powerful LLMs to estimate sample quality, but these methods often require costly full-dataset inference, resulting in substantial computational overhead (Albalak et al., 2024).
Model-aware Methods.
In summary, existing methods primarily rely on repeated full-dataset inference or heuristic scoring with high computational cost, making them often impractical for large-scale full-parameter fine- tuning (Liu et al., 2024c; Yin Rush, 2025). Moreover, most approaches lack a solid theoretical foundation clarifying why they effective, leaving reliability and generalizability uncertain.
CONCLUSION AND FUTURE WORK
In this paper, we propose SPICE, conflict-aware greedy data selection framework. We have the- oretically proven that our method selects data with higher information gain, and have empirically demonstrated that these gains translate into better downstream performance. Our method attains full-data-level accuracy with 10% data at lower total time while outperforming other baselines, and theory (5 (cuucuratrrre explains when and why it works. Looking forward, future work could extend approach to more tasks, like multimodal reinforcement learning. By making the decay rate of marginal information explicit and controllable, SPICE offers a principled path toward compact, high-quality training corpora.
10

---

Published as a conference paper at ICLR 2026
8 ETHICS STATEMENT
Our work primarily focuses on data selection and efficient fine-tuning for large language models. All datasets used are publicly available and widely used prior research, with personally identifiable information included.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, we provide a code repository (link in the abstract code) and have fixed the random seed in configs/ fault. yaml. Specifically, we conducted ex- periments using three different seeds: 2024, 2025, and 2026. Details of the experimental setup, including the configuration files and scripts needed reproduce the results, are included in the code repository.
In addition, the appendix contains further details facilitate reproducibility, including complete theoretical proofs, descriptions of the datasets and methods, and additional experimental results and ablation studies:
Complete theoretical proofs for AdaFisher complexity, perturbation bounds, greedy guar- antecs, and submodularity
Empirical validations of gradient conflict analysis, s-decomposition, and curvature param- eter experiments
Detailed experimental setup including dataset composition, evaluation metrics, bench- marks, and baseline methods
Additional results on LLaMA2, SPICE+ early stopping strategy, and ablation studies of proxy models and values.
More data statistics and case studies about selected data.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
A.A. Ageev and M.I. Sviridenko. Pipage rounding: A new method of constructing algorithms with proven performance guarantee. Journal of Combinatorial Optimization, 833):377332,, 2004. ISSN 1382-6905. doi: 10. /155/:/occo.00000388111 13.607..91..9607.2222 [Online; accessed 2025-08-12]]
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangmnng Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey on data selection for language models, 2024. URL / arxiv. org/ abs 2402.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0775: 2021.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models, 2024. URL https: /ttp:///aavvvvv./ia org/ abs//aa///// 220...330000000 06290.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. survey on evaluation of large language models, 2023. URL https: org/abs/23077 03109.
Sahil Chaudhary, Code alpaca: An instracttion-folloiingg Ilama model for code generation. https: //g//t/ttt github..coooo com/ sah www.eahiiihiaa.a 1280114/ cttp////ooaaaaaccccacco 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Broekman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
Yicheng Chen, Yining Li, Kai Hu, Zerun Ma. Haochen Ye, and Kai Chen. Mig: Automatic data selection for instruction tuning by maximizing information gain in semantic space. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 9902-9915, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5 doi: 18.53/ v1/2025. 1122055findings-al..555 URL 515/.
htthttss://aclaane http:://aaaaaallaa...aaaaa .tn.://..or.g022.....n........ findings-ach
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/ 103.00553, 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021. URL https: https://arxiv. htt//bs//2111 14168.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, 2020. URL ht / //aaaaxivvvorr./.. hbap/abs//a / 906. 11829.
Michele Conforti and Gérard Cornuéjols. Submodular set functions, matroids and the greedy algo- rithm: Tight worst-case bounds and some generalizations of the rado-edmonds theorem. Discrete Applied Mathematics, 7(3):5112744 3 1984. ISSN 0166-218X. doi: 10.101600166-218x(88) 90003-9. [Online; accessed 2025-08-12).
12

---

Published as a conference paper at ICLR 2026
Rohan Deb, Kiran Thekumparampil, Kousha Kalantari, Gaurush Hiranandani, Shoham Sabach, and Branislav Kveton. Fishersft: Data-efficient supervised fine-tuning of language models using in- formation gain, 2025. URL httpt:://aaiiiiiiiii org. /tabb//299 ab555 14826.
Uriel Feige and Jan Vondrák. The submodular welfare problem with demand queries. The- ory of Computing, 6 6(11):247-290, 2010. doi: 10.006//oc.2211.00666 // /ttheoryofcooputtooo...ooo
URL https:
R. A. Fisher. On the mathematical foundations of theoretical statistics. Philosophical Transac- fions of the Royal Society of London. Series A, Containing Papers Mathematical or Physical Character, 222:309-3000 368, 1922. ISSN 02643952. URL http: www. jstor org/ stabl// 91208.
Rajiv Gandhi, Samir Khuller, Srinivasan Parthasarathy, and Aravind Srinivasan. Dependent round- ing and its applications to approximation algorithms. Journal of the ACM, 534)):324-360, 2006. ISSN 0004-5411.
doi: 10. 1 145/1 147954. 147956. [Online; accessed 2025-08-12).
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter; Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https: w records 12608602.
Damien Martins Gomes, Yanlei Zhang, Eugene Belilovsky, Guy Wolf, and Mahdi S. Hosseini. Adafisher: Adaptive second order optimization via fisher information, 2025. URL https: / ///axiiv.org/aaa /ttrr//ab////// 2405. 16397.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprint arXiv:2503.0010 0100007777 2025. doi: 10.48550/arXiv. 2503.01807. URL https::////....... /apps:///rxivvv aariiv org abs / 2503 01 807.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfitering: Weak-to-strong data filtering for fast instruction-tuning In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pp. 14255-14273, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/11/224. acl-long.766 769. URL https: hwtpa/aaaaanntoooo..cco.o org/ 2024 acl-long. 769/
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume Long Papers), pp. 7602-7635, Mexico City, Mexico, June 2024b. Association for Computational Linguisties. doi: 10.18653/v1/ 2024.naacl- long.lllong.2111 URL https: / /tteppp://allleeeeeeeeeee ac aclanthologyy org 2024. www.nnannllaaaa long. 421/.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic hu- man falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computa- lional Linguistics (Volume 1: Long Pupers), pp. 3214-3252, Dublin, Ireland, May 2022. As- sociation for Computational Linguisties. doi: 11188///1222222acllloo / //aclanthology wttp:eyy.rgg/2222/// act- 229.
URL https:
Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, and Yuyu Luo. Lead: Iterative data selection for efficient Ilm instruction tuning, 2025. URL https: // //arxiv.or org. abs 2505. 07437.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu, Conffict-averse gradient descent for multi-task learning, 2024a. URL https: // www.ivivvvv... www..org/abs.... 2110. 14048.
13

---

Published as a conference paper at ICLR 2026
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning, 2024b. URL https://arxiiy org/ abs/ 20t./2110.. 14048.
Liangxin Liu, Xuebo Liu, Derek F. Wong, Dongfang Li, Ziyi Wang, Baotian Hu, and Min Zhang. Selectit: Selective instruction tuning for Ilms via uncertainty-awar aware self-reflection, 2025. URL
https:: //axxiv. www.iiiiig///i.eeeo org/oorgsss. 2240216606
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? comprehensive study of automatic data selection in instruction tuning, 2024c. URL https: /tapps:///rxiv..////. org/ abs///as///// / 2312 15685.
Zifan Liu, Amin Karbasi, and Theodoros Rekatsinas, Tsds: Data selection for task-specific model 11303. Lovász. Submodular functions and convexity. Springer Berlin Heidelberg, 1983, doi: 10.1007/ 978-3-642- 1--42-688844--100 [Online; accessed 2025-08-07].
finetuning, 2024d. URLhttpsss/aarxvv..rr//aaas
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Instag: Instruction tagging for analyzing supervised fine-tuning of large language models, 2023. URL https: aaaiivivivv org/ abs 2200 07074.
James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature, 2020. URL
G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functions i. Mathematical Programming. 14(1):265--2) 294, 1978. ISSN 0025- 5610. doi: 10. 10.1000/01188897/ [Online; accessed 2025-08-07].
Guangyuan Shi, Qimai Li, Wenlong Zhang. Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing conflicting gradients from the root for multi-task learning, 2023. httsttp::// /arxiv..rg// abs/2302.. 11289.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:22 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction- foolowiwng llama model. 2023.
https::/githun com/ caatsu-la//a saaabsstanfon
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / arxiv. orgoorgggooo.... /agg/b//// 09288...
Jingtan Wang, Xiaoqiang Lin, Rui Qiao, Pang Wei Koh, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Nice: Data selection for instruction tuning in lims with non-differentiable evaluation metric. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. URL forum? sediiidwww 8m5HUBs. ICML 2025 poster.
https: cpenreview .nt/ wwwpeeeeeeoorutttteeeeeee
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danq Chen. Less: lecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference Machine Leaniigg, ICML'24, JMg.. 2024a.
14

---

Published as a conference paper at ICLR 2026
Tingyu Xia, Bowen Yu, Kai Dang, An Yang. Yuan Wu, Yuan Tian, Yi Chang, and Junyang Lin. Rethinking data selection at scale: Random selection almost all you need, 2024b. URL https: // //axxiivorggaaaaaaeaeaa wtwpg///aaaaaaaaaa 2415.09335.
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai. Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL https: arxiv org/ alss
Yu Yang, Siddhartha Mishra, Jeffrey N Chiang, and Baharan Mirzasoleiman. Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models, 2024b. URL https: //tp://aaaivv../iai .ogg/ abs////3..23./ 07384.
Yuming Yang. Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Mingqi Wu, Tao Gui, Qi Zhang, and Xuanjing Huang. Measuring data diversity for instruction tuning: A systematic analysis and reliable metric, 2025. URL 17184.
org/ abs/2502.
Junjie Oscar Yin and Alexander M. Rush. Compute-constraineed data selection, 2025. URL https: //arxiv..c ooro abs 2410.1. 6208.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020. URL https://arxiv, wtwp:///wwwrr/b///////00 06782.
Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pre- training with data influence models, 2024. URL httttpps//axxi..oxee 6240 06006,
Yuxuan Zhang, Chao Xu, Howard H. Yang, Xijun Wang, and Tony Q. S. Quek. Dpp- based client selection for federated learning with non-iid data, 2023. URL https: /tte:///xxxiv. orgg / 2303. 17358.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- tics. URL http: arxiv org/ alss 2403..... 13372.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuming Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023a. URL https::: //rxiiivvv .or. hbtg//bb//// 2200 11206.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuming Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023b. URL /ttpcttss//axxxi...e .rw. / abs/ 2305 11206.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follovin evaluation for large language models, arXiv preprint arXiv:2311.07911, 2023c.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOF: COMPLEXITY OF ADAFISHER INFORMATION MATRIX
(Section 2.1) Next, we will briefly introduce AdaFisher and how to reduce computational complexity compared to Fisher. The specific content is referenced from the paper "AdaFisher: Adaptive second order optimization via fisher information" (Gomes et al., 2025).
A.1 ADAFISHER: AN EFFICIENT RECOND-ORDER OPTIMIZER
Traditional Fisher Information Matrix (FLM) based optimizers, while theoretically superior in terms of convergence properties, face significant computational challenges when applied to modern deep neural networks. The primary bottleneck lies in the quadratic complexity: computing and storing the full FIM requires (dd2) memory and (dd)) operations for matrix inversion, where d represents the total number of parameters (Martens & Grosse, 2020). For contemporary deep networks with millions to billions of parameters, this computational burden renders traditional FIM-based methods practically infeasible.
AdaFisher addresses these fundamental limitations through novel diagonal block- Kronecker ap- proximation of the Fisher Information Matrix. By leveraging this diagonal concentration property, AdaFisher achieves a remarkable reduction in computational complexity from ( to (d) while preserving the superior convergence characteristics of second-order methods. This breakthrough makes Fisher Information-bant optimization practically viable for large-scale deep learning appli- cations, offering computational efficiency comparable to first-order methods.
A.2 METHOD OVERVIEW AND COMPLEXITY REDUCTION
AdaFisher employs systematic approach to approximate the Fisher Information Matrix through Kronecker factorization. For a neural network layer i with weight matrix W; E R the empirical Fisher Information Matrix can be expressed
[[(ee(gg) (=] H 1 S,
(16)
represents the second-orde statistics of layer activations, S; = s captures the second-order statistics of pre-activation gradients, h = [M_ 1]" denotes the aug- mented activation vector, and represents the Kronecker product.
where
The key insight of AdaFisher lies in the diagonal concentration phenomenon: the Kronecker factors H,-1 and S; exhibit strong diagonal dominance, meaning their energy is primarily concentrated along the diagonal elements. Based on this observation, AdaFisher introduces the diagonal approx- imation:
F = H
& S.D + AI
(17)
where HiD, Diag(H,-) and S.D Diag( (S,) retain only the diagonal elements, and À is regularization parameter for numerical stability.
We now provide rigorous mathematical proof of the complexity reduction from O(d2) to O(d).
Traditional FIM Complexity. For network with L layers, where layer has input dimension 74-1 and output dimension ni, the traditional FIM computation requires:
((5 14-1
3 )) O(d²) (18)
))
Storage:
M-
n,)² =
Σ n n Inversion: O
=
where d = ELe 14-178 the total number of parameters.
16

---

Published as a conference paper at ICLR 2026
AdaFisher Diagonal Approximation. putational requirements:
The diagonal approximation fundamentally alters the com-
Storage:
nn-1 +m) = O(d), Inversion:
n + n4) =dd)
(19)
Therefore we can analyze the detailed per-iteration proof: Firstly, computing diagonal statistics requires:
Cupdato
(n-1 n.) = O(d),
(20)
then for diagonal matrices, inversion reduces to element-wise reciprocal:
Cinversion
(n=-1 + nz) O
(21)
so the preconditioned gradient computation:
g( = (1
(22)
requires only dlementwwise operations, yielding (dd) complexity.
AdaFisher reduces the per-iteration computational complexity of Fisher Information-baxed opti- mization from (dd)) (dd) while maintaining OllogT/V7 convergence rate.
Proof. The total per-iteration complexity dominated by:
Crotal
Cupdase
Civversion Cpnocoudiiiioning (d) O O(d) (d)
(23)
This represents quadratic-to- reduction compared traditional FIM methods, making second-order optimization computationally feasible for large-scale applications. If want to see the more detailed proof process and method, please refer to AdaFisher (Gomes al., 2025).
B PROOF: SUBMODULARITY OF FISHER-BASED OBJECTIVES
(Section 2.2) We need to prove that the Fisher-basse objective
F(S) log det + aFFs)
(24)
monotone submodular under mild conditions, and is strictly submodular under a natural nonde- generacy condition.
Proof. Let the empirical Fisher matrix over a S be
Fs
Σ
w. 9:9
(25)
iES
where the sample weights satisfy w; (covering standard Fisher with w; = 1 and AdaFisher via appropriate effective vectors; see below).
Fix a Since Fs is positive semidefinite (PSD) and I > 0. the matrix aFs is positive
definite (PD) and hence invertible for any S.
For any S and element x, define the marginal gain
A = F(S (z)) F(S).
(26)
Since Fsu(z) Fs + w.gag. the matrix determinant lemma (applied with /wa gwv gives
(S) = log aw g. (I oFF)) gx)
(27)
17

---

Published as a conference paper at ICLR 2026
Now let A S B _ D. Since all w, 2 0. FA Fg in the Loewner order, hence
I aF 3 I oFg.
(28)
The order-reversing property of matrix inversion for PD matrices then gives
aI+aFa)~+ (I+ (=F)
(29)
Pre- and post-multiplying by 9r yields
9 (I+ aF aF Ix 2 91 (I+ oFB) Jn.
(30)
Since log(1 + o.w,2) ) is strictly increasing for aw 2 0, we obtain
A² 2 (B),
(31)
establishing monotonicity and submodularity (diminishing returns).
Strictness. The inequality is strict, (AA > A, whenever
((I + aF
(I + (I+oFB)) > 0.
(32)
sufficient condition for this that Fø F and Oz has a nonzero projection onto its range (i.e.. is not orthogonal to the new directions introduced from A to B).. Under this nondegeneracy condition, the Fisher utility is strictly submodular.
AdaFisher. For AdaFisher, each summand takes the form diag(19d) g:9. diag( (ggg) Define ef- fective vectors OS := diag(1g [t )g; and weights w; = 1. Then F Eies W:0hū, and the same argument applies verbatim with g: replaced by gt.
with OR projecting nontrivially onto its range holds except in rare degenerate cases. Gradients gi in typical datasets are high-dimensional and not orthogonal, so enlarging S almost always expands their span. As 9r comes from the same distribution, it is almost never orthogonal to the new directions continuous spaces. Empirically, (A) (B) occurs in over 99% of our test data, showing that striet submodularity is both theoretically valid and dominant in practice.
Prevalence. In practice, the condition F8 FA
C PROOF: CLASSICAL GREEDY APPROXIMATION
From the Theorem 2, we let F be a normalized, monotone submodular function. Under a cardinality constraint k the greedy solution S greedy satisfies
F(Sgreddy) 2 (1-2)
(33)
where S* an optimal k-subset.
proof. Let the greedy algorithm produce the chain of sets
So Ø, S1, S..... Sk = Sgreedys
(34)
where at step the algorithm selects an element
F+41 E arg max ZEDiS,
(S.),
(S,) F(S F(S.).
(35)
Let S* denote an optimal with | k and define the residual
R₂ A F(S*) F(S.).
t= 0, l...
(36)
We will show that R.41 S (1 R,, from which the claim follows by induction.
First, by submodularity and monotonicity we have for any T' C D and any S C D:
F(SUT) F(S) < Σ AS)
(37)
18

---

Published as a conference paper at ICLR 2026
Apply equation 37 with S St and T S" (S. Since F is monotone and S* C S1 US St), we get
F(S)) - F(S)) s
Σ
(S,)
(38)
Consequently there exists some S" \. St with
F(S*) F(S))
A,(S)) 2 5 Scl
(39)
Using |S* \S/ < | S k yields the lower bound
R2
max wES-
A,(S,)
2
(40)
k
By the greedy choice,
Bet (Si)
max (S)) sED(S,
WE (Si).
(41)
Combining this with equation 40 gives
R Since Azw.i (Si) F(S+11) - F(S.), we obtain the recursive inequality
Aon (S.)
(42)
Re
Ri41 S (1 (1-1)
1 k.
F(S++1) - F(S)) 2
R.
(43)
k
Unrolling the recursion from t = 0 (noting Ro F(S*) because F(Ø) 0) yields
(1-1) k 1))² F(S*).
R 1 1 k
Ro =
(44)
Hence
F(Sx) = F(S*) R > - -1-11 1.) (1-1 )
(45)
Using the inequality (1 1)*² < - for all integers & 2 1 gives the standard bound
((ggreeyy) F(St) 2 (1-)) (-'))F(S))
(46)
This completes the proof.
Remarks. The proof only uses normalization, monotonicity and submodularity; no continuity or differentiability required.
D PROOF: GREEDY GUARANTEE WITH E-CURVATURE
From the Theorem 3, we know that let F be normalized and monotone submodular with total cur- vature c = minzeD (Z)
(D))) E [0, 1]. Under cardinality constraint k, the greedy solution
Spreedy satisfies
F(Sgreddy) 2 F(S*).
(47)
proof. The proof follows the multilinear-extension and continuous-greedy approach (see Feige & Vondrák (2010) and Conforti & Cornuéjols (1984) for discrete derivations).
The original problem a discrete optimization problem, but to enable further optimization, we need to make continuous that we can subsequently use calculus tools.
19

---

Published as a conference paper at ICLR 2026
Multilinear extension. Let X be the ground set and [X]] Define the multilinear extension G: [= + R20 by
G(y) BR-y [[R)],
(48)
where ~ y denotes the product distribution that includes element independently with probabil- ity Standard properties: G(1s) = f(S) for integral vectors, each partial derivative 8,G(v) is
nonnegative, and cross-partial derivatives are nonpositive because of the submodularity.
We need to find a direction (specifically, the direction of choosing S*) that makes the growth rate of G(y) at least ((S*) - Gyy). This crucial because the continuous greedy algorithm moves the direction that maximizes the increase of G(y).. If the optimal direction has this lower bound, then the entire Ordinary Differential Equation (ODE) can be solved.
Let PR {u [0,1]" E; 2/ k} denote the cardinality polytope. The following inequality is the crucial bridge between curvature and the continuous-gred greedy dynamics.
Lemma 2. For every y [0. 1|"h it holds that
max U VG(V) 2 F(5")-cCo
(49)
vEP.
where St any optimal k-set and e is the total curvature.
Proof. Take v 1s- (the indicator the optimal set) which belongs Ps. By linearity of expec- lation and the definition of the partial derivative of the multilinear extension, one may write
1s- VG(y)
Σ [
f(R (0)] ERt [(- f(R)), (50)
where the second equality is the standard rearrangement identity for the multilinear extension (see Feige& & Vondrák (2010) for formal derivation). Now apply the curvature definition: for any fixed realization R.
((RUS*) S(S*) c (R),
(51)
which follows from the fact that curvature bounds how much the presence of R can reduce the singleton marginals comprising f(S*). Taking expectation over R ~ y yields
1s- VG E(f(RUS") - f(R] 2 F(S*) cGly)
(52)
Since the left-hand side is most the maximum over E Pk. equation 49 follows. For fully detailed, line-by-line algebraic proof, please refer to Feige & Vondrák (2010) and Conforti & Cornučjols (1984).
Next we formulate the continuous greedy trajectory differential cquation, which upon solving yields lower bound for the continuous solution.
Continuous-greedl ODE.
Run the continuous-greedy trajectory vtt) for E [0, 1J with y(0) 0 and
dy E arg max v VG(((t)). vEr,
(53)
dt
By Lemma 2,
G(v(t)) = dy vG(y(t)) 2 dt
(54)
dt
Let H(!) = G(v(O). Then H satisfies the linear differential inequality
((t)= 2 F(S*) cH)(), Then applying Grönwall's inequality, if H(I) satisfies:
H(0)=
(55)
(() 2 (t)))
(56)
where ((1) are continuous, then:
H(()) + f B(s)edi a(t)dt a(s)ers((t)ttss)
H(L) 2 e- f3 a(s)ds
(57)
20

---

Published as a conference paper at ICLR 2026
So solve the ODE:
J a(s)ds = / cds = ot + ((
alr)er =
=
(58)
H(1) 2 E E (0+ d(
(59)
=e F(5") [-" e
(60)
.=..F(() ect
(61)
F(S*)
c (1 c c
(62)
Thus can get
H1) = G(v(1) 2 F(S*) (1 (1-0-9.
(63)
Rounding to an integral solution. Finally, we need to convert the continuous optimization solu- tion into actual discrete set without losing performance.
Based on the dependent rounding procedure (Ageev & Sviridenko, 2004; Gandhi et al., 2006), we can convert the fractional solution y(1) into an integral vector 1s with |S| while preserving the expected value of the multilinear extension:
B[((S) = G(v(1)),
(64)
and there always exists one satisfy:
f(S) 2 G(v(1)).
(65)
Hence by the solution of ODE, we can get:
F(S) G(1s) G(v(1)) 2 F(S*)
(1 (1---9) 99.
(66)
c
Taking Sgreedy to be the constructed feasible set
1 - F(S*).
F F(Sgreddy) 2
(67)
This completes the proof.
E PROOF: PERTURBATION BOUND VIA GRADIENT ALIGNMENT
We provide the detailed proof of Theorem 4 and the technical assumptions that were compressed in the main text.
E. TECHNICAL ASSUMPTIONS AND CONSTANTS
The perturbation bound relies the following assumptions: (1) Bounded gradients: [|g.|] & Gmax for all samples x E 'D; (2) Fisher scaling condition: all Fsll S p < ensuring Neumann expansion convergence
The constant Cp,, Gmax) has the explicit form:
1
C(p, G
(68)
aG?nad
where P means that upper bound Fsll (typically p E [0.1,0.5] in practice), Gmax max, llgx.l that maximum gradient norm across all samples, and a > is that Fisher informa- tion scaling parameter
21

---

Published as a conference paper at ICLR 2026
E.2
MAIN PERTURBATION BOUND
Theorem 5 (Detailed perturbation bound). Under the above assumptions, the perturbation term satisfies:
a² Eyes(o gy)2
(c S Clp, Gmax)
(69)
1 al
The cumulative perturbation in greedy selection is bounded by:
S22 (St-1))
k-
max max
Σ 1 gu)
(70)
18|Sk
BES
proof. Let A aFs. By the assumption, we have ||A|| aallssl S < 1, so the Neumann series expansion for (I + A)-1 valid:
(I+ = Σ (-))a.
(71)
m=u
Set u Os and D2 = 1 + + as above. Then
0x (I aFs) Ix 9T A) In Σ
1) T A Jr
Or
=110=112 In Agu +
(-1)" 9. A 9z.
(72)
m=
Observe that
T Ag. = a 9. Fsg. = a (S).
(73)
And we know that the perturbation c, (S) from the Definition 3
+ 9, (1 aFs) 9x
E ( log
(74)
+ a [/ 12
Substituting equation 72 into the E (S) yields
E (S) = log 1 (S) + a Em=a( (-1)m² 9 A
(75)
D₂
Define the numeratoo-ssmall-errm
-x) + Em 2m=2(-1) 11m A gr
u
(76)
D₁
so that Eμ (S) = log(1 + u).
We next bound the absolute value of the tail series. For PSD matrix A and any integer m one has the spectral-order incquality
A" = ||A|| m-1 A.
(77)
Using this and ||A||| S P we obtain for every m 2 2
| AT gxrl ||||| m-1 02 AgN |||||| m-1 (E)((S)
(78)
Hence the infinite tail is bounded by the geometric series
Σ (-1)TT AT 9x
< (S)
Σ
||||==1 aEx(S) 1- |A] < (E
|||||
P
(79)
1 1-p
m=
m=
Therefore the numerator of satisfies
- -x +a Σ (-1)
A gn S (5) 1 1+_ 1- =
(1+-P)
a2))(()
(80)
1-
72
22

---

Published as a conference paper at ICLR 2026
Dividing by D2 yields the uniform bound
(5) Jul - (1-p) D
(81)
[u] [u
At this point we invoke the elementary inequality |log(1 + u) 1
valid for |u|
1. We
(S) < To guarantee a simple, data-independent sufficient condition
therefore require (1 D,D,r use the crude bound
a ES) S 11 < G max
P
aG
(82)
a.max
where we used || I p/a. Hence a sufficient (mild) technical condition this condition we obtain the data-independent upper bound
1. Under
1-
aG aGmax 1 P
| S
<1
(83)
Combining the bound on |u] with the logarithm incquality gives
a (S)
[s(( = log(1 + u)l m
D)
a 222x(S)
(84)
aG2 max
D2
1 p aG2 a
max
1-p
This proves equation 4 with the explicit constant
1
C(p, Giax, a) 1 1
(85)
aG
aG2 maxx
P
Finally, summing the pointwise bound over a greedy sequence F1,.... Th yields
Es, (S C(p, Gmax, a). Σ Ex, (St-1) < C(p, Gmax, a max max ssls
Σ
(9 gv)?
Dr.
yES
(86)
which establishes the stated cumulative O(-) bound and completes the proof.
L
For completeness, we provide the exact curvature bound that was simplified in Corollary 1:
C(p. Giaxx) a Eytu g, 9²)
max (1 + allg>111 I12) log( 1 + a 113)
(87)
where C(p, Gmax) is given above. This shows the precise dependence of the approximation guar- antee gradient inner products and problem parameters.
F DETAILED EMPIRICAL EXPERIMENTS
In order to verify the correctness of the theoretical analysis in Section 2, conducted multiple experiments, which comprehensively validated our idea in the actual data selection process and laid the groundwork for the rational implementation of our subsequent methods.
In Section 3, we conducted three groups of experiments to verify that gradient conflicts lead to faster marginal decrease in information gain during greedy selection, thereby reducing the in- formation content of the selected samples. Next, we briefly introduce the experimental setup and procedures, well additional experiments for validating the theoretical analysis.
23

---

Published as a conference paper at ICLR 2026
Lcacortie
$2.5
16
50
16
25
19
36
180
50
160
25
15
Eabarie SHCW
Salation decaey
Seaticn
13
3.2
HumanEval
GSM8K
MMLU
Code Alpaca
Stanford Alpaca
ShareGPT
Figure 4: Decay of the high and low conflict subsets experiments in HvmanEval, GSM8K, MMLU, Code Alpaca, Stanford Alpaca and ShareGPT. Obviously, the overall results shows highly consistent with our corollary
F.1 GRADIENT VISUALIZATION
In this paper, we propose that certain samples in data selection exhibit gradient conflicts, visual inspection is necessary to guide subsequent processing. In the experiments, we randomly selected 512 data points for 3D and 2D visualization analyses, and computed the information content of each gradient sample using Fisher information, which is represented in the figures by varying shades.
As shown in Figure 2 (a), we observe that most samples are consistent with the average gradient of the actual step updates, while few samples conflict with the average gradient trrection-this is the so-called gradient conflict phenomenon. Interestingly, some of these conflicting samples pos- sess high information content, which aligns with our intuition that opposing gradients can pull the model out of local optima and may even represent the highest-quality gradients in the data (Shi et al., 2023). Therefore, simply discarding these conflicting samples, although it may accelerate conver- gence and improve the greedy approximation, could result in the loss of high-information samples. This highlights the need to balance gradient conflict and gradient information.
F.2 DECAY OF THE HIGH/LOW CONFLICT SUBSETS
(Section 3) This experiment aims to compare the marginal gain decay between high-conflict and low- conffict sample groups during the Fisher greedy selection process. We analyzed the results from two perspectives. First, we ensured the reliability of the experiments by normalizing and repeating them six times. Second, we examined the decay rate and the marginal decrease in information content.
Setup and Procedures. For a randomly sampled dataset in number of 256 data, compute the gradient Si of each sample and the overall mean gradient g. and derive a confiict score for each sample based on the deviation from the mean. Then divide the samples into two groups using a 20% threshold: the low-conflict group (the bottom of samples with the lowest conflict scores) and the high-conflict group (the top of samples with the highest conflict scores). Finally, independently run the Fisher greedy selection within each group with the same budget 1=128, and record the marginal gain sequence each step.
Result. As shown in Figure 2 (b), the marginal gain in the low-conflict group decays significantly slower than in the high-conflict group, resulting much higher greedy approximation of infor- mation content for the same & steps. This observation confirms the data-dependent approximation guarantee discussed in our thcoretical analysis. To further validate our findings, we conducted the same experiments on common test datasets MMLU (Hendrycks et al., 2021), GSM8K (Cobbe al., 2021) and HumanEval (Chen et al., 2021) and common train datasets Code Aplaca (Chaudhary, 2023), Stanford Alpaca (Taori et al., 2023) and ShareGPT (Lu et al., 2023). The results are pre- sented below:
The results in Figure 4 are consistent with the previous observations, supporting our corollary.
24

---

Published as a conference paper at ICLR 2026
36
335
Mon 3320
Mop p=000
36-
Map p44
Porton 455
Stat
Sept
Pap
Sep:
See
Sept
contict and varppitonn
corfect vereps
confict verepsionn
corfict and varagsiton
confict verepatee
confect
6312
1000
36
1005
3004
6000
03
3/87
6.4
OA
06 87 03 4+
D4 x4
02 0.1
HumanEval
GSM8K
MMLU
Code Alpaca
Stanford Alpaca
corleggo
ShareGPT
Figure 5: Spearman correlation analysis experiments in HumanEval, GSM8K, MMLU, Code Al- paca, Stanford Alpaca and ShareGPT. The overall conflict negatively correlated with marginal gain A, and positively correlated with E, which confirms our conclusion.
F.3 SPEARMAN CORRELATION ANALYSIS
In this set of experiments: (1) At each step of the greedy selection, we examine the correlation between the conflict scores of samples in the candidate pool and their current marginal gains A,; (2) with fixed initial set Sn, we investigate the relationship between the variance E₂ (So) of individual samples and their degree of conflict.
Setup and Procedures. For Experiment (1), we primarily focus intra-group correlations. In the first step, we compute the conflict score Conflict((gz) = cos(ge- g) for each sample in the candidate pool relative to the mean gradient of the pool. Then, based on the current information content, calculate the marginal gain for each candidate sample and compute the correlation coefficient Ps Spearmani (Con CCnflict, A) the current step. The sample with the highest marginal gain is selected and added to the chosen set, and removed from the candidate pool. This process is repeated until & steps are completed and then plot
line chart.
For Experiment (2), we focus more on the correlation between individual-level |e] and conflict. First, we sample 256 data points from the dataset and compute the gradient for each sample. Then, random subset Sy of the specified size is selected, and the conflict score Con fliftii((gz) and (So)) for each sample is calculated. For clearer visualization, we applied some pre-processing to the data in le (So)l log(1 (1 + (So))l and Con (gx) = 1 Con fonclllc((g))) ). Finally, we compute the (ConrmmmnCCooffliie le" and then plot seatter chart.
correlation coefficient p
Result. As shown in Figure (c), both experiments demonstrate that gradient conflict Conflict(an exhibits strong correlation, namely with the step-wise marginal gain A (7=-0.792) and the individual-level Ezz (0=0.901). This strong correlation further validates the conclusion de- rived from our e-curvature proposed in Section 2. Similarly, we also conduct the additional compre- hensive experiments in other dataset MMLU, GSMSK, EvaanEval, Code Alpaca, Stanford Alpaca and ShareGPT. From Figure we can see that this correlation is generally consistent, which also supports the correctness of our hypothesis and theoretical analysis.
F.4 ADDITIONAL THEORETICAL EXPERIMENTS
In the above experiments, we have verified that theoretical analysis holds in practical data selec- tion. However, we still need to empirically validate the various boundary conditions and assumptions made in our theoretical analysis, which will be addressed below.
F,4.1
g- --ECOMPOSITION EMPIRICAL VERIFICATION
Setup and Procedures. We empirically validate the e-decomposition proposed in Definition and the perturbation bound established in Theorem 4. Using Qwen2-7B (Yang et al., 2024a) model
25

---

Published as a conference paper at ICLR 2026
checkpoints, compute gradients for N = 512 randomly sampled Instruction-responee pairs from our training dataset.
We test subsets S of varying sizes |S] {16, 32, 64, 128, 256} with 10 random trials per size. For each configuration, we compute the value of base marginal base,, true marginal gain Az(S), perturbation term (x)S), gradient conflict Conflict(1;, 5) and the bound:
a² Σ yEs E gv)
[e S C(p, Gmax)
+ a119211
Result. Table 4 presents our validation results across different subset sizes. The e-decomposition demonstrates mathematical precision with perfect decomposition accuracy (relative error 1e for all 20,640 computed values), confirming the correctness of our theoretical formulation.
We find a strong correlation between (e=(S) and gradient conflicts, with a Spearman coefficient of 0.78 (p < 0.001). The correlation steadily increases with subset size, rising from p = 0.72 to P 0.82 at SS] = 256, consistent with our prediction that larger subsets amplify gradient conflict effects and thus strengthen their link to marginal utility perturbations. Moreover, the assumption a Fisl < p < is empirically validated: the maximum observed value satisfies a Fsss I 0.80 for 0.8)
The theoretical bound violation rate 10% represents acceptable deviation for empirical validation on real instruction-tuni tunnnngtttnnnnggn data, where practical conditions naturally deviate from idealized theoreti- cal assumptions. Due to computational constraints with high-dimensional gradients, we employ the AdaFisher diagonal approximation Fggh (Gomes et al., 2025) for tractable computation. This ap- proximation introduces systematic deviations from the exact Fisher formulation while maintaining the fundamental structural relationships theory predicts.
Correlation
Bound
Decomposition
S
Spearman
Pearson
Violation
Error
16
0.723
0.501
6.4%
< 1e-6
32
0.756
0.515
7.1%
< le-6
64
0.789
0.528
10.3%
< 1e-6
128
0.801
0.534
12.8%
< 1e-6
256
0.818
0.541
13.4%
< le-6
Overall
0.777
0.524
10.00%
1e-6
Table 4: e- -Decooppsition Empirical Validation Results. All correlations significant at p < 0.001.
Overall, these results support the practical applicability of our conflict-aware selection framework, demonstrating that the theoretical relationship holds meaningfully in real-world scenarios despite computational approximations.
F.4.2 CURVATURE e PARAMETER ANALYSIS
Setup and Procedures. We empirically validate the curvature parameter and its relationship with gradient conflicts as established in Lemma and Corollary Using the same 512 gradient samples, we select the data into three conflict levels based negative alignment with the mean gradient: (1) Low conflict, (2) Medium conflict and (3) High conflict.
For each conflict group, compute the empirical curvature parameter 1 min, (x)))AA (0) and validate the theoretical bound e < max, (D) \ [=))l/base, by calculating the theoretical value and the actual marginal revenue decay rate. Next, randomly sample 1000 times experiment to compute the actual approximation ratio.
Result. From the Table 5, it presents comprehensive validation of curvature parameter anal- ysis across three conflict levels. The results provide strong empirical support for our theoretical framework while revealing important practical boundaries.
26

---

Published as a conference paper at ICLR 2026
Our core theoretical prediction that gradient conflict positively correlates with submodular curva- ture receives robust empirical support. As conflict levels progress from low to high, we observe monotonic increase in curvature parameters: demonstrates that conflicting gradients indeed lead to higher curvature the submodular objec- tive function, validating the fundamental insight underlying conflitt-waaee selection framework. And the spearman p=0.8593 also verified its strong correlation relationship.
0.032 - 0.074 0.092. This progression
Conffict Level
Curvature
Bound Holds
Actual Ratio
Theoretical Ratio
Prediction Error
Low
0.032
1.000
0.984
1.57%
Medium
0.074
1.000
0,964
3.59%
High
0.092
1.000
0.955
4.46%
Overall
0.066
3/3
1.000
0.968
3.21 %
Table 5: Curvature Parameter Analysis Results. All empirical curvature values satisfy the theoretical bound e & max, (Ex(D) {=>)s//baee.. The conflict-curvature correlation achieves Spearman p 0.86 (p < 0.001), validating our core theoretical prediction.
All empirical curvature values satisfy our theoretical upper bound e < max,, ( (=b)e//bse... confirming the correctness of our curvature characterization in Lemma 1. The technical assump- tion a Fssl < 1 holds for low-conflict data but is violated for medium and high-conflict groups. This violation directly impacts prediction accuracy: errors increase from 1.57% (low-conflic)) to 4.46% (high-conflict), demonstrating the practical boundaries of our theoretical guarantees. The theory successfully captures structural relationships while highlighting the importance of technical assumption validity for quantitative predictions.
G DETAILED EXPERIMENTAL SETUP
G.1 TRAINING DATA
(Section 5.1) To ensure comprehensive coverage diverse capabilities, we construct our training dataset with consideration for three key aspects: mathematical reasoning, code generation, and gen- cral knowledge. Our dataset incorporates established sources including Alpaca Code (Chaudhary, 2023) for coding tasks, GSM8K (Cobbe et al., 2021) for mathematical reasoning, and WizardL.M, which encompasses ShareGPT (Lu et al., 2023) and Alpaca (Taori et al., 2023), for general knowl- edge and instruction following. Our final training contains 97,495 samples. Then will provide some statistical information as follows and we will release the data soon:
Datasets
Aspects
# Nums
# Instruction Len.
# Response Len.
LiZardLLM (Lu al., 2023)
General Knowledge
122K
192
1,234
Alpaca Code (Chaudhary. 2023)
Code Generation
20K
74
197
GSM8K (Cobbe et al., 2021)
Math Reasoning
8.5K
290
321
Final Data
97,495
405
1,038
Table Statistical information of our training instruction-tuning data.
G.2 METRICS
Half-life t//2. The half-life is defined as the number of steps needed for the cumulative marginal gain to reach half of the total gain:
T Σ
11 = min(t:
A
Z 2
A
(88)
1=1
where A, = A,(S;-1) is the marginal gain at step é and T is the total number of steps. The short half-life means that early steps contribute most, marginal gain decays quickly.
27

---

Published as a conference paper at ICLR 2026
Cumulative Marginal Gain (AUMG). The cumulative marginal gain, or Area Under Marginal Gain (AUMG), is the sum of marginal gains across steps:
AUMG = Σ A. ! A())dl
(89)
Am1
where the higher MG leads higher total accumulated utility.
PASS@k. is a standard metric for evaluating code generation models. For each problem, the model is allowed to generate k candidate solutions. The metric estimates the probability that at least one of those k samples passes all the unit tests for the problem:
if c k
PASS@k
(90)
1
if n - c<c k
where n is the total number of generated solutions, is the number correct solutions, and & the number of solutions sampled for evaluation.
Exact Match (EM). Exact Match Accuracy measures whether model's prediction exactly matches the ground truth answer. Formally, it can be defined as:
N EM 10==)))
(91)
where N is the total number of samples, Ús is the model's prediction for the i-th sample, y: is the corresponding ground truth, and 1(-} is the indicator function, which equals I if the condition is true and otherwise.
Accuracy (ACC). Accuracy measures the fraction of correct predictions among all predictions. Formally, is defined as:
N
ACC = N $111 1 E Y),
(92)
where N is the total number of samples, D; is the model's prediction for the i-th sample, Y; is the set correct answers for that sample, and this metric is less strict than Exact Match, as it allows multiple acceptable answers for single sample.
Prompt level Strict (Pr(S)). Prompt level Strict measures the percentage of prompts where all verifiable instructions are correctly followed. Formally, it is defined as:
Pr(S) = N Σ fall instructions in prompt are satisfied),
(93)
where N is the total number of prompts, and 1 } is an indicator function that returns if all verifi- able instructions within prompt are correctly followed by the model's response, and otherwise. This metric provides a strict evaluation of catruction-follwwiin adherence every instruction within given prompt (Zhou et al., 2023c).
capability, as it requires perfect
Multi-choice (MC2). MC2 (Lin et al., 2022) evaluates whether a model assigns higher proba- bility mass to the set of correct answers compared to the set of incorrect ones. Formally, is defindd as:
MC2
Σ
Σ Pi.o
Σpia Pi.a aEF.
(94)
where N is the total number of questions, T; and F; denote the sets of correct and incorrect answers for question i, and Pi,a is the normalized probability that the model assigns to answer a. Unlike MC1, which requires selecting single correct answer, MC2 emphasizes the model's ability to collectively assign more probability mass to truthful answers when multiple correct answers may exist, thereby providing finer-grained measure of truthfulness.
28

---

Published as a conference paper at ICLR 2026
G.3 BENCHMARKS
We evaluated the model from three aspects: Code Generation, Math Reasoning, and Multi task Knowledge and Reasoning. The following benchmark is evaluated through Im-evaluation (Gao al., 2024), and the detailed prompts please refer to Gao et al. (2024).
Datasets
Aspects
Metrics
# Nums
Evaluate
GSM8K (Cobbe et al., 2021)
Math Reasoning
Exact Match
1,319
4-shot
BBH (Suzgun al., 2022)
Complex Reasoning
Exact Match
6,511
3-shot
MMLU (Hendrycks et al., 2021)
Multi Task Understanding
Accuracy
14,042
ARC-Challenge (Clark et al., 2018)
Common Sense Knowledge
Accuracy
1.172
TruthfulQA (Lin al., 2022)
Factual Accuracy
MC2
[FEval (Zhou et al., 2023c)
Instruction Following
Pr(S)
477
HumanEval (Chen et al., 2021)
Code Generation
PASS @
164
3-shot
MBPP (Austin et al., 2021)
Code Generation
PASS@1
3-shot
Table 7: Statistical information of our benchmark datasets.
Grade School Math 8K (Cobbe et al., 2021) is benchmark dataset for evaluating the mathematical reasoning ability of language models. It contains grade school level math word prob- lems with step-by-step solutions. Models are typically evaluated using the EM metries. Given natural language word problem, the model is required to generate solution that includes the correct numerical answer. Problems range from simple arithmetic to more complex multi-step reasoning. And we will use ICI. to evaluate the performance under 4-shot.
GSM8K.
BBH. Big-Bench Hard (Suzgun et al., 2022) is challenging subset of tasks from the BIG-bench evaluation suite, specifically curated to include problems where language models typically perform below average human performance. logical reasoning, mathematics, world knowledge, and language understanding. Models are eval- uated using EM on each task. The dataset designed test complex reasoning capabilities that require multi-step thinking, pattern recognition, and domain -omain--pecific knowledge. Each task presents unique challenges, from formal fallacy identification to geometric reasoning and causal judgment. We will also evaluate the performance under 3-shot setting.
contains 23 diverse reasoning tasks spanning areas such as
MMLU. Massive Multitask Language Understanding (Hendrycks al., 2021) a benchmark for evaluating the broad knowledge and reasoning abilities of language models across multiple domains. contains 57 subjects covering topies from STEM, humanities, social sciences, and professional knowledge. Models are evaluated using the ACC metric. The dataset consists of multiple-choie questions with correct answer based on its knowledge and reasoning. No external tools calculators are allowed during evaluation.
answer options per question. For each subject, model is required to select the
ARC-C. A12 Reasoning (Clark et al., 2018) is dataset of 7,787 genuine grade-school level, multiple-choice science questions from grade 3 to grade 9, assembled to encourage research in ad- vanced question-answering The dataset is partitioned into Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both retricval-basee and a word co-occurrence algorithm. Models are evaluated using ACC metrics. Most questions have 4 answer choices, with less than 1% having either 3 or 5 answer choices. The questions cover diverse scientific domains and require genuine reasoning rather than simple fact retrieval pattern matching
algorithm
TruthfulQA. Lin et al. (2022) is a benchmark designed to measure whether language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. Questions are crafted that some humans would answer falsely due to false beliefs or misconceptions. Models are evaluated using MC2 metrics.
29

---

Published as a conference paper at ICLR 2026
IFEval. Instruction instruction-Folloving Evaluation (Zhou et al., (023c) is benchmark designed to evaluate the instruction- following capabilities of large language models. It contains around 500 prompts with verifiable instructions such "write in more than 400 words" and "mention the keyword of Al at least times". The benchmark identifies 25 types of verifialle instructions, with each prompt containing one more verifiable instructions. Models are evaluated using Pr(S) metrics that be verified by heuristics.
HvmanEval. Chen al. (2021) introduced the benchmark dataset for evaluating the functional correctness of code generation models. It consists of 164 Python programming problems, and com- monly used metries PASS@1 1. Given problem description and the function signature. the model is required to generate Python code that implements the specified function. A generated solution is considered correct if passes all the provided unit tests.
MBPP. Mostly Basic Python Problems (Austin et al., 2021) is benchmark for evaluating code generation capabilities language models. The benchmark consists around 1,000 crowd, sourced Python programming problems, designed to be solvable by entry, level programmers, covering pro- gramming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and automated test cases. Models are evaluated using PASS @ met- rics by executing generated code against the provided test cases. For the code generation, we will evaluate the benchmark of MBPP and HumanEval in 3-shot setting.
G.4 BASELINES
We selected multiple representative sfute-of-thh-ar data selection methods as baselines. Overall, can be divided into two selection methods: Model agnostic methods (Full, Random, IFD, and Fisher) and Model aware methods (LESS, SelectlT) (Albalak et al., 2024; Zhou et al., 2023a; Yu al., 2024). To ensure the fairness of the data screening experiment, we chose to select 10% of the data as the training set. The following introductions are all summarized from the original papers.
Full. Full data selction method uses the entire available training dataset without any data selection.
Random. A simple method where subset of the training data is randomly selected (Xia et al., 2024b). In the paper, they proposed that even only 1-2% of the total dataset is randomly selected, its performance can be comparable or better than other complex data selection methods. To ensure the fairness of the data selection experiment, we chose to randomly select 10% of the data as the training set
PPL. Ferplexity-basee selection method (Li et al., 2024a) ranks training samples according to their perplexity computed by pretrained language model. For a sequence (Wi,...: Wr), the perplexity is defined as
log Sogp(w. we meec)
PPL(x) exp
(95)
Samples with lower perplexity are considered closer to the model distribution, while higher perplex- ity samples are considered harder or less aligned with the model. Li et al. (2024a) achieved the best results when selecting the 10% of data with high perplexity as we used in the baselines. But simple perplexity "cannot directly represent the difficulty or quality of instruction tuning samples", they introduces the IFD (Li et al., 2024b).
IFD. Instruction Eitruction-Foolowing Following Difficulty (IFD) (Li al., 2024b), metric devised to evaluate the challenge each instructional sample presents, calculated the ratio between s(A) and s(AlQ) given a question-answer (Q, A) pair:
IF) s(A(Q) 1 ΣN Di=1 log P(a 10 s(A)
(96)
DE log P(z/1af
where s(A) denotes the Direct Answer Score, which quantifies the LLM's intrinsic capability to generate the response independently. s(A (Q) represents the Conditioned Answer Score, which is
30

---

Published as a conference paper at ICLR 2026
computed by sequentially predicting subsequent tokens given the instruction Q and their preceding context (Li et al., 2024b; Xia et al., 2024b).
We use the base model to calculate the IFD score for each sample, and then select the top 10% of samples with the highest IFD score and less than I for training. This method selects samples with instructions that indeed helpful (IFD<1) but still challenging (high IFD score), avoiding data that is too simple or instructions that are invalid.
Fisher. Deb al. (2025) adopted the Fisher matrix for greedy selection, using the GPT2 model for selection, It was found that selecting 50% of the 10000 data points had the best performance. However, he made some tracks to estimate the matrix and reduce computational complexity. The approximated formula is followed
score log det + ΣΣ )o.
977)
In order to better reproduce his results, we adopted the method of greedy and Fisher matrix for data selection, selecting 10% the data with more information. But only selects larger information gains which will cause the conflict problem shown in Figure 6.
Low-rank gradiEnt Similarity Search (LESS) (Xia et al., 2024a) estimates the influence of each training sample on target validation set by leveraging gradient similarity under the Adam optimizer. To improve efficiency, it uses LoRA to reduce parameter dimensionality and random projections to construct a reusable oow-dimensional gradient datastore.
LESS.
Given few-shot validation set Deat, the influence of a candidate training datapoint 2 is defined as
coll (VE(D 0,), F(2; 0.))
InfAdam Dea))) max
(98)
VE(D vall (1 0
where VE(D()) sat 02)
denotes the average projected gradient of validation subtask j at checkpoint O,, and [(2;; 0c) is the projected Adam update for 2. So selects the top 10% of training data with the highest influence scores for fine-tuning.
SelectIT. Liu al. (2025) exploits the intrinsic uncertainty of large language models select high-quality instruction tuning (IT) data without external resources. The method uses three levels of self-reflection: token-level uncertainty from rating confidence, sentence-level uncertainty across different prompts, and lovel-level uncertainty by combining evaluations from multiple foundation models. Given sample S = (X.Y) and K rating prompts (RPo,
}, the Selec-
IIT score is: Smodel Σ
N
Avg(Stoken (RP&))
(99)
O2 1 x Std|Stoken Stoken (RP&)
where Stoken (RPx) Shase X K-1 Σド= PPJ P Ps is the token-level score from model i using prompt k, 02 is the parameter count of model i, and a controls uncertainty weighting, Samples with highest 5modee scores are selected for training. So we replicated the selected data according to the config and code provided by the author in 3-level selection.
H DETAILED EXPERIMENTAL RESULTS
H.1 METHOО VISUALIZATION
In Section 4, we visualize the results of the method on the selected data, as shown in Figure When À 0, it is normal greedy search, and although high information samples selected, the gradient directions often conflict. Intuitively, this is not a phenomenon that can bring stable convergence training results.
31

---

Published as a conference paper at ICLR 2026
Score = Ay
Conflict
Nea-C Noa-Coufiitt
Conflict
Average Step
Direction
Ax
High
Greedy Search
À > 0: SPICE Search
X
Redundant Conflicting
Diverse & Non-Conflicting
Figure 6: Conceptual illustration showing how SPICE achieves better selection quality by avoid- ing high-conflict samples while maintaining step direction compared to greedy selection when selecting samples from one batch data.
H.2 TRAINER LOG
For all selected data, we perform SFT using the Llamafactory (Zheng et al., 2024) framework under the same configuration. Next, will present the loss graph of our training in Figure
shows training loss curves for all data selection methods on Qwen2-7B fine-tuning. SPICE demon- strates the most substantial loss reduction (2.33 70% decrease) with stable convergence, indicating effective identification of informative examples that challenge the model appropriately. While methods with lower initial losses converge smaller final values, SPICE's higher starting point reflects its ability to select more complex, potentially generalizable training instances rather than easier examples that may lead superficial optimization. This line with the phenomenon of reducing gradient conflicts, and the experiment also proves that our method has higher perfor- mance. Similarly, for the LLaMA2-7B model, did not clearly exhibit convergence mode of Qwen2-7B, but it also completed the training very well.
H.3 RESULTS OF SPICE+
SPICE+ (w = 0.5) means that through a data-driven early stopping strategy, the selection method adaptively select data compared to fixed selection of k per round. When the information enters the half-life, the selection stops, as described in Section 4.2. Therefore, we conducted more detailed analysis.
Due to our observation in the empirical experiment that the model often reaches less than half of the data when the half-life reached during greedy selection, we conducted the experiment with the following configuration (S = 0.5B, T 10, k 64, N 128) on Qwen2-7B and (S 7B, T 10, k 64, N Naxhaet = 128) on LLaMA2-7B, which means that we selected 60 out of 120 data, but stopped early if the half-life was reached. Through the results, we observed that most of the data reached their half-life at around 25-30% for early cessation. Based on this, we also conducted experiments
approximately 26% of the selected data. In Table 8. we that the sample with twice the amount of information brought by spending twice the time, but from the results, there was only slight improvement, but it is still very good result.
from it
Additionally, we also conducted experiments on other stopping thresholds w to validate the effec- tiveness of our method in the same settings below. The results shown in Table 9.
From the results, we can see that the data selection method SPICE+ with w 0.5 achieves the best performance, and the time cost is also the most appropriate. Moreover, it can also be seen that when
32

---

Published as a conference paper at ICLR 2026
Training Loss Comparison of Qwen2-7B
Training loss of random
Training loss of ifd
Training loss of fisher
0.50
orighnl
0.78
orggnnl smoothed
originl
0.76
1.75
0.45
0.74
Loss
1.25
0.72
0.40
1.00
0.70
0.35
0.68
0.50
20
40
60
10
20
30
40
50
10
20
30
40
50
Steps
Steps
Steps
Training
Training selectIT
Training
onignal smrothed
0.55
original smosthed
original
0.28
2.0
smoothed
0.26
0.50
losc
0.22
1.0
0.20
20
40
60
10
30
40
Steps
Steps
Steps
Training Loss Comparison of LLaMA-2-74
Training
Training
Training
0.9
0.90
3.0
smoothed
2.5
0.85
L053
LoGS
0.80
1.5
0.75
1.0
20
40
60
40
50
10
20
40
50
Steps
Steps
Training loss of less
Training loss of selectIT
Training loss of SPICE
0.60
sm oothed
orghha
orginal smoothed
3.0
smocteed
origiaa
0.55
2.5
Loss
Lass
2.0
0.7
1.5
0.45
0.6
1.0
40
30
Steps
Steps
Steps
Figure 7: The training loss comparison of various methods on Qwen2-7B and LLaMA2-7B, We trained for 3 epochs, with 5 steps record the loss. Each method was kept around 60 steps, with 20 steps per epoch. SPICE achieves the steepest and most stable loss reduction (2.33-40.69 70%), reflecting effective identification of informative and challenging examples that enhance convergence. Its higher starting loss indicates selection of more complex samples, which promotes generalization compared methods biased toward easier instances. On Qwen2-7B. SPICE shows the clearest convergence advantage, while on LLaMA2-7B the trend is less pronounced but training remains effective.
33

---

Published as a conference paper at ICLR 2026
Method
GSM8K
BBH
MMLU
ARC
T-QA
IFEval
H-Eval
MBPP
T(h)
Owen2-7B
SPICE(10.0%)
86.7
61.0
67.1
51.8
55.5
38.6
47.1
56.2
2:56
SPICE+(26.19)
86.7
61.3
67.3
52.0
55.5
38.6
47.5
56.2
5:49
A
+0.0
+0.3
+0.2
+0.2
+0.0
+0.0
+0.4
+0.0
+2:53
LLaMA2-7B
SPICE(10.0%)
13.8
40.8
41.9
47.7
40.3
22.6
16.7
24.6
6:10
SPICE+(29.8%)
13.8
41.5
42.1
47.3
40.3
22.6
16.9
24.6
10:55
A
+0.0
+0.7
+0.2
-0.4
+0.0
+0.0
+0.2
+0.0
+4:45
Table 8: Evaluation of data selection methods between SPICE and SPICE+. Among them, T-QA represents TruthfulQA, H-eval represents HumanEval dataset, and T(h) represents Time Cost (h). It can be seen from this that selecting more data under SPICE+ brings almost doubled time cost, but the same time it brought little improvement.
Data Rate
Average
Time Cost
0.1
55.1%
67.1
9:55
0.3
34.5%
67.2
7:01
0.5
26. 1%
67.2
5:49
0.7
9.3%
67.1
2:45
Table The results of the data selection method SPICE+ with different stopping thresholds w. The data rate the ratio the selected data to the total data. The average is the average score of the benchmarks. The time cost is the time cost of the data selection method.
we set w to 0.1, only half of the data is collected, indicating that the data quickly reaches very low marginal gains in the later stages of greedy selection.
H.4 MORE STUDY PROXY MODEL.
In Section 5.4, conducted ablation study the proxy model of Qwen2 (Yang et al., 2024a), and the results are shown below in Table 10. In addition, considering the parameters of Ilama, we also used LLaMA2-7B (Touvron et al., 2023) proxy model select data and use it SFT Qwen2-7B.
Proxy Model
Step Intervals
Qwen2-0.5B
Qwen2-1.5B
Qwen2-7B
LLaMA2-7B
67.0
67.1
67.2
65.9
5
67.0
66.9
67.2
66.1
50
66.6
66.9
67.1
66.
Average
66.9
66.8
67.1
66.1
Table 10: Results of the average performance GSM8K and HumanEval) of different struc- ture and size of Proxy model for selecting data SFT Qwen2-7B.
Obviously, when using Qwen2-7B itself as a proxy model, it will bring better performance, which also line with my method. Table we have also conducted experiments on the proxy model of LLaMA2-7B, and it shows that the data selected by the proxy model of LLaMA2-7B did not bring good results compared to Qwen2-7B (limited cross-architectue transfer). which line with the model training and selection of our method itself.
For scaling to Larger Models, we extend our experiments to 70B+ target models while using small fame-family proxies (0.5B/7B). Even when fine-tuning 70B-scale models, SPICE subsets selected by small proxies match or outperform full-data LoRA, showing that very large proxies are not nec- essary in practice. Using models from 30B-70B proxies would require significant computational
34

---

Published as a conference paper at ICLR 2026
SPICE (Proxy Model)
SFT Model
Null
Full
Qwen2-0.5B
Qwen2-7B
LLaMA-7B
Qwen2-0,5B
31.1
30.8
29.9
29.8
30.5
Qwen2-7B
67.0
67.2
66.1
62.0
65.2
Qwen2-72B
77.9
78,0
77.4
77.2
77.5
LLaMA2-7B
23.6
23.4
24.1
22.2
23.6
LLaMA2-70B
51.4
51.4
50.9
50.9
51.1
Table 11: Performance across target SFT models with different SPICE proxy models (10% budget).
Average Instruction Length Method
Muthod Similarity
Instruction Semantic Embeddings 1-SNE
471.6
Rober Selac
035
20033
TIME COrpOtMnt
0691
0007
232-8
335.1
247.8
0199
5.mh
5:89
1403
6477
-075
Ramsary
8686
3504
1550
1009
1,07
8002
35
SROE
0047
2A85
1653
1477
1332
1/01
Roher SelectiT
Less
FD
Random SPICE
Figure& 8: (a) Average instruction length of selected subset data by various methods. (b) Method similarity heatmap of instruction embeddings. (c) Instruction semantic embeddings t-SNE.
and memory overhead, going against SPICE's design goal of being lightweight selection mecha- nism.
H.5 STATISTICS OF SELECTED DATA
Statistical Analysis. In Section 5.4, we conducted statistical analysis on the selected 10% of data, first calculating the average instruction length of the data, and then random sample 10% of selected subset, to evaluate the semantic properties of the selected subsets, we embedding samples using all-Mini M embeddings and visualize the embeddings with t-SNE and similarity heatmap.
As shown in Figure (a), can see that information based SPICE and Fisher tend to choose data with lower length, while Less selects data with the longest length; In (b) and (c), the data we selected also have a relatively high similarity with Fisher, but through the trade-off of gradient conflict, we can select the data with better performance among them. In addition, we found that out of over 90000 data, only over 40000 data were selected under all data selection methods, and nearly half of the data points were not selected. Among them, math problems were selected the most.
Baseline
Jaccard
Overlap @ Ours
Overlap @ Base
Fisher
0.47
0.64
0.64
Random
0.05
0.10
0.09
IFD
0.02
0.03
0.03
SelectI
0.08
0.15
0.15
LESS
0.01
0.02
Table 12: Pairwise overlap between baselines and our selection: Jaccard, Overlap Ours, and Over- lap@ Base. Among them, Fisher, as non-conffict-wware information-bast method, has the highest overlap with SPICE's data.
Overlap Analysis. We summarize pairwise agreement with three set-overlap scores: Jaccard, Overlap @ Ours, and Overlap @ Base. Table 12 shows that Fisher exhibits the strongest agreement with our selector (Jaccard 0.47; Overlap@Ours 0.64; Overlap Base = 0.64), indicating that both methods consistently capture shared core of high-information examples. By contrast,
35

---

Published as a conference paper at ICLR 2026
Cluster-Coverage Analysis
100
95
92.1
92.6
Converage@1000
90.6
90
84.5
84.5
85
79.8
80
75
70
ifd
random
less
SelectIT
fisher
Figure 9: Brief cluster-hit histogram over K =-1000 clusters; Coverage @ 1000 denotes the fraction of clusters with least one selected item (higher indicates broader semantic coverage).
Random (Jaccard = 0.05; Overlap@ Ours = 0.10; Overlap@ Base = 0.09), IFD (0.02/0.03, 10.00.03000000 /0.03,, SelectIT (0.08/ (0.08/0.15/00 /0.15/0.15), and LESS 00.010.00/00.00 show substantially lower overlap with our picks, suggesting that their selection criteria emphasize different regions of the data space. This pat- tern supports the region that our method preserves the high-information "core" while differing more marginal contentious items. Building this observation, next turn to targeted case studies contrasting items jointly selected by non-conflict-and information -based selection and ours.
Cluster-Coverage Analysis. To verify that our selected subset spans the major semantic modes in the pool, we use the full data into K=1000 clusters (built once on instruction embeddings). We then map the already selected items onto these clusters and report single coverage statistic, Coverage @ 1000, together with a brief cluster-hit histogram for context. Broad coverage (high Cov- erage@ 1000 with non-collapsed hits) indicates that method surfaces high-quality examples from many data modes rather than concentrating on a few niches. This descriptive evidence helps explain why 10% budget can outperform using all data: it preserves representative, informative items across clusters while avoiding redundant or contentious samples.
As shown in Figure under the Coverage@@ 1000 metric all methods cover 80% of clusters. Random reaches 92.6, while ours is close behind at 92.1. Even with 1 1,000-way partition, our subset still hits 92.1% of clusters, indicating broad semantic dispersion, i.e.. we consistently select at least one high-quality item per cluster. This result helps explain how ~10% budget can match or surpass full-data training by capturing representative, informative examples while avoiding redundancy.
Case Studies. As shown above, we preserve the Fisher's informational core while removing boundary samples whose gradients strongly oppose the set's mean gradient direction, thereby mitigating marginal-gain decay and leading to a larger cumulative information. Accordingly, we will analyze the high-con high-conflict cases that were excluded and -igh-mnformtiinn cases that were overlapped; Table 13 lists 6 such cases.
H.6 EXPANDED RESULTS AT 1% ~30% DATA BUDGETS
In the main experiment (Section 5.2), we have shown that our method can outperform the full data on Qwen2-7B and LLaMAA-7B with 10% data. To assess robustness of our main conclusions under different data budgets, replicate the full graining-and-sllecti subset fraction of the training pool. Concretely, we fix all model, optimization, and hyperparam- eters to the main-experiment (10% values and run other three budget levels: 1%, 5% and ~30% (SPICE+) compared to 100% (full data), Fisher (non-conflict--ne conrlicttaware information- method), DPP and Null (No fine-tunning). The results shown in Table 14.
pipeline while varying only the
As shown in Table 14, across both base models the performance increases monotonically as the selection budget grows from 5% to 300% with larger gains from 5% 10% and smaller gains from
36

---

Published as a conference paper at ICLR 2026
Type
Instruction
Response
Given the following list of words, come up with a sentence using as many of them as possible. Rainbow, Song, Cat, Inception
The rainbow appeared after the storm, the cat curled up next to me as we listened to a beautiful song from the movie Inception.
E
What is a linked list?
A linked list is a data structure in which each element contains data well link to the next element in the list. Linked lists provide dynamic allocation memory which can useful for cer..
Select all rows a pandas dataframe where at least one value exists in the column named 'name".
selected_rows = df' [dfnname.notullO]
S
(4 7 2
The answer this equation is 41. Resilient.
S
Create word that best describes the following definition. person who loyal despite
difficulties
S
capitalize all the words in a string input: hello world
Hello World.
Table 13: Case study of selections between non-conflict-wware information-based selection and SPICE: 3 excluded (E) and 3 shared (S) cases on default settings.
10% +0%%. For example, on Qwen2-7B (SPICE) the average score rises from 56.5-+58.0-458. where the 10%-+~30% gain is clearly smaller than the 5%-100% gain. LLaMA2-7B shows the same trend.
Compared with Full Data (100%), our method at 10% and ~30% outperforms the full-data baseline on both models; at 1% and 5%, it is close to full data on Qwen2-7B but lower on LLaMA2-7B. This indicates that even with tiny data, the diminishing-retura benefits of information- selection can outperform budget expansion. Moreover, relative to Fisher, an information-base greedy selection that does not consider conflicts, our approach remains stronger even at the 1% and 5% budget, suggesting that conflicting samples lower overall gains. At the task level, our method de- livers notably larger improvements IFEval, indicating stronger benefits for Eetructiion-ollloviin while reasoning and knowledge benchmarks improve more steadily with the budget. Overall, the ctaptive-stopping method (SPICE+) attains the best performance.
H.7 COMPUTATIONAL COMPLEXITY
Standard Fisher-based greedy selection scales as Okk D(k)D)) due to repeated log-det evaluations, which is prohibitive for modern LLMs. In contrast, SPICE leverages (i) AdaFisher (Deb et al., 2025) to reduce the Fisher update cost from (dd) to (d). (ii) a simple inner-product based conflict penalty that also costs O(d) per sample, and (iii) proxy models for gradient computation. Conse- quently, the overall complexity is reduced to O (D)))d) Did), which shown in Section 1. This linear-in-d complexity makes awwrr data selection feasible at billion-parameter scale (10B+), highlight- ing its scalability advantage.
Specifically, the detail pipeline is as followed:
Step 1: Single-pase gradient computation Each example D is used once to compute its per-sample gradient at a fixed proxy checkpoint. These gradients Ja are cached and reused for all subsequent greedy steps.
Gradient compute: O(ID)d)
Memory: 0(md)
where d is the proxy model dimension and is the candidate-pool size.
Step 2: In-pool scoring For each candidate pool C with |C| = m:
Fisher marginal gain (diagonal AdaFisher): Odd) per example;
37

---

Published as a conference paper at ICLR 2026
Budget
Method
GSMSK
BBH
MMLU
ARC
T-QA IFEval
H-Eval
MBPP
Avg
Qwen2-7B
0%
Null
77.8
60.3
64.4
48.3
54.3
25.8
43.9
53.8
53.6
1%
DPP
77.9
60.4
64.3
48.4
54.2
26.0
44.8
53.8
53.5
10%
DPP
86.5
61.0
66.0
51.0
55.0
35.4
45.0
55.7
57.0
1%
Fisher
77.8
60.4
64.2
48.5
54.1
26.3
45.1
54.0
53.8
5%
Fisher
83.4
65.2
50.1
54.5
30.6
43.7
54.0
55.3
10%
Fisher
86.5
60.8
65.2
50.3
55.0
30.6
44.5
54.6
55.9
~30%
Fisher
86.5
61.1
67.0
50.8
55.2
33.8
45.2
56.2
$7.0
1%
SPICE
78.1
60.3
64.8
49.2
54.1
26.3
46.9
54.0
54.2
5%
SPICE
83.5
60.4
64.8
50.5
55.2
35.5
46.7
55.0
56.5
10%
SPICE
86.7
61.0
67.1
51,8
55,5
38.6
47.1
56.2
58.0
330%
SPICE+
86.7
61.3
67.3
52.0
55.5
38.6
47.5
56.2
58.1
100%
Full Data
84.2
61.3
65.7
50.5
54.8
33.5
45.7
55.2
56.4
LLaMA2-7B
0%
Null
12.7
39.9
39.9
43.1
38.8
18.8
14.0
23.0
28.8
1%
DPP
12.7
40.0
40.0
44.0
38,9
19.1
14.5
23.0
28.9
10%
DPP
13.6
40.2
41.0
47.0
40.0
22.2
15.9
23.5
30.4
1%
Fisher
13.0
40.0
40.3
43.5
38.8
19.0
14.4
22.8
29.0
5%
Fisher
13.4
40.1
41.0
46.2
39.3
21.8
15.1
23.0
30.0
10%
Fisher
13.6
40.5
41.5
46.7
39.3
22.0
15.2
23.2
30.3
300%
Fisher
13.7
41.4
41.9
47.1
40.1
22.5
16.7
24.3
31.0
1%
SPICE
12.7
40.1
40.1
45.9
38.9
20.0
14.6
22.4
29.3
5%
SPICE
12.8
40.5
40.5
46.0
40.5
21.0
16.0
24.0
30.2
10%
SPICE
13.8
40.8
41.9
47.7
40.3
22.6
16.7
24.6
31.1
~00%
SPICE+
13.8
42.1
47.3
22.6
16.9
24.6
31.2
100%
Full Data
13.6
41.3
40.8
46.1
43.5
19.4
16.5
25.2
30.8
Table 14: SPICE per-benchmark scores at 1 1%, 5%, 10%, and ~-30% budgets compared to 0% (Null), 100% (full data), Fisher (non-conffict-awar) information-based method) and DPP. Among them, T- QA represents TruthfulQA, H-eval represents HumanEval dataset, and ~30% means that SPICE+ used (26.15 % for Qwen2-7B and 29.6 29.8% for LLaMA2-7B).
* Conflict score (cosine w.r.t. mean gradient): (dd) per example.
Thus, one scoring pass costs
((m)).
Step 3: In-pool greedy selection For each pool, selecting kipool examples requires rescoring the remaining candidates:
Okkpoal m d).
Summing across pools with Σ Apoot = k gives
()d).
Step 4: Proxy model updates (interval T) which adds
SPICE updates the proxy model once every T pools,
O(kd),
negligible relative to the selection cost.
Step 5: Overall time and space complexity
Thaal O(D))d) OkkDD))) O(kd) (D]
Peak memory usage is
O(md),
since only store gradients for pool time.
38

---

Published as a conference paper at ICLR 2026
Summary SPICE is single-pass, streaming procedure: (i) each example in D produces exactly one par-sample gradient; (ii) all subsequent selection steps reuse cached gradients; (iii) no full- dataset recomputation of gradients is performed; (iv) complexity scales linearly with dataset size and model dimension.
H.8
TIME COST
All experiments run on the same 8xH20 GPU node with a shared LoRA SFT setup across methods (3 epochs, same batch size, max length, optimizer, LR schedule, and LoRA hyper-parameters) Under a fixed 10% data budget, fine-tuning cost is therefore identical across methods, and the main difference comes from selection time.
Table reports selection time on this 8-GPU node together with average performance. SPICE needs only 2:56 of selection time, and its total time (selection + 10% LoRA SFT) is lower than full-data LoRA, while achieving higher performance on both Qwen2-7B and LLaMA2-7B.
Method
Selection Time Cost
Selection Computation
Performance (Qwen2-7B)
Performance
(hour:min)
Complexity
(LLaMA2-7B)
Full
00:00
O
56.4
30.8
Random
00:00
O(k)
55.5
30.1
SPICE
02:56
O[k] DIld)
58.0
31.1
IFD
04:32
55.4
30.0
LESS
16:22
O(lm)|Dild)
55.0
30.3
Fisher
17:01
55.2
30.2
SelectIT
25:19
55.8
30.1
DPP
23:32
WUND-NND ND)
56.3
30.5
TSDS
00:05
O(MLlog
55.4
29.9
LEAD
12:02
56.5
31.0
Table 15: Time cost and computation complexity.
39

---

Published as a conference paper at ICLR 2026
1 LARGE LANGUAGE MODELS USAGE
Did you use Large Language Models (LLMs) in paper writing? Yes, LLMs were used solely for polishing writing and formatting of equations. No content, data analysis, interpretation was gen- erated or influenced by LLMs. All research results and conclusions were independently developed by ours.
40


Reviews:
[{"review_id": "bhmDs0CUVF", "reviewer": "Reviewer_Y2FZ", "summary": "The paper proposes SPICE, a conflict-aware data selection method for instruction tuning. It starts from the submodular log-det(Fisher) objective, shows that marginal information gains decay faster when gradient conflicts are high, and formalizes this using an ε-decomposition → curvature analysis. SPICE scores each sample by (Fisher marginal) − λ·conflict (conflict = negative cosine to the running mean gradient), supports early stopping, and allows proxy models for efficiency. On LLaMA2-7B and Qwen2-7B, using ~10% of data, SPICE matches or exceeds full-data and several selectors (LESS, Fisher, SelectIT, IFD) across 8 benchmarks.", "questions": "Benchmark coverage. Please add stronger baselines, e.g., the method in arXiv:2402.02318 and a LoRA-only tiny-data baseline (e.g., 0.5–2% data) to show SPICE’s advantage at very small budgets.\n\nTiny-data behavior. How does SPICE compare to straightforward LoRA with small training sets across tasks? Any regime where plain LoRA beats SPICE-selected 5–10%?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear theory–practice link: ε-decomposition → curvature explains greedy degradation under gradient conflicts.\n\nSimple, practical selector: Fisher-marginal − λ·conflict with early stopping and proxy models; easy to drop into pipelines.\n\nSolid empirical sweep: two bases (Qwen2-7B, LLaMA2-7B), 8 benchmarks, cost/ablation studies; strong gains on IFEval/MMLU at ~10% data.", "weaknesses": "Assumption fragility: bounds rely on α‖F‖<1 and AdaFisher approximations; reported violation rates in higher-conflict regimes weaken guarantees.\n\nLimited baselines: several strong recent selectors and tiny-data LoRA baselines are missing; Random is competitive in places.\n\nConflict proxy is heuristic (−cosine to mean gradient); sensitivity to optimizer/state/batch is underexplored, and cross-architecture transfer is weak.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zA2leRccyl", "reviewer": "Reviewer_rHBr", "summary": "The paper studies selecting instruction-tuning datasets by proposing to avoid gradient conflicts. The authors develop an epsilon-decomposition that splits the Fisher gain into a baseline and a perturbation term and showed that the perturbation is upper bounded by squared gradient inner products. The authors proposed SPICE, a greedy data selection algorithm that scores a candidate example by the marginal gain as well as a gradient penalty term. Empirically, the authors demonstrated that at 10% data matches or outperforms full-data SFT and several baselines while reducing selection/training cost.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper is very well written - clear and flows well from theory to an a practical algorithm inspired by the theory. The experiments seemed pretty complete as well.\n- The proposed SPICE selection algorithm is simple and intuitive as well\n- The experiments compared several baselines on multiple benchmarks. The gain is pretty consistent.", "weaknesses": "- It would be interesting if the authors can demonstrate whether the finding can be extended to larger corpus / base LMs as behaviour might change as we scale up the model size.\n- It would be nice if the authors could provide us with qualitative examples to better understand what constitute examples that has low/high gradient conflict. Is there some intuition as to what they might imply to the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epudZa9BN2", "reviewer": "Reviewer_Xhgm", "summary": "This paper proposes a submodular framework for data-efficient language model fine-tuning, introducing a conflict-aware selection mechanism that balances information gain and gradient disagreement. The method improves greedy subset selection efficiency and achieves competitive results.", "questions": "1. *Theorem 1 (rows 167–169):* Why do large perturbations lead to faster decay? Shouldn’t it be the difference between successive perturbations, not the absolute magnitude of a perturbation, that drives faster decay?\n2. *Definition 4:* Corollary 1 penalizes both similar and opposite gradients via squared inner products, while Definition 4 only penalizes opposite ones. Why are similar gradients (redundancy) ignored, given that the theory penalizes both?\n3. *Pipeline clarity:*\n    - Section 4.3 (row 346): When stating “at each iteration, we select one sample using our conflict-aware greedy algorithm,” does ‘one sample’ refer to a single example or a mini-batch of k samples? If it refers to a single example, does the model get updated after each selection (get updated after seeing a new example) when T=1?\n    - Section 5 (row 372): How is the 120-sample candidate pool formed? Is it randomly drawn from D with size k×T?\n    - Is the proxy model updated after each cycle?\n4. *Baselines and related work:*\n    - If the proxy model is periodically updated and selection occurs within a randomly sampled “candidate pool”, the setup seems closer to online batch selection, making comparisons to FisherSFT, LESS, or IFD, a non-periodic selection mechanism, potentially unfair. It remains unclear whether SPICE’s performance gains stem from the periodic schedule or the proposed selection mechanism.\n    - Representation-based selection methods [1] and other recent instruction-tuning data selection works [2-4] are not discussed.\n5. *Complexity analysis:* Could the authors provide an explicit asymptotic analysis of time complexity? Algorithm 1 appears to require gradient computation over the entire dataset D for each selection, which seems computationally expensive.\n\n[1] Ivison, H., Zhang, M., Brahman, F., Koh, P. W., & Dasigi, P. (2025). *Large-Scale Data Selection for Instruction Tuning*. arXiv preprint arXiv:2503.01807.\n\n[2] Liu, Z., Karbasi, A., & Rekatsinas, T. (2024). *TSDS: Data Selection for Task-Specific Model Finetuning*. In *The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024)*.\n\n[3] Wang, J., Lin, X., Qiao, R., Koh, P. W., Foo, C.-S., & Low, B. K. H. (2025). *NICE Data Selection for Instruction Tuning in LLMs with Non-differentiable Evaluation Metric*. In *Forty-second International Conference on Machine Learning (ICML 2025)*.\n\n[4] Chen, Y., Li, Y., Hu, K., Ma, Z., Ye, H., & Chen, K. (2025). *MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space*. In *Findings of the Association for Computational Linguistics: ACL 2025*.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Novel insight into the fast decay of marginal contribution to enhance greedy submodular optimization.\n2. Comprehensive experiments and ablations supporting the method’s effectiveness.\n3. Decent performance on various benchmarks.", "weaknesses": "1. High computational cost due to gradient retrieval for each selection step.\n2. Possible misalignment between the theoretical motivation and empirical design; the overall selection pipeline remains somewhat unclear (see questions).\n3. Limited baseline comparisons (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gOB8Vf7voN", "reviewer": "Reviewer_KYWw", "summary": "This paper addresses the problem of selecting a small subset of instruction-tuning data for large language models in order to fine-tune efficiently. The authors observe that while maximizing the log-determinant of the empirical Fisher information matrix yields a submodular objective, in practice marginal gains collapse quickly due to gradient misalignment among samples. They formalize this by decomposing the marginal information gain into a base term and an interaction term, and conclude that controlling gradient conflict is key to sustaining information gain. Based on this, they propose SPICE: a selection algorithm that (1) uses a scoring function that subtracts a conflict penalty from the marginal information gain, (2) optionally stops early once the marginal gain falls below a threshold, and (3) uses a proxy (smaller) model to compute the gradients efficiently. They empirically show that on multiple benchmarks, using only ~10% of data, SPICE matches or exceeds full-data fine-tuning and outperforms several baselines, while reducing computation cost.", "questions": "How sensitive is SPICE to the choice of proxy model? If the proxy model differs in architecture or domain from the fine-tune target, how does performance vary?\n\nIn domains with heterogeneous instruction types (e.g., chat, coding, planning) where gradient directions may naturally differ, how does the conflict penalty trade off between “reducing harm” vs “reducing diversity”? Have you analysed domain-coverage of the selected subset?\n\nCould you provide more detailed hardware/GPU-hour breakdowns (selection + fine-tune) for each baseline method (e.g., LESS, SelectIT, FisherSFT) under identical hardware, to strengthen the cost-efficiency claim?\n\nHave you tested SPICE on larger models (>30 B) or other modalities (vision+language) or RLHF settings? If not yet, what do you foresee as the main challenge in scaling?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The method addresses both effectiveness (maintaining or improving performance with fewer training samples) and efficiency (using proxy model selection & early stopping) — a nice combination.\n\nEmpirical results are broad (multiple benchmarks, models, tasks) and show impressive savings (≈10% data) with no performance loss and even gains in some cases.\n\nThe algorithm is extensible: the idea of “penalize conflict” could be applied in other data-selection or multi-task contexts.", "weaknesses": "The current scope of experiments is limited to ~7 B-parameter models and instruction-tuning; extension to larger models (>30 B), multimodal tasks, or RLHF settings remains to be seen.\n\nThe proxy-to-target model transfer is shown only within same architecture family; cross-architecture transfer (e.g., completely different model family) may degrade and is less explored.\n\nThe cost comparison, while present, could be strengthened with more granular breakdowns (selection cost vs fine-tune cost) across all baselines under identical hardware settings.\n\nThe penalty on “conflict” implicitly biases toward samples aligned with current gradient direction—there is a risk that samples with contradictory but important signals might be under-selected; more analysis of diversity vs conflict trade-offs would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)