Title: SPICE: Submodular Penalized Information–Conflict Selection for Efficient Large Language Model Training
Primary area: foundation or frontier models, including LLMs
Keywords: Data selection; Submodular; Log-determinant Fisher information; Instruction tuning

Abstract: Information-based data selection for instruction tuning is compelling: maximizing the log-determinant of the Fisher information yields a monotone submodular objective, enabling greedy algorithms to achieve a $(1-1/e)$ approximation under a cardinality budget. In practice, however, we identify alleviating gradient conflicts, misalignment between per-sample gradients, is a key factor that slows down the decay of marginal log-determinant information gains, thereby preventing significant loss of information. We formalize this via an $\varepsilon$-decomposition that quantifies the deviation from ideal submodularity as a function of conflict statistics, yielding data-dependent approximation factors that tighten as conflicts diminish. Guided by this analysis, we propose SPICE, a conflict-aware selector that maximizes information while penalizing misalignment, and that supports early stopping and proxy models for efficiency. Empirically, SPICE selects subsets with higher log-determinant information than original criteria, and these informational gains translate into performance improvements: across 8 benchmarks with LLaMA2-7B and Qwen2-7B, SPICE uses only 10% of the data, yet matches or exceeds 6 methods including full-data tuning. This achieves performance improvements with substantially lower training cost.
Code is available at https://anonymous.4open.science/r/SPICE-6DF7/README.md.

TL;DR: We prove gradient conflicts accelerate marginal log-det FIM decay through ε-analysis , and introduce SPICE—an adaptive conflict-penalized greedy selector that matches full-data results with 10% data.