Title: Quadratic Coreset Selection: Certifying and Reconciling Sequence and Token Mining for Efficient Instruction Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: instruction tuning;principled data selection for LLM
Abstract: Instruction-Tuning (IT) was recently found the impressive data efficiency in post-training large language models (LLMs). While the pursuit of efficiency predominantly focuses on sequence-level curation, often overlooking the nuanced impact of critical tokens and the inherent risks of token noise and biases. Drawing inspiration from bi-level coreset selection, our work provides the principled view of the motivation behind selecting instructions' responses. It leads to our approach Quadratic Coreset Selection (QCS) that reconciles sequence-level and token-level influence contributions, deriving more expressive LLMs with established theoretical result. Despite the original QCS framework challenged by prohibitive computation from inverted LLM-scale Hessian matrices, we overcome this barrier by proposing a novel QCS probabilistic variant, which relaxes the original formulation through re-parameterized densities. This innovative solver is efficiently learned using hierarchical policy gradients without requiring back-propagation, achieving provable convergence and certified asymptotic equivalence to the original objective. Our experiments demonstrate QCS's superior sequence-level data efficiency and reveal how strategically leveraging token-level influence elevates the performance ceiling of data-efficient IT. Furthermore, QCS's adaptability is showcased through its successes in regular IT and challenging targeted IT scenarios, particularly in the cases of free-form complex instruction-following and CoT reasoning. They underscore QCS's potential for a wide array of versatile post-training applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nQuadratic Coreset Selection (QCS) addresses the data efficiency bottleneck in instruction-tuning large language models by jointly leveraging information at both the sequence and token levels. The authors first present a coreset perspective on next-token prediction, demonstrating that existing sequence-level subset selection and emerging token-level mining methods can both be formulated as distinct bi-level optimization problems. They then introduce QCS as a unified objective whose solution set strictly subsumes those of the individual problems—guaranteeing that a model trained on a QCS-selected subset is at least as expressive as one trained using either sequence-only or token-only selections.\n\nSolving the original QCS objective requires inverting Hessians at the scale of the model, which is computationally infeasible. To overcome this, the authors propose a probabilistic relaxation: inclusion indicators for sequences and tokens are modeled as Bernoulli random variables with learnable parameters. These parameters are optimized using a hierarchical policy gradient estimator that avoids backpropagating through the language model. A projection step enforces subset size constraints. The authors prove that this relaxed formulation is asymptotically equivalent to the deterministic QCS objective, and that the projected updates converge under mild conditions.\n\nEmpirical results show that QCS achieves comparable or superior performance to the targeted instruction-tuning baseline LESS using only 5% of the data, while requiring significantly less memory and computation. When the token-level component is enabled, QCS even matches or surpasses full-data performance on benchmarks such as TYDI-QA and BBH. Further experiments demonstrate QCS’s advantages in capturing critical tokens, handling noisy labels, and supporting continual and transfer instruction tuning—highlighting the value of integrating sequence- and token-level influence in a single coreset-based framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The authors propose a quadratic coreset objective that unifies sequence- and token-level subset selection. The resulting solution set is provably at least as expressive as either granularity alone, representing a clear conceptual advance for data-efficient instruction tuning.\n\n- A Bernoulli relaxation with a hierarchical policy-gradient estimator is introduced to avoid intractable Hessian inversions while preserving asymptotic equivalence to the original QCS formulation. This makes the method practically feasible on standard hardware.\n\n- The method achieves full-data or better accuracy on TYDI-QA and strong parity on MMLU and BBH using only 5% of the training sequences, reducing memory from approximately 17.7 GB to 176 MB and eliminating tens of GPU-hours typically required for gradient-based data mining—demonstrating strong practical value.\n\nWeaknesses\n\n- The experimental scope is limited, with results reported on only three benchmarks. QCS also does not outperform LESS on TYDIQA or BBH, suggesting that broader validation is needed to establish generality.\n\n- The wall-clock efficiency and latency benefits remain unclear. While memory and compute savings are reported, the outer-loop optimization takes considerably longer than LESS warm-up, making the real-world efficiency somewhat speculative. Due to the introduced quadratic coreset selection scheme, it is not surprised to see the performance may outperform LESS. However, with a more complex scheme, the stability may be worse.\n\nQUESTIONS:\nGeneralizability beyond current benchmarks\n- You report results on MMLU, TYDI-QA, and BBH. Could you evaluate QCS on at least one additional instruction-tuning benchmark with a distinct task profile—such as GSM8K (math reasoning) or AlpacaEval (open-ended generation)—to test its generalizability? If performance drops in these settings, could you provide an error analysis to help understand where the quadratic coreset formulation may be falling short?\n\nWall-clock efficiency and scaling behavior\n- While Table 3 details memory savings, the runtime benefits remain unclear. You mention that the QCS outer-loop takes approximately 30 minutes, compared to ~6 minutes for LESS warm-up. Could you provide a comprehensive report of end-to-end GPU hours and wall-clock time for QCS, LESS, and full-data fine-tuning at both the 7B and 13B model scales? Additionally, would you consider including a plot of accuracy versus training time across data budgets (e.g., 1% to 10%) to better illustrate the cost-efficiency trade-offs?\n\nTypos and formatting issues\n- There are several textual inconsistencies that may impact readability. Would you be able to fix the following?\n\nLines 61–65: \"OCS is further evidenced...\" → should be \"QCS\".\n\nLines 280–282: \"Benorlli distribution\" → should be \"Bernoulli distribution\".\n\nTable 1 caption: \"fined-tune\" → should be \"fine-tuned\".\n\nLines 269, 280, etc.: Standardize model name as either \"LLaMA-2\" or \"LLAMA-2\" consistently.\n\nLine 220: \"as we previously discuss\" → should be \"as we previously discussed\".\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of coreset selection for LLM instruction tuning, both at the sequence and the token levels. The authors propose an optimization problem that optimizes both sequence selection and token selection within the selection sequence. To overcome the prohibitively high computation cost of solving the original optimization problem, the authors proposed a probabilistic variant, which assigns probabilistic densities to the training set. In addition, the authors further extend the proposed method for transfer instruction tuning,\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- To the best of my knowledge, this work is the first that focuses on both sequence-level and token-level coreset selection, providing a more fine-grained approach to instruction selection for LLM tuning.\n- The paper demonstrates strong technical depth. The relaxation of the original optimization problem is innovative and reasonable, with a detailed analysis.\n- The extension to transfer instruction tuning shows the broader applicability of the proposed method.\n\nWeakness:\n- While the paper provides extensive technical detail, it lacks sufficient high-level intuition and explanation, which makes it difficult to grasp the main idea. For example, the absence of a step-by-step algorithm and a concrete example makes the method harder to follow and implement.\n- The proposed method does not significantly outperform the baseline method in the transfer tuning setting. The results in Table 1 show that the performance of the proposed method is similar to or even worse than that of LESS.\n\nQUESTIONS:\n- I suggest adding an illustration of the end-to-end algorithm with examples.\n- For the results in Table 4, could you also provide the selection and training costs? If the proposed method consumes significantly more resources in the selection stage than the baselines, it may be unfair to compare under the same selection ratio.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes Quadratic Coreset Selection (QCS) , a novel framework for efficient LLM instruction tuning that simultaneously optimizes sequence-level and token-level data selection. By reconciling both selection granularities through a bi-level optimization objective, QCS achieves superior data efficiency while mitigating token noise and bias. To overcome computational barriers, the authors develop a probabilistic re-parameterized solver using hierarchical policy gradients that avoids prohibitive Hessian inversions. The authors conduct extensive experiments to demonstrate the effectiveness of QCS.

STRENGTHS AND WEAKNESSES:
Strengths:  
1. The proposed method is quite innovative. QCS innovatively reconciles sequence-level and token-level data selection within a unified bi-level framework.
2. The probabilistic re-parameterization and hierarchical policy gradients elegantly bypass prohibitive Hessian inversions, which is a clever approach.

Weaknesses:  
See "Questions" below.

QUESTIONS:
1. It seems like the selected strategy requires iterative updates. If the number of iterations is large, I’m concerned whether the computation time will significantly exceed the time for training with Full data using LoRA. Is there any direct data on this?
2. QCS’s performance is sensitive to the token selection ratio. Is there any plan to design an adaptive strategy?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Quadratic Coreset Selection (QCS), a novel approach designed to improve the efficiency of instruction-tuning (IT) for large language models (LLMs). QCS uniquely combines sequence-level and token-level data selection, addressing the limitations of methods that focus solely on one or the other. The paper proposes a probabilistic solver for QCS that bypasses computationally intensive traditional methods, which ensures provable convergence and asymptotic equivalence to the original objective. Experimental results demonstrate QCS's superior data efficiency in various IT scenarios, including targeted and continual IT, which leads to more expressive and better-performing LLMs with reduced data requirements.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper introduces a principled approach that jointly optimizes sequence- and token-level data selection, which addresses limitations of treating them separately.\n\n- By reparameterizing coreset weights as Bernoulli distributions and applying hierarchical policy gradients, the method avoids costly gradient computations and scales to large models.\n\n- QCS achieves strong results using only a small amount of training data, often matching or surpassing full-data baselines across multiple instruction-following benchmarks.\n\n- The method is evaluated in regular, targeted, and continual instruction tuning settings, which shows consistent benefits and adaptability.\n\nWeaknesses\n\n- While QCS combines sequence- and token-level selection, it builds heavily on existing coreset and influence-function frameworks, which offers incremental rather than fundamental innovation.\n\n- The paper does not thoroughly investigate or explain scenarios where QCS underperforms (in Table 1, QCS underperforms compared to Full and LESS on MMLU and BBH across both LLAMA-2-7B and 13B).\n\n- Section 7.2 shows that QCS performance is highly sensitive to the elastic parameter ϵ and token selection ratio α; for instance, α = 0.2 yields substantially worse results than α = 0.5 or 1, which indicates fragility.\n\n- Most experiments focus on instruction-following tasks; the generality of QCS to other post-training settings remains untested.\n\n- While the paper claims that QCS is more efficient than LESS, the memory and time comparisons in Table 2 are minimal and lack full context (e.g., training time for LoRA or convergence behavior), which makes it hard to assess practical efficiency gains.\n\nQUESTIONS:\nPlease see the concerns noted in the paper's weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Quadratic Coreset Selection (QCS), a novel framework for improving the data efficiency of instruction tuning for large language models. The central idea is to reconcile the two approaches: sequence-level data curation and token-level selection. The authors formalize both approaches from a bi-level coreset selection perspective and propose a unified QCS objective that jointly optimizes for selecting influential instruction-response sequences and critical tokens within them. The experimental results demonstrate the efficacy of our QCS algorithm from two perspectives: (1) its instruction-following ability with the sequence-level data efficiency, showcasing the generalization with limited data; (2) analyze the token-level benefits and inner-loop adjustments within QCS, revealing the cause of its success.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The paper addresses the crucial challenge of data efficiency in instruction tuning. The contribution is both novel and important by unifying sequence and token-level selection into a single principled framework. \n\n2. Experimental results showcase that the proposed method can achieve performance close to the full set while using only 5% data, which is also better than the compared baselines.\n\n\nWeakness:\n\n1. The paper is conducted on Llama 2 base models, which I think is quite outdated. I strongly encourage the authors to perform experiment on newly-realeased foundation checkpoints like Llama 3 or Llama 3.1, or Qwen 3 family models, since the conclusions on such checkpoints may make more sense and be more insightful. I am looking forward to new results on newer model checkpoints in the rebuttal.\n\n2. The result section in the work is somehow weak. I think case studies are always desired for this topic since readers want to understand from specific cases how this method works. I also look forward to more example cases in the rebuttal.\n\nQUESTIONS:\nAs pointed out in my review, I want to know why old checkpoints like Llama 2 are used for experiments but not the newer models.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}