Title: Computational Budget Should Be Considered in Data Selection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data Selection

Efficient Learning

Computation Budget

Resource-Constrained Learning
Abstract: Data selection improves computational efficiency by choosing informative subsets of training samples. 
However, existing methods ignore the compute budget, treating data selection and importance evaluation independently of compute budget constraints. Yet empirical studies show no algorithm can consistently outperform others (or even random selection) across varying budgets. We therefore argue that compute budget must be integral to data-selection strategies, since different budgets impose distinct requirements on data quantity, quality, and distribution for effective training. 
To this end, we propose a novel Computational budget-Aware Data Selection (CADS) method and naturally formulate it into a bilevel optimization framework, where the inner loop trains the model within the constraints of the computational budget on some selected subset of training data, while the outer loop optimizes data selection based on model evaluation. 
Our technical contributions lie
in addressing two main challenges in solving this bilevel optimization problem: the expensive Hessian matrix estimation for outer-loop gradients and the computational burden of achieving inner-loop optimality during iterations. 
To solve the first issue, we propose a probabilistic reparameterization strategy and compute the gradient using a Hessian-free policy gradient estimator. To address the second challenge, we transform the inner optimization problem into a penalty term in the outer objective, further discovering that we only need to estimate the minimum of a one-dimensional loss to calculate the gradient, significantly improving efficiency. To accommodate different data selection granularities, we present two complementary CADS variants: an example-level version (CADS-E) offering fine-grained control and a source-level version (CADS-S) aggregating samples into source groups for scalable, efficient selection without sacrificing effectiveness.
Extensive experiments show that our method achieves performance gains of up to 14.42\% over baselines in vision and language benchmarks. 
Additionally, CADS achieves a 3-20× speedup compared to conventional bilevel implementations, with acceleration correlating positively with compute budget size.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework that incorporates **computational budget constraints** into data selection, presenting the Computational Budget-Aware Data Selection (CADS) method. Formulated this as a bilevel optimization problem, CADS consists of an inner loop that trains a model on a selected data subset within a defined computational budget and an outer loop that optimizes data selection based on the model's performance. \n\nThe authors address two key challenges in solving this bilevel problem: **the high computational cost of Hessian estimation and the complexity of achieving inner-loop optimality**. To overcome these, CADS employs a probabilistic reparameterization strategy with a Hessian-free policy gradient estimator and reformulates the inner optimization as a penalty term in the outer objective, requiring only a one-dimensional loss estimation for enhanced efficiency. This loss estimation is performed using a log-linear function interpolated from losses across different subset sizes.\n\nCADS is implemented in two variants: CADS-E, which enables fine-grained, example-level selection, and CADS-S, which offers group-level sampling ratio estimation. Experimental results across vision (MNIST, CIFAR10, DomainNet) and language tasks (instruction-tuning dataset) demonstrate that CADS achieves up to 14.42% performance improvements and 3–20x training speedups compared to baseline methods, showcasing its effectiveness and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n* Quality: The first half of the paper (up to Section 5) is well-structured, effectively motivating the reader with a clear rationale for the proposed approach. However, the experimental section is less clear, making it difficult to follow the setup and key takeaways from each experiment. I notice that Section 5.2, which discusses the CIFAR-10 experiment, missing results referencing the relevant table, only the experimental setup.\n* Clarity: The mathematical formulation is clear, making the challenges easy to understand.\n\n[Weaknesses]\n* About runtime concerns, while incorporating computational budget constraints is a strength, the runtime of the selection algorithm may scale poorly with the number of subsets (K), potentially offsetting the computational efficiency gains of the method. For example, in algorithm 1, we have to sample K subsets and train K models and evaluate them to access K losses so we can interpolate.\n* Figure 7, for example in the appendix illustrates the loss estimation process. It remains unclear about sampling requirements, especially how many subsets (K) are needed to generate sufficient sample losses for reliable interpolation and how robust this interpolation.\n* It's unclear of the selection of the baseline method (PBCS) or provide details on how it operates. Could the authors clarify it?\n\nQUESTIONS:\n* The definitions of low-frequency and high-frequency data are unclear, particularly how these characteristics are identified or quantified in the context of the dataset (image or instruction-tuning dataset). \n* In Section 3, how do we construct validation dataset? how it is sourced from Group A and Group B, as referenced in Appendix B.1?\n* What is the dataset and model being used to have Figure 7?\n\nLIMITATIONS:\nYes. It's documented in their appendix (see section A and G).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to incorporate a computational budget into the data selection process by proposing a bilevel formulation. The formulation uses a computational constraint to train a model on the subset of the dataset in the lower level, with the upper level optimizing the selection of the subset. Apart from providing insights into how the computational budget affects the selection of data, the paper also presents technical contributions for solving the bilevel problem better than previous methods. Empirical evaluation on datasets like MNISt, DomainNet and instruction tuning tasks shows the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper addresses a crucial problem of compute-constrained data selection.\n2. While the bilevel formulation of the problem is well known the use of the probabilistic reparametrization strategy along with the insight of avoiding the training of the lower level problem to convergence can be useful for solving other bilevel problems.\n3. The paper clearly discusses the proposed enhancements to solving the bilevel problems along with the computational complexity analysis of the problem.\n\nWeaknesses\n1. The empirical evaluation section of the paper is rather weak with evaluation primarily shown on datasets like Mnist and Domainnet. This makes the paper weak since data selection is desirable on larger and more complicated datasets. It is unclear whether the results presented in the paper inform us about the success of the method on larger datasets.\n2. The computational complexity of the method is compared against other bilevel based methods for dataset selection. Adding comparison to other dataset selection methods such as Forgetting scores could be useful. \n3. Performance comparison is very limited. Other bilevel approaches as well as non-bilevel approaches should be included in the paper\n4. While the related work mentions the paper [67] a direct comparison with their approach is essential.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper demonstrates that computational budget constraints significantly impact data selection. The authors propose Computational budget-Aware Data Selection (CADS), formulating it as a bilevel optimization framework. To address two key challenges, they introduce a probabilistic reparameterization strategy and transform the inner optimization into a penalty term within the outer objective. Experimental results and analyses appropriately support the claims.

STRENGTHS AND WEAKNESSES:
## Strengths

1. This paper reveals an interesting and novel perspective on data selection.

2. This paper elegantly formalizes the problem as a bilevel optimization problem.

3. The proposed CADS is well-motivated, and its effectiveness is thoroughly demonstrated.

## Weaknesses

1. In Sec. 3, how to generate low-/high-frequency datasets? How to define low-/high-budget phases and the difference between them?

2. It seems that there is no direct relevance between the frequency favoritism of deep neural networks and the proposed method. Authors need to further explain the relationship and provide more evidence to support this motivation.

3. One major concern is the lack of comparison between the proposed CADS and established baselines regarding performance and computational cost on widely-used datasets (e.g., ImageNet) and general-purpose models.

4. It seems that CADS is proposed in a general manner. Authors need to discuss the scalability of CADS to other bilevel optimization problems.

QUESTIONS:
See above weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a two-step algorithm to incorporate the computational budget of a training model in  their data selection procedure. It advocates to keep an eye on the compute budget during data selection and not treat it as a fixed hyper parameter. I like the hypothesis of the paper, but I have some concerns over the overall exposition and validation of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper raises a very important aspect of the training process. Computation budget should be part of the training process and the data selection\n- Some initial results are shown on the MNIST dataset, and some other datasets to validate the hypothesis.\n\n\nWeakness:\n- The text did not make it clear what does compute budget refers to\n- The two-step algorithm in the introduction misses the context, like why Hessians are needed in the first step, what optimality refers to in the data selection.\n- The paper seems to be motivated based on the observation of spectral bias from Rahman et al. This is insightful but does that generalizes to different problems in machine learning?\n- Solid validation is missing, for example, how the generalization of a network gets affected with the compute budget (in terms of FID scores). \n\nOverall I think the idea is not sufficiently validated.\n\nQUESTIONS:\n- Why the first step needs Hessians computation?\n- How is optimality defined in the second step?\n- How can one know how much time a model would need to train? \n- It is not clear to what the compute budget refers to? Is it the compute budget for the model or the compute budget for the data mining process?\n- What computational overhead the algorithm adds on top of the computational cost of training a network?\n- Can you provide more validations on generative models showing different metrics, how they get affected by the compute budget?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}