Kati8/ICLR_2023_2025 huggingface dataset:
authors: Ziqing Fan
authors: Siyuan Du
authors: Shengchao Hu
authors: Pingjie Wang
authors: Li Shen
authors: Ya Zhang
authors: Dacheng Tao
authors: Yanfeng Wang
reviews: 
Comments: This paper proposes a novel file selection method for LLM pretraining to avoid the dimensional collapse issue. The method can achieve greater uniformity across feature dimensions, while the other methods usually get a long narrow feature space, limiting their performance on all the tasks.
Strengths: 1. This paper observes the drawbacks of the other file selection methods for LLM pretraining, which usually obtain quite narrow feature space, and are hard to obtain good results for all the tasks.

2. The proposed method is interesting, which converts the uniformity of the eigenvalues of the covariance matrix into minimizing the F norm of the covariance matrix, then uses a submodular optimization to solve it.

3. Experimental results on 7 tasks show the proposed method achieves a higher average perfomance than the other baselines, and is quite data efficient compared to full data pretraining.
Weaknesses: 1. The observation in figure 2 is evaluated on the SlimPajama. Do these methods have the same phenomenan on the other pretraining corpus.

2. In the efficiency analysis, comparing with full data pretraining is not enough, the comparison with the other baselines is not given.
Recommendation: 8
Reviewer confidence: 2
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The authors propose a diversified submodular file selection algorithm (DiSF) to select sample set with more diverse data. It can alleviate dimensional collapse problem in feature space and enable LLM model to have generic performance.
Strengths: The paper addresses an important problem in LLM pretraining and proposes a simple yet effective method. The writing is clear and the narrative is easy to follow, facilitating understanding of complex concepts.
Weaknesses: No major weakness.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 4
reviews: 
Comments: This paper introduces DiSF, a method aimed at selecting diverse training data to enhance the effectiveness of pre-training large language models (LLMs). The authors begin by analyzing recent data selection methods and identify a critical issue: the selected data often suffer from dimensional collapse in the feature space. This collapse leads to improved performance on domain-specific tasks but causes a significant degradation in general performance due to a lack of diversity.

To address this problem, the authors propose improving the diversity of the data selection process by formulating it as a submodular optimization problem. Specifically, their goal is to prevent dimensional collapse by ensuring more uniform eigenvalues in the covariance matrix of the selected samples. They employ a classical greedy algorithm for submodular optimization, combined with mini-batch sampling, to make the approach tractable.

Due to hardware limitations, the authors tested their approach on models with sizes of 120M, 560M, and 1.1B parameters. Despite the smaller scale, the DiSF method achieves superior performance over six baselines on nine standard benchmarks consistently, across different training budgets (10B and 50B tokens). The authors also provide extensive ablation studies to analyze the effects of data selection budget, different architectural backbones, feature extractors, and more.
Strengths: + Well-Motivated Research: The paper addresses a significant issue in data selection for LLMs—dimensional collapse due to lack of diversity—which has substantial implications for model performance.
+ Clear Presentation: The methodology is clearly explained, making the paper easy to read and understand.
+ Strong Experimental Results: The results are convincing, demonstrating that DiSF outperforms existing methods consistently across multiple benchmarks and model sizes.
+ Thoughtful Comparisons: Comparisons with six baseline methods are thorough, providing a solid context for the effectiveness of DiSF.
+ Comprehensive Ablation Studies: The extensive ablation studies offer valuable insights into the impact of various factors such as data selection budget, model architecture, and feature extractors.
Weaknesses: + Limited Scale of Experiments: The experiments are limited to models up to 1.1B parameters. While the results are promising, it would be beneficial to see how the method scales to larger models commonly used in current LLM research.
+ Scalability Concerns: There is uncertainty about whether the proposed method can maintain its efficiency and effectiveness when applied to larger datasets.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3