reviews: 
Comments: The paper introduces the DiSF (Diversified Submodular File Selection) algorithm to combat dimensional collapse in Large-Language Model (LLM) pretraining. The DiSF algorithm is designed as a subset selection task with a goal of selecting a diverse subset from an unlabeled training corpus and is achieved by maximizing a submodular information function over the complete unlabeled set with sample interactions modeled through a normalized covariance matrix. The paper demonstrates upto 1.5x training efficiency and upto 5x data efficiency on several benchmark datasets.
Strengths: The strengths of this paper can be summarized as below - 

1. The paper address the task of data-efficient LLM pretraining while boosting performance on several benchmark tasks by minimizing the impact of dimensionality collapse which is a critical challenge in this domain. 

2. The authors clearly demonstrate the impact of dimensionality collapse in existing approaches targeting data-efficient LLM training and establish the requirement for decorrelated text samples during the training process to minimize its effect.

3. The DiSF algorithm models this requirement as a diversity driven selection problem and achieves this my training the underlying LLM on a subset of data selected by maximizing a submodular function $C(U, M)$ over the complete unlabeled set $\mathbb{D}$. This proposed methodology is strongly supported by experimental evidence on multiple benchmark datasets.
Weaknesses: Although the results demonstrated in the paper are impressive the following weaknesses/ questions need to be addressed.

1. Although the subset selection mechanism in DiSF depends on submodular maximization of $F_{M}^{DiSF}(U)$ but the proof of submodularity of $F_{M}^{DiSF}(U)$ has not been provided. The paper presents only an empirical proof of monotonicity through Figure 4 but that is not deemed sufficient (Fujishige, 2005) to prove a function to be submodular. The authors must provide proof of $F_{M}^{DiSF}(U)$ demonstrating the diminishing marginal returns property described in lines 232- 234.

2. Diversity driven subset selection for data-efficient LLM pretraining is not novel to the current paper. Prior work like INGENIOUS (Renduchintala et al., 2023) have demonstrated significant improvements in model performance while being data-efficient. The authors should compare against such methods to study the impact of dimensionality collapse and the diversity selection algorithm.

3. Submodular Functions like Log-Determinant, Facility-Location etc. have been shown to model diversity in several tasks like summarization (Kumari et al., 2024, Kothawade et al., 2022) and subset selection (Jain et al., 2023, Lin and Bilmes, 2011) to name a few. What is the justification of introducing only $F_{M}^{DiSF}(U)$ (Equation 7 in the paper) rather than any known submodular functions ? The The authors should compare the performance of DiSF when the underlying function is chosen to be some/all of the popular submodular functions proven to be effective for diversity selection.

4. Although the experimental results in Table 2 show a strong average performance across datasets, when the budget increases from 10B to 50B (5x increase in number of available training examples) the performance gain (overall) is quite small. Did the authors perform any additional experiments or theoretical analysis to justify this result ?

Some minor questions/ suggestions -
1. Section 2.3 can be moved higher up in the paper or a brief explanation on the cause and effects of dimensionality collapse should be highlighted in Section 1 for better readability.

2. Phrases like "Diversified Submodular File Selection" (line 19, 88, 183-184 ...), "decorrelated text files" (line 20, 89, 184 ...) has been repeated several times in the paper which should be reduced to improve readability.

3. Do the authors plan on releasing the code for reproducing the results from the paper in the near future ?

4.  In Equation 8 the $\text{argmax}$ is applied over $\mathbb{D} \setminus U$ which seems to be redundant. To the best of my knowledge, during the course of the greedy optimization process in Algorithm 1 of the paper, DiSF selects the sample with the highest information gain which inherently excludes duplicate samples which are already selected in U.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3