Does ‘Deep Learning on a Data Diet’ reproduce? Overall yes, but GraNd at Initialization does not
Authors: Andreas Kirsch
Venue: Accepted by TMLR ()

Abstract: Training deep neural networks on vast datasets often results in substantial computational demands, underscoring the need for efficient data pruning. In this context, we critically re-evaluate the data pruning metrics introduced in `Deep Learning on a Data Diet' by Paul et al. (2021): the Gradient Norm (GraNd) (at initialization) and the Error L2 Norm (EL2N). Our analysis uncovers a strong correlation between the GraNd scores at initialization and a sample's input norm, suggesting the latter as a potential baseline for data pruning. However, comprehensive tests on CIFAR-10 show neither metric outperforming random pruning, contradicting one of the findings in Paul et al. (2021). We pinpoint the inconsistency in the GraNd at initialization results to a later-fixed bug in FLAX's checkpoint restoring mechanism (https://github.com/google/flax/commit/28fbd95500f4bf2f9924d2560062fa50e919b1a5). Altogether, our findings do not support using the input norm or GraNd scores at initialization for effective data pruning. Nevertheless, EL2N and GraNd scores at later training epochs do provide useful pruning signals, aligning with the expected performance.

Published in Transactions on Machine Learning Research (09/2023)
Does 'Deep Learning on a Data Diet' reproduce? Overall yes, but GraNd at Initialization does not
Andreas Kirsch
wndross.kirschhecssoroo
OATML, Department of Computer Science University of Orford
Reviewed on OpenReview: https: openreview. net/ forun? id= 1 duxaa
Abstract
Training deep neural networks on vast datasets often results in substantial computational demands, underscoring the need for efficient data pruning. In this context, we critically re-evaluate the data pruning metrics introduced in 'Deep Learning Paul et al. (2021): the Gradient Norm (GraNd) (at initialization) and the Error L2 Norm (EL2N). Our analysis uncovers a strong correlation between the GraNd scores at initialization and a sample's input norm, suggesting the latter as a potential bascline for data pruning. However, comprehensive lests on CIFAR-10 show neither metric oulperforming random pruning, contradicting one of the findings in Paul et al. (2021). We pinpoint the inconsistency in the GraNd at initialization results to later-fixed bug in FLAX's checkpoint restoring mechanism' Altogether, our findings do not support using the input norm GraNd scores at inittaiizoion for effective data pruning. Nevertheless, FL,2 and GraNd scores at later training epochs do provide useful pruming signals, aligning with the expected performance.
Data Dict' by
1 Introduction
Deep neural networks have achieved state-of-the-ar performance on tasks like image classification and text generation, but require massive datasets that demand extensive computational resources for training (Deng et al., 2009; Brown al., 2020). This motivates developing techniques like data pruning to identify effective subsets of the training data (Bachem al., 2017; Sorscher el al., 2022).
Data Diet' (Paul et al., 2021) introduced two new pruning metrics, Gradient Norm (GraNd) at initialization and Error L2 Norm (EL2N). During talk that was the given on the paper, the surprising effectiveness of GraNd at inilialization before any training led us to hypothesize on a potential correlation with a sample's s input norm. The input norm cheap to compute and could thus provide an intriguing new bascline for data pruming experiments.
Recently, 'Deep Learning
In this work, we set out reproduce the results for GraNd scores initialization and explore the icacticality of using the input norm proxy. Through extensive analysis CIFAR-10, we find a strong correlation between the input norm and the gradient norm at initialization. However, neither metric outperform random pruning, failing to reproduce the original findings. While we replicate the reported results for EL2N and GraNd scores at later epochs, the performance of GraNd scores at initialization is not reproducible despite testing with multiple codebases. We trace this back to a bug the checkpoint restoring code of the underlying FLAX library. By rectifying the claims about GraNd scores at initialization and uncovering limitations of the input norm, our reproduction provides updated recommendations for efficient data pruning.
In summary, this reproduction contributes new insight the relationship between the input norm and the gradient norm at initialization and fully explains a failure to reproduce the performance of using the gradient. norm at initialization for pruning, one of the six contributions of Paul et al. (2021).
Details httpa:///gitabb..oooooggggeeeeee http:///ee/.laaaccooo//////aaaaa

---

Published in Transactions on Machine Learning Research (09/2023)
Outline. In $2, provide background on the GraNd and EL2N scores and discuss their use for data pruning. In $3.1, we begin by discussing the correlation between input norm and gradient norm at initialization. We empirically find strong correlation between GraNd scores at initialization and imput norms as we average over models. In $3.2, we explore the implication of this insight for dataset pruning and find that both GraNd at initialization and input norm scores do not outperform random pruning, but GraNd scores after a few epochs perform similar to EL.2N scores these later epochs.
2 Background
DDeep Learning on a Data Diet'. Paul et al. (2021) introduce two novel metrics: Error L2 Norm (EL2N) and Gradient Norm (GraNd). These metrics aim to provide a more effective means of dataset pruning. It important to emphasize that the GraNd score at initialization calculated before any training has taken place, averaging over several randomly initialized models. This fact has been met with skepticism by reviewers", but Paul et al. (2021) specifically remark on GraNd at initialization:
Pruning at initialization. In all settings, GraNd scores can be used to select a training subset at. initialization that achieves test accuracy significantly better than random, and in some cases, competitive with training on all the data. This is remarkable because GraNd only contains information about the gradient norm initializion, averaged over initializations. This suggests that the geometry of the training distribution induced by random network contains surprising amount of information about the structure of the classification problem.
GraNd. The GraNd score measures the magnitude of the gradient vector for a specific input sample in the context of neural network training over different parameter draws. The formula for calculating the (expected) gradient norm is:
GraNd(x) = Eo, L(((
(1)
where Lo, 0)),) y) is the gradient of the loss function L with respect to the model's parameters 0g epoch 4, (x; is the model's prediction for input H, and y is the true label for the input. We take an expectation over several training runs. The gradient norm provides information about the model's sensitivity to particular input and helps in identifying data points that have strong influence on the learning process.
EL2N. The EL2N score measures the squared difference between the predicted and (one-hot) true labels for a specific input sample, also known as the Brier score (Brier, 1950):
EL2N(x) Ee,11I((;; v1
(2)
where f(x;0) 0) is the model's prediction for input x, y is the (one-hot) true label for the input, and ||-1|2 denotes the Euclidean (L2) norm. The EL2N score provides insight into the model's performance on individual data points, allowing for a more targeted analysis of errors and potential improvements.
The GraNd and EL2N scores are proposed in the context. of dataset pruning, where the goal is to remove less informative samples from the training data. Thus, can create a smaller, more efficient dataset that maintains the model's overall performance while reducing training time and computational resources.
While GraNd at initialization does not require model training, it requires a model and is not cheap to compute. In contrast, the input norm of training samples incredibly cheap to compute and would thus provide an exciting new bascline to use for data pruning experiments. We investigate this correlation next and find positive evidence for However, we also find that the GraNd score at initialization does not outperform random pruning, unlike the respective results in Paul et al. (2021).
2

---

Published in Transactions on Machine Learning Research (09/2023)
Rank Conr. 0.96
Ronk Corr: 0.39
00-
04
Graldd
0.3
40
0.1
20
00
00
150
(a) Original Repo (10 trials)
(b) Hilb (10 trials)
Rank Cor:: 0.97
140
1750
120
100
1000
Input
750
500
20
125 GraNd at Init
1.1
12
ExptGredient h Input nomi)
(c) Minimal (120 trials)
(d) Minimal (120 trials)
Figure 1: Correlation between GraNd at Initialization and Input Norm for CIFAR-10's training set. (a, b, c): The original repository and the 'minimal' implementation have very high rank correlation- 'hlb'' has lower but still strong rank correlation. (d): Ratio between inpul norm and gradient norm. In the 'minimal' implementation, the ratio between input norm and gradient norm is roughly log- normal distributed.
0.93
0.92
Metfic at Epoch
Mibricc
0.89
at
Random Nomm
Random
0.91
0.88
Input Korm
Gmaid
GraNd at
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0,2
0.3
0.4
0.5
0.6
Pruned Fraction
Pruned Fraction
(a) Original Repo (2 trials each)
(b) Hlb (10 trials)
Figure Reproduction of Figure (second row) from Paul et al. (2021). In both reproductions, GraNd initialization performs as well the input norm. However, it does not perform better than random pruning. Importantly, it also fails to reproduce the results from Paul et al. (2021). However, GraNd epoch 20 (respectively at epoch 1 for 'hlb') performs similar to EL2N and like GraNd at initialization in Paul et al. (2021).
3

---

Published in Transactions on Machine Learning Research (09/2023)
3 Investigation
We examine the correlation between the input norm and GraNd at initialization, as well as other scores on CIFAR-10 (Krizhevsky al., 2009), through three distinct approaches:
First, we update the original paper repository" (attps://github. con/maansheej/date diet), which utilizes JAX (Bradbury al., 2018). We rerun the experiments for Figure (second row) in Paul et al. (2021) on CIFAR-10, training for 200 epochs using GraNd at initialization, GraNd at epoch 20, E2LN at epoch 20, Forget Score at epoch 200, and input norm.
Second, we reproduce the same experiments using 'hlb' (Balsam, 2023), a significantly modified version of ResNet-18 that achieves high accuracy in 12 epochs, taking approximately 30 seconds total on an Nvidia RTX 4090 with PyTTrch (Paszke et al., 2019). For this approach, we compare GraNd at initialization, GraNd epoch (= 20/200- 12 epochs), EL2N at epoch and input norm". We analyze the rank correlations between the different scores for the two repositories mentioned above.
3. Third, we employ another 'minimal' CIFAR-10 implementation (van Amersfoort, 2021) with standard ResNet18 architecture for CIFAR-10 to compare the rank correlations.
3.1 Correlation between GraNd at Initialization and Input Norm
To gain deeper understanding of the relationship between the input norm and the gradient norm at initialization, we first consider toy example and then provide empirical evidence. Let us examine lincar softmax classifier with C classes (without a bias term). The model takes the form:
f(x) = softmax(
(3)
together with the cross- entropy loss function:
L log f fz)
(4)
The gradient of the loss function concerning the rows wy of the weight matrix W is:
Vw, L = (w(==(((z) - 1(j = y})2,
(5)
where LÍj = y} the indicator function that if j = y and otherwise. The squared norm of the gradient is:
V v Σ ((()), 16 -1
(6)
Taking expectation over W (different initializations), the norm of the gradient is:
1/27
C [ 10 1 y)}²
Ew vo Ew
||
(7)
As a result, the gradient norm is a multiple of the input norm. The factor depends f(x), which we could typically expect to be 1/C in expectation over different weights at initialization (due to class symmetry).
2Sc also http:p///pppaaaaeee.aaeee/aaaaaa
Sgee https://github. www.blaccccccaaaaaaaaaaaaaaaaaaa diet hhttps://github http:/hhww...oolccccccc//ootorrr
4

---

Published in Transactions on Machine Learning Research (09/2023)
1.0
Input Nor
0.39
0.039
0.067
4 C24
-3 02002
4.033
0.5
Gralis at Irit
+0 40.060
+0.1
2.4
-M4
Graldd Epoch
0.94
ELa lppeet 201
2.80
EL2N Epoch
Irast horm
bud tosooh Sbaech
Input
Grate at Init Graño at Epoch EL2N Epoch
(a) Original Repo (2 trials each)
(b) Hib (10 trials)
Figure 3: Rank Correlations of the Scores. Cf. Figure 12 in the appendix of Paul et al. (2021). In both reproductions, GraNd at initialization and imput norm are positively correlated, while GraNd and EL2N at later epochs are strongly correlated with each other and the Forget Score (at epoch 200).
3.2 Empirical Results
Correlation Analysis. We first examine the correlation between the input norm and GraNd score at initialization. Across three separate codebases and implementations (original JAX, 'hlb' PyTorch, 'minimal' PyTorch), we consistently find strong Spearman rank correlation between the two metrics when averaged over multiple random initializations (Figure 1). For example, the original JAX and 'minimal' PyTorch implementation have rank correlation coefficient of 0.96 between input norm and GraNd the CIFAR-10 training set over 10 trials, suggesting input norm could potentially serve inexpensive proxy for GraNd at initialization. 'hlb' uses different input preprocessing and other tricks, which might explain the lower but still strong rank correlation coefficient of 0.39.
Reproducing Figure of Paul al. (2021) on CIFAR-10. However, our altempls to reproduce the data pruning results originally reported for GraNd at initialization are unsuccessful. As shown Figure 2, neither input norm nor GraNd at initialization outperform random pruning, with both achieving approximately 2 percentage points less accuracy than random pruning when pruning 60% of the CIFAR-10 training data. This directly contradicts the original findings of Paul et al. (2021), where GraNd at initialization markedly exceod random pruning when pruning 60% of the CIFAR-10 training data. While we are able to reproduce the expected pruning performance for GraNd after 20 epochs of training (respectively at epoch 1 for "hlb') and also match the original EL2N results, GraNd at initialization does not match expectations despite testing across codebases.
Score Rank Correlations. To further analyze relationships between the different scores, we visualize their Spearman rank correlations CIFAR-10 using correlation matrix (Figure 3). The results confirm the strong correlation between input norm and GraNd at initialization. They also reveal high correlations between GraNd at later epochs, EL2N, and the Forget Score after full training. However, GraNd at initialization and input norm show little correlation with the later epoch scores. This aligns with our reproduction results, where only the late-epoch GraNd and EL2N scores exhibit the expected pruning performance. The distinct correlation profile provides additional evidence that GraNd at initialization is measuring something fundamentally different. from the trained model pruning metrics.
4 Discussion
While input norm inexpensive to compute, given its model independence and lower computational cost compared GraNd or other scores, our results imply it should be used for data pruning. Similarly, since only GraNd scors later epochs appears to perform expected, we cannot recommend using GraNd scores at initialization for data pruning either.
5

---

Published in Transactions on Machine Learning Research (09/2023)
Regarding the failure to reproduce the GraNd at initialization results, initially could not rerun the code using the original JAX version due to GPU incompatibility. However, the authors of Paul et al. (2021) managed to reproduce their original results by setting up an old VM image with the original dependencies. This led us to discover a later-fixed bug in the FLAX checkpoint restore function flax. baaaaiig..esttoe chwoorreeccckoiitt' as the underlying cause of the reported GraNd at initialization results: the code was restoring different. checkpoints than expected. Our experience reinforces the importance of preserving code environments and dependencies to enable reproducibility.
Our findings highlight the critical need for reproductions, especially when initially surprsiing results are reported. As state-of-the-art techniques become more complex, it is essential that different rescarch groups thoroughly verily results across frameworks and implementations before informing best practices.
5
Conclusion
This work makes multiple contributions. We uncovered a strong correlation between the inexpensive input norm and the proposed GraNd metric at initialization. However, through extensive analysis we found neither imput norm nor GraNd scores at initialization reproduced the originally reported pruning performance. Only GraNd and EL2N scores at later training epochs provided useful pruning signals, aligning with expected behavior. We traced the non-reproducible GraNd at initialization results to a now-fixed bug in the checkpoint restoring code of the underlying FLAX framework. Our findings rectify claims around GraNd scores initialization, provide updated iecommendations on utilizing these data pruning metrics, and underscore the importance of rigorously confirming initial surprising results across implementations. As deep learning advances, maintaining high standards for reproducibility will ensure progress builds solid foundations.
Acknowledgments
Many thanks to Mansheej Paul and Karolina Dziugaite for very helpful feedback, discussions, and in particular, for their assistance investigating the source of the discrepancy in reproducing the GraNd at initialization results. Upon being informed of our findings, they worked to trace the issue and shared their methodology for reproducing the original results using the original framework versions. This helped uncover the checkpoint restoring bug the underlying cause. We appreciate that they took the time to rerun experiments with the original codebase and dependencies, and for quickly preparing an updated version of their paper.
AK is supported by the UK EPSRC CDT in Autonomous Intelligent. Machines and Systems (grant reference EP/L015897/I). ChatGPT (GPT-4) and Claude were used to suggest edits and improvements.
References
Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXic: 06476, 2017.
Tysam Balsam. CIFARIO,, 2023. URL
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://githul com/ com/google/jax.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1 3, 1950.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877-1901, 2020.
passing a step (i.c., initial- Ixation) would trigger loading the latest checkpoint instead the zero-th checkpoint because the internal implementation was checking step: instead of ie step is not when deciding whether to fallback loading the latest checkpoint.
58ce https://github /ttpg///e/aaaoooooeaaaaaaa/a///a

---

Published in Transactions on Machine Learning Research (09/2023)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 TEEE conference compuler vision and pollern recognation, pp. 248 255..55. leee, 2009.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Autiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperalive style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL hire--hrfroprpaaccecccoooo weweformaneedeep-leerree Melearnnng-iibrryy pdf.
9015-pytorch-an anterchhan-iiie www.:porrrraaaaaaaaaaaeeeeeeeeee
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advonces in Neural Information Processing Systems, 34:20596-200 20607, 2021.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523 19536,, 2022.
Joost van Amersfoort. Minimal CIFAR-10, 5 2021. URL tree//main/minimaa,_cc cifar.

---

Published in Transactions on Machine Learning Research (09/2023)
A Appendix
200
GraNd at init
350
175
300
150
125
100
150
100
50
10000
Sorted by Aug Normled Score
ExplGradient norm Input noml
(a) Minimal (1000 trials)
(b) Minimal (1000 trials)
Figure 4: Correlation between GraNd at Initialization and Input Norm the Test Set. (a): We sort the samples by their average normalized score (i.c., the score minus its minimum divided by its range), plot the scores and compute Spearman's 's rank correlation on CIFAR-10's test data. The original repository and the 'minimal' implementation have very high rank correlation- 'hlb' has a lower but still strong rank correlation. (b): Ratio between inpul norm and gradient norm. In the 'minimal' implementation, the ratio between input norm and gradient norm roughly log-nooe log-normal distributed


Reviews:
[{"review_id": "FFI94WGuab", "reviewer": "Reviewer_iV8N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nAll in all, the paper is clearly written and the the statement seems to be technically sounds. I have noticed that Paul et al. (2021) has also updated their arxiv version of the \"Deep Learning on a Data Diet,\" paper to account for the mistake pointed out. \n\nWeaknesses:\nAs a paper solely submitted to point out a bug from a previous paper, I don't think there is much major weaknesses. The bug pointed out seems to be real and the conclusion from this paper seems to be correct. Nevertheless, I am not sure if such a paper is worth publishing, especially after Paul et al. (2021) has fixed the issues in their version of the paper. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines one of the novel approaches introduced by Paul et al. (2021) in their paper titled \"Deep Learning on a Data Diet,\" known as the GradND. GradND is a method used to prune away unnecessary data in the training dataset for more effective training using the reduced dataset. In the present study, the authors shed light on a significant issue discovered within the experiments conducted by Paul et al. (2021), which has resulted in misleading experimental outcomes. Specifically, the authors of this paper show that the results of GradND obtained in the original paper by Paul et al. (2021) at initialization of the model is wrong and caused by a bug in their implementation. \n\n\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't think there are any changes needed. ", "broader_impact_concerns": "No concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xQvYkkKide", "reviewer": "Reviewer_UzQW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n* The paper contributes to the community by reproducing and valdiating established results. \n\n*Weaknesses*\n* I don't know if, in this case, a TMLR paper is the best format for sharing the results. The issue seems to be a software bug, not a conceptual error. It's important to know that GraNd doesn't work at iteration 0, but it works at other iterations so unless Pauls et al _predicted_ that GraNd must have worked at iteration 0 (as I understand, that's not the case) correcting the original paper feels enough to me. I wonder if authors can suggest what could be the other consequences of their experimental findings. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper reports on the failure to fully reproduce results of Paul et al, 2021. In particular, gradient at initialization (GranD) scoring did not perform better than random pruning of training examples. Authors used the original source code as well as another optimized ResNet-18 implementation. It seems that the cause for the non-reproducibility is a (now fixed) bug in the Flax library. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In my opinion, the second paragraph of the paper is not neutral to the principle of double-blind review and contains details that are irrelevant for the majority of readers. I request these details to be removed. ", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2XHDefS37Z", "reviewer": "Reviewer_LnNb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe authors provided a clear motivation for why they started checking whether GraNd (norm of the gradients at the initialization step) is useful, which is due to its strong correlation to the input-norm metric and therefore may lead to a dataset pruning metric that is both cheap to compute and powerful.\nThe authors also used both the original repo and a re-implementation repo to confirm that the failure in reproducing the result is not due to an error in their implementation.\nThe strong correlation between the input-norm and the GraNd metrics also indicates that this GraNd metric is unlikely to work as reported in the DataDiet paper, as it would be difficult to imagine that the input norm would be important in deciding how good the example is for training.\nFinally, the authors also identified the possible cause of the original successful result in the Data-Diet paper. \nTaken together, these results help the community further explore which metric is important for data pruning.\n\nWeaknesses:\nThe first figure is a bit confusing. If the authors want to show the strong correlation between the two metrics (input-norm and GraNd), why not show a scatter plot with X-axis being one metric and Y-axis being another metric? \nThe writing can be improved. It’s currently informal and can be clearer. For example, the abstract is a bit influent in its logic, with the motivation not clearly explained before the result.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates whether one of the key metrics (GraNd) introduced by the Data-Diet paper is indeed useful in dataset pruning, as reported in that paper. Through thorough experiments on two architectures using both the original repo and a new reimplementation, the authors find that GraNd cannot outperform random pruning, while the other metric (EL2N) works as reported. The authors further reported that a bug fix in the supporting training framework of Data-Diet repo (JAX) is likely the reason for the inconsistency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness.", "broader_impact_concerns": "no clear negative society impacts from this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is written well. The experiments are convincing. The paper explains why it failed to replicate some of the experiments, and the results are interesting. Especially, I am quite surprised at how strong the random pruning performs. Also, the strong correlation between the GraND score and input norms makes sense. It is well explained too. The paper is short, but I think that is appropriate since the conclusions that can be drawn from it are straightforward and no need to be obfuscated.
Decision: Accept (poster)