Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper provides a method to infer various meta-data features of datasets such as typical/atypical data points, noisy labels, noisy and out-of-distribution data points. 
strength_and_weaknesses: *Strengths*:
1. The paper proposes, to the best of my knowledge, the first method which jointly infers multiple meta-data features of a dataset. 
2. Simplicity: The method is simple and intuitive. 
3. Paper is very well-written. 

*Weaknesses*:
1. While I do appreciate the simplicity of the methods, it is hard to believe that the method performs on par with state-of-the-art label correction methods. I believe this can only be the case when the label corruptions occur uniformly at random, which might very well not be the case, as shown by many recent paper on noisy label correction. Moreover, the papers that the authors compare with are not the state-of-the-art since methods like Confident learning proposed later outperform the methods the authors compare with. 
2. The method does not seem to detect out-of-distribution samples very well (Fig. 4 (d)). I would like the authors to explain the underwhelming performance. 
3. It would be nice and instructive to discuss failure of the method in detecting these meta-data features. Like the authors mention in the limitations, the assumption that the learning dynamics.
4. The authors mention experiments on two different kinds on ResNet models, but (1) I do not see results from both the models, (2) experimenting with different model architectures might make for a stronger experimental result, showing the method's applicability beyond ResNet architectures. 

*Questions*
1. How are out-of-distribution probes generated? 
clarity,_quality,_novelty_and_reproducibility: The paper is clear and very well written. The paper solves a novel problem. It seems to be reproducible if the code is released upon acceptance of the paper.
summary_of_the_review: I like the paper, I think it is novel, simple, intuitive, well-written with strong experimental results, but can benefit from some more experiments and clarifications.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Metadata Archaeology, a unifying and general framework for uncovering latent metadata categories.
This paper introduces and validate the approach of Metadata Archaeology via Probe Dynamics (MAP-D): leveraging the training dynamics of curated data subsets called probe suites to infer other examples’ metadata.
This paper shows how MAP-D could be leveraged to audit large-scale datasets or debug model training, with negligible added cost. This is in contrast to prior work which presents a siloed treatment of different data properties.
The authors use MAP-D to identify and correct mislabeled examples in a dataset. Despite its simplicity, MAP-D is on-par with far more sophisticated methods, while enabling natural extension to an arbitrary number of modes.
The authors show how to use MAP-D to identify minority group samples, or surface examples for data-efficient prioritized training.
strength_and_weaknesses: Pros: 
1. The motivation is clear. 
2. The paper is well-written and organized. 
Cons: 
1. The main contributions are not clear. 
2. Some related works are missing, e.g., 3D Face Reconstruction from A Single Image Assisted by 2D Face Images in the Wild.
clarity,_quality,_novelty_and_reproducibility: See Strength And Weaknesses
summary_of_the_review: See Strength And Weaknesses
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Note: Score updated from 6 to 8 after authors' response.**

The authors propose a data sample annotation method they term Metatdata Archaeology via Probe Dynamics (MAP-D). In this method, one first creates data subsets with known properties (“probes”), such as corrupted, mislabeled, and atypical/typical samples. One then checks whether other samples have these properties by performing kNN classification on the time series of losses of unmodified data samples, with the time series of losses of the probes serving as the cluster labels. The authors apply MAP-D to six different image classification datasets and find that MAP-D performs similarly to SOTA baselines for identifying and correcting mislabeled examples, and can be used to speed up training by prioritizing training samples.
strength_and_weaknesses: **Strengths**

Simple method

Convincing results; method seems promising for multiple different applications

Well-written and mostly easy to follow

**Weaknesses**

Training speed-ups not measured in wall clock time

A few clarity issues, mostly regarding explanation of the noise correction methodology

Some of the more important content is in the Appendix, and some of what I would consider follow-up experiments are presented in the main text
clarity,_quality,_novelty_and_reproducibility: **Measuring Training Speed-Ups**

Speedups in units of steps or epochs do not consistently yield real-world (wall-clock) speedups. I think it would significantly strengthen the work if the authors demonstrated a wall-clock time improvement using their method for prioritized training, instead of just a step-wise speedup. At the very least, be specific about the units of speedup (e.g. "10x stepwise speedup" instead of "10x speedup")

**Clarity and Organization**

Related work section is thorough, but may want to include the recently-published Sorscher et al., 2022: Beyond neural scaling laws.

The text annotations in Figure 3 are difficult to read. I recommend enlarging them.

The main text makes it seem like Figures 5 and 6 are based on using the binary (argmax) method for label noise correction with MAP-D. It requires reading the appendix and comparing Appendix Figure 8 to Figures 6 and 6 to ascertain that the values in Figures 5 and 6 are obtained using the probabilistic method. I did not notice the probabilistic method mentioned in the main text at all. Please clarify this.

I think the noise correction correction protocol could be a little bit better explained in the main text. Specifically, does “number of epochs before noise correction” mean that the model is pretrained for N epochs (105 or 10 in the experiments here) in addition to the normal training duration, or that the total training duration is identical unchanged and the noise correction does not begin until N epochs into training?

The sample prioritization results are strong. Given that they’re one of the key results (and mentioned in the abstract and introduction), I think it undersells them to not describe the results, and to mention in passing at the beginning of the results section simply that they’re presented in the appendix. I understand space is limited, but I would suggest at least including a brief summary of the results in a dedicated subsection—it could be as little as one or two sentences.

I think some of the main text figures could be moved to the Appendix: specifically, Figure 3d-h, one of Figure 4a&b or Figure 4c&d, and Figure 6. This would allow moving the sample prioritization results and full related work section to the main text, as well as permitting more explanation and discussion of results.

summary_of_the_review: Straightforward method that shows promise for multiple different applications. I have a few suggestions to improve clarity and one to more convincingly measure a claimed training speed-up, but overall great work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces MAP-D, a novel approach to data archeology that leverages training dynamics to uncover a dataset's salient meta-data. In contrast to already existing approaches, MAP-D enables the simultaneous auditing of a dataset across multiple dimensions (e.g., typical data, corrupted inputs, etc).
strength_and_weaknesses: The paper's main strengths are the novelty, simplicity, and effectiveness of MAP-D. It is also reasonably well-written and organized.

The paper's main weakness is the lack of an organic, real-world dataset on which to compare MAP-D against a high-quality, human-annotated test set. The probes described in section 2.2 (i.e., typical, atypical, random labels, random input & labels, and corrupted inputs) are intuitive and easy to compute w/o human annotations. However, understanding MAP-D's performance against a human-curated test set would make the paper significantly more compelling, while also highlighting its practical strengths and weaknesses in a real-world scenario.
clarity,_quality,_novelty_and_reproducibility: The approach proposed in the paper appear to be novel. The paper is reasonably well-written and organized. However, it could be further improved by ...
(1) making Figure 3.a/b/c readable by using six or nine graphs instead the current three, so that the curves are not so "squashed" against each other; 
(2) adding a paragraph to discuss in-depth the insights of what happens in Figure 3.d/e/f; 
(3) expanding the last paragraph before "3.3" to a comprehensive discussion of the insight from Fig. 4; 
(4) emphasizing earlier (beginning with the abstract!) MAP-D's impact on noise correction, prioritizing points for training, and identifying minority-group samples (in particular, the last two should be brought in the main paper, rather than relegated to appendices). The paper should find a way to summarize (in a compelling manner) across all 6 datasets, rather than just the two CIFARs. To save space in the main paper, the author(s) could shorten the introduction and move Figure 6 in an appendix (after all, its results could be summarize in one sentence, with the interested reader referred to the appropriate appendix). 
summary_of_the_review: The paper introduces a novel, simple, effective approach to an important problem: meta-data archeology. Overall, it is well written and organized, and it makes a strong empirical case. The paper could be further improved by (i) adding more in-depth discussions of the results, and (ii) further emphasize MAP-DF's impact on noise correction, prioritizing points for training, and identifying minority-group samples.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This is a well written paper presenting a novel contribution accompanied with a well designed set of convincing experiments (with some room for improvements in he empirical scope and process). It has been assessed by four knowledgeable reviewers, three of whom recommend a straight accept an one rates the work as marginally rejectable. Based on my review, I would like to side with the majority of my colleagues and recommend acceptance of the paper in its current form. The authors have actively communicated with the reviewers during the rebuttal period and they seem to have addressed most of the pressing concerns already.
justification_for_why_not_higher_score: Reviewers have brought up some issues highlighting limits in empirical evaluation scope and design.

justification_for_why_not_lower_score: This is a good, novel and potentially useful work with a good impact potential to enable comprehensive meta-learning and related tasks.
The reviewers' concerns are not major and some of them have been already addressed by the authors via the rebuttal.
Decision: Accept: notable-top-25%