Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a consistency score (C-score) that measures the expected accuracy of a held-out instance averaged over different training sample sizes, which can be useful for analyzing learning dynamics and generalization performance.

I see some problems with the proposed approach. First of all, the proposed metric is computationally very expensive since 2,000 models need to be trained for each s. Can't we simply use the learning speed of each test sample to achieve similar goals? Measuring learning speed should be computationally much simpler since it does not require training 2,000 models if we measure the learning speed for test or validation samples. Those test samples are automatically held out since they are not in the training dataset. In fact, authors show strong correlations between C-score and other measures based on learning speed in Fig. 9. Are there *real* benefits of using C-score over simpler alternatives such as measures based on learning speed, e.g., better outlier detection performance, better accuracy for detecting mis-labeled samples, better analysis of learning dynamics and generalization performance, etc.? If there's no such *real* benefit, then it's hard to justify the use of C-score considering its high computational cost.

At s=70,80,90%, there will be a lot of overlap of samples among 2,000 subsets and I don't see much value in distinguishing such cases (70%, 80%, 90%) with such fine granularity. In Fig. 4(a), they (70%, 80%, 90%) do not show much different anyway while there's a huge difference in the histograms between 10% and the rest (20% ~ 90%). To solve this problem, I suggest using something like s=1%, 2%, 4%, 8%, ..., 64% (log scale) instead of 10%, 20%, ..., 90% (linear scale). This way, we can also see the effect of very small n, which I believe is an interesting regime to study. Using log scale may also be better for comparing consistency profiles for different datasets such as CIFAR-10 and CIFAR-100. Fig. 2 shows the consistency profile curves start at around 0.2 for CIFAR-10 at s=10% while they are more spread from 0.0 to 0.85 for CIFAR-100 at s=10%. This may be an artifact of having coarse-grained intervals for s. By using log scale, we may be able to see a similar behavior between CIFAR-10 and CIFAR-100 only shifted in log scale, e.g., the consistency profile curves for CIFAR-100 may also start around 0.2 when s=1%.

I am not sure how to justify the definition of C-score that simply averages the consistency profile over uniform s from 10% to 90%. Why uniform instead of non-uniform? Why consider averaging instead of max, min, etc.? There are many other possibilities. Taking an average over uniform s seems ad hoc to me. Can authors provide a good justification? If there's no good justification, it may make sense to define C-score at a particular value of s (what authors call a point estimate) without even defining C-score averaged over s. But, this issue does not seem crucial.

How about using a higher learning rate for samples with low C-scores to facilitate learning from small number of irregular samples? This seems to be the opposite of what authors are suggesting in Figs. 8 and 9. 

It would be great if authors can improve the performance of optimizers based on the observations made in Figs. 8 and 9.
review_title: Some problems

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes yet another score, the consistency score, to evaluate the importance of individual data samples built on held-out performance. This time the authors focus on a dynamic training perspective by studying the effect of different sample sizes. The paper is generally well written and easy to follow, despite some long confusing lines. The experiments are quite comprehensive, equipped with proper discussions. In particular, this work offers an insightful characterization of the better generalization ability of SGD through sample-level stratification according to C-score. 

Nevertheless, I still have a few concerns about the motivation and the novelty of the paper:
1. Lack of theoretical or empirical justification of the proposed metric. As motivated by the authors, consistency score is designed to tell regular examples from exception examples, measured by the held-out performance. But it seems more reasonable to use the relative accuracy (standard acc - held-out acc), i.e., the influence value (Feldman and Zhang 2020), to measure the learning difficulty of each individual example. Perhaps the authors should put more discussions on this topic and show the difference between the two metrics. Similarly, the authors also lack a discussion of the necessity of designing such a new metric based on held-out acc, which is hard to compute. As pointed out in Figure 9, there is a high correlation between C-score and accuracy (or prediction entropy), which also seems a good criterion for evaluating the quality of an individual example and is also quite easy to compute. Hence, an involved discussion/comparison of the choice of the particular formulation is needed, either theoretically or empirically. Without this, the novelty and motivation of this work would be questionable.
2. Despite the inspiring introduction, the main part of the paper lacks a clear storyline and seems verbose sometimes. For example, Figure 1,3,4,5 basically tell the same natural story (C-score proportional to sample complexity) without much additional insight on the problem. The authors should focus on discussing the interesting phenomena indicated by the new metric, rather than listing every possible experiment result. As for as I can tell, a major difference/contribution of this work is that it involves the dependence on sample size, which allows it to study the dynamic training behavior. I think the authors should focus on this perspective. For example, I do not see the necessity of discussing the average C-score of different sample sizes. Except for computational stability, I wonder whether this metric really means something, not to mention that the authors do it in a rather heuristic way, by assuming a uniform distribution over sliced sample sizes. Also, Figure 6 seems to deal with a technical issue with limited computation (again a trivial phenomenon), and it could be deferred to the appendix.
3. Personally, I find two insightful phenomena in this work worth more involved discussion. First, Figure 2 demonstrates quite different behaviors of examples with different C-scores, that the bad ones can get even worse with a large subset! Second, Figure 8 shows the C-score helps understand the superiority of SGD from a (not only intuitively understood but also empirically shown) sample-level perspective. If the authors could stick to these two stories and dig deeper, I believe the paper would be more impressive.

In all, I find this paper is well written with thorough experiments around the proposed metric. However, as mentioned above, it would be a regret that the paper lacks more in-depth discussion. I would be pleased to upgrade the score with positive feedbacks from the authors.

 
Minor points:
1. The interplay between generalization and overparameterization is not discussed at all in this paper, so the authors had better not include it in the title.


Reference:
Feldman, V., & Zhang, C. (2020). What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv preprint arXiv:2008.03703.
review_title: Interesting paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper formulates a consistency score (C-score, which characterizes
the expected accuracy for a held-out instance given training sets of varying size
sampled from the data distribution) to measure the regularity of an example. It also proposes approximations for C-score and study structural regularities of MNIST, CIFAR-10, CIFAR-100, and ImageNet. C-score is also used to learn dynamics of different optimizers and consequences for generalization. It further provides proxies (in particular, the learning speed) for C-score for practical tasks such as outlier-detection. 

################################################

Reasons for score:
The paper is overall well-written and presents some interesting findings about a newly proposed measure C-score. However, the novelty of the consistency score and the experiments on utilizing it for outlier detection are relatively limited. 


################################################

Pros:

+well-written and easy to follow

+interesting observations and good visualizations

+experiments in general support claims

Cons:

-Not enough results shown to demonstrate how useful this measure is and no baselines are provided.

-The idea of regular/irregular is not new.

-Pure empirical findings; more discussions on the theoretical side is needed


One of my main concerns is the practical implication of the proposed measure. Only a simple mislabelling experiment on one model and dataset is shown and no baseline methods are compared with.

The idea of attributing examples into regular examples and irregular examples is also not new. The learning speed and generalization as the authors mentioned has been previously observed. Using C-score to analyze the dynamics of different optimizers is interesting but is not explored deeper / more extensively.

################################################

Questions:

-Why choosing s=70 for ImageNet? rather than 50 or other numbers? Since ImageNet is much more complex than CIFAR, it is hard to tell if similar observations (other than the examples given) hold.

-Ablation study on the number of models (2000 is used currently for all experiments) to train for each subset ratio should be provided at least for one dataset.

-I believe the code can be made public anonymously?

################################################

Post-Rebuttal:

I want to thanks the authors for their detailed responses. It addresses my concern of hyperparameter choices. However, after going through the responses and paper again, I decide to maintain my initial assessment. The main reasons are that 1.conceptually C-score is not very novel. The addition of this paper to our existing understanding is not that much. 2.practically the C-score does not improve the-state-of-art of outlier detection (comparison with traditional methods is lacking).

I encourage the authors to further explore using C-score on improving outlier detection and comparing with existing traditional methods. Besides, diving a bit deeper into using C-score to analyze the dynamics of different optimizers will be very interesting. 
review_title: Intuitive measure but not enough practical implications

Meta-review: This paper proposes the c-score, which is the aggregation of a "consistency profile" that measures per-instance generalization.  Naive computation of the c-score is expensive and thus requires an approximation.  The paper then uses the c-score to analyze several image benchmarks and their learning dynamics.  

While the reviewers found the experiments to be well-done, their primary concern was over the novelty and ultimate usefulness of the c-score.  As R1 and R4 point out, the c-score correlates with other known measures such as accuracy and training speed.  The authors claim this is a contribution.  In turn, it is hard to tell if the c-score is a true metric of interest or a recapitulation of what is already known.  No reviewer was in favor of acceptance.
Decision: Reject