Deconstructing Distributions: A Pointwise Framework of Learning
Authors: Gal Kaplun
Nikhil Ghosh
Saurabh Garg
Boaz Barak
Preetum Nakkiran
Venue: ICLR 2023 poster (2023)
Keywords: understanding deep learning
empirical investigation
distribution shift

Abstract: In machine learning, we traditionally evaluate the performance of a single model, averaged over a collection of test inputs. In this work, we propose a new approach: we measure the performance of a collection of models when evaluated at *single input point*. Specifically, we study a point's *profile*: the relationship between models' average performance on the test distribution and their pointwise performance on this individual point. We find that profiles can yield new insights into the structure of both models and data---in and out-of-distribution. For example, we empirically show that real data distributions consist of points with qualitatively different profiles. On one hand, there are ``compatible'' points with strong correlation between the pointwise and average performance. On the other hand, there are points with weak and even *negative* correlation: cases where improving overall model accuracy actually *hurts* performance on these inputs. As an application, we use profiles to construct a dataset we call CIFAR-10-NEG: a subset of CINIC-10 such that for standard models, accuracy on CIFAR-10-NEG is *negatively correlated* with CIFAR-10 accuracy. Illustrating for the first time an OOD dataset that completely inverts ``accuracy-on-the-line'' (Miller et al., 2021).

Published as a conference paper at ICLR 2023
DECONSTRUCTING DISTRIBUTIONS: FRAMEWORK OF LEARNING
A POINTWISE
Gal Kaplun"
Nikhil Ghosh" UC Berkeley nikhil ghosh@berkeley.edu
Saurabh Garg
Harvard
Carnegie Mellon University
sgarg2@andrew.cmm.edu
Boaz Barak
Preetum Nakkiran
Harvard
UC San Diego
b@boazbarak.org
preetum@nakkiran.org
ABSTRACT
In machine learning, we traditionally evaluate the performance of a single model, averaged over collection of test inputs. In this work, we propose new ap- proach: we measure the performance of collection of models when evaluated on a single inpul point. Specifically, we study a point's profile: the relationship between models' average performance on the test distribution and their pointwise performance on this individual point. We find that profiles can yield new insights into the structure of both models and data-in and nutrof-distribution. For example, we empirically show that real data distributions consist of points with qualitatively different profiles. On one hand, there are "compatible" points with strong corre- lation between the pointwise and average performance. On the other hand, there are points with weak and even negative correlation: cases where improving overall model accuracy actually hurts performance on these inputs. As an application, we use profiles to construct dataset we call ---AR- subset of CINIC-10 is negatively correlated
such that for standard models, accuracy on CIFAR-IO-NE with accuracy on CIFAR-IO test. This illustrates, for the first time, an OOD dataset that completely inverts "accuracy-on-the-linn (Miller et al. 2021).
INTRODUCTION
central question in machine learning is: what are the machines learning? ML practitioners produce models with surprisingly good performance inputs outside of their training distribution- exhibiting new and unexpected kinds of learning such as mathematical problem solving, code generation, and unanticipated forms of robustness However, current formal performance measures are limited, and do not allow us to reason about or even fully describe these interesting settings.
When measuring human learning using exam, we do not merely assess single student by looking at their final grade on an exam. Instead, we also look at performance on individual questions, which assess different skills. And consider the student's 's improvement over time, to see richer picture of their learning progress. In contrast, when measuring the performance a learning algorithm, we typically collapse measurement of its performance to just a single number. That is, existing tools from learning theory and statistics mainly consider a single model (or a single distribution over models), and measure the average performance
a single test distributions Shalev-Shward & Ben- 2014 Tsybakov] 2009 Valiant 1984). Such coarse measurement fails capture rich aspects learning. For example, there are many different functions which achieve 75% test accuracy ImageNet, but it is crucial to understand which one of these functions we actually obtain when training real models. Some functions with 75% overall accuracy may fail catastrophically on certain subgroups of inputs (Buolamwini & Gebru 2018 Koenecke et al. 2020
"Equal contribution.
For example, [evlin et al. 2018 Brown et al. 2020 Radford et al.
2021 Hendrycks et al. 2021a
2020a 2021c).

---

Published as a conference paper at ICLR 2023
Models
10% Acc
3056 Ape
50% Aze
70% Ace
0.29
Figure 1: Learning Profiles. We consider the "input points vs. model" matrix of ac- curacies (i.e., probabilities of correct classi- fication), with rows corresponding to inputs and columns corresponding to models from some parameterized family, sorted according their global accuracy. A 70%-accurate model is on average more successful than a 30%-accurate one, but there are points on which it could do worse. In this case, the soft- max probabilities of the bottom image show that only higher accuracy models recognize the existence of the soccer ball, throwing them off the "Dalmatian" label. Label noise am- biguity is the reason behind some but not all such "accuracy non-monotonicities.
Points
0.05
0.07
0.24
0.18
0.38
0.2
0.05
Soaffcction
Sacce Bal
Alndan
0.2
0.5 6.6
Global Accuracy
Hooker et al., 2019); yet other functions may fail catastrophically on "outtof-distriiuttinn" inputs. The research program of understanding models as functions, and not just via single scalars, has been developed recently (e.g. in (Nakkiran & Bansal 2022).. ) and we push this program further in our work.
Figure [ Tillustrates approach. Instead of averaging performance take a "distribution free" approach, and consider pointwise performance on one input at a time. For each input point
distribution of inputs,
consider the performance of collection of models ing resources (e.g., training time, training set size, model size, etc.). While more-resources models have higher global accuracy, the accuracy profile for single point 2 i.e., the row corresponding to in the points vs. models matrix-is not always monotonically increasing. That is, models with higher overall test accuracy can perform worse on certain points. The pointwise accuracy also sometimes increases faster (for easier points) or slower (for harder ones) than the global accuracy. We also consider the full softmax profile of point 2. represented by a stackplot on the bottom of the figure depicting the softmax probabilities induced on z by this family of models. Using the softmax profile we can identify different types of points, including those that have non- monotono accuracy due to label ambiguity (as in the figure), and points with softmax entropy non-monotonicity, for which model certainty decreases with increased resources. And since our framework is "distribution free," " it applies equally well to describe learning on both in-distribution and "outtof-distrriutiin" inputs.
function of increas-
1.1 OUR CONTRIBUTIONS
In this paper, we initiate systematic study of pointwise performance in ML (see Figure D. We show that such pointwise analysis can be useful both as conceptual way to reason about learning, and as practical tool for revealing structure learning models and datasets.
We introduce a mathematical object capturing pointwise performance: the "profile" of a point with respect to parameterized family of classifiers T and test distribution D (see Section 2. Roughly speaking, a profile is the formalism of Figure T ie.. mapping the global accuracy of classifiers to the performance on an individual point.
Framework:
Definition of learning profiles (Section 2.1
Taxonomy of points (Section 3). Profiles allow deconstructing popular datasets such as CIFAR-10. 10, CINIC-10, ImageNet, and ImageNet-R into points that display qualitatively distinct behavior (see 3 and 4). For example, for compatible points the pointwise accuracy closely tracks the global accuracy, whereas for non-monotone points, the pointwise accuracy can be negatively correlated with the global accuracy. We show that a significant fraction standard datasets display noticeable non-monotonicity, awhich is fairly insensitive to the choice of architecture.
Figures
Pretrained End-to-End Methods (Section 3.2). Our pointwise measures reveal stark differences between pre-trained and randomly initialized classifiers, even when they share not just identical architectures but also identical global accuracy. In particular, we that for pre-trained classifiers
2

---

Published as a conference paper at ICLR 2023
the number of points with non-monotone accuracy is much smaller and the fraction of points with non-monotone softmax entropy is vamishing small.
Accuracy the line and LIAAR-I-NNCC (Section 4 Using profiles, provide novel pointwise perspective on the accuracy-on on-the-linnee phenomenon of (Miller et al. 202T). As an application of our framework, we construct new "out-of-distribution dataset CIFAR- TO-NEG: set of 1000 labeled images from CINIC-10 on which performance of standard models trained on CIFAR-10 is negatively correlated with CIFAR-I0 accuracy. In particular, a 20% improvement in test accuracy on CIFAR-10 is accompanied by a nearly 20% drop in test accuracy on CIFAR-IO-NEG. This shows for the first time a dataset with low noise which completely inverts "accuracy-on-ee oneteee---.-
Theory: Monotonicity in models of Learning (Section C). We consider three different theoretically tractable models of learning, including Bayesian inference and a few models previously proposed in the scaling law and distribution-shift literature (Recht et al. 2019; Sharma & Kaplan 2020 Bahhi et al., (02T). For these models, we derive predictions for the monotonicity of certain pointwise performance measures. In particular, all of these models imply pointwise monotonicity behaviors that (as we show empirically) are not always seen in practice.
We demonstrate that pointwise analysis of learning is possible and promising. However, we present only an initial study of this rich landscape. In Section we discuss how our conceptual framework can guide future work in understanding in- and -ut-t--distribution learning, in theory and practice.
1.2 RELATED WORKS.
The line of work on Accuracy-on-the -o--to---tt--eeee (AoL) (Recht et al., 2019 Miller et al. 2021 studies the performance of models under distribution shift, by examining the relation (if any) between in-distribution and -utto--distribution accuracy of models. Similar to us, some works examine instance behavior in training: (Zhong al. 2021) propose studying instance- performance in the NLP setting, and also take expectations over ensembles of models. Our framework is considerably more general, however, and we give new applications of this general approach. (Feldman & Zhang 2020) and (Ilyas et al. (2022) propose tools for measuring the effect of specific training samples on the pointwise accuracy of given example for models trained with a set amount of resources. (Swayamdipta et al.) (0200) analyse individual examples in the training set using information based on the training dynamics. (Tnevaa al., 2018) look "forgetting events", i.e., when training examples move from being classified correctly to incorrectly, resembling our notion of non-monotonicity.
OOD Robustness (Hendryeks et al. 2020b Radford et al., 202Th show that large pretrained models are more robust to distributions shift and (Desai & Durrett (020) show that large pretrained models are better calibrated on OOD imputs. There is a also long line of literature on OOD detection (Hendrycks)
& Gimpel 2016 Geifman & Yl-iv 2017 Liang et al. 2017: Lakshminaray ynnan et al. 2016]
Jiang et al. 2018; Zhang et al. (2020), uncertainty estimation (Ovadia et al. 2019). and accuracy prediction (Deng & Zheng] 2021]Guilee et al. [202]] Garg al.] under distribution shift. Our work can be seen as an extreme version of "distribution shift", using distributions focused on a single point.
Example difficulty Much work was made recently to understand example difficulty for deep learning (e.g.. (Jiang et al.) 2020 Agarwal Hooker 2020 Lalor al., (017)). Several works study deep learning (Nakkiran et al. 2019b Baldock et al. 2021) through the lens of example difficulty to uncertainty) of deep models, while others try to modify the training distribution via either removing mislabeled examples (Pleiss et al. 2020 Northeutt et al., 202T), or controlling for hardness (Shrivastava et al. 2016; Hacohen & Weinshall 2019). The main difference with our work is that we focus on the shape of the curve of example accuracy with respect to a parameterized family of models.
understand certain properties (e.g.. generalization
Model Similarity Several works demonstrated that the best supervised models tend to make similar predictions. (Mania et al. 2019) measure the prediction agreement between standard vision models on ImageNet and CIFAR-10 and concluding that agreement levels are much higher than they would be under the assumption of independent mistakes. (Gontijo-Lopes et al. 2021 study the effect of different training methodologies similarity between models independently trained on different data subsets. In contrast, we focus
model similarity. (Nixon et al. 2020) shows high
not on comparing different types of models, but rather comparing models that span large interval
3

---

Published as a conference paper at ICLR 2023
Imagetitt vs Imagelen-cc
Imagehal vs wwwgemee.ccceeeeee
Image (P3480 sutt aratiiii
Image fer Imagellet 1924033
Iregett wwwpphossssnooooooooooo www.aagoeeiieeerrn.. Rodes
Beuke sp Derammecis
E
17 03 05 Imagelett Accunecy
87
63 36 14 53
01
54 DA
Accurwcy
Acturaryy
Figure 2: Zooming In. Average ImageNet accuracy for different models (ResNet-50,, DenseNet- 121, and DenseNet-169) on the x-axis with the y-axis corresponding (from left to right) to 1) ImageNet-v2 accuracy; 2) ImageNet-v2 dog-superclass accuracy; 3) the performance on single image of dog (i.e., the accuracy profile). The rightmost panel zooms in further and shows the ooftmax-profile of single image for ResNet-50. This image is a "compatible point" the sense that as global ImageNet accuracy increases, the pointwise accuracy increases, and the entropy of the softmax distribution decreases.
of accuracies. However, the above results, as well as our investigations, suggest most points' profiles remain similar under varying architectures or subsets of data.
2 ACCURACY-ON-THE-
ZOOMING IN
We first explore the pointwise perspective through the distribution shift from ImageNet to ImageNet-v2 (Recht et al. 2019). To start, in the left panel of Figure 2 2021) and show that for a wide variety of models, accuracy on ImageNet-v2 is well-approximated that such
we replicate (Miller et al.
by simple monotone function of the ImageNet accuracy. In the middle panel, a relation holds even when we consider accuracy only on the ImageNet-v2 dog super-class. That is, zoom-in the y-axis, and go from averaging over the entire ImageNet-v2 distribution to averaging over only dog classes. We see that accuracy on this sub-distribution also obeys a strong correlation with the global accuracy. This is interesting, since priori classifiers with equally-good global performance could have very different performance on dogs.
Zooming in even further, in the third panel of this figure we evaluate the same models on an just one individual dog sample. That is, we compute the accuracy profiles (per Definition 2.2 of this particular point with respect to several parameterized learning algorithms. This example illustrates and puts into context the type of object we are interested in studying. namely general learning profiles which measure pointwise statisties of learning algorithms as function of global performance.
2.1 FORMAL DEFINITIONS
We now formally define our central objects which are the learning profiles of a point (a,, 3) with respect to some parameterized family of learning algorithms and a test distribution. These objects, visually represented in the bottom of Figure D capture the behavior of models from the parameterized family on as function of their global performance on the test distribution. A classifier (or model) a function f X - A()) that maps an input x E X' into a probability distribution over the set of labels )/ For example, for a DNN, J (I:) denotes the softmax probabilities on input ar. We denote by f(x) the prediction of the classifier on a, obtained by outputting the highest probability label. We consider a parameterized family T (n) of algorithms, where n corresponds to some measure of resources: number of samples, model size, training time, etc., and T(n) denotes the distribution of models obtained by running the (randomized) learning algorithm T with n amount of resources. For the purposes of this formalism, we consider the training set to be part of the algorithm. and make no assumptions on how it is chosen or sampled. Generally, the expected performance of ((n) w.r.t. a global test distribution DD will be a monotonically increasing function of n., and there are many works on "scaling laws" for quantifying this dependence (Rosenfeld et al., 2019: Heniihhn et al.) 2020; Kaplan et al., 2020: Bahri et al. 2021). In particular, we consider standard training set-ups which avoid so called ddouble-descent" (Nakkiran et al.) 2019a) pathologies when taking n to be number of samples or training time. For reasons of computational efficiency, we use training time as resource measure in our experimental results. However, an increasing body of works suggests that different resource measures such as time, sample size, and model size, have qualitatively similar
impacts (Nakkiran et al. 2019a 2020 Ghosh et al. 2021
Kaplan et al.] 2020
4

---

Published as a conference paper at ICLR 2023
Compatible
Easy
Hard
Non Monotone
In Dist
OOD
Imagalet
Skotch
Imagelitt R
Imagenet
ImageNet
Figure 3: Taxonomy of Samples. Different qualitative profiles for ResNet-50 trained on ImageNet, roughly classified into 4 categories: 'Compatible' samples where sample performance traces global performance; 'Easy' samples which are classified correctly even by poor models; 'Hard' samples that even the best models fail on and; Non-monotone ones where performance behaves unpredictably w.r.t. global performance. Top row is ImageNet's validation dataset and bottom row is OOD examples. Remarkably, similar profiles emerge for both in-dist and OOD examples.
The pointwise accuracy AcC2,T(1) (1) of T on point = = (xr, y) is the probability that the output classifier f = ((n) makes correct prediction, i.e., f(x) y. The global accuracy Accp,7 (n)) of T(n) with respect to a distribution D over Z : X' x 3' the expected accuracy of points sampled from D, i.e., Eg~D [Acc,,7(1) (n)]. Throughout this paper, will omit the test distribution D from subscripts when it is clear from the context. We will assume that our family is globally monotonic in the sense that ACCD,7(1) 2 ACOD,T(11) for 2 n'. This assumption is merely for convenience, and can be ensured e.g., by early stopping.
The accuracy profile of a parameterized algorithm T and point 2 is the curve that maps global accuracy p E [o, 1] to the expected pointwise accuracy of (n) at 2, that is n is set so the global accuracy is p. accuracy profile of particular point. Formally:
For example, the third panel of Figure represents
Definition 2.1 (Accuracy profile). Let T, D be as above. The accuracy profile of point 2 = (z, y) the (possibly partial) function AI,T,o [0. 1] ~ [0, 1] that maps global accuracy p E [0, 10 to Acc,,T(M) (n), where n(p) is chosen such that AccT, AccT,p(n(p)) (n(p)) p.
As we will see, to get more insight on model performance we sometimes need to go beyond the accuracy and observe the full softmax probabilities induced by the model at particular point. This motivates the following definition of sofimax profiles, which are visually represented as stackplots in both the fourth panel of Figure 2 and bottom of Figure [
Definition 2.2 (Softmax profile). Let T, D be as above. The softmax learning profile of a point
2 (z. y) the function St,TI [0, - A()) that maps a global accuracy p [0. 1] to the averaged softmax distribution of predictions at 2, among classifiers with global accuracy Specifically, with = n(()) as above, we define S2,T,o(y) == (n)))..
We use the general name learning profile of a point to describe any map from p E [0, 1] to some statisties of the distribution (=) Defining learning profiles as a function the global accuracy (as opposed n)) allows to compare different resource measures on the same axis.
3 STRUCTURE AND DIVERSITY OF DATA AND MODELS
We now conduct systematic study of the structure of profiles, exploring what they can teach us about data samples and training algorithms. Profiles are joint functions of an input point and a training procedure. Below, we will first fix training procedure and vary the choice of input points: this reveals structure in data sets, through the lens a given model. Afterwards, we will fix an input point and vary the training procedure: this reveals structure in training procedures, through the lens of test point.
5

---

Published as a conference paper at ICLR 2023
ImageNet
ImageNet-v2
ImageNet-R
ImageNet-Sketch
ImageNet-A
DenseNet-12 ResNet-50
8.6%
5.3%
2.1% %/
25.1%
18.9%
30.4%
15.9%
14.7%
54.1%
18.7%
74.5%
9.4%
85.3%
47.6%
non-monotone
easy
hard
compatible
11.4% 29.4%
7.1% 29.2%
2.9%
2.9%
16.4%
28.7%
29.8%
54.2%
17.1%
8.6%
40.5%
Figure 4: Each pie-chart gives pointwise decomposition of dataset according profile types. Each point is classified based on the accuracy profile of an ImageNet trained classifier (see the end of Section 3.1
The decompositions are similar for both ResNet-50 and DenseNet-12 architectures.
3.1 STRUCTURE IN DATA
We first fix a training procedure and use the resulting profiles to study both and cittof--distibbution samples. From this analysis we broadly sketch the landscape of the various profile types. Note that the type of a point is dependent on the training procedure. [ several "prototypical" profiles encountered in real data and the corresponding samples. We highlight the following qualitative types:
1. Easy points for which even low global accuracy classifiers succeed with high probability, Note that there are out-of-distribution points which are "easy" for ResNet-50, such as the shed painted as a school-bus in Figure Further, not all easy points are alike: some samples are "harder-than- average" for weak models, that become "easier-thann-verre
for strong models (e.g. Figure O
2. Hard points for which even high-accuracy classifiers fail. By looking at the softmax probabilities, can disentangle the causes for the difficulty. Some points are simply ambiguous or mislabeled. For other points the softmax entropy remains high even at high global accuracies, and even the top-5 accuracy low.
3. Compatible points for which the accuracy profile is close the identity (y x)) function, i.e., pointwise accuracy closely tracks the average performance. not a priori clear that compatible points should exist. For example, one might expect the accuracy profile always be step function, with the individual accuracy a sample jumping from 0 to when global accuracy crosses some threshold. That is, the model could have "grokked" the sample at some global accuracy level, but performed trivially before this level (in the terminology of (Power et al.) 2202))).
Non-monotone points for which the pointwise accuracy is anti-correlated with the global accuracy in some intervals. Again, underlying reasons for the non-monotonicity of such points. Some are mislabeled or have an ambiguous label then the classifier struggles with choosing the correct label. Other points even have non-monotone softmax entropy which implies that higher global accuracy classifiers are actually less certain about this point than lower global accuracy classifiers. The existence of non monotone points may not always be an issue, since it is possible that powerful enough models can eventually reach high accuracy on such points. However, when individual point accuracy is important, non- monotonicity may be undesirable in resource constrained settings. Nonetheless, non-monotonicity is certainly unusual and such points may provide rich examples of interesting learning behavior.
the softmax profile to better understand the potential
To try to get better quantitative understanding dataset structure through the lens of our taxonomy on learning profiles, use the following procedure to classify profile either easy, hard, compati- ble, or non-moootone. To evaluate if point is non-monee mon-oonttone, we compute the non-monotonicin score of its profile. which measures the cumulative drop in pointwise performance as global perfor- mance increases (see Appendix BB. If the point has a non monomooontii score greater than 0.1, which indicates noticeable non-monotonicity, then classify it non-monotone. Otherwise, we classify

---

Published as a conference paper at ICLR 2023
the profile as easy, hard, compatible based the L2 distance of the profile to a corresponding "template" profile. Easy points are represented by the profile f(p) = 1, hard points by f(p) = 0. and compatible points by f(p) p. In Figure [4 plot the decompositions of various datasets according to the described classification of their accuracy profiles. Each profile is computed from the predictions of an ImageNet trained architecture. We see that as expected ImageNet and ImageNet-v2 contain significantly many compatible points, although there are still many points that are not most accurately described as compatible. The datasets ImageNet-R, ImageNet-Sketch, and ImageNet-A are significantly less compatible with ImageNet and are progressively harder.
The examples above are meant to illustrate the potential of the learning profiles as means of better understanding data and learning in particular, considering entire profiles can often reveal more insight than just the final pointwise accuracy. Although the choice of profile types in our taxonomy may not be the optimal classification, we can nevertheless see that it allows us to gain insight into the structure of datasets. We hope our initial investigation can inspire future work in this area.
3.2 STRUCTURE IN TRAINING PROCEDURES
Just as we can understand different samples by fixing a training procedure, we can also understand different training procedures by their behaviors on fixed sample.Takee Taking this viewpoint, we investigate standard architectures (ResNet-18 and 1enseNet-121) on CIFAR-10, considering both models trained from scratch and those pre-trained on ImageNet. (See full experimental protocol in Appendix [A]. In Figure 3 we decompose the CIFAR- 10 test set similarly to Figure 4 but now with the perspective of understanding model differences through the dataset. We can that the models trained from scratch and the pre-trained models exhibit very different decompositions, but very similar between architectures. In particular, we see that with pre-training the number of non-monotone examples decreases and points become overwhelmingly compatible. We now further probe the observed model similarity and monotonicity induced by pre-training.
Model Similarity We start by introducing a distance measure to compare two training procedures. Given two softmax profiles, we define the profile distance to be the average over all test points and accuracies E [0, 1] of the L1 distance between the softmax distributions at accuracy (see Appendix B The rightmost panel of Figure 6 shows the pairwise profiles distances between several architectures and their pretrained variants. We find that profiles of pretrained models significantly differ from non-pretree non-pretrained ones. However, controlling for the presence of pretraining, model architecture does not seem to significantly affect profiles.
Pretraining Induces Monotonicity We also show specific way in which pretraining affects profiles: it drastically reduces the number of points which are non-monotonic. To quantify this effect, we use the non-monotonicity score of profile which is the negative variation of the profile and is large when negative-slope regions exist (see Appendix B). The right panel of Figure Figure compares the CDFs of the accuracy profile non-monotonicity scores for both from scratch training and fine-tuning (see also specific example in the middle panel). We observe that models trained from scratch prone to significant amounts of non non-monotonicity. while pretraining eliminates non- non-monotoniccty almost completely. Further, this "elimination of non-monotonicity" by pretrained models applies for both the accuracy and entropy profiles (see Appendix Figure 10), suggesting that pretrained models display an inductive bias similar "idealized" Bayesian inference, which always displays monotonicity (see Theorem C.Ip. Note that we have only considered a few specific pre-training settings and it would be interesting for future work explore wider range of settings see what conclusions still hold.
4 POINTWISE PERSPECTIVE ON DISTRIBUTION SHIFTS
We now show that the pointwise perspective can shed light on distribution shifts. An important open question in this area is to understand the relationship between in- and out-of-distribution (OOD) performance, and how it depends on different factors such as pre-training. To probe this relationship, it is a common practice to evaluate methods on many OOD test sets, and measure in-dist vs. OOD performance (Recht et al. 2019 Radford et al. (021). In several cases, these metrics are linearly correlated (after probit scaling), phenomenon known "accuracy-on-the one (Recht et al. 2019
Miller et al. 202I). However, this phenomenon does not hold universally, and we do not yet have good understanding of when a distribution pair is linearly-correlated.
7

---

Published as a conference paper at ICLR 2023
8
Densenet121
Resnet18 preetrriedd
Densenet121-pretrainee
30.6%
31.9%
18.6%
7.2%
5.5%
7.7%
9.1%
9.7%
87.2%
84.3%
41.6%
40.7%
non-monotone
easy
hard
compatible
Figure 5: Taxonomy of the CIFAR-I0 test set computed analogously Figure using the models trained on CIFAR-10 from Section 3.2 Pretraining significantly increases compatibility.
CIFAR-10 Image #665 (ship)
Accurady Kon-manotoricity
Softmax TV Distance
Fraction Excending Sizze
Pointate Accunicy
witeonti paraarad
Dursoncti21
Resnetia-pt go
0.12
up Ios sall
Dunseritt22 go
6.12
02
Homett6
Demamet223
0.3
0.4
0.6 Accuracy CIFAR-IO
0.5
0.7
0.8
13ah
Nun-monstoriccy
Figure 6: Scratch vs. Pretrained. A study of model similarity on CIFAR-10. Left panel: Averaged CIFAR- 10 test dataset, profiles cluster together based on pretraining and not architecture. For example, ResNet-18 profiles are much closer to DenseNet-12 121 ones than to ResNet-18 -8 pretrained ones. Middle panel: Illustration of the effect of pretraining on single image. Right panel: Pretraining reduces accuracy non-monotonicity across many points in the CIFAR-10 10 test set.
In the previous section, our pointwise analysis demonstrated the existence of non-monotone instances where pointwise and global performance are anti-correlated. Interestingly, such examples occur often enough for us to construct non-degenerate out-of-distribute test set which breaks "the line" in much stronger ways than were previously known (see Figure 77.
Using pointwise perspective, construct CIFAR- 10-NEO (see Figure 7. CIFAR-101 10-like, class balanced and correctly labeled dataset datasst 10 accuracy. Specifically, improving test accuracy by 20% (from 60% 80%) CIFAR-10 hurts test accuracy by R 20% on CIFAR- 10-NEG, when training with ResNet-18 and DenseNet- behavior of R8 and DenseNet-121 121 are largely similar expected, and we believe the same behavior will be exhibited for standard architectures trained from scratch. In contrast, CLIP fine-tuned models on CIFAR- 10-NEG are linearly correlated with CIFAR- performance Since the models are only trained to 80% accuracy on CIFAR-10, 10, it is possible that the anti correlationn can be corrected higher accuracies, however the anti-correlation exhibited is already significant as it persists for wide range of fairly high accuracies and the drop in 1-FAR-10-NEEL performance is significant.
1000 samples, which is anti-correlated with CIFAR-
The
To identify a dataset points with negative correlation, we start with the CINIC-10 test (Darlow al. 2018). To avoid ambiguous and mislabeled points in our new dataset, we perform CLIP-fitterig: we restrict the CINIC-10 test set to points correctly predicted by a CLIP model fine-tuned on the CIFAR-10 train set. We train several ResNet- 18 models on CIFAR-10 dataset and obtain per-sample monotonicity scores (defined in Appendix B) for the CINIC-10 test set. After sorting points with nun-monotonicity score, select perfectly balanced dataset of 1000 points consisting of the top 100 most non-monotonic samples from each class. We remark that it likely that such an approach likely be used to construct analogous datasets which demonstrate similar anti-correlated behavior.
Dataset: httts:://aoonmmoooo//o
4open. sotteeeeeeeee/e///aaa--------
'Correct labeling is essential; incorrectly labeled examples will naturally he negatively correlated with global performance. As heuristic, we use CLIP to filter such samples. *Fully fined-tuned CL.IP achieves 100% accuracy CTFAR-IO-NEG by design.
8

---

Published as a conference paper at ICLR 2023
CEAR- at CEARIO
Frataned
(a)
(b)
(c)
(d)
Figure 7: CIFAR-IO-NEG. (a) We construct CIFAR- 10-NEG, a clean dataset that has negative correlation with CIFAR-10 10 for standard models (e.g., ResNet- 18 and DenseNet- 121): improving test accuracy on CIFAR-10 hurls test accuracy CIFAR-1O-NEG, (b) Pretraining restores monotonicity: pretrained models (e.g.. CLIP and ResNet- 18 and DenseNet-121 pre-trained on Imagenet) have strong positive correlation with CIFAR-IO-NEG, (c) A learning profile of one LIFAR-1O-NEL example, showing how profiles reveal more about the evolution of predictions across learning. (d) Samples from CIFAR-10-NEGE We contrast CIFAR-10 samples with CIFAR- 10-NGG samples in Appendix
Previous works observed weak correlation under distribution shift, but we are the first to observe anti- correlation between in-distribution and out-of-distribute performance for natural (non-adversaria) and correctly labeled images. FigureZi) shows a sample from this dataset. We juxtapose CIFAR- 10 test set examples with more examples from CIFAR- 1A-R-1 in Appendix It is an interesting question for future work to generate such datasets using more natural constructions and to better understand the source of the anti-correlated behavior.
DISCUSSION AND CONCLUSIONS
We conclude by discussing why we believe the pointwise perspective in general and learning profiles in particular are central understanding both on- and off-distribution learning.
Out-of Distriiiitiiittonn Inputs When deploying ML systems, inputs are rarely drawn from exactly the same distribution as the train set. Many existing frameworks try to model this as distribution shift: they assume test inputs are drawn from a distribution DD, that is related to the train distribution DD in some way (e.g. covariate shift (Heckman 1977, Shimodaira 2000), label shift (Lipton et al.. 2018, Garg et al. 2020), or closeness in some divergence (Ben- David et al. (010)). However, in practice the distribution is often not well-specified or indeed, a distribution at all. We often care about performance image. Furthermore, the inputs to our system may change in arbitrary and unmodeled ways (with weather, country, wildfires, etc). An mnstancewwis perspective is crucial in these settings.
particular instances: e.g., on correctly recognizing a pedestrian in this specific
Lessons for Theory We outline several concrete lessons for theory, and some speculative ones. Concretely, our experiments have identified arguably unexpected behaviors of real models and real datasets, which any potential theory of deep learning must be consistent with. For example, in Section 3 we found significant number of real in-distribution samples on which DNNs are accuracy non-monotone: where networks with higher average accuracy (e.g., trained on more samples, or for more time) actually perform much worse. This behavior is impossible in many toy models of learning, as we prove in Appendix C Thus, our experiments serve as guidelines for future theory work.
Lessons for Practice We expect that pointwise profiles are an interesting new measurement in many settings, which may reveal effects obscured by coarser metrics. For example, studying the softmax pro- file of point can reveal not only a model's final accuracy on this point, but how its predictions evolved as it learnt, and potential causes of confusion along the way (e.g., texture bias or ambiguous objects). Our finding of non-monotone samples also suggests that current learning techniques are suboptimal in certain ways, but gives hope they can be improved. Specifically, non-monotone samples are those for which weaker models perform well (and thus, we know learning is possible), but stronger models for some reason regress. It may be possible to fix this "irrational" behavior in practice, since we know such samples are not fundamentally difficult. Indeed, we find that some techniques such as pretraining why is important question for future work.
also eliminate most non

---

Published as a conference paper at ICLR 2023
REFERENCES
Chirag Agarwal and Sara Hooker. Estimating example difficulty using variance of gradients. CoRR, abs/2008. 1600, 2020. / absp://bbs//000./ 11600
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws, arXiv preprint arXiv:2102.06701, 2021.
Robert N. Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. CoRR, abs/2106.09647, 2021. URL
/abss//ab/// /2006
096471 7
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2), 2010.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HlerbertVooss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR. 0bbs/2005.1416.. 2020. URL BtLpttps: arxiv org abs attaaabs//000.00 1015.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference fairness, accountability and transparency. PMLR, 2018.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv: 170..088/8 2017.
Luke Nicholas Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey. CINIC-10 not imagenet or CIFAR-1O, CoRR, abs/1810.03505, 2018. URL
1810. 03505
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
15069-15078 2021.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 295-302, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/1/2020. emnlp-main.21. URL Bttpttttst////cattiioo
org/2020. emnlp main.21
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep 2018. URL
bidirectional transformers for language understanding. CoRR,
04805
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891, 2020.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Saurabh Garg, Sivaraman Balakrishnan, Zachary Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of natoo--distrbutionn performance. In International Conference on Learning Representations (ICLR), 2022.
Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. arXiv preprint 1755::1005.08500, 2017.
Nikhil Ghosh, Song Mei, and Bin Yu. The three stages of learning dynamics in high-dimensional kernel methods, 2021.
10

---

Published as a conference paper at ICLR 2023
Raphael Gontijo-Lopss, Yann Dauphin, and Ekin D Cubuk. No one representation rule them all: Overlapping features of training methods. arXiv preprint arXiv:2110.12899, 2021.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. arXiv preprint arXiv:2107.033/5, 2021.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. CoRR, abs/1904.03626, 2019. URL. http:ttt://arxi..rrr/////////
03626
James J Heckman. Sample Selection Bias as a Specification Error (With an Application to the Estimation of Labor Supply Functions), 1977.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1610000 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR, abs/2009.03300, 2020a. URL http:httt///xiiiii org/ abs/ 2009 03300
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve -uttof-distributionn robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2744-2752 Online, July 2020b. Association for Computational Linguistics. doi: 10. 118633/1122020 mainn URL https
//aclantho ogy. org. 220gg/2220..m acl -m..ma...mm2
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burs,, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. CoRR, 0aax22155.09936 2021a. URL
2105 09938
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. CoRR, abs/2103.03874 2021c.
wwwporgggabb//ooo 03874
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardr, and Dawn Song. Natural adver- sarial examples. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 15262-15277 2021d.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCan- dlish. Scaling laws for autoregressive generative modeling. CoRR, abs/2010.1470.1 2020. URL
https: //pp:i://axvvv.// wttp//////o.....oo./ http:p////axx...rr////////0..01 14700
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- 2019.
pressed deep neural networks forget? arXiv preprint
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting predictions from training data. arXiv preprint arXiv:2202.00622. 2022.
Heinrich Jiang, Been Kim, Melody Y Guan, and Maya R Gupta. To trust or not to trust a classifier. In NeurlPS, pp. 5546-5557, 2018.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Exploring the memorization- generalization continuum in deep learning. CoRR, abs/2002.03206, 2020. URL hitpps:// arxiv. arxiv.org/abal /ttggbbbbb////// 03206
11

---

Published as a conference paper at ICLR 2023
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford. Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08311, 2020. URL/ttts:: petttps::/aaxiv.crtt/ wwtp///gg/aaa/aeeoooo.ooo 2001 08361
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):768--768, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv:1662.00147,, 2016.
John P. Lalor, Hao Wu, Tsendsuren Munkhdalai, and Hong Yu. An analysis of machine learning intelligence. CoRR, abs/1 702.04811, 1, 2017. httttp::/aaxi..or////eaeeee
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.0200 2017.
Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In International Conference on Machine Learning (ICML), 2018.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. arXiv preprint 1rXiv:/905./25500 2019.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference Machine Learning, pp. 7721-7735. PMLR, 2021.
Preetum Nakkiran and Yamini Bansal. Distributional generalization: Structure beyond test error, 2022. [KL[itttps://peerevele net/ http://wwweee/fooommmdm-d---
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. CoRR, abs/1912.0029) 2019a. URL
abs//915s http://arxiv.ort wwtp://wwwwwggo/////.//.... 02292
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang. Benjamin L. Edelman, Fred Zhang, and Boaz Barak. SGD on neural networks learns functions of increasing complexity. CoRR, abs/1905.116600 2019b. URL/http://arxiv.org
www./a/s.s..... 1905. 11604
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In International Conference on Learning Representa- tions, 2020.
David Navon. Forest before trees: The precedence of global features in visual perception. Cognitive psychology, 93):353-383, 1977.
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles 2020 workshop, 2020.
not better? In "I Can't Believe It's Not
Curtis Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks, 2021.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint 1906.00:900022555 2019.
Geoff Pleiss, Tianyi Zhang, Ethan R. Elenberg, and Kilian Q. Weinberger. Identifying mislabeled data using the under the margin ranking. CoRR, abs/2001. 2020. URL https
//arxitr org/ abs/ 2001. 10528
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- cralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv.2201.02177, 2022.
12

---

Published as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 0rXiv:2003.0002: 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Jonathan Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. constructive prediction of the generalization enror across scales. CoRR, abs/1909. 12673, 2019. BtRL:/tp:///ax
org/abs/1909. 12673
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Utkarsh Sharma and Jared Kaplan. neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004. 2020.
Hidetoshi Shimodaira. Improving Predictive Inference Under Covariate Shift by Weighting the Log-L.ikelihood Function. Journal of Statistical Planning and Inference, 2000.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), June 2016.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint (rXiv:2009.10795 2020.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study example forgetting during deep neural network learning. CoRR, abs/1812.05159, 2018. URL httt:::///arxx: org
05159
Alexandre B Tsybakov. Introduction to nonparametric estimation., 2009.
Leslic G Valiant. theory of the learnable. Communications of the ACM, 27(11):1134-1144 1984.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518, 2019.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In European Conference on Computer Vision, pp. 102-117. Springer, 2020.
Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. Are larger pretrained language models uniformly better? comparing performance at the instance level. CoRR, abs/2105.000 10..0200,, 2021. URL https::////// htt.p//://.aalvooor////////.....
13

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
CIFAR-10 Experiments In the experiments of Section 3.2 models trained from scratch on CIFAR-
10 were trained from random initialization using SGD with Cosine Annealing learning rate schedule with initial learning rate 7) 0.01, batch size 128, and weight decay 5 x 10-4, for 30 epochs. The pre-trained models were first trained from scratch on the full ImageNet 32 x 32 (Chrabaszzz et al., (017) training set using those same hyperparameters but for 100 epochs. For fine-tuning pre-trained models on CIFAR-10, the linear classification layer was initialized randomly and then trained using SGD with learning rate of 0.001 and batch size of 128 for 3 epochs with no weight-decay. For all training, we used standard data augmentation (i.e., random horizontal flip, random crop of size 32 x 32 with padding size 4, and mean/std normalization). For each CIFAR-I0 training run a single model was trained on a random subset of the training set (50,000 total samples) of size 10,000 for models trained from scratch and size 5,000 for pre-trained models. Profiles were evaluated on the CIFAR- 10 test set (10,000 total samples) twice per epoch for scratch models and 10 times per epoch for pre-trained models. We computed profiles based on the evaluations of 50 independent runs. The final profiles were computed by performing Gaussian filter smoothing with and linear interpolation an equally spaced grid of length 50. The architectures used were ResNet- 18 and 1e1
ImageNet Experiments For ImageNet, we train randomly initialized seeds for three standard architectures: ResNet-50, DenseNet121 12 and DenseNet-169 for 90 epochs with SGD with momentum 0.9, weight decay 0.0001 and learning rate schedule of [0.1, 0.01, 0.001] for 30 epochs each and batch size of 256 (128 for DenseNet-169), We use standard data augmentations (i.c., flip and random crop to 224x224 images). To produce softmax-profiles we use 30 equally spaced checkpoints (adding 10 checkpoints around learning-rated drops increase plot resolution) evaluated both and off distribution (ImageNet-A, ImageNet-R, ImageNet-sketch, ImageNet-v2 (Hendrycks et al. 2021b
Wang al. 2019 Recht et 2019)).
14

---

Published as a conference paper at ICLR 2023
B ADDITIONAL DETAILS
Profile Distance We define the distance between the P-profiles (e.g. accuracy-profiles, softmax-
profiles, etc.) of training procedures Ti and T2 to be i fo d(P2.T (p), Pz,7a (p)) dp, where d is some distance measure on A()) and E denotes averaging over points the CIFAR-I0 test set. left Σyey By 4yl
panel of Figure 6 we take d to be total variation (TV) distance, defined as d(p,
Plots for other choices of are shown in Figure 11
Non-Monotonicity Score Given a profile P. [0, - aa b), we can measure how much the curve p Pp(p) deviates from being monotonically increasing by computing the non-monotonicity score.
nmono = f max (0, fo-- P=(p)) dp.
-0:-ap dp
Note that the non-monotonicity score is always non-negative. It is zero if and only if always increasing and is bounded by b In the right panel of Figure 6 we plot the 1 CDF of the non-monotoniccty scores of the accuracy profiles Ag on the CIFAR- test set. In Figure I0 we show the respective plots for the negative entropy profiles p EEH (T))((() where
np)) such
that ACcT, DD (p)) = p and the soft-accuracy profiles + (p)ly where for т E A()). Ty is the probability assigned
y under π.
Pointwise Accuracy-on-the-CCiee Curre The plots in Figure 2 suggest the conjecture that families of algorithms {7,} which have the same global accuracy curve, also have approximately similar pointwise accuracy profile: the pointwise accuracy As,T, (p) on is well approximated by function gz (p) that only depends on the point z (and not the algorithm T.). One way to test such conjecture is to look at two algorithms T and T' and measure the average absolute difference of their pointwise accuracies at a given global aceuracy (i.e., d(p) E2 (A, A..T (p))). In Figure T2 we plot from CIFAR-10.2 when training ResNet-18 18 and DenseNet-121
d(p) when evaluating
CIFAR-
10. We see that d(p) is non-negligible but still much smaller than we would expect pointwise performance between T and T' was completely uncorrelated giving some weak evidence in support of the conjecture.
15

---

Published as a conference paper at ICLR 2023
C MONOTONICITY IN MODELS OF LEARNING
We proceed to show that accuracy and softmax profiles of several models of learning obey natural monotonicity properties. This in contrast with our experimental results of Section that demonstrate the existence of points with non-monotone accuracy and softmax entropy, in particular when training models from scratch (as opposed to fine-tuning). This mismatch between theory and practice can be interpreted in two (non-mutually exclusive) ways. One is that we need better models to capture realistic learning methods. The second is that m--oooonooiitty suggests sub-optimal behavior in practical methods, and as they improve we might expect profiles to become monotone. In particular, the fact that the accuracy and softmax entropies of Bayesian inference with a correct prior are monotone (see below), suggests that practical non-monode con-mootonnicty might arise due to "mismatched priors". We start by defining the following three natural monotonicity properties with respect to set of possible instances 2 and a set of algorithms ALG:
1. Accuracy monotonicity: We say that a parameterized learning algorithm T satisfies accuracy monotonocity if Vz E 2: 2 n' - (n) Z (n')..(n')..... That is, improving global accuracy cannot hurt on any specific instance. We also consider a weaker version which we call a pointwise scaling law, whereby there are constants C, > 0 such that for all E Σ, (n) 2 1 C. for all n 2 n/.
2. Universality of instance difficulty: We say that ALG satisfies universality of sample
difficulty w.r.t. 2 if for all 2, E 2 and all pairs of algorithms T, T.T' (T) Acc2((7) = Acc=(T') S Acc.. (T') That is, if = is harder than z' w.r.t. one algorithm in ALG, then it is harder than 2' w.r.t all algorithms in ALG, implying an inherent "difficulty ordering" of a point.
3. Entropy monotonicity:
We say that a parameterized learning algorithm T (that produces distributions over labels) satisfies entropy monotonicity if for every (z, y) E Z: 2 n' (T() S (T (x)) That is, for f drawn from T(n). the expected entropy of f(x) is non-increasing function of the resource
All three properties are incomparable with one another, in the sense that there exist learning methods that satisfy any subset of these. The main result of this section is that several natural models of learning satisfy the above monotonicity properties. These include standard Bayesian inference (with correct priors) as well as certain "toy models" that were proposed in the literature to explain some puzzling global features of deep learning. The latter are highly simplified models designed to match certain global behaviors of DNNs such as scaling laws and accuracy on the line. While these models were designed to capture global phenomena, show they also satisfy certain pointwise properties well:
Theorem C.I.
1. The "skills vs difficulty" " model of (Recht et al. 2019) satisfies the universality of instance difficulty and accuracy monotonicity properties.
2. The "manifold partition" model of (Sharma & Kaplan 2020 Bahri el al. 2027) satisfies the pointwise scaling law property.
3. Any general Bayesian inference model satisfies accuracy monotonicity and entropy monotonicity. Specific models, such as Bayesian Gaussian Process with a fixed kernel, also satisfy universality of instance difficulty with respect to a fixed training set.
The skill vs. difficulties model. Recht et al. present highly simplified model for explaining distribution shift phenomena (Recht et al. 2019, Appendix B). In this model, each point = has a "difficulty level" d, E R. Each classifier f has an accuracy function, A₁ R - [0. 1] which is a monotonically non non-increasing function mapping the difficulty (of some point 2) the probability that the classifier is successful (on 2). Note that A only depends on the "skill" of f so if f and g have the same skill the accuracy function will be the same. Now, for any two points 2, 2', we have that for every f output by some procedure T, Ay(ds) 2 Ay(d)) and only if d, < dgr Then, by definition, every model of this type satisfies universal instance difficulty. In their paper, they specifically considered a restricted version where the accuracy function of a classifier f has the form (d) = (bss d) where & is the CDF of standard normal and s R is a parameter measuring the "skill" of a model. In such a case, the global accuracy is a monotonically increasing function of the skill (n) (since increasing skill improves accuracy on every point and versa), and hence
16

---

Published as a conference paper at ICLR 2023
for every collection (()), the skill will be an non-decreasing function of n, meaning that it satisfies accuracy monotonicity as well. In other words,
n 2 n' = sy(n) 2 sy(n') = AceT, = (s)
((sy(n')-d, dz) 2 ACoT,
The partitioned manifold model. 2020). For completeness, we sketch their argument here, simply observing that the existing proof continues to apply in the pointwise setting.
This proof follows directly from the proof of (Sharma & Kaplan
(Sharma & Kaplan 2020) and (Bahri et al. 2021) propose tractable theoretical models to explain the ubiquity of scaling laws. In the notation of this paper, this is the observation that for many natural data distribution D and learning methods T, the global accuracy Accp, (n) scales as 1 - C-n for some exponent a that depends on the data distribution rather than particular features of the learning methods. Specifically, (Sharma & Kaplan (020) present simple toy model, in which the concept learned is some Lipschitz function f 0, I R. and they assume that ((n) corresponds a piecewise linear approximation on n = Cd cubes of side length 1/C. They prove that such models satisfy a global scaling law with regression error scaling as n -1/d. However, because of the symmetry between points in this model, their proof (as well as the proofs in (Bahri et al., 202I) implies also the stronger notion of a pointwise scaling law.
Bayesian inference model. In a general Bayesian inference model where we are performing inference with respect to the true distribution, Pr. For any fixed instance x. the label is random variable Y and we have a sequence of correlated random variables Z1, Zz, Za..... For every n E N, the Thth posterior distribution pa. is a random element in A()) obtained by sampling 21,....3 Fn and letting Pr. (y] .1,-.., Zn) ) be the distribution of Y Z1 Z1,... Zn Zn. The prediction algorithm after observing 21...., 2n is then arbitrarily choosing from the set arg maXyey Pn+1. Since different inference methods could correspond to completely different random variables, in general such models do not satisfy universal instance difficulty. However, they do satisfy entropy monotonicity and accuracy monotonicity. This is shown by the following lemma:
Lemma C.2 (Entropy and accuracy monotonicity of Bayesian inference). Let Y, Z1. Zz... be defined as above, and consider the process of sampling z = (21, (21,22,..> .... Then for every n, defining the posterior distribution Pn as above, EH (p.) EH (Pn+1) and E Il S E || llooo where the expectations are over the sampling of z and for q E ( I|al maxgey (1)-
Lemma C.2 clearly implies the Bayesian inference satisfies entropy monotonicity. The reason it also implies accuracy monotonicity is the following: If Ip_]]se = a and there are A labels ,,,..., for which po. (y:) a, then given this posterior Pa. WLOG we will predict that the label is y with probability 1/k. But since we are in the Bayesian setting, we assume that the posterior correctly models the world, that for each i [l....,k] k} the probability the true label was fact y a. and hence the probability for correct prediction is ΣiE( Pr[y = yi and Y = yil = Ek 1 =a= |pp.l.z0
Proof of Lemma write Pa- E ESSpp((Z.+z) 0zp= where p= obtained by conditioning pr, Zu+1 z and 2| 21...., Zn, 2n]- Hence Pn+1 = Ps with probability ag. But now the result follows from the concavity of entropy and convexity of the infinity norm: EH (p.) >
When Pa+1 is obtained by conditioning Px. on the value 2 of Zn+1, then we
0g Pr|Zn+1
Σ (p.) = (px+1) and El[pa]|> E az || = EBlpa |loc
17

---

Published as a conference paper at ICLR 2023
D EXTRA FIGURES
Label MONARCH BUTTERFLY
Label
Htage 2231 Sontarnii
Irouge ROCS Srote
1038
24
Imageled
Imagohat
Inagelett Accuracy
Figure 8: Example plots softmax profiles obtained from ResNet-50 training on ImageNet, display- ing variety of interesting behaviors.
10
10
O
0.8
0.8
Accuracy
0.4
Point
802
cOssesMin Damsiia
CORSSFMIN
0.65
0.50
0.56
6.40
0.75
0.45
6.60 0.65 6,70
0.55
Global Accuracy
Global Accuracy
Figure 9: Global shape vs. local features. Lower accuracy models tend be more sensitive to local features than the global shape of an image. For example, in this coffee mug, lower accuracy classifiers are thrown off by the illustration images whose local texture features conflicts with the global shape. In this sense, higher-accuracy classifiers behave more closely to humans, for whom global structure dominates local one (Navon 1977).
Dalmatian dog. Similar results can be seen with other
Softmax Entropy Non-monotonicity
Soft-Accuracy Non-monatoniciit
1.0
1.0
Fraction Exceeding Score
0.8
0.8
Fraction jiceeding
0.6
0.6
0.4
0.4
0.2
0.0
10
10
10-1
10-3
Non-monotonicity Score
Non-monotonicity Score
Figure 10: Plot of CDFs of the non-monotonicity score similar to the right panel of Figure 6 for the softmax entropy (left panel) and the soft-accuracy (right panel). For both measures, non-monotonicity is sharply reduced by pre-training, just as for accuracy.
18

---

Published as a conference paper at ICLR 2023
Softmax Divergence
Softmax Cosine Distance
Resnet18
0.0041
0.072
Resnet18
0.021
Densenet121
0.0041
0.071
Densenet121
0.11
0.11
0.0055
Resnet18 pt
0.19
Densenec121-pt
0.16
0.16
Densenet121-pt
0.27
0.041
Resnet18
Densenet121
Resnet18-pp
Denseneti21-pt
Resnet18
Densenat121
Resnet18-pt
Densenet121
Figure 11: Heatmaps of profile distances similar to the left panel of for the KL-Divergence (left panel) and the Cosine Distance (right panel). For both distances from-scratch models display higher similarity to each other than to pre-trained models.
CIFAR-10.2 Pointwise Acc. Difference
0.12
Average Difference
0.11
0.09
0.08
0.07
0.3
0.4
0.5
0.6
0.7
0.8
Accuracy on CIFAR-10
Figure 12: For each accuracy p on the x-axis plot the average pointwise absolute difference of the accuracies of a Resnet-18 18 model and a Densenet-12 121 model on the CIFAR-10.2 dataset. The values fairly small, especially for higher accuracies.
19

---

Published as a conference paper at ICLR 2023
E SAMPLES FROM CIFAR-10 AND CIFAR-IO-NEG
plne
Oet
(a)
(b)
(c)
(d)
(c)
oog
13g
ship
(f)
(g)
(h)
Figure 13: Random samples from CIFAR-1O-NER set.
(a)
(b)
(c)
(d)
(e)
62g
home
chip
trckk
(f)
(g)
(h)
Figure 14: Random samples from CIFAR-10 test set.
20


Reviews:
[{"review_id": "zAZlmP6dZKb", "reviewer": "Reviewer_JEZe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the how the accuracy of a fixed point changes when multiple models with increasing sizes are used for predictions which it defines as the learning profile of a point. They show that there can be points which are positively correlated with the average accuracy of models and there are certain points which are negatively correlated. They also observe that there are differences between pretained models and randomly initialized models. In particular, the number of points with non-monotone accuracy is much lower for pretained models  as compared to randomly initialized models.", "summary_of_the_review": "The extensive study of point wise profiles seems interesting and the observation about the difference between pretrained and randomly initialized models is interesting. ", "strength_and_weaknesses": "The idea of studying the accuracy of fixed points with respect to changing models is interesting. There exists points which have a negative correlation with average accuracy is quite interesting. The observation about the difference between pretrained models and randomly initialized models is also quite surprising and would be interesting to look into. \n\nHowever, I feel that some of the previous works have not been properly cited. [1] already had this observation that a few points are misclassified by larger models but correctly classified by smaller models and hence, this observation is not really new. The authors cite this paper but it would be good to state the differences more clearly.\n\n[1] Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qE6GScL8CD1", "reviewer": "Reviewer_5UuZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a new performance measuring method that measures the performance of a collection of models when evaluated on a single input point. The authors compare the models' average performance on the test distribution and their pointwise performance on an individual point. The empirical results show that for some individual test points, the average performance has a weak and even negative correlation to the performance on the individual point. Then the authors construct a dataset called CIFAR-10-NEG out of CIFAR-10 so that accuracy on CIFAR-10-NEG is negatively correlated with the accuracy on CIFAR-10 test.", "summary_of_the_review": "I like this paper in general, due to the novelty of the topic and the new phenomenon. However, I do have some concerns about whether such a negative correlation phenomenon is common in general or only a corner case for specific models and datasets.", "strength_and_weaknesses": "Strength: the paper is novel and the demonstration is pretty clear. The authors provide a lot of figures that are very helpful to understand the paper.\n\nWeakness: \n\n1. The explanation of why such a negative correlation phenomenon appears is not provided. I suspect that the existence of this phenomenon may depend on the specific models and dataset. A possible explanation might be that the models do not have enough many parameters and thus do not have the capability to fit all samples in the dataset. In other words, the models have to sacrifice the performance on a small portion of data to get a better overall average performance. If so, then I am afraid this phenomenon is less interesting.\n\n2. It may be hard to identify the different types of points mentioned on page 6 beforehand since multiple models need to be trained and evaluated on each sample. This process may take a very long time, yet is not helpful to improve the average performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Z7ycoU8sGs4", "reviewer": "Reviewer_NDc7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is an empirical paper that proposes a new way to examine the predictions of machine learning models. In particular, it proposes to compare how well a variety of models predict a single, fixed test point. The authors note that models have a few surprising features with respect to this measure: in commonly used datasets, a large number of datapoints can have prediction quality negatively correlated with overall accuracy on the model's training set, and that pretraining often removes this negative correlation. The authors also note that various simplified theoretical models of machine learning training do not capture this behavior.", "summary_of_the_review": "Overall, I think this paper demonstrates a few interesting empirical results, and would be a good addition to the community.", "strength_and_weaknesses": "**Edit 10-29-2022 after reading other reviews.** I think reviewer diDC brings up a really good point that I think the authors should really address -- what about the behavior observed in this paper is dissimilar from more basic situations? E.g. take the fitting of the MLE of the mean of a Gaussian with gradient descent (I know you can just take the mean and skip gradient descent, but the point is to get a series of better and better \"models\"). If you randomly initialize at one of the datapoints, it a reasonable fraction of datapoints will be negatively correlated with overall \"model\" performance (basically, everything that lies further away from the mean than the initialization). Is there more complex behavior going on that this toy example can't capture? If so, do the authors know if this more complex behavior will / will not show up in basic regression models (linear / logistic regression)?\n\n\n-------------------\n\nThis paper is a little different from many machine learning papers in that it does not prove any new theory, it does not provide state of the art empirical results, and it does not answer any open questions in the field. What it does provide is some new interesting / mysterious empirical observations about the behavior of machine learning models that (to my knowledge) were not previously well known; I especially appreciated the fact that the authors have found some new ways in which toy theoretical models for explaining machine learning do not match actual machine learning models. From that perspective, I think this is a solid paper that could lead to thought provoking discussions at the conference. \n\nI had a few scattered issues with the current draft of the paper:\n\n- The paper only constructs families of models with increasing training accuracy by using more training time. The authors argue that other ways of constructing families of models (e.g. more complex models / more training data) might be expected to perform similarly. But there aren't experiments in the paper to demonstrate this, and I think this limitation should be emphasized more in the paper.\n- I think the definitions of easy / hard / compatible / non-monotone points should go before the discussion of those different types of points. I also felt the enumerated discussion of these points was a little unnecessary, and could be cut to move more important things into the main text.\n- Model similarity uses $L_1$ distance, whereas the definition of easy / hard / compatible uses $L_2$. Why the difference?\n- Right under Fig 7, there is a reference to Figure 7(b). Should this be 7(c)?\n- Why was 0.1 chosen as a threshold for nmono$(P_z)$?\n\nA few comments on Appendix C\n- I thought the results in here were pretty interesting. It might be worth adding a paragraph or two in the main text to summarize these results  (currently references to them are a little scattered and unclear).\n- I don't think $H$ was ever defined as the entropy.\n- \"... obtained by sampling $z_1, \\dots, z_n$ and letting $p_n(y \\mid z_1, \\dots, z_n)...$ I think you mean to condition on $x$ as well here?\n- Is Lemma C.2 meant to hold for all $x$? This should be stated explicitly.\n- \"we assume that the posterior correctly models the world, that is...\" This is not always true (I would say it is almost never true). It could be true asymptotically as $n \\to \\infty$ if the model's likelihood is correctly specified. I think this could use some more discussion.\n- \"Hence $p_{n+1} = p_z$ with probability $\\alpha_z$.\" I don't think there's really a sense in which a distribution equals another distribution with a certain probability; the distributions are equal or they are not. This sentence doesn't seem necessary for the rest of the proof in any case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "AYFVaOi8b5", "reviewer": "Reviewer_diDC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In their manuscript entitled, \"Deconstructing distributions: a pointwise framework of learning\", the authors present an approach to identifying a certain class of outliers in training sets: those points for which improvements to the global model accuracy correspond to degradation in individual accuracy. \n", "summary_of_the_review": "An extensive and capable investigation, which might be highly interesting although I reserve judgement for now!", "strength_and_weaknesses": "Perhaps the greatest strength of this paper is the comprehensiveness of its investigation in terms of considering a good selection of the relevant literature and presenting a decent slice of empirical examples and theoretical considerations.  At face value the system presented for characterising the outlying data points and/or understanding the model families considered seemed novel and interesting to me; however, on further considering I found myself wondering whether the opposite was true.  I give the following criticisms from the perspective of a statistician who works with large datasets and complex models, not as a practitioner of the particular deep learning models for (e.g.) image classification to which this manuscript is addressed. \n\nThat the 'fitting' (generally considered) of certain points in a dataset degrades as the overall performance improves is in the most general sense a trivial observation.  Even in a well-specified model there will be expected that some data points are from the tails of the distribution and as we learn to represent the distribution overall these tail points naturally become 'less expected' under the overall distribution.   More relevant is the miss-specified cases where we might think of these as contaminating data points for instance, e.g. long tailed or shot noise not expected under a white noise assumption; in the literature on 'robustness' and 'robust regression' we have many ways to classify these so-called 'leverage' points and quantify their influence on the model fit (e.g. Rousseeuw & Hubert, 2011).  Similarly, in the Bayesian literature we have many visualisations and posterior predictive checking methods to do the same and stimulate model development (e.g. Gabry et al., 2017). \n\nAdmittedly, in these examples above, we are primarily considering these points with regard to the best-performing model (e.g. in a Bayesian sense, at the empirical Bayes hyper-parameter estimate), rather than in terms of a sequence including a long tail of poorly performing models.  To this end I suppose the crux of assessing this paper must come down to convincing the reviewer/reader that there is additional value in constructing these curves of training behaviour as a function of global model accuracy rather than just looking at which points are hard to fit for the best trained models (ie., best overall accuracy).  Here I would be interested in the authors' reply and the other reviewers' thoughts. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster