A Unified Framework for Comparing Learning Algorithms
Authors: Harshay Shah
Sung Min Park
Andrew Ilyas
Aleksander Madry
Venue: Submitted to ICLR 2023 (2023)
Keywords: algorithm comparison
model comparison
data-centric
influence
datamodels
data augmentation
pretraining
learning rate

Abstract: We propose a framework for {\em (learning) algorithm comparisons}, wherein the goal is to find similarities and differences between models trained with two different learning algorithms. We begin by formalizing the goal of algorithm comparison as finding {\em distinguishing feature transformations}, input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present a two-stage method for algorithm comparisons based on comparing how models use the training data, leveraging the recently proposed datamodel representations [Ilyas et al., 2022]. We demonstrate our framework through three case studies that compare models trained with/without standard data augmentation, with/without pre-training, and with different optimizer hyperparameters. 

Under review as a conference paper at ICLR 2023
A UNIFIED FRAMEWORK FOR COMPARING LEARNING ALGORITHMS
Anonymous authors Paper under double-bling review
ABSTRACT
We propose a framework for (learning) algorithm comparisons, wherein the goal is find similarities and differences between models trained with two different learning algorithms. We begin by formalizing the goal of algorithm comparison as finding distinguishing feature transformations, input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present a two-stage method for algorithm comparisons based on compar- ing how models use the training data, leveraging the recently proposed datamodel representations (Ilyas et al., 2022). We demonstrate our framework through three case studies that compare models trained with/without standard data augmenta- tion, with hith/without pre-training and with different optimizer hyperparameters.
INTRODUCTION
Building a machine learning model involves a series of design choices. Indeed, even after choosing dataset, for example, one must decide on model architecture, optimization method, and data augmentation pipeline. These design choices together define a learning algorithm, a function mapping training datasets machine learning models.
Even if they do not affect accuracy, design choices determine the biases of the resulting models. For example, Hermann et al. (2020) find significant variation in shape bias (Geirhos al., 2019) across group of ImageNet models that vary in accuracy by less than 1%. In order to understand the impact of design choices, we thus need to be able to differentiate learning algorithms in more fine-grained way than accuracy alone.
Motivated by this observation, we develop a unified framework for comparing learning algorithms. Our proposed framework comprises (a) precise, quantitative definition of learning algorithm com- parison; and (b) a concrete methodology for comparing any two algorithms. For (a), we frame the algorithm comparison problem as one of finding input transformations that distinguish the two algo- rithms. This goal is different and more general than quantifying model similarity (Ding al., 2021; Bansal et al., 2021; Morcos et al., 201 8a) or testing specific biases (Hermann et al., 2020). For (b), we propose two-stage method for comparing algorithms in terms of how they use the training data.
In the first stage of this method, we leverage datamodel representations (Ilyas et al., 2022) to find weighted combinations of training examples (which we call training directions) that have disparate impact on test-time behavior of models across learning algorithms. In the second stage, we filter the subpopulation of test examples that are most influenced by each identified training direction, then manually inspect them to infer shared feature (e.g., we might notice that all of the test images contain a spider web). We then tie this intuition back to our quantitative definition by designing distinguishing feature transformation based on the shared feature (e.g., overlaying spider web pattern to the background of an image).
We illustrate the utility of our framework through three case studies (Section 3), motivated by typical choices one needs to make within machine learning pipelines, namely:
Data augmentation: We compare classifiers trained with and without data augmentation on the LIVING17 (Santurkar et al., 2021) dataset. We show that models trained with data augmenta- tion, while having higher overall accuracy, are more prone to picking up specific instances of co-occurrence bias and texture bias compared to models trained without data augmentation. For

---

Under review as a conference paper at ICLR 2023
(A) Data augmentation
Pre-training
(C) Optimizer hyperparameterss
Living17
Spider web
Waterbirds
Human face
CIFAR-10
Rectangle patch
With standard sugmentation
With ImageNat pre-trainigg P("landbrrd"] P("Indbidd) 85%
Traning with low SGD notee
P["spider"] P!"spider"| 52%
P("buck") P("truck"] 61%
Withou stancied dats augmentation
Without Imagahiee pre-training
Trining with high 5GD ho
P!"spider"] x 3% + Pl "["spddrr"] x 6%
Pi"lanbbird] s 42% "!"anddbidd] is 42% Pl"truck") >: 0%
P/ Pl"truck") > 25%
Figure 1: A visual summary of our case studies. We use our method to study the differences between training with and without standard data augmentation, with and without ImageNet pre-training, and with different choices of SGD optimizer hyperparameters. In all three cases, our framework allows us pinpoint concrete ways in which the two algorithms being compared differ.
instance, adding spider web (Figure IA}-which co-occurs with spiders-to random images in- creases the "spider" confidence augmentation- the confidence of non-augmented models by less than 1 1.
models by 15% on average, but increases
ImageNet pre-training: We compare classifiers first pre-trained on ImageNet (Deng et al., 2009; Russakovsky et al., 2015) then fine-tuned on WATERBIRDS (Sagawa et al., 2020) with classifiers trained from scratch on WATERBIRDS. We demonstrate that pre-training can either suppress or amplify specific spurious correlations. As an example of the former, adding a yellow patch to random images increases confidence in the "landbird" label of models trained from scratch by 12%, but actually decreases the confidence of pre-trained models by 4%. As an example of the latter, adding human face (Figure IB) to the background increases the "landbird" confidence of pre-trained models by 4%, but decreases the confidence of models trained from scratch by 1%.
Optimizer hyperparameters: Finally, compare classifiers trained CIFAR- 10 (Krizhevsky, 2009) using stochastic gradient descent (SGD) with different choices of learning rates and hatch sizes. Our analysis pinpoints subtle differences in model behavior induced by small changes to these hyperparameters. For example, adding a small pattern that resembles windows (Figure 1C) random images increases the "truck" confidence by 7% on average for models trained with smaller learning rate, but increases the confidence by only 2% for models trained with a larger learning rate.
Across all three case studies, our framework surfaces fine-grained differences between models trained with different learning algorithms, enabling better understand the role of the design choices that make up a learning algorithm.
COMPARING LEARNING A ALGORITHMS
In this section, we describe our (learning) algorithm comparison framework. In Section 2.1, we formalize algorithm comparison as the task of identifying distinguishing transformations. These are functions that-when applied test exanples-significane significantly and consistently change the predic- tions of one model class but not the other. In Section 2.2, we describe our method for identifying distinguishing feature transformations by comparing how each model class uses the training data.
2.1 FORMALIZING ALGORITHM COMPARISONS VIA DISTINGCISHING TRANSFORMATIONS
The goal of algorithm comparison is understand the ways in which two learning algorithms (trained on the same data distribution) differ the models they yield. More specifically, we are interested in comparing the model classes induced by the two learning algorithms:
Definition 1 (Induced model class), Given input space X, label space Y, and model space M C X +, a learning algorithm A: (X y)) - M is (potentially random) function mapping set of happtt-labl pairs to a model. Fixing a data distribution D, the model class induced by algorithm A is the distribution over M that results from applying A to randomly sampled datasets from D.
2

---

Under review as a conference paper at ICLR 2023
The perspective adopt here is that model classes differ insofar they different features to make predictions. We make this notion precise by defining functions that we call distinguishing transformations:
Definition 2 (Distinguishing feature transformation). Let AI, A2 denote learning algorithms, S dataset of input-label pairs, and L a loss function (e.g. correct-class margin). Suppose M1 and M2 are models trained on dataset D using algorithms A1 and Ag respectively. Then, (e, distinguishing feature transformation of M1 with respect to M2 is a function F X + such that for some label ye EV.
Countrfactual effect of F on M1
Counterfaciual effect of F on M2
(F Mo) - L Ux) 26 8
and
EL (e) L2(2, yo)l Se
where Li(2, y) ((M (z), y). and the expectations above are taken over both inputs x and ran- domness in the learning algorithm.
Intuitively, a distinguishing feature transformation is just a function that, when applicd to test data points, significantly changes the predictions of one model class but not the other-in a consistent way. Definition 2 also immediately suggests a way to evaluate the effectiveness of distinguish- ing feature transformation. That is, given hypothesis about how two algorithms differ (e.g., that models trained with A, are more sensitive to texture than those trained with A2), one can design a corresponding transformation F (e.g., applying style transfer, as Geirhos al. (2019)), and directly measure its relative effect on the two model classes.
Informative distinguishing feature transformations. Not every distinguishing transformation F sheds the same amount of light model behavior. For example, given any two non-identical learn- ing algorithms, we could craft a transformation F that imperceptibly modifies its input to satisfy Definition 2 (e.g., by using adversarial examples). Alternatively, one could craft an F that arbi- trarily transforms its inputs into pathological out-of-distribution examples on which the two model classes disagree. While these transformations satisfy Definition 2, they yield benefit in terms of qualitatively understanding the differences in salient features used by the model classes. More con- cretely, an informative distinguishing feature transformation must (a) capture feature that naturally arises in the data distribution and (b) be semantically meaningful.
2.2 IDENTIFYING DISTINGUISHING FEATURE TRANSFORMATIONS
We now describe our two-stage method for comparing learning algorithms based on how they use the training data. Recall that our goal is to identify informative distinguishing transformations -that is, those that (a) capture a feature that arises naturally in the data distribution and (b) are semantically meaningful. Rather than search for distinguishing feature transformations directly, instead first identify training directions, or weighted combinations of training examples, that impact test perfor- mance of models trained with one learning algorithm but not the other. Then, the-loop approach to extract semantically meaningful features and corresponding transformations from these directions.
human-in-
Stage I: An algorithm for finding distinguishing training directions. First, we find weighted combinations of training examples that influence the predictions models trained with one learning algorithm but not the other. Our algorithm leverages datumodel representations (Ilyas et al., 2022) and comprises the following three steps:
Compute datamodels for each algorithm. Our point of start is computing datamodel repre- sentation for each example in the test set T. Given training set S and learning algorithm A, datamodel representation for test example x; is vector 0g E IIS), where O,, measures the extent to which models trained with A depend' on the j-th training example to correctly classify example Tz. In other words, 0, is a training direction that strongly influences the prediction for example
'or readers familiar with influence functions (Koh Liang, 2017: Hampel et al., 2011), an intuitive (but not quite accurate) way to interpret datamodel weight 6Gg is as the influence of the j-th training example on test example x;, In Appendix A, we discuss additional properties of datamodel representations that make them particularly well-suited for model comparison.
3

---

Under review as a conference paper at ICLR 2023
Ty- We compute two sets of datamodels g(1) and ((2) corresponding model classes induced by learning algorithms A1 and Ag respectively.
2. Compute residual datamodels. Next, we compute residual datamodel for each test example Ty, which is the projection of the datamodel representation (!4) onto the null space of datamodel representation 0
((12)) = - ) 8.
where 8, / / denotes the normalized version of datamodel 8, Intuitively, the residual datamodels of algorithm A1 with respect to A2 correspond to the training directions that influence A₁ after "projecting away" the component that also influences A2.
3. Run principal component analysis. Finally, the highest-variance directions the space of residual datamodels. That is, we run
principal component analysis (PCA) find
& E-PCA((O . 2
find the top principal components of the residual datamodels. Intuitively (deferring formal analysis to Appendix A), we expect the returned principal components to be the most distinguish- ing training directions across the
We illustrate our algorithm visually in the top half of Figure 2.
Stage II: Human-in- -n-man-n-tthe-lop analysis. With these distinguishing directions in hand, we use human-in-the-loow analysis to identify informative distinguishing features transformations in three steps. First, given a principal component, we inspect the test examples whose residual datamodels most aligned with that component. We view these examples representing the subpopulation that depends most heavily on the training direction. We then use visual inspection (see, e.g., Figure 2C) and if needed, additional analysis², to infer distinguishing feature shared by examples in the surfaced subpopulation. Finally, we design a distinguishing feature transformation (as in Figure 2D) counterfactually verify the effect of the inferred feature on model behavior.
(A) Compute datamodel representations each algorithm
(B) Analyze residual datamodels
Train set space
R
(1)
0()
(())))
Stage
coordinate dependence on training example
(2)
(
Example
All N training examples
Run PCA on (6
m122
(C) Each principal component surfaces subpopulation
(D) Infer and test distinguishing feature
Stage II
do(orange
Compare Pr [y plane do(orange tint)] with models trained using algorithms af, ot
Examples this subpopulation share orange background!
Figure 2: A visual summary of two-stage approach algorithm comparison. In the first stage (top row) we use examples' datamodel representations (Ilyas et al., 2022) to find so-called dis- tinguishing training
sirecttonnsweiihheee combinations of training examples that impact the two algorithms disparately across the test set. In the second stage (bottom row) we surface subpopu- lations that rely on the identified directions, and use a comma--nn-hh--lee to go from the identified distinguishing training direction to testable feature transformation.
Visual inspection may he insufficient to identify single distinguishing feature. In such cases, resort to additional numan-in-the-lone analysis (see Appendix D) identify the distinguishing feature.
4

---

Under review as a conference paper at ICLR 2023
Explained Vartanoo under Algaribon Az (85)
of 0-133
02
PC's (22))
(Withuut Augrnmmbaii rngrommationn (ifioo))
Line
A
0.1
B
Explained Variance under Algorithun di (5)
(Wit Augmentation)
Figure 3: Comparing LIVING17 models trained with and without data augmentation. (Left) Each green (resp., red) point is a training direction (i.c., vector v E ØISl representing a weighted combination of training examples) that distinguishes A₁ from A2 (resp., A2 from Ai) as identified by the first stage of our framework. The and y coordinates of each point represent the importance of the training direction models trained with A1 and As respectively. (Right) Test examples most impacted by the distinguishing training directions annotated A and B. A seems to correspond to spiders on spider webs, and B to salamanders with yellow polka dots.
APPLYING THE ALGORITHM COMPARISON FRAMEWORK
We demonstrate our comparison framework through variety of case studies. Specifically, we consider three key aspects of the standard training pipeline-data augmentation, pre-training, and aptimization-and characterize their effect in terms of the features that they amplify or suppress. In each case study, we follow the same procedure: (i) use the algorithm from Stage above (Section 2) to identify examples training directions; (ii) inspect the test images that depend most heavily these training directions to come up with a candidate distinguishing transformation; and (iii) apply these candidate transformation test examples and compare its effect models trained with the two learning algorithms.
3.1
CASE STUDY: DATA AUGMENTATION
Data augmentation key component of the standard computer vision training pipeline, as its application can significantly improve model performance. However, the effect of using data aug- mentation on models' learned features remains elusive. In this case study, we study the effect of data augmentation in the context of classifiers trained on the ImageNet-derived LIVING17 dataset (Santurkar et al., 2021). Specifically, we compare two classes of ResNet-18 models trained on this dataset with the exact same settings modulo the use of data augmentation. That is, we consider the following two learning algorithms:
Algorithm A₁: Training with standard augmentation, i.e., horizontal flip and resized random crop with torchvis on default parameters. Resulting models attain 89% average test accuracy (where the average is taken over randomness in training).
Algorithm A2:
Training without data augmentation. Models attain 81 % average accuracy.
Appendix C.I provides further experimental details.
Identifying distinguishing features. We compare algorithms A₁ and Ag using our method from Section 2. The first stage of this method finds set of distinguishing training directions v, RISI, As we will show (in Section A), for given training direction the fraction of variance that v explains* the datamodel representations {oL") } captures the importance the corresponding
'The fraction of explained variance given vector E Idd in a set vectors (0, E Id) } the empirical variance of 8, divided by the total amount of variance in (0t) (i.e., trace(Cov(O.))). In other words, this measures what fraction of the total variation in (0,) is along the direction
5

---

Under review as a conference paper at ICLR 2023
Data Augmentation
Dats Augmentation
Perturh
Pemurr
Purcant in Prialamande))
20
Percent Incrense in Priepden)
25
Pecturb
16
Peruat
0.
20
Perturh
Perturs
0.6 Perturbation Intensity
0.4
022
0.3
0.4
Perturbation Intensity
(a) "Spider web" feature
(b) "Polka dots" feature
Figure 4: Effect of data augmentation on LIVING 17 models. Standard data augmentation ampli- fies specific instances of co-occurrence (panel (a)) and texture (panel (b)) biases. The left side of each panel illustrates the distinguishing feature transformation at three different levels of intensity 5. On the right, we plot the average treatment effect of the transformation on the predicted confidence of models trained with and without data augmentation, for varying 5. We find that aat moderate intensity 8 a-ddingg spider web pattern images makes models trained with (without) data aug- mentation, average, 13% (1%) more confident in predicting the class "spider," while overlaying a polka dot pattern makes them 14% (2%) more confident the class "salamander." In both cases, increasing the intensity 8 widens the gap between the two model classes.
combination of training examples to model predictions for algorithm A,, Thus, we would hope for training directions that distinguish A₁ from A2 to explain a high (resp., low) amount of the variance in datamodel representations of algorithm A1 (resp., A2).
Figure displays the identified distinguishing training directions surfaced in the first stage. On the left, we can see that the training directions distinguishing A from A2 (in green) indeed explain a significant amount variance in the datamodels of Ai but not in those of A2. Visualizing the subpopulations corresponding to two of the distinguishing directions (Figure 3 right) suggests the following distinguishing features:
Spider web: Direction A surfaces a subpopulation of test images that contain spiders. In stark contrast to random images of spiders in LIVING17 data, all the images in the surfaced subpop- ulation contain white spider web in the background. This leads us hypothesize that models trained with standard data augmentation-mmorss moresso than those trained without it it-use spider webs predict the class "spider". We test this hypothesis using a feature transformation that overlays spider web pattern onto entire image (see Figure 4a).
Polka dots: Direction B surfaces a subpopulation of test images that contain salamanders. Again, comparing these images to random salamander images in LIVING17 test data reveals a shared property of the surfaced subpopulation, namely, the presence of yellow-blac polka dots. This sug- gests that this texture is a distinguishing feature-modek trained with data augmentation strongly rely on the polka dot texture to predict the class "salamander." To test this hypothesis, we design a feature transformation that adds yellow-black polka dots to the entire image (see Figure 4b).
Findings. In Figure 4, compare the effect of the above feature transformations mod- els trained with and without data augmentation. The results confirm our hypotheses put forth above. In particular, overlaying spider web pattern with varying (30%/40%/50%) opacity increases P("spider") (i.e., models' average softmax confidence in the spider label) predicted in models trained with data augmentation (11%//3%/115% increase) signifantly more than in those without (2%/1%/19). Similarly, overlaying the yellow polka dot texture with varying opacity (20%/30%/40%%) increases P("salamander") far more when we use data augmentation (0%6/14%/119 %) than when we do not (1%/2%/2%) Furthermore, increasing the transformations' intensity consistently widens the gap between the predictions of the two model classes. Overall,

---

Under review as a conference paper at ICLR 2023
Esplained Vartance urater Algaritom Ag (%)
PCs of ol
PCs of (P Line xey
(Withont Pie Pietraiiin training
A
B
Explained under Algoritlum A1 (7)
(With ImageNet P'r training)
Figure 5: Comparing Waterbirds models trained with and without ImageNet pre-training. An analogous figure to Figure 3 for our second case study, see Figure 3 for a description. In this case, direction A seems to correspond to yellow birds and direction B to human faces in the background.
these differences verify that the feature transformations constructed are indeed able to distin- guish the two learning algorithms as according to Definition 2.
Connection to previous work. Our case study demonstrates how training with standard data aug- mentation on LIVING17 data can amplify specific instances of co-occurrence bias (spider webs) and texture bias (polka dots). These findings are consistent with prior works that show that data augmen- tation can introduce biases (e.g., (Hermann et al., 2020)). Also, the distinguishing features that we find are specific to certain classes- may explain why data augmentation can have disparate im- pacts performance across different classes (Balestriero et al., 2022). Finally, our findings, which demonstrate how data augmentation alters relative importance of specific features, corroborate the view of data augmentation as feature manipulation (Shen et al., 2022).
3.2 CASE STUDY: PRE-TRAINING
Pre-traing Pre-training models on large datasets is standard transfer learning approach to improve perfor- mance on downstream tasks where training data is scarce. Our case study focuses the effect of Ima- geNet pre-training (Deng et al., 2009; Korbblith et al., 2019b) in the context of classifiers trained on the WATERBIRDS dataset (Sagawa et al., 2020). The task classify images of birds as "waterbird" or "landbirdd" thesee labels spuriously correlate with "land" and "water" backgrounds respectively. Pre-training significantly improves the "worst group" accuracy on images where the background conflicts with the bird-but how does it impact the fine-grained features learned by the models? To study this, we compare two classes of ResNet-50 models trained with the exact same settings modulo the use of ImageNet pre- pre-training. That is, consider the following two learning algorithms:
Algorithm A₁: Pre- fre-training on ImageNet, followed by full-network finetuning on Waterbirds. Resulting models attain 89.1% average accuracy and 63.6% worst-group accuracy on the test set.
Algorithm A2: Training from scratch Waterbirds data. Corresponding models attain 63.9% average accuracy and 5.7% worst-group accuracy on the test set.
Identifying distinguishing features. Applying our framework analogously as done in Section 3.1, we identify two candidate distinguishing features:
Yellow color: Direction A surfaces a subpopulation of test images that contain yellow birds be- longing class "landbird." This leads hypothesize that models trained from scratch (i.e., without ImageNet pre-training) spuriously rely on the color yellow to predict the class "landbird," whereas ImageNett-retrainee models do not. Additional analysis (Appendix D) supports this hy- pothesis, as training images that contain other yellow objects strongly influence the predictions of models trained from scratch on this subpopulation. To test this hypothesis, we design a feature transformation that adds a yellow square patch to images (see Figure 6a).
7

---

Under review as a conference paper at ICLR 2023
ImageNiet Prestraining
ImageNer Pretraining
Willout
Add Patch Sieo 20px
Add Patch
Poreent Inceaaee in Prlandhted)
Size SOOK
Percet hecrronn in Pw Pellandird)
Add Fatch
Add Poldh
Siee 3lpx
Sine 100px
Add Fatch
Add Poldh
14
Size 90pt
Sin 120ps
Patch Stre
Patch Stoe
(a) "Yellow" feature
(b) "Human face" feature
Figure 6: Effect of ImageNet pre-training WATERBIRDS classification. Analogously to Fig- ure 4, we use our framework to identify spurious correlations that are either suppressed or am- plified by ImageNet pre-training. (Left) Adding yellow patch to images makes models trained without (with) pre-training, on average, 9% more (2% less) confident in predicting the label "land- bird." (Right) Adding human faces image backgrounds makes models trained with (without) pre-training, on average, 3% (0%) more confident in predicting "landbird." In both cases, increasing the transformation intensity widens the gap in treatment effect between the two model classes.
Human face: Direction B surfaces subpopulation of "landbird" that have human faces in the background. This suggests that ImageNet pre-training introduces a spurious dependence on hu- man faces to predict the label "landbird." Additional analysis supports this hypothesis, as training images containing face(s) strongly influence the predictions of ImageNet-pretrained models on this subpopulation (see Appendix D). To test this hypothesis, we design transformation that inserts patches of human faces in WATERBIRDS image backgrounds (see Figure 6b).
Findings. In Figure 6, we compare the effect of the above feature transformations on models trained with and without ImageNet pre-training The results confirm both of our hypotheses. Adding yellow square patch with varying size (20/30/40 px) to test images increased P P("landbird") by 4%/9%/122 12% for models trained from scratch but decreased P("landbird") for models pre-trained on ImageNet. Similarly, adding human face patch* to image backgrounds increased P( P("landbird") by 2%/3%/4% for pre-trained models, but did not significantly affect models trained from scratch. Once again, increasing the intensity (i.e., patch size) of these feature transformations further widens the gap in sensitivity between the two model classes.
Connections to prior work. This case study pinpoints how pre-training can alter the importance of different spurious correlations. In particular, our results show that ImageNet pre-training reduces dependence on some spurious correlations (e.g.. yellow color - landbird) but also introduces new ones (e.g., human face + landbird). Our findings thus shed light on two seemingly contradictory phenomena: pre-training can simultaneously improve robustness spurious features (Ghosal al., 2022; Tu et al., 2020) in target data well as transfer new spurious correlations (Salman et al., 2022; Neyshabur al., 2020) from the pre-training dataset.
3.3
CASE STUDY: SGD HYPERPARAMETERS
The choices of optimizer and corresponding hyperparameters can affect both the trainability and the generalization of resulting models (Hoffer et al., 2017). In this case study, we focus on stochastic gradient descent (SGD), and its hyperparameters--eeee Icarning rate and batch size- that control the effective scale of the noise in SGD. We defer our findings to Appendix B due to space constraints.
*We collect a bank of human face patches using ImageNet validation examples and their corresponding face annotation (Yang al., 2022); Appendix C.4 for more information.
8

---

Under review as a conference paper at ICLR 2023
4
RELATED WORK
We now compare and contrast our approach to algorithm comparison with approaches to the related problem of model comparison, where one tries characterize the difference between two (usually fixed) machine learning models. A long line of work has sought to design methods for characterizing the differences between models:
Representation-batn based comparison. A popular approach (particularly in the context of deep learn- ing) to compare models using their internal representations. Since the coordinates of these repre- sentations do not have a consistent interpretation, representation-base studies the degree to which different models' representations can be aligned. Methods based this ap- proach include canonical correlation analysis (CCA) and variants (Raghu et al., 2017; Morcos et al., 2018a; Cui et al., 2022), centered kernel alignment (CKA) (Kornblith et al., 2019a), graph-based methods (Li et al., 2015; Chen et al., 2021), and model stitching (Csiszarik et al., 2021; Bansal et al., 2021). Prior works have used these methods to compare wide and deep neural networks (Nguyen et al., 2021); vision transformers and convolutional networks (Raghu et al., 2021); pre-trained and trained-from-serracch models (Neyshabur et al., 2020); and different language models (Wu et al., 2020). Though they often useful, prior work shows that representation-basee similarity mea- sures are not always reliable for testing functional differences in models (Ding et al., 2021). Our approach to algorithm comparison differs from these methods in both objective and implementation:
model comparison typically
Learning algorithms,, not fixed models: Rather than focusing on single fixed model, our objective here is to compare the class of models that result from a given learning algorithm. In particular, we aim to find only differences that arise from algorithmic design choices, and not those that arise from the (sometimes significant) variability in training across random seeds (Zhong et al., 2021).
Feature-base based, not similarity-based based: Methods such as CCA and CKA focus on outputting a single score that reflects the overall similarity between two models. On the other hand, the goal of our framework is to find fine- meeggrieedd differences in model behavior. Still, in Appendix F.1 we show that we can also use our method for more global comparisons, for instance by computing the average cosine similarity of the datamodel vectors.
Model-agnostic: Our framework agnostic to type of model used and thus allows one to easily compare models across learning algorithms-on our method extends even to lcarning algorithms that do not have explicit representations (c.g., decision trees and kernel methods).
Example-level comparisons. An alternative method for comparing models is to compare their pre- dictions directly. For example, Zhong et al. (2021) compare predictions of small and large language models (on a per-example level) to find that larger models are not uniformly better across exam- ple. Similarly, Mania et al. 22 study the agreement between models, i.e., how often they output the same prediction a per-example level. In another vein, Meding et al. (2022) show that after removing impossible or trivial examples from test sets, different models exhibit more variations in their predictions. Our framework also studies instance-level predictions, but ultimately connects the results back to human- cimaaninterpreetbble distinguishing features.
Comparing feature attributions. Finally, another line of work compares models in terms of how they use features at test time. In the presence of known set of features, one can compute fea- ture importances and compare them across models (Wang et al., 2022). In cases where we do not have access to high-level features, we can use instance-level explanation methods such as saliency maps to highlight different parts of the input, but these methods generally do not help at distin- guishing models (Denain & Steinhardt, 2022). Furthermore, multiple evaluation metrics (Adebayo et al., 2018; Hooker et al., 2018; Shah et al., 1) indicate that common instance-specific feature attribution methods can fail at accurately highlighting features learned by the model.
5 CONCLUSION
We introduce a unified framework for fine-grained comparisons of any two learning algorithms. Specifically, our framework compares models trained using two different algorithms in terms of how the models rely training data to make predictions. Through three studies, showcase the utility of our framework in pinpointing how three aspects of the standard training pipeline- -dtin augmentation, pre-training, optimization- shape model behavior.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In arXiv preprint vrXiv:2106.12223 2022.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Neural Information Processing Systems (NeurIPS), 2018.
Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmen- tation are class dependent. arXiv preprint arXiv:2204.03632. 2022.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In Neural Information Processing Systems (NeurIPS), 2021.
Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. Advances in Neural Information Processing Systems, 14:2556-2569,
2021.
Zuohui Chen, Yao Lu, JinXuan Hu, Wen Yang. Qi Xuan, Zhen Wang, and Ziaoniu Yang. Revisit similarity of neural network representations from graph perspective. arXiv preprint arXiv:2111.11165, 2021.
Adrian Csiszarik, Peter Korosi-Szabo, Akos Matszangosz, Gergely Papp, and Daniel Varga. Simi- larity and matching of neural network representations. Neural Information Processing Systems (NeurIPS), 2021.
Tianyu Cui, Yogesh Kumar, Pekka Marttinen, and Samuel Kaski. Deconfounded representation similarity for comparison of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Neural Information Processing Systems (NeurIPS), 2017.
Jean-Stanislas Denain and Jacob Steinhardt. Auditing visualizations: Transparency methods strug-
gle detect anomalous behavior. arXiv preprint arXiv:2206. 13498, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Frances Ding, -tan-Ssassslas Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Sabri Eyuboglu. Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross- modal embeddings. arXiv preprint arXiv:2203. 14960, 2022.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 2881-2899 2020.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. arXiv preprint 190000990,,0329, 2019.
Soumya Suvra Ghosal, Yifei Ming, and Yixuan Li. Are vision transformers robust to spurious correlations? arXiv preprint arXiv:2203.09/25, 2022.
Frank R Hampel, Elvezio M Ronchetti, Peter J Roussceuw, and Werner A Stahel. Robust statisties: the approach based on influence functions, volume 196. John Wiley & Sons, 2011.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Katherine Hermann, Ting Chen, and Simon Korbblith. The origins and prevalence of texture bias in convolutional neural networks. In Advances in Neural Information Processing Systems, 2020.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. arXiv preprint arXiv: 10758, 2018.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Data- models: Predicting predictions from training data. In International Conference Machine Learning (ICML), 2022.
Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. arXiv preprint arXiv:2206. 224i::2226..4544, 2022.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations (ICLR), 2017.
Been Kim. Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning (ICMI.), 2018.
Pang Wei Koh and Percy Liang Understanding black-box predictions via influence functions. In International Conference on Machine Learning, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural net- work representations revisited. In Proceedings of the 36th International Conference Machine Learning (ICML), 2019a.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In computer vision and pattern recognition (CVPR), 2019b.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical report, 2009.
Guillaume Leclere, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vineet, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang. et al. 3db: A framework for debugging computer vision models. In arXiv preprint arXiv:2106.03800 2021.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. https: github. com/ wwwfffi/fffeeeooeoooo 2022.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations? In Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015, 2015.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Neural Information Processing Systems (NeurIPS). 2019.
Scott Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Neural Information Processing Systems (NeurIPS), 2017.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. In Advances in Neural Information Processing Systems (NeurIPS), pp- 9993-10002, 2019.
Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, and Felix A. Wichmann. Trivial or impossible dichotomous data difficulty masks model differences (on imagenet and beyond). In International Conference Learning Representations (ICIR), 2022.
11

---

Under review as a conference paper at ICLR 2023
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 1995.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018a.
Ari Morcos, David GT Barrett, Neil Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations (ICLR), 2018b.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? Advances in neural information processing systems, 33:512-523, 2020.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In arXiv preprint arXiv:2 00020, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Korbblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Neural Information Processing Systems (NeurIPS), 2021.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. wwhy should i trust you?" explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. In International Journal Computer Vision (JJV), 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations, 2020.
Hadi Salman, Saachi Jain, Andrew Iyyas, Logan Engstrom, Eric Wong, and Aleksander Madry. When does bias transfer in transfer learning? In arXiv preprint Nixiv:2207.028444 2022.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. In International Conference on Learning Representations (ICLR), 2021.
Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems, 34, 2021.
Ruoqi Shen, Sebastien Bubeck, and Suriya Gunasekar. Data augmentation feature manipulation. In International Conference on Machine Learning, pp. 19773- 19808. PMLR, 2022.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
Sahil Singla and Soheil Feizi. Salient imagenet: How discover spurious features in deep learning?
In International Conference on Learning Representations, 2021.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study robustness spurious correlations using pre-trained language models. Transactions of the Association for Computa- fional Linguistics, 8:621-633, 2020.
12

---

Under review as a conference paper at ICLR 2023
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Junpeng Wang, Liang Wang, Yan Zheng, Chin-Chia Michael Yeh, and Shubham Jain Wnd Zhang. reaming-from-ddiagrrees preprint arXiv:2201.077 .08849, 2022.
model comparison and visual analytics framework. arXiv
Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An empirical study of large-batch stochastic gradient descent with structured covariance noise. arXiv preprint
2019.
Eric Wong, Shibani Santurkar, and Aleksander Madry- Leveraging sparse linear layers for debug- gable deep networks. In International Conference on Machine Learning (ICML), 2021.
John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similar- ity analysis of contextual word representation models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint Stt::2000.0099440 2020.
Kaiyu Yang, Jacqueline Yau, Fei-Fei, Jia Deng, and Olga Russakovsky. A study face obfuscation in imagenet. In International Conference Machine Learning, pp.
PMLR, 2022.
Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. Are larger pretrained language mod- els uniformly better? Comparing performance the instance level. Findings of the Association for Computational Linguistics (Findings of ACL), 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 2017.
13

---

Under review as a conference paper at ICLR 2023
Appendices
A Algorithm analysis
15
A primer on datamodel representations
15
Residual datamodels
A.3 Finding global trends with PCA
16
B Case study: SGD hyperparameters
17
C Experimental Setup
19
C.I Datasets
19
Models, learning algorithms, hyperparameten
................................
C.3 Datamodels
20
C.4 Feature transformations
21
C,5 Training infrastructure
21
D Additional human-in-the-lopp analysis
23
D.1 Tools for Inferring distinguishing features from PCA subpopulations
23
D.2 Case study: Standard data augmentation
24
Case study: ImageNet pre-training
25
D.4 Case study: SGD noise
26
E
Additional evaluation of distinguishing feature transformations
28
E.1 Case study: Standard data augmentation
28
E.2 Case study: ImageNet pre-training
29
E.3
Case study: SGD hyperparameters
30
F
Miscellaneous results
31
F.1
Aggregate metric for algorithm comparison
31
F2
Explained variance of residual datamodel principal components
32
F.
Subpopulations surfaced by principal components of residual datamodels
33
F44
Using CLIP to identify distinguishing fcature candidates
37
F5
Additional analysis on the effect of ImageNet pre- vre-trainin
39
On comparisons with model predictions and penultimate-lap representations...
41
Effect of sample size on datamodel estimation
G Related work model interpretation
44
14

---

Under review as a conference paper at ICLR 2023
A ALGORITHM ANALYSIS
In the main paper, we applied our comparison framework to identify feature transformations that distinguished three pairs of learning algorithms. Here, we describe in more detail the algorithmic stage of that framework, i.e., the stage whose purpose is to find distinguishing training directions. To this end, walk through each of the three steps of the algorithm presented in Section 2.2 and provide intuition for how they identify distinguishing training directions.
A.1 A PRIMER ON DATAMODEL REPRESENTATIONS
The first step in method is to compute datamodel vectors ( TR one for each test input zy- Each datamodel vector encodes the importance of individual training examples S to model's loss input žj when trained with learning algorithm A, More specifically, each vector corresponds to the solution to a specific regression problem these regression problems (explained below) form the basis of our analysis.
Setting up the regression problem. Let us fix a single learning algorithm A (being A1 or A2). For a given training set (ai...., J. a test input z, and subset S' C S of the training set we define the model output function as:
(() = the loss after training a model on 5' and evaluating on T.
For example, (xx, S*) can encode training a deep neural network on the subset S', then computing the network's correct-class margin the input For fixed the corresponding regression problem is to predict the model output f(z, S") given a subset S*.
Datamodels. Ilyas et al. (2022) show that-for deep neural networks trained on standard image classification tasks wes we can solve the regression problem above with a simple linear predictor. More specifically, they showed that
Eif(x, = 0, 1s
(1)
where 0₂ is (learned) parameter vector (called the datamodel for r)), and 1g (0, 13 is binary indicator vector of the set S", encoding whether each example of S is included in S" i.e.,
{
if r; S
(1s));
otherwise.
In this way, datamodels consistitute linear approximations of model output functions.
Datamodels as a representation space. While each datamodel is specific to an individual test input, can treat collection of datamodels embeddings representations of test inputs {ry} into common |S] -idmensiioa space. By comparing the representations, we can analyze the structure of the data-as used by the specific learning algorithm under study. Furthermore, these representations have a number of properties that make them useful for algorithm comparisons:
(a) Consistent basis: A datamodel representation for fixed train set always has the same basis: coordinate i corresponds to the importance of the i-th training example. This consistency makes datamodels convenient medium for algorithm comparisons, as representations are automatically aligned across different learning algorithms, and even across models that lack explicit representa- tions (e.g.. decision trees).
(b) Predictiveness:
Datamodel vectors are causally predictive of model behavior. That is, as Ilyas et al. (2022) show, we can use them to predict the counterfactual impact of removing or adding different training examples model output for given test example. As result, any trends find across the datamodel representations come with a precise quantitative interpretation in terms of model outputs (to which we will come back to later in thise section).
(c) Density: Datamodel representations also have a high effective dimensionality; that is, one needs thousands of components to explain significant fraction of variance, for instance, on CIFAR- 10 Ilyas et al. (2022). This suggests that datamodel representations encode fine- grained information about how each learning algorithm uses the training data, making them useful for uncovering subtle differences in model behavior.
"This is in stark contrast to "standard" representations derived from the penultimate layer of trained model, which tend to have effective dimensions that much lower, typically equal to number of classes minus one.
15

---

Under review as a conference paper at ICLR 2023
A.2 RESIDUAL DATAMODELS
In Step 2 of our algorithm, using the two sets of datamodels { and (0(2),, we compute the residual datamodel vectors:
0(112) = -.0) (0.")) 0 0
(Note that this operation only makes sense because the two sets of datamodels live in the same vector space-as see property (a) above.) As we demonstrate, they correspond to datamodels for certain residual model oulput function that we define below.
Recall that our overarching goal to find training directions that that strongly influence models trained with learning algorithm A1 but not A2 (or vice-versa) when classifying x. More specifically, we want to find training directions w that strongly influences (x, while ignoring directions that also influence f(8) (x, In other words, what we care about is the "residual" of f(1) after removing the part that is correlated with f(2), To capture this, consider the residual model output function of Ai relative to A2:
(z, (z) Pfo) (z,
where pyw is the correlation between two model output functions (across varying S). It turns that the datamodel for the residual output function is given precisely by the residual datamodel defined in Step 2 of our algorithm: the projection of one (normalized) datamodel representation into the nullspace of other representation. That is, residual datamodels correspond to linear approximation of the residual model output:
( (x, 5)] g(1/2) 1g
In summary, Step 2 of our procedure reduces understanding the differences in learning algorithms Ai and A2 to analyzing their residual model outputs via residual datamodels. The residual data- models highlight directions that influence learning algorithm A1 after projecting away directions that also influence learning algorithm A2.
A.3 FINDING GLOBAL TRENDS WITH PCA
The output of Step of our procedure is two set of residual datamodels one set looking at the residual of A, with respect to A2, and vice versa. These residual datamodels capture directions that each algorithm most sensitive to, though still on a per-example level. On the other hand, our goal is to find directions that each algorithm is most sensitive to on an aggregale level (as in Definition 2). We now show how the final step in our procedure, computing PCA on the residual datamodel representations, achieves this goal.
For given input x with datamodel 0, and for a training direction u IR (S), we can estimate the example's sensitivity to u as [(0- [hht is, the (estimated) effect on [(x S) of upweighting and downweighting the training samples according to Since our goal is to find direction that the residual model output function f ((122) is most sensitive in aggregate, our objective is
(((
arg
WES
The direction that maximizes this objective exactly the dominant principal component of the set of residual datamodels! Similarly. the top i principal components correspond to directions that algorithm Ar is most sensitive to after accounting for A2.
In summary, Step 3 of our procedure finds directions that strongly influence only learning algo- rithm by examining directions of highest explained variance of the residual datamodel vectors.
'Over the distribution of S.
16

---

Under review as a conference paper at ICLR 2023
B CASE STUDY: SGD HYPERPARAMETERS
The choices of optimizer and corresponding hyperparameters can affect both the trainability and the generalization of resulting models (Hoffer al., 2017; Keskar et al., 2017). In this case study, we focus on stochastic gradient descent (SGD), and its hyperparameters-teannig rate and batch size- that control the effective scale of the noise in SGD. We study the effect of these hyperparameters in the context of CIFAR- 10 (Krizhevsky, 2009) classifiers by comparing the following two learning algorithms:
Algorithm Ai: Training with high SGD noise: large learning rate (0.1) and small batch size
(256). Resulting models attain 93% average test accuracy.
Training with low SGD noise: small large rate (0.02) and large batch size (1024). Resulting models attain 89% average test accuracy.
Algorithm Ay:
Identifying distinguishing features. Applying our framework analogously as done in previous case studies, we identify two candidate distinguishing features:
Black-and-white texture: Direction A surfaces a subpopulation of black-and-white dogs (Fig- ure 5, top right), Additional analysis in Appendix D shows that subset of training images with black-and-whit objects (e.g., ships) influence predictions on this subpopulation only when mod- els are trained with low SGD noise (algorithm A2). This leads us to hypothesize that models trained with low SGD noise rely more on black-and-whit textural features to predict the class "dog." To test this hypothesis, we design a feature transformation that modifies given image with small wlahk-and-white patch, which loosely resembles the face and nose of dogs in the surfaced subpopulation (see Figure 8a).
Direction surfaces a subpopulation of front-facing trucks (Figure 5, bottom right) with shared characteristic: rectangular-shaped cabin and cargo area. Additional analysis Appendix shows that a subset training images with rectangular patterns influence predic- tions on this subpopulation only when models trained with low SGD noise. This suggests that models trained with low SGD noise partially rely on rectangular-shapp feature transformation modifies given image with
Rectangular shape:
patterns to predict the
class "truck". To test this hypothesis, patch of high-contrast rectangles, which loosely resembles the cabin / cargo shape of trucks in the surfaced subpopulation (see Figure 8h).
Findings. In Figure 8, we compare the effect of the above feature transformations on models trained with high and low SGD noise. The results again confirm both hypotheses. Adding a black-and- white patch of varying size (4/5/6 px) to test images increased P("'ogg)) by 8/12/14% for models trained with low SGD noise, but only by 6/8/9% for models trained with high SGD noise. Similarly, applying rectangular-shat shape patch of varying size (6/7/8 px) increased P("truck")) by 4/6/7% for models trained with low SGD noise, but only by 1/2/2% for models trained with high SGD noise. Once again, increasing the intensity (i.e., patch size) of these feature transformations widens the gap in sensitivity between the two model classes.
Connections to prior work. This case study shows how reducing the scale of SGD noise in- crease reliance on certain low-level features (e.g., rectangular shape - trucks). While prior works show that lower SGD noise worsens aggregate model performance (Keskar al., 2017: Wen al., 2019), our methodology identifies specific features that are amplified due to low SGD noise. Fur- thermore, the simplistic nature of the identified features corroborate the theoretical results put forth in Li et al. (2019): the learning rate scale determines the extent to which models memorize patterns that are easy-to-fit but hard-to-gent -eneeto-eeeeralize. More broadly, our framework motivates a closer look at how features amplified via low SGD noise collectively alter aggregate model performance.
17

---

Under review as a conference paper at ICLR 2023
Explnited Algoritton (S0)
of (1-12) PCs at (AU0)
Line x-y
A
0.35
0.10
B
com
Explained (High Noise)
Algonithm As (%)
Figure 7: Comparing CIFAR-10 models trained with high and low SGD noise. An analogous figure Figure 3 for our third case study, see Figure for a description. In this case, direction A seems to correspond dogs with particular black-and-whit texture, and direction B to front- facing trucks with prominent rectangular component(s).
Fligh
Hfigh
Line
Add Paich
Add Patch
Sioe Sprc
Peccant Incruse in Peedog
12
Size fipx
Add Futch
Add Fatch
Siee Sps
Sizw Pps
W
Perent
Folch
Putch Size
Sizee
Patch Size
Patch Size
(a) "Black-an white texture" feature
(b) "Rectangular shape" feature
Figure 8: Effect of SGD hyperparameters on CIFAR-10 models. Analogously to Figures 4 and 6, use our framework identify features that distinguish models trained with lower SGD noise from models trained with higher SGD noise. (Left) Adding black-and-white patch to images makes models trained with low (high) SGD noise, on average, 11% (8%) more confident in predicting the label "dog." (Right) Adding high-contrast rectangles to images makes models trained with low (high) SGD noise, on average, 5.5% (1.5%) more confident predicting the label "truck." " both cases, increasing the intensity of feature transformations widens the gap in treatment effect between the two model classes.
18

---

Under review as a conference paper at ICLR 2023
C EXPERIMENTAL SETUP
In this section, we outline the experimental setup-datasee datasets, models, training algorithms, hyperpa- rameters, and datmodels- used for case studies in Section 3.
C.I DATASETS
Li The 17 dataset (Santurkar et al., 2021) is an ImageNet-derived dataset, where the task is classify images belonging to 17 types of living organisms (e.g., salamander, bear, fox). Each Living17 class corresponds to an ImageNet superclass (i.e., set of ImageNet classes aggregated using WordNet (Miller, 1995)). Santurkar et (2021) introduce Living four benchmark to evaluate model robustness to realistic subpopulation shifts. In our case study, we study the effect of data augmentation using a variant of this dataset, wherein the training and test images belong to the same set of subpopulations (i.e., no subpopulation shift).
one of
Waterbirds. The Waterbirds dataset (Sagawa al., 2020) consists of bird images taken from the CUB dataset (Wah et al., 2011) 1) and pasted on backgrounds from the Places dataset (Zhou et al., 2017). The task here is to classify "waterbirds" and "landbirds" in the presence of spurious corre- lated "land" and "water" backgrounds in the training data. Sagawa et al. (2020) introduce Waterbirds as a benchmark to evaluate models under subpopulation shifts induced by spurious correlations. In case study, compare how models trained from scratch Waterbirds data differ from MaageNet-pretrainet models that are fine-tuned on Waterbirds data.
CIFAR-10. We consider the standard CIFAR-10 (Krizhevsky, 2009) image classification dataset in order to study the effect of two SGD hyperparameters: learning rate and batch size.
Summary statistics of the datasets described above outlined in Table 1.
Table 1: Summary statistics of datasets
Dataset
Classes
Size (Train/Test)
Input Dimensions
Living17
17
88, 400030033400
3 x 224 224 x 224 x 224
Waterbirds
2
4.795/5,794
CIFAR-10
10
50,000/10,000
3 x 32 x 32
C.2 MODELS, LEARNING ALGORITTMS, AND HYPERPARAMETERS
Living17. We use the standard ResNet18 architecture (He et al., 2015) from the torchvision library. We train models for 25 epochs using SGD with the following configuration: initial learning rate 0.6, batch size 1024, cyclic learning rate schedule (with peak at epoch 12), momentum 0.9, weight decay 0.0005, and label smoothing (with smoothing hyperparameter 0.1). To study the effect of data augmentation, we train models with the following algorithms:
Algorithm Ai (with data augmentation): Models trained with standard data augmentation: random resized cropping (with default torchvis s sion hyperparamters) and random horizontal flips. On average, models attain 89.2% average test accuracy.
Algorithm A2 (without data augmentation): Models are trained without data augmentation. On average, models attain 81.9% average test accuracy.
Waterbirds. We use the standard ResNet50 architecture from the torchvision library. We train models using SGD with momentum 0.9 and weight decay 0.0001 for maximum of 50 epochs (and stop early if the training loss drops below 0.01). For model selection, choose the model check- point that has the maximum average accuracy on the validation dataset. As in Sagawa et al. (2020), do not use data augmentation. In our case study pre- pre-training, consider ImageNet pre- trained models from torchvision. We consider models trained using the following algorithms:
Algorithm A₁ (ImageNet pre-training): Models pre-trained ImageNet are fully fine-tuned on Waterbirds data with a fixed SGD learning rate 0.005 and batch size 64. On average, models attain 89. 59.1% (non-adjusted) average test accuracy and 63.9% worst-group test accuracy.
19

---

Under review as a conference paper at ICLR 2023
Algorithm Ag (Training from scratch): Models are trained from scratch (i.e., random initializa- tion) on Waterbirds data with SGD: initial learning rate 0.01, batch size 64, and a linear learning rate schedule (0.2x every 15 epochs). On average, models attain 63.6% average test accuracy and 5.7% worst-group test accuracy.
CIFAR-10. We use the ResNer9 architecture from Kakao Brain', which is optimized for fast train- ing. We train models using SGD with momentum 0.9 and weight decay 0.0005 for a maximum of 100 epochs (and stop early if the training loss drops below 0.01). We augment training data with a standard data augmentation scheme: random resized cropping with 4px padding and random hori- zontal flips. To study the effect of SGD noise in our case study, we vary learning rate and batch size. Specifically, we compare models trained with the following algorithms:
Algorithm A1 (high SGD noise): Models are trained with SGD using a large initial learning rate (0.1), small batch size (256), and a linear learning rate schedule (0.5x every 20 epochs). On average, models attain 93.3% test accuracy.
Algorithm A₂ (low SGD noise): Models are trained with SGD using small fixed learning rate (0.02) and large batch size (1024). On average, models attain 89.5% test accuracy.
C.3 DATAMODELS
Now, we provide additional details datamodels which, recall, are used in the first stage of our algorithm comparison framework (see Section 2).
Estimating linear datamodels. Recall from Appendix A that the datamodel vector for example Ty, 0f E RRIS, encodes the importance of individual training examples S to model's loss at ex- ample 2; when trained with algorithm A,, Concretely, given test example zj and training set f za). the datamodel 0, is a sparse linear model (or surrogate function) trained on the following regression task: For training subset S' C S, can predict the correct-class margin fa(z;; 5") of a model trained on S' with algorithm A?? This task can be naturally formulated as the following supervised learning problem: Given training set {(S,, fa(z; St)) of size m, the datamodel 8 (for example æz) is the solution to the following problem:
1 m
0j = min
Σ
w 1w..
S.)x;; + A|ww/|,,
WERISI 17A
where 1s; is a boolean vector that indicates whether examples in the training dataset ar E belong to the training subset S, Note that each datamodel training point (S;, St)) is obtained by (a) training a model f (e.g.. ResNet9) on a subset of data S; (e.g., randomly subsampled CIFAR data) and (b) evaluating the trained model's output on example Tz. Ilyas et al. (2022) demonstrate that linear datamodels can accurately (and counterfactually) predict outputs of deep image classifiers.
Datamodel estimation hyperparameters. Recall that our algorithm comparison framework in Sec- tion 2 involves estimating two sets of datamodels (0(1) and {0(2)) for learning algorithms Ai and A respectively. In our case studies, we estimate two datamodels, 0!!) and o(²) for every example s; in the test dataset. Estimating these datamodels entail three design choices:
Sampling scheme for train subsets: Like in Ilyas et al. (2022), we use o-random subsets of the training data, where denotes the subsampling fraction; set 50% as it maximizes sample efficiency (or model reuse) for empirical influence estimation (Feldman & Zhang, 2020), which is equivalent variant of linear datamodels.
Sample size for datamodel estimation: Recall that datamodel training set of size m corre- sponds to training m models (e.g., m ResNet18 models on CIFAR-10) on independently sampled train subsets (or masks). We estimate datamodels on LIVING17, WATERBIRDS, and CIFAR-10 using 120k, 50k, and 50k samples (or models) per learning algorithm respectively; we make a validation split using 10% of these samples.
pettps://githan com htt.://ww.....aeooooo/cc/ecaa/// ci eifarl0.py
20

---

Under review as a conference paper at ICLR 2023
4 sparsity regularization: We cross-validation to select the sparsity regularization parameter Specifically, for each datamodel, we evaluate the MSE on a validation split to search over A = 50 logarithmically spaced values for along the regularization path. We then co-puu the datamodel on the entire dataset with the optimal À value and all m training examples.
We provide pseudocode for estimating datamodels in Algorithm 1.
Algorithm An outline of the datamodeling framework introduced llyas et al. (2022).
1: procedure ESTTMAATEDAATAMODDLLLAarggg example 2, train set S size d, subsampling frac. E ())
2:
T 7+1
Initialize datamodel training set
3:
for E (1...,mm do
4:
Sample a subset S; C S from Ds where |S| = a d
5:
Yi - fa( St) Define 1s. {0, 13dd as (1s,), = 1 if zy E S clse 0
Train a model on S₁ using A, evaluate on
6:
7:
T T + (1s.,
b Update datamodel training set
8:
+ RUNREGRESSION(T)
b Predict the from the 1s, vectors Result: a weight vector 9 R
9:
return 0
C.4 FEATURE TRANSFORMATIONS
As discussed in Section 2, counterfactually verify distinguishing features (inferred via human- in-the-loop analysis) by evaluating whether feature transformations change model behavior as hy- pothesized. Here, we describe the feature transformations used in Section 3 in more detail
Designing feature transformations. We design feature transformations that modify examples by adding specific patch or perturbation. We vary the intensity of patch-based and iitturrbtiinn-baeee transformations via patch size and perturbation intensity respectively. Additional details specific each case study:
Pre-training. We use patch-based transformations in this case. For the yellow color feature, we add k x k square yellow patch the input. For the human face feature, add k x k image of human face to the input. To avoid occlusion with objects in the image foreground, we add the human face patch to the background. We make bank of roughly 300 human faces using ImageNet face annotations (Yang et al., 2022) by (a) cropping out human faces from ImageNet validation examples and (b) manually removing mislabeled, low-resolution, and unclear human face images.
Data augmentation. We design perturbation bituurbation-bbsed transformations to verify the identified dis- tinguishing features: spider web and polka dots. In both cases, 8-perturb each input with random crop of a fixed grayscale spider web or yellow polka dot pattern.
SGD hyperparameters. We use patch- patch-based transformations in this case study. For the black- white texture feature, we add a k-sized patch that loosely resembles black-white dog nose. Sim- ilarly, for the rectangular shape feature, we add k-sized patch that loosely resembles windows.
Evaluating feature transformations. As shown in Section 3, given two learning algorithms A and Ag, we evaluate whether a feature transformation changes predictions of models trained with A₁ and Ag hypothesized. To evaluate the counterfactual effect of transformation F on model M, we evaluate the extent to which applying F' to input examples æ increases the confidence of models in particular class y. In experiments, all test examples and over 500 models trained with each learning algorithm.
estimate this counterfactual effect by averaging
C.5 TRAINING INFRASTRUCTURE
Data loading. We use FFCV (Leclere al., 2022), which removes the data loading bottleneck for smaller models, gives a 3-4x improvement in throughput (i.c., number of models day per GPU).
"hhe code for these feature transformations available at anonymi zed-ur1. Wehpaage: http: ffev.io
21

---

SAGA-based Under review as a conference paper at ICLR 2023
Datamodels regression. In addition to FFCV, the fast-11 package "0 GPU solver for t e1-reuularized regression to parallelise datamodel estimation.
Computing resources. We train our models cluster of machines, each with 9 NVIDIA A100 or V V100 GPUs and 96 CPU cores. We also use palf-prreision to increase training speed.
"Github /
22

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL HUMAN-IN-THE-LOOR ANALYSIS
As outlined in Section 2, the second stage of our framework applies human-in-the-lopg analysis to infer distinguishing feature transformations from training directions extracted via PCA residual datamodels. In this section, we present additional human-in-the-loop analysis in order to substantiate the distinguishing features inferred in each case study.
D.1 TOOLS FOR INFERRING DISTINGUISHING FEATURES FROM PCA SUBPOPULATIONS
In this section, we outline additional human- comman-in--hh--ee in-the-loop tools that use to analyze subpopulations surfaced by principal components (PCs) of residual datamodels.
Class-specific visual inspection. As shown in Section the subpopulation of test examples whose datamodels have maximum projection onto PCs of residual datamodels largely belong same class. So, a simple-yet--feetive way to identify subpopulation-specee distinguishing feature(s) is to just visually contrast the surfaced subpopulation from set of randomly sampled examples that belong the same class.
Relative influence training examples. Given subset of test examples S' C S, can we identify a set of training examples T' C T that strongly influence predictions on S' when models are trained with algorithm Ar but not when trained with A₂? Given datamodel representations {0, (e for Ai and {0,-) for A2, we apply two-step heuristic approach identify training examples with high influence on A₁ relative to AA:
First, given learning algorithm Ar and test subset S' we estimate the aggregate (positive or negative) influence of training example on subset S by taking the absolute sum over the corresponding datamodel weights: Ejes' [0
Then, we take the absolute difference between the aggregate influence estimates of training example Fy using ((1) and g(2), This difference measures the relative influence of training example 2g on predictions test subset 5 when models are trained with algorithm A instead of algorithm A2.
In our analysis, we (a) identify training examples that have top-most relative influence estimates and then (b) visually contrast the subsets of test examples (one for each learning algorithm) that are most influenced by these training examples.
23

---

Under review as a conference paper at ICLR 2023
D.2 CASE STUDY: STANDARD DATA AUGMENTATION
Our case study on LIVING17 7 data in Section 3.1 shows that standard data augmentation can amplify co-occurrence bias (spider web) and texture bias (polka dots). We further substantiate these findings with relative influence analysis (Figure 9) and class-specific visual inspection (Figure 12).
Subset of axamples training data
Subset Influence on test data
020
6:00
0.00
0.00
0.0
6.1
0.3
Algoritim
Most influenced examples in test (Learning llorithm Ai)
Most influenced examples test data (Learning algerithm Ap)
(a) "Spider web" feature
examples training
on data
Algeithimm A.
32
Algorthm
Most Influenced examples in test data (Leariing algorithm Aj)
Most influenced examples in test data (Learning algorithm A.)
KMA
(b) "Polka dots" feature
Figure 9: Relative influence of training data LIVING17 subpopulations. Panel (a): Training images that contain web-like patterns have high relative influence on the "spider web" test sub- population (see Figure 3). These images strongly influence model predictions on test images that contain spider webs (in bottom row) only when models are trained with augmentation (algorithm A1). Panel (b): Training images that contain yellow-black texture have high relative influence on the "polka dots" test subpopulation (see Figure 3). These images strongly influence model predic- tions on test images of salamanders with yellow polka dots (in bottom row) only when models are trained with augmentation (algorithm A1).
24

---

Under review as a conference paper at ICLR 2023
D.3 CASE STUDY: IMAGENET -RAIIIINN
Our case study on WATERBIRDS data shows that ImageNet pre-training reduces dependence on the "yellow color" feature, but introduces dependence the "human face" feature. We support these findings with relative influence analysis in Figure 10 and additional visual inspection in Figure 13.
Subset of examples training data
Subset influence on test data
34
26.0
0.4
34
10
Algorttim
Most influenced examples in test (Learning a lgorithm A₁)
Most influenced examples test data (Learning algerithm As)
(a) "Yellow color" feature
examples training
on
Algonthim A.
Algorthm
Most Influenced examples in test data (Learning a algorithm Ar)
Mast influenced examples in test data (Learning algerithm 4.)
'Human face" feature
Figure 10: Relative influence of training data on WATERBIRDS subpopulations. Panel (a): Training images with yellow objects in the background have high relative influence on the "yellow color" test subpopulation (see Figure 5). These images strongly influence model predictions test images that have yellow birds objects (bottom row) only when models are trained from scratch (algorithm A2). Panel (b): Training images that contain human faces in the background have high relative influence on the "human face" test subpopulation (see Figure 5). These images strongly influence model predictions on test images (in bottom row) with human face(s) only when models are pre-trained on ImageNet (algorithm A1).
25

---

Under review as a conference paper at ICLR 2023
D.4 CASE STUDY: SGD NOISE
We analyze relative influence (Figure 11), and class-specific subpopulations (Figure 14) to hone in on instances of distinguishing features-black-and-wy texture and rectangular shape
in CIFAR-10 10 data that are amplified by low SGD noise.
Subset of examples training data
Subset Influence on test data
640
6:30
6:32
8.64
5.0
Algorithm
Most influenced examples in test (Leariin a lgorithm Au)
Most influenced examples test data (Learning algerithm Aq)
(a) "Black-white texture" bias
examples training
6.05
0:91
Algorithm À
Most influenced examples in test data Learning Igorithm Ax)
Most influenced examples in test data (Learning algerithm Ag)
"Rectangular shape" bias
Figure 11: Relative influence of training data on CIFAR-10 subpopulations. Panel (a): Training images with black-white objects have high relative influence on the "black-white" dog subpopula- tion (see Figure 7). These images influence model predictions on test images of black-white dogs (in bottom row) only when models are trained with low SGD noise (alg. Ag). Panel (b): Training images with high-contrast rectangular components in the background have high relative influence on the "rectangular shape" truck subpopulation (see Figure 7). These images influence model predic- tions on test images of front-facing trucks with prominent rectangular components (in bottom row) only when models are trained with low SGD noise (alg. Ag).
26

---

Under review as a conference paper at ICLR 2023
Subpopulation surfaced via direction (spider web)
Random sample spider test data
Subpopulation surfaced via direction (polka dot texture)
Random sample of sala mander Images in test data
Figure 12: Class-specific visual inspection of LIVING17 17 subpopulations. (Top) In contrast to ran- dom LIVING 17 images of spiders, the "spider web" subpopulation surfaces spiders with a promi- nent spider web in the background. (Bottom) Unlike random LIVING17 images of salamanders, the "polka dots" subpopulation surfaces salamanders that have a yellow-blace polka dot texture.
Subpopulation surfaced via direction A (human face)
Random sample of test images in class landbird
Subpopulation surfaced dirrection (yellow color)
Random sample test class landbird
Figure 13: Class-specific visual inspection of WATERBIRDS subpopulations. (Top) In contrast to random "landbird" images, the "human face" subpopulation surfaces landbirds with human face(s) in the background. (Bottom) Unlike random "landbird" images, the "yellow color" subpopulation surfaces images with yellow birds or yellow objects in the background.
Subpopulation surfaced vin direction texture)
sample dog images
Subpopulation surfaced direction frectang eantaar shape)
Random sample truck imag es
Figure 14: Class-specific visual inspection of CIFAR- 10 subpopulations. In contrast to random images of dogs (top) and surface images of black-white dogs and front-facing trucks with multiple rectangular components respectively.
truck (bottom). the "black-white" and "rectangular shape" subpopulations
27

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL EVALUATION OF DISTINGUISHING FEATURE TRANSFORMATIONS
Distinguishing feature transformations, which recall from Section 2, functions that, when applied to data points, change the predictions of one model class-but not the other-in a consistent way. In our case studies, we design distinguishing feature transformations that counterfactually verify features that are identified via human-in-the-lon feature transformations to quantitatively measure the relative effect of the identified features on models trained with different learning algorithms. In this section, we present additional findings on feature transformations for each case study:
analysis. Our findings in Section 3 use
E. CASE STUDY: STANDARD DATA AUGMENTATION
In Section 3.1, showed that standard data augmentation mugmentation-hooizonnet flips and random crops- amplifies LIVING17 models' reliance on "spider web" and "polka dots" to predict spiders and salamanders respectively. Figure 15 verifies tion intensity 5 values. We also observe that decreasing the minimum allowable crop size in RandomResizenn zedCrop from 1.0 (i.e., no random cropping) 0.08 (default torchvision hy- perparameter) increases models' sensitivity to both feature transformations.
findings
larger range of perturba-
Effect of spider web feature transformation Living17 models
Percent increase Prispidor
15.0
12.5
Perturbation Intensity 5
10.0
0.1
7.5
0.4
5.0
0.5
2.5
0.0
1.)
cropping
(0.9.
(0.6,
(0.08,1.) (Default
RandomReszzdCrog scale hyperparameter
(a) "Spider web" feature
polka feature transformation on Living17 models
Porcent increase Pr(salamanden
20
Perturbation Intensity 5
2.5
0.05
0.15
10
0.2
0.25
5
0.35
(1., 1.1 (No croppingl
1.1 1.1
(0.6, RandomResizeddCapp scale hyperparameter
(0.3, (0.08.1.) [Default]
(b) "Polka dots" feature
Figure 15: Additional evaluation LIVING feature transformations. The top and bottom row evaluate the effect of "spider web" and "polka dot" feature transformations on models trained with different data augmentation schemes. Increasing the intensity of the transformations and the min- imum crop size of RandomRest zedCrop augmentation (via scale hyperparameter) increases the sensitivity models to both feature transformations a consistent manner.
28

---

Under review as a conference paper at ICLR 2023
E.2 CASE STUDY: IMAGENET PRE-TRAINING
In Section 3.2, we showed that fine-tuning ImageNet-pretraiine ResNer50 models on WATERBIRDS data instead of training from scratch alters the relative importance of spurious features: "yellow color" and "human face". In Figure 16, we show that both feature transformations alter the pre- dictions of preeraet- ResNet18 8 and ImageNet-pretraing ResNet50 models in similar way.
Effect of yellow color feature transfomation Waterbirds models
12.5
Increase in Pr(landbird)
10.0
7.5
5.0
Learning Algorithm
ImageNet-pretrainee ResNet18
2.5
ImageNet pretrained ResNet50
mmageNet-pretraine
Randomly initialized
0.0
-25
10.0
20.0
30.0
40.0
Patch
(a) Yellow color" feature
Effect of human face feature transformation on Waterbirds models
Percent Increase in Pr(landbird)
Learning Algorithm
ImageNet-pretraing ResNet18 Imageeet-pretrained ResNet50
Randomiy initialized ResNet50
120
Patch Size
"Human face" feature
Figure 16: Additional evaluation of WATERBIRDS feature transformations. The top and bot- tom evaluate the effect of "yellow color" and "human face" feature transformations models trained with and without ImageNet pre-training. In both cases, unlike ResNet50 models trained from scratch, ImageNet- ResNet18 and ResNet50 models sensitive the "human face" transformation but not to the "yellow color" transformation.
29

---

Under review as a conference paper at ICLR 2023
E.3 CASE STUDY: SGD HYPERPARAMETERS
In Section 3.3, we showed that reducing SGD noise results in CIFAR-10 models that are more sensitive certain features, such rectangular shape bias and black-white texture predict trucks and dogs. In Figure 17, we evaluate how feature transformations change class-wise predictions of models trained with different SGD learning rate and batch size hyperparameters.
effect of black-white texture feature transtormation CIFAR10 models
40
Percent increase in Prtoo)
SCD hyperparameters IR:0.11 LR:0.05, 85:512
LR:0.02, BS:1024
3.0
cat
ship
airplanc
automobile
truck
dear
bird
(a) "hhack-white texture" feature
wise effect of rectangular shape feature transformation on CIFAR10 models
Percent increase in Pritruck)
Hyperparamaters
LR:0 BS:128
LR:0 05, 8S:512
LR:0.02 85:1024
bird
airplane
horse
truck
automobile
deer
dog
cal.
frog
ship
class
(b) "Rectangular shape" feature
Figure 17: Additional evaluation of CIFAR-10 feature transformations. The top and bottom row evaluate the effect of -black-white texture" and "rectangular shape" feature transformations on CIFAR- 10 models trained with high (light blue), medium, and low (dark blue) SGD noise. In both cases, models trained with higher SGD noise are, on average, more sensitive to these transforma- tions across all classes. Furthermore, the effect of the transformations are class dass-dependent model predictions on transformed examples from semantically similar classes differ to greater extent.
30

---

Under review as a conference paper at ICLR 2023
F MISCELLANEOUS RESULTS
F.1 AGGREGATE METRIC FOR ALGORITHM COMPARISON
As discussed in Section 4, we can repurpose our framework as a similarity metric that quantifies the similarity of models trained with different learning algorithms in more global manner. A straight- forward approach to output a similarity score, (or distribution) is to compute the cosine similarity of datamodel vectors. More concretely, let 0.1) and 0, denote the datamodels of example æ; with peapect to models trained using learning algorithms Ai and A2. Then, the cosine similarity between 8(1 and 6! measures the extent which models trained with A1 and A2 depend on the same set of training examples to make predictions on example Z₁.
We apply this metric to two case studies cesttes--retuutee ppe--trainin (WATERBIRDS) and SGD noise (CIFAR- 10>-in Figure 18. Specifically, Figure 18 plots the distribution cosine similarity datamodels for multiple learning algorithms and over all test examples. The left subplot shows that WATER- BIRDS, ImageNet-pretraines ResNet50 models are, on average, more similar to ImageNet-pretraine ResNet18 models than ResNet50 models pretrained on synthetically generated data (Baradad Ju- rjo et al., 2021) and models trained from scratch. The right subplot shows that on CIFAR- 10, ResNet9 models trained with high SGD noise are more similar to smaller-widhh ResNet9 models trained with high SGD noise than to ResNet9 models trained with low SGD noisc.
Casine similarity w.rt TetegeNet-pretraiiee ResNet50
Cosine similarity w.rt ResNets w/ high SGD noise
0.08
Leaming algorithmm (average cosine similarity)
Leaminn algorithms (average cosine similanit)
RcsNet18 pretrained (0.59)
0.07
Wiahho.55 BesNutt wi high 50D roise (0.76)
Besheib0 preteimed
0.06
ResNets w/ low SGD noise (0.59)
9.04
0.03
0.82
0.01
0.
0.4
0.5
3.5
0.7
0.0
Cosine simlarity between datamodiel vectors
Costne simiarty between datamodia vectors
Figure 18: Datamodel cosine similarity. We use cosine similarity between two datamodel vectors as an aggregate metric to quantify the similarity of models trained with different learning algorithms. (Left) On WATERBIRDS data, datamodels of ImageNet-pretraine ResNet50 and ResNet18 models are more similar to each other than to models pretrained on synthetically generated data and models trained from scratch. (Right) On CIFAR- 10 data, ResNet models trained with high SGD noise are more similar to each other to ResNet models trained with low SGD noisc
31

---

Under review as a conference paper at ICLR 2023
F.2 EXPLAINED VARIANCE OF RESIDUAL DATAMODEL PRINCIPAL COMPONENTS
Recall from Appendix A that the fraction of variance in datamodel representations {0 explained by training direction signifies the importance of the direction (or, combination of training exam- ples) to predictions of models trained with algorithm A,- Through our case studies in Section 3, we show that the top 6 principal components (PCs) of residual datamodels ((1)2) correspond to training directions that have high explained w.r.t datamodels of algorithm A1 but not A2, and vice versa. Figure 19 shows that the top-100 PCs of residual datamodel (())2) (resp., ((?(1)) have more (resp., less) explained variance datamodel ((1) than on datamodel ((2)
Explained Vartane under Algorthm 4,
Living17 Data augmentation
Explained Varzance under Algonthum Ag tha
Waterbirds
Pretraining
Explained Variansa undder Algosttum Ag C3a
CIFARIO SGD hyperaramettess
oC (1
of (211)
PC (211))
PC of #2111 Lime x-y
(Nithows Augnentation
06
Tine xay
Tine xey
EEou SCD Natser
0.3
(Vilbout
0,00
0001
d15
Explained Varlance unden Algoeithun
Explained Varlance under Algositiun (WSi Image Nett Pre-trxtming)
Explained Varlance under Algorithun (Righ SGD Noiwe)
(With Augmentation)
Figure 19: Explained variance of residual datamodels' principal components. Highlighted in green (resp. red), the top-100 PCs of residual datamodel g(1\2) (resp. 0(2)1)) explain a larger (resp- smaller) fraction of datamodel variance under algorithm A₁ than under algorithm A across all three case studies.
32

---

Under review as a conference paper at ICLR 2023
F.3 SUBPOPULATIONS SURFACED BY PRINCIPAL COMPONENTS OF RESIDUAL DATAMODELS
As outlined in Section 2, the human-in-the-loop stage of our framework involves extracting test data subpopulations from principal components (PCs) of residual datamodels. Specifically, these sub- populations correspond to test examples whose residual datamodel representations have the most positive (top-k) and most negative (bottom-k) projection onto a given PC. Here, we show that the top-k and bottom-k subpopulations corresponding to the top few PCs of residual datamodels con- sidered in Section 3 surface test examples with qualitatively similar properties.
Principal component d1|
subpopulation
Principal component Bottom-k pulation
Principal component #2 Top- subpopulation
Principal component #2| Bottom k subpo pulla tion
Principal compo Top- subpopulation
Principal component Bottom subpo pula tion
Principal component di Top- subpoputatiin
Principal component Bottem-k puatttion
Principal component #5 Top- subpopulation
Principal component #5 Bottom k subpopulation
Figure 20: Top five PC subpopulations of LIVING17 residual datamodel ((122), where learning algorithms Ar and Ag correspond to training models with and without standard data augmentation respectively. Our case study in Section 3.1 analyzes PC #2 (direction A) and PC #5 (direction B).
33

---

Under review as a conference paper at ICLR 2023
Principal component #1 | Top-k su bppopuattioon
Prin drinl component #1 Bottom * sub populatton
Principal component #2 Top-k su bpopulation
Principal component #2 Bottom-k & sub populaatioo
Principal bpopulation
Bottom populaatin
component Top-k bpopuuattion
population
Figure 21: Top four PC subpopulations of CIFAR- 10 residual datamodel ((2(1), where learning algorithms Ai and A₂ correspond to training models with high and low SGD noise respectively. Our case study Section 3.3 analyzes PC #1 (direction A) and PC #2 (direction B).
34

---

Under review as a conference paper at ICLR 2023
Principal component #1 Top-k su bupopulatioon
Principal component A1 Bottom & sub subpopulaiin
Principal component Top-k bpopulattion
crial Bottom-k populattio
Principal component Top-k k suboopulation
Principll component population
Figure 22: Top four PC subpopulations of WATERBIRDS residual datamodel ((122), where learning algorithms A, and Ag correspond to training models with and without ImageNet pre-training re- spectively. Our study in Section 3.1 analyzes PC #3 (direction A).
35

---

Under review as a conference paper at ICLR 2023
Principal component #1
Top-k su boopulatioo
Pipal component A1 Bottom & sub popullttin
Principal bpopulation
Princ Bottom-k populaatin
component Top-k bpopulation
Principal population
Figure 23: Top four PC subpopulations of WATERBIRDS residual datamodel ((2)1), where learning algorithms A, and A₂ correspond to training models with and without ImageNet pre-training re- spectively. Our study in Section 3.1 analyzes PC #3 (direction B).
36

---

Under review as a conference paper at ICLR 2023
F.4 USING CLIP TO IDENTIFY DISTINGUISHING FEATURE CANDIDATES
As discussed in Section 2 and Appendix D. the second stage of our framework applies human-in- the-loop analysis infer distinguishing feature transformations that disparately impact predictions of models trained with different learning algorithms. In this section, we demonstrate that for image classifiers, shared rision-languaa models such as CLIP (Radford al., 2021) provide a streamlined alternative to manual human-in -omaa--n-thheloop identification of distinguishing features.
Approach. Before we describe our approach, note that CLIP contrastive learning method that embeds text and natural language into shared embedding space. Our approach leverages CLIP embeddings to identify multiple distinguishing captions-reye repressnatrreeeeeeeveeee descriptions that best contrast given subpopulation of images from set of images sampled from the same distribution. In the context of our framework, our CLIPP-ased approach takes as inputs a distinguishing training direction v, set of images D, and set of captions S and outputs set of distinguishing captions S' in four steps:
Pre-compute image and text embeddings. Use the image encoder of CLIP model to compute set of normalized embeddings for all images in D. Analogously, use the text encoder of a CLIP model to compute a set of normalized embeddings for all captions in S.
Record image-text pairwise cosine similarity. Let vector C, E RI denote the pairwise cosine similarity between the embedding of image E D and all captions j E S.
Compute mean cosine similarity over dataset and top-k subpopulation. Compute the mean cosine similarity vector C = EieD C; over all images in D. Similarly, given distinguishing training direction compute the mean cosine similarity vector ((o) over the top-k images whose residual datamodel vectors are most aligned with v.
Extract distinguishing captions S'. Use cosine similarity vectors C' and (()) to extract captions in S that have the maximum difference between cf and C.
Intuitively, the set of distinguishing captions S' correspond to representative captions (or, descrip- tions) that best contrast the top-k images surfaced by distinguishing direction from the dataset.
Results. We now apply this approach to our case study on ImageNet pre-training, where we compare WATERBIRDS models trained with and without ImageNet pre-training (see Section 3). Specifically, we evaluate whether the CLIP-based approach surfaces distinguishing captions that similar distinguishing features "yellow color" (direction A) and "human face" (direction B) inferred via manual human- cenman--n-tthe-eee in-the-loop analysis. Figure 24 illustrates that for direction A). the CaiP-based approach highlights distinguishing captions such as yellow, lemon, and sulphur, all of which are similar to the "yellow color" feature that we infer via human-in-the-lonw analysis. Similarly, Figure 25 shows that the distinguishing captions for direction B (e.g., florist, faces, counselors) are similar to the identified "human face" feature.
To summarize, we demonstrate how the second stage of our framework can be easily specialized to comparisons of vision classifiers trained on ImageNet-like data via vision-language embeddings such as CLIP.
We use filtered list of roughly 20,000 most common English words in order of frequency, taken from /tt:::///ttttt wtt.bb.comm/irrt/toorrrrre e- 00000=eg]1 1gg ish.
37

---

Under review as a conference paper at ICLR 2023
Subpopulation surfaced direction
Randem Waterbirds dataset
Distinguishing captions CLI space
evlevvnc metric
0.04
0.03
0.02
0.00
yellow
lemon
sulphur
suifur
amarilio
mustard
underoath
pollen
Top-k captions
Direction A. The CLIP-based approach extracts distinguishing captions such as yellow, lemon, and sulphur. all of which contrast the residual subpopulation on the left to of random images from the WATERBIRDS dataset on the right. These distinguishing captions match the "yellow color" feature that we infer and counterfactually verify via suman-in-the-lowe analysis in Section 3.2.
Figure 24:
Subpopulation surfaced via direction
Random images from Waterbirds detaset
Distinguishing captions CLIP space
Relevance metric
0.020
0.015
0.010
0.005
0.000
Mlorisss
florist
faces
counselors
actresses
translator
Iranslators
handler
designers
presenters
Topsk caplions
Figure 25: Direction B. The CLIP-based approach extracts distinguishing captions such as florists, faces, and counselors, all of which contrast the residual subpopulation (left) of images with human faces in the background to set of random images (right) from the WA- TeRBIRds dataset. These distinguishing captions match the "human face" feature that we infer and counterfactually verify via human-in-the-lon analysis in Section 3.2.
38

---

Under review as a conference paper at ICLR 2023
F.5 ADDITIONAL ANALYSIS ON THE EFFECT OF IMAGENET PRE-TRAINING
In this section, we evaluate our algorithm comparisons framework with a controlled experiment on WATERBIRDS data, wherein we compare learning algorithms that differ in one more axes. Through this experiment, we show that our framework identifies feature transformations that (a) impact similar algorithms similarly and (b) distinguish dissimilar algorithms even when they result in models with similar (and relatively high) test accuracies.
Setup. To design controlled experiment, we extend our case study on ImageNet pre-training (see Section 3.2) in two ways:
Learning algorithms. We consider four learning algorithms in this experiment. Similar to our case study in Section 3.2, algorithms A₁ and Ag correspond to ResNet50 models pre-trained on ImageNet and ResNet50 models trained from scratch. The two additional algorithms control for (a) choice of pre-training data and (b) usage of pre-training. In partieular, algorithm Ag corre- sponds to R8 models pre-trained on ImageNet and algorithm A₁ corresponds to ResNet50 models pre-trained on synthetically generated data (Baradad Jurjo et al., 2021).
Dataset. We restrict our analysis to the majority group (i.e., landbirds land) of the WATER- BIRDS dataset. With this modification, we control for aggregate performance-moder trained with any of the four algorithms attain at least 97% accuracy on this group.
In other words, the learning algorithms described above output models with similar aggregate per- formance but differ in multiple axes; A, and A3 pre-train on the same dataset (ImageNet), but differ in model architecture; algorithms A1 and A4 pre-train but on different datasets; algorithms A and Ag differ in their usage of pre-trainn pre-training.
Explained variance. Recall that fraction of explained variance of given vector R in set of vectors {0, E [d] is the empirical variance of v 8, divided by the total amount of variance in (0:} (i.e., trace(Cov (Q,])) In other words, this measures what fraction of the total variation in {0:) is along the direction t.
Comparing direction-specifi explained variance. Recall that through our case study ImageNet pre-training in Section 3.2, we identified (and counterfactually verified) a "human face" distinguish- ing feature that impact models pre-trained ImageNet (algorithm Ai) but not models trained from scratch (algorithm A2). Here, we consider the fraction of datamodel variance explained by the training direction corresponding to the "human face feature across all four learning algorithms. Figure 26 shows that the fraction of datamodel variance explained by this direction is maximum under algorithms A1, followed by algorithm A3-bothh algorithms that pre-train ResNet models on ImageNet.
Explained variance (%) Direction (Human face)
ResNet50 pretrained on ImageNet
Datamodel
ResNet18 pretrained ImageNet
ResNet50 trained from scratch
ResNet50 pretrained on synthetic data
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Explained variance (%)
Figure 26: Fraction of datamodel variance explained by the training direction corresponding to the "human face" feature in Section 3.2. Each set of datamodel representations maps to one of four different learning algorithms, which differ in terms of architecture, pre-training, and/or choice of pre-training dataset.
39

---

Under review as a conference paper at ICLR 2023
Comparing top-A explained variance. For given set of training directions {UI, Uk the frac- tion of explained variance datamodel representations captures the importance of the training di- rections to model predictions for algorithm A, In Figure 27, compute the top-k principal com- ponents of datamodels corresponding to algorithms that pre-train on ImageNet-Ai and A3-and evaluate the fraction of explained variance in datamodel representations of directions and the learning algorithm A, As expected, the PC directions have high explained variance on datamodels corresponding algorithms that pre-train ImageNet and low explained variance on datamodels corresponding to algorithms that pre-train on synthetic data and train from scratch.
function of the number
Datamodee PCA on Resherl pretra ImageNet
Datamodel PCA on Restecll pretrained imagetiet
Oumullltion www.ulatiue explined veriance (84
CUTTA wowvlllvv oxpllined variance (N6
80
80
Datamodel
60
80
ResNet50 pretrained on ImageNet
ResNeel3
ResNets pretrained on synthtiic data ResNNtt0 trained from scratch
100
200
300
4.00
500
180
200
3.0
400
500
Top-k principal companentts
Top-k principal components
Figure 27: Given number of directions & and the learning algorithm A., compute the top-k prin- cipal components of datamodels corresponding to algorithms that pre-train on ImageNet-AA (left) and A3 (right)-and evaluate the fraction of explained variance in datamodel representations corre- sponding to algorithm A, These PCs have high explained variance on datamodels corresponding algorithms pre-trained on ImageNet and low explained variance datamodels corresponding to models pre- preitrained on synthetic data and models trained from scratch.
Note that these results are consistent with our findings (a) aggregate algorithm comparisons via datamodel cosine similarity in Appendix F.1 and (b) additional counterfactual analysis of distin- guishing feature transformations Appendix C.4.
40

---

Under review as a conference paper at ICLR 2023
F.6 ON COMPARISONS WITH MODEL PREDICTIONS AND PENULTIMATE LAYER REPRESENTATIONS
In this section, contrast our algorithm comparisons framework comparisons based model predictions and penultimate-layee representations. Note that there are no existing methods that can be directly reused for comparing learning algorithms the best of our knowledge. Therefore, we design experiments to evaluate whether model predictions and penultimate-layer representations can identify distinguishing training directions surfaced using our framework.
Model predictions. Through this experiment, we show that example-leve differences in predic- tions (Zhong et al., 2021; Meding et al., 2022) of models trained with different algorithms are not necessary to identify subpopulations analysed in our case studies. First, we re-run the first stage of our framework only on test examples on which models trained with different algorithm have the average. Then, we compare distinguishing training directions (i.e., output of the first stage) before and after controlling for prediction-leve agreement. Our results in Table show that for each case study, our framework identifies similar training directions (i.e., high cosine similarity) even after removing test examples on which model predictions differ. This experiment shows that our framework identify fine-grained differences between learning algorithms that persist even after controlling for prediction-level disagreement across models trained with different algorithms.
same prediction
Dataset Case study
Direction
(Absolute) Cosine Similarity
1iving17 / Data augmentation
A (Spider web)
0,999
B (Polka dots)
0.998
Waterbirds / ImageNNr pre-training
A (Yellow color)
0.977
B (Human face)
0.740
CIFAR-I0/ SGD hyperparameters
A (Black-white texture)
0,998
B (Rectangular shape)
0.999
Table 2: Distinguishing training directions before and after filtering out digh-disagreement test ex- amples exhibit high cosine similarity and surface subpopulations of images that share the same distinguishing feature.
Penultimatc-layer representations. Representation-base comparisons (Raghu al., 2017; Mor- cos et al., 2018b; Kornblith et al., 2019a) measure the degree to which different models' represen- tation can be aligned. Unlike datamodel representations, penultimate-laye aligned-coordinatt
representations are not representations do not share a consistent interpretation
of penultimate- across different models. So, we first introduce a variant based on penultimate-laye representations that has consistent basis. Specifically, similar to how the datamodel weight 0;p denotes the in- fluence of training example j over the prediction on test example i, we set 0, to equal the cosine similarity between the penultimate-layer representation of test example and train example We then compare two properties of datamodel representations and penultimate-
representations:
Effective dimensionality representations: We first compare the effective dimensionality (i.e., cumulative fraction of variance explained by top-k components) of datamodel representations and penultimate-layer representations g(r), Figure 28 shows that for all datasets and learning algorithms, datamodel representations have significantly higher effective dimensionality than the penultimate-layer alternative. For example, on CIFAR-10 10 data, more than 99% of the variation in penultimate- layer representations is captured by the first 10 components.
Explained variance of distinguishing training directions: We now re-run the first stage of our framework to compare distinguishing directions obtained via datamodel representations 0 and penultimate-Iun layer representations g(r). Here, we evaluate the extent to which these training di- rections distinguish models trained with different algorithms. Specifically, shown in Figure 29, we compare the difference in the cumulative fraction of variance explained by the top-k training directions across representations corresponding algorithms A, and Ag (higher the better). Fig- ure 29 shows that (a) training directions obtained from datamodel representations have signif- icantly higher gap in explained variance across learning algorithms and (b) directions obtained
41

---

Under review as a conference paper at ICLR 2023
from penultimate-layy layer representations can have close to negative gap in explained vari- ance across learning algorithms.
Effective dimensionalitt of (datamodel, penultimate-layer> representations
Dataset: CIFAR-10
Dataset: Living17
Dataset: Waterbirds
1.0
1.3
1.0
Cumulattve facc
Oumulative explined varsmoe tee
Gamulatio batir engaiind Voal Sarcac flat.
0.0
9.4
0.
0.
Ponultimate
Penutimmte
0.6
For Imagelett
Fenultimate rep.:
rep: sug
Ponutimate rep::
Datamodei:: High 5GD nolse
Datamodels: with aug
Datamodels: imageliet pre-tmihing
04
Detemodelic Low SGD noise
Detamodels: Without wag.
Delamodels: Trained from scratch
02
00
100
210
2013
407
500
105
200
530
4.00
300
120
2003
2003
500
Top-k pr Iicipall compenantts
Top-k principal compunents
Top-k c.p.kpprincie pwi.pppippppll componentts
Figure 28: Effective dimensionality (i.e., cumulative fraction of variance explained by top-k com- ponents) of datamodel representations significantly more than that of eenuttmmae-layee represen- tations across all datasets and learning algorithms considered in Section 3.
Dataset: C CIRAR110
Resitus High SED notse Low 550 noise
Resious Low SED nolse High 5G0 noise
Gce Kop wwg. var (ttc)
0.013
0.10
Representations Datamodels Penutimate
0.05
0.000
Dataset: Living17
Residua With bug. Without
Residual Without With aug
8:925
Representations Datamndes Perulimate
0:38
-0.32
-6.355
-0.00
prinipal
Dataset: Waterbinds
Residua = Imageliet pre ere-training- Trained from scratch
Residual Traned from scratch- mageltet pre-traming
0.20
0.15
Representatiors Dacamodets henuttmmtte-leee
0.30
190
0.00
Figure 29: Difference in the cumulative fraction of variance explained by the top-k training direc- tions across (datamodel or penultimate-layer) representations corresponding to learning algorithms Ai and Ay; higher the better. Top-k distinguishing training directions obtained from datamodel representations have significantly higher gap in explained variance across learning algorithms (c.g.. CIFAR-10, WATERBIRDS) and (b) directions obtained from penultimate-layer representations can have close to zero (e.g.. WATERBIRDS) negative gap (e.g., CIFAR- 10, LIVING17) in explained variance across learning algorithms.
42

---

Under review as a conference paper at ICLR 2023
F.7 EFFECT OF SAMPLE SIZE ON DATAMODEL ESTIMATION
In this section, we analyze the effect of datamodel sample size on datamodel estimation.
Setup. Recall from Appendix C.3 that datamodel training set of size corresponds to train- ing m models on independently sampled training data subsets. For our case study on ImageNet pre-training in Section 3.2, we estimate datamodels on WATERBIRDS data with 50, 000 sam- ples (i.e., m ResNet50 models trained on random subsets of the WATERBIRDS training dataset). In this experiment, we analyze how the estimated datamodels vary as function of sample size m E (5000, 10000, 25000, 50000).
Cosine similarity between datamodels. Our algorithm comparisons framework uses normalized datamodel representations to compute distinguishing training directions in the first stage. So, we first analyze the alignment between datamodel representations that are estimated with different sam- ple sizes. Specifically, we evaluate the cosine similarity between 0 m and o5 where vector ofm) E R |s| corresponds to the linear datamodel for example x estimated with m samples. As shown in Figure 30, the average cosine similarity between datamodels is greater than 0.9 even when the sample size is reduced by factor of 10, from 50000 to 5000.
Cosine similarity w.r.t datamodels estimated with 50000 samples
0.08
Sample size
0.07
5=000 (0.1x), mean: 0.92
0.06
1100000 (0.2x), mean: 0.95
Fracionn
0.05
25000 (0.5x), 0.98
0.04
0.03
0.02
0.01
0.00
0.05
1.00
Cosine similarity between datamodel vectors
Figure 30: Histogram over cosine similarity between datamodels @fmu) and o!ma), where vec- tor m)) !s corresponds to the linear datamodel for example estimated with m {5000, 10000, 25000, 5000} samples.
Explained variance of principal components. As discussed in Section 3, for a given training direction v, the fraction of variance that explained in datamodel representations (Ø(") } captures the importance of the corresponding combination of training examples to model predictions for algorithm A,, Here, we show that principal components of datamodel representations trained with smaller sample size (e.g., m 500) have similar explained variance on datamodel representations estimated with larger sample size, and vice-versa. As shown in Figure 31, the explained variance of the top-10 principal components of datamodels estimated with m E {500, 5000} have similar explained variance on datamodels estimated with m E (5000, 10000, 25000, 50000).
PCs of datamodels estimated with 5000 samples
PCs of datamodels estimated with 50000 samples
Fraction of explained variance
0.030
Sample size
Sample size
5000
25000
5000
25000
0.025
10000
50000
10000
50000
0.020
0.015
0.010
0.005
0.000
0
8
9
0
9
8thh principal component
8thh principal component
Figure 31: Explained variance of the top-10 principal components of datamodels estimated with m (500, have similar explained variance on datamodels estimated with sample size m e (5000, 10000, 25000, 50000).
43

---

Under review as a conference paper at ICLR 2023
G RELATED WORK ON MODEL INTERPRETATION
In the main paper, we focused our discussion of related work to those directly related to model comparisons. Here, we give discuss additional related work on interpretability and debugging of model biases.
Interpretability, explainability, and debugging. Stage II of our method hinges the interpretabil- ity of the extracted subpopulation. A long line of prior work propose different interpretability and explainability methods for models.
Local explanation methods include saliency maps (Simonyan et al., 2013; Dabkowski & Gal, 2017; Adebayo et al., 2018), surrogate models such as LIME (Ribeiro al., 2016), and Shapley values (Lundberg & Lee, 2017). Our method is similar to per-example based interpretability methods such as influence functions (Koh & Liang. 2017) that our interpretation based on data; however, our analysis differs from these priors methods in that it looks at entire subpopulations of inputs.
Global interpretability and debugging methods often leverage the rich latent space of neural net- works in order to identify meaningful subpopulations or biases more automatically. Concept activa- tion vectors and its variants (Kim et al., 2018; Abid al., 2022; Ghorbani et al., 2019) help decom- pose model predictions into set of concepts. Other recent works (Eyuboglu et al., 2022; Jain al., 2022) leverage the recent cross-modes representations along with simple models mixture mod- els and SVMs, respectively et al., 2021; Singla & Feizi, 2021) analyze the neurons of the penultimate layer of (adversarially robust) models identify spurious features. Our framework can be viewed as leveraging a different emhedding space, that of datamodel representations, to analyze model predictions.
identify coherent subpopulations
slices. Other methods (Wong
Robustness to specific biases. In applying our framework across the three case studies, we identify a number of both known and unknown biases. A large body of previous work aims at finding and debugging these biases: Priors works investigate specific biases such as the role of texture (Geirhos et al., 2019) or backgrounds (Xiao et al., 2020) by constructing new datasets. Leclere et al. (2021) automate many of these studies in the context of vision models with a render-based framework. Crucially, these works rely on having control over data generation and having candidate biases ahead of time, See the "Connections to prior work" within each case study for more specific references.
44


Reviews:
[{"review_id": "tcxPa8NmR0k", "reviewer": "Reviewer_UoE7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe authors present a way of comparing two learning algorithms by\nfiguring out which feature transformations/alterations, of test\ninstances, can alter the output (probability) of model trained by one\nmethod (on average) while not changing the output of another (as\nmuch).  This approach can shed light on how two different algorithms\n(such as whether or how to use pretraining, or data-augmentation, ...) can\ndiffer in specific learning problems, beyond just looking at model accuracy\n(summary performance numbers) and beyond just looking at the model\nbehavior (outputs) on a specific (test) instances, but getting insights into how features\nare being used differently (the patterns of emphasis on various features) by the two algorithms.", "summary_of_the_review": "\nThe approach is interesting in getting insights into how/why a model/algorithm behaves a certain way, \nin part by comparing it to another model/algorithm! (and going deeper than just looking at final output\non specific instances).  I raised a few issues (see strengths/weaknesses), but overall I am positive.\n\n", "strength_and_weaknesses": "Strengths:\n\n-- The contribution can be readily seen to be useful (diagnostics,\n could be important for troubleshooting).\n\n-- Mostly clear and easy to read paper.\n\nWeaknesses (More details after the list):\n\n-- Computational cost or complexity of the approach is not clear (eg\n   each of the 3 steps in stage 1), and may be too expensive.\n\n-- It would be good to have further experiments with regards to\n understanding the properties of the proposed technique:\n limits/weaknesses, strengths, say in discrimination power between two\n algorithms, etc.\n\n\n* Regarding computational cost: you say that the expectation of the\nmodel output on instances is taken with respect to random choises\nwithin algorithm when training as well as instance (Definition 2).\nBut it's not clear how many times you had to sample and retrain, and\nhow that impacts the variance (stability) over the results you\nobserved. I looked at the appendix, and that didn't help much.\n\n\n* With regards to how discriminating of your technique, among two\nalgorithms: If the two learning algorithms are very similar, does the\nmethod find a useful difference? can one set up controlled experiments\nchanging the degree of similarity? For instance, the accuracies of the\ntwo models can be nearly identical and relatively high, eg both at say\nnear 95%, and yet the two models could behave very differently.\n\nHow many dimensions of difference, via PCA, does the approach find?\n(as a function of problem and algorithm characteristics)\n\n-- Any thoughts on how your approach might help one diagnose a model's behavior (a single\n model) and remedy certain undesired model behavior?  (biases, etc).\n\ntypo:\n\nAppendix B.   \"small large rate\"  --> small learning rate \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FBIJhJ9FAv_", "reviewer": "Reviewer_QoXq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a technique for comparing two machine learning algorithms in terms of the most important features/data that influence the prediction of the model the most. The technique is based on the datamodel representation model (Ilyas et.al., 2022). Based on the distinguished features produced by the model, the technique utilizes human-in-the-loop distinguished feature transformation that represents the characteristics of the compared algorithms. The authors tested the proposed technique on three case studies: data augmentation, pretraining, and hyperparameter tuning.", "summary_of_the_review": "It's an interesting paper overall. But I have some concerns about its applicability. Therefore, I recommend weak rejection.", "strength_and_weaknesses": "Strengths:\n- Interesting technique in exploring the characteristics of the algorithm and the relation to the datasets for algorithm comparison.\n- The technique is applicable to multiple scenarios.\n\nWeakness:\n- The authors claimed that the technique is applicable universally across different learning algorithms and any domain. However, within the paper, the authors only discuss neural network models and limit the technique to computer vision problems. Even in the human-in-the-loop stage, the technique involves visual inspection, which makes the technique only applicable to computer vision problems.\n- The authors do not introduce any baselines in the experiments. Therefore, it's hard to assess the contribution of the paper completely. \n- The technique involves a human-in-the-loop stage for designing feature transformation. However, there is no clear explanation of how to perform the step.\n\n\n------\n==Post rebuttal==\n\nThanks the authors for the feedback.\nI have read the authors' feedback and other reviewer comments. The feedback addressed some of my concerns. However, some other concerns remain.\n\nI think one of the weaknesses of the paper is in the human-in-the-loop part, where it makes the method not easily applicable to other cases. The method requires the analyst to infer a feature shared by given a subpopulation of inputs. Depending on the application, this stage may be hard to perform.\n\nThe application discussed in the paper is also limited to computer vison task. However, the authors branded the paper as a 'unified framework' for comparing (any) learning algorithms. I think the authors should provide more example from other domains, like text/tabular data, or tone down the claim they made.\n\nTherefore, my recommendation remain the same.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2PvJbhSuf-", "reviewer": "Reviewer_1DZF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a method to find the differences of how differently trained models make predictions. More precisely, it gives human-interpretable features (or patterns) on how different models recognize different classes. The proposed method heavily relies on how influences of training data on test data are measured. Authors showcase three applications of the proposed method.", "summary_of_the_review": "Although being well-written, lacking enough discussion on related work and utility of the proposed method, I would like to reject the paper.\n", "strength_and_weaknesses": "Strength\n- The algorithm is simple and the paper is overall well-written.\n- Comparison among models, instead of studying a single model, is a promising direction for model interpretation.\n\nWeakness\n- This work seems like to be related to model interpretation, or explainable machine learning, because it gives a human-understandable interpretation, or explaination of how models predict specific classes. However, related literature reivew and comparions are missing.\n- The process seems to heavily rely on the method used in Stage I, 1. Authors also mentioned the influence function, why it is not used but the datamodel is used?\n- Stage II seems to be non-deterministic for the human-in-the-loop part. What if different results maybe given by the same output of stage I? This weakens reproducibility somehow.\n- Authors lack describing how the method would benefit users in the three application cases. After finding the transformations, how would they give hints on improving any aspects of each scene?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-0-PH4AXxp", "reviewer": "Reviewer_SMZu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a human-in-the-loop framework that first tries to find most distinguishable samples for each algorithm, and then let human summarize observable patterns from those samples. ", "summary_of_the_review": "In summary, the paper seems to be an incremental work built on top of the datamodel idea. However, some details are missing in the paper and the paper is an interesting engineering idea rather than a scientific idea. Therefore, I do not think the paper meets the acceptance bar. ", "strength_and_weaknesses": "Strength: The paper provides several applications of this method, which seems interesting.\n\nWeakness: It is unclear how exactly is the datamodel computed; It is unclear computation-wise whether this framework is practical in general settings; Regardless of computation front, it is unclear whether the framework can work in general settings, e.g., what if human can not find the patterns, does it also dependent on sample size? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject