Distinguishing rule- and exemplar-based generalization in learning systems
Authors: Ishita Dasgupta
Erin Grant
Thomas L. Griffiths
Venue: ICLR 2022 Submitted (2022)
Keywords: inductive bias
combinatorial generalization
cognitive psychology
robustness to spurious correlation

Abstract: Despite the increasing scale of datasets in machine learning, generalization to unseen regions of the data distribution remains crucial. Such extrapolation is by definition underdetermined and is dictated by a learner’s inductive biases. Machine learning systems often do not share the same inductive biases as humans and, as a result, extrapolate in ways that are inconsistent with our expectations. We investigate two distinct such inductive biases: feature-level bias (differences in which features are more readily learned) and exemplar-vs-rule bias (differences in how these learned features are used for generalization). Exemplar- vs. rule-based generalization has been studied extensively in cognitive psychology, and in this work we present a protocol inspired by these experimental approaches for directly probing this trade-off in learning systems. The measures we propose characterize changes in extrapolation behavior when feature coverage is manipulated in a combinatorial setting. We present empirical results across a range of models and across both expository and real-world image and language domains. We demonstrate that measuring the exemplar-rule trade-off while controlling for feature-level bias provides a more complete picture of extrapolation behavior than existing formalisms. We find that most standard neural network models have a propensity towards exemplar-based extrapolation and discuss the implications of these findings for research on data augmentation, fairness, and systematic generalization.

Under review as a conference paper at ICLR 2022
DISTINGUISHING RULE- AND EXEMPLAR-BASED GEN- ERALIZATION IN LEARNING SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the increasing scale of datasets in machine learning, generalization to unseen regions of the data distribution remains crucial. Such extrapolation by definition underdetermined and is dictated by a leamne''s 's inductive biases. Machine learning systems often do not share the same inductive biases as humans and, result, extrapolate in ways that are inconsistent with our expectations. We investigate two distinct such inductive biases: feature-level bias (differences which features more readily learned) and exemplar-vs-rule bias (differences in how these learned features are used for generalization). Exemplar- vs. rule-based generalization has been studied extensively in cognitive psychology. and, in this work, we present a protocol inspired by these experimental approaches for directly probing this trade-off in learning systems. The measures changes in extrapolation behavior when feature coverage is manipulated a com- binatorial setting. We present empirical results across a range of models and across both expository and real-worid image and language domains. We demonstrate that measuring the exemplar-rule trade-off while controlling for feature-level bias pro- vides more complete picture of extrapolation behavior than existing formalisms. We find that most standard neural network models have propensity towards exemplar-based extrapolation and discuss the implications of these findings for research on data augmentation, fairesss, and systematic generalization.
propose characterize
INTRODUCTION
Extrapolation or generalization- decisions on unseen datapoints always underdetermined by data; which particular extrapolation behavior an algorithm exhibits is determined by the algorithm's 's inductive biases (Mitchell, 1980). For modern deep learning systems, these inductive biases often deviate from those in humans. When the inductive biases of ML systems are opaque, and guarantees on extrapolation are not possible- as is often the case with many modern ML systems (D' Amour al., 2020)-220 about the system's operation. Cognitive psychology provides rich basis for experimental designs to study the often-opaque human cognitive system via its external behavior. These can be leveraged to distinguish between competing hypotheses about a machine learning system's 's inductive biases in the same manner (Ritter al., 2017b: Lake al., 2018; Dasgupta al., 2019).
instead turn empirical study of the behavior of system to derive principles
In this paper, we draw on methods from cognitive psychology to define a protocol that teases apart the different inductive biases that go into informing how an opaque learning system extrapolates outside its training distribution. We focus in particular on combinatorial generalization for classification in the presence of spurious correlation. Our protocol goes significantly beyond existing work by controlling for various confounds. We isolate two distinct kinds of inductive bias fass-feature-eevee bias and exemplar-rule bias- -hat have different effects on model extrapolation. We examine these inductive biases across models in expository points-in-a-plant setting, well as in naturalistic image and language domains. Finally, we discuss the implications of these inductive biases and their relation to previous work on data augmentation and spurious correlation.
Feature-level bias measures which features a system finds easier or harder to learn. This informs which feature system will generalize the basis of when both features are correlated confounded. This kind of feature-level bias has been studied extensively in human cognition (Landau et al., 1988; Hudson Kam & Newport, 2005). There has also been recent work directiyy inspired by these

---

Under review as a conference paper at ICLR 2022
observations
ratio of predictions
condition
training examples
extrapolation
humans (shape-biasedl
rule-based
exemplar-based
(mo feature blas)
(no feature blas)
cue
"dax"
"fop"
conflict
"dax"
"Fop"
"dax"
"fep"
"dax"
"lep"
zero shot
"dax"
"fop"
7
"dax"
"Fep"
"dax" "Tep"
"dax" "lep"
partial exposure
daxx
"fep"
"fep"
"dax"
"top"
dat
"lop"
"aax"
"Iep"
Figure Illustrative category learning experiment: Training examples from the 3 independent training conditions, the extrapolation test, and characteristic behavior for learners with different inductive biases.
cognitive psychology studies that examines similar biases in artificial neural networks, most notably the "shape-bias", the tendency generalize image category labels according shape rather than according to color or texture (Ritter et al., 2017a; Hermann et al., 2019; Geirhos et al., 2018). While there exists previous work examining specific feature biases in deep learning, we present a more general measure of feature-level bias as well as demonstrate how it interacts with but is distinct frothenother kind of inductive bias, viz. exemplar-vs vs-rule bias.
Exemplar-vs-rule bias measures how system uses features to inform decisions by trading off between exemplar- and rule-based generalization. rule- based categorization decision made on the basis of minimal features that support the category boundary (e.g. Ashby & Townsend, 1986), while an exemplar-based decision-maker generalizes a category on the basis of similarity to category exemplars (e.g., Shepard & Chang. 1963), and therefore may invoke many or all features that underlie a category. Extensive empirical work in cognitive psychology has found evidence of both kinds of generalization in humans (Nosofsky al., 1989; Rips, 1989; Allen & Brooks, 1991; Rips & Collins, 1993; Smith & Sloman, 1994). This trade-off can be understood intuitively as a continuum that varies the number of features employed to discriminate between categories (Pothos, 2005). This continuum also plays a role in representation learning systems such as deep neural networks (Hinton & Salakhutdinov, 2006), where feature selection automated.
AN ILLUSTRATIVE EXAMPLE
We first examine intuitive example that highlights the distinct inductive biases care about. Consider the category learning paradigm in Fig. (1). The stimuli vary along two feature dimensions, shape and color. Color determines the label of an object (i.e,, green objects "fep"), and shape is unrelated to the underlying category structure and acts as distractor. Participants (either humans artificial learning systems) are independently placed in three different conditions- cue conflict, zero shot, and partial exposure- -tht vary in coverage of the feature space. After observing the training examples, the participant presented with an extrapolation test consisting of an example outside the support of feature combinations observed during training (i.e., to classify the green circle as "dax" or a "fep." using arbitrary names to demonstrate that which feature relevant to the category boundary is not given). We explain below how differences in classification behavior on this extrapolation isolate feature-level bias as well as exemplar-vs-ru bias. We encourage the reader try the experiment themselves to examine their intuitions.
"dax"; purple
Cue conffict (CC, top row, Fig. (1)). The data presented in this condition confound color and shape (i.e., color and shape equally predictive of the category boundary). How system generalizes here directly measures its feature-level bias towards color or shape.
We leave to future work the details of mathematically formalizing the properties of statistical learners that result in exemplar-vs-rule bias. We instead focus the behavioral manifestations of this inductive bias and present empirical protocol measure even in opaque systems.
2

---

Under review as a conference paper at ICLR 2022
Characteristic behavior (right half of Fig. (1)). Since humans have an established shape bias (Landau et al., 1988), we expect that humans will classify the test item according to the object that shares its shape, not its color; in this case, as a "fep." However, this inductive bias is not shared by rule- and exemplar-base reasoners that have no a priori propensity for features, and are equally likely to classify the test item as "dax" or "fep."
Zero shot (ZS, middle row, Fig. (1)). This condition requires extrapolation to a new feature value "zero-shot" (i.e., without prior exposure). This setting is often used to examine out-of-domain (OOD) and compositional generalization in machine learning (Xian et al., 2018). Behavior in this condition reveals whether the model has learned the discriminating features and whether it can extrapolate to new feature values, and thus acts as a baseline.
Characteristic behavior (right half, Fig. (I)). Rule- and exemplar-basee behavior in this condition is confounded. A rule-based learner infers the minimal rule that color determines label, does not assign any predictive value to shape, and therefore classifies the extrapolation stimulus based on color as a "dax." An exemplar-base based learner categorizes based on the similarity along all feature dimensions of the extrapolation stimulus to category exemplars. Both training exemplars have no overlap with the test stimulus along the shape dimension, but the "dax" overlaps along the color dimension, and the learner categorizes it as a "dax." "
Partial exposure (PE, bottom row, Fig. (1)). Compared to zero shot, participants in this condition also receive "partial exposure" to a new feature value (i.e., circle) along the shape dimension. This setting is most similar to combinatorial zero-shot generalization (e-g.. Lake & Baroni, 2018a), where the learner is exposed independently to all feature values but has to generalize to a new combination.
Characteristic behavior (right half of Fig. (1)). This setting meaningfully distinguishes rule- and exemplar-base -aaed generalization. To understand the behavior of these two systems, we contrast this condition the cue-conflict condition. The addition of the purple diamond-shaped "fep" means the learner has seen both a diamond and a circle labeled "fep". A rule-based learner takes this as direct evidence that shape is not predictive of category label and classifies the extrapolation stimulus on the basis of color as "dax." This is typically also how humans extrapolate. This additional training example, however, does not impact an exemplar- based system, since does not share any features with the extrapolation stimulus. The exemplar- based reasoner classifies on the basis of feature-overlap with training exemplars and is therefore indifferent, exactly as in the cue-conflict condition.
From behavior to inductive bias. Feature-level bias is measured as deviation from chance perfor- mance the CC condition. Fxemplar-vs-rul bias is measured by the difference between performance in the PE and ZS conditions- difference indicates rule-based generalization, while the magnitude of the difference measures exemplar propensity. Pure exemplar-base reasoning implics no difference between the PE and CC conditions, while non-zero difference indicates partial rule propensity.
Implications. A purely exemplar based system doesn't 't learn decision boundaries that operate over minimal features. It instead favors decision boundary that weights all features. This is undesirable in domains where not all feature combinations will be observed, and systematic generalization to unobserved combinations is expected (Lake et al., 2018; Mareus, 2018; Arjovsky al., 2019). On the other hand, a rule-based system that applies the same category decision rules across all data regions might over-generalize, which is undesirable in some naturally occurring long-tailed distributions (Feldman & Zhang, 2020; Feldman, 2020; Brown et al., 2020). In such cases, flexible exemplar- based learning that generalizes based on dense similarity is preferable (Zhang et al., 2016; Arpit et al., 2017). The protocol we present allows us to empirically measure the exemplar-vs-rule bias of a learner and therefore navigate the exemplar-vs- vs-rule trade-off in various scenarios.
3 A PROTOCOL FOR EXAMINING INDUCTIVE BIAS
We embed the structure of the category learning problem discussed in Section (2) into a statistical learning problem that can be applied across domains to test black-box learners.
Problem setting, We consider a setting where inputs are a composition of categorical attributes (oracle setting in Andreas, 2019) with two latent binary features, Zdise. Zdists E {0, that jointly determine the observation x via some mapping g {0,1)² > X; see Fig. (2). These features can be derived from richer set, e.g., the median of continuous feature (see Appendix). We consider the
3

---

Under review as a conference paper at ICLR 2022
training condition
feature space xt
eue conflict
zero shot xh
partial exposure m.
extrapolation xy
m.
distractor (shipe)
discriminant (color)
Figure Formalizing the illustrative experiment: The experiment from Fig. (1) expressed in of formalism Section (3) with Zdist = color and Zdis shape. Background colors indicate true category boundary.
binary classification task of fitting a model f X' {0, 1) from given model family FF predict binary label for each observation. One of the underlying features, the discriminant, Adise- defines the decision boundary: the other one, the distractor, Zdist, is not independently predictive of the label.
This specifies a generative process x, Zdisc: Zdist p(x Zdise: Zdist) p(Zdise- Zdist). p(x Zdise Zdist) either generated (e.g., in Section (4)), or the empirical distribution of the subset of datapoints x with the corresponding underlying feature values (assuming access to these annotations, e.g., in Sections (5) and (6)). p(Zaise; Zdist) varied across training conditions, as outlined below.
Training conditions. The upper-right quadrant in all subfigures of Fig. (2), for which p(Zaisc acts as hold-out set on which we can evaluate generalization to an unseen combination of attribute values. We produce multiple training conditions with the remaining three quadrants of data by manipulating p(dise Zdis). All the analyses in this paper compare model extrapolation to the held-out test quadrant across various training conditions.
Zdis = 1)
To equalize the class base rates we balance all training conditions across the discriminant; i.e., we enforce p( ZZdis 0) pl Zdisc 1) 0.5. We also fix the number of datapoints across all conditions at N; With these constraints, we can control ZZdise Zdist) via two degrees of freedom: Zdise 0) (this implicitly fixes p(Zais
Zaisc 0) 1 To balance the dataset); and 1 = pppppee Ziiis = ddis: = 1).The The three conditions Section (2), as well as the held-out test set, correspond particular settings of To and (shown in Fig. (2), more in Appendix (A.2)).
To ddist
Measuring inductive bias. For given model family F, let p denote the result of selecting model from FF by training in the zero-shot condition, and similarly f and fCC, Feature-level bias (FLB) and exemplar-vs- vs-rule propensity (EVR) are measured as:
FLB(F) B[(A(y, PC (x)) 0.5,
EVR(F) B[A(y, f (x))) E[A(
(2)
where the expectation is taken with respect to the the data distribution under the extrapolation region (i.e., p(x, To = 1, = 1)). is the 0-1 accuracy. FLB takes values between -0.5 and 0.5 (indicating bias toward Zuis Zuiscs respec- tively); 0 represents no feature bias. EVR takes values between and (indicating rule-based and exemplar- based extrapolation, respectively).
10
Related formalisms and spurious correlation. This binary formula- tion of discriminant and distractor features has previously been studied in the context of spurious correlation (Sagawa et al., 2020). Rather than independently varying occupancy in the four quadrants, they directly manipulate the (spurious) linear correlation between the distractor and the discriminant features (Pmaj). In combinatorial feature spaces, scalar spurious correlation insufficiently specifies the data distribution. The linear correlation coefficient o between Zaise and Záist henceforth "spurious correlation" can be written in terms of mp and m::
0.6
Figure 3: Spurious correlation
data distributions (Fq. (3)).
To
+ W3
p(TO, R1) = /3(1
8
(3)
2
2
4

---

Under review as a conference paper at ICLR 2022
Different To and combinations can give equal p (see contours in Fig. (3), dots indicate points along the qqui-cooreltton contour that intersects with the PE condition: (mg = 0.5, म = 0.0, p 0.58)). In this paper we demonstrate that different data distributions with the same spurious correlation can result in vastly different generalization behavior to the under-represented extrapolation quadrant. This indicates that sensitivity to spurious correlation is underspecified inductive bias-we risk conflating conceptually distinct sources of inductive bias by focusing on this single metric. We argue for formulation like ours-based on manipulating feature combinations- that can tease apart distinct inductive biases: at the level of what features a system finds easier to learn (FLB) as well as how to use these features inform a decision boundary (EVR). We discuss how these biases can be interdependent, but capture distinct behaviors that sensitivity to spurious correlation cannot explain, thereby providing a more comprehensive picture of how a system generalizes.
EXEMPLAR-VS-ROE BIAS IN A 2-D CLASSIFICATION EXAMPLE
To illustrate our framework in a simple statistical learning problem, to quantitatively confirm the intuitions outlined in Section (2), we consider a two-dimensional classification problem. The feature dimensions are orthogonal bases in 2D space. We specify:
p(x Zeiicc; Edist) = (1,, 1.0);
p= x [22diisc - 1, 2.dii - 1]
(4)
where, as specified in Section (3), Zuisc: Zuiat E (0, 13. pl Zuise Zdiss) is determined by the training condition. Zdis: determines class labels, Zaia is a distractor, a is fixed at 3, and N = 300 datapoints are each class. The group with Zdist
Ziiicc = assigned the test set.
4.1 MODEL FAMILIES AND NOMENCLATURE.
Neural network (NN): We train feedforward ReLU classifiers with varying numbers of hidden layers and hidden units. We use the scikit-learn implementations with default parameters, run 20 times for confidence intervals.
Generalized linear model (GLM): Parametric models allow us to formalize the feature-sparsity that characterizes rule-based learners. Linear logistic regression is sparse by definition (it has access to only linear features). We generalize this model by cxpanding the feature space to include a nonlinear interaction & and examine L1 and L.2 regularization in GLM over this altered feature space.
Gaussian process (GP); Non- con-paraettric kernel methods allow us to formalize exemplar exemplar-based based generalization, where generalizations are made the basis of dense similarity to training data. A direct implementation of exemplar-based reasoning is only possible in the synthetic setting in which features which compute similarity are known. We examine the performance of GPs with Radial Basis Function (RBF) kernels. We fit the kernel length-scale (Rasmussen, 2003) (giving 5.2) as well vary it (adjusting "locality" in decision boundaries); GP:8.0 denotes lengthscale 8.0 GP.
4.2 COMPARING CUE CONFLICT, ZERO SHOT, AND PARTIAL EXPOSURE.
We consider one model from each class: NN with hidden layer, 2 units (NN:2h1d); linear GLM (GLM:lin): RBF GP with fitted lengthscale (GP:fit). The decision boundaries learned by these models are shown in Fig. (4a). Ziia, 7as are equivalent by design, and permit no FLB, CC is exactly at chance. This lets us focus on validating our novel protocol for measuring EVR without confounds. We generalize to cases with FLB in later sections. The GLM, sparse and therefore rule-based by definition, can only learn a linear boundary. It is therefore unaffected by the distractor dimension, showing no difference in extrapolation behavior between ZS and PE (zero EVR). On the other hand, the GP is exemplar baase by definition and displays a high EVR. The NN shows an intermediate EVR, more aul-bbsee than the sereel--xeepllr-beeee based GP but not entirely rule-based like the GLM.
4.3 THE INFLUENCE OF MODEL PROPERTIES ON THE EVR
We first examine EVR in our control model classes (GLMs and GPs) to validate that it tracks rule- vs exemplar-basee based extrapolation, followed by analyses of various NNs.
Regularized GLMS: EVR reduces with rule propensity, A key property of rule propensity is sparsity in feature space. A linear GLM (GLM:lin) is sparse by definition, examine GLM on
5

---

Under review as a conference paper at ICLR 2022
conflict
shot
pantial expesamm (PR)
GLM m
EVR(F)
GP
GLMHI GLM-12 GLM
GLM:lln
GP80
GP
GP.R
NN:22 AW-1chld NWsshild
model family J
NN
(b) EVR reflects exemplar-o-rule propensity both within and across model families. The EVR acmes model families, computed 20 runs,
bars represent 95% Cls. The GLMs are largely hased and show low EVR, Even within Ms. sparsity regulariza- tion gives lower EVR. GPs are largely exemplar-based and show high EVR, Even within GPs, more 'local' GPs with lower lengthscales have higher EVR. NNs lie in-between, with larger NNs giving higher EVRs.
(a) Decision boundaries averaged across 20 runs. Training data- points are green or purple by label: lest are white.
Figure 4: Simple 2-D classification (Scction (4)) The spocific model used in (a) are bolded in (b).
expanded feature set so we can manipulate this sparsity. The additional feature + x Zdist Zudisc is computed by taking the product of the observed features and normalizing by a. We compute EVR for this GLM with different regularizers (regularization weight 1.0), shown in Fig. (4b).
GLM with no regularization (GLM:0) displays significant EVR. L2 regularization reduces it but LI (which directly induces feature sparsity") brings it to zero (or perfectly rule-based). This shows that a low EVR tracks rule propensity i.e., feature-level sparsity in model.
Lengthscales in GPs: EVR increases with exemplar propensity. A sufficient condition for exem- plar propensity is the locality of decision boundaries. We can directly manipulate this in GP with its lengthscale. We evaluate EVR in GPs with RBF kernels of different lengthscales in Fig. (4b). We find that the EVR lowest with high length-scales and grows as the lengthscale reduces. This shows that a high EVR tracks exemplar propensity, i.e., having local decision boundaries.
NNs: The necessary but not sufficient role of expressivity. The results from GLMs and GPs above indicate that some ways to reduce expressivity (Ll regularization in GLMs and high lengthscale in RBF-GGs) encourage rule propensity over exemplar propensity (thereby a lower EVR). We manipulate the most common variable in NN expressivity-iit size.
We increase the width of an NN with fixed depth of (Fig, (4b)) and find that the EVR increases. A deep NN with the same number of units, however, shows comparable EVR to wide network. Deeper networks with the same number of units are more expressive than wide ones (Raghu et al., 2017), indicating that excess expressivity-whin necessary-is not the sole driver of EVR.
4.4 EVR IS DISTINCT FROM SENSITIVITY TO SPURIOUS CORRELATION
A crucial difference between the ZS and the PE conditions is that the PE condition creates (spurious) correlation p = 0.58 between Zaisa and Zuise. Is sensitivity to this spurious correlation (p) the sole the driver of the difference in performances between the PE and 7S conditions, i.e., of the EVR? We show that this is not the case; the EVR is measuring something distinct. As described in Section (3), there are multiple data-settings with the same p. We consider training conditions specified by other To, that have the same as the PE condition (dots along the solid contour in Fig. (3)). We find that performance on the extrapolation quadrant after training on these new data distributions is much higher (and closer to ZS performance) than when trained on the PE condition even though p is exactly the same. This indicates that performance on the PE condition (normalized by ZS performance to give the EVR) is uniquely indicative of something different from sensitivity to spurious correlations measures the inductive bias toward exemplar-vs-rue based extrapolation.
Weight sparsity from I.I I.-rrugulrr is equivalent feature-sparsity only in special cases, including GL.M.

---

Under review as a conference paper at ICLR 2022
We also find that different ways to reduce P give different extrapolation behavior (e.g., increasing I is more effective than reducing To, see Appendix for details). This has implications for data manipula- tion methods (e.g., subsampling augmentation) that manipulate this p to control extrapolation. This also supports that spurious correlation alone cannot explain extrapolation behavior, highlighting the importance of FLB and EVR that measure behavior under different feature combinations in training.
Conclusions. The EVR tracks exemplar-vs rule-based extrapolation, as validated by evaluating on interpretable models like GLMS and GPs. It decreases with reductions in expressivity mediated by regularization and lengthscale. The EVR in NNs also decreases with (some kinds of) expressivity: more sensitive to increases in width than depth. Finally, sensitivity to spurious correlation cannot explain the EVR. These results lay groundwork for future theoretical work in formalizing the EVR.
CREATING COMPOSITIONALITY BY ADDING DISTRACTORS TO IMDB
In this section, we show how we can apply our approach to any classification task. We consider sentiment analysis on Internet Movie Database Movie Reviews (IMDb) (Maas et al., 2011).
Selecting features. The sentiment label ("positive" or "negative") is the discriminant Zaisc- We manufacture an orthogonal distractor Zdist as the presence/absence of a word that a) occurs in roughly 50% of the sentences in the dataset and b) does occur more frequently for either positive or negative reviews. Some examples are "film" and "you": we use the word "film," see Figure 5.
Models. We train single layer LSTM (20 hidden units; default hyperparameters) on each condition and test on the held-out quadrant.
We exclude models that do not reach 80% validation accuracy.
Tacishands
Ag tin gene,, the
distractor (cuntains Tilmm")
Feature-level bias. The distractor Zaia is easier to learn than the discriminant zdisc. as reflected in the CC condition (19.7% FLB -0.3): the system preferentially classifies on the basis of the distractor.
esseci
Whur sortos
modee bed nolt rrelidkeon
Exemplar-rule bias. We see good performance in ZS (84%): despite never having seen the word "film," the system can generalize to reviews containing it. The performance PE drops significantly (30.1%) 1)) giv- ing a large EVR (EVR = 0.54), indicating exemplar-based based reasoning. Note that the condition has access to superset of feature coverage compared to 7S, and contains all the same information required to generalize well. However, the ceemplar-bssed tendency to nonethe- less index on additional features (e.g., the presence of the word "film") hurts performance on the extrapolation quadrant. Performance in PE is higher than in CC, indicating that the system can learn to use the discriminant (i.e., it is not purely relying on FLB).
discriminant (seatiment)
Figure 5: Example stimuli from the IMDb dataset.
A COMPOSITIONAL DATASET DERIVED FROM MULTI-LABEL CELEBA
We now test our protocol on a standard classification task on a large-scale image dataset, CelebFaces Attributes (CelebA) (Liu et al., 2015). Each image in this dataset is labelled with 40 binary attributes. each of which can be leveraged as discriminant or distractor. We examine FLB and EVR for standard models across different feature pairs, and discuss the practical implications of our findings.
Selecting features. We select feature pairs that split the data roughly evenly and thus maximizing the number of training datapoints in each quadrant. We carry out our analyses across range of feature pairs; an example is depicted in Fig. (6a), and further details are in the Appendix.
Models. We train ResNets of various depths ({10, 18, 34)) and widths ((2, 4, 8, 16, 32, 644) on 6 different choices for feature pairs, with standard hyperparameters. We limit our analyses to networks that achieve at least 75% validation accuracy ensure that, despite differences in data variability across training conditions, all models learn a meaningful decision boundary.
Feature-level bias. There is wide range of FLB in the feature pairs; e.g., "male" is easier to learn than hhgh checkbones" giving high FLB, "mouth open" and "wearing lipstick" are equally difficult and give FLB of close to O. These FLBS were consistent across ResNet widths and depths.
7

---

Under review as a conference paper at ICLR 2022
Exemplar-rule bias. We observe good ZS performance: the models generalize to new feature values outside the training support. We see a wide range of EVR across feature pairs, Fig. (6b). Across all feature pairs, the EVR is non-negative: generalization in the PE condition is always worse (or not significantly better) than in the ZS condition. Further, we see a linear correlation between EVR and FLB in logit space across feature pairs. EVR therefore depends on how easy or hard the features are to learn. The key, however, is that this regression of the EVR onto FLB has a positive intercept: there a positive EVR even for feature pairs with no FLB. There is tendency for lower performance in PE compared to ZS (a nonzero EVR, exemplar propensity) even when FLB is controlled for.
We find no differences in EVR across ResNet widths and depths: Fig. (6b) plots EVR and FLB averaged over ResNet sizes smodel-specifie results in Appendix). One explanation is that the features in CelebA are complex; to learn these, we need reasonably high model expressivity, and differences in parameter count do not further modulate EVR. This is consistent with findings in Section (4) where expressivity is necessary but not sufficient for increases in EVR: we see a jump in EVR going from NN:2hld to N6:16h1d, but no further change going to the even more expressive NN:4h4d.
Controlling spurious correlation. We replicate the findings in Section (4): the EVR cannot be explained by sensitivity to spurious correlation p. This is shown in Fig. (6c), where we substitute performance in the PE condition with performance in a different data-setting (To = 0.825, To = 0.25) with the same م 0.58 as in the PE condition. We find none of the effects discussed above, indicating v--rule propensity-whic is not
that the PE condition is measuring something accounted for by sensitivity spurious correlation. Further, EVR does not increase with model expressivity, unlike sensitivity to spurious correlation (Sagawa et al., 2020).
Practical implications of the EVR. The nonzero EVR, i.e., ceemplar-basedieg models are better at extrapolating zero-shot to a new feature value than when they have partial exposure to that feature value, even though the additional data need not change the learned decision boundary. The training examples added in PE can be classified with the decision function from ZS without incurring additional training loss. A rule-based system recognizes this and bases its generalization on the minimal features that support the category boundary, i.e., the ones also learned in the zero-shot case. However, an exemplar-ban based model changes its decision boundary in response to this additional data. This has implications for fairness, as we outline with example features below.
reveals that our
PE-approximating data distributions (To R 0.5, गं RV 0.0) occur naturally. For example, as Sagawa et al. (2020) observe, "blond" "male"s are under-represented in CelebA. Consistent with the rest of our results, we find better classification for the extrapolation quadrant (blond males) if we discard data from an adjacent quadrant (blond non-males, condition, as opposed to the PE condition if such data is included: ResNet10, width 2, gives 7S 75.12 + 3.09% PE 60.22 + 7, 7.27% for Zdis: ="male" (discard blond non-males to get ZS) and ZS = 68.18. + 3.34%; PE = 49.78 + 3.76% for Ziise =bllndd' (discard non-blond males to get ZS). Previous sub-sampling approaches (Sagawa et al., 2020; Haixiang et al., 2017) that manipulate spurious correlation underspecify feature distributions (in the sense that there are many ways to alter feature distributions to change spurious correlation (Fig. (3))) and the aforementioned results demonstrate that these distinctions are important in determining extrapolation behavior to held-out feature combinations. Moreoever, differences in extrapolation behavior between the PE and 7S conditions, in particular, give insight into a system's 's exemplar-vs-rut bias.
non- oon-blond males) simulating the zero-shot
7 RELATED WORK AND FUTURE DIRECTIONS
Model design for systematic generalization. Rule-based generalization permits systematic extrap- olation in combinatorial domains. This systematicity has been found lacking in neural networks (Lake & Baroni, 2018b; Barrett al., 2018), leading to renewed interest in hybrid symbolic-connectiinnn architectures (Garnelo& & Shanahan, 2019). However, works evaluating, proposing, new algo- rithms for systematic generalization have not performed a thorough investigation of how feature co-oceurrences modulate extrapolation behaviors. Evaluating the exemplar-rule trade-off in these models by evaluating their EVR is a promising future direction.
Learning causal features. Feature sparsity, exemplified by rule-based generalization, is equivalent to learning causal features under the assumption that the causal model is the simplest model that explains the data. Recent work has investigated data settings that permit the separation of causal
8

---

Under review as a conference paper at ICLR 2022
no foutum-desrl
no featureedeeel
distractor (roouth open)
EVR(F)
discriminant (wearing lipstick)
raleshsord
(a) Example CelebA stim- uli: we test 6 discriminant- distractor pairs (with sulfi- cient coverage quadrants in Fig. (2)) of features.
FLB(F)
FLB(FF)
Main effect, logit scale: Exempllar-s- propensity (EVR) trends lincar with non-zero in- lercept (lit in red) in feature-level bias (FLB).
(c) Control, logit scale: Performance deviation
from ZS under equi-correlatoo interpolation close to zero; linear lit (red) has intercept at zero.
Figure 6: CelebA results, Stimuli and results on various feature pairings from the CelebA domain (Section (6)). Error bars represent 95% confidence intervals across ResNets of various sizes. See figure sub-captions and main text for details.
features from spurious correlations (Arjovsky et al., 2019; Bengio et al., 2019; Hill et al., 2019). Here, demonstrate that a model with exemplar propensity makes more rule-based ("causal") extrapolation for certain training feature combinations (i.e., zero shot as opposed to partial exposure). Investigating how feature coverage impacts causal generalization is fruitful future direction.
Simse generalization and kernels. We use similarity- based kernels such as the RBF to exemplify exemplar-based extrapolation. Recent work has interpreted neural networks as kernel regression (Jacot et al., 2018). Theoretically formalizing what NN properties, under a kernel framing, lead to being exemplar-base based (as behaviorally tracked by the EVR) an exciting future direction.
Data augmentation. The EVR demonstrates that increased data variation in the form of feature coverage worsens systematic generalization. The negative effect of data variation on generalization has been documented for adversarial augmentations (Raghunathan et al., 2020). Our results show that this phenomena persists even when augmentation is not adversarial (not maximizing a loss), rendering it generally relevant for the design of data augmentations (Perez & Wang. 2017).
8 CONCLUSIONS
Taking inspiration from- and going psyccolooiil studies, we design behavioral protocol to distinguish the effects of two inductive biases (FLB and EvR) that is easily applicable any classification domain. This follows in promising line of recent work that analyses and interprets deep learning systems based on their external behavior (Ritter et al., 2017b; Dasgupta et al., 2019). It complements other approaches that follow in the neuroscience tradition of analyzing internal representations (Zeiler& & Fergus, 2014; Karpathy et al., 2015), or make approximations of these internal workings to support theoretical results (Jacot al., 2018: Li & Liang, 2018; Allen-Zhu et al., 2019; Du et al., 2018). The behavioral approach has the advantage that it makes no assumptions about the model, allowing comparisons across models that differ in complexity and architecture.
Both rule- and exemplar-based extrapolation are valuable depending on domain, underscoring the importance of diagnosing this bias in systems. We show that this inductive bias is distinct from sensitivity to spurious correlation (Sections (4) and (6)), highlighting the impact of imbalanced feature combinations on extrapolation in neural networks (as captured by FLB and EVR).
Finally, the EVR is also a novel empirical phenomenon. As demonstrated in two real-world domains, we find that more feature coverage (as in PE compared to ZS) hurts generalization for exemplar basse models. This has implications for methods that manipulate data distributions to improve performance; e.g., data subsampling (Haixiang et al., 2017), data augmentation (Perez. Wang, 2017; Antoniou et al., 2017), and contrastive learning (Chen et al., 2020). Since an exemplar- based model is easily distractor feature, EVR is potentially useful as a diagnostic in application
biased by the inclusion settings (e.g., fairness) in which the goal is to control algorithmic behavior non-representative factors of a dataset (i.e., algorithmic bias Mitchell al. (2019)).
A limitation of the present work is that do not provide conclusive answer as to what properties of a model family influence the EVR. A broader study on these controlling factors (e.g.. hyperparameters, optimizers), and theoretical work formalizing this effect, are exciting avenues for future work.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We evaluate reproducibility according to the criteria from Pineau et al. (2020) in Appendix (D). We additionally provide source code in zip file supplementary material.
REFERENCES
Scott W Allen and Lee R Brooks. Specializing the operation of an explicit rule. Journal of experimental psychology: General, 120(1):3, 1991.
Zeyuan Allen-Zhu. Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252 PMLR, 2019.
Jacob Andreas. Measuring compositionality in representation learning. arXiv:1902.07181, 2019.
arXiv preprint
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint
2017.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 0028990,........ 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242. PMLR, 2017.
Gregory Ashby and James Townsend. Varieties of perceptual independence. Psychological Review, 93(2):154, 1986.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pp. 511-520, 2018. URL https::tp/.o
wwwp:iii.orr//aaooeoe / 807. 04225.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Séhastien I Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv:1901. 10912, 2019.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When memorization of irrelevant training data necessary for high-accuracy learning? arXiv preprint arXiv:20/2.0642// 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:001..03395. 2020. URL https: /ttp///axvvvv//a///////ea./aaav/ org/ abs 20111.01111111 03395.
Ishita Dasgupta, Demi Guo, Samuel J Gershman, and Noah D Goodman. Analyzing machine-learned representations: A natural language case study. arXiv preprint arXiv: 1909.0588, 2019.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes 2018
over-parameterized neural networks. arXiv preprint
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Under review as a conference paper at ICLR 2022
Marta Garnelo and Murray Shanahan. Reconciling deep learning with symbolic artificial intel- ligence: representing objects and relations. Current Opinion in Behavioral Sciences, 29:17- 23, 2019. ISSN 2352-1546. doi: http::/oooooggll....ff....eeea.. 12.001. URL https: //www. sci encedi rect. wtt.c//occcccieeee http://ccooc..eeeeeeaaaaaaaaaaaa 53300 943. Ar- tificial Intelligence.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 12231, 2018.
Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue, and Gong Bing. Learning from class- iaaas-sebal data: Review of methods and applications. Expert Systems with Applications, 73:220-233 2017.
Katherine L Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. arXiv preprint (tt:::911.09077 2019.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. arXiv preprint arXiv:1910.00577,, 2019.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 3 31365786):044-077.22
Carla L Hudson Kam and Elissa L Newport. Regularizing unpredictable variation: The roles of adult and child learners in language formation and change. Language learning and development, 1(2): 151-195, 2005.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization neural networks. In Advances neural information processing systems, pp. 8571-8580, 2018.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. Proceedings of the 3rd International Conference on Learning Representations, Workshop Track, 2015.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of pp. 2873-2882. PMLR,
recurrent networks. In International Conference Machine Learning, 20.
Brenden M Lake and Marco Baroni. Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. Proceedings of the 6th International Conference on Learning Representations, Workshop Track, 2018b.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2018.
Barbara Landau, Linda B Smith, and Susan S Jones. The importance of shape in early lexical learning. Cognitive development, 3(3):299-33) 321, 1988.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent structured data. arXiv preprint arXiv:1808.0123 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 150, 2011.
Gary Mareus. Deep learning: A critical appraisal. arXiv preprint arXiv: 2018.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman. Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-229, 2019.

---

Under review as a conference paper at ICLR 2022
Tom M Mitchell. The need biases in learning generalizations, Department of Computer Science, Laboratory for Computer Science Research, 1980.
Robert M Nosofsky, Steven E Clark, and Hyun Jung Shin. Rules and exemplars in categorization, identification, and recognition. Journal of Experimental Psychology: Learning, memory, and cognition, 15(2):282, 1989.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint
2017.
Joelle Pineau et al. The machine learning reproducibility checklist, v2.0, 2020. URL https: //www.ca cs aeggill... http://pp.aaaaappppprooooaaaaaaa http://wwcb.bbttttccccccii-t---- pdf.
Emmanuel M Pothos. The rules versus similarity distinction. Behavioral and brain sciences, 28(1):1,
2005.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. international conference on machine learning, pp. 2847-2854. PMLR, 2017.
Aditi Raghumathan, Sang Michael Xie, Fanny Yang. John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002.10716. 2020.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Lance J Rips. Similarity, typicality, and categorization. 1989.
Lance J Rips and Allan Collins. Categories and resemblance. Journal of Experimental Psychology: General, 122(4):468, 1993.
Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: shape bias case study. International Conference Machine Learning, pp. 2940-2949, 2017a.
Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. International conference on machine learning, pp. 2940-2949. PMLR, 2017b. URL https: /ttp://aaavvv./ee owt.org/////....
httpp/////wwwwwoo/aa//////////// 1talbs/1100 0866.........
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356. PMLR, 2020.
Roger N Shepard and Jih-Jie Chang. Stimulus generalization in the learning of classifications. Journal of Experimental Psychology, 65(1):94, 1963.
Edward E Smith and Steven A Sloman. Similarity- rule-based categorization. Memory & Cognition, 22(4):377-386, 1994.
Yongqin Xian, Christoph H Lampert, Bernt Schicle, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-222 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. European conference on computer vision, pp. 118-833,20 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1611vv1611.0333 2016.
12

---

Under review as a conference paper at ICLR 2022
training condition
feature space xt
eue conflict
zero shot xh
partial exposure
extrapolation
m.
m.
We 1.0,
distractor (shipe)
discriminant (color)
distructor (ara
011
discriminant (x1
Figure We expand Fig. from the main by including realization the abstract training conditions simple 2D points-in-Be in-a-planc setting. (Top) Formalizing the illustrative experiment: The experiment from Fig. (1) expressed terms of the formalism in Section (3) with Adia color and kasse = shape. Background colors indicate true categury boundary. (Bottom) The conditions realized via binarization of continuous feature values. Here, the discriminant is binarized as > 0 and the distractor input.
0; this setting is further investigated Section (4), Color here depicts the label but part of the
A ADDITIONAL FORMALIZATIONS
A.1 GENERALIZING THE FRAMEWORK FROM TWO BINARY ATTRIBUTES TO MANY CATEGORICAL ATTRIBUTES
In the most general terms, we consider setting in which each observation x X' is underlied by categorical variables 21...., 2n E (O..... C} with C Z4. henceforth attributes whose concatenation z = (z1,..., xn.) determines the observable input x via some mapping g Z + X. We consider the binary classification task of fitting model ff X {0. 1} from given model family F to predict a binary label for each input. A subset of the attributes in z, without loss of generality (=o, -.. 2;), is taken to define the decision boundary, while the remaining attributes, 2(+1, Zn, are assumed to not be independently predictive of the true classification y E (0,1). 1). We therefore denote the discriminant, Zdisc (zo, ................................ 21), and the distractor Zdist (24+1,.... 2.). For simplicity, we assume that the attributes are binary (i.e., C' 2 and E {0, 1}, Vi), and that the discriminant attributes must be jointly active for the classification to change from the null class y (i.e., = > Zdisc 1); the latter simplification allows us to redefine Zuisc 2g A ... A and Zaia = 2141 ... Zn. which is equivalent to the earlier discussion of illustrative two-attribute case.
A.2 TRAINING CONDITIONS EXPRESSED IN TERMS OF THE JOINT DISTRIBUTION
express training conditions displayed Fig. (2) and realized Figure terms of the joint distribution instead of the parameters No. Ti.
1. The cue-conffict condition the upper left and lower right quadrants in Figure and defines the distribution of attributes as
Pee Zuise
0, Zdist
1) = 0.5 (Zuisc 1, Zdist 1)
Pee Zddssc
0, Zdist
0) = Pee (Zdisc 1, Zdist 0) 0.5
2. The zero-shot condition populates the bottom left and right quadrants Figure 6 and defines the distribution of attributes
13

---

Under review as a conference paper at ICLR 2022
Pis Zdise = 0, Zdist 1) =
Pzs (diss = 1, Zdist = 1)
Pas Zdise
0, dist 0) 0.5
Pzs Zdiss
Zdiitt
0) 0.5
3. The condition populates all quadrants but the upper right in Figure 6 and defines the distribution of attributes as
Ppe Zrise
0, Zuisst 1) = 0.25
(2dise 1, Zidist = 1)
Pee Zulic O, Zdist
0) 0.25
Ppe (Zdisc
1, Zdist = 0) = 0.5.
B MORE CELEBA RESULTS
We include model-specifie results, split by ResNet depth and width, in Fig. (8). We find no systematic relationship between exemplar- vs. rule-based generalization and depth or width.
ResNet10
ResNet18
ResNeI34
witt
Witt
With
logg
logit
loght
Figure 8: CelebA EVR and FLB across feature pairs, averaged across 30 runs, split by depth and width of ResNet.
C SPURIOUS CORRELATION UNDERDETERMINER FEATURE DISTRIBUTIONS
The partial-exposune condition (To 0.5, क. = 0.0) in Section (2) results in a spurious correlation between the discriminant ddss and the distractor diss (p 0.58). examine behavior wider range of data settings, vary To and и described in Section (3), thereby also changing the degree of spurious correlation.
I. Interpolation towards zero shot. We interpolate from 0.5 towards 0.0, keeping 0.0.
This moves us closer to To = πi = 0.0, where we have no exposure to Zdisc 1 in training. Intuitively, condition.
reducing the exposure to the new distractor feature value from the -xppial-xpposure
II. Interpolation to full exposure. We interpolate π from 0.0 towards 0.5, keeping mo 0.5. This moves us closer to Ap Ti = 0.5, where we have equal exposure to all quadrants in training. Here, rather than reducing the exposure the new distractor feature value, we are equalizing the exposure to it across the discriminant dimension.
III. Interpolation with matched correlation. We report results on this in Sections (4) and (6). As also depicted in Fig. (3), we generate training conditions by changing and иı such that we follow a p-contour away from the partial-exposure condition (To = 0.5, कं 0.0, & 0.58): solid contour in Fig. (3). We also match the spurious correlation across the two interpolations in Appendix (C)A and Fig. (10) shows these additional p-contours as dashed lines.
These different interpolations are depicted in Fig. (10a) with different shape/colors.
C.1 GENERATING INTERPOLATION POINTS
We generate points along all three interpolation lines: from partial exposure towards zero shot ((C)D; from partial exposure towards full exposure ((C)II); and the equi-correlation line originating from partial exposure (II)III). The interpolating points along each line are selected to balance spurious correlation and feature exposure. In particular, we follow the following procedure:
1. We choose a point that interpolates towards full exposure. We do this by choosing a value of mi between 0.0 and 0.5, This gives data setting, along with a corresponding spurious correlation, computed via Eq. (3):
mn 0.5
FE
p == p (0.5, _ FE
14

---

Under review as a conference paper at ICLR 2022
partial exposure
interpolant
interpolant #2
0.5,m = 0.0,
0.32, my 0.0,
0.125, = 0.0,
0.58
p= 0.436
0.208
zero-shot interp. (C)l
-5
0.5, 11
0.0.
Kg 0.5. IT' = 0.1.
No 0.5. W1 = 0.25.
p 0.2588
full-exposure interp. (C)II
-5
0.5, f1 0.58
0.0,
0.06, #1 0.1,
No 0.825, T1 = 0.25
P
p 0.58
oqui-correlation interp. (C)llI
Figure We visualize several of interpolants used the interpolation analyses.
2. We generate corresponding point that interpolates towards zero shot. Given the data setting above, we set mi 0.0 and compute the mn to produce the same p as the full-exposure interpolations in Step 1. This gives the data setting:
To
75
n
FE
ল 0.0;
P p= p-p(mT
π 0.0) (0.5,
3. Finally, also derive the equi-correlation interpolation from the full-exposure interpolation as follows. We retain from the full-exposure condition, but recompute the such that the correlation matches the spurious correlation of the pure glspec (p 0.58). This gives an additional data setting:
( FF)
P (0.5, 0.58
Note that, despite there being three different interpolation lines, the specific interpolants we use are constrained along single degree of freedom freedom-choosing FE (Step 1). The data settings for zero shot (Step 2) and equi-correlation (Step 3) are derived from this value.
C.2 SPECIFIC INTERPOLATION VALUES USED
For all data settings, generate points along the interpolation lines using the procedure in Ap- pendix (C.1).
For the simple 2D classification setting, examine two interpolants. In this simple domain, we keep the interpolation distances small, since we expect changes in extrapolation behavior even from small changes.
15

---

Under review as a conference paper at ICLR 2022
zemo-shot inserp (C)A
12S equa
full-exposuee interp. (C)B
equi-corelation interp (CXC
0.25
0.25 0.38
0.5
interpolation distance
interpolation distance
(v (0.5 xo) + x?)
( - mg)² + m?)
(a) A heatmap of the spurioans correlation (Eq, (3)), showing different interpolation. The partial-exposure condition is identified with star outline; points along the three interpolation line identified with filled shapes.
for 66:16hld 2D classification of points-in-a-plane
Interpolations
(c) Interpolations, ResNet-
18 CelehA ("weuring lip
stick", "month open")
Figure 10: Interpolations away from the PE: changes extrapolation behavior under data distribution with the same spurious correlation as in PE, as well as different ways to change spurious correlation.
interpolant 1
interpolant 2
p
T
p
interpolation to zero shot ((C)I)
0.481
0.0
0.563
0.32
0.0
0.436
interpolation to full exposure ((C)II)
0.5
0.01
0.563
0.5
0.1
0.436
equi-correlation interpolation ((C)III)
0.519
0.01
0.58
0.661
0.1
0.58
For CelebA, we increase the interpolation distance to reflect the wider range of natural data distribu- tions among feature pairs. The data these interpolation values generate is visualized the equivalent points-in-a-plane setting in Figure 7.
interpolant 1
interpolant
interpolation to zero shot ((C)I)
0.32
0.0
0.436
0.125
0.0
0.258
interpolation full exposure ((C)II)
0.5
0.1
0.436
0.5
0.25
0.258
equi-correlation interpolation (CCIIII
0.66
0.1
0.58
0.825
0.25
0.58
C.3 INTERPOLATION ANALYSES
C.3.1 IN THE 2-D CLASSIFICATION EXAMPLE
In the simple setting from Section (4), we vary To. for an NN model (NN:16hld, the NN with lowest ER level overall). Results are in Fig. (10b) and discussed below.
EVR * sensitivity to spurious correlation. As also discussed in the main text, along the equi- correlation interpolation, the "effective EVR" drops drastically (i.e., the learner generalizes in more rule-based manner) despite no change in spurious correlation.
Implications for controlling extrapolation. Despite both having the same p. interpolating towards full-exposure increases the EVR more than towards zero-shot. This further supports that spurious correlation cannot fully characterize extrapolation behavior. This shows that different ways to reduce p have different effects extrapolation, and has important implications for data manipulation methods (e.g., subsampling or augmentation) that aim to directly control this p.
C.3.2 IN CELEBA
We see the same effects as in the linear setting: as also discussed in the main text, we see much smaller gap to the ZS condition despite change in spurious correlation. We don't 't find clear effects distinguishing different ways to reduce spurious correlation (interpolation to zero shot ((C) and interpolation to full exposure ((C)II)..
16

---

Under review as a conference paper at ICLR 2022
D REPRODUCIBILITY DETAILS
We use the criteria from Pineau et al. (2020) omitting the non-applicable theory component.
D.1 MODELS & ALGORITHMS
An analysis of the complexity (time, space, sample size) of any algorithm. The algorithms employ (LBFGS and stochastic gradient descent on convex and nonconvex problems) are standard, and refer the reader to other references determine their complexities.
D.2 DATASETS
The relevant statistics, such as number of examples.
2D
IMDb
Ce CelebA
dataset size (train)
75
21,215
4,000 40,000
dataset size (valid)
75
21,027
8,000
dataset size (test)
75
13,995
20,000
input space
IR.
1R:
II18x218x3
The details of train validation test splits. We do not use validation set for the simple 2D classification setting and IMDb datasets, but hold out examples for a test set. For CelebA, we follow authors' division of images in train, validation and test splits.
An explanation of any data that were excluded, and all pre-processing step. As described in the main text, subsample data balance attributes within each training condition. For the CelebA domain, we use the following feature pairs to produce the results in Section (6):
discriminant
distractor
mouth open
male mouth open mouth open
wearing lipstick
male
male
high checkbones
male
blond hair
male
arched eyebrows
link downloadable version of the dataset or simulation environment. available datasets whose links can be found online.
We use publicly
For new data collected, complete description of the data collection process, such as instruc- tions to annotators and methods for quality control. We do collect any new data.
D.3 CODE
We provide code in a folder of the supplementary entitled code.
Specification of dependencies. See requirements. txt.
Training and evaluation code. See exposure-bias train.
main. py, which imports the module
'The numbers for CelehA are approximate because there are deviations in the availability of images across attribute combinations.
17

---

Under review as a conference paper at ICLR 2022
(Pre-)trained mo)el(s). We do not provide pre-trained models because of the large number of models evaluated to compute average performances across models.
README file includes table of results accompanied by precise commands run to produce those results. We will clean and provide Jupyter notebooks to generate all plots in the publicly available code repository after publication.
D.3.1 EXPERIMENTAL. RESULTS
The range of hypeer-parameters considered, method to select the best hyper-parameter config- uration, and specification all hyper-parameters used to generate results. We use default hyperparameter settings whenever possible; all hyperparameter settings can be found in the accompa- nying code submission in the files in the folder exposure- bwwssiiiiaaaaee htteeeeiiaasooooo/sseeeeeooooeee
The exact number training and evaluation runs. For the points-in-a-plane and IMDb settings, we use 20 random seeds, which randomize the model weight initialization. For the CelebA domain, we run seeds for each model configuration, and discard runs that achieve below 75% accuracy on validation set images that belong to the data conditions (quadrants) observed during training.
clear definition of the specific measure or statistics used to report results. We report accuracy performance metric each of the four quadrants depicted in Fig. (2) well interpolating data settings. We additionally report measures that are the performance difference between data settings.
description of results with central tendency (e.g., mean) variation (e.g., error bars). See the main text for description of results. We include 95% confidence interval on all reported measures.
The average runtime for each result, estimated energy cost. For the points-in-a-plane, IMDb and CelebA datasets, the average runtime (time to train and evaluate a single model) 5, 10 and 30 minutes, respectively.
A description of the computing infrastructure used. We run experiments serially on an NVIDIA P100 GPU.
18


Reviews:
[{"review_id": "gNz0BNNARxZ", "reviewer": "Reviewer_TPBn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found that this paper was quite difficult to read since the authors present discussions without clear definitions. Using many acronyms makes the reading difficult too. \n\nFor example, “feature-level bias is measured as deviation from chance performance in the CC condition.” When reading CC condition, it says “the data presented in this condition confound color and shape”. What does confound mean here? CC condition is unclear and I do not know feature-level bias.\n\nSimilarly, “Exemplar-vs-rule bias is measured by the difference between performance in the PE and ZS conditions—…” When reading PE and ZS, they are not defined either. \n\nReading on, I do not see formal definitions of feature-level bias exemplar-vs-rule bias. There are measures obtained from models trained by some specific methods. So the proposed approach deals specifically with three types of methods. Why do the authors focus on three types of methods? Why are they representative? \n\nAlso, I do not know what spurious correlation means in this paper. \n\nThe authors present quite some examples to illustrate their points, but I am confused by their examples. For example, in Figure 1, only two training examples are given. Any prediction is incorrect since there is not enough evidence to learn a model. I cannot see the differences between rule-based and example-based. What rules can we generate based on the two examples?\n", "summary_of_the_paper": "This paper studies the extrapolation of machine learning models to unseen regions, and specifically studies two types of biases: feature-level bias (differences in which features are more readily learned) and exemplar-vs-rule bias (differences in how these learned features are used for generalization). Motivated by the studies of exemplar vs. rule-based generalization in cognitive psychology, the authors present a protocol directly probing this trade-off in machine learning systems. The authors present empirical results across a range of models in both expository and real-world image and language domains and demonstrate that using the trad off provides a more complete picture of extrapolation behaviour than existing methods. \n", "summary_of_the_review": "The paper is unclear based on the current presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_WCl8DFjdrQ", "reviewer": "Reviewer_RJtk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some of the ideas introduced by the paper to think about certain inductive biases is interesting. For example, feature-level bias being which features are easier or harder to learn. Here it could be interesting to go into continuous vs. categorical features.\n\nI found some of the exposition unnecessarily confusing. For example, since in Section 2, the color of objects determines their label (i.e. green objects are “dax” and purple objects are “fep”), why use the labels \"dax\" and \"fep\" at all rather than just green and purple?\n\nThere are many claims that could be substantiated or explained further. For example, why do GPs help with formalizing exemplar-based generalization (Section 4.1)? It would also be clearer to specify which statements have been substantiated through experiments and which have not. For example, Section 4.2 starts off by saying that one NN, one GLM, and one GP has been trained and then make a jump to Figure 4a which shows decision boundaries that are not surprising; I am not sure how it validates the protocol proposed to measure EVR without confounds. Section 4.4 makes a substantial claim (\"We also find that different ways to reduce ρ (e.g. by reducing π0 or by increasing π1), give different extrapolation behavior (see Appendix)\") based on results pushed to the appendix. \n\nSome sentences could be better worded:\n- (Page 6) \"EVR increases with exemplar-basedness\" \n- (Page 7) What is the \"held-out quadrant\"?\n", "summary_of_the_paper": "The paper proposes measures of how two types of explanation methods -- rule based and exemplar based -- generalize and extrapolate to unseen data regions.", "summary_of_the_review": "The paper is ambitious but falls short in clearly explaining the framework proposed, and substantiating the idea with strong experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tdmh4Vu7vn", "reviewer": "Reviewer_UbRf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generalization in the domain of extrapolation, which the authors address in this paper, is one of the critical issues in machine learning. Therefore, they propose a protocol to investigate the problem of inductive bias in extrapolation. Specifically, inspired by psychological research, they propose a protocol to investigate the inductive bias of the learning system towards different features (FLB) and the inductive bias of the learning system towards different ways of using features, either by rule-based or exemplar-based generalization (EVR).\n\nThe approach, inspired by psychological research, is interesting. However, we are not convinced that it is a practically valid method, as we have only shown experiments on two datasets.", "summary_of_the_paper": "The generalization of data distributions to unseen regions, i.e., extrapolation, remains one of the critical challenges in machine learning. The inductive biases of the learner determine such extrapolation. Unfortunately, machine learning systems often do not share the same inductive biases as humans and, as a result, may extrapolate in ways that do not match the analyst's expectations. The authors investigated two different types of such inductive bias: feature-level bias (differences in which features are more easily learned) and exemplar-based and rule-based bias (differences in how learned features are used for generalization). Inspired by these experimental approaches, we have proposed a protocol to investigate this trade-off in learning systems directly. We present empirical results for a range of models and the domains of explanatory images and language. We demonstrate that controlling for feature-level bias while measuring the trade-off between exemplars and rules provides a complete picture of extrapolative behavior than existing formalisms.", "summary_of_the_review": "The authors' treatment of inductive bias in extrapolation is interesting and will interest many researchers. The proposed protocol, which takes its ideas from psychology, is also interesting. However, we are not convinced that the proposed protocol is practical.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "26h4hnV19sn", "reviewer": "Reviewer_yoH5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ Probing the behavior of the learning system by comparing its generalization performance when trained on different datasets is interesting and novel.\n+ The synthetic experiment on 2D inputs demonstrate the effectiveness of the measure. \n+ The paper is very well-written.\n\nWeakness:\n+ Unclear utility.\nThe main contributions of the paper are the two proposed measures. However, I found it hard to utilize these measures in practical applications. \n  + In order to compute the measures, the values for the discriminant features and the distractor features are required. In this situation, there are many existing methods for learning models that are robust against the distractor features [1]. What will be the point of evaluating the bias of a non-robust classifier?\n  + For the exemplar-vs-rule propensity (EVR), my understanding is that it captures the extent to which the distractor features are utilized in the learning system. What will be the benefit of EVR compared to evaluating the worst-group accuracy as in [1]? Let’s use CelebA as an example. The training data contains mostly {male, dark_hair}, {female, blond_hair}, {female, dark_hair} images. At test time, [1] evaluates the worst-group accuracy, which is likely to be the performance on {male, blond_hair} (since it is underrepresented in the training data).\n\nQuestions and suggestions:\n+ How do you interpret the values of the FLB and EVR measures? For instance, in IMDB, the FLB score is -0.3 while the EVR score is 0.54. What do these values mean?\n+ I think both measures can be defined in a more generalized setting (where you don’t need to assume the label marginal is uniform).\n+ You argued in the paper that the measure is capturing something more than the spurious correlation. I think it will be more precise to state it as **linear** correlation in Figure 3.\n+ It will be very interesting to see a contour plot of EVR over $\\pi_0$ and $\\pi_1$ even in the synthetic experiments.\n\nI am happy to adjust my ratings after the rebuttal period.\n\n[1] Sagawa, Shiori, et al. \"Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.\" arXiv preprint arXiv:1911.08731 (2019).", "summary_of_the_paper": "This paper proposes two measures for evaluating the feature-level bias and the exemplar-vs-rule bias in learning systems. Specifically, the authors designed three independent training conditions:\n    i. cue conflict: {x1 = 0, x2 = 1, y = 0}, {x1 = 1, x2 = 0, y = 1}\n    ii. zero shot: {x1 = 0, x2 = 0, y = 0}, {x1 = 1, x2 = 0, y = 1}\n    iii. partial exposure: {x1 = 0, x2 = 0, y = 0}, {x1 = 1, x2 = 0, y = 1}, {x1 = 0, x2 = 1, y = 0}\nand one testing condition:\n    iv. extrapolation: {x1 = 1, x2 = 1, y = 1}.\nThe inductive bias of a given learning system is measured by its extrapolation performance difference when trained on different training conditions. \n\nEmpirically, the authors first verified their framework on a synthetic dataset with 2D inputs. The results confirm that generalized linear model favors rule-based generalization while Gaussian process favors exemplar-based generalization. On IMDB, the authors show that LSTM models exhibit high feature-level bias (overfitting to the spurious token features) favors exemplar-rule bias. On CelebA, the authors show that ResNet exhibits a wide range of feature-level bias for different features (‘male’ is easier to learn than ‘high cheekbones’). However, across all feature pairs, the model prefers exemplar-based generalization to rule-based generalization.", "summary_of_the_review": "The authors proposed an interesting approach for measuring the rule- and exemplar-based  generalization for a given learning system. The perspective is novel and the writing is clear. My only concern is the practical utility of the proposed measures.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel protocol for examining the inductive biases in learning systems, by quantifying the exemplar-rule trade-off (as measured by the exemplar-vs-rule propensity (EVR) defined in Eq. (2)) while controlling for feature-level bias. 

Reviewers mostly agree that the problem studied in this paper is practically relevant and that the two bias measures are potentially interesting and (jointly) more informative than existing measures such as spurious correlation. However, a shared concern among the reviewers (with confidences scores >=3) is the clarity of the exposition (e.g., many key concepts such as the data conditions are informally specified [Section 2 (Reviewer TPBn)], some key messages not clearly conveyed in the main paper [Section 3 (Reviewer RJtk)], and results inconclusive or not sufficiently supported by the experimental results [for both the synthetic setting (Reviewer RJtk) and the real-world setting (Reviewer yoH5)]. Based on the above concerns, the reviewers were not convinced that this work is well supported in its current state to merit acceptance for publication.
Decision: Reject