Learning De-biased Representations with Biased Representations
Authors: Hyojin Bahng
Sanghyuk Chun
Sangdoo Yun
Jaegul Choo
Seong Joon Oh
Venue:  (2020)
Keywords: Generalization
Bias
Dataset bias

Abstract: Many machine learning algorithms are trained and evaluated by splitting data from a single source into training and test sets. While such focus on in-distribution learning scenarios has led interesting advances, it has not been able to tell if models are relying on dataset biases as shortcuts for successful prediction (e.g., using snow cues for recognising snowmobiles). Such biased models fail to generalise when the bias shifts to a different class. The cross-bias generalisation problem has been addressed by de-biasing training data through augmentation or re-sampling, which are often prohibitive due to the data collection cost (e.g., collecting images of snowmobile on a desert) and the difficulty of quantifying or expressing biases in the first place. In this work, we propose a novel framework to train a de-biased representation by encouraging it to be different from a set of representations that are biased by design. This tactic is feasible in many scenarios where it is much easier to define a set of biased representations than to define and quantify bias. Our experiments and analyses show that our method discourages models from taking bias shortcuts, resulting in improved performances on de-biased test data.

Under review as a conference paper at ICLR 2020
LEARNING DE-BIASED REPRESENTATIONS WITH BIASED REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Many machine learning algorithms are trained and evaluated by splitting data from a single source into training and test sets. While such focus on in-distribution learning scenarios has led interesting advances, has not been able tell if mod- els are relying on dataset biases as shortcuts for successful prediction (e.g., using snow cues for recognising snowmobiles). Such biased models fail generalise when the bias shifts to different class. The cross-bias generalisation problem has been addressed by de- biasing training data through augmentation or re-sampling. which are often prohibitive due to the data collection cost (e.g., collecting images of a snowmobile on a desert) and the difficulty of quantifying or expressing biases in the first place. In this work, propose a novel framework to train a de- riebbised representation by encouraging it to be different from a set of representations that are biased by design. This tactic is feasible in many scenarios where it is much easier to define set of biased representations than to define and quantify bias. Our experiments and analyses show that method discourages models from taking bias shortcuts, resulting in improved performances on de-biased test data.
INTRODUCTION
Most machine learning algorithms are trained and evaluated by randomly splitting a single source of data into training and test sets. Although this is a standard protocol, is blind to critical problem: the existence of dataset bias (Trralla & Efros 2011). For instance, many frog images are taken in swamp scenes, but swamp itself is not a frog. Nonetheless, neural network will exploit this bias (i.e.. take "shortcuts") if it yields correct predictions for the majority of training examples. If bias is sufficient to achieve high accuracy, there is little motivation for models to learn the complexity of the intended task, despite its full capacity to do so. Consequently, model that relies bias will achieve high in-distribution accuracy, yet fail to generalise when the bias shifts.
We tackle this "cross-biat generalisation" problem where model does not exploit its full capacity due to the "sufficiency" of bias cues for prediction of the target label in training data. For example, language models make predictions based on the presence of certain words (e.g., "not" for "contra- diction")) (Gururangan et al. 2018)) without much reasoming on the actual meaning of sentences, even if they are principle capable of sophisticated reasoning. Similarly, convolutional neural networks (CNNs) achieve high accuracies on image classification by using local texture cues as shortcut, as opposed to more reliable global shape cues (Geirhos et al. 2019 Brendel & Bethge 2019).
Existing methods attempt to remove model's dependency on bias by de-biasing the training data through data augmentation (Geirhos et al.] 2019) or re-sampling tacties (Li & Vasconcelos) 2019). Others have introduced pre-defined set of biases that a model is trained to be independent against (Wang et al., (201)). These prior works assume that bias can easily be defined or quantified, but real-world biases often do not (c.g., texture bias above).
To address this limitation, we propose a novel framework to train a de-biased representation by encouraging it to be "different" from set of representations that are biased by design. Our insight that biased representations can easily be obtained by utilising models of smaller capacity (e.g., bag of words for word bias and CNNs of small receptive fields for texture bias). Experiments show that our method is effective in reducing a model's s dependency on "shortcuts" in training data, as evidenced by improved accuracies in test data where the bias is either shifted removed.

---

Under review as a conference paper at ICLR 2020
signal
siccel
signai
Irsining sample test sample
target
colour
conditional
Bls
distribution
in-distribution 1(5 "1""" , (rr 28s WSS (PT) p15" R2,, x( o p(
cross-domain
cross-bias
( ("")"" 7 p(5% (")
y
Figure I: Learning scenarios. Different distributional gaps may take place between training and test distri- butions. Our work addressing the cross-bias generalisation problem. Background colours the right three figures indicate the decision boundaries models trained on given training data.
2 PROBLEM DEFINITION
We provide rigorous definition of our over-arching goal: overcoming the bias in models trained on biased data. We show that the problem we tackle is novel and realistic.
2.1 CROSS-BIAS GENERALISATION
We first define random variables, signal S and bias B cues for the recognition of an input X as certain target variable Y. Signals are the cues essential for the recognition of X as Y; examples include the shape and skin patterns of frogs for frog image classification. Biases B's, hand, are cues not essential for the recognition but correlated with the target Y; many frog images taken in swamp scenes, SO swamp scenes that intervening on B should not change Y; moving a frog from swamp to dessert scene does change the "frogness". We assume that the true predictive distribution p(Y) (X) factorises as S P(Y S, (S, B(X), signifying the sufficiency of p(S, B X) for recognition.
the other
be considered B. A key property of B is
Under this framework, three learning scenarios are identified depending on the change of relation- ship p(S, B, Y) across training and test distributions, p(S" B", y") and p(See B y*), respec- tively: in-distribution, cross-domain, and cross-bias generalisation. See Figure for a summary.
In-distribution. p(S", B", y") = p(S", Ber Y²). This is the standard learning setup utilised in many benchmarks by splitting data from single source into training and test data at random.
Cross-domain. p(S" Y") * , y*) and furthermore p(B") * p(B"). B in this case is often referred to as "domain". For example, training data consist of images with (Y"=frrog, B"=wilderness) and (Y"=bird, ="=wilderness), while test data contain (Y"=frog, =i=indoors) and (Y**=bird, =nddors), This scenario is typically simulated by training and testing on different datasets (Ben-David et al. 2007).
Cross-bias. p(BBr) 1L. p(y and the dependency changes across training and test distributions: P(B", Y") * p(BBK, ).²) We further assume that p(B²) p(B"), to clearly distinguish the scenario from the cross-domain generalisation. For example, training data only contain images of two types (Y""frrog, ====wamp) and (Y"=bird, B"==sy),, but test data contain unusual class-bias combinations (Y""=frog, =ly= and (Y'*=bird, =w==mppp). Our work addresses this scenario.
2.2 EXISTING CROSS-BIA GENERALISATIN METHOODS AND THEIR ASSUMPTIONS
Under cross-bias generalisation scenarios, the dependency pf B") J.. p(Y") makes bias B viable cue for recognition. The model trained on such data becomes susceptible to interventions on B, limiting its generalisabililty when the bias is changed or removed in the test data. There exist prior approaches to this problem. but with different types and amounts of assumptions on B. We briefly recap the approaches based on the assumptions they require. In the next part 22.3 we will define our novel problem setting that requires an assumption distinct from the ones in prior approaches.
Il and denote independence and dependence, respectively.
2

---

Under review as a conference paper at ICLR 2020
When an algorithm to disentangle bias B and signal S exists. Being able to disentangle B and S lets one collapse the feature space corresponding to B in both training and test data. A model trained on such normalised data then becomes free of biases. As ideal as it is, building model to perfectly disentangle B and S is often unrealistic.
When a generative algorithm or data collection procedure for pl B)) exists. When additional examples can be supplied through p( X BB), the training dataset itself can be de-biased, i.e., B Y. For example, one can either collect swamp to balance out the bias. Such a data augmentation strategy is indeed a valid solution adopted by many prior studies (Panda et al., 2018: Geirhos et al., 2019) Shhttyy et al., 2019). However, collecting unusual inputs can be expensive (Peyre et al.) 2017). and building a generative model with pre-defined bias types (Geirhos et al., 2019) may suffer from bias mis-specification and the lack of realism.
synthesise unusual images like frogs in sky and birds in
When predictive algorithm or ground truth for (BB|) exists. Conversely, when one can tell the bias B for every input X, two approaches are feasible. (1) The first is a data re- re-weighting solution: we give greater weights on frogs in sky than frogs in swamps to even out the correlation in p(B, Y) (Li et al. 2018 Li & Vasconcelos) 2019). (2) The second approach removes the depen- dency between the model predictions f(X) and the bias B. Many existing approaches for fairness in machine learning have proposed independence based regularisers to encourage ((X) _II. B (Zemel
et al. 2013) or the conditional independence f(X) IL B Y (called the "separation" constraint, Hardt et al. (2016)). Other approaches have proposed to remove predictability of p( ((B)X) based on ((X) through domain adversarial losses (Li & Vasconcelos 2019 Wang et al. 2019) or projec-
tion (Wang et al. 2019 Quadrianto et al.) 2019)).
The knowledge on x(B is provided in many realistic scenarios. For example, when the aim is to remove gender biases B in a job application process p p(Y),), applicants genders p(B(X) are supplied as ground truths. However, there exist cases when / is difficult to even be defined or quantified but can only be indirectly specified. We tackle such a scenario in the next part.
2.3 OUR SCENARIO: CAPTURING BIAS WITH A PARTICULAR SET OF MODELS
Under the cross-bias generalisation scenario, certain types of biases are not easily addressed by the above methods. Take texture bias as an example I Geirhos et al. (019)): (1) texture B and shape
cannot easily be disentangled, (2) building a generative model x(X ( or collecting unusual
images is expensive, (3) building the predictive model p(B for texture requires enumeration (classifier) or embedding (regression) of all possible textures, which is not feasible.
However, slightly modifying the third assumption results in a problem setting that allows interesting application scenarios. Instead of assuming explicit knowledge on X)),,,, we approximate B by defining set of models G' that are biased towards B by design. For texture biases, for example, define G to be the set of convolutional neural network (CNN) architectures with < 5 x 5 overall receptive fields. Then, any learned model g G can by design make predictions g(a)) based on the patterns that can only be captured with small receptive fields (i.c., textures).
More precisely, we define G to be bias-characterising model class for the bias-signal pair (B,S) S) if for every possible joint distribution PBB, X) there exists a E G such that p(B) X) s 9(X) (recall condition) and every g G satisfies g(X) IL B (precision condition). In practice, G may not necessarily include all biases and may also capture important signals (i.e., imperfect recall and precision). With this in mind, our framework is formulated so that f(X) does not ignore signals captured by G we do not require G to be perfect. Further detail is provided in 22.3
There exist many scenarios when such G can be defined, as we are given several evidence for the type of bias. For instance, action recognition models have been reported to rely heavily on static cues without learning temporal cues (Li et al., 2018 8 Li & Vasconcelos 2019); though the actual bias may not precisely be any static cue, we can still regularise the 3D convolutional networks towards better generalisation across static cue biases by defining G to be the set of 2D convolutional architectures. It has been argued that visual question answering (VQA) models, too, rely overly on language biases rather than the visual cues (c.g. without looking at the image, one knows the answer to what colour is the banana is yellow) (Agrawal et al. 2018). We can define G as the set of models looking at
3

---

Under review as a conference paper at ICLR 2020
the language modality only. Entailment models are biased towards the presence of certain words (e.g. when there are many nots, the sentence is contradictory), rather than really understanding the underlying meaning of sentences (McCoy al. 2 Niven & Kao (019)). We can design G to be the set of bag-of- words classifiers (He et al. 2019 2 et al. 2019). Generally, these scenarios exemplify situations when the added architectural capacity is not fully utilised due to the sufficiency of simpler cues for solving the task in the given training set.
3 PROPOSED METHOD
We present a solution for the cross-bias generalisation when the biaacccaracterising model class G is known (see (2.3); the method is referred to REBI' 3 The solution consists of training model for the task p( ({ with regularisation term encouraging the independence between the prediction f(X) and the set of all possible biased predictions {x(X) I E G). We will introduce the precise definition of the regularisation term and discuss why and how leads to the unbiased model.
3.1 REBI' S: REMOVING BIAS WITH BIAS
(( is fully known, we can directly encourage f(X) IL B. Since we only have access to G. Simply E F by designing set of biased models G and letting ʃ run away from G. This leads to the independence from bias cues B while leaving signal cues S as valid recognition cues; see 22.3 We will specify REBI'S S learning objective after introducing our independence criterion, HSIC.
the set of biased models G 2...) 3 we seek to promote ((X) 1L g(X) for every
put, we de-bias representation
Hilbert-Schmidt Independence Criterion (HSIC). Since we need to measure degree of inde- pendence between continuous random variables s(X) and g(X) in high dimensional spaces, it is infeasible to resort to histogram- measures; we use HSIC (Gretton et al. 2005). For two random variables U and V and kernels k and I, HSIC is defined as HSIC*.(UU, = =ICCVv llis where Cki is the cross-covariance operator in the Reproducing Kernel Hilbert Spaces (RKHS) of and (Gretton et al., 2005), an RKHS analogue of covariance matrices. || Ilus is the Hilbert- Schmidt norm, Hilbert-space analogue of the Frobenius norm. It is known that for two random variables U and V and radial basis function (RBF) kernels k and l, HSIC* V) 0 if and only if U 1I. V. A finite-sample estimate HSIC. (U, V) has been used in practice for statistical test- ing (Grettnn et al. (2005) (200)) feature similarity measurement (Kornblith et al., (2019), and model regularisation (Quadrianto et al. 2019 Zhang et al. 2018). HSIC (U, V) with m samples is de- fined (m 1)-2I((UVT) where U is mean-subtracted matrix of pairwise kernel similarities U, = k(u,, u,) - m E, k(u;, up)) among samples (u.} ~ U. V defined similarly.
Minimax optimisation for bias removal. In our case, we compute
(f(C), (F()) == max HSIC (f(X). gEG
with an RBF kernel for the degree of independence between representation f E F and the bi- ased representations G. We write HSIC.(f, G) and HSIC,(J, .) as shorthands. Since the prob- const, we use the canonical kernel align-
lem miny HSIColf, G) allows trivial solutions f ==
ment (CKA) (Shawe-Taylor & Cristianini 2004 Kornblith al. 2019) criterion defined by CKAo(f.g) HSIC,(f. (SSCo(f. HSICo(g.g)#) The learning objective for f is then
defined as
{ ((F,X,Y)+ X max CKAo(F.9) gEG
min JeF
where X(f,X,Y) is the loss for the main task XX) and À > 0. We consider replacing the inner optimisation with L2 minimisation mingeg Ilf- 99l2.. while retaining the CKA regularisation for the outer optimisation for f. Intuitively, Lg minimisation poses a stronger similarity condition be- tween f and g (in fact identity) than does C.A.A leading to better de- de-biasing performances (44..3)
'bbrevviation of "removing bias": pronounced like Levi's.
4

---

Under review as a conference paper at ICLR 2020
Independence versus separation. The CKA regularisation in equation 2 encourages f(X) T I (X). This may lead to less stable optimisation as removing bias often increases the main task loss L. Furthermore, g(X) may also capture signals along with bias (imperfect precision), and suppressing important signals is not desirable. To avoid such cases, we formulate our ob- jective as conditional independence f(X) I g(X) by computing the separation CKA, SCKAo(f: | > CKAo = (XX v)), where the term "separation" (or "equalised odds") comes from the fairness literature (Hardt et al.] 2016). Unlike independence that requires f f(X) ignore gXX) altogether, conditional independence allows s(X) to utilise cues captured by g(X) (i.e., whether it is bias or signal) if it highly correlates with Y. In other words, REBI'S S encourages f(X) to learn features beyond those already captured by g(X), so that it can generalise well to test sets with different biases (i.c., cross-bias generalisation).
The final learning objective for REBI' S is then
min JEF
((UXX,Y)
max (F.Au(f.9))
(3)
BEG
3.2
WHY AND HOW DOES IT WORK?
Independence describes relationships between random variables, but use it for function pairs. Which functional relationship does statistical independence translate to? In this part, we argue with proofs and observations that the answer to the above question is the dissimilarity of invariance types learned by a pair of models.
Linear case: Equivalence between independence and orthogonality. We study the set of func- tion pairs (f,g) satisfying f(X) IL g(X) for suitable random variable X ~ p(X). Assuming linearity of involved functions and the normality of X, we obtain the equivalence between statistical independence and functional orthogonality.
Lemma 1. Assume that and g affine mappings [(x) Ax + and g(x) Bx b where A E mm and B E R(x Assume further that X is a normal distribution with mean and covariance matrix E. Then, f(X) IL (() if and only if ker(A)- 5y ker( For positive semi-definite matrix E, we define (r, = (r, Es), and the set orthogonality Ly likewise. Proof in A
In particular, when f and have 1-dimensional outputs, the independence condition is translated to the orthogonality of their weight vectors and decision boundaries. From a machine learning point of view, f and g are models with orthogonal invariance types.
Non-linear case: HSIC as metric learning objective. We lack theories fully characterise general, possibly non-lime non-linear, function pairs (f.g) achieving f(X) 1 9(X); it is an interesting open question. For now, we make a set of observations in this general case, using the finite-sample independence criterion HSICo(f.9) = (m 1r-2 ÿT) = O, where f is the mean-subtracted m- ((f(())())))) f(xx)) and likewise for (see 3.1
kernel matrix Л.j ((f())
Note that tr(F T²)) ) is an inner product between flattened matrices f and g. We consider the inner product minimising solution for f on an input pair * given a fixed g. The problem can be written min (zo.())).((=))) tr(f G²), which is equivalent to min minl(re)).((.)) for T10.
Suppose T10 > 0. It indicates relative invariance of on (I1, since k(g(z)). > m- M. k(g(11), (zx)). Then, the above problem boils down min (ao)) foi. signifying the relative variance of on (zt, 21). Following a similar argument, obtain the converse statement: g is relatively variant on a pair of inputs, invariance of f on the pair minimises the objective.
We conclude that min/ HSICo(f. g) against fixed g is metric-Ieammin objective for the embed- ding f. where ground truth pairwise matches and mismatches are relative mismatches and matches for g, respectively. As result, and learn different sorts of invariances.
Effect of HSIC regularisation on toy data. We have established that HSIC regularisation encour- ages the difference in model invariances. To how it helps de-bias model, we have prepared
5

---

Under review as a conference paper at ICLR 2020
synthetic doodimenniooall training data following the cross-domain generalisation case in Figure I X = (B.S) E R² and Y {red, yellow, green]. Since the training data is perfectly biased, a multi-layer perceptron (MLP) trained on the data only shows 55% accuracy on de-biased test data (see decision boundary figure in Appendix (B). To overcome the bias, we have trained another MLP with equation 3 where the bias-characterising class G' is defined the set of MLPs that take only the bias dimension as input. This model exhibits de-biased decision boundaries (Appendix (B) with improved accuracy of 89% on the de-biased test data.
EXPERIMENTS
In the previous section, REBI' S has been introduced and theoretically justified. In this section, we present experimental results of REBI'S, S. We first introduce the setup, including the biases tackled in the experiments, difficulties inherent to the cross-bias evaluation, and the implementation details ($4.)) Results on Biased MNIST ($4.2) and ImageNet (#4.3) are shown afterwards.
4.1
EXPERIMENTAL SETUP
Which biases do tackle? There is broad spectrum of bias types to be addressed under the cross-bias generalisation setting. Our work is targeting the biases that arise due to the existence of shortcut cues that are sufficient for recognition in training data. In the experiments, we tackle a representative bias of such type: "local pattern" biases for image classification. Even if a CNN image classifier has wide receptive fields, empirical evidence indicates that they heavily rely on local patterns (i.e., color and texture) as opposed to global shape cues (Geirhos et al. 2019).
While difficult to precisely define and quantify all local pattern biases, it is easy to capture it through a class of CNN architectures: those with smaller receptive fields. This is precisely the setting where we benefit from REBI'S. S.
Evaluating cross-bias generalisation is difficult. To measure the performance of a model across real-world biases, one requires an unbiased dataset or one where the types and degrees of biases can be controlled. Unfortunately, data in real world arise with biases. To de-bias frog and bird image dataset with swamp and sky backgrounds (see 2.1.. either rare data samples must be collected (search for photos of frog on sky) or must intervene with the data generation process (throw frog into the sky and take a photo). Either way, it an expensive procedure (Peyre et 2017).
Preparing unbiased data is feasible in some natural language corpus of unbiased gender pronouns, Webster et al. (2018)). However, we are addressing biases that are expressed in terms of class representations but perhaps are difficult to precisely express in language, such as texture bias of image classifiers (..3.
when the bias type is simple (e.g., collecting
We thus evaluate our method along two axes: (1) Biased MNIST and (2) ImageNet. Biased MNIST contains synthetic biases (colour and texture) which we freely control in training and test data for in- depth analysis of REBI' S. particular, we can measure its performance on perfectly unbiased test data. On ImageNet, we evaluate our method against realistic biases. Due to the difficulty of defining and obtaining bias labels on real images, we use proxy ground truths for the local pattern bias to measure the cross-bias generalisability. MNIST and ImageNet experiments complement each other in terms of experimental control and realism.
Implementation of REBI'S S. We describe the specific design choices in REBI' S implementation (equation 3) in our experiments. We will open source the code and data.
To train a model that overcomes local pattern biases, first define biased model architecture fami- lies G such that they precisely and sufficiently encode biased representations: CNNs with relatively small receptive fields (RFs). The biased models in G will by design learn predict the target class of an image through only local cues. On the other hand, we define a larger search space with larger RFs for unbiased representations.
In our work, all networks ff and are fully convolutional networks. f(z) and g(x) denote the final convolutional layer outputs (feature maps), on which we compute the independence measures like

---

Under review as a conference paper at ICLR 2020
We perform global average pooling and learnable linear classifier on f(),, trained along with the outer optimisation, to compute the cross-entropy loss in equation 3
HSIC, CKA, and SCKA (.11)
For Biased MNIST, F' is the LeNet F (LeCun et al. 1998) architecture with RF 28. It has two 5- c after each of which max max-pooling layers of 2 x kernels are applied, followed by three linear layers. G has the same number of layers but either with all convolutional layers of 1 kernels and without max-pooling operations, called B1 iniddeet1, or with one 3-conv and one 1-conv, called BlindNet3, each with RFs of I and 3, respectively. On ImageNet, we use ResNet (He al. 2016) architecture for F (either ResNet 18 with RF of 435 or ResNet50 with RF of 427). G defined as BagNet (Brendel & Bethge 2019) architectures with the same depth as ResNet's 's (either BagNet:88 with RF of 43 or BagNet 50 with RF of 91). More implementation details are provided in [C
4.2
BIASED MNIST
We first verify our model on a dataset where we have full control over the type and amount of bias during training and evaluation. We describe the dataset and present the experimental results.
4.2.1 DATASET AND EVALUATION
2
3
4
5
6
7
3
9
We construct new dataset called Biased
MNIST designed to measure the extent to which models generalise bias shift. We modify MNIST (LeCun et al. 1998) by in- troducing two types of bias B colour and texture that highly correlate with the la- bel Y during training. With alone, a CNN can achieve high accuracy without having to learn inherent signals for digit recognition S, such as shape, providing little motivation for the model to learn beyond these superficial cues.
O /
2
3
S
7
8
Figure 2: Biased MNIST. construct synthetic dataset with two types of biases colour and texture which highly correlate with the label during training. Upper row: colour bias. Lower row; colour and texture biases.
We inject colour and texture biases by adding colour or texture patterns on training image back- grounds (see Figure 2b. We pre-select 10 distinct colour or 3 x 3 texture patterns b(y) for each digit y E {0, ... ... 9}. Then, for each image of digit y, we assign the pre-defined pattern b(y) with proba- bility p E [o, 1] and any other pattern (pre-defined for other digits) with probability (1 p)) P then controls the bias-target correlation in the training data: P p= 1.0 leads to complete bias and p = 0.1 leads an unbiased dataset. We consider two datasets: Single-bias MNIST with only colour bias and Multi-bias MNIST with both colour and texture biases (p 0.99 in all the experiments).
We evaluate the models generalisability bias shift by evaluating under the following criterion: Biased. p(S" B y²) p(S", B" y") (an in-distribution case in 2..1. Whatever bias the training set contains, it is replicated in the test set. This measures the ability de-biased models to maintain high in mn-dittribution performance while generalising to unbiased settings.
Unbiased. Bi IL yte_ We assign biases on test images independently of the labels. Bias is no longer predictive of Y and a model needs to utilise actual signals S to yield correct predictions.
We have additional fine-grained measures on Multi-bias MNIST: removed colour bias (colour; tex- ture bias remains) and removed texture bias (texture; colour bias remains) cases. Colour and texture biases are marginalised out in the test set, respectively, to factorise generalisability across different types of bias.
4.2.2 RESULTS
Results on Single- and Multi-bias MNIST are shown in Table [
REBI' S lets a model overcome bias. We observe that vanilla LeNet F achieves 100% accuracy under the "biased" metric (the same bias between training and test data) in Single- and Multi-bias MNIST. This is how most machine learning tasks are evaluated, yet this does not show the extent
We refer convolutional layers with & k kernels k-conv.

---

Under review as a conference paper at ICLR 2020
Single-bias MNIST
Multi-bias MNIST
Model
Description
biased
unbiased
biased
colour
texture
unbiased
Vanilla
100.
27.5
100.
45.1
85.8
29.6
Biased
10.8
100.
10.0
58.5
4.7
FaEx
Wang at al
2019
97.2
14.7
95.0
31.1
91.1
18.1
FIG
REBI'S (ours)
96.1
74.9
94.7
91.9
90.9
88.6
Table 1: Biased MNIST results. Architecture families are set as F' LeNet and G =B BlindNet1 for Single-bias and BlindNet for Multi-bias MNIST. Accuracy results are shown.
to which the model depends on bias for prediction. When the bias the label at evaluation, vanilla LeNet accuracy collapses to 27.5% and 29.6% under the "unbiased" metric on Single- and Multi-bias MNIST, respectively. The intentionally biased BlindNet models G' result in an even lower accuracy of 10.8% on Single-bias MNIST, close to the random chance 10%. This reveals that the seemingly perfoorrf model has in fact overfitted to bias and has not learned beyond this fallible strategy.
randomly assigned to
REBI'S on the other hand, achieves robust generalisation across settings by learning to be different from BlindNet representations G. REBI'S achieves a +47.4 pp (Single) and +59.0 pp (Multi) higher performances than the vanilla model under the cross-bias generalisation setup (the unbiased metric), with slight degradation in original accuracies (-3.9 pp and -5.3 pp. respectively).
Comparison against HEX. Previously, HEX (Wang et al. 2019) has attempted to reduce the depen- dency of model "superficial statistics", high-frequency textural information. HEX measures texture via neural grey-level co-occurrence matrices (NGLCM) and projects out the NGLCM fea- ture from the output of the model of interest. We observe that FREX, where HEX is applied F, is effective in removing texture biases (85.8% to 91.1% 1% cross-texture accuracy), but still vulnerable colour biases (accuracy drops from 45.1% % to 31.1%). Hand-crafting texture features done by HEX has resulted in its limited applicability beyond the hand-crafted bias type. By designing the model family architecture, instead of specific feature extractor, REBI' ' achieved a representation free of broader types of biases.
4.2.3 FACTOR ANALYSIS
Some design choices exist, leading to our final model ($3.1) the final performance. See Table 2 for ablative studies the Single-bias MNIST.
We examine how the factors contribute
Impact of the independence criterion. Three independence measures, HSIC, CKA. and SCKA, have been considered. SCKA, the separation CKA, used in REBI'S, s, shows a supe- rior de-biasing performance (74.9% against baseline choices. It improves upon HSIC by avoiding the trivial solution (a constant function), and upon CKA via more stable optimisation due to the milder conditional independence f(X) |L B that does not contradict the classification objective.
Indep.
biased unbiased
HSIC
83.5
51.1
CKA
97.4
22.0
SCKA
96.1
74.9
Inner opt.
biased unbiased
SCKA
95.8
62.4
L2
96.1
74.9
Updating y
biased unbiased
Impact of the I22 objective in the inner optimisation. We then study the effect of our choice to replace the inner SCKA optimisa- tion with the L2 objective. L2 considered, as it poses stronger convergence condition than SCKA does. We confirm indeed that the L2 objective results better de-biasing performance.
Fixed g
94.6
44.7
Multiple g
95.9
58.1
Updated g
96.1
74.9
Table 2: Factor analysis. Default
REBI'S S parameters: last rows.
Impact of updating G. The advantage of specifying class of models G' instead of single, fixed model go is that HSIC(f, Gi) can be computed more precisely ()..1). We quantify this benefit. Fixing the biased representation to go results in sub-optimal de-biasing performance, 44.7%. By including multiple fixed biased models (gu. ... 9T C G. de- de-biasing improves to 58.1 58,1% but not good the updated case, 74.9%. It is thus important to precisely compute the representation- to-set independence. More detailed analysis around the receptive field sizes of models in is in Appendix F
8

---

Under review as a conference paper at ICLR 2020
Model
Description
Biased
Unbiased
IN-A
F
ResNet18
93.3
85.8
30.5
G
RagNet18
72.4
58.6
19.5
FIL
REBI'S
93.7
88.4
31.7
F'
Geirhos et al.
92.5
87.6
29.7
F
ResNet50
91.7
78.3
29.5
G
5a
60.9
21.4
FaG
REBI'S 5
88.7
89.2
31.3
Table 3: ImageNet results. We show results with F and Fun corresponding (F' is ResNet1 and G is BagNet 8) and is kesNet50 and G is B5gNet550). IN-A indicates ImageNet-A.
4.3 IMAGENET
In ImageNet experiments, we further validate the applicability of REBI' on the local pattern bias in realistic images (i.e.. objects in natural scenes). The local pattern bias often lets model achieve good in-distribution performances by exploiting the local cue shortcuts (e.g, determining a turtle class by not seeing its shape but the background texture).
4.3.1 DATASET AND EVALUATION
We construct 9-Class IetgeNet, subset of ImageNet (Russakovsky et al. 2015) containing super-classes as done in llyas et al. (2019), since full-scale analysis on ImageNet not scalable. We additionally balance the ratios of sub-class images for each super-class to solely focus the effect of the local pattern bias.
Since it is difficult to evaluate cross-bias generalisability realistic unbiased data ( 4.11 we settle for alternative evaluations:
Biased. p(S 8e y²) p(S D", Y") Accuracy is measured on the in-distribution validation set. Though widely-used, this metric is blind to a model's generalisability to unseen bias-target combinations.
Unbiased. B yte As proxy to the perfectly de-biased test data, which difficult to collect the ground truth labels for local pattern bias. For full details of texture clustering algorithm, see Appendix { For an unbiased accuracy measure- ment, compute accuracies for every of images corresponding to target-texture combination (c, y). The combination- accuracy Ac,y is computed by Corr(c, /Ppppcc..))) y), where Corr(c, (c, y) the number of correctly predicted samples in (c, y) and Pop(c, y) is the total number of samples in (c, v), called the population at (c, y). The unbiased accuracy is then the mean accuracy over all A where the population is non-zero Pop(c. y) # O. This measure gives more weights on samples of unusual texture-class combinations (smaller Pop(c, y)) that are less represented in the usual biased accuracies. Under this unbiased metric, biased model basing its recognition on textures is likely to show sub- sub-optimal results on unusual combinations, leading to drop in the unbiased accuracy.
(4..1). texture clusters IDs e {1.-.. K}
ImageNet-A. ImageNet-A (Hendryeks et al., 2019) contains failure cases of ImageNet pre- trained 50 among web images. The images consist of many failure modes of networks when "frequently appearing background elements" become erroneous cues for recognition (e.g. a bee image feeding on hummingbird feeder is recognised as hummingbird). Improved performance on ImageNet-A is an indirect signal that the model learns beyond the bias shortcuts.
4.3.2 RESULTS
We measure performances of ResNet18 and ResNet each trained to be different from BagNet18 8 and BagNet 50 respectively using REBI' S, under the metrics in the previous part. Results are shown in T
Vanilla models are biased. Both ResNet and 50 show good performances on the bi- ased accuracy (93.3% and 91.7%, respectively), but dropped performances on the texture-ume unbiased accuracies (85.8% and 78.3%, respectively). BagNet 18 and BagNet50 perform worse than the vanilla ResNets as they are heavily biased towards texture by design (i.e., small receptive field size). The drop signifies the biases of vanilla models towards texture cues; by basing their predictions

---

Under review as a conference paper at ICLR 2020
on texture they obtain generally better accuracies texture-class pairs (c, y) that repre- sented more. The drop also shows the limitation of current evaluation schemes where cross-bias generalisation is not measured.
REBI'S S leads to less biased models. When REBI' s is applied on ResNet18 and ResNet to encourage them to unlearn cues learnable by BagNet and BagNet 50, respectively, observe general boost in unbiased accuracies. ResNet18 8 improves from 85.8% to 88.4%; ResNet50 from 78.3% to 89.2%. Our method thus robustly generalises to less represented texture- target combinations at test time. We observe that our method also shows improvement on the chal- lenging ImageNet-A subset (e-g. from 29.5% to 31.3% for ResNet which further shows superiority on generalisability to unusual texture-class combinations. Similarly, data augmentation via Stylised ImageNet (Geirhos et al. 2019) attempts to reduce model's dependency on texture by augmenting data with texturised images. While it shows improvements in reducing texture bias (87.6% for unbiased accuracy), does not increase the generalisability to the challenging natural adversarial examples (29.7% for ImageNet-A), More detailed analysis on per-texture and per-class accuracies are included in Appendix [ learning curves for the baseline and REBI'S S are in Ap-
pendix H
Qualitative analysis. We qualitatty present the cases where our method successfully de-biases a texture-target dependency. Figure 3 shows examples common and uncommon texture-target combinations for "grass" and "close-up" texture clusters. The shown uncommon in- stances are the ones ResNet18 has incorrectly predicted the class. For example, crab in the grass has been pre- dicted turtle, presumably because turtles co-occur lot with grass backgrounds training data. On the other hand, REBI'S s (ours) robustly generalises to unusual texture-class combinations.
common images
uncommon
images
turtle in grass baseline: turtle ours: turtle
crab in grass baseline: turtle ours: crab
CONCLUSION
We have identified practical problem faced by many ma- chine learning algorithms that the learned models exploit bias shorteuts to recognise the target (cross-biar bias gener- alisation problem in $2) In particular, models tend to under-utilise its capacity extract non-bias signals (e.g. global shapes for object recognition) when bias short- cuts provide sufficient cues for recognition in the training data (e.g. local patterns and background cues for object recognition) (Geirhos et al. 2019). We have addressed this problem with the REBI'S S framework, which does not rely on expensive, if not infeasible, training data de- biasing schemes. Given an identified encourages a model f to be statistically independent of G (33). We have provided theoretical justifi- cations for the use of statistical independence (3.2) and have validated the superiority of REBI'
insect close-up baseline: insect ours: Insect
frog close-up baseline: insect ours: frog
Figure 3: Qualitative results. Common
and uncommon images shown according class-texture relationship. Predictions of ResNet and REBI S are shown as well.
of models G that encodes the bias be removed, REBI'
in removing biases from models through experiments on modified MNIST and ImageNet (14)
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971-49800 2018
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pp. 137-144, 2007.
Wieland Brendel and Matthias Bethge. Approximating CNNs with ofg-ol-local-- features models works sur- prisingly well on imagenet. In Intermational Conference on Learning Representations, 2019. URL.
https: https://openreviin net. net/forummii d fwwhAgyO
10

---

Under review as a conference paper at ICLR 2020
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. EMNLP, 2019.
Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. In Advances in neural information processing systems, pp. 262-270, 2015.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Bren- del. Imagenet- trained CNNs are biased towards texture; increasing shape bias improves accuracy robust- ness. In International Conference on Learning Representations, 2019. URL htttp:://oparroiii.c..
net het/foruri Bid-Bygh9 109XX
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidr norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Arthur Gretton, Kenji Fukumizu, Choon H Teo, Le Song. Bernhard Schölkopf, and Alex J Smola. A kernel statistical test of independence. In Advances in neural information processing systems, pp. 585-592, 2008.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pp. 112, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. URL htt..:tt.://// /wwww ac www..b.......... org/ anthol N
Moritz Hardt, Eric Price, Nati Srebro, et al. Equality of opportunity in supervised Icarning. In Advances in neural information processing systems, pp. 3315-3323, 2016.
He He, Sheng Zha, and Haohan Wang. Unleam dataset bias natural language inference by fitting the residual. EMNLP, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv: 1907.077747 2019.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint 120.009955. 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Diederik P Kingma and Jimmy Ba. arX.v:1412.6980, 2014.
Adam: A method for stochastic optimization. arXiv preprint
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Pruceedings of the IEEE, 86(1 (1:21))::272233 1998.
Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of 2019.
the IEEE Conference Computer Vision and Pattern Recognition, pp.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias.
In Proceedings of the European Conference Computer Vision (ECCV), pp. 513-528, 2018
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. ACI., 2019.
Timothy Niven and Hung-Yo Yu Kao. Probing neural network comprehension of natural language arguments. ACL, 2019.
Rameswar Panda, Jianming Zhang, Haoxiang Li, templating visual emotions: Understanding and overcoming dataset bias. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 579-595, 2018.
Lec, Xin Lu, and Amit K Khy-Chhodhury. Con-
Julia Peyre, Josef Sivie, Tvan Laptev, and Cordelia Schmid. supeiies learning of visual relations. In Proceedings of the IEFF International Conference on Computer Vision, pp. 5179-5188, 2017.
11

---

Under review as a conference paper at ICLR 2020
Novi Quadrianto, Viktoriia Sharmanska, and Oliver Thomas. Discovering fair representations in the data do- main. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8227-8236. 2019.
Olga Russakovsky, Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, An- drej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):21 1-252, 2015.
John Shawe- Shawe-Taylor and Nello Cristianini. Kermel methods for pattern analysis. Cambridge university press, 2004.
sidewalk-quantifying Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk quantifying and con- trolling the effects of context in classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8218 8226, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1xXtv:100.15530 2014.
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1528, June 2011.
Haohan Wang, Zexue He, and Eric P. Xing. Learning robust representations by projecting superficial statistics out. In International Conference Learning Representations, 2019. URL https:://ppexxew....
net/forum? (idrummdddddrreeeeeee
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-617, 2018.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, pp. 325 033,,,,,222 2013.
Changqing Zhang, Yeqing Liu, Yue Liu, Qinghua Hu, Xinwang Liu, and Pengfei Zhu. Fish-mml: Fisher-hsic multi-view metric learning. IJCAI, pp. 3054-3060, 2018.
12

---

Under review as a conference paper at ICLR 2020
A STATISTICAL INDEPENDENCE IS EQUIVALENT TO FUNCTIONAL ORTHOGONALITY FOR LINEAR MAPS.
We provide proof for the following lemma in 3.2
Lemma 1. Assume that and are affine mappings f(x) = Az + a and g(x) Bx + where A |mxx and B E R Assume further that X is normal distribution with mean μ and covariance matrix Σ. Then, f(X) TT g(X) if and only if (er(A)= Ls; ker(B)+. For positive semi-definite matrix Σ, we define (r, S) (r, Es), and the set orthogonality Ly likewise.
Proof. Due to linearity and normality, the independence f(X) TT g(X) is equivalent to the co- variance condition (o)((()).(()) = 0. Covariance is computed as:
Cov(J(X), 1x A(X xx)(X T
BB AEBT
(4)
Note that
AEB =0 = (u, (u, (TABB"")) = = (A Bd w> = 0 vwww
(5)
im(A²) Ly im(ii) = ker ker(A)+ 1y (B)
B DECISION BOUNDARY VISUALISATION FOR TOY EXPERIMENT
We show the decision boundaries of the toy experiment in 3 See Figure decision boundary changes over training can be found at anonymous link
GIF animations of
signal
bias B
Figure 4: Decision boundaries. Left to right: training data, haseline model, and our model.
C IMPLEMENTATION DETAILS
solve the minimax problem in equation 3 through alternating stochastic gradient descents (ADAM Kingma & Ba (2014)) where we alternate between 5 epochs for the outer problem and 5 epochs for the inner one. The regularisation parameter À is set to 7, 0.00 for CKA and 100 for MNIST and 0.05 for ImageNet. Note that degree of synthetic bias excessively large (bias-target correlation set to = 0.99) compared to the degree of bias in realistic settings. We have used batch sizes 256 (32 for Re and learning rates 0.001 with linear decay. We train all models for 80 epochs for Biased MNIST. For 9-class ImageNet, we train ResNet18 for 200 epochs and ResNet for 100 epochs.
L2 minimisation
large for MNIST as the
BagNet 18 and BagNet have 18 and 50 convolutional layers which are the same structure (Basic Bottleneck blocks) as ResNet18 and ResNet50. respectively. The internal kernel sizes of BagNet are set to x following the original paper's 's philosophy (Brendel & Bethge (009
D PERFORMANCE BY BIAS AND LABEL ON BIASED MNIST.
In the Biased MNIST experiments (44.2) we have shown either biased or unbiased set statistics. In this section, provide more detailed results where model accuracies computed per bias class
13

---

Under review as a conference paper at ICLR 2020
B b and per target label Y y- We visualise the case-wise accuracies of the baseline LeNet and the REBI'S trained LeNet BlindNet in Figure [ The diagonals in each matrix indicate the pre-defined bias-target pair (44.2.1). Thus, the biased accuracies be computed by taking the mean of diagonal entries in each matrix and the unbiased accuracies through the mean of all entries in each matrix.
The vanilla LeNet's 's tendency to have higher accuracies on diagonal entries and near-zero perfor- mances many other off-diagonal entries indicate the fact that LeNet is relying a lot on the colour (bias) cues. REBI' 3 successfully resolves this tendency in the vanilla model, exhibiting more uni- form performances across all bias-target pairs (b, y). Note that accuracies below the main diagonal are relatively high as they are the classes that pre-defined patterns are assigned to at probability (1 p). Figure 6 demonstrates that our method successful across different degrees of bias given training data.
100
100
100.0
0.0
0.0
50.0
0.0
0.0
2.4
24.2
1.4
93.4
96.0
97.1
96.4
36.7
83.8
21.3
94.3
95.0
92.0
03.1
97.0
0.0
0.0
11
3.8
75.7
41.1
0.0
0.0
97.4
97.4
95.5
50.2
88.0
29.1
96.0
96.9
91.4
49.8
96.4
-2.00
0.0
67.3
100
0.0
0.2
06.7
0.0
0.0
82.0
15.4
72 3
99
95.2
74.0
92
83.0
01.1
75.1
60
60
Class
0.0
0.0
0.0
11.1
100.0
0.0
0.0
2.3
14.3
0.0
Clas
92.6
1.7
79.5
86.4
98.8
46.4
92.8
91.1
93.7
33.1
-40
40
32.0
26.0
64.1
62.3
0.2
77.7
100.050.
0.1
50.3
96.7
90.9
95.2
85
92.6
30
96.6
96.7
93.9
44.5
11.4
0.3
8.9
0.0
0.3
21.7
86.0
0.0
0.0
96.9
95.2
95.4
64.7
92.4
30.3
96.5
85.7
94.7
64.3
95.7
0.2
0.0
29.7
1.7
0.0
0.0
8.0
0.0
64.3
100.0
31.4
0.0
23.0
06.0
69.0
95
02.4
56.3
97.8
2
5
00
2
7
00
Blas
Blas
(a) Vanilla trained LeNet.
(b) REBI S trained.
Figure 5: Bias-target- accuracies. We show accuracies for each bias B and target Y y pair in Single-bias MNIST.
Vanilla
Biased
REBI'S
Accuracy
20
0.70
0.75
0.80
0.90
0.95
1.00
Blas-target correlation
Figure 6: Impact of p. REBI'S S effective in de- le-biasing across different degree of bias in data.
E IMPACT OF RECEPTIVE FIELDS OF FOR BIASED MNIST
is conceptually important to design a the set biased models that encode the bias as precisely as possible. See precision and recall conditional for G in 2.3 To see if this is empirically true, we have measured the performance of REBI'S S with = LeNet and G = Bli inneet where the BlindNet receptive fields controlled by replacing convolutional layers with 1 x convolutions, resulting in receptive fields (1.5, 7, 7.2 The 28 x 28 receptive field indicate the case when LeNet used as G. We measure the biased and unbiased set performances Single-bias MNIST (44.2.1
14

---

Under review as a conference paper at ICLR 2020
100
Accuracies
20
biased unbiased
28
Receptive Fields
Figure 7: Receptive fields of C. Biased and unbiased accuracies of REBI'S S with F LeNet and BlindNct with varying receptive fields.
monkey
dog
turtle
monkey
turtle
"Face" Cluster Samples
"Water" Cluster Samples
(c) "Grass" Cluster Samples
Figure 8: Texture-class correlation. We show samples from each texture cluster. For each cluster, we visualise its top-3 correlated classes in rows.
Results are shown in Figure 7 We observe relatively stable biased set performances and decreasing unbiased performances. The decrease in de-biasing ability is attributed to the violation of the precision condition for G' (:2.3 x receptive fields are sufficient to capture any colour bias variations, and de-biasing against models of larger receptive fields will further make f not see the meaningful signal S (cues beyond 1 x 1-expressible colours). In the extreme case, when F LeNet is trained be independent against itself G LeNet, the de-biasing performance drops significantly.
TEXTURE CLUSTERING
In our ImageNet experiments (4.3). obtained proxy ground truths for the local pattern bias using
texture clustering. We extract texture information from images by clustering the gram matrices of low-layer feature maps as done standard texturisation methods (Gatys al.) 2015 Johnssnn al.
2016); we use feature maps from layer relul 2 of a pre-trained VGG16 (Simonyan & Zisserman
2014 As we intend evaluate whether given model biased towards local pattern cues, we only utilise features from the lower layer encoding lower layer features like edges and colours rather than high-level semantics. Figure g shows that each cluster effectively captures similar local patterns across different classes. For each cluster, visualise its top-3 correlated classes. We can see that certain classes share a common texture: cat, monkey, and dog share a "face" -like texture. If a certain class is biased towards particular texture during training, model can take the shortcut to utilise texture cues for recognising the target class, leading to sub-optimal cross- tross-bias generalisation to unusual class-texture combinations (e.g. crab grass).
15

---

Under review as a conference paper at ICLR 2020
"Water"
"Face"
"Vivid"
"Mammal"
"Grass"
"Shell"
"Spotled"
"Close-up"
"Dog"
"Human"
Figure 9: More clustering samples. Extended version of Figure g
Dog
100
100
99
100
100
100
89
0.7
ye
100
91
10D
75
100
93
99
92
67
75
40
100
83
0.6
80
72
Monkey
100
0.6
100
99
100
95
100
Normalons Aordd Chaster Stae
ImaggeNet Cless
Frog
98
95
78
78
29
43
58
75
100
100
97
84
50
50
Crab
94
69
75
100
75
100
0.4
0.3
Bird
ns
95
100
100
100
57
95
93
96
100
80
Turtle
100
94
74
87
100
82
100
95
91
83
100
73
0.2
Fish
02
95
:77
62
96
87
79
0.1
Insect
50
85
90
100
100
98
67
Dog
Face
Mamuaal
Spotted
Shall
Water
Grass
Vivid
Classupp
Human
Texture Cluster
Figure 10: Data and model bias. Rows correspond to image labels and columns correspond texture clusters. Cells are colour-coded according the population of samples of corresponding texture-class pair, each cell, ResNet1 8 and REBI' accuracies are shown pairs.
16

---

Under review as a conference paper at ICLR 2020
100
Unbiased accuracy
PW/
RecNet50
REDI'S 'S
20
40
60
80
100
Epech
Figure 11: Learning curve. De-biased ImageNet accuracies of vanilla ResNet50 and REBI' s trained against B0
G BIAS IN DATA AND MODELS
We study the bias in the 9-class ImageNet data and the models trained them (44.3) Figure 10 shows the statistics of texture biases in data and the model biases that result from them. To measure the dataset bias, empirically observe correlations between texture and target classes by counting the number of samples for each texture-clas pair (c, y) (denoted "population" Pop(c, y) in the main paper). We observe indeed that there does exist strong correlation between texture clusters and class labels. We say that a class has dominant texture cluster if the largest cluster for the class contains more than half of the class samples. out of classes considered has the dominant texture cluster: ("Dog", "Dog"), ("Cat", "Face"), ("Monkey", "Mammal"), ("Frog", "Spotted"), ("Crab", "Shell") and ("Insect", "Close-up").
Figure 10 further shows the accuracies of the baseline ResNet18 18 and REBI' S to indicate the presence of bias in the models, and how REBI' overcomes the bias despite the bias in data itself. We measure the average of accuracies in classes with dominant texture clusters (biased classes) and the average less biased classes. We observe that ResNet shows higher accuracy on biased classes (89.4% than less biased classes (83.9), signifying its bias towards texture. On the other hand, REBI I' S achieves similar accuracies 88. 7%6basee classes) and 88.2% (unbiased classes). We stress that REBI' overcomes the bias even if the training data itself biased.
H LEARNING CURVES ON IMAGENET
We visualise the learning curves for the baseline vanilla 5e and REBI'S trained ResNet50 against BagNet in Figure [1 We observe that REBI' S gradually de-biases rep- resentation beyond the baseline model.
17


Reviews:
[{"review_id": "S1g5ZEg6KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###  Summary \n\nThe paper proposes a method for regularizing neural networks to mitigate certain known biases from the representations learned by CNNs. The authors look at the setting in which the distribution of biases in the train and test set remains the same, but the distribution of targets given biases changes.\n\nTo remove the bias, they propose hand-designing a set of models that rely exclusively on these biases to make predictions. They then train a new unbiased model by making sure that this new model uses sufficiently different information than the biased models to make predictions (By adding a regularization term)\n\n### Decision and reasons\n\nI vote for a weak accept. \n\nStrengths: \n1: The paper is well written with a clearly defined problem-setting. The proposed method is sound and interesting, and the empirical results, thorough. \n\nWeaknesses:\n2: The solution just pushes the problem of 'learning a model that only uses the true signal to make predictions' to 'learning a set of models that only use the noise to make predictions.' It's not clear why the latter is easier than the former. \n\n3: The paper does not have any baselines that directly try to remove the bias (Instead of using the two-step process). As a result, it's hard to judge how meaningful the improvements are. \n\n\n### Supporting arguments for the reasons for the decision.\n\nStrengths: \n1: The paper does a really good job of defining the problem-setting. Contrasting cross-bias with cross-domain and in-distribution makes the goal of the paper very clear. The notation used to formalize the problem setting in Section 2.1 is also clear and concise. Moreover, the experiments on the toy dataset help clarify the proposed solution whereas experiments on Biased MNIST and Imagenet show that it successfully mitigates the bias. Finally, the authors show the importance of each component of the proposed solution by factor analysis.\n\nWeaknesses:\n2: In the most general case, it is not obvious why it is easier to define and learn a set of models that only use noise to make predictions (which is the required first step for their proposed solution) as opposed to learning a model that only uses signal (which is the goal of the problem). These two problems seem equally hard. The paper builds on the premise that in some cases former is easier (i.e. in some cases, it is easier to learn a set of models that use only noise as opposed to learning a debiased model directly). The authors give two such examples (They only explore the first experimentally.) \n1. Learning a model that relies on local texture. They achieve this by limiting the receptive field of the features. \n2. Learning a model that relies on static images to make predictions about actions in videos. \nI feel that the two given examples are very narrow. It would be nice if the authors could identify a broader class of problems for which G is given or can easily be defined. Moreover, even in these two examples, I'm not convinced that the proposed biased models only use B for making predictions. For example, for some classification tasks, the local texture could be part of the signal and not just the bias. Similarly, static images from a video do contain important information for making the prediction. \n\n3: The authors only compare their method to a baseline that does nothing to debias the representations. Even though this is an important comparison (as it shows that the proposed method can debias the representations), it does not tell the reader how effective the proposed method is compared to other possible solutions. The results would be more meaningful if the authors could include at-least a simple baseline that tries to remove the bias in other ways (For example, they could use the style-transfer baseline used by Geirhos et al., 2019). \n\nI vote for accepting the paper as a poster. It introduces an interesting approach for debiasing representations. However, due to its narrow scope and missing baselines, I would not recommend the paper for an oral presentation. \n\n\n### Questions\n\n1. What are some broader class of problems for which defining G is easier than directly regularizing for debiased representations?\n\n2. How well do Bagnets alone perform on the benchmarks in Table 3? I would expect to see that Bagnets alone do worse than vanilla Resnets on Unbiased and IN-A. Is that so? \n\n3. How well do other methods do in these domains? (Such as methods that directly debias the training data against texture by applying style-transfer). \n\n\n### Update after Author's response\n\nThe author's response has clarified the motivation behind the proposed approach to an extent. They have also added a comparison with a method that directly promotes learning the shape as opposed to the texture ( by training on stylized Imagenet) \n\nI agree with R1 on all accounts (i.e. it is very hard to define the family of biased feature extractors, the proposed approach is ad-hoc, the authors need to compare to texture-shape disentanglement methods, etc), however at the same time, I can see that proposed approach can act as a useful heuristic for regularizing neural networks to pay attention to certain kind of information. \n\nAn interesting use-case of the proposed method (which the authors indirectly mentioned in their response to my review) is in a multi-modal setting. It's not trivial to enforce deep learning systems to utilize all data modalities in a multi-modal setting. By defining G to be models trained on individual modalities, it would be possible to nudge our models to pay attention to the information in all modalities. \n\nSome important results in deep learning have been ad-hoc (For example skip connections in deep networks, ReLUs) and have nonetheless progressed the field. This work is not as widely applicable as skip connections or ReLUs, but it is, nonetheless, providing a heuristic for solving an important problem. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "ByewI_D15B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a methodology for reducing model dependance on bias by specifying a model family of biases (i.e. conv nets with only 1x1 convs to model color biases), and then forcing independence between feature representations of the bias model and the a full model (i.e. conv nets with 3x3 convs to also model edges). \n\nOverall the method is very interesting. A few very related recent works were missing: https://arxiv.org/pdf/1908.10763.pdf and https://arxiv.org/abs/1909.03683. These works provide different formulations for factoring out know \"bias oriented\" models (and focus more on NLP, although the second is applied to VQA). Although, I do appreciate that the bias models used in this work are slightly more general in terms of family than those studied before, they do encode significant intuition about the target bias to be removed and perhaps in this context, the methods cited above could also be compared. That being said, I don't feel the paper needs to provide this comparison given how recent those works are, but it would improve the quality of the paper. \n\nOne aspect that worries me about this paper is that most of the biases studied are synthetic, so specification of the bias family is trivial, in contrast to the works I mentioned above (where the bias model is potentially somewhat misspecified and needed to be discovered by different researchers). But I do really like the experiments on Imagenet-a. \n\nOverall the paper presents an interesting contribution that would be useful for future study in reducing bias dependance in ml.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gpPXAK5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript discusses the problem of bias shortcut employed by many machine learning algorithms (due to dataset problems or underlying effects of any algorithmic bias within an application). The authors argue that models tend to underutilize their capacities to extract non-bias signals when bias shortcuts provide enough cues for recognition. This is an interesting and important aspect of machine learning models neglected by many recent developments. \n\nThe only problem is that the paper seems to be a bit immature as the exemplar application is too naive for illustrating the idea. The authors’ idea is to assume that\n‘there is a family of feature extractors, such that features learned by any of these extractors would correspond to pure bias. Then in order to learn unbiased features, the goal is to construct a feature extractor that is ''as different as this family\" as possible’\nin practice, their claim is texture and color are biases; one should learn shape instead of texture and color. So the family of feature extractors are the ones with small receptive field that can only capture texture and color. Therefore, what they eventually achieved is the unbiased feature extractor only learns the shape of object and avoids learning any texture and color. \n\nSo, the problem is, in practice, it is very hard to define the family of biased feature extractors. It really depends on the dataset and the goal. Texture and color, in general, are still important cues for object recognition, removing this information is NOT equivalent to removing bias. Just as a suggestion, the background scene might be a better definition of bias. However, with the proposal in this paper, it would be unclear how to define the family of feature extractors for describing background. Therefore, the solution given for this important problem seems to be too ad-hoc and not generalizable. \n\nThe second example (that does not have an experiments on) is action recognition; the family of biased feature extractors is 2D-frame-wise CNNs (object recognition). The authors claim that objects are biases for action recognition systems, but again a large part of action recognition is indeed object recognition. Many actions are defined based interaction of humans with objects (e.g., opening bottle or pouring water from bottle). Some objects may be instroducing bias in the task, but not all. Again, the proposed solution in this paper cannot disentangle this. \n\nThe authors need to survey previous texture-shape disentanglement works and then compare with those methods.\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides and analyzes an interesting approach to "de-biasing" a predictor from its training set.  The work is valuable, however unfortunately just below the borderline for this year.  I urge the authors to continue their investigations, for instance further addressing the reviewer comments below (some of which are marked as coming after the end of the feedback period).
Decision: Reject