Adversarial Filters of Dataset Biases

Large-scale benchmark datasets have been among the major driving forces in AI , supporting training of models and measuring their progress . The key assumption is that these benchmarks are realistic approximations of the target tasks in the real world . However , while machine performance on these benchmarks advances rapidly — often surpassing human performance — it still struggles on the target tasks in the wild . This raises an important question : whether the surreal high performance on existing benchmarks are inflated due to spurious biases in them , and if so , how we can effectively revise these benchmarks to better simulate more realistic problem distributions in the real world . In this paper , we posit that while the real world problems consist of a great deal of long-tail problems , existing benchmarks are overly populated with a great deal of similar ( thus non-tail ) problems , which in turn , leads to a major overestimation of true AI performance . To address this challenge , we present a novel framework of Adversarial Filters to investigate model-based reduction of dataset biases . We discuss that the optimum bias reduction via AFOPTIMUM is intractable , thus propose AFLITE , an iterative greedy algorithm that adversarially filters out data points to identify a reduced dataset with more realistic problem distributions and considerably less spurious biases . AFLITE is lightweight and can in principle be applied to any task and dataset . We apply it to popular benchmarks that are practically solved — ImageNet and Natural Language Inference ( SNLI , MNLI , QNLI ) — and present filtered counterparts as new challenge datasets where the model performance drops considerably ( e.g. , from 84 % to 24 % for ImageNet and from 92 % to 62 % for SNLI ) , while human performance remains high . An extensive suite of analysis demonstrates that AFLITE effectively reduces measurable dataset biases in both the synthetic and real datasets . Finally , we introduce new measures of dataset biases based on K-nearest-neighbors to help guide future research on dataset developments and bias reduction . 1 INTRODUCTION . Large-scale neural networks have achieved superhuman performance across many popular AI benchmarks , for tasks as diverse as image recognition ( ImageNet ; Russakovsky et al . ( 2015 ) ) , natural language inference ( SNLI ; Bowman et al . ( 2015 ) ) , and question answering ( SQuAD ; Rajpurkar et al . ( 2016 ) ) . Yet these deep models struggle when taken out of these dataset environments and evaluated on adversarial data or problems in the wild ( Eykholt et al. , 2018 ; Jia & Liang , 2017 ) . This raises a key question : Does high model performance on today ’ s benchmark datasets indicate the underlying task is solved , or do those datasets overestimate the true capabilities of current AI systems ? Answering this question is key because benchmarks serve important roles in the community . Not only do they direct progress on core tasks , they also make it easier to tackle the lofty target tasks such as image recognition in the wild through a more practically-scoped dataset such as ImageNet . However , the closed-world assumption of most existing datasets is subject to significant bias ( Torralba & Efros , 2011 ) . Much of the data that is easy to obtain and label isn ’ t necessarily representative of the task we seek to measure . Thus , if left unchecked , artifacts from data collection ( Fouhey et al. , 2018 ) or human labeling ( Gururangan et al. , 2018 ; Poliak et al. , 2018 ; Tsuchiya , 2018 ; Geva et al. , 2019 ) can significantly inflate model performance . Though there exist task- and dataset-specific ap- proaches for addressing these biases ( Goyal et al. , 2017 ; Geirhos et al. , 2018 ) , the complex artifacts that emerge from large-scale dataset creation are challenging to exhaustively identify and remove . In this paper , we present AFLITE – a computationally efficient dataset reduction algorithm , aimed at systematically reducing spurious artifacts in a dataset . AFLITE is general and can be applied to any task and dataset . Our approach leverages a high capacity model to learn dataset specific biases on a small subset , then uses it to identify and filter artifact-prone instances in the remainder of the dataset to yield a final dataset that is possibly closer to the intended task . We first evaluate the effectiveness of our method on synthetic data and show that AFLITE lowers the performance of models relying on annotation artifacts while preserving the performance of models whose representation captures the underlying tasks . In addition , while AFLITE aims to retain the more challenging , confusing instances , our experiments show that it can successfully remove biased instances that are adversarial to the correct representation of the data . Finally , we apply the method to several benchmark datasets across various tasks and domains . In language understanding , we apply AFLITE to the SNLI ( Bowman et al. , 2015 ) and MNLI ( Williams et al. , 2018 ) datasets for natural language inference , and to QNLI ( Wang et al. , 2018 ) for question answering . We show a 30 % absolute gap in performance in the current state-of-art methods before and after AFLITE . In computer vision , AFLITE reduces the performance of image classification neural methods on ImageNet ( Russakovsky et al. , 2015 ) , showing a 49 % absolute gap . 2 DATASET REDUCTION FOR REPRESENTATION-BIAS MINIMIZATION . In this section , we introduce AFLITE , a general approach for reducing the scope of bias in datasets . Large datasets run the risk of prioritizing performance on the data-rich head of the distribution , where examples are plentiful , and discounting the tail . Our goal is to minimize the ability of a model to exploit biases in the head of the distribution , while preserving the inherent complexity of the tail . Let Φ represent a feature representation , defined over a datasetD = ( X , Y ) . With AFLITE , we seek a subset S ⊂ D of size |S| ≥ n that is maximally resilient to the features uncovered by Φ . For any identically-distributed train-test split of D , the features extracted by Φ should not generalize to the held-out set . Our approach allows for any choice of feature representation . Formalization More formally , let M denote a family of classification models ( e.g. , logistic regression , SVM , or a particular neural architecture ) that can be trained on subsets S of D = ( X , Y ) using features Φ ( X ) . We define the representation bias of Φ in S w.r.tM , denotedR ( Φ , S , M ) , as the best possible out-of-sample classification accuracy achievable by models inM when predicting the true labels Y using features Φ ( X ) . For a given target reduced dataset size of at least n , the goal is to find a subset S ⊂ D , |S| ≥ n that minimizes this representation bias in S w.r.t.M : min S⊂D , |S|≥n R ( Φ , S , M ) ( 1 ) Eq . ( 1 ) corresponds to the optimum bias reduction , referred to as AFOPTIMUM . R ( Φ , S , M ) can be formulated as the expected classification accuracy resulting from the following process . Let q : 2S → [ 0 , 1 ] be a probability distribution over subsets T = ( XT , Y T ) of D. The process is to randomly choose a subset T with probability q ( T ) , train a bias estimator MT ∈ M on D \ T , and evaluate its classification accuracy fMT ( Φ ( X T ) , Y T ) on T . Note that the resulting classification accuracy on T itself is a random variable , since the training set D \ T is random . We define the expected value of this classification accuracy to be the representation bias : R ( Φ , S , M ) , ET∼q [ fMT ( Φ ( X T ) , Y T ) ] ( 2 ) While this expression formalizes the intended objective function , it involves a large summation over subsets T ⊂ S just to compute the representation bias present in a single set S. It does not suggest a practical way to compute the minimization in Eq . ( 1 ) without further considering each of the exponentially many subsets S ⊂ D individually – thus an optimal solution for Equation ( 2 ) is intractable . To get around this difficulty , we reformulate the representation bias in S as a sum factored over the |S| individual instances i ∈ S. This will allow us to efficiently decide whether or not to include i in the targeted , reduced subset we are constructing . The idea is to aggregate the contribution of each i towards the representation bias expression across all random choices of the training set D \ T . We call this the predictability score p ( i ) for i : on average , how reliably can the label yi be predicted using features Φ ( xi ) when a model fromM is trained on a randomly chosen training set D \ T not containing i . The higher the value of p ( i ) , the easier it is to correctly classify the instance ( xi , yi ) using model familyM . This is the signal we will use to decide whether to include i in the reduced subset S we are constructing . With some abuse of notation , for i ∈ D , let q ( i ) , ∑ T3i q ( T ) denote the marginal probability of choosing a subset T that contains i . The ratio q ( T ) q ( i ) is then the probability of T conditioned on it containing i . Let fMT ( Φ ( xi ) , yi ) be the classification accuracy of MT on i . The reformulation of representation bias in terms of predictability scores of individual instances works as follows : ET∼q [ fMT ( Φ ( X T ) , Y T ) ] = ∑ T⊂S q ( T ) · 1|T | ∑ i∈T fMT ( Φ ( xi ) , yi ) = ∑ T⊂S ∑ i∈T q ( T ) · fMT ( Φ ( xi ) , yi ) |T | = ∑ i∈S ∑ T⊂S T3i q ( T ) · fMT ( Φ ( xi ) , yi ) |T | = ∑ i∈S q ( i ) ∑ T⊂S T3i q ( T ) q ( i ) fMT ( Φ ( xi ) , yi ) |T | = ∑ i∈S q ( i ) ET⊂S , T3i [ fMT ( Φ ( xi ) , yi ) |T | ] = ∑ i∈S p ( i ) where p ( i ) is the predictability score of i defined as : p ( i ) , q ( i ) ET⊂S , T3i [ fMT ( Φ ( xi ) , yi ) |T | ] ( 3 ) While the method works for any probability distribution q with non-zero support on all samples , for simplicity of exposition , we restrict q to be the uniform distribution over all subsets T ⊂ S of a fixed size . This makes both |T | and q ( i ) fixed constants ; in particular , q ( i ) = ( |S|−1 |T |−1 ) / ( |S| |T | ) = |T ||S| . This reduces the predictability score expression of Eq . ( 3 ) to the simplified variant p̃ ( i ) : p̃ ( i ) , 1 |S| ET⊂S , T3i [ fMT ( Φ ( xi ) , yi ) ] ( 4 ) Putting the pieces together , we have a factored reformulation of the representation bias in Eq . ( 2 ) : R ( Φ , S , M ) = ∑ i∈S p̃ ( i ) ( 5 ) Armed with this factored representation , we return to the task of identifying an S ⊂ D , |S| ≥ n that minimizes the representation bias . We use the simplified predictability scores ( henceforth simply referred to as predictability scores ) as a heuristic metric to decide which i ∈ D to include in S. We consider three approaches that iteratively filter out the most predictable instances fromD to arrive at S. In all cases , we use a fixed training set size |S \ T | = t < n. Further , since a larger filtered set is generally desirable , we terminate the filtering process early ( i.e. , while |S| > n ) if the predictability score for every i falls below a pre-specified early stopping threshold τ ∈ [ 0 , 1 ] . The three approaches are as follows . ( A ) A simple greedy approach starts with the full set S = D , identifies an i ∈ S with the highest predictability score , removes it from S , and repeats up to |D|−n times . ( B ) A greedy slicing approach identifies the instances with the k highest predictability scores , removes all of them from S , and repeats the process up to b |D|−nk c times . ( C ) A slice sampling approach where , instead of greedily choosing the top k instances , it randomly samples k instances with probabilities proportional to their predictability scores.1 In this paper , we use the greedy slicing approach in our experiments and refer to it as AFLITE . While the optimum bias reduction via AFOPTIMUM is intractable , its light-weight version , AFLITE , is applicable in practice . The slice sampling approach can be efficiently implemented using what is known as the Gumbel method or Gumbel trick ( Gumbel & Lieblein , 1954 ; Maddison et al. , 2014 ) , which uses random perturbations to turn sampling into a simpler problem of optimization . This has recently found success in several probabilistic inference applications ( Kim et al. , 2016 ; Jang et al. , 2016 ; Maddison et al. , 2016 ; Balog et al. , 2017 ; Kool et al. , 2019 ) . Starting with the log-predictability scores log p̃ ( i ) for various i , the idea is to perturb them by adding an independent random noise γi drawn from the standard Gumbel distribution . Interestingly , the maximizer i∗ of γi+log p̃ ( i ) turns out to be an exact sample drawn from the ( unnormalized ) distribution defined by p̃ . Note that i∗ is a random variable since the γi are drawn at random . This result can be generalized ( Vieira , 2014 ) for slice sampling : the k highest values of Gumbel-perturbed log-predictability scores correspond to sampling , without replacement , k items from the probability distribution defined by p̃ . The Gumbel method is typically applied to exponentially large combinatorial spaces , where it is challenging to scale up . In our setting , however , the overhead is minimal since the cost of drawing a random γi is negligible compared to computing p̃ ( i ) . Implementation Algorithm 1 provides an implementation of AFLITE . The algorithm takes as input a dataset D = ( X , Y ) , a representation Φ ( X ) we are interested in minimizing the bias in , a model family M ( e.g. , linear classifiers ) , a target dataset size n , size m of the support of the expectation in Eq . ( 4 ) , training set size t for the classifiers , size k of each slice , and an early-stopping filtering threshold τ . Importantly , for efficiency , Φ ( X ) is provided to AFLITE in the form of precomputed embeddings for all of X . To obtain Φ ( X ) in practice , we train a first model on a small fraction of the data based on the learning curve in low-data regime , and do not reuse this data for the rest of our experiments . Moreover , this fraction corresponds to the training size t for AFLITE and it remains unchanged across iterations . We follow the iterative filtering approach , starting with S = D and iteratively removing some instances with the highest predictability scores using the 1All of these approaches can be improved further by considering not only the predictability score of the top-k instances i but also ( via retraining without these instances ) how their removal would influence the predictability scores of other instances in the next step . We found our computationally lighter approaches to work well even without the additional overhead of such look-ahead . Algorithm 1 : AFLITE Input : dataset D = ( X , Y ) , pre-computed representation Φ ( X ) , model familyM , target dataset size n , number of random partitions m , training set size t < n , slice size k ≤ n , early-stopping threshold τ Output : reduced dataset S 1 S = D 2 while |S| > n do // Filtering phase 3 forall i ∈ S do 4 Initialize a multi-set of out-of-sample predictions E ( i ) = ∅ 5 for iteration j : 1 .. m do 6 Randomly partition S into ( Tj , S \ Tj ) s.t . |S \ Tj | = t 7 Train a classifier L ∈M on { ( Φ ( x ) , y ) | ( x , y ) ∈ S \ Tj } ( L is typically a linear classifier ) 8 forall i = ( x , y ) ∈ Tj do 9 Add the prediction L ( Φ ( x ) ) to E ( i ) 10 forall i = ( x , y ) ∈ S do 11 Compute the predictability score p̃ ( i ) = | { ŷ ∈ E ( i ) s.t . ŷ = y } | / |E ( i ) | 12 Select up to k instances S′ in S with the highest predictability scores subject to p̃ ( i ) ≥ τ 13 S = S \ S′ 14 if |S′| < k then 15 break 16 return S greedy slicing strategy . Slice size k and number of partitions m are determined by the available computation budget . At each filtering phase , we train models ( linear classifiers in our implementation ) on m different random partitions of the data , and collect their predictions on their corresponding test set . For each instance i , we compute its predictability score as the ratio of the number of times its label yi is predicted correctly , over the total number of predictions for it . We rank the instances according to their predictability score and use the greedy slicing strategy of removing the top-k instances whose score is not less than the early-stopping threshold τ . We repeat this process until fewer than k instances pass the τ threshold in a filtering phase or fewer than n instances remain .

Summary: This paper proposes to learn a subset of a given dataset that acts as an adversary, that hurts the model performance when used as a training dataset. The central claim of the paper is that existing datasets on which models are trained are potentially biased, and are not reflective of real world scenarios. By discarding samples that add to this bias, the idea is to make the model perform better in the wild. The authors propose a method to do so, and then refine it so that the resulting solution is tractable. They implement the method on several datasets and show that by finding these adversarial samples, they indeed hurt model performance.