Missingness Bias in Model Debugging

1 INTRODUCTION . Model debugging aims to diagnose a model ’ s failures . For example , researchers can identify global biases of models via the extraction of human-aligned concepts ( Bau et al. , 2017 ; Wong et al. , 2021 ) , or understand the texture bias by analyzing the models performance on synthetic datasets ( Geirhos et al. , 2019 ; Leclerc et al. , 2021 ) . Other approaches aim to highlight local features to debug individual model predictions ( Simonyan et al. , 2013 ; Dhurandhar et al. , 2018 ; Ribeiro et al. , 2016a ; Goyal et al. , 2019 ) . A common theme in these methods is to compare the behavior of the model with and without certain individual features ( Ribeiro et al. , 2016a ; Goyal et al. , 2019 ; Fong & Vedaldi , 2017 ; Dabkowski & Gal , 2017 ; Zintgraf et al. , 2017 ; Dhurandhar et al. , 2018 ; Chang et al. , 2019 ) . For example , interpretability methods such as LIME ( Ribeiro et al. , 2016b ) and integrated gradients ( Sundararajan et al. , 2017 ) use the predictions when certain features are removed from the input to attribute different regions of the input to the decision of the model . Dhurandhar et al . ( 2018 ) find minimal regions in radiology images that are necessary for classifying a person as having autism . Fong & Vedaldi ( 2017 ) propose learning image masks that minimize a class score to achieve interpretable explanations . Similarly , in natural language processing , model designers often remove individual words to understand their importance to the output ( Mardaoui & Garreau , 2021 ; Li et al. , 2016 ) . The absence of features from an input , a concept sometimes referred to as missingness ( Sturmfels et al. , 2020 ) , is thus fundamental to many debugging tools . However , there is a problem : while we can easily remove words from sentences , removing objects from images is not as straightforward . Indeed , removing a feature from an image usually requires approximating missingness by replacing those pixel values with something else , e.g. , black color . However , these approximations tend to not be perfect ( Sturmfels et al. , 2020 ) . Our goal is thus to give a holistic understanding of missingness and , specifically , to answer the question : How do missingness approximations affect our ability to debug ML models ? OUR CONTRIBUTIONS . In this paper , we investigate how current missingness approximations , such as blacking out pixels , can result in what we call missingness bias . This bias turns out to hinder our ability to debug models . We then show how transformer-based architectures can enable a more natural implementation of missingness , allowing us to side-step this bias . More specifically , our contributions include : Pinpointing the missingness bias . We demonstrate at multiple granularities how simple approximations , such as blacking out pixels , can lead to missingness bias . This bias skews the overall output distribution toward unrelated classes , disrupts individual predictions , and hinders the model ’ s use of the remaining ( unmasked ) parts of the image . Studying the impact of missingness bias on model debugging . We show that missingness bias negatively impacts the performance of debugging tools . Using LIME—a common feature attribution method that relies on missingness—as a case study , we find that this bias causes the corresponding explanations to be inconsistent and indistinguishable from random explanations . Using vision transformers to implement a more natural form of missingness . The tokencentric nature of vision transformers ( ViT ) ( Dosovitskiy et al. , 2021 ) facilitates a more natural implementation of missingness : simply drop the corresponding tokens of the image subregion we want to remove . We show that this simple property substantially mitigates missingness bias and thus enables better model debugging . 2 MISSINGNESS . Removing features from the input is an intuitive way to understand how a system behaves ( Sturmfels et al. , 2020 ) . Indeed , by comparing the system ’ s output with and without specific features , we can infer what parts of the input led to a specific outcome ( Sundararajan et al. , 2017 ) —see Figure 1 . The absence of features from an input is sometimes referred to as missingness ( Sturmfels et al. , 2020 ) . The concept of missingness is commonly leveraged in machine learning , especially for tasks such as model debugging . For example , several methods for feature attribution quantify feature importance by studying how the model behaves when those features are removed ( Sturmfels et al. , 2020 ; Sundararajan et al. , 2017 ; Ancona et al. , 2017 ) . One commonly used method , LIME ( Ribeiro et al. , 2016a ) , iteratively turns image subregions on and off in order to highlight its important parts . Similarly , integrated gradients ( Sundararajan et al. , 2017 ) , a typical method for generating saliency maps , leverages a “ baseline image ” to represent the “ absence ” of features in the input . Missingness-based tools are also often used in domains such as natural language processing ( Mardaoui & Garreau , 2021 ; Li et al. , 2016 ) and radiology ( Dhurandhar et al. , 2018 ) . Challenges of approximating missingness in computer vision . While ignoring parts of an image is simple for humans , removing image features is far more challenging for computer vision models ( Sturmfels et al. , 2020 ) . After all , convolutional networks require a structurally contiguous image as an input . We thus can not leave a “ hole '' in the image where the model should ignore the input . Consequently , practitioners typically resort to approximating missingness by replacing these pixels with other , intended to be “ meaningless ” , pixels . Common missingness approximations include replacing the region of the image with black color , a random color , random noise , a blurred version of the region , and so forth ( Sturmfels et al. , 2020 ; Ancona et al. , 2017 ; Smilkov et al. , 2017 ; Fong & Vedaldi , 2017 ; Zeiler & Fergus , 2014 ; Sundararajan et al. , 2017 ) . However , there is no clear justification for why any of these choices is a good approximation of missingness . For example , blacked out pixels are an especially popular baseline , motivated by the implicit heuristic that near zero inputs are somehow neutral for a simple model ( Ancona et al. , 2017 ) . However , if only part of the input is masked or the model includes additive bias terms , the choice of black is still quite arbitrary . In ( Sturmfels et al. , 2020 ) , the authors found that saliency maps generated with integrated gradients are quite sensitive to the chosen baseline color , and thus can change significantly based on the ( arbitrary ) choice of missingness approximation . 2.1 MISSINGNESS BIAS . What impact do these various missingness approximations have on our models ? We find that current approximations can cause significant bias in the model ’ s predictions . This causes the model to make errors based on the “ missing ” regions rather than the remaining image features , rendering the masked image out-of-distribution . Figure 2 depicts an example of these problems . If we mask a small portion of the image , irrespective of which part of the image that is , convolutional networks ( CNNs ) output the wrong class . In fact , CNNs seem to be relying on the masking pattern to make the prediction , rather than the remaining ( unmasked ) portions of the image . This type of behavior can be especially problematic for model debugging techniques , such as LIME , that rely on removing image subregions to assign importance to input features . Further examples can be found in Appendix C.1 . There seems to be an inherent bias accompanying missingness approximations , which we refer to as the missingness bias . In Section 3 , we systematically study how missingness bias can affect model predictions at multiple granularities . Then in Section 4 , we find that missingness bias can cause undesirable effects when using LIME by causing its explanations to be inconsistent and indistinguishable from random explanations . 2.2 A MORE NATURAL FORM OF MISSINGNESS VIA VISION TRANSFORMERS . The challenges of missingness bias raises an important question : what constitutes a correct notion of missingness ? Since masking pixels creates biases in our predictions , we would ideally like to remove those regions from consideration entirely . Because convolutional networks slide filters across the image , they require spatially contiguous input images . We are thus limited to replacing pixels with some baseline value ( such as blacking out the pixels ) , which leads to missingness bias . Unlike CNNs , vision transformers ( ViTs ) ( Dosovitskiy et al. , 2021 ) operate on sets of image tokens , each of which correspond to a positionally encoded region of the image . ViTs thus enable a far more natural implementation of missingness : simply drop the tokens that correspond to the regions of the image we want to “ remove. ” Instead of replacing the masked region with other pixel values , we can modify a forward pass of the ViT to directly remove the region entirely . We will refer to this implementation of missingness as dropping tokens throughout the paper ( see Appendix B for further details ) . As we will see , using ViTs to drop image subregions will allow us to side-step missingness bias , and thus enable better model debugging . 3 THE IMPACTS OF MISSINGNESS BIAS . Section 2.1 featured several qualitative examples where missingness approximations affect the model ’ s predictions . Can we get a precise grasp on the impacts of such missingness bias ? In this section , we pinpoint how missingness bias can manifest at several levels of granularity . We further demonstrate how , by enabling a more natural implementation of missingness through dropping tokens , ViTs can avoid these biases . To systematically measure the impacts of missingness bias , we iteratively remove subregions from the input and analyze the types of mistakes that our models make 1 . Our assessment of missingness bias , from the overall class distribution to individual examples , is guided by the following questions : To what extent do missingness approximations skew the model ’ s overall class distribution ? . We find that missingness bias affects the model ’ s overall class distribution ( i.e the probability of predicting any one class ) . In Figure 3 , we measure the shift in the model ’ s output class distribution before and after image subregions are randomly removed . The overall entropy of output class distribution degrades severely . In contrast , this bias is eliminated when dropping tokens with the ViT . The ViT thus maintains a high class entropy corresponding to a roughly uniform class distribution . These findings hold regardless of what order we remove the image patches ( see Appendix C.2 ) . Does removing random or unimportant regions flip the model ’ s predictions ? We now take closer look at how missingness approximations can affect individual predictions . In Figure 4 , we plot the fraction of examples where removing a portion of the image flips the model ’ s prediction . We find that the ResNet rapidly flips its predictions even when the less relevant regions are removed first . This degradation is thus more likely due to missingness bias rather than the removal of individual 1See Appendix A for experimental details . We perform an extensive study across various missingness approximations , architectures , and subregion sizes . Here we present the findings on a single representative setting : removing 16 × 16 patches from an image through blacking out ( ResNet-50 ) and dropping tokens ( ViTS ) . We defer the remaining experiments , which lead to similar conclusions , to Appendix C. We further consider other datasets in Appendix E regions . In contrast , the ViT maintains its original predictions even when large parts of the image are removed . Do remaining unmasked regions produce reasonable predictions ? When removing regions of the image with missingness , we would hope that the model makes a “ best-effort ” prediction given the remaining image features . This assumption is critical for interpretability methods such as LIME ( Ribeiro et al. , 2016a ) , where crucial features are identified by iteratively masking out image subregions and tracking the model ’ s predictions . Are our models actually using the remaining uncovered features after missingness approximations are applied though ? To answer this question , we measure how semantically related the model ’ s predictions are after masking compared to its original prediction using a similarity metric on the WordNet Hierarchy ( Miller , 1995 ) as shown in Figure 6 . By the time we mask out 25 % of the image , the predictions of the ResNet largely become irrelevant to the input . ViTs on the other hand continue to predict classes that are related to the original prediction . This indicates that ViTs successfully leverage the remaining features in the image to provide a reasonable prediction . Can we remove missingness bias by augmenting with missingness approximations ? One way to remove missingness bias could be to apply missingness approximations during training . For example , in RemOve and Retrain ( ROAR ) , Hooker et al . ( 2018 ) suggest retraining multiple copies of the model by randomly blacking out pixels during training ( see Appendix F for an overview on ROAR ) . To check if this indeed helps side-step the missingness bias , we retrain our models by randomly removing 50 % of the patches during training , and again measure the fraction of examples where removing image patches flips the model ’ s prediction ( see Figure 5 ) . While there is a significant gap in behavior between the standard and retrained CNNs , the ViT behaves largely the same . This result indicates that , while retraining is important when analyzing CNNs , it is unnecessary for ViTs when dropping the removed tokens : we can instead perform missingness approximations directly on the original model while avoiding missingness bias for free . See Appendix F for more details .

Summary: The authors focus on the problem of model debugging (for image recognition). They identify that the proposed tools (that rely on CNNs and ResNets) might suffer from the ‘missingness’ issue, i.e., the absence of features due to masking objects of interest. The authors exhibit how the method of masking pixels/patches can lead to the missingness bias and this can happen even if the masked values are replaced with some other 'dummy' values. To mitigate that, they propose to use a recent transformer since it does not use convolutions, but linearized patches as inputs. Then, they demonstrate that the proposed method can simply 'skip' those patches that are masked and it does not result in a skewed result.