Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset

1 INTRODUCTION . Deep neural networks have been widely adopted in many domains . Yet , for some applications , their use may be limited by how little we understand which features are relevant . Whether engineer or user , insurance company , or regulatory body ; all require reliable information about what the model has learned or why the model provides a certain output . Numerous methods have been proposed to explain deep neural networks ( Gilpin et al. , 2018 ; Molnar et al. , 2020 ) . Ultimately , to evaluate whether such explanations are helpful , we need user studies ( Doshi-Velez & Kim , 2017 ; Wortman Vaughan & Wallach , 2020 ) . In fact , some studies provided evidence that interpretable ML techniques may be helpful to find biases or spurious correlations ( Ribeiro et al. , 2016b ; Adebayo et al. , 2020a ) . However , a substantial body of work shows that they may not be as helpful as claimed ( Kaur et al. , 2020 ; Alqaraawi et al. , 2020 ; Chu et al. , 2020 ; Shen & Huang , 2020 ) . Consequently , it seems that in real-world applications , biases are often found by simply inspecting the model ’ s predictions rather than applying interpretable ML . A recent example is the Twitter image cropping algorithm : it was the users who discovered that it favored white people over people of color ( Yee et al. , 2021 ) . In this work , we ask : do modern interpretability methods enable users to discover biases better than by simply inspecting input/output pairs ? To investigate this question empirically , we first propose TWO4TWO : a synthetic dataset depicting two abstract animals . Its data-generating factors can be correlated with the binary target class , thereby creating arbitrarily strong biases . We designed a baseline explanation technique for bias discovery using only the model ’ s output : input images are arranged in a grid grouped by the model ’ s logit predictions . This design allows a user to inspect all attributes that potentially predict the target class . In an initial user study ( N=50 ) , we validated that participants were struggling to find both biases contained in our dataset using this technique . Hence , more elaborate methods can improve over the baseline on this dataset . In the main study ( N=240 ) , we compared the baseline against two state-of-the-art explanations : automatically-discovered concepts and counterfactual interpolations generated with an invertible neural network . We found that none of these explanations outperformed the baseline , even though some features were identified more accurately with counterfactuals . The textual justifications of participants revealed several usability issues in all methods . This highlights the necessity to validate any claims about the benefits of explanation techniques in user studies . This work represents substantial empirical novelty and significance in the field of interpretable ML : • The TWO4TWO dataset generator provides control over features and biases . It is designed specifically for human subject studies and to challenge existing interpretability approaches , • Methods to quantify ground-truth feature importance when the data-generating process is known , • A study design that provides a unified approach to evaluating interpretable vision methods on the task of bias discovery . It is suitable for lay-users and includes several measures to ensure high-quality crowd-sourced responses , • A strong and simple baseline explanation technique using only the model output , which we propose as a benchmark for future studies , • We open-source our dataset , explanation techniques , model , study design , including instructions and videos to support replicating our results as well as adapting our design to other explanation techniques . 2 RELATED WORK . Interpretable ML for Vision Different explanation approaches have been proposed : saliency maps ( Bach et al. , 2015 ; Ancona et al. , 2018 ; Sundararajan et al. , 2017 ) , example-based explanations ( Cai et al. , 2019 ) , counterfactual examples ( Singla et al. , 2020 ) , activation-concept approaches ( Kim et al. , 2018 ) , or models with built-in interpretability ( Chen et al. , 2019 ; Brendel & Bethge , 2018 ) . For a detailed review about the field , we refer to ( Gilpin et al. , 2018 ; Molnar et al. , 2020 ) . Our work focuses on counterfactual explanations and automatically-discovered concepts . Counterfactual explanations are samples that change the model output , e.g. , flip the output class ( Wachter et al. , 2018 ) . We generated counterfactuals with invertible neural networks ( INNs ) ( Jacobsen et al. , 2018 ; Kingma & Dhariwal , 2018 ) . This approach has recently gained momentum ( Hvilshøj et al. , 2021 ; Dombrowski et al. , 2021 ; Mackowiak et al. , 2020 ) . Previous works have also used GANs and VAEs for counterfactual generation ( Goyal et al. , 2019 ; Mertes et al. , 2020 ; Sauer & Geiger , 2021 ; Singla et al. , 2020 ; Liu et al. , 2019 ; Baumgartner et al. , 2018 ; Chang et al. , 2019 ) . The main advantage of using INNs for counterfactuals is that the generative function is perfectly aligned with the forward function , as an analytic inverse exists . Concepts represent abstract properties , which can be used to explain a model . For example , the classification of an image as ” zebra ” could be explained by a pronounced similarity to the ” stripe ” concept . This similarity is determined by the dot product of the network ’ s internal activations with a concept vector . TCAV ( Kim et al. , 2018 ) required manually defined concepts . Recent works proposed to discover concepts automatically ( Ghorbani et al. , 2019 ; Zhang et al. , 2021 ) . Animal Color Block Shape Body PostureBackgroundLegs stretched outHead peeking out Peeky Stretchy Figure 2 : The left panel depicts the main difference between Peeky and Stretchy : the legs ’ position . While Peeky shows one pair of legs moved inwards , Stretchy ’ s legs are moved outwards . TWO4TWO offers different attributes : animal color , background color , the shape of the blocks and the animal ’ s body posture . All of which can be controlled and biased separately . User Studies for Interpretability Previous works with the task of bias discovery have mainly evaluated saliency maps and used datasets with a single , simple bias , e.g . background Adebayo et al . ( 2020a ) ; Ribeiro et al . ( 2016a ) or image watermarks Kim et al . ( 2018 ) . User studies for concept-based methods tested only the accessibility of the explanations by asking users to assign images to a concept ( Zhang et al. , 2021 ; Ghorbani et al. , 2019 ) . Counterfactual explanations have been evaluated by Mertes et al . ( 2020 ) on a forward-prediction task . We thus believe that we are the first to extensively test counterfactual-based and concept-based explanations on bias discovery using a challenging dataset . Recently , a study on exemplary-based explanations focused on understanding internal activations of a neural network ( Borowski et al. , 2020 ) . It showed that for this task , examples could be more beneficial than complex feature visualizations ( Olah et al. , 2017 ) . Similarly , there is evidence that participants often rely on model predictions rather than on explanations ( Alqaraawi et al. , 2020 ; Adebayo et al. , 2020a ) . Synthetic Datasets for Interpretable Vision Datasets with known ground-truth biases have been proposed before . BAM is an artificial dataset ( Yang & Kim , 2019 ) where spurious background correlations are introduced by pasting segmented objects on different textures , e.g . dogs on bamboo forests . However , the resulting images are unsuitable for user studies as they look artificial and make it easy for participants to suspect that the background is important . Additionally , it would be difficult to introduce more than one bias . A limitation that also the synthetic dataset in ( Chen et al. , 2018 ) shares . The synthetic dataset created by Arras et al . ( 2021 ) created a dataset to technically evaluate saliency methods on a visual question answering task technically . TWO4TWO is the first dataset designed explicitly for human subject evaluations . To the best of our knowledge , we provide the first unified approach to evaluate interpretable vision on a bias-discovery task . 3 TWO4TWO : DATASETS WITH KNOWN FEATURE IMPORTANCE . Dataset Description Datasets generated with TWO4TWO consist of two abstract animal classes , called Peeky and Stretchy . Both consist of eight blocks : four for the spine and four for the legs . For both animals , one pair of legs is always at an extended position . The other pair moves parallel to the spine inward and outward . The attribute legs ’ position , a scalar in [ 0,1 ] , controls the position . At a value of 0.5 , the pair of legs are at the same vertical position as the last block of the spine . Peekies have a leg position ≤ 0.52 which means legs are moved mostly inwards to the body center . In the same fashion , Stretchies are extended outwards , legs ’ position ≥ 0.48 . We added some ambiguity to ensure a model has an incentive to use possible biases . Therefore , Peekies and Stretchies are equally likely for a legs ’ position between 0.48 and 0.52 . It is also difficult for humans to tell if the legs are outward or inwards in this range . Besides the legs ’ position , the dataset has the following parameters : body posture ( bending and three rotation angles ) , position , animal color ( from red to blue ) , blocks ’ shape ( from cubes to spheres ) , and background color ( from red to blue ) . Each can be changed arbitrarily and continuously ( See Appendix Table 5 ) . When designing the dataset , we wanted to ensure that ( 1 ) participants can become experts within a few minutes of training , ( 2 ) it allows for the creation of multiple biases that are difficult to find , and ( 3 ) that it provides a challenge for existing interpretability methods . Goal ( 1 ) is met as participants can be instructed using only a few examples ( see the tutorial video in Appendix C ) . The high number of controllable attributes achieve Goal ( 2 ) . We biased the attributes such that they do not stand out , which we validated in the first user study . Goal ( 3 ) is met by spatially overlapping attributes and long- range image dependencies . Spatially overlapping attributes , like color and shape , directly challenge saliency map explanations . Long-range image dependencies , like the legs ’ positions relative to the spine , can not be explained when analyzing patches separately as done in ( Chen et al. , 2019 ; Brendel & Bethge , 2018 ) . Both properties are common in real-world datasets : For example , race and gender in facial datasets are encoded by spatially overlapping features . Long-range image dependencies are particularly relevant for pose estimation and visual reasoning ( Johnson et al. , 2017 ) . Introducing Biases For our studies ’ dataset , we sampled the block ’ s shape in a non-predictive biased fashion . This means that for legs ’ positions that clearly showed a Peeky [ 0 , 0.45 ] most blocks were rather cubic , while for legs ’ positions that clearly showed a Stretchy [ 0.55 , 1 ] most blocks were rather round . However , for the legs ’ positions between [ 0.45 , 0.55 ] the blocks shape was uniformly distributed . In particular , in the even narrower interval [ 0.48 , 0.52 ] where a classifier can only be as good as random guessing , the block ’ s shape does not provide any additional information about the target class . In Figure 3 , we show the joint distribution of the block ’ s shape and legs ’ position . We sampled the animals ’ color to be predictive for the target class . At the small interval where the legs overlap [ 0.48 ; 0.52 ] , we distributed the animal color to provide additional class information . Stretchies were more likely to be red , and Peekies were more likely to be blue . Outside of this centered interval , the color gradually became uniformly distributed ( see Figure 3 ) . Hence , color was more equally distributed than the shape , making the color bias harder to detect visually . The remaining attributes , background color and body posture , were sampled independently of the class , and we expected our model to ignore them . Measuring Ground-Truth Feature Importance Even if a dataset contains biases , it is unclear how relevant they will be to a neural network after training . Feature importance also depends on the network architecture , the optimization process , and even the weight initialization . As TWO4TWO allows us to change any parameter in isolation , we can directly compare the model prediction between two images that differ in only one parameter . For these two images , we measured both the median absolute logit change and also for how many samples the predicted class was flipped . Both measures quantify how influential each parameter is ( see Table 1 ) . As expected , the legs ’ position had a strong influence on the prediction . The model relied more on animal color than on the blocks ’ shape , which is expected as the color contains additional information about the class . Surprisingly , the prediction flip for unrelated attributes such as background was only slightly lower than for blocks ’ shape . To analyze this further , we calculated a linear fit for each parameter change to the logit change . We reported the coefficient of determination R2 , which indicates how much of the variance in the prediction can be explained linearly by the analyzed property . While the unrelated properties sometimes flip a prediction , the direction of that flip is random ( R2 ≈ 0 ) . In contrast , the biased parameters influence predictions in a directed fashion , with animal color ( R2=0.751 ) being clearly more directed than blocks ’ shape ( R2=0.307 ) .

Summary: The paper introduces an experiment design and an approach to synthesizing the dataset for the experiment. The experiment asked the systems to classify whether the shape of the "animal" is Peaky or Stretchy. The advantages are that authors can controllably generate trainable examples under arbitrary biases of the predefined features (shape, color, etc). For experiments, human subjects are asked to predict the systems' output. The authors compared the visual explanation (concept explanation) and counter-factual explanation with the baseline explanation that uses the output logits and found that the the current good explanation approaches do not make humans understand the system significantly better