On the Effects of Data Distortion on Model Analysis and Training

Data modification can introduce artificial information . It is often assumed that1 the resulting artefacts are detrimental to training , whilst being negligible when2 analysing models . We investigate these assumptions and conclude that in some3 cases they are unfounded and lead to incorrect results . Specifically , we show current4 shape bias identification methods and occlusion robustness measures are biased5 and propose a fairer alternative for the latter . Subsequently , through a series of6 experiments we seek to correct and strengthen the community ’ s perception of how7 distorting data affects learning . Based on our empirical results we argue that the8 impact of the artefacts must be understood and exploited rather than eliminated.9 1 Motivation10 Modifying data has become commonplace both when training and analysing models , yet the wider11 implications are often disregarded . We delve into some of the side-effects and point out that this12 practice has resulted in the creation of biased model interpretation tools and poorly informed theories.13 On the analysis side , we take as examples occlusion robustness and shape bias identification methods.14 On the training side , we focus on some instances of Mixed Sample Data Augmentation ( MSDA ) ,15 where two images are combined to obtain a new training sample . Visual examples of each can16 be found in Figure 1 . In this paper we study a number of assumptions that lie at the heart of the17 aforementioned methods , which we briefly introduce below.18 Shape-texture bias : Deep models are known to be sensitive to interventions that are imperceptible19 to humans [ 35 , 13 ] , as well as to other forms of distribution shifts [ 1 , 6 , 8 ] . It has been argued that20 this is intimately linked to networks tending to use texture rather than shape information [ 2 , 11 ] .21 Recently , input distortions have become a popular way of assessing a model ’ s texture bias . To this end,22 images are divided into a grid and the resulting patches are randomly shuffled such that information23 is preserved locally , while the global shape is altered [ 32 , 27 , 25 , 41 ] . It is implicitly assumed that24 patch-shuffling does not introduce misleading shape or texture that could affect model evaluation.25 Occlusion robustness : A widely adopted method for measuring occlusion robustness is through the26 accuracy obtained after superimposing a rectangular patch on an image [ 5 , 9 , 39 , 42 , 20 ] . We refer27 to this approach as CutOcclusion throughout the paper . Just as for shape bias , this method relies on28 information introduced not to interfere with a model ’ s learnt representations such that a decrease in29 performance can be directly attributed to lack of robustness.30 Data augmentation studies : In statistical learning , training with augmented data is termed Vicinal31 Risk Minimisation ( VRM ) [ 37 , 4 ] and it is seen as injecting prior knowledge about the neighbourhood32 of the data samples . The intuition behind augmentation caused researchers to interpret its effect33 through the similarity between original and augmented data distributions . This perspective is often34 challenged by methods which , despite generating samples that do not appear to fall under the35 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . distribution of natural images , lead to strong learners . Gontijo-Lopes et al . [ 12 ] argue it is the36 perceived distribution shift that needs to be minimised , while maximising the sample vicinity.37 Formalising these concepts , they introduce augmentation “ diversity ” and “ affinity ” . Diversity is38 defined as the training loss when learning with artificial samples , while affinity quantifies the39 difference between the accuracy on original test data and augmented test data for a reference model.40 The latter penalises augmentations that introduce artificial information to which the model is not41 invariant , implicitly assuming that training with that information is detrimental to generalisation.42 In summary , it is currently assumed that the artefacts introduced by changes in the data are negligible43 when evaluating models , while those introduced when training are important and undesirable . Does44 the artificial information added by analysis methods not have major side-effects or does it lead to45 biased results ? Conversely , are the artefacts important when training with modified data ? Do they46 cause models to learn better or worse representations ? 47 We set out to answer these questions and find that when the secondary effects of data manipulation48 are not accounted for , results can be misleading , especially in comparative studies . Subsequently , we49 construct empirical counter-examples which disprove common beliefs in the literature and highlight50 the importance of understanding the changes MSDAs introduce . Our contributions are:51 • We show that increasingly popular model interpretation and analysis methods are biased,52 relying on unfounded assumptions ( Section 2 ) ; 53 • For measuring occlusion robustness , we propose a fairer alternative ( Section 3 ) ; 54 • We show that , in contrast to what is widely assumed , not preserving the data distribution55 can lead to learning better representations ( Section 4 ) .56 2 Are artefacts negligible when analysing classifiers ? 57 To verify whether distorting data at evaluation time could have side-effects previously not considered,58 we look at the increase in misclassifications per category . That is , from the number of incorrect59 predictions of a model evaluated on modified data , we subtract the incorrect predictions when testing60 on original data . If there is a significant increase for a specific class , it indicates that the distortion61 introduces features the model associates with that class . We refer to this phenomenon as “ data62 interference ” . Considering only positive differences , we denote the increase in the percentage of63 misclassifications for class c of a given model m by imc . We define the Data Interference DI index as64 DI = icmax∑ c ic icmax , ( 1 ) where cmax is the class with highest mean increase across all runs . The DI index measures the65 percentage represented by the dominant class cmax weighted by its increase . A high index value66 indicates a sharp increase for a particular class which is consistent across runs . We associate this with67 an overlap between introduced artefacts and learnt representations . In Appendix B.1 we experiment68 To obtain models with different behaviours in a controlled manner , we make use of data augmentation.71 Since it is sufficient to identify some common cases in which models are disfavoured , we choose to72 reduce our environmental impact by restricting the analysis to simple MSDAs that combine images73 without incurring additional computation time or external models . As will be argued in Section 3 , we74 expect the unfairness to be present in most settings , thus the exact choice of augmentation is irrelevant.75 We focus on two popular MSDAs , MixUp [ 40 ] and CutMix [ 39 ] . MixUp linearly interpolates between76 two images to obtain a new training example , while CutMix masks out a rectangular region of an77 image with the corresponding region of another image . Besides the aforementioned methods , we78 also employ FMix due to its irregularly shaped masks sampled from Fourier space , which will play79 an important role in our analysis . Note that although the masking methods sample the size of the80 occluding patch from the same distribution , in CutMix part of the rectangle can be outside the image,81 which leads to less occluded samples overall . We refer to models by the augmentations they were82 trained with and use “ basic ” to label the models trained without MSDA.83 Throughout the paper , we do 5 runs of each experiment with PreAct-ResNet18 [ 15 ] as the default84 architecture . We include results for BagNet [ 2 ] and VGG [ 33 ] in the Appendices . The main data sets85 we report results on are CIFAR-10/100 [ 21 ] , Tiny ImageNet [ 34 ] , FashionMNIST [ 38 ] , ImageNet [ 29 ] .86 For ImageNet we use pretrained ResNet-101 models made publicly available by Harris et al . [ 14 ] .87 Note that the only experiments for which we are unable to run repeats are those on ImageNet , since88 only one model per augmentation is provided . For full experimental details , see Appendix A. Total89 emissions of training the models evaluated in this paper are estimated [ 22 ] to be 38.07 kgCO2eq , to90 which 13.11 kgCO2eq more are added during the analysis . The hope is that our findings will lead to a91 better understanding which in the long run would help reduce erroneous research directions without92 needing to empirically disprove them , lessening the future carbon footprint of the community.93 2.1 Shape bias94 For assessing shape bias through sample manipulation , the standard procedure is to choose between95 dividing the image in 4 , 16 or 64 patches to be shuffled . Since FashionMNIST images are smaller,96 we choose a 2× 2 grid , for CIFAR-10/100 and Tiny ImageNet 4× 4 , while for ImageNet we use97 an 8 × 8 grid . However , similar results are obtained for different grid sizes ( Appendix B.2 ) . The98 large DI index in Table 1 indicates that either basic or MixUp models tend to associate the features99 artificially introduced by patch-shuffling with a certain class . We take a closer look at the distribution100 of misclassifications for CIFAR-10 and notice that the basic model tends to wrongly predict the class101 “ Truck ” ( Figure 2 ) . This is not at all surprising , given that the strong horizontal and vertical edges are102 highly indicative of this class . Similar observations can be made for other data sets ( Appendix B.3 ) .103 Thus , we believe the grid-shuffling approach is causing models which are not invariant to strong104 horizontal and vertical edges to appear to rely more heavily on shape information . A model not105 affected by this transformation could be considered texture-biased if we accept the larger definition106 of texture as local information . However , there is a question about the extent to which the reciprocal107 is true ; A model can be invariant to the aforementioned edges because it is indeed relying on texture108 information or simply because it uses different shape-related features.109 Is a model necessarily more affected by patch-shuffling if it has a higher shape bias ? To answer110 this question , we can use another method of determining shape and texture bias to find a counter-111 Table 2 : Accuracy of augmentation-trained ImageNet and Tiny ImageNet models on the GST data set when the label is taken to be either the shape or texture . There is no clear correlation between masking methods and low texture bias . ImageNet Tiny ImageNet Shape Texture Shape Texture basic 20.31 53.28 10.56±0.65 26.04±1.77 MixUp 24.14 60.31 12.02±0.33 27.77±1.56 FMix 21.25 53.43 10.40±0.39 19.90±2.12 CutMix — — 10.54±0.38 23.72±2.42 Figure 2 : Difference in incorrect predictions on CIFAR-10 for the basic model . example . We analyse the ImageNet models on the Geirhos Style-Transfer ( GST ) [ 11 ] data set . GST112 contains artificially generated images where the shape belongs to one class and the texture to another.113 There are 16 coarse classes that encompass a number of ImageNet categories to which they are114 mapped . The bias of the models is given by the accuracy obtained when the label is set to either the115 shape or texture . Using this well-known method of identifying shape bias we want to find models116 which have similar biases but different DI indices when patch-shuffling . This would indicate that117 sensitivity to shuffling is not necessarily linked to increased shape bias.118 The results in Table 2 show that the basic model does not have a higher shape bias than masking119 methods although it has a significantly higher DI index , as we have seen in Table 1 . We repeat the120 same experiment on the Tiny ImageNet [ 34 ] data set . Geirhos et al . [ 11 ] use WordNet [ 26 ] to map the121 1000 categories to the 16 classes of the GST data set . A number of ImageNet categories that belong122 to the 16 higher-level classes of GST are missing . For this reason , a poorer overall performance is123 expected and the results could differ slightly given a better fit between the sets . Nonetheless , we find124 again no significant correlation between masking augmentation and texture bias . We also include in125 Table B.2 the results for BagNet models , which have smaller receptive fields and so are forced to use126 more local information . Even in this case , we find a high DI for the basic model and no difference127 in texture bias compared to MSDA . Thus , a model which is more affected by patch-shuffling is not128 necessary more shaped-bias . In other words , models can appear to have vastly different shape bias129 when evaluated on randomly rearranged patches , albeit in reality their bias is similar . The inability of130 this method to account for artefacts makes it unfair and unreliable.131 2.2 Occlusion measurement132 We want to determine whether the same issue identified in the case of shape bias evaluation applies133 to occlusion robustness measures . We focus on CutOcclusion , where a rectangular black patch is134 superimposed on test images and the robustness is given by the resulting accuracy . We perform135 the same experiment as before , where the DI index is now measured when testing on rectangle-136 occluded images . There is no standardised distortion when measuring CutOcclusion , with the size137 and positioning of the obstructing patch varying between studies . Most often in prior art a lack of138 robustness is noted for large occluders [ e.g . 5 , 42 ] . For this reason , we sample the size of the patch139 from a Beta ( 2,1 ) distribution , allowing the occluding patch to lie outside the image ( as it is done for140 augmenting with CutMix and CutOut [ 7 ] ) . This allows us to capture both the cases in which either141 the centre or the border area is masked out but requires a non-uniform distribution to counter for the142 patches existing outside the image . We also sample from a uniform distribution where the occluder is143 restricted to be positioned within the image boundaries and obtain similar results ( See Appendix C.1 ) .144 Table 3 shows that a significant gap in the DI index can be identified for each of the data sets . This145 indicates that some models will again be disadvantaged . We additionally find that data interference is146 present for different architectures , when overlapping patches from external images or using differently147 shaped masks ( Appendix B.4 ) . Thus , the result of CutOcclusion and its variants is highly dependent148 on the problem at hand . That is , whether the artefacts introduced by the artificial occlusion are149 salient features of the model depends on what features are naturally distinctive for the model . Just as150 for randomly shuffling tiles , by occluding images using a particularly shaped patch , one implicitly151 measures a model ’ s affinity to certain features , albeit those features might be discriminative . This152 deems such methods inappropriate for fairly assessing robustness and texture bias.153 A related observation was made by Hooker et al . [ 17 ] who note the pitfalls of manipulating data to154 determine feature importance . They point out that when simply superimposing uniform patches over155 image features , it is difficult to asses how much of the reduction in accuracy is caused by the absence156 of those features and how much is due to images becoming out of distribution . To address this , the157 most important features identified by an estimator are masked out both on train and test data , closing158 the gap between the two sets . Hooker et al . then train and evaluate models on the newly generated159 images . Unlike for interpretability methods , the subject of occlusion robustness studies is the model160 itself , which makes training with a modified version of the data an inviable option . In the following161 section we explore ways of overcoming this bias when measuring occlusion robustness.162 3 What are fairer alternatives ? 163 We propose a simple , more carefully defined measure that aims to decouple the machine ’ s edge bias164 from the occlusion robustness , which we refer to as “ interplay occlusion ” ( iOcclusion ) . Interplay165 occlusion reflects the change in the interplay between performance on seen and unseen data . Formally,166 iOcclusioni = ∣∣∣∣A ( Ditrain ) −A ( Ditest ) A ( Dtrain ) −A ( Dtest ) ∣∣∣∣ , ( 2 ) where A ( D ) denotes the accuracy on a given data set D , and Di is the data set resulting from167 removing i % pixels of each image . The intuition is that on train data robust models are less sensitive168 to the artefacts of the occlusion policy for small levels of occlusion , resulting in a large difference in169 accuracy from that on unseen data . The performance of both train and test gets close to random as170 the percentage of occluded data approaches 90 % and we expect the gap to fall off quicker for less171 robust models . This change in interplay is taken with respect to the generalisation gap of the model,172 such that the quality of the model fit in itself does not interfere with the robustness measure.173 Although iOcclusion reduces data interference , other factors have to also be considered when choosing174 a masking method for computing Di , such as the number of contiguous components or the amount of175 salient information masked out . In this paper we choose to generate masks using Grad-CAM [ 30 ] ,176 such that the area with most salient i % pixels is covered . It must be noted that this method implicitly177 assumes there could be multiple occluders and has the downside of incurring a higher environmental178 cost . For this , we also experiment with using rectangular or Fourier-sampled masks and conclude179 that although random masking makes the process noisier , the exact choice of masking method is of180 secondary importance as long as the occluder ’ s granularity is accounted for . Appendix C.4 provides181 discussion and results on these alternative instances of iOcclusion , as well as differences in their182 carbon footprint . For a fair comparison , throughout this section we do not allow the obstructing patch183 when measuring CutOcclusion to lie outside the image such that the fraction removed is exact.184 Assessing the correctness of such a measure is difficult in the absence of a baseline . For the remainder185 of this section we will build varied experiments to attest the validity of our method . We focus on the186 key results , but include additional ones and further experiments in Appendix C. Since occlusion in187 real-life scenarios could be caused by non-uniformly coloured objects , an appropriate measure must188 generalise across colour patterns . When computing iOcclusion and CutOcclusion , we superimpose189 patches from images belonging to a different data set and compare the results to those obtained190 when occluding with black patches only . For visual clarity , Figure 3 presents the results for the191 least and most robust models ( see Appendix C.2 for a full comparison ) . For iOcclusion , using192 uniform occluders gives similar results to its non-uniform version , whereas the CutOcclusion measure193 provides an inconsistent model evaluation.194 As we have argued , in addition to not being sensitive to the colour pattern of the patch , a fair measure195 must also be invariant to the shape of the patch . To empirically confirm iOcclusion reduces the196 importance of edge information , we aim to obtain a model that is robust to occlusion , but at the same197 time has a high DI index ( it is sensitive to edge information ) . To this end , we create a variation of198 FMix , Random Masks ( RM ) , where at the beginning of the training process three masks are randomly199 sampled from Fourier space . For each batch , one of the three is chosen uniformly at random . While200 the RM models are not sensitive to black-patch occlusion , when masking with patterned patches they201 have a higher DI index than FMix , as desired . Table 4 gives results for a fraction of 0.3 pixels covered202 by a non-uniform occluder . Our measure reflects the robustness of training with RM , situating it close203 to other masking methods . On the other hand , because CutOcclusion implicitly penalises models204 with high DI index , according to this measure RM appears almost as sensitive to occlusion as MixUp.205 Figure B.4 shows results for a wider range of fractions.206 Another problem that occurs when purely looking at post-masking accuracy is weaker models would207 erroneously appear less robust . We show this by reversing the problem : we evaluate the same model208 on two different subsets of the CIFAR-100 data set : typical and tail images as categorised by Feldman209 and Zhang [ 10 ] . They consider a train-test sample pair to belong to the tail of the data distribution if210 the test sample is correctly classified when a model is trained with the train sample and incorrectly211 without it . CutOcclusion would indicate that models are significantly more robust to occluding typical212 examples . However , a closer analysis makes us doubt this conclusion . The raw accuracy on both213 train and test data for tail examples is lower than for the typical ones . In fact , the performance when214 occluding images decreases at the same rate for the two subsets . By way of definition , iOcclusion215 allows a fair comparison of robustness regardless of the overall performance of a model ( Figure 4 ) .216 As we evidenced through controlled experiments , there are many cases that CutOcclusion does not217 properly address . From a model analysis perspective , correctly assessing the occlusion robustness218 could lead to better understanding and development of models and training procedures . Equally219 important , it has applicability for real-world deployments where no prior knowledge exists about the220 Table 5 : Augmentation comparison on CIFAR-10 . We consider two variants when calculating diversity . One is computing the cross-entropy loss using the label of the majority class ( Diversity ) , as for mixing in [ 19 ] . The alternative , MixDiversity , takes a linear combination of the two cross-entropy losses . Affinity Diversity MixDiversity MixUp −12.58±0.14 0.41±0.01 0.84±0.00 FMix −25.55±0.26 0.34±0.01 0.65±0.00 Table 6 : DI index ( % ) measured for nonuniform occlusion when training without the class with highest increase in incorrect predictions . Again , a gap can be noted , supporting the idea that data interference is not specific to peculiar cases . CIFAR-10 CIFAR-100 MixUp 0.39±0.15 1.22±0.19 FMix 0.08±0.06 0.50±0.21 possible shapes of the obstructions . While this aspect of generality is the strength of our approach,221 it must be stressed that when there exists a limited set of known possible occluders , evaluating222 robustness specifically to them could be safer . Incorrectly assessing robustness can have severe effects223 especially when applied to autonomous vehicles or medical imaging . We do not propose a universal224 solution , but rather suggest an alternative to the biased approach for the common scenario in which225 the environment is not controlled and little is known about all the potential occluders . However , even226 in this case , our metric should be taken as a guide when analysing models . Although iOcclusion aims227 to address data interference , since a ground-truth does not exist , it can not be guaranteed that this228 method provides fair results in the absolute.229 The strength of the bias will depend on the data in question and some applications will be more heavily230 affected than others . We have seen that for natural images this bias does exist . To confirm that we have231 not just identified isolated cases , we remove the class that has the highest increase in mispredictions232 and retrain the models on the remaining classes . We find that the bias is again present ( Table 6 ) ,233 but with respect to another class . For example , in the case of CIFAR-10 , after removing the “ Truck ” 234 class , models mispredict rectangle-occluded images as “ Boat ” ( Appendix D ) . Thus , the edge artefacts235 are very likely to interfere with learnt representations since they are such fundamental features . From236 an evaluation perspective , as we have seen , this impacts assessment methods and must be accounted237 for . From the training perspective , such a widespread data interference of masking distortions would238 indicate a large perceptual shift in the data when performing MSDA . In the following section we239 investigate the importance of the artefacts in this case and their implications.240 4 Is the magnitude of the distribution shift important ? 241 Traditionally it was believed that a good augmentation should have minimal distribution shift . Most242 recently , it has been argued that it is the degree of the perceived shift that determines augmentation243 quality [ 12 ] . We start with the perceptual gap of training with MSDA , as proposed in Gontijo-Lopes244 et al . [ 12 ] . Reiterating , this is given by the difference between the performance of the baseline model245 when presented with original test data and augmented test data and is termed “ affinity ” . Subsequently,246 we address the gap in the wider sense , as is often sought in prior art . We first argue that high affinity247 and high diversity are not necessarily desirable . Indeed , on CIFAR-10 , we find FMix , a better248 performing augmentation , to have both lower affinity and lower diversity than MixUp ( Table 5 ) . For249 diversity , we compute the cross-entropy loss where the label is taken to be that of the majority class.250 Similar results are obtained with the MixUp loss , where a weighted average of the true labels is taken.251 While intuitively for a high level of affinity , high diversity could correspond to better methods,252 the converse does not hold . We argue this is because affinity is rather an analysis of the learnt253 representations of the reference model and can not give an insight into the quality of the augmentation254 or its effect on learning . The limitations of affinity are intimately linked to those of CutOcclusion.255 We have seen in Section 3 that the bias of the basic model is present not only when obstructing an256 image with a uniform patch , but also when mask-mixing . As such , an augmentation will have a257 lower affinity if it introduces artefacts that could otherwise lead to learning better representations258 when used in the training process . We believe this issue extends to other approaches that aim to259 motivate the success of MSDA through reduced distribution shift . Henceforth , we focus on bringing260 further supporting evidence that the importance lies in the invariance introduced by the shift and its261 interaction with the given problem rather than its magnitude.262 4.1 If it is not the magnitude that matters , is it the direction ? 263 We use empirical evidence to argue against previous assumptions behind the success of MSDA and264 propose the study of introduced bias as a more informative research direction . Here we use the term265 “ bias ” to refer to a drift in the learnt representations introduced by the change in the training procedure.266 A fundamental difference to classical training is that in the case of augmentation the samples are no267 longer independent . Mixed-sampling takes this even further . An immediate question is , does the268 added correlation lead to more meaningful representations ? It is claimed that the strength of MixUp269 lies in causing the model to behave linearly between two images [ 40 ] or in pushing the examples270 towards their mean [ 3 ] . Both of these claims rely on the combined images to be generated from the271 same distribution . We want to verify to which extent this is necessary for a successful augmentation.272 It has been argued that label mixing has a negligible effect on the final model performance [ 19 , 18,273 14 , 24 ] . We use the reformulated objective setting [ 18 , 14 ] , where targets are not mixed and the274 mixing coefficient is drawn from an imbalanced Beta distribution . This allows us to apply MSDA275 between data sets . Thus , for training a model on a data set , we use an additional one whose targets276 will be ignored . As an example , a model that is learning to predict CIFAR-10 images will be trained277 on a combination of CIFAR-10 and CIFAR-100 images , with the target of the former . This scenario278 breaks the added correlation between training examples . Note that when mixing between data sets279 we use the same procedure as when performing regular MSDA , without improving the process.280 Table 7 contains the results of this experiment , showing that an accuracy similar to or better than that281 of regular MSDA can be obtained by performing inter-dataset MSDA.This invalidates the argument282 that the power of MixUp resides in causing the model to act linearly between samples . Another283 observation is that for FMix and MixUp , introducing elements from CIFAR-100 when training284 models on the CIFAR-10 problem does not harm the learning process . The reciprocal , however , does285 not hold . Hence , the “ distribution shift ” is more intimately linked to the problem at hand and aiming286 to characterise an augmentation based on the distance from the original distribution is a limiting287 approach , especially when the distance is measured as perceived by a reference model.288 We believe an explanation is that the artefacts created when putting together images from CIFAR-10289 with those of CIFAR-100 could introduce information that makes the separation of the 10 classes290 easier . However , if the information happens to interfere with a feature that is important for separating291 the CIFAR-100 categories , the performance could degrade on this data set . This singular experiment292 is not sufficient to draw any general conclusions . However , it does show that shifting two distributions293 by the same amount can have different effects on the model performance . Thus , the specifics of the294 bias introduced could be more important than its magnitude . While some level of data similarity has295 to be preserved when performing MSDA , it is far from being the objective of such data-distorting296 approaches which , as we will argue further , should be rather seen as forms of regularisation.297 We have seen that for all considered data sets , artefacts introduced by masking methods seem to298 overlap with common features . This has led us to believe that MSDA training could help bypass299 some of the simplicity bias . The simplicity bias refers to the tendency of deep models to find simple300 representations and has been used to justify the success of deep models [ 28 , 36 ] . Recent research301 shows that this propensity causes models to ignore complex features that explain the data well in302 favour of elementary features , even when they lead to worse performance [ 31 , 16 ] .303 Although it could seem natural that since MSDAs are not augmentations in the VRM sense , they will304 increase the complexity of the problem , we design an experiment to support this claim . Similarly305 to Shah et al . [ 31 ] , we combine CIFAR-10 and MNIST [ 23 ] samples . Since they have the same306 number of classes , we can easily associate each class of one data set with a corresponding one from307 the other . Thus , we stack a padded image from the kth class of MNIST on top of a sample from the308 kth class of CIFAR-10 , such that a 3× 64× 32 image is obtained . We then randomly combine the309 test images and separately compute the accuracy with respect to the targets of each data set.310 The predictions with respect to the CIFAR-10 labels are no better than random ( 10.04±0.11 ) , while311 the accuracy with respect to the MNIST images remains high ( 99.57±0.72 ) . Thus , models trained312 on this combination are mostly relying on MNIST images to make predictions . Similar behaviours313 have previously been associated with simplicity bias . Subsequently , when training , we perform FMix314 only on MNIST images and observe that this is enough to reverse the results . Evaluating against the315 CIFAR-10 label gives an accuracy of 86.60±0.34 , while testing against the MNIST label only gives316 11.61±0.30 . We find that this also holds true for the other MSDAs . Thus , performing these distortions317 on the simpler data set increases its complexity to the point where it surpasses that of CIFAR-10.318 Previously , we presented evidence that masking MSDA does not necessarily promote learning neither319 more shape nor texture information . In the light of this fact along with the results from this section,320 we believe image distortions force the model to learn more complex both shape and texture-specific321 features . Thus , in this paper we pointed out that the shift in learnt representations can lead to better322 models and simply quantifying the distribution shift can be misleading . An open question remains:323 How can we better capture the bias that is introduced and its quality ? We believe understanding how324 a relatively small change in the data distribution impacts learnt representations could lead the way to325 characterising the relationship between data and model generalisation.326 5 Conclusions327 Distorting data is such a commonplace procedure , yet little effort has been devoted to investigating its328 broader effects . This is particularly problematic when image modifications are applied in analyses . We329 show a number of cases in which this leads to biased results . For occlusion robustness measurement,330 we propose an alternative . The insights we gain from this endeavour point towards the study of data331 characteristics as a cornerstone of our understanding and raise a number of questions about mixed332 sample data augmentation , on which we subsequently focus . We note that they interfere with features333 that are consistently found across a number of data sets and conclude that the methods commonly334 used are forms of mixed sample regularisation rather than augmentation . A limitation of previous335 studies that aim to explain their success is the focus on trying to argue similarity with original data,336 rather than explaining the bias introduced by the distortion . Correctly interpreting it is important337 not only for making the models trustable but also for injecting more informed prior knowledge in338 future applications . Beyond their practical benefits , we believe MSDAs have the potential to help339 characterise the interplay between data and learnt representations . Overall , the purpose of our paper340 is to encourage better practice when dealing with all forms of data distortions.341 References342 [ 1 ] Rocío Alaiz-Rodríguez and Nathalie Japkowicz . Assessing the impact of changing environments343 on classifier performance . In Conference of the Canadian Society for Computational Studies of344 Intelligence , pages 13–24 . Springer , 2008.345 [ 2 ] Wieland Brendel and Matthias Bethge . Approximating CNNs with bag-of-local-features models346 works surprisingly well on imagenet . In International Conference on Learning Representations,347 2019 . URL https : //openreview.net/forum ? id=SkfMWhAqYQ.348 [ 3 ] Luigi Carratino , Moustapha Cissé , Rodolphe Jenatton , and Jean-Philippe Vert . On mixup349 regularization . arXiv preprint arXiv:2006.06049 , 2020.350 [ 4 ] Olivier Chapelle , Jason Weston , Léon Bottou , and Vladimir Vapnik . Vicinal risk minimization.351 In Advances in neural information processing systems , pages 416–422 , 2001.352 [ 5 ] Sanghyuk Chun , Seong Joon Oh , Sangdoo Yun , Dongyoon Han , Junsuk Choe , and Youngjoon353 Yoo . An empirical evaluation on robustness and uncertainty of regularization methods . arXiv354 preprint arXiv:2003.03879 , 2020.355 [ 6 ] David A Cieslak and Nitesh V Chawla . A framework for monitoring classifiers ’ performance:356 when and why failure occurs ? Knowledge and Information Systems , 18 ( 1 ) :83–108 , 2009.357 [ 7 ] Terrance DeVries and Graham W Taylor . Improved regularization of convolutional neural358 networks with cutout . arXiv preprint arXiv:1708.04552 , 2017.359 [ 8 ] Logan Engstrom , Brandon Tran , Dimitris Tsipras , Ludwig Schmidt , and Aleksander Madry.360 Exploring the landscape of spatial robustness . In International Conference on Machine Learning,361 pages 1802–1811 . PMLR , 2019.362 [ 9 ] Alhussein Fawzi and Pascal Frossard . Measuring the effect of nuisance variables on classifiers.363 In Edwin R. Hancock Richard C. Wilson and William A. P. Smith , editors , Proceedings of the364 British Machine Vision Conference ( BMVC ) , pages 137.1–137.12 . BMVA Press , September365 2016 . ISBN 1-901725-59-6. doi : 10.5244/C.30.137 . URL https : //dx.doi.org/10.5244/366 C.30.137.367 [ 10 ] Vitaly Feldman and Chiyuan Zhang . What neural networks memorize and why : Discovering368 the long tail via influence estimation . Advances in Neural Information Processing Systems , 33,369 2020.370 [ 11 ] Robert Geirhos , Patricia Rubisch , Claudio Michaelis , Matthias Bethge , Felix A. Wichmann,371 and Wieland Brendel . Imagenet-trained CNNs are biased towards texture ; increasing shape bias372 improves accuracy and robustness . In International Conference on Learning Representations,373 2019 . URL https : //openreview.net/forum ? id=Bygh9j09KX.374 [ 12 ] Raphael Gontijo-Lopes , Sylvia J Smullin , Ekin D Cubuk , and Ethan Dyer . Affinity and diversity:375 Quantifying mechanisms of data augmentation . arXiv preprint arXiv:2002.08973 , 2020.376 [ 13 ] Ian J Goodfellow , Jonathon Shlens , and Christian Szegedy . Explaining and harnessing adversar-377 ial examples . arXiv preprint arXiv:1412.6572 , 2014.378 [ 14 ] Ethan Harris , Antonia Marcu , Matthew Painter , Mahesan Niranjan , Adam Prügel-Bennett , and379 Jonathon Hare . Understanding and enhancing mixed sample data augmentation . arXiv preprint380 arXiv:2002.12047 , 2020.381 [ 15 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Identity mappings in deep residual382 networks . In European conference on computer vision , pages 630–645 . Springer , 2016.383 [ 16 ] Katherine Hermann and Andrew Lampinen . What shapes feature representations ? exploring384 datasets , architectures , and training . In H. Larochelle , M. Ranzato , R. Hadsell , M. F. Balcan,385 and H. Lin , editors , Advances in Neural Information Processing Systems , volume 33 , pages386 9995–10006 . Curran Associates , Inc. , 2020 . URL https : //proceedings.neurips.cc/387 paper/2020/file/71e9c6620d381d60196ebe694840aaaa-Paper.pdf.388 [ 17 ] Sara Hooker , Dumitru Erhan , Pieter-Jan Kindermans , and Been Kim . A benchmark for inter-389 pretability methods in deep neural networks . In Advances in Neural Information Processing390 Systems , pages 9737–9748 , 2019.391 [ 18 ] Ferenc Huszár . mixup : Data-dependent data augmentation , 2017 . URL http : //www.392 inference.vc/mixup-data-dependent-data-augmentation/.393 [ 19 ] Hiroshi Inoue . Data augmentation by pairing samples for images classification . arXiv preprint394 arXiv:1801.02929 , 2018.395 [ 20 ] Narine Kokhlikyan , Vivek Miglani , Miguel Martin , Edward Wang , Bilal Alsallakh , Jonathan396 Reynolds , Alexander Melnikov , Natalia Kliushkina , Carlos Araya , Siqi Yan , et al . Captum : A397 unified and generic model interpretability library for pytorch . arXiv preprint arXiv:2009.07896,398 2020.399 [ 21 ] Alex Krizhevsky et al . Learning multiple layers of features from tiny images . 2009.400 [ 22 ] Alexandre Lacoste , Alexandra Luccioni , Victor Schmidt , and Thomas Dandres . Quantifying401 the carbon emissions of machine learning . arXiv preprint arXiv:1910.09700 , 2019.402 [ 23 ] Yann LeCun and Corinna Cortes . MNIST handwritten digit database . 2010 . URL http:403 //yann.lecun.com/exdb/mnist/.404 [ 24 ] Daojun Liang , Feng Yang , Tian Zhang , and Peter Yang . Understanding mixup training methods.405 IEEE Access , 6:58774–58783 , 2018.406 [ 25 ] Tiange Luo , Tianle Cai , Mengxiao Zhang , Siyu Chen , Di He , and Liwei Wang . Defective407 convolutional layers learn robust CNNs . arXiv preprint arXiv:1911.08432 , 2019.408 [ 26 ] George A Miller . Wordnet : a lexical database for english . Communications of the ACM , 38 ( 11 ) :409 39–41 , 1995.410 [ 27 ] Chaithanya Kumar Mummadi , Ranjitha Subramaniam , Robin Hutmacher , Julien Vitay , Volker411 Fischer , and Jan Hendrik Metzen . Does enhanced shape bias improve neural network robustness412 to common corruptions ? In International Conference on Learning Representations , 2021 . URL413 https : //openreview.net/forum ? id=yUxUNaj2Sl.414 [ 28 ] Preetum Nakkiran , Gal Kaplun , Dimitris Kalimeris , Tristan Yang , Benjamin L Edelman , Fred415 Zhang , and Boaz Barak . SGD on neural networks learns functions of increasing complexity.416 arXiv preprint arXiv:1905.11604 , 2019.417 [ 29 ] Olga Russakovsky , Jia Deng , Hao Su , Jonathan Krause , Sanjeev Satheesh , Sean Ma , Zhiheng418 Huang , Andrej Karpathy , Aditya Khosla , Michael Bernstein , Alexander C. Berg , and Li Fei-Fei.419 ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision420 ( IJCV ) , 115 ( 3 ) :211–252 , 2015. doi : 10.1007/s11263-015-0816-y.421 [ 30 ] Ramprasaath R Selvaraju , Michael Cogswell , Abhishek Das , Ramakrishna Vedantam , Devi422 Parikh , and Dhruv Batra . Grad-cam : Visual explanations from deep networks via gradient-based423 localization . In Proceedings of the IEEE international conference on computer vision , pages424 618–626 , 2017.425 [ 31 ] Harshay Shah , Kaustav Tamuly , Aditi Raghunathan , Prateek Jain , and Praneeth Netrapalli . The426 pitfalls of simplicity bias in neural networks . In H. Larochelle , M. Ranzato , R. Hadsell , M. F.427 Balcan , and H. Lin , editors , Advances in Neural Information Processing Systems , volume 33,428 pages 9573–9585 . Curran Associates , Inc. , 2020 . URL https : //proceedings.neurips.429 cc/paper/2020/file/6cfe0e6127fa25df2a0ef2ae1067d915-Paper.pdf.430 [ 32 ] Baifeng Shi , Dinghuai Zhang , Qi Dai , Zhanxing Zhu , Yadong Mu , and Jingdong Wang . Infor-431 mative dropout for robust representation learning : A shape-bias perspective . In International432 Conference on Machine Learning , pages 8828–8839 . PMLR , 2020.433 [ 33 ] Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale434 image recognition . In International Conference on Learning Representations , 2015.435 [ 34 ] Stanford . Tiny imagenet visual recognition challenge , 2015 . URL https : //tiny-imagenet.436 herokuapp.com/.437 [ 35 ] Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Goodfel-438 low , and Rob Fergus . Intriguing properties of neural networks . arXiv preprint arXiv:1312.6199,439 2013.440 [ 36 ] Guillermo Valle-Perez , Chico Q. Camargo , and Ard A. Louis . Deep learning generalizes because441 the parameter-function map is biased towards simple functions . In International Conference on442 Learning Representations , 2019 . URL https : //openreview.net/forum ? id=rye4g3AqFm.443 [ 37 ] Vladimir Vapnik . The nature of statistical learning theory . Springer science & business media,444 1999.445 [ 38 ] Han Xiao , Kashif Rasul , and Roland Vollgraf . Fashion-MNIST : a novel image dataset for446 benchmarking machine learning algorithms . arXiv preprint arXiv:1708.07747 , 2017.447 [ 39 ] Sangdoo Yun , Dongyoon Han , Seong Joon Oh , Sanghyuk Chun , Junsuk Choe , and Youngjoon448 Yoo . Cutmix : Regularization strategy to train strong classifiers with localizable features . In449 Proceedings of the IEEE International Conference on Computer Vision , pages 6023–6032,450 2019.451 [ 40 ] Hongyi Zhang , Moustapha Cisse , Yann N. Dauphin , and David Lopez-Paz . mixup : Beyond452 empirical risk minimization . In International Conference on Learning Representations , 2018.453 URL https : //openreview.net/forum ? id=r1Ddp1-Rb.454 [ 41 ] Tianyuan Zhang and Zhanxing Zhu . Interpreting adversarially trained convolutional neural455 networks . In International Conference on Machine Learning , pages 7502–7511 . PMLR , 2019.456 [ 42 ] Zhun Zhong , Liang Zheng , Guoliang Kang , Shaozi Li , and Yi Yang . Random erasing data457 augmentation . In Proceedings of the AAAI Conference on Artificial Intelligence , number 34 ( 07 ) ,458 pages 13001–13008 , 2020.459 Checklist460 1 . For all authors ... 461 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s462 contributions and scope ? [ Yes ] 463 ( b ) Did you describe the limitations of your work ? [ Yes ] See lines 174–184 , 221–229,464 292–294.465 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] See lines466 221–229 as well as 89–93 and Appendix C.4.467 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to468 them ? [ Yes ] 469 2 . If you are including theoretical results ... 470 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 471 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 472 3 . If you ran experiments ... 473 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-474 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] 475 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they476 were chosen ) ? [ Yes ] See Appendix A.477 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running exper-478 iments multiple times ) ? [ Yes ] The only exception is when using publicly available479 models , as mentioned in Section 2.480 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type481 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See Appendix A.482 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 483 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See lines 84–93 and484 Appendix A.485 ( b ) Did you mention the license of the assets ? [ N/A ] 486 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 487 488 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re489 using/curating ? [ N/A ] 490 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable491 information or offensive content ? [ N/A ] 492 5 . If you used crowdsourcing or conducted research with human subjects ... 493 ( a ) Did you include the full text of instructions given to participants and screenshots , if494 applicable ? [ N/A ] 495 ( b ) Did you describe any potential participant risks , with links to Institutional Review496 Board ( IRB ) approvals , if applicable ? [ N/A ] 497 ( c ) Did you include the estimated hourly wage paid to participants and the total amount498 spent on participant compensation ? [ N/A ] 499

Summary: This paper studies empirically the effect of artefacts and biases in model analysis and training methods relying on data manipulation. They argue that common metrics used for analysing shape bias and occlusion robustness also (undesirably) include the effect of artefacts (such as edge sharpness) introduced by the corresponding image manipulations, and propose a new metric, iOcclusion, which is more robust to these effects.  Further, it is argued that these artefacts also manifest themselves in training schemes based on mixed sample data augmentation (MSDA), and can be beneficial by introducing a systematic bias towards complexity.