On the Effects of Data Distortion on Model Analysis and Training
Authors: Antonia Marcu
Adam Prugel-Bennett
Venue: NeurIPS 2021 Submitted (2021)
Keywords: occlusion
shape bias
evaluation fairness
model interpretation
augmentation
robustness
model bias
training bias
distribution shift

Abstract: Data modification can introduce artificial information. It is often assumed that the resulting artefacts are detrimental to training, whilst being negligible when analysing models. We investigate these assumptions and conclude that in some cases they are unfounded and lead to incorrect results. Specifically, we show current shape bias identification methods and occlusion robustness measures are biased and propose a fairer alternative for the latter. Subsequently, through a series of experiments we seek to correct and strengthen the community's perception of how distorting data affects learning. Based on our empirical results we argue that the impact of the artefacts must be understood and exploited rather than eliminated.

Modifying data has become commonplace both when training and analysing models, yet the wider On the Effects of Data Distortion on Model Analysis and Training
Anonymous Author(s)
Affiliation
Address email
Abstract
Data modification can introduce artificial information. It is often assumed that the resulting artefacts are detrimental to training, whilst being negligible when amalysing models. We investigate these assumptions and conclude that in some cases they are unfounded and lead to incorrect results. Specifically, we show current shape bias identification methods and occlusion robustness measures biased and propose a fairer alternative for the latter. Subsequently, through a series of experiments seek correct and strengthen the community's perception of how distorting data affects learning. Based on our empirical results we argue that the impact of the artefacts must be understood and exploited rather than eliminated.
2
4
6
Motivation
10
12 implications are often disregarded. We delve into some of the side-effects and point out that this practice has resulted in the creation of biased model interpretation tools and poorly informed theories.
14 On the analysis side, we take as examples occlusion robustness and shape bias identification methods.
15 On the training side, we focus on some instances of Mixed Sample Data Augmentation (MSDA),
18 where two images are combined to obtain new training sample. Visual examples of each can be found in Figure T In this paper we study number of assumptions that lie at the heart of the aforementioned methods, which we briefly introduce below.
17
18
Shape-texture bias: Deep models are known be sensitive interventions that are imperceptible 20 to humans [35 13), as well as to other forms of distribution shifts [[] 6 g) It has been argued that
21 this is intimately linked to networks tending use texture rather than shape information 22] [T]. Recently, input distortions have become a popular way of assessing a model's texture bias. To this end,
22
23 images divided into a grid and the resulting patches are randomly shuffled such that information
24 is preserved locally, while the global shape is altered (32 27 25 41). It is implicitly assumed that patch-shuffling does not introduce misleading shape or texture that could affect model evaluation.
25
26
Occlusion robustness: A widely adopted method for measuring occlusion robustness is through the image [5] 9 39 42 20) We refer
accuracy obtained after superimposing rectangular patch to this approach as CutOcclusion throughout the paper. Just as for shape bias, this method relies on information introduced not interfere with model's 's learnt representations such that decrease in performance can be directly attributed to lack of robustness.
28
30
31 Data augmentation studies: In statistical learning, training with augmented data is termed Vicinal
32 Risk Minimisation (VRM) (37 and it seen as injecting prior knowledge about the neighbourhood of the data samples. The intuition behind augmentation caused researchers interpret its effect
34 through the similarity between original and augmented data distributions. This perspective is often challenged by methods which, despite generating samples that do not appear to fall under the
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

BAREI SHOP
BARE SHOP
SHOP
Image 1
Image 2
Occlusion
Shuffled grid
MixUp
CutMix
FMix
Figure I: Examples image distortions. For test-time distortions (Occlusion and Shuffled grid) only Image I was used. For mixing augmentations, the first row was generated with mixing factor of 0.2, while the second one with 0.5.
36
distribution of natural images, lead to strong learners. Gontijo-Lopes et al. 12 argue it is the perceived distribution shift that needs to be minimised, while maximising the sample vicinity. Formalising these concepts, they introduce augmentation "diversity" and "affinity". Diversity is defined as the training loss when learning with artificial samples, while affinity quantifies the difference between the accuracy on original test data and augmented test data for a reference model. The latter penalíses augmentations that introduce artificial information which the model is not invariant, implicitly assuming that training with that information is detrimental to generalisation.
38
40
41
42
43 In summary, it is currently assumed that the artefacts introduced by changes in the data are negligible when evaluating models, while those introduced when training are important and undesirable. Does the artificial information added by analysis methods not have major sidc-effects does it lead to biased results? Conversely, are the artefacts important when training with modified data? Do they cause models to learn better worse representations?
44
46
48 We set out to answer these questions and find that when the secondary effects of data manipulation not accounted for, results can be misleading, especially in comparative studies. Subsequently, we construct empirical counter-examples which disprove common beliefs in the literature and highlight the importance of understanding the changes MSDAs introduce. Our contributions are:
50
51
52
We show that increasingly popular model interpretation and analysis methods are biased, relying unfounded assumptions (Section
53
54
For measuring occlusion robustness, we propose a fairer alternative (Section B):
55
We show that, in contrast to what is widely assumed, not preserving the data distribution can lead to learning better representations (Section
56
2
Are artefacts negligible when analysing classifiers?
57
To verify whether distorting data at evaluation time could have side-effects previously not considered, 59 we look at the increase misclassifications per category. That is, from the number of incorrect predictions of model evaluated on modified data, subtract the incorrect predictions when testing 61 on original data. If there is a significant increase for a specific class, it indicates that the distortion introduces features the model associates with that class. We refer this phenomenon as "data 63 interference". Considering only positive differences, we denote the increase in the percentage of misclassifications for class of given model by i
We define the Data Interference DI index as
iomaz Ee
DI
icmax
85 where Cmar is the class with highest mean increase across all runs. The DI index measures the percentage represented by the dominant class Cmez weighted by its increase. A high index value 67 indicates a sharp increase for particular class which is consistent across runs. We associate this with an overlap between introduced artefacts and learnt representations. In Appendix B. we experiment

---

Table 1: DI index (%) for PreAct-ResNet18 on grid-shuffled images for four different types of models. Results with the highest average are given in italic and the lowest in bold. Information introduced when shuffling tends to interfere less with the representations of FMix and CutMix models.
basic
MixUp
FMix
CutMix
CIFAR-10
2.90+1.10
2.54=1.29
0.60 2.22
0.33
CIFAR- 100
1 1.0510.59
0.93
0.11
PashionMNIST
1..63
2. 2.7311.64
1..21
0.70+0.22
Tiny ImageNet 2. 24.814.70
1..4
0.14
ImageNet
0.82
1.49
0.58
69 with an alternative index, where we weight by the highest increase a model across the runs, so as to obtain worst-case analysis. As expected, we observe a more accentuated bias in this case.
71 To obtain models with different behaviours in a controlled manner, we make use of data augmentation. Since it is sufficient to identify some common cases in which models are disfavoured, we choose to reduce our environmental impact by restricting the analysis to simple MSDAs that combine images without incurring additional computation time or external models. As will be argued in Section 3 we expect the unfairness to be present most settings, thus the exact choice of augmentation irrelevant. We focus on two popular MSDAs, MixUp 440 and CutMix (39]. MixUp linearly interpolates between two images to obtain a new training example, while CutMix masks out a rectangular region of an image with the corresponding region of another image. Besides the aforementioned methods, we also employ FMix due to its irregularly shaped masks sampled from Fourier space, which will play an important role in our analysis. Note that although the masking methods sample the size of the occluding patch from the same distribution, in CutMix part of the rectangle can be outside the image, which leads to less occluded samples overall. We refer to models by the augmentations they were trained with and use "basic" to label the models trained without MSDA.
72
73
74
75
76
77
78
79
80
81
89
84
Throughout the paper, we do 5 runs of each experiment with PreAct-ResNetll 15 as the default architecture. We include results for BagNet (2 and VGG (33 in the Appendices. The main data sets 85
we report results on are CIFAR- 10/100 221] Tiny ImageNet (34], FashionMNIST (38), ImageNet 22]
86
87
For ImageNet we use pretrained ResNet- 101 models made publicly available by Harris et al. [1]
88
Note that the only experiments for which we are unable to run repeats are those on ImageNet, since only one model per augmentation is provided. For full experimental details, see Appendix AA] Total emissions of training the models evaluated in this paper are estimated [22) to be 38.07 kgCOgeq. to which 13.11 kgCOzeq more are added during the analysis. The hope is that our findings will lead to a better understanding which in the long run would help reduce erroneous research directions without needing to empirically disprove them, lessening the future carbon footprint of the community.
89
91
93
2.1
Shape bias
For assessing shape bias through sample manipulation, the standard procedure is to choose between dividing the image in 4, 16 or 64 patches to be shuffled. Since MNSS images are smaller, 97 we choose a 2 x 2 grid, for CIFAR-10/101 and Tiny ImageNet 4 x 4, while for ImageNet we use an 8 x 8 grid. However, similar results are obtained for different grid sizes (Appendix [B.2). The large DI index in Table indicates that either basic or MixUp models tend to associate the features artificially introduced by patch-shuffling with certain class. We take a closer look at the distribution of misclassifications for CIFAR- 10 and notice that the basic model tends to wrongly predict the class "Truck" (Figure 2). This is not at all surprising, given that the strong horizontal and vertical edges are highly indicative of this class. Similar observations can be made for other data sets (Appendix B.3)) Thus, we believe the grid-shuffling approach is causing models which are not invariant to strong horizontal and vertical edges to appear to rely more heavily shape information. A model not affected by this transformation could be considered texture- if we accept the larger definition of texture as local information. However, there is question about the extent to which the reciprocal is true; A model can be invariant to the aforremenionedd edges because it is indeed relying on texture information or simply because uses different shape-related features.
95
96
98
99
100
102
104
106
108
109
110 Is model necessarily more affected by patch-shuffling if it has higher shape bias? To answer this question, we can use another method of determining shape and texture bias to find counter-
3

---

0.5
Table 2: Accuracy of ougmentation-trainee ImageNet
difference between incorrect predictions
and Tiny ImageNet models on the GST data set when the label is taken be either the shape or texture. There is no clear correlation between masking methods and low texture bias.
01
ImageNet
Tiny ImageNet
Shape
Texture
Shape
Texture
airllane
automobile
Cat
diear
dog
fog
horse
ship
track
basic 20.31
53.28
10.56+0.65
1..04
MixUp
24.14
60.31
12.02+0.33
27.77+1.56
FMix 21.25
53.43
10.40
19.90
Figure 2: Difference in incorrect predic-
CutMix
10.54
23.72+2.42
tions on CIFAR- 10 for the basic model.
112
example. We analyse the ImageNet models on the Geirhos Style-Transfen (GST) 11 data set. GST contains artificially generated images where the shape belongs to one class and the texture to another. There are 16 coarse classes that encompass number of ImageNet categories to which they are mapped. The bias of the models is given by the accuracy obtained when the label is set to either the shape or texture. Using this well-known method of identifying shape bias we want to find models which have similar biases but different DI indices when patch-shuffling. This would indicate that sensitivity to shuffling is not necessarily linked to increased shape bias.
113
115
117
118
119
The results in Table 2 show that the basic model does not have a higher shape bias than masking methods although it has a significantly higher DI index, as we have seen in Table T We repeat the
120
121
same experiment on the Tiny ImageNet (34] data set. Geirhos et al. [T use WordNet [26] to map the
122
1000 categories to the 16 classes of the GST data set. A number of ImageNet categories that belong
123
to the 16 higher-level classes of GST are missing. For this reason, a poorer overall performance is expected and the results could differ slightly given a better fit between the sets. Nonetheless, we find again no significant correlation between masking augmentation and texture bias. We also include in Table B.2 the results for BagNet models, which have smaller receptive fields and so are forced to use more local information. Even in this case, we find a high DI for the basic model and no difference in texture bias compared to MSDA. Thus, model which is more affected by patch-shuffling is not necessary more shaped-bias. In other words, models can appear to have vastly different shape bias when evaluated randomly rearranged patches, albeit in reality their bias is similar. The inability of this method to account for artefacts makes it unfair and unreliable.
125
127
129
130
131
132
2.2
Occlusion measurement
133
We want to determine whether the same issue identified in the case of shape bias evaluation applies to occlusion robustness measures. We focus on CutOcclusion, where a rectangular black patch is superimposed on test images and the robustness is given by the resulting accuracy. We perform the same experiment as before, where the DI index now measured when testing on rectangle- occluded images. There is no standardised distortion when measuring CutOcclusion, with the size and positioning of the obstructing patch varying between studies. Most often in prior art a lack of robustness is noted for large occluders [e.g. 42). For this reason, we sample the size of the patch from a 1 distribution, allowing the occluding patch to lie outside the image (as it is done for augmenting with CutMix and CutOut (7). This allows us to capture both the cases in which either the centre or the border area is masked out but requires a non-uniform distribution to counter for the patches existing outside the image. We also sample from uniform distribution where the occluder is restricted to be positioned within the image boundaries and obtain similar results (See Appendix C.I].
134
135
136
137
138
140
142
144
145
3 shows that a significant gap in the DI index can be identified for each of the data sets. This indicates that some models will again be disadvantaged. We additionally find that data interference is present for different architectures, when overlapping patches from external images or using differently shaped masks (Appendix B..4). Thus, the result of CutOcclusion and its variants is highly dependent on the problem at hand. 'hht is, whether the artefacts introduced by the artificial occlusion are salient features of the model depends on what features are naturally distinctive for the model. Just as for randomly shuffling tiles, by occluding images using particularly shaped patch, one implicitly
Table
146
147
148
150

---

Table DI index (%) when occluding with black patches. The highest results are given in italic and the lowest in bold. For each data set, there exists non noglngggig gap in the DI index.
basic
MixUp
FMix
CutMix
CIFAR-10
5. 5.48+1.91
0. 0.75+0.55
0.87
2.87
CIFAR-I00 100
3. 1.25+1.31
0.80=0.46 16
1.28
1.36 6.66
FashionMNIST
0.4410.34
2.5911.05
0.92
1.82
Tiny Imagenet 1.28 4.50
2.4010.82
1.881
0.47 10.41
4.6214.93
1.02
152
measures a model's affinity to certain features, albeit those features might be discriminative. This deems such methods inappropriate for fairly assessing robustness and texture bias.
154
related observation was made by Hooker et al. [7 who note the pitfalls of manipulating data to determine feature importance. They point out that when simply superimposing uniform patches over image features, is difficult to asses how much of the reduction in accuracy caused by the absence of those features and how much is due to images becoming out of distribution. To address this, the most important features identified by an estimator are masked out both on train and test data, closing the gap between the two sets. Hooker et al. then train and evaluate models on the newly generated images. Unlike for interpretability methods, the subject of occlusion robustness studies is the model itself, which makes training with modified version of the data an inviable option. In the following section we explore ways of overcoming this bias when measuring occlusion robustness.
155
156
157
158
160
162
163
3
What are fairer alternatives?
164
We propose a simple, more carefully defined measure that aims decouple the machine's 's edge bias from the occlusion robustness, which we refer to as "interplay occlusion" (iOcclusion). Interplay occlusion reflects the change in the interplay between performance on seen and unseen data. Formally,
185
186
Occlusion; = A(Dirainn A(Di test)
A(Dirain) - A(Drnst)
167
where A(D) denotes the accuracy on given data set DD and Dí is the data set resulting from removing i% pixels of each image. The intuition is that on train data robust models are less sensitive to the artefacts of the occlusion policy for small levels of occlusion, resulting in large difference in accuracy from that on unseen data. The performance of both train and test gets close to random as the percentage of occluded data approaches 90% and we expect the gap to fall off quicker for less robust models. This change in interplay is taken with respect to the generalisation gap of the model, such that the quality of the model fit in itself does not interfere with the robustness measure.
168
169
170
172
174 Although iOcclusion reduces data interference, other factors have to also be considered when choosing a masking method for computing D', such as the number of contiguous components or the amount of salient information masked out. In this paper we choose to generate masks using Grad-CAM 30) such that the area with most salient 1% pixels is covered. It must be noted that this method implicitly assumes there could be multiple occluders and has the downside of incurring a higher environmental
175
176
177
178
cost. For this, we also experiment with using rectangular or Fourier-sampled masks and conclude that although random masking makes the process noisier, the exact choice of masking method is of secondary importance as long as the occluder's ' granularity is accounted for. Appendix discussion and results on these alternative instances of iOcclusion, as well as differences in their carbon footprint. For fair comparison, throughout this section we do not allow the obstructing patch when measuring CutOcclusion to lie outside the image such that the fraction removed is exact.
180
182
184
185
Assessing the correctness of such a measure is difficult in the absence of baseline. For the remainder of this section we will build varied experiments to attest the validity of our method. We focus on the key results, but include additional ones and further experiments in Appendix [ Since occlusion in real-life scenarios could caused by non-unifornly coloured objects, an appropriate measure must generalise across colour patterns. When computing iOcclusion and CutOcclusion, we superimpose patches from images belonging to different data set and compare the results to those obtained when occluding with black patches only. For visual clarity, Figure 3 presents the results for the
196
187
188
190
5

---

2,00
unifomm
FMix uniform
PMix nonuniform
1.75
FMix nonuniormm
80
bask uniform
basic uniform
basic nonuniform
150
basic nonunifomm
CunDcctusion
(acclusion
125
1.00
0.75
40
0.50
20
0.25
0.00
0.3
0.4
0.5
0.6
0.7
0.1
fraction removed
fraction removed
Figure 3: CutOcclusion (left) and iOcelusion (right) when occluding with black patches (uniform)
and patches taken from other images (nonuniform). iOcclusion gives more consistent results.
Table 4: DI index (%) and occlusion robustness for models trained on CIFAR-10 when obstructing 30% of the image pixels with non-uniform patches. When measuring the robustness with CutOc- clusion, RM appears significantly less robust than CutMix due to its sensitivity to patching with rectangles, while iOcclusion highlights the robustness specific to training with FMix-like masks. Given in bold is the closest result to that of RM for each evaluation.
basic
MixUp
CutMix
FMix
RM
DI index
1.67+0.27
1.00
0.17
0.15
0.39
CutOcclusion
47.97
58.65+1.01
76.56+6.36
78.00
60.79+5.08
iOcclusion
0.21+0.10
0.57 10.18
1.09
1.46
1.20
192 least and most robust models (see Appendix C.2 for a full comparison). For iOcclusion, using uniform occluders gives similar results to its non-uniform version, whereas the CutOcclusion measure provides an inconsistent model evaluation.
193
195 As we have argued, in addition not being sensitive to the colour pattern of the patch, a fair measure must also be invariant to the shape of the patch. To empirically confirm iOeclusion reduces the importance of edge information, we aim to obtain a model that is robust to occlusion, but at the same time has a high DI index (it is sensitive to edge information). To this end, we create a variation of Fixx Random Masks (RM), where at the beginning of the training process three masks are randomly sampled from Fourier space. For each batch, one of the three is chosen uniformly at random. While the RM models are not sensitive to black-pator occlusion, when masking with patterned patches they have higher DI index than FMix, as desired. Table gives results for fraction of 0.3 pixels covered by a non-uniform occluder. Our measure reflects the robustness of training with RM, situating it close to other masking methods. On the other hand, because CutOcclusion implicitly penalises models with high DI index, according to this measure R.M appears almost as sensitive to occlusion as Mix Figure B.4shows results for wider range of fractions.
196
197
198
199
201
203
205
206
207
Another problem that occurs when purely looking at post-masking accuracy is weaker models would erroneously appear less robust. We show this by reversing the problem: we evaluate the same model on two different subsets of the CIFAR- 100 data set: typical and tail images as categorised by Feldman and Zhang [10]. They consider train-test sample pair to belong to the tail of the data distribution if the test sample is correctly classified when a model is trained with the train sample and incorrectly without it. CutOcclusion would indicate that models are significantly more robust to occluding typical examples. However, a closer analysis makes us doubt this conclusion. The raw accuracy on both train and test data for tail examples is lower than for the typical ones. In f'ct, the performance when occluding images decreases at the same rate for the two subsets. By way of definition, iOcclusion allows fair comparison of robustness regardless of the overall performance of model (Figure 4
209
211
213
215
216
217 As we evidenced through controlled experiments, there are many cases that CutOcclusion does not properly address. From a model analysis perspective, correctly assessing the occlusion robustness could lead to better understanding and development of models and training procedures. Equally important, it has applicability for real-world deployments where prior knowledge exists about the
219

---

typical
EMix typical
FMix taill
FMix tail
basic typical
basic typical
basic
CunDccluiion
60
10cduuion
0.6
40
20
0.2
0.0
0.4 0.5
0.6
0.7
0.1
0.4
0.5
0.6
fraction removed
removed
Figure 4: CutOcelusion (left) and iOcclusion (right) for the basic and FMix models on two subsets of the same data set: tail and typical. Evaluating the models with iOcclusion on the two types of samples leads to mostly overlapping robustness levels. That is, they do not differ outside the margin of error. On the contrary, CutOcclusion incorrectly finds the models to be less robust on tail data.
Table 5: Augmentation comparison on CIFAR-I0. We consider two variants when calculating diversity. One is computing the cross-entropy loss using the label of the majority class (Diversity), as for mixing in [1)]. The alternative, MixDiversity, takes a linear combination of the two cross-entropy losses.
Table 6: DI index (%) measured for non- uniform occlusion when training without the class with highest increase in incor- rect predictions. Again, a gap can be noted, supporting the idea that data inter- ference is not specific to peculiar cases.
Affinity
Diversity
MixDiversity
CIFAR-10
CIFAR-100
MixUp
-12.5840.1 0.41
0. 0.8400.00
MixUp
0.39
1.22+0.19
FMix -25.5510.28
0.04
0.65
FMix
0.08 10.06
0.50
221
possible shapes of the obstructions. While this aspect of generality is the strength of our approach, must be stressed that when there exists limited set of known possible occluders, evaluating robustness specifically to them could be safer. Incorrectly assessing robustness can have severe effects especially when applied autonomous vehicles medical imaging. We do not propose universal solution, but rather suggest an alternative to the biased approach for the common scenario in which the environment is not controlled and little is known about all the potential oceluders. However, even in this case, our metric should be taken as guide when analysing models. Although iOcclusion aims to address data interference, since a ground truth does not exist, it cannot be guaranteed that this method provides fair results in the absolute.
222
223
224
225
226
227
228
229
The strength of the bias will depend on the data in question and some applications will be more heavily affected than others. We have seen that for natural images this bias does exist. To confirm that we have not just identified isolated cases, we remove the class that has the highest increase in mispredictions and retrain the models on the remaining classes. We find that the bias is again present (Table (). but with respect to another class. For example, in the case of CIFAR- 10, after removing the "Truck" class, models mispredict rectangle-occluded images as "Boat" (Appendix D). Thus, the edge artefacts 236 are very likely interfere with learnt representations since they are such fundamental features. From an evaluation perspective, as we have seen, this impacts assessment methods and must be accounted for. From the training perspective, such a widespread data interference of masking distortions would indicate large perceptual shift in the data when performing MSDA. In the following section we investigate the importance of the artefacts in this case and their implications.
231
232
233
235
237
238
239
240
Is the magnitude of the distribution shift important?
241
242
Traditionally it was believed that a good augmentation should have minimal distribution shift. Most 243 recently, it has been argued that it is the degree of the perceived shift that determines augmentation quality [1] We start with the perceptual gap of training with MSDA, as proposed in Gontijo-Loper 245 et al. 12.. Reiterating, this given by the difference between the performance of the baseline model when presented with original test data and augmented test data and is termed "affinity". Subsequently,
7

---

247
we address the gap in the wider sense, as is often sought in prior art. We first argue that high affinity and high diversity are not necessarily desirable. Indeed, on CIFAR-10, we find FMix, a better performing augmentation, to have both lower affinity and lower diversity than MixUp (Table For diversity, we compute the cross-entropy loss where the label is taken to be that of the majority class. Similar results are obtained with the MixUp loss, where weighted average of the true labels is taken.
248
250
251
252
While intuitively for a high level of affinity, high diversity could correspond to better methods, the converse does not hold. We argue this is because affinity is rather an analysis of the learnt representations of the reference model and cannot give an insight into the quality of the augmentation or its effect on learning. The limitations of affinity are intimately linked to those of CutOcclusion. We have seen in Section 3 that the bias of the basic model is present not only when obstructing an As such, an augmentation will have a lower affinity if it introduces artefacts that could otherwise lead to learning better representations
253
254
255
256
257
image with a uniform patch, but also when mask-mixing
258
259
when used in the training process. We believe this issue extends to other approaches that aim to motivate the success of MSDA through reduced distribution shift. Henceforth, we focus on bringing
260
further supporting evidence that the importance lies in the invariance introduced by the shift and its interaction with the given problem rather than its magnitude.
262
263
4.1
If it is not the magnitude that matters, is it the direction?
We use empirical evidence to argue against previous assumptions behind the success of MSDA and propose the study of introduced bias as a more informative research direction. Here we use the term 285
bias" to refer to drift in the learnt representations introduced by the change in the training procedure. 267 A fundamental difference to classical training is that in the case of augmentation the samples are no longer independent. Mixed-sampling takes this even further. An immediate question is, does the added correlation lead to more meaningful representations? It is claimed that the strength of Mix lies in causing the model behave linearly between two images [40] or in pushing the examples towards their mean 33. Both of these claims rely on the combined images to be generated from the same distribution. We want verify to which extent this is necessary for successful augmentation.
268
269
270
271
272
273
has been argued that label mixing has a negligible effect on the final model performance [19 18 14 24]. We the reformulated objective setting [18 [4]. where targets not mixed and the mixing coefficient is drawn from an imbalanced Beta distribution. This allows us to apply MSDA
275
276
an additional one whose targets will be ignored. As an example, a model that is learning to predict CIFAR-10 10 images will be trained on combination of CIFAR-10 and CIFAR- 100 images, with the target of the former. This scenario breaks the added correlation between training examples. Note that when mixing between data sets
between data sets. Thus, for training model data set,
277
279
280
we use the same procedure as when performing regular MSDA, without improving the process.
281
Table contains the results of this experiment, showing that an accuracy similar to or better than that of regular MSDA can be obtained by performing inter-dataset MSDA.This invalidates the argument that the power of MixUp resides in causing the model to act linearly between samples. Another observation is that for FMix and MixUp, introducing elements from CIFAR- when training models on the CIFAR- 10 problem does not harm the learning process. The reciprocal, however, does not hold. Hence, the "distribution shift" is more intimately linked to the problem at hand and aiming to characterise an augmentation based on the distance from the original distribution is a limiting approach, especially when the distance is measured as perceived by a reference model.
282
283
285
286
287
288
289
We believe an explanation is that the artefacts created when putting together images from CIFAR-10 with those of CIFAR- 100 could introduce information that makes the separation of the 10 classes casier. However, if the information happens to interfere with a feature that is important for separating the 100 categories, the performance could degrade on this data set. This singular experiment
291
292
293 is not sufficient to draw any general conclusions. However, it does show that shifting two distributions by the same amount can have different effects on the model performance. Thus, the specifics of the
295 bias introduced could be more important than its magnitude. While some level of data similarity has to be preserved when performing MSDA, it is far from being the objective of such data-distorting approaches which, as we will argue further, should be rather seen as forms of regularisation.
297
We have seen that for all considered data sets, artefacts introduced by masking methods seem to overlap with common features. This has led us to believe that MSDA training could help bypass some of the simplicity bias. The simplicity bias refers to the tendency of deep models to find simple
299
8

---

Table 7: Accuracy on CIFAR-I0 (left) and CIFAR-100 (right) upon mixing with samples from a different data set. The baseline is the accuracy when training with a single data set using the reformulated objective. In the interest of space, CIFAR- 110 is used to refer to mixing with CIFAR- 100 when training on the CIFAR-10 problem and vere.
MixUp
FMix
CutMix
MixUp
FMix
CutMix
baseline
94.18 10.34
94.36
94.67 10.20
74.68
75.75 10.31
74.19
CIFAR-110
94.70 10.27
94.80
94.66
12.3611..4
10.55
74.47 100.9
Fashion
92.28 10.28
95.0310.10
94.61 10.19
66.4011.86
74.46
74.06
301
representations and has been used to justify the success of deep models [28 36 Recent research
302
shows that this propensity causes models to ignore complex features that explain the data well in favour of elementary features, even when they lead to worse performance [31 16]
304
Although it could seem natural that since MSDAs are not augmentations in the VRM sense, they will increase the complexity of the problem, we design an experiment to support this claim. Similarly 31 we combine CIFAR-10 10 and MNIST [3 samples. Since they have the same to Shah et al.
305
306
307
number of classes, we can easily associate each class of one data set with a corresponding one from the other. Thus, we stack a padded image from the keth class of MNIST on top of a sample from the ktt class of CIFAR-10, such that a 3 x 64 x 32 image is obtained. We then randomly combine the test images and separately compute the accuracy with respect to the targets of each data set.
308
309
310
311
The predictions with respect to the CIFAR-I0 labels are no better than random (10.04+0.11) 11). while the accuracy with respect to the MNIST images remains high (99.5720.72). Thus, models trained on this combination are mostly relying on MNIST images to make predictions. Similar behaviours have previously been associated with simplicity bias. Subsequently, when training, we perform FMix
312
313
314
315
only on MNIST images and observe that this is enough to reverse the results. Evaluating against the
316
CIFAR-10 label gives an accuracy of 86 20.31 while testing against the MNIST label only gives 11.61+0.30 30- We find that this also holds true for the other MSDAs. Thus, performing these distortions on the simpler data set increases its complexity to the point where it surpasses that of CIFAR-10.
317
318
319
Previously, we presented evidence that masking MSDA does not necessarily promote learning neither more shape texture information. In the light of this fact along with the results from this section, we believe image distortions force the model to learn more complex both shape and texture-specific features. Thus, in this paper we pointed out that the shift in learnt representations can lead to better
320
321
322
323
models and simply quantifying the distribution shift can be misleading. An open question remains:
324
How can we better capture the bias that introduced and its quality? We believe understanding how a relatively small change in the data distribution impacts learnt representations could lead the way to characterising the relationship between data and model generalisation.
325
328
327
5
Conclusions
328
Distorting data is such a commonplace procedure, yet little effort has been devoted investigating its broader effects. This is particularly problematic when image modifications are applied in analyses. We show number of cases in which this leads biased results. For occlusion robustness measurement, we propose an alternative. The insights we gain from this endeavour point towards the study of data characteristics as a cornerstone of our understanding and raise a number of questions about mixed sample data augmentation, on which we subsequently focus. We note that they interfere with features that are consistently found across a number of data sets and conclude that the methods commonly used are forms of mixed sample regularisation rather than augmentation. A limitation of previous studies that aim to explain their success the focus on trying to argue similarity with original data, rather than explaining the bias introduced by the distortion. Correctly interpreting it is important
329
330
331
332
333
334
335
336
337
338
not only for making the models trustable but also for injecting more informed prior knowledge in future applications. Beyond their practical benefits, we believe MSDAs have the potential to help
340
characterise the interplay between data and learnt representations. Overall, the purpose of our paper
is to encourage better practice when dealing with all forms of data distortions.
9

---

References
342
343
[1] Rocío Alaiz-Rodríguez and Nathalie Japkowicz. Assessing the impact of changing environments on classifier performance. In Conference of the Canadian Society for Computational Studies of Intelligence, pages 13-24. Springer, 2008.
345
[2 Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-locall-eature models works surprisingly well on imagenet. In International Conference on Learning Representations. 2019. http::tt::///// htttps://openrreiieew.... wttp://wwweeeetoorr//ddttttaaaae
347
319
[3] Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regularization. arXiv preprint arXiv:2006.06600 2020.
350
[4] Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In Advances in neural information processing systems, pages 412, 2001.
352
[5] Sanghyuk Chun, Scong Joon Oh, Sangdoo Yun, Dongyoon Han, Junsuk Choe, and Youngjoon Yoo. An empirical evaluation on robustness and uncertainty of regularization methods. arXiv preprint MXXiv:2003.08877 2020.
354
356
[6] David A Cieslak and Nitesh V Chawla. framework for monitoring classifiers' performance: when and why failure occurs? Knowledge and Information Systems, 18(1):83-108, 2009.
357
358
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout, arXiv preprint DexXiv:/708.000 2017.
359
[8] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry, Exploring the landscape of spatial robustness. In International Conference on Machine Learning, pages 1802-1811. 1. PMLR, 2019.
381
363
[9] Alhussein Fawzi and Pascal Frossard. Measuring the effect of nuisance variables on classifiers.
In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, P Proceedings of the British Machine Vision Conference (BMVC), pages 137.1- 12.7.12.. BMVA Press, September
365
366
2016. ISBN 1-901725-59-6. doi: 10.5244/C.30.1377 URL http:::t///// //xx. doi org/ 10. 5244/ 1.7
367
388
[10]
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33, 2020.
389
370
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations,
372
373
374
2019. URL/pttps//oppnrrriie http:p///ppppueeeeeeeeee////////
[12] Raphael Gontijo-Lopes, Sylvia J Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.089/1 089777 2020.
376
377
[13] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv: 1xiiv:4422.66 14266572, 2014.
378
379
[14] Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, Adam Prügel--ennett, and Jonathon Hare. Understanding and enhancing mixed sample data augmentation. arXiv preprint MxXiv:2002.12047. 2020.
380
381
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
383
384
[16] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9005- Curran Associates, Inc., 2020. URL ptcttpp:///pooceeings.eeeeee ppep://000iii///eeec0000000000e0
385
387
ec/
pdf
10

---

389
[17] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for inter- pretability methods in deep neural networks. In Advances in Neural Information Processing Systems, pages 9737-9748, 2019.
390
382
[18] Ferenc Huszár. mixup: Data-dependent data augmentation, 2017.
URL http://wwx www
393
inference. wwt.w////iiiieee. siiip--dta-dppppeee aeaa-eendent-datt---ggeeeeeee
[19] Hiroshi Inoue. Data augmentation by pairing samples for images classification. arXiv preprint
395
arXiv: 022929, 2018.
396
[20]
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896 2020.
397
398
399
400
[21]
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[22]
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 1rX1v01910.09700, 2019.
402
403
[23]
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:
404
/ //yann.1ecuu com/exdb/mnist/
405
[24] Daujun Liang, Feng Yang. Tian Zhang, and Peter Yang. Understanding mixup training methods. IEEE Access, 6:58774-58783, 2018.
407
[25] Tiange Luo, Tianle Cai, Mengxiao Zhang, Siyu Chen, Di He, and Liwei Wang. Defective
408
convolutional layers learn robust CNNs. arXiv preprint arXiv:1911.08432 2019.
[26] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11); -9-41,11 1995.
410
[27]
Chaithanya Kumar Mummadi, Ranjitha Subramaniam, Robin Hutmacher, Julien Vitay, Volker Fischer, and Jan Hendrik Metzen. Does enhanced shape bias improve neural network robustness to common corruptions? In International Conference on Learning Representations, 2021. URL
412
414
415
[28) Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. SGD neural networks learns functions of increasing complexity. arXiv preprint arXiv: 1905. 1604, 2019.
416
417
[29]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 115(3))211-222, 2015. doi: 1007/sl 1263-015--8---------
419
421
[30] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
423
424
425
426
[31] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9573-9585 Curran Associates, Inc., 2020. URL /ttph/tt/// neurips cc/paper 222200/0000000077777700002222220 Paper...app.... pdf
427
428
429
430
[32] Baifeng Shi, Dinghuai Zhang. Qi Dai, Zhanxing Zhu, Yadong Mu, and Jingdong Wang. Infor- mative dropout for robust representation learning: A shape-bias perspective. In International Conference Machine Learning, pages 8828 8839. PMLR, 2020.
432
438
434
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
11

---

436
[34]
Stanford. Tiny imagenet visual recognition challenge, 2015. URL
437
herokuapp.com com/
438
[35]
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprini arXiv: 131.:191,.6199, 2013.
439
440
[36]
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the uurameter-function map is biased towards simple functions. In International Conference on Learning Representations, 2019. URL URL htt.tpps///ppnnriieee.eeee/////s idrr:rrmmrrr-
442
444
[37] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
445
[38] Han Xiao, Kashif Rasul, and Roland Vollgraf. Pashion-MNIST: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1XXiv:170.00710 2017.
447
448
[39]
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference Computer Vision, pages 6023-6032, 2019.
449
451
452
[40] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
453
454
URL https:://pperrrviier.. wwt.:eee/tfoooeee meet//oorumiiidrrd0p111111
[41] Tianyuan Zhang and Zhanxing Zhu. Interpreting adversarially trained convolutional neural networks. In International Conference on Machine Learning, pages 7502-7511. PMLR, 2019.
456
457
[42] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, number 34(07), pages 13001-13008, 2020.
458
459
Checklist
480
461
1. For all authors...
462
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
463
464
(b) Did you describe the limitations of your work? [Yes] See lines 174-184, 221 221-229, 292-294.
465
466
(c) Did you discuss any potential negative societal impacts of your work? [Yes) See lines 221-229 as well as 89-93 and Appendix C.4
468
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
470
2. If you are including theoretical results...
471
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
472
(b) Did you include complete proofs of all theoretical results? [N/A]
473
3. If you ran experiments..
474
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
475
476
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] The only exception is when using publicly available models, as mentioned in Section
479
480
481
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A
12

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
484
(a) If your work uses existing assets, did you cite the creators? [Yes] See lines 84-93 and Appendix A
485
486
(b) Did you mention the license of the assets? [N/A]
487
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
489
(d) Did you discuss whether and how consent was obtained from people whose data you'te ro using/curating? [N/A]
480
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
492
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
495
486
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
498
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "eay4oeifuzi", "reviewer": "Reviewer_RRFF", "summary": "This paper studies impacts of data augmentation. Authors propose new measures for evaluating data interference and occlusion robustness. Through analyzing changes in accuracy induced by data distortion, authors argue that current practice about using and treating data augmentation has led to biased model analysis and interpretation methods. ", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors deal with an important area of research that discovers hidden biases in data augmentation. However, unfortunately, this paper is hard to follow. Vague words uncommon in the community are given without context or explanation. For instance, in L 43-44, what does mean by “artefacts introduced by changes in the data are negligible when evaluating models”? Similarly, what are “the artefacts introduced when training”?\n\nAlso, key measures proposed in the paper require more explanations and motivations. For instance, for data interference (Eq. 1), why is the percentage represented by the dominant class important? Similarly, why is this percentage required to be weighted by the maximum increase? In addition, it is unclear why the authors consider only increases in the error rates when we are evaluating the impacts of “data interference.” Similar issues exist for the interplay occlusion (Eq. 2).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed limitations and societal impact in the paper. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "IKvx7OGS85M", "reviewer": "Reviewer_iusD", "summary": "In this paper, the authors examine the effects of distorting data and its effects on  biased results. They emphasize on image modifications, and they propose an alternative measure of occlusion robustness (called the interplay occlusion based on the interplay between seen and unseen data.) The authors also focus on questions about mixed sample data augmentation. They mention the need for arguing about bias from distortion and its role in trusting models and injecting informed knowledge as well.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It would have been helpful to discuss more applications in real world that would suffer from such distortion bias. Also, in the end the authors ask for better practice when dealing with all forms of data, so it could help discuss the side-effects of  this kind of bias when we have different kinds of datasets compared to the image datasets the authors explore. Some of the main experiments are mentioned (and referred to a lot) in the Appendix. So although the idea is new and the goal is clear, the writing could still improve. The significance would be more pronounced if the applications/real-world ruinous after-effects are mentioned in a separate paragraph or section.\n\n-It is mentioned that occlusion in real-life scenarios could be caused by non-uniformly colored objects, so a suitable measure should generalize to all color patterns. It would be helpful to elaborate how this was actually done.\n\n-It is mentioned that mostly simple features (similar to the common features) are impacted. But if the information happens to interfere with a feature that is important for separating, the performance could degrade. It could be useful to add the reasons as to why the experiment is not sufficient for general conclusions about this point and how one might try examine this issue.\n\n-Is there a way to see what proportion of the features are the ones in the simplicity bias? That is, what proportion of them are similar to the common features.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention the limitations of their proposed approach and new measure (interplay occlusion). But a separate paragraph on this topic could improve readability.\nHowever, the societal impact still needs more details. As the real world applications and how the side effects of such bias would be  harmful need more detailed explanations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "80wM8aXcVwf", "reviewer": "Reviewer_kAg1", "summary": "This paper delves into the assumption that data modification is detrimental to training, while negligible when analysing models. It aims to encourage better practice when dealing with data distortions rather than elimination. With experiments and analyses, the authors show that current shape biased identification methods and occlusion robustness measures are biased. The authors then propose a fairer alternative to measure occlusion robustness. A series of experiments are conducted to disprove common assumptions and put forward the argument that not preserving the data distribution can lead to learning better representations. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The authors question the common assumption that data modality is negligible when evaluating models and important when training, and construct counter-examples against the common belief. The idea is novel and challenges the common assumption. \n\n2. The paper does not follow the common structure of ‘Introduction - Related Work - Methodology - Experiments - Conclusion’. It is organized by posing and answering questions, which would strongly disprove common beliefs and put forward the authors’ arguments. \n\n3. The clarity of this paper is good in general. However, the paper lacks an easy-to-read summary for each experiment. Maybe a brief and clear answer to the question should be stated ahead of the experiment and analysis part. \n\n4. The paper aims to encourage better practice when dealing with data distortions rather than elimination. It proposes a fairer alternative named ‘interplay occlusion’ to measure occlusion robustness. It points out the limitation of previous studies and argues that correctly interpreting the effect of data distortion is of critical significance not only to make the models more trustable but also to inject more informed prior knowledge in future applications. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. As the authors mentioned in the paper, there still remains a question: \tHow can we better capture the bias that is introduced and its quality. \n\n2. It is meaningful that this work corrects and strengthen the community’s perception of how distorting data affects learning. It argues that the impacts should be understood and exploited, however, the future applications of this work are not fully investigated.\n\n3. Some conclusion is drawn with observation rather than strict proof, which is not convincing enough, e.g. shape-biased model and texture-biased model. \n\n4. The paper lacks an easy-to-read summary for each experiment. It is organized by posing and answering questions creatively, but it lacks a brief summary of each answer. When reading the paper, I could only understand the answers to the questions after I have finished read the whole texts. Maybe a brief and clear answer to the question should be stated ahead of the experiment and analysis part. \n\n5. There are some minor typos in the paper, e.g. ‘asses’ in line 156 should be ‘assess’\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "51agICEJVE_", "reviewer": "Reviewer_VZ6B", "summary": "This paper studies empirically the effect of artefacts and biases in model analysis and training methods relying on data manipulation. They argue that common metrics used for analysing shape bias and occlusion robustness also (undesirably) include the effect of artefacts (such as edge sharpness) introduced by the corresponding image manipulations, and propose a new metric, iOcclusion, which is more robust to these effects.  Further, it is argued that these artefacts also manifest themselves in training schemes based on mixed sample data augmentation (MSDA), and can be beneficial by introducing a systematic bias towards complexity.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper makes a number of interesting and novel insights regarding current practices in data modification for image classification. These are well supported for the most part by (clear and well-designed) experimental evaluations, though the lack of formal definitions and analysis means that some claims remain somewhat speculative. There are some issues with the clarity of the narrative, though I believe this is non-critical and can be fixed.\n\nThe paper firstly introduces the DI metric and show that models can have similar shape biases (according to GST dataset), but perform differently under DI w/ shuffling. It is then argued that this shows measures based on patch-shuffling are unreliable. However, since DI was introduced by the authors, I think a better representative for “effect of patch-shuffling” would be accuracy after shuffling, and how this compares to GST. At least, it could be discussed how the class-focused DI compares to accuracy. **AR**: I am satisfied with the author response and it is good to see that there is still a discrepancy between accuracy after shuffling and GST.\n\nThe authors then make a similar argument for occlusion robustness. However, it is not clear to me what \"occlusion robustness\" actually means, since there is no formal definition, nor clear empirical gold standard like GST dataset for shape bias. In particular, since the model is treated as a black-box for all occlusion robustness metrics (including iOcclusion), how can one distinguish whether the model’s representation is “not occlusion robust” or “using edge bias/artefacts”, if this leads to the same input-output behaviour, i.e. misclassification? **AR**: Thank you for the explanation. If I understand correctly, the distinction is between whether the performance only drops for some types of occlusion such as a box (indicating sensitivity to artefacts), or all types of occlusion (indicating lack of robustness/inability to classify without the occluded information). Firstly, I would suggest the authors define this formally as the target/proxy to the intuitive notion of \"how much information can be hidden from the network\". Secondly, I accept the authors' point that iOcclusion has been shown to be less sensitive to extraneous factors such as edge artefacts. However, invariance is only a necessary, and not sufficient condition to show that it captures occlusion robustness. For instance, a function which randomly assigns a value to each network would also be invariant. While Figure 3 does indeed show that FMix is given a greater iOcclusion robustness than the basic network, more analysis is needed to definitively prove this link/correlation between occlusion robustness and iOcclusion. For instance, one could assess, for a fixed occlusion type, how correlated *accuracy drop* is with *iOcclusion*, with respect to different networks. \n\nThe definition of iOcclusion using the train-test generalization gap to improve consistency (w.r.t. colour/shape pattern of occlusion patch) of the metric is a very interesting and, to the best of my knowledge, novel idea. I find the argument regarding consistency with weaker models (Lines 207-216, Figure 4) comparatively weaker as I expect CutOcclusion to also have this property if it was instead defined as the drop in accuracy, as is commonly done.\n\nFinally, the authors argue against existing model-based measures of data augmentation quality such as diversity and affinity, and suggest to instead focus on introduced biases. This is well supported by the inter-dataset experiments.\n\nIn terms of clarity, I found the paper very readable and well-written on the small scales, but I felt the narrative of the paper is sometimes muddled and unclear, and the \"big picture\" can get lost in the minutiae. For instance, a clear statement of the desiderata of iOcclusion at the start of Section 3 would be helpful. Also, I found the Introduction to be quite difficult to follow without having read the rest of the paper beforehand. For instance, I found the statement in Lines 43-44 “In summary, it is currently assumed that the artefacts introduced by changes in the data are negligible when evaluating models, while those introduced when training are important and undesirable.” to be very difficult to interpret without having read the whole paper to understand what is meant by “negligible” and “important and undesirable”. I think the paper would be improved if the Introduction focused more on examples and the intuitive argument. **AR**: The authors have agreed to address clarity issues (also raised by other reviewers); I would once again strongly encourage the authors to carefully reassess each section to make sure a) the idea is clear without reading the whole section and b) all quantities and concepts  are fully defined and the logical argument is made as clear as possible. Clarity is always important but especially paramount for papers such as this which follow an unconventional structure and whose main contribution is a critical analysis of previous work.\n\n**AR**: In light of the authors' response, I have increased my score and vote to accept the paper. As the other reviewers have pointed out and the authors have noted, the study mostly focuses on negative results showing counterexamples to prior work rather than new solutions. However, in my opinion this is not a key limitation as such results are also important, and the paper makes a number of interesting observations, such as the bias of artefacts in robustness evaluation metric, backed up with empirical evidence. If the logical arguments are laid out more clearly, along with the adjustments that the authors have agreed to make, I believe that the paper consititutes a good contribution to the area which will be of utility to other researchers. \n\nOther questions:\n\n• In Figure 3, for fraction removed 0.1 and the FMix model, iOcclusion is much greater than one, meaning the generalization gap is larger for occluded images than for the original images. Do the authors have an explanation for why this is? \n\n• In the definition of DI, is increase in misclassification for class $c$ calculated for images with true label $c$, or predicted as $c$ by the classifier?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, limitations and societal impact are adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the interesting topic of the artefacts and biases that can be introduced by approaches that rely on data manipulation.

After the discussion period, the paper received final scores of 7663, but where the 3 was a very low confidence review with little detail and for which the reviewer did not participate in discussions or requests to add more detail to their review.  I have predominantly thus discounted this review from my deliberations, but not entirely as I do think the failure of this reviewer to understand the paper is actually partly representative of clarity issues of the work raised by other reviewers.  It is also worth noting that this was a paper where it proved extremely difficult to recruit appropriate reviewers, with very few bids received in the bidding process and numerous assigned reviewers pulling out because they did not feel they could sufficiently well understand the work to assess it; something which I think is again reflective of clarity issues as discussed below.

Overall, I see this paper as very much on the borderline, with reviewers praising some of its underlying content and insights, but repeated issues raised on clarity and reviewers sometimes unsure about the paper's contributions.  Indeed, I myself struggled substantially when I tried to read the paper and believe it currently requires substantial effort to understand.  In particular, I think the introduction (and to a lesser extent the abstract) are both very poor at outlining what the paper is trying to achieve and giving motivation and context for the work.  I think one reviewer puts it very well when they say "I found the Introduction to be quite difficult to follow without having read the rest of the paper beforehand" which is exactly the opposite of how a paper should read: the introduction is usually the most critical part of the paper and lays the foundation for the reader to understand the work.  Similarly, issues then percolate down through the paper, which, amongst other things, is regularly too vague and imprecise; often lacks a central narrative and fails to properly link the lower level ideas back into the higher level aims; and generally lacks the hand-holding and high-level glue that make a paper easy to follow.

In light of this, the keys questions become whether the strengths of the paper are sufficient for these shortfalls on clarity to be overlooked and whether the required adjustments can reasonably be expected to be made in time for the camera-ready.  Though I think this is a very tight decision (which I mulled over for some time), unfortunately, my final conclusion is that too much needs to be done for the paper to be accepted at this time (e.g. I think the introduction needs a complete rewrite), bearing in mind the highly competitive nature of this review process.  I hope though that the reviews and discussions have been helpful to the authors and that they will make the required updates and successfully resubmit to another venue.
Decision: Reject