FMix: Enhancing Mixed Sample Data Augmentation
Authors: Ethan Harris
Antonia Marcu
Matthew Painter
Mahesan Niranjan
Adam Prugel-Bennett
Jonathon Hare
Venue:  (2021)

Abstract: Mixed Sample Data Augmentation (MSDA) has received increasing attention in recent years, with many successful variants such as MixUp and CutMix. We analyse MSDA from an information theoretic perspective, characterising learned models in terms of how they impact the models’ perception of the data.  Ultimately, our analyses allow us to decouple two complementary properties of augmentations that are useful for reasoning about MSDA. From insight on the efficacy of CutMix in particular, we subsequently propose FMix, an MSDA that uses binary masks obtained by applying a threshold to low frequency images sampled from Fourier space.  FMix improves performance over MixUp and CutMix for a number of models across a range of data sets and problem settings,  obtaining new state-of-the-art results on CIFAR-10 and Fashion-MNIST.

Under review as a conference paper at ICLR 2021
FMIX: ENHANCING MIXED SAMPLE DATA AUGMEN- TATION
Anonymous authors Paper under double-blind review
ABSTRACT
Mixed Sample Data Augmentation (MSDA) has received increasing attention in recent years, with many successful variants such as MixUp and CutMix. We anal- yse MSDA from an information theoretic perspective, characterising learned mod- els in terms of how they impact the models' perception of the data. Ultimately, our analyses allow us to decouple two complementary properties of augmenta- tions that are useful for reasoning about MSDA. From insight on the efficacy of CutMix in particular, we subsequently propose FMix, an MSDA that uses binary masks obtained by applying threshold to low frequency images sampled from Fourier space. FMix improves performance over MixUp and CutMix for a num- ber of models across range of data sets and problem settings, obtaining new state-of-the-art results on CIFAR-10 and MaShion-MNIST.
INTRODUCTION
Recently, a plethora of approaches to Mixed Sample Data Augmentation (MSDA) have been pro- posed which obtain state-of-the-art results, particularly in classification tasks (Chawla et al.) 2002 Zhang et al. 2017] Tokozume et al.] 2017: 2018 Inoue 2018 Yun et al. 2019; Takahashi et al, Summeers and Dinneen 2019). MSDA involves combining data samples according some policy to create an augmented data set on which to train the model. The policies so far proposed can be broadly categorised either combining samples with interpolation (e.g. MixUp) masking (e.g. CutMix). Traditionally, augmentation is viewed through the framework of statistical learning as Vicinal Risk Minimisation (VRM) [Vapnik] 1999: Chapelle al. 200I). Given some notion of the vicinity of a data point, VRM trains with vicinal samples in addition to the data points them selves. This is the motivation for MixUp (Zhag et al., 2017); to provide a new notion of vicinity based on mixing data samples. In the classical theory, validity of this technique relies on the strong assumption that the vicinal distribution precisely matches the true distribution of the data. As re- sult, the classical goal of augmentation is to maximally increase the data space, without changing the data distribution. Clearly, for all but the most simple augmentation strategies, the data distribu- tion is in some way distorted. Furthermore, there may be practical implications to correcting this, as is demonstrated in Touvron et al.) (2019). In light of this, three important questions arise regarding MSDA: What is good measure of the similarity between the augmented and the original data? Why is MixUp so effective when the augmented data looks so different? If the data is distorted, what impact does this have on trained models?
To construct a good measure of similarity, we note that the data only need be 'perceived' similar by the model. As such, we measure the mutual information between representations learned from the real and augmented data, thus characterising how well learning from the augmented data sim- ulates learning from the real data. This measure clearly shows the data-level distortion of Mix by demonstrating that learned representations are compressed in comparison to those learned from the un-augmented data, To address the efficacy of MixUp, we look the information bottleneck theory of deep learning (Tishby and Zaslavsky 2013). This theory uses the data processing inequal- ity, summarised as "post-processing cannot increase information', to suggest that deep networks progressively discard information about the input whilst preserving information about the targets. Through this lens, we posit that the distortion and subsequent compression induced by MixUp pro- motes generalisation by preventing the network from learning about highly sample-specific features in the data. Regarding the impact on trained models, and again armed with the knowledge that MixUp distorts learned functions, show that Mix acts a kind of adversarial training (Good-

---

Under review as a conference paper at ICLR 2021
fellow al., 2014), promoting robustness to additive noise. This accords with the theoretical result of ArehambbalA et al. (200) and the robustness results of Zhang et al. (0117) However, we further show that MSDA does not generally improve adversarial robustness when measured as worst case accuracy following multiple attacks as suggested by Carlini et al. (211). In contrast to our findings regarding MixUp, show that CutMix causes learned models to retain a good knowl- edge of the real data, which we argue derives from the fact that individual features extracted by a convolutional model generally only derive from one of the mixed data points. At the same time Cut- Mix limits the ability of the model to over-fit by dramatically increasing the number of observable data points, in keeping with the original intent of VRM. We go on to argue that by restricting to only masking a square region, CutMix imposes an unnecessary limitation. Indeed, it should be possible to construct an MSDA which uses masking similar to CutMix whilst increasing the data space much more dramatically. Motivated by this, we introduce FMix, masking MSDA that uses binary masks obtained by applying a threshold to low frequency images sampled from Fourier space. Using our mutual information measure, we show that learning with FMix simulates learning from the real data even better than CutMix. We subsequently demonstrate performance of FMix for a range of models and tasks against series of augmented baselines and other MSDA approaches. FMix obtains new state-of-the-ar performance on CIFAR-I0 (Krizhevsky et al. 2009) without external data and Fashion MNIST (Xiao et al. 2017) and improves the performance of several ofte-of-theeart mod- els (ResNet, SE-ResNext, DenseNet, WideResNet. PyramidNet, LSTM, and Bert) on range of problems and modalities.
In light of our analyses, and supported by our experimental results, we go on to suggest that the compressing qualities of MixUp are most desirable when data limited and learning from individual examples is easier. In contrast, masking MSDAs such as FMix are most valuable when data is abundant. We finally suggest that there is no reason to see the desirable properties of masking and interpolation as mutually exclusive. In light of these observations, we plot the performance of MixUp, FMix, a baseline, and a hybrid policy where we alternate between batches of MixUp and FMix, as the number of CIFAR-10 10 training examples is reduced. This experiment confirms our above suggestions and shows that the hybrid policy can outperform both MixUp and FMix.
2 RELATED WORK
In this section, we review the fundamentals of MSDA. Let px (x) denote the input data distribution. In general, we can define MSDA for a given mixing function, mix(X1, A). where X1 and X₂ are independent random variables on the data domain and A is the mixing coefficient. Synthetic minority over-sampling (Chawla et al. 2002). a predecessor to modern MSDA approaches, can be seen as a special case of the above where X, and X₂ are dependent, jointly sampled as nearest neighbours in feature space. These synthetic samples are drawn only from the minority class to be used in conjunction with the original data, addressing the problem of imbalanced data. The mixing function is linear interpolation, mix (21, x222] + 2) and PA (10,, 1). More recently,
Zhang et al. (2017), Tokozume et al.] (2017). Tokozume et al. (2018) and Inoue (2018) concurrently proposed using this formulation (as MixUp, Between- Class (BC) learning. BC+ and sample pairing
respectively) on the whole data set, although the choice of distribution for the mixing coefficients varies for each approach. We refer to this interpolative MSDA, where, following Zhang et al. 2017) we use the symmetric Beta distribution, that is PA = Beta(a,
Recent variants adopt binary masking approach (Yun et al.. 2019 Summers and Dinneen 2019 Takahashi et al. 2019). Let M = mask(A) be a random variable with mask(A) E {0, 1)" and (mask(()) = X, that is, generated masks are binary with average value equal to the mixing coeffi cient. The mask mixing function is mix(x,, X2, m) m C x1 + (1 m) O x2, where @ denotes wise multiplication. A notable masking MSDA which motivates our approach CutMix (Yun)
et al. 2019). CutMix is designed for two dimensional data, with mask(A) E {0, 1) Juxh, and uses mask(A) ()) = pand_rect(ww11 X, hi - A), where aand_rect(r_.. w. rn) (0. 10,1,ax yields a binary mask with a shaded rectangular region of size Tw X Th at a uniform random coordinate. CutMix im- proves upon the performance of MixUp
a range of experiments. For the remainder of the paper we focus on the development of a better input mixing function. Appendix A provides a discussion of the importance of the mixing ratio of the labels. For the typical case of classification with cross entropy loss, the objective function is simply the interpolation between the cross entropy against each of the ground truth targets.
2

---

Under review as a conference paper at ICLR 2021
Table Mutual information between VAE latent spaces (Z and the CIFAR-I0 test (Z X)), and the CIFAR-I0 10 test set as reconstructed by a baseline VAE (T( X)) for VAEs trained with range of MSDAs. MixUp prevents the model from learning about specific features in the data. Uncertainty estimates are the standard deviation following 5 trials.
X(
MSE
Baseline 78.05 10.53
74.40 10.45
0.256.10.002
MixUp
70.38+0.00
68.58 1..12
0.288
CutMix
83.17+0.72
79.4640.75
0.254
Uniform Noise
DecpFool
Worst Case
1.00
Baseline
Accuracy (9)
0.75
MixUp
CutMix
0.50
0.25
0.00
10
10"
-
10"
10a 10
10
10
10 10
10
2
10
109
Figure Robustness of PreAct-ResNetle models trained on CIFAR-10 with standard augmen- tations (Baseline) and the addition of MixUp and CutMix to uniform noise, DeepFool (Moosavi- Dezfooli et al. 2016), and the worst case performance after multiple attacks including the additive uniform noise and DeepFool. MixUp improves robustness to adversarial examples with similar properties to images generated with MixUp (acting as adversarial training). but MSDA does not improve robustness in general. Shaded region indicates the standard deviation following repeats.
ANALYSIS
We now analyse both interpolative and masking MSDAs with a view to distinguishing their impact on learned representations. We summarise previous analyses and theories (Zhang al. 2019 Yun et al., 2019)) in Appendix G For desire measure which captures the extent which learning about the augmented data simulates learning about the original data. To achieve this, we propose training unsupervised models on real data and augmented data and then measuring the mutual information, the reduction in uncertainty about one variable given knowledge of another, between the representations they learn. In particular, we propose using Variational Auto-Encoders (VAEs) (Kingma and Welling 2013), which provide rich depiction of the salient or compressible information in the data (Higgins al.). Denoting the latent space VAE trained on the original data as Zx and on some candidate aug- mentation A as AA in Appendix B we show that
2017
Liang
et al. 20188) Guo et al. 2019 He et al. [2019] Verma et Verma our analysis,
obtain tractable lower bound, I AZ X), and upper bound, (ZZ:; X) where X is the original data as reconstructed by baseline VAE, for the intractable quantity 1 1(ZA; x. Table gives these quantities for MixUp, CutMix, and baseline. The results show that MixUp consistently reduces the amount of information that is learned about the original data. In contrast, CutMix manages to induce greater mutual information with the data than is obtained from just training on the un-augmented data. Crucially, the results present concrete evidence that interpolative MSDA differs fundamentally from masking MSDA in how it impacts learned representations.
Having shown this is true for VAEs, we now wish to understand whether the finding also holds for trained classifiers. To this end, in Figure 4 in the appendix we visualise the decisions made by classifier using Sradhent-weighhed weighted Class Activation Maps (Grad-CAMs) (Selvaraju et al. 2017). Grad-CAM finds the regions in an image that contribute the most to the network's ' prediction by taking the derivative of the model's output with respect the activation maps and weighting them according to their contribution. If MixUp prevents the network from learning about highly specific features in the data would expect more of the early features to contribute to the network output.
3

---

Under review as a conference paper at ICLR 2021
Clearly, it is difficult to ascertain whether this is the case from the examples in the figure, although there is some indication that it may be true. To verify empirically we compute the average sum of Grad-CAM heatmaps over the CIFAR-10 test set for 5 repeats (independently trained PreAct- & models). We obtain the following scores: baseline - 14615. MixUp 16213, CutMix 131 16. It is clear that average more of the early features contribute to the decisions made by MixUp trained models and that this result is consistent across independent runs.
Following on from these observations, it is now pertinent to ask whether these different represen- tations learned from MixUp give rise to practical differences other than just improved generalisa- tion. Since is our assessment that models trained with MixUp have an altered 'perception' of the data distribution, we suggest an analysis based on adversarial attacks, which involve perturb- ing images outside of the perceived data distribution to alter the given classification. We perform fast gradient sign method, standard gradient descent, projected gradient descent, additive uniform noise, and DeepFool (Moosavi-Dezfoool et al.) 2016) attacks over the whole CIFAR- 10 test set on PreAct-ResNet18 models subject to &o constraints using the Foolbox library (Rauber et al. 2020 2017). The plots for the additive uniform noise and DeepFool attacks, given in Figure || show that MixUp provides improvement over CutMix and the augmented baseline in this setting. This is because MixUp acts as a form of adversarial training (Goodfellow et al.] (014), equipping the models with valid classifications for images of similar nature those generated by the additive noise and DeepFool attacks. In Figure 1. we additionally plot the worst case robustness following all attacks as suggested by Carlini al. (2019). These results show that the adversarial training effect of MixUp is limited and does not correspond to general increase in robustness. We provide an enhanced depiction of these results in Appendix C
4 FMIX: IMPROVED MASKING
Our finding is that the masking MSDA approach works because it effectively preserves the data distribution in a way that interpolative MSDAs do not, particularly in the perceptual space of Con- volutional Neural Network (CNN). We suggest that this derives from the fact that each convolutional neuron at a particular spatial position generally encodes information from only one of the inputs at a time. This could also be viewed as local consistency in the sense that elements that are close to each other in space typically derive from the same data point. To the detriment of CutMix, it would be easy for a model to learn about the augmentation since perfectly horizontal and vertical artefacts are unlikely to be salient feature of the data. We contend that a method which retains the masking nature of CutMix but increases the space of possible shapes (removing the bias towards horizontal even greater knowledge of the un-augmented data in
and vertical edges) may be able to induce trained models as measured by our mutual information analysis. This should in turn correspond with improved accuracy. If we can increase the number and complexity of masks then the space of novel features (that is, features which occur due to edges in the mask) would become significantly larger than the space of features native to the data, As result, it is highly unlikely that a model would be able to 'fit' to this information. This leads to our core motivation: to construct a masking MSDA which maximises the space of edge shapes whilst preserving local consistency.
For local consistency, we require masks that are predominantly made up of a single shape or con- tiguous region. We might think of this as trying to minimise the number of times the binary mask transitions from '0' to '1' or vice-versa. For our approach, we begin by sampling a low frequency grey-scale mask from Fourier space which can then be converted to binary with a threshold. We will first detail our approach for obtaining the low frequency image before discussing our approach for choosing the threshold. Let denote a complex random variable with values on the domain Z C with density PR(Z) (0, Iw)h) and P3(2) (0, Iwxk). where R and 3 return the real and imaginary parts of their input respectively. Let freq(w, h) [ denote the magnitude of the sample frequency corresponding to the i, j'th bin of the w discrete Fourier transform. We can apply low pass filter to Z by decaying its high frequency components. Specifically, for a given decay power d, we use
filter(z, 0),, freq(w, h)) [i.
Defining F-1 as the inverse discrete Fourier transform, we can obtain a grey-scale image with
G BRF (fitter (Z, 6)))
4

---

Under review as a conference paper at ICLR 2021
Mask
Image
Image 2
FMix
Figure 2: Example masks and mixed images from CIFAR-IO for FMix with 8 3 and À 0.5.
All that now remains is to convert the grey-scale image to binary mask such that the mean value some given X. Let top(n, x) return a containing the top n elements of the input x. Setting the top Awh elements of some grey-scale image to have value "I' and all others to have value '0" we obtain a binary mask with mean A. Specifically, we have
1, if g E top(Awch, g)
mask(2,g)(t..))
(3)
0, otherwise
To recap, we first sample random complex tensor for which both the real and imaginary part are independent and Gaussian. We then scale each component according to its frequency via the parameter 8 such that higher values of correspond to increased decay of high frequency informa- tion. Next, we perform inverse Fourier transform on the complex tensor and take the real part to obtain a grey-scale image. Finally, we set the top proportion of the image to have value '1' and the rest have value '0' to obtain binary mask. Although we have only considered two di- mensional data here it is generally possible to create masks with any number of dimensions. We provide some example two dimensional masks and mixed images (with 5 = 3 and À 0.5) in Fig- ure 2 We can see that the space of artefacts is significantly increased, furthermore, FMix achieves I(Z = 83.67 10.89. xx) = 80.28 1077.. and MSE 0.255 showing that learning from FMix simulates learning from the un-augmented data to an even greater extent than CutMix.
EXPERIMENTS
We now perform a series of experiments to compare the performance of FMix with that of MixUp, CutMix, and augmented baselines. For each problem setting and data set, we provide exposition on the results and any relevant caveats. Throughout, our approach has been to use the which yield the best results in the literature for each setting. Unless otherwise stated, we use = 1 for the distribution X. For FMix, we use 8 = 3 since this was found to produce large artefacts with sufficient diversity. We perform an ablation of both parameters in Appendix H. reporting results for fold cross validation. We perform repeats where possible and report the average performance and standard deviation after the last epoch of training. A complete discussion of the experimental set-up can be found in Appendix F along with the standard augmentations used for all models on each data set. Additional experiments
point cloud and audio classification given in Appendix [ In all tables, we give the best result and results that are within its margin of error in bold. We discuss any cases where the results obtained by us do not match the results obtained by the authors in the accompanying text, and give the authors results in parentheses. Uncertainty estimates are the standard deviation over 5 repeats. Code for all experiments is given in the supplementary material.
Image Classification We first discuss image classification results on the CIFAR- 10/100 (Krizhevsky et al. 2009), Fashion MNIST (Xiao et al. 2017), and Tiny-ImageNet (Stanford, 2015) data sets. We train: PreAct-ResNet18 (He et al. (016)). WideResNet-28--10 (Zagoruyko and Komodakis 2016), ManseNet-BC-199 (Huang et al., 2017) and PyramidNet-272-2222
(Han et al.
2017). For PyramidNet, we additionally apply Fast AutoAugment (Lim et al. 2019). successor to Automugment (Cubuk et al., 2019a), and ShakeDrop (Yamada et al.) 2018) following Lim et al.
2019). The results in Table 2 show that FMix offers significant improvement over the other
5

---

Under review as a conference paper at ICLR 2021
Table 2: Image classification accuracy for our approach, FMix, against baselines for: PreAct-
ResNet18 (ResNet), WideResNet-28-10 (WRN), DenseNet-BC-190 (Dense), 2722-000 SrokeDrop + Fast AutoAugment (Pyramid). Parentheses indicate author quoted result.
Data set
Model
Baseline
FMix
MixUp
CutMix
ResNet
94.63
96.14_0.10
95.66-0.11
96.00
WRN
95.25
96.38
(97.3) 96.60
96.53
CIFAR-10
Dense
96.26
97.30=
(97.3) 97.05
96.96
Pyramid
98.31
98.64
97.92
98.24
ResNet
75.22
79.85
(78.9) 77.44..0.5.
79.5 +0.38
CIFAR-100
WRN
78.2640.25
82.03
(82.5) 81 31.0940.33
81.9640..00
Dense
81.73
83.95
43.33+0.30
82.79
ResNet
95.70=0.09
96.36=0000
96.28 96.28
+0.08
96.03
Fashion
WRN
95.29 10.17
96.00
95.75 10.09
95.64 10.20
Dense
95.84
96.26
96.30 10.00 10.04
96.12 10.13
Tiny- Imageeet
ResNet
55.94
61.43=0.37
55, 55,9640.41
64.0820.30
Table 3: Classification performance for ResNet101 trained ImageNet for 90 epochs with a batch size of 256, and evaluated on ImageNet and ImageNet-a, adversarial examples to ImageNet. Note
that /Zang al. (2017) (MixUp) batch size of 1024 and Yun et al. (2019) (CutMix) train for 300 epochs, so these results should not be directly compared.
Baseline
FMix
MixUp
CutMix
Data set
Q:
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
1.0
77.42
93.92
75.89
93.06
76.92
93.55
ImageNet
77.28
93.63
0.2
77.70
93.97
77.23
93.81
76.72
93.46
1.0
7.19
33.65
8.69
34.89
6.92
34.03
ImageNet-a
4.08
28.87
0.2
5.32
31.21
5.81
31.43
6.08
31.56
methods on test, with the exception of the WideResNet on CIFAR-10/100 and the PreAct-ResNet on Imn--mmaeNNet. In combination with PyramidNet, FMix achieves, to the best of our knowledge, new state-of-the-an classification accuracy on CIFAR-10 without use of external data. By the addition of Fast AutoAugment, this setting bares some similarity to the recently proposed AugMix (Hendrycks et al. 2019ab which performs MixUp on heavily augmented variants of the same image. With the PreAct-ResNetl on Fashion MNIST. Note that Zhang et al. (207)) also performed experiments with the PreAct- ResNet18, WidcResNNet28--U and ManseNett-CC1999 on CIFAR-10 10 and CIFAR-100. There are some discrepancies between the authors results and the results obtained by our implementation. Whether any differences are significant is difficult to ascertain as no measure of deviation is provided in Zhang et al. (2017). However, since our implementation based on the implementation from Zhang et al. (0117) and most of the differences are small, we have no reason to doubt it. We speculate that these discrepancies to differences in reporting or training configuration.
FMix obtains a new state-of-the-e classification accuracy
simply result of random initialisation, but could also be due
Next, we obtain classification results on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC2012) data set (Russakovsky et al., 2015). We train ResNet- 101 on the full data set (Ima- geNet), additionally evaluating on ImageNet-a (Hendryeks et al., 2019b). set of natural adversarial examples to ImageNet models, to determine adversarial robustness. We train for 90 epochs with batch size of 256. We perform experiments with both a = 1.0 and a 0.2 (as this was used by Zhang et al. (2017). The results, given in Table 3 show that FMix was the only MSDA to provide an improvement over the baseline with these hyper-parameters Note that MixUp obtains an accu- racy of 78.5 in Zhang al. (2017) when using batch size of 1024. Additionally note that Mix obtains an accuracy of 79.48 and CutMix obtains an accuracy of 79.83 in Yun et al. (2019) when training for 300 epochs. Due to hardware constraints cannot replicate these settings and is

---

Under review as a conference paper at ICLR 2021
Table 4: Classification performance for FMix against baselines on Bengali grapheme classification.
Category
Baseline
FMix
MixUp
CutMix
Root
92.86 10.20
96.13 97.05
94.80.1.10
95.74.10.2
Consonant diacritic
96.23 10.35
96.42.0.2
96.9611021
Vowel diacritic
96.91 10.19
97.77
96.74
97.37
Grapheme
87.60
91.87++030
89.23
91.08
Table 5: Classification performance of FMix and baselines on sentiment analysis tasks.
Data set
Model
Baseline
FMix
MixUp
CNN
96.04_0.16
96.80 00.6
96.62
Toxic (ROC-AUC)
BiLSTM
96.72.0.04
97.35 20.0
97.15
Bert (a=0.1)
98.222.003
98.26 =0003
CNN (a=0.2)
86.68
87.31 -0.34
88.94_0.13
IMDb
BiLSTM (a=0.2)
88.29
88.47
88.72
CNN
95.47 +0.08
95.80
95.91
Yelp Binary
BiLSTM
96.41 10.05
96.68_..066
96.71 20.07
CNN
63.78
64.46_0007
64.56
Yelp Fine-grasd fine-grained
BiLSTM
62.96
66.46_0.13
66. 1 1
not known how FMix would compare. On ImageNet-a, the general finding is that MSDA gives a good improvement in robustness to adversarial examples. Interestingly, MixUp with = 1.0 yields a lower accuracy on ImageNet but a much higher accuracy on ImageNet-a, suggesting that models trained with MixUp learn a fundamentally different function.
For final experiment with image data, we use the Bengali.Al handwritten grapheme classification data set (Bengali.Al 2020), from a recent Kaggle competition. Classifying graphemes is a multi- class problem. they consist of a root graphical form (a vowel or consonant, 168 classes) which is modified by the addition of other vowel (11 classes) consonant (7 classes) diacritics. To cor- rectly classify the grapheme requires classifying each of these individually, where only the root is necessarily always present. We train separate models for each sub-class, and report the individual classification accuracies and the combined accuracy (where the output is considered correct only if all three predictions are correct). We report results for folds where 80% of the data is used for training and the rest for testing. We extract the region of the image which contains the grapheme and resize to 64 x 64, performing no additional augmentation. The results for these experiments, with an SE-ResNeXt-50 (Xie et al.) [2017: Hu et al. 2018), are given in Table FMix and CutMix both clearly offer strong improvement over the baseline and MixUp, with FMix performing significantly better than CutMix on the root and vowel classification tasks. As result, FMix obtains significant improvement when classifying the whole grapheme. In addition, note that FMix was used in the competition by (Singer and Gordeev (2020) in their second place prize- winning solution. This was the best result obtained with MSDA.
Sentiment Analysis Although typically restricted classification of two dimensional data, can extend the MSDA formulation for classification of one dimensional data. In Table [ we perform a series of experiments with MSDAs for the purpose of sentiment analysis. In order for MSDA to be effective, we group elements into batches of similar sequence length as is already a standard practice. This ensures that the mixing does not introduce multiple end tokens or other strange artefacts (as would be the case if batches were padded to a fixed length). The models used are: pre- trained FastText-300d (Joulin et al. (0166) embedding followed by a simple three layer CNN (LeCun) et al. 1995), the FastText embedding followed by two layer bi-directional LSTM (Hochreiter and Schmidhuber 1997). and pre-trained Bert (Devlin et al. 2018 provided by the HuggingFace transformers library (Wolf et al. 2019). For the LSTM and CNN models we compare MixUp and FMix with a baseline. For the Bert fine-tuning we do not compare to MixUp as the model input is series of tokens, interpolations between which meaningless. We first report results the

---

Under review as a conference paper at ICLR 2021
Accuracy (56
90
FMix
96
Mixup
Baseline
80-
95
FMix + MixUp
94
70
0.0
0.2
0.4
0.6
0.8
1.0
Fraction of Total Datasel
Figure 3: CIFAR-10 performance for R change the amount of training data.
Toxic Comments (Jigsaw and Google 2018) data set, a Kaggle competition to classify text into one of 6 classes. For this data Note that these results are computed over the whole test set and are therefore not comparable to the competition scores, which were computed over subset of the test data. In this setting, both Mi and FMix provide an improvement over the baseline, with FMix consistently providing a further improvement over MixUp. The improvement when tueettunin Bert with FMix is outside the margin of error of the baseline, but mild in comparison to the improvement obtained in the other settings. We additionally report results on the IMDb (Maas et al., 2011). Yelp binary, and Yelp fine- grained (Zhang et al. 2015) data sets. For the IMDb data set, which has one tenth of the number of examples, we found a 0.2 to give the best results for both MSDAs. Here, MixUp provides a clear improvement over both FMix and the baseline for both models. This suggests that MixUp may perform better when there are fewer examples.
report the ROC-AUC metric, this was used in the competition.
Combining MSDAs We have established through our analysis that models trained with interpola- tive MSDA perform fundamentally different function to models trained with masking. We now wish to understand whether the benefits of interpolation and masking mutually exclusive. We therefore performed experiments with simultaneous action multiple MSDAs, alternating their application per batch with TreAct-ResNetle masking, particularly FMix+MixUp (96.30 gives the best results, with CutMix+MixUUp per- forming slightly worse (96.2610.04). In contrast, combining FMix and CutMix gives worse results (95.85...1) than using either method on its own. For a final experiment, we note that our results suggest that interpolation performs better when there is less data available (e.g. the IMDb data set) and that masking performs better when there is more data available (c.g. ImageNet and the Ben- Al data set). This finding is supported by our analysis since it is always easier for the model to learn specific features, and so we would naturally expect that preventing this is of greater utility, when there is less data. We confirm this empirically by varying the size of the CIFAR- 10 training set and training with different MSDAs in Figure 3 Notably, the FMix+MixUp policy obtains superior performance irrespective the amount of available data.
CIFAR-10. A combination of interpolation and
CONCLUSIONS AND FUTURE WORK
In this paper we have introduced FMix, a masking MSDA that improves classification performance for series of models, modalities, and dimensionalities. We believe the strength of masking meth- ods resides in preserving local features and we improve upon existing approaches by increasing the number of possible mask shapes. We have verified this intuition through novel information theoretic analysis. Our analysis shows that interpolation causes models to encode more general features, whereas masking causes models to encode the same information as when trained with the original data whilst eliminating memorisation. Our preliminary experiments suggest that combin- ing interpolative and masking MSDA could improve performance further, although further work needed to fully understand this phenomenon. Future work should also look to expand finding that masking MSDA works well in combination with Fast AutoAugment (Lim et al. 2019), perhaps by experimenting with similar methods like AutoAugment (Cubuk et al. (019a) or Ran- dAugment (Cubuk et al. 2019b). Finally, our early experiments resulted in several lines of enquiry that ultimately did not bare fruit, which we discuss further in Appendix
the
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Bengali.Al. Bengali.ai ai handwritten grapheme classification competition, 2020. URL https://
www. www.kaggle.co wwt.c////ccc/eeeeee httm////engaaaiiaaaammaaaaaaaaaa cv199/
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 2019.
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regular- ization. arXiv preprini arXiv:2006.06049, 2020.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In Advances in neural information processing systems, pages 416-422, 2001.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer, Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113-123, 2019a.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. arXiv preprint arXiv:1909. 13719, 2019b.
Jacob Devlin, Ming-Wos Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 0rXiv:1810.048805 2018.
Raphael Gontio-Lopes, Sylvia Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.088 08973, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint urXiv:1412.665 2014.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear larization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3714-3722, 2019.
regu-
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5927-5935, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
Zhuoxun He, Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data. arXiv preprint arXiv:1909.099 1909.09149, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07172. 2019b.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the varia- tional evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, page 2, 2016.

---

Under review as a conference paper at ICLR 2021
Jie Hu, Li Shen, and Gang Sun. Sgueeze-and-excitatin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and patlern recognition, pages 4700-4708, 2017.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. arXiv preprint arXiv:/801.0292) 2018.
Jigsaw and Google. Toxic comment classification challenge, 2018. URL https://wwwg
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016.
Diederik P Kingma and Max Welling. arXiv:1312.6114,22 2013.
Auto-encoding variational bayes. arXiv preprint
Artemy Kolchinsky and Brendan D Traccy. Estimating mixture entropy with pairwise distances. Entropy, 19(7):361, 2017.
Alex Krizhevsky et Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series.
The handbook of brain theory and neural networks, (16) 1995.
Daojun Liang, Feng Yang, Tian Zhang, and Peter Yang. Understanding mixup training methods. IEEE Access, 1158774-58783, 2018.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-voluee volume 1, pages 142-150 Association for Computational Linguistics, 2011.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puler vision and pattern recognition, pages 2574-2582, 2016.
Guillaume Perrault-Archambaule Yongyi Mao, Hongyu Guo, and Richong Zhang. Mixup as directional adversarial training, 2020. URL https://pperreveer.ct ferm:nrmm//f-
Skq JKRYY
Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human un- certainty makes classification more robust. In Proceedings of the IEEE International Conference on Computer Vision, pages 9617-9626, 2019.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652 660, 2017.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017.
1707. 04131
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensor- flow, and jax. Journal of Open Source Software. 5(53):2607, 2020. doi: 10.21105/joss.00 URL httpt::///// //dtoi .org/ 10. 21105 joss 02607
10

---

Under review as a conference paper at ICLR 2021
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 1 11(3))211-252, 2015. doi: 10. 1007/s1 1263-015-0816-)
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
Philipp Singer and Dmitry Gordeev. Bengali.al ai handwritten grapheme classification competition Second place solution, 2020. URL
Dnscuusiinn135500
Stanford. Tiny imagenet visual recognition challenge, 2015. URL BolLattps://ttby-mmageeee
herokuapp. com/
Cecilia Summers and Michael J Dinneen. Improved mixed-example data augmentation. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1262-1270. IEEE, 2019.
Ryo Takahashi, Takashi Matsubara, and Kumiaki Uchara. Data augmentation using random image cropping and patching for deep cnns. IEEE Transactions on Circuits and Systems for Video Technology, 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Learning from between-class examples for deep sound recognition. arXiv preprint 17X1v:1711.1025: 2017.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classi- fication. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5486-5494, 2018
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems, pages 8250-8260. 2019.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Rexearch, pages 6438-6447, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface' e's trans- formers: State-of-the the-thterrttt art natural language processing. ArXiv, abs/1910.03771, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 1912-1920, 2015.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 170.007777 2017.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1 1492-1500, 2017.
Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and Koichi Kise. Shakedrop regularization for deep residual learning. arXiv preprint arXiv: 0802.02375 2018.

---

Under review as a conference paper at ICLR 2021
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. arXiv preprint 900,09990 2019.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07146, 2016.
Wide
residual networks.
arXiv preprint
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 171.:9171 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pages 649-657, 2015.
12

---

Under review as a conference paper at ICLR 2021
A ON THE IMPORTANCE OF TARGETS
The standard formulation for classification with MSDA weights the cross entropy losses computed with each of the true labels by the corresponding input mixing ratio. It could be suggested that by mixing the targets differently, one might obtain better results. However, there are key observations from prior art which give us cause to doubt this supposition; in partieular, Liang et al. (218)) per- formed number of experiments concluded that when the targets are not mixed in the same proportion as the inputs the model can be regularised to the point of underfitting. However, despite this conclusion their results show only mild performance change even in the extreme event that targets are mixed randomly, independent of the inputs. That doesn't 't mean that the target space is always insignificant. For example, we might care about how calibrated the outputs are. Calibration is the extent to which an output 'probabil- ity' corresponds to the actual probability of being correct. Clearly, this is challenging property to evaluate since we have no notion of ground truth uncertainty in the data. Peterson et al. (2019) suggest using human uncertainty as baseline on the CIFAR- 10 data set. Specifically, they intro- duce the CIFAR- 10H data set which consists of human soft-labels for the CIFAR-10 10 test set, i.e. the distribution resulting from many different humans labelling each image. We evaluate a series of CIFAR-10 pretrained PreAct-ResNet18 8 models on CIFAR-10F in Table The metric used is the relative entropy of the model outputs with respect to the soft-labels. The results show that the masking MSDA approaches induce a notion of uncertainty that is more similar to that of human observers. An important weakness of this claim derives from the cross entropy objective used to train models. We note that
the importance of the mixing ratio of the labels in MixUp. They
H(P PY x) (PP D(PF) Ix
In other words, the model is jointly required to match the target distribution and minimise the en- tropy of each output. The result of this that trained models naturally output very high confidence predictions as an artefact of their training process. The above claim should therefore be taken with a pinch of salt since is likely that the improved results derive simply from the lower entropy targets and model outputs. Furthermore, we expect that significant improvement would be gained in this test by training MSDA models with a relative entropy objective rather than the cross entropy.
Table 6: Mcan and standard deviation divergence scores on CIFAR- 10H, using the PreAct ResNet18 model trained on CIFAR-I0.
Baseline
FMix
MixUp
CutMix
D(p PY (x)
0.71610.032
0.22010.009
0.23941.000
0.211 10.005
B VAE MUTUAL INFORMATION
Recall from the paper that we wish to estimate the mutual information between the representa- tion learned by a VAE from the original data set, Zx, and the representation learned from some augmented data set, Z.. written [(Zx: AA) = Ezx [D((z, 2x) [l VAEs comprise an encoder, P(z) and a decoder, P(x 2)- We impose a standard Normal prior on Z, and train the model to maximise the Evidence Lower BOund (ELBO) objective
L= Ex [ZIx x [log(p(x12)) [ (P(z)) (l,
(5)
Denoting the outputs of the decoder of the VAE trained on the augmentation as X decode(Zx).
and by the data processing inequality, we have I(Za;X) < I(Z,; Zx) with equality when the decoder retains all of the information in Z. Now, we need only observe that we already have model of P(ZA |x), the encoder trained on the augmented data. Estimating the marginal pz, presents challenge it is Gaussian mixture. However, information that is equivalent up to an additive constant, and for which the divergence has a closed form solution, with
measure alternative form of the mutual
F [D(P [pz))] Ex (D(PizA18)) (1 D)] D(PZ, (I((0, 1))
(6)
13

---

Under review as a conference paper at ICLR 2021
The above holds for any choice of distribution that does not depend X. Conceptually, this states that we will always lose more information on average if we approximate Piz, 18) with any constant distribution other than the marginal Pža- Additionally note that we implicitly minimise D(zz |O, I)) during training of the VAE (Hoffman and Johnson 2016). In light of this fact, can write ((ZA; Ex (1 (O,
We can now easily obtain a helpful upper bound of (Z Zx) such that bounded on both sides. Since ZA is just function of X, again by the data processing inequality, we have T(ZA; X) 2 A( Zx). This is easy to compute since the ELBO objective. To summarise, we can compute our measure by first training two VAEs, one on the original data and one on the augmented data. We then generate reconstructions of data points in the original data with one VAE and encode them in the other. We now compute the expected value of the relative entropy between the encoded distribution and an estimate of the marginal to obtain estimate of lower bound of the mutual information between the representations. We then recompute this using real data points instead of reconstructions to obtain an upper bound.
just the relative entropy term from
C SUPPLEMENTARY ANALYSES
Input
Bascline
MixUp
CutMix
Figure 4: Grad-CAM at the output of the first convolutional layer of a PreAct-ResNet1 trained with
range of MSDAs.
D ADDITIONAL EXPERIMENTS
Table 7: Classification performance for our approach, FMix, against baseline for a PointNet QQi et al. 2017) on ModelNet10 (Wu et al. 2013)
Data set
Model
Baseline
FMix
MixUp
CutMix
ModelNet10
PointNet
89.10
89.57 20.44
ResNet (a=1.0)
98.59
98.461.008
98.464008
Commands
97.69
ResNet (a=0.2)
98.44_0.06
98.31 +0.08
98.48
Point Cloud Classification We now demonstrate the extension of FMix to 3D through point cloud classification ModelNet10 (Wu et al. 2015). We transform the pointclouds to voxel representa- tion before applying 3D FMix mask. Table 7 reports the average median accuracy from the last epochs, due large variability in the results. shows that FMix continues improve results within significance, even in higher dimensions.
The Google Commands data set was created to promote deep learning re- search speech recognition problems. It is comprised of 65,000 one second utterances of one of 30 words, with 10 of those words being the target classes and the rest considered unrelated or background noise. We perform MSDA Mel-frequency spectrogram of each utterance. The
Audio Classification
14

---

Under review as a conference paper at ICLR 2021
FGSM
Gradient Descent (GD)
Projected GD
1.00
Baseline
Accuracy (%6
0.75
MixUp
CutMix
0.50
FMix
10
10"
10
10a
10
10
10
10
10
10
10
10P
800
Uniform Noise
DeepFool
Worst Case
1.00
Baseline
Aceuracy (%)
0.75
MixUp
CutMix
FMix
0.25
0.00
10-3
10-2
10
10a
10
10
10
10
10
10
10
10
8.0
Figure 5: Full set of adversarial robustness results over the CIFAR-10 10 test set for range of MSDAs, including our approach FMix.
results for PreAct ResNet- 18 given in Table We evaluate FMix, MixUp, and CutMix for the standard a = used for the majority of our experiments and a = 0.2 recommended by Zhang et al.) (2017) for MixUp. We see in both cases that FMix and CutMix improve performance MixUp outside the margin of error, with the best result achieved by FMix with a 1.
15

---

Under review as a conference paper at ICLR 2021
E EXPERIMENTAL DETAILS
In this section we provide the experimental details for all experiments presented in the main paper. Unless otherwise stated, the following parameters chosen: a 6 = 3, weight decay of 1 x 10a and optimised using SGD with momentum of 0.9, For cross validation experiments, 3 or 5 folds of 10% of the training data are generated and used for single run each. Test set experiments use the entire training set and give evaluations on the test sets provided. If no test set is provided then constant validation of 10% of the available data used. Table provides general training details that were present in all experiments.
All experiments were run on single GTX10800 or V100, with the exceptions of ImageNet ex- periments (4 x GTX1080ti and DenseNedPyramiddNe experiments (2 V100). ResNet18 and LSTM experiments ran within 2 hours in all instances, PointNet experiments ran within 10 hours, WideResNNetDenseNNe experiments ran within 2.5 days and auto-augment experiments ran within 10 days.
For all image experiments perform random crops and random horizomtal flips. For the google commands experiment we used the transforms and augmentations implemented Setprrcc-sspeeehc-cooooe
standard augmentations to normalise the image to [0.1] and
/ wwt.t.......g.gg.gg...g..... com/ tugstuggll for their solution to the tensorflow speech regonition challenge.
Table 8: General experimental details present in all experiments. Batch Size (BS), Learning Rate (LR). Schedule reports the epochs which the learning rate multiplied by 0.1. Adam optimiser used.
Experiment
Model
Epochs
Schedule
BS
LR
PreAct-ResNet18 88
200
100, 150
128
0.1
WideResseet-2--U1
200
100, 150
128
0.1
CIFAR-10/ 100
DenseNet-BCC-99
300
100, 150, 225
32
0.1
PyramidNe1:222-200
1800
Cosine-Annealed
0.05
8
200
100, 150
128
0.1
FaSSionMNIST
WideRResNet22-110
300
100, 150, 225
32
0.1
DenseNet-BCC-900
300
100, 150, 225
32
0.1
Google Commands
&
90
30, 60, 80
128
0.1
ImageNet
ResNet101
90
30, 60, 80
256
0.4
TinylmageNet
PreActtResNNe118
200
150, 180
128
0.1
Bengali.Al
PreAct-ResNet188
100
50, 75
512
0.1
CNN
15
10 3
Sentiment Analysist
LSTM
15
10
64
10
Bert
5
3
32
10 10-5
Combining MSDAS
8
200
100, 150
128
0.1
ModelNet10
PointNet
50
10, 20, 30, 40
10
Ablations
PreAct-ResNet18
200
100, 150
128
0.1
16

---

Under review as a conference paper at ICLR 2021
F THINGS WE TRIED THAT DIDN'T WORK
This section details a number of experiments and modifications we attempted which did not lead to significant results. Our aim here to prevent future research effort being devoted to approaches that have already been explored by us. It may also be the case that better versions of these could be constructed which obtain better results.
F.1 SALIENCE PRIOR
It is clear that we should care about how the mixing coefficient relates to the relative amount of salient information from each data point in the outcome. This presents a challenge because getting A of the salient information in the first data point does not imply that we have 1 À of the salient information in the second. We could consider making an assumption that the expected distribution of salient information in each data point the same. In such a case, the above problem no longer exists. For images, a simple assumption would be that the salient information is roughly Gaussian about the centre. To apply a salience prior to our mask generation process, we need change the binarisation algorithm. Specifically, we iterate over the values in descending order until the mass over the prior equal to A. We experimented with this approach and found no significant performance gain, and so did not pursue it any further. That said, there may still be some value to the above motivation and a more complex, data point specific, salience distribution could work.
F.2 MASK SOFTENING
Following the observation that combining interpolation and masking provides the best results, and particularly the experiments in Summmers and Dinneen (2019), we considered a grey-scale version of FMix. Specifically, we explored a method which softened the edges in the mask. To achieve this, after sorting the low frequency image by pixel value, instead of choosing a threshold and setting one side to 1 and the other O, we choose an equal distance either side of the threshold and linearly value the mask between and 0 for some number of pixels. The number of grey pixels is chosen to ensure that the mean mask value is retained and that the fraction of the image that non-binary does not exceed some present value.
We found that softening the masks resulted in no performance gains, and in fact, occasionally hin- dered training. We considered it again for the toxic comments experiments since we assumed smooth transitions would be very important for text models. It did offer minor improvements over default FMix, however, we judged that the gain was not worth the added complexity and diluting of the core idea of FMix for us to present it in the paper. Furthermore, proposing it for the singular case of toxic comments would have been bad practice, since we only observed improvement for one model, on one data set. That said, we feel mask softening would be interesting to explore further, certainly in the case of text models. We would need to experiment with softened FMix masks in multiple text data sets and observe improvement in most or all of them over base FMix in order to formally propose softening an FMix modification.
F.3 TARGET DISTRIBUTION
A final alteration that we experimented with relates to the distribution of targets. The idea was that we could change the distribution of the target mixing coefficients to obtain better 'calibrated" model outputs. The way this is done is simple, we pass the sampled À through its CDF and then through the inverse CDF of the target distribution. This allows us to, for example, encourage confident outputs by choosing symmetric Beta distribution with a z 0.1. The issue with this approach is two fold. First, changing the distribution of the outputs in this way has no bearing on the ordering, and so no effect on the classification accuracy. Second, any simple transform of this nature can be trivially learned by the model or applied in post. In other words, it is equivalent to training model normally and then just transforming the outputs. As result, it is difficult argue that this approach does anything particularly clever. We trained models with different target distributions at several points and found that the performance was not significantly different.
17

---

Under review as a conference paper at ICLR 2021
G CURRENT UNDERSTANDING OF MSDA
Attempts to explain the success of MSDAs were not only made when they were introduced, but also through subsequent empirical and theoretical studies. In this section we review these studies to paint picture of the current theories, and points of contention, on how MSDA works. In addition to their experimentation with the targets, Liang et al. (2018) argue that linear interpolation of inputs limits the memorisation ability of the network. [Gontijo-Lon Lopes et al. (2020) proposes two measures to explain the impact of augmentation on generalisation when jointly optimised: affinity and diver- sity. While the former captures the shift in the data distribution as perceived by the baseline model, the latter measures the training loss when learning with augmented data. somewhat more math- ematical view on MSDA was adopted by Guo et al. (2019), who argue that MixUp regularises the model by constraining it outside the data manifold. They point out that this could lead to reducing the space of possible hypotheses, but could also lead to generated examples contradicting original ones, degrading quality. Upon exypadi- the objective, a more recent study that also focuses on MixUp motivates its success by the co-action of four different regularisation factors (Carratino iet al. 2020).
Following Zhang al. (017). He et al. (2019) take statistical learning view of MSDA, basing their study on the observation that MSDA distorts the data distribution and thus does not perform VRM in the traditional sense. They subsequently propose separating features into 'minor' and 'ma- jor', where a feature is referred to as 'minor' if it is highly sample-specific. Augmentations that significantly affect the distribution are said to make the model predominantly learn from 'major' features. From an information theoretic perspective, ignoring these 'minor' features corresponds to increased compression of the input by the model. Although He et al. (2119) noted the importance of characterising the effect of data augmentation from an information perspective, they did not explore any measures that do so. Instead, He et al. (2019) analysed the variance in the learned represen- tations. It can be seen that this is analogous to the entropy of the representation since entropy can be estimated via the pairwise distances between samples, with higher distances corresponding to both greater entropy and variance (Kolchinsky and Tracey 2017). In proposing Manifold MixUp, Verma et al. (2019) additionally suggest that MixUp works by increasing compression. The authors compute the singular values of the representations in early layers of trained networks, with smaller singular values again corresponding to lower entropy. The issue with these approaches is that the entropy of the representation is only an upper bound on the information that the representation has about the input.
An issue with these findings is that they relate purely to interpolative MSDAs. is also the case that there is disagreement in the conclusions of some of these studies. If interpolative MSDA works by preventing the model from learning about so called 'minor' features, then that would suggest that the underlying data distribution has been distorted, breaking the core assumption of VRM. Furthermore, Yun et al. (2019) suggested that masking MSDA approaches work by addressing this distortion. this is the case then we should expect them to perform worse than interpolative MSDAs since the bias towards compressed representations has been removed. Clearly, there is some contention about the underlying mechanisms driving generalisation in MSDAs. In particular, it is necessary to provide an explanation for masking MSDAs that is complementary to the current explanations of interpolative MSDAs, rather than contradictory them.
18

---

Under review as a conference paper at ICLR 2021
H HYPERPARAMETER CHOICE
Figure 6a gives the relationship between validation accuracy and the parameter a for three MSDA methods. Validation accuracy the average over folds with validation set consisting of 10% of the data. This ablation was performed on the CIFAR-10 data set using the PreAct ResNet18 model from the previous experiments. In the cases of FMix and Mix there exists an optimal value. In both cases, this point is close to = 1, although for MixUp it is skewed slightly toward 0, as was found for their ImageNet experiments. The choice of decay power is certainly more significant. Figure [bb] shows that low values of drastically reduce the final accuracy. This is unsurprising since low 5 corresponds to a speckled mask, with no large regions of either data point present the augmentation. Larger values of ó correspond smoother marks with large cohesive regions from each donor image. We note that for 6 2 there little improvement to gained, validating our decision to 5 3. 3.
Aceuracy (%))
Accuracy (96)
91
911
FMix
Mixup
CutMix
FMix
2
2
(a) Performance of masking MSDAs (FMix and Cut- Mix) remains with increased mixing increases). Performance interpolative MSDAs (MixUp) does degrade, since data level distortion increases.
(b) Performance of FMix increases with the decay power 8. Using lower frequency grey-scale image (increasing 5) increases local consistency up to a point (8 = 3).
Figure 6: CIFAR-10 accuracy for PreAct-ResNet11 with varying a trained with FMix (ours), MixUp and CutMix (Figure ba), and with varying 6 trained with FMix (Figure 6b
19


Reviews:
[{"review_id": "dDAQqn0WkeE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, authors provide an analysis of mutual information for MSDA and the develop a new variant of mixup. The effectiveness is demonstrated by experiments.\n\nStrength\n1.\tAuthors study the difference between masking MSDA and interpolative MSDA, which is helpful for understanding the power of mixup and its variants.\n2.\tThey develop a new augmentation method and improve the performance of masking MSDA.\n\nWeakness\n1.\tThe proposed measurement is not helpful for designing new methods. Note that the mutual information in mixup is lower than baseline while mixup still outperforms baseline.\n2.\tCompared to mixup and cutmix, the improvement reported in Table 2 is marginal.\n3.\tThe experiments on ImageNet is unconvincing. Both of mixup and cutmix are worse than baseline, which contradicts the existing results.\n4.\tThere lacks the discussion for the saliency based mixup methods, e.g., Puzzle Mix [1]. It is closely related to fmix but equipped with a learnable strategy to obtain patches for mixing.\n\n[1] J-H Kim, et al. Puzzle mix: Exploiting saliency and local statistics for optimal mixup", "extra_scores": {}, "extra_text": {}}, {"review_id": "OM9oOeLT16", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an advanced masking strategy for CutMix augmentation based on the low-pass filter. The authors provide an interesting mutual information analysis for different augmentation strategies to describe their motivation. The experiments include many vision tasks (CIFAR-10, CIFAR-100, Fashion-MNIST, Tiny-ImageNet, ImageNet, Bengali datasets) and language tasks (Toxic, IMDb, Yelp).\n\n**Pros**\n\n\\+ The mutual information analysis provides us a new perspective to understand different data augmentations.\n\n\\+ Various experiments.\n\n**Cons**\n\n**[Contradictory results between mutual information and performances]**\nIf we believe the VAE experiments in section 3, we have another paradox: the mutual information measurement and the real performance are not related.\nTable 1 shows that in terms of mutual information, MixUp < Baseline < CutMix (and < FMix with a very small gap).\nHowever, many experiments in this paper show that baseline < mixup < cutmix in terms of the performances.\nThis paper cites information bottleneck theory to justify the deceases shown by Mixup, but it is still contradictory to the performances.\nIt makes me confused to understand the meaning of mutual information. What is good for an augmentation method if we have high or low mutual information? It is still unclear to me.\n\nA similar comment also can be applicable to the \"adversarial robustness\" experiments. Aside from that mixup is hard to say \"adversarial training\" (what is the threat model in this scenario?), I feel that this result is irrelevant to FMix motivation.\n\n\n**[Weak logical connection between the motivation and the method]**\nIn my opinion, the connection between the analysis in the motivation and the proposed method is too weak. This paper proposes a CutMix variant where the mask is sampled by a low-pass filter. Why the low-pass filter approach can solve the motivation, i.e., enhancing mutual information between input and augmented images? There could be other possible variants as discussed in my \"related works\" comment\n\n\n**[Related works]**\nThere are a few CutMix variants that employ a non-random masking strategy. Especially, I believe these two variants, which have similar motivation, should be compared:\n\n- Walawalkar, Devesh, et al. \"Attentive Cutmix: An Enhanced Data Augmentation Approach for Deep Learning Based Image Classification.\" ICASSP 2020\n- Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. \"Puzzle mix: Exploiting saliency and local statistics for optimal mixup.\" ICML 2020\n\nwhere Attentive CutMix uses CAM to extract masks, and PuzzleMix employs an optimization problem to optimize masks.\nIf it is possible, please provide more comparison between these two papers.\n\n\n**[Too small performance gap, less convincing experiments]**\nIn Table 2, the performance gap between FMix and CutMix is too small, usually less than 0.3%. Note that the performance gaps are almost neglectable in these tasks.\n\nFurthermore, FMix is often worse than CutMix in many tasks (Table 2 TinyImageNet, Table 3 ImageNet-A, Table 4, CIFAR-10H Table 6). I wonder what is the advantage to use FMix comparing to CutMix if FMix shows worse performance than CutMix.\n\nEspecially, I believe Table 3 is problematic. This paper argues that \"Mixup uses 1024 batch size and CutMix uses 300 epochs\". However, in PuzzleMix Table 5, CutMix-trained ResNet50 (top1 err 22.92) outperforms baseline ResNet50 (top1 err 24.31) with only 100 epochs. Thus, to me, this table is not convincing enough.\n\n\n**[Potential issues in VAE analysis]**\nThe mutual information analysis is heavily relying on the learned VAE model. I wonder the quality of the generated images by VAE, in terms of both qualitatively (please provide generated samples in the supplementary) and quantitatively (e.g., FID).\nIf the VAE is not optimized well, the analysis will not be convincing enough.\n\n\n**Minor comments**\n- Why CutMix experiments are missed in Table 5?\n- I suggest avoiding using the words, \"clear\" and \"clearly\".\n\n---\n\n**Post-rebuttal update**\n\nMy main concerns in the initial review were three-folds:\n\n- Potential flaws in the analyses based on VAE and adversarial attacks\n- Unclear connection between the MI analysis and the proposed method\n- Small performance gap, and even sometimes worse performance, compared to the baseline methods (Mixup, CutMix)\n\nAfter having discussions with the authors, I will keep my initial score because:\n\n- I am still confused about the MI-based analysis conclusion. The authors mentioned *\"We make no claim that increasing or decreasing the mutual information measure will have a strong impact on performance. Instead, we contend that MixUp works by forcing the model to ignore sample specific features (thus learning compressed representations – the reason for discussing the information bottleneck theory) and that CutMix works by mimicking the real data whilst preventing example memorization.\"* in the rebuttal, but these two conclusions are not trivial to me (by the MI analysis).\n- Even if we ignore the first part, my second concern still remains. The authors mentioned *\"That is the problem FMix tries to solve by removing the horizontal and vertical edge artefacts from cutmix. Our belief is that cutmix biases models towards these edges as they are a guaranteed feature of the data and learning about them would reduce the loss since these edges can tell you how much of each source image is present in the input (a key part of the objective).\"*. But if this paper assumes that the rectangle masking strategy of CutMix makes bias, then I think other CutMix variants such as AttentiveCutMix or PuzzleMix should be considered as the comparison methods. Hence, I disagree with this statement *\"A comparison to masks generated using additional models (and, thus, significant additional computation) does not seem fair to us.\"*\n- For my last concern, the small performance gap, the authors claimed that this method *\"was also used by the second place team in the BengaliAI Kaggle competition\"*. It is good evidence that FMix can sometimes offer benefit to real-world applications, but I think more evidence that FMix can really solve problems of previous MSDA in a certain scenario, e.g., the edge bias as pointed by the authors.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ND9Xrp8upVc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new mixup method that builds masks by first sampling a grey-scale mask from fourier space, which is subsequently transformed into a binary mask. This improves results against several baselines and achieves state-of-the-art on a few important vision benchmarks.\n\nMy first remark is about the masking. The procedure seems fine, but why not compare to the way masks are sampled in context encoders [1]. This seems like an important baseline masking method to compare to. In addition, one could try sampling masks from a standard segmentation model, e.g., R-CNN.\n\nMy second remark is about the MI bounds. On page 14 in the Appendix, you state that the MI between Z_A and X_hat is approximately equal to the KL divergence between the posterior and the normal distribution, but in general this wont be true as in training the Gaussian mixture p_Za wont match the normal distribution, so you have an upper bound. So you have a lower bound of an upper bound to the MI, not a lower bound.\n\nI'm curious though why not just use one of the recent neural estimators, e.g., found in [2] or [3]. In general, using VAEs for MI estimators depends heavily on the quality of the generator, so these neural estimators might be better suited.\n\nOther comments:\nP1:\n* \"'post-processing cannot increase information'\" if such processing is deterministic, no?\n\nP2:\n* \"CutMix imposes an unnecessary limitation\": what limitation? Could you clarify?\n\nFinally, do you plan to have updated results that compare to the 1024 batch size / 300 epoch settings?\n\n[1] Pathak, Deepak, et al. \"Context encoders: Feature learning by inpainting.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n[2] Belghazi, Mohamed Ishmael, et al. \"Mine: mutual information neural estimation.\" arXiv preprint arXiv:1801.04062 (2018).\n[3] Poole, Ben, et al. \"On variational bounds of mutual information.\" arXiv preprint arXiv:1905.06922 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "3u8821TDeo1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an interesting analysis of CutMix and MixUp data augmentation techniques. It also presents an improvement to CutMix that removes the horizontal/vertical axis bias. The idea to use fourier noise to construct masks for a variant of CutMix is interesting and well-motivated.\n\nMy main concern is whether the conclusions drawn by the analyses are fully grounded. The paper performs an analysis of the effect of the augmented data on learned representations by training unsupervised models on the augmented or clean data and measuring their mutual information. This analysis has the undesirable property of not matching the supervised case in a number of ways, such as different learning objectives, model architectures, etc. Even ignoring this, if we take the result that “MixUp consistently reduces the amount of information that is learned about the original data”, what then explains the improved generalization accuracy MixUp showcases in their original paper?\n\nMoreover, after claiming that the analysis indicates that MixUp learns different representations, the paper asks “whether these different representations learned from MixUp give rise to practical differences other than just improved generalisation.” The issue is that they do this via an adversarial attack analysis, rather than a more realistic non-worst-case robustness analysis. This leads to the conclusion “MixUp (...) does not correspond to a general increase in robustness.” But it does not answer the original question of whether “MixUp gives rise to practical differences other than just improved generalisation.” The finding that MixUp yields greater ImageNet-A robustness (presented later in the paper) also contradicts this early claim.\n\nThe finding that MixUp provides more compressed representations does not necessarily mean that masking augmentation methods are better than interpolation ones. The paper seems to acknowledge this as well, in the final paragraph of the introduction, where it describes an experiment in which combining FMix+MixUp gives the best results (presumably because their representations of data are different and therefore combining them would yield the best of both worlds). This seems to contradict the previous adversarial analysis in which MixUp was found to not yield significantly more robustness. Further, the combination experiment has the two leading combination methods (FMix+MixUp and CutMix+MixUp) yield very similar results (within the margin of error), which opens the question of whether FMix meaningfully improves over CutMix.\n\nOverall, I find the paper very easy to read and presenting some interesting ideas and even some exciting improvements in performance. I just wish the presentation and the claims made in the analysis of MSDA methods accounted for some of the inconsistencies described above.\n\nUpdate after rebuttal: I appreciate the authors' response and clarifications. I maintain my original score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes the space of mixed sample data augmentation approaches, and proposes a new variant, FMix, based on a new masking strategy. Reviewers point to the fact that FMix is only marginally better than previous approaches, that the experimental setup is unconvincing, and that the proposed analysis might not be grounded. This is a really borderline paper but I see the issues as more important than the benefits, so I recommend rejection.
Decision: Reject