<figure><img src="image_1.png" /></figure>

HAL
open science

# An Investigation of Methods for Improving Spatial Invariance of Convolutional Neural Networks for Image Classification

David Noel, Sumitra Mukherjee

► To cite this version:

David Noel, Sumitra Mukherjee. An Investigation of Methods for Improving Spatial Invariance of Convolutional Neural Networks for Image Classification. 2024. hal-04484940

HAL Id: hal-04484940

https://hal.science/hal-04484940v1

Preprint submitted on 1 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Open licence - etalab
# An Investigation of Methods for Improving Spatial Invariance of Convolutional Neural Networks for Image Classification

David Noel

College of Computing and Engineering
Nova Southeastern University
3300 S University Dr, Fort Lauderdale, FL 33328, USA

dn632@mynsu.nova.edu

Sumitra Mukherjee

College of Computing and Engineering
Nova Southeastern University
3300 S University Dr, Fort Lauderdale, FL 33328, USA

sumitra@nova.edu

## Abstract

Convolutional Neural Networks have achieved impressive results on complex visual tasks such as image classification. They are commonly assumed to be spatially invariant to small transformations of their input images. Spatial invariance is a fundamental property that characterizes how a model reacts to input transformations, i.e., its generalizability - and deep networks that can robustly classify objects placed in different orientations or lighting conditions have the property of invariance. However, several studies have shown that slight rotations, translations, or rescaling of their input images significantly reduce the network's predictive accuracy.

Data augmentation is a mainstream technique used to improve invariance in Convolutional Networks by artificially increasing the amount of training data by generating new and diverse data points with additional properties extracted from the existing dataset. This article provides a rigorous comparative analysis of two novel data augmentation techniques used to improve spatial invariance and reduce overfitting in Convolutional Networks: RandAugment (a stochastic technique that applies geometric transforms to images) and Conditional Generative Adversarial Networks (models that create synthetic samples).

Empirical results on three benchmark datasets using a base model developed for this study and finetuned pre-trained versions of ResNet50 and InceptionV3 indicate that RandAugment is more effective.

**Keywords:** spatial invariance, convolutional neural networks, generative adversarial networks, randaugment, deep learning
# 1. Introduction

Convolutional Neural Networks (ConvNets, or CNNs) [1], [2] have achieved remarkable success in many visual tasks, such as image classification [3]. They are assumed to be spatially invariant because of their unique architectures. *Spatial invariance* is a highly desirable property of CNNs in which the network can disentangle the poses and deformations of objects from their texture and shape [4]. It is a crucial property of image representation where the objects in an image are invariant to viewpoint changes [5]. According to [6], a deep network that can robustly classify objects despite the objects being placed in different orientations or lighting conditions is said to have the property of invariance.

However, several authors [7] - [10] have shown that contemporary ConvNets are not spatially invariant to small perturbations in their input images. For instance, [7] showed a 30% change in the accuracy of predictions made by a CNN when the same sample images had their pixels shifted by as little as one pixel. This lack of robustness significantly reduces their predictive abilities, which could lead to fatalities, racial discrimination, or other unexpected outcomes where reliability, security, and unbiasedness are critical [11].

Data augmentation [3] is a ubiquitous solution for improving spatial invariance in CNNs. Data augmentation artificially increases the training dataset by generating new and diverse data points extracted from existing training data. Furthermore, in so doing, it directly imbues sources of invariances within the training dataset [12]. While simple augmentation techniques are effective, they require domain expertise and do not capture enough variations within the training data. In contrast, advanced augmentation techniques that use deep learning, such as RandAugment [13] and Generative Adversarial Networks (GANs) [14], learn invariances from data by sampling from a continuous distribution space, leading to more diverse and varied training samples.

This study provides a rigorous comparative analysis of data augmentation using a Conditional GAN (CGAN) developed for this work, along with RandAugment, which is a stochastic technique that applies geometric transforms to images, in their propensity to improve the classification performance of modern Convolutional Networks such as ResNet50 [15] and InceptionV3 [16]. This enhancement is achieved by addressing spatial invariance and reducing overfitting. The study evaluated and compared combinations of these networks across a base model (NoelNet) developed for this study, using three benchmark image datasets: MNIST [2], FMNIST [17], CIFAR-10 [18] using the following metrics: training/validation loss, training/validation accuracy, test accuracy, precision, recall, and training latency. The experimental setup for each analysis was guided by five policies: Policy 1 (no data augmentation), Policy 2 (augmentation using RandAugment), Policy 3 (augmentation using GAN), Policy 4 (combine GAN generated samples with non-augmented samples), Policy 5 (apply RandAugment to GAN generated samples).

# 2. Literature Review

Neural Networks are biologically inspired computational models introduced by [19] that combine multiple nonlinear processing layers in an acyclic graph that consists of an input layer, zero or more hidden layers, and an output layer.

Deep neural networks are excellent at discovering intricate structures in high-dimensional data using deep learning techniques [10], [20]; these techniques automate feature detection in deep models, unlike in traditional works where domain experts must painstakingly hand-engineered these features. Deep neural networks have made considerable progress in many areas, such as computer vision [3], speech recognition [21], and natural language processing [22].
Advances in deep network architectures, powerful computation, and open access to extensive training datasets have fueled that success [12].

ConvNets are specialized neural networks that extract features from data with grid-like topologies, such as images [20]. They typically comprise three main layers: Convolutional layers, Pooling Layers, and a Fully Connected Layer. Neurons in the convolutional layer are combined in feature maps, where each neuron is connected to local patches in the feature maps of the previous layer through a set of weights called a filter (or kernel). In CNNs, different feature maps utilize different filters to capture distinct patterns in images. This approach takes advantage of the local correlation present in images, making it more effective for the network to detect a variety of features by employing specialized filters for different aspects of the input data. Since the same pattern can appear anywhere within an image, neurons within the same feature map share the same weight [23], [24], [25]. The latter imbues convolutional networks with some translational invariance [26]. The pooling layer merges semantically similar features into one via a down-sampling operation (typically max-pooling), which adds translational invariance and reduces the number of parameters within the network [20]. The fully connected layer is the output layer, where all neurons connect to all the neurons in the previous layer. Once the input traverses the fully connected layers, the final layer uses the softmax activation function to categorize the input.

Spatial invariance in ConvNets is somewhat satisfied by their ability to exploit the natural translational symmetries in data through weight sharing and pooling [4], [20]. However, researchers like [10] observed that applying an imperceptible (to humans) non-random perturbation to a test image arbitrarily changed the prediction of state-of-the-art CNNs that should be robust to small perturbations in the input.

One way to improve spatial invariance in CNNs, thereby enhancing their generalizability, is data augmentation [9], [10], [27], a powerful technique used to artificially increase the training set size to capture more sources of invariance in the data by applying a transform to an example without changing its label. Formally, let $q(\hat{x}|x)$ be the augmentation transform from which to draw an augmented example $\hat{x}$, based on some original example $x$. For a valid augmentation, it is required that an example $\hat{x} \sim q(\hat{x}|x)$ drawn from the distribution shares the same ground-truth label as $x$ [28]. Data augmentation aims to bake invariances into the dataset, such as rotations, lighting, occlusion, and scale. This way, models trained using that dataset learn these invariances and perform well despite transformations of their input [12].

Traditional transforms [3], such as horizontal flipping, have proven very effective for simple cases but not enough to significantly improve accuracy, reduce overfitting, or minimize adversarial attacks in deep networks [29]. However, using advanced data augmentation techniques such as RandAugment and Generative Adversarial Networks can significantly improve the robustness of CNNs because they provide stronger augmentations, leading to more diverse samples and greater sources of invariance from which to learn [28].

To imbue deep networks with spatial invariance, some early techniques, such as the deep belief network [30] and sparse convolutional DBN [31], achieved good results on simple datasets but were limited in handling more complex datasets. Other models like the Neocognitron [26], HMAX model [32], and Convolutional Network [2] achieved invariance through alternating layers of feature detectors and pooling/subsampling. Recent strategies for improving invariance include Dropout [33], Batch normalization [34], and specialized deep network architectures [35], [36], [37]. Quantitative approaches [6] and Adversarial training [27] have also been explored. Bayesian model selection and Gaussian processes [38] were used to select helpful invariances, but they faced
limitations on complex datasets. GANs [39], traditional transformations, and combinations of augmentation techniques [40] have been studied to improve performance.

This research uses Deep Conditional GANs and RandAugment to enhance spatial invariance in Convolutional Networks. This approach avoids manual feature engineering, geometric transforms, and inflexible modifications of network architecture, providing invariances within the data.

## 3. Methodology

The experiments used three benchmark datasets: MNIST, FMNIST, and CIFAR-10. Three synthesized datasets were generated using a Conditional GAN. RandAugment, ResNet50, and InceptionV3 were imported via Keras API, while the Conditional GAN and NoelNet (a 9 convolution layer network) were built using Keras Sequential API. A personal computer with a quad-core CPU, 8GB GTX 1080 GPU, 32GB RAM, and 512GB SSD storage was used for each experiment with Python 3.9, TensorFlow, and Keras software components.

### 3.1 Datasets

Table 1 shows the 6 labeled datasets used, each comprising 10 distinct categories: MNIST (28x28 grayscale images of handwritten digits), FashionMNIST (28x28 grayscale images of fashion items), CIFAR-10 (32x32 color images), and their CGAN generated equivalents (MNIST_GAN, FMNIST_GAN, CIFAR10_GAN). Additionally, the table includes the total sample size of each dataset and their train/validation/test splits.

<table><caption>**Table 1**

*Description of datasets employed by each policy*</caption><thead><tr><td>dataset</td><td>sample size</td><td>train size</td><td>validation size</td><td>test size</td></tr></thead><tbody><tr><td>MNIST</td><td>70,000</td><td>54000</td><td>6000</td><td>10,000</td></tr><tr><td>FMNIST</td><td>70,000</td><td>54000</td><td>6000</td><td>10,000</td></tr><tr><td>CIFAR-10</td><td>60,000</td><td>45000</td><td>5000</td><td>10,000</td></tr><tr><td>MNIST_GAN</td><td>70,000</td><td>54000</td><td>6000</td><td>10,000</td></tr><tr><td>FMNIST_GAN</td><td>70,000</td><td>54000</td><td>6000</td><td>10,000</td></tr><tr><td>CIFAR10_GAN</td><td>60,000</td><td>45000</td><td>5000</td><td>10,000</td></tr></tbody></table>

### 3.2 Data Preparation

Each dataset was partitioned into a training/validation/test set. The validation set was created by splitting the training dataset in a 90:10 ratio (i.e., 90% for training and 10% for validation). The test set retained their default splits (i.e., 10,000 testing samples per dataset). Each benchmark dataset is size-normalized and centered by default, and each sample was converted to float32 datatype, normalized in the range [0,1], and resized to the minimum input volume required by ResNet50 and Inception: 32x32x3 and 75x75x3, respectively. Class labels of each dataset were one-hot encoded.
## 3.3 Convolutional Networks

ResNet50 and InceptionV3 used their default hyperparameters except for having a 10-neuron output layer instead of 1000. All models used the Adam optimizer with 1e-3 learning rate and categorical cross-entropy loss function. The networks were trained for 100 epochs using 512 batch sizes. Several callback functions, such as early stopping if the validation loss had not improved after 40 epochs, annealing of the learning rate by a factor between 2 and 10 [41] if there is no improvement in validation loss after 10 epochs, and the model with the best-performing weights (measured by the minimum validation loss) was saved to file.

## 3.4 Data Augmentation

Augmentation was implemented using RandAugment, and a Deep Conditional GAN was developed for this work.

### 3.4.1 RandAugment

Random augmentation (RandAugment) is a data augmentation technique that introduces random augmentations into the training process of neural networks. Its primary aim is to improve model generalization by applying diverse data transformations during training, allowing the model to better handle variations in the input data. These random augmentations also function as a regularization technique, helping to reduce overfitting, and are applicable across tasks such as image classification, object detection, and segmentation. Additionally, RandAugment provides significant flexibility through the choice of the number of augmentations to apply and the strength of each augmentation, allowing practitioners to fine-tune the level of augmentation based on the specific characteristics of their dataset or task.

While RandAugment is versatile, the choice of augmentation policies might still require task-specific tuning since not all augmentations may be beneficial for every task or dataset. Furthermore, RandAugment increases the computational cost of training due to the random application of many augmentations, its performance can be sensitive to the choice of hyperparameters such as the number of augmentations and their magnitudes.

RandAugment requires two parameters: N (number of random transforms to apply sequentially) and M (strength of each transform). By specifying these parameters, RandAugment generates a sequence of N transformations, each applied with a certain magnitude M, and then applies this sequence to augment the training data. For example, a transformation can be [(Rotate, 0.6, 2), (Translate, 0.3, 9)]. For each operation, we uniformly sampled a transform from 14 possible transformations: *identity*, *autoContrast*, *equalize*, *rotate*, *solarize*, *color*, *posterize*, *contrast*, *brightness*, *sharpness*, *shear-x*, *shear-y*, *translate-x*, *translate-y*), with a magnitude in [1, 10], and fixed the probability to 0.5. We followed the approach [13] suggested and set the hyperparameter N to equal 2 transforms with Magnitude 9, i.e. (N = 2 and M = 9).

### 3.4.2 Conditional GAN

Generative Adversarial Networks (GANs) are classes of neural networks consisting of two models, a generator (G) and a discriminator (D), trained simultaneously in a minimax game. The generator aims to produce synthetic samples mimicking the original set, while the discriminator tries to distinguish between real and generated samples. GANs have been extended
to Conditional GANs (CGANs) [51], where both G and D are conditioned on an extra input, *the sample label*, providing flexibility to generate samples per specified class.

GAN-based oversampling addresses the challenges of acquiring large datasets in domains like medical image analysis, where data collection is constrained by factors such as patient privacy, disease rarity, and the expertise needed for labeling [52]. However, drawbacks include the potential introduction of synthetic artifacts and biases, as GANs may struggle to capture the complexity of the original data distribution. Additionally, ensuring the quality and relevance of generated samples remains a crucial consideration, as poorly generated data could adversely impact model performance.

The CGAN was trained separately on each benchmark dataset. A total of 1000 epochs was used for training, as this number was observed to produce visually plausible samples to the original [39]. To minimize the effects of overfitting, one-sided label smoothing was implemented as recommended by [16], [42]. Samples after each training epoch and after convergence were printed on the screen to demonstrate that the CGAN was not simply producing samples via memorizing training examples [43]. At the end of the training, the generator model was saved to a file and used to synthesize equivalently sized training sets used by Policies 3, 4, and 5.

**Discriminator.** Implemented as a moderate CNN using GAN best practices such as LeakyReLU activation function with a slope of 0.2, 2x2 stride to downsample, and Adam optimizer with learning rate 0.0002 and momentum 0.9. Each discriminator took as input one 28x28 grayscale image (MNIST, FMNIST) or one 32x32 color image (CIFAR-10) and an integer class label of the image and output a binary classification of whether the image was real (class=1) or fake (class=0). The CIFAR-10 model had three CONV layers with 5x5 kernels, a stride of 2, and same padding. It had 64 kernels in CONV1, 128 in CONV2, and 256 in CONV3. LeakyReLU followed each CONV layer. Similarly, Batch Normalization followed each CONV layer except for the first layer. Finally, a dropout of 40% was used before the one-unit output layer. The F/MNIST model had the same setup, except two CONV layers were used, both with 128 filters.

**Generator.** Implemented as a moderate CNN and took as input a point in latent space ($z = 100$) and the number of classes and output a 28x28 (MNIST, FMNIST) or 32x32 (CIFAR-10) image.

* The generator takes, as input, a point in the latent space with a dimensionality of 100.

* The fully connected layer interprets this point in the latent space and produces a tensor with dimensions of 7 x 7 x 128 for (F/MNIST) or 4 x 4 x 256 for CIFAR-10.

* These activations are then reshaped into many copies of a low-resolution output image. For F/MNIST, this results in 128 copies, and for CIFAR-10, it results in 256 copies.

Following this reshaping step, the generator uses transpose convolutional layers for up-sampling in the case of CIFAR-10:

* Three sequential transpose convolutional layers are employed.

* The first layer has 256 filters with a size of 5x5, a stride of 2, and the same padding.

* The second layer has 128 filters with a size of 5x5, a stride of 2, and the same padding.

* The third layer has 64 filters with a size of 5x5, a stride of 2, and the same padding.

Each of these convolutional layers is followed by LeakyReLU activation with a slope of 0.2 and Batch Normalization with a momentum of 0.9. Finally, the output layer consists of a convolutional layer with a single 5x5 kernel, a tanh activation function, and same-padding. The output of this layer produces the generated image.
**Embedding Layer.** As a best practice, an embedding layer was used to encode class labels into the Discriminator and Generator models. A fully connected layer followed this layer with linear activation that scales the embedding to the image size before concatenating it in the model as an additional feature map [43].

**3.5 Data Input Pipeline**

Each policy was implemented over nine stages using a data input pipeline from TensorFlow's data API. **Stage 1:** Load the dataset. **Stage 2:** Partition the dataset into train/validation/test splits. **Stage 3:** Preprocess dataset samples. **Stage 4:** Configure the dataset using batching, prefetching, shuffling, and augmenting. **Stage 5:** Import and configure CNN. **Stage 6:** Train CNN using the training dataset. **Stage 7:** Evaluate CNN performance on training/validation data. **Stage 8:** Evaluate CNN on the testing dataset. **Stage 9:** Compute and save performance metrics to file.

**3.6 Evaluation Metrics**

The following metrics were used to analyze the performance robustness of ResNet50, InceptionV3, and NoelNet across the five augmentation policies: *Training/Validation Loss, Training/Validation Accuracy, Test Accuracy, Precision, Recall, Training Latency, and F1 Score.* Consistent with previous studies [8], [44], [45], we used testing accuracy to measure spatial invariance in ConvNets. Since the problems in this study are multi-class, each class was treated as a positive class or class of interest. Precision and recall was then computed for each individual class then averaged across all classes to provide an overall assessment of the model's performance.

**3.7 Experiments**

The following describes each of the five policies evaluated in this study.

**Policy 1** was the baseline for each experiment. It established a reference for loss, accuracy, precision, recall, f1-score, and training latency for all experiments and did not use data augmentation. The training, validation, and testing datasets generated in Stage 2 of the pipeline were saved as TensorFlow datasets to be used by the following policies.

**Policy 2** used RandAugment to generate transformation policies stochastically. First, it set the "augment" flag in Stage 4 of the pipeline to "true" to activate the RandAugment function within that stage. Once activated, the input pipeline generated augmented samples fed into the CNN.

**Policy 3** used a Conditional GAN to synthesize additional training examples for a given dataset. The CGAN was first trained using the pipeline's training dataset generated in Stage 2. Once trained, its generator was used to synthesize labeled training examples equal to the number of samples in the training dataset. Finally, these synthesized samples were saved as a TensorFlow dataset, used as the training dataset for Stage 3 of the pipeline, and later used in Policies 4 and 5.

**Policy 4** combined Policy 3 and Policy 1. This policy concatenated the GAN-generated training dataset from Policy 3 with the training dataset from Policy 1. The resulting dataset was then used as the pipeline's training dataset for Stage 3.
**Policy 5** combined Policy 3 and Policy 2. This policy stacked different augmentation styles. It applied RandAugment to the GAN-generated training dataset from Policy 3. The GAN-generated training dataset was used as the training dataset for Stage 3 of the pipeline, and the augment flag in Stage 4 was set to "true" to activate the RandAugment function.

## 4. Results

A total of 45 experiments were conducted (3 benchmark datasets * 3 ConvNets * 5 Policies = 45). The results of the experiments were used for analyzing improvements in spatial invariance in ConvNets and for hyperparameter optimization during training of the conditional DCGAN, ResNet50, and InceptionV3. Parameter tuning was required to avoid gradient explosion and mode collapse [35].

### 4.1 Deep Conditional GAN

The GAN model was trained 30 times on CIFAR-10 and 10 times on F/MNIST for 200 epochs to select good hyperparameters, which were then used to train the final GAN model for 1000 epochs. The generated images improved in quality from epoch 1 to epoch 1000, and stabilization was achieved after approximately 500 epochs of training. This progression of image quality, seen in Figure 1, was significant because it demonstrated that the GAN was not simply memorizing the images but learning their salient features to synthesize additional samples. The final hyperparameters used to train the CGAN are shown in Table 2.

<caption>**Table 2**</caption>

Final hyperparameters used to train the CGAN model

<table><thead><tr><td>hyperparameter</td><td>value</td></tr></thead><tbody><tr><td>epochs</td><td>100</td></tr><tr><td>batch size</td><td>128</td></tr><tr><td>classes</td><td>10</td></tr><tr><td>latent dim.</td><td>100</td></tr><tr><td>optimizer</td><td>Adam (lr = 0.0002, beta1 = 0.5, beta2 = 0.9)</td></tr><tr><td>momentum</td><td>0.9</td></tr><tr><td>kernel</td><td>5x5</td></tr><tr><td>activation</td><td>LeakyReLU (alpha = 0.2)</td></tr><tr><td>loss function</td><td>Binary Cross Entropy</td></tr><tr><td>channels</td><td>1:F/MNIST, 3:CIFAR-10</td></tr></tbody></table>

The generated images for F/MNIST were of higher quality than those for CIFAR-10 due to the simpler features of the former. Training the CGAN took approximately 29.21 hours, i.e., MNIST took 9.02 hours, FMNIST took 8.66 hours, and CIFAR-10 took 11.53 hours.
<caption>**Figure 1**</caption>

Sample Images at Various Epochs Generated Using the Conditional DCGAN

<figure><img src="image_3.png" /></figure>

## 4.2 NoELNet

The following results are obtained for NoELNet, a 9-convolution neural network.

### 4.2.1 MNIST on NoELNet

Table 3 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having a slightly higher latency than Policy 1. Policy 3, which only uses the GAN-generated samples, performed worst.

<caption>**Table 3**</caption>

Performance Test Metrics of each Policy for MNIST on NoELNet

<table><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>mnist</td><td>noelnet</td><td>1</td><td>0.9943</td><td>0.9944</td><td>0.9942</td><td>0.9943</td><td>350</td><td>0</td></tr><tr><td>mnist</td><td>noelnet</td><td>2</td><td>0.9961</td><td>0.9961</td><td>0.9961</td><td>0.9961</td><td>948</td><td>0.18</td></tr><tr><td>mnist</td><td>noelnet</td><td>3</td><td>0.9894</td><td>0.9894</td><td>0.9894</td><td>0.9894</td><td>558</td><td>-0.49</td></tr><tr><td>mnist</td><td>noelnet</td><td>4</td><td>0.9944</td><td>0.9944</td><td>0.9943</td><td>0.9944</td><td>955</td><td>0.01</td></tr><tr><td>mnist</td><td>noelnet</td><td>5</td><td>0.9913</td><td>0.9915</td><td>0.9913</td><td>0.9914</td><td>935</td><td>-0.3</td></tr></tbody></table>

*Note.* The table displays the performance metrics for the test samples, where the delta column shows the percentage change in accuracy for each policy relative to Policy 1. Higher percentages imply improved performance and generalizability.

### 4.2.2 FMNIST on NoELNet

Table 4 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having the highest latency. Policy 3, which only uses the GAN-generated samples, performed worst but had the fastest latency.
<caption>**Table 4**</caption>

*Performance Test Metrics of each Policy for FMNIST on NoelNet*

<table><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>fmnist</td><td>noelnet</td><td>1</td><td>0.9288</td><td>0.9292</td><td>0.9288</td><td>0.9287</td><td>323</td><td>0</td></tr><tr><td>fmnist</td><td>noelnet</td><td>2</td><td>0.9424</td><td>0.9422</td><td>0.9424</td><td>0.9421</td><td>979</td><td>1.46</td></tr><tr><td>fmnist</td><td>noelnet</td><td>3</td><td>0.8634</td><td>0.8635</td><td>0.8634</td><td>0.8617</td><td>252</td><td>-7.04</td></tr><tr><td>fmnist</td><td>noelnet</td><td>4</td><td>0.9307</td><td>0.9303</td><td>0.9307</td><td>0.9304</td><td>587</td><td>0.2</td></tr><tr><td>fmnist</td><td>noelnet</td><td>5</td><td>0.8724</td><td>0.8719</td><td>0.8724</td><td>0.871</td><td>570</td><td>-6.07</td></tr></tbody></table>

**4.2.3 CIFAR-10 on NoelNet**

Table 5 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having a higher latency than Policy 1. Policy 3, which only uses the GAN-generated samples, performed worst but had the fastest latency.

<caption>**Table 5**</caption>

*Performance Test Metrics of each Policy for CIFAR-10 on NoelNet*

<table><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>cifar10</td><td>noelnet</td><td>1</td><td>0.8516</td><td>0.8517</td><td>0.8516</td><td>0.8509</td><td>647</td><td>0</td></tr><tr><td>cifar10</td><td>noelnet</td><td>2</td><td>0.8895</td><td>0.8913</td><td>0.8895</td><td>0.8901</td><td>1077</td><td>4.45</td></tr><tr><td>cifar10</td><td>noelnet</td><td>3</td><td>0.5783</td><td>0.5992</td><td>0.5783</td><td>0.5814</td><td>427</td><td>-32.09</td></tr><tr><td>cifar10</td><td>noelnet</td><td>4</td><td>0.8364</td><td>0.837</td><td>0.8364</td><td>0.8364</td><td>1188</td><td>-1.78</td></tr><tr><td>cifar10</td><td>noelnet</td><td>5</td><td>0.6859</td><td>0.6963</td><td>0.6859</td><td>0.6892</td><td>711</td><td>-19.46</td></tr></tbody></table>

**4.2.4 MNIST on ResNet-50**

Table 6 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having a higher latency than Policy 1. Policy 3, which only uses the GAN-generated samples, performed worst but had the fastest latency.

<caption>**Table 6**</caption>

*Performance Test Metrics of each Policy for MNIST on ResNet-50*

<table><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>mnist</td><td>resnet</td><td>1</td><td>0.9914</td><td>0.9916</td><td>0.9913</td><td>0.9914</td><td>914</td><td>0</td></tr><tr><td>mnist</td><td>resnet</td><td>2</td><td>0.9937</td><td>0.9938</td><td>0.9936</td><td>0.9937</td><td>1392</td><td>0.23</td></tr><tr><td>mnist</td><td>resnet</td><td>3</td><td>0.9874</td><td>0.9873</td><td>0.9874</td><td>0.9873</td><td>852</td><td>-0.40</td></tr><tr><td>mnist</td><td>resnet</td><td>4</td><td>0.9930</td><td>0.9931</td><td>0.9929</td><td>0.9930</td><td>1731</td><td>0.16</td></tr><tr><td>mnist</td><td>resnet</td><td>5</td><td>0.9894</td><td>0.9895</td><td>0.9893</td><td>0.9893</td><td>1425</td><td>-0.20</td></tr></tbody></table>
### 4.2.5 FMNIST on ResNet-50

Table 7 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having a higher latency than Policy 1. Policy 3, which only uses the GAN-generated samples, performed worst but had the fastest latency.

<caption>**Table 7**</caption>

*Performance Test Metrics of each Policy for FMNIST on ResNet-50*

<table><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>fmnist</td><td>resnet</td><td>1</td><td>0.8985</td><td>0.9002</td><td>0.8985</td><td>0.8968</td><td>753</td><td>0</td></tr><tr><td>fmnist</td><td>resnet</td><td>2</td><td>0.9266</td><td>0.9279</td><td>0.9266</td><td>0.9269</td><td>1300</td><td>3.13</td></tr><tr><td>fmnist</td><td>resnet</td><td>3</td><td>0.8267</td><td>0.8289</td><td>0.8267</td><td>0.8206</td><td>691</td><td>-7.99</td></tr><tr><td>fmnist</td><td>resnet</td><td>4</td><td>0.9159</td><td>0.917</td><td>0.9159</td><td>0.9162</td><td>1587</td><td>1.94</td></tr><tr><td>fmnist</td><td>resnet</td><td>5</td><td>0.85</td><td>0.8582</td><td>0.85</td><td>0.8518</td><td>970</td><td>-5.4</td></tr></tbody></table>

### 4.2.6 CIFAR-10 on ResNet-50

Table 8 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having a higher latency than Policy 1. Policy 3, which only uses the GAN-generated samples, performed worst but had the fastest latency.

<caption>**Table 8**</caption>

*Performance Test Metrics of each Policy for CIFAR-10 on ResNet-50*

<table><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>cifar10</td><td>resnet</td><td>1</td><td>0.7339</td><td>0.7476</td><td>0.7339</td><td>0.7358</td><td>617</td><td>0</td></tr><tr><td>cifar10</td><td>resnet</td><td>2</td><td>0.8517</td><td>0.8535</td><td>0.8517</td><td>0.8523</td><td>1416</td><td>16.05</td></tr><tr><td>cifar10</td><td>resnet</td><td>3</td><td>0.6286</td><td>0.6351</td><td>0.6286</td><td>0.6258</td><td>587</td><td>-14.35</td></tr><tr><td>cifar10</td><td>resnet</td><td>4</td><td>0.7671</td><td>0.7759</td><td>0.7671</td><td>0.7683</td><td>1222</td><td>4.52</td></tr><tr><td>cifar10</td><td>resnet</td><td>5</td><td>0.6915</td><td>0.7006</td><td>0.6915</td><td>0.6937</td><td>960</td><td>-5.78</td></tr></tbody></table>

### 4.2.7 MNIST on InceptionV3

Table 9 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having a higher latency than Policy 1. Policy 3, which only uses the GAN-generated samples, performed worst.

<caption>**Table 9**</caption>

*Performance Test Metrics of each Policy for MNIST on InceptionV3*

<table><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>mnist</td><td>inception</td><td>1</td><td>0.9943</td><td>0.9944</td><td>0.9942</td><td>0.9943</td><td>1781</td><td>0</td></tr><tr><td>mnist</td><td>inception</td><td>2</td><td>0.9958</td><td>0.9959</td><td>0.9957</td><td>0.9958</td><td>3201</td><td>0.15</td></tr><tr><td>mnist</td><td>inception</td><td>3</td><td>0.9892</td><td>0.9894</td><td>0.9891</td><td>0.9892</td><td>1852</td><td>-0.51</td></tr><tr><td>mnist</td><td>inception</td><td>4</td><td>0.9956</td><td>0.9956</td><td>0.9955</td><td>0.9956</td><td>3422</td><td>0.13</td></tr><tr><td>mnist</td><td>inception</td><td>5</td><td>0.9911</td><td>0.9911</td><td>0.9911</td><td>0.9911</td><td>2024</td><td>-0.32</td></tr></tbody></table>
### 4.2.8 FMNIST on InceptionV3

Table 10 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having a higher latency than Policy 1. Policy 3, which only uses the GAN-generated samples, performed worst but had the fastest latency.

<table><caption>**Table 10**
Performance Test Metrics of each Policy for FMNIST on InceptionV3</caption><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>fmnist</td><td>inception</td><td>1</td><td>0.9167</td><td>0.9166</td><td>0.9167</td><td>0.9163</td><td>1643</td><td>0</td></tr><tr><td>fmnist</td><td>inception</td><td>2</td><td>0.9482</td><td>0.9484</td><td>0.9482</td><td>0.9482</td><td>2471</td><td>3.44</td></tr><tr><td>fmnist</td><td>inception</td><td>3</td><td>0.8527</td><td>0.8595</td><td>0.8527</td><td>0.8543</td><td>1543</td><td>-6.98</td></tr><tr><td>fmnist</td><td>inception</td><td>4</td><td>0.9353</td><td>0.9364</td><td>0.9353</td><td>0.935</td><td>3217</td><td>2.03</td></tr><tr><td>fmnist</td><td>inception</td><td>5</td><td>0.8825</td><td>0.8831</td><td>0.8825</td><td>0.8821</td><td>1843</td><td>-3.73</td></tr></tbody></table>

### 4.2.9 CIFAR-10 on InceptionV3

Table 11 shows that Policy 2, which only uses RandAugment, performed best by achieving the highest accuracy and f1 score at the expense of having the highest latency. Policy 3, which only uses the GAN-generated samples, performed worst.

<table><caption>**Table 11**
Performance Test Metrics of each Policy for CIFAR-10 on InceptionV3</caption><thead><tr><th>dataset</th><th>model</th><th>policy</th><th>accuracy</th><th>precision</th><th>recall</th><th>f1score</th><th>latency(s)</th><th>delta(%)</th></tr></thead><tbody><tr><td>cifar10</td><td>inception</td><td>1</td><td>0.8196</td><td>0.8229</td><td>0.8196</td><td>0.8192</td><td>1288</td><td>0</td></tr><tr><td>cifar10</td><td>inception</td><td>2</td><td>0.9134</td><td>0.9142</td><td>0.9134</td><td>0.9135</td><td>2913</td><td>11.44</td></tr><tr><td>cifar10</td><td>inception</td><td>3</td><td>0.6483</td><td>0.668</td><td>0.6483</td><td>0.6441</td><td>1371</td><td>-20.9</td></tr><tr><td>cifar10</td><td>inception</td><td>4</td><td>0.8292</td><td>0.8389</td><td>0.8292</td><td>0.829</td><td>2541</td><td>1.17</td></tr><tr><td>cifar10</td><td>inception</td><td>5</td><td>0.6954</td><td>0.7068</td><td>0.6954</td><td>0.6981</td><td>1429</td><td>-15.15</td></tr></tbody></table>

### 4.2.10 Training Latency

Figure 2 shows that InceptionV3 took the most time to train, at 9.0383 hours, while NoeINet took the least at 2.9183 hours. ResNet-50 took 4.560 hours and was twice as fast to train as InceptionV3. Taking the longest time to train has led to InceptionV3 achieving the highest accuracies across all datasets used by the policies, as shown in Figure 3.
<figure><img src="image_4.png" /><figcaption>**Figure 2**

*Comparing Dataset Training Times Across Different Models*</figcaption></figure>

## 4.3 Highest Accuracy

Figure 3 shows the accuracies achieved by Policy 2, which consistently produced the highest accuracy and f1 score across all models and datasets. The chart shows that InceptionV3 resulted in the highest accuracy across the three datasets, closely followed by NoelNet and then by ResNet-50. Policy 2 achieving the highest accuracies and f1 scores relative to all Policies means that it has consistently improved the spatial invariance of each model. Notably, NoelNet, which is a basic CNN (~1 million parameters), consistently achieved higher accuracies than ResNet-50 (~23.6 million parameters) and competitive accuracies with InceptionV3 (~21.8 million parameters).

<figure><img src="image_5.png" /><figcaption>**Figure 3**

*Highest Dataset Accuracy Across Different Models*</figcaption></figure>
# 5. Discussion

## Does data augmentation lead to more spatially invariant and robust networks?

Of the five Policies used in this research, four Policies: 2, 3, 4, and 5 implemented a form of data augmentation. Policies leading to test accuracies and f1 scores higher than those achieved by Policy 1 (baseline) are considered to have increased the spatial invariance of the network. First, Policies 2 and 4 have consistently achieved higher accuracy and f1 scores than Policy 1 on a basic model, NoelNet, as seen in Tables 3, 4, and 5. Similar patterns were observed on ResNet-50 (Tables 6, 7, and 8) and InceptionV3 (Tables 9, 10, and 11) for all three datasets. For instance, from the results of CIFAR-10 on ResNet-50 shown in Table 8, Policy 2 achieved a 16.05% improvement over the baseline, while Policy 4 achieved a 4.52% improvement over the baseline. The results from Policies 2 and 4 on each model indicate that advanced data augmentation does lead to more spatially invariant networks.

Secondly, Figure 3 shows that NoelNet has consistently achieved higher accuracies than ResNet-50 and competitive accuracies with InceptionV3 across all datasets, even though it is a basic CNN (~1 million parameters) without optimizations. ResNet-50 has over 23 million parameters, InceptionV3 has over 21 million, and both are state-of-the-art ConvNets with novel and innovative architecture. NoelNet being competitive with the more advanced models occurs because augmentation imbues invariances within the training data, which models with simple or sophisticated architectures can then learn. This shows that advanced augmentation techniques are very effective in improving the robustness and spatial invariance of ConvNets irrespective of the complexity of the underlying model.

## Do synthesized samples improve invariance?

Tables 3 through 11 show that Policy 3 consistently had the lowest accuracies of each policy and never resulted in accuracies, precisions, or recalls higher than the baseline. This implies that the synthesized samples led to significant overfitting, thus not improving the invariance of the models. Independently, the GAN samples did not improve the model's spatial invariance. However, combined with the unperturbed samples (Policy 4), it improved the spatial invariance of the models because it created a more diverse dataset that led to accuracy and f1 scores higher than the baseline.

## Does the number of parameters in a CNN affect its spatial invariance?

InceptionV3 has 21,823,274 parameters, NoelNet has 1,091,594, and ResNet-50 has 23,585,290. From Figure 3, the maximum accuracy on InceptionV3 was 99.58%; on NoelNet, it was 99.61%; and on ResNet-50, it was 99.37%. These accuracies were achieved using MNIST, but similar patterns were seen when using FMNIST or CIFAR-10. Interestingly, InceptionV3 and NoelNet had higher accuracy and f1 scores for all three datasets than ResNet-50, even though ResNet-50 has the highest number of parameters. Furthermore, NoelNet, which has the lowest number of parameters, i.e., ~23 times smaller than ResNet-50, and ~21 times smaller than InceptionV3, achieved very competitive scores with InceptionV3. The results show that having more
parameters does not necessarily indicate better predictive abilities or increased spatial invariances, as intuition may suggest.

## 6. Conclusion

RandAugment is an effective data augmentation technique for improving spatial invariance and reducing overfitting in ConvNets. The study also found that combining synthesized and unaugmented samples led to increased spatial invariance of the resulting model. Additionally, wider models such as InceptionV3 were found to be more spatially invariant than deeper models such as ResNet-50 and NoelNet, although deeper models were faster to train.

As part of future research to improve spatial invariance, we could use GANs with more versatile loss functions such as Wasserstein GAN [46], generative techniques like Neural Style Transfer [47], augmentation strategies like Cutout [48], CutMix [49], and GridMask [8], all applied to larger datasets like ImageNet [50].

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data Availability

All datasets used for experimentation are public benchmarks.
## References

[1] LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., & Jackel, L. (1989). Handwritten Digit Recognition with a Back-Propagation Network. In Advances in Neural Information Processing Systems. Morgan-Kaufmann.

[2] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. *Proceedings of the IEEE*, 86(11), 2278-2324.

[3] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. *Neural Information Processing Systems*. 25.

[4] Jaderberg, M., Simonyan, K., Zisserman, A., & Kavukcuoglu, K. (2016). Spatial Transformer Networks. arXiv. https://doi.org/10.48550/arXiv.1506.02025

[5] Lenc, K., & Vedaldi, A. (2015). Understanding image representations by measuring their equivariance and equivalence. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 991-999.

[6] Goodfellow, I., Lee, H., Le, Q., Saxe, A., & Ng, A. (2009). Measuring invariances in deep networks. *Advances in neural information processing systems*, p. 22.

[7] Azulay, A., & Weiss, Y. (2018). Why do deep convolutional networks generalize so poorly to small image transformations?. arXiv. https://doi.org/10.48550/arXiv.1805.12177

[8] Chen, S., Dobriban, E., & Lee, J. H. (2020). A group-theoretic framework for data augmentation. *The Journal of Machine Learning Research*, 21(1), 9885-9955.

[9] Engstrom, L., Tran, B., Tsipras, D., Schmidt, L., & Madry, A. (2018). A rotation and a translation suffice: Fooling cnns with simple transformations.

[10] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014). Going deeper with convolutions. arXiv.org. Retrieved January 19, 2023, from https://arxiv.org/abs/1409.4842

[11] Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? *Advances in neural information processing systems*, p. 30.

[12] Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1), 60. doi:10.1186/s40537-019-0197-0

[13] Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020). RandAugment: Practical automated data augmentation with a reduced search space. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops* (pp. 702-703).

[14] Goodfellow, I. J., Mirza, M., Xu, B., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative Adversarial Networks. arXiv. https://doi.org/10.48550/arXiv.1406.2661

[15] He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv. https://doi.org/10.48550/arXiv.1512.03385

[16] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2015). Rethinking the inception architecture for computer vision. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 2818-2826).

[17] Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747.

[18] Krizhevsky, A., & Hinton, G. (2009). Learning multiple layers of features from tiny images.
[19] McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. *The bulletin of mathematical biophysics*, 5(4), 115-133.

[20] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.
doi:10.1038/nature14539

[21] Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A. R., Jaitly, N., ... & Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6), 82-97.

[22] Torfi, A., Shirvani, R. A., Keneshloo, Y., Tavaf, N., & Fox, E. A. (2020). Natural language processing advancements by deep learning: A survey. *arXiv preprint arXiv:2003.01200*.

[23] Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning*. Amsterdam University Press.

[24] Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., ... & Chen, T. (2018). Recent advances in convolutional neural networks. *Pattern recognition*, 77, 354-377.

[25] Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. *nature*, 323(6088), 533-536.

[26] Fukushima, K., & Miyake, S. (1982). Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In *Competition and cooperation in neural nets* (pp. 267–285). Springer, Berlin, Heidelberg.

[27] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks. *arXiv preprint arXiv:1706.06083*.

[28] Xie, Q., Dai, Z., Hovy, E., Luong, T., & Le, Q. (2020). Unsupervised data augmentation for consistency training. *Advances in Neural Information Processing Systems*, pp. 33, 6256–6268.

[29] Su, J., Vargas, D. V., & Sakurai, K. (2019). One pixel attack for fooling deep neural networks. *IEEE Transactions on Evolutionary Computation*, 23(5), 828-841.

[30] Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. *Neural computation*, 18(7), 1527-1554.

[31] Ranzato, M. A., Boureau, Y. L., & Cun, Y. (2007). Sparse feature learning for deep belief networks. *Advances in neural information processing systems*, 20.

[32] Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object recognition in cortex. *Nature Neuroscience*, 2(11), 1019-1025.

[33] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. *J. Mach. Learn. Res.*, 15(1), 1929–1958.

[34] Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In *International conference on machine learning* (pp. 448-456). PMLR.

[35] Yu, S., Zhang, K., Xiao, C., Huang, J., Li, M., & Onizuka, M. (2022). HSGAN: Reducing Mode Collapse in GANs by the Latent Code Distance of Homogeneous Samples. Comput. Vis. Image Underst., 214(C).

[36] Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017). Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision (pp. 764-773).

[37] Cohen, T., & Welling, M. (2016). Group equivariant convolutional networks. In *International conference on machine learning* (pp. 2990-2999). PMLR.
[38] Van der Wilk, M., Bauer, M., John, S. T., & Hensman, J. (2018). Learning invariances using the marginal likelihood. *Advances in Neural Information Processing Systems*, 31.

[39] Perez, L., & Wang, J. (2017). The Effectiveness of Data Augmentation in Image Classification using Deep Learning. arXiv. https://doi.org/10.48550/arXiv.1712.04621

[40] Shijie, J., Ping, W., Peiyi, J., & Siping, H. (2017). Research on data augmentation for image classification based on convolution neural networks. In *2017 Chinese automation congress (CAC)* (pp. 4165-4170). IEEE.

[41] Smith, L. N. (2017). Cyclical learning rates for training neural networks. In *2017 IEEE winter conference on applications of computer vision (WACV)* (pp. 464-472). IEEE.

[42] Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016). Improved techniques for training gans. Advances in neural information processing systems, 29.

[43] Denton, E., Chintala, S., Szlam, A., & Fergus, R. (2015). Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks.

[44] Arjovsky, M., Bottou, L., & Gulrajani, I. (2019). Invariant Risk Minimization. arXiv. https://doi.org/10.48550/arXiv.1907.02893

[45] Gowal, S., Qin, C., Huang, P. S., Cemgil, T., Dvijotham, K., Mann, T., & Kohli, P. (2020). Achieving robustness in the wild via adversarial mixing with disentangled representations. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 1211-1220).

[46] Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein generative adversarial networks. In *International conference on machine learning* (pp. 214-223). PMLR.

[47] Gatys, L. A., Ecker, A. S., & Bethge, M. (2015). A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576.

[48] DeVries, T., & Taylor, G. W. (2017). Improved regularization of convolutional neural networks with Cutout. arXiv preprint arXiv:1708.04552.

[49] Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. In *Proceedings of the IEEE/CVF international conference on computer vision* (pp. 6023-6032).

[50] Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). ImageNet: A largescale hierarchical image database. In *2009 IEEE conference on computer vision and pattern recognition* (pp. 248-255). IEEE.

[51] Mirza, M., & Osindero, S. (2014). Conditional Generative Adversarial Nets. arXiv. https://doi.org/10.48550/arXiv.1411.1784

[52] Frid-Adar, M., Diamant, I., Klang, E., Amitai, M., Goldberger, J., & Greenspan, H. (2018). GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing, 321, 321-331.