# FreMix: Frequency-Based Mixup for Data Augmentation

Authors: Yang Xiu; Xinyi Zheng; Linlin Sun; Zhuohao Fang
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Deep learning models have attracted tremendous attention in computer vision in recent years, while most of them heavily rely on massive data for training. As one of the solutions to the sparse data problem, data augmentation techniques, such as image translation and rotation, can substantially increase the model’s generalization ability and performance. However, on one hand, these approaches primarily work under the pixel domain, which is limited to fully mining and fusing picture data from the frequency viewpoint. On the other hand, the fusion weighting factors are primarily modified in a manual fashion, which increases the application costs in practice. To this end, we propose a novel method termed as frequency-based Mixup (FreMix) that allows images to be fused in the frequency domain and to improve the efficiency of data augmentation by adaptively adjusting the weighting coefficients in this paper. In FreMix, first, a fast Fourier transformation (FFT) is performed on the input image, such that the frequency information rather than raw pixel information can be extracted for further augmentation. Besides, an exploration-exploitation training paradigm is exploited, such that the FreMix can be trained periodically to facilitate learning and avoid manually hyperparameter settings. We conduct comparing experiments on three benchmark datasets including CIFAR, ImageNet, and ILSVRC2015, and the experimental results validate the effectiveness of the proposed method.


---

## Body

## 1. Introduction

Existing deep neural networks rely on a considerable quantity of data with low confidence due to the enormous series of tests and the risk of overfitting [1]. Data augmentation [2] approaches have been proposed since deep learning [3] models require big data. In the field of data augmentation, horizontal/vertical flip [4], rotation [5], scaling, cropping, clipping [6], panning, contrast, color dithering [7], noise, pixel domain fusion, and other techniques are available. However, scaling distortion distorts the image [8], and most of these methods suffer from the low-accuracy problem, which further limits their applications in practice.Existing advances show that automatic enhancement and bit-image blending are the more advanced data augmentation techniques. For example, hybrid-based techniques such as Mixup [9] regulate the neural network enhancing the linearity between training samples. The robustness of the adversarial samples is increased and can stabilize the training process for generating adversarial networks [10]. Mixup uses a convex combination of two pairs of instances and labels to train the network to function linearly in the training instances. This simple learning process gives rise to robustness to adversarial examples and improved calibration capability. Mixup can implicitly control the complexity of the model. However, since the coefficients are manually adjusted, there is a problem that the operation is very inconvenient and there is a cost for experimenting with the parameters. Pixel-based blending affects the nature of the pixel-based image, which is not consistent with the pixel problem [11]. And there is also the obvious problem that using Mixup introduces some very unnatural pseudopixel information [12]. To reduce the effect of pseudopixel information, Cutout [13] or CutMix [14] methods are developed, which differ in the pixel values of the filled areas. Cutout enables the model to focus on the regions (abdomen) that are harder to distinguish from the target, but there is a part of the region without any information, which will affect the training efficiency. Cutout randomly masks the square part of the input image during the training process so the classification algorithm cannot adapt well to the masked data. CutMix uses a region-based enhancement strategy that exploits a binary mask to select the blended regions. Such an operation allows the model to identify two targets from a local view of an image. Also, CutMix allows the learned model to retain a good understanding of the actual data, which can improve the training efficiency and does not have unnatural image blending. However, when CutMix was used to train MobileNetV2 [15] on Tiny imagenet [16], we observed severe performance degradation. Only a tiny section of an image can affect the results in fine-grained image classification tasks. Hence, CutMix blurs the features that are critical for establishing the picture class. This makes the model confusing for classification.To improve the existing sample blending-based strategy, we propose a frequency-based Mixup (FreMix) which fuses with extracted frequencies and adaptive exploration of the fusion coefficients. Previous related methods are mainly based on the pixel domain mining of the images while lacking the frequency information exploration, which thus leads to incomplete exploitation of the raw data. Thus, in the proposed FreMix, we study a frequency-based method which does not directly sum images in the pixel space but adapts to the interaction of different textures and high and low information of images by fusion in the natural domain of frequencies. Besides, unlike the previous way that requires manual adjustment of parameters, we design an online update method of parameters based on exploration-exploitation mechanism from reinforcement learning, which can solve the problem of manual adjustment of parameters.Our research was aimed at better mining image data in the field of data augmentation and providing improved methods to avoid manual adjustment of weighting coefficients and also improving the efficiency and accuracy of mining image data by experimentally exploring new solutions to adjust coefficients online. The approach is based on the fusion of natural domains of frequencies to fit the interaction of different textures and high and low information of images, instead of summing directly in pixel space. The automatic hyperparameter optimization is performed using exploration-exploitation mechanism from reinforcement learning. The results of our method on different datasets and different networks improve the accuracy extensively and effectively, such as ResNet [17], VGG [18], and WRN [19]. Our contribution can be summarized as follows: (1) we developed a novel FreMix method in which the frequency domain information of images is leveraged for better data augmentation. (2) An exploration-exploitation mechanism is leveraged in FreMix, such that the proposed method can avoid manually setting hyperparameters and be more applicable in practice. (3) Experiments on three datasets show that the proposed method achieves superior performance in terms of data augmentation.The remainder of this paper is organized as follows. In Section2, we present related work in the field of data augmentation and hyperparameter tuning methods. In Section 3, our proposed FreMix is presented with a detailed algorithm procedure provided. In Section 4, the proposed FreMix is verified through two real-world datasets. Finally, Section 5 concludes the paper.


## 2. Related Work

### 2.1. Data Augmentation

Data augmentation is a typical machine learning approach that is mainly used to increase the size of the training dataset to make it as diverse as possible so that the trained model can generalize better. Currently, data augmentation mainly includes horizontal/vertical flipping, rotation, scaling, cropping, clipping, panning, contrast, color dithering, and noise. The main existing methods are Mixup, CutMix, and Cutout. Mixup mixes two random samples proportionally, and the proportional distribution of classification results may cause underfitting. Cutout enables the model to focus on the region (abdomen) where the target is difficult to distinguish. Still, there is a part of the region without any information, which will affect the training efficiency. Mixup utilizes all of the pixel data while also introducing some extremely strange pseudopixel data. Mixup uses a convex combination of two pairs of instances and labels to train the network so that the network functions linearly across the training instances. Notably, this simple learning procedure leads to robustness to adversarial examples and improved calibration. AdaMixup [20] diagnoses flow intrusions in Mixup, where a mixed model collides with another example in the data flow, which can cause underfitting. This risk is regulated, and the loss term penalizes the intrusion by an intrusion discriminator. Manifold Mixup [21] uses two intermediate representations as examples at the kth layer. When k=0 implies the input layer, it reduces to vanilla Mixup. CutMix uses a region-based augmentation method that selects the mixing zone using a binary mask for better performance in spatial situations. Note that our approach is related to these methods, but clearly, since we do not use real labels for Mixup supervision. Besides, our proposed FreMix in this paper does not directly sum in pixel space but fits the different textures of images and the interaction of high and low information through the fusion of natural domains of frequencies. It achieves very good performance on several datasets and different models such as CIFAR [22] and ImageNet [23] and greatly exceeds the previous methods.


### 2.2. Hyperparameter Optimization Method

The setting of hyperparameters has a direct impact on model performance and its importance cannot be overstated. To maximize model performance, it is critical to understand how to optimize hyperparameters. Several standard hyperparameter optimization methods are described. Most of the time, engineers depend on trial-and-error methods which could tune hyperparameters for optimization. On another side, this method is time-consuming and needs a lot of experience. Therefore, many automated hyperparameter optimization methods have been developed. Grid search [24] is arguably the most basic hyperparameter optimization method. Using this technique, we simply construct separate models for all hyperparameters possible, evaluate the performance of each model, and select the model and hyperparameters that yield the best results. This method enables training for individual hyperparameter combination models and evaluating the performance of each. Each model is independent, so it is easy to perform parallel computation. However, the fact that each model is independent also results in no guidance between models, and the computational results of the former model do not influence the choice of hyperparameters for the latter model.In contrast, Bayesian optimization methods [25] can draw on existing results to influence the selection of hyperparameters for subsequent models. Only combinations of hyperparameters that are likely to increase model performance are computed, which reduces the number of calculations required for model training assessment. Bayesian optimization works by establishing a posterior distribution of the function that best describes the function to be optimized. The posterior distribution improves as the number of observations grows, and the algorithm gets more confident about which portions of the parameter space are worth examining and which are not. Gradient-based optimization methods [26] are often used in neural network models, where the gradient of the hyperparameters is mainly calculated and optimized by a gradient descent algorithm. The idea of evolutionary optimization methods [27] comes from biological concepts, and since natural evolution is a dynamic process occurring in a constantly changing environment, it applies to the hyperparameter search problem, since hyperparameter search is also a dynamic process.


## 2.1. Data Augmentation

Data augmentation is a typical machine learning approach that is mainly used to increase the size of the training dataset to make it as diverse as possible so that the trained model can generalize better. Currently, data augmentation mainly includes horizontal/vertical flipping, rotation, scaling, cropping, clipping, panning, contrast, color dithering, and noise. The main existing methods are Mixup, CutMix, and Cutout. Mixup mixes two random samples proportionally, and the proportional distribution of classification results may cause underfitting. Cutout enables the model to focus on the region (abdomen) where the target is difficult to distinguish. Still, there is a part of the region without any information, which will affect the training efficiency. Mixup utilizes all of the pixel data while also introducing some extremely strange pseudopixel data. Mixup uses a convex combination of two pairs of instances and labels to train the network so that the network functions linearly across the training instances. Notably, this simple learning procedure leads to robustness to adversarial examples and improved calibration. AdaMixup [20] diagnoses flow intrusions in Mixup, where a mixed model collides with another example in the data flow, which can cause underfitting. This risk is regulated, and the loss term penalizes the intrusion by an intrusion discriminator. Manifold Mixup [21] uses two intermediate representations as examples at the kth layer. When k=0 implies the input layer, it reduces to vanilla Mixup. CutMix uses a region-based augmentation method that selects the mixing zone using a binary mask for better performance in spatial situations. Note that our approach is related to these methods, but clearly, since we do not use real labels for Mixup supervision. Besides, our proposed FreMix in this paper does not directly sum in pixel space but fits the different textures of images and the interaction of high and low information through the fusion of natural domains of frequencies. It achieves very good performance on several datasets and different models such as CIFAR [22] and ImageNet [23] and greatly exceeds the previous methods.


## 2.2. Hyperparameter Optimization Method

The setting of hyperparameters has a direct impact on model performance and its importance cannot be overstated. To maximize model performance, it is critical to understand how to optimize hyperparameters. Several standard hyperparameter optimization methods are described. Most of the time, engineers depend on trial-and-error methods which could tune hyperparameters for optimization. On another side, this method is time-consuming and needs a lot of experience. Therefore, many automated hyperparameter optimization methods have been developed. Grid search [24] is arguably the most basic hyperparameter optimization method. Using this technique, we simply construct separate models for all hyperparameters possible, evaluate the performance of each model, and select the model and hyperparameters that yield the best results. This method enables training for individual hyperparameter combination models and evaluating the performance of each. Each model is independent, so it is easy to perform parallel computation. However, the fact that each model is independent also results in no guidance between models, and the computational results of the former model do not influence the choice of hyperparameters for the latter model.In contrast, Bayesian optimization methods [25] can draw on existing results to influence the selection of hyperparameters for subsequent models. Only combinations of hyperparameters that are likely to increase model performance are computed, which reduces the number of calculations required for model training assessment. Bayesian optimization works by establishing a posterior distribution of the function that best describes the function to be optimized. The posterior distribution improves as the number of observations grows, and the algorithm gets more confident about which portions of the parameter space are worth examining and which are not. Gradient-based optimization methods [26] are often used in neural network models, where the gradient of the hyperparameters is mainly calculated and optimized by a gradient descent algorithm. The idea of evolutionary optimization methods [27] comes from biological concepts, and since natural evolution is a dynamic process occurring in a constantly changing environment, it applies to the hyperparameter search problem, since hyperparameter search is also a dynamic process.


## 3. Method

### 3.1. Frequency-Based Data Mix

Convolutional neural networks (CNNs) have shown excellent performance in computer vision tasks, particularly in classification tasks. To increase robustness in real-world scenarios, CNNs often adopt two practical strategies: data augmentation and model integration. Data augmentation reduces overfitting and improves the generalization of the model. Traditional image enhancement is label preserving, e.g., flipping and cropping. Multiple inputs and their labels are proportionally mixed to create artificial samples in mixed-sample data augmentation (MSDA) approaches [28]. MSDA is simple to implement and really helps to improve performance, so it is widely used in areas such as image recognition, sound recognition, GAN, and semisupervised learning.Recently, hybrid-based enhancement techniques have achieved highly accurate prediction performance. They generate input data by a linear combination of two randomly selected training data. Likewise, their corresponding labels are generated by the same linear combination of two labels. By doing so, they effectively improve prediction accuracy while preventing some undesirable behaviors such as memory and sensitivity to adversarial examples. Moreover, Mixup training encourages the output of the DNN [29], i.e., the estimated label distribution, as a better indicator of the actual likelihood of correcting the predictions. Specifically, for generating an enhanced sample, the calculation for Mixup training is as follows
(1)xmix=λx1+1−λx2,M⊙x1+1−M⊙x2,ymix=λy1+1−λy2,λ∼Betaα,α,where x and y denote a training sample and its label, M∈0,1W×H denotes a binary mask indicating the position of the reject and fill from the two images, ⨀ is an element multiplication, Beta∙,∙ implies the beta distribution, and α∈0,∞ is the parameter controlling the shape of the beta distribution. Using mixed inputs and mixed labels, the model minimizes the following equation.
(2)Lmix=λHy~mix,y1+1−λHy~mix,y2,where y~=σfx denotes the predicted label distribution from the model, f is the model, σ is the activation function, usually a softmax function, σsmz=expz/∑i=1Nexpzi, and H is the cross-entropy function formulated by Hp,q=−∫xpxlogqx.Because the Mixup’s raw picture exploration is restricted, the suggested FreMix performs a fast Fourier transform (FFT) on the input image first, extracting the frequency information rather than the original pixel information for subsequent augmentation. Figure1 depicts the flow of our technique. The fast Fourier transform (FFT) is a technique for computing a sequence’s discrete Fourier transform (DFT) or inverse transform fast [30]. Fourier analysis converts a signal’s original domain (typically time or space) into a frequency domain representation or vice versa. By reducing the DFT matrix into the product of sparse (mainly zero) elements, the FFT can quickly perform such a transform.Figure 1
Schematic diagram of our approach.The DFT requires the computation of approximatelyN2 multiplications and N2 additions. This computation is large when N is large. The N-point DFT is decomposed into two N/2-point DFTs using the symmetry and periodicity of WN; therefore, the total computation of two N/2-point DFTs is just half of the original, i.e., N/22+N/22=N2/2. This can be continued by decomposing N/2 into N/4-point DFTs again, etc. For N=2 m, points of DFT can be decomposed into 2 points of DFT, so that its computation can be reduced to N/2log2N times multiplication and Nlog2N times addition. Here are the steps of the operation.The DFT of a finite length discrete signalxn,n=0,1,⋯,N−1 is defined as
(3)Xk=∑n=0N−1xnWNknk=0,1,⋯,N−1,WN=e−j2πN.Decomposingxn into the sum of two sequences of even and odd numbers, i.e.,
(4)xn=x1n+x2n,where x1n and x2n are both of length N/2, x1n is an even sequence, and x2n is an odd sequence, then
(5)Xk=∑n=0N/2−1x1nWN2kn+∑n=0N/2x2nWN2n+1kk=0,1,⋯,N−1.BecauseWN2kn=e−j2π/N2kn=e−j2π/N/2kn=WN/2kn, then
(6)Xk=∑n=0N/2−1x1nWN/2kn+WNk∑n=0N/2−1x2nWN/2kn=X1k+WNkX2kk=0,1,⋯,N−1,where X1k and X2k are the N/2-point DFTs of x1n and x2n, respectively. Since both X1k and X2k have period N/2 and WNk+N/2=−WNk, Xk can again be expressed as
(7)Xk=X1k+WNkX2kk=0,1,⋯,N2−1,Xk+N2=X1k−WNkX2kk=0,1,⋯,N/2−1.The principle of the FFT algorithm is to achieve large-scale transformations by many small more easily performed transformations, reducing the operational requirements and increasing the speed with the operation. The FFT is not an approximation of the DFT, they are exactly equivalent. Figure2 shows the flow chart of 8-point FFT decomposition.Figure 2
Illustration of an 8-point FFT decomposition procedure.


### 3.2. Adaptive Fusion Algorithm

FreMix creates new samples by linearly interpolating pairs of FFT information of raw examples, and it is simple to do and to be effective in picture classification problems. There are two issues. First, FreMix asks for more epochs to converge. The reason is that it needs more extended training and explores more regions of the data space. Second, FreMix has a condition of hyperparameter value to sample mixing coefficients, while various hyperparameter values usually cause large differences in model accuracy. To mitigate this problem, inspired by Mixup without hesitation (mWh) [31], we integrate the exploration-exploitation mechanism from reinforcement learning into our FreMix to overcome both problems, rather than enriching data by using Mixup during the model training process. FreMix will be speeded up since exploration-exploitation will turn the mixing operation off. And it also makes it robust to the hyperparameter α. Note that mWh is a pixel domain-based method, which has the disadvantage of requiring extraction from the original pixels compared to the frequency domain. We utilize the exploration-exploitation (EE) mechanism to operate in the frequency domain in the proposed FreMix. Our method can extract frequency information directly instead of the original pixels for further data augmentation.Specifically, we propose an adaptive fusion algorithm, which assumes that the number of minibatches ism during training and defines two parameters p and q (0≤p<q≤10≤p<q≤1), which divide the training process into three phases:Step 1.
From 1 topm minibatch, train using Mixup.Step 2.
Frompm+1pm+1 to qm, switch between Mixup and base data enhancement algorithms.Step 3.
Run Mixup with probabilityϵ, where ϵ decreases linearly from 1 to 0.In Step1, the FreMix algorithm uses Mixup to search in the large percentage of the sample representation area. In Step 2, there is a trade-off between exploration and exploitation. Step 3 gradually switches from the exploration model to the exploitation model. As for effective training, FreMix is a general and straightforward training policy. To balance exploration and exploitation, we use the method of reintroducing essential data augmentation. When comparing CutMix and mWh in CutMix with ResNet-50, experimental results showed that the convergence rate of diverse dataset instances is improved by mWh and the accuracy on ImageNet is robust. Compared with the baseline, it also gets significant improvement in various tasks and models. Our FreMix method based on mWh in the frequency domain gives further improved results.


## 3.1. Frequency-Based Data Mix

Convolutional neural networks (CNNs) have shown excellent performance in computer vision tasks, particularly in classification tasks. To increase robustness in real-world scenarios, CNNs often adopt two practical strategies: data augmentation and model integration. Data augmentation reduces overfitting and improves the generalization of the model. Traditional image enhancement is label preserving, e.g., flipping and cropping. Multiple inputs and their labels are proportionally mixed to create artificial samples in mixed-sample data augmentation (MSDA) approaches [28]. MSDA is simple to implement and really helps to improve performance, so it is widely used in areas such as image recognition, sound recognition, GAN, and semisupervised learning.Recently, hybrid-based enhancement techniques have achieved highly accurate prediction performance. They generate input data by a linear combination of two randomly selected training data. Likewise, their corresponding labels are generated by the same linear combination of two labels. By doing so, they effectively improve prediction accuracy while preventing some undesirable behaviors such as memory and sensitivity to adversarial examples. Moreover, Mixup training encourages the output of the DNN [29], i.e., the estimated label distribution, as a better indicator of the actual likelihood of correcting the predictions. Specifically, for generating an enhanced sample, the calculation for Mixup training is as follows
(1)xmix=λx1+1−λx2,M⊙x1+1−M⊙x2,ymix=λy1+1−λy2,λ∼Betaα,α,where x and y denote a training sample and its label, M∈0,1W×H denotes a binary mask indicating the position of the reject and fill from the two images, ⨀ is an element multiplication, Beta∙,∙ implies the beta distribution, and α∈0,∞ is the parameter controlling the shape of the beta distribution. Using mixed inputs and mixed labels, the model minimizes the following equation.
(2)Lmix=λHy~mix,y1+1−λHy~mix,y2,where y~=σfx denotes the predicted label distribution from the model, f is the model, σ is the activation function, usually a softmax function, σsmz=expz/∑i=1Nexpzi, and H is the cross-entropy function formulated by Hp,q=−∫xpxlogqx.Because the Mixup’s raw picture exploration is restricted, the suggested FreMix performs a fast Fourier transform (FFT) on the input image first, extracting the frequency information rather than the original pixel information for subsequent augmentation. Figure1 depicts the flow of our technique. The fast Fourier transform (FFT) is a technique for computing a sequence’s discrete Fourier transform (DFT) or inverse transform fast [30]. Fourier analysis converts a signal’s original domain (typically time or space) into a frequency domain representation or vice versa. By reducing the DFT matrix into the product of sparse (mainly zero) elements, the FFT can quickly perform such a transform.Figure 1
Schematic diagram of our approach.The DFT requires the computation of approximatelyN2 multiplications and N2 additions. This computation is large when N is large. The N-point DFT is decomposed into two N/2-point DFTs using the symmetry and periodicity of WN; therefore, the total computation of two N/2-point DFTs is just half of the original, i.e., N/22+N/22=N2/2. This can be continued by decomposing N/2 into N/4-point DFTs again, etc. For N=2 m, points of DFT can be decomposed into 2 points of DFT, so that its computation can be reduced to N/2log2N times multiplication and Nlog2N times addition. Here are the steps of the operation.The DFT of a finite length discrete signalxn,n=0,1,⋯,N−1 is defined as
(3)Xk=∑n=0N−1xnWNknk=0,1,⋯,N−1,WN=e−j2πN.Decomposingxn into the sum of two sequences of even and odd numbers, i.e.,
(4)xn=x1n+x2n,where x1n and x2n are both of length N/2, x1n is an even sequence, and x2n is an odd sequence, then
(5)Xk=∑n=0N/2−1x1nWN2kn+∑n=0N/2x2nWN2n+1kk=0,1,⋯,N−1.BecauseWN2kn=e−j2π/N2kn=e−j2π/N/2kn=WN/2kn, then
(6)Xk=∑n=0N/2−1x1nWN/2kn+WNk∑n=0N/2−1x2nWN/2kn=X1k+WNkX2kk=0,1,⋯,N−1,where X1k and X2k are the N/2-point DFTs of x1n and x2n, respectively. Since both X1k and X2k have period N/2 and WNk+N/2=−WNk, Xk can again be expressed as
(7)Xk=X1k+WNkX2kk=0,1,⋯,N2−1,Xk+N2=X1k−WNkX2kk=0,1,⋯,N/2−1.The principle of the FFT algorithm is to achieve large-scale transformations by many small more easily performed transformations, reducing the operational requirements and increasing the speed with the operation. The FFT is not an approximation of the DFT, they are exactly equivalent. Figure2 shows the flow chart of 8-point FFT decomposition.Figure 2
Illustration of an 8-point FFT decomposition procedure.


## 3.2. Adaptive Fusion Algorithm

FreMix creates new samples by linearly interpolating pairs of FFT information of raw examples, and it is simple to do and to be effective in picture classification problems. There are two issues. First, FreMix asks for more epochs to converge. The reason is that it needs more extended training and explores more regions of the data space. Second, FreMix has a condition of hyperparameter value to sample mixing coefficients, while various hyperparameter values usually cause large differences in model accuracy. To mitigate this problem, inspired by Mixup without hesitation (mWh) [31], we integrate the exploration-exploitation mechanism from reinforcement learning into our FreMix to overcome both problems, rather than enriching data by using Mixup during the model training process. FreMix will be speeded up since exploration-exploitation will turn the mixing operation off. And it also makes it robust to the hyperparameter α. Note that mWh is a pixel domain-based method, which has the disadvantage of requiring extraction from the original pixels compared to the frequency domain. We utilize the exploration-exploitation (EE) mechanism to operate in the frequency domain in the proposed FreMix. Our method can extract frequency information directly instead of the original pixels for further data augmentation.Specifically, we propose an adaptive fusion algorithm, which assumes that the number of minibatches ism during training and defines two parameters p and q (0≤p<q≤10≤p<q≤1), which divide the training process into three phases:Step 1.
From 1 topm minibatch, train using Mixup.Step 2.
Frompm+1pm+1 to qm, switch between Mixup and base data enhancement algorithms.Step 3.
Run Mixup with probabilityϵ, where ϵ decreases linearly from 1 to 0.In Step1, the FreMix algorithm uses Mixup to search in the large percentage of the sample representation area. In Step 2, there is a trade-off between exploration and exploitation. Step 3 gradually switches from the exploration model to the exploitation model. As for effective training, FreMix is a general and straightforward training policy. To balance exploration and exploitation, we use the method of reintroducing essential data augmentation. When comparing CutMix and mWh in CutMix with ResNet-50, experimental results showed that the convergence rate of diverse dataset instances is improved by mWh and the accuracy on ImageNet is robust. Compared with the baseline, it also gets significant improvement in various tasks and models. Our FreMix method based on mWh in the frequency domain gives further improved results.


## 4. Experiments

We evaluate our method in a variety of tasks and compare it to the competition. We emphasize that our approach can be combined with any model structure to enjoy its performance benefits. This is simply done by adding our data augmentation technique before training the model.


### 4.1. Model Architecture

We choose five CNN architectures as backbone networks: three of them are traditional CNN (i.e., VGGNet, ResNet, and ResNeXt) [32] and the others are lightweight CNN (i.e., MobileNetV2 and ShuffleNet) [33] datasets. We validated the effectiveness of our method on three benchmark datasets ranging from small to large: CIFAR100 (32×32 RGB images, 100 classes), Tiny imagenet (64×64 RGB images, 100 classes) and ILSVRC2015 (256×256 RGB images, 1000 classes).


### 4.2. Evaluation Settings

The best training method for all networks was stochastic gradient decay with a momentum of 0.9. All methods used for our comparisons followed the same training schedule and dataset. For CIFAR100, we set the initial learning rate to 0.1 and decayed the learning rate by 0.2 every 60, 120, 160, and 200 epochs. In Tiny imagenet and ILSVRC2015, we set the initial learning rate to 0.1 and decayed it by 0.1 in 75, 150, and 225 calendars. Because lightweight models have a different ideal training scheme, we followed the procedure described in their paper. We used a weight decay of 4e-5 for CIFAR100 and 1e-4 for the other models to regularise the model. For the CIFAR100, Tiny imagenet, and ILSVRC datasets, the batch sizes for each model are 256.


### 4.3. Classification Accuracy and Expected Calibration Error

We will show that our approach improves classification accuracy (i.e., high confidence on test samples). Besides, in this subsection section, we experiment with a more realistic scenario. For the quantitative analysis of confidence calibration, we use two popular metrics, expected calibration error (ECE) and excess confidence error (OE).The ECE represents the average difference between true confidence and predicted confidence. If ECE is zero, it means that the network is correctly calibrated. OE is similar to ECE, but it only measures the difference in confidence when overconfidence is indicated. Overconfidence is primarily a critical factor in high-risk systems. Therefore, this metric is a good indicator to assess system reliability for high-risk applications. These two metrics were calculated on the validation set.The experimental findings employing various networks and data sets are shown in Tables1, 2, and 3. After combining our methods, we consistently achieved better accuracy and confidence calibration. In particular, our gains in prediction accuracy are substantial, with the gap between the baseline and the baseline combined with our method being as large as the gap between vanilla and other competitors. As a result, our method exceeds the performance of existing methods under most experimental conditions. A key observation can be made by experimenting with compact models like MobileNetV2. In general, a Mixup-like approach is an enhancement method that populates training examples to prevent overfitting. However, if it injects examples that are far from the training distribution, such an augmentation method can cause underfitting. Underfitting usually does not degrade the performance of high-volume networks, but it can impair the performance of low-volume networks. Since MobileNetV2 is a low-volume model, it requires less regularization than larger models, and applying weak regularization (i.e., small weight decay) may be sufficient. When CutMix was used to train MobileNetV2 on Tiny imagenet, we observed severe performance degradation, and we speculate that the accuracy degradation is due to the strong regularization induced underfitting.Table 1
The results of our experiments on CIFAR100.
NetworkMetricVanillaMixupFreMixCutMixFreMix+EEVGG16Acc74.3075.0276.2275.3476.10ECE0.180.060.030.060.06OE0.160.040.020.030.08ResNet50Acc78.3279.8280.9680.5781.02ECE0.090.040.020.080.07OE0.070.030.010.060.06ResNeXt50Acc79.1881.1081.6381.1681.46ECE0.060.040.020.0590.03OE0.050.010.000.0470.02MobileNetV2Acc69.6969.9673.9068.8269.91ECE0.060.010.040.050.04OE0.040.010.010.010.00ShuffleNetV2Acc72.1774.1775.5373.6073.73ECE0.080.060.0420.010.02OE0.060.000.0000.010.00Table 2
The results of our experiments on Tiny imagenet.
NetworkMetricVanillaMixupFreMixCutMixFreMix+EEResNet50Acc66.668.3470.7169.0869.87ECE0.090.0320.030.0290.03OE0.070.0220.010.0150.05MobileNetV2Acc57.6259.5562.1253.5457.66ECE0.080.090.030.090.08OE0.050.020.000.000.00Table 3
The results of our experiments on ILSVRC2015.
NetworkMetricVanillaMixupFreMixCutMixFreMix+EEResNet50Acc76.1377.3778.3878.4378.51ECE0.370.040.030.030.02OE0.030.010.010.030.03In contrast, when our method is combined with CutMix, the impact of this underfitting is greatly reduced, filling the degradation gap caused by CutMix. From this result, we believe that our method does not penalize vanilla training to prevent overfitting. Instead, our method helps the model understand the hidden relationships by weak supervision. Since hidden relationships can provide a reasonable explanation for understanding examples far from the training distribution, our method can also prevent underfitting. This observation is consistent with our motivation that interclass correlations help improve prediction accuracy and prediction confidence estimates.


## 4.1. Model Architecture

We choose five CNN architectures as backbone networks: three of them are traditional CNN (i.e., VGGNet, ResNet, and ResNeXt) [32] and the others are lightweight CNN (i.e., MobileNetV2 and ShuffleNet) [33] datasets. We validated the effectiveness of our method on three benchmark datasets ranging from small to large: CIFAR100 (32×32 RGB images, 100 classes), Tiny imagenet (64×64 RGB images, 100 classes) and ILSVRC2015 (256×256 RGB images, 1000 classes).


## 4.2. Evaluation Settings

The best training method for all networks was stochastic gradient decay with a momentum of 0.9. All methods used for our comparisons followed the same training schedule and dataset. For CIFAR100, we set the initial learning rate to 0.1 and decayed the learning rate by 0.2 every 60, 120, 160, and 200 epochs. In Tiny imagenet and ILSVRC2015, we set the initial learning rate to 0.1 and decayed it by 0.1 in 75, 150, and 225 calendars. Because lightweight models have a different ideal training scheme, we followed the procedure described in their paper. We used a weight decay of 4e-5 for CIFAR100 and 1e-4 for the other models to regularise the model. For the CIFAR100, Tiny imagenet, and ILSVRC datasets, the batch sizes for each model are 256.


## 4.3. Classification Accuracy and Expected Calibration Error

We will show that our approach improves classification accuracy (i.e., high confidence on test samples). Besides, in this subsection section, we experiment with a more realistic scenario. For the quantitative analysis of confidence calibration, we use two popular metrics, expected calibration error (ECE) and excess confidence error (OE).The ECE represents the average difference between true confidence and predicted confidence. If ECE is zero, it means that the network is correctly calibrated. OE is similar to ECE, but it only measures the difference in confidence when overconfidence is indicated. Overconfidence is primarily a critical factor in high-risk systems. Therefore, this metric is a good indicator to assess system reliability for high-risk applications. These two metrics were calculated on the validation set.The experimental findings employing various networks and data sets are shown in Tables1, 2, and 3. After combining our methods, we consistently achieved better accuracy and confidence calibration. In particular, our gains in prediction accuracy are substantial, with the gap between the baseline and the baseline combined with our method being as large as the gap between vanilla and other competitors. As a result, our method exceeds the performance of existing methods under most experimental conditions. A key observation can be made by experimenting with compact models like MobileNetV2. In general, a Mixup-like approach is an enhancement method that populates training examples to prevent overfitting. However, if it injects examples that are far from the training distribution, such an augmentation method can cause underfitting. Underfitting usually does not degrade the performance of high-volume networks, but it can impair the performance of low-volume networks. Since MobileNetV2 is a low-volume model, it requires less regularization than larger models, and applying weak regularization (i.e., small weight decay) may be sufficient. When CutMix was used to train MobileNetV2 on Tiny imagenet, we observed severe performance degradation, and we speculate that the accuracy degradation is due to the strong regularization induced underfitting.Table 1
The results of our experiments on CIFAR100.
NetworkMetricVanillaMixupFreMixCutMixFreMix+EEVGG16Acc74.3075.0276.2275.3476.10ECE0.180.060.030.060.06OE0.160.040.020.030.08ResNet50Acc78.3279.8280.9680.5781.02ECE0.090.040.020.080.07OE0.070.030.010.060.06ResNeXt50Acc79.1881.1081.6381.1681.46ECE0.060.040.020.0590.03OE0.050.010.000.0470.02MobileNetV2Acc69.6969.9673.9068.8269.91ECE0.060.010.040.050.04OE0.040.010.010.010.00ShuffleNetV2Acc72.1774.1775.5373.6073.73ECE0.080.060.0420.010.02OE0.060.000.0000.010.00Table 2
The results of our experiments on Tiny imagenet.
NetworkMetricVanillaMixupFreMixCutMixFreMix+EEResNet50Acc66.668.3470.7169.0869.87ECE0.090.0320.030.0290.03OE0.070.0220.010.0150.05MobileNetV2Acc57.6259.5562.1253.5457.66ECE0.080.090.030.090.08OE0.050.020.000.000.00Table 3
The results of our experiments on ILSVRC2015.
NetworkMetricVanillaMixupFreMixCutMixFreMix+EEResNet50Acc76.1377.3778.3878.4378.51ECE0.370.040.030.030.02OE0.030.010.010.030.03In contrast, when our method is combined with CutMix, the impact of this underfitting is greatly reduced, filling the degradation gap caused by CutMix. From this result, we believe that our method does not penalize vanilla training to prevent overfitting. Instead, our method helps the model understand the hidden relationships by weak supervision. Since hidden relationships can provide a reasonable explanation for understanding examples far from the training distribution, our method can also prevent underfitting. This observation is consistent with our motivation that interclass correlations help improve prediction accuracy and prediction confidence estimates.


## 5. Discussion

### 5.1. The Difference between Our Method and Other Methods

The coefficients of Mixup need to be adjusted manually. They are fused in the pixel domain. The coefficients of CutMix also need to be adjusted manually, and the trained model is fixed for both. Mixing in the pixel domain affects the pixel-based image properties. Mixup injects examples that are far from the training distribution, leading to the underfitting of the data augmentation. CutMix generally leads to underfit problems and may lead to confusion in model classification. In our method, the coefficients can be adjusted automatically within a preset range (for a comparison of Mixup and CutMix with our method in terms of coefficients, fusion domain, and training patterns, see the Table4 for details).Table 4
The difference between our method and other methods.
MethodCoefficientFusionTrainingMixupManualPixel fieldFixedCutMixManualPixel fieldFixedOursAutomaticFrequency fieldAdaptive


### 5.2. Shortcomings and Future Work

Although our method mines the image data and expands the experimental data in the field of data augmentation, it will add extra computation. Another point is that although our method can automatically adjust the coefficients, the range of adjustment needs to be set. Then, the parameters are adjusted within the range we set. That means the adaptive adjustment of the parameters is within a certain range that we set manually. It is not completely free of manual operation. Our future research direction is to add more automatic adjustment settings to avoid manual hyperparameter adjustment as much as possible.


## 5.1. The Difference between Our Method and Other Methods

The coefficients of Mixup need to be adjusted manually. They are fused in the pixel domain. The coefficients of CutMix also need to be adjusted manually, and the trained model is fixed for both. Mixing in the pixel domain affects the pixel-based image properties. Mixup injects examples that are far from the training distribution, leading to the underfitting of the data augmentation. CutMix generally leads to underfit problems and may lead to confusion in model classification. In our method, the coefficients can be adjusted automatically within a preset range (for a comparison of Mixup and CutMix with our method in terms of coefficients, fusion domain, and training patterns, see the Table4 for details).Table 4
The difference between our method and other methods.
MethodCoefficientFusionTrainingMixupManualPixel fieldFixedCutMixManualPixel fieldFixedOursAutomaticFrequency fieldAdaptive


## 5.2. Shortcomings and Future Work

Although our method mines the image data and expands the experimental data in the field of data augmentation, it will add extra computation. Another point is that although our method can automatically adjust the coefficients, the range of adjustment needs to be set. Then, the parameters are adjusted within the range we set. That means the adaptive adjustment of the parameters is within a certain range that we set manually. It is not completely free of manual operation. Our future research direction is to add more automatic adjustment settings to avoid manual hyperparameter adjustment as much as possible.


## 6. Conclusion

Generally, existing deep learning models rely on a great number of data. Data augmentation has a strong generalization ability to raise the training data set as well as make the data set as diverse as possible. The current class of augmentation methods based on mixing different samples, including Mixup and CutMix, can be very effective in improving the model’s accuracy. However, since these methods operate mainly in the pixel domain, they cannot process successfully mined and fused image data, and the fused weighting factors are mainly modified manually, so they are not suitable for practical applications.To better exploit the frequency information for data augmentation and improve the existing sample-based fusion strategies, we propose a novel FreMix method that fuses with extracted frequencies and performs adaptive exploration of the fused coefficients. Our method achieves very good performance on several datasets with different model structures, such as CIFAR and ImageNet, which greatly surpasses previous methods. Further, despite the effectiveness, it should be noted that the proposed method increases the computational effort and the coefficients of adaptive exploration fusion are not fully automatic, and a range needs to be set manually. Our future work will focus on improving the efficiency of frequency fusion as well as adding more automatic adjustment settings to avoid manual adjustment of hyperparameters.



---

*Source: 5323327-2022-04-14.xml*