Improving Native CNN Robustness with Filter Frequency Regularization
Authors: Jovita Lukasik
Paul Gavrikov
Janis Keuper
Margret Keuper
Venue: Accepted by TMLR ()

Abstract: Neural networks tend to overfit the training distribution and perform poorly on out-of-distribution data. A conceptually simple solution lies in adversarial training, which introduces worst-case perturbations into the training data and thus improves model generalization to some extent. However, it is only one ingredient towards generally more robust models and requires knowledge about the potential attacks or inference time data corruptions during model training. This paper focuses on the native robustness of models that can learn robust behavior directly from conventional training data without out-of-distribution examples. To this end, we study the frequencies in learned convolution filters. Clean-trained models often prioritize high-frequency information, whereas adversarial training enforces models to shift the focus to low-frequency details during training. By mimicking this behavior through frequency regularization in learned convolution weights, we achieve improved native robustness to adversarial attacks, common corruptions, and other out-of-distribution tests. Additionally, this method leads to more favorable shifts in decision-making towards low-frequency information, such as shapes, which inherently aligns more closely with human vision.

Published in Transactions on Machine Learning Research (12/2023)
Improving Native CNN Robustness with Filter Frequency Regularization
Jovita Lukasik" University of Siegen
jovita. jonta..ukasikkuunisiieee...
Paul Gavrikov IMLA, Offenbury University
paul.gaurkkavrthhsseffeebbbbaa de
Janis Keuper
Jenis.eewuerrhss-offeebyyy de
IMLA, Offenburg University
Margret Keuper
keaper@mpiiinf.mppp de
University of Monnheim
Max Planck Institude for Informatics, Suarland Informatics Campus
Reviewed on OpenReview: https: // http:/pppreeeeeeeeeeee openrevie.. net/ forun? id- 2wecNCp27Y
Abstract
Neural networks tend to overfit the training distribution and perform poorly out-of- distribution data. A conceptually simple solution lies in adversarial training, which intro- duces worst-case perturbations into the training data and thus improves model generalization to some extent. However, is only one ingredient towards generally more robust models and requires knowledge about the potential allacks or inference time data corruptions during model training. This paper focuses on the native robustness of models that can learn robust behavior directly from conventional training data without out-of-distribution examples. To this end, we study the frequencies in learned convolution filters. Clean-trained models often prioritize high-frequency information, whereas adversarial training enforces models to shift the focus to low-frequency details during training. By mimicking this behavior through fre- quency regularization learned convolution weights, we achieve improved native robustnoss to adversarial attacks, common corruptions, and other out-of-distributin lests. Addition- ally, this method leads more favorable shifts in madisionnmaking towards low-frequency information, such shapes, which inherently aligns more closely with human vision.
1 Introduction
Modern convolutional neural networks (CNNs) (He et al., 2016; Liu et al., 2022; Tan & Le, 2019) show steady increase in performance in terms of test accuracy models suffer from a low generalization ability, even when faced with small domain shifts. To improve the low generalization ability, previous work focused on aspects such as aliasing (Zhang, 2019; Zou et al., 2020; Li et al., 2020; Grabinski et al., 2022b;a), the padding operations (Gavrikov & Keuper, 2023), the training schedule (Lopes et al., 2019; Saikia al., 2021), or analyzing the image feature feature spectrum (Geirhos al., 2019; Wang al., 2020). In addition, introducing perturbed images into the training data, known as adversarial training (AT) (Madry et al., 2018), alleviate low generalization to some extent. However, AT is not the cure-all to improve network robustness and tends to overfit on training attacks (Tramèr & Boneh, 2019; Rice et al., 2020; Yu et al., 2022). Intuitively, the adversarial attack used during training becomes an in- domain sample of the model, while its robustness to new out-of-domain samples (e.g. a different adversarial
wide range of learning tasks. Yet, most
* "Equal contribution.

---

Published in Transactions on Machine Learning Research (12/2023)
+000
vertcal Iriq.
Beg percocooy
ALD
Layers
harionna fequanio
Regular Convolution
1A
(c) Regularization of the first third of the network.
(d) Regularization of all layers.
(a) Computation frequency distributions
Figure Our proposed regularization decreases the reliance on high-frequency information of a ResNet-20 trained on CIFAR-10 (b-d) visible in mean DCT-II coefficients magnitudes (= frequency distribution;
computation shown in (a)).
attack) is hard to anticipate. Saikia et al. (2021) show that AT can even increase the mean corruption error ImageNet-C (Hendrycks & Dietterich, 2019). Therefore, we argue that AT only be one ingredient towards building more robust models, while the main focus should rather be to encourage behavior that we call native robustness. We expect from natively robust models that they can learn robust behavior directly from the conventional training data. Thereby, robust. behavior includes, on the one hand, a certain degree of adversarial robustness without being confronted with adversarial attacks during training, É.e. the model should not casily be fooled using attacks with very small perturbation budgets. Similarly, they should be robust against other perturbations such common corruplions (Hendrycks & Dielterich, 2019) long corruption severities are low. On the other hand, robust behavior implies a better aligument with human perception, i.e. models should decide for specific class more by the shape of an object than by its texture (Geirhos et al., 2019). Note that the expected degree of specific robustness can not be compared to the one obtained by techniques that specifically optimize for them, such as adversarial training. For instance, adversarial samples remain out-of-domain samples for such natively robust models. Yet, additionally training these natively robust models with AT should be complementary and have a further beneficial effect.
In this paper, we propose new perspective on improving native robustness by investigating the frequencies in the learned network filters directly. Specifically, we propose to project CNN convolution filter weights into the frequency domain by applying discrete cosine transformation (DCT-II). Although the resulting formulation is in principle equivalent to the commonly adopted CNN formulation, it provides direct access to the learned filter froquencics. Thereby, we aim to invostigate the following rescarch questions: (i) Which filter frequencies are predominantly learned in the layers of CNNs? (ii) Can we regularize the frequencies during the training process such as increase the native robustness of the learned model?
We investigate these questions in the context of image classification yet our approach bears the potential to be expanded to other lasks such object detection and segmentation. First, analyze the learned filter frequencies of modern CNNs and observe that they tend to have low-frequency bias in deep layers, while fil- ters of earlier layers of the network are either uniformly distributed in frequency space or even biased towards higher frequencies. In the latter cases, the convolution thus relies on high-frequency information. Contrary, adversarial training appears to shift the focus low filter frequencies havior, we introduce a rogularization scheme, which increases the bias to fow-frequencics in these carly layers (see Figure for visualization). We evaluate the proposed spectral decomposition and regularization on different CNNs under distribution shifts in test data. Results on CIFAR-10 10, CIFAR-100 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), MNIST (LeCun et al., 2010), Tiny-Im (Le & Yang, 2015) and ImageNet (Deng et al., 2009) show increased native robustness In summary, we make the following contributions:
early layers. To leverage this be-
(ode:: http:///ggthaa.ccc////oaaaaaaaat
2

---

Published in Transactions on Machine Learning Research (12/2023)
We observe that adversarial training results in shift towards low-frequency bias in the filter weights of early layers during the early phases of training (Section 3).
Based on this observation, we propose high-frequency penalization term in the weight space of convolution layers (Section 4) to miligate the reliance on high-frequency information.
Networks trained with this regularization become gradually, yet consistently, more robust against a wide array of out-of-distribution generalization tasks without reliance on AT or additional data í.e. networks increase their native robustness (Section 5). Additional AT is complementary and further improves the measurable adversarial robustness to a variety of attacks.
2
Related work
Robustness. While modern neural networks yield accuracies close to or even beyond human performance, they seem to struggle with generalization to out-of-distribution data. In the context of adversarial at it has been shown that minor, for the human eye barely perceivable perturbations can cause models to make wrong predictions with high confidence. Formally, let f be model parameterized by 8, æ an input sample with the corresponding class label y, and & the loss function. Then adversarial attacks will attempt to maximize the loss L by finding an additive perturbation to an imput sample x'. To constrain their intensity, perturbations are sought in an B, (2) ball centered at x with a radius of e.
max C (f (1(: xr E'EB, (x))
B.(x) {z' :1 x'Ilp sc..
with ||. ||p depicting the -n-norm. The most successful adversarial attacks are whito-box attacks, where the attacker has full access to the allacked model. Often, these methods rely on gradient information, such as projected gradient descent (PGD) (Kurakin et al., 2017) where the attacker follows the gradient that maximizes the loss and then projects the perturbations back to (x)). Since PGD is computationally expensive, a faster, yet less successful attack that approximates the perturbations by the gradient sign and only performs one step has been proposed: fast gradient sign method (FGSM) (Goodfellow et al., 2015).
Regarding defenses in general, the most successful approach to tackle out-o--distribution shifts is adversarial training (Madry et al., 2018) where worst-case perturbations are reintroduced to the training data, Often, these methods are accompanied by additional external data (Carmon et al., 2022). Evaluating the defenses a single attack can be mislcading, due to the possibility of attack overfitting (Rice et al., 2020). Towards more reliable benchmarks, and black- box attacks such as APGD (Croce& & Hein, 2020a), FAB (Croce & Hein, 2020b), and Square (An- driushchenko et al., 2020) and establishes the public RobustBBncc leaderboard. Benchmarks are constraincd to p = 2, e = 0.5 on CIFAR- p = 00, = 8/255 on CIFAR- 10000,, and p = x0, = 4/255 on ImageNet, respectively. However, these large thresholds are disputed as they generate easily detectable perturbations (Lorenz et al., 2022).
Allcck (AA) (Croce & Hein, 2020a) proposes an ensemble of various white-
Unfortunately, the possibility of adversarial attacks is only symptom of larger generalization issues. For example, neural networks fail to generalize under various corruptions such as weather conditions, changes in lighting, noise, and blurring (Dodge & Karam, 2017; Hendrycks & Dietterich, 2019). For fast and comparable benchmarks, common corruption datasets SIFAR-10-C, 100-C, and ImageNet-CC C' have been proposed (Hendrycks & Dietterich, 2019), which include 15 (+4 extra) types of corruption at increasing severity level (from to 5).
Additionally, Geirhos et al. (2019) observed that CNNs are biased towards detecting textures of an image instead of the shape (cue-conftict), which is in contrast to human vision behavior that focuses on shape information, f.e. shape bias. To overcome this texture bias, they train on a stylized version of ImageNet to increase the shape bias of CNNs. For fast evaluation of out-of-distribution (OOD) generalization Geirhos et al. (2021) proposed a benchmark including 17 OOD datasets, from which 12 contain image perturbations and the other 5 are single manipulations of ImageNet (Deng et al., 2009): cuc-conflicted texture vs. shape data, sketches (Wang et al., 2019), stylized images, edges, and silhouettes. They evaluate and compare more than 50 different. networks to human performance to determine the gap between human and machine vision.
3

---

Published in Transactions on Machine Learning Research (12/2023)
Frequencies and robustness. Recent work demonstrated the importance of learned frequencies for net- work robustness. Wang et al. (2020) demonstrated that CNNs initially rely on low-frequency information for prediction, but shift towards high-frequency information as training progresses. On the other hand, AT models predominantly classify based on low-frequency information. As texture information typically resides in higher frequency bands, this a suitable explanation for the observations by Geirhos et al. (2019). As such, there is also a correlation between AT and a reduced texture bias (Geirhos et al., 2021; Gavrikov et al., 2023). Duan et al. (2021) exploit these findings by proposing an adversarial attack that drops DCT coefficients corresponding to high frequencies from inputs to fool neural networks. Yet despite the common assumption, adversarial attacks are not always targeting high-frequencies and the behavior depends on the dataset (Maiya et al., 2021; Abello et al., 2021; Bernhard et al., 2021; Ortiz-Jiménez et al., 2020).
Multiple works explore the desensitization of neural networks to HF from various angles to avoid AT: Lopes et al. (2019) randomly add noise to image patches, Saíkia. et al. (2021) regularize the feature maps pro- ducod by convolution layers in a dedicated two-stream architecture, and Grabinski et al. (2022a) introduce a downsampling approach within the frequency domain that removes aliasing-related high-frequency informa- tion. Huang et al. (2023) propose frequency regularization in the context of adversarial training, which contrary to the previous works- amplifics high frequencies, resulting in higher robust accuracy, at an impair- ment of clean accuracy. In contrast, we regularize high-frequency information directly in convolution filters to improve the native robustness and OOD generalization of the model, which is not limited to effects on adversarial training, but indeed can also improve the robust accuracy in the context of adversarial training, and eventually mitigate robust overfitting.
Feature map vs. weight regularization. We have previously outlined the role of HF in generalization within the existing literature. In this paragraph, we aim to emphasize the distinction between our proposed weight regularization method and prior research.
Regularization us. bandpass filtering: Previous methods often lowpass-filer signals which leads to a hard smoothing of the resulting feature maps and the active delction of information that may be necessary to predictions - especially in fine-grained classification problems. Instead, we only regularize the allenuation and thus effectively force the network to reweigh information without having to discard information.
Data-independent and explicit attenuation: By regularizing weights, we induce an explicit causal bias in the operator. Alternatively, an attemuation of feature maps would be implicit and would highly depend on the frequency distribution of the inputs. Additionally, attenuating the filters results in a local suppression (i.e., in the patch) of HF, while a (global) feature-map regularization would affect the entire scene.
Basis decomposition. The decomposition of convolution filters is typically studied in the context of compression, see Varoslavsky (2014) for an overview. The majority of decomposition approaches convert the convolution layer weights to the frequency domain e.g. by utilizing the DCT-II-basis (Chen, 2004; Chen et al., 2016; Lo & Hang, 2019; Cheinski & Wawrzynski, 2020; Chen et al., 2022: Ulieny et al., 2022) to prune and compress the number of frequency components. But works also exist that transform the input images directly for better performance and generalization (Xu et al., 2020; Hossain et al., 2019). In detail, the discrete cosine trunsform (DCT) (Ahmed et al., 1974) maps an input signal into a frequency domain represented by cosine basis functions. In particular, the common DCT-II variant is used in JPEG compression, where it successfully compresses natural images (Wallace, 1992). These works mainly explore the fact, that data of multiple domains is not uniformly distributed in the frequency domain and is typically biased towards low frequencies (Singh & Theunissen, 2004; Ruderman, 1994). Gavríkov & Keuper (2022a:b) showed that the basis of convolution filter kernels obtained via SVD is often highly similar and independent of the architecture, learned task, or datasct. The identified bases have striking similarity to the DCT-II basis.
Our realization of the DCT-II basis is similar to Ulieny et al. (2022) and other previous work, however, instead of compression, we explore an orthogonal direction and study the role of individual frequencies in training and apply regularization in the frequency space to improve generalization. DCT merely serves as a tool in our study.
4

---

Published in Transactions on Machine Learning Research (12/2023)
3 Frequency analysis
In this initial analysis, we transform learned convolution filters to the frequency domain. We implement this by changing the basis of convolution weights to DCT-II, revealing the coefficients and therefore frequency information. Formally we define this as follows. Let. denote the k x k-DCT-IT basis. Then every basis vector Viy with horizontal frequency j and vertical frequency is defined as:
Ti k
1 2) cOs 2
Tj
2 2 )
V V.j.m.n cos
(2)
k
Every basis vector is additionally normalized to its L length: V.g Vzy/ M basis vectors for different kernel sizes k in Figure 2. In principle, DCT-II could be replaced by any other
i l11- We show the DCT-II
1:00
setiaal Eequasis
veterl Enqueecie
vetica Ireparace
00
(a) 3x3
(b) 5x5
(c) 7x7
Figure 2: The full DCT-II basis for different resolutions.
frequency base such as discrete Fourier or sine transform. Following the basis change, we visualize the average magnitude of coefficients in every convolution layer by heat maps (as shown in Figure 1). Having the frequency information at hand, we can directly analyze its distribution common CNNs.
3.1 Analyzing learned convolution weights
We start by analyzing two modern networks trained on ImageNet. without any robustness optimization techniques: EfficientNet-BO (Tan & Le, 2019) and ConvNeXt-Tiny (Liu et al., 2022) (Figure 3). Our visualizations show that these CNNs do not always learn a uniform frequency spectrum utilization throughout the network. Earlier layers show a more uniform distribution of magnitude or are biased towards higher frequencies. However, deeper convolution layers instead reveal a salient bias towards low frequencies. Some layers even appear discard majority of high-frequency information.
In addition, filters. As shown from various angles (Wang et al., 2020; Geirhos et al., 2019; Saikia et al., 2021) robust models shift their bias to low-frequencies, as this reduces the possibility of overfitting on high-frequencies and therefore provides better generalization abilities. Thus, we expect that these results transfer to the frequency utilization weight space some extent. Indeed, Wang al. (2020) stated that the very first
interested in how adversarial training affects the frequency utilization in convolution
Layers
(a) RifficientNet-
(b) ConyNeXt-Ting
Figure 3: Frequency distribution of each layer for trained (a) EeficientNNtt-BO (containing x 3 and 5 kernels) and (b) ConyNeXt-Timg (containing 2 2 [downsampling layers], 5 5 [stem] and 7 kernels).

---

Published in Transactions on Machine Learning Research (12/2023)
Training
Training
Eo:
20
Co43
to
7
to120
Cr1
20
0:20
30
Co:0
Co
To
Co
200
Gows
(a) Regular training.
(b) Adversarial training.
Figure 4: Evolution of the frequency distributions in the first three convolution layers of an CelicienNNet-Bi in comparison between (a) regular and (b) advesaaiial training with CIFAR-10. Evolution plots for all layers and other architectures can be found in Appendix L.
convolution layer of AT CNNs learns smoother filters which equals to filters that are less reliant on high- Trequency information than the equivalents normally trained models. However, their frequency analysis was limited to the first initial layer, while aim to provide holistic analysis over the entire network. This is also backed by our previous observations showing that frequency utilization varies by depth. Further, their results do not appear to be representative of modern models that trained under - Such models predominantly learn thresholding filters (Madry et al., 2018) independent of architecture and dataset (Gavrikov & Keuper, 2022b) that do not resemble "common" first layers as shown by Yosinski et al. (2014). As such, they are hardly smooth.
Exemplarily, we proceed by comparing an adversarially trained FfficienttNe--BO with regularly-trained counterpart (more comparisons are included in Appendix L). We observe that adversarial training leads to characteristically different distribution of learned frequencies during training (Figure 4). Especially in the first layers, the network learns predominantly from low frequencies, which enables the network to preserve the global image content, rather than overfitting on high-frequency details such as texture. Interestingly, the addverarrially-tranned model learns this behavior in the early training stages, and faster than under normal training conditions (Rahaman et al., 2019). Deeper layers on the other hand show no salient differences.
Based on these findings, we propose a transformation approach of convolution weights into the frequency domain to interact with frequency information. Secondly, based on the latter finding we propose high- frequency regularization, to further enforce the low-frequency bias in the first network layers and thus increase the native robustness.
4 Modifications to the convolution layers
Let us first formalize the computation flow in a conventional 2D convolution layer fommod((;; W), fronw2d transforming an imput signal æ with dyn input-channels into a signal with dout output-channels using a convolution kernel with a size of kn x kj- Further, let W E Rdoa xda x xka denote the learned weights (L.e. the set of all kernels Waj in the respective layer, without bias). Without loss of generality, we assume ko k1 in this paper. The output of feoneed (z; W) is then defined as:
d
W..d & La for 8 E (o..... dout - 1}.
(3)
d=0
In the following, we propose a simple representation in the frequency space by replacing the convolution weight W with a combination of learned coefficients on the DCT-II basis. In this work, we limit ourselves to kernels with & 2 We realize this by two common implementations seen in related literature (e.g. Ulieny al. (2022)). Schematic visualizations of both approaches can be found in Appendix E.
Weight decomposition (WD). Our first approach decomposes the weight in a convolution layer into learnable coefficients C Rdo Xde xkck and the basis V defined in Equation 2: W = CC V. Then, the

---

Published in Transactions on Machine Learning Research (12/2023)
vertical frequencies
hovizonttl frequencles
0,0
0,1
0,2
R(C) 1Cl
1,2
Paif max (1l I.),id 1001-11iii [1s.,,d,.0000l.. 0),
(7)
2,0
2,1
2,2
for I,J [[k/2] k). M, M {0, 1),
Figure 5: Regularization area of the coef- ficients in an individual 3 x 3 filter kernel. Colors match Equation 7.
[s] = {o...., dout - 13, [d] = ......, 1}.
convolution can be rewritten as:
Y. C) Fz +
Σ (C,,d,m V Vmn) Ed-
(4)
d.m.n
This increases the parameters be kept in memory a factor of 2 and adds one additional tensor multi- plication per layer. However, these additional parameters are constant and do not need to be learned.
Signal decomposition (SD). Alternatively, our second approach does not replace the convolution weight W directly but performs depthwise convolution of all combinations of inputs and the fixed basis vectors which is then aggregated by a learnable pointwise (1 x 1) convolution.
y, Σ Cs.d.m,n (mm, zd). d.m.n
This increases the parameter number by factor of dink? to be kept in memory. Again, the number of learnable parameters is not increased. Also, note that the associativity property of convolution reveals the equivalence of both formulations in the forward pass:
(6) However, due to different learning dynamics, the modifications may converge to different solutions. In both approaches, the initial coefficient weights sampled from uniform distribution with adjusted scale per He et al. (2015). For the weight decomposition approach, we use di as fan information. The basis vectors are initialized as defined in Section 3 without any further adjustments.
ys
C.ddmmm (Vm,n F..)
Σ (C d,m,n
V Vmma) Ba.
d,n.n
4.1 Frequency coefficient regularization
As we have seen in Section 3.1 neural networks are biased towards low-frequency information, while early layers also introduce more magnitude on high frequencies. However, adversarial training increases the low- frequency bias already in the early training stages resulting in an overall low-frequee freffrequnccy dominance after convergence in the first. layers. To make use of this finding and increase the robustness of CNNs directly without adversarial training, we propose regularize the DCT-II coefficients and explore the frequency shift and performance.
The proposed regularization (Equation 7 and Figure 5) regularizes the highest frequencies and additionally forces the first coofficient to have a higher magnitude than the subsequent frequency. Such behavior, that the weight coefficients decay with their corresponding frequencies, can also be observed in the adversarially robust weights in Figure 5. Interestingly, this is also in line with the theoretically expected behavior of frequency spectra that represent shapes rather than textures. We provide more thooretic background on this aspect in Appendix The occurrence of this latter constraint is determined by the binary hyperparameter Pasg, with Pag 1 throughout the paper, if not stated otherwise. The multiplicative term ブ increases

---

Published in Transactions on Machine Learning Research (12/2023)
penalization of higher frequencies. Let coefs(8, h) denote function that returns the set of convolution coefficient weights of the learnable parameters 0 in the first 1/h section of the network depth. To enforce the dominance of low frequencies in early layers, we set h = 3 as our default value. We train the network with the following modifications to the objective:
min C (f (2:0),.) +A
R(C).
(8)
CEcoefs(B.A)
Where x, y DD denotes the training dataset and is the original objective. An exemplary visualization of the learned coefficients under regularization given Figure for {1,3}.
Table 1: Performance evaluation of various networks (ResNeet-20/9 and EfficientNet-Bi. BO) on multiple
datasots (CIFAR-10/100, SVHN, TinyImageNet, MNIST) before and after our applied regularization. Wo report clean accuracy on the respective test sets as well as adversarial robustness against FGSM, PGD-40, and AutoAttack for _ 1/255 (e 16/255 for MNIST). We report averages over 5 runs.
Variant
Clean (t)
Adversarial Acc. (1)
Variant
Clean (1)
Adversarial Acc. (f)
Val Acc.
FGSM
PGD-40
AA
Val Acc.
FGSM
PGD-10
AA
CNN
91.29
50.49
30.92
10.78
CIFAR-100
ResNel 20
CNN WD
60.41
14.36
5.45
1.17
ResNett-2
WD
91.04
48.40
30.37
10.72
58. 90
12.84
4.68
1.01
SD
91.36
50.83
32. 22.98
11.97
60.31
13.87
5.18
1.13
Rrg-
50.85
41.81
26.79
WD + Reg.
SD
90.54
53.12
44.42
29.14
SD Reg.
58.19
17.20
12:24
5.11
CIFAR-10
ResNNtt99
CNN
WD
93.73
59.58
55.51
49.84
53.0 53.04
35.23
37.49
SVHN
ReeNee 20
WD
CNN
96.35
96.31
83.52
83.84
80.01
79.91
71.25
69.81
SD
93.97
55.73
50. 29
36.0
SD
96.34
84.07
80.64
71.74
WD Reg-
93.18
59.25
56.08
43.62
WD + Reg.
96,
84.11
81.21
73.27
SD) Reg.
03.09
59.87
56.89
44.80
+ Reg.
96.34
84.17
81.23
73.03
BE.Net- B30
CNN
WD
90.51
90.38
53.55
49.87
54.05
49.97
45.51
40.76
TinyImNNet
ResNeei-
CNN
53.20
52.08
17.79
16:76
9.57
90.44
51.04
51.77
43.39
SD
52.12
16.85
15.88
9.15
WD Reg.
85.97
57.91
39.60
53.30
WD + Reg.
51.25
18.10
17.34
10.23
SD) Reg.
89.18
57.83
59.68
53.50
81) + Rog.
51.22
1 18.26
1 17.40
10.39
MNIST
ResNel 200
CNN
WD
99.69
99.68
90.15
89.74
47.
46.37
10.22
8.92
SI>
99.65
91.23
54.08
16.80
WD + Reg.
99.69
90.70
55.84
25.92
SD Reg.
99.69
88.98
50.02
5
Experiments
In the following, we compare different architectures, with regular convolutions, and both decomposition variants (WD/SD) at varying frequency regularization (+ Reg.) (Equation 7). For each combination, we report results clean accuracy, as well as robustness to various aspects.
Models and datasets. We evaluate low-resolution datasets such CIFAR-10/100 (Krizhevsky, 2009), MNIST (LeCun et al., 2010), SVHN (Netzer et al., 2011), and Imag-lma (Le & Yang, 2015) on ResNet- 20 (as introduced for CIFAR He et al. (2016)), ResNet-9 a regular and larger ResNet with optimization for CIFAR and reduced number of layers (see Appendix D for architecture details), and an FfficiinnNNe--R (Tan & Le, 2019) where we remove striding from the stem convolution. For ImageNet (Deng al., 2009), we evaluate NetieoentNet-BO (Tan & Le, 2019) and Con<NeXt-Tiny (Liu et al., 2022). We test h E {1,3) and X e {0.01, 0.05, 0.1} and report results for the best performance over the mean of 5 runs except for ImageNet (1 run). Details regarding the training can be found in Appendix B.
Note that we have selected models with different kernel sizes e.g. after the stem, ResNets use k = 3, EfficientNNes-BO mix = 3 and = 5, and ConvNeXts & = 7 (and & = 2 downsampling layers). The variance in kernel size allows us demonstrate the transferability of our proposed regularization beyond the common k 3 kernels.
8

---

Published in Transactions on Machine Learning Research (12/2023)
Table 2: Comparison against other rosustness techniques (Grabinski et al., 2022a; Lopes et al., 2019) of ResNet-20,, ResNet-9, and EfficientNet-BO on CIFAR-10. We report the mean clean validation accuracy and robust accuracy against adversarial attacks: FGSM, PGD-40, and AutoAttack for L 1/255, and the mean corruption accuracy CIFAR-IO-C. We report averages over 5 runs.
Variant
Clean (t)
Adversarial Ace. (1)
CC (t)
Val Acc.
FGSM
PGD-40
AA
Acc.
CNN
67.96
FLC (Grabinski et al., 2022u)
91.52
52.49
30.25
8.48
68.75
PaGA (Lopes al., 2019)
11.35
Blur Pooling (Zhang, 2019)
89.89
41.61
29.43
15.58
66.73
ResNet-
Wavelet Pooling (Li al., 2020)
Adaptive Blur Pooling (Zoa et al., 2020)
89.89
89.48
41.17
41.94
32.09
27.88
18.22
13.78
67.40
67.17
WD SD
91.04
18
30.37
10.72
66.92
91.36
50.83
32.98
11.97
67.48
WD + Reg.
89.86 99.86
11.81
26.79
74.01
SD Reg.
90.34
53.12
44.42
29.14
74.14
CNN
29.29
59.58
53
37.49
73.38
FLC (Grabinski et al., 2022a)
94..24
59.64
53.47
38 665
73.81
PaGA (Lopes al., 2019)
ResNet-9
93.73
72.87
SD
93.97
55.73
50.29
36.00
73.48
WD + Reg.
93.18 18
59.25
56.08
43.62
76.41
SD Reg.
93.09
59.87
56.89
44.80
77.72
90.38
53.55
54.05
FLC (Grabinski al., 2022a)
89. 68.
51.92
53.09
45.37
69.72
PaGA (Lopes et al., 2019)
90.72
54.18
34.97
46.64
69.31
BefiiientNet-B0
WD SD
90.51
49.89 87
49.97
40.76
67.10
90:44
01
43.39
66.65
WD + Reg. Reg-
88.97
57.91
59.60
53.30
72.14
57.83
71.87
Table 3: A benchmark of a ResNet-20 (CIFAR-10) with various architecturial frequency attemnation tech- niques evaluated with a batch size 512 on an NVIDTA A100 GPU.
Variant
Total
Learnable Params (+)
Throughput (k img/see) (t)
Batch Update
Params (1)
(ms) (4)
CNN
272.5k
272.5k
128.0
24.7
FLC (Grabinski et al., 2022a)
272 5k
272.5k
82.7
30.2
Blur Pooling (Zhang. 2019)
5k
Adaptive Blur Pooling (Zou al., 2020)
272.7k
272.7k
31.4
41.0
Wavelet Pooling (Li et al., 2020)
272.5k
272.5k
103.0
26.5
WD (+ Reg.) SD
271.0k
272.5k
121.3
21.9
Reg.)
28.0
Robustness evaluation. To understand the effect on robustness and generalization of our proposed de- composition and regularization approaches, we run the standard AutoAttack test suite (AA) (Croce Hein, 2020a) and additional FGSM- and PGD-attacks al 1/255 We use Foolbox (Rauber et al., 2017) to run both FGSM and PGD at the default setting (e-g. 40 steps for PGD). We do not include AA results for ImageNet, these models barely withstand any attacks and measure robust accuracies of 0% even at this small without adversarial training. Further, we evaluate the robustness of common corruptions CIFAR-10 and ImageNet models on the respective corrupted datasets (Hendrycks Dietterich, 2019). In addition, we are interested the behavior of the methods towards texture bias (Geirhos et al., 2019) and OOD generalization tests (Geirhos et al., 2021). Hence, we evaluate our ImageNet (Deng et al,, 2009) models on 5 of these OOD datasets: texture-shape cue-conflict, ImageNet- of ImageNet using the implementation of
16/255 for MNIST) under the -
Sketch, Stylized-Imaaehhe Geirhos et al. (2021).
and eaige//slhonnttte-aansooommmm
5.1 Low-resolution datasets
CIFAR-10. As be expected, switching from regular to either decomposition variant has an insignificant impact the clean accuracy and small effect robust accuracy (at all adversarial attacks) (Table 1).
9

---

Published in Transactions on Machine Learning Research (12/2023)
+2
Block1.1
Block3.1
Block3.2
Block4.1
Block4
Block5.
Block5.2
Block6.
Block6.2
Block7.3
Block7.2
Blocka
Block8.2
Figure 6: Frequency heat maps of feature maps: Regularized layers (top row) of a ResNet-20 show a shift towards LF after regularization, compared to non-regularized layers in rows two and three.
However, applying the regularization clearly improves robustness towards all attacks, while slightly decreasing clean accuracy. We can also observe that SD slightly outperforms WD on almost all tested architectures. Hence, it may be tempting to only proceed with SD. However, the additionally created channels account for more parameters, a large memory overhead, and slower inference and training performance. E.g. on ResNet-20 we see a 4.4x slower forward pass and 18% more total parameters, while WD has a minimal overhead, both, in parameters and throughput. (Table 3).
Regarding robustness, we see the largest gains on models that initially performed worst (+18.36% on ResNet- 20 CIFAR-10). Out of all our tested models, EfficientNet-BO is the most robust, both, before and after regularization. Noticeably, even the worst hyperparameter combination for ResNet-20 (WD, X = 0.01, h 1, Pay 0) still achieves 14.22% higher AA accuracy than the baseline. A complete overview of tested hyperparamnters is given in Appendix F.
Common corruptions of CIFAR-10. For common corruptions (CC) (the last column in Table 2 corre- sponds to the CIFAR-10 results in Table 1), we analyze the mean accuracy over all corruptions and severities, as well as individual results for corruptions at the highest severity level. complete overview is given in Appendix C. Similar to the results on adversarial robustness we observe that on average both regularizod variants outperform the baseline. Additionally, regularized models become significantly more robust against corruptions having predominantly high frequency (HF) perturbations (see Yin et. al. (2019) for spectrums) such pizelate and defocus/glann http//lllsssssss//e gaussian blur. Perhaps less surprisingly, regularized models become less sensitive to increased JPEG compression, as they rely on (quantized) DCT-II coefficients. For corruptions with larger variance in the frequency spectrum, regularized performance remains largely unchanged. We see a slight degradation performance in low frequency (LF) corruptions such as brightness, saturation, contrast, and impulse noise. However, the accuracy drop is relatively low considering the evaluation at the highest severity level.
Other datasets. Although several works reported shift the frequency band of adversarial attacks depending on the dataset (Maiya et al., 2021; Abello al., 2021; Bernhard al., 2021; Ortiz-Jiménez al., 2020), consequently see improvement due to HF regularization Arguably, smaller improvements for SVHN/Ting ImageNte which are also the datasets that show more LF perturbations than HF. Contrary to our CIFAR-10 results, WD outperforms SD on all datasets except SVHN.
multiple datasets (Table 1).
Spectrum of feature maps. Further, we aim to understand the implications of the regularization on the computed feature maps. Exemplarily, we compare an SD + Reg. ResNet-20 against a CNN baseline and analyze the magnitude shift in the DCT-II coefficients the feature maps (Figure 6) of a clean validation batch. Our regularization causes a clear shift towards lower frequencics in regularized layers. Interestingly, in the stem layer, we also see large shifts from entirely vertical or horizontal frequencies to more balanced ones. Contrary, non-regularized (deeper) layers appear to slightly shift towards higher frequencies.
10

---

Published in Transactions on Machine Learning Research (12/2023)
Table 4: Results on ImageNet for Efficient.Net-B0 and ConwNeXt-Tiny on clean data, FGSM, PGD-40 (L c 1/255), ImageNet-CC and out-of-distribution generalization test. All regularization hyperparameters are À 0.05 and h 3.
Variaul
Adverurial Allacks
Corruption
Cae-Cuullec (1)
Skutch (1)
Styliaad (1)
Edge
Silbowulle
Acc. (T)
Ezror (1)
Top
Top
FGSM
PCD-40
ImageNet-C
Top
Top
Top
Top 5
Top
Top
CSN
16,89
14.86
63.90
86.25
48.25
55.00
BllickoNNes-B0
8D>
75.02
78.50
08.12
WD Reg.
70.44
92.15
18.45
4.03
02.03
29.38
06.38
86.83
47.02
79.75
06.23
18.18
SD + Rog
74.42
92.19
18.70
5.30
51:12
25.75
64.75
87.88
40.12
77.12
32:30
58.75
CNN
81.52
96.53
55.53
3.03
41.92
26.51
71.50
88.00
56.00
78.35
46.12
02.50
CanvNeXi-Tiay
$5.90
77.02
06.76
82.88
Table 5: FGSM-Adversarin Training on CIFAR-10 with I < 8/255. We report the mean over runs for the FGSM train and validation accuracy of the epoch of the best PGD-40 validation accuracy well as the AutoAttack accuracy. We also report the corresponding mean accuracy on CIFAR-10-C and report the difference to the clean-trained evaluations.
Clean Val. Acc. (f)
FGSM (1) Train Acc.
Adversarial Acc. (t)
Corruption Acc.
PGD-10
AA
Mean (1)
A (AT-Normal) (t)
CNN WD + Reg. + Rog.
73.73
50.39
46.14
36.09
66.99
-0.97
ResNet-20
71.69
18. 16
15.38
35.64
65.48
-8, 56
73.01
49.93
-7.41
CNN
81.70
60.27
52.77
0.00
74.06
0.68
ResNet-9
WT) Reg.
81.56
61.66
51.52
39.97
74.47
-1.94
SD + Reg.
82.56
63.39
51.80
40.14
75.40
-2.32
CNN WD + Reg.
63.00
42.34
42.49
34.04
57.35
-10.74
EfficientNet-Bi
68.50
45.87
02.66
SD + Reg.
68.89
46.76
45.57
36.76
63.07
-8.8
Table 6: PGD-Adversarial Training on ImageNet of ResNet-50 with L c 4/255. We report the train and validation accuracy under PGD attacks, validation accuracy under AutoAttack, corruption error, and cue-conflict. Results are from one rum.
Variant
(t) Train Ace.
Adversarial Acc. (f)
Corruption Error (4)
Cue-Conffict. (f)
Val Acc. (1)
PGD
AA
CNN
56.85
33.88
36.04
22.33
78.65
38.83
WD
55.82
33.91
35.06
22.06
78.90
38.83
+ Reg.
Comparison to other methods. We compare our method to FreueecyLooCCut Pooling (FLC) (Grabinski et al., 2022a), Patch Gaussian Augmentation (PaGA) (Lopes et al., 2019), and, on ResNet-20, Blur Pooling (Zhang, 2019), Adaplive Blur Pooling (Zou al., 2020), and Wovelet Pooling (Li al., 2020) (Table 2) these methods also aim Hlf-regularizatiion Regarding AA and CC performance, we observe that our method consistently outperforms these other approaches in standalone comparisons with small degradation of clean validation accuracy. Additionally, imposing regularization top of other methods improve their robustness significantly, as we show for FLC and PaGA in Appendix I. Interestingly, we often get the highest levels of robustness in combination with another method, proving that our regularization can be complementary to other robustness techniques. For a comparison to Wang et al. (2020), please refer to Appendix where we show favorable behavior of our approach.
5.2 ImageNet
Next, we aim to explore how our regularization performs on the common ImageNet datasct (Deng et al., 2009). In particular, more OOD tests exist. for this dalaset which allows us to study aspects outside adver- sarial robustness, and robustness against common corruptions. Similar to our results on other datasets, we see an improvement
adversarial robustness slight (1-2%) degradation of clean performance (Table 4). While we see an improvement CC performance on EfficientNee,, we see an equal decrease for ConyNeXt. This may be due to the larger kernels (7 x 7) that ConvNeXt utilizes and may, thus, require other hyper-
11

---

Published in Transactions on Machine Learning Research (12/2023)
parameters. Importantly we see significant improvement of the eue-conflict in both cases which is also reflected in the increased accuracy of silhovelle (T.F) and the decrease in performance of edge (HF). This indicates that our regularization favorably shifts models toward shape bias (Geirhos et al., 2019).
5.3 Integration into adversarial training and impact robust overfitting
So far, we investigated the effect of our proposed HF regularization on native robustness. For completeness, we aim to explore the rolc of our regularization in AT. We train our models against FGSM-adversaries on CIFAR-10 (L e = 8/255) (Table 5). To avoid robust overfitting, we use early stopping based on PGD-40 test performance. We observe, that our regularization has a beneficial effect on the out-of-domain attacks (Le, AA) and all runs show an increased performance after regularization. We furthermore observe that our regularization appears to mitigate robust overfitting of training attacks (similarly to Grabinski et al. (2022a)) on ResNet-9: without regularivation the AT-trained CNN achieves high FGSM train accuracy and high PGD-40 validation accuracy but fails to generalize to other attacks (AA) and stagnates at 0%. With regularization, all rums show comparable or even better accuracy than the best non-regularized models. However, similarly to the observations by Saikia et al. (2021), we generally see a significant decrease in CC accuracy due to AT. Again, this demonstrates that AT not the cure-all to improve network robustness and there is need for other approaches such as our proposed frequency regularization.
Additionally, extend experiments to AT on ImageNet (Table 6). Here switch to single-step PGD training with the common c <= 4/255 and train the ResNet-50 architecture. Again we report PGD, AA, and CC performance but this time also the cue-conffict score. Our regularized WD architecture outperforms the baseline in all metrics: adversarial robustness, corruption error, and cue-conflict. This demonstrates that our method can mitigate some of the overfitting aspects of adversarial training and leads to an improved OOD gencralization performance. We do not report the (regularized) SD performance due to the lack of tuned hyperparameters but expect similar gains when tuned properly.
6 Conclusion
We have shown a first step towards improving the native robustness of CNNs to multiple distribution shifts such as adversarial attacks, corruptions, and shape-biased datasets, as well as the benefit of our regularization for adversarial training. In particular, our regularization decreases the sensitivity to high- frequency perturbations. Albeit our results do not approach SOTA levels, we emphasize that we improve robustness wide range of tests, whereas SOTA methods like AT often overfit to one specific type of robustness, such as adversarial attacks, and often even impair performance on other tests compared to normal baselines. Additionally, our method does not rely on OOD examples but intrinsically strengthens the model. Our approach has shown to generalize to different networks with various kernel sizes, that were trained on different datasots, and different mcasures of robustness. We have also shown that our method can be used in combination with other approaches such as PaGA (Lopes al., 2019), FLC (Grabinski et al., 2022a), and even AT (Madry et al., 2018) to further improve robust performance. In combination with AT, our approach shows promise to mitigate robust overfitting (Rice et al., 2020).
Limitations. We observed that on some architectures switching to WD/SD introduces a significant drop in accuracy (before regularization). Although the forward pass of both methods is mathematically equivalent to baselines, the backward pass is not. E.g. weight updates on linear combinations of decomposed convolution filters and feature maps are in different backward pass stages and under different quant izaniization conditions due to limited bit precision. While we observe that our regularization generally improves a multitude of robustness aspects, the regularized counterparts may underperform CNN basclines due to the initial impairment due to the architecture change. We aim to explore more rool causes and alternatives in future work.
Acknowledgments
JI. and MK acknowledge support from the DFG research unit 5336 "Learning to Sense".
12

---

Published in Transactions on Machine Learning Research (12/2023)
References
Antonio A. Abello, Roberto Hirata, and Zhangyang Wang. Dissecting the high-frequency bias in convolu- tional neural networks. In IEEE Conference Computer Vision and Puttern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pp. 863-871 Computer Vision Foundation / IEEE, 2021.
Ahmed, T. Natarajan, and K.R. Rao. Discrete cosine transform. IEEE Transactions on Computers, C-23 (1):90-93, 1974.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficient black-box adversarial attack via random search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020. Springer International Publishing, 2020.
Rémi Bernhard, Pierre-Alain Moöllic, Martial Mermillod, Yannick Bourrier, Romain Cohendet, Miguel Soli- nas, and Marina Reyboz. Impact spatial frequency based constraints on adversarial robustness. International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021. IEEE, 2021.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness, 2022.
Karol Cheinski and Pawel Wawrzynski. Det-conv: Coding filters in convolutional networks with discrete cosine transform. In 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020, 2020.
Wai Chen. The Electrical Engineering Hondbook. Academic Press, October 2004. ISBN 978-0-08047748-0.
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen. Compressing convo- lutional neural networks in the frequency domain. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, Sun Francisco, CA, USA, August 13-17, 2016, pp. 1475-1484. ACM, 2016.
Yaosen Chen, Renshuang Zhou, Bing Guo, Yan Shen, Wei Wang, Xuming Wen, and Xinhua Suo. Discrete cosine transform for filter pruning. Applied Intelligence, 2022.
Francesco Croce and Matthias Hein. Reliable ovaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206 2216. PMLR, 2020a.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: D0.109/CVVP.2009.5522
Samuel F. Dodge and Lina J. Karam. A study and comparison of human and deep learning recognition performance under visual distortions. CoRR, abs/1705.02498, 2017. URL http://arxiv. wttpa/awwwi.rrraa//araraaaaaaasa wttg/as///////ee 1705. 02498.
Ranjie Duan, Yuefeng Chen, Dantong Niu, Yun Yang, A. K. Qin, and Yuan He. Advdrop: Adversarial attack to dnns by dropping information. In Proceedings of the IEEF/CVI International Conference on Computer Vision (ICCV), pp. 7506-7515, October 2021.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zissorman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.
wetwork.org/challeeee WVOC////CO1///00/0/ htt.///001////w/ooopo/// workshop/index..eo /index.hml..
13

---

Published in Transactions on Machine Learning Research (12/2023)
Paul Gavrikov and Janis Keuper. CNN Filter DB: empirical investigation of trained convolutional filters. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern. Recognilion (CVPR), June 2022a.
Paul Gavrikov and Janis Keuper. Adversarial robustness through the lens of convolutional filters. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. Workshops (CVPRW), June 2022b.
Paul Gavrikov and Janis Keuper. On the interplay of convolutional padding and adversarial robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp- 3981 3990, October 2023.
Paul Gavrikov, Janis Keuper, and Margret Keuper. An extended study of human-like behavior under adver- sarial training, In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pottern Recognition (CVPR) Workshops, pp. 2360-2367, June 2023.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, 2021.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.
In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Represenlations, ICLR 2015, Son Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv.org haigg/bb/11/0 absp///ss/11........ 657.2.62...
Julia Grabinski, Steffen Jung, Janis Keuper, and Margret Keuper. Frequencyloveut pooling - plug and play against catastrophic overfitting. In Computer Vision ECCV 2022 17th European Conference, Tel Avie, Israel, Oclober 23-27, 2022, Proceedings, Part XIV, pp. 36 57. Springer, 2022a.
Julia Grabinski, Janis Keuper, and Margret Keuper. Aliasing and adversarial robust generalization of cmns. Machine Learning, 2022b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. CoRR, abs/ 1502.01852, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dan Hendryeks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Md Tahmid Hossain, Shyh Wei Teng, Dengsheng Zhang, Suryani Lim, and Guojun Lu. Distortion robust image classification using deep convolutional neural network with discrete cosine transform. In 2019 IEFE International Conference Image Processing, ICIP 2019, Taipei, Taiwan, September 22-25, 2019, pp. 659-663. IEEE, 2019.
Binxiao Huang, Chaofan Tao, Rui Lin, and Ngai Wong. Frequency regularization for improving adversarial nosustness. In Praclical-DL, Workshop, Association for the Advancement of Arlificial Intelligence (AAAT), 2023.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale, 2017.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Yann LcCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit databasc. ATT Labs [Online]. Available: http://yann.lee www.anannee...o...m conneoddm///tooo /ondb/i,,,,, 2, 2010.
Qiufu Li, Linlin Shen, Sheng Guo, and Zhihui Lai. Wavelet integrated cnns for noiso-robust image clas- sification. In 2020 IEEE/CVF Conference on Compuler Vision and Pallern Recognation (CVPR), pp. 7243-7252, 2020. doi: CBB00//CVR42200000000
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Shao- Lo and Hsuch-Ming Hang. Exploring semantic segmentation on the DCT reprosentation. In MMAsia '1: ACM Mullimedia Asia, Beijing, Chino, December 16-18, 2019, pp. 59:1- 59:6. ACM, 2019.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D. Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. CoRR, abs/1906 abs/1906.02611, 2019.
Peter Lorenz, Dominík Strassel, Margret Keuper, and Janis Keuper. Is abbustbenchhaautoattack /aatoattack a suitable benchmark for adversarial robustness? In The AAAI-22 Workshop on Adversarial Machine Learning and Beyond, 2022.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openrrevir http://purreeeeeeeeeeee/e/reeeea nett/forrumr31dd3keRR..CY..
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Shishira R. Maiya, Max Ehrlich, Vatsal Agarwal, Ser-Nam Lim, Tom Goldstein, and Abhinav Shrivastava. A frequency perspective of adversarial robustness. CoRR, abs/ 201/2111000861,, 2021. URL https: // //arxiv.. org/abs/2111.00 00861.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised fcature learning. NIPS, 01 2011.
Guillermo Ortiz-Jiménez, Apostolos Modas, MohsenMohsen Moosavi- Moosavi-Dezfooli, and Pascal Frossard. Hold me tight! influence of discriminative features on deep network boundaries. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Informalion Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville, On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robust- ness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017. URL http::///axxi..o/aaa//////
04131.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In Hal Daumé 111 and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8093-8104. PMLR, 13-18 Jul 2020.
Daniel L. Ruderman. The statistics of natural images. Network: Computation In Neural Systems, 5:517 548, 1994.
Tonmoy Saikia, Cordelia Schmid, and Thomas Brox Improving robustness against common corruptions with frequency biased models. In 2021 IEEE/CVE International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 10191-10200. IEF., 2021.
15

---

Published in Transactions on Machine Learning Research (12/2023)
Nandini C. Singh and Frédéric Theunissen. Modulation spectra of natural sounds and ethological theories of auditory processing, The Journal of the Acoustical Sociely of America, 114:3394-411, 01 2004. doi: 10.1121/L.162406.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise, 2017.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. 2019. doi: 10.48550/ARMN ARXIV 1995.11946. URL https://arxiv org/abs/1905. 11946.
Florian Tramer and Dan Boneh. Adversarial Training and Robustness for Multiple Perturbations. Curran Associates Inc. Red Hook, NY, USA, 2019.
Matej Ulieny, Vladimir A. Krylov, and Rozenn Dahyot. Harmonic convolutional networks based discrete cosine transform. Poltern Recognition, 129:108707, 2022. ISSN 0031-3203.
G.K. Wallace. The jpeg still picture compression standard. IEEE Transactions on Consumer Electronics, 38(1)xxvii xxxiv, 1992.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advonces in Neural Information Processing Systems, pp. 11606-10518, 2019.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High-frequent component helps explain the generalization of convolutional neural networks. In 2020 IEEE/CV /CV Conference on Compuler Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. Computer Vision Foundation / IEEE, 2020.
Eric Wong, Leslic Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Leurning Representations, 2020.
Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In 2020 IGEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seatile, WA, USA, June 13-19, 2020, pp. 1737-1746. Computer Vision Foundation / IEFE, 2020.
Leonid Yaroslavsky. Fast transforms in image processing: Compression, restoration, and resampling. Ad- vances in Flectrical Engineering, 2014:1 07 2014. doi: 10.1155/2014227621.
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancoucer, BC, Canada, pp. 13255-13266 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? volume 4, 2014.
Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Understanding robust overfitting of adversarial training and beyond. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 Proceedings of Machine Learning Research, pp. 17-23 Jul 2022.
PMLR,
Richard Zhang. Making convolutional networks iift-invariant again. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 7324-7334. PMLR, 09-15 Jun 2019. URL https: //proceedings.... .mw.ainng.m.......mmmmmm..m..m prtps/ /ttps//199//aa111a.... html.
/
Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. In British Machine Vision Conference (BMVC), 2020.
16

---

Published in Transactions on Machine Learning Research (12/2023)
Organization of the Appendix. In the following, provide additional information and details that accompany the main paper. We first describe the broader impact of our method. Secondly, describe the details of training setups for the different models and datasets. We give more information on the results of common corruptions, i.e. CIFAR-IO-C. We provide an overview the used ResNet-9 architecture. Further, we visualize both our proposed decomposition approaches, weight decomposition and signal decomposition. Then, we include all hyperparameter exploration results for both our methods and further motivate our regularization approach by ablating two other regularization types. We show results of adversarial training on ImageNet and additional comparisons to other robustness techniques and show the impact of corruptions and adversarial samples in the frequency domain. Also, we include details on the shape bias visualization. Next, we provide detailed plots on the coefficient distributions of clean-trained and adversarial-trained networks, e.e. ResNet-20, ResNet-9, and Net--0, over all layers and epochs. Then, we include a detailed version of Table in the main paper. We visualize and interpret the attributions before and after regularization, and show the Fourier spectrum of activations (and their differences). Finally, we provide thcoretical motivation for our proposed regularization based on the frequency spectra of shape signals.
Broader impact in Appendix A
Training details in Appendix B
Details on corruption experiments in Appendix C
ResNet-9 architecture in Appendix D
Architecture diagram of weight/signal decomposition in Appendix
Hyperparameter exploration in Appendix F
Adversarial training ImageNet in Appendix G
Ablation with other regularization approaches in Appendix H
Comparison to other robustness techniques in Appendix I
Analysis of corruptions and adversarial samples in the frequency domain in Appendix J
Details on shape bias Appendix K
Evolution plots in Appendix
Detailed table including standard deviation in Appendix M
Qualitative analysis of attribution maps in Appendix N
Fourier spectrum of activations in Appendix O
Background on the link between shape and low-frequency bias as motivation for our proposed reg- ularization in Appendix P
A Broader Impact
Arms race: Adversarial training, which is common technique to enhance model robustness, involves introducing worst-case perturbations into the training data. However, the very same tech- niques used for adversarial training can also be employed by malicious actors to craft adversarial attacks. As AI models become more robust to conventional adversarial attacks, adversaries may develop more sophisticated and potent attack strategics. This arms race in adversarial techniques could potentially lead to an escalation cyber threats, with negative consequences for cybersocurity and data privacy.
17

---

Published in Transactions on Machine Learning Research (12/2023)
Native robustness: In theory, could attempt to characterize all possible perturbations of train data and perform adversarial training on those. However, characterizing all possible perturbations is already very hard for specific problems and potentially insolvable for vision in general (e.g., all the possible perturbations that could happen in autonomous driving). This problem hasn't been solved to date. Further, even if we were able characterize all possible perturbalions, we could only train on a discrete set, which could only approximate the continuous search space and, thus, risk overfitting. Lastly, to effectively integrate this into current training approaches, we would have to significantly increase the training time to compensate for the new data. In contrast, by native robustness, we seek to add a low-frequency prior to the network to steer the training to a more generalizable representation without additional (perturbated/ (aaaatttddduumnneeeeee data. Still, we remain compatible with these methods and even adversarial training. In Section 5.3 we even show a reduced risk of robust overfitting after regularization.
Ethical and bias: While the paper's s approach priorilizing low-frequency information decision- making can mitigate certain types of bias, it crucial recognize that bias generally multi- faceted issue. Shifting the focus toward low-frequency information may introduce its own set of biases. For example, in computer vision, prioritizing shapes might lead to underrepresentation misrepresentation of fine-grained details, which can have implications in fields like medical diagnos- tics and object recognition. It is essential to carefully consider and address these potential biases to ensure fair and equitable AI systems.
B
Training details
B.1 Low resolution: CIFAR-10/106 MNIST, SVHN, Tiny-lmagenet
Training setup. We train models for all low-resolution datasets with the same hyperparameters. Models are trained for 120 epochs. For both ResNets we use an SGD optimizer (with Nesterov momentum of 0.9) with an initial learning rate of le-2 that we downscale by 0.1 every 30 epochs, and weight decay of le-2. For Efficient.Ne-BOO
AdamW (Loshchilov & Hutter, 2019) optimizer with initial learning rate of le-4 that follows cosine annealing schedule, and weight decay of 5e-2. In all cases, use batch size of 256, and categorical cross entropy as loss function with our regularization. We analyze models with weights learned after the last gradient update.
We use the following augmentations for datasets:
Training images are zero-padded by 4 px along each dimension, apply random horizontal flips, and procced with 32 x px random crops. Test images are not modified.
CIFAR-10/100:
Tiage Training images are obtained using randomly resized 56 px crops. Test images are 56 56 px center crops.
MNIST: Train and test. images are upscaled to 32 32 px.
SVHN: Train and test images are not modified.
For all datasets, samples are normalized by the channel mean and standard deviation.
B.2 ImageNet.
We train all ImageNet models with the default hyperparameters and angmentations for ConyNeXt-Ting (Liu et al., 2022). In particular, we train 300 epochs with an effective batch size of 4096. For EfficientNet-Bll we reduce the batch size to 1024 due memory constraints. Again, we evaluate model parameters learned after the last gradient update.
18

---

Published in Transactions on Machine Learning Research (12/2023)
C CIFAR10-C details
In this section we provide detailed information about the the corruption results on CIFAR-10-C (Hendrycks & Dietterich, 2019) in Table 7.
Table 7: Results (in % and averaged over 5 runs) CIFAR-IO-C for ResNet-20, ResNet-9 and EffcientNet- BO. Reported is the mean over all severities and corruptions and all corruptions at severity level 5.
(f)
Accuracy
Goussian
Shot
Imopulse
Defoeus
Clase
Motion
Zoom
Snow
Prost
Fog
Brightuess
CNN
67 96
23.77
28.61
52.01
51.67
36.31
56.85
56.99
65.77
50.60
62.74
85.5d
20.06
27.02
64.38
85.63
ResNet 20
SD
67. 18
20:31
20.60
27 27.09
52.37
37.85
56.75
56.62
66.24
51.23
63.66
85.96
WD Rog. (A 0.06, 3)
74.04
34.40
25.13
64.51
73.90
70.38
60.49
84.52
Reg. (A 0.01.
74.14
31.55
37,28
25.09
63.91
54.21
05.01
71.05
74.28
00.51
62.38
85.17
CNN
73.38
27.16
34.61
24.02
53.98
47.80
64.43
62. 64
75.08
63.46
07.74
80.48
67.67
ResNet-9
73. 48
25.36
35.52
27.47
57.91
45.10
65.93
03.20
75.29
04.31
68.32
88.69
WD Reg.
1 0.01
26.41
32.77
38.91
25.21
61.58
60.63
87.13
21.71
79.12
25.45
66.25
87.85
Reg.
77-72
04.71
65.65
80.68
CNN
68
25. 02
20.04
20.41
43.76
47.17
52.34
63.
65.74
54.80
54.81
84.06
WD
67. 10
18.56
22.33
25.02
68.55
41.17
55.67
58.24
67.11
56.59
58.52
63.96
ElficienNNet-B0
17.59
21.15
21.70
47.46
44.97
55.47
53.16
57.59
84.02
WD Reg. (A 0.01
72.14
28.06
33.15
21.30
61:23
50:35
62.72
08.09
73.79
08
52.39
83.10
61.23
50.22
Varlant
Digital
Extra Gauseion Bluz
Coutrast
Elastic Transform
Pizelate
JPEG
Compressioo
Speckle Noise
Spatter
Saturate
26.18
64.51
34.30
35-35
60:75
WD
25.54
64.21
37.42 /2
66.84
29.57
30.21
70.87
90.97
RosNot-2U
SI)
36.73
WD Reg. () 0.05.
23.63
71.65
81.36
76.15
48.227
38.97
74.83
78.53
SD Reg. (A 0.01, =
22.10
71.43
80.04
76.23
41.07
57.60
75.24
70.91
26.25
76.
30.51
74.78
72.83
WD
25.02
75.09
19.01
73.59
$0.04
45.16
73.29
62.65
RosNet-9
SD)
45.00
WD Reg. (A 0.01. -
24.10
78.52
77.45
70.45
42,94
26.66%
70:25
81.16
SD Reg. (A b
24.24
78.58
81.09
80.67
45.90
59.61
60.49
$1,28
CNN
20.16
70
76066
73.33
33
$5.53
73,
81.92
WD
19.34
69:50
47.27
72.11
28.27
37.72
72.20
80.45
ElficientNei-BB
SD
WD Reg. (1 0.01. A 3)
18.75
73.92
26.93
77.89
37.34
04.08
70.21
78.72
SD Reg. (A 0.01.h33)
17.37
74.41
76.30
77.85
50.56
51. .9
76.72
78.10
19

---

Published in Transactions on Machine Learning Research (12/2023)
D ResNet-9 architecture
In this section, we show the architectural topology of the ResNet-9 (Figure 7), used in the main paper in Section 5.
3xd Cone BM RALU
3e Cane BN RELu
MaPPoll Sw2
3/ Cone BN RALU
MaxPool 6m2
2x MosPool 5-2
30 Conv BN RALU
3:3 ReLLU
12B
2598
512
512
612
512
Figure 7: Architecture ResNet-9. The numbers on the arrows indicate the number of channels. The initial and final number of channels depends on input data and the number of classes.
E Decompositions
In the following, we visualize both convolution decomposition approaches as presented in Section Figure 8 shows the weight decomposition approach and Figure the signal decomposition approach.
coefficients
k2 basis
Gn
Cout
Figure 8: Flow of the weight decomposition implementation. Instead learning the weight directly, we learn coefficients of DCT-II basis and construct the weight. via linear combination.
On
pointwise Ieamed coerricents
depthsise coneciutinn with Soed besis
Figure 9: Flow of the signal decomposition implementation. Each input channel is convolved with all basis vectors in a depthwise convolution layer. The outputs then aggregated by pointwise (1 1) convolution.
20

---

Published in Transactions on Machine Learning Research (12/2023)
F Hyperparameter exploration
In the main paper, only included the results using h = 3 regularization depth and À = 0.05. For the sake of completeness, we further provide additional results on other regularization hyperparameters. In the following, we compare different regularization hyperparameters and evaluate them on the clean accuracy in CLEAR-10 (Krizhevsky, 2009) and against adversarial attacks, FGSM (Goodfellow et al., 2015), PGD-40 (Kurakin et al., 2017) and AutoAttack (Croce Hein, 2020a) as well as against corruptions in CIFAR-10-C (Hendrycks & Dietterich, 2019). The evaluation setting the same as in the main paper in Section 5. Table 8 presents overview of different regularization depths for ResNct-20 (He et al., 2016), from regularizing only the first eighth layers (the first 3 layers in this case, / 8) to regularizing the complete network (h-1). Note, h = 2 regularizes half of the network. As we can see, regularizing more than half of the network leads to a decrease in clean validation accuracy, with no overall benefit to robust accuracy. Whereas regularizing only within the first half of the network leads to the trade-off slightly decreasing the clean accuracy compared to the non-regularized versions, but substantially increasing the robust accuracy, which we aim for in this paper. Also, our default setting in the main paper (h 3) leads overall to the best robust accuracy.
In Table 9 and Table 10 provide results CIFAR-10 and CIFAR-100 for ResNet-20, ResNet-9 (He al., 2016) and Efficient (Tan & Le, 2019) with different regularization hyperparameters:: X as the loss parameterization Equation 8, has the network depth regularization (first 1/h is regularized) and the binary paramcter Paiat Equation 7. The evaluation on clean accuracy and adversarial robustness in the same as in the main paper in Section
Table 8: Ablation of ResNet-20 with different regularization depths on CTFAR-10 and evaluation against adversarial attacks, FGSM, PGD-40, AutoAttack, and corruption CIFAR-IO-C. Reported is the mean accuracy in % and standard deviation over runs.
Varlant
Clean Val Acc.(1)
FCSM < 1/256 (1)
PCD-40 < 1/256 (1)
AA 1/255(1)
Moan Accuracy CIFAR-10-C (*)
CNN WD
91.29 + D.14
50.49 + 0.62
30.92 0.88
10.78 + 0.33
67.96
91.01 i 0:21
46.40 + 0.36
30.37 0.98
10.72
0.91
66.92
0.23
WD Reg. (A - A
80.86
0.14
50.85 + 0.71
41.81 1.15
26.70
1.52
74.04
SD Reg (A - 0.03,
90.34 + 0.14
53.07 0.29
44.04
0.56
28.62
0.96
73.98
WD Reg. (A = 0.05. =10
88.84 + 0.08
48.99 + 0.90
40.67 + 1.34
26.19 = 1.52
73.96
WD Reg. (A 0.05, 1.2)
88.78 + 0.21
48.46 + 0.57
39.98 0.86
25.77
1.30
73.56
Reg.
1.4)
WD + Reg. (X 0.05. 1.6)
88.77 1: 0.32
48.56 1: 0.94
40.57 1.22
26.20 =: 1.09
73.94
WD Reg. (X 0.05, 1.8)
88.03 + 0.32
48.42 + 0.90
39.42 1.58
24.74
2.55
73.39
Reg.
WD + Reg. (X 0.05. 4)
90.13 1 0.10
51.20 1 0.75
41.85 1.15
26.06
1.30
72.05
WD - Reg. (X = 0.05. A.
90.05 + 0.51
50.87 + 0.52
41.17 2 0.66
25.06 = 0.69
71.79
WD Reg.
A
0.21
50.45
39.65 0.89
22.82
70.88
Reg. (A 0.05,
80.26 0.32
49.48
0.35
41.23 0.43
26.61
74.32
SD Reg. (A = 0.05, h 1.2)
89.09 + 0.21
50.11 + 0.74
42.10 + 1.14
27.75 = 1.86
74.35
SD Reg.
1.4)
0.22
50.54 0.53
42.19 1.06
1.29
74.34
Reg.
1.6)
0.27
SD Reg. () = 0.05. A 1.8)
89.52 1 0.34
51.25 + 0.95
43.04 1.49
28.34
2.09
74.67
81) Reg. (A 0.05, A 2)
89.32 + 0.13
50.90
0.68
42.60 1.34
27.76
1.68
74.74
Reg.
SD Reg. () = 0.06.
90.61 4 0.25
52.35
4 0.63
42.46
0.77
25.89
0.63
71.88
SD Reg. (A = 0.05. 1
90.63 1 0.05
51.92
1 0.98
40.82 4 1.21
23.43
1.18
71.15
21

---

Published in Transactions on Machine Learning Research (12/2023)
Table 9: Ablation Results (in % and averaged over 5 runs) ResNet-22,, ResNet-9, and Neticient trained with different convolution implementations and varying regularization on CIFAR-10, and evaluation against adversarial attacks, FGSM, PGD-40 and AutoAttack and corruptions in CIFAR-10-C.
Variant
Val (RA())
PGD-40
1/288(1)
CIFAR-10-0
WD
0.21
48. 00.00000
0.58
0.91
WD Bog. (A 0.01
59.00
0.27
48.55
0.70
30.77
1.02
25.00
1.21
13.55
WD Reg. (A
65
0.33
48 03
0:42
10.30
0.98
25.95
1:37
73.97
WD Reg. [A 0.01
3.P Q)
10.56
0.53
60.74 0.67
41.00
0.00
25.56
1.23
73.60
WD lin. 0.01
0.79
Beg. 0.00
WD
WD
0.35
WD Reg. (A 0.15
89.00
0.14
50.
0.42
41.11
0.52
25.88
0.50
74.
ResNet-20
WD Reg. (A 0.1.A.A.. m.
50 65
0.13
00.00 65
0:30
41 57
0.40
95.33
0.77
75 23.65
SD)
11 36
0.233
50 85
98
1/00
11.97
0.52
07
Rox. [A 0.016 1, 1.g 0)
59.14
0.10
40.5 67
40.00
1.55
25.55
1.59
75
Bey [A
0.01.
0.05,
03
44.00
0.56
SD Bax [A 0.1. 1,
89.32
40.3
0.01
40.68
1.35
25.73
1.50
74.14
8D Reg [A 0.1.M
70
0
50 06
0.75
11 90
1/00
27 26
1.008
74.20
SD Rox. [A 0.1.
5.005
90.32
0.20
62: 0.48
42.64
0.60
27:11
0.57
75.75
Bay [A 0.1,
0.144
59.5
13.90
27.75
71.00
CNN
94-20 d: 0.00
50 68 0.41
53.00
0.50
57.49
0.53
73.55
55.51
72.87
WD 0.01.
06.
1.29 67.
SD+ Rog.
0.05, 83
92.41
0.24
53. 71
46.79
2:7.77
78.81
CNN
00.38
0.30
59.55 0.78
$4.05
1.44
45. 2:1.9
85 08.00
W1)
90.51
0.34
40. 87
1.59
49 97
2.44
40 76
2:3.27
07.
SD
00.44
0.11
61.04
0.80
54. 77
1:25
45.30
2:1.0
06
WD Rog, (A 6
$5.07
0.51
ST.91 2: 0.06
60
1.01
68.3721.10
72.14
WD Tee.. (A
56.
51.
0.46
Bag. (A 6.05,
88.08
0.35
58.72
58.2
2.21
51.012 2.61
74.12
22

---

Published in Transactions on Machine Learning Research (12/2023)
Table 10: Ablation Results (in % and averaged over 5 runs) of ResNet-20, ResNet-9, and EfficientNet- BO trained with different convolution implementations and varying regularization on CIFAR-100, and evaluation against adversarial attacks, FGSM, PGD-40, and AutoAttack.
Variant
Clean Val Acc. in (961(1)
FGSM 1/255 (t)
PGD-40
1/255(1)
AA
1/255 (1)
CNN
60.41
14.36
5.45
0.10
Reg. 0.01
Pug
55.31
-046
12.65
0.45
0.38
WD Reg. (A 0.01,
55.23
0.26
18.75
0.49
13.01
0.60
5.93 0.34
WD Reg. LA 0.01.
RAG 0)
56.50
0.25
16.10
0.72
11.63 = 0.97
4.78 # 0.58
WD Reg. (A 0.01. =
56.50
0.27
16.01
0.38
11.59 + 0.42
4.96
0.31
WD Reg. (A 0.85. h-
55.55
0.16
17.03
0.46
13.26 1 0.48
6.07 0.46
WD Reg. 0.06,
==33)
56.50
0.29
16,00
0.60
12.39
0.56
5.28
0.28
WD
0.1.
55. 41
0.13
16.64
12.71 0.94
5.69 0.72
0.57
Beg.
3.
0.35
RisNNet-20
Reg. (A 0.1,
12.73
0.4G
SD
60.34 0.44
13.87
0.18
5.18 0.09
1:13 0.05
SD Regg (A 0.01, h Pay 0)
56.32 0.26
17.31 - 0.37
13.25 2: 0.48
5.53 0.39
SD> Reg. (A = 0.01, 6
56.88 0.50
17.40 + 0.38
13:41 0.60
6.03 0.48
SD Reg. (A 0.01, h
3. Pag
57 0.31
17.25
0.23
12.47 0.74
5.24 0.50
8D3 Heg. 0.05,
0.03,
3)
08.19 0.23
17.20 0..43
12.24 0.51
5.11 0.27
ST
56.71 0.24
17.74 0.33
13.73 0.32
0.27 0.52
Reg.
Pdg
Hog.
0.87
0.44
ST> Reg.
PMg
58.32 0.59
17.10
0.38 12.03
12.46
5.03 0.32
SD Reg. (A 0.1.
58.17 0.41
17.00
0.35
0.60
5.01 0.47
CNN
75.80 0.28
30.41
24.12
0.27
10.
29.97
23.68
Beg.
0.29
24.11
Reg.
0.05
0.23
33.76
26.72
WD Beg (A 0.1.
59 0.28
28
25.43 0.40
13:04
0.38
WD Reg. (A 0.1.
72.7 74 0.17
30.02
0.19
26.52 -: 0.29
14.71
0.36
Re-Net-0
ST)
76.06 0.28
30.73
0.16
24.65 0.32
11.48
0.37
S13 Reg, 0.01, 1)
74.00- 0.27
31.97 0.18
28.64 0.57
16 0.29
WD Reg 0.01.
74.75 0.31
33.66
30.31 0.30
17.62
SD
0.32
0.17
Bog.
0.28
ST>
Reg.
0.1,
0.24
31.07
27.20
15.08
CNN
62.09 0.75
28.33 1.07
24.99
2.70
18.76
3.57
WD
63.49 0.32
23.35
18.13
2.33
2.48
26.35
0.01.
Beg.
0.05.
Reg. (A 0.1.
0.29
26.35
21.
WD Reg. (A 0.1. A
46 0.07
27.27 + 0.53
27.55
0.57
22 57
0.69
Net-iintNeet-O0
ST)
62. 85
0.20
25. + 0.30
20.73
+: 1:95
13:21
2:27
SD Reg. (A 0.01,
24.24 0.44
28.0
0.62
24.70
3: 0.78
23. 43
1.19
WD Reg. = 0.06, J
29.21 0.42
27.44
01.91
27.73
1.03
22.93
0.97
SD Reg, 0.01,
0.44
1: 0.61
1: 0.80
1.21
27.79
Reg.
0.27
22
Reg.
0.1,
59.72
0.35
28.93
29.36
0.92
24.73
23

---

Published in Transactions on Machine Learning Research (12/2023)
G Adversarial Training on ImageNet
We show the results of our regularization on a ResNet-50 trained on ImageNet with adversarial training Ta- ble 11, based on the implementation in attps://githiu is execuled against a single-sep ,,8, e 4/255 PGD adversary. Our regularization improves both, clean and robust performance against nou-regularized baselines. On ImageNet, we observe better results with WD for the same regularization hyperparameters.
torch. Training
Table 11: Performance of Adversarial Training of a ResNet-50 trained on ImageNet.
Variant
Clean Val Acc. (t)
Adversarial Ace. (t)
PGD c= 4/255
AA 4/255
CNN
56.85
36,01
22.33
56.50
35.96
18.66
SD Reg. 0.05, 3)
57.61
35.85
22.25
WD
WD + Reg. (A 0.05, A 3)
58.09
37.09
24.32
Reg.
0.01,
57.22
35.75
WD Reg. (A 0.1, 3)
56.99
36.17
21.10
WD + Reg. (A 0.2, A 3)
57.45
36.52
24.03
WD Reg. 0.05,
54.72
31.94
19.94
H Extended Ablation: Other regularization approaches
Table 12:
Ablation on the hierarchical regularization approach Efficient.Net-BO trained with differ- ent convolution implementations CIFAR-10 and evaluation against FGSM and PGD attacks. Results in % and averaged over 5 runs.
Variant CNN
Clean
Adversarial Acc. (f) PGD-40 1/255
Val Ace. (t)
FGSM = 1/255
90.38 + 0.10
53.55 = 0.76
54.05 = 1.44
WD
90.51 + 0.34
49.87 = 1.59
49.97 : 2.44
SD
90.44 0.11
51.01 0.89
51.77 1.25
WD Reg. (A = 0.05, A 3)
88.27 + 0.11
56.78 = 0.69
58.35 = 0.86
Reg. ()
hierarchical)
0.12
SD Reg. (} 0.05, à 3)
88.08 + 0.35
56.72 + 1.86
58.21 + 2.21
SD Reg. (A 0.05, h 3, hierarchical)
90.36 + 0.43
52.21 = 0.57
53.13 = 0.54
Ablation. In order further support the regularization approach, we included an ablation with different regularization (Table 12). Here, instead of regularizing the coefficients belonging to the highest frequencies, impose penalty that enforces a hierarchy of frequencies based on magnitude starting with the highest magnitude on the lowest frequency. This regularization has slightly positive impact on the robustness compared to its non-regularized counterpart but does not suppress higher frequencies. Therefore, it is intrinsically limited in improving robustness. We compare the rogularization as described Scction 4.1 with regularization approach, which ignores the first part. of Equation 7 but rather forces the lowest frequencies to have a higher magnitude than subsequent frequencies for all frequencies, i.c. frequency Co.0 has higher frequency magnitude than f 11,13 and further {co.1 , C1,13 has higher magnitude than (co,2, C1,2, 2,,1, 22,2) and Table 12
We denote this type of regularizaton with pasp- As we can see in
In this section, we additionally compare our proposed regularization with two other regularizations. First, to motivate the regularization of high frequencies, we regularize inverse to our original approach: instead of regularizing high frequencies, we regularize the low frequencies. In the case of kernel size k 3 the penalty becomes:
Pinverse (C) = C..oool2 2 max Σ (IC...sll2)
(9)
(1=))
Second, we observe that some layers learn predominantly coefficients in the first row and column. To further investigate this observation, we regularize the lower quadrant, i.e. keeping the frequencies in the
24

---

Published in Transactions on Machine Learning Research (12/2023)
Ablation of ResNet-22 with different regularization types on CIFAR-10 and evaluation against adversarial attacks, FGSM, PGD-40, AutoAttack, and corruption CIFAR-10-C. Reported is the mean accuracy in % and standard deviation over runs.
Table 13:
Variant
Clean Val Ace. (f)
PGSM c 1/255 (t)
PGD-40 <= 1/255 (t)
1/255(1)
Mean Accuracy
CIFAR-IO-C (1)
30.92
91.04
0.21
48.10 1: 0.36
30.37
0.98
10.72
0.91
68.92
WD Reg. [A 0.01,
86.01 + 0.37
48.03 1: 0.42
40.50 +: 0.98
25.94 2: 1.37
73.97
WD- Reg. [A 0.01.
90.02 0.22
50.38 + 0.37
41.34 + 0.79
25.86
1.07
73.43
[A 0.06,
0.27
Reg. [A 0.05.
0.14
1.15
1.62
WD Rog. [A 0.1.
88.91 0.15
49.16 1: 0.95
40.95
1.56
26.39
1.84
73.84
WD Reg. (A 0
89.85 1::55-100 0.12
50.65 1: 0.39
41.57 1 0.40
26.33
0.77
73.85
SD
91.36 0.23
50.83 + 0.49
32.98
1.06
11.97
0.92
67.48
SD Reg. (A 0.01,
42.61
Reg. (A
0.14
63.12
0.63
SD Reg. (A 0.06,
89.20 + 0.24
49.50 + 0.74
41.05 + 0.89
26.28
1.04
74.24
SD + Reg. (A 0.05,
90.34 1: 0.14
53.07 +: 0.29
44.04 + 0.56
28 62
0.56
73.95
SD (A 0.1.h (1)
89.29 0.25
50.06 1:
41.90
1.03
27. 58
1.08
74.29
Reg. (A
WD Reg. [A 0.01.
3, laverse)
90.45 + 0.22
40.09 + 0.85
23.85 + 1.13
6.22 = 0.75
05.41
WD
A 0.05,
lnverse)
0.14
03.95
20.39
0.42
Reg.
inverse)
Reg. À 0.01,
3, quadrant)
90.50 0.27
51.70 76 +: 0.87
2.05
24.37
2.96
71.02
WD Reg. A 0.06, à 3, quadrnot))
90.06 0.20
51 93 + 0.40
40.20 1.10
24.08
1.26
71.55
WD Reg. ( 0.1,
3. quulrant)
90.10 0.17
01.14 + 0.42
40.81 0.79
24.79
1.05
71.62
SD (A
laverse)
Reg. ()
lnverse)
0.12
21.81
SD + Reg. =A= 0.1.h = 3, inverse)
90.56 0.45
44.10 1: 0.97
20.17
1.18
3.60
0.42
64.05
SD- Reg. (A 0.01, h = 3, quadrant)
90.67 1 0.06
53.95 4: 0.49
43.04
+: 0.77
27.23
1.19
72.20
SD Reg. (A 0.05, 3, quadrant.)
90.49 0.11
52.08 + 0.54
42.31
1.21
25.65
1.77
72.22
SD Reg. (A
quadrant)
42.06
first column and first row (c,0, Co.i; for i k)), with being the kernel size, quadrant. This way do not regularize all coefficients corresponding to high frequencies. As a result, quadrant suppresses high frequencies diagonal structures but allows certical/horizone regularization enforces regularization independent of the orientation.
high frequencies, while our original
We present the results in Table 13 on ResNet-20, The inverse approach does not increase robustness, quite contrary even decreases the accuracy under adversarial attacks and CIFAR-10-C, which strengthens the formulation of our original regularization. The quadrant regularization increases robustness but performs subpar compared to our original approach.
Comparison-State-of-thee
including Wang et al.
In the main paper, we compared our method to other robustness techniques such as FrequencyLow Pooling (Grabinski et al., 2022a) and Patch Gaussian Augmentation (Lopes al., 2019) in Section 5.1. In Table 14 we additionally compare our proposed method to Smoothing Kernel (HFC) proposed in Wang et al. (2020). However, in contrast to the results in Wang et al. (2020), we do not see favorable behavior of this method, but a decrease of clean validation accuracy and robust accuracy evaluated against PGD-40, FGSM, and AutoAttack. Since we can not replicate their results, we have reached out to the authors but have not received a reply. Furthermore, Table 14 also demonstrates, combining other robustness techniques with our proposed regularization even further increases the robustness.
Table 14 also provides the standard deviation over 5 runs to the tables Table 1 and Table 2 from the main paper.
25

---

Published in Transactions on Machine Learning Research (12/2023)
Comparison results (in % and averaged over runs) of ResNet-20,, ResNet-9, and EfficientNet- BO against other robustness techniques (Grabinski et al., 2022a; Lopes et al., 2019; Wang al., 2020) on CIFAR-10. We report the mean clean validation accuracy and robust accuracy against adversarial attacks: FGSM, PGD-40, and AutoAttack for L ; 1/255, and the mcan corruption accuracy CIFAR-10-C. The proposed regularization improves over other robustness techniques all models.
Table 14:
Variant.
Robust Acc. 1 PGD.40
1/256]
Corruption Moan Acc.
Clean Val (cc.(f)
FGSM
CNN
91
0.14
50.49 + 0.62
30.92
0.88
10.78
+ 0.33
67.96
CNN FLC (Grsbinski al, 2022s)
91.52
62.48
5.48
0.74
68.75
CNN PaGA (Lopes ul., 2019)
91.29 0.35
50.36
1.00
31.50
1.82
11:38
1.00
67.73
WD
91.04 0.21
18.40 0.36
30.37
0.08
10.72
0.91
66.92
WD FLC
96 0.17
1.36
0.51
WD) PACI
91.01 0.18
48.61
0.53
31.44
1.29
11.72
1.45
67.87
ResNet-20
8D)
91 36 0:23
50.83
0.49
32.98
1.0G
11.97
0.92
67.48
SD FLC
91 42 0.14
02.99
0.43
33.21
0.61
9.32
1 0.29
08.44
SD PaGA
91 0.27
50.68
0.84
1.20
10.91
1.09
87.71
WD Reg
0.14
50.65
0.71
1.15
26.79
1.52
WD Reg FLC
80.65 0.24
52.21 0.48
42.02
1.03
25.39
1.69
75.28
WD Reg PAGA
89.84 0.15
50.09 0.64
40.92
1.24
25.70
1.80
73.65
8D + Reg.
90.54 0.14
63.12 0.34
44.42
0.63
29.14
1.16
74.14
SD
FLC
90.37 0.54
3.56
Reg PaGA
90.32 0.15
52.47 0.89
1.35
27.30
1.30
73.19
CXN
94.29 0.09
59.58 0.41
53.04
0.59
37.40
0.53
73.38
CXN FLC (Grabinski at al, 2022a)
0.14
69.64
0.36
PaGA (Lopes al., 2019)
94.33 0.17
09.12
0.00
52.62
0.61
17.50
0.48
735.72
CNN + HFC (Wang et al., 2020)
87.62 0.34
0.87 47.27
42.41
1.03
29. 16
1.08
32.85
WD
0.06
0.33
0.53
WD
91. 0.37
00.52
0.23
50.07
0.42
35.53 0.25
733000
WD PaGA
91.71 0.07
55.62
0.62
50.17
0.64
35.50
0.59
73300
ResNet.9
SD
90.97 1 0.15
55.73
0.56
50.29
1: 0.55
36 38.02.1
0.86
73.48
SD FLC
93 93.98 0.10
56.07 0:30
0.44 50.73
36. 36
0.87
73.41
PaGA
WD Reg.
93.18 0.39
69.25
2.24
56.08
43.62
4.53
76.41
WD Reg FLC
93.20 0.19
59.64
0.68
56.39
0.84
44.24
1.09
76.78
WD Reg PaGA
92.15 0.22
56.48
0.80
52.64
0.58
39.78
1.23
78.14
SD Res.
93.09 0.30
59.87
1.29
56.89
1.79
44,81
2.29
77.72
Reg FLC
0.35
SD + Reg PaGA
93.1 0.40
1.21 69.86
57:24
1.49
45.00 46.06
1.62
77.64
CXN
90.38 0.10
53.35
0.76
54.05
1.44
45.51
88.09
FLC (Grabiaski et al., 2022a)
0.28
51.92
0.93
1.10
69.72
CNN + PaGA (Lopes et al., 2019)
90.72 0.33
54.18
0.71 42.40
54.97
1.29
46. 64
1.52
69.31
CNN HFC (Wang et al., 2020)
79.96 4: 1.30
1.90
43.85
2.03
37 344
1: 1.94
22.80
WD
90.51 0.34
1.59
67.10
WD FLC
89 80 0.23
46.29
1.30
46.68
1.70
37.32
2.21
68.54
WD PaGA
90 + 0.21
48.7T
0.84
49.14
1.50
39.50
2.46
67.89
Net--io
80)
04 0.11
51.04
0.89
51.77
1.25
43.30
1.63
66.65
8D FLC
47.91
1.62
67.61
PAGA
90.91
50.92
0.63
51.74
0.58
42.65
0.75
68.13
WD Reg.
88. 97 0.31
57.91 1: 0.98
59.60
1.01
53.37
1.19
72.14
WD Reg FLC
87 67 0.32
52.68
2.67
54.66
2.63
48.14
3.58
71.17
WD Reg PaGA
88 49 0.20
55.15 0.77
56.70
0.71
50.05
0.85
74.16
Reg.
0.52
57.83
0.33
59.68
0.44
53.50
0.59
71.87
8D + Reg FLC
87. 0.32
54.15
1.03
0.97 58.56
56.38
50.31
1: 1.25
71.10
SD + Reg PaGA
80.00 66 : 0.22
56.72 4 0.79
0.56
51.87 + 1.13
72.39
J Perturbations in the frequency domain
In this section, we visualize common corruptions and FGSM attacks in frequency space. Therefore, we compute the average error in the FFT spectrum between the attack images and their clean counterparts.
Common Corruptions. Figure 10 shows an example image of the CIFAR-10-C' test with various cor- ruptions while Figure 11 shows the error of these corruptions in FFT space (as already shown in Yin et al. (2019)). Corruptions affect the frequency spectrum differently e.g. pizelate affects only high frequencies, brightness only low frequencies, and others like Gaussian noise affect the entire spectrum.
FGSM. Figure 12 shows the attack spectrum of an FGSM-allack with = 1/255, L on a ResNet-20, ResNet-9, and Efficientnet-Be trained with CIFAR-10. We find perturbations for the models under the regular implementation of convolutions (non-robust), and regularized Regular convolutions are attacked in almost the entire spectrum, but primarily in the highest frequencies (top-left). Under regularization, we observe a shift of attacks to lower frequencies which indicates successful
decomposition layers.
26

---

Published in Transactions on Machine Learning Research (12/2023)
defense against high-frequency perturbations. architecture.
We observe differences in the spectrum depending the
antic tnrrfom
Mur
nobe
Figure 10: Visualizations of the corruptions in CLEAR-1O-C.
1H9
spatter
200TH BiLr
refoous ENIT
Speckle 9966
COPFERSSION
Test
SIMPSANY node
Dighiiees
claet: transfom
contrast
gaussan bar
shot nobe
saturate
2 bur
mition 2Mr
ples ads
Imsube roles
Mean absolute error in FFT spectrum of the corrupted CIFAR-10 test (severity level 5) and clean test set. Corruptions affect the frequency spectrum differently e.g. pizclate affects only high frequencies, brightness only low frequencies, and others like Gaussion noise affect the entire spectrum. Black indicates no error, and bright colors indicate larger errors. The lowest frequencies are located in the center, high frequencies are located at the edges.
Figure 11:
27

---

Published in Transactions on Machine Learning Research (12/2023)
CNN
WD + Reg.
SD + Rog.
(a)
ResNet-20
(b) ResNet-9
(c) EfficientNet-R30
Figure 12: Mean absolute error in the FFT spectrum of FGSM-attacks (c 1/255, I, CIFAR-10 models with different (regularized) convolution modifications and the clean test set. Regular convolutions are at- tacked in almost the entire spectrum, but primarily in the highest frequencies (top-left). After regularization, the adversarial attacks shift from high frequencies to lower ones. The architectures are attacked differently. Black indicates no error, and bright colors indicate larger errors. The lowest frequencies are located in the center, high frequencies are located at the edges.
28

---

Published in Transactions on Machine Learning Research (12/2023)
K
Details on shape bias
Accompanying Table 4, we also plot the texturo-shape bias in Figure 13 using Geirhos et al. (2021) for our ImageNet (Deng et al., 2009) models, EeficinetNet-BO and ConvNext-Ting (Liu et al., 2022) for our decompasition approaches with and without regularization.
Fraction of 'shape' decisions
0.9
0.8
0.7
0.6
0.5
0,4
0.3
0.2
0.1
Shape categories
0.1 0.2
0.7
0.8
0.9
Fraction of 'texture' decisions
shape
tieeete 29300
Enderdee 80 (9000
Encerto 20 30:50 Rog
Contlabi MDD-lppi
Figure 13: Top: Shape vs. texture bias: calegory-level plot. Bottom: Box-plot: shape vs. texture bias
for all variants of EfficientNet-B0 and ConvNeXt-Ting Both plots were generated by Geirhos et al. (2021). High values indicate a shape bias, and low values a texture bias
L Evolution Plots
In the following, we show further and more detailed evolution plots of the DCT-II coefficients in all convolu- tion layers of ResNet-20 (Figure 14), ResNet-9 (Figure 15), and EfficientNet-BO0 (Figure 16) between regular and adversarial training. The latter adversarial training is based on FGSM with an early stopping approach w.r.t the PGD-40 accuracy as presented Wong al. (2020).
29

---

Published in Transactions on Machine Learning Research (12/2023)
4: 1433 D- 1S-9 9313
19:00
00-99
12.00
10.00 19,000 10133
69:39
11.0
1039
10133
(a) Regular Training.
(b) Adversarial Training.
Figure 14: Evolution of the coefficients ResNet-20 trained on CIFAR-10.
1053 10-40
19:09
200 10
19:00 40
105T
19
(a) Regular Training.
(b) Adversarial Training.
Figure 15: Evolution of the coefficients in ResNet-9 trained on CIFAR-10.
30

---

Published in Transactions on Machine Learning Research (12/2023)
(a) Regular Training.
(b) Adversarial Training.
Figure 16: Evolution of the coefficients in a EfficientNet-BO trained on CIFAR-10.
31

---

Published in Transactions on Machine Learning Research (12/2023)
M Tables from Main Paper with Standard Deviation
In this section, we provide a more detailed version of Table from the main paper. In Table 15 we also show the standard deviation over runs.
Table 15: Results (in % and averaged over 5 runs) of ResNet-20 trained with different convolution imple- mentations and regularization CIFAR-100, SVHN, TinyImageNet, MNIST. We report the mean clean validation accuracy and robust accuracy against adversarial attacks: FGSM, PGD-40, and AutoAttack for L 1/255. For MNIST we set 16/255. The usage of the proposed regularization improves the robustness on all datasets.
Dataset
Model
Variant
Clean (1)
Robust Acc. FGSM
[L 16"
(/225)
Val Acc.
PGD-10
AA
CNN
60.41
0.33
14.36 + 0.35
5.45 + 0.30
1.17 + 0.10
WD
58.90
0.19
12.84
0.39
4.68 + 0.16
1.01 + 0.09
CIFAR-100
ResNet-20
SD
60.34
0.44
13.87
0.18
5.18 1 0.09
1.13 1 0.05
WD + Reg.
56.65 0.39
0.67 16.85
0.65 12.73
5.59 + 0.46
Reg-
5.11
CNN
96.31 0.09
83.84 0.65
79.94
0.80
09.81
1.12
WD
96.35 0.15
83.52 0.27
80.01
0.41
71.25
0.71
SVHN
ResNet-20
SD WD Reg.
96.34 0.05
84.07 0.27
80.64
0.57
71.74
0.85
96.26 0.10
84.11 0.27
81.21
0.32
78.27
0.64
SD + Reg.
96.34 + 0.07
84.17 = 0.19
81.23 + 0.22
73.03
0.84
CNN
53.20 # 0.46
17.79 + 0.14
16.76 0.11
9.57 + 0.24
WD
52.08 0.26
17.11 0.42
0.43 16.19
9.40 + 0.33
TinyImageNet
ResNet-9
SD WD + Reg.
52.12 0.40
16.85 0.28
15.88 0.25
9.15 + 0.22
51.25 + 0.20
18.10 + 0.28
17:34 0.30
10.23 0.28
SD Reg.
0.36
CNN
99.68
89.74
45.37 + 18.85
8.92 8.33
WD
99.69 0.05
90.45 = 6.09
17.06 + 13.02
10.22
7.35
MNIST
ResNet-20
SD WD Reg.
99.65
0.04
91.23 2.87
54.08
+ 11.61
16.80
1 11.47
99.69
0.01
90.70 3.92
55.84
6.63
25.92
3.25
SD + Reg.
99.69 + 0.02
88.98 + 3.35
50.02 + 12.53
21.71
8.26
N
Qualitative Analysis of Attribution Maps
In the following section, aim to understand how our regularization affects decisions by visualizing and interpreting attribution maps via SmoothGrad (Smilkov et al., 2017). We study the attributions of some selected ImageNet validation samples for the true label. We generate attributions from regular ConvNeXt- Tiny, and the same network with WD and regularized WD layer as in Table 4. For all samples, we assert that all 3 models correctly predict the top-1 label. Our results in Figure 17 show that our proposed regularization (WD + Reg.). appears to shift attributions from edges and local texture shortcuts seen in unregularized networks (CNN or WD) more global understanding of the scene.
32

---

Published in Transactions on Machine Learning Research (12/2023)
Orignnal
CNN
+ Reg.
Original
Rog.
Orignnal
Original
CNN
WD Reg.
Original
CNN
WD
WD + Reg.
Orignaal
CNN
WD + Rog.
Original
Reg.
Figure 17: Smooth attributions (Smilkov al., 2017) of ConvNeXt-Tiin without changes convolution layens (left), with unregularized WD (center), and regularized WD (right). The regularization appears shift attributions from edges (first rows) and local texture shortcuts (last 4 rows) to global understanding of the scene.
33

---

Published in Transactions on Machine Learning Research (12/2023)
O Fourier Spectrum of Activations
Complementary the results in Figure we investigate the activations immediately after convolution layers of ResNet-20 on a batch of clean CIFAR-10 test samples. However, this time we plot the centered magnitude of the Fourier magnitude (FFTT). Note that the regularization only affects layers up and including B1.2.2. The results in Figure 18 show shift towards lower frequencies in the affected layers of regularized models compared to non-regularized models. The shift is not as strong as for AT models. Figure 19 shows couple of differences for better understanding. Note that in general, the shift is relatively small but still significantly
affects robustness.
Stem
01.0.1
01.02
B1..1
01.1.
01.211
01.2.2
DI3.1
01.32
61.4.1
01.4.2
01.5.1
B1.5.2
01.5.1
B16.2
D1.7.1
Bl7.2
81.8.1
01.8.2
CWN
50
Figure 18: Visualization the magnitude of the Fourier Spectrum of Activations in each convolution layer within ResNet-20 trained on CIFAR-10.
Stem
B10011
B10.2
B11.1
6112
612.1
612.2
B13.1
B13.2
BI.4.1
B1.422
B1.5.1
B1..2
616611
01.62
01.7.1
B1.72
B1.0.1
B1.8.2
20:33
04420
(a)
WD + Reg.
V8.
AT
Stem
B10.1
BL0.2
81111
BL1.2
B12.1
B122
BL3.1
B23
BLA.1
B1.422
BL5.1
B15.2
B16.1
816.2
B17.1
BL7.2
BL8.1
B1.822
1000
1000
1000
1000
2e03
5:03
2303
5.00
000
0000
3:02
5:0
(b)
WD + Reg. vs. CNN
Stem
B10.1
810.2
811.1
BL1.
B12.1
Bl22
816.2
B17.1
BL7.2
B18.1
B18.2
00000
00
T-00%
1.00
2000
-ed
2000
2200
2000
1000
2009
28
52
36
500T
200
2000
000
mm
00
28
(c) WD + Reg. vS.
SD Reg.
Figure 19: Comparison of Fourier domain activation magnitudes in a ResNet-20 trained on CIFAR-10, highlighting the distinctions between the WD Reg. approach and various alternative methods
P Bridging Shape and Low-Frequency Bias
Our proposed regularization is based on the thcoretical background of the relationship between shapes and frequencies. Geirhos el. al. (2019) showed that neural networks tend to classify based on textures and not shapes which not only different from human vision but also results in non-robustness due to the locality of the information. On the other hand, detection based on global information such as shapes must be more robust (Geirhos et al., 2021). Therefore, we want to induce a shape bias directly by our proposed regularization approach.
34

---

Published in Transactions on Machine Learning Research (12/2023)
10 fheeee
DC Shece Funciee
OCT4 coudure velro of Fundiinn
Sepuliza
Figure 20: Shape and DCT-II spectrum for an object and its shape.
Tay Shape Example
Tay Shape
1.00
0.00
d0
Figure 21: 1D shape loy example with its correspouding DCT-II basis transformation
Low-frequency components dominate shape signals. For simplicity, let us consider shape function in 1D, ie. rectangular pulse. For the Discrete Fourier Transform (DFT), it is well known that rectangular pulse in the spatial domain directly corresponds to sinc function, defined frequency domain and vice versa. The absolute value of this function assigns the highest values to the lowest frequency and the amplitude of the wave decreases as the frequency increases. The behavior of the DCT is related to that of the DFT and similar in practice.
sine(w) )wn(w) in the s
Figure 20 visualizes the behavior for the DCT-II: Transforming the shape function a 1D selected area the image (Everingham et al.) into frequency domain using the DCT-IT basis results in a wave function whose amplitudes decrease as the frequency increases. We furthermore visualize this relationship on toy example of rectangle function and its transformation into the frequency domain (Figure 21). In the following, we provide the mathematical motivation of the resulting regularization term, which is given in Equation 7 in the main paper and which is inspired by this behavior. Specifically, we show the behavior of the DCT-II of rectangular pulse.
Let I E N a rectangular pulse of width 2M in the domain of symmetric, periodic signals length N, approximating a square wave when M N/4, i.c. æ(n) for -M < n < M and r(n) otherwise.
The DCT-IT of is X DCT(x), for k 1-....
N
35

---

Published in Transactions on Machine Learning Research (12/2023)
M Σ
X(k)= =
COs
π
π2
+
2 k)
N
11=
M
1
M
+ 2/2 k) exp
(-ix7
( n
1 2
exp
i N
k)
(10)
2
M
M
M
exp
π 2N
k
exp i ' ))
+ 2 exp
1
ग 1x) (-i
(-2
Σ
exp
(-i
- ))
2
2N
1= M
n=-M M
exp (1 k))
exp (15 M+1
2
exp
2N
k
e-exp (1 (x
(11)
i xk -M - eep (ki M+1
1 exp
(11 π
exp
2
2N
1 exp
exp (-ixk)
cxp (1))) -
exp (i 7. k))
M+1
2
exp
2N k
exp (1)) 1/2 exp (1))
(1 1b) M+ exp (1) M
+ exp
1
-11 π
exp
(12)
2
2N
exp (1 k) _ exp (i -
1
cxp (-i(M + 2) k)) exp (1(M + 2) kk)
2 exp
2N k
cxp (-1) N k:) cxp
(12 Rb)
1
(- π 2N ki)
exp (i(M + *)) - exp
(-i(M + ())
2
exp
exp (ig N k)
exp
(- k)
1
exp
2N k
π
sin ((M + ()
2 exp
π (-i
2N k
sisin (IM + (1))
(13)
2
sis
x k)
2i
2N k)
1 sin ((M + 7-)
exp
i
. 2N
k:
+ exp
1 k:
(14)
2
sin (2 k)
(M- 3M-+bam k:
sin
sin ((M + ())
JT k 2N
cos
2N
π
k.
=
(2M +1)
(M + 5
cos
2N
π
(15)
(M
1
π ki
(2M + sine
+
COS
2
N
2N
In the second equality (10), we apply Fuler's formula. Moving to (11) we apply the finite sums formula and cancel exp (i5+)* à and exp ki - in line (12). Then, we again apply again Fuler's formula in (13) and (14). The step to arrive at (15) is an approximation used for the DFT, following the reasoning that for small w, sin(w) = w and the term in the sin function in the denominator is small. As a result, the DCT of the rectangular pulse corresponds to a sine function overlayed with a cosine. The amplitude of the resulting function decays as the frequency increases. For small k. the function approximates sinc function. In the regularization term we propose in Equation 7, we, therefore, regularize the lower frequencies with the hierarchical term that encourages the maximal amplitudes of higher bands to be lower than the maximal frequency of lower bands, i.e. we regularize the max (I|C [e. ll |C (.1)oo)))))) for L,, M = (0, 1}. For high frequencies, we just encourage the amplitudes to be low in general, so that the full regularization is
R(C)= = |C I+ Paitf max (II (.). ll
||C |[C()..e).ooool2)
for J,J= [[K/2], ... k), L, M = 10 and [s] = (0.... dont 1}, [d] = {0............ ... (0....,dn 1}.
36


Reviews:
[{"review_id": "3p3zAXljSx", "reviewer": "Reviewer_TkKe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\nThe proposed method effectively improves the networks’ native robustness without training on adversarial samples.\n\n[ Weaknesses & Suggestions to Authors]\n1.\tIf the authors want to claim the relationship between the network’s robustness and the frequency spectrums of the convolutional kernels, there are at least three key problems the authors need to clarify: \n(1) It is generally believed that the adversarial attack on the input image mainly affects the high frequencies of the input image. However, is there any evidence to show the adversarial attack mainly affects the high frequencies or low frequencies of the intermediate-layer feature? Spectrums of input images are not equivalent to spectrums of feature maps.\n(2) What’s the relationship between the network’s robustness and the intermediate-layer feature spectrums?\n(3) What’s the relationship between the intermediate-layer feature spectrums and the frequency spectrums of the convolution kernels?\nIt would be better if the authors theoretically analyze, or discuss, or conduct some experiments towards the above three concerns.\n\n2.\tIn Equation (8) of this paper, the second term of the function R(C) seems to be ad-hoc.\n\n3.\tIn a CNN, most kernels are of the size 3*3. Based on Equation (8), it’s hard to tell which frequencies are penalized. Is it meaningful to distinguish low frequencies and high frequencies from such a small 3*3 map?\n\n4.\tThe size of ResNet-9 is too small. We suggest the authors to conduct experiments on more large SOTA models, such as ResNet-50, ResNet-101, ….\n\n[ Minors]\n1.\tIn Equation (8) of this paper, the authors use an informal index representation, such as “:, :, [k/2]:, [k/2]:”\n\nThe overall rating is major revision.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors observe that the cleanly-trained models often prioritize high-frequency information, whereas adversarial training enforces models to shift the focus to low-frequency details during training. Based on this observation, the authors propose a high-frequency penalization term in loss function to improve robustness to adversarial attacks. The authors conduct experiments to illustrate their findings and the effectiveness of their proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see \"Strengths And Weaknesses.\"\n\nI suggest the major revision.", "broader_impact_concerns": "No concern on ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BVhnIotEBg", "reviewer": "Reviewer_b5AR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:   \n(+) 1. The idea proposed by the author is interesting and worth investigating.  \n(+) 2. This paper conducts experiments across various datasets with different image sizes.   \n(+) 3. As shown in Table 1, the method improves both adversarial and corruption accuracy across all 3 architectures tested.  \n(+) 4. The results in Table 2 indicate that the proposed method is more robust than the previous Frequencylowcut pooling (FLC) and Patch gaussian augmentation (Paga).  \n\n**Weaknesses**:    \n(-) 1. The high-frequency regularization approach is intended to emulate the behavior seen in adversarial training. When analyzing the spectrum of feature maps, in Section 5.1, how do the layers subjected to regularization compare to those in a CNN baseline trained using adversarial methods?  \n(-) 2. Missing citation. Similar assumptions regarding the types of features learned by adversarially trained models have been made in a closely related study [4]. Please cite this study and discuss the two approaches in the related work section.  \n(-) 3. Important missing citations. Previous works have examined anti-aliasing to improve robustness, with a particular focus on mitigating aliasing in the downsampling operation in [1], [2], and [3]. However, this paper does not discuss or compare against these methods. It would be important to benchmark the proposed method against these methods.  \n(-) 4. In terms of training, how does the performance of the proposed method compare to FLC pooling, and also others? The discussion of the training time comparison will be essential for determining the practical feasibility of employing this method in downstream computer vision tasks.  \n\n[1] Richard Zhang. Making convolutional networks shift-invariant again. In International Conference on Machine Learning (ICML), pp. 7324-7334. PMLR, 2019.  \n[2] Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. In British Machine Vision Conference (BMVC), 2020.  \n[3] Qiufu Li, Linlin Shen, Sheng Guo, and Zhihui Lai. Wavelet Integrated CNNs for Noise-Robust Image Classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 7245-7254.  \n[4] Frequency Regularization for Improving Adversarial Robustness. In Practical-DL Workshop, Association for the Advancement of Artificial Intelligence (AAAI), 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* This paper suggests introducing frequency regularization for DCT-II coefficients, based on the observation that adversarially trained models tend to focus more on low-frequency details in their initial layers. The application of this frequency regularization enhances the robustness of CNNs to both adversarial perturbations and common corruptions.\n* This method for improving robustness does not require additional training data.\n* The authors perform evaluations using modern CNN architectures like EfficientNet and ConvNext, assessing their effectiveness against both traditional attack methods like FGSM and PGD, as well as the most recent auto-attack.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major:**   \n* This paper needs a summary section that consolidates all the results. Without the summary it is hard to appreciate the results.\n* Compare against [1], [2], and [3]. The proposed method outperforms FLC, and prior research has already demonstrated FLC's superior robustness compared to [1], [2], and [3]. Nevertheless, it would be advantageous to include all these methods in a unified benchmark for future research purposes.\n* Please see the weaknesses for any additional changes.\n\n**Minor:**   \n* Please change references such as Tab. 1 to Table 1 for better readability.\n* Kindly specify the relevant section in the appendix when making references in the main paper.", "broader_impact_concerns": "Since the study deals with adversarial attacks and defenses, it should include a statement addressing broader impacts statement", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hZTodgtR81", "reviewer": "Reviewer_KFrx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe specific method proposed by the authors is novel. Moreover, the robustness of their method to many types of image corruptions is a major strength. The method is also motivated by solid frequency analysis of existing networks in section 3, which is also a plus. The experiments are generally comprehensive and well-chosen. Finally, the writing and presentation of the paper are generally good, particularly the table format, figures, and notation.\n\n**Weaknesses**\n\nOne weakness in my view is the motivation and explanation of native robustness. It's not clear to me what the practical value of native robustness is; if the goal is to produce networks that are robust to a variety of perturbations, one possibility is to simply adversarially train on all the possible perturbations. Moreover, there are existing works on training techniques robust to *unforeseen* adversarial attacks. The authors argue that, unlike adversarial training, their method does not require out-of-distribution training data. However, from another perspective, adversarial training does not require out-of-distribution training data either- instead, the process of generating adversarial examples for training can be viewed as part of the learning algorithm itself.\n\nAnother weakness is the similarity of this paper's perspective with other works. As the authors point out in section 2, there is literature already exploring the relationship between frequency and robustness, including works that propose using frequency as a tool to improve robustness. Moreover, the basis decomposition idea is not new either, as the authors note. Ideally, the distinctions between this work and prior work would be delineated more sharply.\n\nRegarding the experiments, I think it is important in most of the tables to compare the proposed method with adversarial training or other strong methods of inducting robustness (even if these do not count as *natively* robust). I understand that the performance of these baselines would likely be expected to be better than the proposed method, but nevertheless, this would be important to have.\n\nAlso, in most cases, the improvement of the proposed method over the baseline of standard training appears relatively modest (although there are cases for which the improvements are quite large). Adding additional stronger baselines will make these improvements easier to evaluate.\n\nFinally, as the authors note, the architecture itself impairs accuracy a little bit, which is not ideal, but overall this is still acceptable since the regularized performance is significantly stronger.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel approach called filter frequency regularization to improve the native robustness of convolutional neural networks (CNNs) to various distribution shifts, including adversarial attacks, corruptions, and shape-biased datasets. The authors show that prioritizing low-frequency information in learned convolution weights leads to better model generalization and decision-making, as well as improved performance on adversarial attacks and out-of-distribution tests. They also demonstrate the effectiveness of their approach on several benchmark datasets and compare it to other state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical for acceptance**\n\n- Clarifying the practical motivation for and definition of native robustness\n- Revise related work to more clearly distinguish the contributions of this work from prior work\n- In results tables, compare with adversarial training or a similarly strong method\n\n**Would strengthen**\n\n- The authors present a purely experimental paper, which demands very strong results; one possibility to strengthen the work without new experiments is to add theory exploring the relationship between frequency and robustness\n- Some of the text in Figures 4 and 5 could be made larger to be more readable\n- The authors mention that their method may be more aligned with human vision; showing this using some interpretability methods could be valuable", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers side with acceptance. The authors gave a thorough response to each reviewer, and the discussion resulted in a number of improvements, which have been included in the revision. The authors have summarized the changes with attribution to each review thread. In particular, the revision addressed issues with the claims and evidence to include citations, further comparisons, and content in the appendices. 

The submission can be accepted essentially "as is" with a last round of trivial changes. This last round should adjust the formatting, to include the changes from the revision as normal text, and proofread the text to correct minor issues (for instance, the grammar glitch of "Exemplary, we show", the vocabulary of "sensibility" vs. "sensitivity", ...).

The AE thanks the authors and reviewers for engaging in the TMLR process to deliver an improved paper with agreement between its claims and evidence and a clear audience.
Decision: Accept (poster)