Revisiting Visual Model Robustness: A Frequency Long-Tailed Distribution View
Authors: Zhiyu Lin
Yifei Gao
Yunfan Yang
Jitao Sang
Venue: NeurIPS 2023 poster (2023)
Keywords: visual models
robustness
frequency domain
long-tailed distribution
TL;DR: Our work analyzes and addresses the robustness of visual models from the perspective of exploring long-tail issues in the image frequency domain.

Abstract: A widely discussed hypothesis regarding the cause of visual models' lack of robustness is that they can exploit human-imperceptible high-frequency components (HFC) in images, which in turn leads to model vulnerabilities, such as the adversarial examples. However, (1) inconsistent findings regarding the validation of this hypothesis reflect in a limited understanding of HFC, and (2) solutions inspired by the hypothesis tend to involve a robustness-accuracy trade-off and leaning towards suppressing the model's learning on HFC. In this paper, inspired by the long-tailed characteristic observed in frequency spectrum, we first formally define the HFC from long-tailed perspective and then revisit the relationship between HFC and model robustness. In the frequency long-tailed scenario, experimental results on common datasets and various network structures consistently indicate that models in standard training exhibit high sensitivity to HFC. We investigate the reason of the sensitivity, which reflects in model's under-fitting behavior on HFC. Furthermore, the cause of the model's under-fitting behavior is attributed to the limited information content in HFC. Based on these findings, we propose a Balance Spectrum Sampling (BaSS) strategy, which effectively counteracts the long-tailed effect and enhances the model's learning on HFC. Extensive experimental results demonstrate that our method achieves a substantially better robustness-accuracy trade-off when combined with existing defense methods, while also indicating the potential of encouraging HFC learning in improving model performance.

Revisiting Visual Model Robustness: A Frequency Long-Tailed Distribution View
Zhiyu Lin', Yifei Gao', Yunfan Yang', Jitao Sang 12-
Beijing Jiaotong University, China
Peng Cheng Lab, Shenzhen 518066, China
fzyllin, yf-gao yfyang, [asaan]{bbjtun edu. edu. cn
Abstract
A widely discussed hypothesis regarding the cause of visual models' lack of ro- bustness is that they can exploit Juman-imperceptibl high-frequency components (HFC) in images, which turn leads to model vulnerabilities, such as the adversar- ial examples. However, (1) inconsistent findings regarding the validation of this hypothesis reflect in a limited understanding of HFC, and (2) solutions inspired by the hypothesis tend to involve tobustness-accured wards suppressing the model's learning on HFC. this paper, inspired by the long-tailed characteristic observed in frequency spectrum, we first formally define the HFC from long-tailed perspective and then revisit the relationship between HFC and model robustness. In the frequency long-tailed tailed scenario, experimental results on common datasets and various network structures consistently indicate that models in standard training exhibit high sensitivity to HFC. We investigate the reason of the sensitivity, which reflects in model's under-fitting behavior on HFC. Furthermore, the cause of the model's under-fitting behavior is attributed the limited information content HFC. Based on these findings, we propose a Balance Spectrum Sampling (BaSS) strategy, which effectively counteracts the long-tailed effect and enhances the model's learning on HFC, Extensive experimental results demonstrate that our method achieves substantially better robustness-accet accuracy trade-off when combined with existing defense methods, while also indicating the potential of encouraging HFC learning in improving model performance.
trade-off and leaning to-
Introduction
The disparity between deep visual models and the human visual system (HVS) in image signal
processing enables the models to outperform the HVS on vision benchmarks [16] 33 However, it is
worth noting that most visual models lack the robustness exhibited by HVS [36] 17. Specifically, the
generalization advancement of visual models can be partially explained by the utilization of human-
imperceptible information, such as high frequency components (HFC) 39 29]. Additionally, the adversarial vulnerability is also closely related to this information, observed in adversarial examples phenomena [36]. The coexistence of both conditions gives rise to a widely discussed hypothesis: the vulnerability of the model siems from the utilization of imperceptible high-frequency information in images.
Based on this hypothesis, a line of research has been conducted to explain and enhance the robustness of the model 43 1 7
35 13. However, these works share a common limitation: most of them solely rely on qualitative analysis of frequency components observed by humans, without further delving into the statistical properties. This limitation gives rise the following issues in existing
"Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

works: (i) the division of the frequency spectrum is intuitively based on manually set radius, resulting in inconsistent conclusions across different datasets and models; and (ii) The understanding of high frequency utilization is limited. For example, improving model robustness often involves suppressing the learning on HFC, which in stark contrast to the promotion of high frequency utilization to improve generalization performance. This leads to a trade-off between robustness and accuracy in some defense methods.
In this paper, to overcome the challenges posed by the aforementioned limitation, we are motivated by the long-tailed characteristics in the spectrum. Specifically, the power spectral density of an image naturally follows power-law distribution, where few low-frequency bands accounts for the majority of the spectrum power while a large number of high- frequenny bands maintains a low power distribution. This new realization encourages us to revisit the significance of high-frequency components in addressing the problem of model robustness under the guidance of deep long-tailed learning.
In deep long-tailed learning [46), the presence of a large number of tail classes plays a critical role in improving model performance. These tail classes offer benefits such as ensuring data integrity and diversity [5] and capturing rare events 23 However, it commonly observed that models tend to under-fit these classes due to their numerical disadvantage. Intuitively, the long-tailed characteristics of the frequency spectrum also exhibit similar phenomena. We confirm the model's s (standard training) under-fitting behavior on high frequencies through the lens of loss landscape [27] Under the context of model ander-fitting, the discussion regarding HFC and model generalization performance can be compatible in most studies (29 39, as the common underlying cause is the low generalization resulting from under-fitting. In this paper, we establish a link between under-fitting behavior and model robustness, specifically identifying that the model's 'ss under-fitting of high-frequency components leads to high sensitivity towards them. We validate this relationship across different datasets and models, consistently confirming our findings. Model's under-fitting behavior on high frequency leads to both low generalization and high sensitivity towards HFC. Thus, it is crucial to improve the under-fitting behavior and further mitigate the trade-off between accuracy and robustness.
Drawing inspiration from deep long-tail taig-tal learning, we introduce spectral entropy as a measure of the information content of frequency components and provide an explanation for model's 's under-fitting behavior on HFC. We attribute model's under-fitting behavior to the low spectral entropy of HFC. Based on this perspective, the principle of maximum entropy 23 3 32 guides us in constructing an image spectrum with a more balanced energy distribution to address the issue of low information entropy. Finally, our quantitative experiments and theoretical analyses inspire us to propose simple yet highly effective Balance Spectrum Sampling (BaSS) strategy, which can work in conjunction with different training paradigms and enables the model to achieve better trade-off between robustness and accuracy.
Our major contributions are as follows: (1) To the best of our knowledge, we are the first to propose focusing on the long-tailed distribution in instance-wise specifically in the frequency domain. This novel perspective provides insights for analyzing and improving model performance. (2) We revisit the relationship between HFC and model robustness, revealing the under-fitting phenomenon and its association with high sensitivity and low generalization at high frequencies. To address the trade-off between accuracy and robustness, we propose to enhance the high-frequency learning. (3) We explain the fnder-fitting behavior through the lens of spectral entropy. Drawing inspiration from the principle of maximum entropy, we propose a simple yet highly effective spectral sampling strategy (BaSS) to improve the model's under-fitiing behavior. By incorporating BaSS with other training paradigms, such as adversarial training and AugMix, model achieves better trade-off between accuracy and robustness. Comprehensive experiments and analyses reveal the effectiveness of our method.
2 The long-Tailed Distribution in Frequency Domain
2.1 Notations and Preliminaries
Image Processing in the Frequency Domain. We perform 2-D discrete Fourier transform (referred to as the DFT) F on image X Hx to get Fourier spectrum X E @# Unless otherwise specified, the low frequencies are shifted towards the center of the Fourier spectrum. We denote X [u, 2] as the frequency component where {O...., . - 1} and v E {O..... W 1}. Addition-
2

---

ally, Wa,e denotes the Fourier Basis function [38 and Tu,o denotes the Euclidean Distance from the spectrum coordinate [u, 2) the center of spectrum, where larger distance correspond to higher frequency, and vice versa.
Power Spectral Density, The power spectral density is defined by azimuthally averaging the magnitude of Fourier coefficients over radial frequencies. Specifically, the value of power density is estimated the azimuthal integration (referred to Af) over radial frequencies e according Eq
2m
AT(rk)
[[x cos o, rh sin Φ) 2d
where Fi is the Euclidean Distance from the hth frequency band to the center the spectrum 6 k E (O......HI1 +
2.2 Definition of Frequency Long-Tailed Tailed Scenario
Real-world data generally demonstrates long-tailed distribution, where small number of dominant categories comprise the majority of the data, and a large number of minority categories possess limited samples. Although previous studies [46] 37] have primarily investigated the degradation in model performance due to the long-tailed problem at the class level, they have not adequately considered the potential existence of long-tailed issue within individual images. Through our analysis of the Fourier spectrum, we observe that the power spectral density follows a power-law distribution, which also exhibits the long-taile -tilled characteristics. Specifically, the distribution (y-axis) of tth frequency band (x-axis) is calculated by the proportion of power in 2th band to the total power of the spectrum, which is denoted as n; Arrra)/E, AT(r)). Without loss of generality, {me) naturally present in decreasing trend (i.e., if h1 < k2, then TA, < TTg))
Consequently, we summarize the statisties of the frequency feature from the long-tailed model perspective as follows: (i) The head part, which is consisted of a few low-frequency bands, accounts for the majority of the spectrum power and is highly semantic and information dense; (ii) The tail part, which is composed of large number of high-frequency bands, however maintains a low power distribution. This part includes frequenyy semantic information (e.g., texture) as well as human-imperceeeeeee imperceptible information.
LFC = {X u|
max
S 80%, Tu, S rnn
t=
(2)
H/2+1
=FC= = (8 u]
min Σ
m 2 20% Fu,t 2 rn
Long Tail based Spectrum Division. Previous analysis 15 frequency components form the image's 's main content, while most high-frequency components comprise a minor part. The rule that few factors determine main outcomes is encapsulated by the Pareto Principle and the "80/20" quantitative rule, found across various domains, where 80% of effects are caused by 20% of the reasons. Guided by this rule, we defined the boundary between frequencies using an 80/20 spectral energy ratio. We formally define the low- and high-frequencier according to the E where the Tn is the Euclidean Distance from the n frequency band to the center of the spectrum. The set composed of X X(u, v) occupying 80% of the total power in the spectrum is defined as the low-frequency component, and the remaining portion of the spectrum is defined as the high-frequenn component. We experimented with various datasets, finding that different resolution images' high-low firquencc boundaries are at a 20% radius from the spectrum's center (as shown in Appendix B.2). This confirms the stability and reasonableness of the classification system using spectral energy and the Pareto Principle.
39 31
indicating that a few low-
3 Investigating and Explaining Model Robustness Behaviors: A Frequency Long-tailed View
Vision models have demonstrated capability learning frequency information Intuitively, model's behavior could be affected by the Long-tailed characteristies in the frequency
3

---

1.0
100
9.
0.5
-0.
-0.5
-1.0
-1.
-1.5 -1.0
0.0 0.5
-15 -1.0 -0.5
1.0
(a)
(b)
Figure I: Loss landscape visualization of HFC (a) and LFC (b). We conduct experiments on ResNet- 18 naturally trained on CIFAR10 dataset. The black star symbol represents model parameters 0. noticeable phenomenon is that the model exhibits learning bias towards different frequency components: specifically, the learning for figh-frequency components has not converged, while the learning for low-frequency components has already reached a local optimum.
domain, which has not been fully explored in previous studies, and is crucial for improving model performance visual tasks. this section, we first summarize the impact long-tailed characteris- ties as a phenomenon of the model's under-fitting of HFC (Sec. 3.I). We then explore the model's sensitivity to HFC from the perspective of under- fitttrrg, and revisit the relationship between HFC and robustness (Sec. 3.2. Finally we explain the model's under-fitting behavior from the perspective of spectral entropy (Sec. 3.3
3.1 Model's Under-fitting Behavior on HFC
In the long-tailed theory, the tail part considered more deserving of attention than the head part. On one hand, the tail part is often overlooked by the system due to its numerical disadvantage(e.g. the sample size of the tail class far less than that of the head class in the context of class-wise long tail). On the other hand, it plays a crucial role in enhancing the system's 's performance, primarily due to two characteristics (i) providing data integrity and diversity [5), and (ii) capturing rare events (28). In this subsection, we explore the learning behavior of models regarding frequency components. Specifically, we visualize and analyze the loss landscape (27 of the classifier on both LFC and HFC. We present the detailed experimental procedure in the Appendix C.
Phenomenon of Model Under-fitting on HFC. The convergence result of LFC and HFC learning exhibits significant differences as shown in Fig. I Fig. [(b] suggests that the loss landscape on LFC exhibits a benign characteristic: the parameter space around the model 0 is relatively flat and close to the local minima, indicating that the model has precisely converged on LFC. In contrast, the loss landscape on HFC shown in Fig. ((a) manifests dramatic non-convexity in certain regions. can be observed that the model 0 are located in steep region of the landscape, with loss rapidly decreasing when moving along the gradient directions (i.e., which are perpendicular to contours).
Additionally, in the bottom-left region of the Fig. T(a] we also observe a better local minimum far away from current model. These observations indicate that the model still has the potential to improve its ability to leam HFC.
Positive Correlation between Generalization and HFC Learning. is widely recognized that under-fitting is cause of low generalization performance. Recent works have confirmed that models exhibit lower generalization ability on HFC compared to LFC [39 29). Meanwhile, ignoring the HFC can lead to inaccurate representation of the data distribution, resulting in a degradation in model generalization 39 4 This correlation can attributed to the first characteristic of tail components mentioned above. Hence, it is vital to devise and employ strategies that alleviate or eliminate the under-fitting of HFC, as substantially contributes to the enhancement of model generalization.
performance 39 29

---

1/5
Kagniti
Gentlee Meggitude
0.:-
1:25
0.6-
1.00
0.4-
0:5-
025
0.00
503
1936
1932
2002
5000
10303
15400
26000
39300
Rank Index
Rank index
Rank Index
2.5
2.-
TPC
0.6
1.0
0.
02
0.
15
25
20
59
Frequency Radius
Fri quarccy Radlus
Radius
(a) CIFARIO
(b) Tiny- ImageNet
(c) ImageNet
Figure 2: Model sensitivity to frequency components on CIFAR1O, Tiny-ImageNet and ImageNet datasets. We present the sorted average gradient magnitude test set in the first row, with the araxis denoting the rank index. In the second row, we select part of rank positions, and display the density distribution of frequeney radius. The gray dashed line served as the boundary between low- and high-frequencies. The overall phenomenon is that naturally trained model is highly sensitive to tail frequency components.
3.2 Revisiting the Relationship between HFC and Model Robustness
3.2.1 Motivation
The phenomenon of under-fitting inspires us to revisit the issue model robustness. Robustness is typically expressed as the sensitivity of model outputs to perturbations in inputs, and it can be evaluated in two ways: (i) changes of the outputs leaded by introducing additional perturbations to the inputs; and (ii) inputs gradients with respect to outputs. The results shown in Fig. I(a) indicate that standard training classifier still exhibit significant loss values in HFC and may have large gradients with respect to fighhfrequencies, thus resulting in model vulnerability. In the context of long-tailed theory, the tail part usually represents special or extreme cases and thus is more likely to be affected by rare events, making HFC crucial for evaluating and analyzing model robustness. Existing studies suggest that the vulnerability of models is attributed to HFC, however, this conclusion is consistent across different datasets and models. We attribute the potential causes of inconsistency to the definition of HFC. We aim to further explore relationship between HFC and model robustness.
[3 7 9
3.2.2 Analyzing Model Sensitivity to Frequency Components
Model sensitivity to different frequency components has been explored using the Fourier heatmap, which quantifies the changes in outputs when applying Fourier basis perturbations images [43). However, there are a few concerns about its applicability: (i) The theory of Fourier basis is based on linear networks while deep neural networks typically exhibit high non-linearity, (ii) The perturbations is excessively sensitive to the magnitude of the basis disturbance. Thus, we take inspiration from
29 77 and employ the loss gradient across various frequency components to examine sensitivity.
Estimation of Model Sensitivity. Given an input image X with label y. the output of classifier f is f(X). The objective function of model's training is formulated gradient the loss w.r.t. the original input X can be expressed as V Cxf((() y) E R whose magnitude can indicate the sensitivity of the classifier pixel domain perturbations. To further assess the model's sensitivity in frequency domain, we employ the set of orthogonal Fourier basis vectors [ and project into frequency domain coordinate system. The corresponding coordinate of projection in wu,v is defined as follows:
((f(X).)) y) Consequently, the
H-1W-1
Cu.y = (g, Wu.u)
Σ
Σ
glm, n expl-122 -i2 12
mi
n
W
m=D n=0
5

---

The sensitivity of model to the frequency components corresponding to the spectrum point [u, u] is measured by the gradient magnitude on Wa, which is defined as || 1. In order to identify the frequency domain sensitivity patterns of the model at the dataset level, we first sort Wa,, according to [|u,,ll2 on a single image. For ease of statistical analysis, we replace We,o with the frequency radius Fu,v as the feature vector. Finally, we get the set of feature vector for all images and estimate the density of frequency radius using kernel density estimation (KDE) within the same rank index. We analyze the frequency sensitivity across the CIFARIO 1O (25), Tiny-ImageNet (26), and ImageNet [12] datasets with resolutions of 32", 642, and 224 pixels, respectively. We show detailed settings in the Appendix D.
Results and Discussions. From the first row of Fig. we observe an imbalanced distribution of gradients in different frequency bands, and the degree of imbalance is quite pronounced, resembling a similar long-tailed distribution. The results in the second row of Fig- indicate that the density peaks of frequency radius in high-magnitude gradients are mainly concentrated in the tail high- frequencies (i.e., represented by the darker colored curves). These observations clearly demonstrate the relationship between HFC and model vulnerability, indicating that perturbations in HFC are more easier to cause errors in the model. In the result of ImageNet as shown in Fig. 2 we notice a particular phenomenon where there are also large gradient magnitude in the head components. This could be due to the wide spectral range causing similar vulnerabilities in the head components. However, this phenomenon does not affect the consistency of our conclusions. It also suggests that addressing robustness in the ImageNet dataset is a more challenging task.
3.3 Reasons leads to the Under-fitting Behavior
In deep long-tailed learning, the lack of sufficient generalizable information on tail classes directly causes biased behavior in models. Similar phenomena inspire us to explain the under-fitting of the tail part from an information-theerrtiic perspective in the case of frequency long-tailed problems.
3.3.1 The Spectral Entropy of HFC is Much Lower than LFC
Spectral Entropy To quantitatively characterize the amount of information contained given image, spectral entropy is utilized analogously to Shannon entropy in information theory. The continuous frequency domain is discretized and represented by a discrete variable &, with & and &n as its minimum and maximum values, respectively. Given that the spectral density of natural images follows a power-law distribution, its density function can be computed and normalized as follows: 1-a k- where is the parameter determining the magnitude of the long- P(E = k)) - effect, and its smaller value indicates a more significant long-tailed effect. We use to estimate Pr. and then the spectral entropy of kah band is derived by applying the Shannon entropy formula: H₂ = n log a Consequently, the flat spectrum corresponds to maximal spectral entropy, while the spectrum of a single frequency signal has minimal spectral entropy, which zero.
Given any natural image X, k represents the order of frequency band, then the spectral entropy Hş is given by: H₁ = (a.A, log + H(((a))) k= where A(a) = =(1 (1-a) (C)
Theorem
- Then we have, lim H = oo and limg-1oo H = 0.
Theorem demonstratsssss an inverse proportional relationship between the frequency band and spectral entropy. decreasing at an order. This indicates that highffrequency bands in the tail contain substantially less information than low-frequency bands in the head. Unlike measuring the number of image samples, we provide a new way to measure image information from the frequency domain perspective, explaining the model's under-learning behavior for low-information data. Moreover, Theorem aalso suggests that high-resolution data, such as ImageNet, often lack effective generalizable information in the tail. This finding aligns with Fig. [ illustrating that the model fails to generate gradient components in the uitra-high frequency range of ImageNet data.
3.3.2 Balanced Spectral Entropy Benefits the Under-fitting Phenomena
The maximum entropy principle seeks to make reliable predictions on unknown data distributions by selecting probability distributions with the highest entropy, without introducing unnecessary biases or prior knowledge. Both generalization and robustness reflect a model's ability to infer unknown samples, which depends on its capacity capture information during training. Based on

---

our analysis, we suggest that this principle explains the model's 's under-fitiing on HFC: the model prioritizes utilizing LFC with higher information entropy to improve its capacity for fitting unknown data, leading to the observed under-fitting in high frequencies. To address under-fitting in HFC, we aim to increase the amount of information, and the maximum entropy principle offers a solution.
Theorem 2 Given the prior data &, and its corresponding probability distribution p(E), the principle of maximum entropy consider the following optimization: p = {arg maxper Hp) p(() 2 0, Ja p(E)dī 1, la (() 0i, i =..... m.). Without considering the moment constraints, the uniform distribution p(E)
En (.))).)((E)) satisfies the principle of maximun entropy.
Theorem 2 demonstrates that a uniform distribution with balanced information adheres to the max- imum entropy principle when fitting unknown distribution. Intuitively, this corresponds to fair learning for tail classes, which has been extensively studied in deep long-tailed learning to improve tail class generalization using techniques like re- re-weighting or re-sampling. This insight drives to tackle the under-fitting fitting issue of HFC by redistributing spectral energy through a balanced sampling strategy.
4 Methodology and Experiments
4.1 Balanced Spectrum Sampling (BaSS)
Previous analyses encourage us to HFC learning by balancing the power spectral density. Nonethe- less, it is essential to consider the potential negative effects of noise present in the image while implementing a balanced spectrum. Noise components, often found in
parts, have been proven less beneficial for classification tasks (39 29 Moreover, uniformly allocating noise and other frequency components may (i) amplify noise influence on model learning. and (ii) allocate less energy to remaining frequency bands, both negatively impacting model learning. To address these challenges and achieve spectral balance, we propose spectral sampling strategy in this subsection.
log, Al(r:) # プラ log, Al(r;)
Sampling Strategy. For the 7hh frequency band, we sample the power spectral density according to Eq. 4 with probability of मे. where 7 2 1 denotes the base and B is the maximum frequency band of the spectrum. Our approach is inspired by two background pieces of knowledge: solutions for the long-tail distribution of image categories and image sampling. The solution for the class-wise long-tail distribution adjusts the sampling frequency of samples with different class quantities through the inverse weighting of category quantities. However, features in images (like spatial pixels and spectral density) can not usually be directly chosen via the sampling theorem. Therefore, we refer to the method of sampling image pixels in the spatial domain: (i) discretizing image features that corresponds to the spectrum obtained from the Fourier transform; (ii) selecting and combining feature values from different regions: we re-weight features at different frequency bands based on the sampling probability îं and obtain a new image through the inverse Fourier transform.
Specifically, employ logarithmic function to: (i) establish a smooth connection between the power density of head and tail parts, mitigating the impact of ultra-high frequency components on the overall distribution, and (ii) adjust spectral power values in the head part to a reasonable range, while diminishing the power disparity between head and tail frequency bands, yielding a more balanced distribution overall.
4.2
Working Mechanism Exploration of BaSS
In this subsection, we validate the potential of the Balanced Spectrum Sampling strategy (BaSS). Specifically, we examine two paradigms for utilizing the sampled images: (i) Data augmentation, where each training image is performed by BaSS with probability of ry In this paradigm, the model will be validated on a test set comprised only of natural images. (ii) New dataset, where both the training and testing sets consist entirely of sampled images.
Experiments Settings. We refer to the dataset performing BaSS on CIFARIO 25 as CIFARIO-B. For different values of 7, we obtain a set of ResNet- 18 [19 models with standard training. We
7

---

100
60
100
Robust Acc(%)
Clean Acc(
75-
60
40
Acc(%)
Robust
60
<5A md. CPAR2O
60
20
0.25
0.5
0:15
40
0.23
0.
0.25
0.5
0.75
CIFAR1O
CIFARIO
(a)
(b)
(c)
Figure 3: Results of exploring potention of BaSS. (a) clean data accuracy, (b) robust accuracy evaluated on CIFARIO-C, (c) robust accuracy against PGD attack with 20 steps, and (d) clean and robust acc comparison on CIFARIO and CIFARIO-B
-55
-55
41
(a) 7 7=000 0.25
(b) 7
(c) 7 0.75
Figure 4: Visualization of Loss Landscape models that training with CIFARIO-B data augmen- tation at ratio of y. The black star symbol represents model parameters 0.
use clean accuracy, corruption robustness and adversarial robustness as the metric to measure the potential of BaSS strategy. Specifically, (i) the clean accuracy is evaluated on both CIFAR10 and CIFAR 10-B test set. (ii) To evaluate the corruption robustness, we first construct a corruption version for CIFAR 10-AR, referring to the construction method of CIFAR (0) and denotes as
IO-BC.. We then calculate the mean accuracy across 19 corruption types and 5 severities from
CIFARIO-C and CIFARIO-B-C. (iii) We perform PGD attack (30] with 20 steps and Eo constraint
E = 1/255, 2/255, 4/255, 8/255 to evaluate ad versariil robustness.
CIFARIO-B Data Augmentation. The results in Fig. Lindicate that as the proportion of CIFARIO- images increases (from Fig. 4(a) to Fig. 4(c)). the model converges more on the high-frequency components (the black stars closer the local minimum). The results from Fig. 33a] Fig((())
and Fig. 3 indicate that the data- augmentdd model achieves a better trade-off between robustness and accuracy. Specially, the corruption robustness is improved across all categories, further achieving a more balanced performance. The enhancement of adversarial robustness under small perturbation e=1/255, 2/55) is quite significant, with the improvement magnitude positively correlated with the mixing ratio. However, the issue of model robustness under large perturbations still remains.
CIFAR10-B as an New Dataset. The naturally trained model achieves a better robustness- accuraacy trade-off on the CIFARIO-B than on the CIFAR- 10 dataset, as shown in Fig 3(J) What attracts us is the improvement in adversarie robustness. From the more detailed results in the Appendix 1.2.1, the performance of CIFAR1O- is superior to that of CIFAR- 10 and all mixed datasets. Based on the anaylsis shown in the Appendix F.2.1, observed that the model trained the CIFARI0-B dataset has a more balanced gradient distribution across different frequency domains. This indicates that when facing adversarial attacks, the model's vulnerabilities aren't concentrated in the high-frequency direction of the image. As a result, the adversarial attack process needs to generate adversarial noise containing patterns from multiple frequency domains. This insight suggests that the CIFAR 10-B dataset in place of the original dataset during the ad adversarial training process.
4.3 BaSS Working in Conjunction with Adversarial Training
Adversarial Training has been the most successful defense strategy, where models are explicitly trained to be robust in the presence of adversarial attacks 30 utilize the BaSS strategy to modify the input image and train models conjunction with PGD-AT
2
45 In this subsection, We
30J, TRADES (45 and MART [4)].
8

---

The model training process Table 1: Clean and Robust Accuracy ()) on CIFARIO and CIFARI00.
ResNet-18 Clean
WRN-34
ResNet-18
WRN-34
CIFARIO
CIFARI00
PGD
Clean
PGD2
AA
Clean
PGD
AA
Clean
PGD"
PGD-AT
83,50
52.72
48.90
86.71
55.13
51.98
PGD-AT
56.33
29.29
24.88
61.78
30.48
27.93
+BaSS
89.22
59.61
55.90
91.08
63.65
60.91
+BaSS
63.91
31.61
27.95
67.07
34.87
32.21
+5.72
+6.89
+7.00
+4.37
+8.52
+8.93
A
+7.58
+2.32
+3.07
+5.29
+4.39
+4.28
TRADES
82.15
52.50
49.05
85.66
56.54
53.77
TRADES
58.30
29.90
25.52
61.06
31.83
27.09
+BaSS
87.20
60.01
56.27
91.20
65.38
62.69
+BaSS
64.17
32.94
27.77
69.01
37.03
33.03
A
+5.05
+7,51
+7.22
+5.54
+8.84
+8.92
A
+5,87
+3.04
+2.25
+7,95
+5.20
+5.94
MART
81.38
54.74
83.78
56.75
51.42
MART
57.80
30.73
26.26
58.10
33.46
28.36
+BaSS
88.75
60.43
56.93
89.63
63.71
59.09
+BaSS
63.94
32.95
28.01
64.02
37.79
32.20
A
+7.37
+5.69
+7.45
+5.85
+6,96
+7.67
A
+6.14
+2.22
+1.75
+5.92
+4.33
+3.84
Table 2: State-of-the-art performance on CIFARIO and CIFARI00.
Table 3: Clean and Robust Accuracy (%) Restricted-ImageNN
CIFAR10
CIFAR100
Restricted-
Method
Method
Clean
AA
Clean
AA
PGD-AT
+BaSS
TRADES
+BaSS
22
88.71
58.04
68.75
31.85
Clean PGD20
65.13
66.76
64.79
66.90
PORT
86.68
60.27
65.93
31.15
49.44
57.63
51.71
$7.96
Ours
91.20
62.69
69.01
33.03
33.37
41.77
35.50
42.80
Experiment Settings: We conducted experiments the following three datasets: CIFAR1O, CI-
FAR 100 (55 and ImageeNe [12), using ResNet-18, WideResNett34-I1 44 and ResNet- 50 [9 as backbone. We compare the performance of our method with the baseline defenses PGD-AT 30J. TRADES [45). MART [40], and also state-of-the-art defenses DAJAT (2) and PORT (34) from the Robust Bench leaderboard [TO]. To fairly evaluate each model, we assessed (a) clean accuracy (i.e., clean test data), robust accuracy under (b) PGD20 attack, the PGD attack (30 with 20 steps and
(c) AutoAttack(AA) [II] The foe attack perturbation was bounded to 8 8/255 with step size of
2/ /55 in both the training and test processes of CIFAR-10 10 and CIFAR- 100. For Restricted Restrieted-ImageNe ImageNet, 6 = 4/255 with a step size of 1 1/255. Detailed settings are available the Appendix F.2.2.
Results. As shown in Tab. T and Tab. 2 by processing the dataset with BaSS strategy, our method consistently achieves better robust-accuracy trade-off across datasets and architectures. Moreover, as shown in Tab. 3, our approach trained with WRN-34-IP 10 achieves state-of-the-ar performance compared with existing defenses on CIFARIO and CIFAR Adversarial training has been observed to exhibit a more pronounced data hunger compared to natural training 22] Existing defense methods that are capable of improving both generalization and robustness performance typically involve adversarial training on an enlarged training set [41 18), which is accompanied by substantial training overhead. It is worth noting that BaSS strategy barely increase the training cost (including the training time and the data size, comparison details are available in the Appendix F.2.2), and achieve a competitive performance. This further demonstrates the effectiveness of BaSS.
4.4 BaSS Working in Conjunction with AugMix
AugMix 21 generates complex and diverse images through data augmentation, improving the model's robustness to different types of corruption. Specifically, for the original data x, Tougmiz are generated by combining multiple simple data augmentation, then consistency constraint is applied to the Fangmiar and x during the training process. Experiments typically involve two AugMix processes to obtain Tougmizl and Taagmiz2. To verify that our method of increasing data information, BaSS, can be combined with the AugMix method, we perform BaSS operations on one of the augmented input data æ to obtain _ and then perform AugMix obtain zb Faugmirll
constrains the consistency of x, = and Taugmiz2-
Experiment Settings: We conduct experiments on CIFAR10 and CIFAR and utilize CIFAR 10-B and CIFAR 100-B B in training process as mentioned above. We use various architectures including ResNet-50, WidcResNeet40--2 DenseNet-29 22] and ResNeXt 42]. During the testing phase,

---

Table 4: AugMix improves both clean accuracy and corruption robustness combined with BaSs.
Robust Acc (%)
Dataset
Model
Method
Clean( %
Noise
Blur
Weather
Digitul
Mean
AugMix +BaSS
95.66
84.62
91.31
91.95
90.90
WRN-40-2
95.83
88.66
91.37
93.04
90.77
90.96
AugMix
95.79
83.08
90.54
91.72
90.00
89.05
DenseNet
CIFARIO
+BaSS
95.99
88.36
91.47
93.14
90.53
90.87
AugMix
96.29
82.99
91.63
92.62
91.38
89.94
ResNeXt-29
+BaSS
96.08
89.37
92.10
93.43
90.94
91.44
AugMix
95.95
85.15
91.56
92.21
91.37
90.29
ResNet-50
+BaSS
96.00
89.85
91.90
93.27
91.53
91.64
AugMix
77.53
55.43
69.08
68.24 28.24
67.13
65.41
WRN-40-2
+BaSS
77.90
60.38
69.07
70.74
66.92
66.91
AugMix
77.46
51.83
67.74
67.49
65.19
63.53
DenseNet
CIFARI00
+BaSS
78.44
59.08
68.89
70.31
65.93
66.19
AugMix
78.68
55.93
70.12
68.82
67.92
66. 16
ResNeXt-29
+BaSS
80.17
60.02
71.27
72.47
68.38
68.24
AugMix
79.01
58.90
71.26
69.95
69.74
67.90
ResNet-50
+BaSS
80.83
64.95
72.24
73.33
69.96
70.21
evaluated the clean accuracy the test sets of CIFARIO and CIFARIO0, and assessed the corruption robustness on CIFAR10-C and CIFAR100-C (0) datasets.
Results. Our method further improves the performance of AugMix as shown in Tab. achieving better trade-off between generalization and robustness across all model structures. Different classes of corruption usually have their own frequency-domain characteristics [43] thus leads to bias towards certain type of corruption existing defense methods (e.g., adversarial training and AugMix). Notably, our method significantly enhances the model's capabilities in defending against high-frequency patterns corruption (c.g., improving Noise corruption robustness by approximately 4%). This improvement enables the model to achieve more balanced defense, indicating that our method has indeed enhanced learning HFC.
We noticed that the BaSS strategy did not help the model's robustness in the "Digital" column of the in Tab. 4 Different types of perturbations usually act on different frequency bands of the image 43 bringing the challenge of diversity of frequency domain information. The Fourier heatmaps 43 show that the type of "Digital" corruption have a common feature: irregular range perturbations exist from low to high frequencies in the image. AugMix resists the above-mentioned irregular frequency band perturbations more efficiently by adopting random data augmentation strategies, producing images with better diversity in frequency domain information. The BaSS method we provided replaced one of the random data augmentation branches AugMix, thus affecting the original performance of AugMix under the "Digital" type perturbations to some extent.
Conclusions
In this paper. we have gained a deeper understanding of the relationship between high frequency components (HFC) and model robustness by defining the frequency long-tailed problem. Observations and analyses have revealed that (i) the high sensitivity to HFC stems from the model's under-fitting behavior of HFC, (ii) the cause of the model's 's under-fitting behavior is attributes to the limited information content of HFC. Based on these insights, we have proposed balanced spectrum sampling strategy (BaSS) that effectively improves the trade-off between robustness and accuracy when combined with different training paradigms. Our current results are mainly focused on the image classification tasks while we believe it is also rewarding to develop new techniques to other vision tasks. We also expect that this work could inspire a future explanations and defense method from model perspective such as, exploring model-side improvement that can work with BaSS strategy.
10

---

Acknowledgments
This work supported by the Beijing Natural Science Foundation (No.JQ20023) and the National Natural Science Foundation of China (No. 61832002).
References
[1] Antonio A Abello, Roberto Hirata, and Zhangyang Wang. Dissecting the high-frequency bias in convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 963-871, 2021.
[2] Sravanti Addepalli, Samyak Jain, et al. Efficient and effective augmentation strategy for adversarial training. Advances in Neural Information Processing Systems, 35:1488-1501 2022.
[3] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recognition. Advances in Neural Information Processing Systems, 34:27826-27839 2021.
[4] Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transformers by revisiting high-frequency components. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 1-18. Springer, 2022.
[5] Christine L Borgman, Milena S Golshan, Ashley E Sands, Jillian Wallis, Rebekah Cum- mings, Peter T Darch, and Bernadette M Randles. Data management in the long tail: Science, software, and service. International Journal of Digital Curation, 11(1): 122-229,1-9 2016.
[6] Mu Cai, Hong Zhang. Huijuan Huang, Qichuan Geng, Yixuan Li, and Gao Huang. Frequency domain image translation: More photo-realistic, better the IEEE/CVF International Conference on Computer Vision, pages 13930-13940, 2021.
In Proceedings of
[7] Alvin Chan, Yew-Soon Ong, and Clement Tan. How does frequency bias affect the robustness of neural image classifiers against common corruption and adversarial perturbations? In Proceedings of the International Joint Conference on Artificial Intelligence, pages 659-665, 2022.
[8] Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. Amplitude-phase recombination: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 458-467,
2021.
[9] Yiting Chen, Qibing Ren, and Junchi Yan. Rethinking and improving robustness of convolutional neural networks: shapley value-based approach in frequency domain. Advances in Neural Information Processing Systems, 35:324-337, 2022.
[10] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670 2020.
[11] Francesco Croce and Matthias Hein. Reliable evaluation of ad adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
[12] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-2888 255.. leee, 2009.
[13] Ranjie Duan, Yuefeng Chen, Dantong Niu, Yun Yang. Kai Qin, and Yuan He. Advdrop: Ad- versarial attack to dnns by dropping information. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7506-7515, 2021.
[14] Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEFF//CV conference on computer vision and pattern recognition, pages 7890-7899, 2020.

---

[15] Lijie Fan, Sijia Liu, Pin- Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining finetuning? Advances in neural information processing systems, 34:21480-21448 2021.
[16] Leon A Gatys, Alexander S Bcker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
[17] lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572) 2014.
[18] Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems,
2021.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[21] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 1tt.iv:1112..07.... 2019.
[22] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[23] Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
[24] Mahyar Khayatkhoei and Ahmed Elgammal. Spatial frequency bias in convolutional genera- tive adversarial networks. In Proceedings of the AAAI Conference Artificial Intelligence. volume 36, pages 7152-7159, 2022.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[27] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[28] Jiawen Li, Lu Meng, 7elin Zhang, and Kejia Yang. Low-frequency, high-impact: Discovering important rare events from ugc. Journal of Retailing and Consumer Services, 70: 103153, 2023.
[29] Zhiyu Lin. Yifei Gao, and Jitao Sang. Investigating and explaining the frequency bias in image classification. In Proceedings of the International Joint Conference on Artificial Intelligence. pages 717-723, 2022.
[30] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 706.06083, 2017.
[31] Shishira R Maiya, Max Ehrlich, Vatsal Agarwal, Ser-Nam Lim, Tom Goldstein, and Ab- hinav Shrivastava. A frequency perspective of adversarial robustness. arXiv preprint arXiv:2111.0086/ 2021.
[32] Gabriel Pereyra, George Tucker, Jan Chorowski, Fukasz Kaiser, and Geoffrey Hinton. Reg- ularizing neural networks by penalizing confident output distributions. arXiv preprint arX1v:1701.06548, 2017.
12

---

[33] Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J Di- Carlo. Large-scale, nigh-resolution comparison of the core visual object recognition behavior of humans, monkeys, and sate-of-the-an deep artificial neural networks. Journal of Neuroscience, 38(33):7255-7269, 2018.
[34] Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang. and Prateck Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? arXiv preprint arXiv:2104.094 2021.
[35] Jiachen Sun, Akshay Mehra, Bhavya Kailkhura, Pin-Yu Chen, Dan Hendrycks, Jihun Hamm, and Z Morley Mao. spectral view of randomized smoothing under common corruptions: Benchmarking and improving certified robustness, In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IV, pages 654-671. Springer, 2022.
[36] Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfel- low, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv: 1312.6199 2013.
[37] Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang. Invariant feature learning for generalized long-tailed classification. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXIV, pages 709-726. Springer, 2022.
[38] Yusuke Tsuzuku and Issei Sato. On the structural sensitivity of deep convolutional networks the directions of fourier basis functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 51-60, 2019.
[39] Haohan Wang, Xindi Wu, Zeyi Huang. and Eric P Xing. frggffreyynccy component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8684 8694, 2020.
[40] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning representations, 2019.
[41] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. arXiv preprint arXiv:2302.04638 2023.
[42] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[43] Dong Yin, Raphael Gontijo Lopes, Jon Shiens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 32, 2019.
[44] Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07146,222 2016.
Wide residual networks
arXiv preprint
[45] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off' between robustness and accuracy. In International conference on machine learning, pages 7472-7482. PMLR, 2019.
[46] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
13


Reviews:
[{"review_id": "KKlIP0E4NC", "reviewer": "Reviewer_fSNP", "summary": "This paper addresses the problem of model robustness, which is of great importance in the machine learning field. The authors revisit this problem and discuss the relationship between high-frequency components and model robustness. They verify the effectiveness of the proposed method on several datasets compared to other baselines.", "questions": "(Just a remark) The figures in the supplementary materials are intuitive, and reorganizing the paper could improve its readability.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper addresses the model robustness problem, which is of great importance to the machine learning field.\n\nThe authors verify the effectiveness of the proposed method on several datasets compared to other baselines.\n\nThe proposed method is theoretically sound with supporting proofs.\n", "weaknesses": "How many times were these experiments conducted, and is it necessary to report the mean and standard variance of the results?\n\nI noticed that the current experiments were mainly conducted on toy datasets, such as CIFAR or ImageNet subsets. What about scaling up the experiments to large-scale datasets in the real world?\n\nThe current experiments mainly focus on the PGD attack. Have you considered experimenting with other types of attacks to evaluate the performance of the proposed method in different scenarios? Additionally, apart from the single baseline PGD-AT, have you tested other baselines? Is the improvement consistent for all variants?\n\nAlthough the proposed method is supported by theoretical proofs, it seems relatively naive. Further robustness analysis is needed for some hyperparameters in Eq. 4.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for your time and effort in evaluating our manuscript. We have responded to each question in detail below：\n- **Q1:Have you considered experimenting with other types of attacks to evaluate the performance of the proposed method in different scenarios? Additionally, apart from the single baseline PGD-AT, have you tested other baselines? Is the improvement consistent for all variants?**\n  - **Rich attack scenarios:** We have evaluated on two types of model robustness that are of primary concern in the community: adversarial robustness and corruption robustness. For evaluations of adversarial robustness, we used  the efficient attack algorithms: PGD, CW, and AutoAttack(AA). The results are presented in Tab1-3 in the paper, and Tab2 in the appendix. For evaluations of corruption robustness, we selected 15 pixel-level image corruption algorithms and set five intensity levels under each perturbation algorithm, summarizing all the algorithms into four categories as shown in Tab4. The results in Tab1-4 show that our proposed method can effectively defend against all the above attack scenarios.\n  - **Adding more baselines:** In the paper, we tested two adversarial training baselines: PGD-AT and TRADES. We report results for **TRADES on the ImageNette below**. In addition, we have report  results on the latest adversarial training baseline, MART in **Fig3 in public pdf**.\n    | Method     | Clean acc | Attacked by PGD-10 | Attacked by AA |\n    | ---------- | --------- | ------------------ | -------------- |\n    | TRADE      | 64.79%    | 51.71%             | 35.50%         |\n    | TRADE+BaSS | 66.90%    | 59.96%             | 42.80%         |\n\n    The above results demonstrate that BaSS method can be applied to enhance the robustness of various  baseline algorithms and ensures stability under various attack algorithm tests.\n- **Q2:  What about scaling up the experiments to large-scale datasets in the real world?**\n  - **R2:** Conducting experiments on adversarial defenses in large-scale datasets is not an easy task. Due to issues with computational resources and time, we were unable to provide you with relevant experimental results during the rebuttal phase. By referring to related work in the same field[1-2], our experimental results on the ImageNet subset already possess convincing evidence for evaluation on large datasets. Additionally, to address your concern, we summarize the advantages of BaSS, which can support training on even larger-scale datasets.\n    - **Low Time-Space Complexity.** BaSS  can be regarded as a pre-processing operation on images. Therefore, it will not incur any additional overhead in memory resources; Moreover, the results in Appendix Tab3 indicate that bass almost does not increase the time overhead during the training process.\n    - **Hyperparameter-Free.** For the hyperparameters of the training process, the algorithm with BaSS adopts the same settings as the baseline. Therefore, it does not have to face issues such as hyperparameter optimization. As a result, even training on larger datasets will not increase the difficulty of optimization.\n    - **Furthermore**, model robustness is widely considered to be sensitive to the resolution of the training images and the number of categories in the dataset. In our experiments, we set up comparative experiments for the above factors. The experimental results show that BaSS can effectively enhance model robustness across datasets with multiple different resolutions and different numbers of categories.\n- **Q3: How many times were these experiments conducted, and is it necessary to report the mean and standard variance of the results?**\n  - **R3:** Thank for your question and suggestion, our experimental results are based on  3 random runs, the standard deviations of 3 runs are very small, which hardly effect the results. In the **fig1 and fig2 of public pdf**, we reported the results and the standard deviations of 5 runs, which are still very stable.\n- **Q4: Although the proposed method is supported by theoretical proofs, it seems relatively naive. Further robustness analysis is needed for some hyperparameters in Eq. 4.**\n  - **R4:** To address the concern of the reviewers, we would like to clarify the motivation behind the theorem  and conclude its importance. \n     - Both theorems share the background of representing image distributions in frequency domain and address the long-tail problem in image features instead of a simple calculation formula, crucial in analyzing image data influence on vision model performance. \n     - Theorem 1 innovates in frequency entropy analysis, being superior to conventional image entropy based on grayscale. Its benefits include: (i) Reducing sensitivity to noise; (ii) Compensating for spatial relationship limitations; (iii) Aligning more with human image perceptions.\n     - Theorem 2 combines maximum entropy and theorem 1 for practical value, inspiring the method in Section 4. The results in Section 5 demonstrate its effective application to enhance model robustness-accuracy.\n    In Eq. 4, where the parameter *B* is determined by the size of  image, and parameter *τ* is the constant *e*, we will provide a clearer explanation during the revision stage. We do not wish to introduce hyperparameter settings to the method, thereby increasing its training difficulty across different datasets. From another perspective, the adjustment of parameters might lead to better results, which could become a direction for optimizing BaSS in our future work.  \n\nWe will continue to follow and respond to any questions regarding the content of the article following the review. Meanwhile, we sincerely hope that the reviewer could reassess the contributions of our work.\n\n**References**\n[1]  Efficient and effective augmentation strategy for adversarial training. NeurIPS 2022\n[2] Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness? .ICLR 2022"}, {"review_id": "oFvzDV0ngn", "reviewer": "Reviewer_48f5", "summary": "This paper studies the effect of power imbalance across frequencies in natural image datasets on the accuracy and robustness of neural net classifiers, and empirically argues that the model is underfitted on frequencies with lower power concentration. Then, the paper proposes an augmentation strategy to improve the balance of power distribution per image, and evaluates its proposed method on several datasets and tasks to illustrate its effectiveness in boosting accuracy and robustness to data corruption and adversarial attacks.\n", "questions": "See the weaknesses section.", "limitations": "See the weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Studying the effect of spectral power imbalance on the accuracy and robustness of neural networks is a novel and significant effort, and the paper provides several interesting experiments to provide insights into this effect. The proposed augmentation method seems to be effective, and works well in combination with other existing augmentations.", "weaknesses": "I think this is a valuable paper overall, however my main concern is a lack of clarity in its explanations and arguments, making several claims that seem incorrect and unjustified to me. Also, the experiments need better organization and more clear and nuanced explanations. I will elaborate my specific concerns and questions below.\n\n1. L115 claims “Inspired by the Pareto principle [34] in long-tailed theory, we propose a stable and quantitative approach”, the paper must be more specific on how the Pareto principle is inspiring the proposed approach and in what sense it is more stable, otherwise such broad statements hinder clarity and readability.\n\n2. Eq 2 set definitions are unclear because r_n is undefined. The explanation given in L117-120 is also unclear since “region” is undefined. Please be more careful and specific when writing definitions that form the foundation of a paper.\n\n3. L133 claims “On one hand, the tail part is often overlooked by the system due to its numerical disadvantage”, what does numerical disadvantage mean? Please be more specific in your statements.\n\n4. In Fig 1 and Section 3.1, the details of how the loss is computed separately for HFC and LFC are missing. Also the colorbar is missing.\n\n5. In Fig 2, top ranks having distributions centered towards the higher frequencies does not mean that the loss is most sensitive to higher frequencies. Please plot Gradient Magnitude versus Frequency Radius to directly observe which frequencies will have the largest effect on loss (largest average magnitude). If lower frequencies receive larger average magnitudes than higher ones, then it cannot be claimed that “perturbations in HFC are more easier to cause errors in the model” as is done in L198.\n\n6. Theorem 1 is just applying the formula plog(p) to the assumed density of p for natural images, this simple application does not need to be a theorem.\n\n7. The paper does not explain or cite why plog(p) is a sensible measure of information on the normalized spectrum (density of p). Note that the normalized spectrum does not have any immediate meaning as a probability distribution, it is a power density function. In particular, the claim in L223 that “This indicates that high-frequency bands in the tail contain substantially less information than low-frequency bands in the head” is misleading, since all that Theorem 1 indicates is that power is concentrated at lower frequencies which is immediate by the assumption on the density of p.\n\n8. Theorem 2 seems immediate from information theory. Could you explain why not simply state that the uniform distribution has maximum Shannon entropy instead of stating Theorem 2?\n\n9. The explanation of the BaSS in section 4.1 is not the same as shown in Algorithm 1 in the Appendix, in that there is no actual sampling happening in Algorithm 1, it is a reweighting of the spectrum based on the ratio of log power density over power density.\n\n10. In the experiments in Tables 1-4, the value of gamma is not specified. It is important to show an ablation study on gamma in all these settings (Fig 3 seems to be doing this, but it is unclear which plot is corresponding to which specific experiment setup from Tables 1-4). Also, the results lack error bars and therefore the significance of the improvements is unclear.\n\n11. Please include ImageNette in Fig 3 results to observe the effect of varying the dataset.\n\n12. In all Tables 1-4, reporting the standalone performance of BaSS is also important. Also, in some situations in Table 4 where BaSS does not help, it is important to comment on why that might be happening. \n\n13. Discussion of related works should be part of the main paper and not in Appendix.\n\n***Some typos***\n\nL1 claims “A widely discussed hypothesis regarding the cause of visual models’ robustness is that they can exploit human-imperceptible high-frequency components”, this seems to be a typo, “models’ lack of robustness” perhaps?\n\nL125: “has not yet to be fully explored” should be “has not been fully explored”.\n\nL136: “regards” be “regarding”.\n\nL321: “worth noted” be “worth noting”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the reviewers for suggestions that help improve the quality of our work，we make the following responses (e.g., R1 is the response to Q1):\n- **R1:In Line113-117, we have revised it as:**\"Previous analysis indicating that a few low-frequency components form the image's main content, while most high-frequency components comprise a minor part. The rule that few factors determine main outcomes is encapsulated by the Pareto Principle and the \"80/20\" quantitative rule, found across various domains, where 80% of effects are caused by 20% of the reasons. Guided by this rule, we defined the boundary between frequencies using an 80/20 spectral energy ratio.\" **In Line120, we added the following:**\"We experimented with various datasets, finding that different resolution images' high-low frequency boundaries are at a 20% radius from the spectrum's center(see Appendix B.2). This confirms the stability and reasonableness of the classification system using spectral energy and the Pareto Principle.\"\n\n- **R2: We added the following on eq2**: \"$r_n$ is the Euclidean Distance from the nth  frequency band to the center of the spectrum\". We give the definition of $r_k$  in L94, $r_n$ and $r_k$ are the differences in footmarks. **We replaced \"the region\"  with** \"the set composed of $\\tilde{X}[u,v]$ \".\n\n- **R3:** Considering that the numerical definition will be different in different long tail, **we added the content in L133:** \"(e.g., the sample size of the tail class  is far less than that of the head class in the context of class-wise long tail)\". \n\n- **R4:**  We have described the details of calculating the loss of high and low frequencies in the Appendix.C. As the complement of  the contour values, we will add colorbar in the revision stage. \n  \n- **R5:**  We have implemented the reviewers' recommended visualization method on **Fig1 in public pdf**, and the results show higher gradient amplitude distributions in the HFCs across three datasets, supporting our paper's conclusions. To further address the reviewers' concerns, we consider two points:\n\n  -  Evolutionary trends on five density curves within the top 80%  show specific patterns(e.g., the peak of the density curve on the Tiny-ImageNet dataset exhibits minimal movement). And data points with density values greater than 0 are predominantly situated in the tail regions, indicating the peak will likely not be in the head region.\n  - Instead of solely observing the mean value, which might overlook distribution diversity and abnormalities, we argue that Kernel Density Estimation offers a more accurate representation of the final results.\n- **Response to concerns about theorem in Q6-Q8:** \n    Spectrum of single image is a statistical observation from a probability distribution, reflecting different frequency modes' probabilities. (e.g., high-frequency components are only found in specific locations, such as contours).  Therefore, our definition of probability distribution is based on different frequency domain patterns in frequency domain space as random variables.\n    - Both theorems share the background of representing image distributions in frequency domain and address the long-tail problem in image features instead of a simple calculation formula, crucial in analyzing image data influence on vision model performance. \n    - Theorem 1 innovates in frequency entropy analysis, being superior to conventional image entropy based on grayscale. Its benefits include: (i) Reducing sensitivity to noise; (ii) Compensating for spatial relationship limitations; (iii) Aligning more with human image perceptions.\n    - Theorem 2 combines maximum entropy and theorem 1 for practical value, inspiring the method in Section 4. The results in Section 5 demonstrate its effective application to enhance model robustness-accuracy.\n  \n- **R9:**  Image features like spatial pixels and spectral density are not explicitly selectable by the sampling theorem. Thus, inspiring from image sampling in pixel domain, we use the method : (1) Discretizing features with Fourier Transform; (2) Selecting and merging values from various regions, re-weighting different frequency band positions as per Eq.4, and creating a new image through inverse Fourier Transform.\n\n- **R10:**  In Section 4, we outline the purposes and significance of the experiments: (1)Section 4.2 details heuristic experiments to explore the bass strategy, using the gamma parameter for comparing strategies and results.(2)Sections 4.3 and 4.4 present solutions ON two robustness problems (adversarial and corruption) using the bass strategy, with algorithms designed for specific training without gamma-related hyper-parameters. For concerns of error bars, we show the results based on 5 random runs in **Fig1-3 of the public pdf**.\n- **R12**: The individual BaSS results are  shown in Appendix Tables 6-9,  we will reorganize the contents of Table 4 in appendix. For the model's decreasing robustness in the \"digital\" column of Table 4,  our analysis is:   [1] shows that the digital subcategory  share a common feature: irregular range corruptions from low to high frequencies. AugMix resists them by adopting random data augmentation, enhancing frequency information diversity. The bass method we provided, replacing one AugMix branch,  may affect diversity of frequency domain information in the augmentation data.\n\n- **Other issues**:\n  - (Q11) We have added ImageNette in Fig3 in Appendix.F2.\n  - (Q13) Due to space limitations, please refer to Q1 response onReviewer P5Z5\n  - typos error: We fixed these issues and reviewed the full article.\n\nWe follow your valuable advice, clarify the statement and add more experimental results. We sincerely hope that you could reconsider the overall performance and contribution of our articles. We continue to pay attention to any follow-up questions and actively discuss them.\n\n**Reference**\n\n[1]  A fourier perspective on model robustness in computer vision, Yin et al."}, {"review_id": "8uU553AlPz", "reviewer": "Reviewer_P5Z5", "summary": "Using the long-tail features in the spectrum, this paper investigates the model's sensitivity to HFC and discovers patterns of underfitting behavior due to limited information content. The authors propose the Balance Spectrum Sampling (BaSS) strategy, which enhances model learning on HFC. Experimental results show that combining BaSS with existing defense methods improves model performance.", "questions": "1. Is the proposed method also applicable to other visual models, such as Vision Transformer (ViT), in addition to CNNs?", "limitations": "No potential negative societal impact", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces an interesting perspective by focusing on the long-tailed distribution in the frequency domain and revisits the relationship between high-frequency components (HFC) and model robustness.\n2. The under-fitting behavior is explained through the lens of spectral entropy, offering a clear understanding of the low information entropy of HFC.\n3. The proposed method achieves improved performance.", "weaknesses": "1. The paper lacks a discussion on the existing literature and its relevance to the topic in details, which could provide a stronger foundation for the research. The content in the Introduction section may not be sufficient.\n2. More experiments are needed to evaluate the proposed method:\n1) The proposed BaSS should be evaluated in combination with more baseline methods. For instance, MART[1] is a method that outperforms TRADES in performance. It would be insightful to see how BaSS performs when incorporated with such stronger baselines.\n2) The paper lacks a comparison with other methods, such as AWP[2]. It would provide a more comprehensive evaluation and analysis of the proposed approach.\n3. There are some small grammar mistakes in the paper. For instance, in the Abstract, it should be \"counteracts and enhances\" instead of \"counteract and enhance\" in lines 16-17. More attention should be given to checking the grammar details throughout the paper.\n\n[1] Improving Adversarial Robustness Requires Revisiting Misclassified Examples. ICLR2020\n[2] Adversarial Weight Perturbation Helps Robust Generalization. NeurIPS2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for affirming the innovativeness of our article and the positive comments about the writing and experiments and we respond to your questions and suggestions below:\n\n- **Q1: The paper lacks a discussion on the existing literature and its relevance to the topic in details, which could provide a stronger foundation for the research. The content in the Introduction section may not be sufficient.**\n\n  - **R1:** Due to the space constraints, the section on related works is presented in the Appendix A. In order to ensure the integrity of the content of the article, we have introduced the core content of the relevant work and the topic in the introduction section. We divided the literatures related to the topic and the corresponding discussions into two parts:\n    - The first section of the related work focuses on the keywords **frequency domain** and **robustness**. In lines 31-40 of the introduction, we presented a discussion on analyzing model robustness in the frequency domain, providing an overview of relevant studies. Additionally, we highlighted two limitations present in existing research, which served as the driving force behind the motivation and problem formulation addressed in our work. A comprehensive description of each individual work is provided in Appendix A.\n\n    - The second part of the related work revolves around the concept of the **long tail**. In lines 48-51 of the introduction, we expounded on the distinct characteristics of the long-tail problem in visual tasks. Building upon this, we drew inspiration from the long-tail distribution observed in the image frequency domain, prompting the formulation of a novel long-tail problem in visual tasks, namely, the instance-wise long-tail problem. Correspondingly, a comprehensive account of the long-tail problem in visual tasks and classical solutions can be found in Appendix A.\n\n    Thank you for your advice, we will carefully consider the location of the references during the revision phase.\n\n  \n\n  \n\n  \n\n- **Q2: Is the proposed method also applicable to other visual models, such as Vision Transformer (ViT), in addition to CNNs?**\n\n  - **R2:** This is a significant and practical question. We endore the importance of considering transformer-based structures (e.x. ViT) and the large-scale dataset (e.x. ImageNet) in data anaylsis,  and we have obtained consistent results regarding the main conclusion of this paper which the robustness issue of vision models mainly stems from the high-frequency components (details is shown in see Appendix D). This indicates that our proposed long-tail scenarios can be well-extended to transformer-based vison models. \n\n    Considering that the ViT model accepts image patches as inputs, wherein each patch itself exists long-tail characteristics in the frequency domain. We believe that the idea of balancing image spectrum energy is also applicable to the improvement of the efficiency of ViT models. In particular, several studies have suggested improving the model's performance by optimizing the utilization of the high-frequency spectrum, which further supports our hypothesis.  However, due to the fact that conducting adversarial training of ImageNet in the ViT model requires substantial computational cost. As a result, we did not perform corresponding experiments in this paper. Nevertheless, we remain committed to exploring and verifying these aspects in our future work.\n\n\n\n- **Q3: More experiments are needed to evaluate the proposed method:**\n\n  - **combination with more baseline methods** \n\n    We supplemented the experiments with MART, as well as the combined applications of experiments on both BaSS and MART. The experiments selected the ResNet18 and WRN34 models and used the CIFAR-10 and CIFAR-100 datasets. We show some of the experimental results below (WRN34, CIFAR-10 over 5 random runs), with detailed results available in a PDF file shared with all authors.\n\n    | Method    | Clean acc  | Attacked by PGD-20 | Attacked by CW | Attacked by AA |\n    | --------- | ---------- | ------------------ | -------------- | -------------- |\n    | MART      | 84.10%     | 57.95%             | 55.53%         | 53.83%         |\n    | MART+BaSS | **89.75%** | **64.43%**         | **63.21%**     | **61.93%**     |\n\n    Our method demonstrates stable performance improvements over superior benchmark methods. We have added MART's results to Table 1 in the main text and included the corresponding references.\n\n  - **comparison with other methods**\n\n    We have compared two types of methods for enhancing robustness in the paper: one is the classical baseline method (which corresponds to a previous question from the reviewers), and the other is the current state-of-the-art (SOTA) method, with SOTA methods referring to the rankings published by Robust Bench. We believe that the author's question corresponds to the comparison of the second type, and following the author's suggestion, we have added Table 3 in the main text to include AWP and other excellent adversarial defense methods, along with comparative analysis and references.\n\n  \n\n- **Q4 There are some small grammar mistakes in the paper. For instance, in the Abstract, it should be \"counteracts and enhances\" instead of \"counteract and enhance\" in lines 16-17. More attention should be given to checking the grammar details throughout the paper.**\n\n  - **R4:** We sincerely thank you for carefully reviewing our article and identifying these issues for us, and we will fix and try to eliminate similar syntax problems in revision.\n\n\n\nWe look forward to following up discussions with you and responding to your follow-up questions, reminding you that detailed  results in Q3 are presented in pdf material shared by all reviewers."}, {"review_id": "bWNgeSpLKC", "reviewer": "Reviewer_oe9n", "summary": "This paper tackles the problem of visual models' robustness from the view of frequency components. Inspired by the long-tailed characteristic observed in frequency spectrum, the authors empirically prove that standardly trained models are sensitive to HFC. After analyzing its reasons, the authors proposed a sampling method and verified its effectiveness. ", "questions": "In Figure 2 second row, why does on Tiny-ImageNet and ImageNet (as opposed to CIFAR10), the tail ranking indices (e.g. \"80%\" and \"Last one\") have a high density in the high frequency area? This inconsistency may draw doubts on the generalizability of the conclusions the paper gets. ", "limitations": "No. The authors should add it.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors focuses on a novel perspective of long-tailed distribution in the frequency domain to analyze and improve model performance.\n2. The paper provides interesting insights into this problem, revealing the underlying properties of the studied problem.\n3. Abundant experiments and analysis are done. \n4. The paper is well-written.\n", "weaknesses": "I am not an expert in this area, thus my comments should be treated with caution.\n1. Baseline comparisons: BaSS is a new sampling strategy, does there exist any other sampling methods for comparison? PGD-AT is from 2017 and is rather an old method.\n2. It would be better to include: the difference in frequency distribution before and after BaSS is adopted.\n3. Flaws in writing. E.g. Typo in Table 2 title.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your recognition of the novelty of our work！We set out our answers to each of your questions.\n\n- **Q1: Baseline comparisons: BaSS is a new sampling strategy, does there exist any other sampling methods for comparison? PGD-AT is from 2017 and is rather an old method.**\n\n  - **Other methods for comparison?** \n\n    BaSS introduces an innovative sampling strategy for instance-level long-tail challenges, i.e., the long-tail distribution is present within the image frequency domain. Actually, we have not encountered any sampling techniques designed for image features. Prevailing efforts have predominantly concentrated on long-tail scenarios delineated by class or attributes, diverging substantially from our framework centered around the frequency long-tail issues. As a result, a comparative analysis with these methodologies was deemed inappropriate.\n\n  - **Concerns about the PGT-AT that is an old method.**\n\n    We would like to reaffirm our positioning on BaSS, highlighting its broad applicability to a wide range of adversarial training frameworks. We selected two of the most basic adversarial training frameworks, PGT-AT and TRADES. To address your concerns, we expand our experimental scope to include the combination of BaSS with MART. MART is the latest adversarial training framework and is widely used after PGD-AT and TRADES methods. The results are presented in publicly visible PDF material a n d provide compelling evidence that BaSS harmonizes effectively with diverse adversarial training frameworks, resulting in substantial improvements in model robustness and generalization performance.\n\n    It should be noted that we did not choose to combine with the current state-of-the-art (SOTA) algorithms, because the aforementioned algorithms usually face huge time and space overhead, which is a problem that needs to be solved in the process of improving adversarial robustness. The experimental results in Tab 1 and Tab 2 show that our sampling strategy significantly improves the performance of adversarial training model without almost increasing the time and space complexity (details in appendix. Tab 3), and even surpasses the current SOTA algorithms in some scenarios (as shown in Tab 3).\n\n- **Q2: In Figure 2 second row, why does on Tiny-ImageNet and ImageNet (as opposed to CIFAR10), the tail ranking indices (e.g. \"80%\" and \"Last one\") have a high density in the high frequency area? This inconsistency may draw doubts on the generalizability of the conclusions the paper gets.**\n\n   - **R2**: We appreciate your careful observation of our experimental results! We also need to point out that on both Tiny-ImageNet and ImageNet, the density peak of the tail ranking index (e.x. last one) is located at the very tail of the spectrum.\n\n     We wish to emphasize that this phenomenon does not impact the consistency of our conclusions drawn from datasets with varying resolutions. Considering the substantial orders of magnitude distinction between the lowest and the highest ranks (illustrated in the first row of Figure 2), we maintain that the model's sensitivity to rankings within this range can be disregarded. Consequently, the perturbations observed in these frequency directions have minimal influence on the model's robustness. Furthermore, it is acknowledged that the ultra-high frequency components in the image often correspond to noise information. This raises the possibility of the model's responsiveness to this particular aspect. Comparable findings can be validated through references [1,2].\n\n- **Other issues:**\n\n  - **It would be better to include: the difference in frequency distribution before and after BaSS is adopted.**\n\n    We visualize the BASS sampled image and the corresponding spectral density distribution curve in Figure 7 of the appendix. Among them, the high-frequency information content of the sampled image is enhanced, and the corresponding spectral energy density is more balanced. \n\n  - **Flaws in writing. E.g. Typo in Table 2 title.**\n\n    We will carefully review the full article and fix these issues during the revision phase.\n\nWe will continue to monitor the follow-up questions and discussions of the reviewer, and once again remind that the results of the Q1 experiment are presented in the pdf material shared with all reviewers.\n\n\n\n\n\n**References**\n\n[1] High-frequency component helps explain the generalization of convolutional neural networks, Wang et al.\n\n[2]  Investigating and explaining the frequency bias in image classification, Lin et al."}, {"review_id": "epcKskPhHV", "reviewer": "Reviewer_zwAk", "summary": "In this paper, the authors investigate the high-frequency components (HFC) in images from a long-tailed perspective. They revisit the relationship between HFC and model robustness and demonstrate that the cause of the model’s under-fitting behavior is attributed to the limited information content in HFC. A Balance Spectrum Sampling (BaSS) strategy, which effectively counteracts the long-tailed effect and enhances the model’s learning on HFC. Extensive experimental results demonstrate the effectiveness.", "questions": "1. Is this paper the first to analyze the problem of model robustness from the perspective of frequency long-tails?\n", "limitations": "The authors do not address the limitations in this paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Investigating the high-frequency components (HFC) in images from a long-tailed perspective for the model robustness task sounds good and very interesting.\n2. The authors give detailed definitions and analysis of the frequency long-tailed scenario.\n3. The proposed solution BaSS is simple and efficient.\n4. Experiments are conducted on a various benchmarks.\n", "weaknesses": "1. To my end, the re-sampling/re-weighting method in the long-tailed situation generally improves the recognition ability of the tails at the expense of a portion of heads, does this phenomenon will weaken the recognition ability of LFC? \n2. Some typos, like:\n    line 16, counteract -> counteracts\n    line 17, enhance -> enhances\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive comments for our work. We set out below our responses to each of the questions.\n\n- **Q1: To my end, the re-sampling/re-weighting method in the long-tailed situation generally improves the recognition ability of the tails at the expense of a portion of heads, does this phenomenon will weaken the recognition ability of LFC?**\n\n  - **R1**: Thank you for your question! Firstly, we would like to introduce a phenomenon that we have observed, which is BaSS does indeed result in a certain decline in low-frequency spectral density, as demonstrated in Figure 7 in the Appendix. This observation aligns with the trend of reducing the sampling rate or weight of the head classes. Consequently, it is valuable and intriguing to discuss whether the model's performance on LFC has been weakened or not. We present the following experimental results and would appreciate discussing with the reviewers. We have selected models based on standard training, AugMix, adversarial training, one with BaSS and the other without BaSS, and report their recognition performance on both LFC and HFC.\n    \n    |           Method                    | Clean acc on LFC  |  Clean acc on HFC  |\n    | :---------------------------: | :---: | :---: |\n    |      standart training.       | 81.59% | 18.19% |\n    |  **standart training+BaSS**   | 76.77% | 28.84% |\n    |            AugMix             | 85.40% | 23.64% |\n    |        **AugMix+BaSS**        | 82.77% | 34.10%  |\n    |     Adversarial training      | 85.57% | 9.88%  |\n    | **Adversarial training+BaSS** | 84.23% | 17.90% |\n    \n    It can be observed that the performance on LFC has shown a decrease trend. This decline can be attributed to BaSS's focus on instance-wise balance, which, in turn, reduces the amount of information available in the low-frequency range. This observation aligns with the analysis presented in section 4. Conversely, it is worth noting that the model has improved the recognition ability on high frequencies. By combining the results from Tables 1 to 4, we can draw the conclusion that performing BaSS benefits the model's recognition on high frequencies and enhances the robust and generalization performance for full image recognition. Nevertheless, it is worth noting that this approach also exhibits some negative impact on low frequencies. This observation serves as inspiration for us to explore more effective data sampling strategies or optimization algorithms on the model side in future endeavors. We have included a detailed discussion of these findings in Appendix F.2.\n\n\n\n- **Q2: Is this paper the first to analyze the problem of model robustness from the perspective of frequency long-tails?**\n\n  - **R2**: Thank you for your question. We would like to confirm that our work represents a pioneering effort in analyzing the robustness and generalizability of vision models from the perspective of frequency long-tails. While some related works have focused on the class- or attribute-wise long-tail problem in image recognition tasks (refer to Line 29-31 in the Appendix), the core contribution of our work lies in introducing a novel instance-wise (frequency) long-tailed problem definition, which also enriches the scene of visual long-tailed tasks. We thoroughly investigate and analyze the impact of the long-tailed spectrum on vision models, and propose an effective solution based on a balanced spectrum sampling strategy to address the challenges posed by the frequency long-tail distribution.\n- **Q3: Some typos, like: line 16, counteract -> counteracts line 17, enhance > enhances.**\n  - **R3**: Thank you for reading our article carefully and  pointing out these issues, and we will fix them accordingly during the revision phase.\n\nWe hope that the above answers will solve your doubts and will continue to pay attention to your follow-up questions."}]

Author rebuttal:
Dear Area Chair and Reviewers,

We would like to thank all reviewers for their thoughtful comments and insights that greatly helped improve our work. We have carefully reviewed and responded to every question from all reviewers, providing a clearer explanation of the content of the article in question, and adding more experimental results to validate our conclusions and discussions. We have summarized the following modifications and additions:

- Added more experiments

  - We supplemented the experiments with MART and MART+BaSS on the CIFAR10 and CIFAR100 datasets, and the results are shown in Tab1-3 of the PDF file.
  - We added experiments with TRADES and TRADES+BaSS on the large-scale dataset ImageNette, and the results are shown in the response to reviewer fSNP.
  - We supplemented the visualization results of fig1 in paper to further validate our conclusions, and the results are shown in fig1 of the PDF.

  Due to computational resource constraints, only partial experimental results are shown, and we will supplement the complete experimental tables during the discussion and revision stage.

- Supplement to the main text and appendix,

  - We have improved some statements in Sections 2-3 of the article, which can be seen in the response to the reviewer 48f5.
  - We added the results of fig2 on imagenette in appendix F.2.
  - We added content and corresponding experimental results of discussions with reviewer zwAk in appendix F.2.
  - We added a table contrasting the individual results of BaSS, according to reviewer 48f5 Q12.

- We have corrected all the typos issues raised by the reviewers.

During the discussion stage, we will continue to pay attention to the reviewers' follow-up questions and suggestions, and supplement the above content.


Meta-review: This paper was reviewed by five experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to NeurIPS 2023. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)