DELVING INTO FEATURE SPACE: IMPROVING ADVER-
Year: 2021

Abstract:
The study of adversarial examples in deep neural networks has attracted great attention. Numerous methods are proposed to eliminate the gap of features between natural examples and adversarial examples. Nevertheless, every feature may play a different role in adversarial robustness. It is worth exploring which feature is more beneficial for robustness. In this paper, we delve into this problem from the perspective of spectral analysis in feature space. We define a new metric to measure the change of features along eigenvectors under adversarial attacks. One key finding is that eigenvectors with smaller eigenvalues are more non-robust, i.e., adversary adds more components along such directions. We attribute this phenomenon to the dominance of the top eigenvalues. To alleviate this problem, we propose a method called Feature Spectral Regularization (FSR) to penalize the largest eigenvalue, and as a result, the other smaller eigenvalues get increased relatively. Comprehensive experiments demonstrate that FSR is effective to alleviate the dominance of larger eigenvalues and improve adversarial robustness on different datasets. Our codes will be publicly available soon.

heading: 1 INTRODUCTION
text: It is shown that the performance of Deep Neural Networks (DNNs) decreases dramatically when confronted with adversarial examples (Biggio et al., 2013; Szegedy et al., 2013; Goodfellow et al., 2015). The vulnerability of DNNs brings potential risk to safety-critical deployments (Finlayson et al., 2019). To mitigate this vulnerability of DNNs, numerous methods are proposed to improve adversarial robustness (Papernot et al., 2016b; Madry et al., 2018; Xie et al., 2019). Among these, adversarial training (AT) (Madry et al., 2018) is the most effective approach that achieves state-ofthe-art performance under various attacks (Croce et al., 2020). Different from standard training, adversarial training trains DNNs on adversarial examples rather than natural examples.
There are numerous methods proposed to eliminate the gap of features between natural examples and adversarial examples (Kannan et al., 2018; Zhang & Wang, 2019; Zhang et al., 2019). However, they constrain the features on the whole without considering the distinction of contribution from an individual feature. This may be inappropriate since every feature may play a different role in robustness. The work of (Ilyas et al., 2019) argued that adversarial examples result from non-robust features (Ilyas et al., 2019), i.e., well-generalizing but brittle features. This inspires us that we could treat each feature separately in the adversarial setting. While the work of (Bai et al., 2021; Yan et al., 2021) considered the influence of different channels on robustness, the problem has not been explored clearly from the perspective of spectral signatures, i.e., the eigenvalues and eigenvectors of feature covariance. The space is split into many spectral components, and then it is worth analyzing which component is beneficial for adversarial robustness and which is fragile under attack.
In this paper, we show that spectral signatures of deep features have a close connection with adversarial robustness. Through applying principal component analysis (PCA) to deep features, we could split feature space to various eigenvectors with its eigenvalues. Our motivation comes from the phenomenon that the standard trained model often results in a sharp distribution of eigenvalues (Yu et al., 2020), i.e., the eigenvalues rapidly become very small as shown in Figure 1. This property may be beneficial for natural accuracy (Lezama et al., 2018; Papyan et al., 2020), while its
impact on robust generalization is unclear. However, we hypothesize that the sharp distribution of eigenvalues impels models to learn less diverse features and is a cause of the vulnerability of DNNs. A minority of eigenvalues occupy the overwhelming majority in the sum of eigenvalues, which may make the model pay little attention to features along the eigenvectors with smaller eigenvalues, so features along such directions are not generalized during training. To verify our hypothesis, we define a new metric to measure the variation of features along different eigenvectors under attacks, as shown in Figure 4. Our observation reveals that the adversary tends to add more components along eigenvectors with smaller eigenvalues, and such huge variation could be distinctly alleviated by AT.
Therefore, we propose to improve adversarial robustness by alleviating the sharp distribution of eigenvalues. Considering that more eigenvalues should occupy the majority, we propose a regularizer named Feature Spectral Regularization (FSR) to penalize the largest eigenvalue of the feature matrix covariance. Empirical evidence shows that FSR increases the overall eigenvalues relatively, making models focus on more spectral components during training. We also provide a theoretical explanation with robust linear regression. Comprehensive experiments confirm that FSR indeed improves the adversarial robustness on several datasets. Our contributions are summarized as follows:
• We find a close connection between spectral signatures of features and adversarial robustness. On one hand, standard trained model presents a sharp distribution of eigenvalues, which is beneficial for natural accuracy while harmful in adversarial setting. On the other hand, the adversary tends to add more quantity along eigenvectors with smaller eigenvalues.
• We propose Feature Spectral Regularization (FSR) to increase the overall eigenvalues relatively in deep feature space, thus alleviating the sharp distribution of eigenvalues. Furthermore, we provide a theoretical explanation based on robust linear regression.
• We empirically show that FSR improves adversarial robustness and alleviates the sharp distribution of eigenvalues, through comprehensive experiments.

heading: 2 RELATED WORK
text: Adversarial Defense. Many defense methods have been proposed to improve adversarial robustness since the discovery of adversarial examples (Papernot et al., 2016b; Xie et al., 2019; Carmon et al., 2019; Zhang et al., 2020). However, many of them are proven to be noneffective because they highly depend on obfuscated gradients (Athalye et al., 2018) or gradient masking (Papernot et al., 2017). Among these, adversarial training (Madry et al., 2018) is now regarded as the state-of-theart method (Rice et al., 2020; Pang et al., 2021). Distinguished form standard training, adversarial training trains DNN on adversarial examples:
min θ max ‖δ‖≤ E(x,y)∈D LCE (x+ δ, y; θ) (1)
where, D is the dataset, the parameters of DNN are denoted as θ, δ means the perturbation within the -ball, and LCE is the cross-entropy (CE) loss.
By introducing a trade-off between robustness and generalization, TRADES (Zhang et al., 2019) is another framework that reaches comparative robustness with AT. Among the proposed methods based on AT, Adversarial Weight Perturbation (AWP) (Wu et al., 2020) explicitly regularizes the flatness of weight loss landscape, and forms a double-perturbation mechanism, which shows huge improvement on adversarial robustness and alleviates the overfitting in AT (Rice et al., 2020).
Spectral Signatures of Feature Representations. Some studies have revealed that the spectral signatures of features influence the performance in various learning tasks. The spectral properties are crucial to detect backdoors (Tran et al., 2018; Hayase et al., 2021). The eigenvectors corresponding to the larger eigenvalues are found to dominate the transferability of features in adversarial domain adaptation (Chen et al., 2019b). The work of (Chen et al., 2019a) explores the correlation between negative transfer and the spectral components of features and weights in inductive transfer learning. By utilizing the principle of Maximal Coding Rate Reduction (MCR2), it is theoretically proven that the larger several singular values of feature matrix for every class should be equal to learn the maximally diverse representation (Yu et al., 2020; Chan et al., 2020).
Different from these studies, we analyse the connection between adversarial robustness and spectral components of deep features. We aim to explore which components are more fragile under attacks, and propose a method to boost adversarial robustness by constraining spectral properties.

heading: 3 SPECTRAL ANALYSIS IN FEATURE SPACE
text: In this section, we investigate the connection between spectral signatures and adversarial robustness. Concretely, we train models by standard training and adversarial training (Madry et al., 2018), and then apply spectral decomposition to attain the spectral signatures. We find that in standard training the eigenvalues reveal a rapidly descending curve while this tendency is alleviated by AT. Then, we find that the adversary tends to add more quantity along eigenvectors with smaller eigenvalues.

heading: 3.1 CURVE OF EIGENVALUES AND ADVERSARIAL ROBUSTNESS
text: Given a dataset D = {(xi, yi)}ni=1 including C classes, xi represents the input data and yi is the label. DNN is composed of a feature extractor h (·) : RD → Rd and a linear classifier g (·) : Rd → RC . After centralizing the learned features (i.e. 1n ∑n i=1 h (xi) = 0), we decompose the learned features by spectral decomposition, which is similar to PCA:
1
n n∑ i=1 h (xi)h (xi) T = d∑ j=1 ujλju T j (2)
where λj means the eigenvalues with index j and uj ∈ Rd represents its eigenvector. Specifically, we train ResNet-18 (He et al., 2016) using both standard training and adversarial training on CIFAR-10. The parameters for AT are the same as (Rice et al., 2020). We calculate the eigenvalues by applying Eq. (2), and the eigenvalues are plotted in Figure 2. The features come from the penultimate layer (512 dimensions). All the features are extracted from the test set in CIFAR-10. A part of the eigenvalues is shown for better visualization.
advori
Difference of models in spectral analysis. As shown in Figure 2(a)(b), the eigenvalues of a standard trained model drop rapidly at some point, while the sharp decrease of eigenvalues is much alleviated by AT. These sharp spectral signature in standard training makes just a few eigenvalues informative from the opinion of PCA. The model fails to explore the influence of eigenvectors with smaller eigenvalues on classification, so the trained model could not recognize the change of features along eigenvectors with smaller eigenvalues. The eigenvectors which may endow useful features are overly penalized. Consequently, we propose a hypothesis that the severe dominance of the top eigenvectors is a cause of vulnerability in DNN, and the adversary adds more components in eigenvectors with smaller eigenvalues. We will verify the proposed hypothesis in the next section.
Connection with intrinsic dimension. We introduce intrinsic dimension (ID) to quantitatively describe the decreasing tendency in eigenvalues. ID (i.e., the minimal number of parameters needed to describe a representation) has a close connection with natural accuracy (Ansuini et al., 2019). It has been found that reduction of ID contributes to an improvement on natural accuracy. We adopt PC-ID proposed by (Ansuini et al., 2019) to estimate ID, which is determined by the number of principal components included to describe 90% of the variance. We mark ID in Figure 2 with solid circles. The results reveal the ID of a standard trained model is very small, while models obtained by AT achieve a higher ID, which is contrary to the influence of ID on natural accuracy. This also verifies that there exists a trade-off between generalization and robustness (Tsipras et al., 2019; Zhang et al., 2019) from the perspective of ID. FSR could further increase ID, based on AT.

heading: 3.2 VARIATION ALONG EIGENVECTORS UNDER ATTACKS
text: In this section, we aim to verify the hypothesis that adversary adds more components along eigenvectors with smaller eigenvalues in attacking stage. We define a metric to quantitatively observe the change of features along different eigenvectors under attack, called variation in this paper.
Definition 1 (Alignment) Given a dataset Ds = {xs,i, ys,i}ni=1 which may be perturbed. The alignment of Ds to the pre-given direction uj is calculated by the expectation over cosine similarity between features extracted by DNN and the direction vector uj:
align (Ds, uj) = E (xs,i,ys,i)∈Ds |〈h (xs,i) , uj〉| ‖h (xs,i)‖ · ‖uj‖
(3)
where the norm ‖·‖ used is Euclidian norm, and uj is calculated by Eq. (2). The calculation of uj is based on features covariance of natural samples, and uj is fixed.
Definition 2 (Variation) Given a dataset D consist of natural examples and its perturbed dataset Dadv . The variation on direction uj is defined as the ratio between alignment on Dadv and D:
r (Dadv, D, uj) = align (Dadv, uj)
align (D,uj) (4)
The alignment is correlated with the distance between subspace spanned by uj and the actual feature space, so the change of alignment under attack is suitable to describe the influence of attacks on direction uj , which is called as variation. Our metric is similar to (Tran et al., 2018; Hayase et al., 2021) used for analyzing backdoors, but we define the alignment by cosine similarity while the latter uses the inner product. Compared with inner product, cosine similarity could eliminate the influence of scale. In Appendix B.1, we provide a complete procedure to calculate the metrics.
We give an intuitive explanation on why the defined variation is suitable to capture the change of features along different eigenvectors under attacks. An toy model for illustration is shown in Figure 3. Suppose the distribution transfers from Figure 3(a) to (b) under attack, we draw an original data x and its shifted data xadv . Take the fixed direction u2 as an illustration. The cosine similarity between x and u2 is equal to cos (θ), i.e., cos (θ) = 〈x, u2〉 / ‖x‖ · ‖u2‖, which is called as alignment above. If the distribution moves from Figure 3(a) to (b), then the cosine similarity cos (θ) increases. Consequently, the change of the cosine similarity can describe how the features change along a direction under attacks. For the vari-
ation defined in Eq. (4), r(Dadv, D, uj) > 1 means the features add more components along direction uj , and vice versa. We could compare r(D,D, uj) with 1 to observe whether the adversary adds or reduces the components along various eigenvectors.
The adversary adds more components along the eigenvectors with smaller eigenvalues. We visualize the variation of features to different eigenvectors {u1, · · · , u512}. The results of CIFAR10 and SVHN are shown in Figure 4. The attacks used include FGSM (Goodfellow et al., 2015),
PGD (step size /10 for 10 steps) (Madry et al., 2018) and C&W attack (Carlini & Wagner, 2017). We set attack budget = 4/255 constrained by `∞ norm. As observed in Figure 4(a), the variation keeps close or smaller than 1 for the several largest eigenvalues in standard trained model. However, the variation of smaller eigenvalues is much larger than 1. This means that FGSM tends to add more components along the eigenvectors uj with smaller eigenvalues in standard training. A similar phenomenon also exists in PGD and C&W, which verifies the hypothesis in Section 3.1. We also provide the curve while changing budget and the variation on CIAFR-100 in Appendix B.2.
For models trained by AT, variation of all eigenvectors keeps close to 1, which is distinctly different from standard training. The high variation of directions with smaller eigenvalues visibly decreases. It is worth noticing that we apply cosine similarity for definition, so the influence from the scale of features is eliminated. Therefore, the components corresponding to the larger eigenvalues are more robust while the smaller ones are more fragile. Adopting the opinion of robust features (Ilyas et al., 2019), the features along the eigenvectors with larger eigenvalues are regarded as robust features, and these along the direction with smaller eigenvalues are non-robust features. The analysis above motivates us to regularize the spectrum signatures to alleviate the rapid decreasing tendency. If we alleviate the dominance of the top eigenvalues, the adversarial robustness is improved.

heading: 4 FEATURE SPECTRAL REGULARIZATION
text: In this section, we propose a method to regularize the eigenvalues, aiming to alleviate the dominance of the top eigenvalues. We first propose our method FSR, and then provide a theoretical analysis.

heading: 4.1 REALIZATION OF FSR
text: Through the detailed analysis above, the sharp distribution of eigenvalues weakens the information contained by the smaller eigenvalues, and causes fragility to adversarial attacks. A promising approach to alleviate such phenomenon is to suppressing the largest eigenvalues, which could mitigate the dominance of the largest eigenvalues relatively. Another straightforward idea is to increase the small eigenvalues during training. However, small eigenvalues are usually numerically unstable, which may be easily affected by noise or round-off error in optimization. In this paper, we propose a method called Feature Spectral Regularization (FSR) by penalizing the largest eigenvalues of feature covariance in a batch of data:
LFSR (F ) = w (τ) · λmax (( F − 1
m 1F
)T ( F − 1
m 1F
)) (5)
where λmax (·) means the largest eigenvalue of a matrix, w (τ) = min {τ/T0, 1} is the weighting function of current epoch τ with a hyper-parameter T0, and 1 ∈ Rm×m is a constant matrix with each element equal to 1 for calculating the mean. Moreover, F ∈ Rm×d is the feature matrix of a batch composed of row vectors h (xi)
T , i.e., F = [h (x1) |h (x2) | · · · |h (xm)]T , and m is the number of samples in a batch of data.
In practice, we can just access the batch of data to approximate the statistics of feature space. However, the eigenvalues change drastically in the early stages of training, which may cause instability on optimization, so we apply a piecewise linear function w(τ) to smooth the training stage. The λmax (·) in Eq. (5) is equal to the square of the largest singular value in feature matrix (F − 1m1F ) by Singular Value Decomposition (SVD). So the realization of FSR is very simple with the help of Pytorch (Paszke et al., 2019) in a few lines of code.
Summarized objective loss. Adversarial training (Madry et al., 2018) has been widely proven to be a strong baseline in adversarial defense. We build and incorporate the proposed FSR into AT training framework for further improving model robustness significantly. The final objective is:
Ladv (x, y; θ) = LCE (xadv, y; θ) + βFSR · LFSR (Fadv) (6) where βFSR is a hyper-parameter that controls the trade-off between two items, xadv represents the adversarial example generated from x by PGD (Madry et al., 2018) using cross-entropy loss, Fadv is the feature matrix of adversarial examples. Besides AT, we also apply FSR to defense methods such as TRADES (Zhang et al., 2019) and AT-AWP (Wu et al., 2020). We provide a pseudo-code of our method in Algorithm 1 in Appendix A.
Computational complexity. SVD is an extra computational cost induced by FSR. For a matrix F ∈ Rm×d, the time complexity of SVD is O ( min { m2 · d,m · d2 }) . Since the batch size is often small, the excess cost in computation is negligible compared to adversarial training.

heading: 4.2 THEORETICAL ANALYSIS
text: Consider a linear regression model ŷ = 〈z, θ〉 with `2 perturbation δ based on the feature z ∈ Rd, the underlying parameter obtained by minimizing mean square error is θ0 ∈ Rd without perturbation. We assume the mean of features E(z) = 0 and covariance matrix Var(z) = Σ. Following the adversarial risk define by (Xing et al., 2021),Radv is expressed in Eq. (7):
Radv (θ, δ) = Ez max ‖zadv−z‖≤δ (〈zadv, θ〉 − 〈z, θ0〉)2 . (7)
The optimal solution of Eq.(7) denoted as θadv has an similar formulation with the ridge regression:
θadv = (Σ + λI) −1 Σ θ0, (8)
where λ can be regarded as a constant (El Ghaoui & Lebret, 1997; Xing et al., 2021).
Given samples {zi, yi}ni=1, the feature matrix composed of row vector zTi is denoted as Z ∈ Rn×d, then Σ = 1n Z
TZ. y = [y1, y2, · · · , yn]T ∈ Rn is a column vector composed of real-valued output yi (i = 1, · · · , n). Previous methods tend to regularize the classifier θ (El Ghaoui & Lebret, 1997; Xu et al., 2008). However, we could directly analyse how the feature Z influences the robustness of model. If the feature is more robust for classifier, then whether using the classifier θ0 or θadv should have the same prediction. Thus, we define the residual risk induced by features Z:
min Z Rres (Z) = ‖Zθadv − Zθ0‖2 , s.t. ‖Z‖ 2 F = s0. (9)
It is essential to normalize the scale of features for comparable representation, so we restrict the norm of Z as used in (Yu et al., 2020). The SVD of matrix Z ∈ Rn×d has the form: Z = UDV T . HereU = [u1, · · · , un] ∈ Rn×n and V ∈ Rd×d are orthogonal matrices. Observe that the constraint for Z only depends on its singular values, i.e., ‖Z‖2F = ∑min{n,d} i=1 σ 2 i = s0. Suppose d < n and σi > 0, thenRres is simplified to the expression by combining Eq. (9) and Eq. (8):
min (σ1,··· ,σd) Rres (σ1, · · · , σd) = min (σ1,··· ,σd) ∥∥∥∥∥∥ d∑ j=1 uj λn σ2j + λn uTj y ∥∥∥∥∥∥ 2 , s.t. d∑ j=1 σ2j = s0. (10)
Theorem 1 Rres (σ1, · · · , σd) is minimum when all the singular values of Z are equal.
As FSR penalizes the largest singular value in feature matrix, it alleviates the dominance of large singular values and helps contribute to equal singular values under the normalized condition, thus helping reduce the residual risk induced by features. Proof of Theorem 1 is provided in Appendix C.

heading: 5 EXPERIMENTS
text: In this section, we evaluate the effectiveness of the proposed FSR on CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011). FSR is applied to two baselines: 1) AT (Madry et al., 2018; Rice et al., 2020); 2) TRADES (Zhang et al., 2019). It is worth noticing that AT is the most effective method to improve adversarial robustness (Rice et al., 2020; Pang et al., 2021; Gowal et al., 2021) in RobustBench (Croce et al., 2020).
Experimental Settings. For CIFAR-10 and CIFAR-100, we set the `∞ perturbation with = 8/255, the step size of attack 2/255, and the inner iteration steps 10. The step size is 1/255 for SVHN. We train ResNet-18 (He et al., 2016) using momentum optimizer with the initial learning rate of 0.1. For AT, we train 200 epochs and the learning rate decays with a factor of 0.1 at 100 and 150 epochs (Rice et al., 2020). For TRADES, we train 120 epochs with the learning rate divided by 0.1 at epochs 75, 90, and 100 (Zhang et al., 2019). The parameter for regularization (1/λ) is set as 4 for TRADES. For FSR, we set βFSR = 0.01. Other hyper-parameters keep the same as their original paper. Considering that the settings have a distinct influence on robustness (Pang et al., 2021; Gowal et al., 2021), the hyper-parameters remain unchanged while adding our FSR.

heading: 5.1 PERFORMANCE UNDER WHITE-BOX ATTACKS
text: To test the adversarial robustness of model in detail, we adopt various white-box adversarial attacks: FGSM (Goodfellow et al., 2015), PGD-20 (step size /4) (Madry et al., 2018), CW (`∞ version optimized by PGD) (Carlini & Wagner, 2017). Following the instruction proposed by (Carlini et al.,
2019), we use 5 random starts at random offsets away from the initial when applying iterative attacks. The settings for evaluation promote a quite strong attack. We report the best checkpoint (the highest robustness under PGD from different checkpoints) as used in (Rice et al., 2020). The test accuracy is reported in Table 1∼3. The results show that FSR can further improve adversarial robustness, compared with the baseline models. We also test the robustness under AutoAttack (AA) (Croce & Hein, 2020), which is now regarded as the strongest attack to evaluate adversarial robustness. As shown in the table, FSR is effective to improve robustness under AA. The detailed results show that our FSR can indeed boost robustness, rather than depending on obfuscated gradients (Athalye et al., 2018) or gradient masking (Papernot et al., 2017). The improvement on various datasets reveals that our method has a consistent promotion, rather than relying on a special dataset.
The performance of final checkpoint is another essential probe to analyse the adversarial robustness of models. We also report the performance of final checkpoint, as shown in Table 4. The results show that FSR could also improve the performance in final checkpoint on various datasets.
Combination with other methods. To validate that FSR is an effective module, we try to combine FSR with Adversarial Weight Perturbation (AWP) (Wu et al., 2020). AWP helps improve adversarial robustness and effectively alleviates the overfitting (Rice et al., 2020) in PGD-based AT. It is one of the strongest defense methods in RobustBench (Croce et al., 2020), which means it is a high baseline. The results are listed in Table 5, and the results show that we could even attain improvement on adversarial robustness based on the high baseline AWP, especially under CW attack and AA.
Performance under WideResNet. To explore the performance of our model in large networks, we use WideResNet-34-10 (depth 34 and width 10) (Zagoruyko & Komodakis, 2017) as another backbone network. In this part, we realize our method based on the well-known effective methods: AT (Madry et al., 2018; Rice et al., 2020) and AT-AWP (Wu et al., 2020) on CIFAR-10. All the parameters of baselines are the same as used in their original paper. The settings of threaten model are the same as ResNet-18 in the previous section. We show the results of both best checkpoint and final checkpoint in Table 6. The “Best” means the checkpoint of best performance under PGD, and the “Last” is the performance of final checkpoint. The results verify that our method is also effective in WideResNet, and we especially attain great promotion in final checkpoint based on AWP.

heading: 5.2 PERFORMANCE UNDER BLACK-BOX ATTACKS
text: Transferability is an intriguing property about adversarial examples, which can be used to implement black-box attack (Papernot et al., 2016a; Liu et al., 2017). Adversarial examples generated by the
substitute model can be used to attack the target model with the weights and architecture unknown to the adversary. Transfer attack is beneficial to verify that our method does not rely on gradient masking (Papernot et al., 2017). As suggested by (Carlini et al., 2019), we adopt another adversarially trained ResNet-18 as the substitute model. The results are shown in Table 1∼ 3, revealing that our methods also improve robustness under black-box attacks.

heading: 5.3 FURTHER UNDERSTANDING OF FSR
text: In this part, we analyse the influence of FSR on the eigenvalue spectrum. The results are shown in Figure 5. Our motivation is that the top eigenvalues dominate the distribution of eigenvalues, so we use FSR to increase the overall eigenvalues relatively. To eliminate the influence from the scale of features, the eigenvalues are divided by the maximum eigenvalue. As shown in the figure, FSR increases the eigenvalues relatively, which is consistent with our intention for FSR. In Appendix D, we show that simply using FSR without AT also promotes a mild improvement on robustness.

heading: 5.4 ABLATION STUDIES
text: Sensitivity analysis of βFSR. We explore how the weight of FSR βFSR influences the performance of models. Following the same settings as the previous section, we train our models on CIFAR-10 using ResNet-18 with the change of βFSR. The results are reported Table 7. As observed in Table 7, FSR can significantly improve robustness with a wide value range, which verifies the stability and effectiveness of the proposed method. In this paper, we choose the parameter by considering both generalization and robustness, so we choose βFSR = 0.01. If we focus merely on adversarial robustness, tuning βFSR could still promote a further improvement on robustness.

heading: 6 CONCLUSION
text: In this paper, we delve into the discrepancy between natural examples and adversarial examples from the perspective of spectral analysis. The variation of different eigenvectors under adversarial attacks is analysed. It is shown that the spectral directions with smaller eigenvalues are more fragile under attack, which is induced by the dominance of the top eigenvectors. Based on the analysis, a method called Feature Spectral Regularization (FSR) is proposed to penalize the largest eigenvalues of the batch covariance matrix, which is numerically stable and could enlarge the overall eigenvalues relatively. We also provide a theoretical explanation in robust linear regression. FSR is a simple module readily pluggable into any defense method. Through comprehensive experiments, we show that FSR can effectively improve adversarial robustness when combined with many defense methods.

heading: A ALGORITHM OF FSR
text: Algorithm 1 Robust Training with FSR Input: Training data D = {(xi, yi)}i=1,··· ,N , DNN composed of feature extractor h (·) and linear
classifier g (·), maximum training steps T Output: Robust model {h (·) , g (·)}
1: for t in [1, 2, · · · , T ] do 2: Read mini-batch {x1, x2, · · · , xb} 3: Generate adversarial robustness using PGD 4: Compute the FSR loss in Eq. (5) using adversarial examples 5: Compute the overall loss in Eq. (6) 6: Optimize all parameters of model using gradient descent 7: end for

heading: B THE DEFINED VARIATION ON MORE DATASETS
text: In this part, we provide more experiments about the variation defined by us. We first provide a detailed algorithm to calculate the variation. More results about the variation are shown following.
B.1 ALGORITHM FOR CALCULATING THE VARIATION
Algorithm 2 Calculating the Alignment and Variation Require: Dataset D = {(xi, yi)}i=1,··· ,N (natural examples), the feature extractor h (·). Ensure: Variation to different eigenvectors r (Dadv, D, uj) (j = 1, · · · , d).
1: Centralize the features h (xi)← h (xi)− 1N ∑N i=1 h (xi)
2: Calculate the covariance matrix 1N ∑N i=1 h (xi)h (xi) T 3: Apply eigendecomposition to the covariance matrix to get the eigenvalues and eigenvectors {λ1, · · · , λd} , {u1, · · · , ud} = eig ( 1 N ∑N i=1 h (xi)h (xi) T ) 4: Generate the adversarial example xadv,i from xi by attack (i = 1, · · · , N ) 5: Centralize the features of adversarial examples h (xadv,i)← h (xadv,i)− 1N ∑N i=1 h (xadv,i) 6: for j = 1, · · · , d do 7: Calculate the cosine similarity between features and eigenvectors
align (D,uj) = 1 N ∑N i=1 |〈h (xi) , uj〉|/ ‖h (xi)‖ · ‖uj‖ align (Dadv, uj) = 1 N ∑N i=1 |〈h (xadv,i) , uj〉|/ ‖h (xadv,i)‖ · ‖uj‖
8: Calculate the variation to different eigenvectors r (Dadv, D, uj) = align (Dadv, uj) /align (D,uj) 9: end for
B.2 MORE EXPERIMENTS ABOUT THE DEFINED VARIATION
In this part, we show more results about variation to eigenvectors on CIFAR-10, CIFAR-100 and SVHN. We compare the models trained on natural examples and adversarial examples. The adversarially trained models adopt the PGD-AT Madry et al. (2018) with `∞ perturbation 8/255, and the parameters are the same as Rice et al. (2020). From the curve shown in Figure 6∼8, we can see that the variation on CIFAR-100 and SVHN reveals the same tendency as CIFAR-10, i.e., adversarial attacks tend to increase the components along the eigenvectors with smaller eigenvalues.
0.8
0.9
1
1.1
1.2
1.3
1 51 101 151 201 251 301 351 401 451 501
v a
ri a
ti o
n r
(j )
index j
FGSM (ε=1/255) FGSM (ε=2/255) FGSM (ε=4/255)
(a) Variation to FGSM (STD)
0.98
0.985
0.99
0.995
1
1.005
1.01
1.015
1.02
1 51 101 151 201 251 301 351 401 451 501
v a ri
a ti o n r
(j )
index j
FGSM (ε=1/255) FGSM (ε=2/255) FGSM (ε=4/255)
(b) Variation to FGSM (AT)
0.8
0.9
1
1.1
1.2
1.3
1 51 101 151 201 251 301 351 401 451 501
v a ri
a ti o n r
(j )
index j
PGD (ε=1/255) PGD (ε=2/255) PGD (ε=4/255)
(c) Variation to PGD (STD)
0.98
0.985
0.99
0.995
1
1.005
1.01
1.015
1.02
1 51 101 151 201 251 301 351 401 451 501
v a ri
a ti o n r
(j )
index j
PGD (ε=1/255) PGD (ε=2/255) PGD (ε=4/255)
(d) Variation to PGD (AT)

heading: C PROOF
text: In this part, we provide the proof of Theorem 1 in Section 4.2.
Consider the linear regression model, ŷ = 〈z, θ〉 , (11)
where z ∈ Rd means the feature, and θ ∈ Rd represents the parameter of model. We assume that the features have zero mean, and the covariance is denoted as Σ, i.e., E(z) = 0,Var(z) = Σ.
Given samples {zi, yi}ni=1, the feature matrix composed of row vectors zTi is represented as feature matrix Z ∈ Rn×d, then Σ = 1n Z
TZ. Assuming that Z has full column rank, if we minimize the residual sum of squares in Eq. (11), then the solution is:
θ0 = ( ZTZ )−1 ZT y, (12)
where y ∈ Rn means the real-valued output, i.e., yT = (y1, y2, · · · , yn). When the feature z is noised by `2 perturbation δ, we can define the adversarial risk induced by the perturbation as (Xing et al., 2021):
Radv (θ, δ) = Ez max ‖zadv−z‖≤δ (〈zadv, θ〉 − 〈z, θ0〉)2 . (13)
The optimal solution to Eq. (13) denoted as θadv is
θadv = (Σ + λI) −1 Σθ0, (14)
where λ can be regarded as a constant. The work of (El Ghaoui & Lebret, 1997; Xing et al., 2021) provides more details about how to get λ.
A better feature should lead to the same prediction whether using classifier θ0 or θadv , so we define the residual risk induced by features
min Z Rres (Z) = min Z ‖Zθadv − Zθ0‖2 , s.t. ‖Z‖ 2 F = s0. (15)
We normalize the scale of features by Frobenius norm to avoid the influence of scale (Yu et al., 2020), which is just correlated to its singular values {σ1, · · · , σd}:
‖Z‖2F = trace ( ZTZ ) = d∑ j=1 σ2j = s0. (16)
The SVD of the n× d matrix Z has the form:
Z = UDV T , (17) where U = [u1, · · · , un] ∈ Rn×n and V ∈ Rd×d are orthogonal matrices. D = [ D0 0 ] ∈ Rn×d
and D0 = diag (σ1, · · · , σd), σ1 ≥ σ2 ≥ · · · ≥ σd. After applying SVD, we can simplify the Eq. (15) to following expression:
min (σ1,··· ,σd) Rres (σ1, · · · , σd) = min (σ1,··· ,σd) ∥∥∥∥∥∥ d∑ j=1 uj λn σ2j + λn uTj y ∥∥∥∥∥∥ 2 , s.t. d∑ j=1 σ2j = s0. (18) According to the property of the subordinate matrix norm, we could derive an upper bound ofRres:∥∥∥∥∥∥ d∑ j=1 uj λn σ2j + λn uTj y ∥∥∥∥∥∥ 2 ≤ ∥∥∥∥∥∥ d∑ j=1 uj λn σ2j + λn uTj ∥∥∥∥∥∥ 2 · ‖y‖2 . (19)
Consider that y can be arbitrary value, so the minimization ofRres requires that the first item to the right of the inequality sign in Eq. (19) is as small as possible, i,e,
min (σ1,··· ,σd) ∥∥∥∥∥∥ d∑ j=1 uj λn σ2j + λn uTj ∥∥∥∥∥∥ 2 = min (σ1,··· ,σd)
λn
σ2d + λn . (20)
The minimization ofRres is simplified to the following expression:
arg min (σ1,··· ,σd) Rres (σ1, · · · , σd) = arg min (σ1,··· ,σd)
λn
σ2d + λn , s.t. d∑ j=1 σ2j = s0. (21)
Since σ1 ≥ σ2 ≥ · · · ≥ σd, the optimal solution of Eq. (21) is that all singular values are equal.

heading: D FSR IMPROVES CONFIDENCE CALIBRATION AND ROBUSTNESS
text: We verify that network trained using only FSR without AT attains a mild improvement on adversarial robustness. We test the performance of model under FGSM, and the results are listed in Table 8.
In addition to robustness, another important but undesirable aspect of DNN is reliability. When using classifiers for prediction, the predictive confidence should be indicative of the actual likelihood of correctness (Guo et al., 2017). Confidence calibration focuses on the mismatch problem between a model’s confidence and its correctness. A popular metric used to measure calibration is the Expected Calibration Error (ECE) (Naeini et al., 2015; Guo et al., 2017), defined as the expected absolute difference between the models’ confidence and its accuracy. We explore the efficacy of FSR on confidence calibration.
As shown in Figure 9, for the baseline, there is a marked rise in the test ECE at the later period of the training process, which indicates the problem of overconfidence. Remarkably, FSR has an obvious improvement on confidence calibration compared with the baseline, e.g., reduces the ECE value from 2.97% to 1.18% for CIFAR-10.

heading: E FURTHER EXPERIMENTS ABOUT ADVERSARIAL ROBUSTNESS
text: In this part, we add two experiments about the proposed FSR on adversarial robustness. We first report the results (i.e., mean and standard deviation) when all experiments in Table 1∼3 are repeated five times. Then, the test accuracy of suppressing more eigenvalues is also provided.
E.1 EXPERIMENTS ABOUT STANDARD DEVIATION
We repeat all the experiments in Table 1∼3 five times, and report the results under white-box attack in following Table. The performance in Table 9∼11 verifies that our proposed method achieves a steady improvement.
E.2 SUPPRESSING MORE EIGENVALUES
In the main body of our paper, the proposed FSR only penalizes the largest eigenvalue. In this part, we explore whether penalizing more eigenvalues could achieve a better performance under attack. The results are listed in Table 12. As we properly increase the number of suppressed eigenvalues k, the adversarial robustness of models could be further improved. However, if we keep increasing k, the robustness of the model declines. We think it is due to that k has some trade-off with the weight of FSR βFSR, i.e., a larger k may require a smaller βFSR.

heading: F MORE DISCUSSIONS BETWEEN SPECTRAL SIGNATURES AND ROBUSTNESS
text: In this section, we discuss the connection between our explanation of adversarial examples in the paper and previous studies. We mainly talk about the connection with interpretations form the perspective of Lipschitz constant and manifold.
F.1 CONNECTION BETWEEN FSR AND LIPSCHITZ CONSTANT
In this part, we first rederive some basic conclusions about Lipschitz constant in DNN as illustrated in (Yoshida & Miyato, 2017). For a feature extractor g(x) : RD → Rd composed of L layers, the weights of each layer in the network are denoted as { W l, bl }L l−1. Then the feature extractor
g (x) = WLφL−1 ( WL−1φL−2 (· · · ) + bL−1 ) + bL, where φl represents the activation function.
We further assume that each activation function is a ReLU. In this case, φl act as input-dependent diagonal matrices ΦlX := diag ( φlX ) , where an element in the diagonal φlX := 1 (x̃ ≥ 0) is one
if its pre-activation x̃l := W lφl−1 + bl is positive (Roth et al., 2020). Then, the global Lipschitz constant of g(x) is bounded by the spectral norm of following expression (Yoshida & Miyato, 2017; Roth et al., 2020; Huang et al., 2021):
Lglob ≤ ∥∥WL · ΦL−1X ·WL−1 · ΦL−2X · · ·Φ1X ·W 1∥∥2 (22)
For better comparison with our proposed method, we do not further simplify Eq. (22) like (Yoshida & Miyato, 2017). With the normalization (Ioffe & Szegedy, 2015) in DNN, the mean of final features are always close to zero. Suppose that the mean of deep features is equal to zero, then the proposed FSR denoted as `FSR could be simplified to
`FSR = λmax ( FTF ) = σmax ( WLφL−1 ( WL−1φL−2 ( · · · · · ·φ1 ( W 1x+ b1 ) · · · · · · ) + bL−1 ) + bL
)2 = ∥∥WLφL−1 (WL−1φL−2 (· · · · · ·φ1 (W 1x+ b1) · · · · · · )+ bL−1)+ bL∥∥22
= ∥∥WL · ΦL−1X ·WL−1 · ΦL−2X · · ·Φ1X ·W 1 · x∥∥22
(23)
where the deep featuresF = WLφL−1 ( WL−1φL−2 ( · · · · · ·φ1 ( W 1x+ b1 ) · · · · · · ) + bL−1 ) +bL, λmax (·) is the largest eigenvalue of a matrix, and σmax (·) is the spectral norm (largest singular value) of a matrix. If we compare Eq. (22) and Eq. (23), FSR has a very similar form with the upper bound of global Lipschitz constant. Since the input data x is a constant, it could be expected that FSR has an effect to decrease the upper bound of the Lipschitz constant. FSR decreases the Lipschitz constant by constraining features rather than directly regularizing weights.
F.2 CONNECTION WITH ADVERSARIAL EXAMPLES ON MANIFOLD
Our observation is consistent with the hypothesis in (Szegedy et al., 2013; Song et al., 2018) that adversarial examples are low-probability (high-dimensional) “pockets” on the manifold from the perspective of spectral signatures. Our paper reveals that the components along eigenvectors with severely smaller eigenvalues are much rare in standard training (seeing Figure 2). When confronted with attacks, the adversary tries to add more components along with such directions (seeing Figure 4). The eigenvectors with smaller eigenvalues could be regarded as “pockets” on the manifold.

Review summaries:
By analyzing the spectral difference between the natural and adversarial examples, this paper finds that eigenvectors with smaller eigenvalues are more non-robust and adversary trends to add more components into these directions. To eliminate the dominance of the top eigenvalues, the paper proposes Feature Spectral Regularization (FSR), which adds more penalties to the largest eigenvalues while relatively increasing the smaller ones. Several experimental results demonstrate that FSR can further improve the robustness when combined with other adversarial defenses.  

The paper has presented a new method to improve the robustness of features under adversarial attacks. Authors developed a new metric for the change of features subject to attacks and key findings are eigenvectors of small eigenvalues are more inclined to change under adversarial attacks, i.e., non-robust. Authors believe the dominance of large eigenvalues and eigenvectors are the primary reasons and a way of flattening the spectrum of features would help mitigate this issue. Authors propose to suppress the largest eigenvalues during training, i.e., spectral regularization, which show positive results in adversarial defense when working together with SOTA defense models.

The authors propose a new defense against adversarial attacks by means of a spectral regularization. The defense is based on an inspection of the embedding of training/test data as returned by the penultimate layer of a neural network. The observation is that the relevant subspace of embedded natural data points (covering e.g. 90% of the variance) is quite low-dimensional, where the use of adversarial training increases the dimensionality of this subspace. In addition, the relevant subspace of embedded adversarial examples is in the provided experiments typically a bit higher dimensional than the subspace of natural examples. Based on these observations, the authors propose a regularizing term, which penalizes the variance in the direction of the first principal component (the largest eigenvalue of the feature covariance matrix).  Experiments are conducted on Cifar-10, -100 and SVHN. The proposed regularizer is applied on top of other defense methods, which typically increases the accuracy under attacks by up to 2%.