On the Relationship Between Adversarial Robustness and Decision Region in Deep Neural Networks
Authors: Seongjin Park
Haedong Jeong
Giyoung Jeon
Jaesik Choi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Decision Region
Adversarial Robustness
Robust Training

Abstract: In general, Deep Neural Networks (DNNs) are evaluated by the generalization performance measured on unseen data excluded from the training phase. Along with the development of DNNs, the generalization performance converges to the state-of-the-art and it becomes difficult to evaluate DNNs solely based on this metric. The robustness against adversarial attack has been used as an additional metric to evaluate DNNs by measuring their vulnerability. However, few studies have been performed to analyze the adversarial robustness in terms of the geometry in DNNs. In this work, we perform an empirical study to analyze the internal properties of DNNs that affect model robustness under adversarial attacks. In particular, we propose the novel concept of the Populated Region Set (PRS), where training samples are populated more frequently, to represent the internal properties of DNNs in a practical setting. From systematic experiments with the proposed concept, we provide empirical evidence to validate that a low PRS ratio has a strong relationship with the adversarial robustness of DNNs. We also devise PRS regularizer leveraging the characteristics of PRS to improve the adversarial robustness without adversarial training.

Under review as a conference paper at ICLR 2023
ON THE RELATIONSHIP BETWEEN ADVERSARIAL ROBUSTNESS AND DECISION REGION IN DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In general, Deep Neural Networks (DNNs) are evaluated by the generalization performance measured on unseen data excluded from the training phase. Along with the development of DNNs, the generalization performance converges to the state-of-the- -ec-o-f-the-rttt and it becomes difficult to evaluate DNNs solely based on this metric. The robustness against adversarial attack has been used metric to evaluate DNNs by measuring their vulnerability. However, few studies have been performed to analyze the adversarial robustness in terms of the geome- try in DNNs. In this work, we perform an empirical study to analyze the internal properties of DNNs that affect model robustness under adversarial attacks. par- ticular, we propose the novel concept Populated Region Ser (PRS), where train- ing samples are populated more frequently, to represent the internal properties of DNNs in a practical setting. From systematic experiments with the proposed con- cept, we provide empirical evidence to validate that low PRS ratio has a strong relationship with the adversarial robustness of DNNs. We also devise PRS regu- larizer leveraging the characteristics PRS to improve the adversarial robustness without adversarial training.
additional
INTRODUCTION
With the steep improvement of the performance of Deep Neural Networks (DNNs), their applica- tions are expanding to the real world, such as autonomous driving and healthcare (Huang & Chen 2020 LeCun al.. 2015, Miotto et al. 2018). For real world application, it may be necessary to choose the best model among the candidates. Traditionally. the generalization performance which measures the objective score on the test dataset excluded in the training phase, used to evaluate the models (Bishop 2006). However, it is non-trivial to evaluate DNNs based on this single metric. For example, if two networks with the same structure have the similar test accuracy,
ambiguous which is better. Robustness against adversarial attacks, measure of the vulnerability. can be alter- native to evaluate DNNs (Szegedy et al.) 2013 Huang et al.) 2015 Jakubovitz & Giryes] 2018: Yuan et al. 2019 Zhong et al. (021). Most previous works were focused
the way to find adversarial samples by utilizing the model properties such as gradients with respect to the loss function. Given that the adversarial attack seeks to find the perturbation path on the model prediction surface over the input space, robustness can be expressed in terms of the geometry of the model. However, few studies have been performed to interpret the robustness with the concept of the geometric proper- ties of DNNs. From a geometric viewpoint, the internal properties of DNNs are represented by the boundaries and the regions (Baughman & Liu 2014). is shown that the DNNs with piece-wise lin- ear activation layers are composed of many linear regions, and the maximal number of these regions mathematically related the expressivity of DNNS (Montúfar et al. 2014 Xiong et al., 2020)). As these approaches only provide the upper bound for the expressivity with the same structured model, it does not explain how much information the model actually expresses.
In this work, we investigate the relationship between the internal properties of DNNs and the ro- bustness. In particular, our approach analyzes the internal characteristics from the perspective of the decision boundary (DB) and the decision region (DR), which are basic components of DNNs (Fawzi et al. (017). To avoid insensitivity of the maximal number of linear regions in the same structure assumption, we propose the novel concept Populated Region Set (PRS), which is set of

---

Under review as a conference paper at ICLR 2023
(A) Warm-up
(B) Standard Training
DEL FLAB
LOSS
fu-1a
ft
fi-ta
loput Spare
(C) Robust Learning with PRS Regularizer
PRS Reg
Ingant
CE Loss
Imput Space
DBs and DO f1
DBs and DR
Figure 1: An illustrative comparison of each training method with CIFAR-10. and visualization for decision boundaries/regions (DBs/DRs) of penultimate layer in the imput space (f)) For visualization, We randomly select three training images in dog class and make hyperplane with these images. The green area indicates DR which the blue boxed image populates. (A) Warm- up stage for VGG-16 with standard training. (B) Standard training after warm up stage. (C) The robust learning with devised PRS regularizer after warm up stage. We identify each training method induces different configuration of DBs/DRs, which represents different internal properties of DNNs.
DRs containing at least one training sample included the training dataset. Since the PRS can be considered the feasible complexity of the model, we hypothesize the size of PRS is related to the robustness of network. To validate our hypothesis, we perform systematic experiments with various structures of DNNs and datasets. Our observations summarized as follows:
The models with same structure can have different size of PRS, although they have similar generalization performance. We empirically show that the model with small size PRS tends to show higher robustness compared to that with a large size. (in Section (.2)
We observe that when the model achieves small size of PRS, the linear classifier which maps the penultimate features to the logits high cosine similarity between parameters corresponding to each class (in Section 3.2)
We verify that the size of intersection of the PRS from the training/test dataset is related to the robustness of model. The model with a large intersection of training/test dataset has higher robustness than the model with small intersection (in Section 3.3).
We devise a novel regularizer leveraging the characteristics PRS to improve the robust
accuracy without adversarial training (in Section
INTERNAL PROPERTIES OF DNNS
This section describes the internal properties of DNNs in the perspective of DBs and DRs. To expand the notion of DBs and DRs the internal feature-level, we re-define the DBs in the classifier that generalizes the existing definition of DBs. Finally, we propose Populated Region Set (PRS) which describes the specific DRs related to the training samples.
2.1 DECISION BOUNDARY AND REGION
Let the classifier with L number of layers be -(- is the sample in the input space X' RD= and (-) denotes the non-linear activation functior For the 1-th layer, f((-) denotes the linear operation and ff (-) denotes the value of the i-th element of the feature vector (x) E Di We define the DB for the i-th neuron of the (-th layer.
(z). where
Definition (Decision Boundary (DB)) The i-th decision boundary at the 1-th layer is defined as.
B = (a (x) 0, Yx E X').
AAthhough there various activation functions, we only consider ReL. activation for this paper.
2

---

Under review as a conference paper at ICLR 2023
1.0
1.0
3.0
0.0
0.2
08
2.0
Layer
0
300
Epochs
133
202
250
503
30
100
Epochs
150
200
200
20
NetwockA
Networks
Training uucuraccy
Tesl accuaity
PRS rato in the peraltimate layer (Laper))))
(a) PRS ratio
(b) Network (Batch size: 2048) and network B (Batch size: 128)
Figure 2: (a) The number of PRS for the depth of each layer. (b) Training/Test accuracy and the PRS ratio on the penultimate layer of the networks with six convolution blocks (CNN-6) with batch size 2048/128. We select the networks at the 300th epoch and denote these two CNN-6 by Network A and B in the paper (PRS ratio of Network A: 0.99, and Network B: 0.007).
We note that the internal DB Bi (1 < L) divides the input space X' based on the hidden representation of the 1-th layer (i.e., existence of feature and the amount of feature activation). There D boundaries and the configuration of the DBs are determined by the training. As input samples in the same classification region are considered to belong to the same class, the input samples placed on the same side of the internal DB Bj share the similar feature representation. In this sense, we define the internal DR, which surrounded by internal DBs.
Definition (Decision Region (DR)) Let Vi E (-1,+11 be the indicator vector to choose pos- itive or negative side of decision boundaries the 1-th layer. Then the decision region D which shares the sign of feature representation, is defined as,
DRv. {alssig((fia (x)) Vi, Vo X}.
Figure 1 presents each training method (in Section 4 for VGG-16 with CIFAR-10 and the inter- nal DRs/DRs of the penultimate layer (fz-1: (z)). We identify the proposed regularizer (A-> C) induces different configuration of DBs/DRs in the input space compared to the standard training (A+B)).
2.2 POPULATED REGION SET
It is shown that the number of DRs is related to the representation power of DNNs (Montúfar et al, 2014). In particular, the expressivity of DNNs with partial linear activation function is quantified by the maximal number of the linear regions and this number is related to the width and depth of the structure. We believe that although the maximal number can be one measure of expressivity, the trained DNNs with finite training date does not handle the regions to solve the task. To only consider DRs that the network uses in the training, we devise the training-related regions where training samples are populated more frequently. We define the Populated Region Set (PRS), which is a set of DRs containing at least one training sample. PRS can be utilized to analyze the relationship between the geometrical properties and the robustness of DNNs in practical aspect.
Definition (Populated Region Set (PRS) From the set of every DRs of the model and given the dataset X, the Populated Region Set for 1-th layer is defined as,
PRS(X, f)) = (DRv, Vr = sign(fua(x)) Vx E XX.
We can then define Populated Region union of decision regions in PRS as,
PR f.() = EP
We note that the size of the PRS is bounded to the size of given dataset X. When PPRS(X, f.)) |X| each sample in training dataset assigned to each distinct DR in the 1-th layer. To compare the PRS among networks, we define the PRS ratio, |PRS(J, XXX))//X,, which measures the ratio between the size of the PRS and the given dataset. Figure presents a comparison between two equivalent neural networks (A and B) with six convolution blocks (CNN-6) trained on CIFAR-10 varying only the batch size (2048/128). Figure 2 (a) presents the PRS ratio for the depth of layers in each model at the 300th epoch. We observe only the penultimate layer (7 8) shows different PRS ratio. Figure 2 (b) shows that the two networks have largely different PRS ratios with similar training/test accuracy. From the above observation and the fact that the penultimate layers are widely used as feature extraction, we only consider the PRS ratio on the penultimate layer in this paper.
In general, the number of training data is smaller than the maximal number of the linear region.
3

---

Under review as a conference paper at ICLR 2023
3 ROBUSTNESS UNDER ADVERSARIAL ATTACKS
In this section, we perform experiments to analyze the relationship between the PRS ratio and the robustness. We evaluate the robustness of the network using the fast gradient sign method (FGSM) [Goodfellow et al. 2014), basic iterative method (BIM) (Kurakin et al., 2016) and projected gra- dient descent (PGD) (Madry et al., 2018) method widely used for the adversarial attacks. The untargeted ad adversarial attacks using training/test dataset are performed for the various perturbations
(e = 0.0313, 0.05, 0.1).
3.1 EXPERIMENTAL SETUP
For the systematic experiments, select three different structures of DNNs analyze: (1) convo- lutional neural network with six convolution blocks (CNN-6), (2) V6G-16 (Simonyan & Zisserman 2014), and (3) ResNet-
18 (He et al.) 2016). We train basic models with fixed five random seeds and
four batch sizes (64, 128, 512 and 2048) over three datasets; MNIST (LeCun & Cortes 2010), F- MNIST (Xiao et al. 2017). and CIFAR-I0 (Krizhevsky et al. 2009). For the extensive analysis on the correlation between the PRS ratio and properties of network, we extract candidates from each model with the grid of epochs. Then we apply the threshold for the test accuracy to guarantee the sufficient and similar performance. Finally, we obtain 947 models for analysis. The details for the network architecture and the selection procedure are described in Appendix A-C. We also perform an ablation study the factors which affects the PRS ratio and the results are provided in Appendix E.
3.2 PRS AND ROBUSTNESS
First, we compare the two models (Network A and in Figure 2 with similar test accuracy but different PRS ratic P and PGD-20 (20-step) on Lo with a = 2/255. We identify Network B (low PRS ratio) is more robust than Network A (high PRS ratio) under all adversarial attacks.
Figure 3 presents the results of robust accuracy under the FGSM, BIM (5-step)
FGSM
PGD-20
Accuray
0.0
Teining (Netwook))
Thiniing (Nitworii R)
0.4
0.4
Test (Neetookk
0.2
0.2
Test (Network B)
Epalen
Figure 3: Robust accuracy under various adversarial attack methods networks A and B. The x- axis indicates perturbation c and the y-axis indicates the training/test robust accuracy.
From the above observation, we measure the PRS ratio and the robust accuracy in various models and datasets to verify the relationship between the PRS ratio and the robustness. Figure 4 presents the experimental results according to the various models under the PGD-20.
To quantify the relation, we calculate the coefficient of the regression line and perform significance test to validate the trend. We also provide the result of robust accuracy against the FGSM attack and AutoAttack (Croce & Hein| 2020) in Appendix F. In Figure E we identify the PRS ratio has an inversely correlated relationship with the robust accuracy in most cases. From this observations, empirically confirm the PRS ratio is related to the robustness against adversarial attacks. To investigate the evidence that the low PRS ratio causes robustness for the gradient-based attack, perform additional analysis of failed attack samples. In the gradient-based attack, as the magnitude of the gradient is a crucial component to success, we observe the failure attack cases for Network A and B. We note the failed attack samples with non-zero gradients maintain the index of the largest logit as the true class after attack. To analyze the reason of failure, we examine the change of the logits under the adversarial attack in Figure 5] (a). To clarify the difference of the change of the logits between Network A and B, we select the examples of successful attack on Network A but failed attack on Network B. In Network B, the logits move on almost parallel direction, which causes the predicted label to be maintained as the true class.
We note different PRS ratios obtained hy different hatch size of Network A (2048) and B (128).
4

---

Under review as a conference paper at ICLR 2023
CNN-66
VGG-16
ResNNtt1B
1.0
Cud.
Gad:: -045
Ood.
Puni 94089-6
1:30-44
Pvol 70LE-4
2/00/
223E-05
Pood: 5238-13
0.6
MNEST
FNNSST
CIBRR-10
0.0
0.0
0-4
0.8
1.0
0.00
0.05
0.10
0.15
0.20
0.25
0.0
0.3
0.5
PVS Ratlo
PRSS Ratio
PRSS Ratio
Figure 4: Relationship between the PRS ratio and robust accuracy attacked by PGD (e: MNIST 0.3, F-MNIST 0.1, and CIFAR 0.0313 on L norm). The colored dots are for the independent models described in Appendix A. The colored dashed lines indicate the trend for each dataset.
Logits
Logits
No Net Om
Epoch
Epoch
Epoch
Epocii
Eppcii 100 Fooch
Fpoch 300
Network
Netwock
Original ligis ackeratzck
lodes
tndes
(a) Illustrative examples for change of logits
(b) Cosine similanty matrix for final layer on Network A/B
Figure 5: (a) Illustrative examples of attacked samples Network A and B which failed on and the corresponding logits before/after the attack. (PGD--0 on T. with e = 0.0313). After the attack, the logits move on almost parallel direction with the original logits in Network B. More examples are provided in Appendix C. (b) Cosine similarity (CS) matrix for final layer on Network A/B. As
the epoch increases, the CS for each parameter increases on Network
CNN-6
VGG-16
ResNet-18
Dataset
Coef
P-val
Coef
P-val
Coef
P-val
MNIST
-0.76
5.17E-29
-14.7
2.70E-08
-2.88
1.361i11
F-MNIST
-0.58
2.43E-46
-3.38
4.11E-19
-2.24
1.66E-16
CIFAR-10
-0.65
4.35E-54
0.28
2.92E-01
-1.29
4.20B-14
Table 1: Coefficient (Coef) and P-value (P-val) of the regression analysis between the PRS ratio and cosine similarity.
To explain the parallel change of the logit vector, we hypothesize the DBs corresponding to each class node have similar configuration in the input space. However, intractable measure the similarity between DBs in the network due to the highly non-linear structure and the high dimen- sional input space. To simplify our hypothesis, we only measure the cosine similarity between the parameters which map the features on the penultimate layer to logits (i.e., final layer). Figure (b) presents the similarity matrices for Networks and B. When we compare the matrices between the two models at the 300th epoch, identify Network B (low PRS ratio) has higher cosine similarity between each parameter in the final layer. We note that the cosine similarity between each parameter in the final layer can be considered as the degree of parallelism for the normal vectors in the linear classifier. We also confirm the decrease of the PRS ratio is aligned with the increase of the similarity of parameters in Figure 5 (b) Network B, when consider the graph in Figure 2 To verify the relationship between PRS ratio and the cosine similarity, we measure the PRS ratio and the cosine similarity between each parameter for all 947 candidate models. Table I shows the results of the regression analysis for the relationship between the PRS ratio and cosine similarity. We identify the PRS ratio has an inverse correlation for the cosine similarity between each parameter in the final layer.
3.3 PRS AND TEST SAMPLES
When we regard the model as a mapping function from the input space to the feature space, the majority of samples from the test dataset assigned to the training PR, the model can be considered
5

---

Under review as a conference paper at ICLR 2023
Polaut Tolostrcumss Accuney
pro Inchton grop
Rocaate grap)
CNN-
VOC-GG
Roc 18
hw--0.7
4179
125 12 Pual
P
PvP 1000-0
Prnk L22
20 PM
0380
Pras L8T1
ou 30
324
Co
004
0.0
1.0
0.00
0.19
P&S
lati
2:00
03
02
0.
(a) Robust Accurayy each group
Relationship between the PRS ratio indussinnnoooo
Figure 6: (a) Test accuracy under adversarial attacks for inclusion/exclusion groups for CNN-6 on CIFAR-10. The blue/orange line indicates the exclusion/inclusion group, respectively. (b) Relation- ship between the PRS ratio and the inclusion ratio for various models and datasets.
to be learned the informative and general concept of feature mapping. However, it is non-trivial to guess which decision will appear when the sample is mapped to out of the training PR. To investigate the differences between the test samples which are included and excluded in the training PR, we evaluate the test accuracy under adversarial attack for each group. For a comparison, we divide both the inclusion and exclusion groups with 1K correctly predicted test samples. Figure 6 (a) shows the robust accuracy under the FGSM and PGD-20 on Los. Although the robust accuracy of each test group decreases as the epsilon becomes larger, we observe the inclusion group is more robust against both attacks compared to the exclusion group. More experimental results are provided in Appendix H. Figure (b) presents the results of the correlation for the relationship between the PRS ratio and the inclusion ratio of the test samples for the training PR. We compute the inclusion ratio the ratio of the test samples mapped to the training PR, and identify the PRS ratio and the inclusion ratio have inversely correlated relationship. As we previously verify that the included test samples show high robustness, we empirically confirm that the low PRS ratio is related to the robustness under adversarial attacks.
3.4 PRS AND TRAINING SAMPLES
From Section 3.3 we empirically observe that the vulnerability of individual test samples is related to PRS defined by training samples. In this section, we categorize the element of the PRS to expand this relationship. At first, we define the major DR for each class which includes the majority of training samples.
Definition 4 (Major Region (MR)) Let the training dataset with class X and the classifier f. The major region for 1-th layer and class e is defined as,
M MRi,
arginax DRy, (Rr,EPRS(X....)
Hx
X
We note that M R2,c denotes the DR and M R nX. represents the training samples which populate in M Rt.c- We refer the remained DRs (i.e., not MR) as the extra regions (ER).
3.4.1 COMPARISON OF MR AND ER
At first, we observe the distribution of training samples for the type of region corresponding each class in VGG-16 trained with CIFAR-1O. Figure (a depicts the distribution of training samples for MR (sky blue bar and black dashed line) and ER for each class. Although the training samples are distributed the various regions, we identify that MR exists for each class. To compare the character- isties of samples populated each region, we randomly selected 10k training samples from MR and ER. We perform adversarial attack for selected samples and measure the confidence of the prediction (logit value for the target class). Figure 7 (b) and (c) show the robust accuracy and confidence for MR and ER, respectively. We empirically verify the training samples in MR have higher adversarial robustness and the network predicts these samples with high confidence.
From the empirical observations that samples belonging to MR are relatively robust, we hypothesize that samples located closer center of MR tends to be more robust. To represent the center of MR. we assume the distribution feature vector fia (x) for populated training samples Vz e X, is gaussian distribution.

---

Under review as a conference paper at ICLR 2023
MR
Number sanptos
MR
0.8
Robus
0.4
CCasss
Fpeilors
Confidence
35
(8) The number of training semples in DR
(b) Robost accuracy for MR and ER
(c) Class confidence for MR and
Figure 7: Experimental results under VGG-16 CIFAR-IO. (a) The number of training samples populated in DRs for each class. The maximum number of samples for each class is 5k and the sky blue bar indicates each MR. (b) Robust accuracy under PGD-20 attacks on Loo for the samples in MR and ER. (c) Class confidence for the samples in MR and ER.
0.010
0.02
0.02
0.005
0.01
Dorsty
Confidence
100
200
100
G
0.02
7
9
0.02
0.01
0.2
0.01
0.01
0.00
0.00
0.00
0.00
0.00
-10
250
250
Attack success
Atlack fiilure
Distmmarto MRV
(b) Relationhip betwoen the distance the MRV confidence
Distribution distance between the each festure vector and the MRV
Figure 8: Experimental results under VGG-16 CIFAR-10. (a) Distribution of distance to MRV for training samples. The blue/red bar indicates the failed/success attack samples and the white box in upper right presents each class. (b) Relationship between the distance to MRV and the confidence. The colored dots represent samples which are vulnerable under adversarial attack per each class.
Definition (Major Region Mean Vectors (MRV) the major region for class e of 1-th layer M and the classifier f. The mean vector of major region is defined as,
MRV,
E
f1a(x).
|M NX
REM
To verify our hypothesis, we measure Euclidean distance between MRV and training samples with success/failure adversarial attack. From Figure (a), identify the samples far from MRV tend to be vulnerable for the adversarial attack in the class. Furthermore, in Figure g (b), we identify the inversely correlated relationship between the confidence and the distance MRV for the failed attack samples.
ROBUST LEARNING VIA PRS REGULARIZER
In previous Sections, empirically verify that PRS is related the adversarial robustness. In particular, (1) inclusion of MR, and (2) distance to MRV are highly related to vulnerability of indi- vidual samples. From these insights, we devise novel regularizer leveraging the properties of PRS to improve the adversarial robustness.
4.1 REGULARIZER PRS
At first, we design the regularizer to reduce the distance between feature vector and MRV. To guar- antee the quality of feature representation which constructs plausible PRS, we utilize the warm-up stage for the classifier. In the warm-up stage, the classifier is trained with cross-entropy loss function Lae during the t-th epoch. After warm-up stage, we construct the MRV for each class and use it after 1-th epoch. Let (1 vi) 2 be training dataset and f be classifier. The regularizer for MRV is

---

Under review as a conference paper at ICLR 2023
Model
Method
Robust Acc
Test Acc
PRS Ratio
Time/Epoch (s)
Standard
38.82+2.73
77.92+0.34
0.101=0.010
8.92+0.32
Resnet-18 18 VGG-16 CNN-6
AT
53.79+0042
70.65:0.17 0.09940.00T (..5500.770000999......
38.63400099
L (Ours)
53.59+0.07
80.30=0.72
0.018+0.003
9.4622.13
LPRS (Ours)
54.52+0.61
80.35+0.11
0.017=0.002
9.78+0.07
Standard
39.94 1.28
80.28*0.24
0.11520.012
11.63+0.18
AT
18.18+0.13
75.22+0.05
0.069=0.002
81,48+0.23
CaRR (Ours)
60.42++0.36
78.61=0.15
0.038+0.007
12.15+0.21
LPRS (Ours)
61.08+0.88
79.09=0.35
0.031#0.005
12.63+0.11
Standard
33.48+0..8
76.9620.15
0.065+0.001
12.11+0.32
AT
50.65+0.20
73.03=0.05
0.046#0.000
58,96+0.15
CMR (Ours)
49.31=0.65
76.51
0.061#0.003
12.97+0.29
LPRS (Ours)
50.54+0.10
76.44=0.18
0.038+0.004
13.0240.35
Table 2: Comparison of robust and test accuracy under PGD-20 attacks on Loo for CIFAR-10.
defined as,
1 N LMRV= = N
Σ (MRVi - fxn()))))
(I)
We note that because LMRV reduce the distance MRV in the feature space, the arbitrary sample can have the opportunity for inclusion of MR. However, it is non-trivial to guarantee for inclusion based on Euclidean distance, because the feature vector encoded (1) in the high dimensional space, and (2) with highly non-linear representations. To ensure that the training samples are included into MR, we devise additional regularizer through Hamming distance.
N
Dr
Cham
sign ((MRV,))))) @ sign (f))))))))
ND.
where Dj is dimension of 1-th layer, and @ is the exclusive operator which returns zero when the target and prediction are identical and one otherwise. Finally, we define PRS regularizer LPRS with the weighted sum of Lee. LMRV and Lham.
LPRS d1 Loe + A2 LMRV >g
-hhe
where \1, A2, and 0u hyperparameters. We perform simple grid search to hyperparameters and use d1 = 0.2, A2 = 0.8, and Ag = for remained experiments. We denote the loss function with 0g by LMR-
4.2 EXPERIMENTAL RESULTS
To verify the effectiveness of our method, we apply the regularizer to various models for the clas- sification task CIFAR-10. We set standard training (Lee) and adversarial training (AT) based on PGD-20 on Loc as the baselines. We freeze the parameters of the final layer for the trained model with standard training in warm-up stage (L 50), and observe the effect of the change of PRS for bust and test accuracy. Figure FigureTdepicts depicts the training procedure for each training method and trained DBs and DRs which represent the state of PRS. In Table we verify that the proposed method can improve the robust accuracy while maintaining the test accuracy. We note that the proposed PRS regularizer does not use the adversarial examples to improve the adversarial robustness. means our method can have strength in the perspective of computation time. We further investigate the change of PRS after training for each method. In Figure 9 we identify the both proposed regularizers can increase the number of populated training samples in MR compared to the standard training (i.e., reduce of PRS ratio). Interestingly, we confirm the similar result for change of PRS in AT. Although AT and PRS regularizer are not related directly, consider this phenomenon can be empirical idence to support relationship between PRS and adversarial robustness. We also provide the results for different networks and large dataset in Appendix F.2.
8

---

Under review as a conference paper at ICLR 2023
Standard
AT
LMR (Ours)
lpps (Ours)
of
Numbee samples
37k
330
Number
Class
Standard
AT
(Ours) (Ous))
Lux
Leas
(a) Companison of the distrbutionn of DR for each class
(b) Comparison of PRS and MR
Figure 9: Experimental results under VGG-16 on CIFAR-10. (a) The distribution of DR for each class in various training methods. The sky blue bar per class indicates each MR. (b) The number of PRS and samples in MR.
5 RELATED WORK
The adversarial attack which reveals the vulnerability of DNNs, is mainly used to validate the relia- bility of the trained network. As an early stage for adversarial attacks, the fast gradient sign method (FGSM) (Goodfellow et al.] (014) based on the gradient with respect to the loss function and the multi-step iterative method (Krrkinn et al. (016) are proposed to create adversarial examples to change the model prediction with small perturbation. Recently, many studies on effective attacks in various settings have been performed to understand the undesirable decision of the networks
(Chen et al. 2020; Madry al. 2018 Shaham et al., 2018). In terms of factors affecting robustness, Yao et al. (2018) provides evidence to argue that training with a large batch size can degrade the robustness of the model against the adversarial attack from the perspective of the Hessian spectrum. With increasing interest in the expressive power of DNNs, there have been several attempts to ana- lyze DNNs from geometric perspective (Choromanska al., 2015 Dauphin al. 2014). In these studies, the characteristics of the decision boundary or regions formulated by the DNNs are mainly discussed. Montúfar et al. (2014) show the cascade of the linear layer and the nonlinear activation organizes the numerous piece-wise linear regions. They show the complexity of the decision bound- ary related to the maximal number of these linear regions, which determined by the depth and the width of the model. Xiong et al. (2020) extend the notion of the linear region to the convolu- tional layers and show the better geometric efficiency of the convolutional layers. Compared the previous work, work focuses the practical decision region which the trained network actually utilizes, interpreting the vulnerability of DNNs with the geometry is another important topic. Croce et al. 2019) discusses the relationship between the max margin of linear regions (LRs) and the bustness in ReLU networks. In particular, they utilize the perpendicular distance to the linearized internal/final decision boundaries (DBs) which comprise the LRs in the input space. In the bounded input space, increasing the distance to DBs expands the volume of LRs, and it eventually reduces the number of LRs. But, as our work mainly focuses on the number of PR on the penultimate layer rather than LRs, LPRS is not limited to the partial linear relation between the input samples and internal DBs, and it can reduce the PRS ratio directly.
CONCLUSION
In this work, we propose the novel concept Populated Region Set (PRS) to derive the relationship between the geometrical properties of DNNs and robustness in a practical setting. From systematic experiments with the proposed concept, we observe the empirical evidences that the PRS is related to the adversarial robustness of DNNs: (1) The network with the low PRS ratio shows high robustness against the gradient-basee based attack compared to the network with a high PRS ratio. In particular, the model with a low PRS ratio has higher degree of parallelism for the parameters in the final layer. which can support robustness. (2) The network with a low PRS ratio includes more test samples in the training PR. We empirically verify this inclusion ratio related to robustness from the observation that included test samples are more robust than excluded test samples. (3) The proposed PRS regularizer can improve the robustness without adversarial examples. We verify the proposed method is valid for various network architecture. We also observe the adversarial training reduces the PRS ratio with the improvement of robust accuracy. It can be considered as another empirical evidence to support the relationship between PRS and adversarial robustness. Our work provides the insight of the geometrical interpretation of robustness in perspective of decision region. We expect the concept of PRS would contribute to the improvement of robustness in DNNs.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
D Richard Baughman and Yih An Liu. Neural networks in bioprocessing and chemical engineering. Academic press, 2014.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer. 2006.
Jianbo Chen, Michael Jordan, and Martin J Wainwright. Hopskipjumpattack: A query-efficieet decision-bar based attack. In 2020 ieee symposium on security and privacy (sp), Pp. 1277-1294.
IEEE, 2020.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204, PMLR, 2015.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustmess with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020.
Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of relu net- works via maximization of linear regions. In the 22nd International Conference Artificial Intelligence and Statistics, pp. 2057-2066. PMLR, 2019.
Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. arXiv preprint arXiv: 14X6::440..272, 2014.
Alhussein Fawzi, cation regions of deep neural networks. arXiv preprint arXiv:1705.09553 2017.
Moosavi-Dezfoolh Pascal Frossard, and Stefano Soatto. Classifi-
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with strong adver- sary. arXiv preprint arXiv:1511.03044 2015.
Yu Huang and Yue Chen. Autonomous driving with deep learning: survey state-of-art tech- nologies. arXiv preprint arXiv:2006.0609), 2020.
Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 514-529, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):466444 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportumities and challenges. Briefings bioinformatics, 1236-1266,, 2018. 8.
10

---

Under review as a conference paper at ICLR 2023
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014.
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Neurocomputing, 307:195-204, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 1556, 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13/2.6999 2013.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 708.0777 2017.
Huan Xiong, Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. International Conference on Machine Learning, pp- 10514-10523. PMLR, 2020.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. arXiv preprint arXiv: 2018.
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2855-2224, 2019.
Ziyuan Zhong, Yuchi Tian, and Baishakhi Ray. Understanding local robustness of deep neural networks under natural variations. Fundamental Approaches to Software Engineering, 12649: 313, 2021.


Reviews:
[{"review_id": "yfDv27ABL5Q", "reviewer": "Reviewer_wNGM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper looks at he notion of _Populated Region Set (PRS)_ which are decision regions with at least one training example in them. The central message of the paper is that networks with lower number of PRS have better robustness.", "summary_of_the_review": "The main drawbacks of the paper  are the following\n\n* The first red flag should have been that for $\\epsilon=0.1$, a CNN network trained via standard training has 80% robust accuracy in Figure 3. This is the main result the authors use to justify that low PRS ratio increases robustness. However, a quick look at Appendix F shows that both Model A and Model B have less than 1% robust accuracy. This shows that it is not that low PRS ratio models are more robust just that, it is harder to find adversarial examples possibly due to ill-conditioning.\n\n*Again Figure 6 shows a similar kind of behaviour as Figure and does not include measurements with AutoAttack.\n\n*Finally table 2 also doesn't contain AA. In fact, even the PGD attack is fairly weak containing only 20 steps. This can also call into question the PRS regularisers are indeed more robust than ST and whether PRS regulariser+AT is more robust than AT.\n\nI did find Appendix G quite interesting, as it does suggest a connection between PRS ratio and robustness but this is the only experiment that is not confounded by the possibility of weak attack.\n\nGiven the above, I am inclined to support rejection unless the authors convince me otherwise.\n", "strength_and_weaknesses": "## Strength\n\n* The concept of PRS is interesting and well motivated.\n* The design of the regulariser is also well motivated. \n* The figures and illustrations are easy to understand and the mathematical formalisations are minimal and kept to the point. Overall, the authors have tried to present a coherent story.\n\n## Weaknesses\n* The message of the paper is not borne by the experimental results. In particular, AA robust accuracy is always zero.\n* It seems that certain networks have not been evaluated under AA and all AA results are relegated to the appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8zogEhhf6j6", "reviewer": "Reviewer_x9iE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first studies the relationship of the decision regions induced by the penultimate layer of a deep net to adversarial robustness and then relies on empirical findings to propose strengthening adversarial robustness without adversarial training. ", "summary_of_the_review": "The empirical findings of this paper are pretty interesting. However, the proposed approach seems not able to generalize to all datasets because for Cifar100, it still needs adversarial training.", "strength_and_weaknesses": "Strength\n- Some empirical findings are quite interesting.\n- If a model can really defend without adversarial training, it is really great. However, it seems not the case for Cifar100 as in the supplementary material.\n\nWeaknesses\n- The concept of decision boundary and decision region as defined in the paper is confusing and misleading.  Because the decision region is normally relevant one class and represents the date examples classified to this class by a deep net, while the decision boundary represent the boundaries of the decision regions. I believe it should be better if the authors use the terminology like activation patterns. \n- Some behavior experiments lack of details and descriptions, hence hard to follow. For example, cosine similarity matrix of the final layer on Network A/B: cosine similarity of what and what (e.g., representations or weights). Also, Figure 5a is hard to interpret. \n- Mathematical notions used is not solid. For example, the one to define $PRS(X, f, l)$: why do we need $\\forall V_l \\in \\{-1, 1\\}^D$; the one to define $MR_{l,c}$: if $DR \\in PRS(X_c, f, l)$ then $DR \\in X_c$, hence $DR \\cap X_c = DR$.\n- It is great if $L_{ham}$ and $L_{MVR}$ can help to defense for all datasets. But it seems that for Cifar100, it still needs adversarial training. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7XSvr_lw6mM", "reviewer": "Reviewer_o1vx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper prooses the novel concept of populated region set, and make connection between the proposed PRS ratio and the adversarial robustness of the model. The paper further propose PRS regualrization that can improve adversarial robustness without adversarial training.", "summary_of_the_review": "Generally the paper provides novel and interesting insight on understanding the adversarial robustness of DNN models. However the lack of technical details make it hard to assess the correctness and practicalness of the proposed method, Thus I would recommend a weak rejection for now.\n\n## Post rebuttal\nI believe the response from the author clearifies my doubts. Given the author response and revision I'm increasing my score.", "strength_and_weaknesses": "## Strength\n1. This paper provides novel concept on PRS and PRS ratio, and make connection between the proposed concept and adversarial robustness\n2. Thorough experiments are conducted to emperically support the claim on the relationship between PRS and robustness\n3. The proposed regualrization indicates the potential benefit PRS can bring to understanding and preventing adversarial attack\n\n## Weakness\nSome technical details of the proposed method are not clearly explained. For example:\n1. How exactly is PRS computed given a DNN model? Is the computation method scalable to a larger/deeper model or a larger dataset?\n2. How is the gradient of the Hamming distance regularization is Eq.(2) computed in the optimization?\n3. As the model is being updated with the proposed regularization, will MRV change in the process? Does MRV need to be recomputed during the training process, and if so what would be the cost of the computation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7V9Se1CTUb", "reviewer": "Reviewer_mBon", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduced the concept of Populated Region Set (PRS) to characterize the complexity of DNNs and build the correction between low PRS ratio and high robustness of models via several experiments.", "summary_of_the_review": "In summary, the novelty of this paper is high, and the concept of the Populated Region Set (PRS) is novel and interesting. However, my main concern is that doesn't provide theoretical analysis, which makes the contribution limited. Also, some definitions are confusing at this point. ", "strength_and_weaknesses": "Strength: 1. The concept of the Populated Region Set (PRS) is novel and interesting.\n2. The experiments are convincing.  \n\nWeaknesses: 1. Lack of theoretical analysis to verify the claims.\n2. Some definitions are confusing. For example, in Definition 2 (Decision Region (DR)), if the i-th layer has \\textbf{only one neuron}, what are $D_\\ell$ and $|DR_{V_\\ell}|$? From the definition, it seems that $D_\\ell =1$ and $|DR_{V_\\ell}|=2$ in this case, but this seems strange. Please give one small example for each definition to make things clear. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject