Empirical Study of the Decision Region and Robustness in Deep Neural Networks
Authors: Seongjin Park
Haedong Jeong
Giyoung Jeon
Jaesik Choi
Venue: ICLR 2022 Submitted (2022)
Keywords: Decision Region
Adversarial Robustness
Deep Neural Networks

Abstract: In general, the Deep Neural Networks (DNNs) is evaluated by the generalization performance measured on the unseen data excluded from the training phase. Along with the development of DNNs, the generalization performance converges to the state-of-the-art and it becomes difficult to evaluate DNNs solely based on the generalization performance. The robustness against the adversarial attack has been used as an additional metric to evaluate DNNs by measuring the vulnerability of them. However, few researches have been performed to analyze the adversarial robustness in terms of the geometry in DNNs. In this work, we perform empirical study to analyze the internal properties of DNNs which affect model robustness under adversarial attacks. Especially, we propose the novel concept Populated Region Set (PRS) where train samples populated more frequently to represent the internal properties of DNNs in the practical setting. From the systematic experiments with the proposed concept, we provide empirical evidences to validate that the low PRS ratio has strong relationship with the adversarial robustness of DNNs.

Under review as a conference paper at ICLR 2022
EMPIRICAL STUDY OF THE DECISION REGION AND ROBUSTNESS IN DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In general, Deep Neural Networks (DNNs) evaluated by the generalization performance measured on unseen data excluded from the training phase. Along with the development of DNNs, the generalization performance converges the state-of-the ofct----te-rtttt art and it becomes difficult to evaluate DNNs solely based on this metric. The robustness against adversarial attack has been used an additional metric to evaluate DNNs by measuring their vulnerability.. However, few studies have been performed to analyze the adversarial robustness in terms of the geome- try in DNNs. In this work, we perform an empirical study to analyze the internal properties of DNNs that affect model robustness under adversarial attacks. par- ticular, we propose the novel concept of the Populated Region Set (PRS), where trained samples are populated more frequently, to represent the internal properties of DNNs in practical setting. From systematic experiments with the proposed concept, we provide empirical evidence to validate that a low PRS ratio has a strong relationship with the adversarial robustness of DNNs.
INTRODUCTION
With the steep improvement of the performance of Deep Neural Networks (DNNs), their applica- tions are expanding to the real world, such as autonomous driving and healthcare (LeCun et al. 2015 Miotto et al., 2018 HHuag & Chen 2020). For real world application, it may be necessary to choose the best model among the candidates. Traditionally, the generalization performance which measures the objective score on the test dataset excluded in the training phase, is used to evaluate the models (Bishop 2006). However, is non-trivial to evaluate DNNs based on this single metric. For example, if two networks with the same structure have the similar test accuracy, it is ambiguous which is better.
Robustness against adversarial attacks, measure of the vulnerability, can be an alternative to evaluate DNNs (Szegedy et al., 2013; Goodfellow et al. 2014 Gu & Rigazio 2014 Huang et al., 2015) Jakubovitz & Giryes! 2018 Yuan et al. [2019] et al. [021]) Adversarial attacks aim to induce model misprediction by perturbing the input with small magnitude. Most previous works were focused on the way to find adversarial samples by utilizing the model properties such gradients with respect to the loss function. Given that the adversarial attack seeks to find the perturbation path on the model prediction surface over the input space, robustness can be expressed in terms of the geometry of the model. However, few studies have been performed to interpret the robustness with the concept of the geometric properties of DNNs.
From a geometric viewpoint, the internal properties of DNNs are represented by the boundaries and the regions (Baughman & Liu| 2014). It is shown that the DNNs with piece-wise linear activation layers are composed of many linear regions, and the maximal number of these regions is mathe- matically related to the expressivity of DNNs (Montáfar al. 2014; Xiong et al., 2020). As these approaches only provide the upper bound for the expressivity with the same structured model, does not explain how much information the model actually expresses.
In this work, we investigate the relationship between the internal properties of DNNs and the ro- bustness. In particular, our approach analyzes the internal characteristics from the perspective of the decision boundary (DB) and the decision region (DR), which are basic components of DNNs (Fawzi cet al., 2017). To avoid insensitivity of the maximal number of linear regions in the same structure assumption, propose the novel concept of the Populated Region Set (PRS), which is set of DRs

---

Under review as a conference paper at ICLR 2022
Degit of Lager
Coay Blocks
Penulimate: Layer
Finsl Layer
High PRIS Ratio
Adversariall
Netwad A High motioio
Network
A Attak sucoess
Attack Callure
Figure 1: An illustrative comparison of two networks that have different PRS ratios (High and Low) with similar test accuracy. (Left column) Each colored box image corresponds to the colored dot in the right column. The orange/red box image is the perturbed image in each network under an adversarial attack respectively. (Middle column) The network with a low PRS ratio learns more sparse feature representation than that with high PRS ratio. (Right column) Decision boundaries and regions in the section of input space for the target layer. Contour indicates the logit value for dog class.
containing at least one sample included in the training dataset. Since the PRS can be considered as the feasible complexity of the model, we hypothesize that the size of PRS is related to the robustness of network. To validate our hypothesis, we perform systematic experiments with various structures of DNNs and datasets. Our observations are summarized as follows:
The models with the same structure can have different size of PRS, although they have similar generalization performance. In experiments, observe that this difference leads to different robustness of the network.
We empirically show that the size of the PRS is related to the robustness against the ad- versarial attack. The model with small size of the PRS tends to show higher robustness compared to that with large size (in Section 1. We further observe that when the model achieves low PRS ratio, the linear classifier that maps the penultimate features the log- its has high cosine similarity between parameters corresponding to each class (in Section 4.2
We verify that the size of intersection of the PRS from the training/test dataset related to the robustness of model. The model with a high PRS inclusion ratio of test samples has higher robustness than that with low PRS inclusion ratio (in Section 5)
We identify that the model with small size of the PRS learns the sparse feature represen-
tation. In quantification, we observe the inversely correlated relationship between the size of the PRS and sparsity of feature representation (in Section 6).
RELATED WORK
Adversarial robustness For the real-world application of DNNs, the adversarial attack, which re- veals the vulnerability of DNNs (Goodfellow et al.. 2014), is mainly used to validate the reliability of the trained network. As an early stage for adversarial attacks, the fast gradient sign method (FGSM) [Goodfellow et al., 2014) based on the gradient with respect to the loss function and the multi-step iterative method (Kurakin et al.) (016) are proposed to create adversarial examples to change the model prediction with a small perturbation. Recently, many studies on effective attacks in various settings (e.g., white-box or black-box have been performed to understand the undesirable decision of the networks (Shaham et al.) 2018 Madry et al. 2018 Chen et al.) 2020). In terms of factors affecting robustness, Yao et al. (2018) provide evidence to argue that training with a large batch size can degrade the robustness of the model against the adversarial attack from the perspective of the
2

---

Under review as a conference paper at ICLR 2022
Hessian spectrum. In contrast, Kamath al. (2019) propose that the model with a constant ratio between the learning rate and batch size does not degrade the model robustness even with a large batch size as it converges to the flatter minima.
Geometric Analysis inside Deep Neural Networks With increasing interest in the expressive power of DNNs, there have been several attempts to analyze DNNs from geometric perspective (Dauphin et al. 2014 Choromanska et al. (013). In these studies, the characteristics of the decision boundary or regions formulated by the DNNs are mainly discussed. [Montáfar al. (2014) show that the cascade of the linear layer and the nonlinear activation organizes the numerous piece-wise linear regions. They show that the complexity of the decision boundary is related to the maximal number of these linear regions, which is determined by the depth and the width of the model. Xiong et al. (020) extend the notion of the linear region the convolutional layers and show the better geometric efficiency of the convolutional layers. FFwzi et al. (2018) reveal that classification regions DNNs are topologically connected and the decision boundary of natural images is flat in most directions. It has also been shown that the manifolds learned by DNNs and the distributions over them are highly related to the representation capability
network (Lei et al., 2018). While these studies highlight the benefits of increasing expressivity of DNNs as the number of regions increases, interpreting the vulnerability of DNNs with the geometry is another important topic. Yang et al. (2020) show that model with thick decision boundaries induces robustness. Moosavi-Dezfooli et al. (2019) show that a decision boundary with a small curvature acquires the high robustness of the model. These approaches focus on the decision boundaries, while this paper suggests to focus on the decision regions, which are composed by the surrounding decision boundaries.
3 INTERNAL PROPERTY OF DNNS
This section describes the internal properties of DNNs from the perspective of decision boundaries (DBs) and regions (DRs). The DBs of the DNN classifier is mainly defined as the borderline between DRs for classification, where the prediction probability of class and the neighboring class j are the same (Fawzi et al., 2018). To expand the notion of DBs and DRs to the internal feature-level, we re- define the DBs in the classifier that generalizes the existing definition of DBs. We then propose the novel concept of the Populated Region (PR) that describes the specific DRs used from the network for training samples. PR is used to analyze the relationship between the trained parameters and the characteristics of networks.
3.1 DECISION BOUNDARY AND REGION
Let the classifier with L number of layers be F(x) r is the sample the input space X C RD= and (-) denotes the non-linear activation functior For the 1-th layer, fi(-) denotes the linear operation and fia denotes the value of the i-th element of the feature vector fia(z) e Pi We define the DB for the i-th neuron of the (-th layer.
fLa(2), where
Definition (Decision Boundary (DB)) The i-th decision boundary at the I-th layer defined as
B = (x) 0, Vx X).
We note that the internal DB BJ (1 L) divides the input space X' based on the hidden representation of the I-th layer (i.e., existence of feature and the amount of feature activation). There are total of Dj boundaries and the configuration of the DBs are arranged by the training. As input samples in the same classification region are considered to belong to the same class, the input samples placed on the same side of the internal DB Bj share the similar feature representation. In this sense, we define the internal DR, which is surrounded by internal DBs.
Definition 2 (Decision Region (DR)) Lei VI E (11,+1) be the indicator vector to choose pos- itive or negutive side of decision boundaries of the 1-th layer. Then the decision region DRv,,, which shares the sign of feature representation, is defined as
DRv, = (a|sign(ffi (x)) = Vi,
Vr X).
'Athhough there various activation functions, we only consider ReL. activation for this paper.
3

---

Under review as a conference paper at ICLR 2022
Figure 1 presents the internal properties for two networks trained CIFAR-10 10 with similar test accuracy. The right column in Figure depicts the internal DBs and DRs in the network with a high/low PRS ratio (top and bottom). We randomly select two test images (blue and green box) and generate adversarial images for blue box (orange and purple box) in each network, respectively. We make hyperplane with these images to visualize the DBs and DRs in the 2D space. We identify that the configuration of DBs and DRs appears to be different, although the two networks have the same structure and similar test accuracy.
3.2 POPULATED REGION SET
It is well-studied that the number of DRs is related to the representation power of DNNs (Montúfar ctt al., 2014 Xiong et al. 2020). In particular, the expressivity DNNs with partial linear activation function is quantified by the maximal number of the linear regions and this number is related to the width and depth of the structure. We believe that although the maximal number can be one measure of expressity, the trained DNNs with finite training date date:canno handle the entire regions to solve the task. To only consider DRs that the network uses in the training process, we devise the train-related regions where training samples populated more frequently. We define the Populated Region Set (PRS), which is a set of DRs containing at least one sample included in the training dataset. PRS will be used to analyze the relationship between the geometrical property and the robustness of DNNs in a practical aspect.
Definition 3 (Populated Region Set (PRS)) From the set of every DRs of the model f and given the dataset X, the Populated Region Set defined as
PRS(X, f.l) (DRv,læ la DRv, 3zex, VV {-1,1)"
We can then define a Populated Region as a union of decision regions in PRS as
PR(X. f. UDREPRS(X...))D
We note that the size of the PRS bounded the size of given dataset X. When (PRS(X, [X|, each sample in training dataset is assigned to each distinct DR in the (-th layer. To compare the which measures the ratio between the size
PRS of networks, we define the PRS ratio, [PRS(f,X)))
of the PRS and the given dataset.
3.0
0.7
0,7
0.6
0.3
0.3
02
100
150
250
250
300
100
250
200
250
350
Layer
Epochs
Epochs
Nebrock A
Neeworiz 3
Tixin axoesay
Tess 4XCMSEY
PRS rulio in the peruiliiatte layer (layse
(a) PRS ratio
(b) Network A (Batch size: 2048) and network B (Batch size: 128)
Figure 2: (a) The number of PRS for the depth of each layers. (b) Training/Test accuracy and the PRS ratio the penultimate layer CNN-6 with batch size 2048/128. We select the networks at the 300th epoch and call these two CNN-6 as Network A and B, respectively, throughout the paper (PRS ratio of Network A: 0.99, and Network B: 0.007).
Figure presents a comparison between two equivalent neural networks (A and B) with six convo- lution blocks (CNN-6) trained on CIFAR-10 varying only the batch size (2048/128). Figure (a) presents the PRS ratio for the depth of layers in each model at the 300th epoch. We observe that only the penultimate layer (1 = 8) shows a different PRS ratio. Figure (b) shows that the two networks have largely different PRS ratios with similar training/test accuracy. From the above observation and the fact that the penultimate layers are widely used as feature extraction, we only consider the PRS ratio on the penultimate layer in the remainder of the paper.
Experimental setups For the systematic experiments, we select three different structures of DNNs to analyze: (1) convolutional neural network with six convolution blocks (CNN-6); (2) VGG-16
In general, the number of training data is smaller than the maximal number of the linear region.
4

---

Under review as a conference paper at ICLR 2022
Simonyan & Zisserman 2014] and (3) ResNet-18 18 (He et al. 2016). We train basic models with fixed five random seeds and four batch sizes (64, 128. 512 and 2048) over three datasets: MNIST (LeCun & Cortes 2010), F-MNIST (Xiao et al. 2017). and CIFAR- 10 (Krizhevsky et al.) 2009). For the extensive analysis on the correlation between the PRS ratio and properties of network, we extract candidates from each basic model with the grid of epochs. Then we apply the test accuracy threshold to guarantee the sufficient performance. Finally, we obtain 947 models for analysis. The details for the network architecture and the selection procedure are described in Appendix A-C.
4 ROBUSTNESS UNDER ADVERSARIAL ATTACKS
In this section, we perform experiments analyze the relationship between the PRS ratio and the robustness. We evaluate the robustness of the network using the fast gradient sign method (FGSM) (Goodfellow et al., 2014), basic iterative method (BIM) (Kurakin et al. 2016) and projected gradient descent (PGD) (Madry et al., (0188) method widely used for the adversarial attacks. The untargeted adversarial attacks using training/test dataset are performed for the various perturbation (e).
4.1 PRS AND ROBUSTNESS FOR MODELS
First, compare the two models (Network A and B in Figure 2 with similar test accuracy but different PRS ratio Figure presents the results robust accuracy (RA) under the FGSM, BIM (5-step), PGD-20 (20-step), and PGD-100 (100-step) on L For each step, a 2/255 is adopted. We identify that Network B (low PRS ratio) is more robust than Network A (high PRS ratio) under all adversarial attacks.
FGSM
BIM
PGD-20
PGD-100
Rolxst Accuacy
0.8
0.8
0.0
0.4
0.4
0.4
0.2
8.0
00:13
0.8
Epsilon
0
0,0313
0.0315
9.56
Triiing (Network
Training (Network B)
Test (Network A)
Test (Network
Figure 3: Robust accuracy under various adversarial attacks networks A and B. The x-axis indi- cates perturbation e and the y-axis indicates the training/test robust accuracy under the attack.
As the PGD attack with 20-step size shows the similar RA compare 100-step size, focus on an analysis under the PGD attack with a 20-step size the rest of the paper. We measure the PRS ratio and the RA in all models and datasets verify the relationship between the PRS ratio and the robustness. For the experiments, we take the magnitude of as follow: MNIST = 0.3, F-MNIST 0.1, and CIFAR10 0.0313 on Lx norm. Figure 4 presents the experimental results according to the model structure under the PGD attack. To quantify the relation, we calculate the slope of the regression line and perform significance test to validate the trend. We also provide the result of RA against the FGSM attack and Auto (Croce & Hein 2020 in Appendix G. From Figure we identify that the PRS ratio has an inversely correlated relationship with the RA most cases. The complicated models (VGG-16 and ResNet-18) have lower PRS ratios compared to CNN-6, and show even lower PRS ratios on the simple datasets (i.c., MNIST and F-MNIST).
PRS and Robustness for Training We also analyze how the PRS ratio and the RA change during the training. We measure the PRS ratio and the RA under the PGD attack for three different networks trained on CIFAR-10. Figure Jigure/spresents presents the test accuracy, PRS ratio and RA for each training epoch. Although the test accuracy of each model converges after the 50th epoch, the PRS ratio continues to decrease after the 50th epoch and the RA continues increase.
'Cross-entropy loss and Adam optimizer with learning rate 10- is used. WWe note that different PRS ratios obtained by different batch size of Network A (2048) and B (128).
5

---

Under review as a conference paper at ICLR 2022
CNN-6
VGG-16
ResNel-8
MINST
Robast Acturay
F-NNIST
CHAR-11
0.0
0.2
0.4
0.6
0.8
1.0
0.00
15
0.00
0.25
0.0
0.1
0:2
0.3
0.4
0.5
Ratio
PRS Ratio
PRS Ratio
Skype 4030
Sign -0.3
Stepe 40221
Skege 16.30
Slege X250
Sipp -0.8
0010
Slege -150
Slege <111
Pvolaa: 9.
Pinlus 12E 63
Poatux 2.00
Piilla 70LE
Pridus: 4.3E 18
Podlu:: 2. U3E
P Pooll:: L.E 02
Piille S22E 13
Figure 4: Relationship between the PRS ratio and RA attacked by PGD method in various models and datasets. The colored dots are for the independent models descrihed in Appendix A. The colored dashed lines indicate the trend for each dataset.
CNN-6
VGG-16
ResNet-18 18
04
Epocts
Epocts
Ppodhs
Auruary
Robust Accuracy (RA)
PRS ratio
Figure 5: PRS ratio and RA for training epochs on three different networks with CIFAR-10. The blue line indicates the test accuracy, the orange line indicates the RA, and the green line indicates the PRS ratio. The shaded lines depict the standard deviation for five fixed random seeds.
4.2 PRS AND FINAL LAYER
From the observations above, we empirically confirm that the PRS ratio is related to the robustness against adversarial attacks. In order to investigate the evidence that the low PRS ratio causes ro- bustness for the gradient-basse
attack, we perform an additional analysis of failed attack samples. In the gradient-based attack, as the magnitude of the gradient is a crucial component to success, we first count the ratio of the zero gradient samples the failed attack samples. The 6 shows the ratio of success samples (light green bar), failure samples with non-zero gradient (blue bar) and zero gradient (red bar) in all samples. We note that the failed attack samples with non-zero gradients maintain the index of the largest logit as the true class after attack. To analyze the reason of failure, we examine the change the logits under the adversarial attack. This change is shown in Figure 6) (b). To clarify the difference of the change of the logits between Network A and B, we select the examples successful attack on Network but failed attack on Network B. Network B, the logits move almost parallel direction, which causes the predicted label to be maintained as the true class.
To explain the parallel change of the logit vector, hypothesize that the DBs corresponding to each class node have similar configuration in the input space. However, it is intractable to measure the similarity between DBs in the entire network due the highly non-linear structure and the high dimensional input space. To simplify our hypothesis, we only measure the cosine similarity between the parameters which map the features on the penultimate layer logits (i.c., final layer). Figure presents that the similarity matrices for Networks A and B. When we compare the matrix between the two models the 300th epoch, we identify that Network B (low PRS ratio) has greater cosine similarity between each parameter in the final layer. We note that the cosine similarity between each parameter in the final layer can be considered as the degree of parallelism for the normal vectors in the linear classifier. We also confirm that the decrease of the PRS ratio is aligned with the increase of the similarity of parameters in the Figure (b). when we consider the graph in Figure 2 To verify the relationship between PRS ratio and the cosine similarity between the parameters in the final layer, we measure the PRS ratio and the cosine similarity between each parameter in all models. Figure & shows the experiment results grouped according the model structure. We identify that

---

Under review as a conference paper at ICLR 2022
Ridlo
simple
Logils
Logils 0)
Logit
10
NLA
NeA
3.
Ne It MEA
3.4
NkB:
Class index
Class inder
Class index
Success
Failure with gradient
Faiuure with gradient
Original logits
Logits alter attacis
Ratio lest samples
(b) Ilustrative examples for change of logits atlack failure cases
Figure 6: (a) Comparison of the ratio of the zero gradient in the failure attack for the test samples under the PGD attack Los with 0.0313 and 20-step size (Network A and B). (b) The illustra- tive examples of attacked samples on Network A and which is failed on B, and the corresponding logits before/after the attack. After the attack, the logits move on almost parallel direction with the original logits in Network B. More examples are provided in Appendix C.
the PRS ratio has an inverse correlation for the cosine similarity between each parameter in the final layer.
30och330 300
Epoch 1
Eooch
Epoch 50
Epoch
Epoch 200
30och330
(8) NetworkA
(b) Network R
Figure 7: (a) Cosine similarity matrix for the final layer on Network A. (b) Similarity matrix for epochs on Network As the epoch increases, the cosine similarity for each parameter increases.
CNN-6
VGG-16
ResNet-1 18
MNIST P-NNIST
Cosine Similartt
0.5
CIPAR-10
0.4
0.3
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.05
0.10
0.15
0.20
0.25
0.0
0.1
0.2
0.3
0.4
PRS Ratio
PRS Ratio
PRS
Ratio
Eupe:
Stope:
Stpe
Stope
Slase
Stoe:
Figure 8: Relationship between the PRS ratio and the cosine similarity in various models and datasets. The colored dots represent the independent models. The colored dashed lines indicate the trends for each the datasets.
PRS AND TEST DATASET
Regarding the model a mapping function from the input space to the feature space, handling unseen data known feature domain is significant with regard to the generalization performance of the model. Hence, if the majority of samples from the test dataset are assigned to the training PR. the model can be considered to learn the informative and general concept of feature mapping. For example, if the arbitrary test sample is mapped to the training PR, expect that similar decision will appear. However, it is non-trivial to guess which type of decision will appear when the test sample mapped out of the training PR.
7

---

Under review as a conference paper at ICLR 2022
FGSM
BIM
PGD-20
PGD-100
Exchaikan
Indeutioon gropp
1.0
Bultaim
1.0
Eoriadion group
RohustAmmrrg
grApp
group
Incheiion grous
0.0
0.0
0.4
0.4
0.4
0.4
0.2
0.3
0.2
0.7
0.00
0.04
5.06
0.00
0.
0.00 0.04
0.66
0.00
0.
Epsilon
0.0
5.00
0.01
0.06
0.20
0.0
0.07
5006
0.28
Figure 9: Test accuracy under adversarial attacks for inclusion/exclusion groups for CNN-6 on CIFAR-10 for five fixed random seeds. The blue/orange line indicates the exclusion/inclusion groups, respectivly.. The exclusion group is shown to be more vulnerable under adversarial attacks.
To investigate the differences between the test samples which are included/excluued in the training PR, we evaluate the test accuracy under adversarial attack for each group. For controlled compar- ison, we divide both the inclusion and exclusion groups with Ik correctly predicted test samples.
Figure 9 shows the RA under the FGSM, BIM with a 5-step size, and the PGD with 20 and 100 step size on Lox- Although the accuracy of each test group decreases as the epsilon becomes larger, we observe that the inclusion group is more robust against all types of attacks compared to the exclusion group. We also provide the result of other networks and datasets in Appendix
Figure 10 presents the PRS ratio and the inclusion ratio of the test samples for the training PR. We compute the inclusion ratio as the ratio of the test samples mapped to the training PR. In Figure TO, we identify that the PRS ratio and the inclusion ratio exist in inversely correlated relationship. As we previously verify that the included test samples show high robustness, we empirically confirm that the low PRS ratio is related to the robustness under adversarial attacks.
CNN-6
VGG-16
R
Incluiio Ratio
0.4
0.3
0.2
MNSS CIFAR-10 10
0.1
0.0
1.0
0.10
0.15
0.3 PRS Ratio
PRS Ratio
PRS Ratio
Skpe 4072
Supe 0079
AS3
4048
Stppe 4.79
.055
Skaee 4071
Peatinn 1038-111
Ponkon 1.700-200
Pintin 6000118
Prelon 4.80-43
Peaton 2:201-01
Penkon 20075440
Pontin 1.509-00
Peston 6.908-3
Figure 10: Relationship between the PRS ratio and the inclusion ratio for CNN-6, VGG-16 and ResNet-18 18 on the various datasets. The colored dashed lines indicate the trend for each dataset.
PRS AND TRAINED FEATURES
This section explores the trained features of the models with the different PRS ratios. First, we visualize the feature maps directly for each depth of layers. Figure I1 shows illustrative examples of feature maps. We identify that the model with the low PRS ratio learns more sparse features compared to the model with the high PRS ratio. As the sparse features are considered as an inde- pendent and informative representation (Ranzato et al., 2007 Lee et al.) 2007), if the PRS ratio can cause sparse feature representation in various cases, conclude that the PRS ratio is related to the informative features. To verify our hypothesis, we measure the trend between the PRS ratio and the average sparsity for each network. The average sparsity of the model is calculated by taking the average the ratio of zero-valued features over the training dataset for all layers.
Figure 12 shows the relationship between the PRS ratio and the average sparsity in all cases. We identify that CNN-6 and VGG-16 show an inversely correlated relationship throughout the dataset, but it is difficult find clear relationship for ResNet-18 18. We conjecture that (1) skip connection, or
8

---

Under review as a conference paper at ICLR 2022
Dpptl Laper
Depdial Lager
Figure 11: Visualization of feature maps for CNN-6 trained on CIFAR-10. (First row) Feature maps
for Network A. (Second row) Feature maps for Network B.
CNN-6
VGG-16
ResNet-18
MINST E-NNIST
0.9
CIBAR-10
Sparsty
0.8
@co
0.6
0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.05
0.10
0.15
0.20
0.25
0.0
0.1
0.2
0.3
0.4
0.5
PRS Ratio
PRS Ratio
PRS Ratio
Supa 4.40
Supee -023
Suyee 4.15
Skea -1.00
Super 3.24
Slype -0.00
Slyee 0.06
Skpx 0.08
Supe -003
Pizton 3877-88
Pratee 9108-11
Pretee 1.000-31
Pretun 2818800
Proton 3750-63
Piakie 92200-1122
Peake 4 01
Peztn 1.50500
Preton 24075433
Figure 12: Relationship between the PRS ratio and the average sparsity for CNN-6, VGG-16 and ResNet- 18 on the various datasets. The colored dashed lines indicate the trend for each dataset.
(2) batch normalization can cause this phenomenon. To investigate the global feature representations further, visualize the projected feature space using t-SNE (van der Maaten & Hinton (008)) for the training dataset. In Figure 6 we can identify that the samples in low PRS ratio cases are more densely located in each class, while the samples show widely spread pattern in each class in high PRS ratio cases.
Low PRS Ratio
High PRS Ralio
Low PRS Ratio
High PRS Ratio
Low PRS Ratio
High PRS Ratio
(a) MNIST
(b) FMNIST
(c) CIBAR 10
Figure 13: Visualization of the projected feature space with t-SNE for CNN-6 trained with various datasets. The colored code indicates the class. The samples in low PRS ratio cases are more densely located in each class compared the samples high PRS ratio.
7 CONCLUSION
In this work, we perform an empirical study to analyze the internal properties of DNNs, which affect the robustness under adversarial attacks. We propose the novel concept populated region set to derive the relationship between the internal properties of DNNs and robustness in a practical setting. From systematic experiments for the proposed concept, we suggest that the PRS ratio is related to the robustness of DNNs and provide empirical evidence of this relationship: (1) The network with the low PRS ratio shows high robustness against the gradient- based attack compared to the network with a high PRS ratio. In particular, the model with a low PRS ratio has higher degree of parallelism for the parameters in the final layer, which can support robustness. (2) The network with a low PRS ratio includes more test samples in the training PR. We empirically verify that this inclusion ratio is related to robustness from the observation that included test samples are more robust than excluded test samples. (3) The network with low PRS ratio learns sparse feature representation to solve tasks. Moreover its global feature representation is more intensive than the network with high PRS ratio.

---

Under review as a conference paper at ICLR 2022
REFERENCES
D Richard Baughman and Yih An Liu. Neural networks in bioprocessing and chemical engineering. Academic press, 2014.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Jianbo Chen, Michael Jordan, and Martin J Wainwright. Hopskinjumpattack: A query-efficient decision-ber based attack. In 2020 ieee symposium on security and privacy (sp), pp. 1277-1294.
IEEE, 2020.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. PMLR, 2015.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. arXiv preprint arXiv:1406.225 2572, 2014.
Alhussein Fawzi, Seyed-Mohsee cation regions of deep neural networks. arXiv preprint arXiv:1705.0953 2017.
Pascal Frossard, and Stefano Soatto. Classifi
Alhussein Fawzi, Seyed- Seyed-Mohsen Moosavi-Dezfoolli Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577. 2014.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint ar 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with strong adver- sary. arXiv preprint arXiv:1511.030:: 2015.
Yu Huang and Yue Chen. Autonomous driving with deep learning: A survey of state-of-art tech- nologies. arXiv preprint arXiv:2006.060% 2020.
Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 514-529, 2018.
Sandesh Kamath, Amit Despande, and KV Subrahmanyam. On adversarial robustness of small vs large batch training. 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
10

---

Under review as a conference paper at ICLR 2022
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area v2. Advances in neural information processing systems, 20: 880, 2007.
Na Lei, Zhongxuan Luo, Shing-Tung Yau, and David Xianfeng Gu. Geometric understanding of deep learning. arXiv preprint arXiv:1805. 10451, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 1966):233-1224 2018.
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In NIPS, 2014.
Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 9070-9078, 2019.
Mare Ranzato, Christopher Poultey, Sumit Chopra, Yann LeCun, et al. Efficient learning of sparse representations with an woergy-based model. Advances in neural information processing systems, 19:1 137, 2007.
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Neurocomputinng 307:195-20 204, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1312.66 2013.
Laurens van der Maaten and Geoffrey Hinton. chine Learning Research, 9:2579-2605. 2605, 2008. URL //tttp jml wrg.papers/v//
Visualizing data using t-SNE. Journal of Ma-
renatenmaattn html
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv:1708.07744 2017.
Huan Xiong. Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. In International Conference on Machine Learning, pp- 10514-10523. PMLR, 2020.
Yaoqing Yang. Rekha Khanna, Yaodong Yu, Amir Gholami, Kurt Keutzer, Joseph Gonzalez, Kannan Ramchandran, and Michael W. Mahoney. Boundary thickness and robustness in learning models. ArXiv, 00./2007.05080 2020.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney, Hessian-based analysis of large batch training and robustness to adversaries. arXiv preprint 10X 2018.
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 3009):2805-2824, 2019.
Ziyuan Zhong, Yuchi Tian, and Baishakhi Ray. Understanding local robustness of deep neural networks under natural variations. Fundamental Approaches to Software Engineering, 12649: 313, 2021.
11


Reviews:
[{"review_id": "ebyho0Am7Xh", "reviewer": "Reviewer_5u8a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe idea of populated regions is interesting, and should be further investigated. Especially, how test samples that fall into them are more robust could be a good direction to study in relation to designing more robust models.\nThe authors investigate PRS from many different directions and show how it is correlated to other important properties across 3 datasets and 3 model types.\n\nWeaknesses:\nThe paper has big problems with clarity. Two networks, one with a large PRS (network A) and one with a small PRS (network B) are used to explore how the size of the PRS is related to other properties, but neither in the main paper nor in the appendix is it explained how they differ from each other. Is it just a different random seed or are they trained differently? In figure 2 it looks like network B reaches a high train accuracy significantly faster than network A.\n\nIn figure 4, 8 and 10 we see scatter plots of lots of models trained but again it is not clear how they are different. The appendix lists clean accuracy for models in table 2,3 and 4 but it only shows 4 different batch sizes and 5 different random seeds. The scatter plots clearly have more than these 20 combinations per dataset.\nAdditionally, I don't understand how the confidence intervals for the accuracies in the appendix tables are calculated, since the accuracies are for a fixed random seed so each run should lead to the same result.\n\nThe paper never says what norm is used for the adversarial attacks. From the epsilon values used I assume it is l_2. Do the results hold for l_inf?\n\nI'm also not sure how useful the PRS metric is. The authors give the example that it is hard to choose between models with similar clean accuracies. But in that case I don't see the advantage of using the PRS ratio instead of directly using robust accuracy. Since there is no explanation on why some models have different PRS ratio it also doesn't give any hints at how one could deliberately train models to achieve a lower PRS ratio and therefore be more robust.\n\nAfter comparing FGSM and iterative attacks (both targeted and untargeted) in figure 3 they use FGSM (untargeted) for the rest of the paper because the difference between the networks is the biggest. I don't think this is a good choice since the comparison clearly shows it is the weakest attack. It would a lot more interesting to see if the results hold for even stronger attacks like PGD or auto-attack[1].\n\nThere are also a lot of grammatical errors in the paper which make it hard to read and understand. There are too many to list here so I would encourage the authors to have someone proofread the draft again.\n\nOther comments & questions:\n\nSince PRS size is correlated with robustness do models trained adversarially have very small PRS compared to when we use standard training?\n\nOn page 1: \"We further observe that the model with the small size of PRS has the relatively parallel parameters compared to the large one in the final layer.\" \nI could only understand what is meant here after reading the section at the end of the paper.\n\nFigure 2: You should make clear that the left and middle plot are only showing the PRS ratio of layer 8. \n\nFigure 6: Shouldn't the ratio of test samples with zero and non-zero gradient sum up to 1 for both networks? I don't understand what the bars show exactly. Additionally, instead of two images for Network B ist would be more interesting two show how a successful attack for Network A changes the \nlogits.\n\nFigure 13: I'm not sure what \"more intensive feature representation\" means here. I find it hard to make any qualitative claims from the plots. For CIFAR-10 you could even argue that the high PRS ratio shows better separation of the classes while they all melt together in the middle for the low PRS ratio model.\n\n[1]\"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks\"\nFrancesco Croce, Matthias Hein\nICML 2020", "summary_of_the_paper": "The paper proposes a new metric, the size of the populated region set (PRS), as an explanation for models with similar clean accuracies reaching very different accuracies under adversarial attacks. PRS is the set of decision regions that have training examples in them. After introducing and defining populated regions show that PRS in the penultimate layer is inversely correlated with robust accuracy. This is shown to hold across MNIST, F-MNIST and CIFAR10 and for 3 different model architectures (simple CNN, VGG and ResNet). They use two example networks with high and low PRS to show how PRS evolves during training and that the higher resistance to adversarial attacks might stem from the neurons in the last layer having similar decision boundaries in input space. \nTest samples that also fall into the PRS are shown to be more robust and models with small PRS have more test samples in them than models with large PRS. Additionally, it is shown that models with small PRS produce more sparse features. ", "summary_of_the_review": "While the concept is interesting the paper doesn't provide enough details about how the models were trained or about the adversarial attacks. I'm not convinced the newly introduced metric will be useful without a deeper evaluation, especially explaining why similar models differ drastically in PRS size. The paper is hard to read and understand as there are a lot of grammatical errors throughout.\n\nGiven all these factors I don't think this paper should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TGRQopjsWZd", "reviewer": "Reviewer_pYmY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nPros: \n \n* Characterizing decision boundaries is an interesting topic. I think there is a lot of interesting empirical work to be done in this area.\n\n* Figure 2 is interesting: the PRS ratio clearly decays to zero for network B. \n\n##########################################################################\n\nCons: \n\n* Why no PGD attacks in the robustness studies?\n\n* Relationship is pretty weak for CNN-6 in Figure 4. Any thoughts why?\n\n* In Section 6, the authors use the word \"intensive\" to describe t-SNE plots. What do the authors mean by this? For instance the authors write: \"In Figure 6, we can identify that the model with the low PRS ratio has more intensive feature representation for each class.\" Is there a quantitative measure for this?\n\n\n* Regression slopes are only visualized, not reported quantitatively. I would also like the see significance tests.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\n\n* \"Robustness score\", the area under the robustness curves, is okay to report, but to the best of my knowledge it is non-standard in the robustness community. Epsilon sweeps are included in the Supplementary.\n\nHow is it actually possible that you can compute your proposed measure? Why does the \"curse of dimensionality\" not apply?\n\n* What is the relationship of your work and those which analyze the linear regions of decision surfaces? E.g Montufar et al. (NeuIPS 2014)\nhttps://arxiv.org/abs/1402.1869\n\n\n##########################################################################\n\nPost rebuttal period: I thank the authors for their thorough response. Having their response and the other reviews, I still harbor some doubts about the validity and usefulness of the proposed PRS metric and its empirical evaluation. I have decided to keep my score marginally below acceptance.", "summary_of_the_paper": "This work empirically studies for deep networks the relationship between (1) model robustness and (2) the decision surface. A novel metric is proposed, the Populated Region Set (PRS) metric, essentially the number of regions in decision space which have at least one training sample. The authors claim the metric has a \"strong relationship\" to robustness, as measured by correlation, and present a number of experiments to support their claim.", "summary_of_the_review": "Interesting empirical work but needs improvement and clarification.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0KTI0sOGU_q", "reviewer": "Reviewer_Yvre", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and clear. The newly introduced geometry metric, the so-called Populated Region Set (PRS) ratio seems promising and intuitive. Although I am not knowledgeable of the related work on the interaction between decision boundaries and model's robustness, I  find the type of analysis/comparisons made (which I indicated in the summary) insightful. \n\nThe conclusions that are drawn related to the decision regions and based on the empirical evidence are consistent with what one may indeed intuit. As useful as I find its intellectual contribution, I am not fully convinced on the informativeness of the PRS ratio when it comes to more challenging settings. For instance, what happens when the number of classes is large? In the experiments, the datasets in consideration have only 10 classes each, which is limiting. As an immediate thought, I would expect that the PRS ratio is generally high in cases where the label space is large. What do the authors think about it?\n\n\n_Typos:_ \"We verify that the the size ...\", \"Traini accuracy\" in Figure 2 first column\n", "summary_of_the_paper": "This paper aims to understand the robustness of DNNs from the perspective of decision regions. Towards that, the authors introduce a new metric, the so-called Populated Region Set (PRS) whose ratio is later used to investigate the robustness of a selection of DNNs empirically. Based on the respective empirical evidence, the paper claims that the lower PRS ratio (roughly #decision regions with at least one training data point/size of training data) leads to enhanced robustness, better representation of test instances in the populated regions, and learns the sparse feature representation. The empirical evidence is collected using the models CNN, ResNet-18, and VGG-16 over datasets MNIST, F-MNIST, and CIFAR-10 and under perturbations with various $\\epsilon$ and untargeted/targeted attacks.", "summary_of_the_review": "The analysis of the interaction between the robustness of ML models and the Populated Region Set (PRS) ratio seems promising in understanding the geometrical interpretation of robustness. Although having an extension to the current experiments would make the use of such a metric more convincing, I am slightly leaning towards acceptance.\n\n\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a concept called Populated Region Set (PRS) as a measure of robustness of deep neural networks (DNNs). The paper provides a suit of empirical results to demonstrate the strong correlation between the PRS ratio and adversarial robustness of DNNs. The authors made great efforts on addressing reviewers' concern, which is greatly appreciated. However, the theory of the work is a bit thin, and it leaves a number of outstanding issues unaddressed. For example, it is not clear the practical advantage of calculating PRS over the direct measure of robust accuracy. What new and better computational procedure can be constructed based on PRS? We encourage the authors keep improving their work for future submission.
Decision: Reject