Empirical Study of the Decision Region and Robustness in Deep Neural Networks

1 INTRODUCTION . With the steep improvement of the performance of Deep Neural Networks ( DNNs ) , their applications are expanding to the real world , such as autonomous driving and healthcare ( LeCun et al. , 2015 ; Miotto et al. , 2018 ; Huang & Chen , 2020 ) . For real world application , it may be necessary to choose the best model among the candidates . Traditionally , the generalization performance which measures the objective score on the test dataset excluded in the training phase , is used to evaluate the models ( Bishop , 2006 ) . However , it is non-trivial to evaluate DNNs based on this single metric . For example , if two networks with the same structure have the similar test accuracy , it is ambiguous which is better . Robustness against adversarial attacks , measure of the vulnerability , can be an alternative to evaluate DNNs ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Gu & Rigazio , 2014 ; Huang et al. , 2015 ; Jakubovitz & Giryes , 2018 ; Yuan et al. , 2019 ; Zhong et al. , 2021 ) . Adversarial attacks aim to induce model misprediction by perturbing the input with small magnitude . Most previous works were focused on the way to find adversarial samples by utilizing the model properties such as gradients with respect to the loss function . Given that the adversarial attack seeks to find the perturbation path on the model prediction surface over the input space , robustness can be expressed in terms of the geometry of the model . However , few studies have been performed to interpret the robustness with the concept of the geometric properties of DNNs . From a geometric viewpoint , the internal properties of DNNs are represented by the boundaries and the regions ( Baughman & Liu , 2014 ) . It is shown that the DNNs with piece-wise linear activation layers are composed of many linear regions , and the maximal number of these regions is mathematically related to the expressivity of DNNs ( Montúfar et al. , 2014 ; Xiong et al. , 2020 ) . As these approaches only provide the upper bound for the expressivity with the same structured model , it does not explain how much information the model actually expresses . In this work , we investigate the relationship between the internal properties of DNNs and the robustness . In particular , our approach analyzes the internal characteristics from the perspective of the decision boundary ( DB ) and the decision region ( DR ) , which are basic components of DNNs ( Fawzi et al. , 2017 ) . To avoid insensitivity of the maximal number of linear regions in the same structure assumption , we propose the novel concept of the Populated Region Set ( PRS ) , which is a set of DRs containing at least one sample included in the training dataset . Since the PRS can be considered as the feasible complexity of the model , we hypothesize that the size of PRS is related to the robustness of network . To validate our hypothesis , we perform systematic experiments with various structures of DNNs and datasets . Our observations are summarized as follows : • The models with the same structure can have different size of PRS , although they have similar generalization performance . In experiments , we observe that this difference leads to different robustness of the network . • We empirically show that the size of the PRS is related to the robustness against the adversarial attack . The model with a small size of the PRS tends to show higher robustness compared to that with a large size ( in Section 4.1 ) . We further observe that when the model achieves a low PRS ratio , the linear classifier that maps the penultimate features to the logits has high cosine similarity between parameters corresponding to each class ( in Section 4.2 ) . • We verify that the size of intersection of the PRS from the training/test dataset is related to the robustness of model . The model with a high PRS inclusion ratio of test samples has higher robustness than that with a low PRS inclusion ratio ( in Section 5 ) . • We identify that the model with a small size of the PRS learns the sparse feature representation . In quantification , we observe the inversely correlated relationship between the size of the PRS and sparsity of feature representation ( in Section 6 ) . 2 RELATED WORK . Adversarial robustness For the real-world application of DNNs , the adversarial attack , which reveals the vulnerability of DNNs ( Goodfellow et al. , 2014 ) , is mainly used to validate the reliability of the trained network . As an early stage for adversarial attacks , the fast gradient sign method ( FGSM ) ( Goodfellow et al. , 2014 ) based on the gradient with respect to the loss function and the multi-step iterative method ( Kurakin et al. , 2016 ) are proposed to create adversarial examples to change the model prediction with a small perturbation . Recently , many studies on effective attacks in various settings ( e.g. , white-box or black-box ) have been performed to understand the undesirable decision of the networks ( Shaham et al. , 2018 ; Madry et al. , 2018 ; Chen et al. , 2020 ) . In terms of factors affecting robustness , Yao et al . ( 2018 ) provide evidence to argue that training with a large batch size can degrade the robustness of the model against the adversarial attack from the perspective of the Hessian spectrum . In contrast , Kamath et al . ( 2019 ) propose that the model with a constant ratio between the learning rate and batch size does not degrade the model robustness even with a large batch size as it converges to the flatter minima . Geometric Analysis inside Deep Neural Networks With increasing interest in the expressive power of DNNs , there have been several attempts to analyze DNNs from a geometric perspective ( Dauphin et al. , 2014 ; Choromanska et al. , 2015 ) . In these studies , the characteristics of the decision boundary or regions formulated by the DNNs are mainly discussed . Montúfar et al . ( 2014 ) show that the cascade of the linear layer and the nonlinear activation organizes the numerous piece-wise linear regions . They show that the complexity of the decision boundary is related to the maximal number of these linear regions , which is determined by the depth and the width of the model . Xiong et al . ( 2020 ) extend the notion of the linear region to the convolutional layers and show the better geometric efficiency of the convolutional layers . Fawzi et al . ( 2018 ) reveal that classification regions in DNNs are topologically connected and the decision boundary of natural images is flat in most directions . It has also been shown that the manifolds learned by DNNs and the distributions over them are highly related to the representation capability of a network ( Lei et al. , 2018 ) . While these studies highlight the benefits of increasing expressivity of DNNs as the number of regions increases , interpreting the vulnerability of DNNs with the geometry is another important topic . Yang et al . ( 2020 ) show that a model with thick decision boundaries induces robustness . Moosavi-Dezfooli et al . ( 2019 ) show that a decision boundary with a small curvature acquires the high robustness of the model . These approaches focus on the decision boundaries , while this paper suggests to focus on the decision regions , which are composed by the surrounding decision boundaries . 3 INTERNAL PROPERTY OF DNNS . This section describes the internal properties of DNNs from the perspective of decision boundaries ( DBs ) and regions ( DRs ) . The DBs of the DNN classifier is mainly defined as the borderline between DRs for classification , where the prediction probability of class i and the neighboring class j are the same ( Fawzi et al. , 2018 ) . To expand the notion of DBs and DRs to the internal feature-level , we redefine the DBs in the classifier that generalizes the existing definition of DBs . We then propose the novel concept of the Populated Region ( PR ) that describes the specific DRs used from the network for training samples . PR is used to analyze the relationship between the trained parameters and the characteristics of networks . 3.1 DECISION BOUNDARY AND REGION . Let the classifier withL number of layers be F ( x ) = fL ( σ ( fL−1σ ( · · ·σ ( f1 ( x ) ) ) ) = fL:1 ( x ) , where x is the sample in the input space X ⊂ RDx and σ ( · ) denotes the non-linear activation function1 . For the l-th layer , fl ( · ) denotes the linear operation and f il:1 ( · ) denotes the value of the i-th element of the feature vector fl:1 ( x ) ∈ RDl . We define the DB for the i-th neuron of the l-th layer . Definition 1 ( Decision Boundary ( DB ) ) The i-th decision boundary at the l-th layer is defined as Bil = { x|f il:1 ( x ) = 0 , ∀x ∈ X } . We note that the internal DBBil ( l < L ) divides the input spaceX based on the hidden representation of the l-th layer ( i.e. , existence of feature and the amount of feature activation ) . There are a total of Dl boundaries and the configuration of the DBs are arranged by the training . As input samples in the same classification region are considered to belong to the same class , the input samples placed on the same side of the internal DB Bil share the similar feature representation . In this sense , we define the internal DR , which is surrounded by internal DBs . Definition 2 ( Decision Region ( DR ) ) Let Vl ∈ { −1 , +1 } Dl be the indicator vector to choose positive or negative side of decision boundaries of the l-th layer . Then the decision regionDRVl , which shares the sign of feature representation , is defined as DRVl = { x|sign ( fl:1 ( x ) ) = Vl , ∀x ∈ X } . 1Although there are various activation functions , we only consider ReLU activation for this paper . Figure 1 presents the internal properties for two networks trained on CIFAR-10 with similar test accuracy . The right column in Figure 1 depicts the internal DBs and DRs in the network with a high/low PRS ratio ( top and bottom ) . We randomly select two test images ( blue and green box ) and generate adversarial images for blue box ( orange and purple box ) in each network , respectively . We make a hyperplane with these images to visualize the DBs and DRs in the 2D space . We identify that the configuration of DBs and DRs appears to be different , although the two networks have the same structure and similar test accuracy . 3.2 POPULATED REGION SET . It is well-studied that the number of DRs is related to the representation power of DNNs ( Montúfar et al. , 2014 ; Xiong et al. , 2020 ) . In particular , the expressivity of DNNs with partial linear activation function is quantified by the maximal number of the linear regions and this number is related to the width and depth of the structure . We believe that although the maximal number can be one measure of expressity , the trained DNNs with finite training data2 can not handle the entire regions to solve the task . To only consider DRs that the network uses in the training process , we devise the train-related regions where training samples are populated more frequently . We define the Populated Region Set ( PRS ) , which is a set of DRs containing at least one sample included in the training dataset . PRS will be used to analyze the relationship between the geometrical property and the robustness of DNNs in a practical aspect . Definition 3 ( Populated Region Set ( PRS ) ) From the set of every DRs of the model f and given the dataset X , the Populated Region Set is defined as PRS ( X , f , l ) = { DRVl |x ∈ DRVl ∃ x ∈ X , ∀Vl ∈ { −1 , 1 } Dl } . We can then define a Populated Region as a union of decision regions in PRS as PR ( X , f , l ) = ∪DR∈PRS ( X , f , l ) DR. We note that the size of the PRS is bounded to the size of given dataset X . When |PRS ( X , f , l ) | = |X| , each sample in training dataset is assigned to each distinct DR in the l-th layer . To compare the PRS of networks , we define the PRS ratio , |PRS ( f , X , l ) ||X | , which measures the ratio between the size of the PRS and the given dataset . Figure 2 presents a comparison between two equivalent neural networks ( A and B ) with six convolution blocks ( CNN-6 ) trained on CIFAR-10 varying only the batch size ( 2048/128 ) . Figure 2 ( a ) presents the PRS ratio for the depth of layers in each model at the 300th epoch . We observe that only the penultimate layer ( l = 8 ) shows a different PRS ratio . Figure 2 ( b ) shows that the two networks have largely different PRS ratios with similar training/test accuracy . From the above observation and the fact that the penultimate layers are widely used as feature extraction , we only consider the PRS ratio on the penultimate layer in the remainder of the paper . Experimental setups For the systematic experiments , we select three different structures of DNNs to analyze : ( 1 ) a convolutional neural network with six convolution blocks ( CNN-6 ) ; ( 2 ) VGG-16 2In general , the number of training data is smaller than the maximal number of the linear region . ( Simonyan & Zisserman , 2014 ) ; and ( 3 ) ResNet-18 ( He et al. , 2016 ) . We train3 basic models with fixed five random seeds and four batch sizes ( 64 , 128 , 512 and 2048 ) over three datasets : MNIST ( LeCun & Cortes , 2010 ) , F-MNIST ( Xiao et al. , 2017 ) , and CIFAR-10 ( Krizhevsky et al. , 2009 ) . For the extensive analysis on the correlation between the PRS ratio and properties of network , we extract candidates from each basic model with the grid of epochs . Then we apply the test accuracy threshold to guarantee the sufficient performance . Finally , we obtain 947 models for analysis . The details for the network architecture and the selection procedure are described in Appendix A-C .

Summary: The paper proposes a new metric, the size of the populated region set (PRS), as an explanation for models with similar clean accuracies reaching very different accuracies under adversarial attacks. PRS is the set of decision regions that have training examples in them. After introducing and defining populated regions show that PRS in the penultimate layer is inversely correlated with robust accuracy. This is shown to hold across MNIST, F-MNIST and CIFAR10 and for 3 different model architectures (simple CNN, VGG and ResNet). They use two example networks with high and low PRS to show how PRS evolves during training and that the higher resistance to adversarial attacks might stem from the neurons in the last layer having similar decision boundaries in input space.  Test samples that also fall into the PRS are shown to be more robust and models with small PRS have more test samples in them than models with large PRS. Additionally, it is shown that models with small PRS produce more sparse features.