Rethinking the Pruning Criteria for Convolutional Neural Network
Authors: Zhongzhan Huang
Xinjiang Wang
Ping Luo
Venue:  (2021)
Keywords: Pruning Criteria
Similarity
Convolution
Weight Distribution

Abstract: Channel pruning is a popular technique for compressing convolutional neural networks (CNNs), and various pruning criteria have been proposed to remove the redundant filters of CNNs. From our comprehensive experiments, we find some blind spots on pruning criteria: (1) Similarity: There are some strong similarities among several primary pruning criteria that are widely cited and compared. According to these criteria, the ranks of filters’ importance in a convolutional layer are almost the same, resulting in similar pruned structures. (2) Applicability: For a large network (each convolutional layer has a large number of filters), some criteria can not distinguish the network redundancy well from their measured filters' importance. In this paper, we theoretically validate these two findings with our assumption that the well-trained convolutional filters in each layer approximately follow a Gaussian-alike distribution. This assumption is verified through systematic and extensive statistical tests. 

Under review as a conference paper at ICLR 2021
RETHINKING THE PRUNING CRITERIA FOR CONVOLU- TIONAL NEURAL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Channel pruning is a popular technique for compressing convolutional neural net- works (CNNs), and various pruning criteria have been proposed to remove the redundant filters of CNNs. From our comprehensive experiments, we found some blind spots pruning criteria: (1) Similarity: There are some strong similarities among several primary pruning criteria that are widely cited and compared. Ac- cording to these criteria, the ranks of filters' importance in convolutional layer are almost the same, resulting in similar pruned structures. (2) Applicability: For large network (each convolutional layer has a large number of filters), some cri- teria can not distinguish the network redundancy well from their measured filters' importance. this paper, we theoretically validate these two findings with our assumption that the well-trained convolutional filters in each layer approximately follow a Gaussian-alike distribution. This assumption verified through system- atic and extensive statistical tests.
INTRODUCTION
Pruning (LeCun et al. 1990 Hassibi & Stork 1993 Han et al. 2015 He et al. 2019) trained neural network is commonly seen in network compression. In particular, for neural networks with convolutional filters, channel pruning refers to the pruning of the filters the convolutional layers. There are several critical factors for channel pruning. Procedures. One-shot method (Li et al., 2016 Train a network from scratch; Use a certain criterion to calculate filters' importance, and prune the filters which have small importance; After additional training. the pruned network can recover its accuracy to some extent. Iterative method (He et al., 2018 Frankle & Carbin 2019): Unlike One-shot methods, they prune and fine-tune network alternately. Criteria. The filters' importance can definded by a given criterion. From different ideas, many types of pruning criteria have been proposed, such borm-based (Li al. 2016), Activation-basee (Hu et al.)
2016 Luo
& Wu 2017), Importance- based (Molchanov et al.) 2016 2019a), BN-based (Liu et al.
2017b) and
on. Strategy. Layer-wise pruning: In each layer, small importance measured by a given criterion. Global pruning: Different from layer-wise pruning, global pruning sort the filters from all the layers through their importance and prune them.
sort and prune the filters, which have
The pruned filters' index ordered by the filters' importance from given pruning criteria, taking VGG16 (3rd Conv) and ResNet18 (12th Conv) as examples. The pruned filters' index (the ranks of filters' importance) are almost the same from different pruning criterion and it will lead to the similar pruned structures.
Table 1:
Criteria
Model
Pruned Filters' Index (Top 8)
Model
Pruned Filters' Index (Top
&
ResNet18
[111, 212, 33, 61, 68, 152, 171, 45]
VGG16
[102, 28, 9, 88, 66, 109, 86, 45] [102, 28, 88, 109, 66, 45]
ResNet18
[1,1, 61. 171, 42, 243, 129]
VGG16
GM
ResNet18
[111, 61, 68, 171, 421
VGG16
[102, 28, 9, 88, 109, 66, 45, 86]
Fermat
ResNet18
[111, 33, 61, 45, 171, 42, 68]
VGG16
[102, 28, 88, 9, 109, 66, 45, 86]
As one of the simplest and most effective channel pruning criteria, é pruning (Li et al. 2016 is widely used in the industry. The core idea of this criterion is to sort the & norm of filters in one layer and then prune the filters, which have small E norm. Similarly, there is &g pruning (Frankle & Carbin 2219 He al.) 2018). Through the study of the distribution of norm, He et al. (2019) demon- strates that these criteria should satisfy two conditions: (1) the variance of the norm of the filters

---

Under review as a conference paper at ICLR 2021
not be too small; (2) the minimum norm of the filters should be small enough. Since these two ditions do not always hold, a new criterion considering the relative importance of the filters (f1 and of algorithm which uses absolute importance of filters) is proposed (He) et a 2019). Since this criterion uses the Fermat point (i.e., geometric median (Cohen et al. 2016)). call this method Fermat. However, due to the high calculation cost of Fermat point, He et al.
l2 norm can be seen
2011 relaxed it and then got another criterion GM method. Let Fly E R Ng represents the j tth filter of the jth convolutional layer, where N, is the number of input channels for jth layer and & denotes the kernel size of the convolutional filter. In 7th layer, there are N filters. The details of these criteria are shown in Table 2 F denotes the Fermat point of Fly in Euclidean space. These four pruning criteria are called Norm-based pruning in this paper as they include norm in their design.
In previous works (Luo et al., 2017 Han et al. 2013 Ding et al. 2019 Dong et al. 2017 Renda et al. 2020), including the criteria mentioned above, they usually fo- cused on (a) How much the model was com- pressed; (b) How much performance was
Table 2: Norm-based pruning criteria.
Criterion
Details of importance
81 Li al. 2016)
Fig la Feg
82 FFannlle Carbin 2019)
Fermat (He el al. 2019)
[F
Feyll2
restored; (c) The inference efficiency of the pruned network and (d) The cost of finding the pruned network. However, there is little work to discuss two blind spots about the pruning criteria:
GM (He et al. 2019
2k
N
|Pa
Fegl22
(1) Similarity: What are the actual differences among these previous pruning criteria? Using VGG16 and 88 ImageNet, we show the ranks of filters' importance under different criteria in Table I It is easy to find that they have almost the same sequence, leading to similar pruned structures. In this situation, the criteria used absolute importance of filters ( (1)2)) and the criteria used relative importance of filters (Fermat, GM) may not be significantly different.
(2) Applicability: What is the Applicability of these pruning criteria to prune the CNNs? There is a toy example w.r.t. & criterion. If the & norm (regarded as importance) of the filters in one layer are 0.9, 0.8, 0.4 and 0.01, according to moaller-norm--ie iifssmiifrrm assumption (Ye et al. 2018), it's easy to know that we should prune the last filter. However, if the norm are close, like 0.91, 0.92, 0.93, 0.92, it hard to know which filter should be pruned even though the first one the smallest. As shown in Fig. | taking Wide ResNet28-11 10 (WRN) example, this is real and existing problem for pruning a network with the lg criterion.
Some similar research methods and opinions on pruning criteria are mentioned in He et al. (2019) and Molchanov et al. (2019a). We make further analysis and observation of them in this paper. Since the criteria in Table are widely
25
bhock1-layer2-comm
meen 0,2462 var: 0.0008 var/meen: 3.0034
afther 160 dim:1440
Frequency
cited and compared (Liu et al. (0220b] [i et al.
15-
Block2 Inyar2-canvi 3012 0.003
2020b He et al.] 2020) Liu al.) [020aa] Li et al. 2020a). therefore it's important to study
var/mean: D0088 Hiterr 820 cim29990
10.
them for the above two issues. In order to rigor-
ously study them: First, we come up with an as- sumption about the distribution the parame- ters of the convolutional filters, called Convolu- tion Weight Distribution Assumption (CWDA), with systematic and comprehensive statistical tests (Appendix PP in Section [ Next, in Sec- tion 3and Section 4 theoretically verify the problem about Similarity and Applicability for some Norm-based criteria in layer-wise pruning- Last but not least, in Section 5 we discuss more issues including: (I) the conditions for CWDA to be satisfied, (2) the Similarity and Applicability when we use other types of pruning criteria, (3) and another pruning strategy, the global pruning.
meam: 0.5111 var D0015
var/maan: 0.0029
Hilter: 640 cim:5780
0.25
0.50
0.75
1.00
1.25
1.75
2.00
Value of la norm
Figure 1: The distribution of la norm (WRN)
Contribution. (1) We propose and verify an assumption called CWDA, which reveals that the well-trained convolutional filters approximately follow Gaussian-alis distribution.
2

---

Under review as a conference paper at ICLR 2021
ResNot18
2003
100
N₂
A:
(b)
(c)
(d)
Figure 2: (a-d) Visualization of correlation matrix FF", More experiments on the different layers of other networks can be found in Appendix Q The structure of Fij.
(2) We analyze the Applicability problem and Similarity from different types of pruning criteria. Under CWDA, we strietly validate these issues of Norm-based criteria in layer-wise pruning.
(3) Under CWDA, some Norm-based criteria using the global pruming strategy have the problem about the inconsistent magnitude of importance between different layers, which explains the phe- nomenon that they sometimes cut off the network. The estimations using CWDA almost coincides to the actual results obtained from the real network, which also demonstrates the effectiveness the CWDA.
WEIGHT DNSTRIBUTION-ASSUMPE
In this section, study the Similarity and the Applicability problem among the pruning criteria shown in Table 2 we propose and verify an assumption about the distribution of the parameters in convolutional filters.
(Convolution Weight Distribution Assumption) Let Fiy E &NNkkk be the jth well-trained filter of the hh convolutional layer. In 7th layer, Fig.j = =1,2 2.., No41 are i.d and follow a Gaussian- alike distribution:
Pg N(O, c². Ebtock),
where e is a constant and Ebtock = diag(Ks [1, K2.... KN.) ) is block diagonal matrix. The values of the off-diagonal elements are close 0 and Ki E R x 1,2 N₁
This assumption is based on the observation shown in the Fig. Let F RRN, xxxk)x denote all the parameters in 7th layer and we use the correlation matrix FFT estimate the shape of Ebtock- example,
Taking the last convolutional layers of ResNet18 and VGG16 trained ImageNet find that FFT is a block diagonal matrix. Specifically, each block is a 12 x k22 matrix and the off- diagonal elements
close O. For jth filter F.j E I N. x in 7th layer as shown in Fig. F this phenomenon reveals that the parameters in the same channel of Fly tend to be linearly correlated, and the parameters of any two different channels (yellow and green channels in Fig. ee)) in Fe only have a low linear correlation. Since the kernel size k is a small constant (like or 3) and N; > 1, the length 2 of each block is much smaller than the length k22 N; of the covariance matrix in most convolutional layers. Therefore, the block diagonal matrix Σ can be regarded as diagonal matrix, as shown in Fig. (b) and (d). For the convenience of analysis, we relax the assumption, i.e., Fi approximately follows Gaussian-alike distributior
Fiy N(O, - INxxkxk),
where IN x identity matrix. The statistical tests in Section 2.1 and the estimation by CWDA in Fig. 10 the reasonability of this relaxation. In the remaining sections, we use Eq represent CWDA unless otherwise specified. In Fig. taking VGG16 and ResNet18 on ImageNet dataset as examples, we visualize the CWDA through the distribution of the convolutional filters.
In Scction 5 make further discussion and analysis on the conditions for CWDA to he satisfied.
3

---

Under review as a conference paper at ICLR 2021
Frequency
Freqquene
Frequency
Freqquency
sie Value weights
Value weights
Value of weights
Value weights (d) VGGI6 (4th Conv)
(a) R8 (11th Conv)
(b) ResNet18 (12th Conv)
(c)VGG16 (3 Conv)
Figure 3: Visualization of the distribution of convolutional filters. The parameters of convolutional
filter approximately follow caussian-alike distribution.
2.1 STATISTICAL TEST
In fact, CWDA is not easy to be verified. For example, for ResNet164 (On Cifar100), the number of filters in the first stage is only 16, which is too small be used estimate the statisties accurately. More other objective reasons are shown in Section 5.1 As these problems, we consider verify- ing three necessary conditions CWDA: (I) Gaussian (i.e., to verify whether Fl approximately follows a Gaussian-alike distribution); (2) Standard Deviation (i.e., to verify whether the standard deviation of each filter in any layers close constant c); (3) Mean (ie.. to verify whether the mean of Fij is close to 0).
illustrate that CWDA holds universally, we consider a variety of factors, such as network structure, optimizer, regularization initialization, dataset, training strategy, and other tasks computer vision (e.g,, semantic segmentation, detection, image matting and so on). The details of these statistical tests are shown in Appendix P
In Table
Table 3: The experiments for having the comprehensive statistical tests on CWDA.
NETWORK STRUCTURE PT
OPTIMIZER P22
REGULARIZATION
ResNet (He al. 60
SGD (Sutskever et al. 2013
L. norm
VGG (Simonyan & Ziserrann 2014
ASGI
[Polyak & Juchsky.
1992
1.2 norm
AlexNet Kiizhevsky 2014
Adam
Kingnn & Ba. 2014
RReLa eu al. 2015
DenseNet (Huaang
Adagrad Duchi al.
Dropact [Lanng
PreResNet (He
201
Adamax Kingma & 2014
Autoaug
Cubuk
2019
WRN IZagoryyko & Komodakis 2016
Adadelta
Zeiler 2012
Cutout Vees & Tay 2017
ResNext (Xie el ul. 2017]
Cutmix
Yun et al. 2019
ATTENTION MECHANISM P4
INIIIALIZATION P.5
DATASET P6
SENet (Hu et al.
2018
Kaiming-normal HH al.
2015
CIFAR10
Krizhevsky et al.
2009
DIANet (Huang et 2019
Kaiming-uniforn (He ct al.
2015
CIFAR100
Krihhessky et a..
2009
SRMNet LLe et al. 2019
Xavicr-normal (Glorot Bengio
2010
ImageNer (Russakvvsky et 2015
CBAM Woo
Ravier-uniform
Hengio
MNIS
LSCunn a., 1998
IEBN Liang 2019
Orthogonal (Saxc al. 2013
SGENer (Lr al.
2019
SEGMENTATION P.7
DETECTION P7
BATCH NORMALIZATION P.
SegNet (Badrinarayannu al., 2017
Faster RCNN Ren
2015
VGG
PSPNet Zhao et al. 2017)
VGG-bn
PYTORCH PRETRAIN P.
MATTING P7
LEARNING RATE P.
8/3/50 VGG11/16/19
Deep image matting
Xu et al. 2017
Schedule150-225 Schedule82-166
AlphaGAN matting
Lutz al. 2018
STYLE TRANSFER P.7
GAN P7
Schedule60-122
Fast neural style Johnson al. 2016
DCGAN Rudfond al. 2015
Cos-Ir (Loshchilov & Hutter, 2016
SIMILARITY
In this section, we further verify the observation that the pruning criteria in Table highly similar from two perspectives. From an experimental point of view, we use more experiments about image classification to investigate the similarities. From a theoretical perspective, we rigorously show the similarities of the criteria in Table in layer-wise pruning under CWDA.
Thhe statistical tests about the situation with without weight decay can he found in Appendix O
4

---

Under review as a conference paper at ICLR 2021
ResNet56 on CIFAR10
ResNet56 on CIFAR100
E0
60
Test accuracy
Test Test accuracy
stuned
pruned
20
20-
tuned
Ratio
Ratio
Figure 4: Test accuracy of the ResNet56 on CIFARIO/100 while using different pruning ratios. L1 pruned and LI tuned denote the test accuracy of the ResNet56 after & pruning and fine-tuning, respectively. If pruning ratio is equal to 0.5, we prune 50% filters in all layers.
Empirical Analysis. (1) Fig. 4 we show the accuracy of the ResNet56 after pruning and fine-tuning under using different pruning ratios and datasets. The test accuracy curves of different pruning criteria different stages very close under different pruning ratios. This phenomenon implies that those pruned networks using different pruning criteria are very similar, and there are strong similarities among these pruning criteria. The experiments about other commonly used prun- ing ratio can be found in Appendix N (2) In Fig. [5] we show the Spearman's 's rank correlation coefficient between different pruning criteria. The Sp in most convolutional layers are more than 0.9, which means the network structures in transition layer (i.e., the layer where the dimensions of the filter change, like the layer between stage I and stage 2 of ResNet164. The number of dimensions in stage and stage 2 are 16 and 32 respectively.) are relatively small. is interesting but will not greatly impact the structural similarity of the whole pruned network. The reason for this phenomenon may be that the layers in these areas are sensitive. The similar observations are shown in Fig. 2 in Ding et al. (2019), and Fig. 6 and Fig. 10 in Li al. (2016).
almost the same after pruning. Note that the Sp
VGGI6-CIFARIO
ImageNlmageNet
Spcarnen's correlation
corcelation
Speurman's correlation Spearman's comelution
UGG 6%%
LIGH
13.04
Index layer
Index of layer
Index layer
ResNet56-CIFAT
ResNet56- CIFARI00
ResNet34 I44geeageeNe
Speormants correiation
Spearman's correlation
12:01 60%
12:01
60%
Index layer
Index of layer
Index of layer
Figure 5: Spearman's rank correlation coefficient (Sp) between different pruning criteria on several networks and datasets (more experiments can be found in Appendix R).
Theoretical Analysis. After the verification by experiments, the similarities via using layer-wise pruning among the criteria in Table 2 are proved theoretically this section. Let C1 and C2 be two pruning criteria to calculate the importance for all convolutional filters of one layer. If they
'Sp is a nonparametric measurement of ranking correlation, and it assesses how well the relationship be- tween two variables can he described using monotonic function, te., filters ranking sequence in the same layer under criteria in this paper.
5

---

Under review as a conference paper at ICLR 2021
can produce the similar ranks of importance, we define C1 and C2 approximatety monotonic to each other and use C1 C2 to represent this relationship. In Section we use the Sp to describe this relationship but it's hard to be analyzed theoretically. Therefore, we consider about stronger condition. Let X = (æ1, T2...., vx)) and Y = (y,, 1/2,..., yk) be two given sequences. we first normalize their magnitude, i.e, let X X X/E(X) and Y Y YYE This operation does not change the ranking sequence of the elements of X and Y, because EE(X) and IE(Y) are constants, i... X Y + X = Y. After that, if both Var(X/Y /) and Var(Y (ar(Y//)) /X) are small enough, then the Sp between X and Y is close to 1, where X/Y (4i/ Tk) The reason is that in these situations, the ratio X/Y and Y/X X will be close to two constants a, b. Note that, for any S i S k, f = a Di and 0i = b Fi. Then, ab = 1 and a, b o O. Therefore, there exists an approximately monotonic mapping from yi E (linear function) and it makes the Sp between X and Y close to 1.
Theorem 1. Let X (Fermat, GM), we have
N(O, e²
I,). and (C1, C2) is one of (l2, E2). (6, Fermat) or
(2(()) Varx (i()))
(((())
max
Varx
(2(x))
B(n),
(3)
where E (X) denotes /B) (X)) and (X) denotes Cg(X))B(Cx(X)), B(n) denotes the upper bound of left-hand side and when is large enough, B(n) 0.
Proof. (See Appendix G
For 7th convolutional layer of neural network, since Fij. = 11.2...Ni+4 meet CWDA and the
dimensions of Fiy are generally large, we can obtain & _ lz, lo = Fermat and Fermat GM
according to Theorem Therefore, we have e = l2 = Fermat = GM, which verifies the strong similarities among the criteria shown in Table 2
4 APPLICABILITY
In this section, we analyze the Applicability problem of the Norm-based criteria when we use these criteria to prune a large network (each convolutional layer has a large number of filters). In Fig. [ can find that l can not distinguish the redundancy of Wide ResNet28-10 (regarded as large network) well from their measured filters' importance, because their importance are very close (the distribution looks sharp). This problem is related the variance of importance. HH al. (2019) argue that a small norm deviation (the values of variance of importance are small) makes it difficult to find an appropriate threshold to select filters to prune. However, even if the values of the variance are large, it may still have the Applicability problems. This is because the magnitude of these importance may be much greater than the values of the variance, where we can use the mean of importance represent their magnitude. We believe that the following two situations may cause Applicability problem: for the filters F, in 7th convolutional layer.
(1) If the mean B(F:) 0 and the variance Var(F:) is close to O; (2) If B(F)) * 0 and Var(F:)/B(R) is close to
For network with large number of convolutional filters, it's easy to know that the dimensions of the filters are also large enough. For lg pruning, according to CWDA (the proof in Appendix [ we can obtain that the mean of (F.) is /2o, I //T deviation and dimension of the parameters in thh layer, respectively. And Var(F.)/B(E)) - 0 when k, is large enough. From these reasons, the importance measured by & norm tends be identical, ie., it's hard to distinguish the network redundancy well in this situation. Moreover, from the proof in Appendix H we know that the Fermat point of F; and the origin 0 approximately coincide. According to Table 2 ||F - Fill2 & ||0 Filla = || Therefore. the importance of Fermat and & are almost equivalent when CWDA holds. Hence, a similar conclusion can be obtained for Fermat criterion. Intuitively, the & criterion should have the same Applicability problem as the
ky + O, where a; and K; are the standard
l criterion. However, from the proof in Appendix
L the mean of (F.)) is 0, k/ * 0, but

---

Under review as a conference paper at ICLR 2021
Var Var(F (FF)) tends be none-zero constant with respect o,. In other words, & criterion does not necessarily have Applicability problems unless o, is small enough.
Except for the Norm-based criteria, we analyze another type pf pruning criteria called RcPr (Prakash
ct al. 2019). This criterion considers the orthogonality among the filters in one layer. Based on the proof in Appendix [ and the fact that the importance measured by &2 norm tend to be identical, also know that this criterion cannot prune the network well when the number of filters is too large. Moreover, in Section 5.2] we study the Applicability problem for more different types of pruning criteria from a numerical perspective.
DISCUSSION
5.1 WHY CWDA SOMETIMES DOES NOT HOLD?
CWDA may not always hold. As shown in Appendix P. small number of convolutional filters may not pass the statistical tests. In this section, we try to analyze this phenomenon.
(1) Need to be trained well enough. The distribution of parameters only be discussed when the network is trained well. If the network does not converge, it is easy to construct a scenario which does not satisfy CWDA, e-g., in [ (Appendix()) we train network with uniform initialization. Although the distribution of parameters converges with the increase of epoch more and more to a normal distribution, when only a few epochs are trained, the distribution of parameters is closer to a uniform distribution. At this time, the distribution obviously does not satisfy CWDA. Moreover, if a network is not trained well, e.g., its parameters converge bad local minima, there may be many unforeseen circumstances causing CWDA to be unsatisfied. In order to eliminate these factors, the network should be trained when studying the parameter distribution.
(2) The number of filters is insufficient. In Appendix the layers that can not pass the statistical tests are almost those whose position is in the front of the network. common feature these layers is that they have few filters, which may not estimate statistics well. In Fig. [ due to the sensibility of layers, the Sp in the transition layer are relatively small. Taking the second convolutional layer (64 filters) in VGG16 on CIFAR 10 alleviate this sensitivity. As shown in Fig. 6 we change the number of filters in this layer from 64 to 128 256. After that, the Sp increases significantly, and it suggests that enough number of filters are essential.
example,
find that increasing the number of filters could
(3) The dimensions of the filter not large enough. The dimension of filter in the 7th layer is closely related to the number of filters in the (i 1)th layer. To eliminate the influence from the number of the filters, we can consider the correlation matrix of each parameter in the filter, As the analysis in Section [ only when the dimension of the convolutional filter is large enough will the matrix approximate a diagonal matrix. It may also be a reason why the layer in the front of the network usually can not pass the statistical test in Appendix P
VGG16-2 Conv (64)
VGG16-2 Conv (128)
VGG16-2nd Conv (256)
Spoannan's corelation
Spearmnn's correatiinn
Spearman's correlation
UGl 20%
1304
LoGN
Index of lyer
Index of layer
Index of layer
Figure 6: The Sp between different pruning criteria on VGG16 (CIFARIO). The number of filters in the second convolutional layers is changed from 64 to 256.
5.2 WHAT ABOUT OTHER PRUNING CRITERIA?
In this section, we study the Similarity and Applicability problem in more types of pruning criteria through numerical experiments, such Activation-based pruning (Hu et al. 2016 Luo & Wu
7

---

Under review as a conference paper at ICLR 2021
2011 7
Importance-based pruming (Molchanov al. 2016 2019a) and BN-based pruning (Luu et al.
201 bb
2
2
B
3
3 6
C35
035
0.40
800
000
077
10
2.3
075
6.0
VCG16 gh Conv
VGG16 10"
Nei56
(4) ResNNeSE
1: 1 norm
f2 norm
3: Taylor &
Taylor &
BN_y
& BN
Entropy
8. APoZ
Figure 7: The Sp between different types of pruning criteria on VGG16 and ResNet56.
For each type, choose two representative criteria and call them: (1) Norm-based: E and
82 (2) Importance-based
Taylor L and Taylor &g (Molchanov et al.] 2016 2019a by (3) BN-
based: BN_ and BN_$ [Liu et al. 2017b); (4) Activation-based: Entropy (Luo & Wu 2017) and
APoZ (Hu et al. 2016). The details of these criteria can be found in Appendix M
The Similarity for different types of pruning criteria. Fig.
show the Sp between different
types of pruning criteria, and only the Sp greater than 0.7 are shown because if Sp < 0.7, it means the Sp (1) As verified in Section 3 L and & maintain strong similarity in each layer; (2) the layers shown in Fig. ( and Fig. d). the Sp between most different pruning criteria are not large in these layers, which indicates that these methods have great differences in the redundancy measurement convolution filters. This may lead to phenomenon that one criterion considers convolutional filter to be important, while another considers redundant. We find a specific example which shown in Appendix E. (3) Intuitively, the same type of criteria should be similar. However, it can be from Fig. (7b) and Fig, (c) that the Sp between Taylor Ê and Taylor & is not large, but Taylor &a has strong similarity with both two Norm-based criteria. Moreover, the Sp between BN y and each Norm-based criteria exceeds 0.9, but is not large in other layers (Fig. (b) and Fig. 7 d)). These phenomena are worthy of further study.
that there is no strong similarity between two criteria in the current layer. According
shown in Fig. 7 we can have the following observations:
mean
var'mean
var
34
02
Index of convolutional
Index convollational
Index convolutional layer
& norm
2: Eg norm
3: Taylor &
Taylor E2
BN
BN_
Entropy
8: APoZ
Figure 8: The visualization of Applicability problem for different types of criteria. (VGG16)
The Applicability for different types of pruning criteria. According to the analysis in Section
the Applicability problem depends on the mean and variance of the parameters. Fig. S shows the result of the importance measured by different pruning criteria each layer of VGG16. VGG16 is a network whose width becomes larger gradually as the depth increases (from 64 to 512). For this reason, we can study the Applicability problem generally wide (shallow layers) and sufficiently wide (deep layers) convolutional layers. First, we analyze the Norm- based criteria. The mean of both & and l2 are relatively large, but most layers, the variance/mean of lg is much smaller than that of &, which means that the & pruning has Applicability problem, while the l does not. This consistent with the conclusion Section Next, for the Activation-basee criteria, the
*The empirical result for slimming training (Liu et al. 201 is shown in Appendix B
8

---

Under review as a conference paper at ICLR 2021
mean and variance/mean both large, which means that these two Activation-based criteria can distinguish the network redundancy well from their measured filters' importance. However, for the importance-based and BN-based criteria, their mean and variance/mean are close to O. According to the condition (1) shown in Section these criteria have Applicability problem, especially in the deep layers.
5.3 WHAT ABOUT GLOBAL PRUNING?
Compared with layer-wise pruning, global pruning is more widely (Liu et al. 2018 Molchanov
al. 2016 Liu et al. 20176) used in channel pruning.
Similarity while using global pruning. In Fig. 9 we show the similarity of different types of pruning criteria using global pruning on VGG16 and ResNet56. Comparing to the results from the layer-wise pruning shown in Fig. 7 we can find that the similarities of most pruning criteria are quite different in global pruning. In particular, for the results of & and 2 in Fig- (a), the similarity between and & is not as strong as the one in the laser-wise case. We argue that this phenomenon may due to the differences between the parameter distri- bution in different convolutional layers. More analysis can be found in Appendix
200
0.55
ResNet58
1: & norm
2: norm
3: Tayllor &
Taylor &2
5. BN_y
6: BN_3
7. Entropy
&: APoZ
Figure 9: Similarity while using global pruning.
Applicability while using global pruning. In fact, for global pruning, Norm-based criteria are not prone to Applicability problems. From Section 4 we have the estimations for the magnitude of importance in 7th layer calculated by & and & o; k/²² and v20; T Aut ) ft). Since o; and &; are quite different, the variance the importance is large in this situation. Fig. 10 shows this kind of difference of the magnitude different convolutional layers.
Our estimation also explains a common problem practical applications of global pruning: the network is easily pruned off. As shown in Fig. T0, take ResNet56 as example. Since the imporiance in first stage is much smaller than the importance in the deeper layer, global pruning will give priority prune the convolutional filters of the first stage. To solve the problem of inconsistent magnitude, we suggest that some normalization methods should be implemented or protection mechanism should be established. e.g., a mechanism which can ensure that each layer has at least certain number of convolutional filters that will not be pruned.
v0015 on CBAR1ED manO
VE006 C CHANSED d momO
Reptocth CRONNOO #armi
Restecthe on CIRANTED marmo
Standbck
Brondbuck
www.
wwW
The
Bolos LoNNEManniiii lugs!
BAke of www.eaaannnlllooo.........
Tolo of aorroaaaora 1987
Tebox of Loreditoond 1apa7
Figure 10: The magnitude of the importance measured by E, and lg criteria. The actual magnitude almost coincides with the estimation obtained by CWDA.
6 CONCLUSION
In this paper, we found two blind spots on pruning criteria: Similarity and Applicability. For Sim- ilarity, some primary pruning criteria can obtain very similar pruning results. For Applicability, some criteria can not distinguish the redundancy of the filters well when the number of filters is large enough. The comprehensive experiments validate these two findings, and our assumption is called CWDA. Under CWDA, these two blind spots are also discussed when using global pruning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, (9(12):2481-249) 2017.
I Bellido and Emile Fiesler. Do backpropagation trained neural networks have normal weight dis- tributions? In International Conference on Artificial Neural Networks, pp. 772-775. Springer, 1993.
Michael B Cohen, Yin Tat Lee, Gary Miller, Jakub Pachocki, and Aaron Sidford. Geometric median in nearly linear time. In Proceedings of the forty-eighth anmual ACM symposium on Theory of Computing, pp. 9-21. ACM, 2016.
Gavin E Crooks. Survey of simple, continuous, univariate probability distributions. Technical report,
Technical report, Lawrence Berkeley National Lab, 2013., 2012.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 113-123, 2019.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 000.0008..45555 2017.
Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, al. Global sparse momen- tum sgd for pruning very deep neural networks. In Advances in Neural Information Processing Systems, pp. 6379-6391, 2019.
Xin Dong, Shangyu Chen, and Simno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, pp. 4857-4867, 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(111)2121--255, 2011.
Bradley Efron. Student's t-test under symmetry conditions. Journal of the American Statistical Association, 64(328): 2e:::128822- 1969.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:// openreview net/ forum?ide xJ-1-333RR
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Jinwook Go, Byungjoon Back, and Chulhee Lee. Analyzing weight distribution of feedforward neural networks and efficient weight initialization. In Joint IAPR International Workshops Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recog- nition (SSPR), pp. 840-849 849. Springer, 2004.
RL Graham, Applications of the fkg inequality and its relatives. In Mathematical Programming The State of the Art, pp. 115-131. Springer, 1983.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 15.0000149 2015.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pp. 164-171, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1034-1034, 2015.
10

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645, Springer. 2016b.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint 1808. 808.06666,, 2018.
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4340 4349, 2019.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter prun- ing criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2009-2001211 2018, 2020.
Lars Hormander. The analysis of partial differential operators. Springer, 1983.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250) 2016.
Jie Hu, Li Shen, and Gang Sun. suueeze-and-excitttinn networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4703 2017.
Zhongzhan Huang, Senwei Liang, Mingfu Liang, and Haizhao Yang. Dianet: Dense-and-implict attention network. arXiv preprint 1905.:9900..077. 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference computer vision, pp. 694-711. Springer, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint 1rX1v:1404.5997, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. Advances neural information processing systems, pp. 598-605, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):278-23324 1998.
HyunJae Lee, Hyo-Eun Kim, and Hyeonseob Nam. Srm: A style-based recalibration module for convolutional neural networks. In Proceedings of the IEEE International Conference on Com- puter Vision, pp. 1854-1862, 2019.
Bailin Li, Bowen Wu, Jiang Su, Guangrun Wang, and Liang Lin. Eagleeye: Fast sub-net evaluation for efficient neural network pruning, arXiv preprint arXiv:2007.02499, 2020a.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1600 1608,0060.008 087/0, 2016.
Xiang Li, Xiaolin Hu, and Jian Yang. Spatial group-wise enhance: Improving semantic feature 2019.
learning in convolutional networks. arXiv preprini
11

---

Under review as a conference paper at ICLR 2021
Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8018-8027, 2020b.
Senwei Liang, Yuehaw Khoo, and Haizhao Yang. Drop-activation: Implicit parameter reduction and
harmonic regularization. arXiv preprini arXiv: D.Xiv:///1.008800 1.05850, 2018.
Senwei Liang, Zhongzhan Huang, Mingfu Liang, and Haizhao Yang. Instance enhancement batch normalization: an adaptive regulator of batch noise. arXiv preprint arXivv 04008, 2019.
Hubert W Lilliefors. On the kolmogorov-smirrov test for normality with mean and variance un- known. Journal of the American statistical Association, 62(318):399-40) 1967.
Yuchen Liu, David Wentzlaff, and SY Kung. Rethinking classddiscimmination based cnn channel pruning. arXiv preprint arXiv:2004. 14492, 2020a.
Zechun Liu, Xiangyu Zhang, Zhigiang Shen, Zhe Li, Yichen Wei, Kwang-Ting Cheng, and Jian Sun. Joint multi-dimension pruning. arXiv preprint urXiv:2005.0880 2020b.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. The IEEE International Con- ference on Computer Vision (ICCV), Oct 2017a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9736-27442 2017b.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810005270, 2018.
Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint ar0iv:/608.0399 2016.
Jian-Hao Luo and Jianxin Wu. An entropy-based pruning method for cnn compression. arXiv preprint arXiv:1706.00 05591, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 505850088000 5058-5066, 2017.
Sebastian Lutz, Konstantinos Amplianitis, and Aljosa Smolic. Alphagan: Generative adversarial networks for natural image matting. arXiv preprint 1rXiv:1807.10088 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv: 16/ 1,06440, 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1264-11272, 2019a.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019b.
Radford M Neal. BAYESIAN LEARNING FOR NEURAL NETWORKS. PhD thesis. University of Toronto, 1995.
Rodrigo R Pescim, Clarice GB Demétrio, Gauss M Cordeiro, Edwin MM Ortega, and Mariana R Urbano. The beta generalized half-normal distribution. Computational statistics & data analysis, 54(4):945-957, 2010.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4):838 855, 1992.
12

---

Under review as a conference paper at ICLR 2021
Aaditya Prakash, James Storer, Dinei Florencio, and Cha Zhang. Repr: Improved training of convo- lutional filters. Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 10666-10675, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.06433 2015.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances neural information processing systems, pp. 91-99, 2015.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing fine-tuning and rewinding in neural network pruning. In International Conference on Learning Representations, 2020. URL https:
//openzeview.. www.eee/eeeeeeee wot.:/ooummma/oo d=S1 lq JONKVB
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115())211-22) 2015.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6122 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning. pp- 1139-1147, 2013.
Yuandong Tian, Tina Jiang, Qucheng Gong, and Ari Morcos. Luck matters: Understanding training dynamics of deep relu networks. arXiv preprint 1905. 2019.
Sanghyun Woo, Jongchan Park, Joon- YounG Lee, and In So Kweon. Cham: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 3-19, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.0085: 2015.
Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2970-2979, 2017.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller-norm- wwwnoo--ess-iffmmmmeee informative assumption in channel pruning of convolution layers. In International Conference on Learning Representations, 2018. URL https: / /tppe/peeevvveeee www.iiewwweeeeeeeeeeeeeee.eeeeee htt////foummmiiimooo f
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 6023-6032, 2019.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
1rXiv:1605.07146, 2016.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 1212::211..701. 2012.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiyy Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890 2017.
13

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL STUDY FOR CWDA
A. IS NORMAL DISTRIBUTION NECESSARY?
In this section, we use the random functions provided by NumPy to generate data manually study the influence of distribution on the similarity between Norm-based criteria. We use these distribution functions generate 100 vectors with 50 dimensions each time. Each vector can be regarded as convolutional filter with dimension 50.
As shown in the Table 4 (Left), we compare the Sp these generated convolutional filters between l,, f2 and GM criteria. If Sp is greater than 0.9, we mark it as green. Otherwise, mark it as red. Under these generated distribution, & and &g can keep larger Sp in most cases, which indicates that they still maintain strong similarity for these distribution. But the similarity between GM and L (or fg) is relatively poor.
Table 4: The similarity when using different distribution.
Distribuiio
Mean
Std
E1-82
GM-62
GM-c;
Mean
f1-62
GM-(2
GM-4
0.29
0.94
8.96
0.95
randn
0.00
1.00
0.93
0.55
0.93
0.00
0.99
0.95
0.93
0.94
1orandn
1,00
1.00
0.95
0.63
0.51
0.00
1.00
0.95
0.96
0:95
1.2)
0.98
b.tar2.51
0.28
0.15
0.6
0.70
0149
0.00
0.15
0.91
0.94
0:95
chisquaro())
2,02
2,03
0.92
0.91
0.73
0.90
2.01
0.97
0.95
0.94
gom.a(2.0.55
gammar2.2)
2,76
0.96
betar (2)+bbnarO 1.3)
0.37
0.26
0.81
0.67
0.31
0.00
0.76
0.81
0.91
0.96
betar 1,2)+bbtaa2255)
[0.1
0.27
0.93
0.44
0..17
0.00
0.27
0.95
0.94
0.94
If we use zero-mean normalization for the generated data X, ie.,, X EX, as shown in Ta- ble4 (Right), we also find that the Sp between these three criteria are greater than 0.9 for all dis- tribution. This phenomenon may indicate that the normality in CWDA is not necessary for the similarity of criteria, but the zero mean the key to the similarity.
Table The similarity when using mixed distribution.
Distribution
Distribution 2
Use zero-mean?
Mean
Std
f1-f2
GM-
GM-6
randn randn rand
randnx8
0.00
5.69
0.98
0.95
0.96
0.26
0.78
0.97
0.96
0.96
randn rand
0.00
0.78
0.99
0.98
0.98
randn rand+5
2.72
2.87
0.99
-0.28
-0.24
randn rand+5
0.00
2,85
0.71
0.91
0.88
randn
log(rand)
-0.49
1.08
0.85
0.14
0.49
randn rand rand
log(rand)
0.00
1.07
0.69
0.90
0.89
log(rand)
-0.26
1.08
0.98
0.87
0.87
log(rand)
0.00
1.01
0.65
0.96
0.81
Next, we consider the mixed distribution, i.e., shown in Table 5 among the 100 vectors we generated, the first 50 vectors are generated by Distribution 1, and the last vectors are generated by Distribution 2. There are several notable phenomena in Table
(1) For the mixed distribution composed of two normal distribution with zero-mean but different variances, the three criteria can maintain strong similarity;
(2) In the mixed distribution, zero-mean normalization may not make the Sp between the three criteria stronger, which is inconsistent with the situation in Table R
(3) For & and f2, the Sp measured by the mixed distribution lower than single distribution (Table in general.
According to CWDA, the mixed distribution in phenomenon (1) actually corresponds to global prun- ing strategy. If CADA holds, the distribution of parameters in different convolutional layers depends
https:///ocs. wwt.:ecs...ipp.oo..ooooooo wec.ppooo///oo/nnggeeee http:///e//r///a/////eeree/////r
random. html
14

---

Under review as a conference paper at ICLR 2021
2 3
140
100
10
639
376
6.72
179
layer
layer
layer
layer
1: norm
2 82 norm
Taylor 4
Taylor &
BNL
6:
BN_3
Entropy
a:
APoZ
Figure 11: The similarity for different type of pruning criteria while using global pruning with different start layer.
on the variance parameters in each layer, and these variances are generally different. Therefore, the global pruning strategy is actually to prune the convolutional filters that satisfy different Gaus- sian distribution. Unlike layer-wise pruning, these filters cannot be represented by a general random variable F, where F N(O, c² IN.xkxk). It needs be described by stochastic process, i.e.,
Fg ~ N(O, (1() [
According to the experiment of phenomenon (I) and CWDA, we expect that & and & should still maintain strong similarity while using the global pruning. However, as shown in Fig. the Sp of l and &g is not very large. This seems to contradict the experiment of phenomenon (1). In fact, however, according to the analysis in Section 9 CWDA may not hold in the first few layers of CNNs. So if these convolutional filters are also considered to be globally pruned in global pruning, then the similarity between & and lz may be weakened, which is consistent with the experiment in Table 5
To verify this statement, Fig. shows the result of VGGI6 via using global pruning strategy. The Start layer means the layer that we start to use for global pruning. For example, when Start layer is equal 0, it is the general global pruning: When Start layer equal 5, the filters from the first five layers are not be considered to be pruned. In Fig. IT we can see that, with the increase of start layer, the similarity between E and &g becomes stronger and stronger. This shows that the similarity between & and f2 is not as strong layer-wise in global pruning, which is mainly caused by the first few layers that do not satisfy CWDA.
Moreover, in phenomenon (2), global pruning is different from layer-wise pruning shown in Table The zero-mean normalization may not make these pruning criteria similar. Therefore, global pruning is worthy of further studies, such using the stochastic process to explore the similarity and the Applicability problem of different types of pruning criteria.
17.5
Block1-layer2--om 0.2437 0.023
Frockl-nnw2--om
20
0.0006
0.0093
varimeen 0.0010
#fiiee 160 44m
Frequency
12.5
Block2. Loyer2-com1l
Frequency
15-
Bockk-laaye2-oon mean 0.3164 var: 0.0010
mean 6.2.70 von 0012 var/mean: 0 00066
var/mean:
7.5
nitter: 320 dm 2680
Bookk-layyr2-ccome
Block3-layer 2eada--lyyr2-ccmme
mean 0-674 0012
5-
0.3704 var/mreem: 0.0022
2.5
#ttteer 843 dim5560
0.0
1.00 Value of norm
1.75
Value of norm
Figure 12: The distribution of l norm when the WRN is not trained well. Left: without data augmentation. Right: trained with uniform initialization and epochs. Like the phenomenon in Fig. I & pruning still has Applicability problems when the network is not trained well.
15

---

Under review as a conference paper at ICLR 2021
B TRAINING THROUGH SLIMMING
7
100
1.00
140
0.35
05
0
0.55
0.55
0.55
6
17
0.00
0.70
0/70
0.70
(a) ReaNet56 global (slimming)
(b) Net5 8th Conv (slimming)
(c) ResNet56 14th Conv (slimming)
1.50
8.05
3.0
0.05
5
40
1.9
0.00
300
P-
6.70
(o) ReaNet56 global
(f) ResNet56 8th Conv
ResNet56 14th Conv
1: norm 2: l norm
Taylor & Taylor 5: BBN. 6: BN B 7: Entropy APoZ
Figure 13: The Similarity for different criteria with/without slimming (Liu et al.
2017a).
As representative of the BN-based pruning method, slimming pruning(Liv Liu et al. be directly compared with the criteria mentioned in the paper because adopts special training method. Therefore, we use the training method in Liu et al. (2017a) to train another ResNet56 on cif Then, the analysis of similarities between 8 different pruning criteria on such model is shown in Fig. 13
2017 7a)
can not
fifth criterion BN_y the method introduced in Liu et al. (011a) From Fig. [3] there is significant difference in the result of the similarity between ResNet56 obtained by slimming method and resnet56 trained in general.
In this situation,
16

---

Under review as a conference paper at ICLR 2021
C
THE RESULT OF SP
70
Spearman: 0.99
1300
Spearman: 0.99
60
1202
Volue of1 00mm
500
802
10-
700
0.25
1.75
1.50
1.75
(a) Sp = 0.99
(b) Sp 0.99
Spearman: 0.99
Spearman: 0.98
Volle Farmor
00
15
0.23
8.5
0.75
1.00
1.15
1.30
1.75
0.0
0.5
1.0
3.5
Value of norm
Value
norm
(c) Sp 0.99
(d) Sp 0.98
Spearman: 0.98
Spearman: 1.00
Femmot.
Volue of
400
150
6.5
300
0.0
05
1.0
2.0
0.0
1.0 1.5
2.0
Value norm
Value of norm
(c) Sp = 0.98
(f) Sp 1.00
Figure 14: The Spearman's rank correlation coefficient (Sp) for different criteria. (a-c) are Sp be- tween E and Eg. GM and E,, Fermat and & from ResNet18 (12 Conv), respectively. The results of VGG16 (3rd Conv) shown in (d-f). If the Sp of two pruning criteria is close to 1, then the sequence of their pruned filters may have strong similarity.
17

---

Under review as a conference paper at ICLR 2021
D OTHER RESULT
RM Bets BM Cartea
The Trd
70-
The 2rel The JSLIPC
60-
Frequeto
30
40-
10-
20-
10-
-0.75 -0.25
0.00
0.25
0.75 1.00
-0.05
0.55
weights
weights
Figure 15: The distribution about other learnable parameters. (Left): The disrtibution about the learnable parameters of batch normalization. (Rihgt): The distribution of the parameters of fully- connected layers (FC). For FC, the Sp between the criteria in Table2 are greater than 0.9. More analysis can be found in Appendix A
In Fig 15 We show the other learnable parameters (i.e. Batch normalization (BN) and fully con- nected neural network (FC)) in VGGI6-BN. For BN, the distribution of its parameters does satisfy CWDA, and similar results are shown in Liu et al. (2017a); Tian et al. (2019). Moreover, the learnable parameters of fully-connected layers also do follow Gaussian-all -lake distribution, which is consistent with the conclusion in previous work Bellido & Fiesler (I993). Neal (1995) Go
al. (2004).
Epoch:
Epoch: 6
Epoch: 14
Epoch: 24
Epoch: 34
Epoch: 42
Epoch: 62
Epoch: 120
Figure 16: The distribution of the convolutional filter (141th Conv) with kaiming- comminggunnffrm initializa- tion for each epoch.
18

---

Under review as a conference paper at ICLR 2021
E AN INTERESTING CASE FOR importance MEASURED BY DIFFERENT CRITERIA
The following results are the index of pruned filters obtained by the filters' importance from different types of pruning criteria. We take VGG16 (2nd) as an example. The 5th filter in this layer is regarded redundant convolutional filter for APoZ criterion, but other criteria consider it to be almost the most important.
Taylor 4 [27, 36, 25, 11, 23, 24, 16, 57, 48, 53, 1, 61, 18. 55, 34, 51, 58, 31, 12, 21, 59, 30, 7. 38, 41, 50, 10, 33. 17, 46, 62, 13, 49, 43, 42, 47, 2, 32, 44, 20, 39, 52, 56, 40, 9, 26, 37, 22, 29, 54, 60, 8, 14, 45, 4, 63, 19, 35, 2]
Taylor fz: [23, 32, 36, 11, 62, 16. 30, 59, 10, 13, 2, 50, 38, 0, 46, 43, 23, 26, 15, 22, 7, 51, 39, 33, 14, 58, 9, 40, 57, 61, 44, 20, 48, 5,53,,1 41, 56, 17, 12, 18, 31, 4, 1, 25, 19, 63, 24, 54, 45, 52, 37, 55, 47, 34, 35, 8, 29, 42, 49, 28, 5]
BN [52, 46, 32, 31, 21, 14, 29, 17, 0, 19, 36, 1, 51, 44, 40, 41, 60, 57, 27, 53, 63, 8, 30, 26, 23, 58, 39, 18, 9, 47, 31, 35, 35, 4,, 61, 6, 4, 33, 25, 15, 48, 43, 28, 56, 2, 13, 16, 34, 20, 59, 10, 24, 50, 62, 12, 49, 38, 5,54
APo7: [5, 10, 38, 42, 62, 24, 13, 12, 7, 29, 15, 23, 11, 56, 35, 57, 19, 49, 43, 25, 6, 63, 61, 36, 9, 27, 33, 20, 48, 58, 55, 18, 51, 31, 1,0, 57, 26, 29, 47, 60, 8, 44, 41, 46, 21, 17, 14, 32, 52, 22, 39, 3, 40, 30, 4, 45, 50, 54]
19

---

Under review as a conference paper at ICLR 2021
F RELATED PROPOSITION
Proposition 1 (Amoroso distribution). The Amoroso distribution is a four parameter, continuous, univariate, unimodal probability density, with semi-infinite range (Crooks 2012 And its probability density function is
1
3
0. exp -
X - 9
Amoroso a, a, a) 8) = T(a)
(4)
0
0
for T. o, O, Or, 3 E IR, a > and range x 2 a if > 0, x S a aif < 0. The mean and variance of
Amoroso distribution
(x a +(a+#)
(5)
T(aa)
and
+(a+ T(a)
T(a ))²
Var (mooAAAm((((aaaaaa. 0²
(6)
T(a)²
Proposition (Half-normal distribution). Let random variable X follow a normal distribution (0, ("), then Y = XX follows half-normal distribution (Pescim al. 2010). Moreover Y also follows Azoroso(z|0, v2a, 2). 2). By Eq. distribution are
and Eq. (6. the mean and variance of half-normal
E Ex 02/2/
and
Var Varx 0² (1 (1-
(8)
Proposition 3 (Scaled Chi distribution). Let X (21, 22 ... and 1...,,,.... are k inde- pendent, normally distributed random variables with mean O and slandard deviation a. The statistic
(2(X) E n? follows Scaled Chi distribution (Crooks 2012). Moreover, (x(X) also fol- lows Amoroso(a V20, 2,,,,,2, 2) By Eq. 5) and Eq. 66. the mean and variance of Scaled Chi distribution are
r( k
E 1x.N0.11111(11)) 20/2
(9)
T
and
[IV +1
I )² T(
2(x = 202
(10)
G PROOF OF THEOREM
Theorem 1. Let X
N(O, x² In) and (C1, C2) one of (2), (6,, Fermat) or (Fermat, GM), we have
(2(x) (i())
(1(X)
max Varx
Varx
< B(n).
(1I)
(x(X)
where (x) denotes (X)) and (2(X) denotes C2(X)/B(C)(X))) B(n) denotes the upper bound of left-hand side and when n is large enough, B(n)
For den layer, we use to represent Púj. J 1, N...N And uy meets CWDA (i.e., V are i.i.d and Uy ~ NO, o² I)). Actually, from the theorctical analysis in Section the fact that two criteria C and C2 meet Eq IT equivalent to C & C2.
20

---

Under review as a conference paper at ICLR 2021
(1) For (1). In fact, 82 = l (their importance rankings are similar) is not trivial. Generally speaking, for convolutional filters, dim(vy (1y) is large enough. Since ey satisfies CWDA, from Theo-
rem we know that the variance of ratio between & and &g have a bound O(dim(r,))) which means lg and l are appropriate monotonic. Specific numerical validation shown in Fig. 17 Appendix Hp.
Theorem 2. Let X N(O, c² In), we have
(x)
(X)
1
max Varx
Varx
(12)
(x())
(x(
n
where (X) denotes ((X)//B(((X)) and (X) denotes (x(X)/B((2(X)),
Proof. (See Appendix H
(2) For (41, Fermat). Since es satisfies CWDA, from Theorem 3 we know that the Fermat point of and the origin O approximately coincide. According to Table Fermat - vall2 [| = 1|0 oill2 ||2/122- Therefore, from Theorem 2 the bound (n)) for the (f1, Fermat) also n' Moreover,
since CWDA, the centroid of v, is G D i= U O. Hence,
N
G
Fermat.
(13)
Theorem 3. Let random variable v; (O.)..g). For E R², we have argminp (Em, - 1| O.
R* and they are i.i.d and follow normal distribution
Proof. (See Appendix Ip.
(3) For (GM, Fermat). First, we show the following two theorems:
Theorem 4. For n random variables ag E IR follow N(0, c² Ig). such estimation:
When k is large enough, we have
Fa;);) Fa(as) 2nk'
F2(as) (a;):) 2nk
Vara;
Vara;
(14)
where = E |ass|///((( BB |1a:112) and P2(a;) = Σm 1a::1///00/ 1a.11
Proof. (See Appendix D.
Theorem 5. Let vo, V1.... up be the lo + vectors in dimensional Euclidean space E". For all P
in E",
k 1 11P- w.B
k
(15)
1=0
=0
where G is the centroid of Us, will hold if satisfies one of the following conditions:
(I) 2 n and rank(U1 Up, U2 vo. ..., Uk Up) n.
(2)if l < n and (w1 Vo. Uo .... Up vo) are linearly independent.
u3)if (0, 2 I.). Eq. 15) holds with probability 1.
Proof. (See Appendix
c² I), we can obtain that a, P 1a NO0.2c2 if P of Uy. According to the analysis in Section and Theorem 4 we have
Let P e (V1, U2 ..., }. Since in
(N
|a l2
|
(16)
21

---

Under review as a conference paper at ICLR 2021
Next, prove (k + P F (Fermat) and Σ ||
-1il2 (GM) are approximately
monotonic, where P E [U1 U2, J.
(k + F = (I: + P) G
Since Eq. 13
P 1:11
Σ
||G 0:l12
Since Theorem
N
|
alle
Σ
||G - v.112
Since Eq. (T6)
P
vill2
(17)
The reason for the last equation is that Σ || - 22l12 is constant for given
H PROOF OF THEOREM 2
Proposition 4 (Stirling's 's formula). For big enough x and IR.t, we have approximation of Gamma function:
x
T(x + 1) V 2TT
(18)
Proposition 5 (FKG inequality). If f and increasing functions R" have
Craham 1983
B(f)E()) S B(fg).
(19)
Say that a function on Inth is increasing if it is an increasing function in each of its arguments.(i.e. for fixed values of the other arguments).
Proposition Let f(X. Y) a two dimensional differentiable function. According to Taylor the- orem Hormander 1983). we have
= ((BX), Σ( E(X))
0x ((I((X)E((Y)) + Remainderl, (20)
eye
-X--B(X) B(X)) A
5x (B(X),
-
eyee
(21)
8²
Σ
- -x
-(())) 9x2 (( - B(X)) + Remainder2
2
Lemma 1. Let X and Y random variables. Then we have such estimation
X
E(X)
Var
VarY
Cov(X,
Var
(22)
E(Y)
E(X)²
E(Y) E(X)E(()
en. wn.wikipedii weweeia..rrgwiii www.//gg////koooo.o /Stii//iiiii ing' sapprox mation
22

---

Under review as a conference paper at ICLR 2021
Proof. Let f(X,)) X/Y. according to the definition of variance, have x) = E| Y) B()(
E(X) a (IB(X),B)) (
=II(X, E ((E(X),
0x
eye
from Eq. 20
E(X EB(X B(X)) a 0x (((())
E[f(X, Y) F(E(()), Σ
eye
Elf(X,Y) - [(B(X),
(X - E(X) a
E
from Eq. 20
0X
eye
2Cov(X, f(E(X) B(Y)) BY ((B(X), + Σ(
Var.X
OX
ege 0X
E(X)
(EX)2 (EYY)²
2Cov(X, B(Y)
(B())
(B(Y))² Var
VarY
[(X)
VarX
VarY Cov(X,y)
E(Y)
E(X)²
B(Y)²
(E(X)E(Y)
Lemma 2. For big enough x and RR, we have
T( 1(
lim ++000
(23)
[(5)
2
And
[ + +1) r(2)
T( T( 2 [(
1
lim 2-++00
(24)
Proof.
2
T(
1
2m
m-1
lim
2
lim
from Proposition.
[(5)
x
x++
1m
2-2
2-2
2e
- 1
I
1
lim
28
x-+x
2
2E
2e
1
1
lim x-+x
1
x.
T
2e
T
2
on the other hand, we have
T 11
T( I(
1
x-2
F.
lim
lim
1 1 x. - 2
+oo I(
I(5)
2
E - 2
2e
1
lim
e- (1 + )
2e
[(-e)
2
23

---

Under review as a conference paper at ICLR 2021
Theorem
2 Let X ~ NN0, c. I,). we have
(x(X)
(x)
2 _
max
Varx
(x(x)
Varx
(x(x)
where (x) denotes (((X)/I((((( and (x( denotes (xxx)/B(2((X))
Proof. For the ratio (x we have
(X)
((G((X)) 2
(X)
Var
Var
(x))
B()2(X))
(x))
(()((X))
(x)(()))
Varl((X)
Varli(X) (ov((2(X),
B()((X))
((e
(x))
E(G( (()
from Lemma.
(X)
Varl(()
from Proposition.
(x))
(X)(
similarly, we also have
(X)
Varl2(X)
Varl
Var
(25)
(x(X)
(()2(
(X))
Therefore,
(x) (i(x))
(x)
Varl2(X)
Varli(X)
max Varx
Varx
(2(x)
[((2(X))²
(X))
20
r((+1)
p( T(T)
T(T)"
σ² (1 2 2)
(v2o
1 T(E)
(n.ov 2/
from Proposition. and
+ - 1) 1
2
from Lemma 2
2n
2n
Because the approximation is widely used in the proof of Theorem it is necessary verify it
numerically. As shown in Fig. 17
we use ResNet56 on Cifar100 and ResNetl 10 on Cifarl respec-
tively verify Theorem From Fig. 17
find that the estimationn of Theorem Tis reliable, iee.
the estimation ( for max
Varx
(2(X)
Varx
(x))
is appropriate.
(x)
(x(X)
I PROOF OF THEOREM 3
Proposition 7. Let (()) (x) denotes generalized Laguerre function, and have following properties:
8m
L (a)
(-1) L (a+n) (x),
(26)
0.nh p
for 0,
L [ (I) > 0.
(27)
24

---

Under review as a conference paper at ICLR 2021
ResNet566 on Cifar100
ResNert 10 on Cifar10
0.040
van/ vanfx()
van
0.035
vant556)
0.030
I112n!
(e- IV/20)
0.025
0.020
0.015
0.010
0.005
0.000
Googe
0.000
10
20
30
40
50
20
40
60
80
100
layer
layer
Figure 17: The approximation of Theorem example about ResNetl10.
(Left) the example about ResNet56; (Right) the
Theorem 3 Let random variable is E RR*, They are i.d and follow normal distribution N N(0, II.)) For F in R., we have
argminp Eu
Σ F 2s 12 = 0.
Proof. Let wi F and we have wa ~ N(F, I+ then
2 Σ11F 1/
-
Σ E, No 2/112
I (F
1=1
π (3-
11)
(-
|1
2
20
The reason for the last equation is that (FF, TT.).. Let ((x)
||w ll follows scaled noncentral chi distribution when wy
7 -1) x²
calculate the minimum of T(x)). From Eq. (66).
d
((())
x2
T(x)
(28)
dx
0²
20
Since Eq. (27) we find that 4T(x) > 0 when x > 0 and x S 0, then 4T(x) S It means that T(x) gets the minimizer at [[11|2 0, i.c., F
de
PROOF OF THEOREM
Lemma 3. For two random variables X, E E follow N(O, e² Ix) and they are i.i.d. When is large enough, we have:
(|ΙII13 ||113)2)
4c +1
2||X||2 ||Y||2 = 20² +
(29)
212
and
(IIX113 111X11-11110 -11118))
8c* 16c o²
Var
(30)
|||X||2 ||X|||
Survey of simple.coniiuonnsuumaitttt prohabiiity distribution and Wikipredia
25

---

Under review as a conference paper at ICLR 2021
Proof. According to Proposition and Lemma it is easy know (similar method in (
when k is large enough, that
E [2| (111)) = 2c
Var (2 || = c² + 4c
(31)
and
E ( - 4c*k;
Var ((I
1 16c (2h² 3h).
(32)
Since Lemmal
1 we have an estimation
(IIX |
E
11118)2
Var [ 1|X11
Var(2||X 1 ||| 112
Var
VI
X |
X
E2
1|8|12
B
Y 12)
E(2
X12 ||||l2)²
4
(
+ 4
16c (2h² + 3k)
20
4cA
1-c
Since Eq. 31 and Eq. 32
16c +
Sc
k2
From Eq.211 and Lemma we also can obtain an estimation E(A/B), where A and B are two random variables. i.e.,
EA
REA
+
Var(B)
(33)
B
EB
(EB)3*
Therefore,
(11 1011)
E 3
EE X 1|
E
~3
+ (2 (1x]1))
X|| 12
||Y||2
E2
X
||
(E2) X |||
Since Eq.(3)
4 + 4
(c2 + kc*k)
Since Eq. 31 and Eq. 32
2
2c + 4 +
2k
Note that, the approximation is widely used in the proof Eq. and Eq. (30p. Hence, it is also
necessary to verify it numerically. As shown in Fig. [8] the estimation is appropriate. According to Lemma 3, the mathematical expectation and variance of the ratio of [||X|| - and 2 X||22 ||Y||2 are both close when k is large enough and is small enough. that is,
2( ||Y|la)
(1 - ||
(34)
By the way, the convolutional filters casily meet the condition that is large enough. Theorem 4, For random variables a, R follow N(0, c² Is). When k is large enough, we have such an estimation:
(as)s) ~ 2nk'
F2(as) ~ 2nk
Var Fa:a:)
Var
(a:):)
where (a,) Em ||
/ E
1 ||ail22) and F2(as) Σi=
E EE 1a.111
Proof.
Since Eq.
9 and Eq. 10 we have
Var Fiia:)
nne2k k nevk)
2
Σ E
|a:
Ez(a:)
=
Var
||:
113
(35)
and
F2(a,)
nevk no²k
|
1
Var F (as)
Var
(36)
2
26

---

Under review as a conference paper at ICLR 2021
(8- 10): Varti 1111111--1110000 |018XX3110] YIst
(km2h Estimation
Esiimatioo
(R- 51:/10//1g |Р|8AAI|ИИ//
08 20) Vartt (MB////AA) YYsl
00-5) Estimation
30-
#l-20listtimation
Ut 100)-ENNPPIE (А:АÍ))
(o-100) Estimation
(K- Estimaaion
20-
12
12
Figure 18: (Left) The numerical verification of Eq 29 and (Right) The numerical verification of Eq 30 and follow N(0, I4).
According to Lagrange's identity, have
|a 12
Σ =
|as
(1|a,112 ||a
1sKzssn
|a:11
112
(Ila:ll2 (|al|1a)))) +
Σ (1a:112 ||ay113²
Isicjsn
ISiRisn
|a:|
1/2 +
Σ
(Ila:122 (11a51aa))
Since Eq. (34)
Isi<ssn
2
n
||ui11a
so we have
== a
V(0.
2 Var Vor
(37)
10:12
Σ 10:112
By central limit theorem, we have use Delta method to find the distribution of g( E [as|l2):
(
v
Ei=1 ||ai|12- - -)) N(O, And let g(x))())))))))
we can
T Di
||للe| ) -((1)) N(0 02 [g(x)) NO 0²
Vn
g
(38)
n
where μ and ²² denote the mean and variance of |aıl|2 respectively. From Eq. 33p. we have
|alla
n
Var N10,
2 Var Var.,N
|a.12
E |0:112
1
=
Since Eq.
38
# n
r +1)
r( 1
20
r
I(4)2
(v2c-
r A+ T
)4 n
Since Eq. and Eq. 10
Since Lemma.
2e . mk2
27

---

Under review as a conference paper at ICLR 2021
0.044
0.014
[l-3) Fat
Van
[kns)itatiiation
0020
In= 5)
Vart!
0.00
Varti-/ (f
0.000
Vk-22r FF)
(0- VartFaF))
[k=201 VartFaF))
0.066
Ih- Estimatio
0.036
0.004
0.02
0.02
0.000
0.000
Figure 19: A numerical verification of Theorem where Fi -E= ||, 12/B(E== [0:112) and F2
Σ ||al1///11//// ||a,112) a, follow N 0.01 ..))
Since Eq. (35 we have
Pa,) Vara: (2(as)
nc²k 2 nevk
n ==
la.12
1
Var
(39)
2nk
1=
Similar to Eq. 37
n
|1:
|0:
1|
Var N(0.
==
Var N(.
ri=l
(40)
|a:l2
12
si=l || 2
N(0.c.
Var e, N(0
Similar to Eq. 37
|
0²
Since central limit theorem
T k +
r 1( )² 1 n
2c
Since Eq. (10)
I(
r
e²
Since Lemma.
Since Eq. (36. we have
nevk 2 ne2k
F2(as) Vara, Fi(a;)
Σ G
||a:113
1
Vara;
(41)
|0:112
2nk
From Eq. 39 and Eq.04] 41
Theorem holds.
In Fig. 19 we also show numerical verification Theorem
K PROOF OF THEOREM
Proposition 8. For m random matrix (asz) mxmm where as N(0 And Eq. holds with probability I.
rank((ai))xxo)) (axm) min(m, n).
(42)
28

---

Under review as a conference paper at ICLR 2021
Lemma 4. Let vo, Ui,..., ..., Uk be the k 1 vectors in dimensional Euclidean space V and k S n.
If Trank(U1 (11 - Up. 02 U0..... Uk - vo) = n. then Vx E V, 3),(0 S is k). s.t.
X
(43)
1=0
and Ei=o A = 1. We call À = (No, Ai...,.. X the generalized barycentric coordinate with respect (No, U1,..., Uk). (In general, barycentric coordinate a concept Polytope)
Proof. Note that by the element of n dimensional linear space V and rank(v1 -0, 12- -o. ... Uk Uo....., U/- 2n) form set of basis in the linear space V. Vr E V,
vo) = n. It means (V1 up can be expressed linearly by them, (1, S i k) s.t.
x= 40
4.444 - vo)
i=1
(1
t4
t,00
i=1
Let Ao = (1- (:) and X; = 4(1 Si S k). Lemma
holds.
Lemma 5. Let Vo, Vi..., Uk be the k + 1 vectors in n dimensional Euclidean space V. Va, b E V. and the generalized barycentric coordinate of a, b with respect (Vo, Vi, Uk) X (Na, Xi,..., Az) and = (Mo, (1,..., ...)..,'?? ,respectively, Then
|a b12 (A 11) D(A 10),
(44)
where D -- and dej = ||0; 2yl1
Proof. Since Lemma let R over,
[vo, U1, ... (B+1))
(k+1), and we have RA and
Ru. More-
||a - b112 = -aa- -
(45)
= =[R(> -1)1= [R(2 - 11]1
(46)
= (A -1) R.. R(A pt).
(47)
Note that, for D +
1
1
2 dif
(U: 1j (T 1z)) 2
(48)
1
oof U1
(u + 0.),)
(49)
2
So we have D = RºR - 2 ((vf i + u ()))))))(())))
can be further simplified to D =
Rs R (Va= + ). where V (1) vo...., ..., of vT) and (1,... I) So
I
|a b112 = (A RT
(50)
= (A (A- + 2 (Va" + +aV ()()))() -p)
(51)
= (A -2- - 11) + 2 - ") + +oVT) -n),
(52)
therefore, only need prove (A (Va + aVV) 1)) =0. From Lemma have
a= (1) = (A - u) a = 0 and the Lemma holds
29

---

Under review as a conference paper at ICLR 2021
Definition (Ultra dimension). For set U composed of vectors in dimensional linear space V, we define (Um as the Ultra dimension of The definition that U has linearly independent vectors and there are no more, then dim(U? = k.
In fact, if U a linear subspace in V, then the Ultra dimension and the dimensions of the linear subspace are equivalent. If U is linear manifold, U {x: + 2o/2 E W}, where uo and W are non-zero vectors and linear subspaces in V, respectively. And dim(W) = r. Then
Vo W dim = r 1, $ W
(53)
In other words, dim(U)& 2 dim(W) always holds.
Lemma 6. For arbitrary k (1 k: 1), let 01, 02, ..., 0g be k. linearly independent vectors in dimensional linear space V. Consider one n- dimensional linear subspace W in and a non-zero vector vo in V. They form linear manifold P tala E W]. If 01 02,,.., 02 do not all belong to P, then there must exist n k vectors pt, 1l.P2.... Pn-k from P, 8.1 (a1, 02 ak. P1, P2, .... Pn. set of basis for the linear space V.
Proof. we use mathematical induction. First, show that the @emma6@holde for n - k = 1. it means we need to find vector P1 E P s.t. d1, a2 ... ag. Pi linearly independent. If P1 does not exist, then vp E P would be linearly represented by 01, 01,02,... ..., Og. In other word,
P Cc& L = span (1, a2. a2...., ax),
(54)
① For the linear manifold P, if to E W. This means that P is equal to the linear subspace W. Since Eq. (54, have W C T. and di) dim(L). Hence, P W L. However, 01 02, Ok do not all belong to P. a contradiction.
① For the linear manifold P, if vo & W, then dim(P) n. Because vo $ W, that is, cannot be represented by a set of basis of W. In other words, vo and a set of basis of W are linearly independent. However, the dimension of W is n - 1, hence dim(P) = n. From Eq. (54). we have P L.
= dim S (L) = k n-n-1
(55)
contradiction. Therefore, Lemma 6 holds for - k = Assume the induction hypothesis that Lemma i is true when n k 1, where S l. when n - k = I Le., = n (1+ (1+1). 1), we
also can find vector P1 E P s.t. 01, u2,..., 0k. linearly independent. Otherwise, Vp E P would be linearly represented by a1, a2 ag. Similarly, we have Eq. (54). Note that, from Definition I
dim P) 2 - 1, hence
n-15 (P) S dim = = =
(56)
a contradiction. At this time, we have k: + = n (1 +1 +1= n - vectors @1 dg, ... .g. which are not all on P. Note that (n (1 (, using the induction hypothesis, the Lemma 6 also holds for n - A = 1. In summary, Lemma holds.
Theorem E
Let to. U1, .i.... Ug be the k + vectors in dimensional Euclidean space E". For all P
k IPP- D (A+ 1)]| G
1=0
where G is the centroid of vg will hold if it satisfies one of the following conditions:
(f)if k Z n and rank(U1 Up, U2 00 ...,.......................... Uk vg) n.
(2)if & < n and (wi 0o, 02 - Up. ..., Us vo) are linearly independent.
(3)if 0; NO., I,), Eq. 15 holds with probability where is constant.
30

---

Under review as a conference paper at ICLR 2021
Proof. For Theorem 5 (1). From Lemma 4 vP E J3 ("o..... s.t. P can be represented
by E No,; where Ei=u 7/ = In fact, for each V, also can be respresented by Σÿ=u 8,y%
where Σ Big 1. We just take (3.g. Bii. Bix) as one of the standard orthogonal basis E (0,0, .0.,,... 0). According to lemma 5
||P 0/ v2112
(7 (5y=)) <t)=((> (2)
(57)
= = Dy -27 Dr; + ci De:
(58)
= - Dr - -2²" Des.
(59)
The final equation is because the diagonal elements of the matrix are all O. On the other hand, we have
||G - v:l12 = k ++1
E) D
D(
Σ 6 e))
(60)
h+1
1=0
1=0
"Da
2
a
D.D + c. DEc
(61)
(h: + 1)2
k+1
1
T Da
2
a Det:
(62)
(k: 11z
k+1
where a = Et 4i ie..a = -1,1,, ..,,,,..,,....... Next, we consider ||P Gl
1).
1
P = (r - k+1 a) D(7 k+1 a)
(63)
- DJ + (k + 1)2
a Da
2
T Do.
(64)
k
In summary, we have
1
Σ P U/ 2
||G
v,113 (k: 1)? "Dy - -27 "Da +
a
Da
(65)
k 1
(k 1)1|P- G
(66)
Therefore, Theorem (1) holds.
For Theorem 5 (2). Next, we prove the case of k < n. Obviously, Lemma does not hold. We
consider about such a linear space W1 span (P- G), i.e., linear space expanded by P G. and its orthogonal complement Wii (in E"), Since dimension formula from linear space, it is easy konw that dim(W,) = n 1.
Two linear manifolds Ti and T2 are constructed as follows,
Ti = {u + Gl E Wit}} T2 {x G - - Wit.P
(67)
(68)
Ve, E Ti, we have (v. - GP - G) = 0, Furthermore,
P - v/ 1 =
(69)
It is easy to know that G - 0o is not O. v 00, ..., Up are all belong to T2, it means UI
are all Ti. Hence, we have Eq. 69). By summing both sides of Eq. (69) for is obvious find
that Theorem (2) holds. If 01 Uo ....
up are not all belong to T2, since Lemma [6) there are
n k vectors P1 uo, P2 no. ... Pn- & Up from T2 s.t. they and 81 Uo Ug - Up are linearly independent, where yæ obviously belongs manifold Ti.
At the same time, we have 2G P; E Tı, we can also construct new vectors 2G -G-p-- 17--8 10 E T2 and calculate the rank that
rank(v1 Uo,..., ..., Uk Up. P1 U0, ..., Pu-k k Un, 2G P1 Up, Uo,...,
to,.....G 2G Pa-k 17)
31

---

Under review as a conference paper at ICLR 2021
rank(U1 U0, ... Uk vo: P1 Up, ..., Pn-k vo, 2(G vu), .... 2(G v6))
(70)
rank(U1 V1- 20 2k
vo P1 - V0, Pn-k V0, O,.....) 0)
(71)
(72)
The reason of the final equation is that
Σ -i=1 (1) 00) (k + 1) - un). Note that there are
total of + (n - k) + (n - k) = n +
(1 k) 2 n vectors, meets the lemma condition. For the
convenience of description, define
_ = 1/ (0 < <is R),,
(73)
L, Pc. (1 sis S -
(74)
(8) 2G p., (1 n k).
(75)
And their centroid is
U/ + nーk [1. +1,
G'
1
Σ kmm0
(76)
2n k +
1
((k + 1)G + 2( k)G)
(77)
2n k +
=G
(78)
That is, the newly added vector does not change the centroid of e, On the other hand, since both I,(2) and (3(3) are in the linear manifold Ti, and it meets the conditions of the Eq. (9). Similar to the derivation in the Theorem (1), we have
(2n - k- - 1)I[P - G =
Σ
(1P- -t())
(79)
1 (1) 1 11 (3)
Σ (11P- -
|| -;1
Σ
(11P-113 - ||G (
(2) (1)0))
(80)
Σ (1I 11P--:13 |G- v.1B3 2(n k)] G
(81)
The final equation is because both L.(3) and L ( are in the linear manifold Ti and satisfy Eq. 69 0 ( = (k 1)l G Therefore,
To simplify Eq. (81) we obtain Σx= (I) will |
Theorem (2) holds.
For Theorem (3). When 2 n, from Proposition S we know that rank( (\1 - Upp 0/2 Up U vo) = holds with probability 1. Hence, if we use the similar deduction from Theorem 5 find that Theorem (3) holds when k 2 n. On the other hand, when A < n, we can get the same result also according to Proposition The reason is that (11 Dp. 82 Up. Uk Uy) are linearly independent with probability 1.
can
32

---

Under review as a conference paper at ICLR 2021
L THE GEOMETRIC STRUCTURE OF CONVOLUTIONAL FILTERS.
Theorem 6. Let vi E TR and 8; ~ N(,, c. [g). Ifk - do and c. + 0, then
(1) ll ~ ||51122 /2c- T((++1)/2) 1 < <
[(k/2)
(2) angle( +
(3) Il: rylla ~ |01 silblliissttss
(4) [[[l:::lli///ar(((l]))]
non-zero constant.
Proof. First, since Chebyshev inequality, for 1 Sis S N and a given M, we have
P (llvallla B([la|la)) v /Maar([vella) 1 M
1
(82)
from Eq. Eq. 10 and Lemma. 2 can rewrite Eq. (82) when l + 00:
P |1ua12 V2c- [[(k 1)/2)
M
1)/2)
M
1
22
/20
2 1
(83)
r(k/2)
T(k/2)
2
M
For small enough e > 0, let M = l/c. Note that / 4 c/v 22 is a constant. When k + 00,
V2c- 1((K+1/22 T(k/2)
M 2
Hence, for any i E [1, NN] and any small enough we have
P Pllballe 22c. [[ + 1)/2)
1
(84)
T(h/2)
So Theorem ( holds.
Let U. (0/1, V/2, Uhk) and Uy (0/1, 2y2, ... Uzz). So U2 >= Ex= Usp Note that, U/ and ey are independent, hence
E(UipUjp) = 0,
(85)
Var(UipUjp) = Var(Uip) Var(u;)) + (IE(vp)) 2 + Var((vi))))
(86)
since central limit theorem, we have
Upp - 0) 10,
Vi
(87)
According to Eq. 9 Lemma 2 and Eq. (8
when A: + 0o, we have
< <;; 01 >
< ve ni N(O, 1 N).
(88)
|1:112 |15//22
Vi
Vi
So Theorem 2) holds. From Theorem 6 1) and Theorem 2),, Theorem 6 3) can be proved through Pythagoras theorem.
For 4),, from Proposition.
we have
B(1ps]11)) Var([ou]]))
k. 8.cV
(89)
ho c2
2) c(1
As shown in Fig. 21 Theorem 6 actually reveals the geometric structure formed by the parameters of the convolutional filters in CNNs. Specifically, from Theorem (1), the convolutional filters v; of each layer locate approximately on the surface of & dimensional sphere with 0 as the origin and
/2c- [((+1)/2) the radius. Then, from Theorem 6 (2), the vectors formed by any two different T(k/2)
33

---

Under review as a conference paper at ICLR 2021
convolutional filters in the same layer are approximately orthogonal. As this result, for any three different filters V1, 222 and Vy. we can use Pythagoras theorem and Theorem (1) to prove that they
equidistant,..e. ||0 12ll2 ||0g 23//2 1|/3 -0 2il12. In fact, Fig. 20 provides another view of the geometric structure of convolutional filters. Since CWDA, [R(x)) O. So the correlation matrix {Corr(e; u;]) c. {(o by N where is constant. That is say, there is only one coefficient difference between correlation matrix and Gram matrix. Therefore, the diagonal elements of the matrix are l|0: and the off-diagonal elements are the dot product between the convolutional filters. This numerical visualization also verifies the conclusion of Theorem
Inder of
of
Pes3 1ot Dory
Photec21 5th Core
Eocriesta 50 Comy
25
53
Figure 20: The correlation matrix of convolutional filters. For clarity, we use the first 64 filters in each layer calculate the Gram matrix.
Cony filters
Origin 0
Cony layers
V/3
Figure 21: The geometric structure of convolutional filters when the network has large number of filters in each layer. for every pair of filters in one layer, (1) Their &a norm are equivalent ([lv ll2 se ||zll22 ~ ||ss lz (2) They are equidistant (Il 02 l = llee 2xll2 R I|us - en eill2); (3) and they are orthogonal (uf U2 Rs o Ui x ug U1 z 0).
34

---

Under review as a conference paper at ICLR 2021
M THE DETAILS OF OTHER PRUNING CRITERIA
For notation, we denote hth convolutional filter in layer as F. and the input feature maps in layer / I' E X''''x'' where N, I',, HH' Wi mean the train set size, number of channels, height and width respectively, i = =1 1.2.... .... A, and = 1,2, .... I. The formulation of the filters' importance under each pruning criteria are illustrated as follows:
Norm-based criteria:
21-Nomm Li et
2 ||
22-Norm Frankle & Carbin 2019)
||F||
BN-based criteria Liu al. (2017b):
BN_y: hel where of is the scaling factor in the Batch Normalization layer I;
BN_B: [8 where & the shifting factor in the Batch Normalization layer 1.
Activation-based criteria:
APoZ Hu et al. (016)) En. ((pp>))))
N Nx1 x x where we set = 0.0001 same as Luo & Wu 2017). and (-)) the indicator function, is convolution operator and I' F. the
i-th output feature map;
Entropy Luo & Wu (2017): we first prepare G GAP(T' F!), where G E R and
GAP(-) is the Global Average Pooling. Then, we estimate statistical distribution for G by dividing all elements G into m bins. Let Py is the probability bin j, and the the importance score is Ej=1P log
First order Taylor based criteria Molchanov et al. 2016 2019a b):
Taylor & NNorm::
Blsss
Fill:
BFY
Taylor NorNo OF
Bloss
Fl
The loss is the Cross Entropy Loss the split training from the original training
35

---

Under review as a conference paper at ICLR 2021
N
ADDITIONAL EXPERIMENTS ABOUT IMAGE CLASIFICATION
Table The accuracy(%) %) of several networks and datasets using different pruning criteria.
Experiment (1)
Experiment (2)
Experiment (3)
Trained
Pruned
Fine-tuned
Trained
Pruned
Fine-tuned
Truined
Pruned
Fine-tuned
CIFARIO
93.61
61.21
93.51
93.21
54.31
93.22
93.26
$7.74
93.32
VGG16
93.61
63.41
93.32
93.21
54.61
93.42
93.26
57.42
93.29
GM
93.61
61.22
93.41
93.21
53.71
93.25
93.26
57.46
93.36
CIFARI00
72.67
25.91
71.50
72.99
20.43
71.36
72.56
24.01
71.07
VGG16
72.67
27.07
71.28
72.99
22.31
71.12
72.56
24.45
70.92
GM
72.67
26.37
71.27
72.99
21.67
71.26
72.56
24,26
70.78
ImageNet
41
71.58
30.33
71.02
71.33
40.33
70.12
72.01
28.07
70.93
VGG16
82
71.58
29.47
70.83
71.33
40.45
70.13
72.01
27.89
71.02
71.58
30.76
70.95
71.33
39.86
70.33
72.01
28.01
70.74
CIFARIO
92.98
77.73
93.08
92.97
76.02
92.82
93.01
79.93
92.81
ResNet56
92.98
79.02
92.83
92.97
77.91
92.72
82.43
92.81
GM
92.98
74.26
92.77
93.2
73.93
92.61
93.01
80.48
92.84
CIFARTO0
71.36
50.64 64.64
70.15
70.02
52.41
69.19
70.48
$2.19
69.77
ResNNet56
f2
71.36
53.44
70.16
70.02
52.73
69.31
70.48
52.16
69.62
GM
71.36
45.12
70.22
70.02
52.62
69.54
70.48
50.74
69.69
ImageNet
4
73.31
62.22
73.06
73.16
54.24
72.99
73.21
63.12
73.02
ResNet34
73.31
62.02
72.91
73.16
53.64
72.78
73.21
62.98
72.86
GM
73.31
61.88
72.96
73.16
53.48
72.94
73.21
62.36
73.04
All the setting of these experiments under be found in http://poooro-.lssssifff...ee
com/
Specifically, for pruning ratio:
VGG16 CIFARIO, CIFARI00 and ImageNet:
ob/
ResNe156 CIFAR10 and CIFAR100:
ResNet34 on ImageNet:
36

---

Under review as a conference paper at ICLR 2021
O
ABOUT WEIGHT DECAY
ResNet110
WRN-28-10
1.
KS test p-value
KS test p-value
wright decay =
weight decay 1e-3
weight docay 2014 0.01
2.4
9.
woight desay
Index of convolutional layer
Index of convolutional layer
Figure 22: KS (Lilliefors) 1967) while using different settings of weight decay.
We train the ResNetl 10 and WRN-28-10 10 on CIFARI00 with different weight decay (1e-3, 3e-4 and 0) and use KS test to verify whether the parameters of different layers follow normal distribution. In Fig. we can find
(1) When weight decay (wd) is non-zero, the normality is higher than that when weight decay is O.
(2) If weight decay is O, the p-value can still be much greater than 0.05, which means that the regu- larization of weight decay may not be the key reason for CWDA. The distribution of the parameters these two networks (weight decay is 0) are shown Fig. 24 and Fig. 23
an 025-005
Froqusto
Values ON
Figure 23: The distribution of parameters in different convolutional filters (WRN-28-10, =)
Wetecoabee 375
-2
05
04
Value paramnters.
Figure 24: The distribution of parameters in different convolutional filters (ResNetl 10, wd 0).
37

---

Under review as a conference paper at ICLR 2021
P STATISTICAL TEST
In this section, according to Table and Section 2.1 we have a series of statistical tests for the necessary conditions of CWDA. let Fij R |Nxxx represent the jth filter of the ith convolutional layer.
(I) Gaussian (i.e., verify whether Fij approximatively follow Gaussian-alike distribution.). In 7th layer,
Kolmogorov-Smireg Smirnov (KS) test (Lilliefors 1967) check if all the weights in the same layer follow a normal distribution.
(2) Standard Deviation (i.e., verify whether the standard deviation of each filter in any layers tends to be a constant c.). Let o denotes the standard deviation of all the weights of filter Fig in yth layer. We use Student's test (Efron 1969) check if the variance of these o, small enough. The null hypothesis Ho and the alternative hypothesis H1 are:
Ho Var 2,, NN.) S og H1 Var(o1, 02, O2, NN.) o.
where N; denotes the number of the filters in 7th layer and do a given real number which small enough, like o 0.0001.
(3) Mean (iee,, verify whether the mean of Fig 0.). Let the mean of all the weights in the same layer is μ.. We use Student's test (Efron 1969h check if is close to O. First, we check the upper bound (Mean-Left) μ, i.e.,
Hu 14 c,,
where c is small constant, like = 0.01. Next, we check the lower bound (lean-Right) and the null hypothesis Ho and the alternative hypothesis H1 are:
Ho μ 2 -,
H1 :ppp<
Of course, the p value for Ho p 2 and Ho S should be the same. There several Notable points:
In all the statistical tests, let the confidence level be 0.95, c 0.01 and of 0.0001. Green color to represent that the convolutional filters in layer pass the statistical test. Conversely, the Red color means that the filters can not pass the tests.
In most layers, the convolutional filters can pass the statistical tests, except for a few layers which are in front of the network. This phenomenon is consistent with the analysis in Section 5.1 and it does not mean CWDA is not true.
Most of the experiments are image classification, except for the tests in Appendix P.7 P.7
p-value and c-value denote p value and critical value (confidence level is 0.95), respec- tively. t-value is t values in Student's t test. If p-value is larger than c-value or t-value is smaller than e-value, we think this filter passes the tests.
In fact, of the experiments in Table are repeated. These experiments are shown in following tables. The (+) means that for the sake of brevity, the repeat experiments are shown only once the experiment with (*).
Table 7: The repeated experiments Table
NETWORK STRUCTURE (...)
OPTIMIZER P.2
INITIALIZATION PP. kaiming-normal
(1) ResNet (*)
SGD
NETWORK STRUCTURE (P.1) WRN
REGULARIZATION P.3
(2)
L2 norm
BATCH NORMALIZATION (P.8) VGG-bn (*)
LEARNING RATE P.10 Schedule 150-225
38

---

Under review as a conference paper at ICLR 2021
1.0
0.8
Passing rate
0.6
0.4
0.01
VGG19
0.2
ResNet34
00
0.005
ResNet50
0.0
0.1
0.08
0.06
0.04
0.02
0.00
00
Figure 25: The passing rate of statistical test in Appendix 2), where 0 S 0.1 and passing rate is the ratio between the number of the filters passed the test and the number total filters.
VGG19 on ImageNet (dn)
ResNet34 on ImageNet (dx)
ResNet50 on ImageNet (a²)
Frequency (normes))
04
Frequency (normed)
B1A
012
810
Frequecyy
6ox
100
59
13
08
190
Indox convelutional layar
Indox convolutional layar
Index convolutional layar
Figure 26: The filters that can not pass the statistical test in Appendix P(2). We record the convolu- tional filters that can not pass the statistical test under all settings of do in Fig. 25 It can be found that the filters that fail to pass the test are concentrated in the first few layers. This IS consistent with the statement in Section 5.1
39

---

Under review as a conference paper at ICLR 2021
P.I
NETWORK STRUCTURE
Table 8: Cifar100 ResNet164
Layer
Number
dim
Gaussian
Mean
Migh.
Sigma
p-value
c-value
p-value
e-value
p-value
e-value
t-value
c-value
Conv1 1
16
16
0.01
0.05
0.56
0,05
0.56
0.05
448.68
26.3
Conv2
16
144
0.04
0.05
0.68
0.05
0.68
0.05
114.12
26.3
Conv3
64
16
0.08
0.05
0.63
0.05
0.63
0.05
661.15
83.68
Conv4
16
64
0.44
0.05
0.63
0.05
0.63
0.05
17.16
26.3
Conv5
16
144
0.31
0.05
0.68
0.05
0.68
0.05
21.0
26.3
Conv6
64
16
0.36
0.05
0.63
0.05
0.63
0.05
112.55
83.68
Conv7
16
64
0.58
0.05
0.63
0.05
0.63
0.05
4.73
26.3
Conv8
16
144
0.35
0.05
0.68
0.05
0.68
0.05
11.9
26.3
Conv9
64
16
0.38
0.05
0.63
0.05
0.63
0.05
57.19
83.68
Conv 10
16
64
0.69
0.05
0.63
0.05
0.63
0.05
2.61
26.3
Conv 11
16
144
0.57
0.05
0.68
0.05
0.68
0.05
5.19
26.3
Conv 12
64
16
0.52
0.05
0.63
0.05
0.63
0.05
41.82
83.68
Conv13
16
64
0.71
0.05
0.63
0.05
0.63
0.05
1.4
26.3
Conv 14
16
144
0.62
0.05
0.68
0.05
0.68
0.05
2.9
26.3
Conv 15
64
16
0.64
0.05
0.63
0.05
0.63
0.05
18.71
83.68
Conv 16
16
64
0.83
0.05
0.63
0.05
0.63
0.05
1.35
26.3
Conv1
16
0.69
0.05
0.68
0.05
0.68
0.05
1.27
26.3
Conv 18
64
16
0.84
0.05
0.63
0.05
0.63
0.05
9.65
83.68
Conv 19
16
64
0.66
0.05
0.63
0.05
0.63
0.05
2.89
26.3
Conv20
16
144
0.7
0.05
0.68
0.05
0.68
0.05
0.77
26.3
Conv21
64
16
0.82
0.05
0.63
0.05
0.63
0.05
8.08
83.68
Conv22
16
64
0.81
0.05
0.63
0.05
0.63
0.05
1.8
26.3
Conv23
16
144
0.88
0.05
0.68
0.05
0.68
0.05
0.51
26.3
Conv24
64
16
0.86
0.05
0.63
0.05
0.63
0.05
4.45
83.68
Conv25
16
64
0.87
0.05
0.63
0.05
0.63
0.05
1.24
26.3
Conv26
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.68
26.3
Conv27
64
16
0.87
0.05
0.63
0.05
0.63
0.05
4.07
83.68
Conv28
16
64
0.47
0.05
0.63
0.05
0.63
0.05
4.45
26.3
Conv29
16
144
0.64
0.05
0.68
0.05
0.68
0.05
4.66
26.3
Conv30
64
16
0.8
0.05
0.63
0.05
0.63
0.05
12.15
83.68
Conv31
16
64
0.69
0.05
0.63
0.05
0.63
0.05
2.59
26.3
Conv32
16
0.84
0.05
0.68
0.05
0.68
0.05
1.03
26.3
Conv33
64
16
0.83
0.05
0.63
0.05
0.63
0.05
10.12
83.68
Conv34
16
64
0.52
0.05
0.63
0.05
0.63
0.05
6.17
26.3
Conv35
16
144
0.38
0.05
0.68
0.05
0.68
0.05
6.31
26.3
Conv36
64
16
0.67
0.05
0.63
0.05
0.63
0.05
12.34
83.68
Conv37
16
64
0.66
0.05
0.63
0.05
0.63
0.05
5.25
26.3
Conv38
16
144
0.69
0.05
0.68
0.05
0.68
0.05
2.07
26.3
Conv39
64
16
0.84
0.05
0.63
0.05
0.63
0.05
7.37
83.68
Conv40
16
64
0.63
0.05
0.63
0.05
0.63
0.05
6.93
26.3
Conv41
16
144
0.52
0.05
0.68
0.05
0.68
0.05
4.62
26.3
Conv42
64
16
0.69
0.05
0.63
0.05
0.63
0.05
18.53
83.68
Conv43
16
64
0.64
0.05
0.63
0.05
0.63
0.05
5.38
26.3
Conv44
16
144
0.6
0.05
0.68
0.05
0.68
0.05
2.7
26.3
Conv45
64
16
0.69
0.05
0.63
0.05
0.63
0.05
17.9
83.68
Conv46
16
64
0.82
0.05
0.63
0.05
0.63
0.05
2.32
26.3
Conv47
16
144
0.82
0.05
0.68
0.05
0.68
0.05
1.98
26.3
Conv48
64
16
0.82
0.05
0.63
0.05
0.63
0.05
9.34
83.68
Conv49
16
64
0.62
0.05
0.63
0.05
0.63
0.05
5.16
26.3
40

---

Under review as a conference paper at ICLR 2021
Conv50
16
144
0.39
0.05
0.68
0.05
0.68
0.05
6.45
26.3
Conv51
64
16
0.82
0.05
0.63
0.05
0.63
0.05
14.56
83.68
Conv52
16
64
0.54
0.05
0.63
0.05
0.63
0.05
6.05
26.3
Conv53
16
144
0.81
0.05
0.68
0.05
0.68
0.05
4.12
26.3
Conv54
64
16
0.67
0.05
0.63
0.05
0.63
0.05
15.76
83.68
Conv55
32
64
0.31
0,05
0.67
0,05
0.67
0.05
26.33
46.19
Conv56
32
288
0.56
0.05
0.83
0.05
0.83
0.05
6.36
46.19
Conv57
128
32
0.47
0.05
0.74
0.05
0.74
0.05
94.73
155.4
Conv58
32
128
0.56
0.05
0.74
0.05
0.74
0.05
7.21
46.19
Conv59
32
288
0.52
0.05
0.83
0.05
0.83
0.05
2.94
46.19
Conv60
128
32
0.63
0.05
0.74
0.05
0.74
0.05
27.0
155.4
Conv61
32
128
0.69
0.05
0.74
0.05
0.74
0.05
1.27
46.19
Conv62
32
288
0.84
0.05
0.83
0.05
0.83
0.05
1.94
46.19
Conv63
128
32
0.83
0.05
0.74
0.05
0.74
0.05
17.24
155.4
Conv64
32
128
0.31
0.05
0.74
0.05
0.74
0.05
7.84
46.19
Conv65
32
288
0.41
0.05
0.83
0.05
0.83
0.05
4.86
46.19
Conv66
128
32
0.81
0.05
0.74
0.05
0.74
0.05
33.53
155.4
Conv67
32
128
0.51
0.05
0.74
0.05
0.74
0.05
5.16
46.19
Conv68
32
288
0.83
0.05
0.83
0.05
0.83
0.05
2.86
46.19
Conv69
128
32
0.68
0.05
0.74
0.05
0.74
0.05
29.78
155.4
Conv 70
32
128
0.83
0.05
0.74
0.05
0.74
0.05
1.27
46.19
Conv71
32
288
0.69
0.05
0.83
0.05
0.83
0.05
1.87
46.19
Conv72 72
128
32
0.84
0.05
0.74
0.05
0.74
0.05
9.81
155.4
Conv73
32
128
0.66
0.05
0.74
0.05
0.74
0.05
1.89
46.19
Conv74 74
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.06
46.19
Conv75 75
128
32
0.87
0.05
0.74
0.05
0.74
0.05
13.07
155.4
C6
32
128
0.83
0.05
0.74
0.05
0.74
0.05
1.93
46.19
Conv77 77
32
288
0.86
0.05
0.83
0.05
0.83
0.05
1.74
46.19
C8nv78
128
32
0.86
0.05
0.74
0.05
0.74
0.05
12.89
155.4
Conv79
32
128
0.46
0.05
0.74
0.05
0.74
0.05
5.41
46.19
Conv80
32
288
0.84
0.05
0.83
0.05
0.83
0.05
1.63
46.19
Conv81
128
32
0.7
0.05
0.74
0.05
0.74
0.05
22.23
155.4
Conv82
32
128
0.57
0.05
0.74
0.05
0.74
0.05
4.95
46.19
Conv83
32
288
0.69
0.05
0.83
0.05
0.83
0.05
2.5
46.19
Conv84
128
32
0.8
0.05
0.74
0.05
0.74
0.05
24.7
155.4
Conv85
32
128
0.82
0.05
0.74
0.05
0.74
0.05
1.88
46.19
Conv86
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.88
46.19
Conv87
128
32
0.82
0.05
0.74
0.05
0.74
0.05
19.06
155.4
Conv88
32
0.05
0.74
0.05
0.74
0.05
2.68
46.19
Conv89
32
288
0.69
0.05
0.83
0.05
0.83
0.05
2.5
46.19
Conv90
128
32
0.69
0.05
0.74
0.05
0.74
0.05
15.94
155.4
Conv91
32
128
0.68
0.05
0.74
0.05
0.74
0.05
3.88
46.19
Conv92
32
288
0.82
0.05
0.83
0.05
0.83
0.05
2.55
46.19
Conv93
128
32
0.66
0.05
0.74
0.05
0.74
0.05
19.66
155.4
Conv94
32
128
0.52
0.05
0.74
0.05
0.74
0.05
6.16
46.19
Conv95
32
288
0.6
0.05
0.83
0.05
0.83
0.05
2.52
46.19
Conv96
128
32
0.81
0.05
0.74
0.05
0.74
0.05
22.61
155.4
Conv97
32
128
0.83
0.05
0.74
0.05
0.74
0.05
3.05
46.19
Conv98
32
288
0.62
0.05
0.83
0.05
0.83
0.05
2.4
46.19
Conv99
128
32
0.7
0.05
0.74
0.05
0.74
0.05
14.79
155.4
Conv100 100
32
128
0.32
0.05
0.74
0.05
0.74
0.05
6.89
46.19
Conv 101
32
288
0.57
0.05
0.83
0.05
0.83
0.05
3.2
46.19
Conv 102v102
128
32
0.65
0.05
0.74
0.05
0.74
0.05
20.08
155.4
Conv10 103
32
128
0.65
0.05
0.74
0.05
0.74
0.05
3.7
46.19
Conv104
32
288
0.66
0.05
0.83
0.05
0.83
0.05
1.77
46.19
Conv 105
128
32
0.7
0.05
0.74
0.05
0.74
0.05
20.8
155.4
Conv 106
32
128
0.55
0.05
0.74
0.05
0.74
0.05
5.14
46.19
Conv 107
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.47
46.19
Conv108
128
32
0.55
0.05
0.74
0.05
0.74
0.05
21.77
155.4
41

---

Under review as a conference paper at ICLR 2021
Conv100
64
128
0.5
0.05
0.82
0.05
0.82
0.05
8.73
83.68
Conv
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.37
83.68
Conv 1 11
256
64
0.47
0.05
0.9
0.05
0.9
0.05
90.92
294.32
Conv1
64
256
0.67
0.05
0.9
0.05
0.9
0.05
2.2
83.68
Conv1 11 13
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.43
83.68
Conv 114
256
64
0.66
0.05
0.9
0.05
0.9
0.05
26.24
294.32
Conv 115
64
256
0.63
0.05
0.9
0.05
0.9
0.05
1.96
83.68
Conv 1 16
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.45
83.68
Conv1
256
64
0.63
0.05
0.9
0.05
0.9
0.05
23.12
294.32
Conv 18
64
256
0.81
0.05
0.9
0.05
0.9
0.05
1.66
83.68
Conv
576
0.82
0.05
0.97
0.05
0.97
0.05
1.15
83.68
Conv 20
256
64
0.8
0.05
0.9
0.05
0.9
0.05
18.65
294.32
Conv 121
64
256
0.71
0.05
0.9
0.05
0.9
0.05
2.41
83.68
Conv 122
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.07
83.68
Conv 123
256
64
0.66
0.05
0.9
0.05
0.9
0.05
19.45
294.32
Conv 1 124
64
256
0.81
0.05
0.9
0.05
0.9
0.05
1.33
83.68
Conv 125
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.86
83.68
Conv 126
256
64
0.81
0.05
0.9
0.05
0.9
0.05
17.83
294.32
Conv 127
64
256
0.83
0.05
0.9
0.05
0.9
0.05
1.5
83.68
Conv 128
64
576
0.86
0.05
0.97
0.05
0.97
0.05
1.33
83.68
Conv 129
256
64
0.81
0.05
0.9
0.05
0.9
0.05
17.73
294.32
Conv 1 30
64
256
0.83
0.05
0.9
0.05
0.9
0.05
1.45
83.68
Conv 131
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.89
83.68
Conv 132
256
64
0.68
0.05
0.9
0.05
0.9
0.05
18.82
294.32
Conv 133
64
256
0.69
0.05
0.9
0.05
0.9
0.05
1.7
83.68
Conv 134
64
576
0.83
0.05
0.97
0.05
0.97
0.05
1.05
83.68
Conv 135
256
64
0.66
0.05
0.9
0.05
0.9
0.05
19.59
294.32
Conv 136
64
256
0.84
0.05
0.9
0.05
0.9
0.05
1.48
83.68
Conv137
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.62
83.68
Conv 1 138
256
64
0.67
0.05
0.9
0.05
0.9
0.05
17.88
294.32
Conv 139
64
256
0.81
0.05
0.9
0.05
0.9
0.05
1.49
83.68
Conv 140
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.59
83.68
Conv 141
256
64
0.82
0.05
0.9
0.05
0.9
0.05
18.42
294.32
Conv 142
64
256
0.67
0.05
0.9
0.05
0.9
0.05
1.39
83.68
Conv 143
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.68
83.68
Conv 144
256
64
0.8
0.05
0.9
0.05
0.9
0.05
19.86
294.32
Conv 45
64
256
0.63
0.05
0.9
0.05
0.9
0.05
1.76
83.68
Conv 146
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.77
83.68
Conv 147
256
64
0.81
0.05
0.9
0.05
0.9
0.05
21.98
294.32
Conv 148
64
256
0.65
0.05
0.9
0.05
0.9
0.05
1.51
83.68
Conv 149
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.82
83.68
Conv Mnn
256
64
0.65
0.05
0.9
0.05
0.9
0.05
24.23
294.32
Conv 151
64
256
0.68
0.05
0.9
0.05
0.9
0.05
1.8
83.68
Conv 152
64
576
0.83
0.05
0.97
0.05
0.97
0.05
1.09
83.68
Conv 153
256
64
0.63
0.05
0.9
0.05
0.9
0.05
22.33
294.32
Conv 154
64
256
0.69
0.05
0.9
0.05
0.9
0.05
1.2
83.68
Conv 1 155
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.36
83.68
Conv 156
256
64
0.53
0.05
0.9
0.05
0.9
0.05
31.15
294.32
Conv 157
64
256
0.64
0.05
0.9
0.05
0.9
0.05
1.6
83.68
Conv 158
64
576
0.82
0.05
0.97
0.05
0.97
0.05
1.88
83.68
Conv 159
256
64
0.49
0.05
0.9
0.05
0.9
0.05
43.56
294.32
Conv 160
64
256
0.5
0.05
0.9
0.05
0.9
0.05
1.94
83.68
Conv161
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.36
83.68
Conv162
256
64
0.36
0.05
0.9
0.05
0.9
0.05
137.02
294.32
Passing rate
98.77%
100.0%
100.0%
97.53%
42

---

Under review as a conference paper at ICLR 2021
Table 9: Cifar100 VGG19
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.24
0.05
0.66
0.05
0.66
0.05
396.02
83.68
Conv2
64
576
0.51
0.05
0.97
0.05
0.97
0,05
1.73
83.68
Conv3
128
576
0.82
0.05
1.00
0.05
1.00
0.05
2.7
155.4
Conv4
128
1152
0.84
0.05
1.00
0.05
1.00
0.05
0.37
155,4
Conv5
256
1152
0.86
0.05
1.00
0.05
1.00
0.05
1.38
294.32
Conv6
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.6
294.32
Conv7 7
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.38
294.32
Conv8
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.26
294.32
Conv9
512
2304
0.92
0.05
1.00
0.05
1.00
0.05
1.35
565.75
Conv10
512
4608
0.93
0.05
1.00
0.05
1.00
0,05
0.95
565.75
Conv 11
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.21
565.75
Conv 12
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.21
565.75
Conv 13
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.19
565.75
Conv 14
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.15
565.75
Conv15 15
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.11
565.75
Conv16
512
4608
0.93
0.05
1.00
0.05
1.00
0.05
0.37
565.75
Passing rate
100.00%
100.00%
100.00%
93.75%
Table 10: Cifar100 AlexNet
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
64
363
0.51
0.05
0.94
0.05
0.94
0.05
54.46
83.68
Conv2
192
1600
0.39
0.05
1.00
0.05
1,00
0.05
44.19
225.33
Conv3
384
1728
0.66
0.05
1.00
0.05
1.00
0.05
6.62
430.69
Conv4
256
3456
0.87
0.05
1.00
0.05
1.00
0.05
1.40
294.32
Conv5
256
2304
0.85
0.05
1.00
0.05
1.00
0.05
8.74
294.32
Passing rate
100.00%
100.00%
100.00%
100.00%
Table 11: Cifar100 DenseNet-he- 100-10
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
e-value
t-value
c-value
Conv1
24
27
0.06
0.05
0.60
0.05
0.60
0.05
294.0
36.42
Conv2
48
24
0.64
0.05
0.63
0.05
0.63
0.05
21.13
65.17
Conv3
12
432
0.63
0.05
0.76
0.05
0.76
0.05
0.52
21.03
Conv4
48
36
0.63
0.05
0.66
0.05
0.66
0.05
4.04
65.17
Conv5
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.16
21.03
Conv6
48
48
0.54
0.05
0.68
0.05
0.68
0.05
12.54
65.17
Conv7
12
432
0.62
0.05
0.76
0.05
0.76
0.05
0.61
21.03
Conv8
48
60
0.64
0.05
0.70
0.05
0.70
0.05
3.88
65.17
Conv9
12
432
0.86
0.05
0.76
0.05
0.76
0.05
0.21
21.03
Conv10
48
72
0.67
0.05
0.72
0.05
0.72
0.05
2.61
65.17
Conv11 11
12
432
0.71
0.05
0.76
0.05
0.76
0.05
0.12
21.03
Co
48
84
0.62
0.05
0.74
0.05
0.74
0.05
3.88
65.17
43

---

Under review as a conference paper at ICLR 2021
Conv 13
12
432
0.85
0.05
0.76
0.05
0.76
0.05
0.59
21.03
Conv 14
48
96
0.82
0.05
0.75
0.05
0.75
0.05
1.89
65.17
Conv 15
12
432
0.86
0.05
0.76
0.05
0.76
0.05
0.27
21.03
Conv 16
48
108
0.70
0.05
0.76
0.05
0.76
0.05
3.75
65.17
Conv17
12
432
0.86
0.05
0.76
0.05
0.76
0.05
0.28
21.03
Conv 18
48
120
0.85
0,05
0.78
0,05
0.78
0.05
2.31
65.17
Conv19
12
432
0.82
0.05
0.76
0.05
0.76
0.05
0.33
21.03
Conv20
48
132
0.59
0.05
0.79
0.05
0.79
0.05
2.91
65.17
Conv21
12
432
0.71
0.05
0.76
0.05
0.76
0.05
0.61
21.03
Conv22
48
144
0.66
0.05
0.80
0.05
0.80
0.05
3.15
65.17
Conv23
12
432
0.86
0.05
0.76
0.05
0.76
0.05
0.36
21.03
Conv24
48
156
0.64
0.05
0.81
0.05
0.81
0.05
1.35
65.17
Conv25
12
432
0.85
0.05
0.76
0.05
0.76
0.05
0.24
21.03
Conv26
48
168
0.70
0.05
0.82
0.05
0.82
0.05
1.15
65.17
Conv27
12
432
0.88
0.05
0.76
0.05
0.76
0.05
0.09
21.03
Conv28
48
180
0.60
0.05
0.82
0.05
0.82
0.05
1.84
65.17
Conv29
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.19
21.03
Conv30
48
192
0.69
0.05
0.83
0.05
0.83
0.05
2.52
65.17
Conv31
12
432
0.87
0.05
0.76
0.05
0.76
0.05
0.11
21.03
Conv32
48
204
0.61
0.05
0.84
0.05
0.84
0.05
1.71
65.17
Conv33
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.33
21.03
Conv34
108
216
0.29
0.05
0.94
0.05
0.94
0.05
5.21
133.26
Conv35
48
108
0.67
0.05
0.76
0.05
0.76
0.05
1.89
65.17
C6nv36
12
432
0.70
0.05
0.76
0.05
0.76
0.05
0.41
21.03
Conv37
48
120
0.67
0.05
0.78
0.05
0.78
0.05
1.54
65.17
Conv38
12
432
0.84
0.05
0.76
0.05
0.76
0.05
0.13
21.03
Conv39
48
132
0.83
0.05
0.79
0.05
0.79
0.05
2.06
65.17
Conv40
12
432
0.82
0.05
0.76
0.05
0.76
0.05
0.67
21.03
Conv41
48
144
0.68
0.05
0.80
0.05
0.80
0.05
1.63
65.17
Conv42
12
432
0.81
0.05
0.76
0.05
0.76
0.05
0.29
21.03
Conv43
48
156
0.67
0.05
0.81
0.05
0.81
0.05
1.16
65.17
Conv44
12
432
0.85
0.05
0.76
0.05
0.76
0.05
0.13
21.03
Conv45
48
168
0.65
0.05
0.82
0.05
0.82
0.05
1.71
65.17
Conv46
12
432
0.69
0.05
0.76
0.05
0.76
0.05
0.35
21.03
Conv47
48
180
0.82
0.05
0.82
0.05
0.82
0.05
1.19
65.17
Conv48
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.14
21.03
Conv49
192
0.84
0.05
0.83
0.05
0.83
0.05
0.93
65.17
Conv50
12
432
0.67
0.05
0.76
0.05
0.76
0.05
0.36
21.03
Conv5
48
204
0.81
0.05
0.84
0.05
0.84
0.05
1.11
65.17
Conv52
12
432
0.81
0.05
0.76
0.05
0.76
0.05
0.25
21.03
Conv53
48
216
0.84
0.05
0.85
0.05
0.85
0.05
0.66
65.17
Conv54
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.2
21.03
Conv55
48
228
0.80
0.05
0.85
0.05
0.85
0.05
1.14
65.17
Conv56
12
432
0.65
0.05
0.76
0.05
0.76
0.05
0.37
21.03
Conv57
48
240
0.82
0.05
0.86
0.05
0.86
0.05
0.91
65.17
Conv58
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.25
21.03
Conv59
48
252
0.67
0.05
0.86
0.05
0.86
0.05
0.98
65.17
Conv60
12
432
0.66
0.05
0.76
0.05
0.76
0.05
0.41
21.03
Conv61
48
264
0.67
0.05
0.87
0.05
0.87
0.05
0.97
65.17
Conv62
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.24
21.03
Conv63
48
276
0.83
0.05
0.88
0.05
0.88
0.05
0.66
65.17
Conv64
12
432
0.85
0.05
0.76
0.05
0.76
0.05
0.13
21.03
Conv65
48
288
0.83
0.05
0.88
0.05
0.88
0.05
0.73
65.17
Conv66
12
432
0.67
0.05
0.76
0.05
0.76
0.05
0.15
21.03
Conv67
150
300
0.51
0.05
0.98
0.05
0.98
0.05
1.5
179.58
Conv68
48
150
0.66
0.05
0.80
0.05
0.80
0.05
0.93
65.17
Conv69
12
432
0.69
0.05
0.76
0.05
0.76
0.05
0.34
21.03
Conv70
48
162
0.63
0.05
0.81
0.05
0.81
0.05
1.04
65.17
Conv71
12
432
0.65
0.05
0.76
0.05
0.76
0.05
0.18
21.03
44

---

Under review as a conference paper at ICLR 2021
Conv72
48
174
0.84
0.05
0.82
0.05
0.82
0.05
0.71
65.17
Conv
12
432
0.82
0.05
0.76
0.05
0.76
0.05
0.53
21.03
Conv74
48
186
0.80
0.05
0.83
0.05
0.83
0.05
0.74
65.17
Conv75
12
432
0.68
0.05
0.76
0.05
0.76
0.05
0.65
21.03
76
48
198
0.33
0.05
0.84
0.05
0.84
0.05
0.9
65.17
Conv77 77
12
432
0.63
0,05
0.76
0,05
0.76
0.05
1.07
21.03
Conv78
48
210
0.82
0.05
0.84
0.05
0.84
0.05
0.77
65.17
Conv79
12
432
0.81
0.05
0.76
0.05
0.76
0.05
0.31
21.03
Conv80
48
222
0.71
0.05
0.85
0.05
0.85
0.05
0.48
65.17
Conv81
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.23
21.03
Conv82
48
234
0.82
0.05
0.86
0.05
0.86
0.05
0.79
65.17
Conv83
12
432
0.84
0.05
0.76
0.05
0.76
0.05
0.37
21.03
Conv84
48
246
0.82
0.05
0.86
0.05
0.86
0.05
1.21
65.17
Conv85
12
432
0.84
0.05
0.76
0.05
0.76
0.05
0.07
21.03
Conv86
48
258
0.84
0.05
0.87
0.05
0.87
0.05
0.85
65.17
Conv87
12
432
0.84
0.05
0.76
0.05
0.76
0.05
0.2
21.03
Conv88
48
270
0.80
0.05
0.87
0.05
0.87
0.05
0.88
65.17
Conv89
12
432
0.84
0.05
0.76
0.05
0.76
0.05
0.05
21.03
Conv90
48
282
0.82
0.05
0.88
0.05
0.88
0.05
0.88
65.17
Conv91
12
432
0.86
0.05
0.76
0.05
0.76
0.05
0.04
21.03
Conv92
48
294
0.83
0.05
0.88
0.05
0.88
0.05
0.75
65.17
Conv93
12
432
0.86
0.05
0.76
0.05
0.76
0.05
0.09
21.03
Conv94
48
306
0.81
0.05
0.89
0.05
0.89
0.05
0.79
65.17
Conv95
12
432
0.85
0.05
0.76
0.05
0.76
0.05
0.04
21.03
Conv96
48
318
0.64
0.05
0.89
0.05
0.89
0.05
0.87
65.17
Conv97
12
432
0.83
0.05
0.76
0.05
0.76
0.05
0.09
21.03
Conv98
48
330
0.82
0.05
0.90
0.05
0.90
0.05
0.94
65.17
Conv99
12
432
0.81
0.05
0.76
0.05
0.76
0.05
0.06
21.03
Passing rate
100.0%
100.0%
100.0%
98.99%
Table 12: Cifar100 PreResNetl
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1 1
16
27
0.05
0.05
0.58
0.05
0.58
0.05
96.78
26.3
Conv2
16
144
0.47
0.05
0.68
0.05
0.68
0.05
26.3
Conv3
16
144
0.64
0.05
0.68
0.05
0.68
0.05
3.3
26.3
Conv4
16
144
0.49
0.05
0.68
0.05
0.68
0.05
2.8
26.3
5
16
144
0.58
0.05
0.68
0.05
0.68
0.05
4.53
26.3
Conv6
16
144
0.53
0.05
0.68
0.05
0.68
0.05
1.88
26.3
Conv7
16
144
0.59
0.05
0.68
0.05
0.68
0.05
1.29
26.3
Conv8
16
144
0.67
0.05
0.68
0.05
0.68
0.05
0.69
26.3
Conv9
16
144
0.70
0.05
0.68
0.05
0.68
0.05
1.2
26.3
Conv10 10
16
144
0.58
0.05
0.68
0.05
0.68
0.05
2.41
26.3
Conv
16
144
0.62
0.05
0.68
0.05
0.68
0.05
1.5
26.3
Conv 12
16
144
0.69
0.05
0.68
0.05
0.68
0,05
0.77
26.3
Conv 13
16
144
0.62
0.05
0.68
0.05
0.68
0.05
2.18
26.3
Conv 14
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.63
26.3
Conv15 15
16
144
0.67
0.05
0.68
0.05
0.68
0.05
1.66
26.3
Conv 16
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.87
26.3
Conv 17
16
144
0.67
0.05
0.68
0.05
0.68
0.05
1.22
26.3
Conv 18
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.72
26.3
Conv 19
16
144
0.82
0.05
0.68
0.05
0.68
0.05
1.29
26.3
Conv20
16
144
0.66
0.05
0.68
0.05
0.68
0.05
1.07
26.3
Conv21
16
144
0.83
0.05
0.68
0.05
0.68
0.05
1.64
26.3
45

---

Under review as a conference paper at ICLR 2021
Conv22
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.93
26.3
Conv23
16
144
0.70
0.05
0.68
0.05
0.68
0.05
1.76
26.3
Conv24
16
144
0.82
0.05
0.68
0.05
0.68
0.05
0.9
26.3
Conv25
16
144
0.83
0.05
0.68
0.05
0.68
0.05
1.51
26.3
Conv26
16
144
0.68
0.05
0.68
0.05
0.68
0.05
1.28
26.3
Conv27
16
144
0.81
0,05
0.68
0,05
0.68
0.05
2.35
26.3
Conv28
16
144
0.68
0.05
0.68
0.05
0.68
0.05
1.25
26.3
Conv29
16
144
0.80
0.05
0.68
0.05
0.68
0.05
2.06
26.3
Conv30
16
144
0.69
0.05
0.68
0.05
0.68
0.05
2.32
26.3
Conv31
16
144
0.85
0.05
0.68
0.05
0.68
0.05
0.81
26.3
Conv32
16
0.56
0.05
0.68
0.05
0.68
0.05
5.68
26.3
Conv33
16
144
0.49
0.05
0.68
0.05
0.68
0.05
9.37
26.3
Conv34
16
144
0.58
0.05
0.68
0.05
0.68
0.05
3.73
26.3
C5
16
144
0.83
0.05
0.68
0.05
0.68
0.05
0.76
26.3
Conv36
16
144
0.49
0.05
0.68
0.05
0.68
0.05
7.17
26.3
Conv37
16
144
0.55
0.05
0.68
0.05
0.68
0.05
4.83
26.3
Conv38
32
144
0.45
0.05
0.75
0.05
0.75
0.05
2.35
46.19
Conv39
32
288
0.54
0.05
0.83
0.05
0.83
0.05
1.89
46.19
Conv40
32
288
0.63
0.05
0.83
0.05
0.83
0.05
1.25
46.19
Conv41
32
288
0.69
0.05
0.83
0.05
0.83
0.05
0.91
46.19
Conv42
32
288
0.71
0.05
0.83
0.05
0.83
0.05
0.58
46.19
Conv43
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.45
46.19
Conv44
32
288
0.56
0.05
0.83
0.05
0.83
0.05
1.58
46.19
Conv45
32
288
0.67
0.05
0.83
0.05
0.83
0.05
0.74
46.19
Conv46
32
288
0.81
0.05
0.83
0.05
0.83
0.05
0.81
46.19
Conv47
32
288
0.70
0.05
0.83
0.05
0.83
0.05
0.59
46.19
Conv48
32
288
0.65
0.05
0.83
0.05
0.83
0.05
1.22
46.19
Conv49
32
288
0.80
0.05
0.83
0.05
0.83
0.05
0.34
46.19
Conv50
32
288
0.69
0.05
0.83
0.05
0.83
0.05
1.18
46.19
Conv51
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.74
46.19
Conv52
32
288
0.67
0.05
0.83
0.05
0.83
0.05
1.18
46.19
Conv53
32
288
0.85
0.05
0.83
0.05
0,83
0.05
0.36
46.19
Conv54
32
288
0.55
0.05
0.83
0.05
0.83
0.05
2.03
46.19
Conv55
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.88
46.19
Conv56
32
288
0.70
0.05
0.83
0.05
0.83
0.05
1.14
46.19
Conv57
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.42
46.19
Conv58
288
0.69
0.05
0.83
0.05
0.83
0.05
1.51
46.19
Conv59
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.57
46.19
Conv60
32
288
0.69
0.05
0.83
0.05
0.83
0.05
0.9
46.19
Conv61
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.43
46.19
Conv62
32
288
0.68
0.05
0.83
0.05
0.83
0.05
1.17
46.19
Conv63
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.61
46.19
Conv64
32
288
0.60
0.05
0.83
0.05
0.83
0.05
1.18
46.19
Conv65
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.6
46.19
Conv66
32
288
0.67
0.05
0.83
0.05
0.83
0.05
1.04
46.19
Conv67
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.41
46.19
Conv68
32
288
0.64
0.05
0.83
0.05
0.83
0.05
1.14
46.19
Conv69
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.58
46.19
Conv70 70
32
288
0.69
0.05
0.83
0.05
0.83
0.05
0.9
46.19
Conv71
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.48
46.19
Conv72
32
288
0.69
0.05
0.83
0.05
0.83
0.05
1.2
46.19
Conv73 73
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.38
46.19
Conv74
64
288
0.41
0.05
0.91
0.05
0.91
0.05
1.27
83.68
Cony75
64
576
0.50
0.05
0.97
0.05
0.97
0.05
1.01
83.68
C6
64
576
0.68
0.05
0.97
0.05
0.97
0.05
1.39
83.68
Conv7 77
64
576
0.80
0.05
0.97
0.05
0.97
0.05
0.96
83.68
Conv78
64
576
0.64
0.05
0.97
0.05
0.97
0.05
1.26
83.68
Conv79
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.99
83.68
Conv80
64
576
0.71
0.05
0.97
0.05
0.97
0.05
0.83
83.68
46

---

Under review as a conference paper at ICLR 2021
Conv81
64
576
0.82
0.05
0.97
0.05
0.97
0.05
0.74
83.68
Conv82
64
576
0.64
0.05
0.97
0.05
0.97
0.05
1.46
83.68
Conv83
64
576
0.71
0.05
0.97
0.05
0.97
0.05
0.75
83.68
Conv84
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.02
83.68
Conv85
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.64
83.68
Conv86
64
576
0.70
0.05
0.97
0.05
0.97
0.05
1.08
83.68
Conv87
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.44
83.68
Conv88
64
576
0.81
0.05
0.97
0.05
0.97
0.05
0.71
83.68
Conv89
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.53
83.68
Conv90
64
576
0.65
0.05
0.97
0.05
0.97
0.05
0.89
83.68
Conv91
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.39
83.68
Conv92
64
576
0.80
0.05
0.97
0.05
0.97
0.05
0.83
83.68
Conv93
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.4
83.68
Conv94
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.58
83.68
Conv95
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.38
83.68
Conv96
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.77
83.68
Conv97
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.37
83.68
Conv98
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.76
83.68
Conv99
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.35
83.68
Conv 100
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.79
83.68
Conv 101
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.42
83.68
Conv 102
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.62
83.68
Conv 103
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.42
83.68
Conv 104
64
576
0.84
0.05
0.97
0.05
0.97
0.05
83.68
Conv 105
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.55
83.68
Conv 106
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.62
83.68
Conv 107
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.6
83.68
Conv 108
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.38
83.68
10nv109
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.47
83.68
Passing rate
100.0%
100.0%
100.0%
99.08%
Table 13: Cifar100 WRN28-10
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.44
0.05
0.58
0.05
0.58
0.05
24.87
26.3
Conv2
160
144
0.92
0.05
0.94
0.05
0.94
0.05
5.96
190.52
Conv3
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.1
190.52
Conv4
160
16
0.67
0.05
0.69
0.05
0.69
0.05
35.0
190.52
Conv5
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.48
190.52
Conv6
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.12
190.52
Conv7
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.03
190.52
Conv8
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.23
190.52
Conv9
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.02
190.52
Conv 10
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.39
190.52
Conv
320
1440
0.95
0.05
1.00
0.05
1.00
0,05
1.13
362.72
Conv 12
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.08
362.72
Conv 13
320
160
0.88
0.05
0.99
0.05
0.99
0.05
2.81
362.72
Conv 14
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv 16
320
2880
0.95
0.05
1.00
0.05
1,00
0.05
0.03
362.72
Conv 17
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.12
362.72
Conv 18
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv19
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.3
362.72
Conv20
640
2880
0.94
0.05
1.00
0.05
1.00
0.05
2.18
699.96
47

---

Under review as a conference paper at ICLR 2021
Conv21
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.26
699.96
Conv22
640
320
0.91
0.05
1.00
0.05
1.00
0.05
1.91
699.96
Conv23
640
5760
0.94
0.05
1.00
0.05
1.00
0,05
0.35
699.96
Conv24
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.24
699.96
Conv25
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.06
699.96
Conv26
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.36
699.96
Conv27
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.04
699.96
Conv28
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
1.23
699.96
Passing rate
100.0%
100.0%
100.0%
100.0%
Table 14: Cifar100 ResNext- 16x64d
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.70
0.05
0.66
0.05
0.66
0.05
32.22
83.68
Conv2
1024
64
0.91
0.05
0.99
0.05
0.99
0.05
1.51
1099.56
Conv3
1024
576
0.91
0.05
1.00
0.05
1.00
0.05
0.41
1099.56
Conv4
256
1024
0.93
0.05
1.00
0.05
1.00
0.05
0.12
294.32
C
256
64
0.86
0.05
0.90
0.05
0.90
0.05
5.91
294.32
Conv6
1024
256
0.95
0.05
1.00
0.05
1.00
0.05
0.13
1099.56
Conv7
1024
576
0.95
0.05
1.00
0.05
1.00
0.05
0.04
1099.56
Conv8
256
1024
0.94
0.05
1.00
0.05
1.00
0.05
0.03
294.32
Conv9
1024
256
0.93
0.05
1.00
0.05
1.00
0.05
0.77
1099.56
Conv 10
1024
576
0.93
0.05
1.00
0.05
1.00
0.05
0.46
1099.56
Conv
256
1024
0.95
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv 12
2048
256
0.95
0.05
1.00
0.05
1.00
0.05
1.79
2154.4
Conv 13
2048
1152
0.96
0.05
1.00
0.05
1.00
0.05
0.55
2154.4
Conv 14
512
2048
0.94
0.05
1.00
0.05
1.00
0,05
0.6
565.75
Conv15 15
512
256
0.93
0.05
1.00
0.05
1.00
0.05
2.92
565.75
Conv16
2048
512
0.94
0.05
1.00
0.05
1.00
0.05
0.94
2154.4
Conv 17
2048
1152
0.95
0.05
1.00
0.05
1.00
0.05
0.82
2154.4
Conv18
512
2048
0.94
0.05
1.00
0.05
1.00
0.05
0.4
565.75
Conv 19
2048
512
0.93
0.05
1.00
0.05
1.00
0.05
1.55
2154.4
Conv20
2048
1152
0.95
0.05
1.00
0.05
1.00
0.05
0.95
2154.4
Conv21
512
2048
0.95
0.05
1.00
0.05
1.00
0.05
0.39
565.75
Conv22
4096
512
0.92
0.05
1.00
0.05
1.00
0.05
3.13
4246.0
Conv23
4096
2304
0.95
0.05
1.00
0.05
1.00
0.05
0.65
4246.0
Conv24
1024
4096
0.92
0.05
1.00
0.05
1.00
0.05
0.64
1099.56
Conv25
1024
512
0.94
0.05
1.00
0.05
1.00
0.05
2.9
1099.56
Conv26
4096
1024
0.88
0.05
1.00
0.05
1.00
0.05
1.72
4246.0
Cony27
4096
2304
0.94
0.05
1.00
0.05
1.00
0.05
1.54
4246.0
Conv28
1024
4096
0.94
0.05
1.00
0.05
1.00
0.05
0.24
1099.56
Conv29
4096
1024
0.95
0.05
1.00
0.05
1.00
0.05
2.25
4246.0
Conv30
4096
2304
0.96
0.05
1.00
0.05
1.00
0.05
1.03
4246.0
Conv31
1024
4096
0.94
0.05
1.00
0.05
1.00
0.05
0.22
1099.56
Passing rate
100.0%
100.0%
100.0%
100.0%
P.2 OPTIMIZER
config:
ht https///hhhhi..oooaaaap///////ea
48

---

Under review as a conference paper at ICLR 2021
Table 15: Cifar100 -eGD-ResNet164
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.13
0.05
0.58
0.05
0.58
0.05
34.35
26.3
Conv2
16
16
0.33
0.05
0.56
0.05
0.56
0.05
9.29
26.3
Conv3
16
144
0.65
0.05
0.68
0.05
0.68
0.05
0.42
26.3
Conv4
64
16
0.58
0,05
0.63
0.05
0.63
0.05
13.54
83.68
Conv5
16
64
0.24
0.05
0.63
0.05
0.63
0.05
2.95
26.3
Conv6
16
144
0.68
0.05
0.68
0.05
0.68
0.05
0.21
26.3
Conv7
64
16
0.58
0.05
0.63
0.05
0.63
0.05
16.09
83.68
Conv8
16
64
0.29
0.05
0.63
0.05
0.63
0.05
4.05
26.3
Conv9
16
0.65
0.05
0.68
0.05
0.68
0.05
0.15
26.3
Conv 10
64
16
0.60
0.05
0.63
0.05
0.63
0.05
15.79
83.68
Conv 11
16
64
0.29
0.05
0.63
0.05
0.63
0.05
5.69
26.3
Conv 12
16
144
0.70
0.05
0.68
0.05
0.68
0.05
0.28
26.3
Conv 13
64
16
0.61
0.05
0.63
0.05
0.63
0.05
20.88
83.68
Conv 14
16
64
0.29
0.05
0.63
0.05
0.63
0.05
8.67
26.3
Conv 15
16
144
0.65
0.05
0.68
0.05
0.68
0.05
0.18
26.3
Conv 16
64
16
0.60
0.05
0.63
0.05
0.63
0.05
17.85
83.68
Conv17 17
16
64
0.29
0.05
0.63
0.05
0.63
0.05
5.21
26.3
Conv 18
16
144
0.65
0.05
0.68
0.05
0.68
0.05
0.23
26.3
Conv 19
64
16
0.49
0.05
0.63
0.05
0.63
0.05
21.6
83.68
Conv20
16
64
0.24
0.05
0.63
0.05
0.63
0.05
2.93
26.3
Conv21
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.15
26.3
Conv22
64
16
0.58
0.05
0.63
0.05
0.63
0.05
15.96
83.68
Conv23
16
64
0.30
0.05
0.63
0.05
0.63
0.05
4.21
26.3
Conv24
16
0.71
0.05
0.68
0.05
0.68
0.05
0.22
26.3
Conv25
64
16
0.58
0.05
0.63
0.05
0.63
0.05
15.38
83.68
Conv26
16
64
0.29
0.05
0.63
0.05
0.63
0.05
5.46
26.3
Conv27
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.2
26.3
Conv28
64
16
0.60
0.05
0.63
0.05
0.63
0.05
17.53
83.68
Conv29
16
64
0.36
0.05
0.63
0.05
0.63
0.05
3.99
26.3
Conv30
16
144
0.70
0.05
0.68
0.05
0.68
0.05
0.25
26.3
Conv31
64
16
0.58
0.05
0.63
0.05
0.63
0.05
17.13
83.68
Conv32
16
64
0.24
0.05
0.63
0.05
0.63
0.05
10.61
26.3
Conv33
16
144
0.71
0.05
0.68
0.05
0.68
0.05
0.21
26.3
Conv34
64
16
0.63
0.05
0.63
0.05
0.63
0.05
21.96
83.68
Conv35
16
64
0.18
0.05
0.63
0.05
0.63
0.05
3.58
26.3
Conv36
16
144
0.68
0.05
0.68
0.05
0.68
0.05
0.33
26.3
Conv37
64
16
0.46
0.05
0.63
0.05
0.63
0.05
21.7
83.68
Conv38
16
64
0.28
0.05
0.63
0.05
0.63
0.05
5.06
26.3
Conv39
16
144
0.70
0.05
0.68
0.05
0.68
0.05
0.27
26.3
Conv40
64
16
0.60
0.05
0.63
0.05
0.63
0.05
23.91
83.68
Conv41
16
64
0.30
0.05
0.63
0.05
0.63
0.05
5.54
26.3
Conv42
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.27
26.3
Conv43
64
16
0.59
0.05
0.63
0.05
0.63
0.05
20.08
83.68
Conv44
16
64
0.24
0.05
0.63
0.05
0.63
0.05
2.2
26.3
Conv45
16
144
0.64
0.05
0.68
0.05
0.68
0.05
0.2
26.3
Conv46
64
16
0.57
0.05
0.63
0.05
0.63
0.05
15.52
83.68
Conv47
16
64
0.31
0.05
0.63
0.05
0.63
0.05
1.59
26.3
Conv48
16
144
0.67
0.05
0.68
0.05
0.68
0.05
0.23
26.3
Conv49
64
16
0.60
0.05
0.63
0.05
0.63
0.05
22.5
83.68
Conv50
16
64
0.27
0.05
0.63
0.05
0.63
0.05
4.37
26.3
Conv51
16
144
0.68
0.05
0.68
0.05
0.68
0.05
0.07
26.3
Conv52
64
16
0.61
0.05
0.63
0.05
0.63
0.05
20.81
83.68
Conv53
16
64
0.27
0.05
0.63
0.05
0.63
0.05
5.5
26.3
49

---

Under review as a conference paper at ICLR 2021
Conv54
16
144
0.67
0.05
0.68
0.05
0.68
0.05
0.36
26.3
Conv55
64
16
0.57
0.05
0.63
0.05
0.63
0.05
20.01
83.68
Conv56
32
64
0.48
0.05
0.67
0.05
0.67
0.05
4.04
46.19
Conv57
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.13
46.19
Conv58
128
32
0.66
0.05
0.74
0.05
0.74
0.05
10.05
155.4
Conv59
32
128
0.40
0,05
0.74
0,05
0.74
0.05
3.91
46.19
Conv60
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.2
46.19
Conv61
128
32
0.68
0.05
0.74
0.05
0.74
0.05
9.64
155.4
Conv62
32
128
0.31
0.05
0.74
0.05
0.74
0.05
2.19
46.19
Conv63
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.15
46.19
Conv64
128
32
0.66
0.05
0.74
0.05
0.74
0.05
10.3
155.4
Conv65
32
128
0.40
0.05
0.74
0.05
0.74
0.05
3.61
46.19
Conv66
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.12
46.19
Conv67
128
32
0.60
0.05
0.74
0.05
0.74
0.05
10.61
155.4
Conv68
32
128
0.37
0.05
0.74
0.05
0.74
0.05
1.97
46.19
Conv69
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.14
46.19
Conv70 70
128
32
0.70
0.05
0.74
0.05
0.74
0.05
9.04
155.4
Conv71
32
128
0.44
0.05
0.74
0.05
0.74
0.05
2.21
46.19
Conv 72
32
288
0.81
0.05
0.83
0.05
0.83
0.05
0.19
46.19
Conv73
128
32
0.68
0.05
0.74
0.05
0.74
0.05
9.69
155.4
Conv 74
32
128
0.40
0.05
0.74
0.05
0.74
0.05
1.31
46.19
Conv75
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.1
46.19
Conv76
128
32
0.68
0.05
0.74
0.05
0.74
0.05
9.75
155.4
Conv77
32
128
0.43
0.05
0.74
0.05
0.74
0.05
2.52
46.19
Conv78
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.17
46.19
Conv79
128
32
0.67
0.05
0.74
0.05
0.74
0.05
10.85
155.4
Conv80
32
128
0.38
0.05
0.74
0.05
0.74
0.05
2.17
46.19
Conv81
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.12
46.19
Conv82
128
32
0.68
0.05
0.74
0.05
0.74
0.05
9.48
155.4
Conv83
32
128
0.41
0.05
0.74
0.05
0.74
0.05
3.89
46.19
Conv84
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.16
46.19
Conv85
128
32
0.62
0.05
0.74
0.05
0.74
0.05
11.02
155.4
Conv86
32
128
0.39
0.05
0.74
0.05
0.74
0.05
2.04
46.19
Conv87
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.11
46.19
Conv88
128
32
0.66
0.05
0.74
0.05
0.74
0.05
10.19
155.4
Conv89
32
128
0.37
0.05
0.74
0.05
0.74
0.05
2.54
46.19
Conv90
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.09
46.19
Conv91
128
32
0.65
0.05
0.74
0.05
0.74
0.05
10.0
155.4
Conv92
32
128
0.40
0.05
0.74
0.05
0.74
0.05
2.28
46.19
Conv93
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.06
46.19
Conv94
128
32
0.69
0.05
0.74
0.05
0.74
0.05
10.16
155.4
Conv95
32
128
0.42
0.05
0.74
0.05
0.74
0.05
2.08
46.19
Conv96
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.11
46.19
Conv97
128
32
0.64
0.05
0.74
0.05
0.74
0.05
10.08
155.4
Conv98
32
128
0.33
0.05
0.74
0.05
0.74
0.05
2.23
46.19
Conv99
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.11
46.19
Conv 1 00
128
32
0.66
0.05
0.74
0.05
0.74
0.05
10.11
155.4
Conv 101
32
128
0.44
0.05
0.74
0.05
0.74
0.05
3.08
46.19
Conv 102
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.09
46.19
Conv 103
128
32
0.65
0.05
0.74
0.05
0.74
0.05
11.58
155.4
Conv 104
32
128
0.38
0.05
0.74
0.05
0.74
0.05
3.37
46.19
Conv 105
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.1
46.19
Conv
128
32
0.68
0.05
0.74
0.05
0.74
0.05
11.59
155.4
Conv107 107
32
128
0.43
0.05
0.74
0.05
0.74
0.05
1.57
46.19
Conv108
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.11
46.19
Conv 109
128
32
0,66
0.05
0.74
0.05
0.74
0.05
8.07
155.4
Conv 110
64
128
0.56
0.05
0.82
0.05
0.82
0.05
3.13
83.68
Conv 111
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.08
83.68
Conv
256
64
0.84
0.05
0.90
0.05
0.90
0.05
4.87
294.32
50

---

Under review as a conference paper at ICLR 2021
Conv 13
64
256
0.55
0.05
0.90
0.05
0.90
0.05
1.13
83.68
Conv
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.07
83.68
Conv 1 115
256
64
0.83
0.05
0.90
0.05
0.90
0.05
4.3
294.32
Conv1 16
64
256
0.52
0.05
0.90
0.05
0.90
0.05
1.52
83.68
Conv1 11 17
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.07
83.68
Conv 18
256
64
0.69
0,05
0.90
0.05
0.90
0.05
4.38
294.32
Conv 1 19
64
256
0.53
0.05
0.90
0.05
0.90
0.05
1.19
83.68
Conv 1 120
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.06
83.68
Conv 121
256
64
0.84
0.05
0.90
0.05
0.90
0.05
4.75
294.32
Conv 122
64
256
0.52
0.05
0.90
0.05
0.90
0.05
1.08
83.68
Conv 123
576
0.88
0.05
0.97
0.05
0.97
0.05
0.04
83.68
Conv 24
256
64
0.84
0.05
0.90
0.05
0.90
0.05
4.98
294.32
Conv 125
64
256
0.50
0.05
0.90
0.05
0.90
0.05
1.36
83.68
Conv 126
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.06
83.68
Conv 127
256
64
0.83
0.05
0.90
0.05
0.90
0.05
4.89
294.32
Conv122
64
256
0.47
0.05
0.90
0.05
0.90
0.05
1.42
83.68
Conv 1 129
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.05
83.68
Convl30
256
64
0.82
0.05
0.90
0.05
0.90
0.05
5.59
294.32
Conv 131
64
256
0.52
0.05
0.90
0.05
0.90
0.05
0.97
83.68
Conv 132
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.06
83.68
Conv 133
256
64
0.84
0.05
0.90
0.05
0.90
0.05
4.4
294.32
Conv 1 34
64
256
0.52
0.05
0.90
0.05
0.90
0.05
0.84
83.68
Conv 135
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.08
83.68
Conv 136
256
64
0.82
0.05
0.90
0.05
0.90
0.05
5.17
294.32
Conv 137
64
256
0.49
0.05
0.90
0.05
0.90
0.05
1.24
83.68
Conv 138
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.05
83.68
Conv 139
256
64
0.84
0.05
0.90
0.05
0.90
0.05
5.18
294.32
Conv 140
64
256
0.52
0.05
0.90
0.05
0.90
0.05
1.94
83.68
Conv 141
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.05
83.68
Conv 142
256
64
0.81
0.05
0.90
0.05
0.90
0.05
5.9
294.32
Conv 143
64
256
0.50
0.05
0.90
0.05
0.90
0.05
1.31
83.68
Conv 144
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.06
83.68
Conv 45
256
64
0.82
0.05
0.90
0.05
0.90
0.05
4.99
294.32
Conv 146
64
256
0.53
0.05
0.90
0.05
0.90
0.05
1.05
83.68
Conv 147
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.04
83.68
Conv 148
256
64
0.83
0.05
0.90
0.05
0.90
0.05
4.7
294.32
Conv 49
64
256
0.50
0.05
0.90
0.05
0.90
0.05
1.34
83.68
Conv 150
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.05
83.68
Conv 151
256
64
0.82
0.05
0.90
0.05
0.90
0.05
5.03
294.32
Conv 152
64
256
0.51
0.05
0.90
0.05
0.90
0.05
1.36
83.68
Conv 133
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.07
83.68
Conv 1
256
64
0.81
0.05
0.90
0.05
0.90
0.05
5.44
294.32
Conv 15: 155
64
256
0.50
0.05
0.90
0.05
0.90
0.05
1.16
83.68
Conv 156
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.06
83.68
Conv 157
256
64
0.81
0.05
0.90
0.05
0.90
0.05
4.65
294.32
Conv 158
64
256
0.47
0.05
0.90
0.05
0.90
0.05
1.52
83.68
Conv 159
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.08
83.68
Conv 160
256
64
0.82
0.05
0.90
0.05
0.90
0.05
5.66
294.32
Conv 161
64
256
0.48
0.05
0.90
0.05
0.90
0.05
1.17
83.68
Conv 162
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.05
83.68
Conv 163
256
64
0.83
0.05
0.90
0.05
0.90
0.05
5.4
294.32
Passing rate
100.0%
100.0%
100.0%
99.39%
Table 16: Cifar100 Meam-ResNet164
Layer
Number dim
Gaussian
Mean Left
Mean Right
Sigma
51

---

Under review as a conference paper at ICLR 2021
p-value
c-value
p-value
c-value
p-value
e-value
t-value
e-value
Conv1
16
27
0.03
0.05
0.58
0.05
0.58
0.05
466.72
26.3
Conv2
16
16
0.12
0.05
0.56
0.05
0.56
0.05
222.95
26.3
Conv3
16
144
0.43
0.05
0.68
0.05
0.68
0.05
31.17
26.3
Conv4
64
16
0.35
0.05
0.63
0,05
0.63
0.05
103.14
83.68
Conv5
16
64
0.34
0.05
0.63
0.05
0.63
0.05
17.8
26.3
Conv6
16
144
0.52
0.05
0.68
0.05
0.68
0.05
5.97
26.3
Conv7
64
16
0.69
0.05
0.63
0.05
0.63
0.05
12.77
83.68
Conv8
16
64
0.33
0.05
0.63
0.05
0.63
0.05
31.39
26.3
Conv9
16
144
0.41
0.05
0.68
0.05
0.68
0.05
16.75
26.3
Conv 10
64
16
0.66
0.05
0.63
0.05
0.63
0.05
26.55
83.68
Conv 11
16
64
0.50
0.05
0.63
0.05
0.63
0.05
8.63
26.3
Conv 12
16
144
0.61
0.05
0.68
0.05
0.68
0.05
3.88
26.3
Conv13 13
64
16
0.67
0.05
0.63
0.05
0.63
0.05
22.11
83.68
Conv 14
16
64
0.66
0.05
0.63
0.05
0.63
0.05
5.01
26.3
Conv 15
16
144
0.69
0.05
0.68
0.05
0.68
0.05
2.1
26.3
Conv 16
64
16
0.87
0.05
0.63
0.05
0.63
0.05
5.14
83.68
Conv 17
16
64
0.28
0.05
0.63
0.05
0.63
0.05
20.05
26.3
Conv18
16
144
0.17
0.05
0.68
0.05
0.68
0.05
7.17
26.3
Conv19 19
64
16
0.86
0.05
0.63
0.05
0.63
0.05
4.1
83.68
Conv20
16
64
0.57
0.05
0.63
0.05
0.63
0.05
17.79
26.3
Conv21
16
144
0.31
0.05
0.68
0.05
0.68
0.05
9.78
26.3
Conv22
64
16
0.91
0.05
0.63
0.05
0.63
0.05
3.07
83.68
Conv23
16
64
0.42
0.05
0.63
0.05
0.63
0.05
21.74
26.3
Conv24
16
144
0.31
0.05
0.68
0.05
0.68
0.05
10.79
26.3
Conv25
64
16
0.65
0.05
0.63
0.05
0.63
0.05
13.76
83.68
Conv26
16
64
0.34
0.05
0.63
0.05
0.63
0.05
14.28
26.3
Conv27
16
144
0.29
0.05
0.68
0.05
0.68
0.05
13.23
26.3
Conv28
64
16
0.54
0.05
0.63
0.05
0.63
0.05
31.67
83.68
Conv29
16
64
0.33
0.05
0.63
0.05
0.63
0.05
14.97
26.3
Conv30
16
144
0.50
0.05
0.68
0.05
0.68
0.05
5.76
26.3
Conv31
64
16
0.46
0.05
0.63
0.05
0.63
0.05
25.54
83.68
Conv32
16
64
0.39
0.05
0.63
0.05
0.63
0.05
14.78
26.3
Conv33
16
144
0.38
0.05
0.68
0.05
0.68
0.05
6.19
26.3
Conv34
64
16
0.84
0.05
0.63
0.05
0.63
0.05
9.25
83.68
Conv35
16
64
0.47
0.05
0.63
0.05
0.63
0.05
11.5
26.3
Conv36
16
144
0.42
0.05
0.68
0.05
0.68
0.05
13.26
26.3
Conv37
64
16
0.55
0.05
0.63
0.05
0.63
0.05
77.76
83.68
Conv38
16
64
0.46
0.05
0.63
0.05
0.63
0.05
20.83
26.3
Conv39
16
144
0.38
0.05
0.68
0.05
0.68
0.05
11.74
26.3
Conv40
64
16
0.68
0.05
0.63
0.05
0.63
0.05
35.12
83.68
Conv41
16
64
0.92
0.05
0.63
0.05
0.63
0.05
0.53
26.3
Conv42
16
144
0.94
0.05
0.68
0.05
0.68
0.05
0.15
26.3
Conv43
64
16
0.94
0.05
0.63
0.05
0.63
0.05
1.15
83.68
Conv44
16
64
0.42
0.05
0.63
0.05
0.63
0.05
20.83
26.3
Conv45
16
144
0.46
0.05
0.68
0.05
0.68
0.05
15.2
26.3
Conv46
64
16
0.45
0.05
0.63
0.05
0.63
0.05
50.71
83.68
Conv47
16
64
0.30
0.05
0.63
0.05
0.63
0.05
25.92
26.3
Conv48
16
144
0.43
0.05
0.68
0.05
0.68
0.05
15.15
26.3
Conv49
64
16
0.53
0.05
0.63
0.05
0.63
0.05
77.85
83.68
Conv50
16
64
0.34
0.05
0.63
0.05
0.63
0.05
23.91
26.3
Conv5
16
144
0.52
0.05
0.68
0.05
0.68
0.05
12.91
26.3
Conv52
64
16
0.45
0.05
0.63
0.05
0.63
0.05
74.94
83.68
Conv53
16
64
0.41
0.05
0.63
0.05
0.63
0.05
30.44
26.3
Conv54
16
144
0.41
0.05
0.68
0.05
0.68
0,05
21.99
26.3
Conv55
64
16
0.41
0.05
0.63
0.05
0.63
0.05
106.69
83.68
Conv56
32
64
0.13
0.05
0.67
0.05
0.67
0.05
167.36
46.19
Conv57
32
288
0.42
0.05
0.83
0.05
0.83
0.05
28.09
46.19
52

---

Under review as a conference paper at ICLR 2021
Conv58
128
32
0.17
0.05
0.74
0.05
0.74
0.05
320.85
155.4
Conv59
32
128
0.20
0.05
0.74
0.05
0.74
0.05
15.24
46.19
Conv60
32
288
0.23
0.05
0.83
0.05
0.83
0.05
10.97
46.19
Conv61
128
32
0.57
0.05
0.74
0.05
0.74
0.05
42.72
155.4
Conv62
32
128
0.22
0.05
0.74
0.05
0.74
0.05
15.01
46.19
Conv63
32
288
0.39
0,05
0.83
0,05
0.83
0.05
9.25
46.19
Conv64
128
32
0.63
0.05
0.74
0.05
0.74
0.05
40.35
155.4
Conv65
32
128
0.33
0.05
0.74
0.05
0.74
0.05
10.85
46.19
Conv66
32
288
0.64
0.05
0.83
0.05
0.83
0.05
4.2
46.19
Conv67
128
32
0.50
0.05
0.74
0.05
0.74
0.05
26.77
155.4
Conv68
128
0.15
0.05
0.74
0.05
0.74
0.05
22.12
46.19
Conv69
32
288
0.37
0.05
0.83
0.05
0.83
0.05
15.38
46.19
Conv70 70
128
32
0.49
0.05
0.74
0.05
0.74
0.05
69.31
153.4
Conv71 71
32
128
0.18
0.05
0.74
0.05
0.74
0.05
10.69
46.19
Conv72 72
32
288
0.26
0.05
0.83
0.05
0.83
0.05
6.71
46.19
C3
128
32
0.56
0.05
0.74
0.05
0.74
0.05
15.51
155.4
Conv74 74
32
128
0.38
0.05
0.74
0.05
0.74
0.05
23.08
46.19
Conv75
32
288
0.43
0.05
0.83
0.05
0.83
0.05
9.83
46.19
Conv 76
128
32
0.49
0.05
0.74
0.05
0.74
0.05
29.79
155.4
Conv77
32
128
0.37
0.05
0.74
0.05
0.74
0.05
20.85
46.19
Conv78
32
288
0.43
0.05
0.83
0.05
0.83
0.05
7.53
46.19
Conv79
128
32
0.58
0.05
0.74
0.05
0.74
0.05
38.54
155.4
Conv80
32
128
0.26
0.05
0.74
0.05
0.74
0.05
18.75
46.19
Conv81
32
288
0.38
0.05
0.83
0.05
0.83
0.05
12.27
46.19
Conv82
128
32
0.62
0.05
0.74
0.05
0.74
0.05
32.84
155.4
Conv83
32
128
0.10
0.05
0.74
0.05
0.74
0.05
20.59
46.19
Conv84
32
288
0.31
0.05
0.83
0.05
0.83
0.05
16.6
46.19
Conv85
128
32
0.38
0.05
0.74
0.05
0.74
0.05
107.99
155.4
Conv86
32
128
0.44
0.05
0.74
0.05
0.74
0.05
18.28
46.19
Conv87
32
288
0.47
0.05
0.83
0.05
0.83
0.05
10.62
46.19
Conv88
128
32
0.62
0.05
0.74
0.05
0.74
0.05
35.83
155.4
Conv89
32
128
0.21
0.05
0.74
0.05
0.74
0.05
21.55
46.19
Conv90
32
288
0.57
0.05
0.83
0.05
0.83
0.05
7.98
46.19
Conv91
128
32
0.49
0.05
0.74
0.05
0.74
0.05
66.43
155.4
Conv92
32
128
0.43
0.05
0.74
0.05
0.74
0.05
17.04
46.19
Conv93
32
288
0.30
0.05
0.83
0.05
0.83
0.05
8.51
46.19
Conv94
128
32
0.58
0.05
0.74
0.05
0.74
0.05
43.25
155.4
Conv95
32
128
0.19
0.05
0.74
0.05
0.74
0.05
14.97
46.19
Conv96
32
288
0.29
0.05
0.83
0.05
0.83
0.05
9.6
46.19
Conv97
128
32
0.60
0.05
0.74
0.05
0.74
0.05
36.05
155.4
Conv98
32
128
0.35
0.05
0.74
0.05
0.74
0.05
7.96
46.19
Conv99
32
288
0.46
0.05
0.83
0.05
0.83
0.05
2.19
46.19
Conv 100
128
32
0.81
0.05
0.74
0.05
0.74
0.05
11.53
155.4
Conv 101
32
128
0.33
0.05
0.74
0.05
0.74
0.05
16.35
46.19
Conv 102
32
288
0.34
0.05
0.83
0.05
0.83
0.05
9.13
46.19
Conv 103
128
32
0.59
0.05
0.74
0.05
0.74
0.05
21.11
155.4
Conv 1 04
32
128
0.39
0.05
0.74
0.05
0.74
0.05
20.91
46.19
Conv 105
32
288
0.64
0.05
0.83
0.05
0.83
0.05
8.36
46.19
Conv 106
128
32
0.54
0.05
0.74
0.05
0.74
0.05
44.06
155.4
Conv 107
32
128
0.31
0.05
0.74
0.05
0.74
0.05
29.98
46.19
Conv 108
32
288
0.45
0.05
0.83
0.05
0.83
0.05
15.03
46.19
Conv 109
128
32
0.50
0.05
0.74
0.05
0.74
0.05
82.36
155.4
Conv1 10
64
128
0.14
0.05
0.82
0.05
0.82
0.05
197.08
83.68
Conv1
64
576
0.46
0.05
0.97
0.05
0.97
0.05
59.29
83.68
Conv1 12
256
64
0.16
0.05
0.90
0.05
0.90
0.05
594.89
294.32
Conv 13
64
256
0.44
0.05
0.90
0.05
0.90
0.05
33.77
83.68
Conv 14
64
576
0.32
0.05
0.97
0.05
0.97
0.05
26.61
83.68
Conv 115
256
64
0.30
0.05
0.90
0.05
0.90
0.05
175.45
294.32
Conv
64
256
0.38
0.05
0.90
0.05
0.90
0.05
24.35
83.68
53

---

Under review as a conference paper at ICLR 2021
Conv 1 17
64
576
0.33
0.05
0.97
0.05
0.97
0.05
16.49
83.68
Conv
256
64
0.51
0.05
0.90
0.05
0.90
0.05
123.0
294.32
Conv 1 119
64
256
0.08
0.05
0.90
0.05
0.90
0.05
35.01
83.68
Conv 1 20
64
576
0.34
0.05
0.97
0.05
0.97
0.05
24.5
83.68
Conv12)
256
64
0.36
0.05
0.90
0.05
0.90
0.05
159.72
294.32
Conv 122
64
256
0.42
0,05
0.90
0.05
0,90
0.05
21.93
83.68
C23
64
576
0.16
0.05
0.97
0.05
0.97
0.05
12.24
83.68
Conv 1 24
256
64
0.54
0.05
0.90
0.05
0.90
0.05
86.27
294.32
Conv 125
64
256
0.24
0.05
0.90
0.05
0.90
0.05
38.93
83.68
Conv 126
64
576
0.44
0.05
0.97
0.05
0.97
0.05
17.46
83.68
Conv 127
256
64
0.50
0.05
0.90
0.05
0.90
0.05
91.81
294.32
Conv 28
64
256
0.32
0.05
0.90
0.05
0.90
0.05
29.94
83.68
Conv 129
64
576
0.33
0.05
0.97
0.05
0.97
0.05
11.34
83.68
Conv 130
256
64
0.58
0.05
0.90
0.05
0.90
0.05
76.05
294.32
Conv 131 131
64
256
0.46
0.05
0.90
0.05
0.90
0.05
38.51
83.68
Conv 132
64
576
0.42
0.05
0.97
0.05
0.97
0.05
13.99
83.68
Conv 133
256
64
0.49
0.05
0.90
0.05
0.90
0.05
78.35
294.32
Conv 1onv134
64
256
0.28
0.05
0.90
0.05
0.90
0.05
55.05
83.68
Conv 135
64
576
0.34
0.05
0.97
0.05
0.97
0.05
22.32
83.68
Conv 136
256
64
0.40
0.05
0.90
0.05
0.90
0.05
127.27
294.32
Conv 137
64
256
0.45
0.05
0.90
0.05
0.90
0.05
73.92
83.68
Conv 138
64
576
0.39
0.05
0.97
0.05
0.97
0.05
20.65
83.68
Conv 139
256
64
0.43
0.05
0.90
0.05
0.90
0.05
79.88
294.32
Conv 140
64
256
0.46
0.05
0.90
0.05
0.90
0.05
57.87
83.68
Conv 141
64
576
0.46
0.05
0.97
0.05
0.97
0.05
14.7
83.68
Conv 142
256
64
0.49
0.05
0.90
0.05
0.90
0.05
62.55
294.32
Conv 143
64
256
0.36
0.05
0.90
0.05
0.90
0.05
64.03
83.68
Conv 144
64
576
0.51
0.05
0.97
0.05
0.97
0.05
17.48
83.68
Conv 14n
256
64
0.45
0.05
0.90
0.05
0.90
0.05
59.58
294.32
Conv 146
64
256
0.36
0.05
0.90
0.05
0.90
0.05
106.22
83.68
Conv 147
64
576
0.53
0.05
0.97
0.05
0.97
0.05
32.45
83.68
Conv 148
256
64
0.39
0.05
0.90
0.05
0.90
0.05
110.05
294.32
Conv 149
64
256
0.43
0.05
0.90
0.05
0.90
0.05
105.01
83.68
Conv 150
64
576
0.43
0.05
0.97
0.05
0.97
0.05
30.32
83.68
Conv 151
256
64
0.40
0.05
0.90
0.05
0.90
0.05
81.71
294.32
Conv 152
64
256
0.34
0.05
0.90
0.05
0.90
0.05
113.29
83.68
Conv 153
64
576
0.42
0.05
0.97
0.05
0.97
0.05
23.49
83.68
Conv 154
256
64
0.37
0.05
0.90
0.05
0.90
0.05
75.03
294.32
Conv 155
64
256
0.30
0.05
0.90
0.05
0.90
0.05
101.39
83.68
Conv 156
64
576
0.63
0.05
0.97
0.05
0.97
0.05
15.84
83.68
Conv 157
256
64
0.49
0.05
0.90
0.05
0.90
0.05
58.98
294.32
Conv 15n
64
256
0.23
0.05
0.90
0.05
0.90
0.05
84.02
83.68
Conv 159
64
576
0.65
0.05
0.97
0.05
0.97
0.05
8.42
83.68
Conv160
256
64
0.54
0.05
0.90
0.05
0.90
0.05
33.29
294.32
Conv 161
64
256
0.57
0.05
0.90
0.05
0.90
0.05
20.57
83.68
Conv 162
64
576
0.82
0.05
0.97
0.05
0.97
0.05
1.41
83.68
Conv16 163
256
64
0.86
0.05
0.90
0.05
0.90
0.05
9.3
294.32
Passing rate
99.39%
100.0%
100.0%
90.18%
Table 17: Cifar100
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Convl Conv2
16
27
0.81
0.05
0.58
0.05
0.58
0.05
1.0
26.3
16
16
0.29
0.05
0.56
0.05
0.56
0.05
27.5
26.3
54

---

Under review as a conference paper at ICLR 2021
C
16
144
0.64
0.05
0.68
0.05
0.68
0.05
0.21
26.3
Conv4
64
16
0.58
0.05
0.63
0.05
0.63
0.05
26.43
83.68
Conv5
16
64
0.22
0.05
0.63
0.05
0.63
0.05
5.73
26.3
Conv6
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.31
26.3
Conv7
64
16
0.56
0.05
0.63
0.05
0.63
0.05
24.75
83.68
Conv8
16
64
0.26
0,05
0.63
0,05
0.63
0.05
2.68
26.3
Conv9
16
144
0.67
0.05
0.68
0.05
0.68
0.05
0.32
26.3
Conv 10
64
16
0.57
0.05
0.63
0.05
0.63
0.05
23.47
83.68
Conv 1
16
64
0.32
0.05
0.63
0.05
0.63
0.05
4.14
26.3
Conv 12
16
144
0.68
0.05
0.68
0.05
0.68
0.05
0.16
26.3
Conv 13
64
16
0.49
0.05
0.63
0.05
0.63
0.05
21.49
83.68
Conv 14
16
64
0.25
0.05
0.63
0.05
0.63
0.05
7.46
26.3
Conv 15
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.22
26.3
Conv 16
64
16
0.56
0.05
0.63
0.05
0.63
0.05
22.21
83.68
Conv1 17
16
64
0.24
0.05
0.63
0.05
0.63
0.05
3.57
26.3
Conv18 18
16
144
0.68
0.05
0.68
0.05
0.68
0.05
0.36
26.3
Conv 19
64
16
0.57
0.05
0.63
0.05
0.63
0.05
27.9
83.68
Conv20
16
64
0.22
0.05
0.63
0.05
0.63
0.05
3.02
26.3
Conv21
16
144
0.64
0.05
0.68
0.05
0.68
0.05
0.18
26.3
Conv22
64
16
0.57
0.05
0.63
0.05
0.63
0.05
18.51
83.68
Conv23
16
64
0.27
0.05
0.63
0.05
0.63
0.05
3.4
26.3
Conv24
16
0.71
0.05
0.68
0.05
0.68
0.05
0.24
26.3
Conv25
64
16
0.58
0.05
0.63
0.05
0.63
0.05
15.57
83.68
Conv26
16
64
0.30
0.05
0.63
0.05
0.63
0.05
3.65
26.3
Conv27
16
144
0.71
0.05
0.68
0.05
0.68
0.05
0.24
26.3
Conv28
64
16
0.51
0.05
0.63
0.05
0.63
0.05
20.17
83.68
Conv29
16
64
0.21
0.05
0.63
0.05
0.63
0.05
5.1
26.3
Conv30
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.23
26.3
Conv31
64
16
0.54
0.05
0.63
0.05
0.63
0.05
21.99
83.68
Conv32
16
64
0.30
0.05
0.63
0.05
0.63
0.05
5.37
26.3
Conv33
16
144
0.65
0.05
0.68
0.05
0.68
0.05
0.41
26.3
Conv34
64
16
0.49
0.05
0.63
0.05
0.63
0.05
22.39
83.68
Conv35
16
64
0.20
0.05
0.63
0.05
0.63
0.05
5.39
26.3
Conv36
16
144
0.63
0.05
0.68
0.05
0.68
0.05
0.15
26.3
Conv37
64
16
0.60
0.05
0.63
0.05
0.63
0.05
22.47
83.68
Conv38
16
64
0.33
0.05
0.63
0.05
0.63
0.05
2.65
26.3
Conv39
16
0.68
0.05
0.68
0.05
0.68
0.05
0.22
26.3
Conv40
64
16
0.56
0.05
0.63
0.05
0.63
0.05
15.34
83.68
Conv41
16
64
0.34
0.05
0.63
0.05
0.63
0.05
4.43
26.3
Conv42
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.28
26.3
Conv43
64
16
0.61
0.05
0.63
0.05
0.63
0.05
20.54
83.68
Conv44
16
64
0.34
0.05
0.63
0.05
0.63
0.05
5.02
26.3
Conv45
16
144
0.67
0.05
0.68
0.05
0.68
0.05
0.37
26.3
Conv46
64
16
0.63
0.05
0.63
0.05
0.63
0.05
19.02
83.68
Conv47
16
64
0.28
0.05
0.63
0.05
0.63
0.05
5.11
26.3
Conv48
16
144
0.68
0.05
0.68
0.05
0.68
0.05
0.29
26.3
Conv49
64
16
0.65
0.05
0.63
0.05
0.63
0.05
19.48
83.68
Conv50
16
64
0.36
0.05
0.63
0.05
0.63
0.05
3.04
26.3
Conv51
16
144
0.69
0.05
0.68
0.05
0.68
0.05
0.41
26.3
Conv52
64
16
0.64
0.05
0.63
0.05
0.63
0.05
21.89
83.68
Conv53
16
64
0.26
0.05
0.63
0.05
0.63
0.05
3.49
26.3
Conv54
16
144
0.70
0.05
0.68
0.05
0.68
0.05
0.07
26.3
Conv55
64
16
0.59
0.05
0.63
0.05
0.63
0.05
20.2
83.68
Conv56
32
64
0.47
0.05
0.67
0.05
0.67
0.05
3.86
46.19
Conv57
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.12
46.19
Conv58
128
32
0.65
0.05
0.74
0.05
0.74
0.05
10.75
155.4
Conv59
32
128
0.48
0.05
0.74
0.05
0.74
0.05
1.97
46.19
Conv60
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.14
46.19
Conv61
128
32
0.64
0.05
0.74
0.05
0.74
0.05
10.13
155.4
55

---

Under review as a conference paper at ICLR 2021
Conv62
32
128
0.48
0.05
0.74
0.05
0.74
0.05
2.12
46.19
Conv63
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.21
46.19
Conv64
128
32
0.67
0.05
0.74
0.05
0.74
0.05
9.34
155.4
Conv65
32
128
0.48
0.05
0.74
0.05
0.74
0.05
4.38
46.19
Conv66
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.23
46.19
Conv67
128
32
0.65
0,05
0.74
0,05
0.74
0.05
9.64
153.4
Conv68
32
128
0.43
0.05
0.74
0.05
0.74
0.05
4.33
46.19
Conv69
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.22
46.19
Conv70
128
32
0.68
0.05
0.74
0.05
0.74
0.05
9.63
155.4
Conv71
32
128
0.48
0.05
0.74
0.05
0.74
0.05
3.55
46.19
Conv72
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.19
46.19
Conv73
128
32
0.81
0.05
0.74
0.05
0.74
0.05
9.45
155.4
Conv74
32
128
0.44
0.05
0.74
0.05
0.74
0.05
5.7
46.19
Conv75
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.41
46.19
Conv76
128
32
0.69
0.05
0.74
0.05
0.74
0.05
10.14
155.4
Conv77
32
128
0.39
0.05
0.74
0.05
0.74
0.05
6.42
46.19
Conv78 78
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.25
46.19
Conv79
128
32
0.67
0.05
0.74
0.05
0.74
0.05
7.54
155.4
Conv80
32
128
0.52
0.05
0.74
0.05
0.74
0.05
3.95
46.19
Conv81
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.24
46.19
Conv82
128
32
0.70
0.05
0.74
0.05
0.74
0.05
8.76
155.4
Conv83
32
128
0.53
0.05
0.74
0.05
0.74
0.05
46.19
Conv84
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.33
46.19
Conv85
128
32
0.70
0.05
0.74
0.05
0.74
0.05
9.76
155.4
Conv86
32
128
0.51
0.05
0.74
0.05
0.74
0.05
3.95
46.19
Conv87
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.39
46.19
Conv88
128
32
0.81
0.05
0.74
0.05
0.74
0.05
10.14
155.4
Conv89
32
128
0.54
0.05
0.74
0.05
0.74
0.05
6.26
46.19
Conv90
32
288
0.86
0.05
0.83
0.05
0.83
0.05
0.62
46.19
Conv91
128
32
0.68
0.05
0.74
0.05
0.74
0.05
10.09
155.4
Conv92
32
128
0.48
0.05
0.74
0.05
0.74
0.05
3.63
46.19
Conv93
32
288
0.86
0.05
0.83
0.05
0.83
0.05
0.33
46.19
Conv94
128
32
0.71
0.05
0.74
0.05
0.74
0.05
9.94
155.4
Conv95
32
128
0.53
0.05
0.74
0.05
0.74
0.05
3.39
46.19
Conv96
32
288
0.87
0.05
0.83
0.05
0.83
0.05
46.19
Conv97
128
32
0.69
0.05
0.74
0.05
0.74
0.05
6.74
155.4
Conv98
32
128
0.52
0.05
0.74
0.05
0.74
0.05
7.89
46.19
Conv99
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.38
46.19
Conv 100
128
32
0.82
0.05
0.74
0.05
0.74
0.05
9.47
155.4
Conv 101
32
128
0.55
0.05
0.74
0.05
0.74
0.05
4.92
46.19
Conv 102
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.37
46.19
Conv
128
32
0.82
0.05
0.74
0.05
0.74
0.05
10.66
155.4
Conv 104
32
128
0.49
0.05
0.74
0.05
0.74
0.05
4.22
46.19
Conv 105
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.34
46.19
Conv 106
128
32
0.81
0.05
0.74
0.05
0.74
0.05
9.11
155.4
Conv 107
32
128
0.51
0.05
0.74
0.05
0.74
0.05
5.88
46.19
Conv 08
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.42
46.19
Conv 109
128
32
0.71
0.05
0.74
0.05
0.74
0.05
7.11
155.4
Conv 10
64
128
0.55
0.05
0.82
0.05
0.82
0.05
3.26
83.68
Conv
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.2
83.68
Conv 1 12
256
64
0.83
0.05
0.90
0.05
0.90
0.05
5.43
294.32
Conv
64
256
0.63
0.05
0.90
0.05
0.90
0.05
2.29
83.68
Conv1 1 14
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.23
83.68
Conv1 15
256
64
0.83
0.05
0.90
0.05
0.90
0.05
6.3
294.32
Conv1 16
64
256
0.62
0.05
0.90
0.05
0.90
0.05
2.39
83.68
Conv 17
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.25
83.68
Conv 118
256
64
0.86
0.05
0.90
0.05
0.90
0.05
6.52
294.32
Conv 119
64
256
0.60
0.05
0.90
0.05
0.90
0.05
3.15
83.68
Conv120
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.28
83.68
56

---

Under review as a conference paper at ICLR 2021
Conv 121
256
64
0.86
0.05
0.90
0.05
0.90
0.05
6.03
294.32
Conv 122
64
256
0.61
0.05
0.90
0.05
0.90
0.05
4.13
83.68
Conv 123
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.22
83.68
Conv 1
256
64
0.85
0.05
0.90
0.05
0.90
0.05
5.46
294.32
Conv123
64
256
0.53
0.05
0.90
0.05
0.90
0.05
5.63
83.68
Conv 126
64
576
0.88
0,05
0.97
0.05
0.97
0.05
0.21
83.68
Conv
256
64
0.85
0.05
0.90
0.05
0.90
0.05
6.59
294.32
Conv 1 228
64
256
0.63
0.05
0.90
0.05
0.90
0.05
4.57
83.68
Conv 1 129
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.21
83.68
Conv 130
256
64
0.85
0.05
0.90
0.05
0.90
0.05
5.21
294.32
Conv 131
64
256
0.68
0.05
0.90
0.05
0.90
0.05
4.42
83.68
Conv 132
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.21
83.68
Conv 133
256
64
0.85
0.05
0.90
0.05
0.90
0.05
5.39
294.32
Conv 134
64
256
0.65
0.05
0.90
0.05
0.90
0.05
6.82
83.68
Conv 135
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.23
83.68
Conv Nonv136
256
64
0.85
0.05
0.90
0.05
0.90
0.05
6.57
294.32
Conv 137
64
256
0.68
0.05
0.90
0.05
0.90
0.05
6.95
83.68
Conv
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.22
83.68
Conv 139
256
64
0.85
0.05
0.90
0.05
0.90
0.05
6.3
294.32
Conv 140
64
256
0.62
0.05
0.90
0.05
0.90
0.05
4.99
83.68
Conv 141
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.21
83.68
Conv 142
256
64
0.86
0.05
0.90
0.05
0.90
0.05
4.94
294.32
Conv 143
64
256
0.62
0.05
0.90
0.05
0.90
0.05
4.11
83.68
Conv 144
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.27
83.68
Conv 145
256
64
0.82
0.05
0.90
0.05
0.90
0.05
5.79
294.32
Conv 146
64
256
0.62
0.05
0.90
0.05
0.90
0.05
4.72
83.68
Conv 147
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.32
83.68
Conv 148
256
64
0.85
0.05
0.90
0.05
0.90
0.05
4.91
294.32
Conv 1
64
256
0.65
0.05
0.90
0.05
0.90
0.05
3.53
83.68
Conv 1 50
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.21
83.68
Conv 151
256
64
0.85
0.05
0.90
0.05
0.90
0.05
4.28
294.32
Conv 152
64
256
0.65
0.05
0.90
0.05
0.90
0.05
4.26
83.68
Conv 153
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.2
83.68
Conv 1 154
256
64
0.86
0.05
0.90
0.05
0.90
0.05
3.77
294.32
Conv 155
64
256
0.62
0.05
0.90
0.05
0.90
0.05
4.45
83.68
Conv 156
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.12
83.68
Conv 157
256
64
0.85
0.05
0.90
0.05
0.90
0.05
3.0
294.32
Conv 158
64
256
0.66
0.05
0.90
0.05
0.90
0.05
5.59
83.68
Conv 159
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.22
83.68
Conv 160
256
64
0.85
0.05
0.90
0.05
0.90
0.05
3.12
294.32
Conv 161
64
256
0.63
0.05
0.90
0.05
0.90
0.05
4.89
83.68
Conv 162
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.17
83,68
Conv16 163
256
64
0.86
0.05
0.90
0.05
0.90
0.05
2.38
294.32
Passing rate
99.39%
100.0%
100.0%
99.39%
Table 18: Cifar100 Adamax-ResNeel64
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
16
27
0.13
0.05
0.58
0.05
0.58
0.05
68.0
26.3
Conv2
16
16
0.48
0.05
0.56
0.05
0.56
0.05
71.45
26.3
Conv3
16
144
0.66
0.05
0.68
0.05
0.68
0.05
11.09
26.3
Conv4
64
16
0.59
0.05
0.63
0.05
0.63
0.05
70.98
83.68
Conv5
16
64
0.55
0.05
0.63
0.05
0.63
0.05
20.74
26.3
Conv6
16
144
0.61
0.05
0.68
0.05
0.68
0.05
13.67
26.3
57

---

Under review as a conference paper at ICLR 2021
Conv7
64
16
0.63
0.05
0.63
0.05
0.63
0.05
78.55
83.68
Conv8
16
64
0.46
0.05
0.63
0.05
0.63
0.05
13.55
26.3
Conv9
16
141
0.69
0.05
0.68
0.05
0.68
0.05
7.71
26.3
Conv10 10
64
16
0.70
0.05
0.63
0.05
0.63
0.05
21.48
83.68
Conv11
16
64
0.66
0.05
0.63
0.05
0.63
0.05
4.16
26.3
Conv 12
16
144
0.89
0.05
0.68
0,05
0.68
0.05
1.72
26.3
Conv
64
16
0.88
0.05
0.63
0.05
0.63
0.05
7.36
83.68
Conv 14
16
64
0.69
0.05
0.63
0.05
0.63
0.05
8.77
26.3
Conv15 15
16
144
0.67
0.05
0.68
0.05
0.68
0.05
2.77
26.3
Conv 16
64
16
0.80
0.05
0.63
0.05
0.63
0.05
7.36
83.68
Conv 17
16
64
0.70
0.05
0.63
0.05
0.63
0.05
6.74
26.3
Conv 18
16
144
0.71
0.05
0.68
0.05
0.68
0.05
2.63
26.3
Conv 19
64
16
0.82
0.05
0.63
0.05
0.63
0.05
11.07
83.68
Conv20
16
64
0.44
0.05
0.63
0.05
0.63
0.05
15.46
26.3
Conv21
16
144
0.51
0.05
0.68
0.05
0.68
0.05
7.75
26.3
Conv22
64
16
0.66
0.05
0.63
0.05
0.63
0.05
24.59
83.68
Conv23
16
64
0.54
0.05
0.63
0.05
0.63
0.05
16.27
26.3
Conv24
16
144
0.62
0.05
0.68
0.05
0.68
0.05
10.25
26.3
Conv25
64
16
0.71
0.05
0.63
0.05
0.63
0.05
44.15
83.68
Conv26
16
64
0.68
0.05
0.63
0.05
0.63
0.05
12.42
26.3
Conv27
16
144
0.64
0.05
0.68
0.05
0.68
0.05
4.52
26.3
Conv28
64
16
0.80
0.05
0.63
0.05
0.63
0.05
21.59
83.68
Conv29
16
64
0.54
0.05
0.63
0.05
0.63
0.05
19.57
26.3
Conv30
16
144
0.60
0.05
0.68
0.05
0.68
0.05
10.01
26.3
Conv31
64
16
0.63
0.05
0.63
0.05
0.63
0.05
50.96
83.68
Conv32
16
64
0.41
0.05
0.63
0.05
0.63
0.05
15.87
26.3
Conv33
16
144
0.58
0.05
0.68
0.05
0.68
0.05
6.75
26.3
Conv34
64
16
0.66
0.05
0.63
0.05
0.63
0.05
52.95
83.68
Conv35
16
64
0.59
0.05
0.63
0.05
0.63
0.05
10.96
26.3
Conv36
16
144
0.66
0.05
0.68
0.05
0.68
0.05
4.91
26.3
Conv37
64
16
0.65
0.05
0.63
0.05
0.63
0.05
21.71
83.68
Conv38
16
64
0.35
0.05
0.63
0.05
0.63
0.05
24.32
26.3
Conv39
16
144
0.50
0.05
0.68
0.05
0.68
0.05
8.62
26.3
Conv40
64
16
0.67
0.05
0.63
0.05
0.63
0.05
49.57
83.68
Conv41
16
64
0.61
0.05
0.63
0.05
0.63
0.05
16.45
26.3
Conv42
16
144
0.63
0.05
0.68
0.05
0.68
0.05
9.32
26.3
Conv43
64
16
0.68
0.05
0.63
0.05
0.63
0.05
38.26
83.68
Conv44
16
64
0.68
0.05
0.63
0.05
0.63
0.05
12.91
26.3
Comv45
16
144
0.65
0.05
0.68
0.05
0.68
0.05
8.54
26.3
Conv46
64
16
0.65
0.05
0.63
0.05
0.63
0.05
53.09
83.68
Conv47
16
64
0.55
0.05
0.63
0.05
0.63
0.05
16.67
26.3
Conv48
16
144
0.56
0.05
0.68
0.05
0.68
0.05
8.19
26.3
Conv49
64
16
0.66
0.05
0.63
0.05
0.63
0.05
39.79
83.68
Conv50
16
64
0.59
0.05
0.63
0.05
0.63
0.05
17.17
26.3
Conv51
16
144
0.61
0.05
0.68
0.05
0.68
0.05
10.77
26.3
Conv52
64
16
0.61
0.05
0.63
0.05
0.63
0.05
51.64
83.68
Conv53
16
64
0.37
0.05
0.63
0.05
0.63
0.05
24.49
26.3
Conv54
16
0.57
0.05
0.68
0.05
0.68
0.05
7.39
26.3
Conv55
64
16
0.62
0.05
0.63
0.05
0.63
0.05
45.71
83.68
Conv56
32
64
0.35
0.05
0.67
0.05
0.67
0.05
84.41
46.19
Conv57
32
288
0.56
0.05
0.83
0.05
0.83
0.05
19.39
46.19
Conv58
128
32
0.42
0.05
0.74
0.05
0.74
0.05
198.37
155.4
Conv59
32
128
0.43
0.05
0.74
0.05
0.74
0.05
16.2
46.19
Conv60
32
288
0.39
0.05
0.83
0.05
0.83
0.05
10.01
46.19
Conv61
128
32
0.62
0.05
0.74
0.05
0.74
0.05
49.07
155.4
Conv62
32
128
0.43
0.05
0.74
0.05
0.74
0.05
11.75
46.19
Conv63
32
288
0.55
0.05
0.83
0.05
0.83
0.05
6.82
46.19
Conv64
128
32
0.62
0.05
0.74
0.05
0.74
0.05
37.0
155.4
Conv65
32
128
0.54
0.05
0.74
0.05
0.74
0.05
8.23
46.19
58

---

Under review as a conference paper at ICLR 2021
Conv66
32
288
0.45
0.05
0.83
0.05
0.83
0.05
5.49
46.19
Conv67
128
32
0.69
0.05
0.74
0.05
0.74
0.05
34.13
153.4
Conv68
32
128
0.41
0.05
0.74
0.05
0.74
0.05
9.57
46.19
Conv69
32
288
0.49
0.05
0.83
0.05
0.83
0.05
6.21
46.19
Conv70
128
32
0.69
0.05
0.74
0.05
0.74
0.05
28.63
155.4
Conv71 71
32
128
0.38
0,05
0.74
0,05
0.74
0.05
12.89
46.19
Conv72
32
288
0.42
0.05
0.83
0.05
0.83
0.05
7.03
46.19
Conv73 73
128
32
0.59
0.05
0.74
0.05
0.74
0.05
34.59
155.4
Conv74
32
128
0.50
0.05
0.74
0.05
0.74
0.05
18.97
46.19
Conv75
32
288
0.68
0.05
0.83
0.05
0.83
0.05
8.88
46.19
Conv76
128
32
0.69
0.05
0.74
0.05
0.74
0.05
40.73
155.4
Conv77 77
32
128
0.57
0.05
0.74
0.05
0.74
0.05
12.85
46.19
Conv78
32
288
0.64
0.05
0.83
0.05
0.83
0.05
4.65
46.19
Conv79
128
32
0.82
0.05
0.74
0.05
0.74
0.05
23.62
155.4
Conv80
32
128
0.59
0.05
0.74
0.05
0.74
0.05
8.78
46.19
Conv81
32
288
0.61
0.05
0.83
0.05
0.83
0.05
3.45
46.19
Conv82
128
32
0.81
0.05
0.74
0.05
0.74
0.05
17.55
155.4
Conv83
32
128
0.55
0.05
0.74
0.05
0.74
0.05
16.4
46.19
Conv84
32
288
0.56
0.05
0.83
0.05
0.83
0.05
8.33
46.19
Conv85
128
32
0.66
0.05
0.74
0.05
0.74
0.05
38.5
155.4
Conv86
32
128
0.39
0.05
0.74
0.05
0.74
0.05
23.91
46.19
Conv87
32
288
0.56
0.05
0.83
0.05
0.83
0.05
12.35
46.19
Conv88
128
32
0.54
0.05
0.74
0.05
0.74
0.05
85.92
155.4
Conv89
32
128
0.56
0.05
0.74
0.05
0.74
0.05
20.14
46.19
Conv90
32
288
0.63
0.05
0.83
0.05
0.83
0.05
8.2
46.19
Conv91
128
32
0.69
0.05
0.74
0.05
0.74
0.05
47.23
155.4
Conv92
32
128
0.54
0.05
0.74
0.05
0.74
0.05
11.75
46.19
Conv93
32
288
0.58
0.05
0.83
0.05
0.83
0.05
5.63
46.19
Conv94
128
32
0.81
0.05
0.74
0.05
0.74
0.05
25.93
155.4
Conv95
32
128
0.61
0.05
0.74
0.05
0.74
0.05
16.71
46.19
Conv96
32
288
0.68
0.05
0.83
0.05
0.83
0.05
7.92
46.19
Conv97
128
32
0.69
0.05
0.74
0.05
0.74
0.05
37.55
155.4
Conv98
32
128
0.40
0.05
0.74
0.05
0.74
0.05
22.82
46.19
Conv99
32
288
0.80
0.05
0.83
0.05
0.83
0.05
6.79
46.19
Conv 100
128
32
0.67
0.05
0.74
0.05
0.74
0.05
49.54
155.4
Conv101
32
128
0.62
0.05
0.74
0.05
0.74
0.05
19.77
46.19
Conv102
32
288
0.67
0.05
0.83
0.05
0.83
0.05
7.45
46.19
Conv 103
128
32
0.62
0.05
0.74
0.05
0.74
0.05
56.52
155.4
Conv 104
32
128
0.47
0.05
0.74
0.05
0.74
0.05
19.67
46.19
Conv 105
32
288
0.59
0.05
0.83
0.05
0.83
0.05
11.28
46.19
Conv 106
128
32
0.66
0.05
0.74
0.05
0.74
0.05
55.28
155.4
Conv
32
128
0.53
0.05
0.74
0.05
0.74
0.05
20.02
46.19
Conv 108
32
288
0.63
0.05
0.83
0.05
0.83
0.05
6.73
46.19
Conv 109
128
32
0.67
0.05
0.74
0.05
0.74
0.05
38.49
155.4
Conv 10
64
128
0.30
0.05
0.82
0.05
0.82
0.05
102.5
83.68
Conv 111
64
576
0.51
0.05
0.97
0.05
0.97
0.05
20.27
83.68
Conv 1 12
256
64
0.18
0.05
0.90
0.05
0.90
0.05
228.72
294.32
Conv
64
256
0.63
0.05
0.90
0.05
0.90
0.05
19.19
83.68
Conv 14
64
576
0.51
0.05
0.97
0.05
0.97
0.05
13.0
83.68
Conv 15
256
64
0.59
0.05
0.90
0.05
0.90
0.05
59.58
294.32
Conv 1 16
64
256
0.67
0.05
0.90
0.05
0.90
0.05
18.46
83.68
Conv
64
576
0.47
0.05
0.97
0.05
0.97
0.05
11.64
83.68
Conv 1 18
256
64
0.64
0.05
0.90
0.05
0.90
0.05
59.16
294.32
Conv1 19
64
256
0.65
0.05
0.90
0.05
0.90
0.05
18.95
83.68
Conv 20
64
576
0.44
0.05
0.97
0.05
0.97
0.05
12.67
83.68
Conv 121
256
64
0.63
0.05
0.90
0.05
0.90
0.05
47.73
294.32
Conv 122
64
256
0.54
0.05
0.90
0.05
0.90
0.05
19.16
83.68
Conv 1
64
576
0.63
0.05
0.97
0.05
0.97
0.05
10.26
83.68
Conv
256
64
0.81
0.05
0.90
0.05
0.90
0.05
42.2
294.32
59

---

Under review as a conference paper at ICLR 2021
Conv 12n
64
256
0.59
0.05
0.90
0.05
0.90
0.05
24.01
83.68
Conv 126
64
576
0.69
0.05
0.97
0.05
0.97
0.05
11.76
83.68
Conv 127
256
64
0.65
0.05
0.90
0.05
0.90
0.05
39.33
294.32
Conv 128
64
256
0.28
0.05
0.90
0.05
0.90
0.05
25.97
83.68
Conv122
64
576
0.60
0.05
0.97
0.05
0.97
0.05
13.68
83.68
Conv 130
256
64
0.51
0,05
0.90
0.05
0.90
0.05
62.74
294.32
Conv133
64
256
0.18
0.05
0.90
0.05
0.90
0.05
37.84
83.68
Conv 132
64
576
0.55
0.05
0.97
0.05
0.97
0.05
16.28
83.68
Conv 133
256
64
0.43
0.05
0.90
0.05
0.90
0.05
98.19
294.32
Conv 134
64
256
0.43
0.05
0.90
0.05
0.90
0.05
39.47
83.68
Conv 135
576
0.47
0.05
0.97
0.05
0.97
0.05
13.86
83.68
Conv 1 36
256
64
0.71
0.05
0.90
0.05
0.90
0.05
65.95
294.32
Conv 137
64
256
0.64
0.05
0.90
0.05
0.90
0.05
44.63
83.68
Conv 138
64
576
0.65
0.05
0.97
0.05
0.97
0.05
13,91
83.68
Conv 1 139
256
64
0.53
0.05
0.90
0.05
0.90
0.05
70.45
294.32
Conv 140
64
256
0.59
0.05
0.90
0.05
0.90
0.05
44.73
83.68
Conv 141
64
576
0.68
0.05
0.97
0.05
0.97
0.05
11.5
83.68
Conv 142
256
64
0.55
0.05
0.90
0.05
0.90
0.05
84.15
294.32
Conv 143
64
256
0.52
0.05
0.90
0.05
0.90
0.05
45.81
83.68
Conv 144
64
576
0.64
0.05
0.97
0.05
0.97
0.05
2.99
83.68
Conv 145
256
64
0.58
0.05
0.90
0.05
0.90
0.05
80.95
294.32
Conv 46
64
256
0.49
0.05
0.90
0.05
0.90
0.05
43.23
83.68
Conv 147
64
576
0.67
0.05
0.97
0.05
0.97
0.05
11.54
83.68
Conv 148
256
64
0.60
0.05
0.90
0.05
0.90
0.05
76.94
294.32
Conv 149
64
256
0.53
0.05
0.90
0.05
0.90
0.05
30.26
83.68
Conv 150
64
576
0.69
0.05
0.97
0.05
0.97
0.05
6.73
83.68
Conv 151
256
64
0.48
0.05
0.90
0.05
0.90
0.05
81.75
294.32
Conv 152
64
256
0.56
0.05
0.90
0.05
0.90
0.05
33.02
83.68
Conv
64
576
0.81
0.05
0.97
0.05
0.97
0.05
8.73
83.68
Conv 1 54
256
64
0.55
0.05
0.90
0.05
0.90
0.05
91.86
294.32
Conv 155
64
256
0.59
0.05
0.90
0.05
0.90
0.05
29.47
83.68
Conv 156
64
576
0.83
0.05
0.97
0.05
0.97
0.05
11.58
83.68
Conv157
256
64
0.60
0.05
0.90
0.05
0.90
0.05
76.05
294.32
Conv 158
64
256
0.47
0.05
0.90
0.05
0.90
0.05
41.88
83.68
Conv 159
64
576
0.82
0.05
0.97
0.05
0.97
0.05
15.7
83.68
Conv 1lw666 160
256
64
0.53
0.05
0.90
0.05
0.90
0.05
64.48
294.32
Conv 161
64
256
0.51
0.05
0.90
0.05
0.90
0.05
68.02
83.68
Conv 162
64
576
0.81
0.05
0.97
0.05
0.97
0.05
12.04
83.68
Conv 163
256
64
0.61
0.05
0.90
0.05
0.90
0.05
51.41
294.32
Passing rate
100.0%
100.0%
100.0%
96.93%
Table 19: Cifar100 Gladelta-ResNetlle
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
16
27
0.11
0.05
0.58
0.05
0.58
0.05
8.49
26.3
Conv2
16
16
0.43
0.05
0.56
0.05
0.56
0.05
9.33
26.3
Conv3
16
144
0.56
0.05
0.68
0.05
0.68
0.05
4.11
26.3
Conv4
64
16
0.44
0.05
0.63
0.05
0.63
0.05
45.93
83.68
C
16
64
0.53
0.05
0.63
0.05
0.63
0.05
3.17
26.3
Conv6
16
144
0.56
0.05
0.68
0.05
0.68
0.05
6.72
26.3
Conv7
64
16
0.64
0.05
0.63
0.05
0.63
0.05
23.93
83.68
Conv8
16
64
0.62
0.05
0.63
0.05
0.63
0.05
3,05
26.3
Conv9
16
144
0.67
0.05
0.68
0.05
0.68
0.05
2.49
26.3
Conv10
64
16
0.67
0.05
0.63
0.05
0.63
0.05
24.31
83.68
60

---

Under review as a conference paper at ICLR 2021
Conv 11
16
64
0.44
0.05
0.63
0.05
0.63
0.05
8.09
26.3
Conv 12
16
144
0.43
0.05
0.68
0.05
0.68
0.05
4.44
26.3
Conv 13
64
16
0.59
0.05
0.63
0.05
0.63
0.05
26.43
83.68
Conv 14
16
64
0.61
0.05
0.63
0.05
0.63
0.05
3.06
26.3
Conv
16
144
0.54
0.05
0.68
0.05
0.68
0.05
2.9
26.3
Conv 16
64
16
0.62
0.05
0.63
0,05
0.63
0.05
21.66
83.68
Conv17
16
64
0.63
0.05
0.63
0.05
0.63
0.05
2.38
26.3
Conv 18
16
144
0.66
0.05
0.68
0.05
0.68
0.05
2.37
26.3
Conv19
64
16
0.62
0.05
0.63
0.05
0.63
0.05
22.61
83.68
Conv20
16
64
0.65
0.05
0.63
0.05
0.63
0.05
2.13
26.3
Conv21
16
144
0.81
0.05
0.68
0.05
0.68
0.05
1.53
26.3
Conv22
64
16
0.80
0.05
0.63
0.05
0.63
0.05
17.96
83.68
Comv23
16
64
0.62
0.05
0.63
0.05
0.63
0.05
3.82
26.3
Conv24
16
144
0.64
0.05
0.68
0.05
0.68
0.05
1.5
26.3
Conv25
64
16
0.70
0.05
0.63
0.05
0.63
0.05
13.74
83.68
Conv26
16
64
0.59
0.05
0.63
0.05
0.63
0.05
3.75
26.3
Conv27
16
144
0.62
0.05
0.68
0.05
0.68
0.05
1.67
26.3
Conv28
64
16
0.63
0.05
0.63
0.05
0.63
0.05
14.36
83.68
Conv29
16
64
0.67
0.05
0.63
0.05
0.63
0.05
1.62
26.3
Conv30
16
144
0.68
0.05
0.68
0.05
0.68
0.05
1.75
26.3
Conv31
64
16
0.80
0.05
0.63
0.05
0.63
0.05
11.87
83.68
Conv32
16
64
0.61
0.05
0.63
0.05
0.63
0.05
3.11
26.3
Conv33
16
144
0.69
0.05
0.68
0.05
0.68
0.05
1.36
26.3
Conv34
64
16
0.71
0.05
0.63
0.05
0.63
0.05
13.17
83.68
Conv35
16
64
0.80
0.05
0.63
0.05
0.63
0.05
1.92
26.3
Conv36
16
144
0.70
0.05
0.68
0.05
0.68
0.05
2.08
26.3
Conv37
64
16
0.70
0.05
0.63
0.05
0.63
0.05
12.17
83.68
Conv38
16
64
0.57
0.05
0.63
0.05
0.63
0.05
5.96
26.3
Conv39
16
144
0.67
0.05
0.68
0.05
0.68
0.05
1.45
26.3
Conv40
64
16
0.80
0.05
0.63
0.05
0.63
0.05
11.82
83.68
Conv41
16
64
0.46
0.05
0.63
0.05
0.63
0.05
5.85
26.3
Conv42
16
144
0.81
0.05
0.68
0.05
0.68
0.05
1.76
26.3
Conv43
64
16
0.67
0.05
0.63
0.05
0.63
0.05
13.14
83.68
Conv44
16
64
0.81
0.05
0.63
0.05
0.63
0.05
2.09
26.3
Conv45
16
144
0.82
0.05
0.68
0.05
0.68
0.05
1.25
26.3
Conv46
64
16
0.67
0.05
0.63
0.05
0.63
0.05
9.63
83.68
Conv47
16
64
0.70
0.05
0.63
0.05
0.63
0.05
3.38
26.3
Conv48
16
144
0.68
0.05
0.68
0.05
0.68
0.05
1.92
26.3
Conv49
64
16
0.81
0.05
0.63
0.05
0.63
0.05
12.43
83.68
Conv50
16
64
0.52
0.05
0.63
0.05
0.63
0.05
5.57
26.3
Conv51
16
144
0.62
0.05
0.68
0.05
0.68
0.05
2.08
26.3
Conv52
64
16
0.70
0.05
0.63
0.05
0.63
0.05
10.8
83.68
Conv53
16
64
0.62
0.05
0.63
0.05
0.63
0.05
4.59
26.3
Conv54
16
144
0.62
0.05
0.68
0.05
0.68
0.05
2.06
26.3
Conv55
64
16
0.80
0.05
0.63
0.05
0.63
0.05
10.21
83.68
Conv56
32
64
0.45
0.05
0.67
0.05
0.67
0.05
10.67
46.19
Conv57
32
288
0.65
0.05
0.83
0.05
0.83
0.05
1.79
46.19
Conv58
128
32
0.41
0.05
0.74
0.05
0.74
0.05
56.17
155.4
Conv59
32
128
0.81
0.05
0.74
0.05
0.74
0.05
2.03
46.19
Conv60
32
288
0.70
0.05
0.83
0.05
0.83
0.05
1.72
46.19
Conv61
128
32
0.62
0.05
0.74
0.05
0.74
0.05
19.79
155.4
Conv62
32
128
0.83
0.05
0.74
0.05
0.74
0.05
1.82
46.19
Conv63
32
288
0.80
0.05
0.83
0.05
0.83
0.05
1.2
46.19
Conv64
128
32
0.64
0.05
0.74
0.05
0.74
0.05
16.34
155.4
Conv65
32
128
0.58
0.05
0.74
0.05
0.74
0.05
2.33
46.19
Conv66
32
288
0.71
0.05
0.83
0.05
0.83
0.05
1.5
46.19
Conv67
128
32
0.61
0.05
0.74
0.05
0.74
0.05
12.64
155.4
Conv68
32
128
0.58
0.05
0.74
0.05
0.74
0.05
2.18
46.19
Conv69
32
288
0.70
0.05
0.83
0.05
0.83
0.05
2.15
46.19
61

---

Under review as a conference paper at ICLR 2021
Conv70
128
32
0.71
0.05
0.74
0.05
0.74
0.05
12.57
155.4
Conv
32
128
0.68
0.05
0.74
0.05
0.74
0.05
2.74
46.19
Conv72
32
288
0.66
0.05
0.83
0.05
0.83
0.05
1.38
46.19
Conv73 73
128
32
0.67
0.05
0.74
0.05
0.74
0.05
12.65
155.4
Conv74
32
128
0.53
0.05
0.74
0.05
0.74
0.05
4.58
46.19
Conv75 75
32
288
0.68
0,05
0.83
0,05
0.83
0.05
1.19
46.19
Conv76
128
32
0.61
0.05
0.74
0.05
0.74
0.05
17.44
155.4
Conv77 77
32
128
0.67
0.05
0.74
0.05
0.74
0.05
1.33
46.19
Conv78
32
288
0.70
0.05
0.83
0.05
0.83
0.05
0.77
46.19
Conv79
128
32
0.68
0.05
0.74
0.05
0.74
0.05
14.49
155.4
Conv80
32
128
0.69
0.05
0.74
0.05
0.74
0.05
2.1
46.19
Conv81
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.19
46.19
Conv82
128
32
0.63
0.05
0.74
0.05
0.74
0.05
8.48
153.4
Conv83
32
128
0.83
0.05
0.74
0.05
0.74
0.05
1.89
46.19
Conv84
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.3
46.19
Conv85
128
32
0.81
0.05
0.74
0.05
0.74
0.05
10.26
155.4
Conv86
32
128
0.70
0.05
0.74
0.05
0.74
0.05
2.94
46.19
Conv87
32
288
0.83
0.05
0.83
0.05
0.83
0.05
0.79
46.19
Conv88
128
32
0.83
0.05
0.74
0.05
0.74
0.05
11.34
155.4
Conv89
32
128
0.82
0.05
0.74
0.05
0.74
0.05
3.05
46.19
Conv90
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.1
46.19
Conv91
128
32
0.84
0.05
0.74
0.05
0.74
0.05
10.63
155.4
Conv92
32
128
0.70
0.05
0.74
0.05
0.74
0.05
1.86
46.19
Conv93
32
288
0.81
0.05
0.83
0.05
0.83
0.05
0.7
46.19
Conv94
128
32
0.84
0.05
0.74
0.05
0.74
0.05
11.17
155.4
Conv95
32
128
0.81
0.05
0.74
0.05
0.74
0.05
2.22
46.19
Conv96
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.9
46.19
Conv97
128
32
0.69
0.05
0.74
0.05
0.74
0.05
10.03
155.4
Conv98
32
128
0.68
0.05
0.74
0.05
0.74
0.05
2.21
46.19
Conv99
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.6
46.19
Conv 100
128
32
0.70
0.05
0.74
0.05
0.74
0.05
12.43
155.4
Conv 101
32
128
0.68
0.05
0.74
0.05
0.74
0.05
4.78
46.19
Conv102
32
288
0.68
0.05
0.83
0.05
0.83
0.05
0.57
46.19
Conv 103
128
32
0.81
0.05
0.74
0.05
0.74
0.05
11.88
155.4
Conv 104
32
128
0.61
0.05
0.74
0.05
0.74
0.05
1.83
46.19
Conv 105
32
288
0.87
0.05
0.83
0.05
0.83
0.05
1.61
46.19
Conv106
128
32
0.80
0.05
0.74
0.05
0.74
0.05
12.72
155.4
Conv 107
32
128
0.60
0.05
0.74
0.05
0.74
0.05
1.77
46.19
Conv 108
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.81
46.19
Conv 109
128
32
0.82
0.05
0.74
0.05
0.74
0.05
11.31
155.4
Conv1 10
64
128
0.50
0.05
0.82
0.05
0.82
0.05
5.15
83.68
Conv 11
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.7
83.68
Conv 12
256
64
0.51
0.05
0.90
0.05
0.90
0.05
50.78
294.32
Conv 13
64
256
0.64
0.05
0.90
0.05
0.90
0.05
3.01
83.68
Conv 14
64
576
0.59
0.05
0.97
0.05
0.97
0.05
2.53
83.68
Conv 15
256
64
0.60
0.05
0.90
0.05
0.90
0.05
26.61
294.32
Conv 1 16
64
256
0.47
0.05
0.90
0.05
0.90
0.05
3.4
83.68
Conv1
64
576
0.67
0.05
0.97
0.05
0.97
0.05
1.73
83.68
Conv 18
256
64
0.66
0.05
0.90
0.05
0.90
0.05
21.71
294.32
Conv
64
256
0.81
0.05
0.90
0.05
0.90
0.05
2.6
83.68
Conv 120
64
576
0.69
0.05
0.97
0.05
0.97
0.05
1.49
83.68
Conv 121
256
64
0.57
0.05
0.90
0.05
0.90
0.05
21.92
294.32
Conv 122
64
256
0.67
0.05
0.90
0.05
0.90
0.05
2.51
83.68
Conv1 123
64
576
0.81
0.05
0.97
0.05
0.97
0.05
0.8
83.68
Conv
256
64
0.70
0.05
0.90
0.05
0.90
0.05
17.85
294.32
Conv 125
64
256
0.63
0.05
0.90
0.05
0.90
0.05
2.41
83.68
Conv 126
64
576
0.71
0.05
0.97
0.05
0.97
0.05
1.37
83.68
Conv 127
256
64
0.65
0.05
0.90
0.05
0.90
0.05
22.35
294.32
Conv
64
256
0.67
0.05
0.90
0.05
0.90
0.05
3.24
83.68
62

---

Under review as a conference paper at ICLR 2021
Conv 1mn
64
576
0.70
0.05
0.97
0.05
0.97
0.05
1.04
83.68
Conv 130
256
64
0.61
0.05
0.90
0.05
0.90
0.05
23.91
294.32
Conv 131
64
256
0.68
0.05
0.90
0.05
0.90
0.05
2.08
83.68
Conv 1 132
64
576
0.81
0.05
0.97
0.05
0.97
0.05
0.98
83.68
Conv133
256
64
0.80
0.05
0.90
0.05
0.90
0.05
19.13
294.32
Conv 134
64
256
0.68
0,05
0.90
0.05
0,90
0.05
2.21
83.68
Conv
64
576
0.81
0.05
0.97
0.05
0.97
0.05
0.9
83.68
Conv 136
256
64
0.68
0.05
0.90
0.05
0.90
0.05
19.27
294.32
Conv 137
64
256
0.68
0.05
0.90
0.05
0.90
0.05
2.7
83.68
Conv 138
64
576
0.83
0.05
0.97
0.05
0.97
0.05
1.1
83.68
Conv 139
256
64
0.69
0.05
0.90
0.05
0.90
0.05
22.22
294.32
Conv 1 140
64
256
0.56
0.05
0.90
0.05
0.90
0.05
1.93
83.68
Conv 141
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.87
83.68
Conv 142
256
64
0.65
0.05
0.90
0.05
0.90
0.05
18.57
294.32
Conv 143
64
256
0.67
0.05
0.90
0.05
0.90
0.05
2.23
83.68
Conv 144
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.94
83.68
Conv 145
256
64
0.63
0.05
0.90
0.05
0.90
0.05
21.58
294.32
Conv l 146
64
256
0.65
0.05
0.90
0.05
0.90
0.05
2.96
83.68
Conv 147
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.1
83.68
Conv 148
256
64
0.66
0.05
0.90
0.05
0.90
0.05
27.04
294.32
Conv 149
64
256
0.61
0.05
0.90
0.05
0.90
0.05
3.3
83.68
Conv 50
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.85
83.68
Conv 151
256
64
0.67
0.05
0.90
0.05
0.90
0.05
23.12
294.32
Conv 152
64
256
0.59
0.05
0.90
0.05
0.90
0.05
2.56
83.68
Conv 153
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.68
83.68
Conv 154
256
64
0.59
0.05
0.90
0.05
0.90
0.05
30.21
294.32
Conv 155
64
256
0.62
0.05
0.90
0.05
0.90
0.05
2.84
83.68
Conv 156
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.85
83.68
Conv
256
64
0.65
0.05
0.90
0.05
0.90
0.05
32.78
294.32
Conv 1 58
64
256
0.55
0.05
0.90
0.05
0.90
0.05
2.77
83.68
Conv 159
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.88
83.68
Conv 160
256
64
0.57
0.05
0.90
0.05
0.90
0.05
37.46
294.32
Conv161
64
256
0.54
0.05
0.90
0.05
0.90
0.05
3.55
83.68
Conv 162
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.82
83.68
Conv163
256
64
0.56
0.05
0.90
0.05
0.90
0.05
48.35
294.32
Passing rate
100.0%
100.0%
100.0%
100.00%
P.3 REGULARIZATION
config:
Table 20: WRN28-10 Cifar100 &
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv
16
27
0.05
0.05
0.58
0.05
0.58
0.05
98.95
26.3
Conv2
160
144
0.69
0.05
0.94
0.05
0.94
0.05
8.19
190.52
Conv3
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.31
190.52
63

---

Under review as a conference paper at ICLR 2021
Conv4
160
16
0.58
0.05
0.69
0.05
0.69
0.05
76.38
190.52
Conv5
160
1440
0.92
0.05
1.00
0.05
1.00
0.05
0.2
190.52
Conv6
160
1440
0.92
0.05
1.00
0.05
1.00
0,05
0.25
190.52
Conv7
160
1440
0.91
0.05
1.00
0.05
1.00
0.05
0.06
190.52
Conv8
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.25
190.52
Conv9
160
1440
0.92
0.05
1.00
0.05
1.00
0.05
0.03
190.52
Conv10
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.95
190.52
Conv 1
320
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.12
362.72
Conv 12
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.19
362.72
Conv13
320
160
0.84
0.05
0.99
0.05
0.99
0.05
4.26
362.72
Conv14 14
320
2880
0.93
0.05
1.00
0.05
1.00
0.05
362.72
Conv 15
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.16
362.72
Conv 16
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 17
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.26
362.72
Conv 18
320
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv19 19
320
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.52
362.72
Conv20
640
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv21
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.64
699.96
Conv22
640
320
0.91
0.05
1.00
0.05
1.00
0.05
2.88
699.96
Conv23
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.1
699.96
Conv24
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.49
699.96
Conv25
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv26
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.46
699.96
Conv27
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.06
699.96
Conv28
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
2.74
699.96
Passing rate
100.0%
100.0%
100.0%
96.43%
Table 21: WRN28-10
Cifar100 RReLU
Layer
Number
dim
Gaussian
Mean
Mean_Righ
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Convl
16
27
0.43
0.05
0.58
0.05
0.58
0.05
40.44
26.3
Conv2
160
144
0.93
0.05
0.94
0.05
0.94
0.05
2.71
190.52
Conv3
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.12
190.52
Conv4
160
16
0.82
0.05
0.69
0.05
0.69
0.05
31.44
190.52
Conv5
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.06
190.52
Conv6
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.12
190.52
Conv7
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.02
190.52
Conv8
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.2
190.52
Conv9
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.02
190.52
Conv10
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.3
190.52
Conv
320
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.09
362.72
Conv 12
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.09
362.72
Conv1 13
320
160
0.90
0.05
0.99
0.05
0.99
0.05
2.73
362.72
Conv 14
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 15
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.07
362.72
Conv 16
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.03
362.72
Conv 17
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.14
362.72
Conv
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 19
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.31
362.72
Conv20
640
2880
0.93
0.05
1.00
0.05
1,00
0.05
0.05
699.96
Conv21
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.33
699.96
Conv22
640
320
0.93
0.05
1.00
0.05
1.00
0.05
1.54
699.96
Conv23
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.07
699.96
Cony24
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.28
699.96
64

---

Under review as a conference paper at ICLR 2021
Conv25
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv26
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.35
699.96
Conv27
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.04
699.96
Conv28
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
1.23
699.96
Passing rate
100.0%
100.0%
100.0%
96.43%
Table 22: WRN28-10 Cifar100 Dropact
Layer
Number
dim
Gaussian
Left_Left
Mean
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 I
16
27
0.54
0.05
0.58
0.05
0.58
0,05
19.13
26.3
Conv2
160
144
0.92
0.05
0.94
0.05
0.94
0.05
1.61
190.52
Conv3
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.11
190.52
Conv4
160
16
0.85
0.05
0.69
0.05
0.69
0.05
17.31
190.52
Conv5
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.06
190.52
Conv6
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.11
190.52
Conv7
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.01
190.52
Conv8
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.14
190.52
Conv9
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.02
190.52
Conv10
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.38
190.52
Conv
320
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv 12
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.1
362.72
Conv 13
320
160
0.90
0.05
0.99
0.05
0.99
0.05
0.92
362.72
Conv 14
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.03
362.72
Conv 15
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.08
362.72
Conv 16
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.03
362.72
Conv 17
320
2880
0.95
0.05
1.00
0.05
1,00
0.05
0.08
362.72
Conv 18
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 19
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.23
362.72
Conv20
640
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv21
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.61
699.96
Conv22
640
320
0.94
0.05
1.00
0.05
1.00
0.05
1.53
699.96
Conv23
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.06
699.96
Cony24
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.43
699.96
Conv25
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv26
640
5760
0.96
0.05
1.00
0.05
1.00
0.05
0.28
699.96
Conv27
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.04
699.96
Conv28
640
5760
0.96
0.05
1.00
0.05
1.00
0.05
0.36
699.96
Passing rate
100.0%
100.0%
100.0%
100.0%
Table 23: WRN28-10 Cifar100 Autoaugment
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Convl 1
16
27
0.05
0.05
0.58
0.05
0.58
0.05
100.18
26.3
Conv2
160
144
0.69
0.05
0.94
0.05
0.94
0.05
8.3
190.52
Conv3
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.31
190.52
Conv4
160
16
0.57
0.05
0.69
0.05
0.69
0.05
77.32
190.52
Conv5
160
1440
0.92
0.05
1.00
0.05
1.00
0.05
0.2
190.52
Conv6
160
1440
0.92
0.05
1.00
0.05
1.00
0.05
0.25
190.52
Conv7
160
1440
0.91
0.05
1.00
0.05
1.00
0.05
0.06
190.52
65

---

Under review as a conference paper at ICLR 2021
Conv8
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.26
190.52
Conv9
160
1440
0.92
0.05
1.00
0.05
1.00
0.05
0.03
190.52
Conv 10
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.97
190.52
Conv1
320
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.12
362.72
Conv 12
320
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.19
362.72
Conv 13
320
160
0.84
0.05
0.99
0.05
0.99
0.05
4.31
362.72
Conv 14
320
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.1
362.72
Conv 15
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.16
362.72
Conv
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 17
320
2880
0.94
0.05
1.00
0.05
1,00
0.05
0.26
362.72
Conv 18
320
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv19 19
320
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.52
362.72
Comv20
640
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv21
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.65
699.96
Conv22
640
320
0.91
0.05
1.00
0.05
1.00
0.05
2.92
699.96
Conv23
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.1
699.96
Conv24
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.5
699.96
Conv25
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv26
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.47
699.96
Conv27
640
5760
0.93
0.05
1.00
0.05
1.00
0.05
0.06
699.96
Conv28
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
2.77
699.96
Passing rate
100.0%
100.0%
100.0%
96.43%
Table 24: WRN28-10 C0 Cutout
Layer
Number
dim
Gaussian
Mean_Left
Mean_ Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.31
0.05
0.58
0.05
0.58
0.05
49.46
26.3
Conv2
160
144
0.88
0.05
0.94
0.05
0.94
0.05
5.66
190.52
Conv3
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.28
190.52
Conv4
160
16
0.81
0.05
0.69
0.05
0.69
0.05
42.43
190.52
5
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.18
190.52
Conv6
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.23
190.52
Conv7
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.02
190.52
Conv8
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.22
190.52
Conv9
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.06
190.52
Conv 10
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.83
190.52
Conv1
320
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.1
362.72
Conv 12
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.18
362.72
Conv 13
320
160
0.89
0.05
0.99
0.05
0.99
0.05
1.71
362,72
Conv 14
320
2880
0.94
0.05
1.00
0.05
1,00
0.05
0.05
362.72
Conv15 15
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.12
362.72
Conv 16
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv 17
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.16
362.72
Conv 18
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.07
362.72
Conv19 19
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.4
362.72
Conv20
640
2880
0.93
0.05
1.00
0.05
1.00
0.05
0.05
699,96
Conv21
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.75
699.96
Conv22
640
320
0.93
0.05
1.00
0.05
1.00
0.05
2.01
699.96
Conv23
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.12
699,96
Conv24
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.54
699.96
Conv25
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.1
699.96
Conv26
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.55
699.96
Conv27
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.09
699,96
Conv28
640
5760
0.96
0.05
1.00
0.05
1.00
0.05
0.63
699.96
66

---

Under review as a conference paper at ICLR 2021
Passing rate
100.0%
100.0%
100.0%
96.43%
Table 25: WRN28-10 C0 Cutmix
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.54
0.05
0.58
0.05
0.58
0.05
21.26
26.3
Conv2
160
144
0.91
0.05
0.94
0.05
0,94
0.05
2.25
190.52
Conv3
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.16
190.52
Conv4
160
16
0.86
0.05
0.69
0.05
0.69
0.05
19.09
190.52
Conv5
160
1440
0.93
0.05
1.00
0.05
1.00
0.05
0.08
190.52
Conv6
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.14
190.52
Conv7
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.02
190.52
Conv8
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.14
190.52
Conv9
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.03
190.52
Conv 10
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.29
190.52
Conv
320
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.07
362.72
Conv 12
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.12
362.72
Conv 13
320
160
0.92
0.05
0.99
0.05
0.99
0.05
1.09
362.72
Conv14 14
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv15 15
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.08
362.72
Conv 16
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 17
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.09
362.72
Conv18
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.05
362.72
Conv 19
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.26
362.72
Conv20
640
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv21
640
5760
0.95
0.05
1.00
0.05
1,00
0.05
0.85
699.96
Conv22
640
320
0.94
0.05
1.00
0.05
1.00
0.05
1.8
699.96
Conv23
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.09
699.96
Comv24
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.51
699.96
Conv25
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.07
699.96
Conv26
640
5760
0.96
0.05
1.00
0.05
1.00
0.05
0.22
699.96
Conv27
640
5760
0.96
0.05
1.00
0.05
1.00
0.05
0.04
699.96
Conv28
640
5760
0.96
0.05
1.00
0.05
1.00
0.05
0.24
699.96
Passing rate
100.0%
100.0%
100.0%
100.0%
P.4 ATTENTION MECHANISM
config:
attes://gttuussccom//rr///sss///
httpes//ggttub.cccmm/uuugggopppp
attes://ggtha..ccom/mmmmp/pppppp
Table SENet (ResNet164) C0
Layer
Number dim
Gaussian
Mean Left
Mean Right
Sigma
67

---

Under review as a conference paper at ICLR 2021
p-value
c-value
p-value
c-value
p-value
e-value
f-value
e-value
Conv1
16
27
0.02
0.05
0.58
0.05
0.58
0.05
181.46
26.3
Conv2
16
16
0.54
0.05
0.56
0.05
0.56
0.05
6.12
26.3
Conv3
16
144
0.86
0.05
0.68
0.05
0.68
0.05
0.42
26.3
Conv4
64
16
0.87
0,05
0.63
0,05
0.63
0.05
11.54
83.68
Conv5
16
64
0.82
0.05
0.63
0.05
0.63
0.05
0.89
26.3
Conv6
16
144
0.89
0.05
0.68
0.05
0.68
0.05
0.37
26.3
Conv7
64
16
0.80
0.05
0.63
0.05
0.63
0.05
10.59
83.68
Conv8
16
64
0.67
0.05
0.63
0.05
0.63
0.05
1.54
26.3
Conv9
16
144
0.82
0.05
0.68
0.05
0.68
0.05
1.34
Conv 10
64
16
0.64
0.05
0.63
0.05
0.63
0.05
11.73
83.68
Conv 11
16
64
0.81
0.05
0.63
0.05
0.63
0.05
1.01
26.3
Conv 12
16
144
0.84
0.05
0.68
0.05
0.68
0.05
1.63
26.3
Conv13 13
64
16
0.81
0.05
0.63
0.05
0.63
0.05
9.92
83.68
Conv 14
16
64
0.43
0.05
0.63
0.05
0.63
0.05
3,99
26.3
Conv 15
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.86
26.3
Conv 16
64
16
0.81
0.05
0.63
0.05
0.63
0.05
11.13
83.68
Conv 17
16
64
0.67
0.05
0.63
0.05
0.63
0.05
0.68
26.3
Conv18
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.61
26.3
Conv19
64
16
0.86
0.05
0.63
0.05
0.63
0.05
12.02
83.68
Conv20
16
64
0.87
0.05
0.63
0.05
0.63
0.05
0.65
26.3
Conv21
16
144
0.84
0.05
0.68
0.05
0.68
0.05
1.02
26.3
Conv22
64
16
0.85
0.05
0.63
0.05
0.63
0.05
7.73
83.68
Conv23
16
64
0.69
0.05
0.63
0.05
0.63
0.05
1.84
26.3
Conv24
16
144
0.89
0.05
0.68
0.05
0.68
0.05
1.07
26.3
Conv25
64
16
0.85
0.05
0.63
0.05
0.63
0.05
7.12
83.68
Conv26
16
64
0.69
0.05
0.63
0.05
0.63
0.05
1.85
26.3
Conv27
16
144
0.82
0.05
0.68
0.05
0.68
0.05
0.77
26.3
Conv28
64
16
0,86
0.05
0.63
0.05
0.63
0.05
14.12
83.68
Conv29
16
64
0.64
0.05
0.63
0.05
0.63
0.05
3.44
26.3
Conv30
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.81
26.3
Conv31
64
16
0.81
0.05
0.63
0.05
0.63
0.05
17.04
83.68
Conv32
16
64
0.68
0.05
0.63
0.05
0.63
0.05
2.15
26.3
Conv33
16
144
0.67
0.05
0.68
0.05
0.68
0.05
0.81
26.3
Conv34
64
16
0.71
0.05
0.63
0.05
0.63
0.05
23.95
83.68
Conv35
16
64
0.69
0.05
0.63
0.05
0.63
0.05
1.03
26.3
Conv36
16
144
0.89
0.05
0.68
0.05
0.68
0.05
0.1
26.3
Conv37
64
16
0.86
0.05
0.63
0.05
0.63
0.05
8.09
83.68
Conv38
16
64
0.68
0.05
0.63
0.05
0.63
0.05
3.55
26.3
Conv39
16
144
0.83
0.05
0.68
0.05
0.68
0.05
0.71
26.3
Conv40
64
16
0.84
0.05
0.63
0.05
0.63
0.05
11.45
83.68
Conv41
16
64
0.81
0.05
0.63
0.05
0.63
0.05
1.84
26.3
Conv42
16
144
0.82
0.05
0.68
0.05
0.68
0.05
26.3
Conv43
64
16
0.83
0.05
0.63
0.05
0.63
0,05
10.49
83.68
Conv44
16
64
0.81
0.05
0.63
0.05
0.63
0.05
2.3
26.3
Conv45
16
144
0.84
0.05
0.68
0.05
0.68
0.05
1.96
26.3
Conv46
64
16
0.46
0.05
0.63
0.05
0.63
0.05
18.47
83.68
Conv47
16
64
0.67
0.05
0.63
0.05
0.63
0.05
2.86
26.3
Conv48
16
144
0.82
0.05
0.68
0.05
0.68
0.05
0.84
26.3
Conv49
64
16
0.64
0.05
0.63
0.05
0.63
0.05
8.9
83.68
Conv50
16
64
0.82
0.05
0.63
0.05
0.63
0.05
3.4
26.3
Conv5
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.92
26.3
Conv52
64
16
0.71
0.05
0.63
0.05
0.63
0.05
14.29
83.68
Conv53
16
64
0.47
0.05
0.63
0.05
0.63
0.05
5.93
26.3
Conv54
16
144
0.86
0.05
0.68
0.05
0.68
0,05
0.92
26.3
Conv55
64
16
0.64
0.05
0.63
0.05
0.63
0.05
16.37
83.68
Conv56
32
64
0.51
0.05
0.67
0.05
0.67
0.05
3.25
46.19
Conv57
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.24
46.19
68

---

Under review as a conference paper at ICLR 2021
Conv58
128
32
0.80
0.05
0.74
0.05
0.74
0.05
14.02
155.4
Conv59
32
128
0.90
0.05
0.74
0.05
0.74
0.05
0.44
46.19
Conv60
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.37
46.19
Conv61
128
32
0.88
0.05
0.74
0.05
0.74
0.05
5.11
155.4
Conv62
32
128
0.85
0.05
0.74
0.05
0.74
0.05
0.72
46.19
Conv63
32
288
0.88
0,05
0.83
0,05
0.83
0.05
0.5
46.19
Conv64
128
32
0.85
0.05
0.74
0.05
0.74
0.05
5.3
155.4
Conv65
32
128
0.82
0.05
0.74
0.05
0.74
0.05
1.19
46.19
Conv66
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.05
46.19
Conv67
128
32
0.81
0.05
0.74
0.05
0.74
0.05
8.62
155.4
Conv68
128
0.87
0.05
0.74
0.05
0.74
0.05
0.41
46.19
Conv69
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.73
46.19
Conv70 70
128
32
0.85
0.05
0.74
0.05
0.74
0.05
5.21
153.4
Conv71 71
32
128
0.89
0.05
0.74
0.05
0.74
0.05
0.38
46.19
Conv72 72
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.5
46.19
C3
128
32
0.88
0.05
0.74
0.05
0.74
0.05
4.26
155.4
Conv74 74
32
128
0.83
0.05
0.74
0.05
0.74
0.05
0.85
46.19
Conv75
32
288
0.87
0.05
0.83
0.05
0.83
0.05
0.41
46.19
Conv 76
128
32
0.68
0.05
0.74
0.05
0.74
0.05
6.07
155.4
Conv77
32
128
0.85
0.05
0.74
0.05
0.74
0.05
0.52
46.19
Conv78
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.44
46.19
Conv79
128
32
0.84
0.05
0.74
0.05
0.74
0.05
4.39
155.4
Conv80
32
128
0.82
0.05
0.74
0.05
0.74
0.05
0.83
46.19
Conv81
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.75
46.19
Conv82
128
32
0.88
0.05
0.74
0.05
0.74
0.05
5.93
155.4
Conv83
32
128
0.68
0.05
0.74
0.05
0.74
0.05
2.33
46.19
Conv84
32
288
0.87
0.05
0.83
0.05
0.83
0.05
0.57
46.19
Conv85
128
32
0.84
0.05
0.74
0.05
0.74
0.05
8.22
155.4
Conv86
32
128
0.82
0.05
0.74
0.05
0.74
0.05
1.0
46.19
Conv87
32
288
0.81
0.05
0.83
0.05
0.83
0.05
0.43
46.19
Conv88
128
32
0.89
0.05
0.74
0.05
0.74
0.05
5.04
155.4
Conv89
32
128
0.86
0.05
0.74
0.05
0.74
0.05
1.08
46.19
Conv90
32
288
0.91
0.05
0.83
0.05
0.83
0.05
0.34
46.19
Conv91
128
32
0.87
0.05
0.74
0.05
0.74
0.05
5.44
155.4
Conv92
32
128
0.86
0.05
0.74
0.05
0.74
0.05
0.93
46.19
Conv93
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.24
46.19
Conv94
128
32
0.86
0.05
0.74
0.05
0.74
0.05
5.26
155.4
Conv95
32
128
0.83
0.05
0.74
0.05
0.74
0.05
0.92
46.19
Conv96
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.19
46.19
Conv97
128
32
0.87
0.05
0.74
0.05
0.74
0.05
5.4
155.4
Conv98
32
128
0.87
0.05
0.74
0.05
0.74
0.05
0.92
46.19
Conv99
32
288
0.86
0.05
0.83
0.05
0.83
0.05
0.4
46.19
Conv 100
128
32
0.88
0.05
0.74
0.05
0.74
0.05
5.19
155.4
Conv 101
32
128
0.85
0.05
0.74
0.05
0.74
0.05
1.2
46.19
Conv 102
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.37
46.19
Conv 103
128
32
0.70
0.05
0.74
0.05
0.74
0.05
6.58
155.4
Conv 1 04
32
128
0.85
0.05
0.74
0.05
0.74
0.05
1.03
46.19
Conv 105
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.15
46.19
Conv 106
128
32
0.87
0.05
0.74
0.05
0.74
0.05
6.01
155.4
Conv 107
32
128
0.86
0.05
0.74
0.05
0.74
0.05
0.98
46.19
Conv 108
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.19
46.19
Conv 109
128
32
0.85
0.05
0.74
0.05
0.74
0.05
6.87
155.4
Conv1 10
64
128
0.61
0.05
0.82
0.05
0.82
0.05
2.17
83.68
Conv1
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.18
83.68
Conv1 12
256
64
0.83
0.05
0.90
0.05
0.90
0.05
12.64
294.32
Conv 13
64
256
0.86
0.05
0.90
0.05
0.90
0.05
0.68
83.68
Conv 14
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.32
83.68
Conv 115
256
64
0.84
0.05
0.90
0.05
0.90
0.05
6.51
294.32
Conv
64
256
0.70
0.05
0.90
0.05
0.90
0.05
0.67
83.68
69

---

Under review as a conference paper at ICLR 2021
Conv 1 17
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.46
83.68
Conv
256
64
0.86
0.05
0.90
0.05
0.90
0.05
7.04
294.32
Conv 1 119
64
256
0.86
0.05
0.90
0.05
0.90
0.05
0.57
83.68
Conv 1 20
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.35
83.68
Conv12)
256
64
0.81
0.05
0.90
0.05
0.90
0.05
6.19
294.32
Conv 122
64
256
0.82
0,05
0.90
0.05
0,90
0.05
0.36
83.68
C23
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.35
83.68
Conv 1 24
256
64
0.86
0.05
0.90
0.05
0.90
0.05
6.66
294.32
Conv 125
64
256
0.87
0.05
0.90
0.05
0.90
0.05
0.4
83.68
Conv 126
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.29
83.68
Conv 127
256
64
0.85
0.05
0.90
0.05
0.90
0.05
7.23
294.32
Conv 28
64
256
0.86
0.05
0.90
0.05
0.90
0.05
0.47
83.68
Conv 129
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.25
83.68
Conv 130
256
64
0.86
0.05
0.90
0.05
0.90
0.05
6.13
294.32
Conv 131 131
64
256
0.86
0.05
0.90
0.05
0.90
0.05
0.56
83.68
Conv 132
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.3
83.68
Conv 133
256
64
0.86
0.05
0.90
0.05
0.90
0.05
6.56
294.32
Conv 1onv134
64
256
0.85
0.05
0.90
0.05
0.90
0.05
0.54
83.68
Conv 135
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.25
83.68
Conv 136
256
64
0.86
0.05
0.90
0.05
0.90
0.05
6.57
294.32
Conv 137
64
256
0.86
0.05
0.90
0.05
0.90
0.05
0.69
83.68
Conv 138
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.2
83.68
Conv 139
256
64
0.83
0.05
0.90
0.05
0.90
0.05
5.95
294.32
Conv 140
64
256
0.84
0.05
0.90
0.05
0.90
0.05
83.68
Conv 141
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.24
83.68
Conv 142
256
64
0.83
0.05
0.90
0.05
0.90
0.05
6.64
294.32
Conv 143
64
256
0.87
0.05
0.90
0.05
0.90
0.05
0.65
83.68
Conv 144
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.19
83.68
Conv 14n
256
64
0.83
0.05
0.90
0.05
0.90
0.05
7.35
294.32
Conv 146
64
256
0.69
0.05
0.90
0.05
0.90
0.05
0.66
83.68
Conv 147
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.14
83.68
Conv 148
256
64
0.84
0.05
0.90
0.05
0.90
0.05
7.95
294.32
Conv 149
64
256
0.85
0.05
0.90
0.05
0.90
0.05
0.59
83.68
Conv 150
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.18
83.68
Conv 151
256
64
0.82
0.05
0.90
0.05
0.90
0.05
6.92
294.32
Conv 152
64
256
0.82
0.05
0.90
0.05
0.90
0.05
0.42
83.68
Conv 153
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.14
83.68
Conv 154
256
64
0.85
0.05
0.90
0.05
0.90
0.05
7.74
294.32
Conv 155
64
256
0.82
0.05
0.90
0.05
0.90
0.05
0.58
83.68
Conv 156
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.16
83.68
Conv 157
256
64
0.84
0.05
0.90
0.05
0.90
0.05
7.89
294.32
Conv 15n
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.47
83.68
Conv 159
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.19
83.68
Conv160
256
64
0.85
0.05
0.90
0.05
0.90
0.05
8.11
294.32
Conv 161
64
256
0.81
0.05
0.90
0.05
0.90
0.05
0.83
83.68
Conv 162
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.15
83.68
Conv16 163
256
64
0.82
0.05
0.90
0.05
0.90
0.05
9.08
294.32
Passing rate
99.39%
100.0%
100.0%
99.39%
Table 27: DIANet (ResNet164) Cifar100
Layer
Number
dim
Gaussian
Mean
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.02
0.05
0.58
0.05
0.58
0.05
218.97
26.3
Conv2
16
16
0.41
0.05
0.56
0.05
0.56
0.05
16.01
26.3
70

---

Under review as a conference paper at ICLR 2021
C
16
144
0.54
0.05
0.68
0.05
0.68
0.05
2.13
26.3
Conv4
64
16
0.83
0.05
0.63
0.05
0.63
0.05
22.48
83.68
Conv5
16
64
0.82
0.05
0.63
0.05
0.63
0.05
1.21
26.3
Conv6
16
144
0.88
0.05
0.68
0.05
0.68
0.05
0.52
26.3
Conv7
64
16
0.83
0.05
0.63
0.05
0.63
0.05
10.96
83.68
Conv8
16
64
0.81
0,05
0.63
0,05
0.63
0.05
1.12
26.3
Conv9
16
144
0.88
0.05
0.68
0.05
0.68
0.05
0.65
26.3
Conv 10
64
16
0.86
0.05
0.63
0.05
0.63
0.05
6.11
83.68
Conv 1
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.32
26.3
Conv 12
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.4
26.3
Conv 13
64
16
0.86
0.05
0.63
0.05
0.63
0.05
10.91
83.68
Conv 14
16
64
0.81
0.05
0.63
0.05
0.63
0.05
2.18
26.3
Conv 15
16
144
0.69
0.05
0.68
0.05
0.68
0.05
1.12
26.3
Conv 16
64
16
0.82
0.05
0.63
0.05
0.63
0.05
24.69
83.68
Conv1 17
16
64
0.83
0.05
0.63
0.05
0.63
0.05
0.87
26.3
Conv18 18
16
144
0.85
0.05
0.68
0.05
0.68
0.05
0.32
26.3
Conv 19
64
16
0.85
0.05
0.63
0.05
0.63
0.05
10.98
83.68
Conv20
16
64
0.62
0.05
0.63
0.05
0.63
0.05
1.4
26.3
Conv21
16
144
0.89
0.05
0.68
0.05
0.68
0.05
0.41
26.3
Conv22
64
16
0.70
0.05
0.63
0.05
0.63
0.05
15.49
83.68
Conv23
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.03
26.3
Conv24
16
0.90
0.05
0.68
0.05
0.68
0.05
0.77
26.3
Conv25
64
16
0.83
0.05
0.63
0.05
0.63
0.05
10.43
83.68
Conv26
16
64
0.82
0.05
0.63
0.05
0.63
0.05
1.59
26.3
Conv27
16
144
0.86
0.05
0.68
0.05
0.68
0.05
0.47
26.3
Conv28
64
16
0.71
0.05
0.63
0.05
0.63
0.05
11.21
83.68
Conv29
16
64
0.65
0.05
0.63
0.05
0.63
0.05
2.22
26.3
Conv30
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.78
26.3
Conv31
64
16
0.86
0.05
0.63
0.05
0.63
0.05
9.33
83.68
Conv32
16
64
0.62
0.05
0.63
0.05
0.63
0.05
2.48
26.3
Conv33
16
144
0.85
0.05
0.68
0.05
0.68
0.05
1.23
26.3
Conv34
64
16
0.81
0.05
0.63
0.05
0.63
0.05
10.78
83.68
Conv35
16
64
0.59
0.05
0.63
0.05
0.63
0.05
3.47
26.3
Conv36
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.78
26.3
Conv37
64
16
0.83
0.05
0.63
0.05
0.63
0.05
17.75
83.68
Conv38
16
64
0.87
0.05
0.63
0.05
0.63
0.05
0.81
26.3
Conv39
16
144
0.85
0.05
0.68
0.05
0.68
0.05
0.57
26.3
Conv40
64
16
0.84
0.05
0.63
0.05
0.63
0.05
12.64
83.68
Conv41
16
64
0.67
0.05
0.63
0.05
0.63
0.05
1.94
26.3
Conv42
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.61
26.3
Conv43
64
16
0.69
0.05
0.63
0.05
0.63
0.05
10.1
83.68
Conv44
16
64
0.63
0.05
0.63
0.05
0.63
0.05
3.55
26.3
Conv45
16
144
0.70
0.05
0.68
0.05
0.68
0.05
1.23
26.3
Conv46
64
16
0.61
0.05
0.63
0.05
0.63
0.05
49.74
83.68
Conv47
16
64
0.88
0.05
0.63
0.05
0.63
0,05
1.3
26.3
Conv48
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.34
26.3
Conv49
64
16
0.85
0.05
0.63
0.05
0.63
0.05
7.77
83.68
Conv50
16
64
0.81
0.05
0.63
0.05
0.63
0.05
1.92
26.3
Conv51
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.67
26.3
Conv52
64
16
0.82
0.05
0.63
0.05
0.63
0.05
13.42
83.68
Conv53
16
64
0.69
0.05
0.63
0.05
0.63
0.05
3.6
26.3
Conv54
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.83
26.3
Conv55
64
16
0.70
0.05
0.63
0.05
0.63
0.05
24.26
83.68
Conv56
32
64
0.62
0.05
0.67
0.05
0.67
0.05
6.86
46.19
Conv57
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.39
46.19
Conv58
128
32
0.61
0.05
0.74
0.05
0.74
0.05
26.14
155.4
Conv59
32
128
0.87
0.05
0.74
0.05
0.74
0.05
0.65
46.19
Conv60
32
288
0.87
0.05
0.83
0.05
0.83
0.05
0.44
46.19
Conv61
128
32
0.85
0.05
0.74
0.05
0.74
0.05
10.89
155.4
71

---

Under review as a conference paper at ICLR 2021
Conv62
32
128
0.86
0.05
0.74
0.05
0.74
0.05
0.76
46.19
Conv63
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.62
46.19
Conv64
128
32
0.84
0.05
0.74
0.05
0.74
0.05
9.18
155.4
Conv65
32
128
0.87
0.05
0.74
0.05
0.74
0.05
0.53
46.19
Conv66
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.56
46.19
Conv67
128
32
0.84
0,05
0.74
0,05
0.74
0.05
6.94
153.4
Conv68
32
128
0.89
0.05
0.74
0.05
0.74
0.05
0.53
46.19
Conv69
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.45
46.19
Conv70
128
32
0.65
0.05
0.74
0.05
0.74
0.05
8.26
155.4
Conv71
32
128
0.88
0.05
0.74
0.05
0.74
0.05
0.62
46.19
Conv72
32
288
0.70
0.05
0.83
0.05
0.83
0.05
0.41
46.19
Conv73
128
32
0.83
0.05
0.74
0.05
0.74
0.05
6.53
155.4
Conv74
32
128
0.69
0.05
0.74
0.05
0.74
0.05
0.77
46.19
Conv75
32
288
0.91
0.05
0.83
0.05
0.83
0.05
0.27
46.19
Conv76
128
32
0.85
0.05
0.74
0.05
0.74
0.05
9.43
155.4
Conv77
32
128
0.68
0.05
0.74
0.05
0.74
0.05
1.07
46.19
Conv78 78
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.36
46.19
Conv79
128
32
0.84
0.05
0.74
0.05
0.74
0.05
10.11
155.4
Conv80
32
128
0.83
0.05
0.74
0.05
0.74
0.05
0.59
46.19
Conv81
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.29
46.19
Conv82
128
32
0.89
0.05
0.74
0.05
0.74
0.05
6.81
155.4
Conv83
32
128
0.82
0.05
0.74
0.05
0.74
0.05
1.28
46.19
Conv84
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.63
46.19
Conv85
128
32
0.84
0.05
0.74
0.05
0.74
0.05
12.87
155.4
Conv86
32
128
0.69
0.05
0.74
0.05
0.74
0.05
0.58
46.19
Conv87
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.28
46.19
Conv88
128
32
0.86
0.05
0.74
0.05
0.74
0.05
8.96
155.4
Conv89
32
128
0.84
0.05
0.74
0.05
0.74
0.05
1.59
46.19
Conv90
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.1
46.19
Conv91
128
32
0.87
0.05
0.74
0.05
0.74
0.05
9.08
155.4
Conv92
32
128
0.84
0.05
0.74
0.05
0.74
0.05
1.02
46.19
Conv93
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.09
46.19
Conv94
128
32
0.85
0.05
0.74
0.05
0.74
0.05
8.1
155.4
Conv95
32
128
0.85
0.05
0.74
0.05
0.74
0.05
0.96
46.19
Conv96
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.19
46.19
Conv97
128
32
0.84
0.05
0.74
0.05
0.74
0.05
8.75
155.4
Conv98
32
128
0.83
0.05
0.74
0.05
0.74
0.05
1.11
46.19
Conv99
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.22
46.19
Conv 100
128
32
0.87
0.05
0.74
0.05
0.74
0.05
8.92
155.4
Conv 101
32
128
0.85
0.05
0.74
0.05
0.74
0.05
1.12
46.19
Conv 102
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.2
46.19
Conv
128
32
0.86
0.05
0.74
0.05
0.74
0.05
8.21
155.4
Conv 104
32
128
0.84
0.05
0.74
0.05
0.74
0.05
1.03
46.19
Conv 105
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.31
46.19
Conv 106
128
32
0.86
0.05
0.74
0.05
0.74
0.05
7.93
155.4
Conv 107
32
128
0.70
0.05
0.74
0.05
0.74
0.05
1.65
46.19
Conv 08
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.32
46.19
Conv 109
128
32
0.86
0.05
0.74
0.05
0.74
0.05
8.7
155.4
Conv 10
64
128
0.65
0.05
0.82
0.05
0.82
0.05
1.96
83.68
Conv
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.18
83.68
Conv 1 12
256
64
0.70
0.05
0.90
0.05
0.90
0.05
15.81
294.32
Conv
64
256
0.60
0.05
0.90
0.05
0.90
0.05
0.85
83.68
Conv1 1 14
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.41
83.68
Conv1 15
256
64
0.82
0.05
0.90
0.05
0.90
0.05
6.26
294.32
Conv1 16
64
256
0.82
0.05
0.90
0.05
0.90
0.05
0.61
83.68
Conv 17
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.31
83.68
Conv 118
256
64
0.84
0.05
0.90
0.05
0.90
0.05
6.85
294.32
Conv 119
64
256
0.69
0.05
0.90
0.05
0.90
0.05
0.54
83.68
Conv120
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.35
83.68
72

---

Under review as a conference paper at ICLR 2021
Conv 121
256
64
0.82
0.05
0.90
0.05
0.90
0.05
6.53
294.32
Conv 122
64
256
0.85
0.05
0.90
0.05
0.90
0.05
0.43
83.68
Conv 123
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.39
83.68
Conv 1
256
64
0.66
0.05
0.90
0.05
0.90
0.05
6.93
294.32
Conv123
64
256
0.67
0.05
0.90
0.05
0.90
0.05
0.6
83.68
Conv 126
64
576
0.88
0,05
0.97
0.05
0.97
0.05
0.25
83.68
Conv
256
64
0.71
0.05
0.90
0.05
0.90
0.05
8.9
294.32
Conv 1 228
64
256
0.85
0.05
0.90
0.05
0.90
0.05
0.47
83.68
Conv 1 129
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.18
83.68
Conv 130
256
64
0.83
0.05
0.90
0.05
0.90
0.05
6.85
294.32
Conv 131
64
256
0.86
0.05
0.90
0.05
0.90
0.05
0.47
83.68
Conv 132
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.17
83.68
Conv 133
256
64
0.86
0.05
0.90
0.05
0.90
0.05
5.22
294.32
Conv 134
64
256
0.71
0.05
0.90
0.05
0.90
0.05
0.4
83.68
Conv 135
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.22
83.68
Conv Nonv136
256
64
0.70
0.05
0.90
0.05
0.90
0.05
7.07
294.32
Conv 137
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.39
83.68
Conv
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.22
83.68
Conv 139
256
64
0.85
0.05
0.90
0.05
0.90
0.05
5.24
294.32
Conv 140
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.55
83.68
Conv 141
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.21
83.68
Conv 142
256
64
0.83
0.05
0.90
0.05
0.90
0.05
7.08
294.32
Conv 143
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.36
83.68
Conv 144
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.26
83.68
Conv 145
256
64
0.82
0.05
0.90
0.05
0.90
0.05
8.28
294.32
Conv 146
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.42
83.68
Conv 147
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.27
83.68
Conv 148
256
64
0.83
0.05
0.90
0.05
0.90
0.05
8.01
294.32
Conv 1
64
256
0.69
0.05
0.90
0.05
0.90
0.05
0.68
83.68
Conv 1 50
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.17
83.68
Conv 151
256
64
0.82
0.05
0.90
0.05
0.90
0.05
7.8
294.32
Conv 152
64
256
0,80
0.05
0.90
0.05
0.90
0.05
0.39
83.68
Conv 153
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.11
83.68
Conv 1 154
256
64
0.81
0.05
0.90
0.05
0.90
0.05
8.1
294.32
Conv 155
64
256
0.80
0.05
0.90
0.05
0.90
0.05
0.55
83.68
Conv 156
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.11
83.68
Conv 157
256
64
0.82
0.05
0.90
0.05
0.90
0.05
8.92
294.32
Conv 158
64
256
0.69
0.05
0.90
0.05
0.90
0.05
0.61
83.68
Conv 159
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.1
83.68
Conv 160
256
64
0.70
0.05
0.90
0.05
0.90
0.05
10.08
294.32
Conv 161
64
256
0.82
0.05
0.90
0.05
0.90
0.05
0.44
83.68
Conv 162
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.09
83.68
Conv16 163
256
64
0.81
0.05
0.90
0.05
0.90
0.05
11.0
294.32
Passing rate
99.39%
100.0%
100.0%
99.39%
Table 28: SRMNet (ResNet164) Cifar100
Layer
Number
dim
Gaussian
Mean_ Left
Mean_Right
Sigma
p-value
c-value
p-value
c-value
p-value
e-value
t-value
e-value
Convl
16
27
0.00
0.05
0.58
0.05
0.58
0.05
1623.9
26.3
Conv2
16
16
0.35
0.05
0.56
0.05
0.56
0.05
33.36
26.3
Conv3
16
144
0.28
0.05
0.68
0.05
0.68
0.05
21.42
26.3
Conv4
64
16
0.31
0.05
0.63
0.05
0.63
0.05
156.52
83.68
Conv5
16
64
0.54
0.05
0.63
0.05
0.63
0.05
9.2
26.3
Conv6
16
144
0.43
0.05
0.68
0.05
0.68
0.05
17.06
26.3
73

---

Under review as a conference paper at ICLR 2021
Conv7
64
16
0.16
0.05
0.63
0.05
0.63
0.05
141.38
83.68
Conv8
16
64
0.71
0.05
0.63
0.05
0.63
0.05
1.14
26.3
Conv9
16
141
0.64
0.05
0.68
0.05
0.68
0.05
0.84
26.3
Conv10 10
64
16
0.68
0.05
0.63
0.05
0.63
0.05
26.28
83.68
Conv11
16
64
0.86
0.05
0.63
0.05
0.63
0.05
0.6
26.3
Conv 12
16
144
0.82
0,05
0.68
0,05
0.68
0.05
1.66
26.3
Conv
64
16
0.71
0.05
0.63
0.05
0.63
0.05
19.29
83.68
Conv 14
16
64
0.68
0.05
0.63
0.05
0.63
0.05
1.02
26.3
Conv15 15
16
0.82
0.05
0.68
0.05
0.68
0.05
1.34
26.3
Conv 16
64
16
0.85
0.05
0.63
0.05
0.63
0.05
6.93
83.68
Conv 17
16
64
0.86
0.05
0.63
0.05
0.63
0.05
1.13
26.3
Conv 18
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.33
26.3
Conv 19
64
16
0.87
0.05
0.63
0.05
0.63
0.05
3.44
83.68
Conv20
16
64
0.82
0.05
0.63
0.05
0.63
0.05
1.76
26.3
Conv21
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.82
26.3
Conv22
64
16
0.86
0.05
0.63
0.05
0.63
0.05
5.63
83.68
Conv23
16
64
0.87
0.05
0.63
0.05
0.63
0.05
1.43
26.3
Conv24
16
144
0.88
0.05
0.68
0.05
0.68
0.05
0.71
26.3
Conv25
64
16
0.86
0.05
0.63
0.05
0.63
0.05
4.67
83.68
Conv26
16
64
0.63
0.05
0.63
0.05
0.63
0.05
2.74
26.3
Conv27
16
144
0.64
0.05
0.68
0.05
0.68
0.05
2.32
26.3
Conv28
64
16
0.84
0.05
0.63
0.05
0.63
0.05
7.18
83.68
Conv29
16
64
0.65
0.05
0.63
0.05
0.63
0.05
3.72
26.3
Conv30
16
144
0.52
0.05
0.68
0.05
0.68
0.05
2.86
26.3
Conv31
64
16
0.84
0.05
0.63
0.05
0.63
0.05
10.89
83.68
Conv32
16
64
0.85
0.05
0.63
0.05
0.63
0.05
2.79
26.3
Conv33
16
144
0.86
0.05
0.68
0.05
0.68
0.05
0.69
26.3
Conv34
64
16
0.86
0.05
0.63
0.05
0.63
0.05
8.82
83.68
Conv35
16
64
0.83
0.05
0.63
0.05
0.63
0.05
3.59
26.3
Conv36
16
144
0.81
0.05
0.68
0.05
0.68
0.05
1.62
26.3
Conv37
64
16
0.86
0.05
0.63
0.05
0.63
0.05
6.81
83.68
Conv38
16
64
0.71
0.05
0.63
0.05
0.63
0.05
2.18
26.3
Conv39
16
0.61
0.05
0.68
0.05
0.68
0.05
2.3
26.3
Conv40
64
16
0.84
0.05
0.63
0.05
0.63
0.05
8.24
83.68
Conv41
16
64
0.81
0.05
0.63
0.05
0.63
0.05
2.79
26.3
Conv42
16
144
0.63
0.05
0.68
0.05
0.68
0.05
3.23
26.3
Conv43
64
16
0.86
0.05
0.63
0.05
0.63
0.05
8.83
83.68
Conv44
16
64
0.62
0.05
0.63
0.05
0.63
0.05
5.95
26.3
Comv45
16
144
0.65
0.05
0.68
0.05
0.68
0.05
4.0
26.3
Conv46
64
16
0.80
0.05
0.63
0.05
0.63
0.05
13.79
83.68
Conv47
16
64
0.81
0.05
0.63
0.05
0.63
0.05
2.02
26.3
Conv48
16
144
0.85
0.05
0.68
0.05
0.68
0.05
1.49
26.3
Conv49
64
16
0.90
0.05
0.63
0.05
0.63
0.05
5.35
83.68
Conv50
16
64
0.69
0.05
0.63
0.05
0.63
0.05
10.35
26.3
Conv51
16
144
0,60
0.05
0.68
0.05
0.68
0,05
2.38
26.3
Conv52
64
16
0.66
0.05
0.63
0.05
0.63
0.05
27.58
83.68
Conv53
16
64
0.87
0.05
0.63
0.05
0.63
0.05
0.87
26.3
Conv54
16
0.89
0.05
0.68
0.05
0.68
0.05
1.13
26.3
Conv55
64
16
0.87
0.05
0.63
0.05
0.63
0.05
6.33
83.68
Conv56
32
64
0.32
0.05
0.67
0.05
0.67
0.05
20.91
46.19
Conv57
32
288
0.64
0.05
0.83
0.05
0.83
0.05
5.09
46.19
Conv58
128
32
0.39
0.05
0.74
0.05
0.74
0.05
86.43
155.4
Conv59
32
128
0.51
0.05
0.74
0.05
0.74
0.05
4.89
46.19
Conv60
32
288
0.46
0.05
0.83
0.05
0.83
0.05
4.81
46.19
Conv61
128
32
0.68
0.05
0.74
0.05
0.74
0.05
37.39
155.4
Conv62
32
128
0.54
0.05
0.74
0.05
0.74
0.05
5.83
46.19
Conv63
32
288
0.62
0.05
0.83
0.05
0.83
0.05
4.73
46.19
Conv64
128
32
0.65
0.05
0.74
0.05
0.74
0.05
25.68
155.4
Conv65
32
128
0.83
0.05
0.74
0.05
0.74
0.05
3.33
46.19
74

---

Under review as a conference paper at ICLR 2021
Conv66
32
288
0.84
0.05
0.83
0.05
0.83
0.05
2.21
46.19
Conv67
128
32
0.83
0.05
0.74
0.05
0.74
0.05
19.46
153.4
Conv68
32
128
0.60
0.05
0.74
0.05
0.74
0.05
3.71
46.19
Conv69
32
288
0.61
0.05
0.83
0.05
0.83
0.05
4.04
46.19
Conv70
128
32
0.71
0.05
0.74
0.05
0.74
0.05
28.1
155.4
Conv71 71
32
128
0.66
0.05
0.74
0,05
0.74
0.05
2.16
46.19
Conv72
32
288
0.66
0.05
0.83
0.05
0.83
0.05
2.64
46.19
Conv73 73
128
32
0.84
0.05
0.74
0.05
0.74
0.05
17.43
155.4
Conv74
32
128
0.82
0.05
0.74
0.05
0.74
0.05
1.55
46.19
Conv75
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.91
46.19
Conv76
128
32
0.05
0.74
0.05
0.74
0.05
24.07
155.4
Conv77 77
32
128
0.66
0.05
0.74
0.05
0.74
0.05
5.42
46.19
Conv78
32
288
0.70
0.05
0.83
0.05
0.83
0.05
2.84
46.19
Conv79
128
32
0.65
0.05
0.74
0.05
0.74
0.05
22.67
155.4
Conv80
32
128
0.61
0.05
0.74
0.05
0.74
0.05
9.26
46.19
Conv81
32
288
0.61
0.05
0.83
0.05
0.83
0.05
5.61
46.19
Conv82
128
32
0.61
0.05
0.74
0.05
0.74
0.05
28.81
155.4
Conv83
32
128
0.83
0.05
0.74
0.05
0.74
0.05
2.49
46.19
Conv84
32
288
0.81
0.05
0.83
0.05
0.83
0.05
2.07
46.19
Conv85
128
32
0.70
0.05
0.74
0.05
0.74
0.05
18.55
155.4
Conv86
32
128
0.67
0.05
0.74
0.05
0.74
0.05
4.81
46.19
Conv87
32
288
0.62
0.05
0.83
0.05
0.83
0.05
3.47
46.19
Conv88
128
32
0.82
0.05
0.74
0.05
0.74
0.05
14.21
155.4
Conv89
32
128
0.69
0.05
0.74
0.05
0.74
0.05
4.38
46.19
Conv90
32
288
0.84
0.05
0.83
0.05
0.83
0.05
2.32
46.19
Conv91
128
32
0.84
0.05
0.74
0.05
0.74
0.05
18.52
155.4
Conv92
32
128
0.60
0.05
0.74
0.05
0.74
0.05
4.86
46.19
Conv93
32
288
0.84
0.05
0.83
0.05
0.83
0.05
2.01
46.19
Conv94
128
32
0.85
0.05
0.74
0.05
0.74
0.05
21.5
155.4
Conv95
32
128
0.44
0.05
0.74
0.05
0.74
0.05
7.02
46.19
Conv96
32
288
0.61
0.05
0.83
0.05
0.83
0.05
3.26
46.19
Conv97
128
32
0.81
0.05
0.74
0.05
0.74
0.05
23.4
155.4
Conv98
32
128
0.84
0.05
0.74
0.05
0.74
0.05
2.63
46.19
Conv99
32
288
0.69
0.05
0.83
0.05
0.83
0.05
1.53
46.19
Conv 100
128
32
0.68
0.05
0.74
0.05
0.74
0.05
22.25
155.4
Conv101
32
128
0.69
0.05
0.74
0.05
0.74
0.05
3.22
46.19
Conv102
32
288
0.85
0.05
0.83
0.05
0.83
0.05
46.19
Conv 103
128
32
0.82
0.05
0.74
0.05
0.74
0.05
15.27
155.4
Conv 104
32
128
0.83
0.05
0.74
0.05
0.74
0.05
3.01
46.19
Conv 105
32
288
0.85
0.05
0.83
0.05
0.83
0.05
1.79
46.19
Conv 106
128
32
0.70
0.05
0.74
0.05
0.74
0.05
21.86
155.4
Conv
32
128
0.58
0.05
0.74
0.05
0.74
0.05
7.47
46.19
Conv 108
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.14
46.19
Conv 109
128
32
0.65
0.05
0.74
0.05
0.74
0.05
24.02
155.4
Conv 10
64
128
0.42
0.05
0.82
0.05
0.82
0.05
10.69
83,68
Conv 111
64
576
0.66
0.05
0.97
0.05
0.97
0.05
2.19
83.68
Conv 1 12
256
64
0.53
0.05
0.90
0.05
0.90
0.05
80.96
294.32
Conv
64
256
0.52
0.05
0.90
0.05
0.90
0.05
83.68
Conv 14
64
576
0.83
0.05
0.97
0.05
0.97
0.05
1.43
83.68
Conv 15
256
64
0.83
0.05
0.90
0.05
0.90
0.05
24.22
294.32
Conv 1 16
64
256
0.82
0.05
0.90
0.05
0.90
0.05
1.18
83.68
Conv
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.89
83.68
Conv 1 18
256
64
0.87
0.05
0.90
0.05
0.90
0.05
12.44
294.32
Conv1 19
64
256
0.61
0.05
0.90
0.05
0.90
0.05
1.29
83.68
Conv 20
64
576
0.62
0.05
0.97
0.05
0.97
0.05
1.29
83.68
Conv 121
256
64
0.68
0.05
0.90
0.05
0.90
0.05
16,31
294.32
Conv 122
64
256
0.71
0.05
0.90
0.05
0.90
0.05
1.4
83.68
Conv 1
64
576
0.87
0.05
0.97
0.05
0.97
0.05
1.46
83.68
Conv
256
64
0.86
0.05
0.90
0.05
0.90
0.05
13.1
294.32
75

---

Under review as a conference paper at ICLR 2021
Conv 12n
64
256
0.83
0.05
0.90
0.05
0.90
0.05
1.03
83.68
Conv 126
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.67
83.68
Conv 127
256
64
0.84
0.05
0.90
0.05
0.90
0.05
13.5
294.32
Conv 128
64
256
0.71
0.05
0.90
0.05
0.90
0.05
1.42
83.68
Conv122
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.45
83.68
Conv 130
256
64
0.82
0,05
0.90
0.05
0.90
0.05
17.02
294.32
Conv133
64
256
0.63
0.05
0.90
0.05
0.90
0.05
1.53
83.68
Conv 132
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.62
83.68
Conv 133
256
64
0.82
0.05
0.90
0.05
0.90
0.05
19.25
294.32
Conv 134
64
256
0.71
0.05
0.90
0.05
0.90
0.05
1.2
83.68
Conv 135
576
0.88
0.05
0.97
0.05
0.97
0.05
0.83
83.68
Conv 1 36
256
64
0.85
0.05
0.90
0.05
0.90
0.05
18.2
294.32
Conv 137
64
256
0.82
0.05
0.90
0.05
0.90
0.05
0.91
83.68
Conv 138
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.59
83.68
Conv 1 139
256
64
0.84
0.05
0.90
0.05
0.90
0.05
16.94
294.32
Conv 140
64
256
0.65
0.05
0.90
0.05
0.90
0.05
1.14
83.68
Conv 141
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.87
83.68
Conv 142
256
64
0.64
0.05
0.90
0.05
0.90
0.05
17.78
294.32
Conv 143
64
256
0.67
0.05
0.90
0.05
0.90
0.05
1.41
83.68
Conv 144
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.78
83.68
Conv 145
256
64
0.62
0.05
0.90
0.05
0.90
0.05
17.51
294.32
Conv 46
64
256
0.52
0.05
0.90
0.05
0.90
0.05
1.45
83.68
Conv 147
64
576
0.86
0.05
0.97
0.05
0.97
0.05
1.19
83.68
Conv 148
256
64
0.67
0.05
0.90
0.05
0.90
0.05
24.19
294.32
Conv 149
64
256
0.60
0.05
0.90
0.05
0.90
0.05
0.89
83.68
Conv 150
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.97
83.68
Conv 151
256
64
0.70
0.05
0.90
0.05
0.90
0.05
23.84
294.32
Conv 152
64
256
0.49
0.05
0.90
0.05
0.90
0.05
1.09
83.68
Conv
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.88
83.68
Conv 1 54
256
64
0.65
0.05
0.90
0.05
0.90
0.05
25.16
294.32
Conv 155
64
256
0.65
0.05
0.90
0.05
0.90
0.05
1.08
83.68
Conv 156
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.74
83.68
Conv157
256
64
0.62
0.05
0.90
0.05
0.90
0.05
35.19
294.32
Conv 158
64
256
0.53
0.05
0.90
0.05
0.90
0.05
1.14
83.68
Conv 159
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.64
83.68
Conv 1lw666 160
256
64
0.52
0.05
0.90
0.05
0.90
0.05
57.16
294.32
Conv 161
64
256
0.39
0.05
0.90
0.05
0.90
0.05
1.63
83.68
Conv 162
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.53
83.68
Conv 163
256
64
0.45
0.05
0.90
0.05
0.90
0.05
136.33
294.32
Passing rate
99.39%
100.0%
100.0%
97.55%
Table 29: CBAM (ResNet164) Cifar100
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.02
0.05
0.58
0.05
0.58
0.05
197.16
26.3
Conv2
16
16
0.61
0.05
0.56
0.05
0.56
0.05
13.19
26.3
Conv3
16
144
0.82
0.05
0.68
0.05
0.68
0.05
2.7
26.3
Conv4
64
16
0.71
0.05
0.63
0.05
0.63
0.05
11.87
83.68
Conv5
16
64
0.89
0.05
0.63
0.05
0.63
0.05
26.3
Conv6
16
144
0.88
0.05
0.68
0.05
0.68
0.05
0.75
26.3
Conv7
64
16
0.88
0.05
0.63
0.05
0.63
0.05
3.67
83.68
Conv8
16
64
0.85
0.05
0.63
0.05
0.63
0.05
1.4
26.3
Conv9
16
144
0.83
0.05
0.68
0.05
0.68
0.05
1.35
26.3
Conv10
64
16
0.68
0.05
0.63
0.05
0.63
0.05
12.85
83.68
76

---

Under review as a conference paper at ICLR 2021
Conv 11
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.46
26.3
Conv 12
16
144
0.82
0.05
0.68
0.05
0.68
0.05
1.31
26.3
Conv 13
64
16
0.71
0.05
0.63
0.05
0.63
0.05
8.4
83.68
Conv 14
16
64
0.89
0.05
0.63
0.05
0.63
0.05
0.71
26.3
Conv
16
144
0.91
0.05
0.68
0.05
0.68
0.05
0.28
26.3
Conv 16
64
16
0.87
0.05
0.63
0,05
0.63
0.05
2.2
83.68
Conv17
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.68
26.3
Conv 18
16
144
0.86
0.05
0.68
0.05
0.68
0.05
0.97
26.3
Conv19
64
16
0.89
0.05
0.63
0.05
0.63
0.05
5.12
83.68
Conv20
16
64
0.65
0.05
0.63
0.05
0.63
0.05
2.8
26.3
Conv21
16
144
0.54
0.05
0.68
0.05
0.68
0.05
1.76
26.3
Conv22
64
16
0.84
0.05
0.63
0.05
0.63
0.05
11.18
83.68
Comv23
16
64
0.82
0.05
0.63
0.05
0.63
0.05
2.65
26.3
Conv24
16
144
0.65
0.05
0.68
0.05
0.68
0.05
1.47
26.3
Conv25
64
16
0.86
0.05
0.63
0.05
0.63
0.05
5.79
83.68
Conv26
16
64
0.70
0.05
0.63
0.05
0.63
0.05
3.49
26.3
Conv27
16
144
0.86
0.05
0.68
0.05
0.68
0.05
0.96
26.3
Conv28
64
16
0.85
0.05
0.63
0.05
0.63
0.05
8.09
83.68
Conv29
16
64
0.91
0.05
0.63
0.05
0.63
0.05
1.29
26.3
Conv30
16
144
0.91
0.05
0.68
0.05
0.68
0.05
1.01
26.3
Conv31
64
16
0.92
0.05
0.63
0.05
0.63
0.05
1.54
83.68
Conv32
16
64
0.90
0.05
0.63
0.05
0.63
0.05
0.92
26.3
Conv33
16
144
0.91
0.05
0.68
0.05
0.68
0.05
0.4
26.3
Conv34
64
16
0.91
0.05
0.63
0.05
0.63
0.05
2.64
83.68
Conv35
16
64
0.88
0.05
0.63
0.05
0.63
0.05
1.48
26.3
Conv36
16
144
0.92
0.05
0.68
0.05
0.68
0.05
0.17
26.3
Conv37
64
16
0.91
0.05
0.63
0.05
0.63
0.05
4.65
83.68
Conv38
16
64
0.66
0.05
0.63
0.05
0.63
0.05
4.56
26.3
Conv39
16
144
0.67
0.05
0.68
0.05
0.68
0.05
5.55
26.3
Conv40
64
16
0.70
0.05
0.63
0.05
0.63
0.05
21.21
83.68
Conv41
16
64
0.87
0.05
0.63
0.05
0.63
0.05
0.91
26.3
Conv42
16
144
0.93
0.05
0.68
0.05
0.68
0.05
0.13
26.3
Conv43
64
16
0.91
0.05
0.63
0.05
0.63
0.05
2.14
83.68
Conv44
16
64
0.92
0.05
0.63
0.05
0.63
0.05
0.71
26.3
Conv45
16
144
0.93
0.05
0.68
0.05
0.68
0.05
0.33
26.3
Conv46
64
16
0.90
0.05
0.63
0.05
0.63
0.05
2.65
83.68
Conv47
16
64
0.69
0.05
0.63
0.05
0.63
0.05
3.17
26.3
Conv48
16
144
0.86
0.05
0.68
0.05
0.68
0.05
0.91
26.3
Conv49
64
16
0.87
0.05
0.63
0.05
0.63
0.05
6.35
83.68
Conv50
16
64
0.89
0.05
0.63
0.05
0.63
0.05
1.24
26.3
Conv51
16
144
0.89
0.05
0.68
0.05
0.68
0.05
1.15
26.3
Conv52
64
16
0.89
0.05
0.63
0.05
0.63
0.05
7.12
83.68
Conv53
16
64
0.53
0.05
0.63
0.05
0.63
0.05
5.77
26.3
Conv54
16
144
0.56
0.05
0.68
0.05
0.68
0.05
3.64
26.3
Conv55
64
16
0.84
0.05
0.63
0.05
0.63
0.05
17.53
83.68
Conv56
32
64
0.43
0.05
0.67
0.05
0.67
0.05
18.41
46.19
Conv57
32
288
0.54
0.05
0.83
0.05
0.83
0.05
10.34
46.19
Conv58
128
32
0.45
0.05
0.74
0.05
0.74
0.05
82.06
155.4
Conv59
32
128
0.55
0.05
0.74
0.05
0.74
0.05
6.51
46.19
Conv60
32
288
0.64
0.05
0.83
0.05
0.83
0.05
6.29
46.19
Conv61
128
32
0.67
0.05
0.74
0.05
0.74
0.05
50.8
155.4
Conv62
32
128
0.55
0.05
0.74
0.05
0.74
0.05
4.23
46.19
Conv63
32
288
0.48
0.05
0.83
0.05
0.83
0.05
4.17
46.19
Conv64
128
32
0.68
0.05
0.74
0.05
0.74
0.05
21.02
155.4
Conv65
32
128
0.70
0.05
0.74
0.05
0.74
0.05
1.86
46.19
Conv66
32
288
0.84
0.05
0.83
0.05
0.83
0.05
1.58
46.19
Conv67
128
32
0.67
0.05
0.74
0.05
0.74
0.05
25.18
155.4
Conv68
32
128
0.61
0.05
0.74
0.05
0.74
0.05
6.87
46.19
Conv69
32
288
0.70
0.05
0.83
0.05
0.83
0.05
2.08
46.19
77

---

Under review as a conference paper at ICLR 2021
Conv70
128
32
0.63
0.05
0.74
0.05
0.74
0.05
36.62
155.4
Conv
32
128
0.62
0.05
0.74
0.05
0.74
0.05
2.77
46.19
Conv72
32
288
0.71
0.05
0.83
0.05
0.83
0.05
1.06
46.19
Conv73 73
128
32
0.82
0.05
0.74
0.05
0.74
0.05
12.76
155.4
Conv74
32
128
0.61
0.05
0.74
0.05
0.74
0.05
4.29
46.19
Conv75 75
32
288
0.70
0,05
0.83
0,05
0.83
0.05
2.75
46.19
Conv76
128
32
0.58
0.05
0.74
0.05
0.74
0.05
36.58
155.4
Conv77 77
32
128
0.81
0.05
0.74
0.05
0.74
0.05
1.8
46.19
Conv78
32
288
0.84
0.05
0.83
0.05
0.83
0.05
1.44
46.19
Conv79
128
32
0.83
0.05
0.74
0.05
0.74
0.05
22.78
155.4
Conv80
32
128
0.88
0.05
0.74
0.05
0.74
0.05
0.69
46.19
Conv81
32
288
0.86
0.05
0.83
0.05
0.83
0.05
0.39
46.19
Conv82
128
32
0.87
0.05
0.74
0.05
0.74
0.05
12.09
153.4
Conv83
32
128
0.85
0.05
0.74
0.05
0.74
0.05
2.09
46.19
Conv84
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.32
46.19
Conv85
128
32
0.84
0.05
0.74
0.05
0.74
0.05
11.39
155.4
Conv86
32
128
0.57
0.05
0.74
0.05
0.74
0.05
1.48
46.19
Conv87
32
288
0.86
0.05
0.83
0.05
0.83
0.05
0.5
46.19
Conv88
128
32
0.83
0.05
0.74
0.05
0.74
0.05
12.88
155.4
Conv89
32
128
0.57
0.05
0.74
0.05
0.74
0.05
2.43
46.19
Conv90
32
288
0.81
0.05
0.83
0.05
0.83
0.05
0.7
46.19
Conv91
128
32
0.81
0.05
0.74
0.05
0.74
0.05
12.91
155.4
Conv92
32
128
0.82
0.05
0.74
0.05
0.74
0.05
1.18
46.19
Conv93
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.58
46.19
Conv94
128
32
0.87
0.05
0.74
0.05
0.74
0.05
13.73
155.4
Conv95
32
128
0.86
0.05
0.74
0.05
0.74
0.05
0.97
46.19
Conv96
32
288
0.87
0.05
0.83
0.05
0.83
0.05
0.85
46.19
Conv97
128
32
0.70
0.05
0.74
0.05
0.74
0.05
14.06
155.4
Conv98
32
128
0.85
0.05
0.74
0.05
0.74
0.05
1.07
46.19
Conv99
32
288
0.82
0.05
0.83
0.05
0.83
0.05
0.67
46.19
Conv 100
128
32
0.86
0.05
0.74
0.05
0.74
0.05
10.9
155.4
Conv 101
32
128
0.65
0.05
0.74
0.05
0.74
0.05
3,05
46.19
Conv102
32
288
0.87
0.05
0.83
0.05
0.83
0.05
0.49
46.19
Conv 103
128
32
0.82
0.05
0.74
0.05
0.74
0.05
14.8
155.4
Conv 104
32
128
0.87
0.05
0.74
0.05
0.74
0.05
0.42
46.19
Conv 105
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.88
46.19
Conv106
128
32
0.85
0.05
0.74
0.05
0.74
0.05
10.12
155.4
Conv 107
32
128
0.81
0.05
0.74
0.05
0.74
0.05
1.94
46.19
Conv 108
32
288
0.86
0.05
0.83
0.05
0.83
0.05
0.25
46.19
Conv 109
128
32
0.85
0.05
0.74
0.05
0.74
0.05
17.64
155.4
Conv1 10
64
128
0.57
0.05
0.82
0.05
0.82
0.05
3.04
83.68
Conv 11
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.62
83.68
Conv 12
256
64
0.46
0.05
0.90
0.05
0.90
0.05
46.96
294.32
Conv 13
64
256
0.70
0.05
0.90
0.05
0.90
0.05
0.81
83.68
Conv 14
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.81
83.68
Conv 15
256
64
0.60
0.05
0.90
0.05
0.90
0.05
13.34
294.32
Conv 1 16
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.78
83.68
Conv1
64
576
0.81
0.05
0.97
0.05
0.97
0.05
0.53
83.68
Conv 18
256
64
0.64
0.05
0.90
0.05
0.90
0.05
12.21
294.32
Conv
64
256
0.65
0.05
0.90
0.05
0.90
0.05
1.I1
83.68
Conv 120
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.76
83.68
Conv 121
256
64
0.81
0.05
0.90
0.05
0.90
0.05
10.67
294.32
Conv 122
64
256
0.69
0.05
0.90
0.05
0.90
0.05
0.65
83.68
Conv1 123
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.43
83.68
Conv
256
64
0.81
0.05
0.90
0.05
0.90
0.05
10.07
294.32
Conv 125
64
256
0.81
0.05
0.90
0.05
0.90
0.05
0.76
83.68
Conv 126
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.4
83.68
Conv 127
256
64
0.84
0.05
0.90
0.05
0.90
0.05
10.13
294.32
Conv
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.64
83.68
78

---

Under review as a conference paper at ICLR 2021
Conv 1onv129
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.32
83.68
Conv 130
256
64
0.81
0.05
0.90
0.05
0.90
0.05
11.56
294.32
Conv 131
64
256
0.81
0.05
0.90
0.05
0.90
0.05
0.75
83.68
Conv 1 132
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.32
83.68
Conv133
256
64
0.58
0.05
0.90
0.05
0.90
0.05
13.42
294.32
Conv 134
64
256
0.69
0,05
0.90
0.05
0,90
0.05
0.67
83.68
Conv
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.27
83.68
Conv 136
256
64
0.68
0.05
0.90
0.05
0.90
0.05
11.61
294.32
Conv 137
64
256
0.65
0.05
0.90
0.05
0.90
0.05
0.65
83.68
Conv 138
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.25
83.68
Conv 139
256
64
0.65
0.05
0.90
0.05
0.90
0.05
12.04
294.32
Conv 1 140
64
256
0.82
0.05
0.90
0.05
0.90
0.05
0.43
83.68
Conv 141
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.39
83.68
Conv 142
256
64
0.69
0.05
0.90
0.05
0.90
0.05
13.46
294.32
Conv 143
64
256
0.81
0.05
0.90
0.05
0.90
0.05
0.51
83.68
Conv 144
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.37
83.68
Conv 145
256
64
0.62
0.05
0.90
0.05
0.90
0.05
14.42
294.32
Conv 146
64
256
0.67
0.05
0.90
0.05
0.90
0.05
0.54
83.68
Conv 147
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.34
83.68
Conv 148
256
64
0.67
0.05
0.90
0.05
0.90
0.05
15.7
294.32
Conv 149
64
256
0.60
0.05
0.90
0.05
0.90
0.05
0.52
83.68
Conv 50
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.34
83.68
Conv 151
256
64
0.69
0.05
0.90
0.05
0.90
0.05
18.04
294.32
Conv 152
64
256
0.54
0.05
0.90
0.05
0.90
0.05
0.74
83.68
Conv 153
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.27
83.68
Conv 154
256
64
0.63
0.05
0.90
0.05
0.90
0.05
22.21
294.32
Conv 155
64
256
0.63
0.05
0.90
0.05
0.90
0.05
0.82
83.68
Conv 156
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.29
83.68
Conv
256
64
0.63
0.05
0.90
0.05
0.90
0.05
23.29
294.32
Conv 1 58
64
256
0.65
0.05
0.90
0.05
0.90
0.05
0.76
83.68
Conv 159
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.27
83.68
Conv 160
256
64
0.61
0.05
0.90
0.05
0.90
0.05
25.39
294.32
Conv161
64
256
0.63
0.05
0.90
0.05
0.90
0.05
0.84
83.68
Conv 162
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.23
83.68
Conv 163
256
64
0.57
0.05
0.90
0.05
0.90
0.05
37.17
294.32
Passing rate
99.39%
100.0%
100.0%
99.39%
Table 30: IEBN (ResNet1 Cifar100
Layer
Number dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
e-value
t-value
e-value
Conv1
16
27
0.01
0.05
0.58
0.05
0.58
0.05
195.79
26.3
Conv2
16
16
0.54
0.05
0.56
0.05
0.56
0.05
4.71
26.3
Conv3
16
144
0.68
0.05
0.68
0.05
0.68
0.05
1.02
26.3
Conv4
64
16
0.81
0.05
0.63
0.05
0.63
0.05
15.02
83.68
C
16
64
0.65
0.05
0.63
0.05
0.63
0.05
0.91
26.3
Conv6
16
144
0.85
0.05
0.68
0.05
0.68
0.05
0.77
26.3
Conv7
64
16
0.71
0.05
0.63
0.05
0.63
0.05
10.97
83.68
Conv8
16
64
0.51
0.05
0.63
0.05
0.63
0.05
2.16
26.3
Conv9
16
144
0.67
0.05
0.68
0.05
0.68
0.05
1.29
26.3
Conv10 10
64
16
0.82
0.05
0.63
0.05
0.63
0.05
13.51
83.68
Conv 11
16
64
0.66
0.05
0.63
0.05
0.63
0.05
1.9
26.3
Conv12
16
144
0.80
0.05
0.68
0.05
0.68
0.05
0.85
26.3
Conv13
64
16
0.80
0.05
0.63
0.05
0.63
0.05
7.16
83.68
Conv14
16
64
0.69
0.05
0.63
0.05
0.63
0.05
2.7
26.3
79

---

Under review as a conference paper at ICLR 2021
Conv 15
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.93
26.3
Conv 16
64
16
0.83
0.05
0.63
0.05
0.63
0.05
14.86
83.68
Conv 17
16
64
0.69
0.05
0.63
0.05
0.63
0.05
2.31
26.3
Conv 18
16
144
0.66
0.05
0.68
0.05
0.68
0.05
1.09
26.3
Conv19
64
16
0.81
0.05
0.63
0.05
0.63
0.05
11.16
83.68
Conv20
16
64
0.82
0,05
0.63
0,05
0.63
0.05
1.1
26.3
Conv21
16
144
0.88
0.05
0.68
0.05
0.68
0.05
0.51
26.3
Conv22
64
16
0.85
0.05
0.63
0.05
0.63
0.05
8.11
83.68
Conv23
16
64
0.86
0.05
0.63
0.05
0.63
0.05
0.49
26.3
Conv24
16
144
0.87
0.05
0.68
0.05
0.68
0.05
1.0
26.3
Conv25
64
16
0.86
0.05
0.63
0.05
0.63
0.05
10.94
83.68
Conv26
16
64
0.71
0.05
0.63
0.05
0.63
0.05
1.04
26.3
Conv27
16
144
0.83
0.05
0.68
0.05
0.68
0.05
1.09
26.3
Conv28
64
16
0.68
0.05
0.63
0.05
0.63
0.05
15.37
83.68
Conv29
16
64
0.83
0.05
0.63
0.05
0.63
0.05
12
26.3
Conv30
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.58
26.3
Conv31
64
16
0.85
0.05
0.63
0.05
0.63
0.05
9.0
83.68
Conv32
16
64
0.70
0.05
0.63
0.05
0.63
0.05
1.73
26.3
Conv33
16
144
0.86
0.05
0.68
0.05
0.68
0.05
1.14
26.3
Conv34
64
16
0.82
0.05
0.63
0.05
0.63
0.05
9.77
83.68
Conv35
16
64
0.66
0.05
0.63
0.05
0.63
0.05
1.8
26.3
Conv36
16
0.84
0.05
0.68
0.05
0.68
0.05
0.69
26.3
Conv37
64
16
0.70
0.05
0.63
0.05
0.63
0.05
8.9
83.68
Conv38
16
64
0.59
0.05
0.63
0.05
0.63
0.05
2.08
26.3
Conv39
16
144
0.65
0.05
0.68
0.05
0.68
0.05
1.38
26.3
Conv40
64
16
0.81
0.05
0.63
0.05
0.63
0.05
10.24
83.68
Conv41
16
64
0.63
0.05
0.63
0.05
0.63
0.05
2.02
26.3
Conv42
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.9
26.3
Conv43
64
16
0.82
0.05
0.63
0.05
0.63
0.05
17.71
83.68
Conv44
16
64
0.81
0.05
0.63
0.05
0.63
0.05
2.86
26.3
Conv45
16
144
0.86
0.05
0.68
0.05
0.68
0.05
1.45
26.3
Conv46
64
16
0.64
0.05
0.63
0.05
0.63
0.05
19.59
83.68
Conv47
16
64
0.83
0.05
0.63
0.05
0.63
0.05
2.27
26.3
Conv48
16
144
0.85
0.05
0.68
0.05
0.68
0.05
0.56
26.3
Conv49
64
16
0.84
0.05
0.63
0.05
0.63
0.05
8.48
83.68
Conv50
16
64
0.63
0.05
0.63
0.05
0.63
0.05
3.11
26.3
Conv51
16
0.81
0.05
0.68
0.05
0.68
0.05
1.48
26.3
Conv52
64
16
0.71
0.05
0.63
0.05
0.63
0.05
17.95
83.68
Conv53
16
64
0.64
0.05
0.63
0.05
0.63
0.05
3.24
26.3
Conv54
16
144
0.87
0.05
0.68
0.05
0.68
0.05
1.26
26.3
Conv55
64
16
0.67
0.05
0.63
0.05
0.63
0.05
12.57
83.68
Conv56
32
64
0.49
0.05
0.67
0.05
0.67
0.05
5.98
46.19
Conv57
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.46
46.19
Conv58
128
32
0.68
0.05
0.74
0.05
0.74
0.05
22.72
155.4
Conv59
32
128
0.88
0.05
0.74
0.05
0.74
0.05
0.28
46.19
Conv60
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.44
46.19
Conv61
128
32
0.87
0.05
0.74
0.05
0.74
0.05
5.9
155.4
Conv62
32
128
0.86
0.05
0.74
0.05
0.74
0.05
0.89
46.19
Conv63
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.72
46.19
Conv64
128
32
0.87
0.05
0.74
0.05
0.74
0.05
5.71
155.4
Conv65
32
128
0.90
0.05
0.74
0.05
0.74
0.05
0.35
46.19
Conv66
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.43
46.19
Conv67
128
32
0.85
0.05
0.74
0.05
0.74
0.05
4.75
155.4
Conv68
32
128
0.85
0.05
0.74
0.05
0.74
0.05
0.67
46.19
Conv69
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.51
46.19
Conv 70
128
32
0.88
0.05
0.74
0.05
0.74
0.05
5.8
155.4
Conv71
32
128
0.84
0.05
0.74
0.05
0.74
0.05
0.97
46.19
Conv72
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.69
46.19
Conv73
128
32
0.85
0.05
0.74
0.05
0.74
0.05
5.58
155.4
80

---

Under review as a conference paper at ICLR 2021
Conv74
32
128
0.88
0.05
0.74
0.05
0.74
0.05
0.67
46.19
Conv 75
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.55
46.19
Conv76
128
32
0.89
0.05
0.74
0.05
0.74
0.05
4.63
155.4
Conv77
32
128
0.84
0.05
0.74
0.05
0.74
0.05
0.9
46.19
Conv78
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.24
46.19
Conv79
128
32
0.87
0,05
0.74
0,05
0.74
0.05
5.47
153.4
Conv80
32
128
0.87
0.05
0.74
0.05
0.74
0.05
0.71
46.19
Conv81
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.73
46.19
Conv82
128
32
0.88
0.05
0.74
0.05
0.74
0.05
5.61
155.4
Conv83
32
128
0.84
0.05
0.74
0.05
0.74
0.05
0.43
46.19
Conv84
32
288
0.91
0.05
0.83
0.05
0.83
0.05
0.35
46.19
Conv85
128
32
0.87
0.05
0.74
0.05
0.74
0.05
5.8
155.4
Conv86
32
128
0.84
0.05
0.74
0.05
0.74
0.05
1.03
46.19
Conv87
32
288
0.88
0.05
0.83
0.05
0.83
0.05
0.47
46.19
Conv88
128
32
0.88
0.05
0.74
0.05
0.74
0.05
5.64
155.4
Conv89
32
128
0.82
0.05
0.74
0.05
0.74
0.05
1.47
46.19
Conv90
32
288
0.89
0.05
0.83
0.05
0.83
0.05
0.33
46.19
Conv91
128
32
0.84
0.05
0.74
0.05
0.74
0.05
5.74
155.4
Conv92
32
128
0.87
0.05
0.74
0.05
0.74
0.05
0.8
46.19
Conv93
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.36
46.19
Conv94
128
32
0.88
0.05
0.74
0.05
0.74
0.05
5.16
155.4
Conv95
32
128
0.68
0.05
0.74
0.05
0.74
0.05
1.01
46.19
Conv96
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.38
46.19
Conv97
128
32
0.82
0.05
0.74
0.05
0.74
0.05
6.46
155.4
Conv98
32
128
0.85
0.05
0.74
0.05
0.74
0.05
0.95
46.19
Conv99
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.36
46.19
Conv 100
128
32
0.84
0.05
0.74
0.05
0.74
0.05
5.54
155.4
Conv 101
32
128
0.85
0.05
0.74
0.05
0.74
0.05
1.57
46.19
Conv102
32
288
0.87
0.05
0.83
0.05
0.83
0.05
0.2
46.19
Conv 103
128
32
0.87
0.05
0.74
0.05
0.74
0.05
4.57
155.4
Conv 104
32
128
0.86
0.05
0.74
0.05
0.74
0.05
0.65
46.19
Conv 105
32
288
0.90
0.05
0.83
0.05
0.83
0.05
0.25
46.19
Conv
128
32
0.86
0.05
0.74
0.05
0.74
0.05
5.92
155.4
Conv 107
32
128
0.67
0.05
0.74
0.05
0.74
0.05
1.33
46.19
Conv 108
32
288
0.84
0.05
0.83
0.05
0.83
0.05
0.3
46.19
Conv 1 09
128
32
0.87
0.05
0.74
0.05
0.74
0.05
4.94
155.4
Conv1
64
128
0.67
0.05
0.82
0.05
0.82
0.05
2.21
83.68
Conv 11
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.29
83.68
Conv
256
64
0.64
0.05
0.90
0.05
0.90
0.05
12.03
294.32
Conv 1 13
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.79
83.68
Conv
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.37
83.68
Conv 15
256
64
0.83
0.05
0.90
0.05
0.90
0.05
6.38
294.32
Conv 16
64
256
0.86
0.05
0.90
0.05
0.90
0.05
0.44
83.68
Conv
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.42
83.68
Conv 18
256
64
0.81
0.05
0.90
0.05
0.90
0.05
6.49
294.32
Conv 1 19
64
256
0.82
0.05
0.90
0.05
0.90
0.05
0.65
83.68
Conv 1 20
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.51
83.68
Conv 121
256
64
0.87
0.05
0.90
0.05
0.90
0.05
7.01
294.32
Conv 122
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.63
83.68
Conv 123
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.34
83.68
Conv 124
256
64
0.85
0.05
0.90
0.05
0.90
0.05
7.76
294.32
Conv 125
64
256
0.87
0.05
0.90
0.05
0.90
0.05
0.62
83.68
Conv 26
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.28
83.68
Conv1 127
256
64
0.84
0.05
0.90
0.05
0.90
0.05
6.24
294.32
Conv129
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.67
83.68
Conv 129
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.26
83.68
Conv 130
256
64
0.84
0.05
0.90
0.05
0.90
0.05
6.71
294.32
Conv 131
64
256
0.87
0.05
0.90
0.05
0.90
0.05
0.66
83.68
Conv
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.23
83.68
81

---

Under review as a conference paper at ICLR 2021
Conv 133
256
64
0.85
0.05
0.90
0.05
0.90
0.05
7.47
294.32
Conv 134
64
256
0.85
0.05
0.90
0.05
0.90
0.05
0.47
83.68
Conv 135
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.26
83.68
Conv 136
256
64
0.87
0.05
0.90
0.05
0.90
0.05
6.6
294.32
Conv
64
256
0.87
0.05
0.90
0.05
0.90
0.05
0.54
83.68
Conv 138
64
576
0.88
0,05
0.97
0.05
0.97
0.05
0.22
83.68
Conv133
256
64
0.86
0.05
0.90
0.05
0.90
0.05
7.99
294.32
Conv 140
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.63
83.68
Conv 141
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.18
83.68
Conv 142
256
64
0.84
0.05
0.90
0.05
0.90
0.05
8.04
294.32
Conv 43
64
256
0.83
0.05
0.90
0.05
0.90
0.05
83.68
Conv 144
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.21
83.68
Conv 145
256
64
0.85
0.05
0.90
0.05
0.90
0.05
7.78
294.32
Conv 146
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.43
83.68
Conv 147
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.23
83.68
Conv 148
256
64
0.85
0.05
0.90
0.05
0.90
0.05
7.64
294.32
Conv 149
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.49
83.68
Conv 150
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.24
83.68
Conv 151
256
64
0.82
0.05
0.90
0.05
0.90
0.05
8.91
294.32
Conv 152
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.75
83.68
Conv 153
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.2
83.68
Conv 1 54
256
64
0.83
0.05
0.90
0.05
0.90
0.05
8.11
294.32
Conv 155
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.5
83.68
Conv 156
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.18
83.68
Conv
256
64
0.81
0.05
0.90
0.05
0.90
0.05
8.0
294.32
Conv 158
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.63
83.68
Conv 159
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.19
83.68
Conv 160
256
64
0.82
0.05
0.90
0.05
0.90
0.05
7.69
294.32
Conv
64
256
0.81
0.05
0.90
0.05
0.90
0.05
0.87
83.68
Conv 1lw166 162
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.17
83.68
Conv 163
256
64
0.83
0.05
0.90
0.05
0.90
0.05
9.42
294.32
Passing rate
99.39%
100.0%
100.0%
99.39%
Table 31: SGENet (ResNet164) Cifar100
Layer
Number
dim
Gaussian
Mean_Lett
Mean.
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.00
0.05
0.58
0,05
0.58
0.05
289.86
26.3
Conv2
16
16
0.35
0.05
0.56
0.05
0.56
0.05
46.81
26.3
Conv3
16
144
0.29
0.05
0.68
0.05
0.68
0.05
25,84
26.3
Conv4
64
16
0.47
0.05
0.63
0.05
0.63
0.05
74.29
83.68
5
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.15
26.3
Conv6
16
144
0.67
0.05
0.68
0.05
0.68
0.05
1.91
26.3
Conv7
64
16
0.48
0.05
0.63
0.05
0.63
0.05
23.94
83.68
Conv8
16
64
0.81
0.05
0.63
0.05
0.63
0.05
1.49
26.3
Conv9
16
0.81
0.05
0.68
0.05
0.68
0.05
2.24
26.3
Conv 10
64
16
0.71
0.05
0.63
0.05
0.63
0.05
14.43
83.68
Conv
16
64
0.88
0.05
0.63
0.05
0.63
0.05
26.3
Conv 12
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.98
26.3
Conv 13
64
16
0.84
0.05
0.63
0.05
0.63
0.05
6.21
83.68
Conv 14
16
64
0.70
0.05
0.63
0.05
0.63
0.05
1.78
26.3
Conv 15
16
144
0.62
0.05
0.68
0.05
0.68
0.05
2.58
26.3
Conv16 16
64
16
0.68
0.05
0.63
0.05
0.63
0.05
9.12
83.68
Conv17 17
16
64
0.69
0.05
0.63
0.05
0.63
0.05
2.25
26.3
Conv18
16
144
0.82
0.05
0.68
0.05
0.68
0.05
0.69
26.3
82

---

Under review as a conference paper at ICLR 2021
Conv 19
64
16
0.85
0.05
0.63
0.05
0.63
0.05
6.68
83.68
Conv20
16
64
0.68
0.05
0.63
0.05
0.63
0.05
3.13
26.3
Conv21
16
144
0.81
0.05
0.68
0.05
0.68
0.05
1.92
26.3
Conv22
64
16
0.81
0.05
0.63
0.05
0.63
0.05
9.84
83.68
Conv23
16
64
0.88
0.05
0.63
0.05
0.63
0.05
0.99
26.3
Conv24
16
144
0.91
0.05
0.68
0,05
0.68
0.05
0.26
26.3
Conv25
64
16
0.91
0.05
0.63
0.05
0.63
0.05
2.74
83.68
Conv26
16
64
0.60
0.05
0.63
0.05
0.63
0.05
3.57
26.3
Conv27
16
0.85
0.05
0.68
0.05
0.68
0.05
1.47
26.3
Conv28
64
16
0.71
0.05
0.63
0.05
0.63
0.05
9.72
83.68
Conv29
16
64
0.58
0.05
0.63
0.05
0.63
0.05
3.86
26.3
Conv30
16
144
0.52
0.05
0.68
0.05
0.68
0.05
4.55
26.3
Conv31
64
16
0.62
0.05
0.63
0.05
0.63
0.05
9.97
83.68
Conv32
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.92
26.3
Conv33
16
144
0.86
0.05
0.68
0.05
0.68
0.05
0.87
26.3
Conv34
64
16
0.87
0.05
0.63
0.05
0.63
0.05
6.02
83.68
Conv35
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.97
26.3
Conv36
16
144
0.83
0.05
0.68
0.05
0.68
0.05
2.01
26.3
Conv37
64
16
0.86
0.05
0.63
0.05
0.63
0.05
3.83
83.68
Conv38
16
64
0.84
0.05
0.63
0.05
0.63
0.05
1.73
26.3
Conv39
16
144
0.83
0.05
0.68
0.05
0.68
0.05
1.99
26.3
Conv40
64
16
0.86
0.05
0.63
0.05
0.63
0.05
7.64
83.68
Conv41
16
64
0.90
0.05
0.63
0.05
0.63
0.05
1.03
26.3
Conv42
16
144
0.91
0.05
0.68
0.05
0.68
0.05
0.3
26.3
Conv43
64
16
0.89
0.05
0.63
0.05
0.63
0.05
3.45
83.68
Conv44
16
64
0.63
0.05
0.63
0.05
0.63
0.05
3.89
26.3
Conv45
16
144
0.61
0.05
0.68
0.05
0.68
0.05
3.72
26.3
Conv46
64
16
0.83
0.05
0.63
0.05
0.63
0.05
12.99
83.68
Conv47
16
64
0.49
0.05
0.63
0.05
0.63
0.05
4.34
26.3
Conv48
16
144
0.53
0.05
0.68
0.05
0.68
0.05
2.15
26.3
Conv49
64
16
0.84
0.05
0.63
0.05
0.63
0.05
10.73
83.68
Conv50
16
64
0.67
0.05
0.63
0.05
0.63
0.05
2.16
26.3
Conv51
16
0.83
0.05
0.68
0.05
0.68
0.05
2.72
26.3
C2nv52
64
16
0.86
0.05
0.63
0.05
0.63
0.05
7.04
83.68
Conv53
16
64
0.82
0.05
0.63
0.05
0.63
0.05
26.3
Conv54
16
144
0.69
0.05
0.68
0.05
0.68
0.05
2.71
26.3
Conv55
64
16
0.81
0.05
0.63
0.05
0.63
0.05
14.14
83.68
Conv56
32
64
0.63
0.05
0.67
0.05
0.67
0.05
11.28
46.19
Conv57
32
288
0.65
0.05
0.83
0.05
0.83
0.05
5.96
46.19
Conv58
128
32
0.36
0.05
0.74
0.05
0.74
0.05
98.65
155.4
Conv59
32
128
0.54
0.05
0.74
0.05
0.74
0.05
5.87
46.19
Conv60
32
288
0.52
0.05
0.83
0.05
0.83
0.05
5.99
46.19
Conv61
128
32
0.64
0.05
0.74
0.05
0.74
0.05
40.2
155.4
Conv62
32
128
0.38
0.05
0.74
0.05
0.74
0.05
3.92
46.19
Conv63
32
288
0.53
0.05
0.83
0.05
0.83
0.05
3.51
46.19
Conv64
128
32
0.70
0.05
0.74
0.05
0.74
0.05
19.17
155.4
Conv65
32
128
0.67
0.05
0.74
0.05
0.74
0.05
1.91
46.19
Conv66
32
288
0.67
0.05
0.83
0.05
0.83
0.05
1.78
46.19
Conv67
128
32
0.82
0.05
0.74
0.05
0.74
0.05
18.69
155.4
Conv68
32
128
0.81
0.05
0.74
0.05
0.74
0.05
2.23
46.19
Conv69
32
288
0.66
0.05
0.83
0.05
0.83
0.05
2.37
46.19
Conv70 70
128
32
0.83
0.05
0.74
0.05
0.74
0.05
22.83
155.4
Conv71
32
128
0.31
0.05
0.74
0.05
0.74
0.05
7.17
46.19
Cony72
32
288
0.59
0.05
0.83
0.05
0.83
0.05
3.55
46.19
Conv73
128
32
0.53
0.05
0.74
0.05
0.74
0.05
28.29
155.4
Cony74
32
128
0.53
0.05
0.74
0.05
0.74
0.05
5.34
46.19
Conv75
32
288
0.64
0.05
0.83
0.05
0.83
0.05
3.14
46.19
Conv76
128
32
0.54
0.05
0.74
0.05
0.74
0.05
36.96
155.4
Conv77
32
128
0.84
0.05
0.74
0.05
0.74
0.05
2.0
46.19
83

---

Under review as a conference paper at ICLR 2021
Conv78
32
288
0.71
0.05
0.83
0.05
0.83
0.05
1.8
46.19
Conv79
128
32
0.85
0.05
0.74
0.05
0.74
0.05
13.94
153.4
Conv80
32
128
0.67
0.05
0.74
0.05
0.74
0.05
1.48
46.19
Conv81
32
288
0.85
0.05
0.83
0.05
0.83
0.05
1.17
46.19
Conv82
128
32
0.71
0.05
0.74
0.05
0.74
0.05
13.44
155.4
Conv83
32
128
0.84
0.05
0.74
0,05
0.74
0.05
1.66
46.19
Conv84
32
288
0.86
0.05
0.83
0.05
0.83
0.05
1.05
46.19
Conv85
128
32
0.84
0.05
0.74
0.05
0.74
0.05
10.81
155.4
Conv86
32
128
0.70
0.05
0.74
0.05
0.74
0.05
1.56
46.19
Conv87
32
288
0.68
0.05
0.83
0.05
0.83
0.05
0.98
46.19
Conv88
128
32
0.84
0.05
0.74
0.05
0.74
0.05
12.75
155.4
Conv89
32
128
0.82
0.05
0.74
0.05
0.74
0.05
2.06
46.19
Conv90
32
288
0.82
0.05
0.83
0.05
0.83
0.05
1.19
46.19
Conv91
128
32
0.84
0.05
0.74
0.05
0.74
0.05
10.13
155.4
Conv92
32
128
0.83
0.05
0.74
0.05
0.74
0.05
2.47
46.19
Conv93
32
288
0.84
0.05
0.83
0.05
0.83
0.05
1.3
46.19
Conv94
128
32
0.80
0.05
0.74
0.05
0.74
0.05
17.69
155.4
Conv95
32
128
0.87
0.05
0.74
0.05
0.74
0.05
1.15
46.19
Conv96
32
288
0.86
0.05
0.83
0.05
0.83
0.05
1.12
46.19
Conv97
128
32
0.84
0.05
0.74
0.05
0.74
0.05
11.38
155.4
Conv98
32
128
0.83
0.05
0.74
0.05
0.74
0.05
1.54
46.19
Conv99
32
288
0.86
0.05
0.83
0.05
0.83
0.05
1.13
46.19
Conv 1 100
128
32
0.85
0.05
0.74
0.05
0.74
0.05
16.13
155.4
Conv 101
32
128
0.59
0.05
0.74
0.05
0.74
0.05
2.81
46.19
Conv 102
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.11
46.19
Conv 103
128
32
0.83
0.05
0.74
0.05
0.74
0.05
11.62
155.4
Conv 104
32
128
0.71
0.05
0.74
0.05
0.74
0.05
2.22
46.19
Conv 105
32
288
0.85
0.05
0.83
0.05
0.83
0.05
1.28
46.19
Conv
128
32
0.82
0.05
0.74
0.05
0.74
0.05
16.64
155.4
Conv 107
32
128
0.68
0.05
0.74
0.05
0.74
0.05
2.62
46.19
Conv 108
32
288
0.85
0.05
0.83
0.05
0.83
0.05
1.0
46.19
Conv 109
128
32
0.62
0.05
0.74
0.05
0.74
0.05
18.47
155.4
Conv 10
64
128
0.58
0.05
0.82
0.05
0.82
0.05
7.8
83.68
Conv 11
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.22
83.68
Conv
256
64
0.50
0.05
0.90
0.05
0.90
0.05
71.76
294.32
Conv 13
64
256
0.84
0.05
0.90
0.05
0.90
0.05
0.75
83.68
Conv
576
0.86
0.05
0.97
0.05
0.97
0.05
0.79
83.68
Conv 115
256
64
0.70
0.05
0.90
0.05
0.90
0.05
15.19
294.32
Conv
64
256
0.82
0.05
0.90
0.05
0.90
0.05
1.21
83.68
Conv 17
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.8
83.68
Conv
256
64
0.87
0.05
0.90
0.05
0.90
0.05
15.06
294.32
Conv 1 19
64
256
0.62
0.05
0.90
0.05
0.90
0.05
1.12
83.68
Conv 120
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.73
83.68
Conv 121
256
64
0.82
0.05
0.90
0.05
0.90
0.05
13.67
294.32
Conv 1 122
64
256
0.59
0.05
0.90
0.05
0.90
0.05
1.17
83.68
Conv 123
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.86
83.68
Conv 1 24
256
64
0.84
0.05
0.90
0.05
0.90
0.05
15.79
294.32
Conv 1 125
64
256
0.84
0.05
0.90
0.05
0.90
0.05
1.12
83.68
Conv 126
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.61
83,68
Conv 127
256
64
0.84
0.05
0.90
0.05
0.90
0.05
10.56
294.32
Conv 1inv128
64
256
0.85
0.05
0.90
0.05
0.90
0.05
0.83
83.68
Conv 129
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.64
83.68
Conv
256
64
0.81
0.05
0.90
0.05
0.90
0.05
13.88
294.32
Conv 131
64
256
0.85
0.05
0.90
0.05
0.90
0.05
1.07
83.68
Conv133
64
576
0.89
0.05
0.97
0.05
0.97
0.05
0.92
83.68
Conv 133
256
64
0.85
0.05
0.90
0.05
0.90
0.05
12.13
294.32
Conv 134
64
256
0.84
0.05
0.90
0.05
0.90
0.05
1.46
83.68
Conv 135
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.8
83.68
Conv
256
64
0.84
0.05
0.90
0.05
0.90
0.05
13.75
294.32
84

---

Under review as a conference paper at ICLR 2021
Conv 137
64
256
0.83
0.05
0.90
0.05
0.90
0.05
0.95
83.68
Conv 138
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.51
83.68
Conv 139
256
64
0.84
0.05
0.90
0.05
0.90
0.05
13.21
294.32
Conv 140
64
256
0.85
0.05
0.90
0.05
0.90
0.05
1.24
83.68
Conv 141
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.75
83.68
Conv 142
256
64
0.81
0,05
0.90
0.05
0.90
0.05
13.98
294.32
Conv 143
64
256
0.81
0.05
0.90
0.05
0.90
0.05
0.93
83.68
Conv 144
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.74
83.68
Conv 145
256
64
0.83
0.05
0.90
0.05
0.90
0.05
15.25
294.32
Conv 146
64
256
0.69
0.05
0.90
0.05
0.90
0.05
1.14
83.68
Conv 147
576
0.85
0.05
0.97
0.05
0.97
0.05
0.67
83.68
Conv 148
256
64
0.82
0.05
0.90
0.05
0.90
0.05
15.25
294.32
Conv 149
64
256
0.81
0.05
0.90
0.05
0.90
0.05
0.89
83.68
Conv 150
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.93
83.68
Conv 151 151
256
64
0.71
0.05
0.90
0.05
0.90
0.05
18.2
294.32
Conv 152
64
256
0.55
0.05
0.90
0.05
0.90
0.05
1.23
83.68
Conv 153
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.97
83.68
Conv 1 154
256
64
0.66
0.05
0.90
0.05
0.90
0.05
22.91
294.32
Conv 155
64
256
0.60
0.05
0.90
0.05
0.90
0.05
1.26
83.68
Conv 156
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.88
83.68
Conv 157
256
64
0.62
0.05
0.90
0.05
0.90
0.05
23.61
294.32
Conv 158
64
256
0.67
0.05
0.90
0.05
0.90
0.05
0.96
83.68
Conv 159
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.91
83.68
Conv 160
256
64
0.61
0.05
0.90
0.05
0.90
0.05
37.47
294.32
Conv 161
64
256
0.54
0.05
0.90
0.05
0.90
0.05
1.39
83.68
Conv 162
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.4
83.68
Conv 163
256
64
0.37
0.05
0.90
0.05
0.90
0.05
133.74
294.32
Passing rate
99.39%
100.0%
100.0%
98.77%
P.5
INITIALIZATION
config:
Table 32: Cifar100 kaiming-unffom-RReNelll-.
Layer
Number
dim
Gaussian
Mean Left
Mean Mean.Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
16
27
0.00
0.05
0.58
0.05
0.58
0.05
425.97
26.3
Conv2
16
16
0.12
0.05
0.56
0.05
0.56
0.05
218.1
26.3
Conv3
16
144
0.01
0.05
0.68
0.05
0.68
0.05
168.11
26.3
Conv4
64
16
0.48
0.05
0.63
0.05
0.63
0.05
64.78
83.68
5
16
64
0.20
0.05
0.63
0.05
0.63
0.05
38.28
26.3
Conv6
16
144
0.31
0.05
0.68
0.05
0.68
0.05
15.42
26.3
Conv7
64
16
0.56
0.05
0.63
0.05
0.63
0.05
41.67
83.68
Conv8
16
64
0.29
0.05
0.63
0.05
0.63
0.05
18.91
26.3
Conv9
16
144
0.48
0.05
0.68
0.05
0.68
0.05
9.02
26.3
Conv10
64
16
0.82
0.05
0.63
0.05
0.63
0.05
9.31
83.68
Conv
16
64
0.42
0.05
0.63
0.05
0.63
0.05
9.62
26.3
Conv 12
16
144
0.48
0.05
0.68
0.05
0.68
0.05
2.79
26.3
Conv 13
64
16
0.69
0.05
0.63
0.05
0.63
0.05
13.65
83.68
Conv 14
16
64
0.59
0.05
0.63
0.05
0.63
0.05
3.04
26.3
Conv15 15
16
144
0.65
0.05
0.68
0.05
0.68
0.05
2.18
26.3
Conv16
64
16
0.68
0.05
0.63
0.05
0.63
0.05
11.98
83.68
85

---

Under review as a conference paper at ICLR 2021
Conv 17
16
64
0.67
0.05
0.63
0.05
0.63
0.05
1.2
26.3
Conv 18
16
144
0.85
0.05
0.68
0.05
0.68
0.05
0.74
26.3
Conv19 19
64
16
0.86
0.05
0.63
0.05
0.63
0.05
7.27
83.68
Conv20
16
64
0.58
0.05
0.63
0.05
0.63
0.05
3.37
26.3
Conv21
16
144
0.50
0.05
0.68
0.05
0.68
0.05
3.47
26.3
Conv22
64
16
0.64
0,05
0.63
0,05
0.63
0.05
19.22
83.68
Conv23
16
64
0.60
0.05
0.63
0.05
0.63
0.05
1.89
26.3
Conv24
16
144
0.66
0.05
0.68
0.05
0.68
0.05
2.33
26.3
Conv25
64
16
0.67
0.05
0.63
0.05
0.63
0.05
10.58
83.68
C6nv26
16
64
0.56
0.05
0.63
0.05
0.63
0.05
9.27
26.3
Conv27
16
144
0.32
0.05
0.68
0.05
0.68
0.05
6.39
26.3
Conv28
64
16
0.66
0.05
0.63
0.05
0.63
0.05
15.84
83.68
Comv29
16
64
0.81
0.05
0.63
0.05
0.63
0.05
3.21
26.3
Co
16
144
0.80
0.05
0.68
0.05
0.68
0.05
2.82
26.3
Conv31
64
16
0.83
0.05
0.63
0.05
0.63
0.05
12.09
83.68
Conv32
16
64
0.64
0.05
0.63
0.05
0.63
0.05
8.7
26.3
Conv33
16
144
0.63
0.05
0.68
0.05
0.68
0.05
3.56
26.3
Conv34
64
16
0.83
0.05
0.63
0.05
0.63
0.05
12.82
83.68
Conv35
16
64
0.38
0.05
0.63
0.05
0.63
0.05
21.3
26.3
C6nv36
16
144
0.66
0.05
0.68
0.05
0.68
0.05
1.08
26.3
Conv37
64
16
0,81
0.05
0.63
0.05
0.63
0.05
13.25
83.68
Conv38
16
64
0.63
0.05
0.63
0.05
0.63
0.05
3.26
26.3
Conv39
16
144
0.82
0.05
0.68
0.05
0.68
0.05
3.46
26.3
Conv40
64
16
0.69
0.05
0.63
0.05
0.63
0.05
14.77
83.68
Conv41
16
64
0.71
0.05
0.63
0.05
0.63
0.05
2.69
26.3
Conv42
16
144
0.84
0.05
0.68
0.05
0.68
0.05
1.58
26.3
Conv43
64
16
0.88
0.05
0.63
0.05
0.63
0.05
10.16
83.68
Conv44
16
64
0.57
0.05
0.63
0.05
0.63
0.05
12.25
26.3
Conv45
16
144
0.80
0.05
0.68
0.05
0.68
0.05
2.95
26.3
Conv46
64
16
0.84
0.05
0.63
0.05
0.63
0.05
12.53
83.68
Conv47
16
64
0.81
0.05
0.63
0.05
0.63
0.05
3.93
26.3
Conv48
16
144
0.81
0.05
0.68
0.05
0.68
0.05
2.59
26.3
Conv49
64
16
0.82
0.05
0.63
0.05
0.63
0.05
8.55
83.68
Conv50
16
64
0.67
0.05
0.63
0.05
0.63
0.05
3.59
26.3
Conv51
16
144
0.70
0.05
0.68
0.05
0.68
0.05
2.53
26.3
Conv52
64
16
0.84
0.05
0.63
0.05
0.63
0.05
14.52
83.68
Conv53
16
64
0.63
0.05
0.63
0.05
0.63
0.05
2.31
26.3
Conv54
16
144
0.70
0.05
0.68
0.05
0.68
0.05
1.73
26.3
Conv55
64
16
0.87
0.05
0.63
0.05
0.63
0.05
10.24
83.68
Conv56
32
64
0.55
0.05
0.67
0.05
0.67
0.05
14.4
46.19
Conv57
32
288
0.55
0.05
0.83
0.05
0.83
0.05
6.19
46.19
Conv58
128
32
0.35
0.05
0.74
0.05
0.74
0.05
73.04
155.4
Conv59
32
128
0.61
0.05
0.74
0.05
0.74
0.05
4.02
46.19
Conv60
32
288
0.56
0.05
0.83
0.05
0.83
0.05
6.32
46.19
Conv61
128
32
0.64
0.05
0.74
0.05
0.74
0.05
34.02
155.4
Conv62
32
128
0.53
0.05
0.74
0.05
0.74
0.05
5.78
46.19
Conv63
32
288
0.59
0.05
0.83
0.05
0.83
0.05
5.53
46.19
Conv64
128
32
0.68
0.05
0.74
0.05
0.74
0.05
25.02
155.4
Conv65
32
128
0.83
0.05
0.74
0.05
0.74
0.05
3.52
46.19
Conv66
32
288
0.65
0.05
0.83
0.05
0.83
0.05
3.72
46.19
Conv67
128
32
0.66
0.05
0.74
0.05
0.74
0.05
17.98
155.4
Conv68
32
128
0.69
0.05
0.74
0.05
0.74
0.05
4.34
46.19
Conv69
32
288
0.69
0.05
0.83
0.05
0.83
0.05
3.5
46.19
Cony70
128
32
0.61
0.05
0.74
0.05
0.74
0.05
27.62
155.4
Conv71
32
128
0.81
0.05
0.74
0.05
0.74
0.05
2.84
46.19
Cony72
32
288
0,69
0.05
0.83
0.05
0.83
0.05
2.3
46.19
Conv73
128
32
0.69
0.05
0.74
0.05
0.74
0.05
20.52
155.4
Conv74
32
128
0.81
0.05
0.74
0.05
0.74
0.05
3.83
46.19
Conv75
32
288
0.83
0.05
0.83
0.05
0.83
0.05
2.45
46.19
86

---

Under review as a conference paper at ICLR 2021
C6
128
32
0.84
0.05
0.74
0.05
0.74
0.05
20.47
155.4
Conv
32
128
0.66
0.05
0.74
0.05
0.74
0.05
9.06
46.19
Conv78
32
288
0.58
0.05
0.83
0.05
0.83
0.05
4.44
46.19
Conv79
128
32
0.60
0.05
0.74
0.05
0.74
0.05
32.04
155.4
Conv80
32
128
0.67
0.05
0.74
0.05
0.74
0.05
3.78
46.19
Conv81
32
288
0.64
0.05
0.83
0,05
0.83
0.05
2.57
46.19
Conv82
128
32
0.65
0.05
0.74
0.05
0.74
0.05
18.89
155.4
Conv83
32
128
0.55
0.05
0.74
0.05
0.74
0.05
9.07
46.19
Conv84
32
288
0.82
0.05
0.83
0.05
0.83
0.05
1.57
46.19
Conv85
128
32
0.70
0.05
0.74
0.05
0.74
0.05
20.75
155.4
Conv86
128
0.43
0.05
0.74
0.05
0.74
0.05
6.63
46.19
Conv87
32
288
0.66
0.05
0.83
0.05
0.83
0.05
3.08
46.19
Conv88
128
32
0.70
0.05
0.74
0.05
0.74
0.05
26.59
153.4
Conv89
32
128
0.67
0.05
0.74
0.05
0.74
0.05
3.37
46.19
Conv90
32
288
0.80
0.05
0.83
0.05
0.83
0.05
2.39
46.19
Conv91
128
32
0.68
0.05
0.74
0.05
0.74
0.05
13.65
155.4
Conv92
32
128
0.66
0.05
0.74
0.05
0.74
0.05
6.03
46.19
Conv93
32
288
0.58
0.05
0.83
0.05
0.83
0.05
2.37
46.19
Conv94
128
32
0.81
0.05
0.74
0.05
0.74
0.05
12.2
155.4
Conv95
32
128
0.82
0.05
0.74
0.05
0.74
0.05
3.9
46.19
Conv96
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.87
46.19
Conv97
128
32
0.80
0.05
0.74
0.05
0.74
0.05
18.32
155.4
Conv98
32
128
0.81
0.05
0.74
0.05
0.74
0.05
3.33
46.19
Conv99
32
288
0.69
0.05
0.83
0.05
0.83
0.05
2.19
46.19
Conv 100
128
32
0.81
0.05
0.74
0.05
0.74
0.05
18.19
155.4
Conv 101
32
128
0.83
0.05
0.74
0.05
0.74
0.05
2.32
46.19
Conv 102
32
288
0.86
0.05
0.83
0.05
0.83
0.05
1.37
46.19
Conv 103
128
32
0.69
0.05
0.74
0.05
0.74
0.05
16.8
155.4
Conv 104
32
128
0.63
0.05
0.74
0.05
0.74
0.05
4.21
46.19
Conv 105
32
288
0.84
0.05
0.83
0.05
0.83
0.05
2.49
46.19
Conv 106
128
32
0.62
0.05
0.74
0.05
0.74
0.05
20.93
155.4
Conv 107
32
128
0.67
0.05
0.74
0.05
0.74
0.05
2.95
46.19
Conv108
32
288
0.63
0.05
0.83
0.05
0.83
0.05
1.17
46.19
Conv 109
128
32
0.65
0.05
0.74
0.05
0.74
0.05
16.5
155.4
Conv
0.50
0.05
0.82
0.05
0.82
0.05
9.42
83.68
Conv 11
64
576
0.70
0.05
0.97
0.05
0.97
0.05
2.0
83.68
Conv
256
64
0.57
0.05
0.90
0.05
0.90
0.05
86.43
294.32
Conv 13
64
256
0.59
0.05
0.90
0.05
0.90
0.05
2.79
83.68
Conv
64
576
0.83
0.05
0.97
0.05
0.97
0.05
1.65
83.68
Conv 11 15
256
64
0.68
0.05
0.90
0.05
0.90
0.05
17.79
294.32
Conv
64
256
0.69
0.05
0.90
0.05
0.90
0.05
3.54
83.68
Conv 17
64
576
0.81
0.05
0.97
0.05
0.97
0.05
2.4
83.68
Conv 18
256
64
0.70
0.05
0.90
0.05
0.90
0.05
19.61
294.32
Conv 1 19
64
256
0.83
0.05
0.90
0.05
0.90
0.05
2.09
83.68
Conv 120
64
576
0.86
0.05
0.97
0.05
0.97
0.05
1.3
83.68
Conv 121
256
64
0.82
0.05
0.90
0.05
0.90
0.05
17.89
294.32
Conv 1 122
64
256
0.81
0.05
0.90
0.05
0.90
0.05
2.49
83.68
Conv 123
64
576
0.64
0.05
0.97
0.05
0.97
0.05
1.33
83.68
Conv 124
256
64
0.70
0.05
0.90
0.05
0.90
0.05
20.03
294.32
Conv 125
64
256
0.83
0.05
0.90
0.05
0.90
0.05
2.66
83.68
Conv 1
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.52
83.68
Conv 127
256
64
0.63
0.05
0.90
0.05
0.90
0.05
22.08
294.32
Conv
64
256
0.60
0.05
0.90
0.05
0.90
0.05
3.27
83.68
Conv1 29
64
576
0.64
0.05
0.97
0.05
0.97
0.05
2.41
83.68
Conv
256
64
0.65
0.05
0.90
0.05
0.90
0.05
26.37
294.32
Conv 131
64
256
0.81
0.05
0.90
0.05
0.90
0.05
2.14
83.68
Conv 132
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.69
83.68
Conv 133
256
64
0.71
0.05
0.90
0.05
0.90
0.05
16.51
294.32
Conv
64
256
0.58
0.05
0.90
0.05
0.90
0.05
2.65
83.68
87

---

Under review as a conference paper at ICLR 2021
Conv 135
64
576
0.84
0.05
0.97
0.05
0.97
0.05
2.34
83.68
Conv 136
256
64
0.83
0.05
0.90
0.05
0.90
0.05
21.06
294.32
Conv 137
64
256
0.81
0.05
0.90
0.05
0.90
0.05
2.17
83.68
Conv 1 138
64
576
0.82
0.05
0.97
0.05
0.97
0.05
1.3
83.68
Conv139
256
64
0.66
0.05
0.90
0.05
0.90
0.05
23.37
294.32
Conv 140
64
256
0.81
0,05
0.90
0.05
0,90
0.05
1.45
83.68
Conv 141
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.02
83.68
Conv 142
256
64
0.70
0.05
0.90
0.05
0.90
0.05
24.32
294.32
Conv 143
64
256
0.68
0.05
0.90
0.05
0.90
0.05
2.39
83.68
Conv 144
64
576
0.83
0.05
0.97
0.05
0.97
0.05
1.32
83.68
Conv 45
256
64
0.81
0.05
0.90
0.05
0.90
0.05
26.7
294.32
Conv 1 46
64
256
0.68
0.05
0.90
0.05
0.90
0.05
2.43
83.68
Conv 147
64
576
0.82
0.05
0.97
0.05
0.97
0.05
1.31
83.68
Conv 148
256
64
0.65
0.05
0.90
0.05
0.90
0.05
29.8
294.32
Conv 149
64
256
0.62
0.05
0.90
0.05
0.90
0.05
2.1
83.68
Conv 1M0
64
576
0.83
0.05
0.97
0.05
0.97
0.05
1.14
83.68
Conv 151
256
64
0.62
0.05
0.90
0.05
0.90
0.05
35.97
294.32
Conv
64
256
0.64
0.05
0.90
0.05
0.90
0.05
1.69
83.68
Conv 153
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.07
83.68
Conv 154
256
64
0.64
0.05
0.90
0.05
0.90
0.05
27.5
294.32
Conv 155
64
256
0.59
0.05
0.90
0.05
0.90
0.05
2.52
83.68
Conv 56
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.11
83.68
Conv 157
256
64
0.59
0.05
0.90
0.05
0.90
0.05
32.33
294.32
Conv 158
64
256
0.60
0.05
0.90
0.05
0.90
0.05
1.95
83.68
Conv 159
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.16
83.68
Conv 160
256
64
0.56
0.05
0.90
0.05
0.90
0.05
50.72
294.32
Conv 161
64
256
0.41
0.05
0.90
0.05
0.90
0.05
2.61
83.68
Conv 162
64
576
0.82
0.05
0.97
0.05
0.97
0.05
0.61
83.68
C6nv163
256
64
0.33
0.05
0.90
0.05
0.90
0.05
194.0
294.32
Passing rate
98.77%
100.0%
100.0%
97.55%
Table 33: Cifar100 Xavier-normal-ResNellee.
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.00
0.05
0.58
0.05
0.42
0.05
4100.8
26.3
Conv2
16
16
0.07
0.05
0.56
0.05
0.56
0.05
263.78
26.3
Conv3
16
144
0.03
0.05
0.68
0.05
0.68
0.05
124.29
26.3
Conv4
64
16
0.24
0.05
0.63
0.05
0.63
0.05
218.9
83.68
Conv5
16
64
0.28
0.05
0.63
0.05
0.63
0.05
42.91
26.3
Conv6
16
144
0.25
0.05
0.68
0.05
0.68
0.05
29.6
26.3
Conv7
64
16
0.41
0.05
0.63
0.05
0.63
0.05
63.85
83.68
Conv8
16
64
0.38
0.05
0.63
0.05
0.63
0.05
10.79
26.3
Conv9
16
144
0.34
0.05
0.68
0.05
0.68
0.05
15.78
26.3
Conv 10
64
16
0.59
0.05
0.63
0.05
0.63
0.05
40.51
83.68
Conv
16
64
0.41
0.05
0.63
0.05
0.63
0,05
9.5
26.3
Conv 12
16
0.43
0.05
0.68
0.05
0.68
0.05
8.63
26.3
Conv 13
64
16
0.55
0.05
0.63
0.05
0.63
0.05
29.22
83.68
Conv 14
16
64
0.47
0.05
0.63
0.05
0.63
0.05
3.81
26.3
Conv 15
16
144
0.57
0.05
0.68
0.05
0.68
0.05
1.54
26.3
Conv 16
64
16
0.80
0.05
0.63
0.05
0.63
0.05
13.88
83.68
Conv 17
16
64
0.68
0.05
0.63
0.05
0.63
0.05
1.34
26.3
Conv 18
16
144
0.83
0.05
0.68
0.05
0.68
0.05
1.23
26.3
Conv19
64
16
0.71
0.05
0.63
0.05
0.63
0.05
7.88
83.68
Conv20
16
64
0.70
0.05
0.63
0.05
0.63
0.05
0.98
26.3
88

---

Under review as a conference paper at ICLR 2021
Conv21
16
144
0.81
0.05
0.68
0.05
0.68
0.05
0.78
26.3
Conv22
64
16
0.83
0.05
0.63
0.05
0.63
0.05
7.01
83.68
Conv23
16
64
0.48
0.05
0.63
0.05
0.63
0.05
4.48
26.3
Conv24
16
144
0.53
0.05
0.68
0.05
0.68
0.05
2.56
26.3
Conv25
64
16
0.48
0.05
0.63
0.05
0.63
0.05
29.86
83.68
Conv26
16
64
0.84
0,05
0.63
0,05
0.63
0.05
0.63
26.3
Conv27
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.48
26.3
Conv28
64
16
0.92
0.05
0.63
0.05
0.63
0.05
2.04
83.68
Conv29
16
64
0.62
0.05
0.63
0.05
0.63
0.05
5.32
26.3
Conv30
16
144
0.53
0.05
0.68
0.05
0.68
0.05
3.17
26.3
Conv31
64
16
0.56
0.05
0.63
0.05
0.63
0.05
23.86
83.68
Conv32
16
64
0.61
0.05
0.63
0.05
0.63
0.05
4.21
26.3
Conv33
16
144
0.53
0.05
0.68
0.05
0.68
0.05
4.85
26.3
Conv34
64
16
0.65
0.05
0.63
0.05
0.63
0.05
12.41
83.68
Conv35
16
64
0.68
0.05
0.63
0.05
0.63
0.05
1.76
26.3
Conv36
16
144
0.89
0.05
0.68
0.05
0.68
0.05
0.46
26.3
Conv37
64
16
0.85
0.05
0.63
0.05
0.63
0.05
4.31
83.68
Conv38
16
64
0.40
0.05
0.63
0.05
0.63
0.05
10.81
26.3
Conv39
16
144
0.43
0.05
0.68
0.05
0.68
0.05
11.21
26.3
Conv40
64
16
0.66
0.05
0.63
0.05
0.63
0.05
21.52
83.68
Conv41
16
64
0.85
0.05
0.63
0.05
0.63
0.05
1.27
26.3
Conv42
16
0.89
0.05
0.68
0.05
0.68
0.05
0.7
26.3
Conv43
64
16
0.86
0.05
0.63
0.05
0.63
0.05
4.92
83.68
Conv44
16
64
0.80
0.05
0.63
0.05
0.63
0.05
3.44
26.3
Conv45
16
144
0.65
0.05
0.68
0.05
0.68
0.05
3.32
26.3
Conv46
64
16
0.67
0.05
0.63
0.05
0.63
0.05
20.05
83.68
Conv47
16
64
0.62
0.05
0.63
0.05
0.63
0.05
5.66
26.3
Conv48
16
144
0.67
0.05
0.68
0.05
0.68
0.05
4.6
26.3
Conv49
64
16
0.81
0.05
0.63
0.05
0.63
0.05
13.5
83.68
Conv50
16
64
0.55
0.05
0.63
0.05
0.63
0.05
6.88
26.3
Conv51
16
144
0.56
0.05
0.68
0.05
0.68
0.05
6.35
26.3
Conv52
64
16
0.59
0.05
0.63
0.05
0.63
0.05
26.53
83.68
Conv53
16
64
0.66
0.05
0.63
0.05
0.63
0.05
11.71
26.3
Conv54
16
144
0.56
0.05
0.68
0.05
0.68
0.05
5.5
26.3
Conv55
64
16
0.55
0.05
0.63
0.05
0.63
0.05
46.57
83.68
Conv56
32
64
0.33
0.05
0.67
0.05
0.67
0.05
32.25
46.19
Conv57
32
288
0.55
0.05
0.83
0.05
0.83
0.05
3.62
46.19
Conv58
128
32
0.42
0.05
0.74
0.05
0.74
0.05
78.52
155.4
Conv59
32
128
0.66
0.05
0.74
0.05
0.74
0.05
2.9
46.19
Conv60
32
288
0.53
0.05
0.83
0.05
0.83
0.05
4.46
46.19
Conv61
128
32
0.60
0.05
0.74
0.05
0.74
0.05
26.11
155.4
Conv62
32
128
0.57
0.05
0.74
0.05
0.74
0.05
6.41
46.19
Conv63
32
288
0.53
0.05
0.83
0.05
0.83
0.05
5.29
46.19
Conv64
128
32
0.67
0.05
0.74
0.05
0.74
0.05
40.74
155.4
Conv65
32
128
0.60
0.05
0.74
0.05
0.74
0.05
5,03
46.19
Conv66
32
288
0.60
0.05
0.83
0.05
0.83
0.05
4.94
46.19
Conv67
128
32
0.68
0.05
0.74
0.05
0.74
0.05
27.32
155.4
Conv68
32
128
0.85
0.05
0.74
0.05
0.74
0.05
2.84
46.19
Conv69
32
288
0.81
0.05
0.83
0.05
0.83
0.05
2.45
46.19
Conv70
128
32
0.82
0.05
0.74
0.05
0.74
0.05
17.34
155.4
Conv71
32
128
0.83
0.05
0.74
0.05
0.74
0.05
3.08
46.19
Conv72 72
32
288
0.66
0.05
0.83
0.05
0.83
0.05
2.93
46.19
Conv73
128
32
0.83
0.05
0.74
0.05
0.74
0.05
15.7
155.4
Conv74
32
128
0.63
0.05
0.74
0.05
0.74
0.05
3.95
46.19
Conv75
32
288
0.67
0.05
0.83
0.05
0.83
0.05
2.9
46.19
Conv 76
128
32
0.63
0.05
0.74
0.05
0.74
0.05
21.4
155.4
Conv77
32
128
0.83
0.05
0.74
0.05
0.74
0.05
1.55
46.19
Conv78
32
288
0.63
0.05
0.83
0.05
0.83
0.05
1.41
46.19
Conv79
128
32
0.80
0.05
0.74
0.05
0.74
0.05
16.74
155.4
89

---

Under review as a conference paper at ICLR 2021
Conv80
32
128
0.84
0.05
0.74
0.05
0.74
0.05
1.6
46.19
Conv81
32
288
0.84
0.05
0.83
0.05
0.83
0.05
2.41
46.19
Conv82
128
32
0.84
0.05
0.74
0.05
0.74
0.05
12.17
155.4
Conv83
32
128
0.80
0.05
0.74
0.05
0.74
0.05
3.14
46.19
Conv84
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.82
46.19
Conv85
128
32
0.85
0.05
0.74
0,05
0.74
0.05
16.09
153.4
Conv86
32
128
0.46
0.05
0.74
0.05
0.74
0.05
7.45
46.19
Conv87
32
288
0.61
0.05
0.83
0.05
0.83
0.05
4.67
46.19
Conv88
128
32
0.82
0.05
0.74
0.05
0.74
0.05
26.75
155.4
Conv89
32
128
0.61
0.05
0.74
0.05
0.74
0.05
5.19
46.19
Conv90
32
288
0.68
0.05
0.83
0.05
0.83
0.05
3.7
46.19
Conv91
128
32
0.67
0.05
0.74
0.05
0.74
0.05
32.92
155.4
Conv92
32
128
0.86
0.05
0.74
0.05
0.74
0.05
2.79
46.19
Conv93
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.96
46.19
Conv94
128
32
0.81
0.05
0.74
0.05
0.74
0.05
16.14
155.4
Conv95
32
128
0.42
0.05
0.74
0.05
0.74
0.05
4.51
46.19
Conv96
32
288
0.68
0.05
0.83
0.05
0.83
0.05
2.83
46.19
Conv97
128
32
0.61
0.05
0.74
0.05
0.74
0.05
22.18
155.4
Conv98
32
128
0.80
0.05
0.74
0.05
0.74
0.05
1.75
46.19
Conv99
32
288
0.86
0.05
0.83
0.05
0.83
0.05
2.14
46.19
Conv 100
128
32
0.88
0.05
0.74
0.05
0.74
0.05
14.54
155.4
Conv 101
32
128
0.80
0.05
0.74
0.05
0.74
0.05
1.6
46.19
Conv 102
32
288
0.85
0.05
0.83
0.05
0.83
0.05
1.48
46.19
Conv 103
128
32
0.85
0.05
0.74
0.05
0.74
0.05
12.08
155.4
Conv
32
128
0.82
0.05
0.74
0.05
0.74
0.05
4.97
46.19
Conv 105
32
288
0.67
0.05
0.83
0.05
0.83
0.05
2.16
46.19
Conv 106
128
32
0.59
0.05
0.74
0.05
0.74
0.05
22.44
155.4
Conv 107
32
128
0.64
0.05
0.74
0.05
0.74
0.05
7.26
46.19
Conv
32
288
0.65
0.05
0.83
0.05
0.83
0.05
1.05
46.19
Conv 109
128
32
0.67
0.05
0.74
0.05
0.74
0.05
32.58
155.4
Conv 110
64
128
0.49
0.05
0.82
0.05
0.82
0.05
6.7
83.68
Conv 11
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.62
83.68
Conv
256
64
0.47
0.05
0.90
0.05
0.90
0.05
76.58
294.32
Conv 13
64
256
0.85
0.05
0.90
0.05
0.90
0.05
2.22
83.68
Conv
64
576
0.70
0.05
0.97
0.05
0.97
0.05
1.93
83.68
Conv 15
256
64
0.69
0.05
0.90
0.05
0.90
0.05
24.15
294.32
Conv
64
256
0.80
0.05
0.90
0.05
0.90
0.05
2.17
83.68
Conv 1 17
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.47
83.68
Conv 18
256
64
0.70
0.05
0.90
0.05
0.90
0.05
23.34
294.32
Conv 1 19
64
256
0.81
0.05
0.90
0.05
0.90
0.05
1.76
83.68
Conv 120
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.17
83.68
Conv 121
256
64
0.69
0.05
0.90
0.05
0.90
0.05
16.53
294.32
Conv 122
64
256
0.70
0.05
0.90
0.05
0.90
0.05
1.66
83.68
Conv 123
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.99
83.68
Conv 1 124
256
64
0.80
0.05
0.90
0.05
0.90
0.05
15.94
294.32
Conv 125
64
256
0.80
0.05
0.90
0.05
0.90
0.05
2.09
83.68
Conv 26
64
576
0.71
0.05
0.97
0.05
0.97
0.05
1.94
83.68
Conv 127
256
64
0.68
0.05
0.90
0.05
0.90
0.05
20.79
294.32
Conv 128
64
256
0.70
0.05
0.90
0.05
0.90
0.05
1.87
83.68
Conv 129
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.45
83.68
Conv 1
256
64
0.68
0.05
0.90
0.05
0.90
0.05
17.26
294.32
Conv 131
64
256
0.68
0.05
0.90
0.05
0.90
0.05
2.05
83.68
Conv Monv132
64
576
0.82
0.05
0.97
0.05
0.97
0.05
1.15
83.68
Conv 1
256
64
0.81
0.05
0.90
0.05
0.90
0.05
15.07
294.32
Conv
64
256
0.61
0.05
0.90
0.05
0.90
0.05
2.3
83.68
Conv 135
64
576
0.67
0.05
0.97
0.05
0.97
0.05
1.2
83.68
Conv 136
256
64
0.65
0.05
0.90
0.05
0.90
0.05
17.54
294.32
Conv 137
64
256
0.67
0.05
0.90
0.05
0.90
0.05
1.75
83.68
Conv
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.06
83.68
90

---

Under review as a conference paper at ICLR 2021
Conv133
256
64
0.69
0.05
0.90
0.05
0.90
0.05
16.9
294.32
Conv 140
64
256
0.66
0.05
0.90
0.05
0.90
0.05
1.9
83.68
Conv 141
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.59
83.68
Conv 142
256
64
0.81
0.05
0.90
0.05
0.90
0.05
18.95
294.32
Conv 143
64
256
0.65
0.05
0.90
0.05
0.90
0.05
1.51
83.68
Conv 144
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.07
83.68
Conv 145
256
64
0.68
0.05
0.90
0.05
0.90
0.05
19.91
294.32
Conv 146
64
256
0.69
0.05
0.90
0.05
0.90
0.05
1.26
83.68
Conv 147
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.73
83.68
Conv 148
256
64
0.62
0.05
0.90
0.05
0.90
0.05
23.67
294.32
Conv 149
64
256
0.70
0.05
0.90
0.05
0.90
0.05
1.44
83.68
Conv 1 50
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.26
83.68
Conv 151
256
64
0.61
0.05
0.90
0.05
0.90
0.05
29.19
294.32
Conv 152
64
256
0.65
0.05
0.90
0.05
0.90
0.05
1.48
83.68
Conv 1 153
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.87
83.68
Conv 154
256
64
0.57
0.05
0.90
0.05
0.90
0.05
29.35
294.32
Conv 155
64
256
0.47
0.05
0.90
0.05
0.90
0.05
2.2
83.68
Conv
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.07
83.68
Conv 157
256
64
0.47
0.05
0.90
0.05
0.90
0.05
42.43
294.32
Conv 158
64
256
0.61
0.05
0.90
0.05
0.90
0.05
2.28
83.68
Conv 159
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.84
83.68
Conv 1 60
256
64
0.49
0.05
0.90
0.05
0.90
0.05
39.92
294.32
Conv 161
64
256
0.47
0.05
0.90
0.05
0.90
0.05
1.59
83.68
Conv 162
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.97
83.68
Conv 163
256
64
0.36
0.05
0.90
0.05
0.90
0.05
78.75
294.32
Passing rate
98.77%
100.0%
100.0%
96.32%
Table 34: Cifar100 -avier-umiffrmmRessSe
Layer
Number
dim
Gaussian
Mean_Left
Mean_Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Convl
16
27
0.00
0.05
0.58
0.05
0.42
0.05
5009.01
26.3
Conv2
16
16
0.02
0.05
0.56
0,05
0.56
0.05
453.3
26.3
Conv3
16
144
0.00
0.05
0.68
0.05
0.68
0.05
370.43
26.3
Conv4
64
16
0.20
0.05
0.63
0.05
0.63
0.05
303.06
83.68
Conv5
16
64
0.24
0.05
0.63
0.05
0.63
0.05
33.98
26.3
Conv6
16
144
0.06
0.05
0.68
0.05
0.68
0.05
61.45
26.3
Conv7
64
16
0.41
0.05
0.63
0.05
0.63
0.05
72.84
83.68
Conv8
16
64
0.42
0.05
0.63
0.05
0.63
0.05
8.9
26.3
Conv9
16
144
0.42
0.05
0.68
0.05
0.68
0.05
11.42
26.3
Conv10
64
16
0.47
0.05
0.63
0.05
0.63
0.05
46.89
83.68
Conv 11
16
64
0.52
0.05
0.63
0.05
0.63
0.05
5.64
26.3
Conv 12
16
144
0.53
0.05
0.68
0.05
0.68
0.05
6.54
26.3
Conv1 13
64
16
0.70
0.05
0.63
0.05
0.63
0.05
11.94
83.68
Conv 14
16
64
0.59
0.05
0.63
0.05
0.63
0.05
4.51
26.3
Conv 15
16
144
0.60
0.05
0.68
0.05
0.68
0,05
2.63
26.3
Conv 16
64
16
0.67
0.05
0.63
0.05
0.63
0.05
13.71
83.68
Conv 17
16
64
0.70
0.05
0.63
0.05
0.63
0.05
2.34
26.3
Conv
16
0.68
0.05
0.68
0.05
0.68
0.05
1.4
26.3
Conv 19
64
16
0.85
0.05
0.63
0.05
0.63
0.05
7.47
83.68
Conv20
16
64
0.52
0.05
0.63
0.05
0.63
0.05
26.3
Conv21
16
144
0.54
0.05
0.68
0.05
0.68
0.05
3.57
26.3
Conv22
64
16
0.63
0.05
0.63
0.05
0.63
0.05
22.62
83.68
Conv23
16
64
0.49
0.05
0.63
0.05
0.63
0.05
9.23
26.3
Cony24
16
144
0.47
0.05
0.68
0.05
0.68
0.05
3.77
26.3
91

---

Under review as a conference paper at ICLR 2021
Conv25
64
16
0.65
0.05
0.63
0.05
0.63
0.05
18.83
83.68
Conv26
16
64
0.52
0.05
0.63
0.05
0.63
0.05
3.82
26.3
Conv27
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.88
26.3
Conv28
64
16
0.82
0.05
0.63
0.05
0.63
0.05
9.15
83.68
Conv29
16
64
0.61
0.05
0.63
0.05
0.63
0.05
5.81
26.3
Conv30
16
144
0.56
0.05
0.68
0,05
0.68
0.05
3,08
26.3
Conv31
64
16
0.85
0.05
0.63
0.05
0.63
0.05
10.1
83.68
Conv32
16
64
0.52
0.05
0.63
0.05
0.63
0.05
8.13
26.3
Conv33
16
0.49
0.05
0.68
0.05
0.68
0.05
4.66
26.3
Conv34
64
16
0.63
0.05
0.63
0.05
0.63
0.05
14.52
83.68
Conv35
16
64
0.70
0.05
0.63
0.05
0.63
0.05
2.11
26.3
Conv36
16
144
0.86
0.05
0.68
0.05
0.68
0.05
1.17
26.3
Conv37
64
16
0.66
0.05
0.63
0.05
0.63
0.05
9.5
83.68
Conv38
16
64
0.59
0.05
0.63
0.05
0.63
0.05
7.3
26.3
Conv39
16
144
0.67
0.05
0.68
0.05
0.68
0.05
2.4
26.3
Conv40
64
16
0.80
0.05
0.63
0.05
0.63
0.05
9.38
83.68
Conv41
16
64
0.84
0.05
0.63
0.05
0.63
0.05
2.45
26.3
Conv42
16
144
0.85
0.05
0.68
0.05
0.68
0.05
2.08
26.3
Conv43
64
16
0.84
0.05
0.63
0.05
0.63
0.05
9.47
83.68
Conv44
16
64
0.38
0.05
0.63
0.05
0.63
0.05
15.34
26.3
Conv45
16
144
0.38
0.05
0.68
0.05
0.68
0.05
6.86
26.3
Conv46
64
16
0.66
0.05
0.63
0.05
0.63
0.05
20.76
83.68
Conv47
16
64
0.53
0.05
0.63
0.05
0.63
0.05
10.64
26.3
Conv48
16
144
0.67
0.05
0.68
0.05
0.68
0.05
3.99
26.3
Conv49
64
16
0.67
0.05
0.63
0.05
0.63
0.05
19.94
83.68
Conv50
16
64
0.50
0.05
0.63
0.05
0.63
0.05
12.16
26.3
Conv51
16
144
0.57
0.05
0.68
0.05
0.68
0.05
2.76
26.3
Conv52
64
16
0.66
0.05
0.63
0.05
0.63
0.05
21.25
83.68
Conv53
16
64
0.59
0.05
0.63
0.05
0.63
0.05
6.63
26.3
Conv54
16
144
0.81
0.05
0.68
0.05
0.68
0.05
2.3
26.3
Conv55
64
16
0.82
0.05
0.63
0.05
0.63
0.05
14.84
83.68
Conv56
32
64
0.52
0.05
0.67
0.05
0.67
0.05
15.63
46.19
Conv57
32
288
0.63
0.05
0.83
0.05
0.83
0.05
3.06
46.19
Conv58
128
32
0.52
0.05
0.74
0.05
0.74
0.05
52.26
155.4
Conv59
32
128
0.82
0.05
0.74
0.05
0.74
0.05
46.19
Conv60
32
288
0.61
0.05
0.83
0.05
0.83
0.05
4.98
46.19
Conv61
128
32
0.66
0.05
0.74
0.05
0.74
0.05
28.38
155.4
Conv62
32
128
0.68
0.05
0.74
0.05
0.74
0.05
3.28
46.19
Conv63
32
288
0.66
0.05
0.83
0.05
0.83
0.05
2.88
46.19
Conv64
128
32
0.82
0.05
0.74
0.05
0.74
0.05
13.36
155.4
Conv65
32
128
0.80
0.05
0.74
0.05
0.74
0.05
3.11
46.19
Conv66
32
288
0.70
0.05
0.83
0.05
0.83
0.05
1.08
46.19
Conv67
128
32
0.80
0.05
0.74
0.05
0.74
0.05
11.6
155.4
Conv68
32
128
0.86
0.05
0.74
0.05
0.74
0.05
1.2
46.19
Conv69
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.38
46.19
Conv70
128
32
0.85
0.05
0.74
0.05
0.74
0.05
13.39
155.4
Conv71 71
32
128
0.86
0.05
0.74
0.05
0.74
0.05
2.05
46.19
Conv72
32
288
0.82
0.05
0.83
0.05
0.83
0.05
2.02
46.19
Conv73 73
128
32
0.70
0.05
0.74
0.05
0.74
0.05
14.13
155.4
Conv74
32
128
0.87
0.05
0.74
0.05
0.74
0.05
1.56
46.19
C5
32
288
0.85
0.05
0.83
0.05
0.83
0.05
1.73
46.19
Conv76 76
128
32
0.83
0.05
0.74
0.05
0.74
0.05
12.27
155.4
Conv77
32
128
0.80
0.05
0.74
0.05
0.74
0.05
2.22
46.19
Cony78
32
288
0.63
0.05
0.83
0.05
0.83
0.05
2.79
46.19
Conv79
128
32
0.67
0.05
0.74
0.05
0.74
0.05
11.46
155.4
Conv80
32
128
0.50
0.05
0.74
0.05
0.74
0.05
5.83
46.19
Conv81
32
288
0.65
0.05
0.83
0.05
0.83
0.05
2.65
46.19
Conv82
128
32
0.83
0.05
0.74
0.05
0.74
0.05
18.09
155.4
Conv83
32
128
0.80
0.05
0.74
0.05
0.74
0.05
3.61
46.19
92

---

Under review as a conference paper at ICLR 2021
Conv84
32
288
0.87
0.05
0.83
0.05
0.83
0.05
1.97
46.19
Conv85
128
32
0.64
0.05
0.74
0.05
0.74
0.05
21.99
153.4
Conv86
32
128
0.87
0.05
0.74
0.05
0.74
0.05
2.38
46.19
Conv87
32
288
0.81
0.05
0.83
0.05
0.83
0.05
1.85
46.19
Conv88
128
32
0.87
0.05
0.74
0.05
0.74
0.05
14.13
155.4
Conv89
32
128
0.85
0.05
0.74
0,05
0.74
0.05
2.08
46.19
Conv90
32
288
0.87
0.05
0.83
0.05
0.83
0.05
1.11
46.19
Conv91
128
32
0.85
0.05
0.74
0.05
0.74
0.05
15.24
155.4
Conv92
32
128
0.83
0.05
0.74
0.05
0.74
0.05
2.64
46.19
Conv93
32
288
0.83
0.05
0.83
0.05
0.83
0.05
1.89
46.19
Conv94
128
32
0.83
0.05
0.74
0.05
0.74
0.05
15.49
155.4
Conv95
32
128
0.82
0.05
0.74
0.05
0.74
0.05
3.04
46.19
Conv96
32
288
0.84
0.05
0.83
0.05
0.83
0.05
1.86
46.19
Conv97
128
32
0.84
0.05
0.74
0.05
0.74
0.05
16.24
155.4
Conv98
32
128
0.55
0.05
0.74
0.05
0.74
0.05
6.03
46.19
Conv99
32
288
0.62
0.05
0.83
0.05
0.83
0.05
4.51
46.19
Conv 100
128
32
0.59
0.05
0.74
0.05
0.74
0.05
24.42
155.4
Conv
32
128
0.57
0.05
0.74
0.05
0.74
0.05
6.83
46.19
Conv 102
32
288
0.70
0.05
0.83
0.05
0.83
0.05
2.47
46.19
Conv103
128
32
0.71
0.05
0.74
0.05
0.74
0.05
25.11
155.4
Conv 104
32
128
0.83
0.05
0.74
0.05
0.74
0.05
2.85
46.19
Conv 105
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.99
46.19
Conv 1 06
128
32
0.81
0.05
0.74
0.05
0.74
0.05
16.7
155.4
Conv 107
32
128
0.61
0.05
0.74
0.05
0.74
0.05
3.72
46.19
Conv 108
32
288
0.85
0.05
0.83
0.05
0.83
0.05
1.75
46.19
Conv 109
128
32
0.64
0.05
0.74
0.05
0.74
0.05
22.31
155.4
Conv 10
64
128
0.55
0.05
0.82
0.05
0.82
0.05
9.69
83.68
Conv 11
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.92
83.68
Conv 1 12
256
64
0.52
0.05
0.90
0.05
0.90
0.05
70.08
294.32
Conv1
64
256
0.62
0.05
0.90
0.05
0.90
0.05
1.6
83.68
Conv 14
64
576
0.84
0.05
0.97
0.05
0.97
0.05
2.01
83.68
Conv 15
256
64
0.59
0.05
0.90
0.05
0.90
0.05
17.41
294.32
Conv 16
64
256
0.81
0.05
0.90
0.05
0.90
0.05
1.63
83.68
Conv 17
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.61
83.68
Conv
256
64
0.66
0.05
0.90
0.05
0.90
0.05
12.81
294.32
Conv 1 19
64
256
0.67
0.05
0.90
0.05
0.90
0.05
2.57
83.68
Conv 120
576
0.68
0.05
0.97
0.05
0.97
0.05
2.84
83.68
Conv 121
256
64
0.68
0.05
0.90
0.05
0.90
0.05
17.96
294.32
Conv 122
64
256
0.70
0.05
0.90
0.05
0.90
0.05
83.68
Conv 1 23
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.95
83.68
Conv 124
256
64
0.69
0.05
0.90
0.05
0.90
0.05
19.88
294.32
Conv
64
256
0.63
0.05
0.90
0.05
0.90
0.05
1.32
83.68
Conv 126
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.81
83.68
Conv 127
256
64
0.63
0.05
0.90
0.05
0.90
0.05
15.33
294.32
Conv 128
64
256
0.59
0.05
0.90
0.05
0.90
0.05
2.0
83,68
Conv 129
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.35
83.68
Conv 1 30
256
64
0.67
0.05
0.90
0.05
0.90
0.05
14.09
294.32
Conv 131
64
256
0.54
0.05
0.90
0.05
0.90
0.05
1.88
83.68
Conv 132
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.32
83,68
Conv 133
256
64
0.67
0.05
0.90
0.05
0.90
0.05
19.46
294.32
Conv 134
64
256
0.63
0.05
0.90
0.05
0.90
0.05
1.51
83.68
Conv 135
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.92
83.68
Conv136
256
64
0.65
0.05
0.90
0.05
0.90
0.05
18.41
294.32
Conv 137
64
256
0.66
0.05
0.90
0.05
0.90
0.05
1.55
83.68
Conv138
64
576
0.85
0.05
0.97
0.05
0.97
0.05
1.18
83.68
Conv 139
256
64
0.65
0.05
0.90
0.05
0.90
0.05
21.62
294.32
Conv 140
64
256
0.80
0.05
0.90
0.05
0.90
0.05
1.64
83.68
Conv 141
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.78
83.68
Conv
256
64
0.63
0.05
0.90
0.05
0.90
0.05
23.34
294.32
93

---

Under review as a conference paper at ICLR 2021
Conv 143
64
256
0.62
0.05
0.90
0.05
0.90
0.05
1.68
83.68
Conv 144
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.95
83.68
Conv 145
256
64
0.67
0.05
0.90
0.05
0.90
0.05
25.4
294.32
Conv 146
64
256
0.69
0.05
0.90
0.05
0.90
0.05
1.81
83.68
Conv 147
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.92
83.68
Conv 148
256
64
0.52
0,05
0.90
0.05
0.90
0.05
30.63
294.32
Conv 149
64
256
0.62
0.05
0.90
0.05
0.90
0.05
2.0
83.68
Conv 150
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.84
83.68
Conv15 151
256
64
0.56
0.05
0.90
0.05
0.90
0.05
33.48
294.32
Conv 152
64
256
0.63
0.05
0.90
0.05
0.90
0.05
1.26
83.68
Conv 153
576
0.86
0.05
0.97
0.05
0.97
0.05
0.69
83.68
Conv 154
256
64
0.51
0.05
0.90
0.05
0.90
0.05
37.11
294.32
Conv 155
64
256
0.66
0.05
0.90
0.05
0.90
0.05
1.72
83.68
Conv 156
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.65
83.68
Conv 157
256
64
0.62
0.05
0.90
0.05
0.90
0.05
28.49
294.32
Conv Comv158
64
256
0.58
0.05
0.90
0.05
0.90
0.05
1.7
83.68
Conv 1 159
64
576
0.86
0.05
0.97
0.05
0.97
0.05
0.82
83.68
Conv160
256
64
0.57
0.05
0.90
0.05
0.90
0.05
42.43
294.32
Conv 161
64
256
0.53
0.05
0.90
0.05
0.90
0.05
2.0
83.68
Conv 162
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.51
83.68
Conv 163
256
64
0.54
0.05
0.90
0.05
0.90
0.05
45.77
294.32
Passing rate
98. 96%
100.0%
100.0%
96.32%
Table 35: Cifar 100 orthogonal-
Layer
Number
dim
Gaussian
Mean Left
Mean.Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1 1
16
27
0.00
0.05
0.58
0.05
0.42
0.05
1011.26
26.3
Conv2
16
16
0.02
0.05
0.56
0.05
0.56
0.05
510.87
26.3
Conv3
16
144
0.05
0.05
0.68
0.05
0.68
0.05
91.21
26.3
Conv4
64
16
0.01
0.05
0.63
0.05
0.63
0.05
837.24
83.68
Conv5
16
64
0.02
0.05
0.63
0.05
0.63
0.05
132.43
26.3
Conv6
16
0.17
0.05
0.68
0.05
0.68
0.05
14.32
26.3
Conv7
64
16
0.19
0.05
0.63
0.05
0.63
0.05
213.05
83.68
Conv8
16
64
0.29
0.05
0.63
0.05
0.63
0.05
25.29
26.3
Conv9
16
144
0.37
0.05
0.68
0.05
0.68
0.05
11.98
26.3
Conv10 10
64
16
0.46
0.05
0.63
0.05
0.63
0.05
59.8
83.68
Conv 11
16
64
0.40
0.05
0.63
0.05
0.63
0.05
8.49
26.3
Conv 12
16
144
0.50
0.05
0.68
0.05
0.68
0.05
8.69
26.3
Conv
64
16
0.31
0.05
0.63
0.05
0.63
0.05
59.75
83.68
Conv 14
16
64
0.44
0.05
0.63
0.05
0.63
0.05
14.31
26.3
Conv 15
16
144
0.67
0.05
0.68
0.05
0.68
0.05
1.84
26.3
Conv 16
64
16
0.65
0.05
0.63
0.05
0.63
0,05
20.38
83.68
Conv 17
16
64
0.59
0.05
0.63
0.05
0.63
0.05
2.67
26.3
Conv 18
16
144
0.64
0.05
0.68
0.05
0.68
0.05
2.39
26.3
Conv 19
64
16
0.65
0.05
0.63
0.05
0.63
0.05
11.96
83.68
Conv20
16
64
0.68
0.05
0.63
0.05
0.63
0.05
2.11
26.3
Conv21
16
144
0.84
0.05
0.68
0.05
0.68
0.05
0.51
26.3
Conv22
64
16
0.86
0.05
0.63
0.05
0.63
0.05
6.61
83.68
Conv23
16
64
0.68
0.05
0.63
0.05
0.63
0.05
2.92
26.3
Cony24
16
144
0.83
0.05
0.68
0.05
0.68
0.05
0.98
26.3
Cony25
64
16
0.80
0.05
0.63
0.05
0.63
0.05
7.3
83.68
Conv26
16
64
0.83
0.05
0.63
0.05
0.63
0.05
1.15
26.3
Conv27
16
144
0.82
0.05
0.68
0.05
0.68
0.05
0.61
26.3
Conv28
64
16
0.88
0.05
0.63
0.05
0.63
0.05
3.66
83.68
94

---

Under review as a conference paper at ICLR 2021
Conv29
16
64
0.83
0.05
0.63
0.05
0.63
0.05
1.14
26.3
Conv30
16
144
0.87
0.05
0.68
0.05
0.68
0.05
0.49
26.3
Conv31
64
16
0.88
0.05
0.63
0.05
0.63
0.05
4.51
83.68
Conv32
16
64
0.64
0.05
0.63
0.05
0.63
0.05
4.88
26.3
Conv33
16
144
0.40
0.05
0.68
0.05
0.68
0.05
4.29
26.3
Conv34
64
16
0.80
0.05
0.63
0,05
0.63
0.05
11.26
83.68
Conv35
16
64
0.66
0.05
0.63
0.05
0.63
0.05
2.42
26.3
Conv36
16
144
0.83
0.05
0.68
0.05
0.68
0.05
0.4
26.3
Conv37
64
16
0.88
0.05
0.63
0.05
0.63
0.05
2.93
83.68
Conv38
16
64
0.67
0.05
0.63
0.05
0.63
0.05
3.22
26.3
Conv39
16
144
0.88
0.05
0.68
0.05
0.68
0.05
0.64
26.3
Conv40
64
16
0.86
0.05
0.63
0.05
0.63
0.05
4.46
83.68
Comv41
16
64
0.67
0.05
0.63
0.05
0.63
0.05
4.24
26.3
Conv42
16
144
0.67
0.05
0.68
0.05
0.68
0.05
2.75
26.3
Conv43
64
16
0.63
0.05
0.63
0.05
0.63
0.05
11.6
83.68
Conv44
16
64
0.70
0.05
0.63
0.05
0.63
0.05
2.71
26.3
Conv45
16
144
0.85
0.05
0.68
0.05
0.68
0.05
0.51
26.3
Conv46
64
16
0.89
0.05
0.63
0.05
0.63
0.05
5.22
83.68
Conv47
16
64
0.55
0.05
0.63
0.05
0.63
0.05
7.37
26.3
Conv48
16
144
0.63
0.05
0.68
0.05
0.68
0.05
3.1
26.3
Conv49
64
16
0,82
0.05
0.63
0.05
0.63
0,05
14.35
83.68
Conv50
16
64
0.53
0.05
0.63
0.05
0.63
0.05
7.6
26.3
Conv51
16
144
0.63
0.05
0.68
0.05
0.68
0.05
4.49
26.3
Conv52
64
16
0.81
0.05
0.63
0.05
0.63
0.05
14.16
83.68
Conv53
16
64
0.70
0.05
0.63
0.05
0.63
0.05
5.13
26.3
Conv54
16
144
0.68
0.05
0.68
0.05
0.68
0.05
1.98
26.3
Conv55
64
16
0.69
0.05
0.63
0.05
0.63
0.05
10.12
83.68
Conv56
32
64
0.47
0.05
0.67
0.05
0.67
0.05
24.35
46.19
Conv57
32
288
0.53
0.05
0.83
0.05
0.83
0.05
7.11
46.19
Conv58
128
32
0.39
0.05
0.74
0.05
0.74
0.05
130.03
155.4
Conv59
32
128
0.64
0.05
0.74
0.05
0.74
0.05
4.38
46.19
Conv60
32
288
0.53
0.05
0.83
0.05
0.83
0.05
5,98
46.19
Conv61
128
32
0.66
0.05
0.74
0.05
0.74
0.05
46.94
155.4
Conv62
32
128
0.65
0.05
0.74
0.05
0.74
0.05
3.71
46.19
Conv63
32
288
0.68
0.05
0.83
0.05
0.83
0.05
1.96
46.19
Conv64
128
32
0.67
0.05
0.74
0.05
0.74
0.05
22.74
155.4
Conv65
32
128
0.68
0.05
0.74
0.05
0.74
0.05
4.41
46.19
Conv66
32
288
0.65
0.05
0.83
0.05
0.83
0.05
4.22
46.19
Conv67
128
32
0.57
0.05
0.74
0.05
0.74
0.05
31.38
155.4
Conv68
32
128
0.81
0.05
0.74
0.05
0.74
0.05
3.85
46.19
Conv69
32
288
0.60
0.05
0.83
0.05
0.83
0.05
3.02
46.19
Conv70
128
32
0.68
0.05
0.74
0.05
0.74
0.05
21.04
155.4
Conv71
32
128
0.44
0.05
0.74
0.05
0.74
0.05
14.29
46.19
Conv72
32
288
0.65
0.05
0.83
0.05
0.83
0.05
5.11
46.19
Conv 73
128
32
0.51
0.05
0.74
0.05
0.74
0.05
58.28
155.4
Conv74
32
128
0.61
0.05
0.74
0.05
0.74
0.05
3.79
46.19
Conv75 75
32
288
0.84
0.05
0.83
0.05
0.83
0.05
3.0
46.19
Conv76
128
32
0.83
0.05
0.74
0.05
0.74
0.05
12.29
155.4
Conv77 77
32
128
0.85
0.05
0.74
0.05
0.74
0.05
2.66
46.19
Conv78
32
288
0.82
0.05
0.83
0.05
0.83
0.05
2.3
46.19
C9
128
32
0.85
0.05
0.74
0.05
0.74
0.05
13.49
155.4
Conv80
32
128
0.61
0.05
0.74
0.05
0.74
0.05
5.51
46.19
Conv81
32
288
0.84
0.05
0.83
0.05
0.83
0.05
2.35
46.19
Conv82
128
32
0.71
0.05
0.74
0.05
0.74
0.05
18.02
155.4
Conv83
32
128
0.58
0.05
0.74
0.05
0.74
0.05
5.69
46.19
Conv84
32
288
0.62
0.05
0.83
0.05
0.83
0.05
3.13
46.19
Conv85
128
32
0.70
0.05
0.74
0.05
0.74
0.05
26.25
155.4
Conv86
32
128
0.85
0.05
0.74
0.05
0.74
0.05
3.26
46.19
Conv87
32
288
0.86
0.05
0.83
0.05
0.83
0.05
1.88
46.19
95

---

Under review as a conference paper at ICLR 2021
Conv88
128
32
0.84
0.05
0.74
0.05
0.74
0.05
11.77
153.4
Conv89
32
128
0.81
0.05
0.74
0.05
0.74
0.05
3.42
46.19
Conv90
32
288
0.81
0.05
0.83
0.05
0.83
0.05
2.45
46.19
Conv91
128
32
0.81
0.05
0.74
0.05
0.74
0.05
16.74
153.4
Conv92
32
128
0.68
0.05
0.74
0.05
0.74
0.05
4.9
46.19
Conv93
32
288
0.68
0.05
0.83
0,05
0.83
0.05
1.94
46.19
Conv94
128
32
0.62
0.05
0.74
0.05
0.74
0.05
28.67
155.4
Conv95
32
128
0.39
0.05
0.74
0.05
0.74
0.05
7.35
46.19
Conv96
32
288
0.62
0.05
0.83
0.05
0.83
0.05
1.82
46.19
Conv97
128
32
0.67
0.05
0.74
0.05
0.74
0.05
29.33
155.4
Conv98
32
128
0.88
0.05
0.74
0.05
0.74
0.05
2.17
46.19
Conv99
32
288
0.88
0.05
0.83
0.05
0.83
0.05
1.18
46.19
Conv 100
128
32
0.87
0.05
0.74
0.05
0.74
0.05
10.3
155.4
Conv 101
32
128
0.71
0.05
0.74
0.05
0.74
0.05
5.73
46.19
Conv 102
32
288
0.67
0.05
0.83
0.05
0.83
0.05
3.25
46.19
Conv
128
32
0.68
0.05
0.74
0.05
0.74
0.05
24.74
155.4
Conv 104
32
128
0.81
0.05
0.74
0.05
0.74
0.05
3.97
46.19
Conv10s
32
288
0.68
0.05
0.83
0.05
0.83
0.05
2.33
46.19
Conv 106
128
32
0.84
0.05
0.74
0.05
0.74
0.05
23.88
155.4
Conv 107
32
128
0.66
0.05
0.74
0.05
0.74
0.05
2.85
46.19
Conv 108
32
288
0.85
0.05
0.83
0.05
0.83
0.05
0.93
46.19
Conv 109
128
32
0.86
0.05
0.74
0.05
0.74
0.05
17.92
155.4
Conv 10
64
128
0.54
0.05
0.82
0.05
0.82
0.05
9.86
83.68
Conv
64
576
0.69
0.05
0.97
0.05
0.97
0.05
3.44
83.68
Conv 12
256
64
0.53
0.05
0.90
0.05
0.90
0.05
72.59
294.32
Conv
64
256
0.81
0.05
0.90
0.05
0.90
0.05
2.17
83.68
Conv 1 14
64
576
0.80
0.05
0.97
0.05
0.97
0.05
4.2
83.68
Conv 115
256
64
0.60
0.05
0.90
0.05
0.90
0.05
27.87
294.32
Conv 16
64
256
0.82
0.05
0.90
0.05
0.90
0.05
2.3
83.68
Conv1
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.58
83.68
Conv 118
256
64
0.83
0.05
0.90
0.05
0.90
0.05
14.32
294.32
Conv 19
64
256
0.82
0.05
0.90
0.05
0.90
0.05
2.93
83.68
Conv 1onv120
64
576
0.70
0.05
0.97
0.05
0.97
0.05
1.69
83.68
Conv 121
256
64
0.80
0.05
0.90
0.05
0.90
0.05
12.83
294.32
Conv 122
64
256
0.68
0.05
0.90
0.05
0.90
0.05
2.67
83.68
Conv 1 123
64
576
0.65
0.05
0.97
0.05
0.97
0.05
2.46
83.68
Conv 124
256
64
0.70
0.05
0.90
0.05
0.90
0.05
16.62
294.32
Conv 125
64
256
0.70
0.05
0.90
0.05
0.90
0.05
2.21
83.68
Conv 126
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.82
83.68
Conv 127
256
64
0.67
0.05
0.90
0.05
0.90
0.05
17.12
294.32
Conv 128
64
256
0.61
0.05
0.90
0.05
0.90
0.05
2.58
83.68
Conv Monv129
64
576
0.70
0.05
0.97
0.05
0.97
0.05
1.92
83.68
Conv 1 130
256
64
0.58
0.05
0.90
0.05
0.90
0.05
18.45
294.32
Conv 131
64
256
0.63
0.05
0.90
0.05
0.90
0.05
2.37
83.68
Conv 132
64
576
0.81
0.05
0.97
0.05
0.97
0.05
1.24
83.68
Conv 133
256
64
0.70
0.05
0.90
0.05
0.90
0.05
21.78
294.32
Conv 1 34
64
256
0.39
0.05
0.90
0.05
0.90
0.05
3.23
83.68
Conv 135
64
576
0.71
0.05
0.97
0.05
0.97
0.05
1.68
83.68
Conv 136
256
64
0.60
0.05
0.90
0.05
0.90
0.05
25.14
294.32
Conv 137
64
256
0.59
0.05
0.90
0.05
0.90
0.05
1.62
83.68
Conv 138
64
576
0.84
0.05
0.97
0.05
0.97
0.05
0.9
83.68
Conv 139
256
64
0.58
0.05
0.90
0.05
0.90
0.05
23.89
294.32
Conv Monv140
64
256
0.43
0.05
0.90
0.05
0.90
0.05
1.81
83.68
Conv 141
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.11
83.68
Conv Linv142
256
64
0.62
0.05
0.90
0.05
0.90
0.05
23.72
294.32
Conv 143
64
256
0.61
0.05
0.90
0.05
0.90
0.05
1.68
83.68
Conv 144
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.68
83.68
Conv 145
256
64
0.62
0.05
0.90
0.05
0.90
0.05
26.18
294.32
Conv
64
256
0.62
0.05
0.90
0.05
0.90
0.05
1.48
83.68
96

---

Under review as a conference paper at ICLR 2021
Conv 147
64
576
0.83
0.05
0.97
0.05
0.97
0.05
0.64
83.68
Conv 148
256
64
0.59
0.05
0.90
0.05
0.90
0.05
26.75
294.32
Conv 149
64
256
0.66
0.05
0.90
0.05
0.90
0.05
1.97
83.68
Conv 1 50
64
576
0.84
0.05
0.97
0.05
0.97
0.05
1.09
83.68
Conv
256
64
0.59
0.05
0.90
0.05
0.90
0.05
27.51
294.32
Conv 152
64
256
0.48
0,05
0.90
0.05
0,90
0.05
2.36
83.68
Conv
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.58
83.68
Conv 1 154
256
64
0.57
0.05
0.90
0.05
0.90
0.05
35.04
294.32
Conv 155
64
256
0.58
0.05
0.90
0.05
0.90
0.05
1.68
83.68
Conv 156
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.6
83.68
Conv 157
256
64
0.53
0.05
0.90
0.05
0.90
0.05
34.74
294.32
Conv 158
64
256
0.48
0.05
0.90
0.05
0.90
0.05
2.37
83.68
Conv 159
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.8
83.68
Conv 160
256
64
0.53
0.05
0.90
0.05
0.90
0.05
37.4
294.32
Conv 161
64
256
0.47
0.05
0.90
0.05
0.90
0.05
1.81
83.68
Conv
64
576
0.85
0.05
0.97
0.05
0.97
0.05
0.58
83.68
Conv 163
256
64
0.49
0.05
0.90
0.05
0.90
0.05
36.85
294.32
Passing rate
97.55%
100.0%
100.0%
96.32%
P.6
DATASET
config:
Table 36: Cifarl0 WRN28-10
Layer
Number
dim
Gaussian
Mean Left
Meft_Left
Mean_Right
Sigma
p-value
c-value
p-value
c-value
p-value
e-value
t-value
c-value
Convl
16
27
0.46
0.05
0.58
0.05
0.58
0.05
27.8
26.3
Conv2
160
144
0.92
0.05
0.94
0.05
0.94
0.05
4.77
190.52
Conv3
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.09
190.52
Conv4
160
16
0.64
0.05
0.69
0.05
0.69
0.05
29.65
190.52
Conv5
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.5
190.52
Conv6
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.15
190.52
Conv7
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.05
190.52
Conv8
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.29
190.52
Conv9
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.08
190.52
Conv10 10
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.42
190.52
Conv 11
320
1440
0.94
0.05
1.00
0.05
1.00
0.05
1.28
362.72
Conv 12
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.09
362.72
Conv 13
320
160
0.90
0.05
0.99
0.05
0.99
0.05
3.15
362.72
Conv 14
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.12
362.72
Conv 15
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv 16
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.03
362.72
Conv17
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.11
362.72
Conv 18
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 19
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.26
362.72
Conv20
640
2880
0.94
0.05
1.00
0.05
1.00
0.05
3.11
699,96
Conv21
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.21
699,96
Conv22
640
320
0.90
0.05
1.00
0.05
1.00
0.05
1.93
699.96
Conv23
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.91
699.96
Conv24
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.21
699.96
Conv25
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.16
699.96
Conv26
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.37
699.96
Cony27
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.05
699.96
97

---

Under review as a conference paper at ICLR 2021
Conv28
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
1.32
699.96
Passing rate
100.0%
100.0%
100.0%
96.43%
Table 37: ImageNet WRN28-10
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
16
27
0.40
0.05
0.58
0.05
0.58
0.05
30.53
26.3
Conv2
160
144
0.91
0.05
0.94
0.05
0.94
0.05
6.08
190.52
Conv3
160
1440
0.95
0.05
1.00
0.05
1.00
0,05
0.13
190.52
Conv4
160
16
0.68
0.05
0.69
0.05
0.69
0.05
25.89
190.52
C
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.34
190.52
Conv6
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.19
190.52
Conv7
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.14
190.52
Conv8
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.28
190.52
Conv9
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.03
190.52
Conv 10
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.36
190.52
Conv 11
320
1440
0.95
0.05
1.00
0.05
1.00
0.05
1.54
362.72
Conv 12
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv1
320
160
0.89
0.05
0.99
0.05
0.99
0.05
3.39
362.72
Conv 14
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.11
362.72
Conv 15
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv 16
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.04
362.72
Conv 17
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.11
362.72
Conv18
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.03
362.72
Conv19 19
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.5
362.72
Conv20
640
2880
0.94
0.05
1.00
0.05
1.00
0.05
3.36
699.96
Conv21
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.17
699.96
Conv22
640
320
0.90
0.05
1.00
0.05
1.00
0.05
1.98
699.96
Conv23
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.74
699.96
Conv24
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.2
699.96
Conv25
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.23
699.96
Conv26
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.36
699.96
Conv27
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv28
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
1.3
699.96
Passing rate
100.0%
100.0%
100.0%
96.43%
Table 38: MNIST WRN28-10
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv 1
16
27
0.46
0.05
0.58
0.05
0.58
0.05
33.06
26.3
Conv2
160
144
0.92
0.05
0.94
0.05
0.94
0.05
5.57
190.52
Conv3
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.1
190.52
Conv4
160
16
0.64
0.05
0.69
0.05
0.69
0.05
30.62
190.52
5
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.5
190.52
Conv6
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.13
190.52
Conv7
160
1440
0.94
0.05
1.00
0.05
1.00
0.05
0.1
190.52
Conv8
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.28
190.52
Conv9
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.02
190.52
Conv10
160
1440
0.95
0.05
1.00
0.05
1.00
0.05
0.43
190.52
98

---

Under review as a conference paper at ICLR 2021
Conv 11
320
1440
0.94
0.05
1.00
0.05
1.00
0.05
1.56
362.72
Conv 12
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.08
362.72
Conv 13
320
160
0.90
0.05
0.99
0.05
0.99
0.05
3.91
362.72
Conv 14
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.17
362.72
Conv15
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.06
362.72
Conv 16
320
2880
0.94
0.05
1.00
0.05
1.00
0.05
0.03
362.72
Conv17
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.09
362.72
Conv 18
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.03
362.72
Conv19
320
2880
0.95
0.05
1.00
0.05
1.00
0.05
0.5
362.72
Conv20
640
2880
0.94
0.05
1.00
0.05
1,00
0.05
2.85
699.96
Conv21
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.14
699.96
Conv22
640
320
0.89
0.05
1.00
0.05
1.00
0.05
2.34
699.96
Comv23
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
0.68
699.96
Conv24
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.23
699.96
Conv25
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.19
699.96
Conv26
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.44
699.96
Conv27
640
5760
0.95
0.05
1.00
0.05
1.00
0.05
0.05
699.96
Conv28
640
5760
0.94
0.05
1.00
0.05
1.00
0.05
1.5
699.96
Passing rate
100.0%
100.0%
100.0%
96.43%
P.7 OTHER TASKS
P.7.1 SEGMENTATION
config:
Table 39: Cityscapes SegNet
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
e-value
t-value
c-value
Conv1
64
27
0.86
0.05
0.66
0.05
0.66
0.05
20.77
83.68
Conv2
64
576
0.88
0.05
0.97
0.05
0.97
0.05
0.51
83.68
Conv3
128
576
0.84
0.05
1.00
0.05
1.00
0.05
155,4
Conv4
128
1152
0.92
0.05
1.00
0.05
1.00
0.05
0.42
155.4
Conv5
256
1152
0.91
0.05
1.00
0.05
1.00
0.05
1.05
294.32
Conv6
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.48
294.32
Conv7
256
2304
0.94
0.05
1.00
0.05
1.00
0.05
0.44
294.32
Conv8
512
2304
0.94
0.05
1.00
0.05
1.00
0.05
0.96
565.75
Conv9
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.39
565.75
Conv 10
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.47
565.75
Conv1
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.48
565.75
Conv 12
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.38
565.75
Conv 13
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.45
565.75
Conv 14
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.36
565.75
Conv
512
4608
0.93
0.05
1.00
0.05
1,00
0.05
0.41
565.75
Conv 16
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.26
565.75
Conv 17
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.57
565.75
Conv 18
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.32
565.75
Conv19 19
256
4608
0.93
0.05
1.00
0.05
1.00
0.05
0.14
294.32
Conv20
256
2304
0.95
0.05
1.00
0.05
1.00
0.05
0.42
294.32
Conv21
256
2304
0.95
0.05
1.00
0.05
1.00
0.05
0.28
294.32
Cony22
128
2304
0.94
0.05
1.00
0.05
1.00
0.05
0.17
155.4
Conv23
128
1152
0.91
0.05
1.00
0.05
1.00
0.05
0.29
155,4
99

---

Under review as a conference paper at ICLR 2021
Conv24
64
1152
0.93
0.05
1.00
0.05
1.00
0.05
0.15
83.68
Conv25
64
576
0.92
0.05
0.97
0.05
0.97
0.05
0.31
83.68
Conv26
19
576
0.87
0.05
0.85
0.05
0.85
0.05
0.9
30.14
Passing rate
100.0%
100.0%
100.0%
100.0%
Table 40: Cityscapes PSPNet
Layer
Number
dim
Gaussian
Mean
Mean Right
Sigma
p-value
e-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.38
0.05
0.66
0.05
0.66
0.05
286.98
83.68
Conv2
64
576
0.49
0.05
0.97
0.05
0.97
0.05
12.81
83.68
Conv3
128
576
0.70
0.05
1.00
0.05
1.00
0.05
1.47
155.4
Conv4
64
128
0.85
0.05
0.82
0.05
0.82
0.05
4.77
83.68
Conv5
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.95
83.68
Conv6
256
64
0.85
0.05
0.90
0.05
0.90
0.05
15.16
294.32
Conv7
256
128
0.60
0.05
0.96
0.05
0.96
0.05
17.95
294.32
Conv8
64
256
0.91
0.05
0.90
0.05
0.90
0.05
1.78
83.68
Conv9
64
576
0.87
0.05
0.97
0.05
0.97
0.05
0.25
83.68
Conv 10
256
64
0.88
0.05
0.90
0.05
0.90
0.05
10.76
294.32
Conv11
64
256
0.91
0.05
0.90
0.05
0.90
0.05
2.01
83.68
Conv 12
64
576
0.91
0.05
0.97
0.05
0.97
0.05
0.35
83.68
Conv 13
256
64
0.89
0.05
0.90
0.05
0.90
0.05
10.08
294.32
Conv 14
128
256
0.86
0.05
0.96
0.05
0.96
0.05
1.76
155.4
Conv 15
128
1152
0.92
0.05
1.00
0.05
1.00
0.05
0.07
155.4
Conv16
512
128
0.87
0.05
0.99
0.05
0.99
0.05
21.04
565.75
Conv1 17
512
256
0.85
0.05
1.00
0.05
1.00
0.05
11.71
565.75
Conv 18
128
512
0.91
0.05
0.99
0.05
0.99
0.05
0.45
155.4
Conv 19
128
1152
0.89
0.05
1.00
0.05
1.00
0.05
1.22
155,4
Conv20
512
128
0.90
0.05
0.99
0.05
0.99
0.05
12.48
565.75
Conv21
128
512
0.93
0.05
0.99
0.05
0.99
0.05
0.22
155.4
Conv22
128
1152
0.90
0.05
1.00
0.05
1.00
0.05
0.17
155.4
Conv23
512
128
0.87
0.05
0.99
0.05
0.99
0.05
8.64
565.75
Conv24
128
512
0.91
0.05
0.99
0.05
0.99
0.05
0.12
155.4
Conv25
128
1152
0.92
0.05
1.00
0.05
1.00
0.05
0.11
155.4
Conv26
512
128
0.89
0.05
0.99
0.05
0.99
0.05
10.47
565.75
Conv27
256
512
0.87
0.05
1,00
0.05
1.00
0.05
0.65
294.32
Conv28
256
2304
0.92
0.05
1,00
0.05
1.00
0.05
0.2
294.32
Conv29
1024
256
0.89
0.05
1,00
0.05
1.00
0.05
16.07
1099.56
Conv30
1024
512
0.84
0.05
1,00
0.05
1.00
0.05
10.07
1099.56
Conv31
256
1024
0.90
0.05
1,00
0.05
1.00
0.05
0.35
294.32
Conv32
256
2304
0.94
0.05
1.00
0.05
1.00
0.05
0.24
294.32
Conv33
1024
256
0.89
0.05
1,00
0.05
1.00
0.05
7.69
1099.56
Conv34
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.26
294.32
Conv355
256
2304
0.93
0.05
1,00
0.05
1.00
0.05
0.35
294.32
Conv36
1024
256
0.91
0.05
1,00
0.05
1.00
0.05
7.32
1099.56
Conv37
256
1024
0.86
0.05
1,00
0.05
1.00
0.05
0.3
294.32
Conv38
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.17
294.32
Conv39
1024
256
0.91
0.05
1,00
0.05
1.00
0.05
6.23
1099.56
Conv40
256
1024
0.91
0.05
1,00
0.05
1.00
0.05
0.18
294.32
Conv41
256
2304
0.88
0.05
1,00
0.05
1.00
0.05
0.06
294.32
Conv42
1024
256
0.81
0.05
1.00
0.05
1.00
0.05
11.36
1099.56
Conv43
256
1024
0.92
0.05
1.00
0.05
1.00
0.05
0.15
294.32
Conv44
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.07
294.32
Conv45
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
7.5
1099.56
Conv46
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.17
294.32
100

---

Under review as a conference paper at ICLR 2021
Conv47
256
2304
0.94
0.05
1.00
0.05
1.00
0.05
0.1
294.32
Conv48
1024
256
0.92
0.05
1.00
0.05
1.00
0.05
5.1
1099.56
Conv49
256
1024
0.88
0.05
1.00
0.05
1.00
0.05
0.17
294.32
Conv50
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.09
294.32
Conv51
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
7.45
1099.56
Conv52
256
1024
0.87
0.05
1.00
0.05
1.00
0.05
0.15
294.32
Conv53
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.13
294.32
Conv54
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
3.5
1099.56
Conv55
256
1024
0.93
0.05
1.00
0.05
1.00
0.05
0.1
294.32
Conv56
256
2304
0.93
0.05
1,00
0.05
1.00
0.05
0.11
294.32
Conv57
1024
256
0.93
0.05
1,00
0.05
1.00
0.05
3.86
1099.56
Conv58
256
1024
0.91
0.05
1,00
0.05
1.00
0.05
0.18
294.32
Conv59
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.18
294.32
Conv60
1024
256
0.92
0.05
1,00
0.05
1.00
0.05
6.96
1099.56
Conv61
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.18
294.32
Conv62
256
2304
0.93
0.05
1,00
0.05
1.00
0.05
0.16
294.32
Conv63
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
7.38
1099.56
Conv64
256
1024
0.91
0.05
1,00
0.05
1.00
0.05
0.23
294.32
Conv65
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.12
294.32
Conv66
1024
256
0.92
0.05
1.00
0.05
1.00
0.05
5.27
1099.56
Conv67
256
1024
0.85
0.05
1.00
0.05
1.00
0.05
0.2
294.32
Conv68
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.18
294.32
Conv69
1024
256
0.90
0.05
1,00
0.05
1.00
0.05
4.64
1099.56
Conv70 70
256
1024
0.92
0.05
1,00
0.05
1.00
0.05
0.1
294.32
Conv71
256
2304
0.94
0.05
1,00
0.05
1.00
0.05
0.16
294.32
Conv72 72
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
4.37
1099.56
Conv73
256
1024
0.91
0.05
1,00
0.05
1.00
0.05
0.1
294.32
Conv 74
256
2304
0.94
0.05
1,00
0.05
1.00
0.05
0.16
294.32
Conv75
1024
256
0.92
0.05
1,00
0.05
1.00
0.05
6.24
1099.56
Conv76 76
256
1024
0.90
0.05
1,00
0.05
1.00
0.05
0.11
294.32
Conv77
256
2304
0.92
0.05
1,00
0.05
1.00
0.05
0.15
294.32
Conv 78
1024
256
0.92
0.05
1.00
0.05
1.00
0.05
3.44
1099.56
Conv79
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.1
294.32
Conv80
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.12
294.32
Conv81
1024
256
0.91
0.05
1,00
0.05
1.00
0.05
4.25
1099.56
Conv82
256
1024
0.90
0.05
1,00
0.05
1.00
0.05
0.09
294.32
Conv83
256
2304
0.92
0.05
1,00
0.05
1.00
0.05
0.15
294.32
Conv84
1024
256
0.89
0.05
1,00
0.05
1.00
0.05
5.13
1099.56
Conv85
256
1024
0.90
0.05
1,00
0.05
1.00
0.05
0.09
294.32
Conv86
256
2304
0.93
0.05
1,00
0.05
1.00
0.05
0.17
294.32
Conv87
1024
256
0.88
0.05
1,00
0.05
1.00
0.05
4.76
1099.56
Conv88
256
1024
0.90
0.05
1,00
0.05
1.00
0.05
0.09
294.32
Conv89
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
0.09
294.32
Conv90
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
3.86
1099.56
Conv91
256
1024
0.92
0.05
1.00
0.05
1.00
0.05
0.07
294.32
Conv92
256
2304
0.92
0.05
1,00
0.05
1.00
0.05
0.03
294.32
Conv93
1024
256
0.92
0.05
1,00
0.05
1.00
0.05
4.57
1099.56
Conv94
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.12
294.32
Conv95
256
2304
0.93
0.05
1,00
0.05
1.00
0.05
0.07
294.32
Conv96
1024
256
0.89
0.05
1,00
0.05
1.00
0.05
6.76
1099.56
Conv97
512
1024
0.90
0.05
1,00
0.05
1.00
0.05
0.08
565.75
Conv98
512
4608
0.94
0.05
1,00
0.05
1.00
0.05
0.02
565.75
Conv99
2048
512
0.89
0.05
1,00
0.05
1.00
0.05
1.67
2154.4
Conv100 100
2048
1024
0.69
0.05
1,00
0.05
1.00
0.05
1.49
2154.4
Conv101
512
2048
0.88
0.05
1.00
0.05
1.00
0.05
0.05
565.75
Conv 102
512
4608
0.90
0.05
1.00
0.05
1.00
0.05
0.02
565.75
Conv 103
2048
512
0.90
0.05
1.00
0.05
1.00
0.05
1.95
2154.4
Conv 104
512
2048
0.85
0.05
1.00
0.05
1.00
0.05
0.07
565.75
Conv105
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.01
565.75
101

---

Under review as a conference paper at ICLR 2021
Conv100
2048
512
0.90
0.05
1.00
0.05
1.00
0.05
1.9
2154.4
Conv 107
512
2048
0.96
0.05
1.00
0.05
1.00
0.05
0.01
565.75
Conv10 108
512
2048
0.96
0.05
1.00
0.05
1.00
0.05
0.0
565.75
Conv 1 109
512
2048
0.96
0.05
1.00
0.05
1.00
0.05
0.0
565.75
Conv1 10
512
2048
0.96
0.05
1.00
0.05
1.00
0.05
0.0
565.75
Conv 111
512
36864
0.95
0.05
1.00
0.05
1.00
0.05
0.0
565.75
Conv 1 12
19
512
0.88
0.05
0.84
0.05
0.84
0.05
0.99
30.14
Conv 1 13
512
9216
0.96
0.05
1.00
0.05
1.00
0.05
0.0
565.75
Conv
19
512
0.87
0.05
0.84
0.05
0.84
0.05
1.06
30.14
Passing rate
100.0%
100.0%
100.0%
99.1 19.12%%
P.7.2 FASTER RCNN
config:
Table 41: COCO ResNet101 max epoch=6
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
e-value
t-value
c-value
Conv1
64
64
0.57
0.05
0.74
0.05
0.74
0.05
23.34
83.68
Conv2
64
576
0.69
0.05
0.97
0.05
0.97
0.05
5.71
83.68
Conv3
256
64
0,80
0.05
0.90
0.05
0.90
0.05
28.22
294.32
Conv4
256
64
0,46
0.05
0.90
0.05
0.90
0.05
77.88
294.32
Conv5
64
256
0.86
0.05
0.90
0.05
0.90
0.05
3.87
83.68
Conv6
64
576
0.67
0.05
0.97
0.05
0.97
0.05
2.11
83.68
Conv7
256
64
0.85
0.05
0.90
0.05
0.90
0.05
22.48
294.32
Conv8
64
256
0.85
0.05
0.90
0.05
0.90
0.05
2.53
83.68
Conv9
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.13
83.68
Conv 10
256
64
0.87
0.05
0.90
0.05
0.90
0.05
30.9
294.32
Conv
128
256
0.62
0.05
0.96
0.05
0.96
0,05
3,99
155.4
Conv 12
128
1152
0.83
0.05
1.00
0.05
1.00
0.05
2.08
155.4
Conv 13
512
128
0.66
0.05
0.99
0.05
0.99
0.05
27.85
565.75
Conv 14
512
256
0.55
0.05
1.00
0.05
1.00
0.05
18.51
565.75
Conv 15
128
512
0.90
0.05
0.99
0.05
0.99
0.05
0.52
155.4
Conv 16
128
1152
0.86
0.05
1.00
0.05
1.00
0.05
0.36
155.4
Conv 17
512
128
0.67
0.05
0.99
0.05
0.99
0,05
15.55
565.75
Conv 18
128
512
0.87
0.05
0.99
0.05
0.99
0.05
0.86
155.4
Conv19 19
128
1152
0.91
0.05
1.00
0.05
1.00
0.05
0.35
155.4
Conv20
512
128
0.71
0.05
0.99
0.05
0.99
0.05
16.78
565.75
Conv21
128
512
0.86
0.05
0.99
0.05
0.99
0.05
0.98
155.4
Conv22
128
1152
0.89
0.05
1.00
0.05
1.00
0.05
0.27
155.4
Conv23
512
128
0.83
0.05
0.99
0.05
0.99
0.05
27.72
565.75
Conv24
256
512
0.87
0.05
1.00
0.05
1.00
0.05
0.85
294.32
Conv25
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.49
294.32
Conv26
1024
256
0.86
0.05
1.00
0.05
1.00
0.05
12.09
1099.56
Conv27
1024
512
0.69
0.05
1.00
0.05
1.00
0.05
12.68
1099.56
Conv28
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.4
294.32
Conv29
256
2304
0.91
0.05
1.00
0.05
1,00
0.05
0.63
294.32
Conv30
1024
256
0.86
0.05
1.00
0.05
1.00
0.05
6.72
1099.56
Conv31
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.27
294.32
Conv32
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.25
294.32
Conv33
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
4.71
1099.56
Conv34
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.39
294.32
Conv35
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.16
294.32
102

---

Under review as a conference paper at ICLR 2021
Conv36
1024
256
0.87
0.05
1.00
0.05
1.00
0.05
9.99
1099.56
Conv37
256
1024
0.92
0.05
1.00
0.05
1.00
0.05
0.26
294.32
Conv38
256
2304
0.93
0.05
1.00
0.05
1.00
0,05
0.2
294.32
Conv39
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
4.41
1099.56
Conv40
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.23
294.32
Conv41
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.24
294.32
Conv42
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
4.76
1099.56
Conv43
256
1024
0.85
0.05
1.00
0.05
1.00
0.05
0.33
294.32
Conv44
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.31
294.32
Conv45
1024
256
0.89
0.05
1.00
0.05
1,00
0.05
7.7
1099.56
Conv46
256
1024
0.89
0.05
1.00
0.05
1.00
0.05
0.25
294.32
Conv47
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.27
294.32
Conv48
1024
256
0.86
0.05
1.00
0.05
1.00
0.05
13.24
1099.56
Conv49
256
1024
0.82
0.05
1.00
0.05
1.00
0,05
0.38
294.32
Conv50
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.21
294.32
Conv51
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
4.57
1099.56
Conv52
256
1024
0.88
0.05
1.00
0.05
1.00
0.05
0.2
294.32
Conv53
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
0.26
294.32
Conv54
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
8.23
1099.56
Conv55
256
1024
0.89
0.05
1.00
0.05
1.00
0.05
0.28
294.32
Conv56
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.37
294.32
Conv57
1024
256
0.85
0.05
1.00
0.05
1.00
0.05
8.72
1099.56
Conv58
256
1024
0.88
0.05
1.00
0.05
1,00
0.05
0.28
294.32
Conv59
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
0.24
294.32
Conv60
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
5.31
1099.56
Conv61
256
1024
0.89
0.05
1.00
0.05
1.00
0.05
0.25
294.32
Conv62
256
2304
0.91
0.05
1.00
0.05
1.00
0,05
0.38
294.32
Conv63
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
5.22
1099.56
Conv64
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.21
294.32
Conv65
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.2
294.32
Conv66
1024
256
0.90
0.05
1.00
0.05
1.00
0,05
2.98
1099.56
Conv67
256
1024
0.92
0.05
1.00
0.05
1.00
0.05
0.26
294.32
Conv68
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.27
294.32
Conv69
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
4.05
1099.56
Conv70
256
1024
0.87
0.05
1.00
0.05
1.00
0.05
0.28
294.32
Conv71
256
2304
0.91
0.05
1.00
0.05
1,00
0.05
0.33
294.32
Conv72
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
8.2
1099.56
Conv
256
1024
0.85
0.05
1.00
0.05
1.00
0,05
0.31
294.32
Conv74 74
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.19
294.32
Conv75
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
5.23
1099.56
Conv76
256
1024
0.86
0.05
1.00
0.05
1.00
0.05
0.3
294.32
Conv77
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.16
294.32
Conv78 78
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
2.86
1099.56
Conv79
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.3
294.32
Conv80
256
2304
0.87
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv81
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
2.53
1099.56
Conv82
256
1024
0.89
0.05
1.00
0.05
1.00
0.05
0.2
294.32
Conv83
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.43
294.32
Conv84
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
5.84
1099.56
Conv85
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.28
294.32
Conv86
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.35
294.32
Conv87
1024
256
0.85
0.05
1.00
0.05
1.00
0.05
3.08
1099.56
Conv88
256
1024
0.87
0.05
1.00
0.05
1.00
0.05
0.25
294.32
Conv89
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
0.36
294.32
Conv90
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
3.51
1099.56
Conv91
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.26
294.32
Conv92
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.19
294.32
Conv93
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
3.62
1099.56
Conv94
512
1024
0.82
0.05
1.00
0.05
1.00
0.05
0.89
565.75
103

---

Under review as a conference paper at ICLR 2021
Conv95
512
4608
0,86
0.05
1.00
0.05
1.00
0.05
0.41
565.75
Conv96
2048
512
0.82
0.05
1.00
0.05
1.00
0.05
4.89
2154.4
Conv97
2048
1024
0.83
0.05
1.00
0.05
1.00
0,05
2.53
2154.4
Conv98
512
2048
0.83
0.05
1.00
0.05
1.00
0.05
0.28
565.75
Conv99
512
4608
0.91
0.05
1.00
0.05
1.00
0.05
0.26
565.75
Conv 100
2048
512
0.88
0.05
1.00
0.05
1.00
0.05
4.5
2154.4
Conv101
512
2048
0.84
0.05
1.00
0.05
1.00
0.05
0.31
565.75
Conv 102
512
4608
0.90
0.05
1.00
0.05
1.00
0.05
0.19
565.75
Conv 103
2048
512
0.87
0.05
1.00
0.05
1.00
0.05
3.81
2154.4
Conv 104
512
9216
0.93
0.05
1.00
0.05
1,00
0.05
0.06
565.75
Conv105
147
0.44
0.05
0.83
0.05
0.83
0.05
90.36
83.68
Passing rate
100.0%
100.0%
100.0%
99.05%
Table 42: PASCAL VOC 2007 ResNet101 bz=24
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
64
64
0.57
0.05
0.74
0.05
0.74
0.05
23.34
83.68
Conv2
64
576
0.69
0.05
0.97
0.05
0.97
0.05
5.71
83.68
Conv3
256
64
0.80
0.05
0.90
0.05
0.90
0.05
28.22
294.32
Conv4
256
64
0.46
0.05
0.90
0.05
0.90
0.05
77.88
294.32
C
64
256
0.86
0.05
0.90
0.05
0.90
0.05
3.87
83.68
Conv6
64
576
0.67
0.05
0.97
0.05
0.97
0.05
2.11
83.68
Conv7
256
64
0.85
0.05
0.90
0.05
0.90
0.05
22.48
294.32
Conv8
64
256
0.85
0.05
0.90
0.05
0.90
0.05
2.53
83.68
Conv9
64
576
0.90
0.05
0.97
0.05
0.97
0.05
0.13
83.68
Conv 10
256
64
0.87
0.05
0.90
0.05
0.90
0.05
30.9
294.32
Conv 11
128
256
0.55
0.05
0.96
0.05
0.96
0.05
4.34
155.4
Conv1 12
128
1152
0.68
0.05
1.00
0.05
1.00
0.05
2.69
155.4
Conv
512
128
0.70
0.05
0.99
0.05
0.99
0.05
28.4
565.75
Conv 14
512
256
0.56
0.05
1.00
0.05
1.00
0.05
14.13
565.75
Conv
128
512
0.89
0.05
0.99
0.05
0.99
0.05
0.6
155,4
Conv 16
128
1152
0.87
0.05
1.00
0.05
1.00
0.05
0,45
155.4
Conv 17
512
128
0.65
0.05
0.99
0.05
0.99
0.05
17.14
565.75
Conv 18
128
512
0.86
0.05
0.99
0.05
0.99
0.05
0.95
155.4
Conv19 19
128
1152
0.89
0.05
1.00
0.05
1.00
0.05
0.39
155.4
Conv20
512
128
0.81
0.05
0.99
0.05
0.99
0.05
16.68
565.75
Conv21
128
512
0.85
0.05
0.99
0.05
0.99
0.05
1.02
155.4
Conv22
128
1152
0.88
0.05
1.00
0.05
1.00
0.05
0.44
155,4
Conv23
512
128
0.84
0.05
0.99
0.05
0.99
0.05
27.01
565.75
Conv24
256
512
0.84
0.05
1.00
0.05
1.00
0.05
0.87
294.32
Conv25
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.5
294.32
Conv26
1024
256
0.84
0.05
1.00
0.05
1.00
0.05
12.16
1099.56
Conv27
1024
512
0.67
0.05
1.00
0.05
1.00
0.05
15.39
1099.56
Conv28
256
1024
0.88
0.05
1.00
0.05
1.00
0.05
0.53
294.32
Conv29
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.7
294.32
Conv30
1024
256
0.86
0.05
1.00
0.05
1.00
0.05
8.23
1099.56
Conv31
256
1024
0.92
0.05
1.00
0.05
1.00
0.05
0.37
294.32
Conv32
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.31
294.32
Conv33
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
5.96
1099.56
Conv34
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.44
294.32
Conv35
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.19
294.32
Conv36
1024
256
0.85
0.05
1.00
0.05
1.00
0.05
13.91
1099.56
Conv37
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.33
294.32
Conv38
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.22
294.32
104

---

Under review as a conference paper at ICLR 2021
Conv39
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
4.78
1099.56
Conv40
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.23
294.32
Conv41
256
2304
0.93
0.05
1.00
0.05
1.00
0,05
0.19
294.32
Conv42
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
4.32
1099.56
Conv43
256
1024
0.86
0.05
1.00
0.05
1.00
0.05
0.37
294.32
Conv44
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.27
294.32
Conv45
1024
256
0.90
0.05
1.00
0.05
1.00
0.05
6.28
1099.56
Conv46
256
1024
0.89
0.05
1.00
0.05
1.00
0.05
0.33
294.32
Conv47
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.27
294.32
Conv48
1024
256
0.83
0.05
1.00
0.05
1,00
0.05
12.8
1099.56
Conv49
256
1024
0.81
0.05
1.00
0.05
1.00
0.05
0.42
294.32
Conv50
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.17
294.32
Conv51
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
5.84
1099.56
Conv52
256
1024
0.89
0.05
1.00
0.05
1.00
0,05
0.26
294.32
Conv53
256
2304
0.83
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv54
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
7.67
1099.56
Conv55
256
1024
0.88
0.05
1.00
0.05
1.00
0.05
0.33
294.32
Conv56
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.3
294.32
Conv57
1024
256
0.87
0.05
1.00
0.05
1.00
0.05
7.21
1099.56
Conv58
256
1024
0.86
0.05
1.00
0.05
1.00
0.05
0.29
294.32
Conv59
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.16
294.32
Conv60
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
4.7
1099.56
Conv61
256
1024
0.90
0.05
1.00
0.05
1,00
0.05
0.25
294.32
Conv62
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.26
294.32
Conv63
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
4.09
1099.56
Conv64
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.28
294.32
Conv65
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.16
294.32
Conv66
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
3.05
1099.56
Conv67
256
1024
0.91
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv68
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.16
294.32
Conv69
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
3.24
1099.56
Conv 70
256
1024
0.86
0.05
1.00
0.05
1.00
0.05
0.21
294.32
Conv71
256
2304
0.93
0.05
1.00
0.05
1.00
0.05
0.17
294.32
Conv72 72
1024
256
0.87
0.05
1.00
0.05
1.00
0.05
7.3
1099.56
Conv73
256
1024
0.86
0.05
1.00
0.05
1.00
0.05
0.42
294.32
Conv74
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.11
294.32
Conv75
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
6.51
1099.56
Conv76
256
1024
0.82
0.05
1.00
0.05
1.00
0,05
0.3
294.32
Conv77 77
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
0.09
294.32
C8ny78
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
3.63
1099.56
Conv79
256
1024
0.87
0.05
1.00
0.05
1.00
0.05
0.24
294.32
Conv80
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
0.09
294.32
Conv81
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
2.6
1099.56
Conv82
256
1024
0.89
0.05
1.00
0.05
1.00
0.05
0.17
294.32
Conv83
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.19
294.32
Conv84
1024
256
0.88
0.05
1.00
0.05
1.00
0.05
4.77
1099.56
Conv85
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.43
294.32
Conv86
256
2304
0.92
0.05
1.00
0.05
1.00
0.05
0.14
294.32
Conv87
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
4.01
1099.56
Conv88
256
1024
0.88
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv89
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.14
294.32
Conv90
1024
256
0.91
0.05
1.00
0.05
1.00
0.05
2.78
1099.56
Conv91
256
1024
0.90
0.05
1.00
0.05
1.00
0.05
0.34
294.32
Conv92
256
2304
0.87
0.05
1.00
0.05
1.00
0.05
0.1
294.32
Conv93
1024
256
0.89
0.05
1.00
0.05
1.00
0.05
2.44
1099.56
Conv94
512
1024
0.82
0.05
1.00
0.05
1.00
0.05
0.29
565.75
Conv95
512
4608
0.84
0.05
1.00
0.05
1.00
0.05
0.11
565.75
Conv96
2048
512
0.66
0.05
1.00
0.05
1.00
0.05
2,4
2154.4
Conv97
2048
1024
0.53
0.05
1.00
0.05
1.00
0.05
2.25
2154.4
105

---

Under review as a conference paper at ICLR 2021
Conv98
512
2048
0.81
0.05
1.00
0.05
1.00
0.05
0.12
565.75
Conv99
512
4608
0.89
0.05
1.00
0.05
1.00
0.05
0.07
565.75
Conv100
2048
512
0.85
0.05
1.00
0.05
1.00
0,05
2.43
2154.4
Conv101 101
512
2048
0.64
0.05
1.00
0.05
1.00
0.05
0.2
565.75
Cony 102
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.04
565.75
Conv 103
2048
512
0.90
0.05
1.00
0.05
1.00
0.05
2,01
2154.4
Conv104
512
9216
0.95
0.05
1.00
0.05
1.00
0.05
0.0
565.75
Conv 105
64
147
0.44
0.05
0.83
0.05
0.83
0.05
90.36
83.68
Passing rate
100.0%
100.0%
100.0%
99.05% %
Table 43: Visual Genome VGG16
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
e-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.50
0.05
0.66
0.05
0.66
0.05
147.34
83.68
Conv2
64
576
0.85
0.05
0.97
0.05
0.97
0.05
3.2
83.68
Conv3
128
576
0.68
0.05
1.00
0.05
1.00
0.05
6.85
155,4
Conv4
128
1152
0.85
0.05
1.00
0.05
1.00
0.05
0.91
155.4
C
256
1152
0.71
0.05
1.00
0.05
1.00
0.05
0.92
294.32
Conv6
256
2304
0.84
0.05
1.00
0.05
1.00
0.05
0.33
294.32
Conv7
256
2304
0.81
0.05
1.00
0.05
1.00
0.05
0.27
294.32
Conv8
512
2304
0.83
0.05
1,00
0.05
1.00
0.05
0.33
565.75
Conv9
512
4608
0.88
0.05
1.00
0.05
1.00
0.05
0.11
565.75
Conv 10
512
4608
0.91
0.05
1.00
0.05
1.00
0.05
0.12
565.75
Conv
512
4608
0.87
0.05
1,00
0.05
1.00
0.05
0.1
565.75
Conv 12
512
4608
0.88
0.05
1,00
0.05
1.00
0.05
0.17
565.75
Conv 13
512
4608
0.88
0.05
1.00
0.05
1.00
0.05
0.6
565.75
Conv 14
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.06
565.75
Conv15 15
4096
25088
0.95
0.05
1.00
0.05
1.00
0.05
0.04
4246.0
Conv16
4096
4096
0.95
0.05
1.00
0.05
1.00
0.05
0.15
4246.0
Passing rate
100.0%
100.0%
100.0%
93.75%
P.7.3 IMAGE MATTING
config:
https://gthuD..c///omm/////
MMatinggMtttnggggg PyTorch PerttiiingPyyoorhhh
httpee//giihubb.com//orr//
Table 44: AlphaGAN matting
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
e-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
64
36
0.10
0.05
0.68
0.05
0.68
0.05
446.15
83.68
Conv2
64
576
0.60
0.05
0.97
0.05
0.97
0.05
2.63
83,68
Conv3
128
576
0.63
0.05
1.00
0.05
1.00
0.05
24.49
155,4
Conv4
128
1152
0.70
0.05
1.00
0.05
1.00
0.05
2.71
155.4
Conv5
256
1152
0.53
0.05
1.00
0.05
1.00
0.05
5.23
294.32
Conv6
256
2304
0.85
0.05
1.00
0.05
1.00
0.05
0.81
294.32
Conv7
256
2304
0.66
0.05
1.00
0.05
1.00
0.05
0.35
294.32
Conv8
512
2304
0.67
0.05
1.00
0.05
1.00
0.05
0.78
565.75
Conv9
512
4608
0.84
0.05
1.00
0.05
1.00
0.05
0.81
565.75
106

---

Under review as a conference paper at ICLR 2021
Conv 10
512
4608
0.87
0.05
1.00
0.05
1.00
0.05
0.87
565.75
Conv 11
512
4608
0.70
0.05
1.00
0.05
1.00
0.05
0.22
565.75
Conv 12
512
4608
0.81
0.05
1,00
0.05
1.00
0.05
0.18
565.75
Conv 13
512
4608
0.87
0.05
1.00
0.05
1.00
0.05
0.45
565.75
Conv 14
4096
25088
0.92
0.05
1.00
0.05
1.00
0.05
4.81
4246.0
Conv 15
512
4096
0.94
0.05
1.00
0.05
1.00
0.05
0.04
565.75
Conv16
512
12800
0.93
0.05
1.00
0.05
1.00
0.05
0.09
565.75
Conv 17
256
12800
0.80
0.05
1.00
0.05
1.00
0.05
0.41
294.32
Conv 18
128
6400
0.70
0.05
1.00
0.05
1.00
0.05
0.87
155.4
Conv19 19
64
3200
0.58
0.05
1.00
0.05
1.00
0.05
0.87
83.68
Conv20
1600
0.46
0.05
1.00
0.05
1.00
0.05
2.43
83.68
Passing rate
100.0%
100.0%
100.0%
95.00%
Table 45: Deep image matting
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
e-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
64
196
0.22
0.05
0.87
0.05
0.87
0.05
25.66
83.68
Conv2
64
64
0.29
0.05
0.74
0.05
0.74
0.05
24.7
83.68
Conv3
64
576
0.54
0.05
0.97
0.05
0.97
0.05
5.84
83.68
Conv4
256
64
0.53
0.05
0.90
0.05
0.90
0.05
46.45
294.32
Conv5
256
0.31
0.05
0.90
0.05
0.90
0.05
93.76
294.32
Conv6
64
256
0.65
0.05
0.90
0.05
0.90
0.05
5.86
83.68
Conv7
64
576
0.48
0.05
0.97
0.05
0.97
0.05
0.51
83.68
Conv8
256
64
0.49
0.05
0.90
0.05
0.90
0.05
19.67
294.32
Conv9
64
256
0.67
0.05
0.90
0.05
0.90
0.05
0.18
83.68
Conv10
64
576
0.81
0.05
0.97
0.05
0.97
0.05
0.1
83.68
Conv
256
64
0.70
0.05
0.90
0.05
0.90
0.05
8.18
294.32
Conv 12
128
256
0.60
0.05
0.96
0.05
0.96
0.05
1.46
155.4
Conv 13
128
1152
0.67
0.05
1.00
0.05
1.00
0.05
0.16
155.4
Conv 14
512
128
0.60
0.05
0.99
0.05
0.99
0.05
89.07
565.75
Conv 15
512
256
0.27
0.05
1.00
0.05
1.00
0.05
83.22
565.75
Conv16 16
128
512
0.39
0.05
0.99
0.05
0.99
0.05
2.41
155.4
Conv 17
128
1152
0.68
0.05
1.00
0.05
1.00
0.05
0.6
155.4
Conv 18
512
128
0.67
0.05
0.99
0.05
0.99
0.05
17.33
565.75
Conv 19
128
512
0.57
0.05
0.99
0.05
0.99
0.05
0.15
155.4
Conv20
128
1152
0.61
0.05
1.00
0.05
1.00
0.05
0.09
155.4
Conv21
512
128
0.52
0.05
0.99
0.05
0.99
0.05
3.61
565.75
Conv22
128
512
0.64
0.05
0.99
0.05
0.99
0.05
0.12
155.4
Conv23
128
1152
0.83
0.05
1.00
0.05
1.00
0.05
0.09
155,4
Conv24
512
128
0.69
0.05
0.99
0.05
0.99
0.05
4.59
565.75
Conv25
256
512
0.63
0.05
1.00
0.05
1.00
0.05
0.69
294.32
Conv26
256
2304
0.67
0.05
1.00
0.05
1.00
0.05
0.1
294.32
Conv27
1024
256
0.61
0.05
1,00
0.05
1.00
0.05
58.75
1099.56
Conv28
1024
512
0.56
0.05
1,00
0.05
1.00
0.05
52.35
1099.56
Conv29
256
1024
0.47
0.05
1,00
0.05
1.00
0.05
0.25
294.32
Conv30
256
2304
0.55
0.05
1.00
0.05
1.00
0.05
0.39
294.32
Conv31
1024
256
0.51
0.05
1,00
0.05
1.00
0.05
13.05
1099.56
Conv32
256
1024
0.57
0.05
1,00
0.05
1.00
0.05
0.14
294.32
Conv33
256
2304
0.80
0.05
1,00
0.05
1.00
0.05
0.14
294.32
Conv34
1024
256
0.66
0.05
1.00
0.05
1.00
0.05
6.25
1099.56
Conv35
256
1024
0.62
0.05
1.00
0.05
1.00
0.05
0.14
294.32
Conv36
256
2304
0.64
0.05
1.00
0.05
1.00
0.05
0.19
294.32
Conv37
1024
256
0.69
0.05
1.00
0.05
1.00
0.05
5.59
1099.56
Conv38
256
1024
0.67
0.05
1.00
0.05
1.00
0.05
0.54
294.32
107

---

Under review as a conference paper at ICLR 2021
Conv39
256
2304
0.69
0.05
1.00
0.05
1.00
0.05
0.68
294.32
Conv40
1024
256
0.67
0.05
1.00
0.05
1.00
0.05
5.87
1099.56
Conv41
256
1024
0.66
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv42
256
2304
0.80
0.05
1.00
0.05
1.00
0.05
0.21
294.32
Conv43
1024
256
0.63
0.05
1.00
0.05
1.00
0.05
6.01
1099.56
Conv44
512
1024
0.69
0.05
1.00
0.05
1.00
0.05
0.51
565.75
Conv45
512
4608
0.64
0.05
1.00
0.05
1.00
0.05
0.28
565.75
Conv46
2048
512
0.57
0.05
1.00
0.05
1.00
0.05
4.17
2154.4
Conv47
2048
1024
0.38
0.05
1.00
0.05
1.00
0.05
2.83
2154.4
Conv48
512
2048
0.28
0.05
1,00
0.05
1.00
0.05
3.58
565.75
Conv49
512
4608
0.67
0.05
1,00
0.05
1.00
0.05
2.97
565.75
Conv50
2048
512
0.70
0.05
1,00
0.05
1.00
0.05
17.54
2154.4
Conv51
512
2048
0.55
0.05
1.00
0.05
1.00
0.05
6.14
565.75
Conv52
512
4608
0.64
0.05
1,00
0.05
1.00
0.05
3.58
565.75
Conv53
2048
512
0.61
0.05
1.00
0.05
1.00
0.05
24.98
2154.4
Conv54
1024
2048
0.84
0.05
1,00
0.05
1.00
0.05
0.92
1099.56
Conv55
1024
18432
0.66
0.05
1.00
0.05
1.00
0.05
3.88
1099.56
Conv56
1024
18432
0.65
0.05
1.00
0.05
1.00
0.05
3.91
1099.56
Conv57
1024
18432
0.68
0.05
1.00
0.05
1.00
0.05
2.13
1099.56
Conv58
1024
5120
0.63
0.05
1.00
0.05
1.00
0.05
14.47
1099.56
Conv59
1024
9216
0.24
0.05
1.00
0.05
1.00
0.05
48.34
1099.56
Conv60
1024
1024
0.63
0.05
1.00
0.05
1.00
0.05
18.39
1099.56
Conv61
1024
9216
0.55
0.05
1,00
0.05
1.00
0.05
59.85
1099.56
Conv62
1024
11520
0.58
0.05
1,00
0.05
1.00
0.05
5.42
1099.56
Conv63
512
9216
0.61
0.05
1,00
0.05
1.00
0.05
1.24
565.75
Conv64
256
4608
0.62
0.05
1.00
0.05
1.00
0.05
0.73
294.32
Conv65
64
2304
0.81
0.05
1,00
0.05
1.00
0.05
0.03
83.68
Conv66
16
64
0.50
0.05
0.63
0.05
0.63
0.05
5.7
26.3
Conv67
64
720
0.41
0.05
0.98
0.05
0.98
0.05
0.37
83.68
Conv68
64
3136
0.37
0.05
1.00
0.05
1.00
0.05
0.9
83.68
Conv69
64
576
0.39
0.05
0.97
0.05
0.97
0.05
6.03
83.68
Conv70 70
64
603
0.32
0.05
0.98
0.05
0.98
0.05
17.08
83.68
Conv71
64
576
0.23
0.05
0.97
0.05
0.97
0.05
46.48
83.68
Passing rate
100.0%
100.0%
100.0%
100.0%
P.7.4 STYLE TRANSFER
config:
Table 46: Fast neural style (candy)
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv 1
32
243
0.14
0.05
0.81
0.05
0.81
0.05
9.84
46.19
Conv2
64
288
0.09
0.05
0.91
0.05
0.91
0.05
9.0
83.68
Conv3
128
576
0.13
0.05
1.00
0.05
1.00
0.05
8.91
155.4
Conv4
128
1152
0.02
0.05
1.00
0.05
1.00
0.05
23.03
155.4
Conv5
128
1152
0.10
0.05
1.00
0.05
1.00
0.05
2.99
155.4
Conv6
128
1152
0.11
0.05
1.00
0.05
1.00
0.05
1.76
155.4
Conv7
128
1152
0.06
0.05
1.00
0.05
1.00
0.05
3.21
155.4
Conv8
128
1152
0.13
0.05
1.00
0.05
1.00
0.05
19.79
155.4
Conv9
128
1152
0.04
0.05
1.00
0.05
1.00
0.05
4.09
155.4
Conv10
128
1152
0.24
0.05
1.00
0.05
1.00
0.05
1.11
155.4
Conv 11
128
1152
0.05
0.05
1.00
0.05
1.00
0.05
3.34
155.4
108

---

Under review as a conference paper at ICLR 2021
Conv 12
128
1152
0.15
0.05
1.00
0.05
1.00
0.05
1.49
155.4
Conv 13
128
1152
0.25
0.05
1.00
0.05
1.00
0.05
2.38
153.4
Conv 14
64
1152
0.30
0.05
1.00
0.05
1.00
0.05
0.93
83.68
Conv15
32
576
0.31
0.05
0.91
0.05
0.91
0.05
1.08
46.19
Passing rate
86.67%
100.0%
100.0%
100.0%
Table 47: Fast neural style (mosaic)
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
32
243
0.18
0.05
0.81
0.05
0.81
0.05
1.44
46.19
Conv2
64
288
0.23
0.05
0.91
0.05
0.91
0.05
7.95
83.68
Conv3
128
576
0.21
0.05
1.00
0.05
1.00
0.05
27.77
155.4
Conv4
128
1152
0.18
0.05
1.00
0.05
1.00
0.05
7.1
155.4
Conv5
128
1152
0.04
0.05
1.00
0.05
1.00
0.05
2.78
155.4
Conv6
128
1152
0.30
0.05
1.00
0.05
1.00
0.05
11.29
155.4
Conv7 7
128
1152
0.13
0.05
1.00
0.05
1.00
0.05
3.61
155.4
Conv8
128
1152
0.19
0.05
1.00
0.05
1.00
0.05
1.48
155.4
Conv9
128
1152
0.12
0.05
1.00
0.05
1.00
0.05
2.61
155.4
Conv10
128
1152
0.28
0.05
1.00
0.05
1.00
0.05
1.44
155.4
Conv 11
128
1152
0.23
0.05
1.00
0.05
1.00
0.05
2.27
155.4
Conv 12
128
1152
0.22
0.05
1.00
0.05
1.00
0.05
8.63
155,4
Conv 13
128
1152
0.26
0.05
1.00
0.05
1.00
0.05
2.33
155,4
Conv 14
64
1152
0.38
0.05
1.00
0.05
1.00
0.05
1.12
83.68
Conv15
32
576
0.52
0.05
0.91
0.05
0.91
0.05
0.23
46.19
Passing rate
93.33%
100.0%
100.0%
100.0%
Table 48: Fast neural style (starry night)
Layer
Number
dim
Gaussian
Mean ReannRiggt
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
32
243
0.13
0.05
0.81
0.05
0.81
0.05
9,92
46.19
Conv2
64
288
0.14
0.05
0.91
0.05
0.91
0.05
19.39
83.68
Conv3
128
576
0.20
0.05
1.00
0.05
1.00
0.05
32.68
155.4
Conv4
128
1152
0.01
0.05
1.00
0.05
1.00
0.05
142.0
155.4
Conv5
128
1152
0.02
0.05
1.00
0.05
1.00
0.05
28.99
155.4
Conv6
128
1152
0.17
0.05
1.00
0.05
1.00
0.05
4.55
155.4
Conv7
128
1152
0.10
0.05
1.00
0.05
1.00
0.05
3.05
155.4
Conv8
128
1152
0.17
0.05
1.00
0.05
1.00
0.05
1.67
155.4
Conv9
128
1152
0.07
0.05
1.00
0.05
1.00
0.05
2.42
155.4
Conv 10
128
1152
0.33
0.05
1.00
0.05
1.00
0.05
6.25
155.4
Conv11 11
128
1152
0.12
0.05
1.00
0.05
1.00
0.05
2.06
155.4
Conv 12
128
1152
0.31
0.05
1.00
0.05
1.00
0.05
1.12
155.4
Conv 13
128
1152
0.16
0.05
1.00
0.05
1.00
0.05
1.58
155.4
Conv 14
64
1152
0.29
0.05
1.00
0.05
1.00
0.05
0.47
83.68
Conv 15
32
576
0.44
0.05
0.91
0.05
0.91
0.05
0.71
46.19
Passing rate
86.67% %
100.0%
100.0%
100.0%
109

---

Under review as a conference paper at ICLR 2021
Table 49: Fast neural style (udnie)
Layer
Number
dim
Gaussian
Mean Left
Mean Mean_Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
32
243
0.22
0.05
0.81
0.05
0.81
0.05
9.61
46.19
Conv2
64
288
0.17
0.05
0.91
0.05
0.91
0.05
13.05
83.68
Conv3
128
576
0.19
0.05
1.00
0.05
1.00
0.05
10.46
155.4
Conv4
128
1152
0.06
0.05
1.00
0.05
1.00
0.05
20.22
155.4
Conv5
128
1152
0.14
0.05
1.00
0.05
1.00
0.05
4.46
155,4
Conv6
128
1152
0.14
0.05
1.00
0.05
1.00
0.05
3.17
155.4
Conv7
128
1152
0.02
0.05
1.00
0.05
1.00
0.05
5.85
155.4
Conv8
128
1152
0.17
0.05
1.00
0.05
1.00
0.05
18.5
155.4
Conv9
128
1152
0.00
0.05
1.00
0.05
1.00
0.05
18.88
153,4
Conv 10
128
1152
0.02
0.05
1.00
0.05
1.00
0.05
3.59
155.4
Conv 11
128
1152
0.00
0.05
1.00
0.05
1.00
0.05
10.35
155,4
Conv 12
128
1152
0.02
0.05
1.00
0.05
1.00
0.05
3.33
155.4
Conv 13
128
1152
0.21
0.05
1.00
0.05
1.00
0.05
3.99
155.4
Conv14 14
64
1152
0.25
0.05
1.00
0.05
1.00
0.05
2.52
83.68
Conv15 15
32
576
0.26
0.05
0.91
0.05
0.91
0.05
2.34
46.19
Passing rate
66.67% %
100.0%
100.0%
100.0%
P.7.5 GAN
config:
http::/gittub..o/ccsiiw/w
pytorch
Table 50: DCGAN MNIST
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
100
8192
0.70
0.05
1.00
0.05
1.00
0.05
1.88
124.34
Conv2
512
4096
0.69
0.05
1.00
0.05
1.00
0.05
0.8
565,75
Conv3
256
2048
0.64
0.05
1.00
0.05
1.00
0.05
0.92
294.32
Conv4
128
1024
0.50
0.05
1.00
0.05
1.00
0.05
3.88
155.4
Passing rate
100.0%
100.0%
100.0%
100.0%
Table 51: DCGAN Cifarl0
Layer
Number
dim
Gaussian
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
100
8192
0.65
0.05
0.05
0.05
1.48
124.34
Conv2
512
4096
0.48
0.05
0.05
0.05
3.56
565.75
Conv3
256
2048
0.32
0.05
0.05
1
0.05
10.52
294.32
Conv4
128
1024
0.44
0.05
0.05
0.05
9.06
155.4
Passing rate
100.0%
100.0%
100.0%
100.0%
Table 52: DCGAN Cifar100
110

---

Under review as a conference paper at ICLR 2021
Layer
Number
dim
Gaussian
Mean
Mean Mean_Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
100
8192
0.53
0.05
1.00
0.05
1.00
0.05
2.13
124.34
Conv2
512
4096
0.33
0.05
1.00
0.05
1.00
0.05
4.32
565.75
Conv3
256
2048
0.48
0.05
1.00
0.05
1.00
0.05
8.49
294.32
Conv4
128
1024
0.27
0.05
1.00
0.05
1.00
0.05
15.21
155.4
Passing rate
100.0%
100.0%
100.0%
100.0%
P.8
BATCH NORMALIZATION
config:
Table 53: Cifarl0 VGG19
Layer
Number
dim
Gaussian
Mean
Mean_Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.24
0.05
0.66
0.05
0.66
0.05
388.19
83.68
Conv2
64
576
0.62
0.05
0.97
0.05
0.97
0.05
1.55
83.68
Conv3
128
576
0.84
0.05
1.00
0.05
1.00
0.05
2.61
155.4
Conv4
128
1152
0.86
0.05
1.00
0.05
1.00
0.05
0.39
155,4
5
256
1152
0.82
0.05
1.00
0.05
1.00
0.05
0.67
294.32
Conv6
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.41
294.32
Conv7
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.35
294.32
Conv8
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.21
294.32
Conv9
512
2304
0.92
0.05
1.00
0.05
1.00
0.05
1.23
565.75
Conv 10
512
4608
0.91
0.05
1.00
0.05
1.00
0.05
1.08
565.75
Convl1 11
512
4608
0.93
0.05
1.00
0.05
1.00
0.05
0.32
565.75
Conv11 12
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.16
565.75
Conv13
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.17
565.75
Conv 14
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.17
565.75
Conv 15
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.12
565.75
Conv 16
512
4608
0.93
0.05
1.00
0.05
1.00
0.05
0.25
565.75
Passing rate
100.0%
100.0%
100.0%
93.75%
Table 54: Cifar10 VGG19-bn
Layer
Number dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
64
27
0.32
0.05
0.66
0.05
0.66
0.05
399.99
83.68
Conv2
64
576
0.53
0.05
0.97
0.05
0.97
0.05
1.69
83.68
Conv3
128
576
0.85
0.05
1.00
0.05
1.00
0.05
3.16
155,4
Conv4
128
1152
0.86
0.05
1.00
0.05
1.00
0.05
0.45
155.4
C
256
1152
0.84
0.05
1.00
0.05
1.00
0.05
1.05
294.32
Conv6
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
0.38
294.32
Conv7
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.45
294.32
Conv8
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.28
294.32
Conv9
512
2304
0.91
0.05
1.00
0.05
1.00
0.05
565.75
Conv 10
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.91
565.75
Conv 11
512
4608
0.92
0.05
1.00
0.05
1.00
0.05
0.22
565.75
Cony12
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.18
565.75
111

---

Under review as a conference paper at ICLR 2021
Conv 13
512
4608
0,94
0.05
1.00
0.05
1.00
0.05
0.29
565.75
Conv 14
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.23
565.75
Conv 15
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.13
565.75
Conv16
512
4608
0.93
0.05
1.00
0.05
1.00
0.05
0.32
565.75
Passing rate
100.0%
100.0%
100.0%
93.75%
P.9
PYTORCHI PRETRAIN
config: http:///aaaa...ooocc///ssssstttt
Table 55: Pytorch pre-trained VGG11
Layer
Number
dim
Gaussian
Mean Left
Mean Mean.Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.08
0.05
0.98
0.05
0.98
0.05
18556.94
83.68
Conv2
128
576
0.43
0.05
1.00
0.05
1.00
0.05
365.45
155.4
Conv3
256
1152
0.55
0.05
1.00
0.05
1.00
0.05
118.19
294.32
Conv4
256
2304
0.42
0.05
1.00
0.05
1.00
0.05
60.64
294.32
Conv5
512
2304
0,67
0.05
1.00
0.05
1.00
0.05
44,8
565.75
Conv6
512
4608
0.83
0.05
1.00
0.05
1.00
0.05
23.01
565.75
Conv7
512
4608
0.82
0.05
1.00
0.05
1.00
0.05
16.13
565.75
Conv8
512
4608
0.88
0.05
1.00
0.05
1.00
0.05
43.86
565.75
Passing rate
100.0%
100.0%
100.0%
75.0%
Table 56: Pytorch pre-trained VGG16
Layer
Number
dim
Gaussian
Mean Left
Mean Mean.Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.20
0.05
0.98
0.05
0.98
0.05
8106.7
83.68
Conv2
64
576
0.59
0.05
1.00
0.05
1.00
0.05
134.8
83.68
Conv3
128
576
0.55
0.05
1.00
0.05
1.00
0.05
152.47
155.4
Conv4
128
1152
0.81
0.05
1.00
0.05
1.00
0.05
64.62
155.4
Conv5
256
1152
0.56
0.05
1.00
0.05
1.00
0.05
40.31
294.32
Conv6
256
2304
0.70
0.05
1.00
0.05
1.00
0.05
21.44
294.32
Conv7
256
2304
0.55
0.05
1.00
0.05
1.00
0.05
25,05
294.32
Conv8
512
2304
0.64
0.05
1.00
0.05
1.00
0.05
33.34
565.75
Conv9
512
4608
0.85
0.05
1.00
0.05
1.00
0.05
11.92
565.75
Conv 10
512
4608
0.88
0.05
1.00
0.05
1.00
0.05
11.49
565.75
Conv 11
512
4608
0.83
0.05
1.00
0.05
1.00
0.05
9.21
565.75
Conv 12
512
4608
0.87
0.05
1.00
0.05
1.00
0.05
8.74
565.75
Conv13 13
512
4608
0.92
0.05
1.00
0.05
1.00
0.05
33.55
565.75
Passing rate
100.0%
100.0%
100.0%
84.62%
Table 57: Pytorch pre-trained VGG19
Layer
Number dim
Gaussian
Mean.Left
Mcan_Right
Sigma
112

---

Under review as a conference paper at ICLR 2021
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
64
27
0.25
0.05
0.98
0.05
0.98
0.05
6679.05
83.68
Conv2
64
576
0.61
0.05
1.00
0.05
1.00
0.05
163.5
83.68
Conv3
128
576
0.58
0.05
1.00
0.05
1.00
0.05
132.27
155,4
Conv4
128
1152
0.58
0.05
1.00
0.05
1.00
0.05
52.71
155.4
Conv5
256
1152
0.51
0.05
1.00
0.05
1.00
0.05
36.91
294.32
Conv6
256
2304
0.66
0.05
1.00
0.05
1.00
0.05
19.93
294.32
Conv7
256
2304
0.85
0.05
1.00
0.05
1.00
0.05
14.54
294.32
Conv8
256
2304
0.64
0.05
1.00
0.05
1.00
0.05
20.24
294.32
Conv9
512
2304
0.64
0.05
1.00
0.05
1.00
0.05
26.91
565.75
Conv 10
512
4608
0.86
0.05
1.00
0.05
1.00
0.05
10.7
565.75
Conv 11
512
4608
0.89
0.05
1.00
0.05
1.00
0.05
8.98
565.75
Conv 12
512
4608
0.89
0.05
1.00
0.05
1.00
0.05
7.87
565.75
Conv1 13
512
4608
0.87
0.05
1.00
0.05
1.00
0.05
7.51
565.75
Conv 14
512
4608
0,90
0.05
1.00
0.05
1.00
0.05
9.71
565.75
Conv 15
512
4608
0.92
0.05
1.00
0.05
1.00
0.05
7.61
565.75
Conv16
512
4608
0.90
0.05
1.00
0.05
1.00
0.05
22.02
565.75
Passing rate
100.0%
100.0%
100.0%
87.50%
Table 58: Pytorch pre-trained ResNet18
Layer
Number
dim
Gaussian
Mean Left
Mean Migh_Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
64
147
0.31
0.05
1.00
0.05
1.00
0.05
3000.89
83.68
Conv2
64
576
0.42
0.05
1.00
0.05
1.00
0.05
90.98
83.68
Conv3
64
576
0.63
0.05
1.00
0.05
1.00
0.05
42.7
83.68
Conv4
64
576
0.52
0.05
1.00
0.05
1.00
0.05
82.51
83.68
Conv5
64
576
0.70
0.05
1.00
0.05
1.00
0.05
32.27
83.68
Conv6
128
576
0.82
0.05
1.00
0.05
1.00
0.05
14.37
155,4
Conv7
128
1152
0.67
0.05
1.00
0.05
1.00
0.05
35.48
155.4
Conv8
128
1152
0.66
0.05
1.00
0.05
1.00
0.05
15.09
155,4
Conv9
128
1152
0.81
0.05
1.00
0.05
1.00
0.05
34.26
155,4
Conv 10
256
1152
0.70
0.05
1.00
0.05
1.00
0.05
11.25
294.32
Conv
256
2304
0.82
0.05
1.00
0.05
1.00
0.05
10.09
294.32
Conv 12
256
2304
0.84
0.05
1.00
0.05
1.00
0.05
12.4
294.32
Conv 13
256
2304
0.82
0.05
1.00
0.05
1.00
0.05
42.29
294.32
Conv 14
512
2304
0.70
0.05
1.00
0.05
1.00
0.05
4.8
565.75
Conv 15
512
4608
0.81
0.05
1.00
0.05
1.00
0.05
8.93
565.75
Conv 16
512
4608
0.86
0.05
1.00
0.05
1.00
0.05
3.92
565.75
Conv17
512
4608
0.85
0.05
1.00
0.05
1.00
0.05
1.47
565.75
Passing rate
100.0%
100.0%
100.0%
88.24%
Table 59: Pytorch pre-trained ResNet34
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv 1
64
147
0.35
0.05
1.00
0.05
1.00
0.05
3020. 16
83.68
Conv2
64
576
0.54
0.05
1.00
0.05
1.00
0.05
76.24
83.68
Conv3
64
576
0.67
0.05
1.00
0.05
1.00
0.05
40.95
83.68
113

---

Under review as a conference paper at ICLR 2021
Conv4
64
576
0.67
0.05
1.00
0.05
1.00
0.05
26.53
83.68
Conv5
64
576
0.84
0.05
1.00
0.05
1.00
0.05
33.62
83.68
Conv6
64
576
0.84
0.05
1.00
0.05
1.00
0.05
14.46
83.68
Conv7
64
576
0.83
0.05
1.00
0.05
1.00
0.05
41.19
83.68
Conv8
128
576
0.83
0.05
1.00
0.05
1.00
0.05
13.86
155.4
Conv9
128
1152
0,81
0.05
1.00
0.05
1.00
0.05
86.4
155,4
Conv10
128
1152
0.66
0.05
1.00
0.05
1.00
0.05
20.55
155,4
Conv 1
128
1152
0.85
0.05
1.00
0.05
1.00
0.05
36.72
155.4
Conv 12
128
1152
0.54
0.05
1.00
0.05
1.00
0.05
9.64
155,4
Conv13
128
1152
0.84
0.05
1.00
0.05
1,00
0.05
18.48
155.4
Conv 14
128
1152
0.69
0.05
1.00
0.05
1.00
0.05
11.52
155,4
Conv 15
128
1152
0.85
0.05
1.00
0.05
1.00
0.05
39.51
155.4
Conv 16
256
1152
0.83
0.05
1.00
0.05
1.00
0.05
9.06
294.32
Conv 17
256
2304
0,81
0.05
1.00
0.05
1.00
0.05
13.99
294.32
Conv 18
256
2304
0.81
0.05
1.00
0.05
1.00
0.05
15.44
294.32
Conv19 19
256
2304
0.64
0.05
1.00
0.05
1.00
0.05
17.56
294.32
Conv20
256
2304
0.68
0.05
1.00
0.05
1.00
0.05
7.88
294.32
Conv21
256
2304
0.69
0.05
1.00
0.05
1.00
0.05
29.32
294.32
Conv22
256
2304
0.87
0.05
1.00
0.05
1.00
0.05
7.87
294.32
Conv23
256
2304
0.87
0.05
1.00
0.05
1.00
0.05
45.76
294.32
Conv24
256
2304
0.86
0.05
1.00
0.05
1.00
0.05
6,93
294.32
Conv25
256
2304
0.84
0.05
1.00
0.05
1.00
0.05
36.02
294.32
Conv26
256
2304
0.89
0.05
1.00
0.05
1,00
0.05
6.78
294.32
Conv27
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
22.54
294.32
Conv28
512
2304
0.83
0.05
1.00
0.05
1.00
0.05
2.31
565.75
Conv29
512
4608
0.86
0.05
1.00
0.05
1.00
0.05
2.42
565.75
Conv30
512
4608
0.85
0.05
1.00
0.05
1.00
0.05
4.27
565.75
Conv31
512
4608
0.86
0.05
1.00
0.05
1.00
0.05
6.3
565.75
Conv32
512
4608
0.88
0.05
1.00
0.05
1.00
0.05
1.64
565.75
Conv33
512
4608
0.87
0.05
1.00
0.05
1.00
0.05
1.24
565.75
Passing rate
100.0%
100.0%
100.0%
96.97%
Table 60: Pytorch pre-trained ResNet50
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
e-value
p-value
c-value
t-value
c-value
Conv1
64
147
0.43
0.05
1.00
0.05
1.00
0.05
1512.0
83.68
Conv2
64
64
0.47
0.05
1.00
0.05
1.00
0.05
512.72
83.68
Conv3
64
576
0.64
0.05
1.00
0.05
1.00
0.05
103.97
83.68
Conv4
256
64
0.69
0.05
1.00
0.05
1.00
0.05
786.24
294.32
C
64
256
0.86
0.05
1.00
0.05
1.00
0.05
52.01
83.68
Conv6
64
576
0.60
0.05
1.00
0.05
1.00
0.05
31.52
83.68
Conv7
256
64
0.84
0.05
1.00
0.05
1.00
0.05
619.68
294.32
Conv8
64
256
0.89
0.05
1.00
0.05
1.00
0.05
14.19
83.68
Conv9
64
576
0.85
0.05
1.00
0.05
1.00
0.05
6.07
83.68
Conv 10
256
64
0.85
0.05
1.00
0.05
1.00
0.05
789.85
294.32
Conv 11
128
256
0.81
0.05
1.00
0.05
1.00
0.05
49.33
155.4
Conv 12
128
1152
0.84
0.05
1.00
0.05
1.00
0.05
6,6
155.4
Conv1 13
512
128
0.70
0.05
1.00
0.05
1.00
0.05
1154.78
565.75
Conv14
128
512
0.86
0.05
1.00
0.05
1.00
0.05
20.26
155.4
Conv15 15
128
1152
0.84
0.05
1.00
0.05
1.00
0.05
59.31
155.4
Conv 16
512
128
0.84
0.05
1.00
0.05
1.00
0.05
897.22
565.75
Conv 17
128
512
0.87
0.05
1.00
0.05
1.00
0.05
14.87
155.4
Conv18
128
1152
0.86
0.05
1.00
0.05
1.00
0.05
10.56
155.4
114

---

Under review as a conference paper at ICLR 2021
Conv 19
512
128
0.81
0.05
1.00
0.05
1.00
0.05
326.18
565.75
Conv20
128
512
0.85
0.05
1.00
0.05
1.00
0.05
5.57
155.4
Conv21
128
1152
0.88
0.05
1.00
0.05
1.00
0,05
7.15
155,4
Conv22
512
128
0.84
0.05
1.00
0.05
1.00
0.05
511.13
565.75
Conv23
256
512
0.82
0.05
1.00
0.05
1.00
0.05
32.44
294.32
Conv24
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
8.35
294.32
Conv25
1024
256
0.83
0.05
1.00
0.05
1.00
0.05
608.91
1099.56
Conv26
256
1024
0.84
0.05
1.00
0.05
1.00
0.05
10.33
294.32
Conv27
256
2304
0.86
0.05
1.00
0.05
1.00
0.05
12.19
294.32
Conv28
1024
256
0.62
0.05
1.00
0.05
1,00
0.05
365.91
1099.56
Conv29
256
1024
0.85
0.05
1.00
0.05
1.00
0.05
294.32
Conv30
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
7.69
294.32
Conv31
1024
256
0.81
0.05
1.00
0.05
1.00
0.05
306.61
1099.56
Conv32
256
1024
0,87
0.05
1.00
0.05
1.00
0.05
5.97
294.32
Conv33
256
2304
0.85
0.05
1.00
0.05
1.00
0.05
6.47
294.32
Conv34
1024
256
0.83
0.05
1.00
0.05
1.00
0.05
312.36
1099.56
Conv35
256
1024
0.85
0.05
1.00
0.05
1.00
0.05
5,81
294.32
Conv36
256
2304
0.89
0.05
1.00
0.05
1.00
0.05
5.38
294.32
Conv37
1024
256
0.83
0.05
1.00
0.05
1.00
0.05
330.13
1099.56
Conv38
256
1024
0.69
0.05
1.00
0.05
1.00
0.05
4.9
294.32
Conv39
256
2304
0.88
0.05
1.00
0.05
1.00
0.05
4.16
294.32
Conv40
1024
256
0.82
0.05
1.00
0.05
1.00
0.05
319.79
1099.56
Conv41
512
1024
0.82
0.05
1.00
0.05
1,00
0.05
6.42
565.75
Conv42
512
4608
0.69
0.05
1.00
0.05
1.00
0.05
1.38
565.75
Conv43
2048
512
0.81
0.05
1.00
0.05
1.00
0.05
97.84
2154.4
Conv44
512
2048
0.48
0.05
1.00
0.05
1.00
0.05
3.4
565.75
Conv45
512
4608
0.88
0.05
1.00
0.05
1.00
0.05
2.7
565.75
Conv46
2048
512
0,87
0.05
1.00
0.05
1.00
0.05
106.86
2154.4
Conv47
512
2048
0.65
0.05
1.00
0.05
1.00
0.05
4.34
565.75
Conv48
512
4608
0.92
0.05
1.00
0.05
1.00
0.05
0.65
565.75
Conv49
2048
512
0.85
0.05
1.00
0.05
1.00
0.05
139.29
2154.4
Passing rate
100.0%
100.0%
100.0%
83.67%
P.10 LEARNING RATE
config:
Table 61: Cifar10 VGG19 bn schedule(82-164)
Layer
Number
dim
Gaussian
Mean Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1
64
27
0.22
0.05
0.66
0.05
0.66
0.05
313.52
83.68
Conv2
64
576
0.62
0.05
0.97
0.05
0.97
0.05
1.52
83.68
Conv3
128
576
0.82
0.05
1.00
0.05
1.00
0.05
1.88
155.4
Conv4
128
1152
0.89
0.05
1.00
0.05
1.00
0.05
0.31
155.4
C
256
1152
0.88
0.05
1.00
0.05
1.00
0.05
0.5
294.32
Conv6
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.24
294.32
Conv7
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv8
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.13
294.32
Conv9
512
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.6
565.75
Conv 10
512
4608
0,92
0.05
1.00
0.05
1.00
0.05
0.4
565.75
Conv 11
512
4608
0.89
0.05
1.00
0.05
1.00
0.05
0.53
565.75
Conv12
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.32
565.75
115

---

Under review as a conference paper at ICLR 2021
Conv 13
512
4608
0,95
0.05
1.00
0.05
1.00
0.05
0.19
565.75
Conv 14
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.11
565.75
Conv 15
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.08
565.75
Conv16
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.32
565.75
Passing rate
100.0%
100.0%
100.0%
93.75%
Table 62: C1 VGG19 bn schedule(60-120)
Layer
Number
dim
Gaussian
Mean_
Mean_ _ight
Sigma
p-value
c-value
p-value
c-value
p-value
e-value
t-value
e-value
Conv1
64
27
0.23
0.05
0.66
0.05
0.66
0.05
292.85
83.68
Conv2
64
576
0,64
0.05
0.97
0.05
0.97
0.05
1.18
83.68
Conv3
128
576
0.85
0.05
1.00
0.05
1.00
0.05
2.03
155.4
Conv4
128
1152
0.88
0.05
1.00
0.05
1.00
0.05
0.34
155,4
Conv5
256
1152
0.85
0.05
1.00
0.05
1.00
0.05
0.54
294.32
Conv6
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.3
294.32
Conv7
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.23
294.32
Conv8
256
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.2
294.32
Conv9
512
2304
0.91
0.05
1.00
0.05
1.00
0.05
0.65
565.75
Conv 10
512
4608
0.92
0.05
1.00
0.05
1.00
0.05
0.38
565.75
Conv 11
512
4608
0.92
0.05
1.00
0.05
1.00
0.05
0.57
565.75
Conv 12
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.34
565.75
Conv 13
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.17
565.75
Conv 14
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.11
565.75
Conv15 15
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.08
565.75
Conv16 16
512
4608
0.94
0.05
1.00
0.05
1.00
0.05
0.26
565.75
Passing rate
100.0%
100.0%
100.0%
93.75%
Table 63: Cifar10 VGG19 bn coslr
Layer
Number
dim
Gaussian
Left
Mean Right
Sigma
p-value
c-value
p-value
c-value
p-value
c-value
t-value
c-value
Conv1 1
64
27
0.35
0.05
0.66
0.05
0.66
0.05
313.71
83.68
Conv2
64
576
0.65
0.05
0.97
0.05
0.97
0.05
1.31
83.68
Conv3
128
576
0.81
0.05
1.00
0.05
1.00
0.05
2.38
155.4
Conv4
128
1152
0.89
0.05
1.00
0.05
1.00
0.05
0.34
155.4
5
256
1152
0.87
0.05
1.00
0.05
1.00
0.05
0.44
294.32
Conv6
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.3
294.32
Conv7
256
2304
0.90
0.05
1.00
0.05
1.00
0.05
0.22
294.32
Conv8
256
2304
0.85
0.05
1.00
0.05
1.00
0.05
0.1
294.32
Conv9
512
2304
0.89
0.05
1.00
0.05
1.00
0.05
0.74
565.75
Conv10 10
512
4608
0.91
0.05
1.00
0.05
1.00
0.05
0.34
565.75
Conv11
512
4608
0.92
0.05
1.00
0.05
1.00
0.05
0.54
565.75
Conv 12
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.32
565.75
Conv 13
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.2
565.75
Conv 14
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.11
565.75
Conv15 15
512
4608
0.95
0.05
1.00
0.05
1.00
0.05
0.08
565.75
Conv16 16
512
4608
0.93
0.05
1.00
0.05
1.00
0.05
0.32
565.75
Passing rate
100.0%
100.0%
100.0%
93.75%
116

---

Under review as a conference paper at ICLR 2021
Q
ADDITIONAL EXPERIMENTS FOR SECTION
In this section, we use more networks (Pytorch Pretrain Models to verify the obersevations in Fig. 2 (a-d). We show the results of three layers for each network.
Q.1 VGG16
VGG16-kemel size 3
10
dim
2000
20
300
4000
4000
dim
aim
VGG16-kernll size
200
400
dim
600
300
180
500
1000
664
1000
dim
6im
VGG16-kernel size -
dim
1000/
mm
1500
2000
500
1000
1500
2000
dim
http::/swaeeooooeeee///tttttttts htm
117

---

Under review as a conference paper at ICLR 2021
Q.2 VGG19
VGG19-kemel size
10
1000
dim
1600
2000
50
500
1000
1500
2000
dim
dimm
VGG19-kemel size = 3
1000
2000
dim
diim
20
3800
4200
60
1000
2000
3000
4000
60
dim
VGG19-kernel size
1000
2000
dim
70
dim
4000
60
1000
2000
3000
4000
20
30
40
50
65
dim
118

---

Under review as a conference paper at ICLR 2021
Q.3 RESNETIS 18
ResNet18- size 3
100
200
300
20
dim
20
400
500
100
200
300
400
500
30
dim
ResNetl8-kemts size =
1000
dim
dim
201
1600
2000
60
500
1000
1500
2000
10
dim
ResNet 8-kernel size
20004
dim
20
dim
3000
4000
001
1000
2000
3000
4000
30
dim
dim
119

---

Under review as a conference paper at ICLR 2021
Q.4 RESNET50
ResNet50-kernel size = 3
10
dim
dim
40
200
400
600
800
1000
dim
dim
ResNet50-kemel size
50
10
dim
20
150
dim
200
50
250
50
100
150
200
250
20
30
50
dim
dim
ResNet50-kkrnel size =
500
1000
dim
dim
501
2000
50g
601
500
1000
1500
2000
dim
dim
120

---

Under review as a conference paper at ICLR 2021
Q.5
ALEXNET
AlexNet-kermel size =
200
400
600
dim
800
1060
dim
40
1600
250
750
1000
1500
dim
AlexNet-kemel size
200
400
600
10
dim
800
1000
dim
1400
1600
0
250
500
750
1000
1250
1500
dim
AlexNet-kernel size
500
10
dim
1000
dim
1300
40
2000
50
500
1000
1500
2000
30
dim
dim
121

---

Under review as a conference paper at ICLR 2021
Q.6 DENSENET
DenseNet-kernel size
200
600
10
dim
800
201
1000
dim
1200
1400
30
1600
250
1000
1250
1500
20
10
40
50
60
dim
dim
DenseNet-kerne size = 3
200
400
600
800
20
1000
2200
1400
1600
00
250
500
750
1000
1250
1500
30
40
60
dim
DenseNet-kernel size
0
400
600
800
20
1000
dim
1200
1400
1800
60
250
500
750
1000
1250
1500
20
30
40
50
63
dim
dim
122

---

Under review as a conference paper at ICLR 2021
Q.7 RESNEXT
ResNext-kernel size 3
20
10
dim
dim
100
40
120
dim
dim
ResNext-kemel size
10
dim
20
300
dim
400
50
500
100
200
300
400
500
50
dim
dim
ResNext-kernel size - 3
dim
296
40
50
100
150
200
250
dim
diim
123

---

Under review as a conference paper at ICLR 2021
Q.8 MOBILENET
MobileNet-kernel size
100
dim
150
dim
300
50
150
200
250
300
dim
dim
MobileNet-kemel size
60
dim
80
20
dim
30
100
120
140
60
25
50
75
100
125
150
dim
MobileNet-kernel size
200
400
dim
20
600
dim
800
200
600
10
20
dim
30
30
60
dim
124

---

Under review as a conference paper at ICLR 2021
R
DETAILS OF EXPERIMENTS
Table 64: Implementation detail for Table 6
Model
Dataset
Batch size
Epoch
Optimizer
schedule
wd
gamma
VGG16
CI
128
180
SGD(0.9)
82/164
1. 1.00E-04
0.1
VGG16
CIFARI00
128
180
SGD(0.9)
82/164
1.00E-04
0.1
VGG16
ImageNet
256
120
SGD(0.9)
30/60/90
1.00E-04
ResNet56
CIFARIO
128
180
SGD(0.9)
82/164
1.00E-04
0.1
ResNet56
CIFARI00
128
180
SGD(0.9)
82/164
1.00E-04 004
0.1
ResNet34
ImageNet
256
120
SGD(0.9)
30/60/90
1.00E-04
0.1
Batch size Epoch Optimizer schedule
train batchsize
number total epochs to run Optimizer
Decrease learning rate at these epochs
wd
weight decay
gamma
learning rate is multiplied by gamma on schedule
There are several additional experiments of Fig. 5 in following figures. They show Spearman's rank correlation coefficient (Sp) among l pruming, & pruning and GM pruning for all the periments in Table 3 These experiments further visualized the strong similarities these pruning methods in different situations.
125

---

Under review as a conference paper at ICLR 2021
FisNeet64-Cifr1000
V3g19-Cifar10
1.0
1.0
0.8
0.-
0.7
0.7
0.6
0.5
0.
Spearmnn
L2/EM
30cammnn
L2/GM
GM
0.3
80%
0.3
80%
92%
92%
0.2
02
140
160
Indes of convolutional layer
Index of convolutional layer
CeexNet-Cffar100
1.0
1.0
0.9
0.9
0.8
0.8
0.7
0.7
66-
0.6
Spearann rank
L1:12
L1:12
0.4
GMLL
GM-L1
0.3
80%
0.3
80%
0.2
0.2
3.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
40
60
80
100
layer
Index convelutional layer
TreResNet110-Cffaff
WRN28- 1RN22-10-Cifar10
1.0
1.0
Spearman Cornalation coofficunns
0.9
0.9
0.7
0.7
0.6
0.5
0.
0.
12:6H
0.4
L2-GM
0.3
80%
0.3
82%
90%
90%
Index of convolutional layer
Indox of convolutional layer
remnext1fxx6dd-Citr11000
1.9
12.-
0.8
Opeermen rank comnetation
0.7
0.5
1142
0.4
GM-11
0.3
50%
90%
0.2
5
30
15
20
30
Index
Figure 27: Network Structure
126

---

Under review as a conference paper at ICLR 2021
Res B4AACC0-Ciarr11000
Reslet1 Aatb
1.0
1.0
63-
0.9
6.8
0.8
Spearman corretaiion
6.7
0.-
0.6
0.5
0.5
0.4
12-CM
0.4
L2-3M
GMLL1
GM411
033
0.3
92%
92%
0.2
02
Index of convolutional layer
Index of convolutional layer
Messlee16--ddamaax--/ftt000
Spearman rank conreltiio coofficions
0.9
0.9
0.8
0.8
6.
0.7
68-
0.6
05
rank
0.5
1141
L1-22
L2:CH
L2:0M
GM411
80%
00%
0.2
0.
120
140
160
20
40
140
160
layer
layer
ResNett1 Siabletta--ddddetttaCCCoooooo
1.0
Spearmen rank comelattio coeffictent
13.
0.8
0.F
0.5
024
GN
GM411
0.3
90%
0.2
160
Index of convorutional layer
Figure 28: Optimizer
127

---

Under review as a conference paper at ICLR 2021
Re5Net164 kaiming unilorm- Clar100
Res 64 Xavier uniform Cifar100
1.0
1.0
0.9
Spaarman rank comreaatoon
0.7
0.7
8.6
0.6
0.5
ronk
0.5
0.4
12:0M
0.4
L2-GM
80%
87%
90%
90%
Index of convolutional layer
Index of comvoluiionll layer
ResNet1 64 Xavier--mai Cofro
ResNet164 wrthoooooo--Cffariii.no
10
correlation confiiient
0.9
comneattin coefteiee
0.9
0.8
0.8-
0.7
0.7
1342
1342
0.4
+ GM-L1
- GML1
0.-
80%
0.3
80%
50%
50%
0.2
0.2
20
40
60
80
100
120
140
160
20
40
60
80
100
120
140
160
Index corvecational layer
Index convelational
Figure 29: Initialization
128

---

Under review as a conference paper at ICLR 2021
SENet (ResNet164) Cifar100
DIANet (Res Cifar100
1.0
1.0
Spearman corretation confiiciunt
69-
0.9
6.8
0.8
0.7
6.-
0.6
0.5
0.5
0.4
12-CM
0.4
12-3M
GMLL1
GML11
0.3
0.3
92%
92%
0.2
02
Index of convolutional layer
Index of convolational layer
SRMNet (ResNet164)
(ResNer114) Cifa1100
Spearman rank cornelation coofficiane
0.9
0.8
0.8
6.7
0.7
68-
0.6
0.
rank
0.5
1141
11-22
L2:CH
L2:0M
GM41
GM411
80%
00%
0.2
0.2
20
120
140
160
20
40
120
140
160
layer
layer
IEBN |ResNeet164) Cifar100
SGENle (leeset1164) Cifar100
1.0
1.0
Spearman rank corneltiion coefficiene
0.9
0.9
0.8
0.8
0.-
0.6
05
0.5
2.4
L2-GM
0.4
L2-GM
GM111
GM4-1
03
0.3
92%
90%
0.2
0.2
20
20
Index of convolutional layer
Indax of convol.tional bayer
Figure 30: Attention mechanism
129

---

Under review as a conference paper at ICLR 2021
SegNet Citycapes
PSPNet Cityscapes
confhilent
mank corretation confficlent
07-
0.7
06-
0.6
0.-
0.5
L142
1142
0.4
- 12-64
0.4
- 12-GM
87%
80%
90%
50%
10c
Index convekutional layet
Irdex convelationl leyet
Figure 31: Other task: segmentation
Faster RCNN |Resteet101] CoCO
Faster RCNN /ResNer1 PASCAL VOC 2007
1.0
1.0
6.9
0.9
Spearman comolution
0.7
0.7
05-
0.6
25
0.5
64-
12
0.4
L2
GMAL1
GM-L1
033
80%
0.3
02%
90%
90%
20
Indox of convolutional layer
Indox of convolutional layer
Visual
1.0
Spearmen rank corre commiation Ineeficient
13.9
0..
0.7
0.6
11-12
2.4
GM-11
03
50%
0.2
10
12
Figure 32: Other task: Faster RCNN
130

---

Under review as a conference paper at ICLR 2021
Fast neural style candy
nestrle style mosaic
1.0
1.0
vank coufficentt
ronk couffictent
0.7
0.7
0.6
0.8
0.5
0.5
6.4
12:0H
0.4
12-GM
80%
80%
90%
90%
Index of convolutional layer
Index of comvolutional layer
Fast-neural-style starry-right.
Fast-neural-styte udnie
1.0
1.0
confieient
cometatton coffteient
0.9
0.9
0.8
0.8
07-
0.7
1342
1342
0.4
GM-L1
+ GML1
03-
80%
0.3
80%
50%
50%
0.2
0.2
10
12
14
10
12
14
Index convelational layer
Index convelational layer
Figure 33: Other task: style transfer
131

---

Under review as a conference paper at ICLR 2021
DOGAN MNIST (Generator)
DCGAN Cifar10 (Generator)
1.0
1.0
Spearman rank comrelation coufficiett
0.7
0.7
2.6
0.6
0.5
0.5
0.4
12-CM
0.4
L2-GM
80%
80%
90%
90%
Index of convolutional layer
Index of comvoltiional layer
DOGAN Cifar100 (Generator)
1.0
Speermen rank cormenation coefitieen
0.9
0.8
0.7
0.5
1142
0.4
GML1
50%
90%
0.2
0.0
0.5
1.0
1.5
3.0
Index convolutional
Figure 34: Other task: GAN
132

---

Under review as a conference paper at ICLR 2021
WRN28 10 Cfar100
WRN28 10 Cfar100 RReLU
1.0
1.0
69-
0.9
0.-
0.8
Spearman corretation
0.6
0.5
05
0.4
12-6M
0.4
L2-3M
GMLL1
GML11
033
0.3
92%
92%
0.2
02
15
15
Index of convolutionll layer
Index of convolational layer
WRN28-10 Cifar100 Dropact
WRN28-10 Autoaugment
Spearman rank correlation coofficiune
0.9
0.8
0.8
6.
0.7
0.6
0.
rank
0.
1142
11-2
L2:CH
12:0M
GM411
80%
00%
0.2
02
15
25
15
25
layer
layer
WRN28-10 Cifar100 Cutout
WAN28-10 Cifar100 Cutmix
1.0
1.0
0.9
0.9
0.8
0.8
Spearman rank Corneltiion
06
0.6
05
0.5
0.4
L2
0.4
L2-GM
GM11
GM411
0.3
0.3
92%
90%
0.2
0.2
15
25
15
25
Indox of convolutionl layer
Indox of convolitional layer
Figure 35: Other task: Regularization
133

---

Under review as a conference paper at ICLR 2021
WRN28-10 10188-00-iifr100
MRN28-10-Cifrr1
1.0
1.0
0.
0.7
0.7
2.6
0.6
rank
0.5
ronk
0.5
6.4
12:0M
0.4
12-GM
80%
87%
90%
90%
Index of convolutional layer
Index of commoluiinnl layer
VRNN2--0--magelle
WRN28-10-MIISST
1.0
1.0
corvelation confeciint
corelation confficiint
0.9
0.9
0.8
0.8
13-12
04-
G2
0.4
GH-L1
-
GM-11
0.3
0.3
90%
90%
0.2
0.2
10
10
Index of conmetutional layer
Index of convolatioall layer
Figure 36: Dataset
VOG19 Cifr100
VGG19 Cifar10
cefficient
Gneermen rank cometetin coefficiiet ceent
0.-
0.9
0.8
0.8
07-
0.7
06-
8.6
11:22
L1-12
G4-13
GM-1
03-
87%
03
80%
02
0.2
0
10
12
14
0
10
12
14
convelutionl layer
convolutiona layer
Figure 37: Batch normalization
134

---

Under review as a conference paper at ICLR 2021
V6611 Votri-Pytorc pre te-ttriined
V6616 VG616-Pytorch pre traitr
1.0
1.0
Spearman corretation coufiiiiunss
coofitclont
0.
0.-
0.7
0.-
0.6
12-6M
Spearmon
L2-3M
GMLL1
GMA11
05
0.
92%
92%
0.4
0.1
Index of convolutional layer
Index of convolationl layer
Pytorch pre-trained
1esyleet8-Pytocch pre-trained
Spearman rank corretation cooffiiinns
0.9
0.9
0.8
0.8
0.7
67-
0.6
rank
0.
06
1142
L1-22
12:CH
L2-0M
GM41
GMAL1
05
80%
00%
0.4
0.
12
10
12
14
layer
layer
Besslee33-Pytorcch pre-trained
ResNet500PPyorcc pre-trained
1.0
1.0
Connelation coofficlonc
69
0.8
Spearman rank corneation
22
0.8
06-
0.7
25
0.6
2.4
L2-GM
Spearann
L2-GM
GMAL1
GM411
0.5
03
92%
90%
0.2
0.4
10
30
Index of convolutional layer
Indox of convolutional layer
Figure 38: Pytorch pre-trained Model
135

---

Under review as a conference paper at ICLR 2021
VGG19 bn Cfar10 scheduie11500222
VGG19 schedulel82-168
1.0
1.0
Spaarman rank coufficiett
0.7
0.7
0.6
0.6
0.5
ronk
0.5
6.4
12:0M
0.4
12-GM
80%
80%
90%
90%
12
Index convolutional layer
Index of comvoluiional layer
VGG19 schedulel60-120)
VGG19 Cifar10 Cosir
1.0
1.0
Spaurman corralation coeffiiiint
0.9
0.9
0.8
0.8
0.7
0.7
0.5
0.5
L142
L142
0.4
0.4
GR0L1
GR0
033
80%
03
80%
92%
92%
0.2
0.2
12
12
Index convolutional
Index comvolttionl
Figure 39: Learning rate
136


Reviews:
[{"review_id": "g08wTdmO6P", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** The goal of the paper is to bring into attention that many norm-based pruning criteria used for structured pruning are very similar, in that their ranking of the redundant filters is highly correlated. The key ingredient is the CWDA assumption that filters in a particular convolutional layer are iid and approximately follow a Gaussian distribution, which is shown based on extensive statistical hypothesis testing. Based on this assumption, they prove that these pruning criteria are roughly the same. \n\n**Pros:**\nI like the nature of the question asked in this paper. These are fundamental questions, like how does the distribution of weights in a particular layer evolve and look like, etc, which do not get as much attention. So in that sense, this paper will be important for the pruning community (where there is a deluge of papers proposing a new criterion) to be aware of, and understanding this aspect they would be able to design a pruning criterion that actually matters. The paper is backed with extensive results testing their CWDA assumption and proving how the criterion are overlapping in their functionality. \n\n\n**Cons:**\nThere is an over-selling of claims, as this \"rethinking\" of criteria is only for norm-based methods, and there are lot of other established methods as well. Then this all holds for layers which are wide enough, but in the end for pruning these layers would not be as sensitive either (and layers like input, last fully-connected, or where there is a transition). \nAlso, at some other places like in the results of different pruning criteria, the experimental methodology and presentation is rather loose. I would have also liked to see at least the empirical results for global pruning because there even some out of these norm criteria might be better than others. \n\n**Detailed comments**:\n\n*Similarity between pruning criteria*:\n\n-First of all, Molchanov et al 2019 https://openaccess.thecvf.com/content_CVPR_2019/papers/Molchanov_Importance_Estimation_for_Neural_Network_Pruning_CVPR_2019_paper.pdf does look at the spearman rank correlations, so I guess you should also mention this in the context of \"blind-spot\". \n\n-Next the criteria that you compare are only based on (absolute/relative) norm of the filters. And, I understand that your main point is showing that many norm based methods are ranking filters in more or less the same way. However, there are several other established and widely used measures (such as Network slimming: Liu et al 2017; EigenDamage: Wang et al 2019). Hence, using the title “Rethinking the pruning criteria…” is too broad of a claim, and should be qualified to include ”norm-based”. \n\n-Also, the paper could have been made stronger by comparing these other measures, empirically at the least.\n\n*Empirical analysis*:\n- Table 4: It is not clear what is the pruning level here?  If you are pruning to small levels, then it does not matter as much of course (also known from results of other papers).  Here, all the methods recover performance after fine-tuning, which shows that this is not yet a good-test bed. The testing should be done precisely in the regime of high pruning ratios (> 90%, 95%), so that you actually see the difference. Can you perhaps provide results in such settings?\n\n- Further, can you share some correlation numbers for global pruning? I know that your focus is layer-wise pruning, which is perfectly fine. However, global pruning methods outperform layer-wise methods (at least this is known for unstructured pruning), and my intuition is that it might also hold for structured pruning (also, global methods would be better from the large minimum norm requirement of He et al 2019).\n\n- Table 1: I do not see much worth of showing that top 8 filters (out of > 100s) are similar for all criterions, as the few most redundant filters will be identified almost equally well by any sensible criterion. \n\n*Other important questions:*\nHow do you choose \\sigma_0 in testing? What is the smallest value that you can set, such that most of the tests pass? \n\"Need to be trained well enough\": Why is this a concern?\n\n*Related work discussion*:\n-Pruning: References go to as back as LeCun et al 1990, Hassibi & Stork 1994, etc. Otherwise it gives an impression that this whole area has just started.\n\n-\"one-shot pruning”: The definition of one-shot does not include fine-tuning. I think saying, \"one-shot pruning + fine-tuning”, is more accurate \n\n*On writing:*\n- There are actually quite a few typos, but still not too bad yet something not be proud of. Towards the last couple of pages, there are many more typos.\n\"about the pruning criteria as follow”; \n\"when the network exists too many filters” -> \"when the network has too many filters;\n“First, two theorems are shown” -> “First, we show the following two theorems\"\n\n- At many places, you have subscripts instead of superscript: \"In ith layer”\n\n- \"In CVPR 2019 Oral”: I don’t think this is a good way of writing or emphasising a paper. Otherwise, we will have things like, “The best paper award winner XYZ et al says, ….”, “The paper XYZ et al, which features a Nobel prize winner,  ….”. I hope you get the point. \n\n**Score**: Right now, owing to the above points, I have some reservations. If you can answer these questions better, I would happily increase my score towards acceptance. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fT5aAdQQ6DU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the current limitations of existing magnitude-based pruning methods. First, the paper focuses on the similarities between three methods and then focuses on the redundancy in large networks. The paper also analyzes the weight distribution for a well-trained network and propose CDWA as a way to prove this distribution.  My main concern with this paper is the contribution to the field. \n\nTwo main comments:\n1. If I understand correctly, the paper focuses on the distribution at the parameter level and shows it follows a Gaussian Distribution. If that is the case, that is a well-known result especially when the network is trained with a weight decay to regularize.\n\nThen, the paper shows the similarities between three magnitude based methods (L1, L2 and GM). I guess I am not very surprised the first two are similar in the structure and slightly different in performance as that is expected. The rank correlation is interesting tho. Would be more interesting to show the same comparison with importance based methods or other methods not using the magnitude only. \n\n2. In terms of redundancy, I am not sure I understand the point there. It is also known that training of neural networks leads to redundancy and that is the main reason for some particular postprocessing or compression algorithms based on that (see Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation -- Denton et al; or Compression aware training of Neural Networks -- Alvarez and Salzmann; and some others in the literature). Magnitude based approaches do not consider the interaction between parameters and therefore still lead to redundancy. Methods aiming at reducing redundancy usually do not provide good compression ratios as the architecture is harder to prune. \n\nThe paper also suggests l2 is regarded as importance. I think that should be reconsidered as L2 is the magnitude and there are many works using other criteria referred as importance (for instance gradient *magnitude). \n\nAll in all, not sure the real contribution to the community. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZYHEgjX4lz7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n \nThe paper discusses various baseline scoring mechanisms used for filter pruning that are norm-based and finds that none of the scoring mechanisms are particularly effective at pruning filters from CNNs. Moreover, all methods seem to perform very similar to each other. These conclusions are based on a theoretical (and experimental) analysis of the various scoring mechanisms under the assumption that trained filter weights follow a Gaussian-like distribution, which reveals that in this case the scoring mechanisms are insufficient to reliably discern the importance of filters since the resulting scores are very similar. The Gaussian-like assumption, which is termed \"Convolution Weight Distribution Assumption\" (CWDA), is validated through a large range of experiments on different architectures and data sets.\n\n## Score\n\nI enjoyed reading the paper and I found the resulting conclusions really interesting since it sheds light on some of the mechanisms behind filter pruning methods, which are not as well investigated as mechanism behind unstructured (weight) pruning. \n\nNonetheless, I think there a is couple of things that the authors should consider before the paper is ready for publication, namely the inclusion of more advanced filter pruning techniques [1, 2, 3] in the discussion and also a set of recommendations for the community that could help design more informative scoring mechanisms for filter pruning in future work. \n\nMore details are provided below. \n\n## Ways to Improve My Score\n\nTo improve the quality of the paper, I believe the authors should include the following two aspects: \n1. Additional filter pruning methods: There has been significant progress in recent years in developing more advanced filter pruning techniques, such as [1, 2, 3] including techniques that are data-dependent, i.e., consider the layer activation in computing the filter score. Do these methods suffer from the same inability to discern the importance of filters? Or are these methods already a stepping stone in the right direction towards building more meaningful pruning criteria? Having the baseline methods alone is not sufficient in order for the community to build on top of this work in a meaningful way. \n2. Recommendations for improved filter pruning methods in the future: As it stands now, the paper provides little guidance what future research into filter pruning should consider when designing novel pruning methods. I think this would be crucial though in order for the community to appreciate the paper. This is also linked to the first point I mentioned since some of the existing advanced filter pruning methods potentially already contain at least partial solutions to the problem presented here and could thus guide future research as well. \n\nI mention additional feedback that affects my score in the \"Weaknesses\" section. The \"minor feedback\" contains feedback that did not significantly affect my score.\n\n## Strengths\n\n* I think the conclusions are very interesting and confirm my own experience with filter pruning. A lot of the baseline approaches (i.e. the norm-based approaches) seem to work more or less the same, especially in a standardized prune+retrain setting.\n\n* The CWDA assumption is verified using a large set of experiments and the authors also discuss the limitations of the assumptions. This is really encouraging to see and I appreciate the rigor. \n\n* The theoretical analysis based on the CWDA assumption seems technically sound and is also back up by appropriate experiments. Figure 1 is really interesting, in particular, as it shows how small the variance of norm distribution is.\n\n## Weaknesses\n\n* The lack of additional filter pruning methods as stated above. The inclusion of those is crucial I believe as it may lead to some really interesting additional conclusions.\n\n* The authors discuss the limitations when CWDA does not hold. I wonder if they could dig a little deeper to see whether in those cases norm-based filter pruning is more effective at identifying important filters. \n\n* The writing in parts is somewhat confusing and as a results it's sometimes hard to discern the meaning. I believe the paper would benefit from an additional revision to ensure that it is accessible to a broader audience.\n\n## Other Minor Feedback\n\n* Typos:\n  * in (2) of the contribution list: \"...when the network exists too many filters.\"\"\"\n  * Page 7, bottom: \"..., which shows that the GM method also exist the same problem ...\"\n  * Page 8, top: \"In CVPR 2019 Oral, ....\"\n  * Page 8, Section 5.1, (1): \"...converges to a bad local minimal ...\"\n  * Page 8, Section 5.1, (2): ...are almost the layers that are ...\"\n  * Please check for other typos as well.\n\n* Is there a difference between $F_{i,j}$ and $F_{ij}$? I saw both notations in the paper and wasn't sure.\n\n* I didn't carefully check all the math but it seems to be reasonable. I think it would help potential readers to include high-level summaries of each theorem in the main part of the paper. That way the paper could be made more accessible to a larger audience.\n\n* Maybe it would be more useful to include the results of the statistical tests as part of a code repo that will be released upon publication. It feels quite difficult and slow to navigate many pages of tabulated results.\n\n * Maybe sort the indices in Table 8 differently. Then it might be easier to see the overlap between the chosen filters for each method.\n\n * What about fully-connected layers? Could the authors observe something similar in those cases? Note that \"filter pruning\" can also be applied to fully-connected layers where neurons corresponds to filters. \n\n## References\n\n1. [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](http://openaccess.thecvf.com/content_iccv_2017/html/Luo_ThiNet_A_Filter_ICCV_2017_paper.html)\n2. [Provable Filter Pruning for Efficient Neural Networks](https://openreview.net/forum?id=BJxkOlSYDH)\n3. [Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection](https://proceedings.icml.cc/paper/2020/hash/8b6a80c3cf2cbd5f967063618dc54f39-Abstract.html)", "extra_scores": {}, "extra_text": {}}, {"review_id": "HINHxUYrpO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### About weight distribution assumption\n\nThe authors assume that all well-trained filters are i.i.d and follow a Gaussian-alike distribution, plus the co-variance matrix is a diagonal matrix. In other words, each weight within the filter follows a Gaussian distribution and is i.i.d because their co-variance equals to 0. **PS: cov(x,y) = 0 is equivalent to independence for Gaussian distribution, though the conclusion does not hold for general cases.** \n\nHowever, CNN works because it takes advantage of the locality of images. In other words, these weights in filters should be highly correlated to extract features (though may not be linearly correlated), which seems to contradicte the assumption.\n\n\n\n### About contributions of the paper\n\nAs I understand, the contributions of the paper are mainly two folded: 1) Weight distribution assumption and its correponding conclusions. 2) The paper reveals the similarity of some previous pruning algorithms and finds they may not be good criteria.\n\nI am kind of worry about the reasonability of the first contribution (Please see last section for details). The second contribution does not seem to be sufficient for a formal paper. We may prefer some more materials. For example, why these findings are important? How do they inspire subsequent research? What should we do to avoid the problem in pruning? etc.\n\nTotally speaking, I think the authors analyze the problem from a very interesting perspective. It will become a good paper with some more improvement, but I do not think it has been good enough for publication now.\n\n### Updates after rebuttal\n\nAbout the contributions of the paper, it does provide some interesting points of view.\n\nThe authors revise the paper and modify their basic assumption. However, it seems to be a hollow assumption without any guarantee. The concerns are as follows: First of all, why the authors claim it as a diagonal matrix when actually observing it as a block-diagonal matrix? Secondly, considering the updated assumption is only tested on one layer of VGG and ResNet, I am really worried about its generality and reproducibility.\n\nTotally, the weight distribution assumption is the cornerstone of the paper, but the assumption seems to be not convicing. Specifically, the assumption in the first version of the paper is a paradox to some extent, and the revised version casually modifies the assumption without too much verfication. I have to decrease my rating to 4.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper works towards analysis to understand the difference -- and primarily the lack thereof -- between different pruning methods. The central observation is that the convolutional filters in a layer are not strongly correlated and -- if the weights of the layer are taken as a matrix -- then the covariance matrix is block diagonal. 

Extending this objective the regime of a large number of filters, then the matrix is approximately diagonal and all weights are -- approximately Gaussian and i.i.d. The point of this analysis is that under this assumption, norm-based metrics, particluarly $\ell_1$ and $\ell_2$, behave quite similarly.

The pros of this paper are the extensive evaluation and -- after revisions -- relatively clear text. The core analysis is nice to have elaborated in detail in the community.

The primary con of this paper is, as the reviewers point out, that there are limited conclusions to take away from this work. Specifically, a plausible default hypothesis is that different pruning criteria result in different pruning decisions. From the results in this paper, that still seems to hold with -- exception of the norm-based metrics. So, while this work does demonstrate that these norm-based metrics are relatively similar -- a nice clarification to see in the community -- the work offers limited comment on the broader space of pruning metrics.

My recommendation is Reject. Despite the strong empirical evaluation, the ultimate results offer limited clarification on the similarity of pruning metrics.

Decision: Reject