Rotate the ReLU to Sparsify Deep Networks Implicitly
Authors: Nancy Nayak
Sheetal Kalyani
Venue: Accepted by TMLR ()

Abstract: Compact and energy-efficient models have become essential in this era when deep learning-based solutions are widely used for various real-life tasks. In this paper, we propose rotating the ReLU activation to give an additional degree of freedom in conjunction with the appropriate initialization of the rotation. This combination leads to implicit sparsification without the use of a regularizer. We show that this rotated ReLU (RReLU) activation improves the representation capability of the parameters/filters in the network and eliminates those parameters/filters that are not crucial for the task, giving rise to significant savings in memory and computation. While the state-of-the-art regularization-based Network-Slimming method achieves $32.33\%$ saving in memory and $26.38\%$ saving in computation with ResNet-$164$, RReLU achieves a saving of $35.92\%$ in memory and $25.97\%$ in the computation with a better accuracy. The savings in memory and computation further increase by $64.67\%$ and $52.96\%$, respectively, with the introduction of $L_1$ regularization to the RReLU slopes. We note that the slopes of the rotated ReLU activations act as coarse feature extractors and can eliminate unnecessary features before retraining. Our studies indicate that features always choose to pass through a lesser number of filters. We demonstrate the results with popular datasets such as MNIST, CIFAR-10, CIFAR-100, SVHN, and Imagenet with different architectures, including Vision Transformers and EfficientNet. We also briefly study the impact of adversarial attacks on RReLU-based ResNets and observe that we get better adversarial accuracy for the architectures with RReLU than ReLU. We also demonstrate how this concept of rotation can be applied to the GELU and SiLU activation functions, commonly utilized in Transformer and EfficientNet architectures, respectively. The proposed method can be utilized by combining with other structural pruning methods resulting in better sparsity. For the GELU-based multi-layer perceptron (MLP) part of the Transformer, we obtain $2.6\%$ improvement in accuracy with $6.32\%$ saving in both memory and computation.

Published in Transactions on Machine Learning Research (05/2024)
Rotate the ReLU to Sparsify Deep Networks Implicitly
Nancy Nayak
e ce17d408 0e
Department of Electrical Engineering Indian Institute of Technology Modras, India
Sheetal Kalyani
skalyani@ee witm ac
Department Electrical Engineering Indian Institute of Technology Madras, India
Reviewed on OpenReview: https: // openreview. net/ formf id- NzyOXmCPuz
Abstract
Compact and energy-effcient models have become essential in this era when deep learning- based solutions are widely used for various real-life tasks. this paper, we propose rotating the ReLU activation give an additional degree of freedom in conjunction with the appro- priate initialization of the rotation. This combination leads to implicit sparsification with out the use of regularizer. We show that this rotated ReLU (RReLU) activation improves the representation capability of the parameters/file /filters in the network and eliminates those parameters/ filters that are not crucial for the task, giving rise to significant savings in mem- ory and computation. While the state-of-the-arl regularization- -baabed Network-Slimming method achieves 32.33% saving in memory and 26.38% saving in computation with Res.Net- 164, RReLU achieves a saving of 35.92% memory and 25.97% in the computation with a better accuracy. The savings in memory and computation further increase by 64.67% and 52.96%, respectively, with the introduction of L1 regularization to the RReLU slopes. We note that the slopes of the rotated ReLU activations act coarse feature extractors and can eliminate unnecessary features before retraining. Our studies indicate that features always choose to pass through a lesser number of filters. We demonstrate the results with popular datasets such as MNIST, CIFAR-10, CIFAR-100, SVHN, and Imagenet with different archi- tectures, including Vision Transformers and EfficientNet. We also briefly study the impact of adversarial attacks on RReLU-based ResNets and observe that we get better adversar- ial accuracy for the architectures with RReLU than ReLU. We also demonstrate how this concept of rotation can be applied to the GELU and SiLU activation functions, commonly utilized in Transformer and EfficientNet architectures, respectively. The proposed method can be utilized by combining with other structural pruning methods resulting in better spar- sity. For the GELU-basod multi-layer perceptron (MLP) part of the Transformer, we obtain 2.6% improvement in accuracy with 6.32% saving both memory and computation.
1 Introduction
Machine Learning has garnered significant attention in recent times for achieving superhuman-level perfor- mance in wide range of problem domains, encompassing real-world applications and intricate tasks. This remarkable progress is primarily attributed to the utilization of Deep Neural Networks (DNNs) endowed with millions of parameters. In the realm of image classification, Convolutional Neural Networks (CNNs) such as AlexNet (Krizhevsky et al. 2012) with 60 million parameters and VGGNet (Simonyan & Zisserman, 2014) with 138 million parameters have set the benchmark. However, deep networks are prone to overfit- ting and encounter exploding and vanishing gradient problems. To address this issue, [e et al. (2016a) introduce ResNet, neural architecture that surpasses VGGNet in depth and can be extended include thousands of layers while maintaining relatively lower computational complexity. Zagoruyko & Komodakis (2066h illustrate that achieving marginal improvement. in ResNet. performance necessitates doubling the

---

Published in Transactions on Machine Learning Research (05/2024)
number of layers. Consequently, they introduce WideResNet (WRN) as an alternative approach to enhance performance by widening the ResNet architectures instead of increasing their depth.
Increasing depth or width to improve accuracy also leads to storing a significantly higher number of parame- ters. For cooourrc-constraineed energy-effcient green networks, the major concerns regarding the deployable network are (i) model size, (ii) run-time memory, and (iii) computation in terms of floating point opera- tions (FLOPs). In order to reduce the complexity without degrading the performance, different compression techniques have been introduced for example, low-rank approximation (Denton et al., 2014), quantization
(Han et al. 2015), binarization (Courbariaux et al. 2015 Rastegari et al. 2016), and transfer learning (Kim
ct al. 2020). Numerous studies have been conducted (Han et al.) 2015 Molchanov et al.] 2016 Li et al.
2017 He et al. 2017 Liu et al.] (017)) that put forth methodologies involving the pruning of connections, weights, and filters in deep neural networks. Typically, regularization techniques are employed to induce
sparsity in weight matrices as a means of facilitating efficient pruning, as demonstrated in previous studies (Han et al.) 2015 Wen et al., 2016 Liu ct al., 2019). However, these techniques primarily target memory reduction without significantly impacting computational efficiency. To address the dual objectives of memory and computation savings, group sparsity- based regularization approach has been introduced in the context of CNNs in various investigations conducted by Meier et al. (2008); Wang et al. 2017); Nayak et al. (2020):
Zhou et al.
2016) Liu et al. (2015): Scardapane et al. (2017).
In Liu et al. (2017), L1 regularization is imposed on the scaling factors Batch Normalization (BN) layers, making it easy to implement without introducing any changes to the existing CNN architectures, By using the idea of the L1 penalty on BN scaling parameters, MorphNet (Gordon et al. 2018) iteratively shrinks and expands a network, followed by retraining. It shrinks by a resource (FLOPP memory)-weighten sparsifying regularizer on activations and expands by uniform multiplicative factor on all the layers. ThiNet is proposed by Luo et al.] (2017) where the algorithm first trains a model, and then, by a greedy algorithm, it determines the filters that can be removed without affecting the performance. In Liu et al. (2019), the authors claim to perform better when the pruned networks are further retrained from scratch. Yang et al. (2020a) propose decomposed training that reduces the rank of the matrix/kernel via SVD training, achieving a higher reduction in computation load under the same accuracy. However, it requires separate regularizers for the orthogonality of the singular vectors and sparsity of the singular values.
Until now, researchers have used regularizers to impose sparsily, forcing the deeper/ wider networks to use fewer filters without degrading the performance. However, can this sparsity be achieved intrinsically without a regularizer? Motivated by this question, in this work, we propose a novel activation function Rotated ReLU (RReLU), where one of the two halves of a ReLU can rotate to have any slope, whereas the other half has a slope as zero. This slope of the rotating half is trained along with the other network parameters. The proposed method is different from dynamic ReLU (Chen et al. 2020) because, in dynamic ReLU, both halves have different slopes determined by euper-function dependent In RReLU, the slopes trained along with the network, each of the filters in the convolution layer with a significant RReLU slope attains more representation power. During the training, some of the RReLU slopes corresponding to the excess filters go to very small values. Therefore, introducing rotation to ReLU has a two-fold advantage- (i) improves representation power corresponding to each significant filter and, therefore, (ii) encourages sparsity. This efficient sparsifying method allows pruning filters efficiently without using any regularizer. Note that rescarchors have come up with different variations of the activation ReLU Nair & Hinton 2010p such as leaky ReLU Maas et al.) 2013), PReLU (He et al. 2015). Exponential Linear Units EELU) Clevert et al. 2015). randomized ReLU (Xu et al. 2015) and randomized leaky ReLU (Srivastaval ct al. 2014h to improve the performance. Dynamic ReLU CChen et al. 2020) is proposed to improve the representation power of deep networks. This activation function is characterized by piecewise linearity with varying slopes, and its parameters are determined by a hyper function over all input elements. Hendrycks & Gimpel (2016) propose Gaussian Error Linear Units (GELU), which is shown to improve the performance compared to ReLU and ELU. However, none of the above induces sparsity.
all the input elements.
We discuss RReLU in detail in Sec. To demonstrate the power of RRRLU, we choose ResNet-bbsed archi- tectures, which are extensively used in diverse areas such computer vision, natural language processing, security, healthcare, remote sensing, and wireless communication (Guo et al.) 2020 Nguyen et al. 2021
Alrabeiah et al. 2020
Mañas et al. 2021 Shankar et al.) 2021 However, irrespective of the architecture,
2

---

Published in Transactions on Machine Learning Research (05/2024)
any model that has ReLU can be retrained by replacing the RcLUs with RReLUs. Further, we show that the use of rotation in activation is not only limited to ReL.U but can be applied to other activations as well, like Gaussian Error Linear Unit (GELU) [Hendrycks & Gimpel 2016) that are used in emerging transformer architectures. Rescarchers use GELU instead of ReLU for many applications to achieve sate--o-ttheaar results using Transformers, including Natural Language Processing (Radlord al. 2018) and Vision Trans- formers (Dosovitskiy al., 2020). By proposing rotated GELU (RGELU) in Vision Transformer, we not only show that sparsity is induced, but the top-1 validation accuracy improves, too, thus making it suitable across different architectures. This makes the RReLU, in our opinion, potent tool for compressing neural networks. Our key contributions are follows.
We propose a Rotated ReLU activation along with an initialization scheme for the rotation. Wo show that this combination is a powerful tool to improve the representation capability of the filters, resulting in intrinsic sparsification. For example, the WRN-40-4 architecture with RReL.U saves a memory of 63.37% and FLOP-coure of 69.7% without any loss in performance.
RRcLU can work with other compression techniques. An example considered in our work imposing L1 regularization of batchnorm parameters. Using RReLU with the L1 regularization of batchnorm scaling parameters achieves better sparsity and accuracy.
RReLU provides better adversarial robustness when compared with ReLU in networks such ResNet-56.
We validate the results with extensive simulation ! with various architectures such as fully connected neural network (FCNN), ResNet, WideResNet, and Transformer using wide variety of datasets such as MNIST, CIFAR-10, CIFAR-100, SVHN, and large-scale image dataset like ILSVRC-2012 (ImageNet-1k).
Finally, show that the idea of rotation is not limited to ReLU and can be applied to activations like GELU used in most popular Transformer architectures.
2 Why and how to rotate the ReLU?
ReLU is a widely used activation function for introducing non-linearity in DNNs. The output of 1hh layer of a DNN of depth L is given by h = (Fii;; Wi)), where JE is either the weights multiplication operation for FCNN or the convolution operation for a CNN followed by batchnorm layer, W, are the trainable weights filters respectively of layer and is the non-linearity
the activation function. The ReLU activation (Glorot et al., 2011) axx) = max x) is composite of piecewise linear approxination, hence
can approximate wide range of functions (Huang 2020 Hanin 2019 Cybenko 1989: Barron 1993). The
non-zero-slope part of ReLU has fixed slope of 1. In this paper, we increase the degrees of freedom of ReLU
+1,00
b>
+1,000
0.5
-1020
0.5
a(2; b)
1.00
((3;;))
-1
-05
0.5
-0.
0.5
(a) (x;; a, b) 6max(0, a.z))
(b) (2;; b) bmax(0,
Figure 1: Rotated ReLU activation
!Code: http::///tttt....c//aa/aaaaaaaaa
TMLR
3

---

Published in Transactions on Machine Learning Research (05/2024)
by rotating the part of the activation where the slope is one, with non-zero slope, thereby increasing the representative power of ReLU. The proposed RReLU activation a(z; (,,) b) = b max(0, ax) has two trainable parameters E (+1,-1} and b E R. RReLU outputs zero for either the positive part or the negative part of its domain, and for the remaining part of the domain, it produces a linear output. The slope of this linear output depends on the values (a, b). With these values of and b, various possible RReLUs can be realized. Fig. la shows all possible cases of RReLU activations with different b).
Employing an RReLU with shared (a, values for all features across all layers is quite similar to using regular ReLU in the architecture, the roles of a and b are compensated by the network weights weights/filters, /filters. Therefore, to fully exploit the power of rotation, instead of using a single (a. across all features in all layers, (a b(() is used for 7th feature x.(1), So, the output of the 9th layer is given by
h141 ax, bi) = by max a1.x:)
(1)
where the feature input to the RReLU is F(h;; Wi). As x is the output from the linear linear/con /convolution layer, it is clear from equation I that any numeric value can take can be adjusted using the weights/filee ffitees itself. Therefore, only two different types of RReLU corresponding to b 2 0 and b < 0 are sufficient for the proposed technique and are shown in Fig lb With these RReLU activations, the output of the 1th layer is denoted as
h = (xi; bi) = b, max(0,
(2)
When the input h is fed to the t layer within a residual unit, the output the unit with a ReLU activation can be represented by
h = ma h + n+ Conv(max(,, (Cnnn, Wi) + Br): W4+1) + B1+1),
(3)
where " and Br are the batchnorm scaling and shifting parameters, respectively. Incorporating by in equa- tion 3 increases the capacity of the piecewise linear approximation to represent the underlying function with more lincar segments. The output of the residual unit with an RReLU activation is given by
h = bi41 max(0, b, + M+1 Conv(b, max(0, Conv((h; Wi) + BB); W4+1) + B141).
(4)
Due to the freedom in choos- ing the value of the slope, the architecture with RReLU needs lesser trainable weights achieve a similar perfor- mance ReLU. The degree of rotation of the non-zero- slope segment of RReLU determines the degree to which the feature is transmitted through the 1th layer. Once fully trained, if the value of bf+t for 7th feature is compar- atively less, then it implies that the convoluted feature 21 (4) not essential for the task and can be ignored keep- ing the performance intact. It has been observed that with the introduction of RRcLU, for deeper/wiid wider archi- tectures, many of the by go to zero, resulting in signif- icant amount of structural sparsity in the model. Note that the value of h.()) is insignificant when 6(")) is approx- imately equal to zero, and this is logically consistent as X; is batch normalized before passing it via RRcLU, and therefore X₁ can take only a bounded value. We do not use magnilude-basee pruning on weights/kernels instead, RReLU inherently learns sparsity, and we prune weights/filtes based on the corresponding RReLU slope values. Now, we describe how rotation helps to achieve lower complexity with lesser computation. We discuss the results using FCNN and ResNet architectures, where the primary operations are matrix multiplication and convolution with skip connection, respectively.
Sparsity and RReLUU:
W
E R hiichess
x
h E R
hq
h E Rha
2
[Fh, W.):
Figure 2: RReLU in FCNN. The figure is bet- ter visible in color. F(.) represents linear op- erations with learnable parameters W,, The RReLU slopes b;, too, are the learnable param- eters.
4

---

Published in Transactions on Machine Learning Research (05/2024)
t layer
- 1)th layer
c[+2
b
6/41
b²
x
c++2
b
b
X
b
h
W
W
F(hm1, W(n1)
hy
F(b,, W,)
Figure 3: RRRLU in CNN. The figure better visible in color. F(.) represents convolution operation with learnable parameters Wi. The RReLU slopes by, too, are learnable.
Let the input and output for the h layer be h, and h respectively and let the number of elements in h be h441- The RReLU activation with trainable slope by E RRh++ is applied on F(h,, Wt). After training, element b1(0) is compared against a threshold and made equal to zero if 6.0)) is less than Ç, indicating that the 1th column of the weight matrix WT can be ignored. The threshold Ç is set by cross-validation. This idea pictorially illustrated with an example in Fig. 2 Here, F(.) is matrix multiplication and denoted by x. In the example in Fig. 2 after the training, the second element b; converges to zero, i.e., b/2) + 0. So the 2nd entry in h is negligible as the RReLU does not allow the second element of X₃ to pass; clearly, the 2nd column of W, can be ignored.
RReLU in FCNN:
RReLU in CNN: Let the number of channels out from the convolution operation at the tt layer be C--1 The RReLU activation with trainable slope by Ran is applied on F(h; W.). After training, b(") is compared against a threshold Ç, and if b( < ( then the 7th feature at the 1 layer can be ignored. Therefore, at the 1th layer, 4h filter out of a set of q+1 filters can be ignored and at 11 layer, 7th sub-filter from each of q+2 filters can be ignored. This is explained pictorially with an example Fig. where é denotes the convolution operation. At the (th layer, four 2D features are convolved with three set of filters denoted by Wt with four sub-filters each, resulting in h4+1- The first 2D feature (yellow) of h141 is calculated by convolving each of the four 2D features in h, with corresponding sub-filters of the first filter (yellow) and adding them. The same occurs at layer ++ 1. Let, after the training, the slope (/3) + 0, then 3rd feature h is also close to zero. Tn this case, the 3rd filter of W, and the 3rd sub-filter of every filter W4T can be ignored (grey).
2.1 Enhanced Filter Representation with RReLU and Batch Normalization Scaling Parameters
Batchnorm scaling and shifting parameters "i and 3r, respectively, in equation 3 are used to scale and shift the batch normalized features to restore the representation power of the network after normalization TToffe Szegedy 2015). The parameters are learned along with the original model parameters. Liu et al.) (2017) propose technique, namely network-slimming for structural pruning, which involves the utilization of the L1 norm on the batch normalization scaling parameters n, VI E L. This approach encourages the Y parameters to approach zero, allowing for the corresponding filters to be discarded. Our proposed method provides increased ficxibility to the 7 by introducing slopes b as shown in equation 4 which enhances the representation power corresponding lo every filter in all the convolution layers. In the context of this study, an increase in filter representation power signifies the achievement of comparable accuracy while utilizing reduced number of filters. As shown in equation 4 when some of the slopes (bx) go to zero, then corresponding BN parameters (n) go to zeros too. In case some elements of b; and the corresponding elements of 71 do not go to zero, then the elements of by give more fleribility to batchnorm parameters
5

---

Published in Transactions on Machine Learning Research (05/2024)
resulting in more representation power to the filters, and a result, many filters become unnecessary. The degree of sparsity is further enhanced through the application of I, regularization to the RReLU slopes. To delve deeper into this, we need a more thorough analysis. Liu et al. (2017) minimizes the L1 norm on the 71 given by Σi=E (it that will force each element " to take lower values and therefore the network may lose its representation power. On the other hand, while minimizing L1 norm RRcLU slopes, although every element of by compelled to adopt smaller values, the lerm 11th b1 remains unconst because the regularization is not directly applied to the scaling parameters, n. Thus, RReLU possesses the potential to augment the reprosentational capabilitiis of every filter within the convolutional layers to greater extent than ReLU in scenarios both with and without the application of regularization. Moreover, we have substantiated this assertion through empirical evidence, as detailed in Section 5.5
2.2 Compatibility of RReLU with other algorithms Featuring ReLU
RReLU is applicable to other architectures as well, where ReLU is used as an activation function. The output of the residual unit of PreAct-ResNet (He et al. 2016b) is given by
2 z + Cony(max(O,, 7+1 Conv(max(O, mz + Br); Wi) + B(+1): Wi+1).
(5)
When ReLU is replaced by RReLU, the same is represented by
ZI+2 = Z + Conv(bi+1 max(0, Y+1 Conv(b, max (0, 722 70 + Bi); Wi) + Bi+1); W411..
(6)
Similarly, the proposed RRcLU activation can be used along with other compression techniques as well, such 2017] and MorphNet (Gordon et al. 2018). The main idea of ThiNet (Luo el al. 2017) is to approximate the output of a layer by using a subset of the filters in that layer to remove the rest of the filters safely. For filter selection, they use a data-driven groedy algorithm. One can use RRcLU instead of ReLU in the ThiNet architecture to study if one can prune more filters with RReLU, which could be an interesting future study. For shrinking network, MorphNet (Gordon et al. 2018 considers a sparsifying regularizer G()) based on discounting the use of batchnorm parameters,
as ThiNet (Luo et al.
0111
O-11 husl
9() 1=1
M-L
B B +
Σ A
Σ
(7)
1=
Here I and On denotes the number of input and output channel, respectively, for the t layer; (BL.)) is an indicator function which equals one if the 7th input (jth output) of layer L is not zeroed out. The same can be done with RReLU in the architecture as well, where the same regularizer G() can be applied. Note that, here RRcLU slopes provides better representation power of every filter in the architecture.
Yang et al. (2000a) propose singular value decomposition (SVD) training to explicitly achieve low-rank DNNs. The authors use that the Li regularization on S can be represented as ( = Σn 8t. Therefore, all the singular values s;, Vi shrink simultaneously to take lower values. To mitigate the proportional scaling associated with the Ln regularization, the authors uses Hoyer regularizer (Hoyer) 2004) defined as I (s) = ||||// = E, /s. However, one can replace the ReLU activations with the proposed RReLU activation while still conducting SVD training to get another degree of freedom as the elements of b, can take any values. This will provide higher representation power to every filtes which may increase the sparsity even more.
regularizers on the singular value vector s of each layer, such
A channel exploration method has been recently proposed in Hou et al. (022 which repeatedly prunes and regrows the channels throughout the training process. Channel pruning selects the most represent at columns from Wt that can reconstruct the original channel matrix with minimal error, therefore formulated as a column subset selection (CSS) problem. The regrowing stage helps to dynamically re-allocating the number of channels across all the layers under a global channel sparsity constraint. CHEX is shown to have excellent pruning capability, and RReLU combined with CHEX can further improve the sparsity. Therefore, the proposed RReLU need not replace the current sparsifying techniques but can also complement them further to enhance sparsity.
2nn this work, every filter represented as a combination of orthogonal singular vectors and singular values.
6

---

Published in Transactions on Machine Learning Research (05/2024)
2.3 Applicability of the idea of rotation to GELU activations
Till now, we have discussed how RReLU can applied to replace ReLU in sparsification techniques. Now, we clucidate how the concept of rotation extends to GELU (Hendrycks & Gimpel 2016). which is defined by the equation:
1 OGELU(X) xP(X S 2z) =d(x)=(())))))))))))(()))) = 2. 1 erf( (22
(8)
It is well-established that Transformers exhibit improved performance when employing GELU activations [Rafford et al. 2018). To introduce varying slopes in the GELU activation, we propose the rotation of the GELU activation, denoted as RGELU, as follows:
X
h = ORGELU bi) = bixi 1 + erf(
(9)
Therefore, any architecture that uses activations similar to ReLU can use the rotated version of the activation of variable slope. For example, SiLU used in EfficientNets can be replaced by rotated SiLU (RSILUU using the above idea of rotation. In Fig. we show how RGELU can have any slope compared to GELU, which fixed at b = 1. In Sec. we provide results with Vision Transformers and EfficientNet the Ima- geNet dataset that prove the intrinsic sparsification properties of RGELU and RSiLU, respectively.
2
GELU, b
RGELU W/ b 0
RGELU W/ 1
3 Saving memory and FLOPs using RReLU
3
2
X
For an FONN, considering W, E xh, the total number of parameters at the 1hh layer is given by huh++1 During the forward pass, the number multiplications and additions is given by h4 and (h4-1 - 1) respectively; therefore, a total of x 1h FLOPs are needed. If values of by are zero, then be ignored. So, the memory saving at
Figure 4: (X;LU(x;))
Rotated GELU activation
columns of Wt the 1 and the (1++1) 11th layer are h and nh441 respectively. The savings in FLOPs for the tth and the (1 (1th) layers are approximately 2h and 2nh141-
The savings are even more for the CNN-based ResNets. The memory and FLOPs for ID-CNN and 2D- CNN were given in Vikas et al. (2011) and Shankar et al. (2021), respectively. Considering L layers in a ResNet architecture, the trainable parameters are given by Φ = [W..... Wz} where Wt 11 is the filter for the 1th layer of 2D CNN. Here, G and G+1 represent the number of input and output channels at the tth layer, respectively, and k the dimension of the filter. The input and output for the (th layer are by E Raxa"" xhf and h1f1 E R844 xh",, (R",, xht respectively. Here, (hy, h() and (M% h/1)) are spatial dimensions (width, height) of the input and the outpul respectively. The lotal number of multiplication for the 1th layer is qqk h4 and the total number of addition for the 7th layer is 941 The total count of FLOPs for the 1th layer of a real-valued 2D CNN is the (q 1) (A22 - 1) x hit Th summation of the number of multiplication and addition that is roughly twice the number of multiplication given by 2 hy 1 1441
1111 Because the addition of residue in the ResNet structure, there a residual
connection at the tth layer, an extra of 14 additions are required. At 1th convolution layer, if the input to RReLU at the 1hh layer has eiu channels and entries of b are insignificant, then only (Q+1 n) channels remain significant. This needs to save only (a+1 n))ckk2 parameters for the 7th layer. The computation for these reduced (c4+ -n) filters at the (th layer is only 2 hy hhf (c44 n) and the same (1 + 1) layer is given by 2(c m hy T
2 20422

---

Published in Transactions on Machine Learning Research (05/2024)
4 RReLU against Adversarial attacks
Deep neural networks are susceptible to adversarial attacks, wherein a seemingly minor malicious alteration in the input can lead to the network misclassifying the data point. Numerous techniques have been investigated with the aim of fortifying neural networks against such vulnerabilities, leading to the concept of adversarial robustness in the network Ciiss et al. 2017; Tramer ot al., 2017). We briefly investigate the adversarial
robustness characteristics of nelworks with RReLU activation. Considering a function f X + RC as a
neural network classifier with C classes, (Cohen et al.) 2019 Hein & Andriushchenko, 2017 Salman et al, 2019) showed that Lipschitzness of f is closely related to its robustness. Yang et al. (2020b) demonstrated
that network could achieve better adversarial robustness by imposing an upper bound/ coostrrint the bound/constraint
network's s local Lipschitz constant. (II.C). A smaller LLC indicates better adversarial robustness (Yang et al. 2020b; Tholeti & Kalyani 2022). Note that Lipschitzness is a function of the gradients of the function f. which in turn is a function of the slopes of the RReLU. A function f R - R" is said to be Lipschitz continuous if Vx, y E RRm"
If(x) - f(y)] - -11l
(10)
where
L = sup][[\\\\>]) x E
(11)
is the Lipschitz Constant (I.C), V (xx is gradient of function [(x), 1/p- 1/4= 1, S and g S oo [atorre
let al. 2020 Paulavićius & Zilinskas 2006). Motivated by the fact that many of the RReLU slopes tend to take smaller values than one, we believed that the local Lipschitz constant for networks with RRcLU should have lower values than the local Lipschitz constant for networks with ReLU. Therefore, the local smoothness should be more for architectures with RReLU. Yang et al. (2020b) also demonstrate that the usual robust training methods such adversarial training (Madry et al. 2017), robust self-training (Raghunathan et al. 2020), and TRADES (Zhai et al., 2019) impose a high degree of local smoothness and therefore are most robust. However, these three methods have a large gap bet ween training and test accuracies as well as adversarial training and test accuracios, leading to poor generalization of the robust methods. As we do not perform any adversarial training, the networks' generalization capability with RReLU unaffected. Here, we study the adversarial robustness of RReLU by computing its LLC as it involves the gradient of the function (x) given by (x)),, which in turn is a function of RReLU slopes b, Based the results in subSec. 5.10 it seems that the architectures with RReLU are inherently more robust than those with ReLU.
5 Results and discussion
In order to demonstrate the effectiveness of our proposed method, we have conducted extensive investigations using various architectures, including a fully connected dense network, ResNet architectures (He et all, b) with depths of 20, 56 layers, and WideResNet (Zagoruyko & Komodakis 2016) with depths of 40 and 16, and widening factor of 4. Inspired He et al. (2016b), we have used improved ResNet architectures to efficiently train deeper ResNets (i.e., with depth 110 and 164). This architecture has identity mapping between two ResNet units where the convolution layer follows the ReLU layer, and we consider this as the baseline. As the activation before the convolution layer, we name them ResNet-1100 110-pre and ResNet-164-prm for depth 110 and 164, respectively.
We Lested our method on various standard datasets such as MNIST, SVHN CIFAR-10, and CIFAR-100. To evaluate our method, we compared its performance to the baseline performances of ResNet and WideRes- Net.E] Towards the end, to show the efficacy of the proposed method and to strengthen our claims, we have demonstrated the results on the Imagenet dataset with both WideResNet and Transformer architec- tures (Vaswani et al. 2017). The experiments with Transformer architecture with the Imagenet dataset are conducted using an NVIDLA-AIR GPU, while the rest of them are conducted on a single NVIDLA-GeForce 2080 Ti GPU.
Even though the number of trainable parameters increases slightly in the architectures with RReLU, intro- ducing RReLU can discard a significant amount of /fighets/ilters. As the optimization process is sensitive
Sa reported by Idelbayev
Kuen
Lin
8

---

Published in Transactions on Machine Learning Research (05/2024)
Dataset
MNIST FONN
CIFAR-10
CIFAR-100
SVHN
Architecture
ResNet-
Not-
ResNet-
ResNet-
WRN-
RosNet- ResNet-
ResNet-
RosNet-
WRN-
WRN.
20
56
110-
164-
40-4
20
56
110-
104-
40-1
16-4
pre
pre
pre
pre
Ace ReLU (standard)
91.25
93.03
93.63
94.58
95.47
68.20
69.99
74.84
75.07
78.82
97.01
Are ReLU (more
96.33
93.12
94.45
95.33
95.51
96.18
67.27
72.91
75.23
78.31
79.75
96.75
training)
#Params ReLU
0.10
0.27
0.85
1.7
1.7
8.9
0.27
0.85
1.7
1.7
8.9
2.78
#FLOPs ReLU
0.40
251
505
489
2605
81
251
506
480
2005
944
Filters pruned (%)
4.8
3.86
8.78
6.05
45.34
43.36
0.0
10.32
1.88
22.63
17.99
20.88
Ace RRtLU (post-
98.24
92.86
94.11
95.11
95.10
96.01
67. 69
71.40
74.87
77.85
80.19
96.77
pruning)
#Params RReLU
0.09
0.25
0.78
1.59
0.85
3.26
0.27
0.798
1.69
1.48
7.23
1.86
#FLOPs RReLU
0.38
188
454
309
1245
81
175
187
109
1841
675
Table 1: Performance of RReLU in terms of accuracy, number of trainable parameters, and computation power (in FLOPs). Architectures with both ReLU and RReLU are trained from scratch without using any regularizer. The number of parameters and FLOPs are in Millions (Mn).
towards bj, the convergence is comparatively slow and, therefore, trained for more time. When some elements of the trainable by take very small values, the resulting network act as sparse model. Research conducted by Kuznedelev et al. 20233 demonstrates that sparse networks exhibit a slower convergence rate and derive greater advantages from prolonged training sessions. Hence, while conventional models described in the literature undergo training for 200 epochs, models utilizing RReLU are trained for an extended duration of 1200 epochs. For a fair comparison, ReLU models are also trained for 1200 epochs, and the corresponding accuracy values are listed in the 2nd row of Table | Please note that the models do not show a notion of overfitting as the validation accuracy keeps increasing till 1200 epochs. The last row of Table I represents the accuracy of networks when trained with RReLU, We prune the RReLU slopes using voss-validation such that the accuracy does not degrade after pruning. Comparing the accuracy of RRRLU (in the last row) and ReLU (in the second row), one can observe that with RReLU, one can achieve an accuracy very close to ReLU even though we offer significant saving in memory and computation, which is evident by comparing #Params and #F.OPs for ReLU and RReLU.
Note that after 1200 epochs of training, the slight gap between the accuracies of ReLU and RReLU is because of slower convergence and not because of crrsiffcatton/coopresstiit ResNet20 and Res.et56 with RReLU for even higher (2000) training epochs; we achieved an accuracy of 93.25 and 94.14, which is comparable to the accuracy of 93.12 and 94.45 of respective architectures with ReLU. The WideResNet architecture WRN-40-4 for CIFARIO and CIFAR100 datasets are trained for 400 epochs, and WRN-16-4 for the SVHN dataset is trained for 1000 epochs. All the architectures are trained using a Cosine Anncaling (CA) learning rate (Ir) scheduler.
To empirically prove this, we trained
5.1 More training improves the validation accuracy of the networks
Here, we make an interesting observation when the networks are trained for a higher duration, the perfor- mance improves significantly for most of them, as highlighted in the second row in Table. [ This observation is consistent with the findings of Nakkiran et al. (2021), which suggest that increasing the duration of train- ing can be beneficial. The distribution of batchnorm parameters (7)) changes too from 200 epochs to 1200 epochs. Without employing any regularizer, an extended training duration compels greater proportion of the elements in % to approach zero. As illustrated in Figure 5a| following 200 training epochs, there are approximately 50 elements in 7 in close proximity to zero. However, with prolonged training, this count rises to approximately 270, as depicted in Figure 5b Hence, the augmentation of training duration facilitates a more pronounced adjustment of the " values, leading to improved representation. When employing RReLU in the architecture, the count of elements of 7t in proximity to zero further escalates, reaching an approxi- mate value of 400, as depicted in Figure [cc] This observation suggests that a network equipped with RRRLU attains greater flexibility, yielding a comparable representation capability with fewer filters compared to a basic Rel.U- ResNet architecture.
9

---

Published in Transactions on Machine Learning Research (05/2024)
1000
1000
800
800
600
400
400
400
200
200
200
-0.5
0.0
0.5
1.0
-0.5
0.0
0.5
1.0
1.5
(a) ReLU, 200 epochs
(b) ReLU, 1200 epochs
(c) RReLU, 1200 epochs
Figure 5: Subfig. (a)-(c) are the distribution of batchnorm parameters 7 after the training when the architecture considered is ResNet56. Subfig. (a)-(b) shows the effect of more training on n. Subfig. (c) shows the distribution of y when ResNet56 is trained with RRcLU for 1200 epochs.
5.2 Initialization of the network parameters
In this experimental setup, the trights/filters are initialized using the Kaiming He initialization method (He et al. 2015). We found that the right initialization of the RRcLU slopes (b;) is crucial for effectively train- ing the proposed method. When slopes are initialized randomly with zero mean Gaussian distribution, many of the slopes by are initialized with very small values and converge to values close to zero very soon, and therefore, the whole network converges to a sub-optimal solution. There- fore, is imperative to implement an effective initialization scheme for the slopes to ensure that the network has an adequate convergence win- dow before certain slopes approach zero. Therefore, the slopes (b, for all E L) are initialized with truncated Gaussian Mixture Model (GMM) with a mean of [+1,-1]] and variance of 3, which depicted in Fig. 6 The utilization of the GMM-type initialization for b provides the slopes with a more exlended duration for convergence, given that they are ini- tialized with substantial values. After training, the RReLU slopes converge to a distribution where many clements of by tend towards zero, as can be observed Fig. across different architectures and datascts.
0.30
0.20
0.15
0.05
0.00
-15
1.0
-0.5
0.0
0.5
1.0
1.5
Figure 6: Initial Distribution of RReLU slopes (b;)
5.3 Distribution of RReLU slopes b
Fig. shows the converged RReLU slopes (by) after the training for different architectures when trained with CIFAR10 and CIFAR100 datasets. A comparison of Fig. 7c and Fig. 7g reveals that for relatively challenging lask, such as CIFAR100 classification using the ResNee-110-prre architecture, smaller number of filters can be pruned using RReLU compared to relatively easier task like CIFARIO classification. Furthermore, the degree of pruning is more pronounced for a deeper architecture such as ResNet--144ppr in Fig. Fd than ResNet- BessNet-110-ph 110-pre in Fig. [cl Readers are encouraged to check the distribution of slopes b, for NedeResNets in Fig. 15 in the Appendix, which shows that many filters are made redundant by the use of RReLU. Fig. 16 in the Appendix shows the distribution of BN parameters y when different architectures are trained with CIFAR10 and CIFAR100 datasets. In the latter part of the paper, discuss the potential
range of values, both "Y and b, can take in Fig. 10
5.4 Discussion on intrinsic sparsity property of RReLU
Table [ presents the percentage of filters that are discarded when utilizing the RReLU activation function. Additionally, a comparison is made between the number of network parameters and FLOPs when using RReLU versus the conventional ReLU activation function. Finally, the paper reports the accuracy achieved after discarding unnecessary weights//ilteer.
10

---

Published in Transactions on Machine Learning Research (05/2024)
400
5000
300
4000
300
200
75
200
2000
50
100
100
1000
-0.2
-0.2
02
(a) ResNet-20
(b) ResNet-56
(c) Res.Net-110pre 110-pre
(d) DosNee-1644ppre
150
125
300
100
100
1500
50
50
25
500
(e) ResNet-20
(1) ResNet-56
(g) ResNet-L10-pre
(h) ResNet-164-pre
Figure 7: Distribution of RReLU slopes (br) with CIFAR-10 (top) and CIFAR-100 (bottom).
The first experiment uses a fully connected neural network (FCNN) with a hidden layer of 500 neurons for MNIST classification. Both ReLU and RReLU architectures are trained for 300 epochs using the same Adam optimizer and achieve a similar range of accuracy. After training, RReLU slopes by less than predefined threshold { can be made without degrading performance. By sctting by cross-validatior!! 24 out of 500 b, with values less than 1.0 are dropped, making 4.8% of the weight matrix at the hidden layer unnecessary. The savings are even greater when the architectures are deeper or have more parameters due to convolution operation.
With the CIFAR-10 dataset, the ResNet-164-pre architecture achieves an accuracy of 95.10% with RReLU, while ReLU achieves 95.51%. The proposed RReLU method leads one to discard 45.34% of the total filters, leading significantly improved sparsity and computational efficiency. The number of parameters reduces from 1.7 Mn to 0.85 Mn, and the number of FLOPs reduces from 489 Mn to 309 Mn. Therefore, RReLU saves 49.1% in memory and 36.81% % in computation without regularizer. Interestingly, for not-so-deep but wider architecture like WRN-40-4 (depth 40, widening factor 4), which has 8.9 Mn trainable parameters, the proposed RReLU activation saves 63.37% in memory and 52.20% in computation for CIFAR-10. While saving memory and computation, the WRN-40-4 architecture with RReLU does not compromise on accuracy; for example, it achieves an accuracy of 96.01% comparable to the 96.18% achieved with ReLU. The performance of the proposed RRRLU activation function is evaluated on CIFAR-100 and SVHN datasets, and the outcomes are presented in Table | demonstrating consistently noteworthy results.
To get an idea about speedup concerning the saving in memory and FLOP reported in the paper, we have done an experiment. First, we have trained the model with RReLU which makes some of the filters insignificant after training. As RReLU can ignore a set of filters and sub-filters in two consecutive layers, for inference we can create a smaller architecture, and overwrite its parameters by the values of the significant filters from the original trained model. The smaller model performs the same as the original model but has lesser number of filters. Therefore, the inference time of this smaller model will be less. As the inference time is a metric for showing speedup (Bianco et al. 2018), therefore we measured the total inference time of both the original model and the smaller model in GPU device. The experiment is is detailed in Sec. A.7 of the Appendix. The corresponding inference time calculations are given in Table 10 and Table I in the Appendix for VGG and ResNet-164-prr respectively. Note that the average inference time of the pruned model using the RReLU slope is lower for both VGG and ResNet-164-pre networks. We have taken a higher batch size to keep the GPU utilization in the same range across all three architectures (i) ReLU (Baseline),
"The value of & picked such that the performance does not degrade after the pruning. The values we found for via cross-aalidtioo are given in Table. in the Appendix.
11

---

Published in Transactions on Machine Learning Research (05/2024)
700
000
100
120
140
Figure 8: Filters alive at different depth for ResNet-164 and ResNet-1644RRRee
(ii) RReLU (Unpruned), and (iii) RReLU (Pruned). Keeping the utilization at the same level, we compare the inference time, which shows that RReLU (Pruned) has the minimum inference time therefore it gives the maximum speedup.
5.5 R enhances the representation capability of the filters
To demonstrate that RReLU provides extra reprosentation capability to batchnorm parameters, we first establish a baseline using the work by Liu et al. (2017), where the authors learn efficient, convolutional networks through network slimming. We denote by ResNet-164-LL1BN. where the authors claim that Lı regularization imposed on the batchnorm parameters, Y/, leads to an increased number of Y values near zero. Note that Liu et al. (2017) trained the ResNet164-LIBN model for 160 epochs, after which only 31% of the filters could be removed without any degradation in accuracy (94.%%), which resulted in 19.3% saving in memory and 26.6% saving in FLOP. We have observed carlier that sufficient training gives risc to better convergence of network parameters, resulting in better sparsity and validation accuracy. Therefore, (MT: more training), after which 44% of the filters could be removed without any degradation in accuracy (95.1%) as reported in Table 2 (third column). In the fourth column of T without any use of regularizer, the proposed RReLU (ResNet-164 RRe.UU can prune 31 31.96% of the filters with even better accuracy (95.54%) than ResNet-16G-LIBN..M.. While ResNee-166-LLBN-MT achieves 32.33% saving in memory and 26.38% saving in computation, the proposed ResNet-164-RRell.
train
the same model for 400 epochs and call it BeaNNet-64-LLBN-AAA
achieves saving of 35.92% in memory and 25.97% in FLOP. All the models in Table 2 are trained for 400 epochs for fair comparison. From these results, it shows that With RReI.U, the models can achieves similar sparsity as LIBN-MT without using any regularizer. One may inquire as to why LIBN-MT exhibits a higher number of pruned filters but a similar degree of savings compared to RReLU, The explanation lies in the fact that toward deeper layers, RReLU primarily targets larger filters of size x 3, whereas LIEN-MT prunes both x and 3 filters while the rest of the convolution operations
filters. In Fig. g the vertical red lines denote
the depths where the convolution operation use filters with kernel size 1. It is evident that within the higher depths, RReLU achieves higher degree of sparsity, resulting in greater savings in memory and FLOP compared to the number of pruned filters.
Whereas ResNee--1GARRReL has RReLU without any regularizer, ResNNet-164-LRRRLLLJ uses a L1 regu- larizer with a sparsity regularizer of 8 = 5e-5 The proposed ResNNet-164-.11RRe accuracy with a great saving of 64.67% in memory and 52.96% FLOP. The proposed ResNet-164- U ResNet-164 model trained with ReLU. As shown
achieves even higher
achieves an accuracy 95. 95.25% which is very close achieves higher degree of sparsity within the ranger of higher depth both
Fig. the in terms of bigger (3 3) and smaller x 1) filters, resulting in higher savings.
Methods
Baseline
Pruning methods
Architecture
1esNet-164
ResNet- Liu et al. 2017
ResNet- 1ieee--66--RR-... (Proposed)
ResNet- IRReLL (Proposed)
Ace (with CIFAR10)
95.77
95.10
95.54
95.25
Filters pruned (%) Params in Mn 9% saving)
57.44
1.67
1313(32.3%%) 300
1.07(35.922) 92%)
(.59(61.67))
FLOPs in Mo(% saving)
489
362 (25. 97%)
280(52.909)
Table 2: Pruning capability of RReLU. All sparse models are trained for 400 epochs before sparsification. Percentage values inside parentheses indicate corresponding savings.
12

---

Published in Transactions on Machine Learning Research (05/2024)
Comparing Fig. 9b with Fig. 9a shows that without regularization, the number of elements of Y close to wero is 1600, whereas with regularization, this number has increased to 4000. Without any use of reg- ularizer, the proposed RReLU ResNet-164LLIBN-MMT,, as depicted in Fig. Dc At the end of 400 epochs, RosNet-164LLIBN-MT has more Yi parameters close to vero than ResNet- 164-RReI.U because of the L regularization applied the 7 The introduction of L1 regularization to the RReLU slopes, denoted as RcsNet-164LLRReeUU leads a pronounced enhancement in sparsity, as evidenced in Figure 3d The implementation of regularization on b; yields substantial increment in the number of elements within Y that approach zero, with this count reaching 6200.
has 2500 elements of 7 close to zero lesser than the
2000
8000
0000
0000
0000
4000
4000
1000
2000
2000
0.0
0.1
0.2
0.3
0.4
0.0
02
C4
0.6
0.8
0.0
0.1
02
0.3
0.4
0.0
0.2
0.4
(a) ResNet-164
(b) ResNet-164LLBB-MM
(c) ResNet-164-RRReLL (d) ResNet-166-LLRRaLLL
Figure Effect of RReLU on BN scaling parameters n. (a) ResNet164 (b) ResNet1664LIBN-MT (c) ResNet 164 with RReLU as activation (d) ResNet166 with RReLU as activation and L regularization imposed b;.
In Fig. 10 we show the lype of values by and take after training Res-Net-164LLIRReeeLL Out of 54 residual blocks, we pick 4 blocks at random and plot by (y-axis) against 7 (x-axis) / Given that the regularization is applied to b it compels these paramcters to adopt smaller values. However, as the clements of " are not regularized and can hence assume wide range of values, the term bb can take any value in the real line shown in the figures. In every block, of values, giving higher representation power to every filter. Therefore, in approach, we prune the filters only based on the values of by If we look at Fig. 9b for ResNet-164LLIBN-MT we see that, when L1 regularization is applied on Y,, many of the batchnorm parameters 7 are pushed to take values the range < of & 0.1. However, looking at the same range in Fig. 9c and 9d while using RReLU, we see that the number of elements of Y that range is less. These values tend to either shift towards higher magnitudes or concentrate around values close zero, indicating more flexibility with RReLU.
that when by takes significant values, Y takes range
0.2
6.1
0.0
0.0
0.0
-0.1
-0.1
-0.1
0.2
-02
010
0.15
0.20
0.25
0.30
0.00
0.35
0.10
0.20
(a) Block 8
(b) Block 20
(c) Block 36
(d) Block 49
Figure 10: Plot of RRcLU slopes (b;) along y-axis vs. BN parameters (mi) along x-axis in different residual
blocks of 16sNeee-14
Also note that, with more depth, more number of filters could be pruned more number of (b,, m) is close to zero 0 For example, in 8th residual block (Fig. 10a), which consists of shallow layers, there relatively few slopes in proximity to zero, limiting the pruning capacity. In contrast, when examining deeper layers,
3In Fig. 17 in the Appendix, show such plots for blocks.
6rhhi trend visible Fig. [7 of the A Appendix.
13

---

Published in Transactions on Machine Learning Research (05/2024)
such the 49th block, numerous pairs of (b)))) gravitate towards zero, indicating greater potential for filter pruning.
Following equation 3 one can observe that when the 11 regularization is applied to the batchnorm parameters Ni enforces all the elements of y to take lower values. This constraint may limit the representation capacity of the ResNeet-164-..BNNNMTT model. But with RReLU, there is no such restriction on " or b and RReLC provides another degree of freedom by the slopes bj. Therefore, When by is significant, the elements of "i take an unconstrained range of values, e.g. in Fig. [Ob] It improves the representation power of the corresponding filters. Therefore, it inherently enables greater number of parameters of by to approach zero, leading to disregarding corrosponding filters, all achieved without the need for explicit regularization, thus resulting in implicit sparsification.
95.0
Accuracy
94.5
94.0
L1BN-MT
RReLU
93.5
LIRReLU
100
125
150
175
200
225
250
275
300
FLOP reduction
Figure 11: Ace vs FLOP reduction. The proposed methods (RReLU, LIRReLU) are compared with LIBN-
MT (MT: more training)
In Fig. 11 we present the top 1% accuracy vs. FLOP reduction trade-off for the ResNet-164 model. The FLOP reduction signifies the difference between the FLOP count the ResNet-164 and the FLOP count of the models employing various sparsifying methods, as depicted in Figure [I] Enhanced performance is indicated by positioning closer the upper-right quadrant of the graph, as it indicates higher accuracy while using lower of FLOPs. Without using regularizer, the RReLU provides same FLOP reduction LIBN-MT with little higher accuracy, indicating that RReLU provides intrinsic sparsification. Further, when Ln regularization is applied on RReLU slopes, L1RReLU provides the best FLOP reduction with a same level of accuracy.
With its excellent sparsification capability, decomposed training (Yang et al. 2020ah achieves exceptional speedup, keeping the accuracy the same. Whereas decomposed training maintains an accuracy of 93.28% for ResNet56, we achieve an accuracy of 94.11% with RReLU (without any regularization). As RReLU demon- strates inherent sparsification properties, can be seamlessly integrated into the architectures employed in the study by Yang et al. (2020a) in conjunction with decomposed training techniques. This integration has the potential to yield significantly improved speedup and enhanced accuracy. In the below paragraph, we show it can be applied to another recent structural pruning work by Hou et al. (0022)
5.6 RReLU combined with other structural pruning methods improves sparsity
Here we show that when RRcLU is combined with the structural pruning methods such CHEX Hou et al. (2022), the sparsity further improves. The pruning and regrowing for the CHEX method are performed during the training after every AT = 2 time step. CHEX considers ResNet50 for performing classification on the ImageNet Datasct. ResNet50 on the ImageNet classification task achieves an accuracy of 7f.16.39 13% using GFLOP count of 4.09. The ResNet50 architecture with RReLU reduces the GFLOP to 3.45 without using any regularizer. This shows that RReLU can provide sparsity intrinsically. Here, both the architectures with
14

---

Published in Transactions on Machine Learning Research (05/2024)
ReLU and RReLU, respectively, are trained for 250 epochs. We have observed that instead of the usual multi- step learning rate (T.R) scheduler, if the cosine annealing LR scheduler is used, then the accuracy improves to 76.28% with 3.15 GFLOPs. The improved performance indicates that if trained well with appropriate hyperparameters and proper initialization, the architectures with RReLU can have better accuracy than ReLU with a lesser number of GLOPs within the same number of time steps.
We acknowledge that compared to recent paarsity-inducing methods such as CHEX, RReLU provides less sparsity. However, it's crucial to recognize that the sparsity induced by RReLU is intrinsic. As highlighted in paper, RReLU is activation function that inherently promotes sparsity and is not pruning technique. Consequently, RReLU can be effectively combined with various structured pruning methods to enhance sparsity, as each filter has better representation power with RReLU. To validate this, we have integrated RReLU with CHEX and compared the GFLOP counts for ResNet50 with ReLU and RReLU below.
Methods
Accuracy
GFLOPs
ReLU (multistep LR) RRcLU (multistep LR)
76.13%
4.09
75.22%
3.45
RReLU (cosine annealing)
76.28%
3.15
CHEX w/ ReLU (s 70%, cosine annealing)
76.28%
1.35
CHEX w/ RReLU (s 70%, cosine anncaling)
75.48%
1.12
CHEX W/ ReLU (s 50%, cosine annealing)
77.98%
2.49
CHEX w/ RReLU (s 50%, cosine annealing) 77.58%
2.20
Table 3: Comparison between (i) ReLU (ii) RReLU (ii) CHEX, and (iv) RReLU combined with structured pruning method CHEX for ResNet50 on ImageNet dataset.
With a target sparsity of s = 70%, the bascline (CHEX w/ RcLU) with ResNet50 on ImageNet classification achieved an accuracy of 76.28% with a GFLOP count of 1.35 when trained from scratch using the codebase and the hyperparameters the authors have provided 7 When CHEX is combined with RReLU (CHEX w/ RRELU), the architecture reduces the computation to 1.12 GFLÖPs, as shown in Table. 3 corroborating our claim. Further, with target sparsity of 50%, CHEX reduces the computation from 4.09 GFLOPs to 2.20 GFLOPs, but when CHEX is combined with RReLU, it reduces the computation to 2.20 GFLOPS. This shows that RReLU improves the sparsity when combined with existing structured pruning techniques. The detailed results are provided in Table. 3
5.7 RReLU as the coarse feature extractor
During training, the learned weights/filters are tuned to extract finer features of the input. The RReLU activation function is observed to extract the coarse features where the weights/filter ffltter are random. In order to substantiate this claim, we conducted an experiment whereby only the RReLU slopes (by) are trained for 500 epochs. At the same time, the weights are fixed, following initialization with the Kaiming He method (He et al.) 2015). The outcomes indicate that while a neural network architccture with untrained weights and a ReLU activation function fails to learn anything, an architecture with solely learned by can allain some degree of learning. Specifically, ResNet-56 with only the learned by (and untrained weights) yielded an accuracy of 51.42% with the CIFAR-10 dataset, and one can remove 7.09% of the filters using Ç 0.1, shown in Table 4
These findings suggest that the RReLU activation function, particularly its learned slopes, extracts the coarse features of the inputs in terms of the untrained filters in the case of CNNs. Fig, 12 shows the histogram of the RReLU slopes of ResNet-56 after training. Once the RReLU is used as a coarse feature extractor, one can drop those RReLU that have nearly zero slopes and prune their corresponding filters. We can then train the remaining weights and retrain the slopes to attain the finer features. After the 2nd step, the models tend to achieve similar accuracy as the baseline, as seen in Table 4
15

---

Published in Transactions on Machine Learning Research (05/2024)
Dataset
CIFAR-10
CIFAR-100
Architecture
ResNet20
ResNet56
ResNet20
ResNet56
Acc. ReLU (standard)
93.03
93.63
68.20
69.99
Acc (1*--tte)
45.12
51.42
8.02
10.54
Filters ignored (%)
7.09
6.84
7.44
#Params RReLU
0.24
0.72
0.23
0.72
#TLOPs RReLU
70.29
Ace (2nd step)
91.93
92.82
66.68
66.39
Table 4: RReLU extracts the coarse features with by being only the trainable parameters.
slopes
le4
140
8
120
Number of paths
ReLU
RReLU
100
6
60
40
2
0
-4
-2
500
1000
Figure 12: Distribution of b, for ResNet-56 with CIFAR-10.
Figure 13: Distribution of filter-path length for WRN-40-4 with CIFAR-100.
5.8
Features choose the shortest path length
Veit et al. (2016) demonstrate residual networks
ensemble of independent paths, and the study show that only shorter paths carry gradients despite using deeper architecture. Our study supports these findings as observe that features not only try to pass through shorter paths, but they try to pass through lesser number of filters as well. While Veit et al. (2016) define path-length as the number of layers the input features pass through, we define filter-path length as the number of filters the features pass through. For WRN-40-4 with ReLU activation, most paths have a filter-path length between 300 to 1100, but using RReLU, we observe a reduction in filter-path length, with most paths having lengths between 100 to 500, shown in Fig. 13 This shows that RReL.U activation allows features to pass through fewer filters, resulting in saved memory and computation.
5.9 Scalable across larger dataset and various architectures
In this part, we show results with the benchmark ILSVRC-2012 (ImageNet-lk) dataset in Table. For testing with TmageNett-kk we consider three architectures: (i) a WideResNet with depth 50 and widening factor 2 denoted by WRN-50-2, (ii) Vision Transformer with 12 layers, a width of 384, the MLP width of 1536, and patch-size of 16 denoted by VIT-s16 (Steiner et al. 2021, and an (iii) EfficientNet. For training WRN-50-2, closely follow the official 1mplementation for image classification with ImageNet dataset with standard Py Porch models. We have selected WRN-50-2 due to its superior performance compared to considerably deeper ResNets such as ResNet-152. We do not use pre-trained parameters for initializing the network and train WRN-50-2 for both ReLU and RReLU from scratch. In both scenarios, whether utilizing ReLU or RReLU activation functions, the architectures of WRN-50-2 undergo training for a total of 120 epochs, the same as the one considered in the official PyTorch repository.
VIT-s16 was successfully trained ImageNet by Dosovitskiy et al. 2020), attaining excellent results pared to familiar convolutional architectures. The VTT-s16 architecture uses only the Encoder block of
httpp://ggthuub.cc/pprrrrrr httett///ghh....cooooo////eeteea
16

---

Published in Transactions on Machine Learning Research (05/2024)
Transformer, which consists of 12 blocks, each having a teaaddddo--roouuc--tteeniioo multi-layer per- ceptron (MIP) block and two Dropout layers and two deyer-normalizatioo layers Even though Dosovitskiy ett al. (2020) proposed Vision Transformer, they published codes only for fine-tuning and pre-trained models. In a subsequent study by Steiner et al. (2021). ) the rescarchers conducted a thorough investigation into the relationship between the amount of training data, model rngularization/dee /data augmentation, model size, and compute budget. They made their code base available with a configuration (Beyer al. 2022) to train plain VIT baselines from scratch for ImageNet-1k. In our research, we utilized this framework to present results with the VIT-s16 model. This plain VIT baseline uses standard data augmentation and no sophisticated regularization techniques such as augmentation or regularization pipeline. It does not use any pre-training procedure, The VTT-s16 model for ReLU is trained for 300 epochs, whereas for better convergence of RReLU slope, VIT-s16 with RReLU is trained for 400 epochs.
EfficientNet has shown superior performance post-ResNett and is famous for efficient computation. For example, for a similar accuracy, EfficientNee-bl is 7.6x smaller and 5.7x faster than ResNet-152 Tan & Le 2019 EfficientNet uses (i) depthwise separable convolution and (ii) sequential squeeze and excitation in inverted bottleneck residual block along with intermediate expansion operation and therefore is better version than MobileNet which has only depthwise scparable convolution. Therefore, we procceded with comparisons only against EfficientNee,, as it represents superior architecture than MobileNet Howard et al. (2017) for studying the performance of RRe.U. EfficientNet uses Sigmoid Linear Unit (SiL.U) activation. We show the intrinsic sparsity achieved by using the idea of rotation in one of the EfficientNet architecture i.e. EfficientNNetb4. We replace SiLU with rotated SiLU (RSILU) and train it. The EfficientNet architectures with ReLU are trained for 400 epochs, whereas we report the performance of Nff with RSILU after 500 epochs.
Arc
Activation
Filters ignored (%)
Accuracy Params(Mn)
Pruned
FLOP(Mn)
Pruned
WRN-50-2
ReLU
76.682
67.4
21563
WRN-50-2
RReLU
25.34
76.58
55.2
18.1%
18648
13.5%
VIT-sIG-MLP
GELU
77.51
14.2
28.3
VIT-s16-MLPP
RGELU
6.32
80.1
13.2
6.32%
26.5
6.32%
EfficientNet-bá
SILU
82.80
19.3
4200
Net--de
RSILU
16.83%
81.67
17.2
10.9%
3500
16.66%
Table 5: Applying Rotation on ReLU, GELU, and SiLU activation with Imagenet dataset,
While WRN-50-2 architecture uses ReLU, each MLP block of VTT-s16 contains two dense layers with a GELU non-lincarity in between two layers. So, for WRN-50-2, we have used RReLU instead of RcLU; for VIT-s16, we have used RGELU instead of GELU. In Table 5 we demonstrate the sparsity property of RReLU or RGELU, Training WRN-50-2 with RReLU for ImageNet demonstrated the implicit regularization property of RReLU, allowing for the removal of 25.34% of the filters. This led to 18.1% reduction in parameters and a 13.5% reduction in FLOPs, keeping the performance the same. The VIT-s16 architecture boasts a total parameter count of 22.14 14 million, with 14.2 million parameters dedicated to the MLP component, while the remaining parameters are attributed to the multihead- propduuuc attention mechanism. So in Table 5 we report the saving in the number of parameters and FLOPs in VIT-s16-MLP. Whereas the VIT-s16 was reported to achieve an accuracy of 77.51% with GELU, the same with RGELU achieved an accuracy of 80.1% with sufficient training. Not only does the accuracy improve, but RGELU provides a saving of 6.32% in both memory and FLOPs. The baseline architecture, EfficientNNetba has 19.3 Mn parameters and 4.2 Bn FLOPs and is reported to have an accuracy of 82.00% Using RSiLU provides intrinsic sparsity to reduce the parameter count and FL.P count to 17.2 Mn and 3.5 Bn, respectively, with a comparable accuracy of 81.67%. We could run only one iteration because of limited computing power, but we believe that the accuracy can be improved with better hyperparameter tuning. These results imply that rotation operation is transferable to other activations if the activation mimics ReLU.
For detailed architecture, please refer httpp:///ghhtaa.ccooooeeerrrrrrr
17

---

Published in Transactions on Machine Learning Research (05/2024)
5.10 RReLU imposes better adversarial robustness
From the distribution of b; in Fig. 7 it is observed that most of the RReLU slopes have values less than one in most cases. As LC can be defined as the upper bound on the gradient of the function f(x) (Latorre et al, 2020 Paulavičius & Zilinskas 2006), it can be argued that the value of LC is typically lower for RReLU than ReLU, resulting in better adversarial robustness. However, practice, calculating the Lipschitz Constant (T.C) for the neural network NP-hard, and an alternative empirical computation of the Local Lipschitz Constant (LLC) is preferred and defined as the following quantity (Yang al., 2020b)
[xf(x,) f(x',))l
max x (X..e)
||x: - x'2lo0
(12)
n
To perform the max operation in equation
12
we adopt a gradient-doscent-lie algorithm with a step-
Architecture
activation
LLC
Attack type
Adv test
FGSM
34.11
ReLU
1.33
PGD
38,20
ResNet-20
FGSM
39.84
RReL.U
1.2
PGD
42.46
FGSM
59.01
ReLU
PGD
16.45
ResNet-56
FGSM
66.85 85.85
RRtLU
1.30
PGD
17.05
Table RReLU to boost local smoothness and hence adversarial accuracy.
size 0.001 and 60 iterations. We have computed the LLC using equation and provided the third column of Table which shows that the LLC for the architectures with RReLU is lower than the LLC for the architectures with ReLU, indicating better smoothness (anng et al. 2020b We consider single-step attack, such as the Fast Gradient Sign Method (FGSM), and an iterative attack, Projected Gradient Descent (PGD), to test the adversarial robustness of architectures with RReLU. FGSM samples considered in our experiments have a perturbation of e = 0.031. The imput variation parameters eg for the PGD attack are considered to be 0.031 and 0.1 for ResNet12 and ResNet55, respectively. The step size for each atlack iteration is <,/20, and the mumber of attack iterations is 10. Observing the adversarial test accuracy (for both FGSM and PGD) the last column of Table 6 we see that for the type of attack we have considered, the architectures with RReLU have better robustness than one with ReLU.
5.11 A study on the sparsification capability of batchnorm scaling parameters versus RReLU slopes
Mathematically, looks like the RReLU slopes can be learned by the outer convolution operation. But the absence of RReLU slopes, just the batchnorm parameters alone cannol render some filters unnecessary to achieve structural sparsity. We suspect the reason is that the batchnorm parameters usually initialized with a positive value; therefore, they do not explore negative values well. Batchnorm parameters cannot be initialized with a range of both positive and negative values because with negative values, during training, there is a high chance that the output from the batchnorm gets mapped to the negative part of the ReLU slope and the information does not pass to the next layer.
To understand the relation between the batchnorm parameters n and RReLU slopes b we have done the following experiments. We study the effect of both 7 and b in performance in terms of both accuracy and sparsity. So, we consider four cases:
The value of the elements of by is fixed to 1 which is nothing but ReLU and we train only n This is the baseline, i.e., experiment with ReLU.
The value of every element of is fixed to the initial value of 0.5 and train only by
We fix elements of both 7 and by to 0.5 and 1, respectively.
18

---

Published in Transactions on Machine Learning Research (05/2024)
3000
2500
2500
2500
2500
2000
2000
2000
2000
1500
1300
1500
2500
500
500
500
0
0.0
0.2
0.4
-0.4
-0.2
0.0
0.4
0.2
-0.4
-0.2
0.2
(a) Experiment distribu-
(b) Experiment distribu- tion of b
(c) Experiment distribu-
(d) Experiment 4: distribu- tion of b
tion of "
lion of n
Figure 14: Distributions of and b
We train both " and by.
All the above models are trained for 400 epochs without using any regularizer. Below are our observations:
The first case nothing but the baseline where we use model with ReLU and the 7 trainable. We achieved accuracy of 95.79. In this case, only 755 elements of " close to not L1 regularization on n. The distribution of elements of n is shown in Fig. 14a clear that the trainable is not able to induce sparsity much by itself.
do So it is
In the second case, we fix every element of n 0.5 and learn the RReLU slopes along with other trainable network parameters. The aim is to understand if b, can do the job of n. We achieve accuracy of 95.70, close to the previous baseline. From this, we can see that even if we keep the fixed, the by are still able to do the job of 7n So, it is not necessary to have to reach similar accuracy as the first case. The distribution of the elements of by are shown in Fig. 14b] One can observe that the number of slopes close to zero 1561, which is higher than the previous case where the number of elements of 7t, close to zero, 755.
In the third case, we fix both Yi and by to see if the network filter weights and biases are enough to reach the same accuracy as the above two cases. We observe that reaches an accuracy of 95.40, almost the same as the previous two methods. But please note that as the elements 7 and by are fixed to values 0.5 and 1.0, respectively, the sparsity could not be induced.
The fourth case has both and by are trainable and the model reaches an accuracy of 95.84. The distributions of Y) and by are shown in Fig. 14c and Fig. [4d] respectively. When both y and b are trained logether, the sparsity is more. We see that 2354 slopes are near zero. But a slightly lesser number of 7 (2282) are close to zero because when the elements of do not take values close to zero, the elements of m take wide range of values. This is more prominent when the network is trained for more epochs. Comparing the Fig. 14d with Fig. 14b] one can observe that with trainable and b,, the sparsity is more than only trainable by even though by can do the job of for achieving good accuracy. Comparing Fig. [4c with Fig. shows shows that with trainable and b,, the number of tt close to zero also incrcases. Further, the sparsity increases if the model is trained for more training epochs as more elements of b, go towards zero. We have seen that once trained for 1200 epochs, the sparsity is even more.
We have also shown how the distributions change during optimization for some of the 7 in Fig. 18 of the Appendix. The elements of Y are initialized with 0.5, and during training, some of the elements z those which correspond the zeroed elements of b, (during training), slowly converge take values close to zero. Fig. 19a] Fig. 196 and Fig. [9c in the Appendix are showing the distribution of by in the first, second, and third layer, respectively of every residual block. Over time, some of the elements of by move towards zero.
19

---

Published in Transactions on Machine Learning Research (05/2024)
6 Conclusion
In this work, we proposed the RReLU activation and novel initialization for the RRcLU slopes. We then demonstrated through extensive experiments that the combination plays a vital role in the implicit sparsification of the architecture by enhancing the representation capabilityy of the architecture. The proposed activation saves resources and improves performance in many architectures. For example, with the CIFAR- 10 dataset, the WRN-40-4 achieves 63.37% saving in memory and 69.7% saving in computation without any loss in accuracy. The proposed RReLU can be used an alternative to ReL.U in all deep architeclures where ReLU had been used carlier including other structural pruning methods too. The RReLU-based architectures are also shown to have lower LLC value and hence better adversarial robustness. Therefore, RReLU is one-stop solution for improved performance, energy efficiency, and robustness. The idea of learning the rotation of the linear component of the activation function can also be applied to other activation functions which have a partly linear component, ie, we have applied the proposed idea to GELU and observed that RGELU ViT-s16 achieves an accuracy better than GELU with 6.32% saving memory and computation of the MLP block.
References
Muhammad Alrabeiah, Andrew Hredzak, and Ahmed Alkhatceb. Millimeter wave base stations with cam- eras: Vision-aided beam and blockage prediction. In 2020 IEEE 91st vehicular technology conference WVTC2020-Spring pp. 1-5. IEEE, 2020.
Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. Transactions on Information theory, 39(3):930-90) 945, 1993.
IEEE
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov.
Big vision.
https://github.. com/
rewearchhbig. vision 2022.
Simone Bianco, Remi Cadene, Luigi Celona, and Paolo Napoletano. Benchmark analysis of representative deep neural network architectures. IEEE access, 6:64270-64215 2018.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic relu. In European Conference on Computer Vision, pp. 351 367. Springer, 2020.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In International conference on machine learning, pp. 854 863. PMT.R, 2017.
Arnr Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint urXiv:1511.07288. 2015.
Jeremy Cohen, Flan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning, pp. 1310 1320. PMLR, 2019.
Matthieu Courbariaux, Yoshua. Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural net- works with binary weights during propagations. Advances in neural information processing systems, 28, 2015.
George Cybenko. Approximation by superpositions of signoidal function. Malhematics of control, signals and systems, 2(4):303-314 1989.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint ar Xiv:2000. 11929, 2020.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maxi- mization. In Proceedings of the TEEE/CV Conference on Compuler Vision and Pollern Recognilion, pp. 9270 9280, 2021.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourleenth international conference on artificial intelligence and statistics, pp. 315 323. JMLR Workshop and Conference Proceedings, 2011.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Morphnet: Fast simple aesorce-conssrrinee structure learning deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognátion, pp. 1586-1595, 2018.
Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, et al. Ghuonev and gluonnlp: deep learning in computer vision and natural language processing. J. Mach. Learn. Res., 21(23):1-7, 2020.
Ramyad Hadidi, Jiashen Cao, Yilun Xie, Bahar Asgari, Tushar Krishna, and Hyesoon Kim. Characteriz- ing the deployment of deep neural networks on commercial edge devices. In 2019 IEEE International Symposium on Workload Characterization (IISWC), pp. 35-48. IEEE, 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Boris Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics, 7(10):992, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part IV 14, pp. 630-645. Springer, 2016b.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of classifier against adversarial manipulation. Advances in neural information processing systems, 30, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1:Xix:6006.00 15, 2016.
Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun-Yu Yuan Kung, Chex: Channel exploration for model compression. In Proceedings of the IEEF/CVF Conference on Compuler Vision and Pallern Recognation, pp. 12287-12288 2022.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arxie preprint orXiv:1704.00 048661, 2017.
Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, 5(9), 2004.
Changeun Huang. Relu networks are universal approximators via piecewise linear or constant functions. Neural Compulation, 32(11):2249 2278, 2020.
21

---

Published in Transactions on Machine Learning Research (05/2024)
Yerlan Idelbayev. Proper ResNet implementation for CIFARIO/CIFARI00 in PyTorch. https://github. com/akkamsster/pytorrh.. resset, cifar10 Accessed: 20xx-xX-XX.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Scunghyeon Kim, Yung-Kyun Noh, and Frank C Park. Efficient neural network compression via transfer learning for machine vision inspection. Neurocomppttngg 413:294 304, 2020.
Alex Krizhevsky, Πlya Sutskever, and Geoffrey Hinton. Imagenct classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
Jason Kuen. Wide residual networks (wideresnels) in pytorch. https://ggthhub.co///ttrraaa WideResNeet-yytorch Accessed: 2023-02-17.
Denis Kuznedelev, Eldar Kurtic, Eugenia Tofinova, Elias Frantar, Alexandra Peste, and Dan Alistarh. Ac- curate neural network pruning requires rethinking sparse optimization. arXiv preprint Xiv:22088000000 2023.
Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arxiv preprint arXin:2004.008: 08688, 2020.
Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruming filters for efficient convnets. In International Conference on Learning Representations, 2017. URL httpp:://penrerviei net/forumiid-rJgqTT
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 806 814, 2015.
Zhuang Liu. Network slimming. hutpp://ggthbbbbo////uuuuaggggg www/1uuzuuagggg11111 smmmmnnnnnnnnnnnnngnnmnmmmmminnn Accessed: 2023-02-17.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming, In Proceedings of the IEEE international conference on computer vision, pp.
2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value network pruning. In International Conference on Learning Representations, 2019. URL htttps://peerreiien net/foorumiidrrJInBB3c
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinct: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5061 2017.
Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Citescer, 2013.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arxiv preprint ar 1706. 06080, 2017.
Oscar Mañas, Alexandre Lacoste, Xavier Giro-i Nieto, David Vazquez, and Pau Rodriguez. Seasonal con- trast: Unsupervised pre-training from uncurated remote sensing data. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 9414-9423, 2021.
Lukas Meier, Sara Van De Geer, and Peter Bühlmann. The group lasso for logistic regression. Journal of the Royal Statistical Society: Series (Statistical Methodology), 70(1):53-71, 2008
Paylo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruming convolutional neural networks for resource efficient inference. arXie preprint 1611.661 2016.
22

---

Published in Transactions on Machine Learning Research (05/2024)
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, 2010.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and 1yya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12):1133440 2021.
Nancy Nayak, Thulasi Tholeti, Muralikrishnan Srinivasan, and Sheetal Kalyani. Green detnet: Computation and memory efficient detnct using smart compression and training. arXiv preprint arXiv:2003.09440 2020.
Gia Nhu Nguyen, Nin Ho Le Viet, Mohamed Elhoseny, K Shankar, BB Gupta, and Ahmed A Abd El-Latif. Secure blockchain enabled cyber-phystee physical systems in healthcare using deep belief network with resnet model. Journal of Parallel and Distributed Computing, 153:150 15::150-160, 2021.
Remigijus Paulavičius and Julius Zilinskas. Analysis of different norms and corresponding lipschitz constants for global optimization. Technolooiool and Economic Development of Economy, 12(())301133) 2006.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint orXiv:2002.17166 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Oclober 11 11-14, 2016, Proceedings, Parl IV, pp. 525 542. Springer, 2016.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019.
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Neurocomputing. 241:81-89, 2017.
Nitin Priyadarshini Shankar, Deepsayan Sadhukhan, Nancy Nayak, and Sheetal Kalyani. Binarized resnet: Enabling automatic modulation classification at the resource-constrained edge. arXiv preprint arXiv:2110 10.1 14357, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprinl Xic: 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting The journal of machine learning research, 15(1): 1929 1958, 2014.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. orXiv preprint arXiv:2106.100 10270, 2021.
Mingxing Tan and Quoe Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Thulasi Tholeti and Sheetal Kalyani. The robust way to stack and bag: the local lipschitz way. arXiv preprint Xiv:22206. 201:0200.00013, 2022.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint ar Xiv: 175:1705.000 2017.
23

---

Published in Transactions on Machine Learning Research (05/2024)
Fengbin Tu, Yiqi Wang, Zihan Wu, Ling Liang, Yufei Ding, Bongjin Kim, Leibo Liu, Shaojun Wei, Yuan Xie, and Shouyi Yin. A 28mm 29.2 tflops/w w bf16 and 36.5 tops/w w int8 reconfigurable digital cim processor with unified ip/int pipeline and bitwise in-memory booth multiplication for cloud deep learning acceleration. In 2022 IEEE International Solid-State Circuits Conference (ISSCC), volume 65, pp. 1-3. IEEE, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural informalion processing syslems, 30, 2017.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29, 2016.
Devannagari Vikas, Nancy Nayak, and Sheetal Kalyani. Realizing neural decoder at the edge with ensembled bnn. IEEE Communications Letters, pp. 1-1, 2021. doi: 10. 10.1109/CCOMM..001...3
Jian Wang, Chen Xu, Xifeng Yang, and Jacek M Zurada. A novel pruning algorithm for smoothing feedfor- ward neural networks based on group lasso method. IEEE transactions on newral networks and learning systems, 29(5):2012-200 2024, 2017.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in newral information processing systems, 29, 2016.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1500 150.000853, 2015.
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. Learning low- rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp.
678 679, 2020a.
Yao- Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. Advances in neural information processing systems, 33:8588 8601, 2020b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ar preprint arXic:1605.07144 2016.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust gencralization just requires more unlabcled data. arXiv preprint ar arXi::1966 00555, 2019.
Hao Zhou, Jose M. Alvarez, and Fatih Porikli. Less is more: Towards compact cnns. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 662-677, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46493-0.
24

---

Published in Transactions on Machine Learning Research (05/2024)
A
Appendix
In this section we provide experiment details so that it is casier for the readers to reproduce the results.
A.1 The sample ResNet and WideResNet architectures considered for our experiments
The architecture of vanilla ResNet-20 is given in Table. 7 where each of the three blocks has 6 layers. For vanilla ResNet-56, each block has layers He et al. (2016a); [delbayev To obtain the architecture with RReLU, the ReLU activations at all the layers (except the first convolution layer) are converted to RReLU. We do not replace the ReLU after the first convolution layer with RReLU because we may not want to miss out any information in the input by deactivating filters at the very first layer. For CIFAR-10 dataset, K = 10 and for CIFAR-100, K 100.
Layer
Description
Output
Dimen-
sion
Input
3 32 32
Conv
Cin 3, Cout 16, Kernel: (3 BN, ReLU
16
BlockI-layerl
Gin 16, Cout 16, Kernel: (3xx 3)]. BN, ReLU
Block1-layers
C.n 16, Cout 16, Kernel: (3 x 3)]. BN, ReLU, addition
Block1-layer3
Cin 16, Cout 16, Kernel: (3 3)]. BN, ReLU
x 32 x 32
Blockl-layerd
Cin 16, Cout 16, Kernel: (3 3)]: BN, ReLU, addition
x 32 x 32
Block1-layer5
Gi 16, Cout 16, Kernel: (3 8)]: BN, ReLU
x 32 x 32
Block1-laycr6
C.n 16, Cout 16, Kernel: (3 x 3)], BN, ReLU, addition
16 x 32 x 32
Block2-layer1
C.n 16, Cout 32, Kernel: (3 x 3)], BN, ReLU
32 X 16 x 16
Block2-layer2
C.n 32, Cout 32, Kernel: (3 x 3)], BNN ReLU, addition
32 X 16 x 16
Block2-layer3
CN 32, Cout 32, Kernel: (3 x 3)], BN, ReLU
32 X 16 x 16
Block2-layerd
C.n 32, Cout 32, Kernel: (3 x 3)], BN, ReLU, addition
32 X 16 x 16
Block2-layer5
C.n 32, Cout
32, Kernel: (3 x 3)], BN, ReLU
32 X 16 x 16
Block2-
C.n 32, Cout
32, Kernel: (3 3)], BN, ReLU, addition
32 X 16 x 16
Block3-layer
Cin 32, Cout
64, Kernel: (3 3)]. BN, ReLU
64 8 8
Block3-layee
Cin 66, Cout 64, Kernel: (3 x 3)]. BN, ReLU, addition
64 8
Block3-layert
C.D. 64, Cout 64, Kernel: (3 x 3).. BN, ReLU
64 8
Block3-layerd
C.N 64, Cout 64, Kernel: (3 x 3)), BN, ReLU, addition
64 8
Block3-layer5
Cin 64, Cout 64, Kernel: (3 3)], BN, ReLU
Block3-layeró
Cin 64, Cout 64, Kernel: (3 3)]. BN, ReLU, addition
Linear
classification into classes with softmax
Table 7: Architecture of vanilla ResNet-20; For the RReLU version, the ReLU activations are replaced by ReLU. Here addition implies the addition of a skip connection with the residual path. For CIFAR-10, K = 10 and for CIFAR-100, = 100.
The Res-ee-110-pre architecture has an identity connection between two ResNet units; the batchnorm layers are followed by RReLU and the RReLU layers are followed by the convolution layers. The "pre' in its name indicates preactivation, i.e., ReLU/RReLU activation before the convolution layer. The ResNet-164-pre also has the above identity connection and preactivation; along with special bottleneck architecture which has different number of channels along the depth than the standard ResNets He et. al. (2016b) Liu
The WedeResNeelKuen architecture WRN-16-4 is given in Table. g that is used with the SVHN dataset. For SVHN K = 10. The other WideResNet architecture WRN-40-4, which has 40 layers and a widening factor of 4. It has the same three blocks but each with 12 layers. WR. N-40-4
A.2 How to set Ç during deployment of the models to discard the slopes that are below Ç and still not degrade the performance?
The value of ( is set by cross- validatii. The network is trained with the train set, and the test set is divided into two parts. The first part of the test set is used to find a Ç such that if the slopes that are below that { are made equal zero, the performance does not degrade. The number of RReLU slopes that can be ignored for different experiments and the corresponding Ç is given in Table g With the latter part of the
25

---

Published in Transactions on Machine Learning Research (05/2024)
Layer
Description
Output
Dimen-
sion
Iuput
3 X 32 32
Cony
Cin 3, Cout 16, Kernel: (3 3), BN, ReLU
16 32
32
Block1-lay
Gin 16, Cout 64, Kernel: (3 3)], BN, ReLU
64
32
32
Block1-layer2
Cin 64, Cout 64, Kernel: (3 8)], BN, ReLU, addition
64
32
BlockI-layer3
C.n 64, Cout 64, Kernel: (3 3)], BN, ReLU
64 32 32
BlockI-layerd
Cin 64, Cout 64, Kernel: (3 3)]: BN, ReLU, addition
Block2-layerl
Cin 64, Cout 128, Kernel:
x 3)], BN, ReLU
16
Block2-layer2
C.n
Cout 128, Kernel:
3)]; BN, ReLUU addition
Block2-layer3
C.n
Cout 128, Kernel:
3)]: BN, ReLU
x
Block2-layerd
Cin 128, Cout 128, Kernel:
3) BN, ReLU, addition
128 x 16 x 16
Block3-layerl
Cin 128, Cout 256, Kernel:
3)), BN, ReLU
256 x 8
Block3-layerg
On 256, Cout 256, Kernel: (3 3)), BN, ReLU, addition
256 x
Block3-layer3
Cin 256, Cout = 256, Kernel: (3 x 3)}, BN, ReLU
256 X x 8
Bloerd-layerd
C.n. = 256, Cout = 256, Kernel: (3 3)], BN, ReLU, addition
256 x x
Linear
classification into classes with softmax
K X
Table Architecture of WRN-16-4. For CIFAR-10, K -0 and for CIFAR-100, K 100.
test set, the pruned network is tested, and accuracy is reported in Table 1 of the main manuscript. For Imagenet and WRN-50-2, the threshold ( is set equal to 0.045 via cross validation.
Table 9:
The value of Ç for different experiments and the corresponding number of filters ignored the 'x/y' format. Here 'x' filters out of 'y' filters could be dropped because RReLU.
Dataset
MNIST
CIFAR-10
CIFAR-100
SVHN
ArchitectureFCNN
ResNet- ResNet- ResNet- ResNet-
WRN-
ResNet- ResNet- ResNet- ResNet-
WRN- 40-4
WRN-
20
56
110-
164-pre
40-4
20
56
110-
164-pre
16-4
pre
pre
1.0
0.1
0.06
0.045
0.08
0.04
0
0.08
0.04
0.04
0.04
0.04
Filters ignored
24
26
179
241
5376
2227
208
75
2684
924
336
300
672
2016
3984
11856
5136
672
3016
3984
11836
$136
1532
300
2000
800
600
200
1000
400
100
500
-0.1
-0.2
0.2
(a) WRN-40-4, CIFAR-10
(b) WRN-40-4, CIFAR-100
(c) WRN-16-4, SVHN
Figure 15: Distribution of slopes b, in WRN-40-4 with dataset CIFAR-10 and CIFAR-100 in (a) and (b) respectively; and distribution of slopes by in WRN-16-4 with dataset SVHN in (c).
A.3 Comparison of the distribution of the slopes of RReLU for WideResNets
In the main manuscript, the distributions of the slope of RReLU are shown for ResNets for CIFAR-10 and CIFAR-100 datasets. The same with WRN-16-4 and WRN-40-4 architectures are shown in Fig. [5] is clear the figure that many slopes take values close to zero, indicating that the corresponding filters are not useful. For WideResNets, a great number of filters could be ignored without any degradation performance. For accuracy and saving in memory and computation, Table 1 in the main manuscript.
26

---

Published in Transactions on Machine Learning Research (05/2024)
400
2000
1500
200
2000
100
1000
0.9
2.0
-3.5
0.5
3.5
0.2
(a) ResNet-20
(b) ResNet-56
(c) Res.et- 10--10-pre
(d) BosNet-1644ppm
250
100
200
1500
2000
1500
1000
1000
50
500
(e) ResNet-20
(1) ResNet-56
(g) ResNet-110-pra
(h) ResNet--64-pre
Figure 16: Distribution of BN parameters (n) with CIFAR-10 (top row) and CIFAR-100 (bottom row).
A.4 Distribution of the batchnorm paramters
The distribution of batchnorm parameters are presented in Fig. 16 for architectures ResNet-20,, ResNet-56, ResNet-110-pre and RosNet-164-pre corresponding to the distribution of the slopes by in Fig. 7 of the main manuscript.
A.5 Typical values taken by " and by
In ResNet-164 architecture, there are total of 163 convolution layers and one linear layer at the end before softmax layer. There are total of 54 Residual blocks out of which we show the scatter plot for couple of Residual blocks in Fig. 17 which shows the typical values taken by 7 and by-
A.6 Finding the filter-path length
Fig. 13 in the main manuscript shows the fitte-path length of WRNN-0044 architecture with RReLU, The metric filter-path length is defined by us, reprosenting the number of filters the features pass through. We get paths of different lengths in ResNets depending whether we choose the direct residual connection because features may pass via the residual or direct connections in every ResNet unit Veit et al. (2016). If we consider the direct path, as the number of filters is zero, the value of the filter-path length metric also zero. Similarly, if we consider the residual path, the features pass via the filters; therefore, the filter-path length is the total number of active filters in that residual unit. the number of filters reduces at every ResNet unit bocause of using RReLU, the filter-path length also reducos, indicating that features choose the shortest paths to pass.
A.7 A discussion speedup with the sparse models
FLOP reduction serves 2017 Gordon et al. 2018). In our specific computational environment, operating within a 64- system, even though the elements of the RReLU slope (bx) converge to zero, they are still represented as 64-bit real numbers. Consequently, under the current system configuration, any significant speedup not anticipated. Further, the same model may demonstrate different exocution times based on various hardware configura- tions. For example, in Fig. 15.5.6 of Tu al. (2022), the peak performance in terms of TFLOPs is different in different deep learning processors. In single chip also, the performance varies based the floating
indicative measure of speedup (Gao et al. 2021 Han et al. 2015 Li et al.
27

---

Published in Transactions on Machine Learning Research (05/2024)
0.1
0.0
0.0
-0.1
-0.1
-0.1
-0.1
0.00
0.05
0:30
0.15
0.05
0.30
0.15
0.00
0.10
0.15
0.20
0.25
0.05
0.30
0.15
0.20
(a) Block 2
(b) Block 8
(c) Block 13
(d) Block 18
0 00
0.0
0.0
0.0
-0.1
-0.1
-0.1
-0.2
0225 0.250
0.00
0.05
0.20
(e) Block 20
(I) Block 26
(g) Block 31
(h) Block 36
0.1
0.1
0.2
0.1
0.0
0.0
-0.1
-0.1
-0.1
-0.2
-0.2
0.00
0.05
0.10
0.15
0.20
0.25
0.00
0.05
0.10
0.15
0.20
0.25
0.0
0.2
0.3
0.0
0.1
0.2
0.3
(i) Block 38
(i) Block
(k) Block 49
(1) Block 54
Figure 17: Plot of BN parameters (n) RReLU slopes (b,) in different residual blocks of ResNet-164- LIRReLU.
point format (BF16 or FP32). Moreover, the same architecture will have different speedups in different frameworks such as PyTorch, Caffe, Tensorflow, etc (Hadidi et al. 2019). As our focus here is primarily on the algorithmic aspect, hardware//rammewokk considerations are currently not the scope of this paper.
However, to get an idea about speedup concerning the saving in memory and FLOP reported in the paper, have done the following. First, we have trained the model with RReLU which makes some of the filters insignificant after training. Here, we have shown the study with VGG and a ResNet network. As RRcLU can ignore set of filters and sub-filters in two consocutive layers, for inference we can create a smaller architecture, and overwrite its parameters by the values of the significant filters from the original trained model. The smaller model performs the same as the original model but has a lesser number of filters. Therefore, the inference time of this smaller model will be less. As the inference time metric for showing speedup Bianco et al. (2018) therefore we measured the total inference time of both the original model and the smaller model in GPU device. The below results are averaged over multiple iterations.
Activation FLOP
ReLU (Baseline)
RReLU (Unpruned)
RReL.U (Pruned)
1270 Mn
1270 Mn
1010 Mn
Memory
20 Mn
20 Mn
6.4 Mn
Average inference time GPU memory utilization
1.06s
1.03s
0.62s
81.7%
81.7%
77.90%
Table 10: VGG
In Table I0 and Table II we reported the average inference time of VGG and ResNet-164-pms respectively in Quadro P2200 5GB RAM GPU while taking average over 200 forward pass. Please note that the
28

---

Published in Transactions on Machine Learning Research (05/2024)
Activation
ReLU (Baseline)
RReLU (Unpruned)
RReLU (Pruned)
FLOP
478 Mn
478 Mn
307 Mn
Memory
1.7 Mn
1.7 Mn
0.92 Mn
Avcrage inference time
3.09s
3.59s
2.14s
GPU memory utilization
84.2%
86.7%
82.9%
Table 11: ResNet-164-pre
average inference time of the prumed model using the RRRLUU slope is lower for both VGG and ResNet- 164-pre networks. The GPU utilization is also reported in the last row. We do not have control over the GPU utilization as the GPU uses optimal cores based on batch size and specific GPU configuration. But we have taken a higher batch size to keep them in the same range across all three architectures (i) RcLU (Baseline), (ii) RReLU (Unpruned), and (iii) RReLU (Pruned). For example for VGG, a batch size of 2048 is used whereas for ResNet- 164-pr, a batch size 3000 is used. Keeping the utilization at the same level, we compare the inference time, which shows that RRcLU (Pruned) has the minimum inference time thercfore it gives the maximum speedup.
A.8 Distribution of batchnorm scaling parameters and RReLU slopes when both are trainable
We plot the change distribution of 7 Fig. 18 and the shift distribution of by in Fig. 19a and Fig, 19c for the first, second and third layers of every residual blocks, respectively.
Fig.
19b
Figure 18: Distribution of for the fourth case where both Y and b trainable
29

---

Published in Transactions on Machine Learning Research (05/2024)
22
7
penmmttrr mrdil kyyrt 13
pramddrrrrrodl lapet 15makia
0
pasnearsrted 17miula
(a)
First layer at every ResNet block
www.rrrtookii...ooo lye a
22
www.rrolooble oe Linela
22
worddaabooki kys 1.7rxbta R
caansasireddddeet NZnddta
7
13
26
(b) Second layer every ResNet block
wwwdor koye Rodulla
12
pars pasmearsioool 1ysttt
monndeaaadd à 1.r
1
pasnessirotddlget 12medta
7 13
24
tnodia byyei 12 ml
1:
poomeamttwoie aye' mumua
4
punmerrcrrodll byerl 1Snak1a
ponnonectodd rakia
pooneauired 17mula
7
(c) Third layer at every ResNet block
Figure 19: Distribution of by for the fourth case where both Y and by are trainable
30


Reviews:
[{"review_id": "Ds9LFGEASO", "reviewer": "Reviewer_ZHyv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. The proposed method is simple, clear, and easy to follow.\n2. The proposed method is applied on both ReLU and GeLU, and across a wide range of common deep learning models and tasks, showing consistent performance improvements.\n3. The proposed method can lead to improvement in sparsity and robustness, which are important properties for deep learning models\n\n## Weakness\n1. The major drawback of the proposed method lies in the need of extensively more training epochs. Results in Table 1-3 are achieved with a mixed of training epochs, making it hard to decide if the comparison is fair. Also the long training epochs itself raise a red flag on the proposed method. It should be explained if the long training needed to just let the model converge or is this needed for RReLU to induce sparsity. Results of RReLU shoudl also be reported at smaller number of training epochs corresponding to common design choices in previous methods.\n2. The paper mentions \"without a regualrizer\" as a motivation in the introduction, but regularizers are still used in a large portion of experimental results of the proposed method. Furthermore, even without explicit L1, regualrizers like weight decay can also draw weight values smaller, leading to sparsity in the ReLU slopes. Whether weight decay is used and what the relationship between these regularizers and the final performance is are not well explained.\n3. Some reasoning in the paper is not well grounded. For example, in Sec. 2.1 the paper discusses the interaction between RReLU slope b and BN scaling factor gamma. Since the two scalars are multiplied directly, without a regularization the gradient passing on them should be symmetric, so having both trainable parameters seems not to be necessary. Also in Sec. 4, the paper argues a small slope can lead to small Lipschitz, yet a small slope can be followed by a large scaling factor of BN or larger weight values in the layer, not necessarily constraining the Lipschitz", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel learnable activation function, RReLU, to halp improve the learning ability of the model while inducing implicit sparsity for model compression. The paper claims the RReLU activation can help choosing less filters for features to pass through, so sparsity can be achieved intrinsically without a regularizer. Better adversarial robustness is also observed from the experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Revision should be made to clearly discuss the impact of regularization on RReLU (both L1 and weight decay), to determine if regularization is necessary for inducing sparsity in the RReLU slopes.\n2. The impact on longer training time should be clearly discussed, and experiments should be conducted fairly with the same training budget.\n3. The theortical reasoning needs strengthen, like how is the RReLU slope interacting with BN, and why exactly is intrinstic sparsity can be induced. Maybe visualizing the optimization curve of some RReLU slop parameter will help.\n4.  The correspondence of results in Table 1-3 needs further explaination, on whether they are trained under the same setting.\n5. (Minor issue) When refering to equations, it would be the best to have () around the equation number, e.g. Equ. (1). Also, there are a lot of in-line equations in Sec. 4, making the paragraph hard to read. Some reformatting will help.", "broader_impact_concerns": "No concerns on the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sRtcyJdf8O", "reviewer": "Reviewer_82c2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The paper introduces a novel method to sparsify vision networks used in image classification. This is achieved during the training phase by incorporating a new trainable parameter into the ReLU activation function. This approach aims to enhance sparsification without necessitating additional post-training or re-training procedures.\n- Additionally, the paper investigates benefits beyond network sparsity, particularly focusing on the aspect of adversarial robustness. The aim is to understand how this method contributes to improving the network's resilience against adversarial attacks.\n\nWeakness\n- Due to added trainable parameters in ReLU function, this method requires longer training epochs to explore changed loss landscape. The expansion of the CIFAR-10 experiment from 200 to 1200 epochs may not seem problematic, but when an experiment's duration increases by 4 to 10 times, it raises concerns. The extended experiment length makes comparisons more challenging, and the figures provided in this paper aren't persuasive.\n- It's unfortunate that the majority of experimental results are focused on small datasets like CIFAR and primarily on highly over-parameterized networks. The experimentation with ImageNet is limited to very large networks suitable for image classification, such as WRN/ViT, while ignoring numerous studies on structured pruning results for ResNet-18/50. Over-parameterized networks are inherently likely to benefit greatly from compression, thus using them as a basis for pruning research may not be the best approach. Also, using ViT for these experiments is questionable due to its over-parameterization.\n- It is a letdown that the research overlooks networks like MobileNet and EfficientNet, which have shown superior results in vision post-ResNet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Rotated ReLU Activation (RReLU), a novel approach to enhance sparsification in vision models. Key highlights include:\n\n* Rotated ReLU Activation (RReLU): This paper propose a Rotated ReLU activation, which involves rotating the ReLU activation function to give it an additional degree of freedom. This rotation, combined with an appropriate initialization scheme, leads to implicit sparsification of the network without the need for regularizers. This method shows significant improvements in the representation capability of network filters and leads to substantial memory and computational savings. For instance, in the WRN-40-4 architecture, RReLU achieved a 63.37% reduction in memory and a 69.7% reduction in FLOP-count, all without compromising performance.\n* Compatibility with Other Techniques: RReLU can be used alongside other network compression techniques. An example explored in the study is the combination of RReLU with L1 regularization of batch normalization scaling parameters, which results in enhanced sparsity and accuracy.\n* Adversarial Robustness: The study also touches upon the adversarial robustness of networks with RReLU activation. It suggests that networks with RReLU could potentially offer better robustness against adversarial attacks than networks with traditional ReLU activations, due to the inherent properties of RReLU affecting the local Lipschitz constant of the network.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Several years ago, the results of such experiments might have held significance, but nowadays, datasets like CIFAR-10, MNIST, SVHN, and models like Wide-ResNet are seen as mere toy examples in the context of pruning research. To truly prove the usefulness of this approach, it's essential to demonstrate its effectiveness on more complex, denser networks and to compare it directly with the most recent Structured Pruning research. It's also vital to have a broader range of studies on ImageNet and to explore how these methods perform in applications beyond image classification.\n2. It's necessary to find ways to mitigate the disadvantages associated with the extended duration of epochs.\n3. I believe there's a need for a 2D graph that effectively shows the relationship between Accuracy and Pruned FLOPS, similar to what's presented in Figure 1 of the EfficientNet paper.\"", "broader_impact_concerns": "This research focuses on the compression of existing networks, so there is no need to discuss additional ethical aspects or social issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N5qG4h9sG2", "reviewer": "Reviewer_7zBv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed approach is sound and quite generic. RReLU networks have higher FLOP reduction for the same accuracy compared to L1-BN baseline. \n\n- This work introduces an elaborate initialization scheme for learnable parameters in activations and provides an empirical study on the impact of the initialization and the evolution of trainable PReLU slopes during training. Authors identify the impact of PReLU on batchnorm parameters and corresponding compressibility. \n\n- Networks with parametric activations are noticeably more robust to standard adversarial attacks compared to vanilla ReLU networks. \n\n**Weaknesses**\n\n- Method requires substantially longer training compared to the original dense training recipe to achieve good performance. Whereas such extended training is affordable for CIFAR-10/100, similar procedure for larger datasets, i.e ImageNet-1k becomes rather expensive. Pruned model achieves performance close to the dense model trained for longer, however, for such long training dense model could have reached optimal performance and already entered an overfitting regime. Sparse networks converge slower and benefit more from extended training, as recent work [1] shows. \n\n- The method is missing comparison with several more recent structured pruning methods - Chex [2], EagleEye [3].  \n\n- The param and FLOP reduction is provided, but the actual speed-ups are not reported.\n\n- ImageNet-1k training setup is not described in sufficient detail. \n\n---\n[1] Kuznedelev, Denis, et al. \"Accurate Neural Network Pruning Requires Rethinking Sparse Optimization.\" arXiv preprint arXiv:2308.02060 (2023).\n\n[2] Z. Hou, M. Qin, F. Sun, X. Ma, K. Yuan, Y. Xu, Y.-K. Chen, R. Jin, Y. Xie, and S.-Y. Kung, “Chex: Channel exploration for cnn model compression,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2022, pp. 12 287–12 298.\n\n[3] B. Li, B. Wu, J. Su, and G. Wang, “Eagleeye: Fast sub-net eval- uation for efficient neural network pruning,” in Proc. Eur. Conf. Comput. Vis. Springer, 2020, pp. 639–654.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a family of activation functions with trainable parameters, where the magnitude of the multiplier, proportional to the slope of activation, serves as a measure of feature importance. The introduced family comprises several widely used activations - ReLU, GELU. Approach is validated on several computer vision models and benchmarks and compared with L1-BN regularization. In addition to learned sparsity networks parametric activations are shown to be more adversarially robust.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Changes \n\n- I would recommend to compare with several stronger baselines to fully appreciate the benefits and contribution of the introduced method. \n\n- The speedups could significantly vary between hardware and details of the environment. Nevertheless, I think this work could benefit from reporting GPU (or CPU) speedups on some configuration. \n\n- ImageNet results are important as proof of the method scalability. The current version doesn’t specify the training recipe (number of training epochs, augmentation/regularization pipeline) used for training of Wide-ResNet and ViT-S/16. The latter model was trained on proprietary JFT data in the original work before finetuning on ImageNet-1k. Does this work apply the same pretraining procedure? Are the sparse models trained with the same compute budget as the dense models?", "broader_impact_concerns": "No need to add Broader Impact Statement", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting way to encourage sparsity. The authors used a wide variety of datasets and architectures. The scale and the relevance of the experiments could be improved in the next iteration of this work.
Decision: Accept (poster)