# Improving Convolutional Neural Networks with Competitive Activation Function

Authors: Yao Ying; Nengbo Zhang; Ping He; Silong Peng
Journal: Security and Communication Networks
 (2021)

---

## Abstract

The activation function is the basic component of the convolutional neural network (CNN), which provides the nonlinear transformation capability required by the network. Many activation functions make the original input compete with different linear or nonlinear mapping terms to obtain different nonlinear transformation capabilities. Until recently, the original input of funnel activation (FReLU) competed with the spatial conditions, so FReLU not only has the ability of nonlinear transformation but also has the ability of pixelwise modeling. We summarize the competition mechanism in the activation function and then propose a novel activation function design template: competitive activation function (CAF), which promotes competition among different elements. CAF generalizes all activation functions that use competition mechanisms. According to CAF, we propose a parametric funnel rectified exponential unit (PFREU). PFREU promotes competition among linear mapping, nonlinear mapping, and spatial conditions. We conduct experiments on four datasets of different sizes, and the experimental results of three classical convolutional neural networks proved the superiority of our method.


---

## Body

## 1. Introduction

Since the convolutional neural network (CNN) [1] was proposed, activation function has always been an important part of CNN. Traditional activation functions such as sigmoid and tanh bring the gradient vanishing problem, which makes the deep convolutional neural network (DCNN) difficult to optimize. Rectified linear unit (ReLU) [2, 3] alleviates the problem of vanishing gradient, which is one of the major factors in the recent renaissance of CNN [4].Compared with the traditional activation functions, ReLU has a stable improvement but still has its own shortcomings. One of the main disadvantages of ReLU is the dead neurons. ReLU makes the original input compete with a constant term 0, thus obtaining the nonlinear transformation ability, resulting in some neurons being untrained during the whole training process. Many subsequent modifications were proposed to avoid the problem of neuron death during training. LReLU [5] and PReLU [6] make the original input compete with the linear mapping term to obtain the ability of nonlinear transformation while also solves the problem of neuron death. ELU [7] makes the nonlinear mapping term compete with the original input for nonlinear transformation capability. Maxout [8] enables multiple linear mapping terms to compete with each other for the capability of nonlinear transformation. The number of linear mapping terms participating in the competition in Maxout is not fixed, which depends on the demand. In order to further enhance the nonlinear transformation ability, Ramachandran et al. [9–11] propose a nonmonotonic activation function: Swish. Compared with the monotonous activation functions, Swish performs better in many tasks and gradually replaces ReLU as the default activation function in CNN. Mish [12] is another monotonic activation function after Swish. Recently, funnel activation (FReLU) [13] makes the original input compete with the spatial condition term. The spatial condition is a simple spatial context feature extractor. After using this condition, FReLU not only has the nonlinear transformation ability like the previous activation functions but also has the pixelwise modeling capacity to grasp the context information. In summary, competition mechanisms are ubiquitous in the activation function, and the number and types of elements participating in the competition are not restricted.In this paper, we summarize the competition mechanism in the activation function and propose a novel activation function design template: competitive activation function (CAF). CAF promotes competition among different elements. The number and types of competing elements in CAF are not fixed; they vary according to demand. CAF generalizes most of the current activation functions. Based on CAF, we propose a concrete instance: parametric funnel rectified exponential unit (PFREU). PFREU promotes competition among linear mapping, nonlinear mapping, and spatial conditions. We conduct experiments using Fashion-MNIST [14], CIFAR-10/100 [15], and Tiny ImageNet [16] datasets to evaluate the effectiveness of our method.The rest of this paper contains the following sections. Section2 presents related works. Section 3 describes our method. In Section 4, we detail our experimentations. Section 5 gives a detailed analysis of PFREU. In Section 6, we conclude this paper.


## 2. Related Work

### 2.1. Conventional Activation Functions

The activation function provides the nonlinear transformation capability required by CNN. As shown in Figure1, the conventional activation function focuses on different nonlinear transformations. ReLU uses identity linear mapping in the positive quadrant, which alleviates the problem of gradient disappearance and makes it possible to train DCNN. However, the constant zero of the negative part of ReLU causes the problem of the zero gradient. Zero gradient will cause some neurons to fail to be trained during training. LReLU uses a small fixed slope value in the negative part of the ReLU to avoid the problem of the zero gradient. However, the performance of LReLU will be greatly affected by the predefined initial value of the slope. In order to avoid the predefined fixed slope values affecting the performance of CNN, PReLU makes the slope value learnable. Different from the above work, linear mapping is used in the negative quadrant of ReLU, while ELU uses nonlinear mapping in the negative quadrant of ReLU to avoid the problem of zero gradient. The exponential term in the negative quadrant of ELU makes the activation mean close to 0. This feature makes the gradient closer to the nature gradient [17] and also speeds up the learning process of the network. The exponential term is saturated on the negative part so that the ELU can learn a more robust and stable representation. Scaled exponential linear unit (SELU) [18] is a modification of ELU. SELU induces self-normalizing properties to normalize its own output. Swish is a recently proposed nonmonotonic activation function, which has a stronger nonlinear transformation capability than the previous monotonic activation functions. As can be seen from Figure 1, Mish is another nonmonotonic activation function similar to Swish. Both Tanh and Softplus [19] are used in Mish, so the computational cost of Mish is higher than that of other activation functions. In short, conventional activation functions by itself bring nonlinear transformation capabilities to the neural network.Figure 1
Conventional activation functions.


### 2.2. Context Conditional Activation Functions

Different from the conventional activation functions, the context conditional activation function brings contextual information into the activation function. As shown in Table1, Maxout [8] expands the input to multiple branches and selects the maximum value in an element-wise way. Maxout generalizes ReLU and does not have the problem of dead neuron. Compared with ReLU in the classification task, Maxout shows a clear improvement. However, multiple branches significantly increase the number of parameters and computational cost. Probout [20] is a modification of Maxout. Probout uses a probabilistic sampling procedure to replace the maximum operation in Maxout to improve its invariance property. The model combined with Probout unit achieves competitive performance on multiple datasets. However, using Probout unit is computationally expensive when testing. FReLU is another activation function integrating the context information. Compared with the conventional activation functions, FReLU adds a spatial condition. The spatial condition provides pixelwise modeling capability, so FReLU can capture contextual information. Similar to the conventional activation functions, FReLU uses max⋅ to obtain the nonlinear transformation capability. FReLU solves the long-standing spatial insensitivity problem in conventional activation functions and only increases the negligible computational cost. Although FRELU compared with the conventional activation functions has the above advantages, FReLU’s nonlinear transformation ability is weaker.Table 1
Common activation functions in a competitive manner.
MethodDefinitionReLUfx=maxx,0LReLUfx=maxx,0.01xPReLUfxi=maxxi,pixiELUfx=maxx,αe−x−1SELUfx=λx,ifx>0,αex−α,ifx≤0Softplusfx=log1+exSwishfx=x⋅sigmoidxMishfx=x⋅tanhsoftplusxMaxoutfx=maxkwkTx+bkFReLUfx=maxx,Tx


### 2.3. Competition Mechanism in CNN

There are many competitive mechanisms in CNN. The widely used maxpooling operation is a typical case of the competition mechanism: the max value in the pooling region is selected. In Table1, we write common activation functions in a competitive manner. Many activation functions contain competition mechanisms. ReLU makes the original input compete with constant 0 to obtain nonlinear transformation capability. LReLU and PReLU make the original input compete with a linear mapping to obtain nonlinear transformation capability. ELU makes the original input compete with nonlinear mapping to obtain nonlinear transformation capability. Unlike the previous activation functions, which had only two terms, Maxout makes multiple linear mapping terms compete with each other. The number of competing terms in Maxout is not fixed. Local winner take all (LWTA) [21, 22] is a work proposed at the same time as Maxout. The difference between LWTA and Maxout is that, in addition to the maximum output value, LWTA sets the remaining values to 0. FReLU makes the linear mapping term compete with the spatial condition term. Liao et al. [23] propose a novel CNN module that promotes competition among different sizes of convolutional filters. This module is used in the classic CNN models to produce state-of-the-art results on the most commonly used datasets.


## 2.1. Conventional Activation Functions

The activation function provides the nonlinear transformation capability required by CNN. As shown in Figure1, the conventional activation function focuses on different nonlinear transformations. ReLU uses identity linear mapping in the positive quadrant, which alleviates the problem of gradient disappearance and makes it possible to train DCNN. However, the constant zero of the negative part of ReLU causes the problem of the zero gradient. Zero gradient will cause some neurons to fail to be trained during training. LReLU uses a small fixed slope value in the negative part of the ReLU to avoid the problem of the zero gradient. However, the performance of LReLU will be greatly affected by the predefined initial value of the slope. In order to avoid the predefined fixed slope values affecting the performance of CNN, PReLU makes the slope value learnable. Different from the above work, linear mapping is used in the negative quadrant of ReLU, while ELU uses nonlinear mapping in the negative quadrant of ReLU to avoid the problem of zero gradient. The exponential term in the negative quadrant of ELU makes the activation mean close to 0. This feature makes the gradient closer to the nature gradient [17] and also speeds up the learning process of the network. The exponential term is saturated on the negative part so that the ELU can learn a more robust and stable representation. Scaled exponential linear unit (SELU) [18] is a modification of ELU. SELU induces self-normalizing properties to normalize its own output. Swish is a recently proposed nonmonotonic activation function, which has a stronger nonlinear transformation capability than the previous monotonic activation functions. As can be seen from Figure 1, Mish is another nonmonotonic activation function similar to Swish. Both Tanh and Softplus [19] are used in Mish, so the computational cost of Mish is higher than that of other activation functions. In short, conventional activation functions by itself bring nonlinear transformation capabilities to the neural network.Figure 1
Conventional activation functions.


## 2.2. Context Conditional Activation Functions

Different from the conventional activation functions, the context conditional activation function brings contextual information into the activation function. As shown in Table1, Maxout [8] expands the input to multiple branches and selects the maximum value in an element-wise way. Maxout generalizes ReLU and does not have the problem of dead neuron. Compared with ReLU in the classification task, Maxout shows a clear improvement. However, multiple branches significantly increase the number of parameters and computational cost. Probout [20] is a modification of Maxout. Probout uses a probabilistic sampling procedure to replace the maximum operation in Maxout to improve its invariance property. The model combined with Probout unit achieves competitive performance on multiple datasets. However, using Probout unit is computationally expensive when testing. FReLU is another activation function integrating the context information. Compared with the conventional activation functions, FReLU adds a spatial condition. The spatial condition provides pixelwise modeling capability, so FReLU can capture contextual information. Similar to the conventional activation functions, FReLU uses max⋅ to obtain the nonlinear transformation capability. FReLU solves the long-standing spatial insensitivity problem in conventional activation functions and only increases the negligible computational cost. Although FRELU compared with the conventional activation functions has the above advantages, FReLU’s nonlinear transformation ability is weaker.Table 1
Common activation functions in a competitive manner.
MethodDefinitionReLUfx=maxx,0LReLUfx=maxx,0.01xPReLUfxi=maxxi,pixiELUfx=maxx,αe−x−1SELUfx=λx,ifx>0,αex−α,ifx≤0Softplusfx=log1+exSwishfx=x⋅sigmoidxMishfx=x⋅tanhsoftplusxMaxoutfx=maxkwkTx+bkFReLUfx=maxx,Tx


## 2.3. Competition Mechanism in CNN

There are many competitive mechanisms in CNN. The widely used maxpooling operation is a typical case of the competition mechanism: the max value in the pooling region is selected. In Table1, we write common activation functions in a competitive manner. Many activation functions contain competition mechanisms. ReLU makes the original input compete with constant 0 to obtain nonlinear transformation capability. LReLU and PReLU make the original input compete with a linear mapping to obtain nonlinear transformation capability. ELU makes the original input compete with nonlinear mapping to obtain nonlinear transformation capability. Unlike the previous activation functions, which had only two terms, Maxout makes multiple linear mapping terms compete with each other. The number of competing terms in Maxout is not fixed. Local winner take all (LWTA) [21, 22] is a work proposed at the same time as Maxout. The difference between LWTA and Maxout is that, in addition to the maximum output value, LWTA sets the remaining values to 0. FReLU makes the linear mapping term compete with the spatial condition term. Liao et al. [23] propose a novel CNN module that promotes competition among different sizes of convolutional filters. This module is used in the classic CNN models to produce state-of-the-art results on the most commonly used datasets.


## 3. Methodology

In this section, we first introduce the definition of CAF and then derive the PFREU.


### 3.1. Competitive Activation Function

As mentioned above, the competition mechanism is widely used in the activation functions. Most of the activation functions compete between two terms or multiple identical types of terms. We summarize the competition mechanism in the activation function and propose a novel activation function design template: CAF. The definition of CAF can be formulated as follows:(1)fx=maxLx,Nx,Tx,C,…,where Lx is the linear mapping term, Nx is the nonlinear mapping term, Tx is the spatial condition term, and C is a constant term. In equation (1), we can see four types of elements, but CAF does not limit the number or types of terms that participate in the competition. Adding elements beyond these four types to the CAF also complies with the definition of CAF. CAF generalizes all activation functions that use max⋅. We use CAF-α to denote CAF with α terms (it should be noted that the original input x (in equation (1)) is a special linear mapping multiplied by 1).As we can see from Table1, most activation functions can be seen as an instance of CAF-2. ReLU makes the linear mapping term compete with the constant term. LReLU and PReLU make competition between two linear mapping terms. ELU makes the linear mapping term compete with the nonlinear mapping term. FReLU makes the linear mapping term compete with the spatial condition term. Maxout can be seen as an instance of CAF-k. We believe that Maxout and FReLU are two representative CAF. Maxout shows that the number of elements participating in the competition is not fixed. FReLU indicates that the types of elements participating in the competition are not fixed, and new element types can be continuously added to enhance CAF. All in all, most current activation functions are constructed by competing among the four types of elements.We propose a simplified version of CAF-3:(2)fx=maxLx,Nx,Tx.It is worth noting that CAF-3 represents all situations where the three elements compete with each other, and here is just one of them. For convenience, we call it CAF-3. It can be seen from Figure2 that the biggest difference between CAF-3 and conventional activation functions is that CAF-3 adds additional spatial conditions, so CAF has the ability of pixelwise modeling to grasp contextual information. The main difference between CAF-3 and FReLU is that CAF-3 adds a nonlinear mapping term, so compared with FReLU, CAF-3 has a stronger nonlinear transformation capability. CAF-3 adopts a competition mechanism among linear mapping, nonlinear mapping, and spatial conditions to achieve a balance between nonlinear transformation capability and spatial information acquisition capability.Figure 2
Graphical description of the corresponding activation function. (a) ReLU. (b) ELU. (c) PReLU. (d) FReLU. (e) CAF-3.
(a)(b)(c)(d)(e)


### 3.2. Parametric Funnel Rectified Exponential Unit

In this section, based on the concept of CAF-3, we propose a new type of activation function: PFREU. We present three variations of PFREU.


#### 3.2.1. PFREU-A

(3)fxc=maxαcxc,βcxc⋅e−xc,Txc,(4)Txc=BNDWConvxc.Herexc is the input of PFREU-A on the cth channel, αc is the coefficient that controls the linear mapping, and βc is the coefficient that controls the nonlinear mapping. The subscript c in αc and βc indicates that we allow linear mapping and nonlinear mapping to vary on different channels. Function T⋅ represents the spatial context feature extractor. In equation (4), DWConv [24, 25] represents the depthwise separable convolutional layer, and BN [26] is an abbreviation for batch normalization operation. We use the Xavier [27] initialization strategy to initialization the depthwise separable convolutional layer. We set the initial values of αc and βc to be 1.


#### 3.2.2. PFREU-B

(5)fxc=maxαcxc,xc⋅e−γcxc,Txc.Comparing equations (3) and (5), we can see that the difference between PFREU-B and PFREU-A lies in the nonlinear mapping term. In PFREU-A, the learnable parameter βc is the coefficient of the exponent. In PFREU-B, the learnable parameter γc is a power exponent. The subscript c in γc indicates that we allow nonlinear mapping to vary on different channels. We set the initial values of αc and γc to be 1.


#### 3.2.3. PFREU-C

(6)fxc=maxαcxc,βcxc⋅e−γcxc,Txc.PFREU-C combines the learnable parameters in PFREU-A and PFREU-B. Therefore, PFREU-C is the activation function with the most parameters among all PFREU variants. Considering that the weight decay tends to push the parameter values to 0, we do not use the weight decay for the learnable parameters in all PFREU variants. It should be noted that all activation functions that conform to equation (2) belong to CAF-3. We propose PFREU to verify the effectiveness of the CAF algorithm. PFREU is just one instance of the CAF-3 algorithm.


## 3.1. Competitive Activation Function

As mentioned above, the competition mechanism is widely used in the activation functions. Most of the activation functions compete between two terms or multiple identical types of terms. We summarize the competition mechanism in the activation function and propose a novel activation function design template: CAF. The definition of CAF can be formulated as follows:(1)fx=maxLx,Nx,Tx,C,…,where Lx is the linear mapping term, Nx is the nonlinear mapping term, Tx is the spatial condition term, and C is a constant term. In equation (1), we can see four types of elements, but CAF does not limit the number or types of terms that participate in the competition. Adding elements beyond these four types to the CAF also complies with the definition of CAF. CAF generalizes all activation functions that use max⋅. We use CAF-α to denote CAF with α terms (it should be noted that the original input x (in equation (1)) is a special linear mapping multiplied by 1).As we can see from Table1, most activation functions can be seen as an instance of CAF-2. ReLU makes the linear mapping term compete with the constant term. LReLU and PReLU make competition between two linear mapping terms. ELU makes the linear mapping term compete with the nonlinear mapping term. FReLU makes the linear mapping term compete with the spatial condition term. Maxout can be seen as an instance of CAF-k. We believe that Maxout and FReLU are two representative CAF. Maxout shows that the number of elements participating in the competition is not fixed. FReLU indicates that the types of elements participating in the competition are not fixed, and new element types can be continuously added to enhance CAF. All in all, most current activation functions are constructed by competing among the four types of elements.We propose a simplified version of CAF-3:(2)fx=maxLx,Nx,Tx.It is worth noting that CAF-3 represents all situations where the three elements compete with each other, and here is just one of them. For convenience, we call it CAF-3. It can be seen from Figure2 that the biggest difference between CAF-3 and conventional activation functions is that CAF-3 adds additional spatial conditions, so CAF has the ability of pixelwise modeling to grasp contextual information. The main difference between CAF-3 and FReLU is that CAF-3 adds a nonlinear mapping term, so compared with FReLU, CAF-3 has a stronger nonlinear transformation capability. CAF-3 adopts a competition mechanism among linear mapping, nonlinear mapping, and spatial conditions to achieve a balance between nonlinear transformation capability and spatial information acquisition capability.Figure 2
Graphical description of the corresponding activation function. (a) ReLU. (b) ELU. (c) PReLU. (d) FReLU. (e) CAF-3.
(a)(b)(c)(d)(e)


## 3.2. Parametric Funnel Rectified Exponential Unit

In this section, based on the concept of CAF-3, we propose a new type of activation function: PFREU. We present three variations of PFREU.


### 3.2.1. PFREU-A

(3)fxc=maxαcxc,βcxc⋅e−xc,Txc,(4)Txc=BNDWConvxc.Herexc is the input of PFREU-A on the cth channel, αc is the coefficient that controls the linear mapping, and βc is the coefficient that controls the nonlinear mapping. The subscript c in αc and βc indicates that we allow linear mapping and nonlinear mapping to vary on different channels. Function T⋅ represents the spatial context feature extractor. In equation (4), DWConv [24, 25] represents the depthwise separable convolutional layer, and BN [26] is an abbreviation for batch normalization operation. We use the Xavier [27] initialization strategy to initialization the depthwise separable convolutional layer. We set the initial values of αc and βc to be 1.


### 3.2.2. PFREU-B

(5)fxc=maxαcxc,xc⋅e−γcxc,Txc.Comparing equations (3) and (5), we can see that the difference between PFREU-B and PFREU-A lies in the nonlinear mapping term. In PFREU-A, the learnable parameter βc is the coefficient of the exponent. In PFREU-B, the learnable parameter γc is a power exponent. The subscript c in γc indicates that we allow nonlinear mapping to vary on different channels. We set the initial values of αc and γc to be 1.


### 3.2.3. PFREU-C

(6)fxc=maxαcxc,βcxc⋅e−γcxc,Txc.PFREU-C combines the learnable parameters in PFREU-A and PFREU-B. Therefore, PFREU-C is the activation function with the most parameters among all PFREU variants. Considering that the weight decay tends to push the parameter values to 0, we do not use the weight decay for the learnable parameters in all PFREU variants. It should be noted that all activation functions that conform to equation (2) belong to CAF-3. We propose PFREU to verify the effectiveness of the CAF algorithm. PFREU is just one instance of the CAF-3 algorithm.


## 3.2.1. PFREU-A

(3)fxc=maxαcxc,βcxc⋅e−xc,Txc,(4)Txc=BNDWConvxc.Herexc is the input of PFREU-A on the cth channel, αc is the coefficient that controls the linear mapping, and βc is the coefficient that controls the nonlinear mapping. The subscript c in αc and βc indicates that we allow linear mapping and nonlinear mapping to vary on different channels. Function T⋅ represents the spatial context feature extractor. In equation (4), DWConv [24, 25] represents the depthwise separable convolutional layer, and BN [26] is an abbreviation for batch normalization operation. We use the Xavier [27] initialization strategy to initialization the depthwise separable convolutional layer. We set the initial values of αc and βc to be 1.


## 3.2.2. PFREU-B

(5)fxc=maxαcxc,xc⋅e−γcxc,Txc.Comparing equations (3) and (5), we can see that the difference between PFREU-B and PFREU-A lies in the nonlinear mapping term. In PFREU-A, the learnable parameter βc is the coefficient of the exponent. In PFREU-B, the learnable parameter γc is a power exponent. The subscript c in γc indicates that we allow nonlinear mapping to vary on different channels. We set the initial values of αc and γc to be 1.


## 3.2.3. PFREU-C

(6)fxc=maxαcxc,βcxc⋅e−γcxc,Txc.PFREU-C combines the learnable parameters in PFREU-A and PFREU-B. Therefore, PFREU-C is the activation function with the most parameters among all PFREU variants. Considering that the weight decay tends to push the parameter values to 0, we do not use the weight decay for the learnable parameters in all PFREU variants. It should be noted that all activation functions that conform to equation (2) belong to CAF-3. We propose PFREU to verify the effectiveness of the CAF algorithm. PFREU is just one instance of the CAF-3 algorithm.


## 4. Experiments

To verify the effectiveness of our method. We use three CNN models to conduct experiments on four commonly used datasets. In order to exclude the situation where complex data expansion and parameter settings affect the final result, we only use conventional settings. For all models, we choose the Xavier [27] initialization strategy and the classification cross-entropy loss function. All of the results listed in this section are the median for five different tests.


### 4.1. Fashion-MNIST

Fashion-MNIST (F-MNIST) [14] is a fashion product dataset released by Zalando Research. In 10 classes, it contains 70000 images. These images are grayscale images, and the image size is 28 × 28 pixels. In the training set, it contains 60000 images. In the test set, it contains 10000 images. The training set and test set are evenly distributed in each class. The format of F-MNIST is the same as MNIST.We use LeNet-5 [28] to evaluate the performance of different activation functions. For data preprocessing, we only divided the original images by 255. The batch size is set to be 128. We train the network for a total of 20 epochs. We use the cosine shape strategy [29, 30] to set the learning rate. The initial learning rate is 0.01. The weight decay and momentum are set to 0.0005 and 0.9, respectively.The experimental results are shown in Table2. Compared with ReLU, FReLU improves the accuracy from 90.34% to 90.98%. FReLU has a better performance compared with the previous nonlinear activation functions. The result shows that the added spatial condition enables FReLU to have the ability of pixelwise modeling, which is not available in the conventional activation functions, so that the performance of LeNet-5 with FReLU is better. All the results of PFREU variants are better than those of FReLU. We think this is because the added nonlinear mapping term makes PFREU have a stronger nonlinear transformation capability than FReLU. LeNet-5 with PFEU-C obtains the best result in all experiments.Table 2
Classification results (%) with LeNet-5 on Fashion-MNIST.
MethodAccuracy rate (%)ReLU90.34LReLU90.37PReLU90.43ELU90.80SELU90.84Softplus88.87Swish89.93Mish90.25FReLU90.98PFREU-A91.09PFREU-B91.04PFREU-C91.21


### 4.2. CIFAR

CIFAR [15] is one of the most widely used color image datasets. CIFAR consists of two subsets: CIFAR-10 and CIFAR-100. CIFAR-10 consists of 10 classes. 50000 training images and 10000 testing images are equally distributed in each class. This dataset contains 32 × 32 pixel images. The size and format of CIFAR-100 are the same as CIFAR-10, but the number of classes has increased tenfold. Therefore, CIFAR-100 is a much more complex task than CIFAR-10.We used the Network In Network (NIN) [31] and the Residual Network (ResNet) [32] to evaluate the performance of different activation functions. For the NIN model, we use simple data preprocessing: divide the image by 255 and then randomly flip it horizontally. The batch size is set to be 128. We train the network for a total of 200 epochs. The initial learning rate is set to 0.01 and divided by 2 at epoch 80 and then divided by 5 at epoch 140. The weight decay and momentum are set to 0.0001 and 0.9, respectively. For the experiment on ResNet-110, we use three epochs to warm up [33] and the other settings are the same to the original settings.As shown in Table3, the NIN model and the ResNet model with the Softplus always fail to converge. With SELU, these models can converge on CIFAR-10 but fail to converge on CIFAR-100. These activation functions with spatial condition behave differently on the NIN model and the ResNet model. In the NIN model, the performance of FReLU is much better than that of the conventional activation functions. We think this is because the spatial conditions are composed of convolutional layers and BN layers, while NIN is a relatively shallow network with only 9 layers. Using FReLU will significantly increase the number of layers of the NIN model. As we all know, depth [34–38] is very important for the expressive ability of CNN. Therefore, the NIN model with FReLU shows a clear advantage over the conventional activation functions. The ResNet model with 110 layers is much deeper than the NIN model. The layer added in FReLU occupies a much smaller proportion in the ResNet model. Therefore, the performance improvement is relatively small. In addition to the depth of the network, the characteristics of the ResNet model itself also affect the performance of the activation function with spatial conditions. The biggest difference between ResNet and traditional CNN is the shortcut connection [38]. The shortcut connection transfers the shallow feature map directly to the deeper layer. This feature weakens the advantage of FReLU over the conventional activation functions to a certain extent. Therefore, for the ResNet model, the nonlinear transformation ability is more necessary than the pixelwise modeling ability. Therefore, on the ResNet model, the performance of FReLU is worse than that of some conventional activation functions with a stronger nonlinear conversion capability. In CIFAR-100, there are more conventional activation functions that perform better than FReLU than in CIFAR-10. We think this is because the number of classes of CIFAR-100 is 10 times that of CIFAR-10, so an activation function with the stronger nonlinear transformation capability is needed to distinguish different classes.Table 3
Classification results (%) with NIN and ResNet-110 on CIFAR.
ModelNINResNet-110DatasetCIFAR-10CIFAR-100CIFAR-10CIFAR-100ReLU86.9359.8492.6467.27LReLU87.7459.7092.8968.16PReLU88.3462.7191.8869.05ELU88.1861.4892.5869.31SELU∗63.10∗66.36Softplus∗∗∗∗Swish84.88∗92.9068.80Mish87.2059.3293.2068.88FReLU90.7967.3692.9868.91PFREU-A91.1167.4393.3669.73PFREU-B91.2467.6793.3870.34PFREU-C90.9867.4893.2870.18From Table3, we can see that, in the NIN model, PFREU outperforms both the conventional activation functions and FReLU. In the ResNet model, PFREU is still superior to the conventional activation functions and FReLU. In the ResNet model, the gap between PFREU and FReLU is larger than that in the NIN model. As mentioned above, the ResNet model requires the nonlinear transformation capability more than the pixelwise modeling capability. The biggest difference between PFREU and FReLU is that PFREU has a nonlinear mapping term. So PFREU has strong nonlinear transformation ability than FReLU. From Table 3, we can see that PFREU has a good balance between nonlinear transformation capability and pixelwise modeling capability. On CIFAR-10, the performance of PFREU-A is better than that of PFREU-C. In contrast, the performance of PFREU-C is better than that of PFREU-A on CIFAR-100. We think the reason is that the CIFAR-10 dataset is a simple recognition task, while the CIFAR-100 dataset is much more complicated. PFREU-C tends to overfit on CIFAR-10. Among all PFREU variants, PFREU-B achieves the best results. We believe that this shows that more parameters do not mean better performance. The core issue of the activation function is design.


### 4.3. Tiny ImageNet

Tiny ImageNet [16] is a subset of the ImageNet [39] dataset. Tiny ImageNet consists of 200 classes. 100000 training images and 10000 validation images are equally distributed in each class. This dataset contains 64 × 64 pixel images. We use the ResNet-110 model used on CIFAR to evaluate the performance of different activation functions on Tiny ImageNet. In order to match the size of the image and model, we extend the stride of the first convolution layer to 2. Other settings are the same as CIFAR.As we can see from Table4, FReLU is better than most conventional activation functions. As mentioned in the previous section, ResNet weakens the advantages of FReLU over conventional activation functions. The number of classes of Tiny ImageNet is twice the number of classes of CIFAR-100. Therefore, the model needs an activation function with a stronger nonlinear transformation capability to distinguish different classes. The performance of all PFREU variants is better than that of FReLU, we think this shows that PFREU has a stronger nonlinear transformation capability than FReLU. The performance of PFREU and conventional activation functions is comparable. Among all PFREU variants, PFREU-B achieves the best result.Table 4
Classification results (%) with ResNet-110 on Tiny ImageNet.
MethodAccuracy rate (%)ReLU52.04LReLU51.25PReLU52.75ELU50.65SELU49.55Softplus49.91Swish51.41Mish51.40FReLU51.92PFREU-A52.27PFREU-B52.53PFREU-C52.43


## 4.1. Fashion-MNIST

Fashion-MNIST (F-MNIST) [14] is a fashion product dataset released by Zalando Research. In 10 classes, it contains 70000 images. These images are grayscale images, and the image size is 28 × 28 pixels. In the training set, it contains 60000 images. In the test set, it contains 10000 images. The training set and test set are evenly distributed in each class. The format of F-MNIST is the same as MNIST.We use LeNet-5 [28] to evaluate the performance of different activation functions. For data preprocessing, we only divided the original images by 255. The batch size is set to be 128. We train the network for a total of 20 epochs. We use the cosine shape strategy [29, 30] to set the learning rate. The initial learning rate is 0.01. The weight decay and momentum are set to 0.0005 and 0.9, respectively.The experimental results are shown in Table2. Compared with ReLU, FReLU improves the accuracy from 90.34% to 90.98%. FReLU has a better performance compared with the previous nonlinear activation functions. The result shows that the added spatial condition enables FReLU to have the ability of pixelwise modeling, which is not available in the conventional activation functions, so that the performance of LeNet-5 with FReLU is better. All the results of PFREU variants are better than those of FReLU. We think this is because the added nonlinear mapping term makes PFREU have a stronger nonlinear transformation capability than FReLU. LeNet-5 with PFEU-C obtains the best result in all experiments.Table 2
Classification results (%) with LeNet-5 on Fashion-MNIST.
MethodAccuracy rate (%)ReLU90.34LReLU90.37PReLU90.43ELU90.80SELU90.84Softplus88.87Swish89.93Mish90.25FReLU90.98PFREU-A91.09PFREU-B91.04PFREU-C91.21


## 4.2. CIFAR

CIFAR [15] is one of the most widely used color image datasets. CIFAR consists of two subsets: CIFAR-10 and CIFAR-100. CIFAR-10 consists of 10 classes. 50000 training images and 10000 testing images are equally distributed in each class. This dataset contains 32 × 32 pixel images. The size and format of CIFAR-100 are the same as CIFAR-10, but the number of classes has increased tenfold. Therefore, CIFAR-100 is a much more complex task than CIFAR-10.We used the Network In Network (NIN) [31] and the Residual Network (ResNet) [32] to evaluate the performance of different activation functions. For the NIN model, we use simple data preprocessing: divide the image by 255 and then randomly flip it horizontally. The batch size is set to be 128. We train the network for a total of 200 epochs. The initial learning rate is set to 0.01 and divided by 2 at epoch 80 and then divided by 5 at epoch 140. The weight decay and momentum are set to 0.0001 and 0.9, respectively. For the experiment on ResNet-110, we use three epochs to warm up [33] and the other settings are the same to the original settings.As shown in Table3, the NIN model and the ResNet model with the Softplus always fail to converge. With SELU, these models can converge on CIFAR-10 but fail to converge on CIFAR-100. These activation functions with spatial condition behave differently on the NIN model and the ResNet model. In the NIN model, the performance of FReLU is much better than that of the conventional activation functions. We think this is because the spatial conditions are composed of convolutional layers and BN layers, while NIN is a relatively shallow network with only 9 layers. Using FReLU will significantly increase the number of layers of the NIN model. As we all know, depth [34–38] is very important for the expressive ability of CNN. Therefore, the NIN model with FReLU shows a clear advantage over the conventional activation functions. The ResNet model with 110 layers is much deeper than the NIN model. The layer added in FReLU occupies a much smaller proportion in the ResNet model. Therefore, the performance improvement is relatively small. In addition to the depth of the network, the characteristics of the ResNet model itself also affect the performance of the activation function with spatial conditions. The biggest difference between ResNet and traditional CNN is the shortcut connection [38]. The shortcut connection transfers the shallow feature map directly to the deeper layer. This feature weakens the advantage of FReLU over the conventional activation functions to a certain extent. Therefore, for the ResNet model, the nonlinear transformation ability is more necessary than the pixelwise modeling ability. Therefore, on the ResNet model, the performance of FReLU is worse than that of some conventional activation functions with a stronger nonlinear conversion capability. In CIFAR-100, there are more conventional activation functions that perform better than FReLU than in CIFAR-10. We think this is because the number of classes of CIFAR-100 is 10 times that of CIFAR-10, so an activation function with the stronger nonlinear transformation capability is needed to distinguish different classes.Table 3
Classification results (%) with NIN and ResNet-110 on CIFAR.
ModelNINResNet-110DatasetCIFAR-10CIFAR-100CIFAR-10CIFAR-100ReLU86.9359.8492.6467.27LReLU87.7459.7092.8968.16PReLU88.3462.7191.8869.05ELU88.1861.4892.5869.31SELU∗63.10∗66.36Softplus∗∗∗∗Swish84.88∗92.9068.80Mish87.2059.3293.2068.88FReLU90.7967.3692.9868.91PFREU-A91.1167.4393.3669.73PFREU-B91.2467.6793.3870.34PFREU-C90.9867.4893.2870.18From Table3, we can see that, in the NIN model, PFREU outperforms both the conventional activation functions and FReLU. In the ResNet model, PFREU is still superior to the conventional activation functions and FReLU. In the ResNet model, the gap between PFREU and FReLU is larger than that in the NIN model. As mentioned above, the ResNet model requires the nonlinear transformation capability more than the pixelwise modeling capability. The biggest difference between PFREU and FReLU is that PFREU has a nonlinear mapping term. So PFREU has strong nonlinear transformation ability than FReLU. From Table 3, we can see that PFREU has a good balance between nonlinear transformation capability and pixelwise modeling capability. On CIFAR-10, the performance of PFREU-A is better than that of PFREU-C. In contrast, the performance of PFREU-C is better than that of PFREU-A on CIFAR-100. We think the reason is that the CIFAR-10 dataset is a simple recognition task, while the CIFAR-100 dataset is much more complicated. PFREU-C tends to overfit on CIFAR-10. Among all PFREU variants, PFREU-B achieves the best results. We believe that this shows that more parameters do not mean better performance. The core issue of the activation function is design.


## 4.3. Tiny ImageNet

Tiny ImageNet [16] is a subset of the ImageNet [39] dataset. Tiny ImageNet consists of 200 classes. 100000 training images and 10000 validation images are equally distributed in each class. This dataset contains 64 × 64 pixel images. We use the ResNet-110 model used on CIFAR to evaluate the performance of different activation functions on Tiny ImageNet. In order to match the size of the image and model, we extend the stride of the first convolution layer to 2. Other settings are the same as CIFAR.As we can see from Table4, FReLU is better than most conventional activation functions. As mentioned in the previous section, ResNet weakens the advantages of FReLU over conventional activation functions. The number of classes of Tiny ImageNet is twice the number of classes of CIFAR-100. Therefore, the model needs an activation function with a stronger nonlinear transformation capability to distinguish different classes. The performance of all PFREU variants is better than that of FReLU, we think this shows that PFREU has a stronger nonlinear transformation capability than FReLU. The performance of PFREU and conventional activation functions is comparable. Among all PFREU variants, PFREU-B achieves the best result.Table 4
Classification results (%) with ResNet-110 on Tiny ImageNet.
MethodAccuracy rate (%)ReLU52.04LReLU51.25PReLU52.75ELU50.65SELU49.55Softplus49.91Swish51.41Mish51.40FReLU51.92PFREU-A52.27PFREU-B52.53PFREU-C52.43


## 5. Analysis

In this section, we first analyze the two most important abilities in the activation function: nonlinear transformation ability and pixelwise modeling ability. Then, we explore the design factors that led to the performance difference among the PFREU variants. Finally, we analyze the parameter computation of PFREU.


### 5.1. Nonlinear Transformation vs. Pixelwise Modeling

Activation function is the source of the nonlinear transformation ability of the neural network. All activation functions have different degrees of nonlinear transformation capability. Recently, FReLU has introduced the ability of pixelwise modeling in the activation function. Two questions emerged naturally:(1)
Which ability is more important to the activation function?(2)
How to balance these two abilities in the activation function?Experiments conducted on CIFAR and Tiny ImageNet provide some observations. Different models behave differently. As the previous analysis, ResNet has a more powerful spatial information acquisition capability. Therefore, compared with NIN, the activation function with spatial conditions has less impact on ResNet. It also depends on the specific task. CIFAR-10 and CIFAR-100 have different levels of difficulty. Therefore, on the CIFAR-100 dataset, the model needs an activation function with a stronger nonlinear transformation capability. Therefore, the network requires both nonlinear transformation capability and pixelwise modeling capability, which is more important depending on the specific model and task.Conventional activation functions do not have the pixelwise modeling capability. Compared with the traditional activation functions, FReLU’s nonlinear transformation ability is relatively weaker. PFREU adds a nonlinear mapping term to enhance the nonlinear transformation ability. The experimental results inSection 4 prove the effectiveness of PFREU. Therefore, our CAF-3 method can construct an activation function that balances nonlinear transformation and pixelwise modeling capabilities.


### 5.2. The Design of PFREU

The difference among the PFREU variants is the nonlinear mapping term. In particular, the number of parameters of the nonlinear term in PFREU-A and PFREU-B is also the same. We constructed two simple exponential functions to explore how different parameter positions affect the change of the exponential function [40–42].(7)f1x=α⋅ex,f2x=eβ⋅x.The difference betweenf1 and f2 is whether the learnable parameter is a coefficient or a power exponent.It can be seen from Figure3 that, under the same parameter amplitude change, the change of function f2 is greater than that of f1. Another major difference between f1 and f2 is that f2 always passes through the origin. As shown in Figure 1, most conventional activation functions pass through the origin, so we think this feature helps improve performance.Figure 3
(a) The shape off1 with different α values. (b) The shape of f2 with different β values.
(a)(b)We calculate the gradient off1 and f2 with respect to input x:(8)∂f1x∂x=α⋅ex,(9)∂f2x∂x=β⋅eβ⋅x.From equations (8) and (9), we can see that if the learnable parameter is used as a power exponent, it will have a greater impact on the input during backpropagation. A previous study [43] has shown that the amplitude change of the learnable parameters in the activation function is very small during the training process. The learnable parameters as power exponents are more beneficial to network optimization than as coefficients. The experimental results in Section 4 show that the performance of PFREU-B is better than that of PFREU-A in most cases. Therefore, if a learnable parameter is added to the exponential term, we think it should be a power exponent.


### 5.3. Parameter Computation

We assume a convolutional network layer and the size of the input feature map isC×H×W, the convolution kernel receptive field is Kh′×Kw′, and the size of the output feature map is C×H′×W′. The number of convolutional parameters is CCKh′Kw′ and the FLOP (floating-point operation) is CCKh′Kw′HW. Taking PFREU-A as an example, PFREU-A has three terms. We assume that the receptive field of the depthwise separable convolutional layer in the spatial condition is Kh×Kw, the number of parameters for depthwise separable convolution is CKhKw, and the FLOP of depthwise separable convolution is CKhKwHW. The number of parameters of the linear mapping term is C, and the FLOP of the linear mapping term is CHW. The linear mapping term and the nonlinear mapping term have the same number of parameters. For simplification, we assume K=Kh=Kw and K′=Kh′=Kw′.So the parameter complexity of the convolutional layer isOC2K′2, and after adopting PFREU-A, the parameter complexity becomes OC2K′2+CK2+2C. The FLOP of the convolutional layer is OC2K′2HW, and after adopting PFREU-A, it becomes OC2K′2HW+CK2HW+2CHW. Since C is much larger than K, K′, and 2, the additional complexity of PFREU can be negligible.


## 5.1. Nonlinear Transformation vs. Pixelwise Modeling

Activation function is the source of the nonlinear transformation ability of the neural network. All activation functions have different degrees of nonlinear transformation capability. Recently, FReLU has introduced the ability of pixelwise modeling in the activation function. Two questions emerged naturally:(1)
Which ability is more important to the activation function?(2)
How to balance these two abilities in the activation function?Experiments conducted on CIFAR and Tiny ImageNet provide some observations. Different models behave differently. As the previous analysis, ResNet has a more powerful spatial information acquisition capability. Therefore, compared with NIN, the activation function with spatial conditions has less impact on ResNet. It also depends on the specific task. CIFAR-10 and CIFAR-100 have different levels of difficulty. Therefore, on the CIFAR-100 dataset, the model needs an activation function with a stronger nonlinear transformation capability. Therefore, the network requires both nonlinear transformation capability and pixelwise modeling capability, which is more important depending on the specific model and task.Conventional activation functions do not have the pixelwise modeling capability. Compared with the traditional activation functions, FReLU’s nonlinear transformation ability is relatively weaker. PFREU adds a nonlinear mapping term to enhance the nonlinear transformation ability. The experimental results inSection 4 prove the effectiveness of PFREU. Therefore, our CAF-3 method can construct an activation function that balances nonlinear transformation and pixelwise modeling capabilities.


## 5.2. The Design of PFREU

The difference among the PFREU variants is the nonlinear mapping term. In particular, the number of parameters of the nonlinear term in PFREU-A and PFREU-B is also the same. We constructed two simple exponential functions to explore how different parameter positions affect the change of the exponential function [40–42].(7)f1x=α⋅ex,f2x=eβ⋅x.The difference betweenf1 and f2 is whether the learnable parameter is a coefficient or a power exponent.It can be seen from Figure3 that, under the same parameter amplitude change, the change of function f2 is greater than that of f1. Another major difference between f1 and f2 is that f2 always passes through the origin. As shown in Figure 1, most conventional activation functions pass through the origin, so we think this feature helps improve performance.Figure 3
(a) The shape off1 with different α values. (b) The shape of f2 with different β values.
(a)(b)We calculate the gradient off1 and f2 with respect to input x:(8)∂f1x∂x=α⋅ex,(9)∂f2x∂x=β⋅eβ⋅x.From equations (8) and (9), we can see that if the learnable parameter is used as a power exponent, it will have a greater impact on the input during backpropagation. A previous study [43] has shown that the amplitude change of the learnable parameters in the activation function is very small during the training process. The learnable parameters as power exponents are more beneficial to network optimization than as coefficients. The experimental results in Section 4 show that the performance of PFREU-B is better than that of PFREU-A in most cases. Therefore, if a learnable parameter is added to the exponential term, we think it should be a power exponent.


## 5.3. Parameter Computation

We assume a convolutional network layer and the size of the input feature map isC×H×W, the convolution kernel receptive field is Kh′×Kw′, and the size of the output feature map is C×H′×W′. The number of convolutional parameters is CCKh′Kw′ and the FLOP (floating-point operation) is CCKh′Kw′HW. Taking PFREU-A as an example, PFREU-A has three terms. We assume that the receptive field of the depthwise separable convolutional layer in the spatial condition is Kh×Kw, the number of parameters for depthwise separable convolution is CKhKw, and the FLOP of depthwise separable convolution is CKhKwHW. The number of parameters of the linear mapping term is C, and the FLOP of the linear mapping term is CHW. The linear mapping term and the nonlinear mapping term have the same number of parameters. For simplification, we assume K=Kh=Kw and K′=Kh′=Kw′.So the parameter complexity of the convolutional layer isOC2K′2, and after adopting PFREU-A, the parameter complexity becomes OC2K′2+CK2+2C. The FLOP of the convolutional layer is OC2K′2HW, and after adopting PFREU-A, it becomes OC2K′2HW+CK2HW+2CHW. Since C is much larger than K, K′, and 2, the additional complexity of PFREU can be negligible.


## 6. Conclusion and Future Works

In this paper, we introduce an activation function design template: CAF. CAF summarizes all current activation functions that usemax⋅ and provide a direction for future design of new competitive activation functions. In order to verify the effectiveness of CAF, we present an instance that conforms to CAF: PFREU. The performance of PFREU is better than that of other activation functions. Experimental results show that, based on the CAF-3 method, an activation function can be constructed that balances the nonlinear transformation capability and the pixelwise modeling capability. In the future, we will use the Neural Architecture Search technique to explore more activation functions that conform to the CAF template and design new modules to enhance the pixelwise modeling capability of CAF.



---

*Source: 1933490-2021-05-13.xml*