# Fractional Rectified Linear Unit Activation Function and Its Variants

Authors: Megha S. Job; Priyanka H. Bhateja; Muskan Gupta; Kishore Bingi; B. Rajanarayan Prusty
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

This paper focuses on deriving and validating the fractional-order form of rectified linear unit activation function and its linear and nonlinear variants. The linear variants include the leaky and parametric, whereas the nonlinear variants include the exponential, sigmoid-weighted, and Gaussian error functions. Besides, a standard formula has been created and used while developing the fractional form of linear variants. Moreover, different expansion series such as Maclaurin and Taylor have been used while designing the fractional version of nonlinear variants. A simulation study has been conducted to validate the performance of all the developed fractional activation functions utilizing a single and multilayer neural network model and to compare them with their conventional counterparts. In this simulation study, a neural network model has been created to predict the system-generated power of a Texas wind turbine. The performance has been evaluated by varying the activation function in the hidden and output layers with the developed functions for single and multilayer networks.


---

## Body

## 1. Introduction

An activation function in the neural network determines whether the neuron’s inputs to the network are relevant or not using simple mathematical operations. Thus, it decides whether a neuron should be activated or deactivated [1]. Over the years, many activation functions have come to light. Among them, the prominently used functions are sigmoid, Hermite, mish, binary step, rectified linear unit (ReLU), softplus, adaptive spline, and hyperbolic tangent (tanh) [2–6]. The activation functions widely considered in the hidden layer of neural networks are ReLU, sigmoid, linear, and tanh [7–10]. And depending on the type of prediction the model requires, the output layer of a neural network will often utilize a different activation function than those used for hidden layers [11]. Moreover, the best solutions are the linear functions for regression-type output layers and softmax for multi-class classification [7, 9].The sigmoid function is smooth and continuously differentiable. However, no symmetry can be seen around zero; therefore, all of the neurons’ output will show identical signs [9]. The tanh function is steeper than the sigmoid and symmetric around the origin [9]. Compared to sigmoid and tanh functions, the ReLU function is computationally efficient. Nevertheless, the gradient of the ReLU function is 0 at the negative axis, which deactivates the neurons in that region [7, 9, 12]. Thus, a leaky ReLU (LReLU) has been proposed to overcome the issue in ReLU by making minor modifications for the negative values of input to get a nonzero value on the negative plane. Hence, there will be no dead neurons in that region using a leaky ReLU [4, 7, 12]. The parametric ReLU (PReLU) is another choice if leaky ReLU fails to solve the issue. The nonlinear variants of ReLU provide a change in the slope of the negative plane of the function by using some nonlinear functions [4, 7, 12].Fractional calculus supports derivatives and integrals to noninteger orders or fractional orders [13, 14]. Over the years, it has been proven that the modeling with fractional differential equations usually results in a more exact and accurate performance than using conventional differentiation [15]. It is worth noting that precise modeling is necessary for systems that need precise damping. Recently, various researchers have incorporated the application of fractional calculus in neural networks. For instance, the issues of robust stability and synchronization for the fractional-order delayed neural networks are presented in [16]. In a similar vein, the researchers introduced an approach in [17], which allows the neural network to search for and improve its activation functions throughout the training phase by defining the fractional-order derivative of given primitive activation functions. Further, the authors of [18] have infused fractional activation functions into feedforward neural networks, providing more tunable hyper-parameters. Besides, the researchers in [19] have considered Caputo’s and Grünwald-Letnikov’s fractional derivatives-based method for feedforward neural network for optimizing fractional-order delay optimum control problems. Another reported work in [20] has used Grünwald-Letnikov’s derivative function as a learning algorithm and fractional calculus to reduce convergence error and speed. The single-point search algorithm reported in [21] has made an efficient global learning machine to determine the optimal search path.Having been motivated by the above literature, this paper derives fractional-order derivation of ReLU and its variants. Firstly, the fractional derivative of ReLU has been calculated, followed by its variants. There are two types of variants that have been derived: linear variants, which consist of LReLU and PReLU. The nonlinear variants comprise exponential linear unit (ELU), sigmoid-weighted linear unit (SiLU), and Gaussian error linear unit (GELU). The original formula has been employed while calculating the fractional-order derivative of linear variants. However, different series expansions have been used first to calculate the fractional-order derivative of nonlinear variants.The remaining contents of the manuscript are organized as follows: Section2 defines the fractional derivative for order α. Section 3 presents the fractional form of ReLU and its variants. The simulation study on performance analysis of developed fractional ReLU functions is given in Section 4. Finally, Section 5 summarizes essential findings.


## 2. Fractional Derivative

This section defines the fractional derivative for the following example functionfx=xk. For this function, the natural n-derivatives are computed as [13, 15, 17, 18],(1)Dfx=ddxfx=kxk−1,(2)D2fx=d2dx2fx=kk−1xk−2,(3)Dnfx=dndxnfx=kk−1k−2…k−n+1xk−n.Thenth derivative in (3) can be rewritten using the factorial function as(4)Dnfx=dndxnfx=k!k−n!xk−n,where k!=k×k−1×⋯×1.It is to be noted that (4) is valid for non-negative integers only. Therefore, the fractional derivative of fx for an order α∈0,1 can be obtained by replacing the factorial with gamma function Γ as follows [13]:(5)Dαfx=dαdxαfx=Γk+1Γk+1−αxk−α,where Γ function is computed as Γk+1=k!.It is worth noting that the fractional derivativeDαfx derived in (5) is valid for k,x≥0.


## 3. Development of Fractional Rectified Linear Unit Activation Function

The ReLU function has become one of the default activation functions for many neural networks. One example of such a type of network is a convolutional neural network. This is because the model with ReLU trains quicker and generally delivers higher performance. This performance of ReLU and its variants can be further improved using fractional calculus. Therefore, this section develops the fractional form of ReLU and its variants.


### 3.1. Fractional ReLU (FReLU)

ReLU is a piecewise-defined linear function in which, if the input is positive, the output value is the same as the input value. Otherwise, it produces the output of zero. As a result, the ReLU mathematical formula is as follows [15, 17, 18]:(6)R1z=max0,z,where z is the input. Thus, in the above function max0,z can also be rewritten as follows:(7)R1z=zforz>0,0forz≤0.The fractional ReLU function for an orderα∈0,1 is defined as(8)R1z=Dαzforz>0,0forz≤0.Using fractional derivativeDαfz derived in (5), the value of Dαz is computed as follows:(9)Dαz=Γ2Γ2−αz1−α,where Γ2=1. Thus, substituting the value of Γ2 the above equation can be written as(10)Dαz=1Γ2−αz1−α.Therefore, the FReLU function for an orderα∈0,1 is obtained as(11)R1z=1Γ2−αz1−αforz>0,0forz≤0.The response of FReLU for various values ofα∈0,1 in comparison with response of conventional ReLU is shown in Figure 1. The figure depicts that the FReLU provides more flexibility than the ReLU. It can also be seen that for smaller values of α, the function behaves like a ReLU, whereas for larger values of α, the function behaves like a step.Figure 1
Response of fractional forms of linear variants for various values ofα∈0,1. (a) Fractional ReLU, (b) fractional LReLU, and (c) fractional PReLU.
(a)(b)(c)


### 3.2. Linear Variants

#### 3.2.1. Fractional Leaky ReLU

The LReLU is the ReLU activation function’s linear variant. Unlike in ReLU, LReLU allows a slight positive gradient when the input is negative, as shown in the following mathematical form [22, 23]:(12)R2z=max0.1z,z.The function in (12) can also be rewritten as follows:(13)R2z=zforz>0,0.1zforz≤0.Therefore, the fractional LReLU (FLReLU) function fromR2z in (13) for an order α∈0,1 is defined as(14)R2z=Dαzforz>0,Dα0.1zforz≤0.In (14), the value of fractional derivative Dαz is computed using (5) as(15)R2z=1Γ2−αz1−αforz>0,0.1Γ2−αz1−αforz≤0.The response of FLReLU for various values ofα∈0,1 in comparison with response of conventional LReLU is shown in Figure 1(b). As shown in the figure, in the positive plane, the response of FLReLU is equivalent to the response of FReLU. However, in the negative plane, the response of conventional LReLU starts at 0.1, whereas the starting point of FLReLU is in −0.1,0.1. Thus, it provides a more flexible behavior compared to conventional LReLU.


#### 3.2.2. Fractional Parametric ReLU

The PReLU is another linear variant of the activation function ReLU. The leakage coefficient is turned into a parameter learned with the other neural network parameters in PReLU. This can be mathematically represented for an inputz as [22, 23](16)R3z=zforz>0,azforz≤0.where a≥0 is the hyper-parameter.Using (16), the fractional PReLU (FPReLU) function for an order α∈0,1 is defined as(17)R3z=Dαzforz>0,Dαazforz≤0.Substituting the fractional derivativeDαz from (5) in (17) yields(18)R3z=1Γ2−αz1−αforz>0,aΓ2−αz1−αforz≤0.The response of FPReLU for various values ofα in comparison with the response of conventional PReLU is shown in Figure 1(c). The figure shows that the positive plane FPReLU shows an equivalent response as in the case of FReLU and FLReLU. However, in the negative plane, the reaction of PReLU starts at a, whereas the starting point of FPReLU is in −a,a depending on α. In all these cases, the hyper-parameter a has been set to 1.5.


### 3.3. Nonlinear Variants

#### 3.3.1. Fractional Exponential Linear Unit

The ELU function is a nonlinear variant of ReLU. ELU becomes smooth slowly until its output equals−a, whereas ReLU is smooth. ELU strives to get the mean activations close to zero, accelerating learning. This can be mathematically represented as [22, 23](19)R4z=zforz>0,aez−1forz≤0.where a≥0 is a hyper-parameter to be tuned.From (19), the fractional ELU (FELU) for an order α∈0,1 is computed as follows:(20)R4z=Dαzforz>0,Dαaez−1forz≤0.In (20), for z>0, the value of Dαz is computed using (5) as(21)Dαz=1Γ2−αz1−α.However, forz≤0, the value of Dαaez−1 is calculated as(22)Dαaez−1=aDαez−Dαa,(23)Dαaez−1=aDα∑i=0∞zii!−aDαz0,(24)Dαaez−1=a∑i=0∞1i!×Γi+1Γi+1−αzi−α−a1Γ1−αz−α.By substituting (21) and (24) in (20), the FELU is obtained as follows:(25)R4z=1Γ2−αz1−αforz>0,a∑i=0∞1i!×Γi+1Γi+1−αzi−α,−a1Γ1−αz−αforz≤0.The response of FELU for various values ofα∈0,1 in comparison with response of conventional ELU is shown in Figure 2. For the analysis, the hyper-parameter a has been set to 1.5. As shown in the figure, for z>0, the response of FELU is equivalent to the response of FReLU. However, for z≤0, the variation in exponential function can be seen for various values of α. Thus, it can be concluded that the FELU provides more flexible behavior compared to conventional ELU.Figure 2
Response of fractional forms of nonlinear variants for various values ofα∈0,1. (a) Fractional ELU, (b) fractional SiLU, and (c) fractional GELU.
(a)(b)(c)


#### 3.3.2. Fractional Sigmoid-Weighted Linear Unit

The SiLU function is another nonlinear variant of the activation function ReLU. It is calculated by multiplying its input by the sigmoid function and thus resembles a continuous and “undershooting” variant of ReLU. This can be mathematically represented as [22, 23](26)R5z=zforz>0,zσzforz≤0,where σz is the sigmoid function defined as(27)σz=11+e−z.From (26), the fractional SiLU (FSiLU) for an order α∈0,1 is computed as(28)R5z=Dαzforz>0,Dαzσzforz≤0,where the value of Dαz for z>0 is computed using (5) as follows:(29)Dαz=1Γ2−αz1−α.However, the procedure for computing the value ofDαzσz for z≤0 is explained underneath. The expansion of σz using Maclaurin series is given as(30)σz=∑i=0∞−1i+2i+1−1Bi+1i+1!zi,where B is Bernoulli’s number. Further, the expansion of zσz is given as(31)zσz=∑i=0∞−1i+2i+1−1Bi+1i+1!zi+1.Therefore, in (31), the value of Dαzσz using (5) is computed as(32)Dαzσz=∑i=0∞−1i+2i+1−1Bi+1Γi+2Γi+2−αi+1!zi+1−α.By substituting (29) and (32) in (28), the FSiLU is obtained as follows:(33)R5z=1Γ2−αz1−αforz>0,∑i=0∞−1i+2i+1−1Bi+1Γi+2Γi+2−αi+1!zi+1−αforz≤0.The response of FSiLU for various values ofα∈0,1 in comparison with response of conventional SiLU is shown in Figure 2(b). The figure depicts that for z>0, FSiLU provides an equivalent response to FReLU. Moreover, for z≤0, a precise variation in continuous and undershooting can be seen as α varies from 1 to 0. Thus, it proves the flexibility of FSiLU compared to conventional SiLU.


#### 3.3.3. Fractional Gaussian Error Linear Unit

GELU is another nonlinear variant of ReLU, and it is a smooth approximation of ReLU. In contrast to ReLU, the negative plane of GELU has a nonmonotonic bump and its nonlinearity weight inputs by value. GELU is the default activation for several natural language processing models. The mathematical model of GELU can be represented as [22, 23](34)R6z=zforz>0,zϕzforz≤0,where ϕz is the Gaussian function which is defined as(35)ϕz=e−z−b2/2c2,where e is Euler’s number, and b and c are the position of the center of the peak and standard deviation, respectively.Therefore, from (34), the fractional GELU (FGELU) for an order α∈0,1 is computed as(36)R6z=Dαzforz>0,Dαzϕzforz≤0.In (36), for z>0, the value of Dαz is computed using (5) as(37)Dαz=1Γ2−αz1−α.However, forz≤0, the value of Dαzϕz is computed using the following procedure. The expansion of ϕz using Taylor’s series is given as(38)ϕz=0.5−12π∑i=0∞1i!−12iz2i+12i+1.In (38), multiplying z on both sides yields(39)zϕz=0.5z−12π∑i=0∞1i!−12iz2i+12i+1.Thus, the fractional derivativeDαzϕz is computed as(40)Dαzϕz=0.5z1−αΓ2−α−12π∑i=0∞1i!−12iz2i+1−α2i+1Γ2i+3Γ2i+3−α.Thus, the mathematical model of FGELU obtained after substituting (37) and (40) in (36) is given as follows:(41)R6z=1Γ2−αz1−αforz>0,0.5z1−αΓ2−α−12π∑i=0∞1i!−12iz2i+1−α2i+1Γ2i+3Γ2i+3−αforz≤0,Figure2(c) depicts the response of FGELU for various values of α∈0,1 in comparison with the reaction of conventional GELU. The response shows that FGELU provides a similar behavior to FReLU for positive values of z. In contrast, for negative values of z, the variation in nonmonotonic bump and nonlinearity can be seen as α varies from 1 to 0. Thus, similar to other variants, the response of FGELU achieves more flexibility than the conventional GELU.The summary of all the proposed activation functions is provided in Table1.Table 1
Summarization of proposed fractional-order activation functions.
CategoryFunctionConventionalFractionalLinear variantReLUR1z=z,forz>0,0,forz≤0R1z=1/Γ2−αz1−α,forz>0,0,forz≤0LReLUR2z=z,forz>0,0.1z,forz≤0R2z=1/Γ2−αz1−α,forz>0,0.1/Γ2−αz1−α,forz≤0PReLUR3z=Dαz,forz>0,Dαaz,forz≤0R3z=1/Γ2−αz1−α,forz>0,a/Γ2−αz1−α,forz≤0ELUR4z=z,forz>0,aez−1,forz≤0R4z=1/Γ2−αz1−α,forz>0,a∑i=0∞1/i!×Γi+1/Γi+1−αzi−α−a1/Γ1−αz−α,forz≤0Nonlinear variantSiLUR5z=z,forz>0,zσz,forz≤0R5z=1/Γ2−αz1−α,forz>0,∑i=0∞−1i+2i+1−1Bi+1Γi+2/Γi+2−αi+2!zi+1−α,forz≤0GELUR6z=z,forz>0,zϕz,forz≤0R6z=1/Γ2−αz1−αforz>00.5z1−α/Γ2−α−1/2π∑i=0∞1/i!−1/2iz2i+1−α/2i+1Γ2i+3/Γ2i+3−αforz≤0


## 3.1. Fractional ReLU (FReLU)

ReLU is a piecewise-defined linear function in which, if the input is positive, the output value is the same as the input value. Otherwise, it produces the output of zero. As a result, the ReLU mathematical formula is as follows [15, 17, 18]:(6)R1z=max0,z,where z is the input. Thus, in the above function max0,z can also be rewritten as follows:(7)R1z=zforz>0,0forz≤0.The fractional ReLU function for an orderα∈0,1 is defined as(8)R1z=Dαzforz>0,0forz≤0.Using fractional derivativeDαfz derived in (5), the value of Dαz is computed as follows:(9)Dαz=Γ2Γ2−αz1−α,where Γ2=1. Thus, substituting the value of Γ2 the above equation can be written as(10)Dαz=1Γ2−αz1−α.Therefore, the FReLU function for an orderα∈0,1 is obtained as(11)R1z=1Γ2−αz1−αforz>0,0forz≤0.The response of FReLU for various values ofα∈0,1 in comparison with response of conventional ReLU is shown in Figure 1. The figure depicts that the FReLU provides more flexibility than the ReLU. It can also be seen that for smaller values of α, the function behaves like a ReLU, whereas for larger values of α, the function behaves like a step.Figure 1
Response of fractional forms of linear variants for various values ofα∈0,1. (a) Fractional ReLU, (b) fractional LReLU, and (c) fractional PReLU.
(a)(b)(c)


## 3.2. Linear Variants

### 3.2.1. Fractional Leaky ReLU

The LReLU is the ReLU activation function’s linear variant. Unlike in ReLU, LReLU allows a slight positive gradient when the input is negative, as shown in the following mathematical form [22, 23]:(12)R2z=max0.1z,z.The function in (12) can also be rewritten as follows:(13)R2z=zforz>0,0.1zforz≤0.Therefore, the fractional LReLU (FLReLU) function fromR2z in (13) for an order α∈0,1 is defined as(14)R2z=Dαzforz>0,Dα0.1zforz≤0.In (14), the value of fractional derivative Dαz is computed using (5) as(15)R2z=1Γ2−αz1−αforz>0,0.1Γ2−αz1−αforz≤0.The response of FLReLU for various values ofα∈0,1 in comparison with response of conventional LReLU is shown in Figure 1(b). As shown in the figure, in the positive plane, the response of FLReLU is equivalent to the response of FReLU. However, in the negative plane, the response of conventional LReLU starts at 0.1, whereas the starting point of FLReLU is in −0.1,0.1. Thus, it provides a more flexible behavior compared to conventional LReLU.


### 3.2.2. Fractional Parametric ReLU

The PReLU is another linear variant of the activation function ReLU. The leakage coefficient is turned into a parameter learned with the other neural network parameters in PReLU. This can be mathematically represented for an inputz as [22, 23](16)R3z=zforz>0,azforz≤0.where a≥0 is the hyper-parameter.Using (16), the fractional PReLU (FPReLU) function for an order α∈0,1 is defined as(17)R3z=Dαzforz>0,Dαazforz≤0.Substituting the fractional derivativeDαz from (5) in (17) yields(18)R3z=1Γ2−αz1−αforz>0,aΓ2−αz1−αforz≤0.The response of FPReLU for various values ofα in comparison with the response of conventional PReLU is shown in Figure 1(c). The figure shows that the positive plane FPReLU shows an equivalent response as in the case of FReLU and FLReLU. However, in the negative plane, the reaction of PReLU starts at a, whereas the starting point of FPReLU is in −a,a depending on α. In all these cases, the hyper-parameter a has been set to 1.5.


## 3.2.1. Fractional Leaky ReLU

The LReLU is the ReLU activation function’s linear variant. Unlike in ReLU, LReLU allows a slight positive gradient when the input is negative, as shown in the following mathematical form [22, 23]:(12)R2z=max0.1z,z.The function in (12) can also be rewritten as follows:(13)R2z=zforz>0,0.1zforz≤0.Therefore, the fractional LReLU (FLReLU) function fromR2z in (13) for an order α∈0,1 is defined as(14)R2z=Dαzforz>0,Dα0.1zforz≤0.In (14), the value of fractional derivative Dαz is computed using (5) as(15)R2z=1Γ2−αz1−αforz>0,0.1Γ2−αz1−αforz≤0.The response of FLReLU for various values ofα∈0,1 in comparison with response of conventional LReLU is shown in Figure 1(b). As shown in the figure, in the positive plane, the response of FLReLU is equivalent to the response of FReLU. However, in the negative plane, the response of conventional LReLU starts at 0.1, whereas the starting point of FLReLU is in −0.1,0.1. Thus, it provides a more flexible behavior compared to conventional LReLU.


## 3.2.2. Fractional Parametric ReLU

The PReLU is another linear variant of the activation function ReLU. The leakage coefficient is turned into a parameter learned with the other neural network parameters in PReLU. This can be mathematically represented for an inputz as [22, 23](16)R3z=zforz>0,azforz≤0.where a≥0 is the hyper-parameter.Using (16), the fractional PReLU (FPReLU) function for an order α∈0,1 is defined as(17)R3z=Dαzforz>0,Dαazforz≤0.Substituting the fractional derivativeDαz from (5) in (17) yields(18)R3z=1Γ2−αz1−αforz>0,aΓ2−αz1−αforz≤0.The response of FPReLU for various values ofα in comparison with the response of conventional PReLU is shown in Figure 1(c). The figure shows that the positive plane FPReLU shows an equivalent response as in the case of FReLU and FLReLU. However, in the negative plane, the reaction of PReLU starts at a, whereas the starting point of FPReLU is in −a,a depending on α. In all these cases, the hyper-parameter a has been set to 1.5.


## 3.3. Nonlinear Variants

### 3.3.1. Fractional Exponential Linear Unit

The ELU function is a nonlinear variant of ReLU. ELU becomes smooth slowly until its output equals−a, whereas ReLU is smooth. ELU strives to get the mean activations close to zero, accelerating learning. This can be mathematically represented as [22, 23](19)R4z=zforz>0,aez−1forz≤0.where a≥0 is a hyper-parameter to be tuned.From (19), the fractional ELU (FELU) for an order α∈0,1 is computed as follows:(20)R4z=Dαzforz>0,Dαaez−1forz≤0.In (20), for z>0, the value of Dαz is computed using (5) as(21)Dαz=1Γ2−αz1−α.However, forz≤0, the value of Dαaez−1 is calculated as(22)Dαaez−1=aDαez−Dαa,(23)Dαaez−1=aDα∑i=0∞zii!−aDαz0,(24)Dαaez−1=a∑i=0∞1i!×Γi+1Γi+1−αzi−α−a1Γ1−αz−α.By substituting (21) and (24) in (20), the FELU is obtained as follows:(25)R4z=1Γ2−αz1−αforz>0,a∑i=0∞1i!×Γi+1Γi+1−αzi−α,−a1Γ1−αz−αforz≤0.The response of FELU for various values ofα∈0,1 in comparison with response of conventional ELU is shown in Figure 2. For the analysis, the hyper-parameter a has been set to 1.5. As shown in the figure, for z>0, the response of FELU is equivalent to the response of FReLU. However, for z≤0, the variation in exponential function can be seen for various values of α. Thus, it can be concluded that the FELU provides more flexible behavior compared to conventional ELU.Figure 2
Response of fractional forms of nonlinear variants for various values ofα∈0,1. (a) Fractional ELU, (b) fractional SiLU, and (c) fractional GELU.
(a)(b)(c)


### 3.3.2. Fractional Sigmoid-Weighted Linear Unit

The SiLU function is another nonlinear variant of the activation function ReLU. It is calculated by multiplying its input by the sigmoid function and thus resembles a continuous and “undershooting” variant of ReLU. This can be mathematically represented as [22, 23](26)R5z=zforz>0,zσzforz≤0,where σz is the sigmoid function defined as(27)σz=11+e−z.From (26), the fractional SiLU (FSiLU) for an order α∈0,1 is computed as(28)R5z=Dαzforz>0,Dαzσzforz≤0,where the value of Dαz for z>0 is computed using (5) as follows:(29)Dαz=1Γ2−αz1−α.However, the procedure for computing the value ofDαzσz for z≤0 is explained underneath. The expansion of σz using Maclaurin series is given as(30)σz=∑i=0∞−1i+2i+1−1Bi+1i+1!zi,where B is Bernoulli’s number. Further, the expansion of zσz is given as(31)zσz=∑i=0∞−1i+2i+1−1Bi+1i+1!zi+1.Therefore, in (31), the value of Dαzσz using (5) is computed as(32)Dαzσz=∑i=0∞−1i+2i+1−1Bi+1Γi+2Γi+2−αi+1!zi+1−α.By substituting (29) and (32) in (28), the FSiLU is obtained as follows:(33)R5z=1Γ2−αz1−αforz>0,∑i=0∞−1i+2i+1−1Bi+1Γi+2Γi+2−αi+1!zi+1−αforz≤0.The response of FSiLU for various values ofα∈0,1 in comparison with response of conventional SiLU is shown in Figure 2(b). The figure depicts that for z>0, FSiLU provides an equivalent response to FReLU. Moreover, for z≤0, a precise variation in continuous and undershooting can be seen as α varies from 1 to 0. Thus, it proves the flexibility of FSiLU compared to conventional SiLU.


### 3.3.3. Fractional Gaussian Error Linear Unit

GELU is another nonlinear variant of ReLU, and it is a smooth approximation of ReLU. In contrast to ReLU, the negative plane of GELU has a nonmonotonic bump and its nonlinearity weight inputs by value. GELU is the default activation for several natural language processing models. The mathematical model of GELU can be represented as [22, 23](34)R6z=zforz>0,zϕzforz≤0,where ϕz is the Gaussian function which is defined as(35)ϕz=e−z−b2/2c2,where e is Euler’s number, and b and c are the position of the center of the peak and standard deviation, respectively.Therefore, from (34), the fractional GELU (FGELU) for an order α∈0,1 is computed as(36)R6z=Dαzforz>0,Dαzϕzforz≤0.In (36), for z>0, the value of Dαz is computed using (5) as(37)Dαz=1Γ2−αz1−α.However, forz≤0, the value of Dαzϕz is computed using the following procedure. The expansion of ϕz using Taylor’s series is given as(38)ϕz=0.5−12π∑i=0∞1i!−12iz2i+12i+1.In (38), multiplying z on both sides yields(39)zϕz=0.5z−12π∑i=0∞1i!−12iz2i+12i+1.Thus, the fractional derivativeDαzϕz is computed as(40)Dαzϕz=0.5z1−αΓ2−α−12π∑i=0∞1i!−12iz2i+1−α2i+1Γ2i+3Γ2i+3−α.Thus, the mathematical model of FGELU obtained after substituting (37) and (40) in (36) is given as follows:(41)R6z=1Γ2−αz1−αforz>0,0.5z1−αΓ2−α−12π∑i=0∞1i!−12iz2i+1−α2i+1Γ2i+3Γ2i+3−αforz≤0,Figure2(c) depicts the response of FGELU for various values of α∈0,1 in comparison with the reaction of conventional GELU. The response shows that FGELU provides a similar behavior to FReLU for positive values of z. In contrast, for negative values of z, the variation in nonmonotonic bump and nonlinearity can be seen as α varies from 1 to 0. Thus, similar to other variants, the response of FGELU achieves more flexibility than the conventional GELU.The summary of all the proposed activation functions is provided in Table1.Table 1
Summarization of proposed fractional-order activation functions.
CategoryFunctionConventionalFractionalLinear variantReLUR1z=z,forz>0,0,forz≤0R1z=1/Γ2−αz1−α,forz>0,0,forz≤0LReLUR2z=z,forz>0,0.1z,forz≤0R2z=1/Γ2−αz1−α,forz>0,0.1/Γ2−αz1−α,forz≤0PReLUR3z=Dαz,forz>0,Dαaz,forz≤0R3z=1/Γ2−αz1−α,forz>0,a/Γ2−αz1−α,forz≤0ELUR4z=z,forz>0,aez−1,forz≤0R4z=1/Γ2−αz1−α,forz>0,a∑i=0∞1/i!×Γi+1/Γi+1−αzi−α−a1/Γ1−αz−α,forz≤0Nonlinear variantSiLUR5z=z,forz>0,zσz,forz≤0R5z=1/Γ2−αz1−α,forz>0,∑i=0∞−1i+2i+1−1Bi+1Γi+2/Γi+2−αi+2!zi+1−α,forz≤0GELUR6z=z,forz>0,zϕz,forz≤0R6z=1/Γ2−αz1−αforz>00.5z1−α/Γ2−α−1/2π∑i=0∞1/i!−1/2iz2i+1−α/2i+1Γ2i+3/Γ2i+3−αforz≤0


## 3.3.1. Fractional Exponential Linear Unit

The ELU function is a nonlinear variant of ReLU. ELU becomes smooth slowly until its output equals−a, whereas ReLU is smooth. ELU strives to get the mean activations close to zero, accelerating learning. This can be mathematically represented as [22, 23](19)R4z=zforz>0,aez−1forz≤0.where a≥0 is a hyper-parameter to be tuned.From (19), the fractional ELU (FELU) for an order α∈0,1 is computed as follows:(20)R4z=Dαzforz>0,Dαaez−1forz≤0.In (20), for z>0, the value of Dαz is computed using (5) as(21)Dαz=1Γ2−αz1−α.However, forz≤0, the value of Dαaez−1 is calculated as(22)Dαaez−1=aDαez−Dαa,(23)Dαaez−1=aDα∑i=0∞zii!−aDαz0,(24)Dαaez−1=a∑i=0∞1i!×Γi+1Γi+1−αzi−α−a1Γ1−αz−α.By substituting (21) and (24) in (20), the FELU is obtained as follows:(25)R4z=1Γ2−αz1−αforz>0,a∑i=0∞1i!×Γi+1Γi+1−αzi−α,−a1Γ1−αz−αforz≤0.The response of FELU for various values ofα∈0,1 in comparison with response of conventional ELU is shown in Figure 2. For the analysis, the hyper-parameter a has been set to 1.5. As shown in the figure, for z>0, the response of FELU is equivalent to the response of FReLU. However, for z≤0, the variation in exponential function can be seen for various values of α. Thus, it can be concluded that the FELU provides more flexible behavior compared to conventional ELU.Figure 2
Response of fractional forms of nonlinear variants for various values ofα∈0,1. (a) Fractional ELU, (b) fractional SiLU, and (c) fractional GELU.
(a)(b)(c)


## 3.3.2. Fractional Sigmoid-Weighted Linear Unit

The SiLU function is another nonlinear variant of the activation function ReLU. It is calculated by multiplying its input by the sigmoid function and thus resembles a continuous and “undershooting” variant of ReLU. This can be mathematically represented as [22, 23](26)R5z=zforz>0,zσzforz≤0,where σz is the sigmoid function defined as(27)σz=11+e−z.From (26), the fractional SiLU (FSiLU) for an order α∈0,1 is computed as(28)R5z=Dαzforz>0,Dαzσzforz≤0,where the value of Dαz for z>0 is computed using (5) as follows:(29)Dαz=1Γ2−αz1−α.However, the procedure for computing the value ofDαzσz for z≤0 is explained underneath. The expansion of σz using Maclaurin series is given as(30)σz=∑i=0∞−1i+2i+1−1Bi+1i+1!zi,where B is Bernoulli’s number. Further, the expansion of zσz is given as(31)zσz=∑i=0∞−1i+2i+1−1Bi+1i+1!zi+1.Therefore, in (31), the value of Dαzσz using (5) is computed as(32)Dαzσz=∑i=0∞−1i+2i+1−1Bi+1Γi+2Γi+2−αi+1!zi+1−α.By substituting (29) and (32) in (28), the FSiLU is obtained as follows:(33)R5z=1Γ2−αz1−αforz>0,∑i=0∞−1i+2i+1−1Bi+1Γi+2Γi+2−αi+1!zi+1−αforz≤0.The response of FSiLU for various values ofα∈0,1 in comparison with response of conventional SiLU is shown in Figure 2(b). The figure depicts that for z>0, FSiLU provides an equivalent response to FReLU. Moreover, for z≤0, a precise variation in continuous and undershooting can be seen as α varies from 1 to 0. Thus, it proves the flexibility of FSiLU compared to conventional SiLU.


## 3.3.3. Fractional Gaussian Error Linear Unit

GELU is another nonlinear variant of ReLU, and it is a smooth approximation of ReLU. In contrast to ReLU, the negative plane of GELU has a nonmonotonic bump and its nonlinearity weight inputs by value. GELU is the default activation for several natural language processing models. The mathematical model of GELU can be represented as [22, 23](34)R6z=zforz>0,zϕzforz≤0,where ϕz is the Gaussian function which is defined as(35)ϕz=e−z−b2/2c2,where e is Euler’s number, and b and c are the position of the center of the peak and standard deviation, respectively.Therefore, from (34), the fractional GELU (FGELU) for an order α∈0,1 is computed as(36)R6z=Dαzforz>0,Dαzϕzforz≤0.In (36), for z>0, the value of Dαz is computed using (5) as(37)Dαz=1Γ2−αz1−α.However, forz≤0, the value of Dαzϕz is computed using the following procedure. The expansion of ϕz using Taylor’s series is given as(38)ϕz=0.5−12π∑i=0∞1i!−12iz2i+12i+1.In (38), multiplying z on both sides yields(39)zϕz=0.5z−12π∑i=0∞1i!−12iz2i+12i+1.Thus, the fractional derivativeDαzϕz is computed as(40)Dαzϕz=0.5z1−αΓ2−α−12π∑i=0∞1i!−12iz2i+1−α2i+1Γ2i+3Γ2i+3−α.Thus, the mathematical model of FGELU obtained after substituting (37) and (40) in (36) is given as follows:(41)R6z=1Γ2−αz1−αforz>0,0.5z1−αΓ2−α−12π∑i=0∞1i!−12iz2i+1−α2i+1Γ2i+3Γ2i+3−αforz≤0,Figure2(c) depicts the response of FGELU for various values of α∈0,1 in comparison with the reaction of conventional GELU. The response shows that FGELU provides a similar behavior to FReLU for positive values of z. In contrast, for negative values of z, the variation in nonmonotonic bump and nonlinearity can be seen as α varies from 1 to 0. Thus, similar to other variants, the response of FGELU achieves more flexibility than the conventional GELU.The summary of all the proposed activation functions is provided in Table1.Table 1
Summarization of proposed fractional-order activation functions.
CategoryFunctionConventionalFractionalLinear variantReLUR1z=z,forz>0,0,forz≤0R1z=1/Γ2−αz1−α,forz>0,0,forz≤0LReLUR2z=z,forz>0,0.1z,forz≤0R2z=1/Γ2−αz1−α,forz>0,0.1/Γ2−αz1−α,forz≤0PReLUR3z=Dαz,forz>0,Dαaz,forz≤0R3z=1/Γ2−αz1−α,forz>0,a/Γ2−αz1−α,forz≤0ELUR4z=z,forz>0,aez−1,forz≤0R4z=1/Γ2−αz1−α,forz>0,a∑i=0∞1/i!×Γi+1/Γi+1−αzi−α−a1/Γ1−αz−α,forz≤0Nonlinear variantSiLUR5z=z,forz>0,zσz,forz≤0R5z=1/Γ2−αz1−α,forz>0,∑i=0∞−1i+2i+1−1Bi+1Γi+2/Γi+2−αi+2!zi+1−α,forz≤0GELUR6z=z,forz>0,zϕz,forz≤0R6z=1/Γ2−αz1−αforz>00.5z1−α/Γ2−α−1/2π∑i=0∞1/i!−1/2iz2i+1−α/2i+1Γ2i+3/Γ2i+3−αforz≤0


## 4. Simulation Study

In this section, the performance of all the developed fractional activation functions is compared with their conventional forms. The comparison has been made in three cases. A single hidden layer feedforward neural network has been used in the first and second cases. In the third case, a multilayer feedforward neural network with two hidden layers is used to evaluate the performance. In the first case, the activation function in the hidden layer has varied between the conventional and proposed activation functions, and the output layer activation function has been kept constant atpurelin. In contrast, in the second case, the activation function in the output layer has varied between the conventional and proposed activation functions, while the hidden layer activation function has been kept constant at tansig. Moreover, in the third case, the activation functions in both the hidden layers have varied between the conventional and proposed activation functions, and the output layer activation function has been kept constant at purelin.For all the case studies, a Texas wind turbine dataset has been chosen consisting of two input parameters: wind speed and wind direction [24]. The dataset consists of 2000 samples in each parameter, as shown in Figure 3. Moreover, the system-generated power is the output parameter. Figure 3 shows that the data are highly nonlinear and chaotic. Further, the dataset is divided into training and testing sets of 80% and 20%, respectively. Three case studies with different architectures of neural networks have been employed for the performance comparison, and each has been discussed in detail underneath. The neural network training algorithm is chosen as the Levenberg-Marquardt algorithm in all cases. The performance measures used in this simulation study are mean square error (MSE) and R2, which are computed as follows:(42)MSE=1n∑j=1nYj−Y^j2,R2=1−∑j=1nYj−Y^j2∑j=1nYj−Y¯2,where “n” is the sample’s number, Yj is the actual value, Y^j is the predicted values, and Y¯ is the average of Y^j.Figure 3
Input and output parameters for training and testing.Case 1.
As mentioned earlier, this case study has used a single hidden layer feedforward neural network. This neural network is used to predict the system-generated power using wind speed, and wind direction is shown in Figure4. The architecture shows that the number of nodes in the input, hidden, and output layers is 2 : 10 : 1. In the figure, bias at the hidden and output layer is denoted as b [10, 25]. The activation function chosen at the output layer is purelin, while the activation function “F” at the hidden layer is varied to evaluate the performance.
Therefore, the performance of the neural network with various activation functions at the hidden layer during training and testing is shown in Table2. The table shows that all the activation functions achieved R2 values of around 0.9 and MSE of approximately 0.02. Thus, the performance indicates that all activation functions performed better on reaching R2 and MSE values close to 1 and 0, respectively. Further, it is to highlight that all the fractional linear variants have performed better than the conventional ones in achieving the highest R2 and lowest MSE values.
Overall, FLReLU and FSiLU have achieved the best performance among all the activation functions. On the other hand, among the three nonlinear variants, FSiLU has performed better than SiLU. However, the conventional ELU and GELU have achieved better performance than FELU and FGELU. Thus, the performance of the neural network with the FLReLU activation function for predicting the system-generated power during training and testing is shown in Figure5. The response in the figure highlights the best tracking ability during both the training and testing phases.Figure 4
Architecture of neural network used for simulation study in Case1.Table 2
Performance comparison of neural network model in Case1 with various activation functions.
CategoryFunctionTrainingTestingR2MSER2MSEConventionalReLU0.96480.02630.96820.0238LReLU0.96110.02980.96210.0289PReLU0.97230.02080.97440.0190ELU0.97600.01830.97720.0179SiLU0.96360.03120.96250.0291GELU0.96030.03010.96200.0285FractionalFReLU0.99010.00770.99290.0072FLReLU0.99420.00440.99330.0046FPReLU0.99330.00510.99310.0051FELU0.92830.05810.96290.0301FSiLU0.99310.00520.99420.0043FGELU0.89020.13600.85640.1861Figure 5
Performance of neural network with FLReLU in hidden layer during training and testing.Case 2.
Similarly, the neural network architecture to predict the system-generated power using wind speed and wind direction as inputs is developed in this case, as shown in Figure6. The architecture shows that the number of nodes in the input, hidden, and output layers is 2 : 10 : 1 [10, 25]. In this case, the activation function chosen at the hidden layer is tansig, while the activation function at the output layer “F” is varied to evaluate the performance.
Therefore, the performance of the neural network model with various activation functions at the hidden layer during training and testing is shown in Table3. The table shows that the fractional-order activation functions have performed better than the conventional ones in terms of R2 values. However, that is not the case with the MSE values. Comparing the functions concerning their R2 value, it is evident that among the three linear variants, LRELU and FLReLU performed better than other linear variants. Thus, the performance of the neural network with the FLReLU activation function in the output layer for predicting the system-generated power during training and testing is shown in Figure 7. The response in the figure highlights the best tracking ability during both the training and testing phases.
Besides, the FELU performs better than ELU among the three nonlinear variants. However, conventional SiLU and GELU performed better than FSiLU and FGELU. Overall, it can be concluded that the performance of the activation functions and their fractional orders has not performed well enough in the output layer. Thus, it has been proven that the activation function in the output layer is always preferably linear.Figure 6
Architecture of neural network used for simulation study in Case2.Table 3
Performance comparison of neural network model in Case2 with various activation functions.
CategoryFunctionTrainingTestingR2MSER2MSEConventionalReLU0.76660.35660.79380.3238LReLU0.90440.13890.90770.1359PReLU0.83271.17480.81450.3338ELU0.74520.76860.64040.7915SiLU0.85343.17580.85193.2667GELU0.81103.45920.80893.6084FractionalFReLU0.77980.35890.81450.3380FLReLU0.92465.02760.91174.6879FPReLU0.87461.19980.88211.0987FELU0.85092.53950.93112.4631FSiLU0.78240.69980.77140.7121FGELU0.72437.54120.74177.2491Figure 7
Performance of neural network with FLReLU in output layer during training and testing.Case 3.
A multilayer feedforward neural network has been used for this case which consists of two hidden layers and one output layer. Thus, the neural network architecture to predict the system-generated power using wind speed and wind direction as inputs is developed in this simulation study, as shown in Figure8. The architecture shows that the number of nodes in the input, hidden layer 1, hidden layer 2, and output layer is 2 : 10 : 10 : 1. The activation function chosen at the output layer is purelin, while the activation function at both the hidden layers “F1” and “F2” is varied to evaluate the performance.
Table4 indicates that keeping ReLU in hidden layer 1, PReLU, and FPReLU performs better than other linear variants. On the other hand, ELU and FELU perform better than other nonlinear variants. Considering the linear variants, the PReLU and FPReLU perform best in the 2nd hidden layer with ReLU in hidden layer 1. With LReLU in hidden layer 1, FLReLU and PReLU have served better than conventional ones. Similarly, with PReLU in a hidden layer 1, FReLU and FLReLU have performed better than conventional ones. Considering the nonlinear variants, all conventional ones performed better than fractional ones with ELU and SiLU in 1st hidden layer. However, all the fractional-order activation functions, namely, FReLU, FPReLU, and FLReLU, have achieved best performance than conventional ones with GELU in the 1st hidden layer. Considering the fractional-order activation functions, the combinations such as FReLU + FPReLU, FLReLU + FLReLU, FLReLU + FELU, FSiLU + FReLU, FSiLU + FLReLU, FSiLU + FPReLU, FGeLU + FReLU, FGeLU + FLReLU, and FGeLU + FPReLU have performed better than the conventional ones. As an example case, the performance of the neural network model with FReLU in 1st hidden layer and FPRELU in 2nd hidden layer during training and testing is shown in Figure 9. The response in the figure depicts the best tracking performance during both the training and testing phases.Figure 8
Architecture of neural network used for simulation study in Case3.Table 4
Performance comparison of neural network model in Case3 with various activation functions.
Hidden Layer 1Hidden Layer 2TrainingTestingR2MSER2MSEReLUReLU0.90580.09910.92030.0971LReLU0.95040.04710.95690.0544PReLU0.98890.01190.98810.0122ELU0.95470.03650.95360.0426SiLU0.94050.06130.93730.0641GELU0.95540.04820.95790.0451FReLU0.90650.13840.88790.1462FLReLU0.89220.08840.87570.0907FPReLU0.98260.01480.97880.0189FELU0.82920.14810.81750.1851FSiLU0.77790.18240.78810.1811FGELU0.67711.78910.66541.6541LReLUReLU0.95690.03310.96010.0319LReLU0.93630.04840.92970.0484PReLU0.91570.08120.90840.0871ELU0.93570.06940.93080.0819SiLU0.93550.06170.93990.0602GELU0.93130.05710.92670.0567FReLU0.94610.04040.94880.0391FLReLU0.94640.04350.94930.0433FPReLU0.95920.03120.95790.0213FELU0.86290.16870.86330.1143FSiLU0.68890.61680.54080.6224FGELU0.51115.36390.51375.2435PReLUReLU0.92610.01060.93960.0109LReLU0.94570.04250.94670.0404PReLU0.93630.06720.92660.0584ELU0.95680.06660.95190.0701SiLU0.92830.09220.93330.0911GELU0.95470.03480.96870.0271FReLU0.95530.03520.95880.0331FLReLU0.96680.02610.96530.0256FPReLU0.93390.09660.93490.0887FELU0.87240.14730.88070.2205FSiLU0.85110.12510.84230.1281FGELU0.41154.32230.41124.1112ELUReLU0.93210.07910.93630.0745LReLU0.93010.53740.93480.0512PReLU0.98340.01440.98290.0143ELU0.87730.15510.86860.1333SiLU0.93830.09320.93560.0921GELU0.91570.15540.91050.1543FReLU0.85680.14970.85290.1433FLReLU0.92880.07690.91050.0823FPReLU0.90960.06570.91660.0633FELU0.84120.17370.85120.1612FSiLU0.46970.64910.56040.6543FGELU0.30724.12450.37864.4532SiLUReLU0.92360.06210.91940.0681LReLU0.95760.03220.95790.0312PReLU0.09790.01510.97450.0212ELU0.91720.11210.91710.1221SiLU0.87710.09430.85430.1177GELU0.93040.06820.92810.0636FReLU0.84110.13210.84410.1285FLReLU0.86080.11110.84410.1153FPReLU0.97890.01660.97550.0167FELU0.87730.49410.87580.4962FSiLU0.80010.36820.81260.3214FGELU0.62671.98710.66911.9887GELUReLU0.87960.14130.87210.1383LReLU0.91720.06290.93880.0431PReLU0.91520.29830.89810.3958ELU0.91520.06310.91620.0594SiLU0.95480.03560.96050.3114GELU0.93240.06670.93170.0637FReLU0.90680.07310.89990.0777FLReLU0.92530.06150.93040.0641FPReLU0.95510.03710.94970.0358FELU0.90680.07940.91440.0706FSiLU0.81030.45280.82430.5552FGELU0.60891.88380.62261.8921FReLUReLU0.97010.02360.96520.0236LReLU0.98870.00840.98840.0092PReLU0.98430.01180.98010.0157ELU0.98370.01310.98450.0113SiLU0.96020.03340.96210.0285GELU0.95810.03210.95410.0353FReLU0.90840.10410.90810.1048FLReLU0.96430.04080.97020.0417FPReLU0.99290.00540.99320.0056FELU0.80751.36030.84861.3801FSiLU0.83420.13560.95110.0431FGELU0.58411.14560.59811.1489FLReLUReLU0.93460.05550.94250.0563LReLU0.91930.06120.92770.0535PReLU0.91750.07030.91690.0596ELU0.89420.11140.87430.2889SiLU0.92710.05460.92870.0626GELU0.91560.06410.91310.0661FReLU0.91710.10650.89910.1368FLReLU0.94560.04090.93450.0498FPReLU0.89690.54410.89311.4355FELU0.93360.05880.92260.0716FSiLU0.79120.15830.78440.1552FGELU0.525250.740.5618330FPReLUReLU0.87940.10370.88570.1098LReLU0.93420.05780.93230.0624PReLU0.98710.02150.99070.0233ELU0.81840.16560.81690.1625SiLU0.95910.09080.95760.0965GELU0.96050.04530.96170.0568FReLU0.81840.13260.80160.1445FLReLU0.95310.04010.95460.0358FPReLU0.85360.12350.88370.1199FELU0.870615.5220.877315.632FSiLU0.74280.75840.76720.6945FGELU0.6598Inf0.71052InfFELUReLU0.92670.12540.93170.1363LReLU0.89280.07840.90880.0695PReLU0.94070.04940.91750.691ELU0.94750.08550.94040.0901SiLU0.89330.13850.88490.1972GELU0.90542.84260.91773.2352FReLU0.88470.11340.89183.6661FLReLU0.82220.12460.83910.7484FPReLU0.90930.06960.90060.0738FELU0.85580.14820.85560.1392FSiLU0.84850.22220.84652.8651FGELU0.525250.740.5618262FSiLUReLU0.84610.29910.82240.311LReLU0.81270.460070.80170.5112PReLU0.1777121.4220.169980.343ELU0.82960.46780.8390.4652SiLU0.80250.14080.79110.1373GELU0.91660.060.89540.0842FReLU0.86030.1120.85360.1102FLReLU0.89150.08640.87360.1194FPReLU0.86511.00250.84611.0591FELU0.0789Inf0.1243InfFSiLU0.203Inf0.1989InfFGELU0.0588Inf0.1122InfFGELUReLU0.464313909732.360.462113909700LReLU0.40334299438.790.418815523900PReLU0.4473402966327.590.4481402966000ELU0.632123082.260.633823082.3SiLU0.507959202784.110.490559202800GELU0.55911203413.610.51591203410FReLU0.570214700123.310.592814700100FLReLU0.4162758904.720.45311620920FPReLU0.578411953493.940.623911953500FELU0.1541Inf0.1499InfFSiLU0.0829Inf0.1043InfFGELU0.0904Inf0.1392InfFigure 9
Performance of neural network with FReLU in hidden layer 1 and FPRELU in hidden layer 2 during training and testing.


## 5. Conclusion

This paper has developed the fractional form of ReLU and its variants to incorporate into the neural network’s layer to improve its performance. The response of all the derived fractional activation functions has been analyzed and compared with their conventional counterparts. From the analysis, it has been observed that the fractional activation functions have provided more flexible behavior compared to the conventional ones. Further, the simulation study to predict the system-generated wind power using a neural network with the developed activation functions in the hidden layer and output layers has shown the best performance. The case studies show that the fractional-order of linear activation functions has performed well, which could be the hyper-parameter they pose in the negative plane. In Case1, PReLU and FPReLU performed better than every other activation function. In Case 2, LReLU and FLReLU performed better than other activation functions. In Case 3, despite keeping any activation function in hidden layer 1, FLReLU performed better in all the instances than its conventional LReLU and other functions. However, the best result has been showcased by keeping PReLU in hidden layer 2 with FReLU in hidden layer 1.The limitation of the proposed activation functions performed at the output layer will be solved in the future part of the work. Also, the fractional form of the most widely used activation function tansig will be developed, and its performance analysis will be done in a real-time case study.



---

*Source: 1860779-2022-06-29.xml*