ARELU: ATTENTION-BASED RECTIFIED LINEAR UNIT
Authors: Dengsheng Chen
Jun Li
Kai Xu
Venue:  (2021)
Keywords: activation function
attention mechanism
rectified linear unit

Abstract: Element-wise activation functions play a critical role in deep neural networks via affecting the expressivity power and the learning dynamics. Learning-based activation functions have recently gained increasing attention and success. We propose a new perspective of learnable activation function through formulating them with element-wise attention mechanism. In each network layer, we devise an attention module which learns an element-wise, sign-based attention map for the pre-activation feature map. The attention map scales an element based on its sign. Adding the attention module with a rectified linear unit (ReLU) results in an amplification of positive elements and a suppression of negative ones, both with learned, data-adaptive parameters. We coin the resulting activation function Attention-based Rectified Linear Unit (AReLU). The attention module essentially learns an element-wise residue of the activated part of the input, as ReLU can be viewed as an identity transformation. This makes the network training more resis- tant to gradient vanishing. The learned attentive activation leads to well-focused activation of relevant regions of a feature map. Through extensive evaluations, we show that AReLU significantly boosts the performance of most mainstream network architectures with only two extra learnable parameters per layer introduced. Notably, AReLU facilitates fast network training under small learning rates, which makes it especially suited in the case of transfer learning and meta learning.

Under review as a conference paper at ICLR 2021
ARELU: ATTENTION-BASER RECTIFIED LINEAR UNIT
Anonymous authors
Paper under double-blind review
ABSTRACT
Element-wise activation functions play a critical role in deep neural networks via affecting the expressivity power and the learning dynamics. Learning-based activation functions have recently gained increasing attention and success. We propose new perspective of learnable activation function through formulating them with element-wise attention mechanism. In each network layer, we devise an attention module which learns an element-wise, sign-based attention map for the pre-activation feature map. The attention map scales an element based on its sign. Adding the attention module with a rectified linear unit (ReLU) results in an amplification of positive elements and a suppression of negative ones, both with learned, data- adtatavee parameters. We coin the resulting activation function Attention-base Rectified Linear Unit (AReLU). The attention module essentially learns an element-wiss residue of the activated part of the input, as ReLU can be viewed an identity transformation. This makes the network training more resis- tant to gradient vanishing. The learned attentive activation leads to well-focused activation of relevant regions of a feature map. Through extensive evaluations, we show that AReLU significantly boosts the performance of most mainstream network architectures with only two extra learnable parameters per layer introduced. Notably, AReLU facilitates fast network training under small learning rates, which makes it especially suited in the case of transfer learning and meta learning.
INTRODUCTION
Activation functions, introducing nonlinearities to artificial neural networks, is essential to networks'
expressivity power and learning dynamics. Designing activation functions that facilitate fast training of accurate deep neural networks is an active area of research (Maas et al. 2013 Goodfellow et al.
2013] Xu et al. 2015a| Clevert al. 2015) Hendrycks Gimpell 2016 Klambauer et al. 2017 7
Barron 2017
Ramachandran et al.) (017). Aside from the large body of hand-designed functions.
learning-basee approaches recently gain more attention and success (Agostinelli et al. 2014 4 He
al. 2015 Manessi & Rozza 2018 Molina et al. 2019: Goyal et al. 2019). The existing learnable a non-learnable activation function (e.g. Rectified Linear Units (ReLU) (Nair & Hinton 2010)) with learnable parameters (He
activation functions are motivated either by rel relaxing/parameteriiin
jet al. 2015), or by seeking for a data-driven combination of a pool of pre-defined activation functions (Manessi & Rozza 2018). Existing cerning-based methods make activation functions data-adaptive through introducing degrees of freedom and/or enlarging the hypothesis space explored.
In this work, we propose a new perspective of learnable activation functions through formulating them with element-wise attention mechanism. A straightforward motivation of this is a straightforward ob- servation that both activation functions and element-wise attention functions are applied as network module of element-wis wise multiplication. More intriguingly, learning element- wise activation functions neural network can intuitively be viewed task-oriented attention mechanism (Chorowski et al. 2015 Xu et al. 2015b), i.e., learning where (which element in the inpul feature map) to allend (activate) given an end task to fulfill. This motivates an arguably more interpretable formulation of attentive activation functions.
Attention mechanism has been a cornerstone in deep learning. It directs the network to learn which part of the input is more relevant or contributes more to the output. There have been many variants of attention modules with plentiful successful applications. In natural language processing, vector wise
attention is developed to model the ct al., 2015) Vaswani et al. (017). Many computer vision tasks utilize pixel-wise or channel- attention modules for more expressive and invariant representation learning (Xu et al.) 201.5b Chen
dependencies in sequence of word vectors (Luong

---

Under review as a conference paper at ICLR 2021
Element-wise attention (Bochkovskiy et al. 2020) is the most fine- grained where each element of a feature volume can receive different amount of attention. Consequently, it attains high expressivity with neuron-level degrees of freedom.
al. 2017).
Inspired by that, we devise for each layer of a network an clement-wise attention module which learns sign-based attention map for the pre-activation feature map. The attention map scales element based on its sign. Through adding the attention and a ReLU module, we obtain Attention based Rectified Linear Unit (AReLU) which amplifies positive elements and suppresses negative ones, both with learned, data-adaptive parameters. The attention module essentially learns an element-wise residue for the activated elements with respect the ReLU since the latter can be viewed as an identity transformation. This helps ameliorate the gradient vanishing issue effectively. Through extensive experiments on several public benchmarks, we show that AReL.U significantly boosts the performance of most mainstream network architectures with only two extra learnable parameters per layer introduced. Moreover, AReLU enables fast learning under small learning rates, making especially suited for transfer learning. We also demonstrate with feature map visualization that the learned attentive activation achieves well-focused, task-oriented activation of relevant regions.
2 RELATED WORK
Non-learnable activation functions Sigmoid a non-linear, saturated activation function used mostly in the output layers of deep learning model. However, it suffers from the exploding/vanie gradient problem. As a remedy, the rectified linear unit (ReLU) (Nair & Hinton 2010) has been the most widely used activation function for deep learning models with the state-of-the-an performance in many applications. Many variants of ReLU have been proposed to further improve its performance on different tasks LReLU (Maas et al., 2013). ReLU6 (Krizhevsky & Hinton 2010), RReLU (Xu et al. 2015a). Besides that, some specified activation functions also have been designed for different usages, such CELU (Barron (017)) ELU (Clevert al. (013)). GELU (Hendrycks & Gimpell 2016). Maxout (Goodfellow et | 2013). 3). SELU (Klambauer et al. 2017). (Softplus) (Glorot et al.
2011 Swish (Ramachandran al. 2017
Learnable activation functions Recently, learnable activation functions have drawn more atten- tions. PReLU (He et al.) 2015), as variants of ReLU, improves model fitting with little extra computational cost and overfitting risk. Recently, PAU (Molina et al., 2019) is proposed to not only approximate common activation functions but also learn new while providing compact representations with few learnable parameters. Several other Iearnable activation functions such as APL (Agostinelli et al. 2014), Comb (Manessi & Rozza 2018). SLAF (Goyal et al. 2019) also achieve promising performance under different tasks.
Attention Mechanism Vector-Wise Attention Mechanism (VWAM) has been widely applied in
Natural Language Processing (NLP) tasks (Xu et al., 2013c Luong et al. 2015: Bahdanau et al.
2014 Vaswani al., 2017 Ahmed et al.) 2017). VWAM learns which vector among sequence of word vectors is the most relevant to the task in hand. Channel-Wii Attention Mechanism (CWAM)
can be regarded as an extension of VWAM from NI.P to Vision tasks (Tang et al.
2019b 2020 Kim
et al. 2019). It learns to assign each channel an attentional value. Pixel-Wise Attention Mechanism (PWAM) is also widely used in vision (Tang et al., (019c a). Element- Wiement-Wise Attention Mechanism (EWAM) assigns different values to each element without any spatial/channel constraint. The recently proposed YOLOv4 (Bochkovskiy et al. 2020) is the first work that introduces EWAM implemented by convolutional layer and sigmoid function. It achieves the state-of-the tth--attttttt art performance on object detection. We introduce a new kind of EWAM for learnable activation function.
3 METHOD
We start by describing attention mechanism and then introduce element-wise sign-based attention mechanism based on which AReLU is defined. The optimization of AReLU then follows.
3.1 ATTENTION MECHANISM
Let us denote V = -w;} E RRP.DDD.... a tensor representing input data or feature volume. Function
R parameterized by θ {0:), is used compute attention map $ {si) E R DD xDD x
2

---

Under review as a conference paper at ICLR 2021
Dimersion
Chonnel peoomto map
V
ReLU
stccniin map
mondion extersion
Post-ectivation feature maps
V
Pre-activatin
V
feature
AReLU
db
Element wite
Figure 1: Left: An illustration of attention mechanisms with attention map different granularities. Right: Visualization of pre-activation and cost-activation feature maps obtained with ReLU and AReLU on testing image of the handwritten digit dataset MNIST LeCun et al. (1988).
over a subspace of (Iet (-)) denote a correspondence function for the indices of dimension):
8; (U;, θ).
+ can be implemented by a neural network with θ being its learnable parameters.
We can modulate the input V with the attention map S using function vr obtaining the output {u:} D!
UG (vs,
is an element-wise multiplication. In order to perform element-wise multiplication, one needs to first extend S the full dimension of V. We next review various attention mechanisms with attention map at different granularities. Figure leett) gives an illustration of various attention mechanisms.
Vector-wise Attention Mechanism In NLP, attention maps are usually computed over different word vectors. In this case, V = { E R NxD represents a sequence of N feature vectors with dimension D. (s;} E R.N is sequence of attention values for the corresponding vectors.
Channel-wise A Attention Mechanism In computer vision, feature volume V = {(i E R #xxC
has spatial dimension of W x and a channel dimension of C. {s; RC is an attention map over the C' channels. All elements in each channel share the same attention value.
Spatial-wise Attention Mechanism Considering again dimension of W H. S = {s:} R is an attention map over the spatial dimension. All channels of a given spatial location share the same attention value.
[wa} WxxCC with a spatial
Element-wise Attention Mechanism Given a feature volume V = {w:} E W containing W x X elements, we compute an attention map over the whole volume (all elements), i.e., S 8t) E BDXXXxC so that each element has an independent attention value.
3.2 ELEMENT- WISE SIGN-BASE BASED ATTENTION (ELSA)
We propose, ELSA, a new kind of element-wise attention mechanism which is used to define our attention-based activation. Considering feature volume V {wi} E RWx HxxHxC compute an element- dlement-wise attention map S = {s.} E W XXCC
C(aa. (3))
0
8; (v,, θ)
(3)
where é = (a, E R² is learnable parameters. C(-) clamps the input variable into [0.01, 0.99). the sigmoid function. The modulation function V is defined as:
U,- V (U,, 9t) s,Ugs
In ELSA, positive and negative elements receive different amount of attention determined by the two parameters a and B. respectively. Therefore, it can also be regarded as sign-wise attention mechanism. With only two learnable parameters, ELSA is light-weight and easy learn.
3

---

Under review as a conference paper at ICLR 2021
3.3 ARELU: ATTENTION-BASER RECTIFIED LINEAR UNITS
We represent the function & in ELSA with a network layer with learnable parameters a and 8:
C(a)aa (x o 3) = (3))2
24 <
(5)
I4 z220
where X = {z, is the imput of the current layer. In constructing an activation function with ELSA, we combine it with the standard Rectified Linear Units
0,
I2 <
R(x))
(6)
I 2220
Adding them together leads to a learnable activation function:
Flz, 0) = R(x;) + C(2,, a, 3) - (1 C(a)
24 <0
(7)
(3))
24 2:20
This combination amplifies positive elements and suppresses negative ones based on the learned scaling parameters 3 and a, respectively. Thus, ELSA learns an element-wise residue for the activated elements w.r.t. ReLU which is an identity transformation, which helps ameliorate gradient vanishing.
3.4 THE OPTIMIZATION OF ARELU
AReLU can be trained using back- pack-propagation jointly with all other network layers. The update formulation of a and B3 can be derived with the chain rule. Specifically, the gradient a is:
30
OF a.
(8)
da OF(z,, a, B)
da
ac where E is the error function to be minimized. The term aF is the gradient propagated from the deeper layer. The gradient of the activation of X with respect is given by: OF(X, a. B)
3;
do
Here, the derivative of the clamp function C(-) is handled simply by detaching the gradient back- propagation when 0.01 a > 0.99.
The gradient of the activation X with respect to /3 is:
aF, = Σ C
(10)
03
2t4 20
The gradient of the activation with respect to input x; by:
OF a)
< 1 (B), x, 2
(11)
Fx:
It can be found that AReLU amplifies the gradients propagated from the downstream when the input activated since 1 1 (B) > 1: it suppresses the gradients otherwise. On the contrary, there is no such amplification effect in the standard ReLU and its variants (c.g., PReLu (He et al. (015))
-only suppression is available. The ability amplify the gradients over the activated input helps avoiding gradient vanishing, and thus speeds up the training convergence of the model (see Figure 3). Moreover, the amplification factor learned to dynamically adapt to the input and is confined with the sigmoid function. This makes the activation more data-adaptive and stable (see Figure rright) for a visual comparison post-activation feature maps by AReLU and ReLU). The suppression part is similar to PReLu which learns the suppression factor for ameliorating zero gradients.
ARcLU introduces a very small number of extra parameters which is 2L for an L-layer network. The computational complexity due AReLU is negligible for both forward and backward propagation.
Note that the gradients a and / depend on the entire feature volume X. This means that ELSA be regarded as global attention mechanism: Although the attention map is computed in an element-we wise manner, the parameters are learned globally accounting for the impact of the full feature volume. This makes A more data-adaptive and hence the whole network more expressive.
4

---

Under review as a conference paper at ICLR 2021
(b)
procadure parameter
Fuy Paa 835,
Play -.5 00
Four m 273,
ARgLU
Rut 2.91 Faa
Figure 2: (a): Plot of accuracy over epochs for networks trained with different initialization of a and B. A larger initial / leads to faster convergence and higher accuracy is obtained when a is initialized to 0.25 or 0.75. (b): The learning procedure of a and / which are initialized to 0.25 and 1.0, respectively. (c): The learned final AReLU's for the three convolutional layers of the MNIST-Conv network. The shaded region gives the range of AReLU curves.
4 EXPERIMENTS
We first study the robustness of AReLU in terms of parameter initialization. We then evaluate convergence of network training with different activation functions on two standard classification benchmarks (MNIST (LeCun et al. 1998)) and CIFARI00 (Krizhevsky et al., (2009.. We compare AReLU with 18 different activation functions including 13 non-learnable ones and 5 learnable ones; the list in Table The number of learnable parameters for each learnable activation function also given in the table. In the end, we also demonstrate the advantages of AReLU in transfer learning, Please refer to supplemental material for more results and experiments details.
4.1 INITIALIZATION OF LEARNABLE PARAMETER AND B
For evaluation purpose, design neural network (MNIST-Comy) with three convolutional layers each followed by max- mnox-nooling layer and an AReLU, and finally fully connected layer followed by softmax layer. Details of this nelwork can be found in the supplemental material. The experiment on parameter initialization is conducted with MNIST-Conv over the MNIST dataset. As shown in Figure 2 a). AReLU is insensitive to the initialization of a and 3. Different initial values result in close convergence rate and classification accuracy. Generally, a large initial value of 3 can speed up the convergence. Figure bb) shows the learning procedure of the two parameters and (c) plots the learned final AReLU's for the three convolutional layers. In the following experiments, we initialize 0.9 and 8 2.0 by default.
4.2 CONVERGENCE MNIST
On the MNIST dataset, evaluate MNIST-Conv implemented with different activation functions and trained with the ADAM or SGD optimizer. The activation function is placed after each max- pooling layers. We compare AReLU with both learnable and non-leamable activation functions under different learning rates of 1 x 10-2, x 10-3 x 10-4, and x 10-5, To compare the convergence speed of different activation functions, we report the accuracy after the first epoch, again taking the mean over five times training: see Table [ In the table, we report the improvement of AReLU over the best among other non-learnable and learnable methods. In Figure we plot the mean accuracy over increasing number of training epochs.
As shown in Table T. AReLU outperforms most existing non-learnable and learnable activation functions in terms of convergence speed and final classification accuracy on MNIST. A note-worthy phenomenon is that AReLU can achieve a more effective training with small learning rate (see the significant improvement when the learning rate is 10-4 4 or 1 x 10-5) than the alternatives. This can also observed from Figure 3 Generally, smaller learning rates would cause lower learning efficiency since the vanishing gradient issue is intensified in such case. AReLU can overcome this difficulty thanks to its gradient amplification effect. Efficient learning with a small learning rate is very useful in transfer learning where a pre-trained model usually fine-tuned on domain/dataset
5

---

Under review as a conference paper at ICLR 2021
Table I: Mean testing accuracy (%) MNIST for five trainings of MNIST-Conv after the first epoch with different optimizers and learning rates, We compare AReLU with 13 non-learnable and learnable activation functions. The number of parameters per activation unit are listed beside the name of the learnable activation functions. The best numbers are shown in bold text with blue color for non-learnable methods (the upper part of the table) and red for learnable ones (the lower part). At the bottom of the table, report the improvement of AReLU over the best among other non-learnable and learnable methods, blue and red color respectively.
Learning Rate
1 x 10
1 x 10 3
1 x
1 10
Optimizer
Adam
SGD
Adam
SGD
Adam
SGD
Adam
SGD
CELU (2017)
97.76
96.12
96.21
62.81
84.01
13.07
24.84
9.60
ELU 2015
97.82
96.17
96.22
58.10
85.67
14.07
19.77
10.13
GELU 6
98,49
94.90
95.79
12.55
83.72
11.49
15.20
10.92
LReLU 2013
97.80
95.59
95.86
35.90
84.08
10.28
15.41
10.73
Maxout 2013
97.04
95.81
96.14
71.75
84.81
10.79
18.83
9.06
ReLU (2010)
97.75
95.02
95.40
36.01
84.02
10.68
15.25
8.73
ReL.U6 (OTO))
97.77
95.32
96.09
43.42
81.39
10.23
14.33
9.56
RReLU (2015a)
98.09
95.88
95.65
53.33
9.57
10.28
SELU 2017
97.25
96.52
96.61
82.36
85.36
16.49
30.04
9.59
Sigmoid
47.16
11.04
83,59
11.35
11.37
9.92
10.52
10.10
Softplus 2011
96.38
90.90
93.83
11.14
51.83
9.19
10.21
9,89
Swish
(2017)
98.10
94.02
95.91
11.44
83.91
10.69
11.39
Tanh
96.93
94.22
96.45
57.70
79.25
11.73
27.05
10.31
APL 2014)
(2)
97.00
95.71
94.67
17.81
76.73
9.39
13.28
11.83
Comb 2018 (1)
98.28
95.97
95.79
35.95
83.91
10.59
20.22
10.18
PAU
2019)
(10)
98.17
97.67
96.73
40.11
87.08
10.54
14.49
11.11
PRcLU
2015) (1)
98.22
95.72
95.87
45.73
85.81
12.08
14.51
9.88
SLAF
2019
96.30
97.07
95.32
83.35
72.67
14.12
10.04
11.32
ARcLU (2)
98.00
97.30
97.13
93.13
90.44
47.78
38.39
14.25
Improvement
-0.49
+0.78
+0.52
+10.77
+4.77
+31.29
+8.35
+3.33
Improvement
0.28
0.37
+0.40
+99.78
+3336
+33.66
18.17
+2242
CELU
Softplus
ELU
Swish
GELU
Tarh
Maxout
Combire
50
RReLLJ
PAU
ReLU
ReLU6
SLAF
SELU
AReLU
Sigmoid
Figure 3: Plots of mean testing accuracy (%) on MNIST for five-time trainings of MNIST-Com over increasing training epochs. The training conducted using SGD with small learning rates (left: 1 x 10-4, right: 1 50-5).
with a small learning rate which difficult for most existing deep networks. Section 4.4 will demonstrate this application of AReLU,
4.3 CONVERGENCE ON 100
In order to better demonstrate the effect of ELSA, we regard the ReLU, without ELSA, as our baseline. For plot clarity, we choose to compare only with those most representative competitive activation functions including PAU, SELU, ReLU, LReLU (LReLU), and PReLU. More results can be found in the supplemental material. We evaluate the performance of AReLU with five different mainstream network architectures CIFAR100. We use the SGD optimizer and follow the training configuration in |Pereyra et al. 2017): The learning rate 0.1, the batch size 64, the weight decay is 10 10-4 and the momentum is 0.9.

---

Under review as a conference paper at ICLR 2021
Proteclen
10
10
10
30.2
20.20
Figure 4: Plots of mean testing accuracy (%) on CIFARIO0 over increasing training epochs, using different network architectures. The training is conducted using SGD with a learning rate of 0.1.
Input
ReLU
ARPLL
Figure 5: Grad-CAM visualization of feature maps extracted by ResNet- 18 with AReLU and ReLU.
The first row is the testing images of CIFARI00.
The results are plotted in Figure 4 Learnable activation functions generally have faster convergence compared to non-learnable ones. AReLU achieves faster convergence speed for all the five network architectures. It is worth to note that though PAU can achieve a faster convergence at the beginning in some networks such as SeResNet- 18, it tends overfit later with a fast saturation of accuracy. AReLU avoids such overfitting with smaller number of parameters than PAU (2 vs 10).
We also conduct qualitative analysis of AReLU by visualizing the learned feature maps using Grad-CAM Selvaraju et al.] (2017) using testing images of CIFAR100. Grad-CAM is a recently proposed network visualization method which utilizes gradients to depict the importance of the spatial locations in a feature map. Since gradients are computed with respect to specific image class, Grad-CAM visualization can be regarded as task-oriented attention map. Figure we visualize the first-layer feature map of ResNet- 18. As shown in the figure, the feature maps learned with AReLU leads to semantically more meaningful activation of regions with respect to the target class. This is due to the data-adaptive attentive ability of AReLU.
4.4 PERFORMANCE TRANSFER LEARNING
We evaluate transfer learning of MNIST-Conv with different activation functions between two datasets: MNIST and SVHNT The data preprocessing for adapting the datasets follows (Shin et al. (017). We train three models and test them on SVHN: 1) one is trained directly on SVHN without any pretraining. one trained on MNIST but finetuned on SVHN, and 3) one pretrained on MNIST and finetuned on SVHN. In pretraining, we train MNIST-Conv using SGD with learning rate of 0.01 for 20 epochs which is sufficient for all model variants converge. In finetuning, we train the model on SVHN with a learning rate of 1 X 10-5, using SGD optimizer for 100 epochs.
The testing results on SVHN are reported in Table 2 where we compare AReLU with several competitive alternatives. Without pretraining, it is hard to obtain good accuracy on the difficult task of SVHN. Nevertheless, MNIST-Conv with AReLU performs the best among all alternatives; some activation functions even failed in learning. In the setting of transfer learning (pretrain finetune), ARcLU outperforms all other activation functions for different amount pretraining, thanks to high learning efficiency with small learning rates.
http://ffdll.ssanffon stanford www.:bnnd..du//hoossuumeee

---

Under review as a conference paper at ICLR 2021
Table 2: Test accuracy (%) SVHN by MNIST-Conv models (implemented with different activation functions) trained directly on SVHN (no pretrain), trained on MNIST but not finetuned (no finetune), as well as pretrained on MNIST and finetuned on SVHN for 5, 10 and 20 epoches. The left part of the table is non-learnable activation functions and the right learnable ones.
Sctting
ELU
GELUU
Maxout
RcLU
SELU
Softplus
APL
Comb|
PAU
PRRLU
SLAF
AReL.U
no pretrain
19.59
19.59
23.01
19.58
19.58
19.58
19.58
19.58
19.58
19.58
19.58
24.95
no finetune
31,95
37.38
36.52
36.87
32.57
14.39
36.20
35.89
24.67
33,45
35.74
31.91
f.1. epochs
70.08
69.19
70.18
69.76
72.81
65.81
71.73
69.63
75.24
66.13
75.91
76.68
Lt. 10 epochs
70.83
71.69
71.31
71.38
72.11
69.14
73.51
70.31
76,26
67.63
76.21
78.12
Ett. 20 epochs
71.48
70.34
73.29
72.06
71.55
71.01
72.41
72.55
74.46
71.99
74.38
78.48
Table 3: Test accuracy (%) on MNIST by MAML with MNIST-Conv models implemented with
different activation functions. The performance is compared on a 5-woos---shooss task and a 5-ways- 5-shots task, respectively.
(ways, shots)
ELU GELU Maxout
ReLU
SELU
Softplus]
APL
Comb
PAU
PReLU
SLAF
AReLU
(5. 1)
82,50
81.88
83.13
70.00
83.75
25.63
71.25
75.63
43.13
88.13
84.38
92.50
(5,5)
94.30
69.37
93.12
78.00
93.50
22.12
63.00
88.25
40.12
91.75
77.00
94.30
4.5
PERFORMANCE META LEARNING
We evaluate the meta learning performance of MNIST-Conv with the various activation functions based on the MAML framework Finn et al. (2177). MAML is a fairly general optimization based algorithm compatible with any model that learns through gradient descent. It aims to obtain meta- learning parameters from similar tasks and adapt the parameters to novel tasks with the same distribution using a few gradient updates. MAML, model parameters are explicitly trained such that a small number of gradient updates over small amount of training data from the novel task could lead to good generalization performance on that task. We expect that the fast convergence of AReLU would help MAML to adapt a model to novel task more efficiently and with better generalization. We set the fast adaption steps as and use 32 tasks for each steps. We train the model for 100 iterations with learning rate of 0.005. We report in Table 3 the final test accuracy for different activation functions on a 5-ways-1-sshee --shoss task and a 5-ways-5-shots task, respectively. The results show that AReLU shows clear advantage compared the alternative activation functions. One noteworthy phenomenon is the performance of PAU (Molina et al. 2019): It performs well in other evaluations but not on meta learning which is probably due to overfitting- prone nature.
4.6 THE GENERALIZED EFFECT OF ELSA
In this experiemnt, we show that ELSA (Element-wise Sign-based Attention) can serve as a general module which can be plugged in to any existing activation function and obtain a performance boost for most cases.
We define new activation F the same as Eq. 7 but replace the ReLU function R with specified activation function. We keep the same experiment settings as Sec. 4.2] As shown in Tab: [ after plugging with ELSA module, Table I indicating the well generalized effect of ELSA.
obtain performance boost for most cases compared with
5 CONCLUSION
We have presented AReLU, a new learnable activation function formulated with element-wise sign- based attention mechanism. Networks implemented with AReLU can better mitigate the gradient vanishing issue and converge faster with small learning rates. This makes it especially useful in transfer learning where a pretrained model needs to be finetuned in the target domain with small learning rate. AReLU can significantly boost the performance of most mainstream network architectures with only two extra learnable parameters per layer introduced. In the future, would like to investigate the application/extensio language translation and even structural feature learning with graph neural networks.
AReLU to more diverse tasks such as object detection,
8

---

Under review as a conference paper at ICLR 2021
Table 4: Mean testing accuracy (%) MNIST for five trainings of MNIST-Cony after the first epoch with different optimizers and learning rates. For each activation function and each learning rate, we show results training with ELSA module. The numbers showing ELSA module improves over the original activation function (shown in Table are highlighted with underline.
I Learning Rate
1 x 10
1 x 10
x 10
1 x 10
Optimizer
Adam
SGD
Adam
SGD
Adam
SGD
Adam
SGD
CELU (2017)
97.81
97.40
97.06
93.43
89.90
65.53
45.99
13.86
ELU (2155)
97.72
97.51
96.88
93.38
89.73
57.85
40.77
12.96
GELU 2016
97.99
97.49
96.93
92.93
89.83
42.01
37.40
T0.22
LReL.U 2013
97.95
97.38
96.88
93.02
89.86
51.62
39.17
12.90
Maxout
97.33
97.47
96.98
93.50
90.16
66.74
48.22
16.37
ReLU (2010)
98.13
97.43
97.00
92.54
89.99
50.00
46.18
11.87
ReLU6 2010
97.89
97.60
97.05
92.83
89.89
45.15
39.17
13,40
RReLU 22015a
97.77
97.37
97.29
92.75
89.72
56.28
37.64
12.61
SELU (017)
97.37
97.32
96.81
93,90
89.91
68.84
46.14
11.79
Sigmoid
96.87
96.06
95.69
81.99
81.90
24.03
22.77
9.81
Softplus 201
96.80
97.11
96.25
91.11
85.68
39.44
23.92
12.55
Swish (0117)
97.72
97.45
96.62
92.47
89.74
61.69
38,90
T3.18
Tanh
97.62
97.24
96.97
91.03
88.89
56.39
44.05
1186
APL 2014)
98.10
97.48
96.81
93.12
89.68
46.81
28.68
11.37
Comb (2018
97.95
97.46
97.08
93.07
89.43
51.74
38.97
13.48
PReLU 2015
97.82
97.45
96.91
93.06
90.41
55.41
43.42
11.96
SLAF (0119
95.36
96.74
96.28
93.80
85.31
64.03
22.26
21.06
REFERENCES
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830, 2014.
Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation, 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate, 2014.
Jonathan T Barron. Continuously differentiable exponential linear units.
arXiv preprint
arXiv:1704.07483, 2017.
Alexey Bochkovskiy, Chien Rao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5659-5667, 2017.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-baser models for speech recognition. In Advances in neural information processing systems, pp. 577-585, 2015.
Djórk-Amé Clevert Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07288 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv: 1703.03400, 2017.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323, 2011.
Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprint arXiv: 1302,4389, 2013.

---

Under review as a conference paper at ICLR 2021
Mohit Goyal, Rajan Goyal, and Brejesh Lall. Learning activation functions: A new paradigm for understanding neural networks, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.084// 08415, 2016.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for amage-to-image translation, 2019.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp- 971-980, 2017.
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar- 10. Unpublished manuscript, 40(7):1-9, 2010.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEFE, 86(11) (0(11):277823222 1998.
Minh-Thang Luong, Hieu Pham, and Christopher 1) Manning. Effective approaches to attention-based neural machine translation. arXiv preprint 1r0xiv:150.04400 2015.
Andrew L Maas, Awni Y Hannun, and Andrew Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3, 2013.
Franco Manessi and Alessandro Rozza. Learning combinations of activation functions. 2018 24th International Conference Pattern Recognition (ICPR), pp. 61 61-66. IEEE, 2018.
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Pad) e activation units: End-to-end learning of flexible activation functions in deep networks. arXiv preprint arXiv:/907.06732 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 1rXiv:1701.063: 2017.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint 17X1v:1710.05941, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via pradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 318-626, 2017.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances Neural Information Processing Systems, pp. 2990-2999, 2017.
Hao Tang, Hong Liu, Dan Xu, Philip HS Torr, and Nicu Sebe. Attentiongan: Unpaired image- to-image translation using attention-guided generative adversarial networks. arXiv preprint 1rXiv:1911.11897, 2019a.
Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang. Jason J. Corso, and Yan Yan. Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation. In CVPR, 2019b.
10

---

Under review as a conference paper at ICLR 2021
Hao Tang, Dan Xu, Nicu Sebe, and Yan Yan. Attention-guia guided generative adversarial networks for unsupervised image-to-image translation. In International Joint Conference on Neural Networks (JJNN), 2019c.
Hao Tang, Dan Xu, Yan Yan, Jason J Corso, Philip HS Torr, and Nicu Sebe. Multi-channel attention selection gans for guided image-to-image translation. arXiv preprint arXiv:2002.01045 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in 2015a.
convolutional network. arXiv preprint
Kelvin Xu, Jimmy Ba. Ryan Kiros. Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-2057, 2015b.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. 02 2015c.


Reviews:
[{"review_id": "tBeLlRSu3I0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Description:\n\nThis work presents a novel learned activation function called Attention-based Rectified Linear Unit (AReLU). Element-wise attention module is developed that learns a sign-based attention (ELSA) which is the novel component of AReLU towards mitigating the gradient vanishing issue. Extensive experiments and analyses have been provided on MNIST and CIFAR100 datasets. Compared with other relevant activation functions, AReLU achieves faster convergence under small learning rate because of the amplification of positive elements and suppression of negative ones with two learnable data-adaptive parameters.   \n\nStrengths:\n- 13 learnable and 5 non-learnable activation function compared with the proposed AReLU. Outperforms most in convergence speed and final classification accuracy on MNIST.\n- More effective training with a small learning rate. \n- Impressive results for transfer learning and meta-learning \n- Grad-CAM on CIFAR100 show semantically more meaningful activations \n\nWeaknesses:\n- Results are reported on CIFAR100 and MNIST datasets. The results would be more compelling if provided on datasets such as ImageNet.  \n- Implementation details are not provided. It would be really important to confirm if the same setup is used when comparing different activations (seed, initialization, hardware, environment)\n- It is not intuitive why two different families of models were used for different datasets - MNIST-Conv (specially designed) for MNIST while Resnet-18 and VGG11 for CIFAR100\n\nQuestions:\n- Could the authors confirm if the code would be made publicly available for reproducibility? \n- It seems like a faster convergence and more improvement observed when SGD (compared to ADAM) is used as the optimizer (Table 1). Was a similar graph observed when using ADAM as the optim for Figure 3?  \n- Continuation of the above point, it would be interesting to see a similar table for CIFAR100 dataset when using ADAM / SGD as the optimizer.  \n- Could the authors clarify the motivation for two different learnable parameters for positive and negative elements compared to only using a single parameter? Also why sigmoid is used for positive elements. \n- Have the authors considered other viable techniques to combine ELSA with ReLU (apart from addition in Section 3.3)?   \n\nSuggestions/Comments:\n\n- Please color code (red) the learnable activation function in Table1\n- Section 3.1 input a data -> input data\n- Figure 1 An visualization -> Visualization\n- Please correct the equation in Section 3.4 (better if equations are numbered)\n- It would be helpful to provide the motivation for clamp function also in Section 3.2 apart from 3.4\n- It would be helpful to be consistent with the nomenclature - MNIST Conv vs ConvMNIST in A1.\n- Please confirm the dataset in A3 - CIFAR10 or CIFAR100\n- It would help to explain “detaching the gradient back-propagation” in Section 3.4, independent of the framework used. \n\nThe paper is clearly written in most parts, It would be interesting to see other comments and discussions on this paper. \n\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nPost Rebuttal update:\n\nI would like to thank the authors for providing relevant details and a thorough rebuttal to all the issues raised by the fellow reviewers. Original rating is maintained.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "sW-xm2FLuJA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\nThis paper is well written and is easy to follow. It focuses on the basic component in neural network, i.e. neural activation function.  Extensive experiments have been conducted and the corresponding experimental results are provided.\n\nCons:\nWhile, in my view, the novelty of this paper is limited and needs more improvements. By changing the actual function performed on the input tensor X has been explored too much, such as the recent proposed FRELU: Funnel Activation for Visual Recognition. What's the difference between this paper and the proposed one? It seems the difference is limited, and they belong to the same family.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jzSxqGrm3KL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Overview]\n\nIn this paper, the authors proposed a new activation function called AReLU which introduces an attention mechanism to the original ReLU function. Based on this new activation function, the output will be adaptively adjusted by the two learnable parameters \\alpha and \\beta. This kind of adaptive adjustment can be thought of as an attention mechanism undertaken over each element in the input feature map. It will in general amplify the positive elements while suppressing the negative ones, and the parameters \\alpha and \\beta will be adjusted adaptively based on the activation values. The experimental results showed that AReLU can achieve much better performance with small learning rates while comparable performance with fairly large learning rates. This inspires another set of transfer learning experiments that demonstrate the effectiveness of AReLU.\n\n[Strength]\n\n1. This paper proposed a simple element-aware activation function. It is built based on ReLU but differs from ReLU in that it has two learnable parameters and will adaptively augment the positive inputs while suppressing the negative ones.\n\n2. The mechanism behind the proposed AReLU seems reasonable. It increases the scale to later than one for positive responses while decreasing the scale for negative responses.\n\n3. The experiments showed that AReLU is more robust to the learning rates compared with other activation functions, either non-learnable or learnable ones. Also, for a lower learning rate, the convergence speed is much faster than other activation functions. This property facilitates the transfer learning scenarios as shown in Sec. 4.4.\n\n[Weakness]\n\n1. It seems that AReLU will hurt the performance when the learning rate is reasonably set, as shown in the first super-column (1e-2) in Table 1. Also, it is not clear about the final performance on CIFAR-100. Overall, it hard for me to determine whether the proposed AReLU can have generic benefits to the training under different settings.\n\n2. In Fig.2 (b),  the values of \\alpha are different for different learning rates, while (1+\\sigma(\\beta)) converges to similar value. This raises the question that whether it is necessary to set a learnable \\beta or not because we can simply fix the value of \\beta at the beginning. Even for \\alpha, it is not clear whether it is necessary to learn it because the authors did not show the final accuracies for different settings.\n\n3. In Table 3, the authors showed that AReLU is more suitable for transfer learning with a low learning rate. However, the comparison is a bit unfair, for other activation functions, we can simply increase the learning rate and it might be the case that they can achieve better performance with a larger learning rate. Another baseline is, as pointed above, we can simply set \\alpha less than 1 (say 0.5) while 1+\\sigma(\\beta) to around 1.5.\n\n4. The proposed AReLU is good for learning the parameters in an adaptive manner. However, does this will introduce another problem, i.e., overfitting? With less amount of data, this adaptiveness may hurt the generalization ability of the learned network. In this paper, the authors did not study this aspect. \n\n5. If the goal of AReLU is mainly to address the vanishing gradient, then what if we remove the sigmoid to have a (1+\\beta)? What will be the outcome in this case? I would like to hear from the authors about this.\n\n[Summary]\n\nOverall I think this paper is well-written and with a fluent flow to follow. The intuition behind the proposed AReLU is clear to understand and I like it. Experimental results demonstrated the effectiveness of AReLU, especially under training+finetuning settings. However, as pointed above, I am still concerned about the generalization of AReLU and the necessity of applying AReLU in the training. I would like to hear more from the authors in the rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-eP82eOQS0Q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper proposed an attention module in which each element of features is scaled based on the sign of its value.  The authors argue that the combination of this attention module and ReLU can be considered as an activation function.\nThis novel activation function amplifies its gradient comparing with ReLU.\nThe experimental results show the proposed method outperforms conventional activation functions in few-shot settings of transfer/meta learning.\n\nStrong points\n* The proposed method consistently improves the accuracy of transfer/meta learning tasks.\nWeak points\n* The technical novelty is limited since the derived activation function is an expansion of PReLU:\n  * PReLU: min(0, a * x_i) + max(0, x_i)\n  * Proposed: min(0, alpha * x_i) +  max(0, (1+sigmoid(beta))*x_i) where 0.01 < alpha < 0.99.\n* Although the very specific form of ELSA is used in Section 3.2, there is no explanation of why C and sigma are introduced for ELSA.\nThe learning rate is not tuned for each activation function in Section 4.4 and 4.5.\n\nDecision reason\nSince the learning rates of the baseline methods seem to be not optimized, the experimental results are not convincing.\nI recommend resubmitting the paper after the tuning of baseline methods.\n\nQuestions\n* Why did you use momentum SGD for alpha and beta instead of using Adam or SGD as other parameters?  Their learning rates are also different from these of other parameters?\n* How is the learning rate determined for the experiments in Section 4.4 and 4.5?   Even if the learning rate is tuned for baseline methods, the proposed method outperforms them?\n\nAdditional Feedback\n* Although it is interesting to formulate activation functions with an element-wise attention mechanism, the merit of this new perspective is not clear for me.\n* It might be easier to understand the proposed method as an expansion of PReLU since the proposed attention mechanism is much different from the attention mechanism in the community.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I would like to thank the authors for the their time and effort on this work. The paper is proposing an activation function that combines RELU like piecewise activation functions and a primitive attention mechanism. Then, they show that their proposed method works better in transfer settings.

I think the approach authors taking here is more akin to a gating mechanism rather than an attention. So I would recommend the authors to change the name perhaps Gated Rectified Linear Units. The paper is interesting, but I agree with AnonReviewer4, that the experiments are not very convincing focusing on small scaled experiments in the supervised learning setting only. I would recommend the authors to compare their approach against other results from the literature as well. As it is right now it is not clear how significant the results in this paper are. I don't think the transfer and meta-learning experiments are very well-motivated in this paper. I would recommend the authors to better motivate those results.

After considering my suggestions above and the comments from the reviewers I would recommend the authors to consider resubmitting to another conference.



Decision: Reject