LAU: A novel two-parameter learnable Logmoid Activation Unit
Authors: Xue-Mei Zhou
Ling-Fang Li
Xing-Zhou Zheng
Mingxing Luo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural network
Learnable activation function
Function approximation
Dilated convolution

Abstract: The activation function in deep neural networks has a major impact on the performance  of the training stage. In this work, we proposed a novel learnable Logmoid Activation  Unit (LAU), $f(x)=x\ln(1+\alpha \textrm{sigmoid}(\beta x))$, with two free parameters  $\alpha$ and $\beta$ that can be optimized via back-propagation algorithm. We design  quasi-interpolation type neural network operators with Logmoid-1 in a given feed-forward neural network for approximating any continuous function in closed spaces. This provides a theoretical basis for the excellent empirical performance of LAUs in experimental simulations. For instance, compared with ReLUs the proposed LAUs improves Top-1 classification accuracy on ImageNet-200 by $7\%$ respectively in ShuffleNet-V2, on CIFAR-10 by 6$\%$ respectively in EfficientNet-B0, and on CIFAR-100 by 5$\%$ respectively in MobileNet-V2. Our simulations show that end-to-end learning deep neural networks with learnable Logmoids can increase the predictive performance.

Under review as a conference paper at ICLR 2023
LAU: A NOVEL TWO-PARAMETER LEARNABLE LOG- MOID ACTIVATION UNIT
Anonymous authors
Paper under double-blind review
ABSTRACT
this work, we proposed a novel learnable Logmoid Activation Unit (LAU), f(x) x In(1 + (3x) by parameterizing Logmoid with two hyper- parameters a and 3 that are optimized via back back-propagation algorithm. We design quasi-interpolation neural network operators with Logmoid-1 for approximating any continuous function in closed spaces. Our simulations show end-to-end learn- ing deep neural networks with learnable Logmoids increase the predictive perfor- mances beyond all well-known activation functions for different tasks.
INTRODUCTION
In recent years, deep learning has achieved remarkable success in various classification problems (LeCun al. 2015; Sriperumbudur et al.) 2010). The main reason the powerful abilities of deep neural networks (DNNs) in representation and learning for the unknown structures. One of most important components in DNNs is activation function. well-designed activation function can greatly improve the predictive performances (Krizhevsky et al. 2017). This has intrigued growing interests in exploring activation functions (Nwankpa et al. 2018 Liang Srikant 2016 Shen al, 2019).
Activation functions are generally classified into linear, nonlinear monotonic and nonlinear non- monotonic functions. Although linear functions including Step Function (Klein et al., 2009). Sign Function (Huang Babri 1998) and Identity Function have been widely used in early results, they useless in practical applications because of the discontinuous derivatives, or lack of biologi- cal motivation and classification ability. These problems are further addressed by using nonlinear monotonic functions such Sigmoid, Tanh and ReLU families. While small derivatives of Sigmoid (Hassell et al. 1977) and Tanh (Kalman & K Kwasny/ 1992) may cause the gradient to disappear (He & Xu 2010 Klambauer al. 2017). Softplus and ReLU functions are designed for solving this prob- lem. Mcanwhile, ReLU shows other characters such as the educed saturation, sparsity, efficiency, and ease of use, but the neural network may lose some valid information because of all negative values of ReLUU being zero. This intrigues designing new activation functions such as Leaky ReLU (Maas al. 2013), RReLUU (Xu al. 2015). ELU (Clevert et al., 2015) and Swish (Ramachandran al. 2017
In general, designing good activation function is still an open question. One method is to find new activation functions by combining different units, such as Mish (Misra 2019) and TanhExp (Liu & Di. 2020). The second is to parameterize some well-known activation functions Biswas et al. (2020): Zhou et al. (2020). which may show better performance beyond parameter-free functions. One example is Logish Zhhu et al. (202T) which exhibits better performance and Top-1 accuracy. Another is Padé Activation Unit (PAU) et al. (2019) using learnable syper-parameter(s) and back-propagation algorithm (LeCun et al.) 1989).
In this work, proposed new family of activation functions by parameterizing Logmoid with fewer trainable parameters for each network layer. It is given by
f(a;; a, () x In(1 + aSigmoid((3z))
where the logarithmic operation reduce the range of Sigmoid, and and / are trainable param- eters. The main contributions are summarized as follows:
1. A new family of activation functions called Logmoid family is proposed.

---

Under review as a conference paper at ICLR 2023
Table
The Top-1 accuracy of Logmoid with parameters on CIFARIO.
Parameter(a 1)
Top-1
Parameter(3
Top-T
76.93%
10
68.48%
5
78.2%
5
73.99%
77.5%
1
77..5%
0.01
36.2%
0.01
-0.01
41.28%
-0.01
77.11%
-10
77.06%
-10
2. Two quasi-interpolation type operators with Logmoid-1 are constructed to approximate univariate continuous functions.
3. The trainable Logmoid Activation Unit performs better than others in simulations.
The rest is organized as follows: Section 2 briefly describes some relative activation functions. Section 3 proposes new activation functions by parameterizing Logmoid. Section 4 analyzes the performances of the subfamily Lgomoid-1. In Section 5, we introduce learnable activation unit called LAUs, and implement several simulations to verify its effectiveness, while the last section concludes the results.
2 RELATED WORKS
This section devotes some related activation functions including Swish, TanhSoft and PAU.
Swish- Swish
(Ramachandran et 2017) 7) is defined by
f(z) zSigmmid(11).
This function shows new features unsaturation, smooth and nonmonotonicity. can potentially present problems in gradient-based optimization because of not continuously differentiable.
SanhSoft-TanhSonn family is indexed by four hyper-parameterr as [a; o, B, 77,77 = tanh(ox + 3)²²) In(8 + e*).
(3)
Based on simulations on CIFARIO dataset, two sub-families of TanhSoft-1 and TanhSoft-2 are de- fined as
TanhSoft-1: :
f(x; 0,7, = tanh(ax) In [1 + e*), 0, 0) tanh(Be?*)).
(4)
TanhSoft-2:
(5)
TanhSoft-1 1 (a 0.87) and TanhSoft-2 0.6, 1) may show higher Top-1 performance than others Biswas et al. (2020).
PAU- Padé Activation Units (PAU) is learnable activation function based on rational function. The padé a rational function of the form
F(x) do ar a22. + Um
+ dy + byx + + b
PAU allows free parameters which can be optimized end-to-end in neural networks.
3 LOGMOID ACTIVATION FUNCTION FAMILY
In this section we present a new family of activation function by parameterizing Logmoid with two hyper-parameters. Specially, we propose family of functions with two parameparseeters as
f 3) = x In(1 + (xx))
where a refers to Sigmoid (zx) = T+e-a Table shows its Top-1 accuracy. For a = 1, the
accuracy decreases faster for -1 < 3 < 1, and reaches to 78.2% for 3 = For 3 = 1, it decreases faster with the gradual increase of This means Logmoid with has better performance.
2

---

Under review as a conference paper at ICLR 2023
p-0.1
5-1
p-2
B-10
0901
0010
B=2
8=10
0110
15
0.5
-1.5
3
-2
2
(a)
(b)
Fig. 1: (a) Numerical Logmoid (solid line) and its derivative (dashed) in terms of and 1. (b) Numerical Logmoid (solid line) and its first derivative (dashed line) with respect and / 1.
ReLU
Swish
TanhExp
Logmoid-1
Fig. 2: Comparison between the output landscapes of ReLU, Swish, TanhExp and Logmoid-1 on 5-layer network.
[a) a) shows Logmoid and its derivative in terms of B. For 1, Logmoid is scaled linear function ((x kr if 3 + 0. Logmoid is then regarded as an interpolation between linear function and ReLU. Here, B shows how fast the first derivative changes asymptotically. When /3
keeps the gradient property of ReLU for solving the dying problem. From Fig Logmoid is far away from the linear function f(z) in the positive part of æ-axis for large a and 1. For practical
applications, one may choose hyper-parameters from 1 & a & 5 and 1 < 8 < 5. This is further regarded as sub-family Logmoid-1 (2; 1).
CONTINUOUS FUNCTION APPROXIMATIONS BY LOGMOID-1
In this section the sub-family Logmoid-1 will be used to approximate any continuous function compact set. Logmoid-1 is smooth and non-monotonic activation function beyond ReLU. Mean- while, it inherits the self-gated property of Swish where the self-gates refers the input itself and function with the input as its argument, i.e., f(x) xz(x).
The output landscapes of five-layer fully connected network are shown in Fig 2 While ReLU has lot of sharp transitions. Logmoid-1 shows continuous and fluent transition shape. This is then useful for optimization and generalization (Li et al. 2018).
Next, we construct bell-shaped function with Logmoid-1 and present main theorems for approxi- mate functions.
Bell-shaped functions-Frood Eq. Logmoid-1 with a 3 = is defined as g(x) x In(1 + (x:)), IR
(8)
According to ref. (Cardaliaguet & Euvrard 1992). define new bell-shaped function
<xxx) + 2|1
Tf < < Tí: if < r < To:
((v(()) + 2(22),
v(x)
(9)
(xx)
if Tı;
Abb(1),
if < T2.
3

---

Under review as a conference paper at ICLR 2023
where (x) is defined by (xx)) f(2 + 2) .(xx - 1/) with bounded function ppx)) = g( +
() -g( 1). A 0.934, To -0.2536 Ti 3.5025 and T2 -3. 3.7326 extreme points, and Y1 = 0.0181 and Y2 -0.0307 are the values of (b) at points Ti and Z2. Here, À is used ensure the continuity of (x) at To.
And then, we define another bell-shaped function as
(xx) (ar
(10)
where ş is defined by Ei= v2 - k) ç. implies that -k= D(x - k) 1. We further define
r(x) = { e 1
i 0:
e 1
Approximation errors-As universal approximation operators, Feed-forward neural networks
(FNNs) (Hornik al. 1989 Cybenko by
1989 with hidden layer and n + neurons represented
(x) = Σ cjs((a) x) bb) + I RR 1=0
(11)
where is activation function, [0, n. and bj R thresholds of the hidden layer, aj R are the connection weights between the input and hidden layers, Cj E IR are the connection weights between the hidden and output layers, dj E IR are the thresholds of the output layer, and (+) denotes the inner product.
In general, any continuous function on a compact set can be theoretically approximated up to any precision by increasing hidden neurons (Cybenko 1989 Hornik et al. 1990 Homnik [991] Funahashi 1989 Leshno et al. 1993). We present tighter estimation of approximation error by Logmoid-1. Let 1|-1, 1] be the space of continuous functions on [1, For any function
Cla, b). define
w(f. = max
[f(x) f(y)l.
(12)
uswvst wi
f is (L, a-Lipschitttz continuous with a E [0, (e.g., f E Lip(L,a) if there is a constant L > such that w(S,)) L8
For any function f E 1[. we construct a FNN operator by ((x)
( = Σ 1 10)x -k)
(13)
Theorem For the operator G LLogmoid-1(F. we have
If(z) G Grogmoid 1 (f,z)) S w(f, 1 na + (Se -8c-3n"" In'-a + 4 +1]1ffloo
where || lloo denotes the uniform norm.
This means the approximation error of f(x) by GLogmoid-1(f1 can be controlled by n and a.
The proof is shown in Appendix A.
Moreover, denote (RR as the set continuous and bounded function on R. For any function (R) define FNN operator as
Σ 10z k). ==-00
G.oggmidd - x= =
(14)
Theorem For the FNN operator 1 we have
f(x) G Logmoid- ( S w(f, _
+ Se tal-a If
na

---

Under review as a conference paper at ICLR 2023
100
0.005
TarhExp
90
0.03
0.025
Accuracy
Test Loos
0.02
60
TanhExp
Sw
RaLU
0.005
Logmodd
10
0
Number of Epoches
Number Epoches
(a)
(h)
Fig. 3: Test of Logmoid-1, TanhExp, Swish, and ReLU with 15-layer basic network MNIST. (a) Test accuracy. (b) Test loss.
90
85
80
Todt Accuracy (S)
Test Acpuraay
70
60
LAUS
Tari-liop
Legmoks-1
Staieh
65
RELU
ReLU
Swish
Logmold--1
Mall
30
19
21
16
32
206
512
Number of Layers
Betch Sire
(a)
(b)
Fig. 4: (a) Test accuracy of Logmoid-1, TanhExp, Swish, and ReLU with the hasic network MNIST. (b)
The test accuracy vs Batch size for LAUs, Logmoid-1, ReL.U, Swish, Mish.
The proof of Theorem is shown in Appendix B. Similar to the bell-shaped function 10 new bell-shaped function based on Swish and Eqs. 9 is defined
[()(x) 2|) if - > Tı; (o((z).
Swish (x)
(15)
otherwise.
where À = 1.758, Ti is the extreme point, and Yi -0034 = <(Ti). ((x) is defined by
(bx)) (w + 1) 2Swish Swish - 1)
(16)
Similar to the FNN operator Googoiid (f, x),, we may define FNN operator Gswish (ʃ, x) by Swiiish (Ir), in order to further verify the performance of Logmoid-1. Two examples are shown in Appendix C.
Table 2: The Top-1 accuracy (in %) of different learning rates (Ir) for LAUs, Logmoid-1, ReLU, Swish, Mish.
Ir
ReLU
Swish
Mish
Logmoid-1
LAUs
10-1
87.42
87.62
87.7
87.18
88.97
10
87.91
88.15
88.38
88.36
88,58
10-3
77.92
80.38
80,99
80.17
80.44
10-4
50.33
51.48
53.66
52.53
55.36
5

---

Under review as a conference paper at ICLR 2023
100
0.04
TomExo
80
Test Accuracy (%6)
Test Loos
0.025
0.02
Txn
Swish
RelU
0.005
0
Number Epoches
Number Epocnes
(a)
(h)
Fig. 5: Test of Logmoid-1, TanhExp, Swish, and ReLU with 15-layer basic network KMNIST. (a) Test accuracy. (b) Test loss.
RAg
1256116405
LnvartSelret
Fig. 6: Schematic VGG-8- --Dlated network. DConv denotes the dilated convolutional.
EXPERIMENTS WITH LOGMOID ACTIVATION UNITS (LAU)
In this section simulate new activation functions for different tasks. We denote Logmoid Activa- tion Units (LAU) with parameters as
f B) x 1n(1 + (Bx))
(17)
where a and 8 are trainable hyper- hyper-paramete parameters being optimized end-to-end to find good one at each layer automatically. Logmoid networks are feed-forward networks including convolutional and residual architectures with pooling layers. Inspired by PAUs, we suppose to learn one LAU per layer. There are 2L parameters with L layers.
5.1 INITIALIZING EXPERIMENTS WITH LOGMOID-1
This subsection for exploring good initializations for all LAUs. We show that Logmoid-1 is good beyond random initializations. Several ablation experiments are performed to evaluate Logmoid- on MNIST (LeCun al. 2010) and KMNIST (Clanuwat et al. 2018) with 10 classes. All the simulations used a basic network with 15 layers (Liu & Di 2020). Each one of the last 12 layers contains a batch normalization (loffe Szegedy 2015). dropout rate (Srivastava et al.) 2014 of 0.25, an activation function, and dense layer with 500 neurons.
Learning speed. We trained the basic 15 layers network with 15 epoches on MNIST and 30 epoches on KMNIST. As shown in Figs and Logmoid-1 performs better than Swish and ReLU both in the convergence speed and final accuracy, and has a similar performance with TanhExp. Logmoid-1 update the parameters rapidly and force the network to fit the dataset in efficient way.
Over-fitting. Experiments are performed with Logmoid-1, ReLU, Swish, and TanhExp on both MNIST and KMNIST using the basic network with blocks from 15 to 25, From Fig the network suffers from over over-fitting by ReLU and Swish because sharp decrease in accuracy. Logmoid-1 and TanhExp maintain a high accuracy in large models and then prevent from over-fitting.

---

Under review as a conference paper at ICLR 2023
5.2 EXPERIMENTS WITH LAUS
This subsection devotes the simulation experiments with LAUs for different tasks. We simulate LAUs and other baseline functions including ReLUU, Swish, TanhExp, ACONC (Ma et al.) 2021)
and Logmoid-1 1 using standard deep neural networks. We benchmark the results in the follow-
ing datasets: MNIST (LeCun et al.) 2010), Kuzushiji-MNIST (KMNIST) (Clanuwat et al.) 2018
Fashion-MNIST (FMNIST) (Xiao et (017). CIFAR (Krizhevsky al. 2009), ImageNet (Rus- sakovsky et al. 2015) and COCO (Lin et 20144)
Ablation experiments with Le demonstrate the image classification performance with LAUs and others. ShuffleNet-V2 V2 network is used as the backbone. All the experiments are sim- ulated on CIFAR-10 dataset.
Batch size. We test LAUs in the same network with the batch size of 16, 32, 64, 128, 256, 512, respectively. LAUs has the best performance compared with others as Fig 4bg
Learning rate. Table 2 shows the Top-1 accuracy in ShuffleNet-V V2 network. For the learning rate 10 LAUs has the highest 88.97% test accuracy while three nonmonotonic functions have similar performance with 10-2, LAUs is best for learning rate 10-2 This means LAUs has good performance in almost all comparable learning rates. Meanwhile, expression ability good at smaller learning rate.
Table 3: Performance comparison of activation functions on FMNIST.
Activation
LeNet(#0.06M)
VGG-8(#9.2M)
VGG-S-Dilated (#3.9
ReLU
c89.84
90.19
90.96
Swish
89.37
90.15
90.80
TanhExp
89.47
90.72
90.45
ACONC
89.27
c91.23
c91.12
Lgomoid-1
88.99
90.56
90.25
LAUS
89..0
92.35
91.84
Actord: 0220370, 0.7444
0.5615
,2 0.52067
Actirt: 0.52M7,
0.59019
1.5
0.5
.0.5
-2
Fig. 7: The estimated activation functions after training LeNet with LAUs on FMNIST.
Empirical results on Fashion-MNIST-We. take use of two basic architectures of LeNet (Lecun al. 1998) and VGG-8 (Simonyan & Zisserman 2014) to evaluate Logmoid-1 on FMNIST. LeNet contains 61,706 trainable parameters while LAUs adds 8 additional parameters. We investigate whether there are smaller sub-networks with similar performances. Some blocks of VGG-8 will be replaced by dilated convolutions, as Fig named VGG-8-Dilated network.
From Table 3 LAUs improves 1.1% Top-1 accuracy compared with the runner-up result on VGG- 8. Comparing the baseline functions on the (GG-8-Dilated and LeNet, LAUS always matches or outperforms the best performance. This shows LAU is a learnalle activation function with- out requirement of additional suboptimal experiments. From 7 some learnable activations

---

Under review as a conference paper at ICLR 2023
smooth versions of Logmoid-1 by controlling hyper- ppper-parameterss and B. So, initialize LAUs with Logmoid-1.
Empirical results CIFAR-WA explore the performance of LAUS difficult datasets, CIFAR-10 and CIFAR-100 KKry et al. 2009p. These 3-channel images require a neuron network with
strong learning ability. We simulate LAUs well other baseline functions on ResNet-20 (He
et al.
2016), (Howard et al. (017)) MobileNet-V2 (Sandler et al., 2018), ShuffleNet (Zhang et al.
2018
ShuffleNet-V2 (Zhang al. 2018), SqueezeNet (landola et 2016), SeNet- (Hu al,
2018
and EfficientNet-BU (Tan & 2019).
From Table LAUUs outperforms the baseline activation functions in most cases. LAUS improves the Top-1 accuracy by around 5% on EfficientNet-BO The predictive performance of LAUs on ResNet- 20 and SqueezeNet still competitive. LAUs gets 2.49% improvement over ReLU on MobileNet while SRS (Zhou et al. 2020) gets 2.33% improvement.
Table Performances of activation functions on CIFAR-10.
Model
ReLU
Swish
ACONC
TanhExp
Logmoid-1
LAUs
ResNet-20
90.96
91.08
s91.16
91.26
90.97
c91.12
MobileNet
86.04
87.93
+88.31
88.24
87.91
88.53
MobileNet-Vi V2
84.92
85.87
686.54
85.83
86.39
.87.95
ShuffleNet
87.55
88.87
88.95
o89.2
87.22
.8.91
ShuffleNet-V2
87.42
87.62
687.64
87.38
87.18
88.99
SqueezeNet
88.50
87.04
088.35
87.86
87.63
c88.22
SeNet-18
88.81
90.53
90.37
90.19
090.95
99,.33
EfficientNet-BO
76.17
77.35
778.55
77.19
78.23
82.52
Table 5: Performances of activation functions CIFAR-100.
Model
ReLU
Swish
ACONC
TanhExp
Logmoid-1
LAUs
ResNet-20
66.16
67.73
67..66
c67.79
67.68
c67.77
MobileNet
59.40
62.11
62.35
c62.21
61.81
c62.05
MobileNet-V
59.40
61.32
o62.13
60.76
60.09
63.66
ShuffleNet
64.53
65.45
66.17
66.7
c66.76
68..01
ShuffleNet-V2
63.65
65.88
o65.94
66.18
65.69
64.73
SqueezeNet
62.64
c63.97
63.45
62.38
63.64
64.26
SeNet-18
66.26
c67.85
67.78
67.68
67.45
68.96
EfficientNer-BO
45.81
46.01
c48.26
47.52
47.08
50.38
The CIFAR-100 has 100 classes with 500 training images and 100 test images per class. From Table LAUs outperforms ReLU on all networks. It improves the top-1 accuracy by 4.6%, 3.3%, 4.4%, 2.1% and 2.9% compared with ReLU, Logmoid-1, Swish, ACONC and TanhExp on EfficientNet- BO. LAUS gets 2.65% improvement over ReLU on MobileNer and is smaller than its from SRS. LAUs shows smaller classification accuracy by 1.4% compared with the best on ShuffleNet-V2. This may be further improved by increasing simulation epochs.
Table 6: Comparison of different activations the ImageNet- dataset. We report the Top-1 and Top-5 accuracies (in %) on ShuffleNet- V2 and MobileNet- V3 (Howard et al. 2019).
ReLU
Swish
ACONC
TanhExp
Logmoid-1
LAU's
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5-1
Top-1
Top-S
ShufleNer-V2 V2
59.81
82.21
60.38
85.15
62.35
84.68
60.05
83.07
c61.07
085.02
63333
056.13
MohileNe-V33
73.88
+77.5
Empirical results on 2mageNet-200-11 We compared LAUs with other baseline activation functions
on ImageNet 2012 (Russakovsky et al. 2015). For quick comparison, randomly choose 200 classes and extracted 500 training images, 50 testing images each class from ImageNet, named as
ImageNet-200. From Table 6 LAUs leads in both the Top-1 and Top-5 accuracies. LAUs gets 4.6%
Top-1 improvements over ReLU on MobileNet-V V3 while PWLU (Zhou et al. 2021) gets 1.91 %
Top-1 improvements.
8

---

Under review as a conference paper at ICLR 2023
Table Performances of activation functions on the COCO object detection task. We report results on Mask R-CNN with Swin Transformer backbone.
Activation
Apboz
AP5o
A AP1
Apmask
AP
Al
ReLU
45.8
64.6
50.3
41.2
63.3
44.7
ACONC
47.4
66.3
50.9
42.4
65.2
45.5
LAUS
48.7
67.5
52.1
43.5
66.8
46.9
Object detection on COCO -bje-ttject detection is a fundamental branch of computer vision. We implement simulation experiments on COCO 2017 (Lin et al. 2014), which contains 118K training. 5K validation and 20K test-dev images. We choose the Mask R-CNN (He et al. 2017) as the detector and Swin Transformer (Liu al., 202T) with different activations as the backhone. We choose batch size of 2, AdamW optimizer (initial learning rate of 0.0001, weight decay of 0.05, and 3x schedual). From Table 7 LAUs gets 2.9%.Apooz and 2.3% P.pes improvements respectively over ReLU while PWLU Zhou et al. 2021 gets 1.42% Aphor and 1.83%A improvements.
CONCLUSIONS
In this work, we presented a novel family of nonmonotonic activation function, named as Logmoid family. We constructed a class of FNN operators with Logmoid-1 to approximate any continu- ous function. We proposed a learnable Logmoid Activation Unit (LAU), which is initialized using Logmoid-1, trainable in end-to-end fashion. One can replace standard activation functions with LAU units in any neural networks. Simulations shows LAUS has the best performance across all ac- tivation functions and architectures. Further work will be performed apply LAUs to other related tasks.
REFERENCES
Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey. Tanhsoft-a a family of activation functions combining tanh and softplus. 2.Xxv:2009,03863, 2020.
Pierre Cardaliaguet and Guillaume Euvrard. Approximation of a function and its derivative with a neural network. Neural Networks, 5(2):207-220, 1992.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXhv:1812.017188 2018.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). 1 2015.
George Cybenko.
Approximation by superpositions of sigmoidal function. Mathematics of Con- trol, Signals and Systems, 2(4):303-314, 1989.
Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural Networks, 2(3): (13):183-192, 1989.
M.P. Hassell, J.H. Lawton, and J.R. Beddington. Sigmoid functional responses by invertebrate predators and parasitoids. The Journal of Animal Ecology, pp. 249-262, 1977.
Kai-Ming He, Xiang- Yu Zhang, Shao-Qing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Xin-Gui He and Shao-Hor Hua Xu. Process neural networks: Theory and applications. Springer Science & Business Media, 2010.

---

Under review as a conference paper at ICLR 2023
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, (2):251-257, 1991.
Kurt Homik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks uni- versal approximators. Neural Networks, 2(5):359- 3603366,,,,,,, 1989.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Networks, 3(5):511- 560, 1990.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Andrew G Howard, Meng-Long Zhu, Bo Chen, Dmitry Kalenichenko, Wei-Jun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural net- works for mobile vision applications. arXiv: 1704.04861, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-eexiiation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2018.
Guang-Bin Huang and Haroon A Babri. Upper bounds on the number of hidden neurons in feed- forward networks with arbitrary bounded nonlinear activation functions. IEEE transactions on neural networks, 9():2244-229 1998.
Forrest landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnct-level accuracy with 50x fewer parameters and ;0.5mb model size. 102:::1602.007360, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-444444 456. PMLR, 2015.
Barry L Kalman and Stan C Kwasny. Why tanh: 1992] IJCNN International Joint Conference on Neural Networks, volume 4, pp. 578-581. IEEE, 1992.
choosing a sigmoidal function. In [Proceedings
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in Neural Information Processing Systems, 30, 2017.
Stefan Klein, Josien PW Pluim, Marius Staring, and Max A Viergever. Adaptive stochastic gradient descent optimisation for image registration. International journal of computer vision, 81 81):)222 239. 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. Commun. ACM, 60(6):84-90, may 2017. ISSN 0001-0782. doi: 10.1145/3065386.
Y. Lecun, L. Bottou, Bengio, and P. Haffner. Gradient- based learning applied to document recog- nition. Proceedings of the IEEE, (6(11):277-2322) 1998.
Yann LeCun, Bernhard Boser. John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989.
Yann LeCun, Corinna Cortes, and J.C. Burges. Mnist handwritten digit database, 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553):436 444, 2015.
10

---

Under review as a conference paper at ICLR 2023
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861-867, 1993.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances Neural Information Processing Systems, 31, 2018.
Shi-Yu Liang and Rayadurgam Srikant. Why deep neural networks for function approximation? 2016.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Xin-Yu Yu Liu and Xiao- Guang Di. Tanhexp: A smooth activation function with high convergence speed for lightweight neural networks. arXiv:2003.09855, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Ningning Ma, Xiangyu Zhang. Ming Liu, and Jian Sun. Activate or not: Learning customized activa- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8032-8042, 2021.
Andrew L Maas, Awni Y Hannun, Andrew Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In International Conference Machine Learning, volume 30, pp. 3. Citeseer, 2013.
Diganta Misra. Mish: self regularized non-monotonic activation function. arXiv:1908.08881, 2019.
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning of flexible activation functions in deep networks. arXiv:1907.067:2 2019.
Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall. Activation func- tions: Comparison of trends in practice and research for deep learning. 1rXiv:/811.03377 2018.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. arXiv:1710.05941, 2017.
Searching for activation functions.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhi-Heng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, (5(3):211-252 2015.
Mark Sandler, Andrew Howard, Meng-Long Zhu, Andrey Zhmoginov, and Liang-Chice Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Zuo-Wo Wei Shen, Hai -hao Yang, and Shi-Jun Zhang. Deep network approximation characterized by number of neurons. arXiv: 1199906.5499 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. 1409 1556. 2014.
Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metrics probability measures. The Journal of Ma- chine Learning Research, 11:1517-1561, 2010.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 1419:1999-1958, 2014.
Ming-Xing Xing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pp. 6105-6114 14. PMLR, 2019.

---

Under review as a conference paper at ICLR 2023
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms. drXiv:/708..0747, 2017.
Bing Xu, Nai- Yan Wang, Tian-Qi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv: 00853, 2015.
Xiang- Yu Zhang, Xin-Yu Zhou, Meng-Xian Xiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Patern Recognition, pp. 6848 6856, 2018.
Yuan Zhou, Dandan Li, Shuwei Huo, and Sun- Yuan Kung. Soft-root-sign activation function. arXiv preprint arXiv:2003.00547 2020.
Yucong Zhou, Zezhou Zhu, and Zhao Zhong. Learning specialized activation functions with the piecewise linear unit. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12094-12104, 2021.
Hegui Zhu, Huimin Zeng, Jinhai Liu, and Xiangde Zhang. Logish: new nonlinear nonmonotonic activation function for convolutional neural network. Neurocomputing, 458:490-499, 2021.
12


Reviews:
[{"review_id": "bj_y0Rc3336", "reviewer": "Reviewer_KSN3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is 16 pages (+references), while the strict upper bound for ICLR is 9 pages. Thus I treat this paper as desk rejected.", "summary_of_the_review": "Exceeds page limit, desk reject.", "strength_and_weaknesses": "Exceeds page limit.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "aBQBcU9b765", "reviewer": "Reviewer_MpkZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new logmoid activation unit, that has nice properties to approximate continuous function and introduce performance gain over other activation functions across multiple datasets.", "summary_of_the_review": "This paper is borderline: interesting idea, good analysis, but weak experiments (see weakness)", "strength_and_weaknesses": "Strength:\n1. an interesting activation function with two parameters that covers the geometry of existed activation functions.\n\n2. theoretical analysis in terms of approximating continuous functions.\n\n3. excellent results achieved over multiple network structures and datasets.\n\nWeakness:\nthe experiments are far from extensive.\n1. we do not know if this activation works for other tasks like detection/segmentation.\n2. it is unknown if this activation works for transformer.\n3. it is unknown if this activation works well with dynamic operators like SE in MobileNet-V3.\n4. it is unknown if this activation works well for large scale dataset (ImageNet-1K, ImageNet-22K).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IwUPDjAcmzW", "reviewer": "Reviewer_Ea8B", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new family of activation functions called LAUs (Logmoid Activation Units). LAUs have two parameters α and β, and the authors comprehensively discuss the change of LAUs and their derivatives with different α and β. The authors also prove that Logmoid-1 (a special case of LAUs) makes Feed-forward neural network achieves lower approximation error than Swish with both mathematical and experimental results. The authors show that LAUs perform better than baseline activation functions on different datasets and architectures.", "summary_of_the_review": "The proposed LAUs are Sigmoid-based activation functions. The overall novelty is limited, and missed essential experiments. The paper is bad-written with the problem of duplication and lack of logical coherence.", "strength_and_weaknesses": "Strength:\nThe proposed LAUs are simple and easy to implement.\n\nWeakness:\n1.\tInsufficient experiments. Pade is introduced in Section Related Works, but the authors did not compare LAUs with Pade among all the benchmark experiments. Besides, there are too many experiments on tiny datasets and networks, making the improvements not very impressive and convincing. Similar to Pade, I suggest the authors provide results of LAUs on ImageNet-1K with MobileNet-V2. Furthermore, there are many powerful activation functions like ACON (Activate or Not: Learning Customized Activation) and PWLU (Learning specialized activation functions with the Piecewise Linear Unit) that the authors did not see. To make the results more convincing, I suggest authors compare LAUs with ACON and PWLU.\n2.\tThis paper is badly written. Section 4.1 suddenly appears without any introduction, which makes readers confused. And Section 4.2 is unnecessary because Section 5 provides experiments about Logmoid-1. All the tables are not centered. Table and figure captions are not clear enough for readers to understand. \n3.\tLimited novelty. There are many Sigmoid-based activation functions like Soft-Root-Sign and ACON, and it is convenient to provide different Sigmoid-based activation functions with search methods. So I do not see much novelty in this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject