Neural networks with trainable matrix activation functions
Authors: Yuwen Li
Zhengqi Liu
Ludmil Zikatanov
Venue: ICLR 2022 Submitted (2022)
Keywords: neural networks
trainable activation function
function approximation
image classification

Abstract: The training process of neural networks usually optimize weights and bias parameters of linear transformations, while nonlinear activation functions are pre-specified and fixed. This work develops a systematic approach to constructing matrix activation functions whose entries are generalized from ReLU. The activation is based on matrix-vector multiplications using only scalar multiplications and comparisons. The proposed activation functions depend on parameters that are trained along with the weights and bias vectors. Neural networks based on this approach are simple and efficient and are shown to be robust in numerical experiments.

Under review as a conference paper at ICLR 2022
NEURAL NETWORKS WITH TRAINABLE MATRIX ACTIVATION FUNCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
The training process of neural networks usually optimizes weights and bias pa- rameters of linear transformations, while nonlinear activation functions are pre- specified and fixed. This work develops systematic approach constructing matrix activation functions whose entries are generalized from ReLU. The acti- vation is based matrix-vector multiplications using only scalar multiplications and comparisons. The proposed activation functions depend on parameters that are trained along with the weights and bias vectors. Neural networks based on this approach are simple and efficient and are shown to be robust in numerical experiments.
INTRODUCTION
In recent decades, deep neural networks (DNNs) have achieved significant successes in many fields such computer vision and natural language processing (Voulodimos al.] 2018), (Otter et al., 2018). The DNN surrogate model is constructed using recursive composition of linear transfor- mations and nonlinear activation functions. To achieve good performance, it is essential choose activation functions suitable for specific applications. In practice, Rectified Linear Unit (ReLU) is one of the most popular activation functions for its simplicity and efficiency. A drawback of ReLU is the presence of vanishing gradient in the training process, known as dying ReLU problem (Lu jet al., 2019). Several relatively new activation approaches are proposed overcome this problem, e.g., the simple Leaky ReLU, and Piecewise Linear Unit (PLU) (Nicolac) 2018). Softplus (Glo-) rot et al. 2011), Exponential Linear Unit (ELU) (Clevert et al. (2016), Scaled Exponential Linear Unit (SELU) (Klambauer et al. 2017), Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel 2016
Although the aforementioned activation functions shown to be competitive in benchmark tests, they are still fixed nonlinear functions. DNN structure, is often hard to determine a priori the optimal activation function for specific application. In this paper, we shall generalize ReLUU and introduce arbitrary trainable matrix activation functions. The effectiveness the proposed approach validated using function approximation examples and well-known benchmark datasets such as CIFAR- 10 and CICAR-10C. There are few classical works on adaptively tuning of parameters in the training process, e.g.. the parametric ReLU (He et al., 2015b). However, our adaptive matrix activation functions are shown to be competitive and more robust in those experiments.
1.1 PRELIMINARIES
We consider the general learning process based given training set { fa)) where the inputs {x,, [ C. Id and outputs {fn ) C. Rd are implicitly related via an unknown target function R RJ with fn f(x)). The ReLU activation function piecewise linear
function given by
(1) = max (t,0), for E
(1) input vector. In a DNN, let L be the number
In the literature is acting component-wise of layers and n, denote the number neurons at the L-th layer for < l < L with no = d and ny = J. Let W = (W1, W2,.., WL) E I gaexue- denote the tuple of admissible weight matrices and B (b,, ba,..., b.) l Th the tuple of admissible bias vectors. The ReLU

---

Under review as a conference paper at ICLR 2022
DNN approximation to f at the L-th layer is recursively defined as
T 7, hw,.dz .... hWa,d2 hwi.di
where hy (a: War by an affine linear transformation at the f-th layer, and denotes the composition of functions. The traditional training process for such DNN is to find optimal W, E I1 R B, E II R"s, (and thus optimal n.,W., such that
N Σ fn TL.W.B(Zn)).
(W.B.. arg min W.B. N
(3)
In other words, NL, WB best fits the data with respect to the e norm within the function class (NL.w.B). In practice, the sum of squares norm used in minimization could be replaced with other convenient norms.
2 TRAINABLE MATRIX ACTIVATION FUNCTION
Having closer look ReLU G, we observe that the activation o o (=( ((e(()) could be realized as a matrix-vector multiplication a ne(r) = Dr (r()())))e((() or equivalently a o n (De a m)), where Tje is column vectur- in T and Dş is a diagonal matrix-valued function mapping from Re to Reexne with entries from the discrete set {0, 1). This is simple but quite useful observation. There is no reason to restrict on {0, 1} and we thus look for a larger set of values over which the diagonal entries of D; are running sampled. With slight abuse of notation, our new DNN approximation to ff is calculated using the following recurrence relation
hw..b:
M hwee [(De o The me)e e 1,...,--11
(4)
Here each Dg is diagonal and is of the form
Dz(y)
(e())))))))))))))))))))))))))))) ae(he))) y R"
(5)
where ae nonlinear function be determined. Since piecewise constant functions can approx- imate a continuous function within arbitrarily high accuracy, we choose az to be the following step function
1,,0-
S (-0,,111)
tg,
s E (51,
1
8 E (mm-1,
te:m:
8 (Sm, oc),
where m is positive integer and { and {sj}m= are constants. For the time being, let De (nn())))e(ez (z))(())) with og being scalar-valued function applied to ne(x) component- wise. the following, we list several oys corresponding to special ag and visualize them in Figure
m 1, 0, 0, 0,
de is ReLU,
m 1, 81 = 0,4, > 0,441 =1 = ae is Leaky ReLU,
m 2, 81 0, 52 1, fz.2 0, f..1
a is discontinuous.
In our case, we will choose the grid points {sy1m priori and train the step values UL (te,, mm of ay. such way, the resulting DNN may use different activation functions for different layers, and these activation functions are naturally adapted to the target function f and the target dataset. Since ReLU and Leaky ReLU are included by our DNN as special cases, the proposed DNN is clearly not worse than the traditional ones in practice. In the following, call the neural network in equation 4 with the activation approach given in equation 5 and equation 6 a DNN based on the "Trainable Matrix Activation Function (TMAF)".
2

---

Under review as a conference paper at ICLR 2022
10
1.0
08-
0.6
66-
04-
02
02
-100
-030
0225
1.0
-10
05
10
(a) m =,81
= 0, > 0, 0, =1
(b) m
2,
0,
1,tE,u 0, te,
fe.2
Figure 1: Graphs de.
REMARK
The activation functions used in TMAF neural network are not piecewise constants. Instead, TMAF activation is realized using matrix-vector multiplication, where entries of those matrices are train- able piecewise constants. There exist several trainable activation functions in the literature, e.g. Parametric ReLU (He et al. 2015b), Padé Activation Unit (PAU) (Molina et al. 2020). Piecewise Linear Unit (PWLU) (Zhou et al. 2021). Swish (Ranachandran et al.) 2017), adaptive rational functions (Boullé et al. 20200, adaptive splines (Bohra el al.) 2020), adaptive activation functions for physics informed neural networks (ada 2020). We point out that TMAF is different from the aforementioned activation approaches. One might think TMAF is similar to PWLU because they are both defined piecewise. However, by construction TMAF includes discontinuous functions as special cases (see Figure] 7 while PWLU always results in a continuous piecewise linear function with trainable slopes.
Starting from the diagonal activation De, we can go one step further to construct more general activation matrices. First we note that D₃ could be viewed nonlinear operator Te )I" [ where
[Te(g)]<z)= = (e(s())))(z), g (g....... In,)
T E ["" x Rd.
There seems be no special reason, aside from keeping the computational cost as low as possible, to consider only diagonal operators D. A more ambitious approach is to consider a trainable nonlinear activation operator determined by more general matrices, for example, by a tri-diagonal operator
(a)(g1(z)) (e(g((r))
O
7e(91(z))
ae(g2(x)) Be(gs(x))
g(x),
ae(gni-1 1e(gm,-(10))
Belgne (x))
Celgn, (x))
for x E Rdd The diagonal entry ag given in equation 6 while the off-diagonal entries 3g, Ye are piecewise constant functions defined in fashion similar ag. Theoretically speaking, even trainable full matrix activation is possible despite of potentially huge training cost. In summary, DNN based on trainable nonlinear activation operators {Te)/en reads
M
hw1.b17
Me+1 hWe.r.b44 o Te(Ne), f. --
(8)
As we indicated above, TMAF can be used with any matrix and this will lead to a great flexibility in training and approximation. fact, a sparse matrix is used, so that the matrix-vector mul- tiplication is proportional to the number of rows in the matrix, then the resulting algorithm will be computationally efficient. We can choose p-diagonal matrices, as well as matrices with prescribed sparsity patterns. Clearly, based the simple examples have considered here, may conclude that such possibilities, when investigated in depth, can increase the robustness and the accuracy of the TMAF-NN.
3

---

Under review as a conference paper at ICLR 2022
REMARK
Our observation also applies to an activation function a other than ReLU. For example, we may rescale (2) to obtain a (w),tv) using set of constants Sonrehssseeeeee and neuron by neuron. Then (w) are used form a matrix activation function and a TMAF DNN, where {ws,e} are trained according to given data and are adapted to the target function.
varying layer by layer
3 NUMERICAL RESULTS
In this section, we demonstrate the feasibility and efficiency of TMAF by comparing it with tradi- tional ReLU-type activation functions. Recall that neurons in the é-th layer will be activated by the matrix Dr. In principle, all parameters in equation Oare are allowed to be trained. To ensure practical efficiency, we shall fix interval grid points { used in ae and only let function values (1,,) in equation b| trained in the following. In each experiment, we use the same learning rates, stochastic optimization methods, and number NE of epochs (optimization iterations). In particular, the learn- ing rate is le-4 from epoch 1 to NE and le-5 is used from epoch NE + to NE. The optimization method is ADAM Kingmaa & Ba 2015)) based on mini-batches of size 500. Numerical experiments performed in py PyTorch (Paszke et al. (2019)). We provide two sets of numerical examples:
Function approximations by TMAF networks and ReL.Utype networks;
Classification problems for MNIST and CIFAR set solved by TMAF and ReLU networks.
For the first class of examples cation problems we consider the cross-entropy that is widely used as loss function in classification models. The cross entropy is defined using training set having p images, each with N pixels. Thus, the training dataset is equivalent to the vector set {z C R.V with each 2j being an image. The j-th image belongs to class Cj (1,...,,M) M). The neural network maps 2j to xy E RRM
the fo-loss function as defined in equation For the classifi-
Ij == 7L.,,8(23) RM, 2j E
j=
The cross entropy loss function of 1,, then is defined by
1 p
exp(xen.t))
E(W, B) =
log
P
ΣM
exp(2xx)
3.1 APPROXIMATION OF A SMOOTH FUNCTION
As our first example, we use neural networks approximate
f(z1.-.. .n,) = sin(TT1 + ......... TΣN). I E 1-2, 2].
k k= 1.....,
The training datasets consist 20000 input-output data pairs where the input data are randomly sampled from the hypercube [- 2)" based on uniform distribution. The neural networks have sin- gle double hidden layers. Each layer (except input and output layers) has 20 neurons. For TMAF Dj in equation
uses intervals (-oo, -1.4) (-1.4, -0.92], (-0.92, -0.66), (-0.56, - 0.26), 0.26, 0). (0, 0.26). (0.26, 0.56]. (0.56, 0.92], (0.92, 1.4]. (1.4, oo) such that prob- ability over each of the ten intervals is 0.1 with respect to Gaussian distribution. Moreover, d in Py Porch to the linear output of neural networks in each hidden layer. The approx- imation results shown in Table 1 and Figures 2 where Para-ReLU stands for the parametric ReLU neural network. It is observed that TMAF is the most accurate activation approach these examples.
5 the function
apply
3.2 APPROXIMATION OF AN OSCILLATORY FUNCTION
The next example is on approximating the following function having high, medium and low fre- quency components
f(x) = (on(100a.)) + cos(50xar)) + sin(T
see Figure 4a for an illustration. In fact. the function in equation | is rather difficult to capture by traditional approximation methods as it is highly oscillatory. We consider ReLU, parametric ReL.U,
4

---

Under review as a conference paper at ICLR 2022
Approximation error
Single hiden layer
Two hiden layers
72
2
5
ReLU
0.09
0.34
0.14
0.48
Para ReLU
0.04
0.11
0.09
0.47
TMAF
0.01
0.05
0.02
0.13
Table 1: Approximation errors for sin(mar + TTn) by neural networks
67
12-
03
68
0.3
0.
00
2002
(a) n
(b) 72
Figure 2: Training errors for sin(TT) + ... + TT..) single hidden layer
and diagonal TMAF neural networks with 3 hidden layers and 400 neurons layer (except the first and last layers). We also consider the tri-diagonal TMAF (denoted by Tri-diag TMAF, see equation]] 7 with 3 hidden layers and 300 neurons per layer. The training datasets are 20000 input- output data pairs where the input data are randomly sampled from the interval [-1, 1] based on uniform distribution.
The diagonal TMAF uses a = a in equation with intervals (-oo, -1], (-1+ kh, -1 (A: 1)4],
(1, oo) for h 0.02, 0 < A < 99. The tri-diagonal TMAF is given in equation 7 where or is the same as the diagonal TMAF, while Be is piecewise constant with respect to intervals (-00, 2201 + 99
h. {{ 2.01 kh, 2201 + co, 0.01), (0.01 + kh, 0.01 (R: }h] k o (2.01, respectively. Numerical results could
1)h]} k= (-0.01, and ye is piecewise constant based on
be found in Figures 4b 5 and Table 2
0.8
0.
0.7
0.-
0.5
6.2
0.1
250
300
1000
1290
1960
1730
2003
250
340
70
1000
1230
1100
1150
2000
7
(b)
Figure 3: Training errors for sin(T ... mT,)) two hidden layers.
5

---

Under review as a conference paper at ICLR 2022
final loss
ReLU
1.00
Para ReLU 1.00
Diag TMAF 6.45e-2
Tri-diag TMAF
5.81e-2
Table 2: Error comparison for si0010) + cos(500mz) + sin(xr)
For this challenging problem, we note that the diagonal TMAF and tri-diagonal TMAF produce high-quality approximations (see Figures and 4b) while ReLU and parametric ReLU are not able to approximate the highly oscillating function within reasonable accuracy, see Figure 4b and Table Moreover, it observed from Figure@Jthat ReLU and parametric ReLU actually approximate the low frequency part of the target function. To capture the high frequency, ReLU-type neural networks are clearly required to use much more neurons, introducing significantly amount of weight and bias parameters. On the other hand, increasing the number of intervals in TMAF only lead to few more training parameters.
ir-d og 1969
Retli
g
04
20.00
-6.35
-0.00
-025
0.00
C25
0.50
0.55
1.00
50
800
1008
opoch
(a) Exact oscillating function
(h) Training loss comparison
Figure 4: Plot of sin(1007z) + cos(500zz) + ssn( and training loss comparison
Bagg TMMF
Triving TMAA
2-
-2
-0.00 -0.35
+0.50
-0.25
0.00
C25
0.25
-1.00 -0.25
-0.0
0.25
1.0
(a) Diag TMAF approximation
(b) Tri-diag TMAF approximation
Figure 5: Approximations to sin(100ma) + cos((50x)) + sin(TT), TMAF-type
3.3 CLASSIFICATION OF MNIST AND CIFAR DATASETS
We now test TMAF by classifying images in the MNIST, CIFAR-10 and CIFAR-100 100 dataset.
For the MNIST set, we implement double layer fully connected networks defined as equation 2 and equation 4 with 10 neurons per layer (except at the first layer no 764), and we use ReL.U

---

Under review as a conference paper at ICLR 2022
P4U
1.00
Apr BA
675
0.5
030
0.0
022
-022
0.50
0.50
4.75
-300
-0.55
-0.0
-0.25
0.00
030
0.75
1000
0.30
-0.25
0.25
1000
(a) ReLU approximation
(b) Para ReLU approximation
Figure Approximations to
+ cos(500mr) sin(Tæ), ReLUU-yye
diagonal TMAF described in Subsection 3.1 Numerical results are shown in Figures 7a 7b and
Table 3 We observe that performance of the TMAF and the ReLU networks are similar. Such behavior clearly should be expected for a simple dataset such as MNIST.
For the CIFAR-10 0 dataset, use the ResNet18 network structure provided by (He et al. 201 5a The activation functions are still ReLU and the diagonal TMAF used in Subsection 3. Numerical
results are presented in Figures 8a 8b and Table Those parameters given in (Paszke et al. 2019)
are already tuned well with respect to ReLU, Nevertheless, TMAF still produces smaller errors in the training process and returns better classification results in the evaluation stage, see Table
For the CIFAR- 100 dataset, we use the ResNet34 network structure provided by (He et al. 2013a) with the ReLU and TMAF activation functions in Subsection 3.1 Numerical results presented in Figures 9a and 9b In the training process, TMAF again outperforms the classical ReLU network.
is possible to improve the performance of TMAF applied to those benchmark datasets. The key point is to select suitable intervals in ae optimize the performance. A simple strategy is to let those intervals in equation varying and adjusted in the training (or the evaluation) process, which will be investigated in our future research.
NCL
Para- Frt.
08
-05
TMA NEL
PXL
spoch
Agach
(a) Training loss
(b) Classification accuracy
Figure 7: MNIST:
Two hidden layers
REFERENCES
Adaptive activation functions accelerate convergence in deep and physics-informed neural net- works. Journal of Computational Physics, 404:109136, 2020. ISSN 0021-9991. doi: https://doi.

---

Under review as a conference paper at ICLR 2022
Evaluation Accuracy
Dataset
ReLU
TMAF
Para-ReLU
MNIST (2 hidden layers)
91.8%
92.2%
91.5%
CIFAR-B 10 (Resnet18)
77.5%
80.2%
78.1%
Table Evaluation accuracy for CIFAR and MNIST
TMA PNLL
TMAE Rell
PS POLJ
PD03 M0.3)
677
ariuary
20-
-25-
20
-25
62
aports
oport
(a) Training loss
(b) Classification accuracy
Figure 8: Comparison among ReLU, Para-ReLU and TMAF for CIFAR
THAP HELL PuLL
TMAP REUL Paraftu
69
10
40
50
10
20
53
spech
spach
(a) Training loss
(b) Classification accuracy
Figure Comparison among ReLU, Para-ReLU and TMAF for CIFAR-100
1t//:00..016//.jppp0010000000000 URL http:pppe///ee.ooooaaaceeeeetrta ariicle/s1//00221991198888
Pakshal Bohra, Joaquim Campos, Harshit Gupta, Shayan Aziznejad, and Michael Unser. Learning activation functions in deep (spline) neural networks. IEEE Open Journal of Signal Processing, 1:295-309, 2020. doi: 10.1 10.110//01PP2200000399977
Nicolas Boullé, Yuji Nakatsukasa, and Alex Townsend. Rational neural networks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. BoLLhttps://proceeer neurin........ss... cc/papprr/222//aah. htw.://sstddb-bbtttctt........
Ajmrk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Yoshua Bengio and Yann LeCun (eds.), 4th Inter-
8

---

Under review as a conference paper at ICLR 2022
national Conference Learning Representations, ICLR 2016. San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Laudendale, USA, April 11-13, 2011, volume 15 of JMLR Proceedings, pp. 315-323, oog, 2011. URL http://procceddnngs m press htt.a//dww....taap...//eeeee//ad
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint, arXiv: 1512.03385, 2015a. URL 1512.03383
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint, arXiv: 1502.01852, 2015b. URL
01852
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint, arXiv: 1606.08415, 2016. htt.tttpp//axxi..rggg abs/
1606. 08415
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http:
7/ //arxiv. org. abaaaass1l11 1412 6980
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self- normalizing neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Gamett (eds.). Ad- vances Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 971-980, 2017. URL http:///pppppaaoo...oe.e..e/e/rp
neuriss.euipp... Abst html
Lu Lu, Yeonjong Shin, Yanhui Su, and George E. Karniadakis. Dying relu and initialization: Theory and numerical examples. arXiv preprint, arXiv: 1903.06733 2019. URL htttp://arxi)
org/abs/ /103 903. 06733
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning flexible activation functions in deep networks. In International Conference on Learn- ing Representations, 2020. URL httltttss////pereeeeee
net hwtp::/wwweeettttteeee www.wooww......ooo.......... Be1.ddd--BBBHDDDDD
Andrei Nicolae. PLU: the piecewise linear unit activation function. 1809.09534, 2018. http://p//aaal...or///////////00
arXiv preprint, arXiv:
Daniel W. Otter, Julian R. Medina, and Jugal K. Kalita. A survey of the usages of deep learning in natural language processing. arXiv preprint, arXiv: 10854, 2018. URL URRp:ttp://arxi
org/abs/1800 10854
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran As- sociates, Inc., 2019. URL htt.//tttps://ppocedinggs rewriiinnnnseeuusss..e.......sss
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. CoRR, abs/1710.05941,, 2017. URL http:::pppa....orr/pp/t/p 1191bb//11111111011111

---

Under review as a conference paper at ICLR 2022
Athamasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Computational Intelligence and Neuro-
science, 2018:7068349, February 2018. ISSN 1687-5265. URL https: http::/doo...ggg.oo
1155/2018/7068344
Yucong Zhou, Zezhou Zhu, and Zhao Zhong. Learning specialized activation functions with the piecewise linear unit. CoRR, abs/2104.03693, 2021. URL pttps://arxxv.orr//aa3.. 2104. 03693
10


Reviews:
[{"review_id": "o5iUN_tTvrW", "reviewer": "Reviewer_mR17", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very poorly written and as is is barely readable. The main ideas are not outlined with sufficient rigor and clarity. It is not clear what are the optimization parameters and how they are optimized. The empirical results are also not well described.\n\nI don't understand the motivation in the beginning of Section 2: \n- application of activation $\\sigma$ returns a vector, while D returns a matrix. How can this be?\n- $\\eta_l$ is defined after it is used in the beginning of section 2.\n- is $m$ a hyperparameter? How is it selected?\n\nFrom what I understand the proposition in eq 6, this is just a series of multipliers to the linear activation of form $tu$, where $u$ is the pre-activated value and $t$ is the current slope. If this is true, then the final activation function won't even be continuous, right?\n\nI also don't understand numerical results: how the params for $\\alpha$ in Section 3.1 are selected? Are those trained? What about results in section 3.2? Why there are 100 intervals? Where does 2.01 came from? Generally, what is being trained? Are the results trained with eq 6 or eq 7? What are the final $t$ and $s$ values? Intervals presented define only the break points of $s$ values, what are the $t$ value?\n\nSome other comments:\n- Unfortunate re-use of the same variable that can be easily avoided: $f$ as a function and $f_n$ as the output.\n- also, t in eq 1 and t in eq 6 mean different things.\n- I would make the mathematical definition more rigor, e.g. a diagonal matrix-valued function mapping is not proper mathematical definition.\n- entries from the discrete set {0, 1} = binary matrix\n- the use of the compositional $\\circ$ operator is very confusing and not-standard. It is quite hard to understand what operator is being applied and what is the resulting matrix dimension. \n", "summary_of_the_paper": "The paper introduces a trainable matrix of activation functions. The authors propose to replace activation with a custom learnable piecewise linear function. The results are evaluated on a custom sine function with different oscillations as well as CIFAR-10 and CIFAR-100.  \n", "summary_of_the_review": "The paper is very poorly written and requires a significant improvements before it is ready for submission. As is, the paper is very far from begin ready even for a proper reviewer's evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "na-ucYH2Fn2", "reviewer": "Reviewer_iGq4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe idea seems to be new.\n\n\nWeakness:\n\n1. The intuition why the proposed method is better is not provided. There are many ways to generalize the existing activation functions, but why would TMAF work?\n\n2. Lack of thorough compassion to support the effectiveness of TMAF. There have been a lot of widely used activation functions, as mentioned in the introduction section: PLU, Leaky ReLU, PLU, Softplus, ELU, SELU, GELU etc. However, the authors did not compare TMAF with most of them, making the effectiveness of TMAF questionable. \n\n3. The task of sin/cos function approximation is quite simple compared to fitting the highly non-convex neural networks. So section 3.1 and 3.2 might not be very informative.\n\n4. The experiments on MNIST/CIFAR is based on very lousy baselines — The 2-hidden-layer network or ResNet18 is by no means the state-of-the-art methods, not even close. So using them in the experiments is not convincing. It would be better to have TMAF plugged in the latest networks, such as EfficientNet and Vision Transformers to test its efficacy. \n\n5. The CIFAR-100 result is missing in Table 3, making the paper look incomplete and probably written in a hurry.", "summary_of_the_paper": "This paper proposes a new type of activation function, called Trainable Matrix Activation Functions (TMAF), to replace the existing activation functions in neural networks, such as ReLU. TMAF is realized using matrix-vector multiplications, where entries of the matrices are trainable piecewise constants. Empirical studies show that TMAF can approximate the sin/cos type of functions better than ReLU/PReLU in terms of approximation error. TMAF also outperforms ReLU in MNIST/CIFAR classifications tasks.", "summary_of_the_review": "All in all, the paper needs a lot more thorough experiments to justify the usefulness of the proposed activation functions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VmolVyEbaIi", "reviewer": "Reviewer_MSbu", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern with this paper is that there is already very similar work on this area of research, namely [1].\nIn [1], we find a learnable piecewise function which leads to the activation function PWLU. If we were to consider a different PWLU for every neuron in the network, we would have basically the same approach as the one introduced here.\n\nThe authors do not mention [1] as related work and of course, the novelty of the contribution is significantly constrained.\n\nOn the experimental section, there is no comparison to other learnable activation functions such as PAU, F-RELU or APL, PWLU, making it difficult for the reader to stack TMAF against the state of the art. \nMoreover, the number of architectures and datasets evaluated is relatively small in comparison to other papers in the same area of research. \nFinally, it seems like the paper only does one run for each of the experiments which might be problematic to estimate the actual performance of the activation function.\n\nThere are other minor improvements that could be done to the paper on the presentation front. It would be helpful for the reader to see the plots of the synthetic experiments overlapped with the ones produced by the networks. Although that space might be better used for a more comprehensive empirical evaluation.\n\n[1] Zhou, Y., Zhu, Z., & Zhong, Z. (2021). Learning specialized activation functions with the Piecewise Linear Unit. arXiv preprint arXiv:2104.03693.\n", "summary_of_the_paper": "The authors propose a new approach to activation functions in DNNs. More precisely, the authors present the trainable matrix activation function (TMAF), which is an activation realized by a matrix-vector multiplication whose entries are trainable.\nThe authors then present a diagonal and tri-diagonal operators. Here, these matrices represent a coefficient that multiplies the output of the layer of the neural network and thus can represent any general piecewise linear activation including relu and leaky relu. \nThese matrices are trainable and therefore add a small number of extra parameters to the network depending on the architecture of the layers. \nFinally the authors demonstrate the benefits of TMAF in synthetic and real world datasets on a couple of architectures.", "summary_of_the_review": "In short, there is very similar work in this area of research which was not mentioned by the authors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Uzgh904Szyw", "reviewer": "Reviewer_MkDq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has several major concerns that are listed below.\n\nNOVELTY: the paper does not provide a review of related works and fails to mention connections to existing trainable (or untrainable) AFs. The form s(x)x is also the base of the Swish AF (where s(x) is the sigmoid) or its variants. Many AFs are built by piecewise linear or constant parts (s-shaped ReLU, adaptive piecewise linear unit, ...) (Apicella et al., 2021). The idea here is to combine Swish with a piecewise constant s(x), but the way it is described in the paper is convoluted (where the original AF is replaced with D(s(x))x, where D is a diagonal matrix).\n\nThe tri-diagonal AF is possibly novel but it is not tested properly, and it also fails to reference related works (e.g., the Maxout, or the bi-dimensional variants of kernel AFs).\n\nCOMPUTATIONAL COST: the paper says \"it is observed that the computational time of D` and T` is comparable to the classical ReLU\", but evaluting (6) or even worse (7) is clearly more expensive than a simple max(0, x). Concrete results on the overhead should be given to justify this sentence.\n\nEXPERIMENTS: more realistic experiments should be provided, by comparing a larger set of trainable/fixed baselines (Swish, SELU, ...), letting the networks train to convergence (see, e.g., Fig. 7), provide standard deviation of the results, etc.", "summary_of_the_paper": "The paper describes an approach to train activation functions. The main idea is to rewrite ReLU as s(x)x (where s(x) is either 0 or 1 depending on the value of x), and then generalize this with a piecewise constant s(x). The paper also proposes a generalization where values at different neurons can be combined. Finally, this is evaluated on two artificial problems and two standard computer vision benchmarks.", "summary_of_the_review": "Some parts of the paper are novel, but the current exposition (and experimental results) are too shallow to warrant publication, which is why I am suggesting a rejection of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "SmLQhJMNpwo", "reviewer": "Reviewer_FB5F", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have several concerns with this paper, which I did not find well-written and significant enough for the following reasons.\n\n- First, the introduction does not cover the literature on adaptive activation functions and lack several references. Hence the authors mention a few classical activation functions but omit to reference the adaptive ones proposed in the past few years such as swish (Ramachandran et al., 2017), rational functions (Molina et al., 2019; Boulle et al., 2020), splines (Bohra et al., 2020), adaptive standard activation functions for physics informed neural networks (Jagtap and Karniadakis, 2019), ...\nMoreover, the authors write that it's hard to determine the optimal activation function for a specific application, while I agree about that, I want to point out the recent works about searching for activation functions as well as the whole literature on the approximation theory of neural networks that can provide some informations regarding the approximation power of neural networks with a specific activation functions. As an example, see the paper by Yarotsky (2017) giving bounds on the size of ReLU networks needed to approximate smooth functions. Papers proposing new activation functions usually try to provide theoretical justification or at least discuss some of these aspects but this is not the case here.\n\n- The section 2, introducing the trainable matrix activation function is not written clearly. As an example, I do not understand the relation $\\sigma \\circ \\eta(x)=D_l(\\eta(x))\\eta(x)$ if $D_l$ is a general diagonal function. I do not understand why we would need a matrix-vector multiplication (especially in the case where the matrix is diagonal), but also for non-diagonal matrix. There are already matrix-vector product before and after each activation functions so the units are already ``mixed'' in the network. Therefore, we can still assume that we need to apply an activation function on each individual unit. Then, to my understanding, the approach proposed by the authors is essentially equivalent to using trainable piecewise constant function with several breakpoints. This is basically a sort of disconnected splines so I do not see the main novelty with respect to the existing literature on this topic. Regardless of the novelty, one of my main concern about this approach is the resulting increase in the number of parameters in the neural networks.\n\n- Later in this section, the authors consider nonlinear activation operators by using tri-diagonal matrices with trainable coefficients. I believe that there is a connection with the recent works trying to learn solution operators of partial differential equations, Green's functions, integral kernels,... (arXiv:2010.08895; arXiv:2003.03485; arXiv:2101.07206; arXiv:2105.00266; Lu et al, DeepONet, 2021). There should be a discussion about this in this section.\n\n- The authors mention that the computational time is comparable to ReLU but do not report any timings to support this claim. Moreover, for the small network size that the authors employ in section 3, the increase of the network parameters will likely impact the training time.\n\n- I do not find the experiments in section 3 significant enough to support the claims of the authors about the gain of the matrix activation functions over ReLU. In sections 3.1-3.2, the authors aim to approximation smooth functions and compare with ReLU. However, they use very shallow neural networks (1 or 2 hidden layers) and so the difference in accuracy is likely due to the increase in terms of trainable parameters. Moreover, it is known that deep neural networks have higher approximation power than shallow ones so I'm wondering why the authors did not test their approach on larger networks. The authors report training errors in their experiments but do not employ a testing set. In Fig.3(c), the constant training loss for ReLU seems to be due to the size of the network employed. The authors comment this by saying that \"increasing the number of intervals in TMAF only lead to a few more training parameters\", I disagree with this since the increase would grow as number of layers x number of neurons per layer so is very significant for the network size used. It is expected that a one/two hidden layer ReLU network won't be appropriate to approximate a sinusoid function as the neural network will basically be a piecewise linear function.\n\n- Similarly, the experiments on CIFAR-10 and CIFAR-100 are not really conclusive as the authors employ a residual network instead of standard convolution networks. Moreover, they don't do any comparison with other (adaptive) activation functions. I also do not think that the reported accuracy is closed to the state-of-the-art. I encourage the authors to read the section 4.3 of (arXiv:1907.06732), which performs comparisons between several activation functions with different architectures on the CIFAR-10 dataset, The reported performances are all higher than 90%, which is way higher than the ones reported by the authors in Table 3. \n\nMinor comments:\n- in the abstract: optimizes weights, they are other typos in the manuscript\n- no conclusions to summarize findings\n- there is no code provided for reproducibility and assessment", "summary_of_the_paper": "The authors introduce trainable matrix activation functions, consisting of trainable matrices at the activation layers to generalize ReLU, in order to increase the approximation performance of deep neural networks. Then, they evaluate their networks on approximation of functions and image classification examples, including the standard CIFAR-10 and CIFAR-100 datasets.", "summary_of_the_review": "I find that the paper is not really well-written as it lacks references and the main contribution is not introduced clearly. Moreover, I think that the contributions are not significant enough and not well supported by experiments for ICLR. Hence, I recommend rejection of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new kind of activation function called matrix activation function that can be learnt jointly with the weights and biases. The paper got 2 strong rejects and 3 rejects. The major challenges include unclear motivation, limited novelty, incomplete related work, weak experiments, and poor paper writing. The author rebuttals did not convince the reviewers. The AC also read through the paper and agreed that the paper is below the bar of ICLR. In particular, the authors neglected a large literature of learning activation functions in the original version, 

(two more examples:

[*] Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, Shuicheng Yan: Deep Learning with S-Shaped Rectified Linear Activation Units. AAAI 2016: 1737-1743.

[#] Yan Yang, Jian Sun, Huibin Li, Zongben Xu: ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI. NIPS 2017.
)

making them unable to compare with existing learnable activation functions thoroughly in the revised version in order to justify the necessity of using matrix activation functions. So the AC recommended rejection.
Decision: Reject