Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new type of activation function, called Trainable Matrix Activation Functions (TMAF), to replace the existing activation functions in neural networks, such as ReLU. TMAF is realized using matrix-vector multiplications, where entries of the matrices are trainable piecewise constants. Empirical studies show that TMAF can approximate the sin/cos type of functions better than ReLU/PReLU in terms of approximation error. TMAF also outperforms ReLU in MNIST/CIFAR classifications tasks.
main_review: Strength:

The idea seems to be new.


Weakness:

1. The intuition why the proposed method is better is not provided. There are many ways to generalize the existing activation functions, but why would TMAF work?

2. Lack of thorough compassion to support the effectiveness of TMAF. There have been a lot of widely used activation functions, as mentioned in the introduction section: PLU, Leaky ReLU, PLU, Softplus, ELU, SELU, GELU etc. However, the authors did not compare TMAF with most of them, making the effectiveness of TMAF questionable. 

3. The task of sin/cos function approximation is quite simple compared to fitting the highly non-convex neural networks. So section 3.1 and 3.2 might not be very informative.

4. The experiments on MNIST/CIFAR is based on very lousy baselines — The 2-hidden-layer network or ResNet18 is by no means the state-of-the-art methods, not even close. So using them in the experiments is not convincing. It would be better to have TMAF plugged in the latest networks, such as EfficientNet and Vision Transformers to test its efficacy. 

5. The CIFAR-100 result is missing in Table 3, making the paper look incomplete and probably written in a hurry.
summary_of_the_review: All in all, the paper needs a lot more thorough experiments to justify the usefulness of the proposed activation functions.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors propose a new approach to activation functions in DNNs. More precisely, the authors present the trainable matrix activation function (TMAF), which is an activation realized by a matrix-vector multiplication whose entries are trainable.
The authors then present a diagonal and tri-diagonal operators. Here, these matrices represent a coefficient that multiplies the output of the layer of the neural network and thus can represent any general piecewise linear activation including relu and leaky relu. 
These matrices are trainable and therefore add a small number of extra parameters to the network depending on the architecture of the layers. 
Finally the authors demonstrate the benefits of TMAF in synthetic and real world datasets on a couple of architectures.
main_review: My main concern with this paper is that there is already very similar work on this area of research, namely [1].
In [1], we find a learnable piecewise function which leads to the activation function PWLU. If we were to consider a different PWLU for every neuron in the network, we would have basically the same approach as the one introduced here.

The authors do not mention [1] as related work and of course, the novelty of the contribution is significantly constrained.

On the experimental section, there is no comparison to other learnable activation functions such as PAU, F-RELU or APL, PWLU, making it difficult for the reader to stack TMAF against the state of the art. 
Moreover, the number of architectures and datasets evaluated is relatively small in comparison to other papers in the same area of research. 
Finally, it seems like the paper only does one run for each of the experiments which might be problematic to estimate the actual performance of the activation function.

There are other minor improvements that could be done to the paper on the presentation front. It would be helpful for the reader to see the plots of the synthetic experiments overlapped with the ones produced by the networks. Although that space might be better used for a more comprehensive empirical evaluation.

[1] Zhou, Y., Zhu, Z., & Zhong, Z. (2021). Learning specialized activation functions with the Piecewise Linear Unit. arXiv preprint arXiv:2104.03693.

summary_of_the_review: In short, there is very similar work in this area of research which was not mentioned by the authors.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a trainable matrix of activation functions. The authors propose to replace activation with a custom learnable piecewise linear function. The results are evaluated on a custom sine function with different oscillations as well as CIFAR-10 and CIFAR-100.  

main_review: The paper is very poorly written and as is is barely readable. The main ideas are not outlined with sufficient rigor and clarity. It is not clear what are the optimization parameters and how they are optimized. The empirical results are also not well described.

I don't understand the motivation in the beginning of Section 2: 
- application of activation $\sigma$ returns a vector, while D returns a matrix. How can this be?
- $\eta_l$ is defined after it is used in the beginning of section 2.
- is $m$ a hyperparameter? How is it selected?

From what I understand the proposition in eq 6, this is just a series of multipliers to the linear activation of form $tu$, where $u$ is the pre-activated value and $t$ is the current slope. If this is true, then the final activation function won't even be continuous, right?

I also don't understand numerical results: how the params for $\alpha$ in Section 3.1 are selected? Are those trained? What about results in section 3.2? Why there are 100 intervals? Where does 2.01 came from? Generally, what is being trained? Are the results trained with eq 6 or eq 7? What are the final $t$ and $s$ values? Intervals presented define only the break points of $s$ values, what are the $t$ value?

Some other comments:
- Unfortunate re-use of the same variable that can be easily avoided: $f$ as a function and $f_n$ as the output.
- also, t in eq 1 and t in eq 6 mean different things.
- I would make the mathematical definition more rigor, e.g. a diagonal matrix-valued function mapping is not proper mathematical definition.
- entries from the discrete set {0, 1} = binary matrix
- the use of the compositional $\circ$ operator is very confusing and not-standard. It is quite hard to understand what operator is being applied and what is the resulting matrix dimension. 

summary_of_the_review: The paper is very poorly written and requires a significant improvements before it is ready for submission. As is, the paper is very far from begin ready even for a proper reviewer's evaluation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes an approach to train activation functions. The main idea is to rewrite ReLU as s(x)x (where s(x) is either 0 or 1 depending on the value of x), and then generalize this with a piecewise constant s(x). The paper also proposes a generalization where values at different neurons can be combined. Finally, this is evaluated on two artificial problems and two standard computer vision benchmarks.
main_review: The paper has several major concerns that are listed below.

NOVELTY: the paper does not provide a review of related works and fails to mention connections to existing trainable (or untrainable) AFs. The form s(x)x is also the base of the Swish AF (where s(x) is the sigmoid) or its variants. Many AFs are built by piecewise linear or constant parts (s-shaped ReLU, adaptive piecewise linear unit, ...) (Apicella et al., 2021). The idea here is to combine Swish with a piecewise constant s(x), but the way it is described in the paper is convoluted (where the original AF is replaced with D(s(x))x, where D is a diagonal matrix).

The tri-diagonal AF is possibly novel but it is not tested properly, and it also fails to reference related works (e.g., the Maxout, or the bi-dimensional variants of kernel AFs).

COMPUTATIONAL COST: the paper says "it is observed that the computational time of D` and T` is comparable to the classical ReLU", but evaluting (6) or even worse (7) is clearly more expensive than a simple max(0, x). Concrete results on the overhead should be given to justify this sentence.

EXPERIMENTS: more realistic experiments should be provided, by comparing a larger set of trainable/fixed baselines (Swish, SELU, ...), letting the networks train to convergence (see, e.g., Fig. 7), provide standard deviation of the results, etc.
summary_of_the_review: Some parts of the paper are novel, but the current exposition (and experimental results) are too shallow to warrant publication, which is why I am suggesting a rejection of the paper.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The authors introduce trainable matrix activation functions, consisting of trainable matrices at the activation layers to generalize ReLU, in order to increase the approximation performance of deep neural networks. Then, they evaluate their networks on approximation of functions and image classification examples, including the standard CIFAR-10 and CIFAR-100 datasets.
main_review: I have several concerns with this paper, which I did not find well-written and significant enough for the following reasons.

- First, the introduction does not cover the literature on adaptive activation functions and lack several references. Hence the authors mention a few classical activation functions but omit to reference the adaptive ones proposed in the past few years such as swish (Ramachandran et al., 2017), rational functions (Molina et al., 2019; Boulle et al., 2020), splines (Bohra et al., 2020), adaptive standard activation functions for physics informed neural networks (Jagtap and Karniadakis, 2019), ...
Moreover, the authors write that it's hard to determine the optimal activation function for a specific application, while I agree about that, I want to point out the recent works about searching for activation functions as well as the whole literature on the approximation theory of neural networks that can provide some informations regarding the approximation power of neural networks with a specific activation functions. As an example, see the paper by Yarotsky (2017) giving bounds on the size of ReLU networks needed to approximate smooth functions. Papers proposing new activation functions usually try to provide theoretical justification or at least discuss some of these aspects but this is not the case here.

- The section 2, introducing the trainable matrix activation function is not written clearly. As an example, I do not understand the relation $\sigma \circ \eta(x)=D_l(\eta(x))\eta(x)$ if $D_l$ is a general diagonal function. I do not understand why we would need a matrix-vector multiplication (especially in the case where the matrix is diagonal), but also for non-diagonal matrix. There are already matrix-vector product before and after each activation functions so the units are already ``mixed'' in the network. Therefore, we can still assume that we need to apply an activation function on each individual unit. Then, to my understanding, the approach proposed by the authors is essentially equivalent to using trainable piecewise constant function with several breakpoints. This is basically a sort of disconnected splines so I do not see the main novelty with respect to the existing literature on this topic. Regardless of the novelty, one of my main concern about this approach is the resulting increase in the number of parameters in the neural networks.

- Later in this section, the authors consider nonlinear activation operators by using tri-diagonal matrices with trainable coefficients. I believe that there is a connection with the recent works trying to learn solution operators of partial differential equations, Green's functions, integral kernels,... (arXiv:2010.08895; arXiv:2003.03485; arXiv:2101.07206; arXiv:2105.00266; Lu et al, DeepONet, 2021). There should be a discussion about this in this section.

- The authors mention that the computational time is comparable to ReLU but do not report any timings to support this claim. Moreover, for the small network size that the authors employ in section 3, the increase of the network parameters will likely impact the training time.

- I do not find the experiments in section 3 significant enough to support the claims of the authors about the gain of the matrix activation functions over ReLU. In sections 3.1-3.2, the authors aim to approximation smooth functions and compare with ReLU. However, they use very shallow neural networks (1 or 2 hidden layers) and so the difference in accuracy is likely due to the increase in terms of trainable parameters. Moreover, it is known that deep neural networks have higher approximation power than shallow ones so I'm wondering why the authors did not test their approach on larger networks. The authors report training errors in their experiments but do not employ a testing set. In Fig.3(c), the constant training loss for ReLU seems to be due to the size of the network employed. The authors comment this by saying that "increasing the number of intervals in TMAF only lead to a few more training parameters", I disagree with this since the increase would grow as number of layers x number of neurons per layer so is very significant for the network size used. It is expected that a one/two hidden layer ReLU network won't be appropriate to approximate a sinusoid function as the neural network will basically be a piecewise linear function.

- Similarly, the experiments on CIFAR-10 and CIFAR-100 are not really conclusive as the authors employ a residual network instead of standard convolution networks. Moreover, they don't do any comparison with other (adaptive) activation functions. I also do not think that the reported accuracy is closed to the state-of-the-art. I encourage the authors to read the section 4.3 of (arXiv:1907.06732), which performs comparisons between several activation functions with different architectures on the CIFAR-10 dataset, The reported performances are all higher than 90%, which is way higher than the ones reported by the authors in Table 3. 

Minor comments:
- in the abstract: optimizes weights, they are other typos in the manuscript
- no conclusions to summarize findings
- there is no code provided for reproducibility and assessment
summary_of_the_review: I find that the paper is not really well-written as it lacks references and the main contribution is not introduced clearly. Moreover, I think that the contributions are not significant enough and not well supported by experiments for ICLR. Hence, I recommend rejection of the paper.
review_title: null

Meta-review: The paper proposed a new kind of activation function called matrix activation function that can be learnt jointly with the weights and biases. The paper got 2 strong rejects and 3 rejects. The major challenges include unclear motivation, limited novelty, incomplete related work, weak experiments, and poor paper writing. The author rebuttals did not convince the reviewers. The AC also read through the paper and agreed that the paper is below the bar of ICLR. In particular, the authors neglected a large literature of learning activation functions in the original version, 

(two more examples:

[*] Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, Shuicheng Yan: Deep Learning with S-Shaped Rectified Linear Activation Units. AAAI 2016: 1737-1743.

[#] Yan Yang, Jian Sun, Huibin Li, Zongben Xu: ADMM-Net: A Deep Learning Approach for Compressive Sensing MRI. NIPS 2017.
)

making them unable to compare with existing learnable activation functions thoroughly in the revised version in order to justify the necessity of using matrix activation functions. So the AC recommended rejection.
Decision: Reject