Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Title: INTELLIGENT MATRIX EXPONENTIATION

Summary of the paper:

The idea is to use the matrix exponential as a layer in a neural network. This basically requires transforming to a latent variable which has a rank two tensor shape (a matrix) then applying the e.g. "expm" function. A robustness result is given (section 3.7) and a universal approximation result (appendix A) and some experiments on mainly toy but also image classification problems, and a univariate regression on seasonal data.

Pros:

The paper is written in a nice easy to read way. I enjoyed reading it because it is very easy to follow, though to be fair the contribution is rather simple so this should be a given.

The robustness result is something, but it follows from a very basic result on matrix exponentials. Also, the universal approximation result is better than nothing, but also quite straightforward and constructive and simply comes from observing the matrix exponential as an arbitrary polynomial.

Section 3 is nice throughout ; simple and easy to follow yet interesting.

Cons:

I hate to be that reviewer, but the paper should go further to motivate the use of the expm function. Figure 2 is very cool indeed, but then the real world experiments are not highly motivating. The seasonality data is neat, but this can presumably be better handled by a Gaussian process with periodic covariance, anyway (which is not compared with). To motivate using expm would require far more extensive experiments, even if this requires some heavy computational resources. 

Suggestions:

Figure 1 - mention the need for a matrix layout (you seem to want to suggest a 3 by 3 matrix with your diagram).

Equation 1 - is it normal to omit j and k from the l.h.s. in tensor notation ?

Recommendation:

This paper is crying out for a more thorough experimental evaluation ; I am near the fence on this paper but I think it could be much more impactful with such and as such lean toward rejection.
review_title: nice but slightly underwhelming

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes a novel machine learning architecture (or M-layer) that is in the form matrix exponential. This architecture can effectively model the interaction of feature components and learn multivariate polynomial functions and periodic functions. The architecture of the M-layer is well described. Its universal approximation capability is explained and proved in the Appendix. Other properties related to periodicity, connection to Lie groups, the interpretation from the perspective of dynamical systems, and the robustness bounds are discussed. Experimental study is conducted on toy datasets and three image classification datasets to demonstrate the properties of the proposed M-layer. 

Overall, this is a nice piece of work. The proposed M-layer is novel, and the properties shown by this kind of architecture are interesting, especially the capability in modelling periodic functions and extrapolating data. Theoretical discussion is generally clear although some places are a bit hard to follow. Experimental study supports the claims. 

Meanwhile, this work could be further improved at the following points:

1. The experimental study does not combine the M-layer with the convolution layers, and it only tests the M-layer on relatively simple image recognition datasets. It will be interesting to know whether the M-layer can be combined with convolutional layers and trained in an end-to-end manner. This is important to check the potential of the M-layer for the applications of visual, audio, and text data analysis, for which end-to-end training has proven to be more effective. This paper could test this setting on some larger-scale benchmark such as ImageNet if possible. 

2. In the experimental study, it will be interesting to see the comparison between the M-layer and Gaussian RBF kernel based SVM (RBF is mentioned in Related Work) on these toy datasets and image benchmarks. RBF-SVM can be regarded as a neural network with one hidden layer. 

3. This work indicates that the M-layer can better consider the cross-terms of feature components. This could be related to the recent work on second (or higher)-order visual representation, in which the correlations of the channels in a convolutional feature map are extracted and used as feature representation for the subsequent fully connected and softmax layers. This paper can discuss the potential connections with this line of research ([R1]-[R4])

[R1] Tsung-Yu Lin; Aruni RoyChowdhury; Subhransu Maji, Bilinear CNN Models for Fine-Grained Visual Recognition, 2015 IEEE International Conference on Computer Vision (ICCV).
[R2] Peihua Li; Jiangtao Xie; Qilong Wang; Wangmeng Zuo, Is Second-Order Information Helpful for Large-Scale Visual Recognition? 2017 IEEE International Conference on Computer Vision (ICCV)
[R3] Melih Engin, Lei Wang, Luping Zhou, and Xinwang Liu, DeepKSPD: Learning Kernel-Matrix-Based SPD Representation For Fine-Grained Image Recognition, European Conference on Computer Vision ECCV 2018
[R4] Piotr Koniusz; Hongguang Zhang; Fatih Porikli, A Deeper Look at Power Normalizations, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

4. The capability of "extrapolate learning" of this M-layer is very interesting. This work uses the results in Figures 2 and 3 to demonstrate it. In addition to the double spiral data and the periodic data, is this "extrapolate learning" capability applicable to other more general data or patterns? Please comment.  

--- Thank the authors for the detailed response. After reading the response and the comments of peer reviewers, the rating is altered as follows. 
review_title: A nice work with theoretical novelty although experimental study can be further expanded

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary

In this work, the authors propose the M-Layer, a neural network "layer" that consists in affinely transforming the input to a matrix, applying its matrix exponential, and applying another affine transformation. The parameters describing the affine transformations are the learnable parameters. 
The authors provide a "universal approximation result" that shows that when using a sufficiently large latent space, the matrix exponential is able to replicate arbitrary polynomial functions of the input data, as well as periodic functions. 
In numerical experiments on synthetic data (determinant of a matrix, swiss roll), time series data, as well as image data (MNIST, CIFAR10, SVHN), a single M-Layer compares favorably to neural ODEs as well as a very simplistic RELU network, while being outperformed by augmented neural ODE's. 
The authors furthermore provide robustness certificates for the M-layer, and point out its connections to Lie-theory and dynamical systems interpretations of deep learning.

### Decision

On the positive I think that the idea of going beyond component-wise nonlinearities is interesting and the matrix exponential, with its rich mathematical structure, makes for an intriguing candidate. The authors analyse this proposal from many different view including robustness, approximation capability, computational implementation, and empirical results. 

On the negative side, I find the comparison to conventional neural networks not entirely fair. In particular, the claims of improved representational power compared to traditional neural network layers seem exagerated and vague. 
The connections drawn to Lie theory and dynamical systems seem superficial in that they mostly restate properties of the matrix exponential instead of providing interpretations of the M layer. 
Finally, the numerical experiments are comparing a single M-layer to a very shallow dense RELU network. While I appreciate the intention of the authors to keep the experimental setup simple, comparing the performance of individual layers is a very artificial situation that does not seem to give any insight if M-layers are useful as layers for deep learning. 
If they are not, however, this casts doubt on the usefulness and relevance of M-layers in general.

While I do think that the idea of an M-layer is interesting and in particular the relative stability bounds are promising, I feel that the message of the paper is diminished by the unclear comparison to existing deep learning methods. 
This makes it difficult to assess the relevance of the work which is why, for now, I tend to recommend rejection. 


### Detailed comments 
 
#### Conceptual comparison to RELU:

"While highly successful in practice, this approach also has disadvantages. In a conventional DNN, any two activations only ever get combined through summation. This means that such a network requires an increasing number of parameters to approximate more complex functions even as simple as multiplication. This approach of composing simple functions does not generalize well outside the boundaries of the training data." 

I would argue that this is misleading, since the results in later layers of a deep neural network depend on the activations in a highly nonlinear way. Furthermore, I believe there exist universal approximation results even for shallow neural networks with arbitrary width, just like the universal approximation results provided for M-layers are true only in the limit of infinitely large matrix exponentials.  They also do not provide any evidence for their claim that the lack of generalization is linked to the composing of simple functions. The authors do provide some experiments to this end on swiss roll and meteorological datasets. However these examples seem somewhat taylor-made for the exponential map approximation. Furthermore, the RELU network used is not even able to fit the training data well, which suggests this issue might be more due to a lack of model capacity.

#### Lie Algebras and dynamical systems

My understanding is that these sections mostly recapitulate how the matrix exponential appears in lie theory and the solution of linear ODE. However, I don't think they provide a compelling interpretation of the M-layer in terms of either of those points of view. In neural ODE for instance the input data to the layer can be thought of as a state of a dynamical system that is determined by the model weights. 
In the M-layer, instead, the input becomes part of the system matrix, then the propagator of this system matrix (the exponential map) is computed, and finally this propagator is applied to initial conditions given by another set of weights by the M-layer. 
I cannot think of a meaningful interpretation and I don't think the authors have provided one, either.

#### Numerical experiments

Reiterating what I wrote earlier, I really think that the performance of deep neural networks constructed with M-Layers needs to be compared to that of conventional deep neural networks to establish the relevance of the M-layer as a neural network layer. 
In a nutshell: if the differences between M-layers and RELUs were to disappear when working with deeper networks, then why should one still use the M-layer. 

=====================================================================================

After reading the rebuttal and the other reviews I still lean towards rejection, the main reason being that I am not convinced at this point that exponential nonlinearities are a direction worth pursuing.
I furthermore find the passages on dynamical systems and lie groups to be still lacking, for the same reasons as detailed above.
review_title: Interesting approach, but some doubts regarding comparisons

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper explores matrix exponentiation as an alternative non-linearity in a neural network. The key idea is to compute an affine transform of the inputs, there by generating an nxn feature map, followed by applying a matrix exponential to this feature map, that is subsequently used for classification. The paper provides several scenarios where matrix exponential could be an interesting non-linearity to use. Experiments are provided on synthetic examples, and standard image recognition benchmarks in a limited setting and show some promise. 

Pros:
1. The idea of using matrix exponential as a non-linearity is an interesting idea that is perhaps not explored in the context of neural networks so far. 
2. The paper provides several contexts in which the matrix exponential can be a good non-linearity to adopt.
3. Experiments on out-of-sample extrapolation shows promising results.

Cons:
1. I think the key factor missing in the paper is perhaps a lack of a solid motivation for using the matrix exponential. What does such an operator capture intuitively in a neural network that is not possible with prior activations, for example, tanh, elu, etc.?

2. Note that the matrix exponentiation operator has a complexity of O(n^1.5) for a feature matrix of nxn, and demands significant computational expense. For large n, which is what is typically used in deep networks, this operation could be impossible. It is unclear if this operation can be done on a GPU and if so to what approximation accuracy? 

3. There are other matrix operations that have been attempted before; for example, matrix log-maps, which can also offer non-linearities, and have been used in bilinear CNN models for fine-grained recognition tasks. The paper may include these topics in the survey and contrast the technical advantages of using the exponential instead. 

4. The experiments are very limited, and networks that use convolutional layers are shown to significantly outperform the proposed non-linearity. 

5. The paper also lacks details in analyzing the feature maps produced by the non-linearity, or provide any gradient derivations, but relies on standard autograd packages for such. It is also not clear why the paper decided to use a full image as input, and why not use convolutional layers within the pipeline. The paper title is also a bit ambiguous as it is unclear what is "intelligent" regarding matrix exponentiation? 

Overall, this paper explores an interesting non-linearity, however lacks theoretical or empirical rigour in showing that the approach is practically useful. 

---------------------------------
Post-Rebuttal: Thanks to the authors for the detailed response. I think the idea has potential, however (resonating with the comments of other reviewers) I still think the paper should be significantly improved for better motivation, stronger theoretical results, and more elaborate and diverse experiments that can demonstrate the effectiveness of the method. 
review_title: Lacks rigour!

Meta-review: This paper was reviewed by 4 reviewers who scored the paper below acceptance threshold even after the rebuttal. Reviewer 4 is concerned about motivation, Reviewer 2 rightly points out that there exist numerous works that use some form of spectral layers in a deep setting on challenging datasets - something lacking in this work. Reviewer 3 is concerned about limited discussion on lie groups and the overall benefit of expm(.). Reviewer 1 reverberates the same comments regarding insufficient experiments,  comparisons and limited motivation. We encourage authors to consider all pointers given by reviewers in any future re-submission.
Decision: Reject