Intelligent Matrix Exponentiation

1 INTRODUCTION . Deep neural networks ( DNNs ) synthesize highly complex functions by composing a large number of neuronal units , each featuring a basic and usually 1-dimensional nonlinear activation function f : R1 → R1 . While highly successful in practice , this approach also has disadvantages . In a conventional DNN , any two activations only ever get combined through summation . This means that such a network requires an increasing number of parameters to approximate more complex functions even as simple as multiplication . Parameter-wise , this approach of composing simple functions does not scale efficiently . An alternative to the composition of many 1-dimensional functions is using a simple higherdimensional nonlinear function f : Rm → Rn . A single multidimensional nonlinearity may be desirable because it could express more complex relationships between input features with potentially fewer parameters and fewer mathematical operations . The matrix exponential stands out as a promising but overlooked candidate for a higher-dimensional nonlinearity for machine learning models . The matrix exponential is a smooth function that appears in the solution to one of the simplest differential equations that can yield desirable mathematical properties : d/dty ( t ) = My ( t ) , with the solution y ( t ) = exp ( Mt ) y ( 0 ) , where M is a constant matrix . The matrix exponential plays a prominent role in the theory of Lie groups , an algebraic structure widely used throughout many branches of mathematics and science . A unique advantage of the matrix exponential is its natural ability to represent oscillations and exponential decay , which becomes apparent if we decompose the matrix to be exponentiated into a symmetric and an antisymmetric component . The exponential of an antisymmetric matrix , whose nonzero eigenvalues are always imaginary , generates a superposition of periodic oscillations , whereas the exponential of a symmetric matrix , which has real eigenvalues , expresses an exponential growth or decay . This fact , especially the ability to represent periodic functions , gives the M-layer the possibility to extrapolate beyond its training domain . Many real-world phenomena contain some degree of periodicity and can therefore benefit from this feature . In contrast , the functions typically used in conventional DNNs approximate the target function locally and are therefore unable to scale well outside the boundaries of the training data for such problems . Based on these insights , we propose a novel architecture for supervised learning whose core element is a single layer ( henceforth referred to as “ M-layer ” ) , that computes a single matrix exponential , where the matrix to be exponentiated is an affine function of the input features . We show that the M-layer has universal approximator properties and allows closed-form per-example bounds for robustness . We demonstrate the ability of this architecture to learn multivariate polynomials , such as matrix determinants , and to generalize periodic functions beyond the domain of the input without any feature engineering . Furthermore , the M-layer achieves results comparable to recently-proposed non-specialized architectures on image recognition datasets . We provide TensorFlow code that implements the M-layer in the Supplementary Material . 2 RELATED WORK . Neuronal units with more complex activation functions have been proposed . One such example are sigma-pi units ( Rumelhart et al. , 1986 ) , whose activation function is the weighted sum of products of its inputs . More recently , neural arithmetic logic units have been introduced ( Trask et al. , 2018 ) , which can combine inputs using multiple arithmetic operators and generalize outside the domain of the training data . In contrast with these architectures , the M-layer is not based on neuronal units with multiple inputs , but uses a single matrix exponential as its nonlinear mapping function . Through the matrix exponential , the M-layer can easily learn mathematical operations more complex than addition , but with simpler architecture . In fact , as shown in Section 3.3 , the M-layer can be regarded as a generalized sigma-pi network with built-in architecture search , in the sense that it learns by itself which arithmetic graph should be used for the computation . Architectures with higher-dimensional nonlinearities are also already used . The softmax function is an example of a widely-used such nonlinear activation function that , like the M-layer , has extra mathematical structure . For example , a permutation of the softmax inputs produces a corresponding permutation of the outputs . Maxout networks also act on multiple units and have been successful in combination with dropout ( Goodfellow et al. , 2013 ) . In radial basis networks ( Park & Sandberg , 1991 ) , each hidden unit computes a nonlinear function of the distance between its own learned centroid and a single point represented by a vector of input coordinates . Capsule networks ( Sabour et al. , 2017 ) are another recent example of multidimensional nonlinearities . Similarly , the M-layer uses the matrix exponential as a single high-dimensional nonlinearity . This mapping satisfies nontrivial mathematical identities , for some of which we here elucidate their application potential in a machine learning context . Feature crosses of second and higher order have also been explored on top of convolutional layers ( ( Lin et al. , 2015 ; Lin & Maji , 2017 ; Li et al. , 2017 ; Engin et al. , 2018 ; Koniusz et al. , 2018 ) . In this line of research , ( Li et al. , 2017 ) introduces the use of a matrix operation ( either matrix logarithm or matrix square root ) on top of a convolutional layer with higher order feature crosses . The M-Layer differs from this kind of approach in two ways : first , it uses matrix exponentiation to produce feature crosses , and not to improve trainability on top of other ones ; secondly , it computes a matrix operation of an arbitrary matrix ( not necessarily symmetric positive semidefinite ) , which allows more expressibility , as seen for example in Section 3.3 . The main differences between the M-layer and other architectures that can utilize feature-crosses are the M-layer ’ s ability to also model non-polynomial dependencies ( such as a cosine ) and the builtin competition-for-total-complexity regularization , which we will explain in Section 3.3 through a “ circuit breadboard ” analogy . Matrix exponentiation has a natural alternative interpretation in terms of an ordinary differential equation ( ODE ) . As such , the M-layer can be compared to other novel ODE-related architectures , such as neural ordinary differential equations ( NODE ) ( Chen et al. , 2018 ) and their augmented extensions ( ANODE ) ( Dupont et al. , 2019 ) . We discuss this in Section 3.6 . Existing approaches to certifying the robustness of neural networks can be split into two different categories . Some approaches ( Peck et al. , 2017 ) mathematically analyze a network layer by layer , providing bounds on the robustness of each layer , that then get multiplied together . This kind of approach tends to give fairly loose bounds , due to the inherent tightness loss from composing upper bounds . Other approaches ( Singh et al. , 2018 ; 2019 ) use abstract interpretation on the evaluation of the network to provide empirical robustness bounds . In contrast , using the fact that the M-layer architecture has a single layer , in Section 3.7 we obtain a direct bound on the robustness on the whole network by analyzing the explicit formulation of the computation . 3 ARCHITECTURE . We start this section by refreshing the definition of the matrix exponential . We then define the proposed M-layer architecture and explain its ability to learn particular functions such as polynomials and periodic functions . Finally , we provide closed-form per-example robustness guarantees . 3.1 MATRIX EXPONENTIATION . The exponential of a square matrixM is defined as : exp ( M ) = ∑∞ k=0 1 k ! M k. The matrix powerMk is defined inductively as M0 = I , Mk+1 = M ·Mk , using the associativity of the matrix product ; it is not an element-wise matrix operation . Note that the expansion of exp ( M ) in Eq . ( 3.1 ) is finite for nilpotent matrices . A matrix M is called nilpotent if there exists a positive integer k such that Mk = 0 . Strictly upper triangular matrices are a canonical example . Multiple algorithms for computing the matrix exponential efficiently have been proposed ( Moler & Van Loan , 2003 ) . We provide an implementation of the M-layer that employs either the TensorFlow function tf.linalg.expm , which uses the scaling and squaring method combined with the Padé approximation ( Higham , 2005 ) , or a simpler and faster approximation of the matrix exponential that only squares a fixed number of times , using the formula exp ( M ) ≈ ( I+ M 2k ) 2 k . This approximation follows from the equivalent definition of exp ( M ) = limn→∞ ( I + Mn ) n , and has O ( kn1.5 ) time complexity , where n is the total number of elements in M . 3.2 M-LAYER DEFINITION . At the core of the proposed architecture ( Figure 1 ) is an M-layer that computes a single matrix exponential , where the matrix to be exponentiated is an affine function of all of the input features . In other words , an M-layer replaces an entire stack of hidden layers in a DNN . We exemplify the architecture as applied to a standard image recognition dataset , but this formulation is applicable to any other type of problem by adapting the relevant input indices . In the following equations , generalized Einstein summation is performed over all right-hand side indices not seen on the left-hand side . Consider an example input image , encoded as a 3-index array Xyxc , where y , x and c are the row index , column index and color channel index , respectively . The matrix M to be exponentiated is obtained as follows , using the trainable parameters T̃ajk , Ũaxyc and B̃jk : Mjk = B̃jk + T̃ajkŨayxcXyxc ( 1 ) First , X is projected linearly to a d-dimensional latent feature embedding space by Ũayxc . Then , the 3-index tensor T̃ajk maps each such latent feature to an n × n matrix . Finally , a bias matrix B̃jk is added to the feature-weighted sum of matrices . The result is a matrix indexed by row and column indices j and k. In order to allow for exponentiation , M must be a square matrix . It is possible to contract the tensors T̃ and Ũ in order to simplify the architecture formula , but partial tensor factorization provides regularization by reducing the parameter count . An output pm is obtained as follows , using the trainable parameters S̃mjk and Ṽm : pm = Ṽm + S̃mjk exp ( M ) jk ( 2 ) The matrix exp ( M ) , indexed by row and column indices j and k in the same way asM , is projected linearly by the 3-index tensor S̃mjk , to obtain a h-dimensional output vector . The bias-vector Ṽm turns this linear mapping into an affine mapping . The resulting vector may be interpreted as accumulated per-class evidence and may then be mapped to a vector of probabilities via softmax . Training can thus be done conventionally , by minimizing a loss function such as the L2 norm or the cross-entropy with softmax , using backpropagation through matrix exponentiation . In general , there is no simple analytical expression for the derivative of exp ( M ) , but one can instead compute derivatives for each step of the algorithm chosen to compute the exponential . For example , tf.linalg.expm uses the scaling-and-squaring algorithm with a Padé approximation , whose steps are all differentiable . We discuss the topic in more detail in Appendix A.2 . The nonlinearity of the M-layer architecture is provided by the Rd → Rh mapping v 7→ Ṽm + S̃mjk exp ( M ) jk . The count of trainable parameters of this component is dn2 +n2 +n2h+ h. This count comes from summing the dimensions of T̃ajk , B̃jk , S̃mjk , and Ṽm , respectively . We note that this architecture has some redundancy in its parameters , as one can freely multiply the T and U tensors by a d× d real matrix and , respectively , its inverse , while preserving the computed function . Similarly , it is possible to multiply each of the n× n parts of the tensors T̃ and S̃ , as well as B , by both an n × n matrix and its inverse . In other words , any pair of real invertible matrices of sizes d×d and n×n can be used to produce a new parametrization that still computes the same function .

Summary: This paper explores matrix exponentiation as an alternative non-linearity in a neural network. The key idea is to compute an affine transform of the inputs, there by generating an nxn feature map, followed by applying a matrix exponential to this feature map, that is subsequently used for classification. The paper provides several scenarios where matrix exponential could be an interesting non-linearity to use. Experiments are provided on synthetic examples, and standard image recognition benchmarks in a limited setting and show some promise.