Intelligent Matrix Exponentiation

1 INTRODUCTION . Deep neural networks ( DNNs ) synthesize highly complex functions by composing a large number of neuronal units , each featuring a basic and usually 1-dimensional nonlinear activation function f : R1 → R1 . While highly successful in practice , this approach also has disadvantages . In a conventional DNN , any two activations only ever get combined through summation . This means that such a network requires an increasing number of parameters to approximate more complex functions even as simple as multiplication . Parameter-wise , this approach of composing simple functions does not scale efficiently . An alternative to the composition of many 1-dimensional functions is using a simple higherdimensional nonlinear function f : Rm → Rn . A single multidimensional nonlinearity may be desirable because it could express more complex relationships between input features with potentially fewer parameters and fewer mathematical operations . The matrix exponential stands out as a promising but overlooked candidate for a higher-dimensional nonlinearity for machine learning models . The matrix exponential is a smooth function that appears in the solution to one of the simplest differential equations that can yield desirable mathematical properties : d/dty ( t ) = My ( t ) , with the solution y ( t ) = exp ( Mt ) y ( 0 ) , where M is a constant matrix . The matrix exponential plays a prominent role in the theory of Lie groups , an algebraic structure widely used throughout many branches of mathematics and science . A unique advantage of the matrix exponential is its natural ability to represent oscillations and exponential decay , which becomes apparent if we decompose the matrix to be exponentiated into a symmetric and an antisymmetric component . The exponential of an antisymmetric matrix , whose nonzero eigenvalues are always imaginary , generates a superposition of periodic oscillations , whereas the exponential of a symmetric matrix , which has real eigenvalues , expresses an exponential growth or decay . This fact , especially the ability to represent periodic functions , gives the M-layer the possibility to extrapolate beyond its training domain . Many real-world phenomena contain some degree of periodicity and can therefore benefit from this feature . In contrast , the functions typically used in conventional DNNs approximate the target function locally and are therefore unable to scale well outside the boundaries of the training data for such problems . Based on these insights , we propose a novel architecture for supervised learning whose core element is a single layer ( henceforth referred to as “ M-layer ” ) , that computes a single matrix exponential , where the matrix to be exponentiated is an affine function of the input features . We show that the M-layer has universal approximator properties and allows closed-form per-example bounds for robustness . We demonstrate the ability of this architecture to learn multivariate polynomials , such as matrix determinants , and to generalize periodic functions beyond the domain of the input without any feature engineering . Furthermore , the M-layer achieves results comparable to recently-proposed non-specialized architectures on image recognition datasets . We provide TensorFlow code that implements the M-layer in the Supplementary Material . 2 RELATED WORK . Neuronal units with more complex activation functions have been proposed . One such example are sigma-pi units ( Rumelhart et al. , 1986 ) , whose activation function is the weighted sum of products of its inputs . More recently , neural arithmetic logic units have been introduced ( Trask et al. , 2018 ) , which can combine inputs using multiple arithmetic operators and generalize outside the domain of the training data . In contrast with these architectures , the M-layer is not based on neuronal units with multiple inputs , but uses a single matrix exponential as its nonlinear mapping function . Through the matrix exponential , the M-layer can easily learn mathematical operations more complex than addition , but with simpler architecture . In fact , as shown in Section 3.3 , the M-layer can be regarded as a generalized sigma-pi network with built-in architecture search , in the sense that it learns by itself which arithmetic graph should be used for the computation . Architectures with higher-dimensional nonlinearities are also already used . The softmax function is an example of a widely-used such nonlinear activation function that , like the M-layer , has extra mathematical structure . For example , a permutation of the softmax inputs produces a corresponding permutation of the outputs . Maxout networks also act on multiple units and have been successful in combination with dropout ( Goodfellow et al. , 2013 ) . In radial basis networks ( Park & Sandberg , 1991 ) , each hidden unit computes a nonlinear function of the distance between its own learned centroid and a single point represented by a vector of input coordinates . Capsule networks ( Sabour et al. , 2017 ) are another recent example of multidimensional nonlinearities . Similarly , the M-layer uses the matrix exponential as a single high-dimensional nonlinearity . This mapping satisfies nontrivial mathematical identities , for some of which we here elucidate their application potential in a machine learning context . Feature crosses of second and higher order have also been explored on top of convolutional layers ( ( Lin et al. , 2015 ; Lin & Maji , 2017 ; Li et al. , 2017 ; Engin et al. , 2018 ; Koniusz et al. , 2018 ) . In this line of research , ( Li et al. , 2017 ) introduces the use of a matrix operation ( either matrix logarithm or matrix square root ) on top of a convolutional layer with higher order feature crosses . The M-Layer differs from this kind of approach in two ways : first , it uses matrix exponentiation to produce feature crosses , and not to improve trainability on top of other ones ; secondly , it computes a matrix operation of an arbitrary matrix ( not necessarily symmetric positive semidefinite ) , which allows more expressibility , as seen for example in Section 3.3 . The main differences between the M-layer and other architectures that can utilize feature-crosses are the M-layer ’ s ability to also model non-polynomial dependencies ( such as a cosine ) and the builtin competition-for-total-complexity regularization , which we will explain in Section 3.3 through a “ circuit breadboard ” analogy . Matrix exponentiation has a natural alternative interpretation in terms of an ordinary differential equation ( ODE ) . As such , the M-layer can be compared to other novel ODE-related architectures , such as neural ordinary differential equations ( NODE ) ( Chen et al. , 2018 ) and their augmented extensions ( ANODE ) ( Dupont et al. , 2019 ) . We discuss this in Section 3.6 . Existing approaches to certifying the robustness of neural networks can be split into two different categories . Some approaches ( Peck et al. , 2017 ) mathematically analyze a network layer by layer , providing bounds on the robustness of each layer , that then get multiplied together . This kind of approach tends to give fairly loose bounds , due to the inherent tightness loss from composing upper bounds . Other approaches ( Singh et al. , 2018 ; 2019 ) use abstract interpretation on the evaluation of the network to provide empirical robustness bounds . In contrast , using the fact that the M-layer architecture has a single layer , in Section 3.7 we obtain a direct bound on the robustness on the whole network by analyzing the explicit formulation of the computation . 3 ARCHITECTURE . We start this section by refreshing the definition of the matrix exponential . We then define the proposed M-layer architecture and explain its ability to learn particular functions such as polynomials and periodic functions . Finally , we provide closed-form per-example robustness guarantees . 3.1 MATRIX EXPONENTIATION . The exponential of a square matrixM is defined as : exp ( M ) = ∑∞ k=0 1 k ! M k. The matrix powerMk is defined inductively as M0 = I , Mk+1 = M ·Mk , using the associativity of the matrix product ; it is not an element-wise matrix operation . Note that the expansion of exp ( M ) in Eq . ( 3.1 ) is finite for nilpotent matrices . A matrix M is called nilpotent if there exists a positive integer k such that Mk = 0 . Strictly upper triangular matrices are a canonical example . Multiple algorithms for computing the matrix exponential efficiently have been proposed ( Moler & Van Loan , 2003 ) . We provide an implementation of the M-layer that employs either the TensorFlow function tf.linalg.expm , which uses the scaling and squaring method combined with the Padé approximation ( Higham , 2005 ) , or a simpler and faster approximation of the matrix exponential that only squares a fixed number of times , using the formula exp ( M ) ≈ ( I+ M 2k ) 2 k . This approximation follows from the equivalent definition of exp ( M ) = limn→∞ ( I + Mn ) n , and has O ( kn1.5 ) time complexity , where n is the total number of elements in M . 3.2 M-LAYER DEFINITION . At the core of the proposed architecture ( Figure 1 ) is an M-layer that computes a single matrix exponential , where the matrix to be exponentiated is an affine function of all of the input features . In other words , an M-layer replaces an entire stack of hidden layers in a DNN . We exemplify the architecture as applied to a standard image recognition dataset , but this formulation is applicable to any other type of problem by adapting the relevant input indices . In the following equations , generalized Einstein summation is performed over all right-hand side indices not seen on the left-hand side . Consider an example input image , encoded as a 3-index array Xyxc , where y , x and c are the row index , column index and color channel index , respectively . The matrix M to be exponentiated is obtained as follows , using the trainable parameters T̃ajk , Ũaxyc and B̃jk : Mjk = B̃jk + T̃ajkŨayxcXyxc ( 1 ) First , X is projected linearly to a d-dimensional latent feature embedding space by Ũayxc . Then , the 3-index tensor T̃ajk maps each such latent feature to an n × n matrix . Finally , a bias matrix B̃jk is added to the feature-weighted sum of matrices . The result is a matrix indexed by row and column indices j and k. In order to allow for exponentiation , M must be a square matrix . It is possible to contract the tensors T̃ and Ũ in order to simplify the architecture formula , but partial tensor factorization provides regularization by reducing the parameter count . An output pm is obtained as follows , using the trainable parameters S̃mjk and Ṽm : pm = Ṽm + S̃mjk exp ( M ) jk ( 2 ) The matrix exp ( M ) , indexed by row and column indices j and k in the same way asM , is projected linearly by the 3-index tensor S̃mjk , to obtain a h-dimensional output vector . The bias-vector Ṽm turns this linear mapping into an affine mapping . The resulting vector may be interpreted as accumulated per-class evidence and may then be mapped to a vector of probabilities via softmax . Training can thus be done conventionally , by minimizing a loss function such as the L2 norm or the cross-entropy with softmax , using backpropagation through matrix exponentiation . In general , there is no simple analytical expression for the derivative of exp ( M ) , but one can instead compute derivatives for each step of the algorithm chosen to compute the exponential . For example , tf.linalg.expm uses the scaling-and-squaring algorithm with a Padé approximation , whose steps are all differentiable . We discuss the topic in more detail in Appendix A.2 . The nonlinearity of the M-layer architecture is provided by the Rd → Rh mapping v 7→ Ṽm + S̃mjk exp ( M ) jk . The count of trainable parameters of this component is dn2 +n2 +n2h+ h. This count comes from summing the dimensions of T̃ajk , B̃jk , S̃mjk , and Ṽm , respectively . We note that this architecture has some redundancy in its parameters , as one can freely multiply the T and U tensors by a d× d real matrix and , respectively , its inverse , while preserving the computed function . Similarly , it is possible to multiply each of the n× n parts of the tensors T̃ and S̃ , as well as B , by both an n × n matrix and its inverse . In other words , any pair of real invertible matrices of sizes d×d and n×n can be used to produce a new parametrization that still computes the same function .

Summary: This work proposes a novel machine learning architecture (or M-layer) that is in the form matrix exponential. This architecture can effectively model the interaction of feature components and learn multivariate polynomial functions and periodic functions. The architecture of the M-layer is well described. Its universal approximation capability is explained and proved in the Appendix. Other properties related to periodicity, connection to Lie groups, the interpretation from the perspective of dynamical systems, and the robustness bounds are discussed. Experimental study is conducted on toy datasets and three image classification datasets to demonstrate the properties of the proposed M-layer.