AlgebraNets

1 Introduction . Nearly universally , the atomic building blocks of artificial neural networks are scalar real-valued weights and scalar real-valued neuron activations that interact using standard rules of multiplication and addition . We propose AlgebraNets , where we replace the commonly used real-valued algebra with other associative algebras . Briefly , this amounts to replacing scalars by tuples and real multiplication by a tuple multiplication rule . For example , by replacing each scalar weight and activation with 2 × 2 matrices , and standard real addition / multiplication with matrix addition / multiplication . These alternative algebras provide three clear benefits for deep learning at scale : Parameter efficiency . One sweeping benefit of AlgebraNets is they are able to match baseline performance on a variety of tasks , spread over multiple domains , with fewer parameters than the competitive real-valued baselines . This means that equivalently capable models can be trained on smaller hardware , and for a given amount of memory , a model with greater effective capacity can be trained . We find some variants of AlgebraNets that are more parameter efficient than the previously considered C and H algebras . Throughout the text , we count parameters as the total number of real values e.g . a complex number counts as two parameters . Computational efficiency . For scaling large models , parameter efficiency is not the only bottleneck : FLOP efficiency – reducing the relative number of floating-point operations to achieve an equivalent accuracy – is also important . We find instantiations of AlgebraNets that are more FLOP efficient than the previously considered C and H algebras and as FLOP efficient as R. Additionally , all of the proposed algebras offer parameter reuse greater than 1 ( see Table 1 ) . That is , the ratio of multiplications performed to values consumed is greater than or equal to 1:1 . By contrast , for multiplication in R it is only 1:2 . Modern hardware requires a high ratio of floating point operations to bytes loaded ( bandwidth ) to become compute bound and saturate the arithmetic units . This is particularly problematic for auto-regressive inference ( dominated by matrix-vector multiplies ) , sparse models , depthwise convolutions and other operations with low arithmetic density . Architectural exploration . The choice of real numbers for weights and activations is usually taken for granted ( with some exceptions , e.g . those discussed in Sec . 3 ) . With AlgebraNets , we challenge this established design choice and open up a vast new space for neural network architecture exploration by showing that real numbers can be easily replaced with a variety of algebraic structures . Leveraging these new building blocks , one can consider different algebraic interactions , different choices of non-linearities , and different network architecture choices . Importantly , as we demonstrate in this work , AlgebraNets are not only scalable to large models and complex tasks , but they in fact offer improvements in model efficiency , which makes them a viable practical choice . We believe we have only begun to scratch the surface of what these alternative building blocks can enable , and we hope that their broader adoption will usher in further progress across the field . In summary , our main contributions are as follows : • We propose AlgebraNets — a novel class of neural networks , which replaces the nearly ubiquitously used real algebra with alternatives . We show that in contrast to previous work , algebra specific initializations and replacement of batch normalization by an expensive whitening procedure ( Trabelsi et al. , 2018 ; Gaudet and Maida , 2018 ; Wu et al. , 2020 ; Pan et al. , 2019 ) is not necessary , making them a near drop-in replacement to real-valued networks . • We evaluate AlgebraNets based on a wide range of algebras on three challenging large scale benchmarks : ImageNet image classification ( Russakovsky et al. , 2015 ) , Enwik8 ( LLC , 2009 ) , and WikiText language modelling ( Merity et al. , 2016 ) . • We explore sparse AlgebraNets to take advantage of their higher compute density . • We find that AlgebraNets offer improved parameter efficiency and FLOP parity compared to the real-valued baselines , which establishes them as a viable choice for efficient deep learning at scale . 2 AlgebraNets . 2.1 Why Algebras ? . We consider algebras because they have the right properties to make them a drop-in replacement for real numbers in typical neural networks . This is not surprising as the real numbers are an algebra over themselves . An algebra A over a field K ( which we take to always be the field of real or complex numbers ) satisfies the following properties1 ( Wikipedia contributors , 2020b ; a ) : 1 . It is a vector space overK . • It has an associative and commutative addition operator with an identity element ( x+ 0 = x ) and inverse element ( x+ ( −x ) = 0 ) . • It is possible to multiply elements of fieldK with vectors.2 2 . There is a right and left distributive multiplication operator • over vectors closed in A . 3 . Scalar multiplication combines with • in a compatible way : ( ax ) • ( by ) = ( ab ) ( x • y ) . We do not claim that these properties are all required as neural network building-blocks , merely that they are convenient . For example , one could imagine not having associative addition – this would require a careful implementation to get right but is possible . One could eliminate the requirement that scalars fromK multiply with vectors from A - this would make various normalizations ( e.g . batch normalization ) impossible , but they are not required . Most importantly , removing some of these requirements does not lead to an obviously useful class of mathematical objects to consider . In addition to the previously considered C and H algebras , we also consider the algebras of n× n matrices over R and C ( i.e . Mn ( R ) orMn ( C ) ) as they have higher compute density than R and map well to the matrix multiplication units that are becoming common in processors ( Oh , 2019 ) . We note 1We use the terminology ‘ vector ’ in the definition as that is the generally accepted mathematical term , however throughout the rest of the paper we use the term ‘ tuple ’ instead . This is to avoid the confusion of calling a matrix a vector , which is technically correct in this context , but rife with potential for confusion . 2a ( bx ) = ( ab ) x ; 1x = x for 1 , the multiplicative identity inK ; a ( x+y ) = ax+ay ; ( a+ b ) x = ax+ bx thatM2 ( R ) is isomorphic to the split-quaternion ( Cockle , 1849 ) algebra andM2 ( C ) is isomorphic to the biquaternion ( Hamilton , 1844 ) algebra , but the matrix algebras are more familiar so we retain that terminology . Lastly , we consider the dual numbers and the cross product of length-3 tuples . 2.2 Algebra Details . R ; Real Numbers All baseline networks are real-valued with scalar weights ; standard multiplication rules apply . For two weight values , we load 2 scalars and perform 1 multiply . C a b a a b b b −a We provide tables describing the multiplicative interaction between tuples . The interaction between two tuples ( oa , ob , ... ) = ( ta , tb , ... ) • ( va , vb , ... ) is described by a matrix where the indices of t are on the left , v are on the top and entries correspond to which component of o the interaction contributes to . A 0 means there is no interaction and a negative index means the result is subtracted . C ; Complex Numbers Each weight , w , is a length 2 tuple ( ta , tb ) representing the complex number ta + tbi . For two weight values we load 4 scalars and perform 4 multiplies . Mn ( R ) ; n×nMatricesEachweight is a lengthn2 tuple , representing ann×nmatrix . Multiplication and addition proceed with standard rules for matrices . We consider up toM4 ( R ) matrices . For two weight values we load 2n2 scalars and perform n3 multiplies . H a b c d a a b c d b b -a d -c c c -d -a b d d c -b -a Mn ( C ) ; n×nComplexMatricesWeights are length 2n2 tuples representing n × n complex-valued matrices . We consider only n = 2 . For two weight values we load 4n2 scalars and perform 4n3 multiplies . The multiplication table is in Appendix A. H ; Quaternions Each weight , wi is replaced by a length 4 tuple , ( ta , tb , tc , td ) . Multiplication is not commutative , with the product of two quaternions given by the Hamilton product ( Hamilton , 1843 ) . For two weight values , we load 8 elements and perform 16 multiplies . D a b c d a a 0 0 0 b 0 b 0 0 c 0 0 c 0 d 0 0 0 d Diagonal Algebra The high FLOP cost of the whitening operation required by ( Trabelsi et al. , 2018 ; Gaudet and Maida , 2018 ; Wu et al. , 2020 ; Pan et al. , 2019 ) makes networks using it inefficient at training and inference in terms of FLOPs . We attempt to design an algebra where using whitening would in fact be competitive by eliminating the interaction of terms through the algebra . Only when combining the ‘ diagonal ’ D algebra with whitening are there interactions between the different tuple components . a b a a b b b 0 Dual Numbers Each weight is represented by a length 2 tuple representing the dual number ( t0 + t1 ) . For a multiplication , we load 4 values and perform 3 multiplies . R3 a b c a 0 c -b b -c 0 a c b -a 0 R3 Cross Product Each weight is represented by a length 3 tuple . We use the cross product between two tuples for the multiplication rule , resulting in 6 different multiplies for 6 values loaded . 2.3 Initialization , Normalization , Non-Linearities , and Pruning . Prior work ( Trabelsi et al. , 2018 ; Gaudet andMaida , 2018 ) has advocated algebra-specific initializations and expensive whitening procedures to replace batch normalization . We find that this is not necessary to achieve good performance , and we are able to use the same initialization , normalization , and non-linearities across all algebras which facilitates exploring a wide variety of options . To initialize all the components of the algebra tuple at the beginning of a network we set the first tuple component to the typical input . For ResNet , MobileNet , and the RNN we initialize the other components of the tuple with a small one or two-layer MLP , i.e . tb , c , ... = MLP ( ta ) . For the transformer , we take advantage of the fact that the embedding is already a learned representation and simply reshape the output embedding appropriately . We find that the specifics of the input initialization do not have a large effect on performance , though allowing a learnable transformation outperformed initializing additional components to 0 or replicating the input . We use standard Glorot ( Glorot and Bengio , 2010 ) weight initialization of each component independently . Comparisons with the algebra specific initializations ( Trabelsi et al. , 2018 ; Gaudet and Maida , 2018 ) can be found in Appendix B . Existing activation functions can be applied component-wise ( t = ( f ( ta ) , · · · , f ( td ) ) ) and we found that ReLU and swish ( Ramachandran et al. , 2017 ) work well ; tanh and sigmoid can also be applied component-wise as part of GRUs and LSTMs . Applying the activation function to the entire tuple has possible computational advantages if it is ReLU-like as it would allow an entire tuple multiplication to be skipped . For example , consider t = f ( g ( t ) ) t. If g ( • ) returns the mean of the tuple , and if f was H the Heaviside step function , then one can remove entire components . Appendix B examines different choices for doing this , but we do not consider it further in the main text . The final logits of an AlgebraNet must be real-valued . We use an Algebra-specific final linear layer and convert the final algebra tuple to a scalar with the tuple-wise L2 norm before applying softmax . More details are in Appendix B . To apply magnitude pruning ( Zhu and Gupta , 2017 ; Gale et al. , 2019 ) to prune tuples we used the tuple L2 norm as the criterion for pruning for all AlgebraNet variants . For theMn ( R ) algebras we also experimented with criteria based on the eigenvalues , λi , and singular values , σi , of each n× n matrix . The Frobenius norm corresponds to ( ∑ i σ 2 i ) 1/2 and the determinant corresponds to ( ∏ i λi ) . We found pruning based on the Frobenius norm to be the most effective , followed by pruning based on the largest eigenvalue . See Appendix C for a comparison between different methods . ( Trabelsi et al. , 2018 ) , ( Gaudet and Maida , 2018 ) , and ( Wu et al. , 2020 ) use whitening in place of batch normalization . Whitening normalizes and de-correlates the different tuple elements from one another . However , this extension results in a substantial increase in both training and test time computational costs , as described in ( Pan et al. , 2019 ) . The inclusion of the whitening cost to the FLOP count in Fig . 1 highlights the substantial cost inference cost . Cholesky decomposition ( Press et al. , 2007 ) of the inverted covariance matrix is required during training and at inference it is not possible to fold the whitening transformation into adjacent convolutions . A contribution from each of the algebra elements contributes to each element in the whitened output . We find that batch normalization does not substantially decrease performance , trains 1.9× faster and has no inference cost , so we use it for all experiments , unless explicitly stated .

Summary: In this paper, the authors propose the usage of complex numbers in deep neural networks. Would be good to know that complex numbers, n x n matrices, quaternions, diagonal matrices, etc. all can be used in neural networks. The authors also claims benchmark performance in large-scale image classification and language modeling.