AlgebraNets

1 Introduction . Nearly universally , the atomic building blocks of artificial neural networks are scalar real-valued weights and scalar real-valued neuron activations that interact using standard rules of multiplication and addition . We propose AlgebraNets , where we replace the commonly used real-valued algebra with other associative algebras . Briefly , this amounts to replacing scalars by tuples and real multiplication by a tuple multiplication rule . For example , by replacing each scalar weight and activation with 2 × 2 matrices , and standard real addition / multiplication with matrix addition / multiplication . These alternative algebras provide three clear benefits for deep learning at scale : Parameter efficiency . One sweeping benefit of AlgebraNets is they are able to match baseline performance on a variety of tasks , spread over multiple domains , with fewer parameters than the competitive real-valued baselines . This means that equivalently capable models can be trained on smaller hardware , and for a given amount of memory , a model with greater effective capacity can be trained . We find some variants of AlgebraNets that are more parameter efficient than the previously considered C and H algebras . Throughout the text , we count parameters as the total number of real values e.g . a complex number counts as two parameters . Computational efficiency . For scaling large models , parameter efficiency is not the only bottleneck : FLOP efficiency – reducing the relative number of floating-point operations to achieve an equivalent accuracy – is also important . We find instantiations of AlgebraNets that are more FLOP efficient than the previously considered C and H algebras and as FLOP efficient as R. Additionally , all of the proposed algebras offer parameter reuse greater than 1 ( see Table 1 ) . That is , the ratio of multiplications performed to values consumed is greater than or equal to 1:1 . By contrast , for multiplication in R it is only 1:2 . Modern hardware requires a high ratio of floating point operations to bytes loaded ( bandwidth ) to become compute bound and saturate the arithmetic units . This is particularly problematic for auto-regressive inference ( dominated by matrix-vector multiplies ) , sparse models , depthwise convolutions and other operations with low arithmetic density . Architectural exploration . The choice of real numbers for weights and activations is usually taken for granted ( with some exceptions , e.g . those discussed in Sec . 3 ) . With AlgebraNets , we challenge this established design choice and open up a vast new space for neural network architecture exploration by showing that real numbers can be easily replaced with a variety of algebraic structures . Leveraging these new building blocks , one can consider different algebraic interactions , different choices of non-linearities , and different network architecture choices . Importantly , as we demonstrate in this work , AlgebraNets are not only scalable to large models and complex tasks , but they in fact offer improvements in model efficiency , which makes them a viable practical choice . We believe we have only begun to scratch the surface of what these alternative building blocks can enable , and we hope that their broader adoption will usher in further progress across the field . In summary , our main contributions are as follows : • We propose AlgebraNets — a novel class of neural networks , which replaces the nearly ubiquitously used real algebra with alternatives . We show that in contrast to previous work , algebra specific initializations and replacement of batch normalization by an expensive whitening procedure ( Trabelsi et al. , 2018 ; Gaudet and Maida , 2018 ; Wu et al. , 2020 ; Pan et al. , 2019 ) is not necessary , making them a near drop-in replacement to real-valued networks . • We evaluate AlgebraNets based on a wide range of algebras on three challenging large scale benchmarks : ImageNet image classification ( Russakovsky et al. , 2015 ) , Enwik8 ( LLC , 2009 ) , and WikiText language modelling ( Merity et al. , 2016 ) . • We explore sparse AlgebraNets to take advantage of their higher compute density . • We find that AlgebraNets offer improved parameter efficiency and FLOP parity compared to the real-valued baselines , which establishes them as a viable choice for efficient deep learning at scale . 2 AlgebraNets . 2.1 Why Algebras ? . We consider algebras because they have the right properties to make them a drop-in replacement for real numbers in typical neural networks . This is not surprising as the real numbers are an algebra over themselves . An algebra A over a field K ( which we take to always be the field of real or complex numbers ) satisfies the following properties1 ( Wikipedia contributors , 2020b ; a ) : 1 . It is a vector space overK . • It has an associative and commutative addition operator with an identity element ( x+ 0 = x ) and inverse element ( x+ ( −x ) = 0 ) . • It is possible to multiply elements of fieldK with vectors.2 2 . There is a right and left distributive multiplication operator • over vectors closed in A . 3 . Scalar multiplication combines with • in a compatible way : ( ax ) • ( by ) = ( ab ) ( x • y ) . We do not claim that these properties are all required as neural network building-blocks , merely that they are convenient . For example , one could imagine not having associative addition – this would require a careful implementation to get right but is possible . One could eliminate the requirement that scalars fromK multiply with vectors from A - this would make various normalizations ( e.g . batch normalization ) impossible , but they are not required . Most importantly , removing some of these requirements does not lead to an obviously useful class of mathematical objects to consider . In addition to the previously considered C and H algebras , we also consider the algebras of n× n matrices over R and C ( i.e . Mn ( R ) orMn ( C ) ) as they have higher compute density than R and map well to the matrix multiplication units that are becoming common in processors ( Oh , 2019 ) . We note 1We use the terminology ‘ vector ’ in the definition as that is the generally accepted mathematical term , however throughout the rest of the paper we use the term ‘ tuple ’ instead . This is to avoid the confusion of calling a matrix a vector , which is technically correct in this context , but rife with potential for confusion . 2a ( bx ) = ( ab ) x ; 1x = x for 1 , the multiplicative identity inK ; a ( x+y ) = ax+ay ; ( a+ b ) x = ax+ bx thatM2 ( R ) is isomorphic to the split-quaternion ( Cockle , 1849 ) algebra andM2 ( C ) is isomorphic to the biquaternion ( Hamilton , 1844 ) algebra , but the matrix algebras are more familiar so we retain that terminology . Lastly , we consider the dual numbers and the cross product of length-3 tuples . 2.2 Algebra Details . R ; Real Numbers All baseline networks are real-valued with scalar weights ; standard multiplication rules apply . For two weight values , we load 2 scalars and perform 1 multiply . C a b a a b b b −a We provide tables describing the multiplicative interaction between tuples . The interaction between two tuples ( oa , ob , ... ) = ( ta , tb , ... ) • ( va , vb , ... ) is described by a matrix where the indices of t are on the left , v are on the top and entries correspond to which component of o the interaction contributes to . A 0 means there is no interaction and a negative index means the result is subtracted . C ; Complex Numbers Each weight , w , is a length 2 tuple ( ta , tb ) representing the complex number ta + tbi . For two weight values we load 4 scalars and perform 4 multiplies . Mn ( R ) ; n×nMatricesEachweight is a lengthn2 tuple , representing ann×nmatrix . Multiplication and addition proceed with standard rules for matrices . We consider up toM4 ( R ) matrices . For two weight values we load 2n2 scalars and perform n3 multiplies . H a b c d a a b c d b b -a d -c c c -d -a b d d c -b -a Mn ( C ) ; n×nComplexMatricesWeights are length 2n2 tuples representing n × n complex-valued matrices . We consider only n = 2 . For two weight values we load 4n2 scalars and perform 4n3 multiplies . The multiplication table is in Appendix A. H ; Quaternions Each weight , wi is replaced by a length 4 tuple , ( ta , tb , tc , td ) . Multiplication is not commutative , with the product of two quaternions given by the Hamilton product ( Hamilton , 1843 ) . For two weight values , we load 8 elements and perform 16 multiplies . D a b c d a a 0 0 0 b 0 b 0 0 c 0 0 c 0 d 0 0 0 d Diagonal Algebra The high FLOP cost of the whitening operation required by ( Trabelsi et al. , 2018 ; Gaudet and Maida , 2018 ; Wu et al. , 2020 ; Pan et al. , 2019 ) makes networks using it inefficient at training and inference in terms of FLOPs . We attempt to design an algebra where using whitening would in fact be competitive by eliminating the interaction of terms through the algebra . Only when combining the ‘ diagonal ’ D algebra with whitening are there interactions between the different tuple components . a b a a b b b 0 Dual Numbers Each weight is represented by a length 2 tuple representing the dual number ( t0 + t1 ) . For a multiplication , we load 4 values and perform 3 multiplies . R3 a b c a 0 c -b b -c 0 a c b -a 0 R3 Cross Product Each weight is represented by a length 3 tuple . We use the cross product between two tuples for the multiplication rule , resulting in 6 different multiplies for 6 values loaded . 2.3 Initialization , Normalization , Non-Linearities , and Pruning . Prior work ( Trabelsi et al. , 2018 ; Gaudet andMaida , 2018 ) has advocated algebra-specific initializations and expensive whitening procedures to replace batch normalization . We find that this is not necessary to achieve good performance , and we are able to use the same initialization , normalization , and non-linearities across all algebras which facilitates exploring a wide variety of options . To initialize all the components of the algebra tuple at the beginning of a network we set the first tuple component to the typical input . For ResNet , MobileNet , and the RNN we initialize the other components of the tuple with a small one or two-layer MLP , i.e . tb , c , ... = MLP ( ta ) . For the transformer , we take advantage of the fact that the embedding is already a learned representation and simply reshape the output embedding appropriately . We find that the specifics of the input initialization do not have a large effect on performance , though allowing a learnable transformation outperformed initializing additional components to 0 or replicating the input . We use standard Glorot ( Glorot and Bengio , 2010 ) weight initialization of each component independently . Comparisons with the algebra specific initializations ( Trabelsi et al. , 2018 ; Gaudet and Maida , 2018 ) can be found in Appendix B . Existing activation functions can be applied component-wise ( t = ( f ( ta ) , · · · , f ( td ) ) ) and we found that ReLU and swish ( Ramachandran et al. , 2017 ) work well ; tanh and sigmoid can also be applied component-wise as part of GRUs and LSTMs . Applying the activation function to the entire tuple has possible computational advantages if it is ReLU-like as it would allow an entire tuple multiplication to be skipped . For example , consider t = f ( g ( t ) ) t. If g ( • ) returns the mean of the tuple , and if f was H the Heaviside step function , then one can remove entire components . Appendix B examines different choices for doing this , but we do not consider it further in the main text . The final logits of an AlgebraNet must be real-valued . We use an Algebra-specific final linear layer and convert the final algebra tuple to a scalar with the tuple-wise L2 norm before applying softmax . More details are in Appendix B . To apply magnitude pruning ( Zhu and Gupta , 2017 ; Gale et al. , 2019 ) to prune tuples we used the tuple L2 norm as the criterion for pruning for all AlgebraNet variants . For theMn ( R ) algebras we also experimented with criteria based on the eigenvalues , λi , and singular values , σi , of each n× n matrix . The Frobenius norm corresponds to ( ∑ i σ 2 i ) 1/2 and the determinant corresponds to ( ∏ i λi ) . We found pruning based on the Frobenius norm to be the most effective , followed by pruning based on the largest eigenvalue . See Appendix C for a comparison between different methods . ( Trabelsi et al. , 2018 ) , ( Gaudet and Maida , 2018 ) , and ( Wu et al. , 2020 ) use whitening in place of batch normalization . Whitening normalizes and de-correlates the different tuple elements from one another . However , this extension results in a substantial increase in both training and test time computational costs , as described in ( Pan et al. , 2019 ) . The inclusion of the whitening cost to the FLOP count in Fig . 1 highlights the substantial cost inference cost . Cholesky decomposition ( Press et al. , 2007 ) of the inverted covariance matrix is required during training and at inference it is not possible to fold the whitening transformation into adjacent convolutions . A contribution from each of the algebra elements contributes to each element in the whitened output . We find that batch normalization does not substantially decrease performance , trains 1.9× faster and has no inference cost , so we use it for all experiments , unless explicitly stated .

Summary: The authors propose AlgebraNets - a previously explored approach to replace real-valued algebra in deep learning models with other associative algebras that include 2x2 matrices over real and complex numbers. They provide a comprehensive overview of prior methods in this direction and motivate their work with potential for both parameter and computational efficiency, and suggest that the latter is typically overlooked in prior literature. The paper is very well-written and follows a nice narrative, and the claims are mostly backed empirically with experimental results.