AlgebraNets
Authors: Jordan Hoffmann
Simon Schmitt
Simon Osindero
Karen Simonyan
Erich Elsen
Venue:  (2021)
Keywords: Sparsity
Pruning
Efficiency
Mathematics

Abstract: Neural networks have historically been built layerwise from the set of functions in ${f: \mathbb{R}^n \to \mathbb{R}^m }$, i.e. with activations and weights/parameters represented by real numbers, $\mathbb{R}$. Our work considers a richer set of objects for activations and weights, and undertakes a comprehensive study of alternative algebras as number representations by studying their performance on two challenging problems: large-scale image classification using the ImageNet dataset and language modeling using the enwiki8 and WikiText-103 datasets. We denote this broader class of models as AlgebraNets. Our findings indicate that the conclusions of prior work, which explored neural networks constructed from $\mathbb{C}$ (complex numbers) and $\mathbb{H}$ (quaternions) on smaller datasets, do not always transfer to these challenging settings.  However, our results demonstrate that there are alternative algebras which deliver better parameter and computational efficiency compared with $\mathbb{R}$. We consider $\mathbb{C}$, $\mathbb{H}$, $M_{2}(\mathbb{R})$ (the set of $2\times2$ real-valued matrices), $M_{2}(\mathbb{C})$, $M_{3}(\mathbb{R})$, $M_{4}(\mathbb{R})$, dual numbers and the $\mathbb{R}^3$ cross product.  Additionally, we note that multiplication in these algebras has higher compute density than real multiplication, a useful property in situations with inherently limited parameter reuse such as auto-regressive inference and sparse neural networks. We therefore investigate how to induce sparsity within AlgebraNets. We hope that our strong results on large-scale, practical benchmarks will spur further exploration of these unconventional architectures which challenge the default choice of using real numbers for neural network weights and activations.

Under review as a conference paper at ICLR 2021
ALGEBRANETS
Anonymous authors
Paper under double-bling review
ABSTRACT
Neural networks have historically been built layerwise from the of functions in f IR + R i.e. with activations and weights/parameetes represented by real numbers, R. Our work considers richer set of objects for activations and weights, and undertakes comprehensive study of alternative algebras as number representations by studying their performance on two challenging problems: large- scale image classification using the ImageNet dataset and language modeling using the enwiki8 and WikiText- 103 datasets. We denote this broader class models AlgebraNets. Our findings indicate that the conclusions of prior work, which explored neural networks constructed from C (complex numbers) and (quaternions) on smaller datasets, do not always transfer to these challenging settings. However, our results demonstrate that there are alternative algebras which deliver better parameter and computational efficiency compared with R. We consider C, H. M22(IR) (the set of 2 x 2 real-valued matrices), Mc, M3 MR(R), M, (R), dual numbers and the R cross product. Additionally, we note that multiplication these algebras has higher compute density than real multiplication, useful property in situations with inherently limited parameter reuse such as auto-regressive inference and sparse neural networks. We therefore investigate how to induce sparsity within AlgebraNets. We hope that our strong results on large-scale, practical benchmarks will spur further exploration of these unconventional architectures which challenge the default choice of using real numbers for neural network weights and activations.
INTRODUCTION
Nearly universally, the atomic building blocks of artificial neural networks are scalar real-valued weights and scalar real-valued neuron activations that interact using standard rules of multiplication and addition.
We propose AlgebraNets, where we replace the commonly used real-valued algebra with other associative algebras. Briefly, this amounts to replacing scalars by tuples and real multiplication by a tuple multiplication rule. For example, by replacing each scalar weight and activation with 2 x matrices, and standard real addition / multiplication with matrix addition multiplication. These alternative algebras provide three clear benefits for deep learning at scale:
Parameter efficiency. One sweeping benefit of AlgebraNets is they are able to match baseline performance on a variety of tasks, spread over multiple domains, with fewer parameters than the competitive real-valued baselines. This means that equivalently capable models can be trained on smaller hardware, and for given amount of memory, model with greater effective capacity can be trained. We find some variants of AlgebraNets that are more parameter efficient than the previously considered C and H algebras. Throughout the text, we count parameters as the total number of real values e.g. complex number counts as two parameters.
Computational efficiency. For scaling large models, parameter efficiency is not the only bottleneck: FLOP efficiency reducing the relative number of floating-pon point operations to achieve an equivalent accuracy - is also important. We find instantiations of AlgebraNets that are more FLOP efficient than the previously considered C and H. algebras and FLOP efficient as R. Additionally, all of the proposed algebras offer parameter reuse greater than 1 (see Table []] That is, the ratio of multiplications performed to values consumed is greater than equal to 1:1. By contrast, for multiplication in IR is only 1:2. Modern hardware requires a high ratio of floating point operations to bytes loaded (bandwidth) to become compute bound and saturate the arithmetic units. This is

---

Under review as a conference paper at ICLR 2021
particularly problematic for auto-regressive inference (dominated by matrix- vector multiplies), sparse models, depthwise convolutions and other operations with low arithmetic density.
Architectural exploration. The choice of real numbers for weights and activations is usually taken for granted (with some exceptions, c.g. those discussed in Sec. 3). With AlgebraNets, we challenge this established design choice and open up by showing that real numbers can be easily replaced with a variety of algebraic structures. Leveraging these new building blocks, one can consider different algebraic interactions, different choices of non-linearities, and different network architecture choices. Importantly, as we demonstrate in this work. AlgebraNets are not only scalable large models and complex tasks, but they in fact offer improvements in model efficiency. which makes them a viable practical choice. We believe we have only begun to scratch the surface of what these alternative building blocks can enable, and we hope that their broader adoption will usher in further progress across the field.
new space for neural network architecture exploration
In summary, our main contributions are as follows:
We propose AlgebraNets - a novel class of neural networks, which replaces the nearly ubiquitously used real algebra with alternatives. We show that in contrast to previous work, algebra specific initializations and replacement of batch normalization by an expensive whitening procedure (Trabelsi et al. 2018: Gaudet and Maida 2018) Wu et al. [2020]
Pan et al., 2019) is not necessary, making them a near drop-in replace real-valued networks.
We evaluate AlgebraNets based on a wide range of algebras on three challenging large scale benchmarks: ImageNet image classification (Russakovsky et al. 2015), Enwik8 (LLC)
2009), and WikiText language modelling (Merity et 2016)..
We explore sparse AlgebraNets take advantage of their higher compute density. We find that AlgebraNets offer improved parameter efficiency and FLOP parity compared the rellev baselines, which establishes them as a viable choice for efficient deep
learning at scale.
2 ALGEBRANETS
2.1 WHY ALGEBRAS?
We consider algebras because they have the right properties to make them a drop-in replacement for real numbers in typical neural networks. This is not surprising as the real numbers are an algebra over themselves. An algebra over a field (which we take to always be the field of real or complex numbers) satisfies the following properties] (Wikipedia contributors: 2020b a):
1. It is vector space over K.
It has an associative and commutative addition operator with an identity element (x + x) and inverse element (x + (-x) = 0).
is possible to multiply elements of field K with vectors
2. There is a right and left distributive multiplication operator over vectors closed in A.
3. Scalar multiplication combines with in a compatible way: (ax) - (by) = (ab)(x x y).
We do not claim that these properties are all required as neural network building- -blockg merely that they are convenient. For example, one could imagine not having associative addition - this would require a careful implementation to get right but is possible. One could eliminate the requirement that scalars from K multiply with vectors from A - this would make various normalizations (e.g. batch normalization) impossible, but they are not required. Most importantly, removing some of these requirements does not lead to obviously useful class of mathematical objects to consider.
In addition to the previously considered and HH algebras, we also consider the algebras of n matrices over R and C (i.e. M, (R) or (C)) as they have higher compute density than R and map well to the matrix multiplication units that are becoming common in processors (Ohl 2019). We note
We use the terminology 'vector' in the definition as that is the generally accepted mathematical term, however throughout the rest of the paper use the term 'tuple' instead. This is to avoid the confusion of calling matrix vector, which is technically correct in this context, but rife with potential for confusion.
Ra(bx)= (ab)x; 1x for 1, the multiplicative identity K; a(x y) = + ay; (a + b)x bx
2

---

Under review as conference paper at ICLR 2021
that (R) is isomorphic the split-quaternion (Cockle 1849) algebra and (C) is isomorphic to the biquaternion (Hamilton 1844) algebra, but the matrix algebras are more familiar so we retain that terminology. Lastly, we consider the dual numbers and the cross product of length-3 tuples.
2.2
ALGEBRA DETAILS
Algebra
Weight Size
Weight Reuse
Multiplies
Values Loaded
Real Numbers
1
Complex Numbers (C)
4:
2 2 Matrix M2 (I))
4
2
8: 8
3 3 Matrix My
9
3
27: 18
Matrix (TR)
64: 32
x 2 Complex Matrix (C)
32:
Quaternions H
4
4
16: 8
Diagonal Algebras
N
N 2NV
Dual Numbers
3
R
Cross Product
3
6:
Table I: Comparison of the different algebras we consider. Weight Size represents the number of components in each tuple, for example 2 x matrix has 4 components. Weight Reuse denotes how many output tuple components each input tuple component is involved in. For example, in a M2(R) matrix multiply, each weight component involved in the update of two components.
I; Real Numbers All baseline networks are real-valued with scalar weights; standard multiplication rules apply. For two weight values, we load 2 scalars and perform multiply.
We provide tables describing the multiplicative interaction between tuples. The interaction between two tuples (Oa, og. ...) = (ta, tb. ...) (Va ...) is the top
C a
b a a b b b
described by matrix where the indices of / are the left, and entries correspond to which component of o the interaction contributes to. A means there is no interaction and negative index means the result is subtracted.
-a
C; Complex Numbers Each weight, w. a length 2 tuple (ta, (a) representing the complex number ta + lg.. For weight values load scalars and perform 4 multiplies.
M, (R); n X n Matrices Each weight is length n² tuple, representing an n xn matrix. Multiplication and addition proceed with standard rules for matrices. We consider up (TR)) matrices. For two weight values we load 2n² scalars and perform n² multiplies.
Mn (C); ; n X Complex Matrices Weights length 2n² tuples representing x complex-valued matrices. We consider only n 2. For two weight values load 4n scalars and perform 4n multiplies. The multiplication table is in Appendix A
H a
e
d
b
d
b
b
d
HI; Quaternions Each weight, w; replaced by length 4 tuple. (tas to, te: ty). Multiplication is not commutative, with the product of two quaternions given by the Hamilton product (Hamilton) (843). For two weight values, we load elements and perform 16 multiplies.
-d
-0
d d
-
-0
Diagonal Algebra The high FLOP cost the whitening operation required by (Trabelsi et al. (2088) Gaudet and Maida 2018 Wu et al. 2020 Pan et al. 2019) makes networks using it inefficient at training and inference in terms of FLOPs. We attempt design an algebra where using whitening would in fact be competitive by climinating the interaction of terms through the algebra. Only when combining the 'diagonal' T algebra with whitening there interactions between the different tuple components.
D
d
a
a
0
b
0
0
C
0
d 0
a b
Dual Numbers Each weight is represented by length 2 tuple representing the dual number (lo + (1e..
b
b 0
For multiplication, load 4 values and perform 3 multiplies.

---

Under review as a conference paper at ICLR 2021
IR3
b
R Cross Product Each weight is represented by length 3 tuple. We the cross product between two tuples for the multiplication rule, resulting in 6 different multiplies for 6 values loaded.
0
c
-b
b
C
0
b
-
0
2.3 INITIALIZATION, NORMALIZATION, NON-LINFARRTTER, AND PRUNING
Prior work (Trabelsi al. 2018 Gaudet and Maida 2018) has advocated algebra- initializations and expensive whitening procedures to replace batch normalization. We find that this is not necessary to achieve good performance, and we are able to use the same initialization, normalization, and non-linearities across all algebras which facilitates exploring a wide variety of options.
To initialize all the components of the algebra tuple at the beginning of a network we set the first tuple component the typical input. For ResNet, MobileNet, and the RNN we initialize the other components of the tuple with a small one or two-layer MLP, i.e. to..... = M (taa). For the transformer, we take advantage of the fact that the embedding already learned representation and simply reshape the output embedding appropriately. We find that the specifics of the input initialization do have large effect on performance, though allowing learnable transformation outperformed initializing additional components to O or replicating the input. We use standard Glorot (Glorot and Bengio 2010) weight initialization of each component independently. Comparisons with the algebra specific initializations (Trabelsi et al. 2018 Gaudet and Maida 2018) can be found in Appendix B
((a))) and we found that ReLU and swish (Ramachandran et al. 2017) work well; tanh and sigmoid can also be applicd component- wise as part GRUs and LSTMs. Applying the activation function to the entire tuple has possible computational advantages if it is ReLU-like as it would allow entire tuple multiplication to be skipped. For example, consider t = f (t)t)).. If g()) returns the mean of the tuple, and if f was H the Heaviside step function, then one remove entire components. Appendix [B] examines different choices for doing this, but we do not consider it further in the main text.
Existing activation functions can be applied component-wise (t = (J(Ia),
The final logits of an AlgebraNet must be real-valued. We an Algebra-specific final linear layer and convert the final algehra tuple to a scalar with the tuple-wise L2 norm before applying softmax. More details are in Appendix B.
To apply magnitude pruning (Zhu and Gupta 2017 Gale et al. 2019) to prune tuples we used the tuple La norm as the criterion for pruning for all AlgebraNet variants. For the (R)) algebras we also experimented with criteria based on the eigenvalues, X; and singular values, d; of each n x n matrix. The Frobenius norm corresponds to (E; o/2 and the determinant corresponds to ([I, A;). We found pruning based on the Frobenius norm to be the most effective, followed by pruning based on the largest cigenvalue. See Appendix C
for a comparison between different methods.
(Trabelsi 2018). (Gaudet and Maida 2018), and (Wu al. 2020) use whitening in place of batch normalization. Whitening normalizes and de-correlates the different tuple elements from one another. However, this extension results in a substantial increase both training and test time computational costs. as described in (Pan et al. 2019). The inclusion of the whitening cost to the FLOP count in Fig. D highlights the substantial cost inference cost. Cholesky decomposition (Press al. 2007) of the inverted covariance matrix is required during training and at inference it is not possible to fold the whitening transformation into adjacent convolutions. A contribution from each of the algebra elements contributes to each element in the whitened output. We find that batch normalization does not substantially decrease performance, trains 1.9x faster and has no inference cost, so we use it for all experiments, unless explicitly stated.
3 RELATED WORK
(Trabelsi et al., 2018) applied complex-valued networks to convolutional neural networks trained on CIFAR-IO, 10, as well as to music transcription and Speech Spectrum Prediction. They find that complex-valued networks with the same number of parameters and more FLOPs perform slightly better than real-valued networks. (Gaudet and Maida 2018) extend the procedure from Trabelsi et al. 2018) to quaternion valued weights, showing that they are able to reduce the parameter count by a factor of two over complex-valie valued networks and a factor of four over real-valued networks, while again slightly increasing the top-1 accuracy on CIFAR-10. (Wu et al. 2020) further extend
4

---

Under review as a conference paper at ICLR 2021
this approach to octonions (which a non-associative algebra), demonstrating that they are able to further reduce the parameter count while increasing the accuracy of their models on CIFAR-IO.
These papers establish the efficacy of some alternative algebras, though they focus purely parameter efficiency, rather than FLOP efficiency which is equally important for image classification tasks. Additionally, the tested datasets are relatively small, and it is unclear how the results scale to larger datasets. Both the quaternion and octonion network papers do not test their models on language modeling tasks where parameter efficiency is often of greater importance.
(Parcollet et al. 2018) propose a quaternion recurrent neural network (QRNN) and quaternion LSTM (LSTM.. They show that quaternion based methods are able to reduce the parameter count while offering better performance on the TIMIT and WSJ phoneme recognition tasks. Associative Long Short-Term Memory leverage complex-vectors to increase the memory capacity of LSTMs without a parameter increase (Danhelka et al. 2016).
Recently, many methods to induce sparsity neural networks have shown that it is possible to train models with an overwhelming fraction of the weights being 0 (Molchanov et al. [017 Gale et al. 2019. Frankle and Carbin 2019 Louizos et al., 2018 Evei et al. 2019 Zhu and Gupta 2017). ) Many of these methods gradually decrease the number of weights in the network through training by using some combination of each weight's gradient and magnitude. Fine grained sparsity is hard to accelerate on modern hardware, although there have been some recent results demonstrating that speedups are possible (Elsen et al,, 2020). (Vecchi et al. 2019) considered inducing sparsity in quaternion networks. Primitives that increase computational density of fundamental interactions would increase the performance of sparse methods as demonstrated on the GPU by (Mueller- HlueleerRRomer et al. 2019) in
scientific computing.
JJyakumar et al. 2020) emphasize the importance of multiplicative interaction layers providing particularly useful inductive bias during the fusion of multiple information streams. Specific AlgebraNets may provide strong, useful domain-specific inductive biases, for example as done by (Worrall al.) al.] 2017). leveraging the rotational invariance of complex numbers in convolutional networks and by (Hinton et al. 2018) where they use a 4 x 4 pose matrix to represent orientations.
4 EXPERIMENTS AND RESULTS
4.1 IMAGENET
We examine the performance of AlgebraNet versions of ResNet-50 (He et al. 2016) and MobileNet-
v1 (Howard et al. (017) on the ImageNet (Russakovsky 2015) dataset. We use a width multiplier on the channels to adjust model capacity. For all experiments we use SGD with momentum of 0.9. We increase the number of training epochs by factor of two to 180, which we also use for the pruning experiments. This did not affect the baseline, but it improves the pruning results. It also resulted in improved performance for HI, so we used it throughout. For batch size of 256, the initial learning rate for the ResNet experiments was set to 2.5 and multiplied by 0.1 at epochs 60, 100, 140, and 160. We find it is useful to reduce the amount of La regularization that is used for AlgebraNets. The baseline value 10-1 was reduced by factor of 0.725 for ResNet-50 and 0.625 for MobileNet-v1. We use the swish activation function for all experiments shown in Figures] D andd] iincluding the baselines. We found it improved performance across the board.
Figures D and 2 compare the trade-offs between accuracy, parameters, and FLOPs for different flavours of AlgebraNet. Notably, we do not find that the parameter reduction without accuracy loss from (Trabelsi al., 2018) and (Gaudet and Maida 2018) on CIFAR translates to ImageNet; we are unable to divide the number of parameters by factor of two/four for C/H and match baseline performance. We hypothesize that this is in part due to over- paaamatriization of many networks trained on CIFAR and that AlgebraNets act as additional regularizer, in part due to the greater weight reuse: each tuple component is now involved in multiple equations. We feel that this highlights the need for testing methods on large-scale datasets.
We find My (R)) AlgebraNets provide the best parameter efficiency of all considered algebras while requiring no more FLOPs than the real baseline on both ResNet-50 and MobileNet-vl. We also find, for both ResNet-50 and MobileNet-vl, M2(C) AlgebraNets provide better FLOP efficiency than the previously studied H. while having the same ratio of multiplies to values. The diagonal
5

---

Under review as conference paper at ICLR 2021
ResNet-50 50 Parameters
Restltt-S0 FLOPs
78
76
766
top accuracy
Cross Product
Dval Numbers
Diagonall Whilaning
70
ResNet 50 Baseline
10
15
20
35
12
14
16
parameters x:I
FLOPS
Parameters
ResNet-50
78
78
accurecs
top-1 sccurcc
MaUR)
72
M,URI
72
M.te]
Writtening
ResNeet Baseline
parameters
PLOPs x10
Figure 1: ResNet-50 vary the width of a ResNet-50 trained on ImageNet for various flavors of AlgebraNet and the real-valued baseline. We separate based algebras with up 1:1 multiplies to values loaded compute density (top), and greater than a 1:1 density (bottom). the left columns, show parameters and ImageNet top-1 accruacy. We count parameters the total number of real values e.g. a complex number counts as two parameters, M2(R) and both count as 4, etc. All runs batch norm (Toffe and Szegedy 2015), unless explicitly stated. Right: For the same algebras, we compare the number of floating point operations (multiply-adds) required at inference. Unlike all previously considered algebras, for M2 (R), we find equivalent computational costs compared to real-valued networks at baseline performance. M2 (C) improves performance compared to the previously considered H with the same compute density.
Mohileket-vl Parameters
MoNieleeod FLOP
74
74
Whithn
Diagonat Whitoring Walletee-vt Basein
62
0.5
3.0
2.5
porameters
Figure 2: MobileNet-vl, Left:
We vary the width of a MobileNet-vl trained on ImageNet for subset of the considered AlgebraNets and the real-valued baseline. Right: For the same algebras, compare the number of FLOPs (multiply-addee required at inference.
algebra is extremely parameter efficient and we do find the interaction between different components through whitening to be important as hypothesized a network trained with whitening achieves 6% higher top-1 accuracy than the same network trained with batch normalization. Unfortunately, adding whitening increases the total number of inference FLOPs by a factor 3x. We are left to conclude that whitening not currently competitive and recommend using batch normalization for all algebras. Future work exploring the role of the interaction from whitening, and alternatives that are more computationally efficient is an interesting direction. An additional benefit is that AlgebraNets applied

---

Under review as conference paper at ICLR 2021
to MobileNet like architectures increase the computational density of the often bandwidth bound depthwise convolutions, while reducing the number of FLOPs the more costly convolution.
100 sccunn
Recite: 50
Nagretad: Puriing
Mudd woulls
MuAAl 405
74
parsmeters 53P
PLOPs 2007
Figure 3: Pruning ResNet-50, Left: Using magnitude pruning, prune entire tuples of M2(R)- ResNet50 to between 50 and 90% sparsity, Different widths correspond to different curves, points along each curve different sparsity levels. In green, show baseline magnitude srngnttud--puunnng results from Gale et al. 2019). Right: For the same pruned networks, we show the FLOP efficiency.
4.2 PRUNING RESNET-50
We use magnitude based pruning, with the schedule proposed (Zhu and Gupta 2017). We always begin pruning at 20% and end pruning at 80% of the total training iterations. We prune every 100 steps. At each pruning step, set tuples with the lowest magnitude, given by VI P to O. We do not prune terms in the final linear layer, the tuple-initialization convolutions, or in the initial convolution of the ResNet. To allow for comparisons with (Gale in pruning experiments, networks are shown Fig. [3 Despite pruning entire tuples, which allows skipping an entire tuple multiplication,
2019). we use ReLU activations opposed to swish as used in Fig. [ Final top-1 accuracies of pruned
still able to find sparse networks that are similarly FLOP efficient to those from Gale et al. 2019), while having higher compute density due to the algebra structure. Pruning individual components, rather than setting entire tuples to 0 does improve performance, though does not provide the same computational benefits offered by AlgebraNets. We provide further results in Appendix
4.3 TRANSFORMER XL ON ENWIK8 AND RNNs ON WIKITEXT-
We perform character level language modeling on the Enwik8 dataset from the Hutter Prize LLC 2009) with the Transformer-Xi XL (Dai et al. 2019) architecture. We tuned the baseline model by halving the embedding size, number heads and feedforward size, resulting "efficient" baseline with only 25% as many parameters as the 24 layer network from (Dai et al. 2019). train with Adam (Kingma and Ba 2014) (learning rate x 10-4), dropout 0.25 (component-wir not suple-wise). and windows of 1536 at train and 4608 at test time. The M2 -ra-ssfoffrreer uses learning rate of .0005 and dropout 0.15. Our 'efficient' baseline 24 layer Transformer-X model has embedding size of 512, 4 heads of size 128 each, and feed-forward hidden size of 1536 for total
more challenging
Name
Algebra MParams
MFL.Ps Validation BPC
Transformer-y XL 18L (see (Dai et 2019))
R
88
256 356/420
1,03
Transformer-Xe XL 24L (see (Dai et al. 2019))
R
277.4
341 568
0.99
Transformer efficient 24L (ours)
IR (2(
69.4
114 227
0.99
My Transfoofreee efficient 42L (ours)
31.2
172 399
0.99
Table Enwik-8: parameter-efficient Transformer-Xi with performance that matches the classic XI.oo on Enwik-8. Switching to M2 (eemmitssssss both increasing the number of layers and reducing parameters at the same time again while keeping the validation bits per character on par. The reported inference FLOPs are representative of incremental token employing memory length 1538 or 4608, corresponding to the training and test regimes.
Transformer-Xi XI.
7

---

Under review as a conference paper at ICLR 2021
parameter count of 69.4 million. It achieves 0.99 bits per character (BPC), matching the results of (Dai et al. 2019) while requiring 75% less parameters. Using the fixed activation size results in further 75% reduction in parameter count. We use these parameter savings to increase the depth of the model from 24 to 42 layers, resulting in a model with 45% as many parameters the 'efficient" baseline. The resulting (2) AlgebraNet also achieves 0.99 BPC, but with only 31.2 million parameters. The character-embeddingg layers are computationally unchanged; they associate each character with a 3-sized M2 embedding which can be thought of as reshaped Rd embedding. Special consideration has be paid the Rixi attention matrix, which is often regarded as a practical memory and compute bottleneck (Child et al. 2019 Roy et al. 2020) Kitaev et al., 2020). Using an / algebra valued attention matrix would increase the memory and compute requirements (e.g. by factor 2 in the Cemplex-Transforinn (R.. Thus, we desire a real valued attention matrix from the sets of -aauul key and query vectors (k: g). We do this by reshaping keys and queries from M2 (R) to R. Formally, we redefine the attentions real-valued scalar product as (k, q) M2()) == (F(k), F(4))2 where F flattens the input into a real vector.
in all linear layers with
(Ynng et al., 2020) or 4 for
Finally, also consider dataset approximately WikiText-103 (Merity et al., 2016) into characters (instead of the more common words). On this dataset we consider single layer GRU architecture followed by 5 linear readout layers with the ReLU non-linearity, skip connections and layer normalization after each layer. We train using a batch size of 16, Adam (learning rate of 10 10-4), and L2 regularization of 10-? for 200,000 steps. Training takes two days for the largest baseline variant on a single V100 GPU. We train using length 512 sequences and back propagation through time. We initialize the different components of each algebra with single linear layers from the input. We report results on the typical validation set. We replace gated recurrent unit (GRU) (Cho et al. 2014) with the AlgebraNet equivalent, as well as replacing the readout layers with AlgebraNet variants. We consider M2 (IR), C, H., and Ma Results are shown in Table 3] A C AlgebraNet with a hidden size of 1024 and 24.1 million parameters achieves a network with 1.45 times the parameter count. We find that M2 (R) with a hidden size of 512 results in a validation BPC of 1.30, comparable to a model with twice many parameters. Again, demonstrating the parameter efficiency of M2(R) and the usefulness of AlgebraNets for problems such as language modeling where parameter efficiency is crucial.
order of magnitude larger by tokenizing
validation BPC of 1. 1.26, comparable
5 CONCLUSION
Conventional neural networks are composed of real-valued weights and activations along with real valued operators. In this work, we proposed AlgebraNets, general paradigm of replacing real valued weights and operators with weights and operators from other associative algebras in general fashion. We show these methods to be more parameter efficient than their real-valued counterparts while having higher compute density. We also find that the M2 (IR) algebra is more FLOP efficient than previously considered algebras in fact it is as FLOP efficient as the reals. The increased compute density of the proposed algebras will prove particularly useful for sparse neural networks and auto-regress regressive inference, due to modern hardware favoring relatively high compute density. We hope that our work enables further development of these methods and promotes broader research into the fundamental design choices upon which modern neural networks are based.
Algebra
Size
MParams
MFLOPs
BPC
Algebra
Size
MParams
MFLOPs
BPC
R 1024
12.1
23.6
1.34
C
1024
24.1
80.2
1.26
R 1280
18.7
36.7
1.32
C
640
9.68
31.5
1.32
R 1536
27.8
52.7
1.29
M2
512
12.6
40.1
1.30
R 1768
35.3
69.7
1.25
IH
512
12.6
73.6
1.32
R 2048
47.2
93.3
1.24
M2
384
16.4
71.9
1.31
Table WikiText-103; We replace a real-valued GRU (and the corresponding linear layers) with AlgebraNet counterparts. We report the minimum validation loss over the last 5% of training. The hidden size is reported as number of tuples, e.g. MR(RR with 512 tuples has 2048 scalars in total.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequenees with sparse transformers, 2019.
Kyunghyun Cho, Bart Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
James Cockle. On systems of algebra involving more than one imaginary. Philossopicaal Magazine (series 3), 35, 1849.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xih Attentive language models beyond a fixed-length context. Proceedings of the 57th Annual Meeting of the Association Computational Linguistics, 2019.
Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves. Associative long short-term memory, 2016.
Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Uiku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro. and Erich Elsen. Rigging the lottery: Making all tickets winners, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.
Chase J. Gaudet and Authony Maida. Deep quaternion networks. In 2018 International Joint Conference on Neural Networks (IJCNN, pages 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256, 2010.
William Rowan Hamilton. On quaternions; or on new system of imaginaries in algebra. Private Correspondence, 1843.
William Rowan Hamilton. Theory of quaternions. Proceedings of the Royal Irish Academy, 1844.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Geoffrey Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. International Conference on Learning Representations, 2018.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 7704 744404461, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift, 2015.
Siddhant M. Jayakumar, Wojciech M. Czameeki, Jacob Menick, Jonathan Schwarz, Jack Rac, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer, 2020.
MultiMedia LLC. Large text compression benchmark, 2009.
Christos Louizos, Max Welling, and Diederik Kingma. Learning sparse neural networks through L_0 regularization. International Conference on Learning Representations, 2018.
Stephen Merity, Caiming Xiong, James Bradhury, and Richard Sacher. Pointer sentinel mixture models, 2016.

---

Under review as a conference paper at ICLR 2021
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. ArXiv, abs/1701.05366 2017.
Johannes Sebastian Bweelle--Remeer, André Stork, and Dieter W. Fellner. Joint Schedule and Layout Autotuning for Sparse Matrices with Compound Entries on GPUs. Hans-Jörg Schulz, Matthias Teschner. and Michael Wimmer, editors, Vision, Modeling and Visualization. The Furographics Association, 2019. ISBN 178-3-03888-098-1 doi:
Nate Oh. The nvidia titan v deep learning deep dive: It's all about the tensor cores, 2019. URL https / com/ http:/hhwww/0/ httpm//////////taaaaa--eeeeeeeee
Xingang Pan, Xiaohang Zhan, Jianping Shi, Xiaoou Tang, and Ping Luo. Switchable whitening for deep
representation learning. 2019 JEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019.
Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi, Renato De Mori, and Yoshua Bengio. Quaternion recurrent neural networks, 2018.
William H. Press, Saul A. Teukolsky, William Vetterling, and Brian P. Flannery. Numerical Recipes 3rd Edition: The Art of Scientific Computing. Cambridge University Press, USA, 3 edition, 2007. ISBN 052 1880688.
Prajit Ramachandran, Barret Zoph, and Quoe V. Le. Searching for activation functions, 2017.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers, 2020.
Olga Russakovsky, Jia Deng. Hao Su. Jonathan Krause. Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICV)), 115(3):211 1-252, 2015, doi: 10.1007/st 1263-015-0816-
Chiheb Trabelsi, Olexa Bilaniuk. Ying Zhang, Dmitrly Serdyuk. Sandeep Subramanian, Joao Felipe Santos. Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks. In International Conference on Learning Representations, 2018.
Riccardo Vecchi, Simone Scardapane, Danilo Comminiello, and Aurelio Uncini. Compressing deep quaternion neural networks with targeted regularization. CoRR, 2019.
Wikipedia contributors. Algebra over a field Wikipedia, the free encyclopedia, 2020a. URL http:t/ps://en [Online; accessed
www.oo/////../p////eeeoo wttp://wwwwpppppt......e..ettto. 90-May-2020,
Wikipedia contributors. Vector space Wikipedia, free encyclopedia, 2020b. URL https: / /n
wikipedia. http:ppppppp..tt..eeeeooooeeeeea 2020).
[Online; accessed 30-May-
Daniel E. Worrall, Stephan J. Garbin, Daniyar TTurmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance, 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), Jul 2017.
Jiasong Wu, Ling Xu, Fuzhi Wu, Youyong Kong, Lotfi Senhadji, and Huazhong Shu. Deep octonion networks. Neurocomputing, 2020.
Muqiao Yang, Dongyu Li Martin Q Ma. Yao-Hung Hubert Tsai, and Ruslan Salakhutdinov. Complex transformer: A framework for modeling complex-valued sequence. CASSP 2020-2020 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
Michael Zhu and Suyog Gupta. To prune, not prune: exploring the efficacy of pruning for model compression, 2017.
10

---

Under review as conference paper at ICLR 2021
A ADDITIONAL ALGEBRA INFORMATION
A.1 M2(R) MULTIPLICATION TABLE
M2
a
b
0
Each tuple (ta; fo tey ta) represents a x real matrix.
0 0
d
0
A.2 Mz(C) MULTIPLICATION TABLE
to to
te tat
Each tuple (tas to tes ta ter &g tog ta) represents the 2 complex matrix:
Le
+
Lyz
+ 42
C2(C)
b
d
s
9
b
d
0
0
0
b
-a
-
0
0
0
0
0
b
c
d
d
d
0
0
0
-a
0
e
g
h
0
f
f
h
-9
0
0
g
0
0
f
9
h
0
0
0
f
-
h
A.3
DUAL NUMBER MULTIPLICATION TABLE
Each tuple (ta, fo) represents the dual number (fa + tge)-
Dual Number
a b 0
b
A.4
CROSS PRODUCT MULTIPLICATION TABLE
Multiplicated uses the cross product between length-3 tuples (fa, fs. fc).
Cross Product
0
-b
-c
a
b
-a 0
A.5 LINEAR LAYER EXAMPLE
We give concrete example of replacing real linear layer with M2 (RR--iina layer such that the activation memory kept identical. Intuitively, this can be thought of as reshaping R input activations to have shape My (R)4/4 that is processed by fM (R)4///4 when flattened with shape Rd. Each such lincar layer fas requires the parameters and of the FLOPS compared real Idd Ed linear layer counterpart.
M2 linear layer resulting in output activations
ALGEBRANET CHOICES: ACTIVATIONS, INITIALIZATIONS, ETC
B.I TUPLE-WISE NONLINEARITY
We consider equations the form:
f(g(t))
11

---

Under review as conference paper at ICLR 2021
We found that if y is the tpple mean, and f' H. the Heaviside function, top-1 performance dropped (R) ResNet-50 AlgebraNet 2.97%. While this drop significant, the resulting activation sparsity might make desirable tradeoff in some circumstances. Other methods, such as setting to the determinant resulted in greater than 10% drop in performance.
B.2 INTTIALIZATION
ResNet-50 HI-AlgebraNet standard number of channels divided by 4, we find a top-1 performance of 74.0 + 0.14 using standard initialization and 74.1 = 0.15 using initialization from Gaudet and Maida (2018). These experiments are done using standard batch normalization instead of the more expensive whitening procedure.
B.3 CONVERSION TO REALS
For all considered algebras, the norm the tuple is mathematically given by VE, 1?. It is possible that the optimal choice for converting to the reals would be different in models with very large final layers, such word based language modeling which do not consider.
C ALGEBRANET PRUNING
C.1 ALTERNATIVE TUPLE PRUNING OF (R)
For (R). we consider a variety of alternative pruning methods to remove entire tuples, based on the two eigenvalues, A1 and A2 and singular values, 01, Specifically, because matrices are square but not symmetric, the Forbenius norm is defined based on the singular values which correspond to the squared eigenvalues of AA", A is the matrix question.
Frobenius Norm: (of + 0z)
Determinant: A1
Smallest Figenvalue min((A1l, [Aa)
Largest Eigenvalue max (/1) [Az]]
In all cases, remove tuples with the minimum magnitude of one of those options.
Sparsity
det
min
max
50
-0.27
-0.76
-0.02
70
-1.26
-1.71
-0.57
90
-2.69
-3.558
-0.73
Table 4: For 50%, 70%, and 90% sparsity, we show the performance relative to the Frobenius norm for different magnitude-based tuple pruning criterion.
Table we show the resulting drop in top-1 accuracy relative to Frobenius norm three different sparsities for three alternative pruning methods. In addition to always achieving the best performance, the Frobenius norm has the additional advantage that it is defined for all Algebra variants that was consider, rather than M.. specific variant, for example.
C.2 PRUNING COMPONENTS OF MR( AND H
For M2 (IK) and H., we also prune individual tuple elements based on element norms. This equally reduces the number of non-zero weights in the network, though does not result in entire matrix multiplies that can be skipped.
12

---

Under review as conference paper at ICLR 2021
Sparsity
M2(RR)
+0.20%
+0.18%
60
+0.37%
+0.32%
70
+0.58%
+0.49%
80
+0.93%
+0.74
90
+2.13%
+1.64 %
Table 5: Performance different from pruning components and entire tuples for M2(R) and Hi-AlgehraNets. Depending on the size of the network, the difference between the methods varies slightly. The main point that pruning elements rather than tuples increases performance, more-so higher sparsities.
In 5 show the resulting increase in top-1 accuracy that results from pruning individual tuple components, rather than entire tuples. However, due the structure M, (R) and H multiplication, setting individual values not result 0 output. Therefore, pruning entire tuples provides more useful computational advantages.
D ALGEBRANET TESTS ON CIFAR
We use network structure based on that described in in[Gaueet and Maida (2018). We begin with the same ResNet structure, with 128, 256, and then 512 channels in each real block. For the C' networks, all channel counts divided hy For the M2(R) and H networks, have half original number channels, all other channel counts are divided by four. Thus, for and M2) we have slightly more than 1/4 the parameters. We train with 24 x 24 random crops and evaluate on 32 x 32 images.
assign the initial convolution, before the residual blacks,
Algebra
Parameters (x10%
FLOPSs (x10%
top-1
3,64
32.8
94.2
C BN
1.85
33.6
94.2
C W
1.86
50.7
94.3
M2 R
0.94
20.7
94.3
H BN
0.94
41.4
93,4
H W
0.97
72.9
94.1
Table A comparison different AlgehraNets on CIFAR-10. BN denotes Batch Normalization, denotes the use of whitening.
We find able divide the channels in the filter valued networks. When reducing the parameter count by a factor of four,, we we are able again match baseline performance with quaternions and 2 x matrices. Regularization has non-trivial effect on performance, and by more finely adjusting the L2 loss for the different algebras may result in higher top-1 accuracy. We note that the relative reduction parameters CIFAR-I0 results from main text also hold here M2 (I) is only algebra that having fewer FLOPs than the baseline real network. For these experiments, we used algebra specific weight initializations, though we again verified that this does not seem to have substantial effect.
and maintain the same performance using complex
something
able replicate ImageNet. to maintain accuracy while
E EXAMPLE M2 [R) CODE
We write the update rule explicitly for readability. Note that is possible to concatenate the relevant terms the channel axis reduce the number of convolutions needed.
CONVOLUTION
Simplified example code for M_2(R).
x: Input with an addi tional algebra axis. the case of a convolution, either (B, H, W, C, A) or (B, C, H, W, A)
Corresponding weight matrix, with addi addalional
13

---

Under review as conference paper at ICLR 2021
algebra axis.
Rule that describes 2x2 matrix multiplication.
mat 22_r [[(O, 0), (1, 2)],
[(0, 1), (1, 3)1,
[(2, 0), (3, 2)].
[(2, 1), (3, 3)]]
Update each of the four algebra components.
new [0. 9, 0] for i in ra)ge(4): for in range(2):
w: weight with extra algebra dimension.
Input with shape [B, A] where is the additional algebra dimension. x_new[i] += Conv2D(x mat_22_rule(ii] [i][1],
w[..., mat mat_22_rulle [i]]000
Add bias if wanted. Add (4.) shape.
LINEAR LAYER
Update each of the four algebra components. x_new [0. 0, 0] for i in ra)ge(4):
for in range(2):
weight with extra algebra dimension.
x: Input with shape [B, A] where is the algebra dimension. x_new[i] += dot mat_22_rule(in [j][1],
w[..., mat mat_22_rule[1 [j]
Add bias if wanted. Add (4,) to shape.
INCREASED ACTIVATION MEMORY
Due to the activations, there will be slight increase memory footprint from AlgebraNets in cases. For example, (R) AlgebraNet for ResNet-50 with channels/4, there will he C/4 convolutions performed. This would, in naive implementation, result twice the activation memory. However, with a properly written kernel, this would not be the case. There is, however, an additional factor: to reach comparable performance slightly larger network than C/4 is needed. In practice about a 1.3x increase in activation memory would be incurred.
14


Reviews:
[{"review_id": "kYdrqZBz_b", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose the usage of complex numbers in deep neural networks. Would be good to know that complex numbers, n x n matrices, quaternions, diagonal matrices, etc. all can be used in neural networks. The authors also claims benchmark performance in large-scale image classification and language modeling.\n\nHowever, this work cannot be appreciated due to the following aspects:\n1. A first question is \"Why it is necessary?\"  Interestingly, the authors already included Section 2.1 Why Algebras?   However,  I am not convinced at all.  A good answer may take either of the two forms: A). simply a math step showing great potential behind;  2) large-scale neural networks that have engineering advantages.  It seems that the authors took the second approach, however, ImageNet is not that challenging and there may be no clear need to switch to complex numbers.  Would the authors be able to justify this?\n\n2. Then, the authors directly go to evaluations. The figures seem to show good advantages.  However, could you please justify your x,y-axis?  The reported results look high biased. As a reviewer, I have to doubt that the authors may have selectively present their results. \n\n    A good research paper on such a big topic, should give clear methodology first, right?  If the methodology is questionable, such good results may become noise to the community.\n    I would hope the authors clarify their methodology, and then present that advantages obtained in the experiments.\n\n3. As a top AI conference, I believe that we are looking for intellectual contributions.\n    This paper is working on a huge title, which is attractive. However, when I try to identify the intellectual contributions (can be theory, algorithm, engineering, applications), I am not convinced at all.  I know such a topic is not easy to handle. I would simple ask the authors to respond to a direct question: how would like the community to appreciate your work?\n\nNOTE: a lot of disputes are around \"the huge title 'AlgebraNets'\". However, I did not receive justification response from the authors. A possible reason may be the authors are not aware of how big the topic it is, and were so attractive/confident in the current experimental improvements (which is also very appreciated).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "k3b56qlIUhz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe authors propose AlgebraNets - a previously explored approach to replace real-valued algebra in deep learning models with other associative algebras that include 2x2 matrices over real and complex numbers. They provide a comprehensive overview of prior methods in this direction and motivate their work with potential for both parameter and computational efficiency, and suggest that the latter is typically overlooked in prior literature. The paper is very well-written and follows a nice narrative, and the claims are mostly backed empirically with experimental results. \n## Pros \n* Empirically justified with experiments on state-of-the-art benchmarks in both computer vision and NLP. \n* Establishes that exploring other algebras is not just an exercise for mathematical curiosity but also practical, and encourages deep learning practitioners to extend the results. \n* Perhaps the most useful aspect is that the experiments fit well into a standard deep learning framework – with conventional operations, initialization, etc. That is, the algebras do not require significant custom ops/modifications to achieve state-of-the-art results. \n* Shows multiplicative efficiency (parameter count and FLOPs) in many cases \n## Cons \n* The authors motivate this work with computational efficiency; however, I did not find any discussion or comments on the total memory footprint. Do any of the algebras require us to keep track of partial computations/intermediate steps - subsequently increasing the total memory footprint? In the case of vision examples, which are dominated by the activations, what are the implications? If the memory footprint is indeed not consistent with a real-valued algebra, then are we trading model/input size for fewer parameters/efficient computation?\n* An intuitive justification of the algebras used in these experiments, along with insight for future algebras might be a nice addition, although I wouldn't consider it a con.\n* Are certain algebras more amenable to specific hardware architectures? If so, a brief discussion would enhance the paper overall.", "extra_scores": {}, "extra_text": {}}, {"review_id": "b13emmFhpZs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an interesting kind of networks, AlgebraNets, which is a general paradigm of replacing the commonly used real-valued algebra with other associative algebras. This paper considers C, H, M2(R) (the set of 2 × 2 real-valued matrices), M2(C), M3(R), M4(R), dual numbers, and the R3 cross product, and investigates the sparsity within AlgebraNets. \n\nThe work in the paper is interesting and this paper is generally written well. However, there are a few issues/comments with the work:\n\n1.The citation of the references in the main body of this paper is not easy to read. It will be better to replace the format “author(s) (year)” with the format “(author(s), year)” ;\n\n2.Some figures and tables do not appear near the discussion, for example, Figure 1 is shown on Page but it is discussed until page 5, which makes it difficult to read;\n\n3.In Figure 1, the subfigure in the second row and first column, it seems that the performance of model with H and whitening the best stable performance.  The subfigure in the second row and second column, it can be seen that the model with H  is not better than the baseline model;\n\n4.There are many inconsistencies in the format of the reference, for example,\n\n1)In some places the author's name is abbreviated, while in others it is not. References “C. J. Gaudet and A. S. Maida. Deep quaternion networks. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2018. ” and “Geoffrey E. Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In ICLR, 2018. ”;\n\n2)In some places the conference’s name is abbreviated with the link, while in others it is not. References “Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=rylnK6VtDH.” and “Geoffrey E. Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In ICLR, 2018. ”.\n\nPlease check carefully and correct the inconsistencies.\n\n+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n\nThe paper replaces the traditional real-valued algebra with other associative algebras and shows its parameter and FLOP efficiency.  In the beginning, \"I think it is an interesting piece of work, and it may be helpful to develop the basic structural design of neural networks. \". However, after getting the response from the author(s), I more doubt the significance of the work in this paper: although many types of models have been proposed in this paper, the improvement over the baseline models is limited. I did not lower the grade on this paper since I thought it would be interesting and important (if effective) to extend the traditional real number field to more complex algebraic structures.\n\n+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to deep neural network models with elements of the weight from algebras, and considers a wide range of algebras and large scale promising experiments. The paper raised a heated discussion.

Pros: 

- Using algebras, one can hope for more efficient architectures 

- Numerical experiments on a wide range of problems

Cons:

 - The theoretical grounding provided in the current version of the paper is not sufficient. The study is empirical (nothing wrong about it), but there is no clear understanding/explanation of why particular choice is better than another, and also why it works in the particular setup. 

- The title does not reflect the content of the paper. It is too broad, and also in some sense “provocative”. The reader expects something much more significant from it.

- Experiment setup: the resulting flops/accuracy figure (main result, Figure 1) does not contain error bars.  I.e., the accuracies should be averaged over several random seeds in order to guarantee the resulting metrics. Also, this figure does not show a clear advantage over the ResNet-50 baseline.
Decision: Reject