Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose the usage of complex numbers in deep neural networks. Would be good to know that complex numbers, n x n matrices, quaternions, diagonal matrices, etc. all can be used in neural networks. The authors also claims benchmark performance in large-scale image classification and language modeling.

However, this work cannot be appreciated due to the following aspects:
1. A first question is "Why it is necessary?"  Interestingly, the authors already included Section 2.1 Why Algebras?   However,  I am not convinced at all.  A good answer may take either of the two forms: A). simply a math step showing great potential behind;  2) large-scale neural networks that have engineering advantages.  It seems that the authors took the second approach, however, ImageNet is not that challenging and there may be no clear need to switch to complex numbers.  Would the authors be able to justify this?

2. Then, the authors directly go to evaluations. The figures seem to show good advantages.  However, could you please justify your x,y-axis?  The reported results look high biased. As a reviewer, I have to doubt that the authors may have selectively present their results. 

    A good research paper on such a big topic, should give clear methodology first, right?  If the methodology is questionable, such good results may become noise to the community.
    I would hope the authors clarify their methodology, and then present that advantages obtained in the experiments.

3. As a top AI conference, I believe that we are looking for intellectual contributions.
    This paper is working on a huge title, which is attractive. However, when I try to identify the intellectual contributions (can be theory, algorithm, engineering, applications), I am not convinced at all.  I know such a topic is not easy to handle. I would simple ask the authors to respond to a direct question: how would like the community to appreciate your work?

NOTE: a lot of disputes are around "the huge title 'AlgebraNets'". However, I did not receive justification response from the authors. A possible reason may be the authors are not aware of how big the topic it is, and were so attractive/confident in the current experimental improvements (which is also very appreciated).


review_title: Huge title without convincing contribution

final_score: rating: 7: Good paper, accept
review_content: ## Summary
The authors propose AlgebraNets - a previously explored approach to replace real-valued algebra in deep learning models with other associative algebras that include 2x2 matrices over real and complex numbers. They provide a comprehensive overview of prior methods in this direction and motivate their work with potential for both parameter and computational efficiency, and suggest that the latter is typically overlooked in prior literature. The paper is very well-written and follows a nice narrative, and the claims are mostly backed empirically with experimental results. 
## Pros 
* Empirically justified with experiments on state-of-the-art benchmarks in both computer vision and NLP. 
* Establishes that exploring other algebras is not just an exercise for mathematical curiosity but also practical, and encourages deep learning practitioners to extend the results. 
* Perhaps the most useful aspect is that the experiments fit well into a standard deep learning framework – with conventional operations, initialization, etc. That is, the algebras do not require significant custom ops/modifications to achieve state-of-the-art results. 
* Shows multiplicative efficiency (parameter count and FLOPs) in many cases 
## Cons 
* The authors motivate this work with computational efficiency; however, I did not find any discussion or comments on the total memory footprint. Do any of the algebras require us to keep track of partial computations/intermediate steps - subsequently increasing the total memory footprint? In the case of vision examples, which are dominated by the activations, what are the implications? If the memory footprint is indeed not consistent with a real-valued algebra, then are we trading model/input size for fewer parameters/efficient computation?
* An intuitive justification of the algebras used in these experiments, along with insight for future algebras might be a nice addition, although I wouldn't consider it a con.
* Are certain algebras more amenable to specific hardware architectures? If so, a brief discussion would enhance the paper overall.
review_title: Impactful paper with strong empirical results

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an interesting kind of networks, AlgebraNets, which is a general paradigm of replacing the commonly used real-valued algebra with other associative algebras. This paper considers C, H, M2(R) (the set of 2 × 2 real-valued matrices), M2(C), M3(R), M4(R), dual numbers, and the R3 cross product, and investigates the sparsity within AlgebraNets. 

The work in the paper is interesting and this paper is generally written well. However, there are a few issues/comments with the work:

1.The citation of the references in the main body of this paper is not easy to read. It will be better to replace the format “author(s) (year)” with the format “(author(s), year)” ;

2.Some figures and tables do not appear near the discussion, for example, Figure 1 is shown on Page but it is discussed until page 5, which makes it difficult to read;

3.In Figure 1, the subfigure in the second row and first column, it seems that the performance of model with H and whitening the best stable performance.  The subfigure in the second row and second column, it can be seen that the model with H  is not better than the baseline model;

4.There are many inconsistencies in the format of the reference, for example,

1)In some places the author's name is abbreviated, while in others it is not. References “C. J. Gaudet and A. S. Maida. Deep quaternion networks. In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2018. ” and “Geoffrey E. Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In ICLR, 2018. ”;

2)In some places the conference’s name is abbreviated with the link, while in others it is not. References “Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=rylnK6VtDH.” and “Geoffrey E. Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In ICLR, 2018. ”.

Please check carefully and correct the inconsistencies.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

The paper replaces the traditional real-valued algebra with other associative algebras and shows its parameter and FLOP efficiency.  In the beginning, "I think it is an interesting piece of work, and it may be helpful to develop the basic structural design of neural networks. ". However, after getting the response from the author(s), I more doubt the significance of the work in this paper: although many types of models have been proposed in this paper, the improvement over the baseline models is limited. I did not lower the grade on this paper since I thought it would be interesting and important (if effective) to extend the traditional real number field to more complex algebraic structures.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
review_title: Interesting study of replacing the traditional real-valued algebra with other associative algebras

Meta-review: The paper proposes to deep neural network models with elements of the weight from algebras, and considers a wide range of algebras and large scale promising experiments. The paper raised a heated discussion.

Pros: 

- Using algebras, one can hope for more efficient architectures 

- Numerical experiments on a wide range of problems

Cons:

 - The theoretical grounding provided in the current version of the paper is not sufficient. The study is empirical (nothing wrong about it), but there is no clear understanding/explanation of why particular choice is better than another, and also why it works in the particular setup. 

- The title does not reflect the content of the paper. It is too broad, and also in some sense “provocative”. The reader expects something much more significant from it.

- Experiment setup: the resulting flops/accuracy figure (main result, Figure 1) does not contain error bars.  I.e., the accuracies should be averaged over several random seeds in order to guarantee the resulting metrics. Also, this figure does not show a clear advantage over the ResNet-50 baseline.
Decision: Reject