Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper builds on the top of standard high-dimensional neural networks (limited to 2,4,8, 16 dimensions) by introducing an elegant way to deal with others dimensions while preserving the internal relation learning capability as well as the reduction of number of parameters. To do so, the linear transformation is turned into a new linear transformation based on the Kronecker product. A sum of Kronecker products is used to "simulate" the different internal relations that could occur in between multi-dimensional components. However, I think that an empirical validation of the ability of the method to recover well-know product, i.e. Hamilton / complex products, is missing. 

According to the results, with experiments conducted on 3 different tasks, the proposed approach definitely seems to work and is an important step further to better understand and manipulate high-dimensional algebras with deep neural networks.

Remarks and questions:
1. While the theoretical aspect of the degeneration of the proposed approach to the Hamilton product "sounds" plausible, I would like to see a more empirical demonstration. Can this approach, correctly parametrize a rotation in a 3D space (simple task of learning a single rotation of an object). 
2. The quaternion case of this method relies on a set of [-1,0,1] to build the S matrices. This is what ensures the geometrical properties of the quaternion space while doing the different manipulations. This method, however, uses real-valued matrices. It is thus almost certain that the geometrical aspect of the "learnt" algebra is impossible to interpret. Do the authors think that a quantisation of the matrices could help finding pure quaternion / complex / octonions / sedenions matrices ? Such an analysis is crucial to validate the fact that the work proposed here is a generalisation of what have been done before to N dimensions for neural networks. 

- Would be great to include the number of parameters in the different Table.
review_title: Kronecker product as a way to parametrize high-dimensional products

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose a novel way of parametrizing hypercomplex multiplications. 
The proposed parametrization helps with: (a) Generalizing the multiplication to arbitrary dimensions, and (b) Reducing the number of parameters.  
Building on this, the authors propose a parameterized hypercomplex multiplication (PHM) layer which essentially replaces the weight matrix of a linear layer with a matrix constructed via sum of Kronecker products. 
They then replace the weight matrix of linear layers in LSTM and Transformer with PHM.
Finally, they show that these PHM-variants of LSTM and Transformer, match or outperform their vanilla counterparts on a variety of NLP tasks, including NLI, MT, text style transfer, etc.,  while reducing the total number of model parameters.  

Overall the paper is quite well written with easy to follow illustrations. 
The proposed parametrization seems reasonable, and the empirical validation lends solid credibility to the idea. 
I have a couple of questions for the authors:
* What are the practical benefits of this parametrization, particularly in comparison to other ways of reducing parameters, say matrix factorization?
* If hypercomplex spaces only exist in $2^k$-D, is the term hypercomplex justified for arbitrary dimensions?
* Have you analyzed the learned hypercomplex spaces? Can they be interpreted for arbitrary dimensions?
review_title: Interesting idea with practical benefits

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors focus on the area of using hypercomplex multiplications (multiplications involving numbers with multiple imaginary components) in deep learning models. Past work in this area has been promising but has been limited to certain dimensions for which there are predefined multiplication operations. The novel contribution of this work is to parameterize the hypercomplex multiplication operations, enabling the model to discover new operations rather than relying on the small number of existing operations and the small number of dimensions for which such operations exist. The authors find that their approach can substantially reduce the number of parameters without reducing performance (and in some cases even improving performance). 

Strengths:

1. The proposed method makes a promising approach from the literature more flexible, helping to pave the way for making this approach more broadly useful.

2. The authors illustrate this flexibility by showing how their approach can be effective for two different architectures (LSTMs and Transformers), making the general point that it can be applied to any architecture that uses feedforward components. They also apply it to multiple tasks, again illustrating the flexibility.

3. As mentioned above, the approach can substantially increase a model’s parameter count without affecting performance. Relatedly, it can also improve inference speed.

4. The paper is generally thorough and clear.

Weaknesses:

1. The specific contribution of this paper is the parameterization of the multiplication operation, but the evidence that this parameterization is helpful is mild, as there are only a few cases where the proposed model noticeably outperforms the Quaternion model. Thus, the evidence presented does not make a strong case for the necessity of this parameterization.

2. Much of the argument hinges on the reduced parameter count, but there was not any mention of exactly how many parameters each model had (at least, not that I saw - I did not check the appendix). I think the paper could be substantially strengthened by adding a “Parameter count” column to each table.

3. There is no clear intuition offered for why this approach might be expected to be effective. Offering such an intuition is certainly not necessary (since results alone are enough), but the paper would be more satisfying if there were such an intuition present.

Overall, I am rating this as a 7, because I find it to be a solid paper but worry that its contribution on top of the existing work that has studied hypercomplex operations may be too small and may not have enough evidence for its usefulness.

review_title: Solid contribution toward making hypercomplex operations more flexible

Meta-review: The authors propose a new parameterization which (across multiple architectures) generalized hypercomplex multiplication and provides for small low dimensions strong performance at substantial parameter savings. All reviewers are happy with the theoretical contributions of the work, but would appreciate additional empirical evidence.
Decision: Accept (Spotlight)