Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters
Authors: Aston Zhang
Yi Tay
SHUAI Zhang
Alvin Chan
Anh Tuan Luu
Siu Hui
Jie Fu
Venue: ICLR 2021 Spotlight (2021)
Keywords: hypercomplex representation learning

Abstract: Recent works have demonstrated reasonable success of representation learning in hypercomplex space. Specifically, “fully-connected layers with quaternions” (quaternions are 4D hypercomplex numbers), which replace real-valued matrix multiplications in fully-connected layers with Hamilton products of quaternions, both enjoy parameter savings with only 1/4 learnable parameters and achieve comparable performance in various applications. However, one key caveat is that hypercomplex space only exists at very few predefined dimensions (4D, 8D, and 16D). This restricts the flexibility of models that leverage hypercomplex multiplications. To this end, we propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predefined. As a result, our method not only subsumes the Hamilton product, but also learns to operate on any arbitrary $n$D hypercomplex space, providing more architectural flexibility using arbitrarily $1/n$ learnable parameters compared with the fully-connected layer counterpart. Experiments of applications to the LSTM and transformer models on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architectural flexibility and effectiveness of the proposed approach.

Published as a conference paper at ICLR 2021
BEYOND FULLY-CONNECTED LAYERS WITH QUATERNIONS: PARAMETERIZATION OF HYPERCOM- PLEX MULTIPLICATIONS WITH 1/n PARAMETERS
Aston Zhang", Yi pay';Shual Zhang", Alvin Chan Anh Tuan Lu Siu Cheung Hui", Jie Fu"
Amazon Web Services Al
Google Research
"ETH Zürich
NTU, Singapore
VinAI
Mila, Université de Montréal
azzaa z0aastonzaang.com com
ABSTRACT
Recent works have demonstrated reasonable success of representation learning in hypercomplex space. Specifically, "fully-connected layers with quaternions" (quaternions are 4D hypercomplex numbers), which replace real-valued matrix multiplications fully-connected layers with Hamilton products of quaternions, both enjoy parameter savings with only 1/4 learnable parameters and achieve comparable performance in various applications. However, one key caveat is that hypercomplex space only exists very few predefined dimensions (4D, 8D) and 16D). This restricts the flexibility of models that leverage hypercomplex multipli- cations. To this end, propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predefined. As result, our method not only subsumes the Hamil- ton product, but also learns to operate on any arbitrary nD hypercomplex space, providing more architectural flexibility using arbitrarily 1/n learnable parameters compared with the fully-connected layer counterpart. Experiments of applica- tions to the L.STM and transformer models on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architec- tural flexibility and effectiveness of the proposed approach.
INTRODUCTION
quaternion is a 4D hypercomplex number with one real component and three imaginary com- ponents. The Hamilton product is the hypercomplex multiplication of two quaternions. Recent works quaternion space and Hamilton products have demonstrated reasonable success (Parcollet et al. 2018b 2019 TTy et al. (019).. Notably, the Hamilton product enjoys parameter saving with 1/4 learnable parameters as compared with the real-valued matrix multiplication. also enables effective representation learning by modeling interactions between real and imaginary components.
One the attractive properties quaternion models its high applicability and universal use- fulness to of the most ubiquitous layers in deep learning, i.e., the fully-connected (or feed- forward) layer. Specifically, "fully-connectee layers with quaternions" replace real-valued matrix multiplications in fully-connected layers with Hamilton products of quaternions, enjoying param- eter savings with only 1/4 learnable parameters and achieving comparable performance with their fully-connected layer counterparts (Parcollet et al. 2018b 2019 Tay et al. 2019).
The fully-connected layer is one of the most dominant components in existing deep learning litera- ture (Goodfellow al. 2016 Zhang al. 2020). Its pervasiveness cannot be understated, given its
"Wrk was done NTU.

---

Published as a conference paper at ICLR 2021
centrality to many building blocks in neural network research. Given widespread adoptions of fully-connected layers, c.g., within LSTM networks (Hochreiter & Schmidhuber 1997) and trans- former models (Vaswani et al. 2017), having flexibility to balance between parameter savings and effectiveness could be extremely y useful to many real-world applications.
Unfortunately, hypercomplex space only exists 4D (quaternions), 8D (octonions), and 16D (sede- nions), which generalizes the 2D complex space (Rishiyur (006). Moreover, custom operators are required at each hypercomplex dimensionality. For instance, the Hamilton product is the hypercom- plex multiplication in 4D hypercomplex space. Thus, no operator in such predefined hypercomplex space suitable for applications that prefer reducing parameters 1/n, where n # 4, 8, 16.
In view of the architectural limitation due to the very few choices of those existing hypercom- plex space, we propose parameterization of hypercomplex multiplications, i.c., learning the real and imaginary component interactions from data in differentiable fashion. Essentially, our method can operate on an arbitrary nD hypercomplex space, aside from subsuming those predefined hy- percomplex multiplication rules, facilitating using up to arbitrarily 1/n learnable parameters while maintaining expressiveness. In practice, the hyperparameter n can be flexibly specified or tuned by users based on applications.
Concretely, our prime contribution is new module that parameterizes and generalizes the hyper- complex multiplication by learning the real and imaginary component interactions, i.c., multiplica- tion rules, from data. Our method, which we call the parameterized hypercomplex multiplication layer, is characterized by sum of Kronecker products that generalize the vector outer products to higher dimensions models (the LSTM and transformer) with our proposed method. We conduct extensive experiments on different tasks, i.e., natural language inference for LSTM networks and machine translation for transformer models. Additionally, we perform further experiments on text style transfer and subject verb agreement tasks. All in all, our method has demonstrated architectural flexibility through dif- ferent experimental settings, where it generally can use a fraction of the learnable parameters with minimal degradation or slight improvement in performance.
real space. To demonstrate applicability, we equip two well-established
The overall contributions of this work are summarized follows:
We propose a new parameterization of hypercomplex multiplications: the parameterized hypercomplex multiplication (PHM) layer. This layer has 1/n learnable parameters com- pared with the fully-connected layer counterpart, where n can be flexibly specified by users. The key idea behind PHM layers is to learn the interactions between real and imaginary components, i.e., multiplication rules, from data using a sum of Kronecker products.
We demonstrate the applicability of the PHM layers by leveraging them in two dominant neural architectures: the LSTM and transformer models.
We empirically show architectural flexibility and effectiveness of PHM layers by conduct- ing extensive experiments on five natural language inference tasks, seven machine transla- tion datasets, together with text style transfer and subject verb agreement tasks.
2 BACKGROUND ON QUATERNIONS AND HAMILTON PRODUCTS
We begin by introducing the background for the rest of the paper. Concretely, we describe quaternion algebra along with Hamilton products, which the heart of our proposed approach.
Quaternion A quaternion Q E H. is hypercomplex number with one real component and three imaginary components as follows:
Q = Q + Qui +
(2.1)
whereby ijk = k2 1. In 2.1 noncommutative multiplication rules hold: ij =
kjjk = i, ki = j.ji= -, kj = -, ik = -. Here, Q, is the real component, Qui Qy Qz are real
numbers that represent the imaginary components of the quaternion Q.
Addition
The addition of two quaternions is defined as Q + P Q, + P, + (Q. + P.)i + (Q, P,Mi + (Q= P.)k, where Q and P with subscripts denote the real and imaginary components of quaternions and P.
2

---

Published as a conference paper at ICLR 2021
Scalar Multiplication
Any scalar multiplies across all the components:
aQ = aQr + aQ2i aQui + aQ2k.
Hamilton Product
The Hamilton product, which represents the multiplication of two quaternions
Q and P. defined as
8 P = -Q.P, + 0::-0-0-------0000--0-------000-
k. (2.2)
The multiplication rule in (2.2 forges interactions between real and imaginary components of Q and P. The benefits of Hamilton products have been demonstrated in recent works where the matrix multiplication in uullycconnected layers is replaced with the Hamilton product: this reduces 75% parameters with comparable performance (Parcollet et al. 2018b 2019;
Tay et al. 2019).
PARAMETERIZATION OF HYPERCOMPLEX MULTIPLICATIONS
The following introduces our proposed parameterized hypercomplex multiplication layer and elab- orates how it parameterizes and generalizes multiplications in hypercomplex space, such sub- suming the multiplication rules of Hamilton products in (2.2).
3.1 FULLY-CONNECTED (FC) LAYERS
Before we delve into proposed method, recall the fully-connected (FC) layer that transforms input x E IRd into an output y E I by
y FC(x Wx
(3.1)
where the weight matrix of parameters W Rxd and the bias vector of parameters E R*
The FC layer in (3.1) is fundamental to many modern and traditional neural network architectures. Note that the degree of freedom for the weight parameters W in (3.1 is kd. Since W dominates parameterization, the parameter size of the FC layer in 3.1 is Okd)).
3.2 PARAMETERIZES HYPERCOMPLEX MULTIPLICATION (PHM) LAYERS
We propose the parameterized hypercomplex multiplication (PHM) layer that transforms an input x into output y
PHM(()) Hx + b
(3.2)
where the same notation from 33.1 is used but the replaced parameter H R is constructed by of Kronecker products. For context, the Kronecker product is a generalization of the vector outer product to higher dimensions in real space. For any matrix X E Rmmx and Y E IRPxq the Kronecker product X & Y is block matrix:
In1 Y
Fin Y
X Y
mpxng
Tml Y
Imn
where Zig is the element of X at its 7th row and jth column. Note that the symbol @ between two matrices is the Kronecker product while the same symbol between two quaternions means the Hamilton product.
Now let revisit 3.2h to explain H. Suppose that both & and d divisible by user-defined hyperparameter n E Z>n- For i 1,..., n, denote by each parameter matrix A, maax and S; E IR & d. The parameter H in is a sum of Kronecker products:
H A S..
(3.3)
3

---

Published as a conference paper at ICLR 2021
Parameters for H:
Size of H:
Parameter size of H:
A1
Si
A2
S2
Figure 1: Illustration of the PHM layer. It uses of Kronecker products of matrices A, and S;
1, 2) to construct H in 3.2 (here n = 2, k = 6, d = 8). Best viewed color.
0.6
2
0.4
loss
loss
1
0.2
0.0
5
10
15
20
1
2
3
4
5
epoch
epoch
(a) Learning rotations in 3D real space
(b) Learning Hamilton products in quatermion space
Figure 2: PHM layers can learn to perform rotations in 3D real space and Hamilton products in quaternion space on artificial datasets.
As illustrated in Figure 1 the parameter matrices A; and S, (i -i= 1...,,...7 ..) that determine the degree of freedom for H. which is kd/n π².. Since H dominates parameterization, the parameter size of the PHM in (322) is Okd//n), where kd ž n' assumed: this condition is mild for real- world problems, such in experiments (e.g., d 512, & 2048, 2, 4, 8, 16). Thus, for the same input and output sizes, the parameter size of a PHM layer is approximately 1/n of that of an FC layer under mild assumptions.
The benefit of parameterization reduction of PHM layers is due to reusing elements of both parame- ter matrices A, and S the Kronecker product. As an alternative perspective, we can equivalently reconstruct H in (3.3) by reusing parameter matrices in real-valued matrix multiplications, followed by more operations. Due to limited space, this more complicated perspective is offered in Ap- pendix A Though simply setting H = A₁ @ S₁ can further save parameters, it does not generalize hypercomplex multiplications hence is out of scope.
To show that PHM layers can learn to perform pre-defined multiplication-rel rerieatttoo--elteed related operations in prac- tice, we perform experiments to learn rotations 3D real space using the PHM layer. Using rotation matrix W R333 we create an artificial dataset (x; E TR:
IR where y; is gener- ated via the 3D rotation of the input: y, = Wx;, Figure 2(a) shows that the loss converges to zero: the PHM layer can learn single rotation of an object in 3D real space.
the following, we show how the proposed PHM layer subsumes and generalizes both hypercom- plex multiplications and real-valued matrix multiplications.
3.3 SUBSUMING HYPERCOMPLED MULTIPLICATIONS
First, we explore how the PHM layer connects to the hypercomplex multiplication. For the sake of illustration, let us take the Hamilton product of two quaternions and P in (.2) example,
4

---

Published as a conference paper at ICLR 2021
which can be rewritten as
QTr -Qx
-Qy
-Q=
Pr
Qr
Q
Q=
Qy
(3.4)
Qu Qz
Q
Q.
[PC-
QQ= Qu
Q
Q
where the 4 output elements are the real values for the quaternion unit basis j..i..kk] Note that for models leveraging Hamilton products of quaternions (Parcollet et al. 2018b 2019 Tay et al. 2019), the components Qr. Qz. Qy Q2 of (3.4) are learnable parameters while the components Pr. Pa. py. P- are the layer inputs. In practice, such a layer usually has more than inputs (d > 4). To apply the Hamilton product, all the inputs evenly split into 4 segments (P,, Pz. Py. Pa) of the right imput vector of (3.4). Then each component in the left matrix of (3.4) can be a block matrix (i) where all the elements take the same value; (ii) whose shape is aligned with the input length d and the output length & of the layer. It is noteworthy that the left 4 x 4 matrix of (3.4 can be rewritten as sum of Kronecker products:
o
0
0
8
Q
0
0
0
Q4
0
0
0
1
8
9u
0
-1
0
@
q.
0
0
0
81
0
0
8/
-1
0
8/
1
0
0
84
A4
(3.5)
According to (355). when n = 4, the PHM layer can be learned to express the Hamilton product of quaternions. Specifically, matrices A....., .,. A4 in (3.3) parameterize the four matrices composed of -1, 0, in (3.5) that reflect interactions between real and imaginary components of quaternions, which are the rule of Hamilton products. The single-element "matrices" S,.... S4, in (3.3) are equal to the learnable components Qr, Qn, Qu Q2 in (3.4). Figure 2(b) shows that PHM layers can learn the rule Hamilton products on artificial data. Likewise, hypercomplex multiplications of octonions sedenions can also be learned by the PHM layer when is set to 8 or 16.
3.4 SUBSUMING VALUED MATRIX MULTIPLICATIONS
Next, we show how the PHM layer subsumes the matrix multiplication in real space. In other words, the PHM layer is a generalization of the FC layer via the hyperparameter n. To explain, referring to (...p. when n = 1, H = A₁ X S aS1. where the scalar is the single element of the 1 x matrix A and S kx Since learning a and S separately is equivalent learning their multiplication jointly, scalar a can be dropped, which is learning the single weight matrix in an FC layer. Therefore,
a PHM layer is degenerated to an FC layer when n
3.5 GENERALIZING HYPERCOMPLEX MULTIPLICATIONS
Though parameter reusing by component-wise partitioning in quaternion space has demonstrated success (Parcollet et al., 20186] Zhuu et al.] 2018, Parcollet et al., 2019: Tay et al. (001)), one key problem that hypercomplex space only exists at very few predefined dimensionalities, such as 4D (quaternions), 8D (octonions), and 16D (sedenions). Within the context of hypercomplex space, specialized multiplication rules, such as the Hamilton product, have be devised and encoded in the network as a fixed inductive bias. As described in Section [ the very few choices over existing hypercomplex space restricts the flexibility of networks that leverage hypercomplex multiplication.
In sharp contrast to relying predefined mathematical rules over limited dimensionality choices, the PHM layer treats the dimensionality n (number of Kronecker products) as tunable hyperparam- eter and learns such specialized multiplication rules from data, manifested in the parameterized matrices A, (i = 1, 1,.... ri) in (333. On one hand, the PHM layer can express hypercomplex mul- tiplications when A, are set to reflect those predefined multiplication rules in hypercomplex space. On the other hand, the PHM layer can be seen as a trainable and parameterized form of nD hy- percomplex multiplications, where generalizes multiplications in hypercomplex space. Since n can be 1, the PHM layer also offers a neat way to bridging multiplication between both real space and hypercomplex space.
be values other than 8, or 16. Thus, the PHM layer
5

---

Published as a conference paper at ICLR 2021
4 NEURAL MODELS WITH PHM LAYERS
To demonstrate the applicability of the PHM layers, we develop the PHM-LSTM and PHM- transformer by equipping two popular neural network models, LSTMs and transformers, with PHM layers.
4.1 PHM-LSTM
Recurrent neural networks such as LSTMs (Hochreiter Schmidhuber 1997) are gated recurrent networks where the gating functions are parameterized by linear transformations. We introduce the PHM-LSTM, which replaces such linear transformations in LSTMs with PHM layers:
Yr PHM (x+) PHM (he-1) +
f i, x/ (vy))
03 (f) C--1 + to (x()))
h, = or o Ct.
where as is the sigmoid activation function, d, is the tanh activation function, Φ Rixd + RAx is four-way split on the last dimension, and Ct2 he are the cell state and the hidden state of the PHM-LSTM unit at any time step L.
4.2 PHM- TRANSFORMER
The transformer is stacked neural network architecture that aggressively exploits linear transfor- mations (Vaswani et al. 2017). Each self-attention layer comprises of Q (query), (key), V (value) linear transformations, along with multiple heads. Each transformer block also has position-wiss feed-forward network composed two FC layers. Since a large majority of the transformer param- eters stem from linear transformations FC layers, we introduce the MEM--rranffrmmrr to replace all the linear transformations or FC layers with PHM layers. The single-head self-attention module rewritten
Q K, V ((PHM(())
QK T VV,
A softmax
Vdg
where de is the key dimension, $ Rxxd__ + 3 is a three-way split on the last dimension, X is the input sequence, and A is the self-attentive representation, For multi-head attention, using PHM layers also enables weight sharing not only among the linear transformations of Q, K, V but also among the linear transformation of multiple heads:
X = :H;
where N₁, is the number of heads and (:) is the column- concatenation. Finally, the position-wise ferd-f network is now defined
Y = PHM(RELU(PHM(X)),.
which transforms X with two PHM layers.
EXPERIMENTS
For context, in the field of representation learning using hypercomplex multiplications, quaternion convolutional neural networks (Zhu et al. 2018). quaternion recurrent neural networks (Parcollet) (et al., 20188), and quaternion transformers (Tay et al., 2019) have all compared themselves with only real-valued counterparts. Therefore, to be consistent with the rest of the literature, we eval- uate PHM-LSTMs and PHM--ransorrmers that are equipped with PHM layers, and compare them with quaternion LSTMs, quaternion transformers, real-valued LSMMs, or real-valued transformers. Both quaternion LSTMs and quaternion transformers replace linear transformations with Hamilton products of quaternions.

---

Published as a conference paper at ICLR 2021
Table 1: Experimental results of natural language inference (accuracy) five different datasets.
The PHM-LSTM reduces the parameters of the standard LSTM model and improves or partially matches performance four out of five datasets.
Model
#Params
MNLI
QNLI
SNLI
DNLI
SciTail
LSTM
721K
71.82 / 71.89 84.44
84.18
85.16
74.36
Quaternion LSTM
180K (-75.0%)
71.57 / 72.19
84.73
84.21
86.45
75.58
PHM-LSTM (n = 2)
361K (-49.9%)
71.82 / 72.08
84.39
84.38
85.77
77.47
PHM-LSTM (n = 5)
146K (-79.7%)
71.80 / 71.77
83.87
84.58
86.47
74.64
PHM-LSTM (n 10)
81K (-88.7%)
71.59 / 71.59
84.25
84.40
86.21
77.84
To demonstrate the architectural flexibility and effectiveness, we evaluate different settings of PHM- LSTMs and PHM-transformets
show that allowing for flexible choices of the hyperparameter in the PHM layer may lead to more effective performance. Details of the setup for the experiments provided in Appendix B
5.1 NATURAL LANGUAGE INFERENCE
The task of natural language inference is determine the logical relationship between two text sequences (MacCartney 2009). It is a fundamental task pertaining to language understanding. To this end, they serve suitable benchmark for evaluating recurrent models.
We run experiments on five datasets: (i) MultiNLI (Williams et al. 2017), (ii) QNLI (Quora) (Wang
al. 2017), (iii) SNLI (Bowman et al. 2015), (iv) Dialogue NLI (Welleck al. 2018), and (v) SciTail (Science Entailment) (Khot et al. 2018). Table
the results on all these datasets. All in all, such results show that the PHM layer can not only reduce the parameters but also improve performance with flexible choices of (four out of five datasets show reasonable improvement or partially match). The only exception is on the QNLI dataset, where the performance drop marginal (< 1%). This is still decent considering the parameter saving: the parameterization cost of the PHM-LSTM is in the order of (L1/n) of that the standard LSTM, where settings of n = and n 10 do not take values of power of 2. As detailed in Appendix B since GloVe (Pennington et al. 2014) embeddings to represent input tokens, we choose multiples of instead of 4 for ease of divisibility. It is also noteworthy that the SNLI, Dialogue NLI, and SeiTail datasets, all the PHM-LSTM variants outperform the standard LSTM model. We think that the element reusing properties of the Kronecker product operation, in addition learning share such reused components amongst recurrent gating functions, may contribute to both effective and efficient representations.
the 300D
5.2 MACHINE TRANSLATION
Machine translation is concerned with translating between source-target language pairs. To this end, sequence transduction models are central to this problem domain. In this experiment, the key goal compare trrrforranformrrs against the standard and quaternion transformer models.
We run experiments on seven datasets: (i) IWSLT' English-Vietnamed English-Indonesian (En-Id), (iii) IWSLT' 14 German-Englist (De-En), (iv) IWSLT' 14 Romanian- English (Ro-En). (v) WMT' 18 English-Estonian (En-Et), (vi) Setimes English- -aglish-Macedonian (En- Mk), and (vii) WMT' 18 English-Romania (En-Ro).
(En-Vi), (ii) IWSLT'
Table reports our results of the machine translation tasks. Overall, these empirical results with different settings demonstrate architectural flexibility and effectiveness of the hypercomplex mul- tiplication parameterization. First and foremost, across six out of seven benchmarks, the PHM- transformer at n = makes reasonable gains over the quaternion transformer, signifying that pa- rameterization of hypercomplex multiplications by learning from data can be more effective than predefining Hamilton product rules mathematically. Second, though increasing n leads to more parameter savings, observe that increasing all the way to 16 does not cause significant degra- dation in performance on datasets such as En- Vi. Third, for most datasets, even with significant parameter savings, we find that the decrease in the BLEU score is mostly manageable (+x 1-3 BLEU

---

Published as a conference paper at ICLR 2021
Table 2: Experimental results of machine translation (BLEU) different datasets. Symbol i represents re-scaling the parameters with factor of 2 by doubling the hidden size. The PHM- transformer does not lose much performance despite enjoying parameter savings. Re-scaling can lead to improvement in performance.
Model
#Params
En- Vi
En-Id
De-En
Ro-En
En-Et
En-MK
En
Transformer (Tm)
44M
28.43
47.40
36.68
34.60
14.17
13.96
22.79
Quaternion Tm
11M (-75.0%)
28.00
42.22
32.83
30.53
13.10
13.67
18.50
PHM-Tm n 2
22M (-50.0%)
29.25
46.32
35.52
33.40
14.98
13.60
21.73
PHM-Tm n 5.5M (-87.5%)
11M (-75.0%)
29.13
44.13
35.53
32.74
14.11
13.01
21.19
PHM-Tm n.
29.34
40.81
34.16
31.88
13.08
12.95
21.66
PHM-Tm n 16
2.9M (-93.4%)
29.04
33.48
33.89
31.53
12.15
11.97
19.63
PHM-Tm'
44M
29.54
49.05
34.32
33.88
14.05
14.41
22.18
PHM-Tm' n 4
22M (-50.0%)
29.17
46.24
34.86
33.80
14.43
13.78
21.91
PHIM-Tm'
11M (-75.0%)
29.47
43.49
34.71
32.59
13.75
13.78
21.43
Table 3: Training time (seconds per 100 steps) and inference time (seconds to decode test sets) with beam size and length penalty of 0.6 on the IWSLT 14 German- German-English dataset.
Model Transformer (Tm) Quaternion Tm PHM-Tm (n 4)
PHM-Tm (n 8)
Training time Inference time
7.61
8.11
7.92
7.70
336
293
299
282
points). However, also note rare occurrence where 16 results in a significant decrease in the BLEU score, such as on the En-Id dataset. Fourth, on several datasets, the traMfrrrafoormrr model improves the performance of the standard transformer model. For example, on datasets such model enjoys a performance boost of about 0.8 BLEU
as En-V Vi and En-Et, the PHM-transforme point with n 2. Finally, by re-scaling with factor of (doubling the hidden size), we are able to improve the performance on three datasets: En-Vi, En-ld, and En-Mk.
Table 3 reports the training and inference time for transformer variants. We observe that the PHM- transformer with 8 has the fastest inference speed amongst all the variants, primarily due to significant reduction of parameters. All all, the training speed is also approximately comparable. This ascertains that the PHM layer does not increase much computational cost in practice.
5.3 TEXT STYLE TRANSFER
We continue to experiment with sequence transduction for text style transfer. The goal of this task is to convert text of a certain style to another style. We use the Modern-Shakespeare corpus] in the experiments. Table 4 reports the results on this text style transfer task. We observe that the best performance is achieved with the PHM-transformer (n 4). Notably, all except the n = 16 variant increases or matches the performance of the standard transformer model. This ascertains ar- chitectural flexibility and effectiveness of the proposed PHM layer. This not only enables parameter savings but also improves the performance the transformer.
5.4 SUBJECT VERB AGREEMENT
We conduct additional experiments on the subject-verb agreement task (Linzen et al., 2016). The task predicts if the sentence, e.g., 'The keys to the cabinel singular. The used dataset can be found online (Linzen et al., 2016). Table reports the results on the subject-verb agreement task. Results promising. demonstrating that all variants with PHM layers outperform the standard and quaternion transformer models. The best performance peaks at 8, despite parameter saving up 1/8.
is followed by plural
https://github.ee
8

---

Published as a conference paper at ICLR 2021
Table 4: Experimental results of text style transfer. The PHM-transformer may reduce the parame- ters of the standard transformer model and improve performance.
Model
#Params
BLEU
Transformer (Tm)
44M
11.65
PHM- Tm (n =2)
22M (-50.0%)
12.20
PHM-Tm (n 4)
11M (-75.0%)
12.42
PHM-Tm (n 8)
5.5M (-87.5%)
11.66
PHM-Tm (n 16)
2.9M (-93.4%)
10.76
Table 5: Experimental results of subject verb agreement. The PHM-transformer may reduce the
parameters of the standard transformer model and improve performance.
Model
#Params
Acc
Transformer (Tm)
400K
94.80
Quaternion Tm
100K
94.70
PHM-Tm (n = 2)
200K (-50.0%)
95.14
PHM-Tm (n = 4)
101K (-74.8%)
95.05
PHM- Tm (n -)
56K (-86.0%)
95.62
RELATED WORK
While neural networks have been a well-established line of research, progress on hypercomplex representations for deep learning still in its infancy and most works this topic
(Gaudet
& Maida 2017 Parcollet et al. 2018a b Zhu et al. 2018 Tay et al. 2019). The hypercomplex Hamilton product provides greater extent expressiveness, similar to the complex multiplication, albeit with 4-fold increase in interactions between real and imaginary components. In the case of quaternion representations, due to parameter savings in the Hamilton product, models also enjoy 75% reduction in the parameter size (Parcollet et al. 2018a Tay et al. (019). A striking caveat is that all quaternions are fundamentally limited to 4D hypercomplex space, which restricts architectural flexibility. The other options would be to scale to octonion (8D) sedenion (16D) space, given the predefined multiplication rules such space. To the best of our knowledge, there no work that attempts to generalize arbitrary nD hypercomplex multiplications to allow for architectural flexibility, where n can be specified or tuned by users.
Our work also be interpreted Quaternion networks (Zhu et al. 2018 Parcollet et al. [018b) 2019) are known to possess weight sharing properties via the Hamilton product operation and have demonstrated reasonable success despite having fewer parameters. To the best of our knowledge, there has been no work that attempts parameterize the hypercomplex Hamilton product for neural networks, i.e., enabling end-to-end learning of real and imaginary component interactions from data.
form of soft parameter sharing, albeit learned from data.
7 CONCLUSION
We proposed parameterized hypercomplex multiplication (PHM) layers that learn and generalize hypercomplex multiplications. practice, the PHM layer has 1/n learnable parameters compared with the fully-connected layer counterpart, where can be flexibly specified by users. PHM layers are applicable to dominant models such as LSTMs and transformers. We evaluated these models equipped by PHM layers comprehensive tasks show architectural flexibility and effectiveness of the hypercomplex multiplication parameterization.
Acknowledgements. We thank the anonymous reviewers for the insightful comments this pa- per. This work was partially supported by the Ministry of Education (MoE) of Singapore under the Academic Research Fund (AcRF) Tier 1 Grant RG135/18.

---

Published as a conference paper at ICLR 2021
REFERENCES
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. arXiv preprint arXiv: 100. 2015.
Chase Gaudet and Anthony Maida. Deep quaternion networks. arXiv preprint arXiv: 2017.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
// www. dww.eeepelaa.. wew.llaanniniiie www.leaannnnooooooooomm org
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521 535, 2016.
Bill MacCartney. Natural language inference. Citeseer, 2009.
Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi, Renato De Mori, and Yoshua Bengio. Quaternion recurrent neural networks. In International Conference on Learning Representations, 2018a.
Titouan Parcollet, Ying Zhang, Mohamed Morchid, Chiheb Trabelsi, Georges Linarès, Renato De Mori, and Yoshua Bengio. Quaternion convolutional neural networks for end-to-end auto- matic speech recognition. arXiv preprint arXiv:1806.07789, 2018b.
Titouan Parcollet, Mohamed Morchid, and Georges Linarès. Quaternion convolutional neural net- works for hetcrogencous image processing. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8514-8518. IEEE, 2019.
Ankur P Parikh, Oscar Täckström, Dipanjan Das. and Jakob Uszkoreit. A decomposable attention model for natural language inference. arXiv preprint 1606.01933 2016.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543 1543, 2014.
Adityan Rishiyur. Neural networks with complex and quaternion inputs. arXiv preprint cs/0607090, 2006.
Yi Tay, Aston Zhang, Luu Anh Tuan, Jinfeng Rao. Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. Lightweight and efficient neural natural language processing with quaternion networks. arXiv preprint arXiv: 190600490 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp.
2017.
Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi pelti-perspeectve matching for natural language sentences. arXiv preprini arXiv: 1702003814 2017.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. Dialogue natural language infer- ence. arXiv preprint 1rXiv:18/1.0067/ 2018
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: 1704.05422, 2017.
Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. Dive into Deep Learning. 2020.
https: //d21. ai
Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 631-647, 2018
10

---

Published as a conference paper at ICLR 2021
B
T2
B1
T4
B2
T2
B2
T+
Size of H:
Columns of H:
Ti T3
B2
Ti
B2
Parameter size of H:
Parameters for H:
Bi
B2
T
T2
T;
T4
Figure 3: Illustration of reconstructing H (32) by reusing parameter matrices B, (i 1, 1,2)
and T3 (j l.....4. 4) in real-valued matrix multiplications, followed by more operations (here
n 2, = 6, d 8). Best viewed in color.
A RECONSTRUCTING THE PARAMETER MATRIX
In the paper, the parameter matrix H in (.2) is constructed sum of n Kronecker products. In the following, we will provide an alternative perspective and show how to equivalently reconstruct H by reusing parameter matrices in real-valued matrix multiplications, followed operations.
A.1
METHOD
The key idea is to operate on partitioned weight blocks and learn dynamic diffusion of weights. There are key parameter blocks and T that central our approach. Intuitively, B maxх=x controls the weight diffusion process and learns the soft interactions between T partitions. Here, n is a user defined hyperparameter.
Suppose that both d and k are divisible by n Z For 1...., n and j = 1...., d denote by each partitioned parameter block T, E Rax and B, E RR the weight diffusion ma- trix assigned to each partitioned parameter block via real-valued matrix multiplication B,T,, The parameter in 3.2) is now constructed by column-wise concatenation (:):
[s(B)); sBB,,)
(A.1)
where each segment s(B;) is also formed by column-wise concatenation:
s (B,)
(A.2)
In (A.2), function v R + RP where (xX) flattens the matrix X E RRxx by concatenating each row of X then transposes the concatenated vector into column vector of dimension pq. is easy to see that, d (B,T,)) R* s(B)) E I w, thus H Bxd
the partitioned parameter blocks B; (i (.....,,) n) and T, Gj = I,..,, A) that determine the degree freedom for H. which is kd/n + n As illustrated in Figure 3 the reuse of parameter matrices B ..... B, and T T.,..., Ta in real-valued matrix multiplications in (A.2) may reduce the degree of freedom for H.
11

---

Published as a conference paper at ICLR 2021
A.2 SUBSUMING HYPERCOMPLEX MULTIPLICATIONS
Similarly, we show how the PHM layer with the reconstructed H in A.1 also subsumes the hyper- complex multiplication. Taking the Hamilton product of two quaternions Q and P as example, can be rewritten as
0
Qr
-1
0
Q-
-1
Q-
Q+
P+
0
0
Q=
1
0
Q+
0
0
0
-1
Q
0
0
Q=
Ps
0
1
0
4u
0
0
Qy
-1
0
Qy
0
-1
Q
Q.
0
0
Q
P.
(A.3)
where the 4 output elements are the real values for the quaternion unit basis i,j, k] According (A.3) when 4, the PHM layer with the reconstructed parameter matrix also be learned to exactly express the Hamilton product of quaternions. Likewise, hypercomplex multiplications of
octonions or sedenions can also learned by the PHM layer when set to 8 or 16.
A.3 SUBSUMING VALUED MATRIX MULTIPLICATIONS
Now we show how the PHM layer with the reconstructed H in (A.I) also subsumes the matrix multiplication in real space. Referring to (3.2). when n = 1. H bW, where the scalar b is the single element of the 1 x matrix B, and elements W fxd come from the concatenation of TI,..., Ta E RI*k Since learning b and W separately is equivalent to learning their multiplication jointly, the scalar can be dropped, which is learning the single weight matrix in an FC layer. Therefore, PHM layer is degenerated to an FC layer when n =
B SETUP FOR EXPERIMENTS
We describe the setup for the experiments as follows.
B. NATURAL LANGUAGE INFERENCE
We implement 300D unidirectional encoders with shared parameters for both premises and hypothe- ses. We take the concatenation of and mean pooled representations as the input two-layer 300D multilayer perceptron for prediction. Our model is trained with the the Adam optimizer with learning rate of 0.0004 and batch size of 256. Word embeddings are initialized with GloVe (Pen- nington et al.) 2014) and are fixed. No cross sentence attention (Parikh et al. 2016) is used, mainly to observe the effectiveness of standalone encoders. For PHM-LSTM, we use {2, 10). Note that in this task, since word embeddings are 300D, we select multiples of 5 instead of 4 for ease of divisibility.
B.2 MACHINE TRANSLATION
For the 15 English- Vietnamese (En-Vi), IWSLT' English-Indonesian (En-Id), IWSLT' 14 Gernan-English (De-En), and 1WSLT'14 Romanian-English (Ro-En) datasets, we run with 50K steps; while for WMT' 18 English-Estonian (En-Et, Setimes English-Maccdonian (En-Mk), and WMT' English-Romanial (En-Ro) datasets, models are trained for 100K steps. For the En-Vi, En-Id, En-Et, En-Mk, and En-Ro datasets, we specify that transformers have 4 layers, 8 heads, and hidden size 512. For the De-En and Ro-En datasets, heads, and hidden size 256. We use beam size of and = 0.6 (length penalty) for decoding. For all PHM models, we benchmark several settings for the hyperparameter {2, 4, 16).
specify that transformers have 2 layers,
12

---

Published as a conference paper at ICLR 2021
B.3 TEXT STYLE TRANSFER
For the used Modemn-Shakespeare corpus in the experiments, the key goal here is to convert modern writing into Shakespeare writing. This dataset comprises of 18, 395 parallel sentences for training, 1, 218 parallel sentences for evaluation (development set), and 462 parallel sentences for testing. We still specify that transformers have layers. heads, and hidden size 512. Similar to machine translation, we experiment with = {2, 4, 8, 16). We train all the models for 10K steps.
B.4 SUBJECT VERB AGREEMENT
In contrast to the previous experimental settings, we use a smaller transformer architecture with 10K training steps. Specifically, transformers here have layers, 4 heads, and hidden size 128. Since the hidden size is smaller than those in the previous experimental settings, we experiment with (2, 8.)
13


Reviews:
[{"review_id": "cYxBqAPaOuI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel way of parametrizing hypercomplex multiplications. \nThe proposed parametrization helps with: (a) Generalizing the multiplication to arbitrary dimensions, and (b) Reducing the number of parameters.  \nBuilding on this, the authors propose a parameterized hypercomplex multiplication (PHM) layer which essentially replaces the weight matrix of a linear layer with a matrix constructed via sum of Kronecker products. \nThey then replace the weight matrix of linear layers in LSTM and Transformer with PHM.\nFinally, they show that these PHM-variants of LSTM and Transformer, match or outperform their vanilla counterparts on a variety of NLP tasks, including NLI, MT, text style transfer, etc.,  while reducing the total number of model parameters.  \n\nOverall the paper is quite well written with easy to follow illustrations. \nThe proposed parametrization seems reasonable, and the empirical validation lends solid credibility to the idea. \nI have a couple of questions for the authors:\n* What are the practical benefits of this parametrization, particularly in comparison to other ways of reducing parameters, say matrix factorization?\n* If hypercomplex spaces only exist in $2^k$-D, is the term hypercomplex justified for arbitrary dimensions?\n* Have you analyzed the learned hypercomplex spaces? Can they be interpreted for arbitrary dimensions?", "extra_scores": {}, "extra_text": {}}, {"review_id": "KhIjhpDlrpB", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds on the top of standard high-dimensional neural networks (limited to 2,4,8, 16 dimensions) by introducing an elegant way to deal with others dimensions while preserving the internal relation learning capability as well as the reduction of number of parameters. To do so, the linear transformation is turned into a new linear transformation based on the Kronecker product. A sum of Kronecker products is used to \"simulate\" the different internal relations that could occur in between multi-dimensional components. However, I think that an empirical validation of the ability of the method to recover well-know product, i.e. Hamilton / complex products, is missing. \n\nAccording to the results, with experiments conducted on 3 different tasks, the proposed approach definitely seems to work and is an important step further to better understand and manipulate high-dimensional algebras with deep neural networks.\n\nRemarks and questions:\n1. While the theoretical aspect of the degeneration of the proposed approach to the Hamilton product \"sounds\" plausible, I would like to see a more empirical demonstration. Can this approach, correctly parametrize a rotation in a 3D space (simple task of learning a single rotation of an object). \n2. The quaternion case of this method relies on a set of [-1,0,1] to build the S matrices. This is what ensures the geometrical properties of the quaternion space while doing the different manipulations. This method, however, uses real-valued matrices. It is thus almost certain that the geometrical aspect of the \"learnt\" algebra is impossible to interpret. Do the authors think that a quantisation of the matrices could help finding pure quaternion / complex / octonions / sedenions matrices ? Such an analysis is crucial to validate the fact that the work proposed here is a generalisation of what have been done before to N dimensions for neural networks. \n\n- Would be great to include the number of parameters in the different Table.", "extra_scores": {}, "extra_text": {}}, {"review_id": "B6o2fKHitDD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors focus on the area of using hypercomplex multiplications (multiplications involving numbers with multiple imaginary components) in deep learning models. Past work in this area has been promising but has been limited to certain dimensions for which there are predefined multiplication operations. The novel contribution of this work is to parameterize the hypercomplex multiplication operations, enabling the model to discover new operations rather than relying on the small number of existing operations and the small number of dimensions for which such operations exist. The authors find that their approach can substantially reduce the number of parameters without reducing performance (and in some cases even improving performance). \n\nStrengths:\n\n1. The proposed method makes a promising approach from the literature more flexible, helping to pave the way for making this approach more broadly useful.\n\n2. The authors illustrate this flexibility by showing how their approach can be effective for two different architectures (LSTMs and Transformers), making the general point that it can be applied to any architecture that uses feedforward components. They also apply it to multiple tasks, again illustrating the flexibility.\n\n3. As mentioned above, the approach can substantially increase a model’s parameter count without affecting performance. Relatedly, it can also improve inference speed.\n\n4. The paper is generally thorough and clear.\n\nWeaknesses:\n\n1. The specific contribution of this paper is the parameterization of the multiplication operation, but the evidence that this parameterization is helpful is mild, as there are only a few cases where the proposed model noticeably outperforms the Quaternion model. Thus, the evidence presented does not make a strong case for the necessity of this parameterization.\n\n2. Much of the argument hinges on the reduced parameter count, but there was not any mention of exactly how many parameters each model had (at least, not that I saw - I did not check the appendix). I think the paper could be substantially strengthened by adding a “Parameter count” column to each table.\n\n3. There is no clear intuition offered for why this approach might be expected to be effective. Offering such an intuition is certainly not necessary (since results alone are enough), but the paper would be more satisfying if there were such an intuition present.\n\nOverall, I am rating this as a 7, because I find it to be a solid paper but worry that its contribution on top of the existing work that has studied hypercomplex operations may be too small and may not have enough evidence for its usefulness.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new parameterization which (across multiple architectures) generalized hypercomplex multiplication and provides for small low dimensions strong performance at substantial parameter savings. All reviewers are happy with the theoretical contributions of the work, but would appreciate additional empirical evidence.
Decision: Accept (Spotlight)