Removing Dimensional Restrictions on Complex/Hyper-complex Convolutions
Authors: Chase John Gaudet
Anthony S. Maida
Venue:  (2021)
Keywords: CNNs
complex
hypercomplex

Abstract: It has been shown that the core reasons that complex and hypercomplex valued neural networks offer improvements over their real-valued counterparts is the fact that aspects of their algebra forces treating multi-dimensional data as a single entity (forced local relationship encoding) with an added benefit of reducing parameter count via weight sharing. However, both are constrained to a set number of dimensions, two for complex and four for quaternions. These observations motivate us to introduce novel vector map convolutions which capture both of these properties provided by complex/hypercomplex convolutions, while dropping the unnatural dimensionality constraints their algebra imposes. This is achieved by introducing a system that mimics the unique linear combination of input dimensions via the Hamilton product using a permutation function, as well as batch normalization and weight initialization for the system. We perform three experiments using three different network architectures to show that these novel vector map convolutions seem to capture all the benefits of complex and hyper-complex networks, such as their ability to capture internal latent relations, while avoiding the dimensionality restriction.

Under review as a conference paper at ICLR 2021
DIMENSIONAL COMPLEX/HYPER-COPPEEX S
REMOVING
RESTRICTIONS
ON
Anonymous authors Paper under double-blind review
ABSTRACT
It has been shown that the core reasons that complex and hypercomplex valued neural networks offer improvements over their real-valued counterparts the fact that aspects of their algebra forces treating multi-dimensional data as single entity (forced local relationship encoding) with added benefit of reducing parameter count via weight sharing. However, both are constrained to a set number of dimensions, two for complex and four for quaternions. These observations motivate us to introduce novel vector map convolutions which capture both of these properties provided by complex/hypercomples convolutions, while dropping the unnatural dimensionality constraints their algebra imposes. This is achieved by introducing a system that mimics the unique linear combina- tion of input dimensions via the Hamilton product using permutation function, as well as batch normalization and weight initialization for the system. We perform three experiments using three different network architectures to show that these novel vector map convolutions seem to capture all the benefits of complex and hyper-complex networks, such as their ability to capture internal latent relations, while avoiding the dimensionality restriction.
1 INTRODUCTION
While the large majority of work in the area of machine learning (ML) has been done using real-valued models, recently there has been an increase in use of complex and hyper-complex models Trabelsi et al.] [018a] Parcollet et al. 2020). These models have been shown to handle multidimensional data more effectively and require fewer parameters than their real-valued counterparts.
For tasks with two dimensional input vectors, complex-valued neural networks (CVNNs) are a natural choice. For example in audio signal processing the magnitude and phase of the signal can be encoded as a complex number. Since CVNNs treat the magnitude and phase as single entity, a single activation captures their relationship opposed to real-valued networks. CVNNs have been shown to outperform or match real-valued networks, while sometimes at a lower parameter count (Trabelsi et al. 2018b Aizenberg Gonzalez, 2018). However, most real world data has more than two dimensions such as color channels of images or anything in the realm of 3D space.
The quaternion number system extends the complex numbers. These hyper-complex numbers are composed of one real and three imaginary components making them ideal for three or four dimensional data. Quaternion neural networks (QNNs) have enjoyed a surge in recent research and show promising results (Takahashi et al., 2017 Corro-oooooohhno et al., 2018 Gaudet & Maida 2018 Parcollet et al. 2017a by 201 8a 2019). Quaternion networks have been shown to be effective at capturing relations within multidimensional data of four or fewer dimensions. For example the red, green, and blue color image channels for image processing networks needs to capture the cross channel relationships of these colors as they contain important information to support good generalization (Kusamichi et al. 2004 [sokawa et al. (2003). Real-valued networks treat the color channels as independent entities unlike quaternion networks. Parcol- let et al. al. (2019) showed that a vallev, capture local (color) and global (edges and shapes) features independently, while the quaternion encoder-decoder can do so. Their conclusion is that the Hamilton product of the quaternion algebra allows the quaternion network to encode the color relation since it treats the colors as a single entity. Another example 3D spatial coordinates for robotic and human-poss estimation. Pavlllo et al. (2018) showed improvement on short-term prediction on the Human3.6 3.6M dataset using network that encoded rotations as quaternions over Euler angles.
sesoddrr-eeoddr fails to reconstruct unseen color images due to it failing to
The prevailing view is that the main reason that these complex networks outperform real-valued networks is their underlying algebra which treats the multidimensional data as a single entity (Parcollet et al.] 2019). This allows the complex networks to capture the relationships between the dimensions without the trade-off of learning global features. However, only the Hamilton product seems to be needed to capture this property and the other aspects of the algebra are only imposing dimensionality constraints. Therefore, the present paper proposes: 1) to create a system that mimics

---

Under review as a conference paper at ICLR 2021
the concepts of complex and hyper-complex numbers for neural networks, which treats multidimensional input as a single entity and incorporates weight sharing, but not constrained to certain dimensions; 2) to increase their local learning capacity by introducing learnable parameter inside the multidimensional dot product. Our experiments herein show that these novel vector map convolutions seem to capture all the benefits of complex and hyper-complex networks, while improving their ability to capture internal latent relations, and avoiding the dimensionality restriction.
2 MOTIVATION FOR VECTOR MAP CONVOLUTIONS
Nearly all data used in machine learning is multidimensional and, to achieve good performance models, must both capture the local relations within the input features (Tokuda et al. 2003; Matsui et al.) (004), as well non-local features, for example edges or shapes composed by a group of pixels. Complex and hyper-complex models have been shown to be able to both capture these local relations better than real-valued models, but also to do at a reduced parameter count due to their weight sharing property. However, as stated earlier, these models are constrained to two or four dimensions. Below we detail the work done showing how hyper-complex models capture these local features as well as the motivation to generalize them to any number of dimensions.
Consider the most common method for representing an image, which is by using three 2D matrices where each matrix corresponds to color channel. Traditional real-valued networks treat this input as a group of uni-dimensional elements that may be related to one another, but not only does need to try to learn that relation, it also needs to try to learn global features such as edges and shapes. By encoding the color channels into a quaternion, each pixel is treated as a whole entity whose color components are strongly related. It has been shown that the quaternion algebra is responsible for allowing QNNs to capture these local relations. For example, Parcollet et al. (2019) showed that real-valued, encoder-decoder fails to reconstruct unseen color images due to it failing to capture local (color) and global (edges and shapes) features independently, while the quaternion encoder-decoder can do so. Their conclusion is that the Hamilton product of the quaternion algebra allows the quaternion network to encode the color relation since it treats the colors single entity. The Hamilton product forces different linear combination of the internal elements to create each output element. This is seen in Fig. frrm Parcollet et al. (201 Sa), which shows how a real-valued model looks when converted to a quaternion model. Notice that the real-valued model treats local and global weights at the same level, while the quaternion model learns these local relations during the Hamilton product. This is because each output unit shares the weights and therefore forced to discover joint correlations within the input dimensions. The weight sharing property can also be seen where each element of the weight is used four times, reducing the parameter count by factor of four from the real-valued model.
The advantages of hyper-complex networks on multidimensional data seems clear, but what about niche cases where there are higher dimensions than four? Examples include applications where one needs ingest extra channels of information in addition to RGB for image processing, like satellite images which have several bands. To overcome this limitation we introduce vector map convolutions, which attempt to generalize the benefits of hyper-complex networks to any number of dimensions. We also add a learnable set of parameters that modify the linear combination of internal elements to allow the model to decide how important each dimension may be in calculating others.
3 VECTOR MAP COMPONENTS
This section will include the work done to obtain working vector map network. This includes the vector map convolution operation and the weight initialization used.
3.1 VECTOR MAP CONVOLUTION
Vector map convolutions use a similar mechanism to that of complex (Trabelsi et al., 2018b) and quaternion (Gaudet & Maida 2018) convolutions but drops the other constraints imposed by the hyper-complex algebra. We will begin by observing the quaternion valued layer from Fig. [ Our goal to capture the properties of weight sharing and each output axis being composed of linear combination of all the input axes, but for an arbitrary number of dimensions
Dvm-
For the derivation we will choose Dym N. Let Va = [U1, U....., Un] be an N dimensional input vector and W* [wn, W22 ..., Wn] be an N dimensional weight vector. Note that for the complex and quaternion case the output vector is set of different linear combinations where each input vector is multiplied by each weight vector element a total of
2

---

Under review as a conference paper at ICLR 2021
Real valued layer
Quaternion valued layer
Whare:
W11
Me M4 My v,
Q
Quss
out
Re FLo 2..0 Ros
Y 210
Foue
Yose
Internal Hamilton product produce Our
W3r
out
Wat
eut
Y
Figure 1: Illustration of the difference between real-valued layer (left) and quaternion-valued layer (right). The quaternion's 's Hamilton product shows the internal relation learning ability not present in the real-valued due sharing weights and being forced to discover joint correlations.
one time over the set. To achieve similar result we will define permutation function:
Un U1-1
T(o:) =
i>1 1.
By applying each element in 4 vector is created that is circular right shifted permutation of V
r(V)) [Uns 01 U2,..., "n- 1n-
Let the repeated composition be denoted as r", then we can define the resultant output vector Vout as:
VB [ V (W")) V V 7"-,.""""""
Vn -" r"-((W") V!!]
(1)
where |. the product of the vectors. The above gives each element of V a unique linear combination of the elements of V and 4 since we never need to compose above times (meaning the original vector W and any permutation only appear once). Also note that we chose right shifting permutation, but any scheme that gives a complete set of linear combinations is admissible.
The previous discussion applies to densely connected layers. The same idea is easily mapped to convolutional layers where the elements of V and n matrices. To develop intuitions, the quaternion convolution operation is depicted in Fig. 3 The top of the figure shows four multichannel inputs and four multichannel kernels to be convolved. The resulting output shown at the bottom of the figure. Each in the middle of the figure shows the calculation of one output feature map, which a 'convolutional" linear combination of one feature map with the four kernels (and the kernel coefficients are distinct for each row). When looking the pattern across the rows, the weight sharing can be seen. Across the rows, any given kernel is convolved with four different feature maps. The only thing constraining the dimension to four is the coefficient values the bottom of the figure imposed by the quaternion algebra (for more detail, see Gaudet & Maida (2018)). We hypothesize that the only thing important about the coefficient values is how they constrain the linear combinations be independent. We also propose that the circularly shifted permutations just described generate admissible linear combinations. In this case, space permitting, Fig- 3 be a x 5 image, where five filters are convolved with five feature maps while the weight sharing properties are preserved. That, there is no longer dimensional constraint.

---

Under review as a conference paper at ICLR 2021
We also define a learnable constant defined as matrix L E R D
= 1
= j
1
(i + (i 1))mod Dvm
-1
else.
The purpose of this matrix is to perform scalar multiplication with the input matrix, which will allow the network some control over how much the value of one axis influences another axis (the resultant axes all being different linear combinations of each axis).
With all of the ahove Here we let the weight filter matrix W
look an example where Dum [A, B, C, Dj by an input vector
then compare the quaternion convolution. w x, z]
D1 W h)
A
B
C
D
W
D2 W h)
D
A
B
C
L
O
(2)
D3 W h)
C
D
A
B
(&W W
B
C
D
A
where
1
1
1
-1
L
(3)
The operator denotes element-wise multiply. The sixteen parameters within L the initial values. They are otherwise unconstrained scalars and intended to be learnable. Thus, the vector map convolution is a generalization of complex, quaternion, or octonion convolution as the case may be, but also drops the constraints imposed by the associated hyper-complex algebra.
For comparison the result of convolving a quaternion filter matrix W h = w + ix +j + kz is another quaternion,
A + B JC AD by quaternion vector
a W *
h)
A
B
-C
D
W
I W
h)
B
D
(4)
W h
C
A
-B
y
(W
C
where A, B, C, and D are real-valued matrices and w, x, y, and are real-valued vectors. See Fig. 3 the Appendix for a visualization of the above operation. More explanation is given in Gaudet & Maida (2018).
The question arises whether the empirical improvements observed in the of complex and quaternion deep networks are best explained by the full structure of the hyper/complex algebra, or whether the weight sharing underlying the generalized convolution responsible for the improvement.
3.2 VECTOR MAP WEIGHT INITIALIZATION
Proper initialization of the weights has been shown be vital to convergence of deep networks. The weight initializa- tion for vector map networks uses the same procedure seen in both deep complex networks (Trabelsi et al. (018b) and deep quaternion networks (Gaudet & Maida 2018). In both cases, the expected value of [W]- is needed calculate the variance:
BIWP - /_ x² f(x) dr
(5)
where fax a multidimensional independent normal distribution where the number of degrees of freedom is two for complex and four for hyper-complex. Solving Eq. 5 gives 2n² for complex and 462 for quaternions. Indeed, when solving Eq.B) for multidimensional independent normal distribution where the number of degrees of freedom is Dvm- the solution will equal D Therefore, in order to respect the Glorot & Bengio (2010) criteria, the variance would be equal to
2
(6)
Dvm (Nin Dout)

---

Under review as a conference paper at ICLR 2021
and in order to respect the He et al. (2015b) criteria, the variance would be equal to:
2
(7)
Pin
This is used alongside vector of dimension Drm that is generated following uniform distribution in the interval [o, 1] and then normalized.
3.3 VECTOR MAP WEIGHT BATCH NORMALIZATION
Batch normalization (lffe & Szegedy 2015) is used by the vast majority of all deep networks to stabilize training by keeping activations of the network at zero mean and unit variance. A formulation for complex batch normalization is given by Trabelsi et al. (0188) in which a whitening approach is used to give equal variance to the two components. This method uses the inverse of the covariance matrix, which for the complex case is 2 x 2 and has an analytical solution. Similarly. Gaudet & Maida (2018) applied the same method for quaternion batch normalization, but since the 4 x 4 covariance matrix does not have an analytical solution for the inverse the Cholesky decomposition is used instead for whitening. We adopt the Cholesky whitening method here since we want to handle any number of dimensions giving us our vector map batch normalization:
(N - + B
where T is the O-centered data given by
h= W(x E(x])
and where W is the Cholesky decomposition of the covariance matrix of the component dimensions. The scaling parameter y is a Dvm x Dvm matrix initialized to everywhere except the diagonal which is set to 1/vDvm in order to obtain modulus of for the variance of the normalized value. The shifting parameter / is a vector of length Dvm that is initialized to 0. It is worth noting that the Cholesky decomposition is expensive and scales O(z²) with the number of dimensions of the imput.
4 EXPERIMENTS AND RESULTS
We perform three sets of experiments designed to baseline performance, compare against some known quaternion results, and to test extreme cases of dimensionality in the data. This is done by simple classífication on CIFAR data using different size ResNet models for real, quaternion, and vector map. The second experiment replicates the results of colorizing images using a convolutional auto-encoder from Parcollet et al. (2019), but using vector map convolution layers. Lastly, the DSTL Satellite segmentation challenge from Kaggle (Kaggle 2014) is used to demonstrate the high parameter count reduction when vector map layers are used for high dimensional data. We purposefully choose commonly used architectures for comparison since we are not aiming for tfate-oftthe-ar results, but rather to show we maintain the benefits of the Hamilton product while dropping dimensionality constraints.
4.1 CIFAR CLASSIFICATION USING RESNET
4.1.1 CIFAR METHODS
These experiments use the ResNet architecture to perform simple image classification using CIFAR- 10 and CIFAR- 100 datasets (Krizhevsky & Hinton 2009). The CIFAR datasets are 32 32 color images of 10 and 100 classes respectively. Each image contains only one class and labels are provided. Since the CIFAR images are RGB, we use Dym 3 for all the experiments.
For the architecture we use different Residual Networks taken directly from the original paper He et al. (2015a). We ran direct comparisons between real-valued, quaternion, and vector map networks on three different sizes: 18N ResNet34, and ResNet50. The only change from the real-valued to vector map networks is that the number of filters at each layer changed such that the parameter count is roughly the same as the real- vealev network. The batch size was 64 for all experiments. All other parret-prraeeters remain the default from the original real-valued implementation.
All use the backpropagation algorithm with Stochastic Gradient Descent with Nesterov momentum set at 0.9. The norm of the gradients were clipped to and custom learning rate scheduler was used. The learning rate was initially set to 0.01 for the first 10 epochs and then set to 0.1 from epoch 1 11-100 and then cut by factor of 10 at epochs 120 and 150.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Percent error for classification on CIFAR-10 and CIFAR-100 where Params is the total number of parameters.
Architecture
Params
CIFAR-10
CIFAR-100
8 Real
11,173,962
5.92
27.81
ResNet18 Quaternion
8,569,242
5.92
28.77
ResNet18 Vector Map
7.376,320
6.05
27.18
ResNet34 Real
21,282,122
5.73
28.18
ResNet34 Quaternion
16,315,610
5.73
27.24
ResNet34 Vector Map
14,044,960
5.55
25.88
ResNet50 Real
23,520,842
6.10
27.40
ResNet50 Quaternion
18,080,282
6.10
27.32
ResNet50 Vector Map
15,559,120
5.72
25.16
4.1.2 CIFAR RESULTS
The results are shown in Table [ for all experiments. The vector map network's final accuracy outperforms the other models in all cases except one and the accuracy rises faster than both the real and quaternion valued networks. This may be due to the ability to control the relationships of each color channel in the convolution operation, while the quaternion is stuck to its set algebra, and the real For future insight into the last statement we investigated the L values after training and observed that they became normally distributed around -1 and 1, meaning they did in fact change some of the relations.
combining the color channels in similar fashion either.
4.2
CONVOLUTIONAL
FOR COLOR RECOVERY
4.2.1 CAE METHODS
This experiment originally was to explore the power of quaternion networks over real-valued by investigating the impact the Hamilton product had on reconstructing color images from gray-scale only training (Parcollet et al. 2019). A convolutional encoder-decoder (CAE) was used to test color image reconstruction. We performed the exact same experiment using quaternions, but also two experiments using vector map layers with Dvm 3, one with L frozen for training and one allowing L to update as normal. This way we can test if we mimic the quaternion results with three dimensions by capturing the important components of treating the input dimensions as single entity with three dimensions and also seeing the potential gain of learning L. The identical architecture is used, two convolutional encoding layers followed by transposed convolutional decoding layers.
A single image is first chosen, then converted to gray-scale using the function GS((z.y). where Pr,y the color pixel at location y. The gray value is concatenated three times for each pixel to create the input to the vector map CNN. We used the exact same model architecture, but since the output feature maps is three times larger in the vector map model reduce their size to 10 and 20. The kernel size and strides 3 and for all layers. The model is trained for 3000 epochs using the Adam optimizer (Kingma & Ba 2014) using learning rate of 5e 44 The weights are initialized following the above scheme in Section 3.2 and the hardtanh (Colloberi) 2004) activation function is used in both the convolutional and transposed convolutional layers.
4.2.2 CAE RESULTS AND DISCUSSIONS
The results can be seen in Fig. 2 where one can see the vector map CAE was able to correctly produce color images like the quaternion CAE. Similar to the quaternion CAE, the vector map CAE appears learn to preserve the internal relationship between the pixels similar to the Hamilton. The reconstructed images were also evaluated numerically using the peak signal to noise ratio (PSNR) (Turaga al. 2004) and the structural similarity (SSIM) (Wang et al. 2004). These evaluations appear in Table 2
The main goal of this experiment was test if there exists property of the quaternion structure that may have not been captured with the attempted generalization of vector map convolutions. Since both vector map networks perform similarly the quaternion network it appears that the way the vector map rules are constructed enable it to capture the essence of the Hamilton product for any dimension size Dum and the additional aspects of the algebraic structure are not important. Since the Dym 3 model matched the quaternion performance, we have shown that the same performance can be achieved with fewer parameters.
6

---

Under review as a conference paper at ICLR 2021
Original
Quaternion
Vector Map
Train
Figure 2: Grey-scale to color results on two KODAK images for quaternion CAE and vector map CAE
Table 2: PSNR and SSIM results for vector map CAE with both frozen L and non-frozen L compared to quaternion CAE.
Quaternion PSNR
D 3 frozen L
Dvm PSNR
learned L SSIM
Image
SSIM
PSNR
SSIM
kodim23 kodim04
31.68dB 28.57dB
0.961
32.03dB 29.18dB
0.973
32.52dB
0.981 0.972
0.954
0.962
30.55dB
4.3 DSTL USING UNET
The DSTL Satellite Imagery Feature Detection challenge was run on Kaggle (Kaggle, 2014) where the goal was to segment, or classify at pixel level, 10 classes from km x km satellite images in both b-nd and 16-band formats. Since satellite images have many more bands of information than standard RGB, it makes good use case for vector map convolutions. We run experiments using the full bands on both real-valued and vector map networks. Note the top scoring solutions used an ensemble of several models, some trained on only particular classes, while we are doing end-to-end training on basic model to do comparison of real-valued to vector map networks.
4.4 DSTL METHODS
Both models use a standard UNet base as described in the original paper Ronneberger et al. (015)). We use the entire 16-band format input, simply concatenating them into one input the models. For the vector map network we choose Dym = 16 to treat the entire 16-band input as a single entity. The real-valued model has a starting filter count of 32, while the vector map has a starting filter count of 96. The images are very large we sample from them in sizes of 82* x 82 pixels, but only use the center 64* x 64 pixels for the prediction. For training, the batch size is set to 32, the learning rate le-3, and we decay the learning rate by factor of 10 every 25 epochs during a total of 75 epochs. The cost function used is Intersection Over union (IOU), also known the Jaccard Index, which is defined as:
TOU = TP/(TP + FP FN)
where TP is the number of true positives, FP is the number of false positives, and FN is the number of false negatives.
7

---

Under review as a conference paper at ICLR 2021
Table 3: IOU score DSTL Satellite segmentation challenge for real-valued and vector map UNet models where Params is the total number of parameters.
Architecture
Params
IOU Score
UNet Real
7,855,434
0.427
UNet Vector Map
5,910,442
0.436
Some of the classes of the data are only in couple of images. For this reason, we train all available images, but hold out random 400 x 400 chunks of the original images. We use the same seed for both the real-valued and vector map runs.
4.5 DSTL RESULTS
The results are shown in Table 3 where one can see that for a lower parameter budget, the vector map achieved better segmentation performance. The loss curves (not shown) display near identical shapes, but the vector map starts a higher score and ends at higher score. Some of the features, like the vegetation and water, stand out more distinctly in the non-RGB bands of information and the vector map seems to have captured this more accurately. The main goal was to show that the vector map convolutions could handle large number of imput dimensions and potentially better capture how the channels relate to one another, which was successful.
5 CONCLUSIONS
This paper proposes vector map convolutions generalize the beneficial properties of convolutional complex and hyper/complex networks to any number of dimensions. We also introduce a new learnable parameter to modify the linear combination of internal features.
The first set of experiments compares performance of vector map convolutions against real-valued networks in three different sized ResNet models on CIFAR datasets. They demonstrate that vector map convolution networks have similar accuracy at reduced parameter count effectively mimicking hyper-comples networks while consuming fewer resources. We also investigate the distribution of the final values of L, the linear combination terms, and see that they also tend to stay around the value they were initialized to, suggesting that balanced magnitude linear combinations are near optimal.
We further investigated if vector map convolutions effectively mimic quaternion convolution in its ability to capture color features more effectively with Hamilton Product with image color reconstruction tests. The vector map convolution model not only can reconstruct color like the quaternion CAE, but it performs better indicated by PSNR and SSIM measures. This shows that other aspects of the quatermion algebra are not relevant to this task and suggests that vector map convolutions could effectively capture the internal relation of any dimension input for different data types.
The final experiment tested the ability of vector map convolutions to perform well very high dimensional input. We compared a real-valued model against a vector map model in the Kaggle DSTL satellite segmentation challenge dataset, which has 12 channels of image information and contains 10 classes. The vector map model was built with Dym- = 12 and not only had fewer learnable parameters than the real-valued model, it achieved a higher Jaccord score and learned faster rate. This establishes advantage of vector map convolutions in higher dimensions.
This set of experiments has shown that vector map convolutions appear to not only capture all the benefits of complex/typer-comples convolutions, but can outperform them using smaller parameter budget while also being free from their dimensional constraints.
REFERENCES
Igor Aizenberg and Alexander Gonzalez. Image recognition using mlmvn and frequency domain features. In 2018 International Joint Conference on Neural Networks (IJCN), pp. 1-8. IEEE, 2018.
Eduardo Bayro- Corro-Corro,,ano, Luis
and Marcela Garza-Burgos. Geometric techniques for robotics and hmi: Interpolation and haptics in conformal geometric algebra and control using quaternion spike neural net- works. Robotics and Autonomous Systems, (04772-44, 2018.
8

---

Under review as a conference paper at ICLR 2021
Ronan Collobert. Large scale machine learning. Technical report, IDIAP, 2004.
Chase Gaudet and Anthony S. Maida. Deep quatertion networks. In 2018 International Joint Conference on Neural Networks (IJCNN), 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249-256, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv 2015a.
preprint arXiv:
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference on computer vision, pp. 102100021000 1034, 2015b.
Sergey loffe and Christian Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprini 100vv:1502.03167 2015.
Teijiro Isokawa, Tomoaki Kusakabe, Nobuyuki Matsui, and Ferdinand Peper. Quaternion neural network and its application. In International conference on nowledge-baseed and intelligent information and engineering systems, pp. 318-324. Springer, 2003.
Kaggle. Kaggle dstl satellite imagery feature detection. https httpswwwwiggoomodddddlllleeeeeee 2014. Accessed: 2020-05-30.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412698) 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Master's thesis, University of Tront, 2009.
Hiromi Kusamichi, Teijiro Isokawa, Nobuyuki Matsui, Yuzo Ogawa, and Kazuaki Maeda. A new scheme for color night vision by quaternion neural network. In Proceedings of the 2nd International Conference Autonomous Robots and Agents, volume 1315. Citeseer, 2004.
Nobuyuki Matsui, Teijiro Isokawa, Hiromi Kusamichi, Ferdinand Peper, and Haruhiko Nishimura. Quaternion neural network with geometrical operators. Journal of Intelligent & Fuzzy Systems, 15(3, 4): 149-164, 2004.
Yurii Nesterov. A method of solving convex programming problem with convergence rate (1/k2).
Titouan Parcollet, Mohamed Morchid, and Georges Linares. Deep quaternion neural networks for spoken language understanding. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 504-511, IEEE, 2017a.
Titouan Parcollet, Mohamed Morchid, and Georges Linarès. Quaternion denoising encoder-decoder for theme identi- fication of telephone conversations. Proc. Interspeech 2017, pp. 3325-3328, 2017b.
Titouan Parcollet, Mirco Ravanelli, Mohamed Morchid, Georges Linarès, Chiheb Trabelsi, Renato De Mori, and Yoshua Bengio. Quaternion recurrent neural networks. arXiv preprint wrXiv:1806.04111 2018a.
Titouan Parcollet, Ying Zhang, Mohamed Morchid, Chiheb Trabelsi, Georges Linarès, Renato De Mori, and Yoshua Bengio. Quaternion convolutional neural networks for end-to-end automatic speech recognition. arXiv preprint arXiv:/806.07700 2018b.
Titouan Parcollet, Mohamed Morchid, and Georges Linarès. Quaternion convolutional neural networks for heteroge- neous image processing. In ICASSP 20/9-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8514-8518. IEEE, 2019.
Titouan Parcollet, Mohamed Morchid, and Georges Linarès. A survey of quaternion neural networks. Artificial Intelligence Review, (3(4))2957-2982 2020.
Dario Pavllo, David Grangier, and Michael Auli. Quaternet: quaternion- recurrent model for human motion. arXiv preprint arXiv: arXiv:/805.0660 1805.06485, 2018.
9

---

Under review as a conference paper at ICLR 2021
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: tation. International Conference on Medical image computing and computer-ussisted intervention, pp. 231- Springer, 2015.
Convolutional networks for biomedical image segmen-
Kazuhiko Takahashi, Ayana Isaka, Tomoki Fudaba, and Masafumi Hashimoto. Remarks quaternion neural network- based controller trained by feedback error learning. 2017 IEEE/SICE International Symposium on System Inte- gration (SII), pp. 875-880. IEFE, 2017.
Keiichi Tokuda, Heiga Zen, and Tadashi Kitamura. Trajectory modeling based on hmmss with the explicit relationship between static and dynamic features. In Eighth European Conference on Speech Communication and Technology. 2003.
C Trabelsi, Bilaniuk, Zhang, D Serdyuk, Subramanian, JF Santos, S Mehri, Rostamzadeh, Y Bengio, and C Pal. Deep complex networks. arxiv 2018. arXiv preprint arXiv: 17X5v:770509792, 2018a.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher Pal. Deep complex networks. networs.. In International Confer- ence Learning Representations 2018 (Conference Track), 2018b. htl/:ttppe//iixr.cc//offeeeeeeee arxiv:1705.099
Deepak S Turaga, Yingwei Chen, and Jorge Caviedes. No reference psnr estimation for compressed pictures. Signal Processing: Image Communication, 19(2):173-184, 2004.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-6) 612, 2004.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
K kernels
kemels
kernlls kernels
K feature maps feature maps
I feature maps
feature taps
M,, "R
M, "K
M, *K
*
*
*
X.Maps
Kommak
Hemeh
K. K,Kapo
KK
M
M *K
Da, "
*
*
EMage
1Mrp
"K
M
M,*K
M, M,
*
MMp
Homel
Mape
1:50pp
M. *K
"K
M, "K
M,",,
*
*
*
*
i.Mapa
9. M.Napa
Heanc
R.Wapa
Maps
M * M M M. *K
M, * M,
M, *
M, K M. M *K K M,
M, *K
M, K,
Figure 3: An illustration of quaternion convolution.
11

---

Under review as a conference paper at ICLR 2021
Figure 4: Randomly selected feature vector maps from the first convolution layer after training. Each row is different image, where the first column are the original input images.
12


Reviews:
[{"review_id": "oKTxWkDew_8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to explain the reason why the quaternion convolutional process reaches better performances than  neural networks with real values. this work is mostly based on previous results obtained in NLP conferences on QNN.\n\nThe idea is worth of interest but the results obtained during the experiments (from other previously already published papers) are not convincing and more efforts have to be provided in the theoretical side. For exemple, why convolution process map in a more efficient way real features in quaternion (color) space? How the Hamilton product emphasis the hidden relations between these features?\n\nIf this is mainly the fact of the Hamilton product, the authors have to give some words on the quaternion space obtained from the real valued feature map. Is this space invariant in regards to the real-valued input observed (even the batch may gives some basic combination of these input features at each tilmestep)?", "extra_scores": {}, "extra_text": {}}, {"review_id": "hWL44137ZYc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe paper proposes a method that generalizes complex and quaternion networks. The proposed method can work with vectors of any dimension. The benchmarks are conducted on CIFAR-10, CIFAR-100 and satellite imaging data DSTL. The paper also conducts an experiment on recoloration.\n\n# Quality\n\nWhile I like the general idea to explore the methods of using complex and hyper-complex representations, I found that this work is weak both in terms of the theoretical and experimental research.\n\nThe theoretical part of the paper proposes a method to multiply n-dimensional vectors similarly to complex and quaternion numbers. This method is not a generalization of the former two (as seen in Eq 2, 4). I am concerned that the paper does not discuss the properties of the operation. Such numbers do not form an algebra and they are not commutative. While quaternions are neither commutative they exhibit useful properties, like associations with rotations. \n\nAnother important aspect missing in the theoretical part of the paper is how to define the batch-norm.\n\nThe experimental part also looks weak. I am not convinced about the significance of CIFAR results. There is also no \"apple to apple\" comparison for the networks with the same amount of parameters. The results reported in Table 2 contradict the claim that the proposed method outperforms the quaternions.\n\nThe DSTL results look significant. I think, this result demonstrates the strength of the proposed method. Therefore I encourage to extend this section to give more details on this experiment. Even though, the result is not SOTA here, the previous results should be mentioned in the table (Kaggle winner entry is ~0.49).\n\n# Clarity\n\nThe paper is clearly written and easy to follow. The math is explained well and helps to build intuition about the proposed method. One point I found unclear if the L is shared or learned separately for each layer. This seems to be an important difference from complex and quaternion numbers. The paper would benefit from discussion of parameter L and ablation experiments on its importance.\n\n# Originality\n\nThe paper is original.\n\n# Significance\n\nThe paper is significant for the community. There are plenty of applications where we need to encode multidimensional data. While I do not believe that this work is a clear cut solution for such applications, it is a step in the right direction.\n\n# Conclusion\n\nThis is an ok paper, but needs many improvements:\n\n- Error bars for experiments\n- Comparisons to previously published results\n- It would greatly benefit from running this method with close to SOTA architecture, for example DenseNet for CIFAR\n- Discussions on the properties of the proposed operation\n- Derivations for batch-norm\n\n## EDIT: Update\n\nThe paper was improved. In particular I like the added explanation of the batch-norm and some improved explanation and phrasing.\n\nNevertheless, the experimentation remains weak both compared to the state of the art and previously published work (e.g. CIFAR reported in Trabelsi et al., 2017 and Gaudet & Maida, 2017).\n\nTherefore, I increase the score by one point, as this work is just very slightly above the acceptance threshold right now. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "WN6PHl3swaH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The purpose of this paper is to analyze the reasons why complex/hyper-complex neural networks yield performance improvements (especially pertaining to generalization). The authors argue that the underlying algebraic structure of complex/hyper-complex coordinate systems enable greater weight-sharing compared to usual real-valued networks. Inspired by this, the authors also propose the idea of vector map convolutions which captures the aforementioned properties but at the same time, are not subjected to the dimensionality constraints. Several empirical studies are also provided to demonstrate the potential effectiveness of vector map convolutions.\n\nI have to first acknowledge that I have not very familiar with much of the background literature on complex and quarternion networks. Also, for the experiments in Sections 4.2 and 4.3, I am not familiar with any of these data-sets as well as the evaluation measures (Jaccard score, etc.). Hence, I am not too confident in my assessment of this paper.\n\nDetailed Comments/Questions:\n\n- I would prefer a clearer and more comprehensive way of introducing the background work and how it leads to the intuition of vector map convolutions. It would be much better if the authors could provide exact mathematical descriptions of usual complex and quarternion networks (esp. as this paper is < 8 pages right now) and how vector map convolutions are the natural generalization- I don't see this as obvious based on looking at the equations in Sections 3.1\n\n- I assume that Equation (1) and the discussion above it is the definition of the vector map convolutions? So only right-shift permutations are used and not other permutations (i.e., a subgroup instead of the full symmetric group S_n?) \n\n- I think that the results for CIFAR-10/100 are potentially interesting as they show that the proposed approach achieves competitive performance with the standard approach while reducing parameter count significantly. However, there's far too little details given about the experimental setup; for example, what are the hyper-parameters used (Batch-size, learning rate, optimizer choices, batch-norm, dropout, etc.) and are they used in a consistent way when comparing between real, quarternion, and vector map? \n\n===============\nPost-rebuttal:\n\nI would like to thank the authors for their rebuttal and addressing some of my concerns. However, after reading the updated manuscript as well as the other reviews, I decide to maintain my current ranting. I would still like to see more rigour in the paper: more of the mathematics need to be fleshed out and how the proposed approach compares with the existing works in complex/quarternion networks in a more mathematical way. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "IglDyOh-F0s", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary\n\nThe authors propose a generalization of complex and quaternion algebra for use in convolutional neural networks, so called vector map convolutions. It is proposed that using these operations in a convolutional kernel will result in higher parameter efficiency for the same accuracy. The authors derive the functional formula for the vector map by generalizing the quaternion algebra proposed in a previous paper in the literature.\n\n---------------------------------------------------------------------------------------------------------------------------\nPositives and negatives\n\n+ The approach is very interesting from a theoretical perspective, and the derivation proposed is well motivated and sufficiently explained in the methods section.\n\n+ The figures are well constructed and illustrate the point very concisely.\n\n- The major problem with this paper is that it severely goes over the 8 page limit, which is hidden by putting a lot of figures that are crucial to understanding the main text in the appendix. I believe this is a violation of the submission rules.\n\n- The paper does not do a good enough job of discussing related approaches that also reduce parameter efficiency (e.g. separable convolutions, dynamic convolutions, stand-alone self-attention) or that mix channels in a similar spirit (e.g. ICLR 2020 “multiplicative interactions and where to find them”) \n\n- The results are not convincing. For the satellite data case, the authors show that their proposal achieves a slightly lower Jaccard Score with fewer parameters, but it’s not clear at all what is the Jaccard Score or what its scale is. So I have no idea how a “UNet Real” Architecture with 5.9M parameters would compare with the proposed Vector Map Unet.\n\n- In the case of CIFAR-10 and CIFAR-100 the architecture does seem to give some improvements in parameter efficiency. But how does it compare to previous work in parameter efficiency as discussed above?\n\n---------------------------------------------------------------------------------------------------------------------------\nRecommendation\n\nWith the paper in its current state I have to recommend a strong reject because it seems to be violating the conference guidelines by putting several crucial images beyond the 8 page limit. If this issue is addressed, the recommendation can be reviewed. In that case, I would still recommend more work in the experiments section (More comparisons for the image classification task, better explanation and comparisons for satellite task).\n\n---------------------------------------------------------------------------------------------------------------------------\nQuestions\n\nWhat is the Jaccard Score?\nWhile the networks are shown to be more parameter efficient, what about the FLOPS of the resulting networks? Are we effectively trading memory for compute?\n\n---------------------------------------------------------------------------------------------------------------------------\nFeedback (not related to the score)\n\nMy suggestion to the authors to gain some space would be to completely remove section 4.2 or move to appendix, and use that space for the actually important figures. Not only is current figure 2 occupying a lot of space while making a weak argument, the whole section does not help in understanding either why Vector Map convolutions work, nor persuade the reader that they are empirically better (in fact the PSNR results look quite a bit worse).\nIn general I would recommend the authors to tighten their writing quite a bit. For example the authors repeat quite a lot of motivation for Hamiltonian algebra. While it’s useful to provide some context in a paper, the authors could summarize that information a bit better so that the reader can get to the actual contribution, the Vector Map convolution faster. On the other hand, it might be worthwhile spending a bit more time explaining  the DSTL challenge, even with a figure, since most readers will not be familiar with it.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the updated version of this manuscript did motivate one reviewer to give the paper a marginal accept rating, all other reviewers really felt that the paper could use more work along the lines of their suggestions. The aggregate view of the reviewers is just not positive enough at this time to warrant an accept recommendation by the AC at this time. The work does seem to have promise and the authors are encouraged to continue to improve the paper for another round of peer review elsewhere.
Decision: Reject