Lightweight Convolutional Neural Networks By Hypercomplex Parameterization
Authors: Eleonora Grassucci
Aston Zhang
Danilo Comminiello
Venue: ICLR 2022 Submitted (2022)
Keywords: Hypercomplex Neural Networks
Lightweight Neural Networks
Quaternion Neural Networks
Parameterized Hypercomplex Convolutions
Hypercomplex Representation Learning

Abstract: Hypercomplex neural networks have proved to reduce the overall number of parameters while ensuring valuable performances by leveraging the properties of Clifford algebras. Recently, hypercomplex linear layers have been further improved by involving efficient parameterized Kronecker products. In this paper, we define the parameterization of hypercomplex convolutional layers to develop lightweight and efficient large-scale convolutional models. Our method grasps the convolution rules and the filters organization directly from data without requiring a rigidly predefined domain structure to follow. The proposed approach is flexible to operate in any user-defined or tuned domain, from 1D to $n$D regardless of whether the algebra rules are preset. Such a malleability allows processing multidimensional inputs in their natural domain without annexing further dimensions, as done, instead, in quaternion neural networks for 3D inputs like color images. As a result, the proposed method operates with $1/n$ free parameters as regards its analog in the real domain. We demonstrate the versatility of this approach to multiple domains of application by performing experiments on various image as well as audio datasets in which our method outperforms real and quaternion-valued counterparts. 

Under review as a conference paper at ICLR 2022
LIGHTWEIGHT CONVOLUTIONAL NEURAL NETWORKS BY HYPERCOMPLEX PARAMETERIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Hypercomplex neural networks have proved to reduce the overall number pa- rameters while ensuring valuable performances by leveraging the properties of Clifford algebras. Recently, hypercomplex linear layers have been further im- proved by involving efficient parameterized Kronecker products. In this paper, we define the parameterization of hypercomplex convolutional layers to develop lightweight and efficient large-scale convolutional models. Our method grasps the convolution rules and the filters organization directly from data without requiring rigidly predefined domain structure to follow. The proposed approach is flexi- ble to operate in any user-defined or tuned domain, from 1D to nD regardless of whether the algebra rules preset. Such malleability allows processing multi- dimensional inputs in their natural domain without annexing further dimensions, done, instead, in quaternion neural networks for 3D inputs like color images. As result, the proposed method operates with 1/nn free parameters as regards its analog the real domain. We demonstrate the versatility of this approach mul- tiple domains of application by performing experiments on various image datasets as well as audio datasets which our method outperforms real and quaternion- valued counterparts.
INTRODUCTION
Recent state-of-the-ard convolutional models achieved astonishing results in various fields of appli-
cation by large-scaling the overall parameters amount (Karras al., 2020; d' d'Ascoli et al. [02T] Dosovitskiy et al. 2021). Simultancously, quaternion neural networks (QNNs) demonstrated to sig- nificantly reduce the number of parameters while still gaining comparable performances (Parcollet et al. 2019c Grassucci et al. 2021a: Tay et al.) 2019). Quaternion models exploit hypercomplex algebra properties, including the Hamilton product, to painstakingly design interactions among the imaginary units, thus involving 1/4 of free parameters with respect to real-valued models. Fur- thermore, thanks to the modelled interactions, quaternion networks capture internal latent relations in multidimensional inputs and preserve pre-existing correlations among dimensions. Therefore, the quaternion domain is particularly appropriate for processing 3D or 4D data, such as color im- ages (up to) 4-channel signals. Unfortunately, most common color image datasets contain RGB images and a zero channel has to be padded to the input in order to encapsulate the image in the four quaternion components. Additionally, while quaternion neural components are widespread and easy to be integrated in pre-existing models, very few attempts have been made to extend models higher domain orders. Accordingly, the development of hypercomplex convolutional models for larger multidimensional inputs, such as magnitudes and phases of multichannel audio signals or 16- band satellite images, still remains painful. Recently, parameterized hypercomplex multiplication (PHM) for fully connected layers have been proposed to generalize hypercomplex multiplications as sum of Kronecker products, going beyond quaternion algebra. However, no solution exists for convolutional layers, which remain the most employed layers when dealing with multidimensional inputs, such as images and audio signals (Wu et al., 2021 Hershey et al. 2017).
In this paper, we propose a parameterized hypercomplex convolutional (PHC) layer to define lightweight large-scale neural models admitting any multidimensional input, whichever the num- ber of dimensions. Our method is flexible to operate in domains from 1D to nD, where can be arbitrarily chosen by the user or tuned to let the model performance lead to the most appropriate do- main for the given input data. Such malleability comes from the ability of the proposed approach

---

Under review as a conference paper at ICLR 2022
to subsume algebra rules to perform convolution regardless of whether these regulations are preset or not. Thus, neural models endowed with our approach adopt 1/n of free parameters with respect to their real-valued counterparts, and the amount of parameter reduction is user choice. This makes PHC layers adaptable to a plethora of applications and hardware, even on the edge where memory saving is a crucial aspect. Additionally, the PHC versatility allows processing multidimensional data in its natural domain by simply setting the dimensional hyperparameter images can be analyzed in their RGB domain by setting n 3 without adding any useless informa- tion, contrary to standard processing for quaternion networks with the padded zero-channel. Indeed, PHC-based models are able grasp the proper algebra from input data, while capturing internal correlations among the image channels and saving 66% of free parameters.
For instance, color
On a thorough empirical evaluation on multiple benchmarks, we demonstrate the flexibility of our method that can be adopted in different domain of applications, from images to audio signals. We devise a set of convolutional models endowed with PHC layers for large-scale image classification and sound event detection tasks, letting them operate in different hypercomplex domain and with various input dimensionality with n ranging from 2 to 16.
The contribution of this paper is three-fold.
We introduce parameterized hypercomplex convolutional (PHC) layer, which aims building lightweight and more efficient large-scale convolutional models. The approach grasps the convolution rules directly from data via backpropagation exploiting the Kro- necker product properties, thus reducing the number of free parameters to 1/n.
We show how the proposed upproach can be employed with any kind of multidimensional data by easily changing the hyperparameter Indeed, by setting n 3 the PHC-based models can process RGB images in their natural domain, while leveraging the properties of hypercomplex algebras, allowing parameters sharing inside the layers and leading to parameter reduction to 1/3. To the best of our knowledge, this is the first approach that pro- cesses color images with hypercomplex-bassee channel. As well, multichannel audio signals can be analysed by simply considering = = for standard first-order ambisonics (which has 4 microphone capsules), of two ambisonics microphones, or even = 16 if we want to include the information of each channel phase.
neural models without adding any padding
for an array
We devise family of PHC neural networks, redefining common ResNets, VGGs and Sound Event Detection networks (SEDnets), operating in any user-defined domain just by choosing the hyperparameter n. which also drives the number of convolutional filters.
The rest of the paper is organized as follows. In Section 2 we recapitulate real and quaternion-valued
convolutional layers. Section 3 theoretically introduces the proposed method and expounds how to process RGB images with n 3, while in Section g we define the PHC models and we present the experimental evaluation, including datasets, experiment details and results. Finally, Section 5 reports the related works and in Section 6 we draw conclusions.
2 REAL AND QUATERNION -VLUUD CONVOLUTIONAL LAYERS
A brief recall on real-valued and quaternion convolutional layers appropriate to better understand the proposed method. A generic convolutional layer can be described by
= Conv(x) = W + x + b,
(I)
where the input x R is convolved (*) with the filter tensor W Raxdxk*k to produce the output E Rddx where k is the filter size. The bias term b does not heavily influence the number of parameters, thus the degrees of freedom for this operation are essentially Osskk2).
Quaternion convolutional layers, instead, build the weight tensor W E Rexdxkxk by following the Hamilton product rule and organize filters according to
Full code is available http::://nnoyymooss.opp...
2

---

Under review as a conference paper at ICLR 2022
Wo W1 W W
Xo
WI + Wo W. + W
X1
W*x-
W2 + W3 + Wo W
W. W + W + Wo.
X3_
where Wa, W1, W2. W3 E R the real coefficients of the quaternion weight matrix = WD + W W2] + W3 and Xo: X1, X2, X; are the coefficients the quaternion input x with the same structure. The imaginary units comply with the property 12 with the non-commutative products ij - -ū; ja -Rj; Ri ii.
R2 -1 and
As done for real-valued layers, the bias be ignored and the degree of freedom computations of the quaternion convolutional layer can be approximated to (sskk /4). The lower number of parameters with respect the real-valued operation is due to the reuse of filters performed by the
Hamilton product in Eq 2 Also, sharing the parameter submatrices forces to consider and exploit
the correlation between the input components (Parcollet et al. 2019a Tay et al. 2019
Gaudet &
Maida 2018).
3
PARAMETERIZING HYPERCOMPLEX CONVOLUTIONS
3.1 PARAMETERIZED HYPERCOMPLEX CONVOLUTIONAL LAYERS
the following, we delineate the formulation for the proposed parameterized hypercomplex convo- lutional (PHC) layer. We also show that this approach is capable of learning the Hamilton product rule when two quaternions are convolved. The PHC layer is based on the construction, by sum of Kronecker products, of the weight tensor H which encapsulates and organizes the filters of the convolution. The proposed method is formally defined as:
y PHC(x) H x
whereby H E Baxdxxxk is built by sum of Kronecker products between two learnable matrices. Here, is the input dimensionality the layer, the output one, and k is the filter size. More concretely,
H Σ OF
(4)
in which A, E R"xn with I...., n are the matrices that describe the algebra rules and F, R RAx x x&xk represents the i-th batch filters that are arranged by following the algebra rules to compose the final weight matrix. The core element of this approach is the Kronecker product, which a generalization of the vector outer product that can be parameterized by n. The hyperparameter be by the user who wants to operate in pre-defined real or hypercomplex domain (e.g., by setting n the PHC layer is defined in the complex domain, or in the quaternion one n
1
0
0
0
0
0
0 0
H
Figure 1: The quaternion convolution rule can be expressed sum of Kronecker products between the matrices A, that subsume the algebra rules and the matrices F; that contain the convolution filters, with = 1, 2, 3, 4. In this example, the parameters of A, are fixed for visualization purposes, PHC layers they are learnable parameters.

---

Under review as a conference paper at ICLR 2022
is set equal to 4, Figure [ [illustrates),, tuned obtain the best performance from the model. The matrices A, and F, are learnt during training and their values are reused to build the definitive tensor H.
The degree of freedom of A; and F, are n² and sdk² /n, respectively. Usually, real world ap- plications employ large number of filters in layers (s, d 256, 512, and small values for k. Therefore, frequently sdk² > n holds. Thus, the degrees of freedom for the PHC weight matrix can be approximated to O(sdk2/n) Hence, the PHC layer reduces the number of parameters by 1/n with respect to a standard convolutional layer in real world problems.
Moreover, when processing multidimensional data with correlated channels, such color images, rather than mulichannel audio or multisensor signals, PHC layers bring benefits due to the weight sharing among different channels. This allows capturing latent intra-channels relations that standard convolutional networks ignore because of the rigid structure of the weights (Grassucci et al. 2021a Parcollet et al.j 2019b). The PHC layer is able to subsume hypercomplex convolution rules and the desired domain is specified by the hyperparameter n. We test this ability with toy problems in Appendix B Interestingly, by setting n = 1 real-valued convolutional layer can be represented too. Indeed, standard real layers does not involve parameters sharing, therefore the algebra rules are solely described by the single A E R 1x1 and the complete set of filters are included Fsxdxkxi We report detailed explanations in Appendix A
3.2 PARAMETERIZED LAYERS FOR COLOR IMAGES
In this section, we describe how the PHC layer be employed process color images in hy- percomplex domains without needing any additional information to the input. Different encodes exist to process color images, however, the most common computer vision datasets are comprised of three-channe images in 1 In the quaternion domain, RGB images are enclosed into quater- nion and processed single elements (Parcollet et al.) (019a).. The encapsulation is performed by considering the RGB channels as the real coefficients of the imaginary units and by padding a zeros channel as the first real component of the quaternion.
Here, we propose to leverage the high malleability of PHC layers to deal with RGB images in hypercomplex domains without embedding useless information to the input. Indeed, the PHC can directly operate in g by easily setting = 3 and process RGB images in their natural domain while exploiting hypercomplex algebra properties such as parameters sharing.
Indeed, the great flexibility of PHC layers allows the user to choose whether processing images in IR or R. On one hand, by setting n = 4, the zeros channel is added to the input even so the layer saves the 75% of free parameters. On the other hand, by choosing = 3 the network does not handle any useless information, notwithstanding. it reduces the number of parameters by solely 66%. This is trade-off which may depend the application Furthermore, the domain on which processing images can be tuned by letting the performance of the network indicates the best choice for
the hardware the needs.
3.3 PARAMETERIZED LAYERS FOR MULTICHANNEL AUDIO
In the following, we expound how the proposed PHC layer can be employed to deal with multi- channel audio signals. For instance, first-order Ambisonics (FOA) is composed of 4 microphone capsules, whose magnitude representations can be enclosed in a quaternion (Comminiello et al. 2019 Ricciardi Celsi et al. 2020). However, the quaternion algebra may be restrictive if more than one microphone employed for registration or whether the phase information has to be included too. Indeed, quaternion neural networks badly fit with multidimensional input with more than channels.
Conversely, the proposed method can be easily adapted to deal with these additional dimensions by handily setting the hyperparameter n and thus completely leveraging each information in the n-dimensional input.
4

---

Under review as a conference paper at ICLR 2022
4 CONVOLUTION-BASED NEURAL MODELS WITH PHC LAYERS
In this section, we expound neural models endowed with PHC layers and we demonstrate how the proposed method can be employed in various spheres of application. To this end, devise deep PHC models for image classification and for sound event detection. For each task, we empirically substantiate the effectiveness and flexibility of the proposed PHC layer. In order be consistent with the literature, we perform each experiment with a real-valued baseline model and then we compare with its quaternion counterpart and with the proposed PHC network. Furthermore, we assess the PHC malleability testing different values of the hyperparameter therefore defining PHC models in multiple hypercomplex domains.
4.1
HYPERCOMPLEX PARAMETERIZATION FOR IMAGE CLASSIFICATION
To begin with, test the PHC layer RGB images and show how, exploiting the correlations among channels, the proposed method saves parameters while ensuring high performances.
4.1.1
PARAMETERIZED HYPERCOMPLEX RESNETS
In recent literature, a copious set of high performance in image classification is obtained with models having residual structure. ResNets (He et al. 2016) pile up manifold residual blocks composed of convolutional layers and identity mappings. generic PHC residual block is defined by
y = F(x, (H,)) +x,
(5)
whereby H, the PHC weights of layer j 1, 2 in the block, and F is
F(x, (H₂)) = PHC (ReLU (PHC(x)).
(6)
4.1.2 PARAMETERIZED HYPERCOMPLEX VGGs
Another family of popular methods for image classification is based the VGG networks (Si- monyan & Zisserman 2015) that stack several convolutional layers and a closing fully connected classifier. To completely define models in the desired hypercomplex domain, we propose to endow the network with PHC layers as convolution components and with Parameterized Hypercomplex Multiplication (PHM) layers (Zhang al. 202T) as linear classifier. The backbone of our PHC
VGG is then
h, ReLU (PHC, (h,-1))
t= 1,...
y ReLU (PHM(h;)).
We also test hybrid model by employing standard real-valued fully connected (FC) layers instead of PHM ones. However, we believe that it is out of the scope of this paper which aims at exploiting hypercomplex algebras properties to define highly malleable neural networks. Nevertheless, for further comparison, report these additional experiments in the Appendix B
4.1.3 EXPERIMENTAL SETUP AND EVALUATION
We perform the image classification task with five baseline models. We consider ResNet18, ResNet50 and ResNet152 from the ResNet family and VGG16 and VGG19 from the VGG one. Each hyperparameter is set according to the original papers (He et al. 2016 Simonyan & Zisser- man 2015). We investigate the performance in four different color images datasets at different scales. We employ SVHN, CIFAR10, CIFAR 100, and ImageNet and any kind of data augmentation applied to these datasets in order to guarantee fair comparison.
We execute initial experiments with VGGs against Quaternion VGGs and two versions of PHC VGGs with n equal to and to 4. Average and standard deviation accuracy over three runs are reported for SVHN and CIFARIO 10 datasets in Table [ Both the VGG16 and VGG19 PHC-based versions clearly outperform real and quaternion counterparts while being built with more than a half the number of parameters of the baseline. Additionally, PHC-based models extraordinarily reduce the number of training and inference time required with respect to the quaternion model which oper- ates in hypercomplex domain well. Furthermore, when scaling up the experiment with VGG19,
5

---

Under review as a conference paper at ICLR 2022
Table 1: Image classification results for VGG. The accuracy mean and standard deviation three runs with different seeds is reported. Training (T) time and inference (I) time required on CIFARIO For training time we report, in seconds per 100 iterations, the mean and the standard deviation over the iterations in one epoch, while the inference time is the time required to decode the test set. The PHC model with n 4 outperforms the quaternion counterpart both in terms of accuracy and time. The PHC with = 2 far exceeds the real-valued baseline in the considered datasets, while both the PHC VGG19 versions with 71 2, 4 are more efficient than the real and quaternion-valuee baselines at inference time.
Model
Params
SVHN
CIFARIO
Time (T)
Time (I)
VGGI6
15M
94.364 + 0.394
85.067 + 0.765
2.2 0.02
1.2
Quaternion VGG16
3.8M (-75%)
93.887 = 0.292
83,997 = 0,493
5.2 0.02
2.2
PHC VGG16 n = 2
7.6M (-50%)
94,831 + 0.257
86.510 0.216
3.2 0.02
1.4
PHC VGGI6 n
3.8M (-75%)
94.639 + 0.121
85.640 + 0.205
3.2
1.4
VGGI9
29.8M
94.140 0.129
85.624 0.257
3.2 0.02
16.0
Quaternion VGG19
7.5M (-75%)
93.983 = 0.190
83.914 0.129
6.2 + 0.02
16.3
PHC VGG19 2
14.9M (-50%)
94.553 0.229
85.750
0.286
4.0 0.02
15.4
PHC VGG19 n = 4
7.4M (-75%)
94.169 = 0.296
84.830 0.733
4.2 1 0.02
15.5
Image Classification results with ResNet models. Each experiment is run three times with different seeds and mean with standard deviation is reported. The proposed models far ceed real-valued and quaternion baselines almost in each experiment we conduct. Interestingly, the PHC model outperform the real-valued counterpart by 4% points in the largest-scale experiment CIFARI00. The time is similar to the claims in Table so we do not add here to avoid redundancy.
Table 2:
Model
Params
FLOPs
SVHN
CIFAR10
CIFAR100
ResNet18
10.1M
1.01G
93,992 + 1.317
89.543 + 0.340
62.634 + 0.600
Quaternion ResNet18
2.8M (-75%)
1.0GG
93.661 1 0.413
88.240 0.377
59,850 +: 0.607
PHC ResNet18 n 5.4M (-50%)
1.03G
94.359 0.187
89.260 0.625
60.320 2.249
PHC ResNet18 n
3.6M (-66%)
1.03G
94.303
1.234
89.603 1 0.563
62.660 1.067
PHC ResNet18 n
2.7M (-75%)
1.03G
94.234 + 0.161
88.847 + 0.874
61.780 0.689
ResNet50
22.5M
2.36G
94.546 + 0.269
89.630 + 0.305
65.514 0.569
Quaternion ResNet50
5.7M (-75%)
2.36G
93.685 = 0.389
89.670 = 0.383
63.760 + 0.717
PHC ResNet50
11.1M (-50%)
2.41G
93.849 + 0.249
89.750 + 0.386
65.884 0.333
PHC ResNet50 n =
7.6M (-66%)
2.1G
93.617 + 0.497
90.423 + 0.145
66.497 1.256
PHC ResNet50 n =
5.7M (-75%)
2,4
94.558 + 0.754
88.897 + 0.645
66.240 1 1.16S
R5sNet152
52.6M
6.62G
94,625 1 0.355
89.580 1 0.173
62.053 0.385
0.3885
Quaternion ResNet152
13.2M (-75%)
6.62G
93.638 + 0.098
89.227 - 0.287
61.267 + 0.784
PHC 55
26.6M (-50%)
6.76G
93.915 0.512
90.540 0.401
65.817 0.327
PHC R5sNet152 n 3
17.8M (-66%)
6.76G
93.955 0.152
90.077 1 0.436
66.347 0.567
PHC ResNet152 n = 4
13.4M (-75%)
6.76G
94.290 = 0.237
89.897 = 0.097
66.437 + 0.064
the proposed methods are more efficient at inference time with respect to the real-valued VGG19. Therefore, PHC models can be easily adopted in applications with disk memory limitations, due to the reduction of parameters, and for fast inference problems thanks to the efficiency testing time. More initial experiments with various networks on other datasets are reported in Appendix B
The malleability of the proposed approach when dealing with color images is expressed in the op- portunity to choose the domain in which operating. Therefore, we test PHC networks in the complex HH (n 2). quaternion 4 (n 4) or H (n 3) domain, where in the latter we do not concatenate any zero padding and process the RGB channels of the image their natural domain. In order to implement this experiment,
change the number of filters in the baseline and in the corresponding compared networks to be divisible by each value of n. More details about the implementations are provided in Appendix B and in the GutHub repository (link in Section

---

Under review as a conference paper at ICLR 2022
Table 3: Storage memory required for ResNet152s checkpoints CIFAR100. Quaternion and PHC models allow a considerable disk memory saving with respect to the real- valued ResNet.
Model
Storage Memory
ResNet152
201 MB
Quaternion ResNet152
51 MB (-75%)
PHC ResNet152 n 103 MB (-49%)
PHC ResNet152 n = 3
70 MB (-65%)
PHC ResNet152 n 4
53 MB (-74%)
Table ImageNet classification with real-valued baseline against our best model PHC 3. Our approach outperform the baseline while saving the 66% of parameters.
Model
Params
ImageNet
ResNet50
25.7M
67.990
PHC ResNet50nn n 3 9.6M (-66%)
68.584
Table 2 presents the the mean and standard deviation accuracy three runs with different seeds for the ResNet-based models. We perform extensive experiments and the PHC models with n = always outperform the quaternion counterpart gaining higher accuracy and being more robust. This underlines the effectiveness of the PHC architectural flexibility over the predefined and rigid structure of quaternion layers. Furthermore, our method distinctly far exceeds the corresponding real-valued baselines across the experiments while saving from 50% to 75% parameters. Focusing on the latter result, the PHC model with proving the validity of processing RGB images in their natural domain leveraging hypercomplex algebra. However, performance with hyperparameter may depend on the application on the hardware employed. On one hand, n 4 may sometimes lead to lower performances, nevertheless it allows saving disk memory, as shown in T thus it may be more appropriate for edge applications. On the other hand, processing color images with n = 3 may bring higher accuracy even so it requires more parameters. Therefore, such flexibility makes PHC models adaptable to a large range of applications. Likewise, PHC networks with n 2 gain considerable accuracy scores with respect to the real-valued corresponding models and, due the larger number of parameters with respect to the PHC with n sometimes outper- form it too. Finally. the PHC with n = 4 obtains the overall best accuracy in the largest experiment of this table. Indeed, considering ResNet152 backbone on CIFARI00 100, our method exceeds the real-valued bascline by more than 4%. This is the empirical proof that, with a very small FLOPs in- crease, PHC models well scale to large real-worke problems by notably reducing the overall number of parameters. What is more, in Table we report the memory required to store models checkpoints for inference. Our method crucially reduces the amount of disk memory demand with respect to the heavier real-valued model.
results to the most suitable choice in many cases,
and
are comparable, thus the choice of this
Further, we perform the image classifcation task on the ImageNet dataset. Since the most valu- able choice for when dealing with RGB images has been proved to be 3, we test the real-valued ResNet50 against the PHC counterpart with n = 3. We train the models for 300k iterations with batch size 256 following the recipes in (Wightman et al. 2021). Table 4 shows that the proposed method achieves comparable, and even slightly superior, performance than the real-valued baseline, while involving 66% fewer parameters. This proves the robustness of the proposed PHC approach, which can be adopted and implemented in models at different scales.
4.2 HYPERCOMPLEX PARAMETERIZATIINN FOR SOUND EVENT DETECTION
Sound event detection (SED) is the task of recognizing the sounds classes and at what temporal instances these sounds active in audio signal (Mesaros et al. 2021). We prove that the PHC layer is adaptable to n-diimnnsiona input signals and, due to parameters reduction and hypercomplex algebra, is more performing in terms of efficiency and evaluation scores.
7

---

Under review as a conference paper at ICLR 2022
Table 5: SEDnets results with microphone (4 channels input). Scores are computed over three runs with different seeds and we report the mean. The proposed method wtih n 2 far exceeds the baselines in each metric considered.
Model
Conv params
F Fscore 1
ER L
Score
score J
P
R
SEDnet Quaternion SEDnet
1.6M
0.637
0,450
0,406
0.756
0,5505
0.4M (-75%)
0.580
0,516
0.468
0.724
0.484
PHC SEDnet 2 0.8M (-50%)
0.680
0.389
0.355
0.767
0.611
PHC SEDnet n = 0.4M (-75%)
0.638
0.453
0.407
0.765
0.547
Table 6: SEDnets results with two microphones (8 channels input). Scores are computed over three runs with different seeds and we report the mean. The PHC SEDnet outperform the baselines.
Model
Conv Params
F Scooee f
ER 4
SED score
P 1 R
SEDnet
1.6M
0.663
0.428
0.383
0.788
0.572
Quaternion SEDnet
0.4M (-75%)
0.559
0.556
0.499
0.754
0.444
PHC SEDnet π 2
0.8M (-50%)
0.669
0,406
0.368
0.767
0.594
PHC SEDnet n 4
0.4M (-75%)
0.638
0.433
0.397
0.729
0.567
PHC SEDnet n2 = 8
0.2M (-87%)
0.553
0.560
0.503
0.747
0.439
4.2.1 PARAMETERIZED HYPERCOMPLEX SEDNETS
Sound Event Detection networks (SEDnett) (Adavanne et al. 2019) are comprised of core con- volutional component which extracts features from the input spectrogram. The information is then passed to a gated recurrent unit (GRU) module and to stack of fully connected (FC) layers with a closing sigmoid which outputs the probability the sound is in the audio frame. Formally, the PHC SEDnet is described by
h. (h--(
t= t......
y (FC (GRU (h₂)))-
After the GRU model, We employ standard fully connected layers, that can be also implemented as PHM layers with n = since the so processed signal loses its multidimensional original structure.
4.2.2
EXPERIMENTAL SETUP AND EVALUATION
For sound event detection models consider the augmented version of the SELDnet (Adavanne cct al., 2019 Comminiello et al. 2019) which was proposed as baseline for of the L3DAS2I Chal- lenge Task 2 (Guizzo al. 2021 and we perform our experiments with the corresponding released 2
d We consider as our baselines the SEDnet (without the localization part) and its quaternion counterpart. The L3DAS21 Task 2 dataset contains 15 hours of MSMP B-format Ambisonics audio recordings, divided in 1-minute-long data points, each of which consists of a simulated 3D office audio environment where up acoustic events may overlap. We perform experiments with mul- tiple configurations of this dataset. We first test the recordings from one microphone considering the magnitudes only (4 channels input), then we test the networks with the signals recorded by two microphones and magnitudes only (8 channels input). In A report the experiments and the results up to channels input, which we consider the phase information too.
We investigate PHC SEDnets in complex, quaternion and octonion domain with = 2, 2,4,8 8 and train each network for 1000 epochs with a batch size of 16. Other hyperparameters are set as sug- gested in the paper (Guizzo al. 2021 I) and more details are provided in Appendix B To assess the performances of our models we take various metrics into account: the Fscore on the detection metric, Precision (P) and Recall (R) suggested in the original paper. For more rigorous evaluation,
also compute the SED woore and the Error Rate (ER) (Adavanne et al. 2019 Mesaros et al. 202T
21.DDS22 datasct and code are available at: https:// http:://github. htt.hu..coom/..mm 20om/1366 1a/a1/////////// 1.302/
8

---

Under review as a conference paper at ICLR 2022
Table 7: SEDnett FLOPPs, training (T) and inference (I) time with 8 channels input. For training time (secondsfiteration) the mean and the standard deviation over one epoch is reported, for inference time we report the time required to perform an iteration on the validation set.
Model
FLOPs
Time (T)
Time (I)
SEDnet
37.3G
1.242 0.088
1.198
Quaternion SEDnet
37.3G
1.308 0.088
1.298
PHC SEDnet n = 2
37.3G
1.091 + 0.074
1.085
PHC SEDnet 4
37.3G
1.091 0.032
1.077
PHC SEDnet n = 8
37.3G
1.142 + 0.042
1.173
The proposed parameterized SEDnets distinctly outperform real and quaternion-valued baselines, as reported in Table and Table 6 Indeed, the PHC SEDnet with n = 2 gains the best results for each score and in both one and two microphone datasets, proving that the weights sharing due to the hypercomplex parameterization is able to capture more information regardless the lower number of parameters.
interesting note that the PHC SEDnet n 4, which operates in the quater- nion domain, achieves improved scores with respect to the Quaternion SEDnet that follows the rigid predefined algebra rules. Further, the malleability of PHC layers allows gaining comparable perfor- mance with respect to the quaternion baseline even so reducing convolutional parameters by 87%, just setting n 8. In Appendix
B we show additional experimental results of PHC models able to save 94% of convolutional parameters while operating in the sedonion domain by involving n 16.
Furthermore, PHC SEDnets are more efficient terms of time required for training and inference, regardless the equal number of FLOPs. Table 7 shows that each tested version of the proposed method is faster regards as the real SEDnet and the quaternion one, both at training and at infer- ence time. Time efficiency is crucial in audio applications where networks usually trained for thousands epochs and datasets are very large and require protracted computations.
5 RELATED WORKS
While a research field deals with scaling up networks (Real et al. 2019), a consistent area in liter- ature aims instead at making these models more efficient and accessible (Tan & Le 2019 Sandler jtt al. 2018). Among the latter, quaternion neural networks (QNNs) have proved to reduce the amount of parameters 1/4 (Parcollet et al. 2019a 2017). QNNs ensure high performance thanks to the sharing of weights due to the Hamil ton product that preserves correlations among channels in multidimensional inputs (Grassucci al. 2021c Tay et al. 2019 Grassucei et al. 202Tb). However, despite the lower number of parameters, QNNs are often slightly slow with respect to real-valued baselines (Hoffmann et al. 2020).. Recently, several attempts have been made to compress neu- ral networks relying on Kronecker product decomposition (Huang et al.) 2020; Tang et al.) 2021). These methods gain considerable results in terms of model efficiency (Wang et al. 20011. Lately, parameterization of hypercomplex multiplications have been proposed to generalize hypercomplex fully connected layers by sum of Kronecker products (Zhang et al., 2021). The latter method ob- tains high performance in various natural language processing tasks by also reducing the number of overall parameters. Lately, other works extended this approach to graph neural networks (Le al.)
2021) and transfer learning (Mahabadi et al. 2021), proving the effectiveness of Kronecker product decomposition for hypercomplex operations.
CONCLUSION
In this paper, we introduce parameterized hypercomplex convolutional (PHC) layer which grasps the convolution rule directly from data and can operate in any domain from 1D to nD, regardless the algebra regulations are preset. The proposed approach reduces the convolution parameters to 1/n with respect to real-valued counterparts and allows capturing internal latent relations thanks to parameter sharing among input dimensions. We show our method is flexible to operate in different fields of application by performing experiments with images and audio signals. We also prove the malleability and the robustness of our approach to learn convolution rules in any domain by setting different values for the hyperparameter from 2 to 16.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
One of the objective of proposing methods that allow a reduction of parameters is make large models more accessible and reproducible, even with lower budget. We strongly believe repro- ducibility should be key ingredient for recent deep learning papers. Therefore, we provide detailed explanations in the Appendix to better understand the proposed method. Moreover, each experi- ment reported the main corpus and in the appendix of this paper is completely reproducible. We provide the full code at htttt::: / monymous 4 4open sci ence/r / Hype www.eyeeeeeetttee rNet.s-Cae CBBB, in- cluding models, train configurations, L3DAS21 preprocessing and information for dataset download and thorough instructions and details are provided in the Appendix B Notebooks tutorials with toy examples are uploaded too, hoping these may help further research on this topic.
CO2 EMISSION RELATED TO EXPERIMENTS
Experiments were conducted using private infrastructure, which has a carbon efficiency of 0.445 kgCOoze/kWhh A cumulative of 2000 hours of computation was performed on hardware of type Tesla V100-SXM2-32GBB (TDP of 300W). Total emissions are estimated be 267 kgCOzeq of which 0 percents were directly offset. Estimations were conducted using the MachinelLcarning Impact caalculatorrprseeeee
in Lacoste et al. (2019).
More in detail, considering experiment for the sound event detection (SED) task, according to Table 7 the real-valued bascline requires approximately 20 hours for training and validation, with corresponding carbon emissions of 2.71 kgCO2eq. Conversely, the proposed PHC model takes approximately 17 hours with a reduction of carbon emissions 16%, being 2.28 kgCO,eq.
In conclusion, we believe that the improved efficiency of our method with respect standard models may be a little step towards reducing carbon emissions.
REFERENCES
S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing, 13:34-43 48, 2019.
D. Comminiello, M. Lella, S. Scardapane, and A. Uncini. Quaternion convolutional neural networks for detection and localization of 3D sound events. IEEE Ini. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp.
Brighton, UK, May 2019.
S. d' Ascoli, H. Touvron, M. Leavitt, A. Morcos, G. Biroli, and L. Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint: arXiv:2103. 10697, 2021.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, An image is worth 16x words: Transformers for image recognition at scale. In Int. Conf. Learning Representations (ICLR), 2021.
C. Gaudet and A. Maida. Deep quaternion networks. In IEEE Int. Joint Conf. on Neural Netw: (IJCN), Rio de Janeiro, Brazil, July 2018
E. Grassucci, E. Cicero, and D. Comminiello. Quaternion Generative Adversarial Networks. arXiv preprint: arXiv:2
2021a.
E. Grassucci, D. Comminiello, and A. Uncini. An information-theoreti perspective on proper quaternion variational autoencoders. Entropy, 23(7), 2021b.
E. Grassucci, D. Comminiello, and A. Uncini. A quatermion-valued variational autoencoder. In IEFE Inf. Conf. on Acoust., Speech and Signal Process. (ICASSP), Toronto, Canada, June 2021c.
E. Guizzo, R. F. Gramaccioni, S. Jamili, C. Marinoni, E. Massaro, C. Medaglia, G. Nachira, L. Nucciarelli, L. Paglialunga, M. Pennese, S. Pepe, E. Rocchi, A. Uncini, and D. Comminiello. L3DAS21 Challenge: Machine learning for 3D audio signal processing. 2021 IEEE Int. Workshop Machine Learning for Signal Process. (MLSP), 2021.
10

---

Under review as a conference paper at ICLR 2022
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
S. Hershey, S. Chaudhuri, P. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson. CNN architectures for large- scale audio classification. In IEEE Int. Conf. on Acoustics, Speech and Signal Process. (ICASSP), pp. 131 135, 2017.
J. Hoffmann, S. Schmitt, S. Osindero, K. Simonyan, and E. Elsen. Algebranets. ArXiv preprint: arXiv:2006.07360, 2020.
C.W. Huang, A. Touati, P. Vincent, G. K. Dziugaite, A. Lacoste, and A. C. Courville. Stochastic neural network with Kronecker flow. In AISTATS, 2020.
T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres. Quantifying the carbon emissions of machine learning. ArXiv preprint: arXiv:1910.09700 2019.
Le, M. Bertolini, Noć, and D. A. Clevert. Parameterized hypercomplex graph neural networks for graph classification. ArXiv preprint: arXiv:2 103. /6584, 2021.
R. K. Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter
layers. ArXiv preprint: arXiv:2106.0464/ 2021.
A. Mesaros, T. Heittola, T. Virtanen, and M. D. Plumbley, Sound event detection: A tutorial. JEEE Signal Processing Magazine, 38(5):67-83) 2021.
T. Parcollet, M. Morchid, and G. Linarès. Deep quatemnon neural networks for spoken language understanding. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 504-501, Okinawa, Japan, December 2017.
T. Parcollet, M. Morchid, and G. Linarès. A survey of quaternion neural networks. Artif. Intell. Rev., August 2019a.
T. Parcollet, M. Morchid, and G. Linarès. Quaternion convolutional neural networks for heteroge- neous image processing. In JEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), pp. 85148511588 8518, Brighton, UK, May 2019b.
T. Parcollet, M. Ravanelli, M. Morchid, G. Linarès, C. Trabelsi, R. De Mori, and Y. Bengio. Quater- nion recurrent neural networks. In Int. Conf. on Learning Representations (ICLR), pp. 1-19, New Orleans, LA, May 2019c.
E. Real, A. Aggarwal, Y. Huang, and Q. Le. Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conf. on Artificial Intelligence, 33:4780-4789 Jul. 2019.
M. Ricciardi Celsi, S. Scardapane, and D. Comminiello. Quaternion neural networks for 3D sound source localization in reverberant environments. In IEEE Int. Workshop on Machine Learning for Signal Process., pp. 1-6, Espoo, Finland, September 2020.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In JEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Int. Conf. on Learning Representations (ICLR), San Diego, CA, USA, 2015.
M. Tan and Q. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Int. Conf. on Machine Learning (ICML), volume 97, pp. 6105-61 14, 09-15 55 Jun 2019.
Z. Tang. F. Jiang, M. Gong, H. Li, Y. Wu, F. Yu. Z. Wang, and M. Wang. SKFAC: Training neural networks with faster sameeckrr-fcctored approximate curvature. /EEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 13479-13487 2021.

---

Under review as a conference paper at ICLR 2022
Y. Tay, A. Zhang, A. T. Luu, J. Rao, S. Zhang, S. Wang, J. Fu, and C. S. Hui. Lightweight and efficient neural natural language processing with quaternion networks. In ACL (1), pp. 1494 1503. Association for Computational Linguisties, 2019.
D. Wang, B. Wu, G. S. Zhao, H. Chen, L. Deng, T. Yan, and G. Li. Kronecker CP decomposition with fast multiplication for compressing RNNs. IEEE Trans. Neural Netw. Learn. Syst., PP, 2021.
R. Wightman, H. Touvron, and Jégou; H. Resnet strikes back: An improved training procedure in timm. ArXiv preprint: arXiv:2110.00476 2021.
H. Wu, B. Xiao, N. C. F. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang. Cvt: Introducing convolutions to vision transformers. ArXiv, abs/2103.15800 2021.
A. Zhang, Y. Tay, S. Zhang. A. Chan, A. T. Luu, S. C. Hui, and J. Fu. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. Int. Conf. Machine Learning (ICMI., 2021.
A THEORETICAL ADD-ONS
A.1 QUATERNION ALGEBRA
quaternion 9 E THT an hypercomplex number of rank It is defined as the composition of four real coefficients with three imaginary units as follows:
go + Qı + q2] + gah go +
being go. Q1 92. 9s and i, J. orthonormal basis in TR3 A quaternion with scalar part go equal to 0 is pure quaternion. Considering two quaternion and the Hamilton product in Eq can be rewritten in concise form as:
qp 4oPo q p 9oP Pod 9
where the bold notation is intended for vectors and the plain notation for scalars. From the above definition, it is easy derive the Hamilton product between two pure quaternions. takes the form:
qp q p p
in which -q p is the scalar part of the new quaternion and its vector part. Therefore, the vector product between two pure quaternions gives rise full quaternion.
As complex numbers, also quaternions have their own complex conjugates that are defined as
q = qu qu i 92ŋ - g3 = - q.
The norm of quaternion is the Euclidean norm in F. of its real coefficients, that is
lql = vqq" = 4 + o + 93 + +
while the inverse is
9
A.2 DEMYSTIFYING PHC LAYERS
We provide a formal explanation the PHC layer to better understand the Kronecker product and how it organizes convolution filters to reduce the overall number of parameters to 1/n. In Eq 10 we show how the PHC layer generalizes from 1D to nD domains. When subsuming vell-valued convolutions in the first line of Eq [0 the Kronecker product is performed between scalar A and the filter matrix F. whose dimension is the same as the final weight matrix H. that is x le x k. Considering the complex with 2 in the second line of Eq 10 the algebra is defined in
12

---

Under review as a conference paper at ICLR 2022
A1 and Ag while the filters contained in F1 and F2, each of dimension 1/4 the final matrix H. Therefore, while the size of the weight matrix H remains unchanged, the parameter size is approximately 1/2 the real one. In the last line of 10] process, in which the size of matrices F, i = 1,-... n is reduced proportionally to n. It is worth noting that, while the parameter size is reduced with growing values of n, the dimension of H remains the same.
see the generalization of this
[A] (1x1)
F
(xxdxkxk)
(xxdxkxk)
[A 60
F
+ A21 60 (2x2)
F2
(2x2)
x xxxxxxx))
(G x xkxk)
(exdxxxk)
0 F, 1 [A2] 2 F2
+
A (nxn)
[
F.,
H
(axn) x (uxn)
xxx#
(#x#x
(oxdxkx))
(10)
B EXPERIMENTAL ADD-ONS
B. TOY EXAMPLES: LEARNING THE MATRIX A
We test the receptive ability of the PHC layer in two toy problems building an artificial dataset. The first task is learning the proper matrix A to build quaternion convolutional layer which properly follows the Hamilton rule in Eq 2 We build the dataset by performing a convolution with matrix of filters W E IHI, that are arranged following the regulation in Eq 2 and quaternion x E HH in input. The target still a quaternion, named y E very fast, meaning that the layer properly learns the matrix A and the Hamilton convolution.
shown Fig. 2 (right), the PHC loss converges
The second toy example a modification of the previous dataset target.. Here, we want to learn the matrix A which describes the convolution among two pure quaternions. Pure quaternions may be. as an example, an input RGB image and the weights of a hypercomplex convolutional layer. Figure 2 (left) displays the convergence of the PHC layer loss during training, proving that the proposed method is able of subsuming hypercomplex convolutional rules when dealing with pure quaternions too..
13

---

Under review as a conference paper at ICLR 2022
510
01005
010
810
0/000
Epoda
Figure 2: Loss plots for toy examples. The PHC layer able to learn the matrix which describes the convolution rule for pure quaternions (left) and for full quaternions (right).
Table VGG16 results with real-valued classifier for quaternion and PHC networks. Extension of Table iin the main corpus. Model
Params
SVHN
CIFAR10
Quaternion VGG16
4.2M (-72%)
94.086
84.126
PHC VGG16 n 2
7.9M (-62%)
94.885
86.147
PHC VGG16
4.2M (-72%)
94.562
85.710
B.2 EXPERIMENTAL ADD-ONS ON IMAGE CLASSIFICATION
B.2.1 IMAGE CLASSIFCCATION IMPLEMENTATION DETAILS
To guarantee the complete reproducibility of our experiments, here we provide more implementation details for the image classification task.
The modified versions of the ResNets built with initial convolutional layer with 60 filters. Then, the subsequent blocks have 60, 120, 240, 516 filters. The number of layers in the blocks depends the ResNet chosen, whether 18, 50 or 152.
Instead, VGG19 convolution component comprise two 24, two 72, four 216, and eight 648 filter layers, with batch normalization. The classifier is composed three fully connected layers of 648, 516 and with the number of classes neurons each.
The of the hyperparameters are as suggested in the original papers. The batch size is fixed 128 and training is performed via SGD optimizer with momentum equal to 0.9, weight decay 5e -4 and a cosine annealing scheduler. For ResNets, the initial learning rate is set to 0.1. For VGG is equal to 0.01. Models on CIFAR 10 and CIFAR 100 are trained for 200 epochs whereas on SVHN networks run for 50 epochs.
For the ImageNet dataset, we follow the recipes in (Wightman et al. 2021 we resize the images for training at 160 x 160 while keeping the standard size of 224 x 224 for validation and test. We employ step learning rate decay every 30 epochs with 0.1, the SGD optimizer and initial learning rate of 0.1 with weight decay 0.0001. The training is performed for 300k iterations with batch size of 256 employing GPUs.
B.2.2 ADDITIONAL EXPERIMENTS ON IMAGE CLASSIFICATION
In the following, we provide additional preliminary experiments with small models equipped with the proposed PHC layer. Table & contains VGGI6s with real-valued final classifier. Table T reports experiments with ResNet20 where we test also n = 1 to replicate the real-valued model. outper- forming it. Experiments with VGG11 I with modified number of filters in order to be divided by each value n is also reported in the same table. Finally, in Table I0] additional experiment on SVHN and CIFAR10 10 with ResNet56 and ResNetl 10, the latter with modified number of filters. PHC-based
14

---

Under review as a conference paper at ICLR 2022
Table 9: Additional preliminary experiments SVHN dataset with ResNer20 and VGGI1, the latter with modified number of filters in order to be divided by cach value of n and FC layers in the closing classifier. We test also the PHC model with n to replicate the real domain which outperform the real-valued ResNet20.
Model
Params
SVHN
ResNet20
0.27M
90.463
Quaternion ResNet20
0.07M (-75%)
93.535
PHC ResNet22
0.27M
93.7962
PHC ResNet20 n
0.14M (-50%)
93.708
PHC ResNet20
0.07M (-75%)
93.669
VGGII
13.8M
93,488
Quaternion VGGI1
3,9M (-71%)
92.888
PHC VGGIl =
7.2M (-48%)
93,958
PHC VGGII n 3
5.0M (-64%)
93,804
PHC VGGI1 4
3.9M (-71%)
93.919
Table 10: Additional preliminary experiments with ResNet56 and ResNetl 10, the latter with mod- ified number of filters in order be divided by each value of n. Accuracy score is the mean three runs with different seeds.
Model
Params
SVHN
CIFAR10
ResNet56
0.9M
94.116
83.700
Quaternion ResNet56
0.2M (-75%)
93.664
81.687
PHC ResNet56 n =
0.4M (-50%
93.722
83.413
PHC ResNer56 4
0.2 (-75%)
94.122
82.720
ResNet110
16.7M
93.461
84.810
Quaternion ResNet110 10
4.2M (-75%)
92.788
83.920
PHC ResNetl 10 n
8.4M (-50%)
93.746
83.220
PHC ResNetl
5.6M (-66%)
94.712
85,200
PHC ResNetl n 4
4.2M (-75%)
94.885
85.280
Table 11: Additional experiments with ResNet-based models. We reduced the number of convo- lutional filters by 75% and then test the models on three datasets to remove the hypothesis that smaller number of parameters leads higher generalization capabilities.
Model
Params
SVHN
CIFAR10
CIFAR100
ResNet18
10.1M
93.992
89.543
62.634
ResNet18 (reduced)
2.7M (-75%)
93.842
88.310
59.590
ResNet50
22.5M
94.546
89.630
65.514
ResNet50 (reduced)
5.7M (-75%)
93.915
89.370
62,450
ResNet152
52.6M
94.625
89.580
62.053
ResNet152 (reduced)
13.2M (-75%)
94.400
89.001
60.850
models gain good performance in each test we conduct while reducing the amount of free param- eters. Finally, in order to further remove the hypothesis that smaller number of neural parameters leads to higher generalization capabilities, we perform experiments with real-valued baselines with number of parameters reduced by 75%. Table [I shows that reducing the number of filters down- grades the performance and thus it is not sufficient to improve the generalization capabilities a model.
15

---

Under review as a conference paper at ICLR 2022
B.3 EXPERIMENTAL ADD-ON. OOD-ONS ON SOUND EVENT DETECTION
B.3.1 L3DAS21 DATASET DETAILS
The L3DAS21 Task 2 dataset is collection of 900 1-minute-long data-points sampled rate of 32 kHz. The sounds classes have been selected from the FSD50K dataset and are representative for office sounds: computer keyboard, drawer open/close, cupboard open/close, finger snapping, keys jangling, knock, laughter, scissors, telephone, writing, chink and clink, printer, female speech, male speech. In this dataset, the volume difference between the sounds is in the range and 20 dB full scale (dBFS). Considering the array of two microphones 1, 2, the channels order is [W1, Z1, YI, XI, W2, Z2, Y2, X2], where WXYZ are the B-format ambisonics channels if the phase (p) information is not considered. Whether we want to include also this information, the order will be [W1, Z1, YI, XI, WIp, ZIp, Ylp. XIp. W2, Z2, Y2, X2, W2p, Z2p. Y2p. X2p] up 16 channels. In Fig 3] we show the 8-channel input when considering one microphone and the phase information. Magnitudes and phases are normalized be centered with standard deviation 1.
WA
Y2
X1
WIP
Z1p
YIp
X1p
Figure 3: Sample spectrograms from L3DAS21 dataset recorded by one microphone with four cap- sules.The first four figures represent the magnitudes while the last four contain the corresponding phases information. The black sections represent silent instants.
B.3.2 SED IMPLEMENTATION DETAILS
To guarantee the complete reproducibility of our experiments, here we provide more implementation details for the sound event detection task.
The extracted features by the preprocessing are fed to the four-layer convolutional stack with 64, 128, 256, 512 filters, with batch normalization, ReLU activation, max pooling and dropout (prob- ability 0.3), with pooling sizes (8, 2). (8, 2), (2, 2). (1, 1). The bidirectional GRU module has three layers, each with an hidden size of 256. The tail has is four-layer fully connected classifier with 1024 filters alternated by ReLUs and with final dropout and sigmoid activation function. The initial learning rate is set to 0.00001.
Following, we define the equations of the evaluation metrics considered in Subsection 4.2 To be consistent with pre-existing literature, we define True Positives as TP, False Positives as FP and False Negatives as FN. These computed according to the detection metric (Guizzo et al. 2021). Moreover, in order to compute the Error Rate (ER), we consider:
min(FN, FP),
D ma FN FP) and I max(0, FP FN), as in (Advvnnee al. 2019).
16

---

Under review as a conference paper at ICLR 2022
2TP 2TP + FP + FN
Soooe
S S+D+1
ER
whereby N is the total number of active sound event classes in the reference.
ER + 1 Fscore
SED Score
score
2
For ER and SoDe the lower scores, the better the performances, while for the Facore higher values stand for better accuracy.
B.3.3 ADDITIONAL EXPERIMENTS ON SOUND EVENT DETECTION
In this section, we provide additional experiments for the sound event detection task. We conduct test considering two microphones and the phase information, For this purposes, we consider as baseline the quaternion model and PHC-based networks with 8,8, 16 test higher order domains. Results shown in Table 12
have input with 16 channels.
Table 12: SED results with microphone: magnitudes and phases (16 channels input). We test
higher order hypercomplex domains up to sedonions by setting = 16. Although the incredible reduction of the number of parameters with respect to the real-valued baseline in Table 6 the PHC with n = 16 still has comparable performances with other models. Furthermore, the PHC with
outperform also the quaternion baseline which has more degrees of freedom.
Model
Conv Params
F Secore
ER
SED
P
R
Quaternion SEDnet
0.4M (-75%)
0.580
0.480
0.450
0.655
0.520
PHC SEDnet
0.4M (-75%)
0.585
0.470
0.443
0.653
0.530
PHC SEDnet 7 =
0.2M (-87%)
0.607
0.466
0.430
0.702
0.534
PHC SEDnet
16
0.1M (-94%)
0.588
0.509
0.461
0.734
0.491
C FUTURE PERSPECTIVES
In this work, demonstrate that the proposed parameterized hypercomplex convolutional (PHC) layer is able to consistently reduce the free parameters of neural models while ensuring very high performances thanks the hypercomplex convolution properties. Therefore, findings pave the way for novel efficient neural networks. As an example, a thorough study on pruning PHC-based models may further reduce the number of parameters and lead to extremely lightweight networks.
believe that these
17


Reviews:
[{"review_id": "o9kLQvXJcWy", "reviewer": "Reviewer_JhvG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* Results are shown on multiple domains and strong performance is achieved.\n* The paper is clearly written.\n* The authors provide code making the results easily reproducible by future groups. \n\nWeaknesses:\n* I think the experimental results could be fleshed out. The networks and datasets are fairly small. It would be interesting to see the results pushed to a greater scale. It would be nice to see, for example, some results with a ResNet50 on ImageNet. \n* It isn't immediately clear to me that parameter count is what needs to be cared about the most. Especially with quaternion, there is a substantial FLOP increase. I think comparisons on FLOPs would be most interesting. If the argument is model size, quaternions will still induce an increase activation memory. \n\n\nComments/Questions:\n* From my understanding of Deep Complex Networks from Trabelsi, and Deep Quaternion Networks from Gaudet, they argue that a whitening procedure should be used instead of BatchNorm. In AlgebraNets (https://arxiv.org/abs/2006.07360) the authors use BatchNorm, arguing the FLOP expense of whitening is not worth it. Is this something that you experimented with?  Similarly, did you consider other algebras, such as matrix rings?\n* Deep Complex Networks, Deep Quaternion Networks, and AlgebraNets all investigate various activation functions beyond ReLU. Is this something that you considered? \n* Other forms of model compression methods such as pruning are quite popular. Have you tried pruning any of these networks for further parameter reductions? \n", "summary_of_the_paper": "Traditionally, networks use real-valued weights. However recent work has shown that in certain domains, weights that obey different algebraic rules can lead to various performance enhancements. The authors introduce a hyper complex convolution block and show that it leads to performance gains and parameter reductions over a variety of baseline models on tasks in various domains. ", "summary_of_the_review": "The authors propose an interesting idea, however I think that the presented comparisons miss a few key comparisons. Specifically, I would have liked to see a measure of FLOPs required to reach given accuracies. Additionally, while the results shown span multiple domains I would have liked to see more ambitious experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RNgcsnNbg-4", "reviewer": "Reviewer_MVqX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. This method significantly improves the time and computational power required for the training of large neural networks while achieving good performance. \n2. The method can be operated in domains from $1D$ to $nD$, where $n$ can be tuned to attain appropriate results on given data. \n3. Neural models redefined with this approach require $1/n$ of free parameters compared to their real-valued counterparts. \n\nWeakness: \n\n1. Although the research has many real-world applications, it is lacking novelty. The paper is an extension of (Zhang et al., 2021) for the convolution case. \n2. Many details required to reimplement these models to achieve similar results are missing in the paper. \n3. The choice of $n$ should also be analyzed using FLOPs. ", "summary_of_the_paper": "The paper proposes Parametrized Hypercomplex convolutional Layers to replace convolutional layers to attain similar performance using fewer parameters. The PHC layer can subsume hypercomplex convolution rules. The main advantage of the PHC Layer is that they enable choosing an arbitrary $n$ to reduce the number of parameters to $1/n$, whereas this was limited to $4$, $8$, and $16$ with quaternions. Hyperparameter $n$ also drives the number of convolutional filters. The proposed network can represent a real-valued convolutional layer by setting $n = 1$.\n\nThe PHC layer differs from the convolutional layer in the way that the weight matrix is obtained. $H$ acts as a weight for input $x$. The $A$ and $F$ matrices which are trainable and used to obtain $H$. Kronecker products of every matrix in the tensors $A$ and $F$ are taken and then these are summed to get $H$.  \n\nAuthors have proposed PHC versions of some common neural networks such as ResNet and VGG and have compared the results with real-valued models, Quaternion models, and PHC models (for $n=2$ and $n=4$). ", "summary_of_the_review": "The paper lacks novelty and is an extension of (Zhang et al., 2021) for the convolution case. This is a major concern for me. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FskoG1Ar2R", "reviewer": "Reviewer_6bRZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper generalize the quaternion neural networks to CNNs and it shows better accuracy v.s. #parameters trade-off than Quaternion counterparts.\n\nWeakness:\nIt's not clear how much real speedup/model size reduction can be achieved. The algorithms firstly use Kronecker product to generate the convolution kernel then use it for the normal conv2d. In my understanding, only the number of free parameters is reduced, instead of the practical resource consumption for inference / training.\nThe comparison is not enough to show the effectiveness of the proposed method. How is the proposed method compared to low-rank based conv kernel decomposition? How is it compared to the depth-wise & point-wise decomposition like in Mobilenets? Without the comparison to other commonly used DNN compression approaches, it's hard to understand the significance of the proposed method.", "summary_of_the_paper": "This paper proposed a lightweight convolutional neural network based on the idea of quaternion neural networks. The paper extended the hypercomplex linear layers to convolution neural networks to apply to many multi-dimensional applications. The authors defined the parameterization of hypercomplex convolutional layers so that convolution neural networks can utilize the quaternion algebra to improve the parameter efficiency.", "summary_of_the_review": "See Main Review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BcQI2P3ZAoX", "reviewer": "Reviewer_KMob", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Summarising this paper is quite straightforward as the contribution is quite simple, yet effective. The concept of hypercomplex parameterization is extended to convolutional layers and common CNN models are also turned into hypercomplex parameterized models. The new model performs better than all baselines with less neural parameters and slightly (always ?) quicker training and inference times compared to hypercomplex models. In practice, the contribution is minimal, but very valuable as the code is given. If the code is open to the community afterwards, it is a win-win situation. Simple, efficient, and works well. \n\nSeing this concept of hypercomplex parameterization extended is very nice considering that is has changed the efficiency capabilities of hypercomplex networks. Indeed, despite promising results and memory footprint, hypercomplex networks suffered from a lack of generalisation AND a crazy training / inference time compared to real-valued neural networks. This is all almost solved with the hypercomplex parameterization introduced in a previous paper and extended to CNN in this work. \n\nMy only concern, that prevents me from assigning a high rank on the current submission relates to the quality of the writing. I will only provide two examples, but the paper should really be proofread carefully and the English must be improved to meet ICLR requirements. If this is done properly, I don't see why this paper shouldn't be accepted. \n\"But the layers saves the 75% of free parameters\" / \"that consider deep layers [...], thus [...]\"\n\nAn other smaller comment concerns Section 3.1. It is really weird to have results presented in formal Section. I would encourage the authors to move this to the experimental part instead, or rewrite the two last paragraphs as they currently sound as \"look, we tried, it works, but we don't show you the results\" even if you give then in Appendix. It is not appropriate to present it in this way and at this stage of the paper. \n\n**Strengths:**\n- Simple idea, yet very effective.\n- Definitely impactful for hypercomplex neural network (and thus the concerned community).\n- If the code is given, it will be very helpful to the community.\n- Good results as expected (since they were already observed with non-convolutional layers). \n\n**Weaknesses:**\n- Writing must be polished.\n- Add time and inference for ALL tables as this is a major point in the discussion between hypercomplex networks and parameterised ones !\n- Add real-valued models with the exact same number of neural parameters than PHC ones to the tables. This will remove the hypothesis that smaller number of neural parameters leads to higher generalisation capabilities.  \n", "summary_of_the_paper": "This work extends the recently proposed hypercomplex parameterization of neural layers to convolutional layers. Then it describe common deep CNN models (ResNet, VGG, SeDnet) based on this novel paradigm in two different tasks (Image Classification and Sound Event Detection). The introduced method offer better performance with a reduced number of neural parameters as well as faster training and inference than equivalent hypercomplex layers. ", "summary_of_the_review": "Simple and impactful idea. Results are convincing. It will be an interesting paper to the hypercomplex networks community. Nevertheless, writing must be improved to get accepted (not the outline of the paper but the content). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In general, the reviewers appreciated the elegant concept behind the paper and the good results. However, they also raised considerable reservations about the significance of a method that decreases the parameter count but not necessarily computational efficiency (FLOPS) or memory. While the additional analysis that the authors provided definitely helps to understand the limitations of the method, the reviewers were in the end quite divided on the significance of the results. In addition, all reviewers agreed that the writing was in somewhat rough shape and needed improvement.

In summary, this is definitely a borderline paper, but given the current reviewer assessment, I would recommend that it is not quite ready for publication.
Decision: Reject