Lightweight Convolutional Neural Networks By Hypercomplex Parameterization

1 INTRODUCTION . Recent state-of-the-art convolutional models achieved astonishing results in various fields of application by large-scaling the overall parameters amount ( Karras et al. , 2020 ; d ’ Ascoli et al. , 2021 ; Dosovitskiy et al. , 2021 ) . Simultaneously , quaternion neural networks ( QNNs ) demonstrated to significantly reduce the number of parameters while still gaining comparable performances ( Parcollet et al. , 2019c ; Grassucci et al. , 2021a ; Tay et al. , 2019 ) . Quaternion models exploit hypercomplex algebra properties , including the Hamilton product , to painstakingly design interactions among the imaginary units , thus involving 1/4 of free parameters with respect to real-valued models . Furthermore , thanks to the modelled interactions , quaternion networks capture internal latent relations in multidimensional inputs and preserve pre-existing correlations among dimensions . Therefore , the quaternion domain is particularly appropriate for processing 3D or 4D data , such as color images or ( up to ) 4-channel signals . Unfortunately , most common color image datasets contain RGB images and a zero channel has to be padded to the input in order to encapsulate the image in the four quaternion components . Additionally , while quaternion neural components are widespread and easy to be integrated in pre-existing models , very few attempts have been made to extend models to higher domain orders . Accordingly , the development of hypercomplex convolutional models for larger multidimensional inputs , such as magnitudes and phases of multichannel audio signals or 16- band satellite images , still remains painful . Recently , a parameterized hypercomplex multiplication ( PHM ) for fully connected layers have been proposed to generalize hypercomplex multiplications as sum of Kronecker products , going beyond quaternion algebra . However , no solution exists for convolutional layers , which remain the most employed layers when dealing with multidimensional inputs , such as images and audio signals ( Wu et al. , 2021 ; Hershey et al. , 2017 ) . In this paper , we propose a parameterized hypercomplex convolutional ( PHC ) layer to define lightweight large-scale neural models admitting any multidimensional input , whichever the number of dimensions . Our method is flexible to operate in domains from 1D to nD , where n can be arbitrarily chosen by the user or tuned to let the model performance lead to the most appropriate domain for the given input data . Such a malleability comes from the ability of the proposed approach to subsume algebra rules to perform convolution regardless of whether these regulations are preset or not . Thus , neural models endowed with our approach adopt 1/n of free parameters with respect to their real-valued counterparts , and the amount of parameter reduction is a user choice . This makes PHC layers adaptable to a plethora of applications and hardware , even on the edge where memory saving is a crucial aspect . Additionally , the PHC versatility allows processing multidimensional data in its natural domain by simply setting the dimensional hyperparameter n. For instance , color images can be analyzed in their RGB domain by setting n = 3 without adding any useless information , contrary to standard processing for quaternion networks with the padded zero-channel . Indeed , PHC-based models are able to grasp the proper algebra from input data , while capturing internal correlations among the image channels and saving 66 % of free parameters . On a thorough empirical evaluation on multiple benchmarks , we demonstrate the flexibility of our method that can be adopted in different domain of applications , from images to audio signals . We devise a set of convolutional models endowed with PHC layers for large-scale image classification and sound event detection tasks , letting them operate in different hypercomplex domain and with various input dimensionality with n ranging from 2 to 16 . The contribution of this paper is three-fold . • We introduce a parameterized hypercomplex convolutional ( PHC ) layer , which aims at building lightweight and more efficient large-scale convolutional models . The approach grasps the convolution rules directly from data via backpropagation exploiting the Kronecker product properties , thus reducing the number of free parameters to 1/n . • We show how the proposed approach can be employed with any kind of multidimensional data by easily changing the hyperparameter n. Indeed , by setting n = 3 the PHC-based models can process RGB images in their natural domain , while leveraging the properties of hypercomplex algebras , allowing parameters sharing inside the layers and leading to a parameter reduction to 1/3 . To the best of our knowledge , this is the first approach that processes color images with hypercomplex-based neural models without adding any padding channel . As well , multichannel audio signals can be analysed by simply considering n = 4 for standard first-order ambisonics ( which has 4 microphone capsules ) , n = 8 for an array of two ambisonics microphones , or even n = 16 if we want to include the information of each channel phase . • We devise a family of PHC neural networks , redefining common ResNets , VGGs and Sound Event Detection networks ( SEDnets ) , operating in any user-defined domain just by choosing the hyperparameter n , which also drives the number of convolutional filters . 1 The rest of the paper is organized as follows . In Section 2 , we recapitulate real and quaternion-valued convolutional layers . Section 3 theoretically introduces the proposed method and expounds how to process RGB images with n = 3 , while in Section 4 we define the PHC models and we present the experimental evaluation , including datasets , experiment details and results . Finally , Section 5 reports the related works and in Section 6 we draw conclusions . 2 REAL AND QUATERNION-VALUED CONVOLUTIONAL LAYERS . A brief recall on real-valued and quaternion convolutional layers is appropriate to better understand the proposed method . A generic convolutional layer can be described by y = Conv ( x ) = W ∗ x+ b , ( 1 ) where the input x ∈ Rt×s is convolved ( ∗ ) with the filter tensor W ∈ Rs×d×k×k to produce the output y ∈ Rd×t , where k is the filter size . The bias term b does not heavily influence the number of parameters , thus the degrees of freedom for this operation are essentially O ( sdk2 ) . Quaternion convolutional layers , instead , build the weight tensor W ∈ Rs×d×k×k by following the Hamilton product rule and organize filters according to it : 1Full code is available at : https : //anonymous.4open.science/r/HyperNets-CBBB . W ∗ x = W0 −W1 −W2 −W3W1 +W0 −W3 +W2W2 +W3 +W0 −W1 W3 −W2 +W1 +W0 ∗ x0x1x2 x3 , ( 2 ) where W0 , W1 , W2 , W3 ∈ R s 4× d 4×k×k are the real coefficients of the quaternion weight matrix W = W0 +W1 ı̂+W2̂+W3κ̂ and x0 , x1 , x2 , x3 are the coefficients of the quaternion input x with the same structure . The imaginary units comply with the property ı̂2 = ̂2 = κ̂2 = −1 and with the non-commutative products ı̂̂ = −̂̂ı ; ̂κ̂ = −κ̂̂ ; κ̂ı̂ = −ı̂κ̂ . As done for real-valued layers , the bias can be ignored and the degree of freedom computations of the quaternion convolutional layer can be approximated to O ( sdk2/4 ) . The lower number of parameters with respect to the real-valued operation is due to the reuse of filters performed by the Hamilton product in Eq.2 . Also , sharing the parameter submatrices forces to consider and exploit the correlation between the input components ( Parcollet et al. , 2019a ; Tay et al. , 2019 ; Gaudet & Maida , 2018 ) . 3 PARAMETERIZING HYPERCOMPLEX CONVOLUTIONS . 3.1 PARAMETERIZED HYPERCOMPLEX CONVOLUTIONAL LAYERS . In the following , we delineate the formulation for the proposed parameterized hypercomplex convolutional ( PHC ) layer . We also show that this approach is capable of learning the Hamilton product rule when two quaternions are convolved . The PHC layer is based on the construction , by sum of Kronecker products , of the weight tensor H which encapsulates and organizes the filters of the convolution . The proposed method is formally defined as : y = PHC ( x ) = H ∗ x+ b , ( 3 ) whereby H ∈ Rs×d×k×k is built by sum of Kronecker products between two learnable matrices . Here , s is the input dimensionality to the layer , d is the output one , and k is the filter size . More concretely , H = n∑ i=1 Ai ⊗ Fi , ( 4 ) in which Ai ∈ Rn×n with i = 1 , ... , n are the matrices that describe the algebra rules and Fi ∈ R sn× dn×k×k represents the i-th batch of filters that are arranged by following the algebra rules to compose the final weight matrix . The core element of this approach is the Kronecker product , which is a generalization of the vector outer product that can be parameterized by n. The hyperparameter n can be set by the user who wants to operate in a pre-defined real or hypercomplex domain ( e.g. , by setting n = 2 the PHC layer is defined in the complex domain , or in the quaternion one if n is set equal to 4 , as Figure 1 illustrates ) , or tuned to obtain the best performance from the model . The matrices Ai and Fi are learnt during training and their values are reused to build the definitive tensor H. The degree of freedom of Ai and Fi are n3 and sdk2/n , respectively . Usually , real world applications employ a large number of filters in layers ( s , d = 256 , 512 , ... ) and small values for k. Therefore , frequently sdk2 n3 holds . Thus , the degrees of freedom for the PHC weight matrix can be approximated to O ( sdk2/n ) . Hence , the PHC layer reduces the number of parameters by 1/n with respect to a standard convolutional layer in real world problems . Moreover , when processing multidimensional data with correlated channels , such as color images , rather than mulichannel audio or multisensor signals , PHC layers bring benefits due to the weight sharing among different channels . This allows capturing latent intra-channels relations that standard convolutional networks ignore because of the rigid structure of the weights ( Grassucci et al. , 2021a ; Parcollet et al. , 2019b ) . The PHC layer is able to subsume hypercomplex convolution rules and the desired domain is specified by the hyperparameter n. We test this ability with toy problems in Appendix B. Interestingly , by setting n = 1 a real-valued convolutional layer can be represented too . Indeed , standard real layers does not involve parameters sharing , therefore the algebra rules are solely described by the single A ∈ R1×1 and the complete set of filters are included in Fs×d×k×k . We report detailed explanations in Appendix A . 3.2 PARAMETERIZED LAYERS FOR COLOR IMAGES . In this section , we describe how the PHC layer can be employed to process color images in hypercomplex domains without needing any additional information to the input . Different encodes exist to process color images , however , the most common computer vision datasets are comprised of three-channel images in R3 . In the quaternion domain , RGB images are enclosed into a quaternion and processed as single elements ( Parcollet et al. , 2019a ) . The encapsulation is performed by considering the RGB channels as the real coefficients of the imaginary units and by padding a zeros channel as the first real component of the quaternion . Here , we propose to leverage the high malleability of PHC layers to deal with RGB images in hypercomplex domains without embedding useless information to the input . Indeed , the PHC can directly operate in R3 by easily setting n = 3 and process RGB images in their natural domain while exploiting hypercomplex algebra properties such as parameters sharing . Indeed , the great flexibility of PHC layers allows the user to choose whether processing images in R4 or R3 . On one hand , by setting n = 4 , the zeros channel is added to the input even so the layer saves the 75 % of free parameters . On the other hand , by choosing n = 3 the network does not handle any useless information , notwithstanding , it reduces the number of parameters by solely 66 % . This is a trade-off which may depend on the application or on the hardware the user needs . Furthermore , the domain on which processing images can be tuned by letting the performance of the network indicates the best choice for n .

Summary: The paper proposes Parametrized Hypercomplex convolutional Layers to replace convolutional layers to attain similar performance using fewer parameters. The PHC layer can subsume hypercomplex convolution rules. The main advantage of the PHC Layer is that they enable choosing an arbitrary $n$ to reduce the number of parameters to $1/n$, whereas this was limited to $4$, $8$, and $16$ with quaternions. Hyperparameter $n$ also drives the number of convolutional filters. The proposed network can represent a real-valued convolutional layer by setting $n = 1$.  The PHC layer differs from the convolutional layer in the way that the weight matrix is obtained. $H$ acts as a weight for input $x$. The $A$ and $F$ matrices which are trainable and used to obtain $H$. Kronecker products of every matrix in the tensors $A$ and $F$ are taken and then these are summed to get $H$.    Authors have proposed PHC versions of some common neural networks such as ResNet and VGG and have compared the results with real-valued models, Quaternion models, and PHC models (for $n=2$ and $n=4$).