Quaternion Recurrent Neural Networks

1 INTRODUCTION . In the last few years , deep neural networks ( DNN ) have encountered a wide success in different domains due to their capability to learn highly complex input to output mapping . Among the different DNN-based models , the recurrent neural network ( RNN ) is well adapted to process sequential data . Indeed , RNNs build a vector of activations at each timestep to code latent relations between input vectors . Deep RNNs have been recently used to obtain hidden representations of speech unit sequences ( Ravanelli et al. , 2018a ) or text word sequences ( Conneau et al. , 2018 ) , and to achieve state-of-the-art performances in many speech recognition tasks ( Graves et al. , 2013a ; b ; Amodei et al. , 2016 ; Povey et al. , 2016 ; Chiu et al. , 2018 ) . However , many recent tasks based on multi-dimensional input features , such as pixels of an image , acoustic features , or orientations of 3D models , require to represent both external dependencies between different entities , and internal relations between the features that compose each entity . Moreover , RNN-based algorithms commonly require a huge number of parameters to represent sequential data in the hidden space . Quaternions are hypercomplex numbers that contain a real and three separate imaginary components , perfectly fitting to 3 and 4 dimensional feature vectors , such as for image processing and robot kinematics ( Sangwine , 1996 ; Pei & Cheng , 1999 ; Aspragathos & Dimitros , 1998 ) . The ∗CIFAR Senior Fellow idea of bundling groups of numbers into separate entities is also exploited by the recent manifold and capsule networks ( Chakraborty et al. , 2018 ; Sabour et al. , 2017 ) . Contrary to traditional homogeneous representations , capsule and quaternion networks bundle sets of features together . Thereby , quaternion numbers allow neural network based models to code latent inter-dependencies between groups of input features during the learning process with fewer parameters than RNNs , by taking advantage of the Hamilton product as the equivalent of the ordinary product , but between quaternions . Early applications of quaternion-valued backpropagation algorithms ( Arena et al. , 1994 ; 1997 ) have efficiently solved quaternion functions approximation tasks . More recently , neural networks of complex and hypercomplex numbers have received an increasing attention ( Hirose & Yoshida , 2012 ; Tygert et al. , 2016 ; Danihelka et al. , 2016 ; Wisdom et al. , 2016 ) , and some efforts have shown promising results in different applications . In particular , a deep quaternion network ( Parcollet et al. , 2016 ; 2017a ; b ) , a deep quaternion convolutional network ( Gaudet & Maida , 2018 ; Parcollet et al. , 2018 ) , or a deep complex convolutional network ( Trabelsi et al. , 2017 ) have been employed for challenging tasks such as images and language processing . However , these applications do not include recurrent neural networks with operations defined by the quaternion algebra . This paper proposes to integrate local spectral features in a novel model called quaternion recurrent neural network1 ( QRNN ) , and its gated extension called quaternion long-short term memory neural network ( QLSTM ) . The model is proposed along with a well-adapted parameters initialization and turned out to learn both inter- and intra-dependencies between multidimensional input features and the basic elements of a sequence with drastically fewer parameters ( Section 3 ) , making the approach more suitable for low-resource applications . The effectiveness of the proposed QRNN and QLSTM is evaluated on the realistic TIMIT phoneme recognition task ( Section 4.2 ) that shows that both QRNN and QLSTM obtain better performances than RNNs and LSTMs with a best observed phoneme error rate ( PER ) of 18.5 % and 15.1 % for QRNN and QLSTM , compared to 19.0 % and 15.3 % for RNN and LSTM . Moreover , these results are obtained alongside with a reduction of 3.3 times of the number of free parameters . Similar results are observed with the larger Wall Street Journal ( WSJ ) dataset , whose detailed performances are reported in the Appendix 6.1.1 . 2 MOTIVATIONS . A major challenge of current machine learning models is to well-represent in the latent space the astonishing amount of data available for recent tasks . For this purpose , a good model has to efficiently encode local relations within the input features , such as between the Red , Green , and Blue ( R , G , B ) channels of a single image pixel , as well as structural relations , such as those describing edges or shapes composed by groups of pixels . Moreover , in order to learn an adequate representation with the available set of training data and to avoid overfitting , it is convenient to conceive a neural architecture with the smallest number of parameters to be estimated . In the following , we detail the motivations to employ a quaternion-valued RNN instead of a real-valued one to code inter and intra features dependencies with fewer parameters . As a first step , a better representation of multidimensional data has to be explored to naturally capture internal relations within the input features . For example , an efficient way to represent the information composing an image is to consider each pixel as being a whole entity of three strongly related elements , instead of a group of uni-dimensional elements that could be related to each other , as in traditional real-valued neural networks . Indeed , with a real-valued RNN , the latent relations between the RGB components of a given pixel are hardly coded in the latent space since the weight has to find out these relations among all the pixels composing the image . This problem is effectively solved by replacing real numbers with quaternion numbers . Indeed , quaternions are fourth dimensional and allow one to build and process entities made of up to four related features . The quaternion algebra and more precisely the Hamilton product allows quaternion neural network to capture these internal latent relations within the features encoded in a quaternion . It has been shown that QNNs are able to restore the spatial relations within 3D coordinates ( Matsui et al. , 2004 ) , and within color pixels ( Isokawa et al. , 2003 ) , while real-valued NN failed . This is easily explained by the fact that the quaternion-weight components are shared through multiple quaternion-input parts during the Hamilton product , creating relations within the elements . Indeed , Figure 1 shows that the multiple weights required to code latent relations within a feature are considered at the same level as for 1https : //github.com/Orkis-Research/Pytorch-Quaternion-Neural-Networks learning global relations between different features , while the quaternion weight w codes these internal relations within a unique quaternion Qout during the Hamilton product ( right ) . Then , while bigger neural networks allow better performances , quaternion neural networks make it possible to deal with the same signal dimension but with four times less neural parameters . Indeed , a 4-number quaternion weight linking two 4-number quaternion units only has 4 degrees of freedom , whereas a standard neural net parametrization has 4 × 4 = 16 , i.e. , a 4-fold saving in memory . Therefore , the natural multidimensional representation of quaternions alongside with their ability to drastically reduce the number of parameters indicate that hyper-complex numbers are a better fit than real numbers to create more efficient models in multidimensional spaces . Based on the success of previous deep quaternion convolutional neural networks and smaller quaternion feed-forward architectures ( Kusamichi et al. , 2004 ; Isokawa et al. , 2009 ; Parcollet et al. , 2017a ) , this work proposes to adapt the representation of hyper-complex numbers to the capability of recurrent neural networks in a natural and efficient framework to multidimensional sequential tasks such as speech recognition . Modern automatic speech recognition systems usually employ input sequences composed of multidimensional acoustic features , such as log Mel features , that are often enriched with their first , second and third time derivatives ( Davis & Mermelstein , 1990 ; Furui , 1986 ) , to integrate contextual information . In standard RNNs , static features are simply concatenated with their derivatives to form a large input vector , without effectively considering that signal derivatives represent different views of the same input . Nonetheless , it is crucial to consider that time derivatives of the spectral energy in a given frequency band at a specific time frame represent a special state of a time-frame , and are linearly correlated ( Tokuda et al. , 2003 ) . Based on the above motivations and the results observed on previous works about quaternion neural networks , we hypothesize that quaternion RNNs naturally provide a more suitable representation of the input sequence , since these multiple views can be directly embedded in the multiple dimensions space of the quaternion , leading to better generalization . 3 QUATERNION RECURRENT NEURAL NETWORKS . This Section describes the quaternion algebra ( Section 3.1 ) , the internal quaternion representation ( Section 3.2 ) , the backpropagation through time ( BPTT ) for quaternions ( Section 3.3.2 ) , and proposes an adapted weight initialization to quaternion-valued neurons ( Section 3.4 ) . 3.1 QUATERNION ALGEBRA . The quaternion algebra H defines operations between quaternion numbers . A quaternion Q is an extension of a complex number defined in a four dimensional space as : Q = r1 + xi + yj + zk , ( 1 ) where r , x , y , and z are real numbers , and 1 , i , j , and k are the quaternion unit basis . In a quaternion , r is the real part , while xi + yj + zk with i2 = j2 = k2 = ijk = −1 is the imaginary part , or the vector part . Such a definition can be used to describe spatial rotations . The information embedded in the quaterion Q can be summarized into the following matrix of real numbers : Qmat = r −x −y −zx r −z yy z r −x z −y x r . ( 2 ) The conjugate Q∗ of Q is defined as : Q∗ = r1− xi− yj− zk . ( 3 ) Then , a normalized or unit quaternion Q/ is expressed as : Q/ = Q√ r2 + x2 + y2 + z2 . ( 4 ) Finally , the Hamilton product ⊗ between two quaternions Q1 and Q2 is computed as follows : Q1 ⊗Q2 = ( r1r2 − x1x2 − y1y2 − z1z2 ) + ( r1x2 + x1r2 + y1z2 − z1y2 ) i+ ( r1y2 − x1z2 + y1r2 + z1x2 ) j + ( r1z2 + x1y2 − y1x2 + z1r2 ) k. ( 5 ) The Hamilton product ( a graphical view is depicted in Figure 1 ) is used in QRNNs to perform transformations of vectors representing quaternions , as well as scaling and interpolation between two rotations following a geodesic over a sphere in the R3 space as shown in ( Minemoto et al. , 2017 ) . 3.2 QUATERNION REPRESENTATION . The QRNN is an extension of the real-valued ( Medsker & Jain , 2001 ) and complex-valued ( Hu & Wang , 2012 ; Song & Yam , 1998 ) recurrent neural networks to hypercomplex numbers . In a quaternion dense layer , all parameters are quaternions , including inputs , outputs , weights , and biases . The quaternion algebra is ensured by manipulating matrices of real numbers ( Gaudet & Maida , 2018 ) . Consequently , for each input vector of size N , output vector of size M , dimensions are split into four parts : the first one equals to r , the second is xi , the third one equals to yj , and the last one to zk to compose a quaternion Q = r1 + xi + yj + zk . The inference process of a fully-connected layer is defined in the real-valued space by the dot product between an input vector and a real-valued M ×N weight matrix . In a QRNN , this operation is replaced with the Hamilton product ( Eq . 5 ) with quaternion-valued matrices ( i.e . each entry in the weight matrix is a quaternion ) . The computational complexity of quaternion-valued models is discussed in Appendix 6.1.2

Summary: The paper takes a good step toward developing more structured representations by exploring the use of quaternions in recurrent neural networks.  The idea is motivated by the observation that in many cases there are local relationships among elements of a vector that should be explicitly represented.  This is also the idea behind capsules - to have each "unit" output a vector of parameters to be operated upon rather than a single number.   Here the authors show that by incorporating quaternions into the representations used by RNNs or LSTMs, one achieves better performance at speech recognition tasks using fewer parameters.