Quaternion Recurrent Neural Networks

1 INTRODUCTION . In the last few years , deep neural networks ( DNN ) have encountered a wide success in different domains due to their capability to learn highly complex input to output mapping . Among the different DNN-based models , the recurrent neural network ( RNN ) is well adapted to process sequential data . Indeed , RNNs build a vector of activations at each timestep to code latent relations between input vectors . Deep RNNs have been recently used to obtain hidden representations of speech unit sequences ( Ravanelli et al. , 2018a ) or text word sequences ( Conneau et al. , 2018 ) , and to achieve state-of-the-art performances in many speech recognition tasks ( Graves et al. , 2013a ; b ; Amodei et al. , 2016 ; Povey et al. , 2016 ; Chiu et al. , 2018 ) . However , many recent tasks based on multi-dimensional input features , such as pixels of an image , acoustic features , or orientations of 3D models , require to represent both external dependencies between different entities , and internal relations between the features that compose each entity . Moreover , RNN-based algorithms commonly require a huge number of parameters to represent sequential data in the hidden space . Quaternions are hypercomplex numbers that contain a real and three separate imaginary components , perfectly fitting to 3 and 4 dimensional feature vectors , such as for image processing and robot kinematics ( Sangwine , 1996 ; Pei & Cheng , 1999 ; Aspragathos & Dimitros , 1998 ) . The ∗CIFAR Senior Fellow idea of bundling groups of numbers into separate entities is also exploited by the recent manifold and capsule networks ( Chakraborty et al. , 2018 ; Sabour et al. , 2017 ) . Contrary to traditional homogeneous representations , capsule and quaternion networks bundle sets of features together . Thereby , quaternion numbers allow neural network based models to code latent inter-dependencies between groups of input features during the learning process with fewer parameters than RNNs , by taking advantage of the Hamilton product as the equivalent of the ordinary product , but between quaternions . Early applications of quaternion-valued backpropagation algorithms ( Arena et al. , 1994 ; 1997 ) have efficiently solved quaternion functions approximation tasks . More recently , neural networks of complex and hypercomplex numbers have received an increasing attention ( Hirose & Yoshida , 2012 ; Tygert et al. , 2016 ; Danihelka et al. , 2016 ; Wisdom et al. , 2016 ) , and some efforts have shown promising results in different applications . In particular , a deep quaternion network ( Parcollet et al. , 2016 ; 2017a ; b ) , a deep quaternion convolutional network ( Gaudet & Maida , 2018 ; Parcollet et al. , 2018 ) , or a deep complex convolutional network ( Trabelsi et al. , 2017 ) have been employed for challenging tasks such as images and language processing . However , these applications do not include recurrent neural networks with operations defined by the quaternion algebra . This paper proposes to integrate local spectral features in a novel model called quaternion recurrent neural network1 ( QRNN ) , and its gated extension called quaternion long-short term memory neural network ( QLSTM ) . The model is proposed along with a well-adapted parameters initialization and turned out to learn both inter- and intra-dependencies between multidimensional input features and the basic elements of a sequence with drastically fewer parameters ( Section 3 ) , making the approach more suitable for low-resource applications . The effectiveness of the proposed QRNN and QLSTM is evaluated on the realistic TIMIT phoneme recognition task ( Section 4.2 ) that shows that both QRNN and QLSTM obtain better performances than RNNs and LSTMs with a best observed phoneme error rate ( PER ) of 18.5 % and 15.1 % for QRNN and QLSTM , compared to 19.0 % and 15.3 % for RNN and LSTM . Moreover , these results are obtained alongside with a reduction of 3.3 times of the number of free parameters . Similar results are observed with the larger Wall Street Journal ( WSJ ) dataset , whose detailed performances are reported in the Appendix 6.1.1 . 2 MOTIVATIONS . A major challenge of current machine learning models is to well-represent in the latent space the astonishing amount of data available for recent tasks . For this purpose , a good model has to efficiently encode local relations within the input features , such as between the Red , Green , and Blue ( R , G , B ) channels of a single image pixel , as well as structural relations , such as those describing edges or shapes composed by groups of pixels . Moreover , in order to learn an adequate representation with the available set of training data and to avoid overfitting , it is convenient to conceive a neural architecture with the smallest number of parameters to be estimated . In the following , we detail the motivations to employ a quaternion-valued RNN instead of a real-valued one to code inter and intra features dependencies with fewer parameters . As a first step , a better representation of multidimensional data has to be explored to naturally capture internal relations within the input features . For example , an efficient way to represent the information composing an image is to consider each pixel as being a whole entity of three strongly related elements , instead of a group of uni-dimensional elements that could be related to each other , as in traditional real-valued neural networks . Indeed , with a real-valued RNN , the latent relations between the RGB components of a given pixel are hardly coded in the latent space since the weight has to find out these relations among all the pixels composing the image . This problem is effectively solved by replacing real numbers with quaternion numbers . Indeed , quaternions are fourth dimensional and allow one to build and process entities made of up to four related features . The quaternion algebra and more precisely the Hamilton product allows quaternion neural network to capture these internal latent relations within the features encoded in a quaternion . It has been shown that QNNs are able to restore the spatial relations within 3D coordinates ( Matsui et al. , 2004 ) , and within color pixels ( Isokawa et al. , 2003 ) , while real-valued NN failed . This is easily explained by the fact that the quaternion-weight components are shared through multiple quaternion-input parts during the Hamilton product , creating relations within the elements . Indeed , Figure 1 shows that the multiple weights required to code latent relations within a feature are considered at the same level as for 1https : //github.com/Orkis-Research/Pytorch-Quaternion-Neural-Networks learning global relations between different features , while the quaternion weight w codes these internal relations within a unique quaternion Qout during the Hamilton product ( right ) . Then , while bigger neural networks allow better performances , quaternion neural networks make it possible to deal with the same signal dimension but with four times less neural parameters . Indeed , a 4-number quaternion weight linking two 4-number quaternion units only has 4 degrees of freedom , whereas a standard neural net parametrization has 4 × 4 = 16 , i.e. , a 4-fold saving in memory . Therefore , the natural multidimensional representation of quaternions alongside with their ability to drastically reduce the number of parameters indicate that hyper-complex numbers are a better fit than real numbers to create more efficient models in multidimensional spaces . Based on the success of previous deep quaternion convolutional neural networks and smaller quaternion feed-forward architectures ( Kusamichi et al. , 2004 ; Isokawa et al. , 2009 ; Parcollet et al. , 2017a ) , this work proposes to adapt the representation of hyper-complex numbers to the capability of recurrent neural networks in a natural and efficient framework to multidimensional sequential tasks such as speech recognition . Modern automatic speech recognition systems usually employ input sequences composed of multidimensional acoustic features , such as log Mel features , that are often enriched with their first , second and third time derivatives ( Davis & Mermelstein , 1990 ; Furui , 1986 ) , to integrate contextual information . In standard RNNs , static features are simply concatenated with their derivatives to form a large input vector , without effectively considering that signal derivatives represent different views of the same input . Nonetheless , it is crucial to consider that time derivatives of the spectral energy in a given frequency band at a specific time frame represent a special state of a time-frame , and are linearly correlated ( Tokuda et al. , 2003 ) . Based on the above motivations and the results observed on previous works about quaternion neural networks , we hypothesize that quaternion RNNs naturally provide a more suitable representation of the input sequence , since these multiple views can be directly embedded in the multiple dimensions space of the quaternion , leading to better generalization . 3 QUATERNION RECURRENT NEURAL NETWORKS . This Section describes the quaternion algebra ( Section 3.1 ) , the internal quaternion representation ( Section 3.2 ) , the backpropagation through time ( BPTT ) for quaternions ( Section 3.3.2 ) , and proposes an adapted weight initialization to quaternion-valued neurons ( Section 3.4 ) . 3.1 QUATERNION ALGEBRA . The quaternion algebra H defines operations between quaternion numbers . A quaternion Q is an extension of a complex number defined in a four dimensional space as : Q = r1 + xi + yj + zk , ( 1 ) where r , x , y , and z are real numbers , and 1 , i , j , and k are the quaternion unit basis . In a quaternion , r is the real part , while xi + yj + zk with i2 = j2 = k2 = ijk = −1 is the imaginary part , or the vector part . Such a definition can be used to describe spatial rotations . The information embedded in the quaterion Q can be summarized into the following matrix of real numbers : Qmat = r −x −y −zx r −z yy z r −x z −y x r . ( 2 ) The conjugate Q∗ of Q is defined as : Q∗ = r1− xi− yj− zk . ( 3 ) Then , a normalized or unit quaternion Q/ is expressed as : Q/ = Q√ r2 + x2 + y2 + z2 . ( 4 ) Finally , the Hamilton product ⊗ between two quaternions Q1 and Q2 is computed as follows : Q1 ⊗Q2 = ( r1r2 − x1x2 − y1y2 − z1z2 ) + ( r1x2 + x1r2 + y1z2 − z1y2 ) i+ ( r1y2 − x1z2 + y1r2 + z1x2 ) j + ( r1z2 + x1y2 − y1x2 + z1r2 ) k. ( 5 ) The Hamilton product ( a graphical view is depicted in Figure 1 ) is used in QRNNs to perform transformations of vectors representing quaternions , as well as scaling and interpolation between two rotations following a geodesic over a sphere in the R3 space as shown in ( Minemoto et al. , 2017 ) . 3.2 QUATERNION REPRESENTATION . The QRNN is an extension of the real-valued ( Medsker & Jain , 2001 ) and complex-valued ( Hu & Wang , 2012 ; Song & Yam , 1998 ) recurrent neural networks to hypercomplex numbers . In a quaternion dense layer , all parameters are quaternions , including inputs , outputs , weights , and biases . The quaternion algebra is ensured by manipulating matrices of real numbers ( Gaudet & Maida , 2018 ) . Consequently , for each input vector of size N , output vector of size M , dimensions are split into four parts : the first one equals to r , the second is xi , the third one equals to yj , and the last one to zk to compose a quaternion Q = r1 + xi + yj + zk . The inference process of a fully-connected layer is defined in the real-valued space by the dot product between an input vector and a real-valued M ×N weight matrix . In a QRNN , this operation is replaced with the Hamilton product ( Eq . 5 ) with quaternion-valued matrices ( i.e . each entry in the weight matrix is a quaternion ) . The computational complexity of quaternion-valued models is discussed in Appendix 6.1.2

Summary: Quality: sufficient though there are issues. Work done in automatic speech recognition on numerous variants of recurrent models, such as interleaved TDNN and LSTM (Peddinti 2017), is completely ignored [addressed in the revision]. The description of derivatives needs to mention the linear relationship between input features and derivatives (see trajectory HMMs by Zen and Tokuda) [addressed in the revision]. TIMIT is a very simple task [addressed by adding WSJ experiments]. Derivations in the appendices could be connected better [addressed in the revision].