EXPLORATION OF EFFICIENT ON-DEVICE ACOUSTIC MODELING WITH NEURAL NETWORKS

1 INTRODUCTION . Acoustic modeling is the fundamental part of automatic speech recognition , and is a representative application of recurrent neural networks ( RNNs ) ( Graves et al. , 2013 ; Amodei et al. , 2016 ; Miao et al. , 2015 ) . The sequence learning algorithms are usually executed sequentially in the time domain to utilize the result of the previous timesteps . For example , in LSTM RNNs , the output of the previous time step , ht−1 , is used for computing the gate signals of the current timestep , t ( Hochreiter & Schmidhuber , 1997 ) . Thus , we need to compute ht−1 , ht , and ht+1 sequentially . This is extremely disadvantageous when implementing the algorithms on embedded devices because the network parameters must be retrieved from the external DRAM at every timestep . The execution of an LSTM RNN on a GPU does not suffer from this problem because multiple LSTM RNNs are executed simultaneously while sharing the network parameters among the RNNs . However , in embedded devices , the algorithm is usually executed as a single stream , and has no way of sharing the parameters with other sequences . One conventional solution to overcome this problem was reducing the size of the parameters by quantization , pruning , and data compression ( Narang et al. , 2017 ; Xu et al. , 2018 ; Prabhavalkar et al. , 2016 ) . However , the size of the parameters can be too big to accommodate on internal memory or cache of embedded systems . A model of the hardware of the embedded systems for executing the acoustic model is shown in Figure 1 . The hardware is based on the current generation of ARM Cortex-A57 CPU that contains four cores and L2 cache memory of 2 MB . The end-to-end acoustic model considered in the research contains at least 10 million parameters , which can reach 40 MB when implemented as a floatingpoint . We consider that the efficient execution of neural networks , with parameter sizes larger than that of the cache or internal memory capacity , is a very important research topic . This work is an algorithmic level optimization approach , and the multi-timestep parallel technique exploited in this research can be combined with the quantization and pruning techniques . Recently , quasi RNNs ( QRNNs ) and simple recurrent units ( SRUs ) were developed for the purpose of fast training and inference of very long sequences ( Balduzzi & Ghifary , 2016 ; Bradbury et al. , 2017 ; Lei et al. , 2017 ; Martin & Cundy , 2018 ) . These RNNs only employ simple feedback that can be represented by linear recurrence equations , and allow fast training as well as multi-timestep parallel inference . QRNNs have a good performance comparable to LSTM in language modeling and machine translation . Convolutional neural networks ( CNNs ) have also been used for sequence learning , such as machine translation , language modeling , and speech recognition ( Gehring et al. , 2017 ; Dauphin et al. , 2017 ; Liptchinsky et al. , 2017 ; Kang et al. , 2017 ) . A CNN has the advantage of parallel processing because there is no dependency between the input and output . However , CNNs only provide a finite context length , while RNN can consult infinite length , at least in theory . Besides these networks , a few different RNN structures have been devised in recent years , and they can be utilized for efficient implementation in embedded systems ( Subakan & Smaragdis , 2017 ; Li et al. , 2018 ) . However , when applied to acoustic modeling , the accuracy of these sequence-modeling algorithms has not been as good as that of LSTM RNNs . In this work , we assess the performance of several sequence modeling algorithms including diagonal LSTM , QRNN , and Gated ConvNet , and compare them with LSTM RNN . The performance was measured using CTC-based end-to-end acoustic modeling and frame-level phoneme recognition tasks . Our initial experiments showed that LSTM RNN performed best , thus we modified the networks by including 1-D convolution to each layer . Note that adding 1-D convolution can increase the length of the context . We find several non-LSTM-based algorithms that yield a much better performance when compared to LSTM RNN . We also vary the depth and width to improve the performance . The execution performances of these algorithms on embedded devices are also compared . This paper is organized as follows . In Section 2 , we introduce a few acoustic modeling algorithms that permit multi-timestep parallelization . Section 3 provides the experimental results of the acoustic models on end-to-end speech recognition and framewise phoneme classification tasks . Section 4 discusses related studies . The concluding remarks are shown in Section 5 . 2 ACOUSTIC MODELING ALGORITHMS . The most well-known and widely used sequence-learning algorithm is the LSTM RNN , represented by Eq . 1 . As illustrated in this equation , the algorithm employs two feedback structures ; one is an element-wise feedback for updating ct , and the other is the complex feedback for computing ft , it , and ot using the previous output ht−1 . Here , we assume that the total sizes of the matrices Uf , Ui , Uo and Uc are larger than that of the cache memory in embedded systems . Thus , when executing the LSTM RNN on embedded devices , the parameters must be retrieved at every inference , which incurs large memory access overhead . ft = σ ( Wfxt +Ufht−1 + bf ) , it = σ ( Wixt +Uiht−1 + bi ) , ot = σ ( Woxt +Uoht−1 + bo ) , ( 1 ) ct = ft ct−1 + it tanh ( Wcxt +Ucht−1 + bc ) , ht = ot tanh ( ct ) . To solve this problem , we explore three algorithms . The first is the diagonal LSTM that employs simple diagonal matrices for Uf , Ui , Uo , and Uc ; the second is the QRNN that does not use Uf , Ui , Uo and Uc ; and the third is the Gated ConvNet that does not employ any recurrent loops . 2.1 DIAGONAL LSTM RNN . Matrix-vector multiplication is widely used in neural networks , however , it demands a large number of parameters as well as arithmetic operations . Thus , element-wise multiplication operations are quite advantageous , and one example is the computation of ct in Eq . 1 . The element-wise multiplication is equivalent to the multiplication of a diagonal matrix and a vector . The diagonal LSTM RNN only allows the diagonal terms of Uf , Ui , Uo and Uc to be non-zero in the LSTM equation ( Subakan & Smaragdis , 2017 ; Li et al. , 2018 ) . In embedded implementations , we can assume that the diagonal terms can be stored in the cache memory because their size is much smaller than that of the full matrices . The number of words for the diagonal terms is a few thousands , while that of the full matrices is millions . This architecture was used for music sequence prediction , and showed a good performance ( Subakan & Smaragdis , 2017 ) . However , the efficacy of this algorithm for acoustic modeling has not been studied . When implementing this algorithm , we compute [ Wf , Wi , Wo , Wc ] ×xt , for t = 1 , 2 , . . . , TP , at one time , which can be completed by fetching the parameters only once from the DRAM . Thus , the overhead of parameter access can be reduced in proportion to TP . Note that the rest of the procedure for computing ht , for t = 1 , 2 , . . . , TP , can be conducted without accessing the external DRAM , particularly because the sizes of Uf , Ui , Uo , and Uc are quite small due to diagonalization . 2.2 QRNN . The element-wise operations , or the multiplication of a diagonal matrix and a vector , are restricted compared to the matrix-vector multiplications ; the former should assume the same basis for the operand and the result , while the latter has the ability of altering the basis . The computation of ct in Eq . 1 uses ct−1 as the operand , thus element-wise operations can work . Simplification of the LSTM RNN using element-wise operations is currently of great interest ( Balduzzi & Ghifary , 2016 ; Bradbury et al. , 2017 ; Lei et al. , 2017 ) . The simplest form of QRNN is shown in Eq . 2. x̂t = tanh ( Wzxt + bz ) , [ ft , it , ot ] = σ ( [ Wf , Wi , Wo ] xt + [ bf , bi , bo ] ) , ct = ft ct−1 + it x̂t , ht = ot ct + ( 1− ot ) xt . ( 2 ) QRNN does not employ the feedback with ht−1 , thus it does not use Uf , Ui , Uo , and Uc . Instead , this algorithm can consult k input samples , xt , xt−1 , . . . , xt−k+1 , while LSTM uses only xt . The QRNN equation when k = 1 is represented by Eq . 2 . The parameter size of QRNN is linearly dependent on k , thus it is not advantageous to use a large value of k. The simplest case , k = 1 , is called the simple recurrent unit ( SRU ) ( Lei et al. , 2017 ) . 2.3 GATED CONVNET . A CNN does not employ any recurrent loops , but it can conduct sequence modeling of finite length . Although this architecture should assume a finite context length , the modeling can solve many practical problems , such as language and acoustic modeling . Although CNNs may have a limited context length due to the feed-forward only structure , the architecture is relatively free from the gradient exploding problems . One variant of CNN-based architecture is the Gated ConvNet , which has a good performance in acoustic and language modeling ( Dauphin et al. , 2017 ; Liptchinsky et al. , 2017 ) . The equation for the Gated ConvNet with the filter length of T is shown in Eq . 3 , h ( X ) = ( X ∗W + b ) σ ( X ∗V + c ) ( 3 ) where ∗ denotes convolution operation and represents element-wise multiplication . W , V ∈ RT×w×w and b , c ∈ Rw are trainable variables where w is the width of the network . X ∈ RN×w is the input vector where N is the sequence length . The parameter size of the Gated ConvNet increases proportional to the filter length T and the depth D , while the context length corresponds to D × T . Thus , there is compromise between the parameter size and the context length . The Gated ConvNet structure can be considered to employ 2-dimensional ( 2-D ) convolutions in multilayers . 2.4 ELEMENT WISE 1-D CONVOLUTION . Increasing the context length in the time domain is critical for an improved performance . However , the parameter size increases proportional to the context length . When the filter length is T and the width of the neural network is w for both the input and output , the parameter size is proportional to T × w2 . One way of reducing the parameter size and increasing the context length is to employ element-wise time domain convolution , which is called 1-D convolution in this work . In this case , the parameter size increase is negligible due to dimensionality reduction . However , this assumes that the feature map elements are independent , and consulting other elements is not beneficial . We conducted experiments by adding the 1-D convolution to the QRNN and Gated ConvNet , and a minimum value of the 2-D filter length T was employed for the sake of reducing the parameter sizes . The 1-D convolution can be causal or consult the future input . We denote the 1-D convolution coefficients as ht . When t is a negative value , it denotes consulting the future input . The 1-D parameter of h0 , . . . , h4 means that it consults the past four inputs including the current one .

Summary: This paper discusses applications of variants of RNNs and Gated CNN to acoustic modeling in embedded speech recognition systems, and the main focus of the paper is computational (memory) efficiency when we deploy the system. The paper well describes the problem of the current LSTM, especially focusing on the recurrent connection matrix operations, which is a bottle neck in this scenario, and introduces variants of RNNs (e.g., QRNN). Also these variants may not yield enough performance compared with LSTM, but 1-D convolution and/or deep structure helps to avoid the degradation. One of the biggest issues of this paper is that they use CTC as an acoustic model, while still many real speech recognition applications and major open source (Kaldi) use hybrid HMM/DNN(TDNN, LSTM, CNN, etc.) systems. Therefore, the paper's claim on CTC is not along with the current application trends. (It may be changed near future, but still hybrid systems are dominant). For example, the WSJ WER performance listed in Table 3 is easily obtained by a simple feed-forward DNN in the hybrid system. The latest Lattice free MMI with TDNN can achieve better performance (~2.X% WER), and this decoding is quite fast compared with LSTM. The authors should consider this current situation of state-of-the-art speech recognition. Also, the techniques described in the paper are all based on existing techniques, and the paper lacks the technical novelty.