Dual-module Inference for Efficient Recurrent Neural Networks

1 INTRODUCTION . Recurrent Neural Networks ( RNNs ) play a critical role in many natural language processing ( NLP ) tasks , such as machine translation ( Bahdanau et al. , 2014 ; Wu et al. , 2016 ) , speech recognition ( Graves et al. , 2013 ; He et al. , 2019 ) , and speech synthesis ( Wang et al. , 2017 ) , owing to the capability of modeling sequential data . These RNN-based services deployed in both data-center and edge devices often process inputs in a streaming fashion , which demands a real-time interaction . For instance , in cloud-based translation tasks , multiple requests need to be served with very stringent latency limit , where inference runs concurrently and individually ( Park et al. , 2018 ) . For on-device speech recognition as an automated assistant , latency is the primary concern to pursue a fast response ( He et al. , 2019 ) . However , serving RNN-based models in latency-sensitive scenarios is challenging due to the low data reuse , and thus low resource utilization as memory-bound General Matrix-Vector multiplication ( GEMV ) is the core compute pattern of RNNs . Accessing weight matrix from off-chip memory is the bottleneck of GEMV-based RNN execution as the weight data almost always can not fit in on-chip memory . Moreover , accessing weights repeatedly at each time-step , especially in sequenceto-sequence models , makes the memory-bound problem severer . Subsequently , the on-chip computing resources would be under-utilized . Although batching is a walk-around for low-utilization , using a large batch size is not favored in latency-sensitive scenarios such as speech recognition and translation . In essence , the RNN inference is not a simple GEMV . With non-linearity followed the GEMV operation as the activation functions , the RNN inference operation is “ activated ” GEMV . These nonlinear activation functions as used in neural networks bring error resilience . As shown in Figure 1 , sigmoid and tanh functions in Gated RNNs such as Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) and Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) have insensitive regions – green shaded regions – where the outputs are saturated and resilient to errors in pre-activation accumulated results . In other words , not all computations in RNNs need to be accurate . Can we leverage this error resilience in RNNs to reduce the memory access and eventually achieve speedup ? To this end , we propose a big-little dual-module inference that regarding the original RNN layer as the big module , and use a parameterized little module to approximate the big module to help reduce redundant weight accesses . The philosophy of dual-module inference is using approximated results computed by the memory-efficient little module in the insensitive region , and using accurate results computed by the memory-intensive big module in the sensitive region . For this reason , the final outputs are the mixture of the big-little module . With the memory-efficient little module computes for the insensitive region , we can reduce the expensive data access and computation of the big module and thus reduce overall memory access and computation cost . The ( in ) sensitive region is dynamically determined using the little module results . Because of the error resilience , using approximated results in the insensitive region has a negligible impact on the overall model quality but creates a significant acceleration potential . Given the trade-off between accuracy and efficiency , the little module needs to be sufficiently accurate while being as much lightweight as possible . To achieve this , we first use a dimension reduction method – random projection – to reduce the parameter size of the little module and thus reducing data accesses . Then , we quantize the weights of the little module to lower the overhead further . Because we only need the little module outputs in the insensitive region that is error-resilient , we can afford aggressively low bit-width . Compared with common sparsification schemes , our hybrid approach avoids indexing overheads and therefore successfully achieves practical speedup . We evaluate our method on language modeling and neural machine translation using RNN-based models and measure the performance , i.e. , wall-clock execution time , on CPU-based server platform . With overall memory access data reduced by 40 % on average , our method can achieve 1.54x to 1.75x speedup with negligible impact on model quality . 2 MOTIVATION . In this section , we discuss the error resilience of RNNs . As shown in Fig . 1 , the nonlinear activation functions – sigmoid and tanh – have insensitive regions where the output activations are resilient to errors introduced in pre-activation accumulation results . We take a single LSTM layer for language modeling over PTB dataset as an illustrative example . The baseline perplexity ( PPL ) is 80.64 . We consider two cases : adding a random error vector under norm distribution into the pre-activation accumulation results in the sensitive regions of four gates ; adding errors to the insensitive regions . We separate the ( in ) sensitive regions by 50 % based on the activation magnitude . As listed in Table 1 , we report the PPL on the testing set and the average cosine similarity between the activations of the baseline model and the error-introduced model . Before applying the nonlinear activation functions , the cosine similarity of two cases – adding errors in the sensitive region or the insensitive region – are in the same level . However , we observe that after the nonlinear gates , the cosine similarity in the insensitive case is much closer to one ( i.e. , fewer output errors ) than that in the sensitive case . We further compare the PPL of these two cases , and we observe that introducing errors in the insensitive region causes little quality degradation . The selection of which neurons should be in the ( in ) sensitive region is dynamic and input-dependent , which can be seen in Figure 2 . Unlike the static weight sparsity that we can prune the unused connections offline in advance , the dynamic region speculation requires a very lightweight criterion for real-time processing . Taking all these into account , we propose a dual-model inference method that efficiently determines ( in ) sensitive region and significantly saves the memory access and computational cost . 3 APPROACH . Firstly , we explain the dual-module inference by taking a fully-connected ( FC ) layer as an example and then extend it to LSTM and GRU . For an FC layer with unit batch size , the operation is typically formulated as a = ϕ ( y ) , y = Wx + b , where W is a weight matrix ( W ∈ Rn×d ) , x is an input vector ( x ∈ Rd ) , b is a bias vector ( b ∈ Rn ) , a is an activated output vector ( a ∈ Rn ) , and ϕ is an activation function . The core computation is matrix-vector multiplication ( GEMV ) , i.e. , Wx . Both the amount of computation and memory access are O ( nd ) ; therefore , it is memory-bounded since the operation intensity is O ( 1 ) according to the Roofline model analysis ( Williams et al. , 2009 ) . Accessing weights from the off-chip memory is the bottleneck in terms of both the latency and energy . 3.1 OVERVIEW OF DUAL-MODULE PHILOSOPHY . Our work aims at reducing the memory access of weight matrices for GEMV-based RNN inference . We show in Section 2 that not all values in y need accurate computation , and those that belong to the insensitive region can afford some level of approximation . In other words , we only need accurate computation and expensive memory access in the sensitive region of y and skip computation and memory access to weights that contribute to the insensitive region of y . With that , we still need approximated results in the insensitive region . Therefore , we propose to learn a lightweight little module from the original trained layer , here we refer the original layer as the big module . Essentially , our little module is executed in a low-dimensional and low-precision space , thus termed as LL module ; by contrast , the original big module with high dimension and high precision is called HH module . Let the outputs from these two modules be yLL and yHH , respectively . If the LL module approximates the HH module well , the final output vector – a mixture of results from the HH and the LL modules – can be assembled by y = yHH m+ yLL ( 1−m ) ( 1 ) where m ∈ { 0 , 1 } n is a binary mask vector for the output switching . mi equals 1 in the sensitive region while it switches to 0 in the insensitive region . The overall saving comes from skipping memory access to the big module while paying the overhead of accessing and computing of the little module . 3.2 CONSTRUCT THE LL MODULE . As the HH module is the original pre-trained layer , we only need to construct the LL module . Delivering a lightweight little module at inference time is crucial to achieving real wall-clock time speedup . As discussed earlier , the sparsification method usually suffers from severe indexing overheads ; therefore , we turn to other approaches . In this work , we propose a hybrid compression with dimension reduction and data quantization to keep the little module as efficient as possible in computation and storage . The low dimension and low precision give birth to the desired LLmodule . We emphasize two objects that should be reached in the design of LL module : ( 1 ) much lower computation and memory overheads than the HH module ; ( 2 ) approximating the outputs of HH module accurately . First , we introduce sparse random projection to reduce the dimension of x from Rd to Rk where k d. Subsequently , the parameter size of the LL module is O ( nk ) , which is much smaller compared with the parameter size O ( nd ) of the HH module . Random projection is a common technique for dimension reduction that preserves distances in Euclidean space ( Achlioptas , 2003 ; Bingham & Mannila , 2001 ; Li et al. , 2006 ; Liu et al. , 2019 ) . The dimension reduction step can be formulated as xLL = PxHH ( 2 ) where P is a sparse random matrix ( P ∈ 1√ 3 · { −1 , 0 , 1 } k×d , the probability of Pij being −1 , 0 , and 1 is 16 , 2 3 , and 1 6 , respectively ) . Note that k is configurable according to actual needs to balance the accuracy loss and inference cost . We choose the value of k according to Achlioptas ( 2003 ) : k = 4logn 2/2− 3/3 ( 3 ) where n is the number of rows in W and is a real number in ( 0 , 1 ) . Second , after the dimension reduction , we quickly construct a lightweight little module in the lowdimensional space to approximate the pre-trained big module . The parameters of the latter ( i.e. , WHH and bHH ) are kept frozen while the parameters of the former ( i.e. , WLL and bLL ) are updated by stochastic gradient descent ( SGD ) to minimize the following loss function : L = 1 S ∑ s ||yHH − yLL||22 = 1 S ∑ s || ( WHHxHH + bHH ) − ( WLLxLL + bLL ) ||22 ( 4 ) where S is the mini-batch size . Essentially , for each pair of big-little modules , we apply linear regression on the little module to approximate the function of the big module and optimize the mean square error of the two . Apparently , the parameter size of WLL is O ( nk ) , much smaller than the original weight WHH of O ( nd ) in the high-dimensional space . Even if further considering the projection cost of O ( kd ) , the overhead is still much lower than the vanilla inference . In this way , the memory-bound issue in GEMV-based models can be greatly alleviated ; the computational complexity is also reduced . The SGD overhead for constructing the above module can be amortized by the pattern of “ construct-once-inference-forever ” . Finally , based on the constructed low-dimensional module , we also apply data quantization technique to reduce the parameter precision . Data quantization can further shrink the storage space of LL parameters due to the shorter bit-width . The input x is also quantized during run-time to reduce the computation cost . In our design , we apply one-time uniform quantization on WLL to avoid complicated calculations . Although some other accurate quantization methods are available as well , we find that one-time quantization works well in our dual-module inference given in Equation ( 1 ) . This error tolerance is benefit from the fact that the computation in the insensitive region has a small influence on the final outputs .

Summary: This manuscript proposes an approach to reduce memory access and computation in Recurrent Neural Networks.  Specifically, they train a second "little" neural network to approximate a pre-trained "big" network and use simple rules to switch between the little and the big network.  The approach can provide some speedups while reducing the total number of memory accesses and the computational cost in exchange for a mild decrease in predictive performance.