Dual-module Inference for Efficient Recurrent Neural Networks
Authors: Liu Liu
Lei Deng
Shuangchen Li
Jingwei Zhang
Yihua Yang
Zhenyu Gu
Yufei Ding
Yuan Xie
Venue:  (2020)
Keywords: memory-efficient RNNs
dynamic execution
computation skipping

Abstract: Using Recurrent Neural Networks (RNNs) in sequence modeling tasks is promising in delivering high-quality results but challenging to meet stringent latency requirements because of the memory-bound execution pattern of RNNs. We propose a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference. Leveraging the error-resilient feature of nonlinear activation functions used in RNNs, we propose to use a lightweight little module that approximates the original RNN layer, which is referred to as the big module, to compute activations of the insensitive region that are more error-resilient. The expensive memory access and computation of the big module can be reduced as the results are only used in the sensitive region. Our method can reduce the overall memory access by 40% on average and achieve 1.54x to 1.75x speedup on CPU-based server platform with negligible impact on model quality.

Under review as a conference paper at ICLR 2020
DUAL-MODULE INFERENCE FOR EFFICIENT RECUR- RENT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Using Recurrent Neural Networks (RNNs) in sequence modeling tasks is promis- ing in delivering high-quality results but challenging to meet stringent latency re- quirements because of the memory-
execution pattern of RNNs. We propose a big-little dual-module inference to dynamically skip unnecessary memory access and computation speedup RNN inference. Leveraging the error-resilient feature of nonlinear activation functions used in RNNs. we propose to use lightweight little module that approximates the original RNN layer, which is referred as the big module, to compute activations of the insensitive region that are more error- resilient. The expensive memory access and computation of the big module can be reduced as the results only used in the sensitive region. Our method can reduce the overall memory access by 40% on average and achieve 1.54x to 1.75x speedup on CPU-based server platform with negligible impact model quality.
INTRODUCTION
Recurrent Neural Networks (RNNs) play a critical role in many natural language processing (NLP) tasks, such machine translation (Bahdanau al.] [014]] Wu al. (016), speech recognition (Graves et al. 2013 He et al. 2019), and speech synthesis (Wang et al. 2017), owing to the capabil- ity of modeling sequential data. These RNN-based services deployed in both data-center and edge devices often process inputs in a streaming fashion, which demands a real-time interaction. For instance, in cloud-based translation tasks, multiple requests need to be served with very stringent latency limit, where inference runs concurrently and individually (Park et al., 2018). For on-device speech recognition as an automated assistant, latency the primary concern to pursue fast response (He et al. 2019).
However, serving RNN-based models in latency-sensitive scenarios is challenging due to the low data reuse, and thus low resource utilization memory-bound General Matrix-Vector multiplica- tion (GEMV) is the core compute pattern of RNNs. Accessing weight matrix from off-chip memory is the bottleneck of GaMV-based RNN execution the weight data almost always cannot fit in on-chip memory. Moreover, accessing weights repeatedly at each time-step, especially in sequence- to-sequence models, makes the memory-bound problem severer. Subsequently, the on-chip com- puting resources would be under-utilized. Although batching is a walk-around for low-utilization, using a large batch size favored in latency-sensitive scenarios such as speech recognition and translation.
In essence, the RNN inference is not a simple GEMV. With non-linearity followed the GEMV op- eration as the activation functions, the RNN inference operation is "activated" GEMV. These non- linear activation functions as used in neural networks bring error resilience. As shown Figure T sigmoid and lanh functions in Gated RNNs such as Long Short-Term Memory (LSTM) ( (Hochreiter & Schmidhuber] 1997) and Gated Recurrent Unit (GRU) (Cho et al. 2014) have insensitive regions- green shaded regions- where the outputs saturated and resilient to errors in pre-activation accu- mulated results. In other words, not all computations in RNNs need to be accurate. Can we leverage this error resilience in RNNs reduce the memory access and eventually achieve speedup?
To this end, we propose a big-little dual-module inference that regarding the original RNN layer the big module, and use parameterized little module approximate the big module help reduce redundant weight accesses. The philosophy of dual-module inference is using approximated results computed by the memory-efficient little module in the insensitive region, and using accurate

---

Under review as a conference paper at ICLR 2020
1.00
1.0
0.75
0.5
0.50
0.0
0.25
0.5
0.00
-1.0
2--5-4-322-1 2
3
4
5
6
-6-5-4-3-2-10 1 2 4 5 6
Figure| 1: Insensitive (green shaded) and sensitive (white) regions of sigmoid (left) and tanh (right) nonlinear functions.
results computed by the memory-intensive big module in the sensitive region. For this reason, the final outputs are the mixture of the big-litle module. With the memory-effiiiett little module reduce the expensive data access and computation of the
computes for the insensitive region, big module and thus reduce overall memory access and computation cost. The (in)sensitive region dynamically determined using the little module results. Because of the error resilience, using approximated results in the insensitive region has negligible impact on the overall model quality but creates significant acceleration potential.
Given the trade-off between accuracy and efficiency, the little module needs to be sufficiently accu- rate while being as much lightweight as possible. To achieve this, we first use dimension reduction method random projection to reduce the parameter size of the little module and thus reducing data accesses. Then, we quantize the weights of the little module lower the overhead further. Because we only need the little module outputs in the insensitive region that is error-resilient, can afford aggressively low bit-width. Compared with common sparsification schemes, our hybrid approach avoids indexing overheads and therefore successfully achieves practical speedup.
We evaluate our method on language modeling and neural machine translation using RNN-based models and measure the performance, i.e., wall-clock execution time, on CPU-based server plat- form. With overall memory access data reduced by 40% on average, our method can achieve 1 1.54x to 1.75x speedup with negligible impact model quality.
MOTIVATION
In this section, we discuss the error resilience RNNs. As shown in Fig. ! the nonlinear activation functions sigmoid and tanh - have insensitive regions where the output activations are resilient to emoor introduced pre-activation accumulation results. We take single LSTM layer for language modeling over PTB dataset as an illustrative example. The baseline perplexity (PPL.) is 80.64. We consider two cases: adding a random error vector under norm distribution into the pre-activation accumulation results in the sensitive regions of four gates; adding errors the insensitive regions. We separate the (in)sensitive regions by 50% based on the activation magnitude.
As listed in Table we report the PPL on the testing set and the average cosine similarity between the activations of the baseline model and the error- introuuccd model. Before applying the nonlinear activation functions, the cosine similarity of two cases adding errors in the sensitive region the insensitive region are in the same level. However, we observe that after the nonlinear gates, the cosine similarity in the insensitive case is much closer one (i.e., fewer output errors) than that in
Table 1: Comparison of adding random errors to the sensitive or insensitive region of LSTM gates.
Cosine similarity before gate
Cosine similarity after gate
Case
PPI.
input
forget
cell
output
input
forget
cell
output
Sensitive
0.953
0.859
0.952
0.932
0.934
0.946
0.882
0.940
85.70
Insensitive
0.944
0.929
0.943
0.947
0.968
0.987
0.969
0.977
81.79
2

---

Under review as a conference paper at ICLR 2020
0
Timesteps
20
20
40
60
80
0
20
40
60
80
Neuron index
Neuron index
Figure 2: Dynamic region distribution across timesteps and inputs. The white and black colors denote neurons in the insensitive and sensitive regions, respectively. The left and right patterns are from different inputs.
the sensitive case. We further compare the PPL. these two cases, and we observe that introducing errors in the insensitive region causes little quality degradation.
The selection of which neurons should be in the (in)sensitive region is dynamic and input-dependent, which be seen in Figure Unlike the static weight sparsity that nections offline in advance, the dynamic region speculation requires very lightweight criterion for real-time processing. Taking all these into account, we propose dual-model inference method that efficiently determines (in)sensitive region and significantly saves the memory access and computa- tional cost.
prune the unused con-
APPROACH
Firstly, we explain the dual-module inference by taking fully-connected (FC) layer an example and then extend it to LSTM and GRU. For FC layer with unit batch size, the operation is typically formulated as a p(y).y = Wx + b, where W is weight matrix (W E R2xd), ). æ is input vector (x R. b bias vector E R"), a an activated output vector E IR",, and ip an activation function. The core computation is matrix-vector multiplication (GEMV), i.e., W a. Both the amount computation and memory access are ((d)); therefore, it is memory-bour the operation intensity is (1) according to the Roofline model analysis (Williams et al. 2009). Accessing weights from the off-chip memory is the bottleneck terms of both the latency and energy.
since
3.1 OVERVIEW OF MOALLM PHILOSOPHY
Our work aims at reducing the memory access of weight matrices for GEMV-based RNN inference. We show in Section [ that not all values in y need accurate computation, and those that belong the insensitive region can afford some level of approximation. In other words, we only need accurate computation and expensive memory access in the sensitive region of y and skip computation and memory access to weights that contribute the insensitive region of y. With that, still need approximated results in the insensitive region. Therefore, we propose to learn a lightweight little module from the original trained layer, here refer the original layer as the big module. Essentially, our little module is executed in low-dimensional and low oww-precision space, thus termed as LL module; by contrast, the original big module with high dimension and high precision is called H module. Let the outputs from these two modules be D and H respectively. If the LLL module approximates the H H module well, the final output vector mixture of results from the H H and the LL modules can be assembled by
= y C 1 Lا @ (1 - -m)
(1)
where m E {0, 1]" region while it switches to 0 in the insensitive region. The overall saving comes from skipping memory access to the big module while paying the overhead of accessing and computing of the little module.
R is a binary mask vector for the output switching. m, equals 1 in the sensitive
3

---

Under review as a conference paper at ICLR 2020
3.2 CONSTRUCT THE LL MODULE
As the HH module is the original pre-trained layer, we only need to construct the LL module. Delivering lightweight little module inference time is crucial to achieving real wall-clock time speedup. As discussed earlier, the sparsification method usually suffers from severe indexing over- heads; therefore, we turn to other approaches. In this work, we propose a hybrid compression with dimension reduction and data quantization to keep the little module as efficient as possible in com- putation and storage. The low dimension and low precision give birth to the desired L.L module. We emphasize two objects that should be reached in the design of LL module: (1) much lower compu- tation and memory overheads than the HH module; (2) approximating the outputs of module accurately.
First, we introduce sparse random projection to reduce the dimension of æ from R to RR* where k < d. Subsequently, the parameter size of the LL module is O(nk), which is much smaller compared with the parameter size ((nd) of the HH module. Random projection is a common technique for dimension reduction that preserves distances in Euclidean space (Achlioptas Bingham & Mannila, 2001: Li et al. 2006 Liu et al. 2019).
2003
The dimension reduction step be formulated as
طLL HH
(2)
where P is a sparse random matrix (P + (-1,0, 1) the probability of P.j being 1, 0.
and 1 is 1. 3. and .. respectively). Note that &s is configurable according to actual needs to balance the accuracy loss and inference cost. We choose the value of & according to |Achliopta (2003):
4logn
k- = 2/2 e²/3
(3)
where n is the number of rows in W and E a real number in (0, 1).
Second, after the dimension reduction, we quickly construct a lightweight little module in the low- dimensional space to approximate the pre-trained big module. The parameters of the latter (i.e., WHH and bdH ) are kept frozen while the parameters of the former (i.e., WLL and b44) are updated by stochastic gradient descent (SGD) to minimize the following loss function:
- 5=55
Σ
11
y J.
a. Σ II "H -W + bz. S
(4)
where S' is the mini-batch size. Essentially, for each pair of big-litle modules, we apply linear regression on the little module to approximate the function of the big module and optimize the mean square error of the two. Apparently, the parameter size of W LL is O(kk), much smaller than the original weight WHH of O(nd) in the high-dimensional space. Even if further considering the projection cost of O(kd), the overhead is still much lower than the vanilla inference. In this way, the memory-boume issue in GEMV -bseed models can be greatly alleviated; the computational complexity is also reduced. The SGD overhead for constructing the above module can be amortized by the pattern
Finally, based the constructed low-dimensional module, we also apply data quantization tech- nique to reduce the parameter precision. Data quantization can further shrink the storage space of LL parameters due to the shorter bit-width. The input is also quantized during run-time to reduce the computation cost. In our design, we apply one-time uniform quantization on W LL to avoid complicated calculations. Although some other accurate quantization methods are available as well, we find that one-tine me-time quantization works well in our dual- inference given in Equation (T). This error tolerance benefit from the fact that the computation in the insensitive region has a small influence on the final outputs.
3.3 DETERMINE THE INSENSITIVE REGION
The dual-module inference relies on a binary mask m to switch between outputs of the "accurate & costly" HH module and the "approximated & efficient" LL module. Hence, the generation of m is a crucial factor to control the overall performance by adjusting the trade-off between accuracy and efficiency. Thanks to the saturation region of the nonlinear activation functions in RNNs, such
4

---

Under review as a conference paper at ICLR 2020
3.5
Frequency (10+)
3
3.0
2-
1.5
1.0
0.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.00 0.25 0.50 0.75 1.00 Cell gate (tanh)
Input gate (sigmoid)
Figure 3: Output activation distribution sigmoid and tanh gates as in LSTM.
as sigmoid and tamh, we observe a unipolar or bipolar distribution of their outputs, as depicted in Figure 3 This affords two excellent opportunities: (1) It is possible to remove the majority of the computation and access from the costly H H module by setting the peak areas in Figure as insensitive regions; (2) The saturation output values in those regions such
0 in sigmoid and near +1 tanh additionally allow inaccurate computations because the outputs are insensitive to approximated values. According to the above observations and analysis, design specific criterion for each activation function. In particular, they are governed by
sigmoid: : if L Osigmoidd m, = 1; otherwise, m,
(5)
Landh: : if Oranh < L < Otaanto m, = 1; otherwise, m.; 0
where sigmoidd > 0, 0 0 < 0, and Ozank > 0 are constant thresholds. Note that these thresholds can be searched to target insensitive ratio using validation dataset or tuned at run-time that acts as knob for accuracy -ffuacy-ffficcenc trade-off.
3.4 OVERVIEW OF DUAL-MODULE INFERENCE ALGORITHM
The overall implementation is provided in Algorithm 1 After the construction of the LL model, the
consequent dual-module inference needs five steps: (1) Dimension reduction and data quantization for each dynamical input æ as x QPx HM where Q(-) is quantization function; (2) Obtain
the approximated output y L by performing yo = ( # L. + bb²) where W & bbd are
stored quantized parameters; (3) Calculate the switching mask vector m according to Equation ():
(4) Obtain a faction of actual output " by performing / (W[,; : H + 6 if
m; 1; (5) Produce the final output y according to the assembling in Equation
Algorithm 1: Dual-module Inference Algorithm
Data: HH module parameters: WHH 8HH, quantized LL module parameters: W Q and bb";
thresholds Os to determine m: random projection matrix P: current input x !1
Result: Final output
Step 2/L
Step 2: yul op( W "Q
LL bb4):
Step 3: Generating m according to Equation
Step 4-5: foreach m; E m do
if m then yi y HH 4 (W [i, HH H + HHH:;
else yi :L:
end
3.5 APPLY TO RECURRENT NEURAL NETWORKS
We discuss how to apply the proposed dual-module inference for an FC layer to RNNs, including LSTM and GRU. We will explain the LSTM implementation for illustration, while the extension to
5

---

Under review as a conference paper at ICLR 2020
70
35
Base Little
Base
Memory access data (MB)
60
30
Little
50
Operations (M)
25
40
20
30
15
31%
20
10-
31%
43%
32%
40%
57%
10
14%
7%
14%
7%
10%
8%
10%
57%
L750
L1500
G750
G1500
L1024
L1024r
Geo Mean
L750
L1500
G750
G1500
L1024
L1024r
Geo Mean
Figure 4: Comparison of memory access data and operations between baseline layers and the litrle
module of dual-module enhanced layers. L750 indicates single-layer LSTM with 750 hidden units; G is short for GRU; L/024r indicates LSTM with residual input.
GRU is quite straightforward. The dynamics of an LSTM cell can be described as
f(1 = ( + W w + Wh,h( 1))
i(I) ab; + Wi + Wiah(1 1))
o(t) = a (b. + Wo + Woth(t 1))
(6)
g(I) 0(b, + W (g + Wl I))
c(t) c(t 1) f(t) +g( о i(t)
Ix(() = ((c(t) @ o(1)
where f. the states of forget, input, and output gate, respectively, and g is the input activation. Each of them has its own bias vector and weight matrices. c and h are the cellular and hidden states of the hidden layer, respectively. (-)) and ((-) are sigmoid function and tanh function, respectively.
The computation of each gate is similar The first difference is the two GEMV computations in each gate; we apply dimension reduction, construction of the LL module, and data quantization on both GEMV computations. The second difference is that there is additional temporal dimension in RNNs. We should guarantee the approximation performance of the LL module at all time steps. Taking the forget gate as example, the linear map works for both (t) P, (t) and h h 1) P.h HH 1). The loss function for constructing the LL module is slightly modified
FC-like layer; therefore, Algorithm [ still holds.
+W L- ST ΣΣ |16 + HH x "**((*W*** HH (1-1))-(b}++ +W I ( 1h (1-1))
(7) Here the minimization considers not only S training samples in each mini- but also T time steps. The data quantization, switching mask (i.e., m) generation, and output assembling is the same Algorithm [ describes. Applying avoid repetition. Note that the input & and hidden state h can have different sizes, termed as de and da, respectively. For simplicity, we set Pz gexd, and Ps. gexd, let LLL and h hL4 to the same length k. For the gate with tanh function, we set 10 Orann! = 10 also for simplicity; however, different magnitudes are allowed.
other gates is similar so do not discuss them to
3.6 SAVING AND OVERHEAD ANALYSIS
The target of our dual-mon module inference method is to reduce the expensive off-chip memory access of the big module with the help of the little module. We introduce an insensitive ratio as the number of outputs using the little module results over entire outputs. The ratio can be interpreted as the zero ratio in mask m as in Equation] I In other words, the higher insensitive ratio will have less memory access to the big module. For example, obtaining ratio of 50% results in reducing 50% of weight matrix accessing in GEMV operation. The choice of accurate ratio determines the model inference quality, and it is knob trade-off model inference quality latency run-time.

---

Under review as a conference paper at ICLR 2020
The overhead of dual-module inference is small due the use of dimension reduction and quanti- zation. When choosing reduced dimension and low-precision bit-width of the little module, we use Equation 3 with = 0.5 and INT8 quantization by default. We also explore different levels of dimension reduction and quantization in Section [43 and Section [44 As shown in Figure 4 we compare memory access data and operations between the single-module the base case and the little module of dual-module inference using a set of LSTM and GRU layers. On average, the little module accounts 10% storage overhead and 40% operation overhead compared with the base case. Note that we count the number of operations in Figure 4 regardless of precision; and the little mod- ule computation overhead can be further reduced using oow-precision compute kernel as we used in performance evaluation.
4 EVALUATION
We first evaluate the model inference quality and execution time under different insensitive ratio and then conduct two sensitivity studies on dimension reduction and quantization.
Our method is evaluated on CPU-based server platform (Intel(R) Xeon(R) CPU E5-2698 v4) as most inference workloads run on CPUs (Park et al. 2018). We use PyTorch to train the little module and evaluate inference quality. The baseline implementation is the PyTorch CPU version with Intel MKL (version 2019.4) as the back-end BLAS kernel library. Our custom kernel implementation uses multi-threaded MKL dot-product kernel at BLAS level-1 to compute the big module instead of BLAS level-2 or level-3 kernels. The kernel-wise performance is measured as wall-clock time and averaged with 1000 runs, assuming cold cache the execution of each RNN cell representing the real-world cases, for example in the decoder of seq2seq model.
We first evaluate our method on single-layer LSTM GRU used in language modeling tasks and then on multi-layer stacked LSTM in GNMT model used in machine translation tasks a standard benchmark model for inference as in MLPerf We train the little module while freezing the param- eters of the big module, and we use the same training set and validation set to run SGD optimization.
4.1 LANGUAGE MODELING
We first evaluate our method on single-layer LSTMs/GPUs. Our implementations are adapted from the word-level language modeling example from PoTch with same hyper- parameters to train base- line models. We report word-level perplexity (PPI.) as the measure of model quality. As listed in Table the baseline LSTM model achieves 80.64 PPL at the latency of 1.477ms. Then, we varying the insensitive ratio to show the quality-performance trade-off; the larger insensitive ratio indicates more results are from the little module and less memory access compute the big module. As we increase the insensitive ratio, we observe the degradation of quality as the perplexity increases during a gradual reduction in execution time. When the insensitive ratio is 50%, the perplexity is slightly increased to 81.36, which is negligible in language modeling tasks, while the inference speedup is 1.67x.
We observe similar the case of base LSTT with 750 hidden units with dual-module LSTM with 1500 hidden units and 50% insensitive ratio, although the memory access reduction is at the same level, our proposed dual- module approach achieves much better model quality because we kept the expressive power of a larger LSTM layer.
trade-off for LSTM with 750 hidden units. Comparing
We further report the results using single-layer GRU on word-level language modeling tasks as in Table 3, Using dual-module inference GRUs expresses the similar quality-performen trade-off as of LSMMs. Our dual-modine method is generally applicable to both LSTMs and GRUs.
4.2 NEURAL MACHINE TRANSLATION
Given the promising results on language modeling, we further investigate Neural Machine Trans- lation (NMT), which is a promising end-to-end learning approach for automated translation (Wu

---

Under review as a conference paper at ICLR 2020
Table 2: LSTT perplexity and execution time (ms).
Insensitive
hidden size:
1500
hidden size:
750
ratio
PPL
Diff.
Time
Speedup
PPL
Diff.
Time
Speedup
Base
80.64
n/a
1.477
1.00x
84.32
n/a
0.546
1.00x
10%
80.72
-0.08
1.315
1.12x
84.42
-0.10
0.448
1.22x
30%
80.56
0.08
1.095
1.35x
84.43
-0.11
0.415
1.32x
50%
81.36
-0.72
0.885
1.67x
84.29
0.03
0.342
1.60x
70%
87.48
-6.83
0.641
2.30x
84.89
-0.57
0.287
1,90x
90%
109.37
-28.73
0.380
3,89x
88.44
-4.12
0.216
2.53x
Table GRU perplexity and execution time (ms).
Insensitive
hidden size:
1500
hidden size:
750
ratio
PPL
DifT.
Time
Speedup
PPL
Diff.
Time
Speedup
Base
85.48
n/a
1.182
1.00x
89.64
n/a
0.466
1.00x
10%
85.62
-0.14
1.024
1.15x
89.81
-0.17
0.383
1.22x
30%
86.01
-0.53
0.869
1.36x
89.63
0.01
0.334
1.40x
50%
88.73
-3.25
0.726
1.63x
89.69
-0,05
0.302
1.54x
70%
98.09
-12.61
0,545
2.17x
92.51
-2.87
0.284
1.64x
90%
122.75
-37.27
0.350
3.38x
102.37
-12.73
0.198
2.35x
et al. 2016). The base model consists four-layer stacked LSTM in both the encoder and the decoder of the sequence-to- -e-uueneee modeling. We focus on the speedup of the decoder since is the most memory intensive and the most time-consuming part (95%). The decoder has a four-layer unidirectional LSTM with hidden size 1024 with residual connections starting from the third layer, i.e., the input size of the third and fourth layer is 2048. Our experiments show de-tokenized BLEU score measure the model inference quality on the public WMT16 English-German German dataset. The baseline model obtains BLEU score of 24.32.
We replace the LSTM layers in the decoder with our proposed dual-module-basee LSTM layers. Similar to single-layer LSTM results, using the little module computed results in the insensitive region can reduce overall memory access while maintaining model quality. As listed Table 4 method can achieve imperceptible BLEU score degradation while speedup inference by 1.75x for the first two LSTM layers and 1.70x for the last two LSTM layers. When compromising more translation quality, i.e., decreasing the BLEU score by 2.4, our method can achieve more than 2x speedup.
4.3 DISCUSSION ON DIMENSION REDUCTION
Dimension reduction an integral part of our dual-module inference method to reduce the number of parameters and memory footprint. Here, we study the impact of different levels of dimension reduction on the model quality and performance. We conduct experiments on language modeling using single-layer LSTM of 1500 hidden units. We quantize the little module to INT8 and reduce the hidden dimension from 1500 to three different levels, which are calculated by Sparse Random Projection, We fix the insensitive ratio to 50% across this set of experiments. As we can see in Table the higher dimension of the little module, the better approximation the little module can perform. For instance, when we reduce hidden size to 966 and quantize to INT8, the dual-module inference can achieve slightly better quality PPL of 80.40 and 1.37x speedup. More aggressive dimension reduction can further have more speedup at the cost of more quality degradation: hidden dimension reduced to 417 and 266 can have 1.67x and 1.71x speedup but increase PPL by 0.72 and 2.87, respectively.
We further show the overhead of performing the computation of the little module. As listed in the last three columns in 3 we measure the execution time of performing dimension reduction on
From http://gthalccmm///////oooooooma
8

---

Under review as a conference paper at ICLR 2020
Table 4: GNMT BLEU score and execution time (ms). (1024, 2048) indicates the hidden size is 1024 and the ippu size is 2048; similarly for (1024, 1024).
Insensitive
Quality
(1024,
I024)
(1024,
2048)
ratio
BLEU
Diff.
Time
Speedup
Time
Speedup
Base
24.32
n/a
0.838
1.00x
1.092
1.00x
10%
24.33
0.01
0.679
1.23x
0.962
1.14x
30%
24.18
-0.14
0.541
1.55x
0.803
1.36x
50%
23.73
-0.59
0.480
1.75x
0.642
1.70x
70%
21.92
-2.40
0.360
2.33x
0.479
2.28x
90%
11.77
-12.55
0.243
3.45x
0.307
3.56x
Table 5: Sensitivity study of dimension reduction.
Dimension
PPL
DifT.
Time
Speedup
SRP
Little
Big
1500 (baseline)
80.64
n/a
1.477
1.00x
0%
0%
100%
966 (e 0.3)
80.40
0.24
1.076
1.37x
8%
14%
44%
417 (e = 0.5)
81.36
-0.72
0.885
1.67x
4%
8%
47%
266 0.7)
83.51
-2.87
0.866
1.71x
3%
5%
46%
inputs by Sparse Random Projection, computation of the little module, and computation of the big module; the execution time is normalized to the baseline case, i.e., the execution time of standard LSTM, to highlight the percentage of overheads. When the hidden dimension is reduced to 966, the overhcad of the little module accounts 22% while the execution time of the big module is cut off by choose 0.5 as the default parameter in sparse random projection
f In our experiments, as it demonstrated good quality and speedup trade-off by our study. When further reducing the hidden dimension 266, there only slight improvement on speedup compared with the hidden size of 417 in the litrle module, where the overhead of the little module is already small enough, but quality dropped significantly.
4.4 DISCUSSION ON QUANTIZATION
Quantizing the weights of the little module is another integral part of keeping memory footprint small. We show different quantization levels the impact on model quality and parameter size. After training the little module, we can quantize its weights to lower precision to reduce the memory accessing on top dimension reduction. As we can see Table 6 more aggressive quantization leads to smaller parameter size that can reduce the overhead of computing the little module; on the other hand, the approximation of the little module is compromised by quantization. We can quantize the little module up to INT4 without significant quality degradation. Using lower precision would degrade the quality while decreasing the parameter size. For performance evaluation, we choose INTS the quantization level since leverage off-the-shelf INT8 GEMM kemee in MKL. We expect more speedup once the little module overhead can be further reduced by leveraging INT4 compute kernels.
Table 6: Inference quality and parameter size comparison under different levels of quantization on the little module
Precision
Base
FP32
INT16
INT8
INT4
INT2
INTT
Perplexity Diff.
80.64
81.28
81.18
81.25
81.28
82.14
94.75
n/a
-0.64
-0.54
-0.61
-0.64
-1.50
-14.11
MSE
n/a
0.408
0.425
0.444
0.465
0.573
3.337
Param. size (MB)
68.7
19.1
9.6
4.8
2.4
1.2
0.6
We measured the exccution time with multi multi-threading.

---

Under review as a conference paper at ICLR 2020
5 RELATED WORK
As we aim at the memory-bound problem of RNN--baee inference applications, we limit the discus- sion related work to RNN inference acceleration. Although we only evaluate our dual-module inference method on standard LSTMs/GRUs, we believe our method can be applied to many newly released sequence modeling networks (Shen et al., 2019; Wu et al., 2019) leverage the com- monly observed crror-resilience of non- non-linear activation functions.
5.1 MODEL COMPRESSION
Compressing DNN models via data quantization, weight sparsity, and knowledge distillation is promising to deliver efficient deployment for inference. Xu et al. (2018) propose a quantization method for RNNs where both weights and activations are quantized to binary or ternary. Wang al. (2018) propose hybrid ternary quantization method based on the different distributions of weights and activations.
Weight pruning, i.c., inducing weight sparsity, has been proposed to reduce the parameter size of pre-trained model (Han et al., (2015b a). While fine-grained pruning element-wise could reduce the number of parameters (Narang et al. 2017 Zhu & Gupta 2017 Dai et al. 2018), indexing non- zero weights causes extra memory cost and would offset the benefits of reducing parameter size; is hard to gain practical acceleration on general-purpot hardware need hardware specialization (Mao et al. 2017). Although structural pruning (Wen et al. (007)) and knowledge distillation (Polino et al. (2018) could achieve speedup, the applicability on more complicated tasks such NMT using large-scale dataset is unstudied; besides, those methods require extensive retraining via regulariza- tion that would increase the training cost and hard to find solution.
Model compression would inevitably compromise the compressive power of RNNs. Our method, by means, is supposed to replace model compression but provides an orthogonal approach to accelerate RNN inference. Using the analogy knowledge distillation, we do not simply deploy a student network learned from the teacher network. Instead, we let the teacher network, applied with model compression or not, help with the student - the little module learned from the base module- and collaboratively perform inference with reduced memory access and computation.
5.2
COMPUTATION SKIPPING
Instead of model compression, many work propose to skip computations dynamically based on certain criterion. Bolukbasi et al. (2017) propose dynamic execution with layer-wise early exit. Zhang et al.) (2018) leverage a special feature of LSTM that using threshold- based pruning on output gates and generate a mask, and then using the mask to skip computation as well as data access of masked-out neurons of the other three gates. Neil et al. (2017) utilize temporal input sparsity but need to enforce input similarity with threshold clipping. Campos et al. (2018) selectively skip updating the hidden states for some inputs. However, these work either depend special cell structure or rely on the temporal similarity of inputs which is not evaluated on NLP tasks such as NMT. We are the first that propose general and principled method to reduce memory access and computation of Gated RNNs, including both LSTMs and GRUs.
CONCLUSION
In this paper, we describe a big-little dual-module inference method mitigate the memory-bound problem in serving RNN-based models under latency-sensitive scenarios. We leverage the error resilience of nonlinear activation functions by using the lightweight little module to compute for the insensitive region and using the big module with skipped memory access and computation to compute for the sensitive region. With overall memory access reduced by near half, our method can achieve 1.54x to 1. 1.75x wall-clock time speedup without significant degradation on model quality.
REFERENCES
Dimitris Achlioptas. Database- frienndl random projections:
with binary
coins. Journal of computer and System Sciences, 66(4):671-687, 2003.
10

---

Under review as a conference paper at ICLR 2020
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.044 2014.
Ella Bingham and Heikki Mannila. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 245-250, ACM, 2001.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learn- ing- Volume 70, ICML'17, pp. 527-536. JMLR.org, 2017. URL http://di acm. acm.org/ org/ citation cfm? 1 d= 330538 3305436
Vetor Campos, Brendan Jou, Xavier Gir i Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: Learning to skip state updates in recurrent neural networks. In International Conference Learn- ing Representations, 2018. URL/atttss:/opeereviii
net/ forum foeerummdd-HHwWWa HKwWHyYY
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Xiaoliang Dai, Hongxu Yin, and Niraj K Jha. Grow and prune compact, fast, and accurate Istms. arXiv preprint arXiv:1805. 2018.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recogmition with deep recur- rent neural networks. In 2013 IEEE international conference on acousties, speech and signal processing, pp. 6645-6649 IEFE, 2013.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv: 15...00149 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming end-to-end speech recog- nition for mobile devices. In ICASSP 2019-20/1 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6381-6385. IEEE, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780) 1997.
Ping Li, Trevor Hastie, and Kenneth W Church. Very sparse random projections. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 287-296. ACM, 2006.
Liu Liu, Lei Deng, Xing Hu, Maohua Zhu, Guoqi Li, Yufci Ding, and Yuan Xie. Dynamic sparse graph for efficient deep learning. In International Conference on Learning Representations, 2019. URR.htttts:://peereeee
net ffot/fouu?? 1 9oBoR
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William Dally. Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint 17Xiv:1705.08922. 2017.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint 1704.05 00.4.01119 19, 2017.
Daniel Neil, Jun Haeng Lee, Tobi Delbruck, and Shih-Chii Liu. Delta networks for optimized recurrent network computation. In Proceedings of the 34th International Conference on Machine Starning-Volume 70, pp. 2584-2593. JMLR. org, 2017.
Jongsoo Park, Maxim Naumov, Protonu Basu, Summer Deng, Aravind Kalaiah, Daya Khudia, James Law, Parth Malani, Andrey Malevich, Satish Nadathur, et al. Deep learning inference in facebook data centers: Characterization, performance optimizations and hardware implications. arXiv preprint arXiv: 18/ 109886, 2018.

---

Under review as a conference paper at ICLR 2020
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. In International Conference on Learning Representations, 2018. URL https:
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Repre- sentations, 2019. URL
Peiqi Wang, Xinfeng Xie, Lei Deng, Guoqi Li, Dongsheng Wang, and Yuan Xie. Hitnet: hybrid ternary recurrent neural network. In Advances in Neural Information Processing Systems, pp- 604-614, 2018.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:
2017.
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. Learning intrinsic sparse structures within long short-term memory, 2017.
Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An insightful visual perfor- mance model for floating-poimt programs and multicore architectures. Technical report, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States), 2009.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019. URL
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 1.X.:0660900814: 2016.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hong- bin Zha. Alternating multi-bit quantization for recurrent neural networks. arXiv preprint 1rXiv:1802.00150, 2018
X. Zhang, C. Xie. J. Wang, W. Zhang, and X. Fu. Towards memory friendly long- short term memory networks (Istms) on mobile gpus. In 2018 51st Annual IEEF/ACM International Symposium on Microarchieccurre (MICRO). pp. 162-174, Oct 2018. doi: 10.1 10.110//ICCO.CC 200.00001000000220
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 17000177, 2017.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX A COMPARISON WITH WEIGHT PRUNING METHOD
We compare our proposed dual-module inference approach with the automated gradual pruning method (Zhu & Gupta 2017), which is a popular pruning method with open implementation Firstly, compared with weight pruning, our method achieves better quality with practical speedup 1.54x 1.75x reduction on wall-clock time on commodity CPUs while element-wise weight pruning requires specialized hardware to gain real speedup of computation given irregular sparsity. Moreover, our dual-module inference method can be further applied on top of pruned models to reduce execution time by reducing memory access.
Comparison of our proposed dual-module inference (using 50% insensitive ratio) with weight pruning using one LSTM layer with 1500 units word language modeling task on WikiText- 2 dataset.
Table 7:
Method Dense
PPL w/o dual- inference
PPL w/ dual-module inference
85.52
86.21
80% weight sparsity
86.42
88.46
90% weight sparsity
88.75
90.96
"From http://jihhhh.mom///raaayyyyeeee
13


Reviews:
[{"review_id": "HkxgREa4Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to compress the networks so as to accelerate the running procedure as well as save the storage. The authors propose a dual-module that is composed of a little module and big module. The big module use the full original data and parameters whereas the little module use small data and parameters by random projecting on the original ones. Through a statistical investigation, the authors provide a method to choose the little or big module dynamically. By applying this method on LSTM and GRU, the authors make them more efficient. Experimental results validate this point.\n\nOverall, I think this paper is well written and easy to follow. The idea of dual-module is interesting and wise. The experiments is valid. However, I would like the authors to answer me two questions. \n1 How do you draw Figure 2? \n2 Why is the distribution of the outputs of sigmoid not bipolar? Intuitively, the distribution should be similar to that of tanh since their functions are similar.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1xpu-VWcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors design a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference.\nIt cannot only mitigate the memory-bound problem to speedup RNN inference but also leverage the error resilience of nonlinear activation functions by using the lightweight little module to compute for the insensitive region and using the big module with skipped memory access and computation. They also conduct several experiments to evaluate their approaches.\n\n\nStrength:\n(1)\tWell written in general.\n(2)\tContributions clearly stated and justified\n\nA couple of minor questions:\n\n(1)\tThe organization in Section 3 can to be improved. It is better if the author can give a brief overview of their method first and then go into details.\n(2)\tSome of the technical details necessary for understanding the soundness of the techniques are either missing or are poorly explained. For example, in Section 3\na.\tthe authors did not mention how to construct the HH module\nb.\tthe authors did not provide detailed information of how to conduct dimension reduction since this will affect the performance\nc.\tmany mathematical notations and equations need to be revised to increase the readability. For instance, there is no information in the paper that explain why the authors design functions in such certain way (such as equation (2) and (3))\nd.\tthe authors did not provide enough detailed information about how to select the quantization methods since there are lots of approaches such as static (uniform) or dynamic quantization, where different methods may have different impacts on the final performance \ne.\tthe authors mentioned that they have tried both sigmoid and tanh activation function to find the sensitive region. However, they do not provide enough reason to do so, how about using other non-linear activation functions\n\n(3)\tThe organization in Section 4 can to be improved. It is better if the author can introduce the motivation of each experiment.\n\n(4)\tParameters of the evaluation are unclear or missing. For example, \na.\twhat is the data size, what is the dropout, learning rate, how many time stamps for the RNN modules\nb.\twhy the authors only use single-layer LSTM and why they select 750 and 1500 hidden units in the experiments\n\n(5)\tWhile the authors have applied their models on other existing method, they do not provide good discussion of results and such model seems old (released in 2016). It would be great if this approach can also be applied on other newly released models\n(6)\tSome tables need to be reorganized. For instance, for table 6, there needs some space between the title and the table.\n(7)\tWhile the paper has good coverage of the prior work, I do suggest the authors can also cite or discuss some newly designed models (in 2019).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1e26lgRYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposes an approach to reduce memory access and computation in Recurrent Neural Networks.  Specifically, they train a second \"little\" neural network to approximate a pre-trained \"big\" network and use simple rules to switch between the little and the big network.  The approach can provide some speedups while reducing the total number of memory accesses and the computational cost in exchange for a mild decrease in predictive performance.\n\nWhile this manuscript proposes a reasonable contribution, it lacks real comparisons to many of the common competing methods that hinder the interpretation.\n\nWeight sparsity/pruning is a very common approach that has shown the ability for larger speedups than what is shown here.  I disagree with the assessment that \"those methods require extensive retraining via regularization.\" Realistically, you can take a pretrained model and add the penalties with mild re-training and extensive reuse of code.  The result is also simpler to implement.  I would argue that this is less work than the proposed approach, which requires switching rules and a second trained network.  I don't know which is better, but the authors should actually evaluate whether their approach improves over the more popular approach.\n\nThe authors should give better discussion and motivation on the random projections.  This is an area with very deep theory, yet the rules are provided without a rationale.  Realistically, where does the sparsity level in the random sparse matrix come from?  Why use the rule for k in (3)?  The authors should motivate and discuss this section more.\n\nAlso note that there is existing literature on learning multiple models and switching between them, for example:\nBolukbasi, Tolga, et al. \"Adaptive neural networks for efficient inference.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\nAs there is a lot of similarity in the motivations, you should discuss that line of research in your related work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an efficient RNN architecture that dynamically switches big and little modules during inference. In the experiments, authors demonstrate that the proposed method achieves favorable speed up compared to baselines, and the contribution is orthogonal to weight pruning. 
All reviewers agree that the paper is well-written and that the proposed method is easy to understand and reasonable. However, its methodological contribution is limited because the core idea is essentially the same as distillation, and dynamically gating the modules is a common technique in general. Moreover, I agree with the reviewers that the method should be compared with more other state-of-the-art methods in this context. Accelerating or compressing DNNs are intensively studied topics and there are many approaches other than weight pruning, as authors also mention in the paper. As the possible contribution of the paper is more on the empirical side, it is necessary to thoroughly compare with other possible approaches to show that the proposed method is really a good solution in practice. For these reasons, I’d like to recommend rejection. 

Decision: Reject