user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Dual-module Inference for Efficient Recurrent Neural Networks

Paper content:
Using Recurrent Neural Networks (RNNs) in sequence modeling tasks is promising in delivering high-quality results but challenging to meet stringent latency requirements because of the memory-bound execution pattern of RNNs.
We propose a big-little dual-module inference to dynamically skip unnecessary memory access and computation to speedup RNN inference.
Leveraging the error-resilient feature of nonlinear activation functions used in RNNs, we propose to use a lightweight little module that approximates the original RNN layer, which is referred to as the big module, to compute activations of the insensitive region that are more error-resilient.
The expensive memory access and computation of the big module can be reduced as the results are only used in the sensitive region.
Our method can reduce the overall memory access by 40% on average and achieve 1.54x to 1.75x speedup on CPU-based server platform with negligible impact on model quality.
Recurrent Neural Networks (RNNs) play a critical role in many natural language processing (NLP) tasks, such as machine translation Wu et al., 2016) , speech recognition (Graves et al., 2013; He et al., 2019) , and speech synthesis , owing to the capability of modeling sequential data.
These RNN-based services deployed in both data-center and edge devices often process inputs in a streaming fashion, which demands a real-time interaction.
For instance, in cloud-based translation tasks, multiple requests need to be served with very stringent latency limit, where inference runs concurrently and individually (Park et al., 2018) .
For on-device speech recognition as an automated assistant, latency is the primary concern to pursue a fast response (He et al., 2019) .
However, serving RNN-based models in latency-sensitive scenarios is challenging due to the low data reuse, and thus low resource utilization as memory-bound General Matrix-Vector multiplication (GEMV) is the core compute pattern of RNNs.
Accessing weight matrix from off-chip memory is the bottleneck of GEMV-based RNN execution as the weight data almost always cannot fit in on-chip memory.
Moreover, accessing weights repeatedly at each time-step, especially in sequenceto-sequence models, makes the memory-bound problem severer.
Subsequently, the on-chip computing resources would be under-utilized.
Although batching is a walk-around for low-utilization, using a large batch size is not favored in latency-sensitive scenarios such as speech recognition and translation.
In essence, the RNN inference is not a simple GEMV.
With non-linearity followed the GEMV operation as the activation functions, the RNN inference operation is "activated" GEMV.
These nonlinear activation functions as used in neural networks bring error resilience.
As shown in Figure 1 , sigmoid and tanh functions in Gated RNNs such as Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Unit (GRU) have insensitive regionsgreen shaded regions -where the outputs are saturated and resilient to errors in pre-activation accumulated results.
In other words, not all computations in RNNs need to be accurate.
Can we leverage this error resilience in RNNs to reduce the memory access and eventually achieve speedup?
To this end, we propose a big-little dual-module inference that regarding the original RNN layer as the big module, and use a parameterized little module to approximate the big module to help reduce redundant weight accesses.
The philosophy of dual-module inference is using approximated results computed by the memory-efficient little module in the insensitive region, and using accurate results computed by the memory-intensive big module in the sensitive region.
For this reason, the final outputs are the mixture of the big-little module.
With the memory-efficient little module computes for the insensitive region, we can reduce the expensive data access and computation of the big module and thus reduce overall memory access and computation cost.
The (in)sensitive region is dynamically determined using the little module results.
Because of the error resilience, using approximated results in the insensitive region has a negligible impact on the overall model quality but creates a significant acceleration potential.
Given the trade-off between accuracy and efficiency, the little module needs to be sufficiently accurate while being as much lightweight as possible.
To achieve this, we first use a dimension reduction method -random projection -to reduce the parameter size of the little module and thus reducing data accesses.
Then, we quantize the weights of the little module to lower the overhead further.
Because we only need the little module outputs in the insensitive region that is error-resilient, we can afford aggressively low bit-width.
Compared with common sparsification schemes, our hybrid approach avoids indexing overheads and therefore successfully achieves practical speedup.
We evaluate our method on language modeling and neural machine translation using RNN-based models and measure the performance, i.e., wall-clock execution time, on CPU-based server platform.
With overall memory access data reduced by 40% on average, our method can achieve 1.54x to 1.75x speedup with negligible impact on model quality.
Dimension reduction is an integral part of our dual-module inference method to reduce the number of parameters and memory footprint.
Here, we study the impact of different levels of dimension reduction on the model quality and performance.
We conduct experiments on language modeling using single-layer LSTM of 1500 hidden units.
We quantize the little module to INT8 and reduce the hidden dimension from 1500 to three different levels, which are calculated by Sparse Random Projection.
We fix the insensitive ratio to be 50% across this set of experiments.
As we can see in Table 5 , the higher dimension of the little module, the better approximation the little module can perform.
For instance, when we reduce hidden size to 966 and quantize to INT8, the dual-module inference can achieve slightly better quality -PPL of 80.40 -and 1.37x speedup.
More aggressive dimension reduction can further have more speedup at the cost of more quality degradation: hidden dimension reduced to 417 and 266 can have 1.67x and 1.71x speedup but increase PPL by 0.72 and 2.87, respectively.
We further show the overhead of performing the computation of the little module.
As listed in the last three columns in Table 5 , we measure the execution time of performing dimension reduction on inputs by Sparse Random Projection, computation of the little module, and computation of the big module; the execution time is normalized to the baseline case, i.e., the execution time of standard LSTM, to highlight the percentage of overheads.
When the hidden dimension is reduced to 966, the overhead of the little module accounts 22% while the execution time of the big module is cut off by half 3 .
In our experiments, we choose = 0.5 as the default parameter in sparse random projection as it demonstrated good quality and speedup trade-off by our study.
When further reducing the hidden dimension to 266, there is only a slight improvement on speedup compared with the hidden size of 417 in the little module, where the overhead of the little module is already small enough, but the quality dropped significantly.
Quantizing the weights of the little module is another integral part of keeping memory footprint small.
We show different quantization levels the impact on model quality and parameter size.
After training the little module, we can quantize its weights to lower precision to reduce the memory accessing on top of dimension reduction.
As we can see in Table 6 , more aggressive quantization leads to smaller parameter size that can reduce the overhead of computing the little module; on the other hand, the approximation of the little module is compromised by quantization.
We can quantize the little module up to INT4 without significant quality degradation.
Using lower precision would degrade the quality while decreasing the parameter size.
For performance evaluation, we choose INT8 as the quantization level since we leverage off-the-shelf INT8 GEMM kernel in MKL.
We expect more speedup once the little module overhead can be further reduced by leveraging INT4 compute kernels.
As we aim at the memory-bound problem of RNN-based inference applications, we limit the discussion on related work to RNN inference acceleration.
Although we only evaluate our dual-module inference method on standard LSTMs/GRUs, we believe our method can be applied to many newly released sequence modeling networks (Shen et al., 2019; as we leverage the commonly observed error-resilience of non-linear activation functions.
In this paper, we describe a big-little dual-module inference method to mitigate the memory-bound problem in serving RNN-based models under latency-sensitive scenarios.
We leverage the error resilience of nonlinear activation functions by using the lightweight little module to compute for the insensitive region and using the big module with skipped memory access and computation to compute for the sensitive region.
With overall memory access reduced by near half, our method can achieve 1.54x to 1.75x wall-clock time speedup without significant degradation on model quality.

assistant: text: We accelerate RNN inference by dynamically reducing redundant memory access using a mixture of accurate and approximate modules.