user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: AntMan: Sparse Low-Rank Compression To Accelerate RNN Inference

Paper content:
Wide adoption of complex RNN based models is hindered by their inference performance, cost and memory requirements.
To address this issue, we develop AntMan, combining structured sparsity with low-rank decomposition synergistically, to reduce model computation, size and execution time of RNNs while attaining desired accuracy.
AntMan extends knowledge distillation based training to learn the compressed models efficiently.
Our evaluation shows that AntMan offers up to 100x computation reduction with less than 1pt accuracy drop for language and machine reading comprehension models.
Our evaluation also shows that for a given accuracy target, AntMan produces 5x smaller models than the state-of-art.
Lastly, we show that AntMan offers super-linear speed gains compared to theoretical speedup, demonstrating its practical value on commodity hardware.
Remarkable advances in deep learning (DL) have produced great models across a wide variety of tasks such as computer vision, machine reading, speech generation and image recognition BID7 .
However, wide adoption of these models is still limited by their inference performance, cost and memory requirements.
On the client side, all pervasive devices like smart-phones, tablets and laptops have limited memory and computational resources to handle large complex DL models.
On the server side, intensive computation can render the models too slow to meet responsiveness requirements and too expensive to scale, preventing their deployment in production.Model Compression is a flourishing area that aims to reduce the computational and memory complexity of DL models to address the aforementioned problems without significantly affecting accuracy.
Compressing Convolution Neural Networks (CNNs) have already been widely explored in the past few years BID3 , while our work focuses on Recurrent Neural Networks (RNNs), which are broadly used among various natural language processing tasks BID17 BID24 BID29 .
It is well known that large RNN models are computation and memory intensive (Zhang et al.) .
In particular, their computation increases linearly with sequence length, and their recurrent unit has to be computed sequentially, one step at a time with limited parallelism, both of which makes long execution time a crucial issue for RNN inference computation.
Compressing RNNs, however, is challenging, because a recurrent unit is shared across all the time steps in sequence, compressing the unit will aggressively affect all the steps.Inducing sparsity is one of the prominent approaches used for RNN compression.
BID18 proposed a pruning approach that deletes up to 90% connections in RNNs.
The obtained sparse matrices, however, have an irregular/non-structured pattern of non-zero weights, which is unfriendly for efficient computation in modern hardware systems BID12 BID25 .
To address this issue, BID19 proposed inducing block-sparsity in RNNs via pruning or group lasso regularization.
Similarly, BID26 introduces ISS, intrinsic structured sparsity for LSTMs BID9 , a type of RNN , such that a sparse LSTM can be transformed into a dense one but with smaller size.
ISS conveniently turns sparsity into efficient execution, but as its sparse structure is quite coarse-grained, it is hard to push out high sparsity without degrading accuracy, especially in RNNs where the hidden dimension is smaller than input dimension (elaborated in Section 5.1).Our
work explores a new line of structured sparsity on RNNs, using predefined compact structures as opposed to pruning and regularization based approaches. We
take inspiration from predefined compact CNN structures such as group convolutions BID11 and depth-wise separable convolutions BID4 . Specifically
, we replace matrix-vector multiplications (MVs), the dominant part of RNN computations, with localized group projections (LGP).LGP divides
the input and output vectors into groups where the elements of the output group is computed as a linear combination of those from the corresponding input group. In addition
, to empower the information flow across multiple groups along the steps of RNN computation, we use a permutation matrix or a dense-square matrix to combine outputs across groups, helping the compact structure to retain accuracy.Furthermore, we combine LGP with low-rank matrix decomposition in order to further reduce the computations. This is possible
as low rank and sparsity are complimentary to each other. Low-rank decomposition
such as SVD approximates a low-rank multiplication Ax as P Qx, where P and Q are dense. By imposing LGP-based
sparsity on P and Q, we reduce the computation further. For a given rank reduction
factor of r, we reduce the computation cost and model size by O(r 2 ), compared to O(r) by using low-rank decomposition methods like SVD BID6 alone.We call our compression approach AntMan -'shrink in scale' by synergistically combining structured sparsity and low-rank decomposition, but 'increase in strength' by enabling the flow across structured groups along RNN sequence to retain accuracy.To train RNN models with AntMan, we use teacher-student training paradigm BID1 by combining the label loss with teacher-MSE-loss and teacher-KL-divergence-loss. To improve the training efficiency
, we develop a new technique to decide proper coefficients to obtain high accuracy efficiently with minimal trials.We evaluate AntMan on multiple RNN based models for machine reading comprehension and language modeling. For a well-known MRC model BID24
, we reduce the computational complexity and model size of LSTMs (a particular type of RNN) by up to 25x with less than 1pt drop in F1 score. For PTB BID29 language model, we
achieve a computational reduction of 50x with no drop in perplexity, and 100x with just a single point drop in perplexity. We also construct language models
for PTB with perplexities ranging from 64 to 70, but with 3x to 5x fewer overall model weights (5x to 25x reduction in RNN weights) than the state-of-art.Last but not least, we develop efficient implementations of inference kernels on CPUs to serve models compressed by AntMan. We show that unlike computation with
unstructured sparsity, AntMan offers significant performance improvement for large RNN models even with modest levels of sparsity. Our evaluations show that a 2x to 10x
theoretical reduction in computation can result in up to 2x to 30x actual speedup, respectively, for moderate to large RNNs, demonstrating attractive practical value of AntMan on commodity hardware.
We develop AntMan, combining structured sparsity and low-rank decomposition, to reduce the computation, size and execution time of RNN models by order(s) of magnitude while achieving similar accuracy.
We hope its compression efficiency and effectiveness would help unblock and enable many great RNN-based models deployed in practice.
We discuss and compare AntMan with several compression techniques as below.Quantization: 16 and 8-bit quantization (original 32-bit) can be supported fairly easily on commodity hardware, resulting in a maximum compression of 4x.
Even more aggressive quantization (e.g., 2-7 bit) hardly provides additional computational benefit because commodity hardware does not support those in their instruction set, while 1-bit quantization does not offer comparable accuracy.In comparison, we demonstrate that AntMan achieves up to 100x reduction in computation without loss in accuracy.
Moreover, quantization can be applied to AntMan to further reduce the computation, and vice versa, as quantization and AntMan are complementary techniques.Pruning: Pruning can be used to generate both unstructured and structured sparsity.
The former is not computationally efficient while the latter requires specialized implementation for efficient execution.While we did not present pruning results in the paper, we did try out techniques on both PTB and BiDAF models to generate random sparsity as well as blocked sparsity.
In both cases, we were able to get more that 10x reduction in computation even in the absence of Knowledge distillation.
Therefore pruning provides excellent computation reduction.However, as discussed in the paper, those theoretical computational reductions cannot be efficiently converted into practical performance gains: Unstructured sparsity resulting from pruning suffers from poor computation efficiency; a 10x theoretical reduction leads to less than 4x improvement in performance while AntMan achieves 30x performance gain with 10x reduction for PTB like models.
TAB7 It is possible to achieve structured sparsity such as block sparsity through pruning.
However, structured sparsity requires implementing specialized kernels to take advantage of the computation reduction.
Its efficiency greatly depends on the implementation, and in general is far from the theoretical computation reduction.On the contrary both ISS and AntMan achieve good computation reduction, and can be efficiently executed using readily available BLAS libraries such as Intel MKL resulting in super linear speedups as shown in the paper.Direct Design: We compared AntMan with smaller RNN models (with smaller hidden dimension) trained using the larger teacher model.
Our results show that for the same level of compression AntMan achieves much higher accuracy.
TAB6 SVD RNN:We constructed compressed models by replacing matrix-multiplication with SVD of various rank, and trained the SVD based models using knowledge distillation.
Once again, we find that for the same level of compression, AntMan achieves much higher accuracy than SVD.
TAB6 Block Tensor Decomposition (BTD) : BTD is designed to compress RNNs whose inputs are produced by convolution based models, and contain certain redundancies.
AntMan, on the other hand, is generic to all RNN based models.
Also, BTD is designed to compress only the input vector and not the hidden vectors.
This hinders the performance of BTD over a range of RNNs, where the hidden vectors are also large.
Here, we compare the performance of AntMan with ISS, without using any knowledge distillation.
Please note that knowledge distillation is part of the training process for AntMan, but it is not for ISS.
Nevertheless, it is interesting to see how AntMan performs in the absence of a teacher.When trained without knowledge distillation, our experiments show that AntMan and ISS have complimentary strengths.
On the PTB dataset, with a 10x compute reduction, AntMan does not generalize well without a teacher, while ISS incurs less than 1pt loss in perplexity compared to the original model.
This is demonstrated by the first row and column in Table 3 , and the third row in TAB2 .
On the contrary, for the BiDAF, AntMan incurs less than 1pt reduction in F1 score for nearly 10x compute reduction 2 , while ISS incurs nearly 2pt reduction in F1 score with less than 5x compute reduction on average.
This is shown in TAB9 .AntMan
can successfully compress BiDAF, while ISS fails because ISS compresses an LSTM by effectively reducing its hidden dimension, while AntMan preserves the hidden dimension size. The LSTMs
in the BiDAF model have large input dimensions making them computationally expensive, but they have very small hidden dimensions. Therefore
, reducing the already small hidden dimension results in significant loss of accuracy. On the contrary
, the PTB model has large input as well as hidden dimensions, allowing ISS to work effectively.

assistant: text: Reducing computation and memory complexity of RNN models by up to 100x using sparse low-rank compression modules, trained via knowledge distillation.