MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression

Deep neural networks (DNNs) have become the state-of-the-art technique for machine learning tasks in various applications. However, due to their size and the computational complexity, large DNNs are not readily deployable on edge devices in real-time. To manage complexity and accelerate computation, network compression techniques based on pruning and quantization have been proposed and shown to be effective in reducing network size. However, such network compression can result in irregular matrix structures that are mismatched with modern hardware-accelerated platforms, such as graphics processing units (GPUs) designed to perform the DNN matrix multiplications in a structured (block-based) way. We propose MPDCompress, a DNN compression algorithm based on matrix permutation decomposition via random mask generation. In-training application of the masks molds the synaptic weight connection matrix to a sub-graph separation format. Aided by the random permutations, a hardware-desirable block matrix is generated, allowing for a more efficient implementation and compression of the network. To show versatility, we empirically verify MPDCompress on several network models, compression rates, and image datasets. On the LeNet 300-100 model (MNIST dataset), Deep MNIST, and CIFAR10, we achieve 10× network compression with less than 1% accuracy loss compared to non-compressed accuracy performance. On AlexNet for the full ImageNet ILSVRC-2012 dataset, we achieve 8× network compression with less than 1% accuracy loss, with top-5 and top-1 accuracies of 79.6% and 56.4%, respectively. Finally, we observe that the algorithm can offer inference speedups across various hardware platforms, with 4× faster operation achieved on several mobile GPUs.  Work in progress. ar X iv :1 80 5. 12 08 5v 1 [ cs .L G ] 3 0 M ay 2 01 8 Deep neural networks (DNNs) have become the leading technique used in machine learning tasks across multiple application domains in computer vision [11, 21, 30], ranging from object perception in autonomous vehicles [3, 25, 24] to medical imaging and diagnosis [5, 13, 16, 17, 6, 23]. However, due to their size and complexity, large DNNs can be quite computationally intensive, which can pose a significant challenge for real-time applications on edge devices [33, 11, 28]. Specifically, state-of-the art DNNs can contain hundreds of layers represented by matrices containing millions of elements, such that both the memory requirements and the computational complexity of matrix multiplication needed for network training and high-accuracy performance rapidly increases with DNN size [13, 16, 29, 10]. This challenge is particularly difficult for DNNs containing fullyconnected (FC) layers characterized by large matrix sizes [13, 16]. DNNs with large FC layers can be found in over 60% of production-side applications running in a cloud services platform, highlighting the practical importance of this challenge [22]. To create a more perfect union between network compression and the underlying compute platform, we propose a matrix permutation decomposition algorithm, termed MPDCompress, that both compresses DNNs and molds them to the underlying hardware-accelerated platform. We experimentally evaluate our MPDCompress algorithm on several DNN models, including LeNet 300-100 (MNIST dataset), Deep MNIST, CIFAR10, and AlexNet (ImageNet ILSVRC-2012 dataset) and achieve less than 1% accuracy loss compared to non-compressed accuracy performance. In the next section, we review related work on network compression and discuss the key challenges for irregular matrix structures that motivated this work. The following sections explain matrix permutation decomposition via random binary mask generation and application, introduce our algorithm architecture for both training and inference modes, and describe our experimental framework. We also present our experimental results on several different DNNs and datasets to confirm the versatility of the approach. Finally, we observe that MPDCompress can offer inference speedups across various hardware platforms, with 4× faster operation on several mobile GPUs .