Towards efficient deep learning
Creators: name: Yin, Miao
Publisher: name: No Publisher Supplied
Year: 2023

description: Deep neural networks (DNNs) have gained significant traction in artificial intelligence (AI), delivering remarkable performance across diverse tasks such as image classification, object detection, natural language processing, video recognition, and speech recognition. However, the growing size of existing DNN models poses a challenge when deploying them on edge devices. State-of-the-art DNN models are often too large for embedded and Internet-of-Things (IoT) systems, resulting in storage and computational demands that hinder practical implementation. This discrepancy between model size and available computational resources impedes the widespread applications of AI in real-world scenarios. To bridge this gap, tensor decomposition-based approaches have emerged as a promising direction for enhancing the efficiency of large DNN models, providing substantial compression ratios. Nonetheless, existing tensor decomposition methods perform poorly when compressing convolutional neural networks (CNNs). To address this challenge, this thesis presents a systematic tensor decomposition-based model compression framework that achieves a high compression ratio while maintaining high task accuracy. Firstly, we formulate model compression as an optimization problem with a tensor rank constraint and propose the utilization of the alternating direction method of multipliers (ADMM) algorithm to solve it. Secondly, to enable automatic model compression and adaptive tensor rank selection, we relax the stringent tensor rank constraint using the tensor nuclear norm, enabling the learning of appropriate tensor ranks from data during the compression process. Thirdly, we develop a novel tensor-format convolutional computation scheme comprising contract-in, core-convolution, and contract-out stages, balancing parameter reduction and computational cost reduction. Together, these components constitute a comprehensive model compression methodology based on higher-order tensor decomposition, significantly enhancing the efficiency of existing DNN models and addressing multiple challenges associated with tensor methods. In addition to the general tensor decomposition-based model compression framework, we develop compact Recurrent Neural Networks (RNNs) based on a fully decomposed hierarchical tucker (FDHT) structure for video recognition applications. This decomposition approach enables comprehensive compression for RNN models while maintaining high accuracy. The newly developed FDHT-based Long Short-Term Memory (LSTM) achieves high accuracy with only a few thousand parameters, marking a significant reduction and accuracy improvement over existing compressed models. Furthermore, to practically improve the efficiency of the recent cutting-edge DNN model, Vision Transformer (ViT), which is dominant in computer vision tasks, we propose a unified graph and optimization-based structured pruning approach via a graph-based ranking system that measures the importance of attention heads, applying heterogeneous structured sparsity patterns to the models. It achieves significant parameter reduction without accuracy loss. Overall, this thesis brings AI closer to real-world applications, facilitating the widespread adoption and democratization of AI technologies.