applied sciences  
LWMD: A Comprehensive Compression Platform for End-to-End Automatic Speech Recognition Models  
[PERSON] 1,2, [PERSON] 12, [PERSON] 1,2,* and [PERSON] 1,2,3  
Received: 12 December 2022  
Received: 29 December 2022  
Received: 3 January 2023  
Published: 26 January 2023  
Received: 12 December 2022  
Received: 29 December 2022  
Received: 3 January 2023  
Received: 29  
output sequence with the CTC criterion. And the AED framework consists of an acoustic encoder and an attention-based decoder that predicts the output sequence in an auto-regressive manner. Based on CTC and AED frameworks, conventional encoders/decoders are usually built by stacking identical blocks, and the multi-headed attention (MHA) based Transformer is the most popular block architecture in existing works [9], which can capture global contextual information effectively with the MHA mechanism. Recently convolution neural network (CNN) also gains increasing attention in building encoders for its advantage in extracting local acoustic information [10]. Even though existing works have achieved promising performance for E2E ASR tasks, most of them tend to adopt large models and ignore resource consumption, which leads to expensive memory and computation costs for real-world applications. There have existed several works that can reduce the resource consumption of neural networks. Refs. [11; 12; 13] design light-weight architectures on top of Transformer for fewer parameters. Refs. [14; 15; 16] propose to reduce model parameter numbers with structural pruning based on fixed architectures. Considering the tensor operation is the basic computation in neural networks, several works employ the low-rank decomposition of tensors to reduce the parameter complexity of neural networks [17; 18]. And the data low-bit quantization also can reduce model size and computation costs by relieving resource consumption of each parameter, which is widely employ in model compression [19; 20; 21; 22]. Even though these works can indeed reduce model parameter numbers or FLOPs by a certain amount, a non-negligible problem is that most of existing light-weight architectures and structural pruning methods can hardly maintain the same performance with the baseline with a significant parameter reduction. In order to reduce resource consumption and obtain more light-weight E2E ASR models while maintaining excellent performance, we propose the light-weight model designing (LWMD) platform. LWMD consists of a light-weight architecture searching (LWAS) framework and a differentiable structural pruning (DSP) algorithm. The LWAS framework employs the neural architecture search (NAS) technique to search architectures with small parameter and FLOPs costs, and the DSP framework can further prune searched architectures for fewer resource consumptions. On the one hand, the LWAS framework designs a topology-fused search space to cover many existing human-designed architectures that proved to be effective and develop a resource-aware differentiable search algorithm to select light-weight architectures from the search space based on popular differentiable search algorithms [23; 24]. On the other hand, the DSP algorithm proposes a differentiable pruning criterion instead of conventional magnitude-aware criteria to build a stronger correlation between the pruning criterion and the model performance, and an attention-similarity loss function is specially developed for the MHA module of E2E ASR models to get better pruning performance. Moreover, since LWMD only makes modifications to the model architecture, it is orthogonal to some other compression techniques (e.g. quantization) and can be easily combined with them. To summarize, the contributions of this paper can be described as follows:  
* LWMD platform. In this paper, we propose the LWMD platform to design light-weight E2E ASR models, which consists of one LWAS framework to search light-weight architectures and one DSP algorithm to prune further searched architectures. * The LWAS framework.