# Variable Scale Pruning for Transformer Model Compression in End-to-End Speech Recognition
## 6 Conclusions and Future Work  
This paper presents a novel approach for pruning transformer models in end-to-end speech recognition, addressing the significant complexity and size of these models that hinder their efficient deployment in Automatic Speech Recognition tasks. While transformer systems exhibit excellent performance, their large size poses challenges for practical implementation. In order to overcome this limitation, our work focused on studying and optimizing pruning techniques specifically tailored to the transformer architecture.  
In this study, we devised a method that centers around analyzing the evolution of weights within the feedforward layers present in both the encoder and decoder blocks of the transformer. Through careful observation, we found that deeper layers tend to possess higher weights compared with earlier layers. Leveraging this insight, our proposed approach, named variable scale pruning, applies a pruning rate that decreases progressively as the layer depth increases.  
To validate the effectiveness of our method, we conducted experimental evaluations on diverse databases. The results showcased the superiority of our approach over traditional local and global pruning methods. Remarkably, our technique achieved an impressive model compression rate of up to 59.5%, all while maintaining a less than 10% relative reduction in accuracy.  
Looking ahead, there are exciting research directions to explore. One such avenue involves investigating the application of variable scale pruning on attention head layers, which play a vital role in the transformer's performance. Additionally, we intend to delve into fine-tuning techniques for the pruned models, aiming to further optimize their performance without sacrificing compression gains. In the present era, there is a growing trend in the adoption of large and pre-trained models, known as foundation models. Through fine-tuning, these models continually improve their efficiency and hold a crucial advantage in being able to work with minimal labeled data. The pruning of such models offers a compelling and promising avenue for further research.  
By advancing the field of transformer model pruning for end-to-end speech recognition, our work contributes to the broader goal of enabling the widespread adoption of efficient and high-performing transformer systems in real-world applications.  
Both authors made substantial contributions to the progress of this research. Specifically, [PERSON] took on a significant role in aspects related to Conceptualization, Investigation, Methodology, Software, and Writing. Meanwhile, [PERSON] made an important effort for Funding acquisition. Additionally, he undertook responsibilities related to Supervision and made substantial contributions to tasks involving Writing--review & editing. All authors have read and agreed to the published version of the manuscript.  
The research presented in this paper was conducted as part of the project FVLLMONTI, which has received funding from the European Union's Horizon 2020 Research and Innovation action under grant agreement No 101016776.  
This research utilizes the Libri-trans and VoxforgeIT datasets, both of which are openly available. The datasets can be accessed at the following URLs: [[https://zenodo.org/record/6482585.YsmV](https://zenodo.org/record/6482585.YsmV)]([https://zenodo.org/record/6482585.YsmV](https://zenodo.org/record/6482585.YsmV)) (accessed on 9 August 2023) and [[http://www.repository.voxforge1.org/downloads/forLibri-trans](http://www.repository.voxforge1.org/downloads/forLibri-trans)]([http://www.repository.voxforge1.org/downloads/forLibri-trans](http://www.repository.voxforge1.org/downloads/forLibri-trans))(accessed on 9 August 2023) and VoxforgeIT respectively.  
The authors declare no conflict of interest.