# Variable Scale Pruning for Transformer Model Compression in End-to-End Speech Recognition
## 5 Experiments and Results
### Pruning Settings  
The adaptive pruning technique was applied in a single step to the feedforward and to the attention layers after the model was trained. The encoder and decoder pruning rates  
Figure 10: Error rate as a function of global and local pruning rate for the VoxforgeIT system. Figure 9: Error rate as a function of global and local pruning rate for the Libri-trans system. are represented by the arithmetic sequences \(U\) and \(V\). To fix the initial pruning amounts \(U_{0}\) and \(V_{0}\) and the differences \(\alpha\) and \(\beta\), adaptive pruning was applied to the development data as follows:  
1. fix \(\alpha\) and \(\beta\);
2. for each pair (\(U_{0}\), \(V_{0}\))-- (a) use Formulas (5) and (6) to prune the feedforward layers; (b) evaluate the system using the pruned model and the development dataset; (c) note the error rate and the Sparsity. The algorithm was executed three times on each dataset, with different pruning rates applied to the attention layers. Specifically, for the Libri-trans dataset, the pruning rates used were 30%, 35%, and 40%. For the VoxforgeIT dataset, the pruning rates employed were 35%, 40%, and 45%. \(\alpha\) and \(\beta\) were chosen empirically. For example, for a pruning rate of 30% and \(\alpha\) = 0.01, the first encoder layer is pruned with the coefficient \(U_{0}\) = 0.3; the second is pruned with \(U_{1}\) = \(U_{0}-\alpha\) = 0.3 - 0.01 = 0.29. Thus, for a 12 th encoder block, the pruning rate of the last layer is \(U_{12}\) = 0.19 (i.e., a pruning rate of 19%). The results of the adaptive pruning technique applied to development data are reported in Appendix A. They are illustrated by Tables A1-A6. Among these results, the best tradeoffs between WER and Sparsity are identified and highlighted in bold. To determine the optimal tradeoff, one must select a specific WER value from the table and then choose the lowest associated sparsity level. Based on the obtained tradeoffs, we deduced the encoder parameters \(U_{0}\) and \(V_{0}\), which correspond to the optimal initial pruning amounts. For both the Libri-trans and VoxforgeIT settings, we extracted the optimal values of the (WER, Sparsity) pairs from Tables A1-A6. Subsequently, we organize and present these optimal values comparatively in Tables 2 and 3. This arrangement allows us to effectively demonstrate the relative trade-offs achieved for each set of parameters. Figures 11 and 12 present the optimal pairs (WER, Sparsity) for different pruning rates, along with the results for global and local pruning for the purpose of comparison. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**PR. hd** & \multicolumn{8}{c}{**Best Tradeoffs (WER, Sparsity)**} \\ \hline
30 (\%) & WER & 6.5 & 6.8 & 7.1 & 7.7 & 8.2 & 8.9 & 9.2 \\  & SP. & 31.62 & 37.87 & 44.12 & 46.62 & 49.12 & 51.62 & 52.87 \\ \hline
35 (\%) & WER & 6.6 & 7.0 & 7.3 & 7.6 & 8.2 & 9.0 & 9.6 \\  & SP. & 34.12 & 41.62 & 44.12 & 46.62 & 50.37 & 52.87 & 55.37 \\ \hline
40 (\%) & WER & 6.8 & 6.9 & 7.0 & 7.1 & 7.5 & 8.4 & 9.5 \\  & SP. & 35.37 & 37.87 & 40.37 & 41.62 & 45.37 & 51.12 & 55.37 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Adaptive pruning of the Libri-trans model: best tradeoffs between the WER and the model Sparsity (SP). The fixed parameters are the pruning rate of the multi-head attention layers (PR. hd) and \(\alpha=\beta=0.01\). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**PR. hd** & \multicolumn{8}{c}{**Best Tradeoffs (WER, Sparsity)**} \\ \hline
35 (\%) & WER & 10.5 & 10.6 & 10.7 & 10.9 & 11.5 & 12.3 \\  & SP. & 44.23 & 48.04 & 49.68 & 51.22 & 57.57 & 61.37 \\ \hline
40 (\%) & WER & 10.4 & 10.5 & 10.8 & 11.1 & 11.4 & 12.2 \\  & SP. & 42.88 & 46.06 & 50.50 & 55.58 & 58.76 & 62.56 \\ \hline
45 (\%) & WER & 10.5 & 10.8 & 11.1 & 11.6 & 12.2 & 12.6 \\  & SP. & 46.82 & 52.53 & 57.30 & 61.11 & 63.96 & 65.87 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Adaptive pruning of the VoxforgeIT model: Best tradeoffs between the WER and the model Sparsity (SP).