## 4 Experiments
### Ablation Study
#### 4.3.2 The DSP Algorithm  
To explore the effectiveness of the DSP algorithm, we conducted more ablation studies as shown in Table 4. Besides the result without any pruning, we further provide experiment results with three different pruning algorithms. Random pruning refers to pruning parameters randomly, structured \(l_{1}\)-pruning and structured \(l_{2}\)-pruning refer to pruning parameters according to the \(l_{1}\)-norm or \(l_{2}\)-norm magnitude. Among these pruning methods, the worst  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Aishell-1**} & \multicolumn{4}{c}{**HKUST**} \\ \cline{2-9}  & **\#Params (M)** & **FLOPs (G)** & **Dev CER** & **Test CER** & **\#Params (M)** & **FLOPs (G)** & **Dev CER** & **Test CER** \\ \hline Transformer & 18.3 & 0.83 & 5.6 & 6.2 & 18.3 & 0.83 & 22.1 & 22.8 \\ Conformer & 20.9 & 0.96 & **5.4** & **5.9** & 20.9 & 0.96 & 21.6 & **22.2** \\ LiteTransformer & 14.1 & 0.60 & 5.9 & 6.5 & 14.1 & 0.60 & 22.8 & 23.3 \\ LightConv & 11.0 & 0.52 & 6.1 & 6.8 & 11.0 & 0.52 & 23.0 & 23.9 \\ Random Search & 12.4 & 0.56 & 5.7 & 6.4 & 11.9 & 0.54 & 22.5 & 23.2 \\ LWAS & 11.4 & 0.51 & **5.4** & **5.9** & 11.8 & 0.54 & **21.4** & **22.2** \\ LWMD (LWAS + DSP) & **8.5** & **0.39** & **5.4** & **5.9** & **9.4** & **0.44** & 21.5 & 22.4 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Experiment results under the CTC framework. FLOPs refer the averages of processing one-second speech. The best results on each framework are in boldface.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Aishell-1**} & \multicolumn{4}{c}{**HKUST**} \\ \cline{2-7}  & **\#Params (M)** & **FLOPs (G)** & **Dev CER** & **Test CER** & **\#Params (M)** & **FLOPs (G)** & **Dev CER** & **Test CER** \\ \hline Transformer & 29.9 & 6.96 & 5.0 & 5.4 & 29.9 & 7.14 & 20.3 & 20.8 \\ Conformer & 32.5 & 7.09 & **4.7** & **5.2** & 32.5 & 7.27 & **19.8** & **20.4** \\ LiteTransformer & 25.0 & 5.72 & 5.2 & 5.7 & 25.0 & 5.87 & 20.7 & 21.3 \\ LightConv & 20.8 & 5.61 & 5.4 & 6.0 & 20.8 & 5.76 & 21.0 & 21.9 \\ Random Search & 19.6 & 3.75 & 5.1 & 5.6 & 19.5 & 4.50 & 21.2 & 22.3 \\ LWAS & 17.7 & 2.81 & **4.7** & **5.2** & 18.1 & 2.90 & 20.0 & **20.4** \\ LWMD (LWAS + DSP) & **13.2** & **2.13** & **4.7** & **5.2** & **14.5** & **2.41** & 20.0 & 20.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experiment results under the AED framework. FLOPs refer the averages of processing one-second speech. The best results on each framework are in boldface.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Search Space** & **\#Params (M)** & **FLOPs (G)** & **Dev CER** & **Test CER** \\ \hline Conformer Baseline & 32.5 & 7.09 & 4.7 & 5.2 \\ Fix MHA as MHA4 & 18.2 & 2.94 & 5.1 & 5.6 \\ Fix CNN as CNN15 & 17.9 & 2.84 & 5.2 & 5.6 \\ Fix FFN as FFN2048 & 23.4 & 4.22 & 4.7 & 5.2 \\ Fix Block Number & 22.3 & 3.95 & 4.7 & 5.2 \\ Complete Search Space & 17.7 & 2.81 & 4.7 & 5.2 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation Studies of the search space on Aishell-1. The complete search space is compared with reduced search spaces.  
performance of random pruning indicates the necessity of a proper pruning algorithm. Even though \(l_{1}\)-pruning and structured \(l_{2}\)-pruning can achieve the same compression ratio with the DSP algorithm, the performance gap demonstrates that the DSP algorithm is a better choice. Further, we compare the results of DSP with/without the attention-similarity loss and observe that the attention-similarity Loss indeed can help to maintain the performance of pruned models. And in Figure 4, we can find the attention-similarity loss results in a more close attention weight with the weight before pruning.