# Environment-Aware Knowledge Distillation for Improved Resource-Constrained Edge Speech Recognition
## 6 Conclusions  
In this paper, environmental awareness is proposed as a method of improving the robustness of compressed universal representations to be used for speech recognition. In particular, we propose three innovations on top of the existing DistilHuBERT distillation recipe: (1) optimize the prediction heads, (2) employ a targeted data augmentation method for different environmental scenarios, and (3) employ a real-time SNR or RT60 estimator to choose the best compressed model for inference. We perform extensive experiments and compare against six benchmark models. When evaluated under noisy conditions (SNR between 0 and 30 dB), the proposed models outperform the benchmarks of comparable size (i.e., DistilHuBERT and RobustDistiller) by as much as 33.04%. The gains are more substantial in very noisy conditions (SNR between 0 and 10 dB), where gains of up to 48.42% \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Clean**} & \multicolumn{3}{c}{**Reverberation Time (WER)**} \\ \cline{3-5}  & & **All** & **High** & **Low** \\ \hline HuBERT Large [3] & 3.62 & 79.99 & 239.42 & 22.46 \\ HuBERT Base [3] & 6.43 & 77.15 & 145.59 & 36.02 \\ Robust HuBERT [15] & 6.75 & 58.36 & 89.13 & 30.29 \\ DistilHuBERT [10] & 13.29 & 156.98 & 648.02 & 74.04 \\ DistilRobustHuBERT & 12.70 & 77.14 & 97.48 & 59.61 \\ RobustDistiller [17] & 14.03 & 67.54 & 90.86 & 43.61 \\ \hline Robust DistilHuBERT & 13.78 & 52.56 & 72.02 & 33.12 \\ Reverb-Aware DistilHuBERT & 13.21 & 51.22 & 70.03 & 32.17 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance comparison across different clean and reverberant conditions with RT60 from 140 ms to 1 s.  
were seen as relative to DistilHuBERT. In turn, for high reverberation levels (RT60 greater than 500 ms), the proposed model was shown to outperform even the teacher models with a 2-6 times greater number of parameters (i.e., HuBERT base and large models) by as much as 89.19%. Lastly, we show the advantages of developing an environment-robust distillation process relative to just compressing robust latent representations from large speech models. Overall, these findings suggest the proposed method can be better suited for edge speech applications under varying environmental conditions. Conceptualization, A.P. and T.H.F. ; methodology, all authors; software, A.P. and H.R.G. ; validation, A.P. and H.R.G. ; formal analysis, A.P. ; resources, T.H.F. ; writing, all authors; supervision, A.A. and T.H.F. ; project administration, T.H.F. ; funding acquisition, T.H.F. All authors have read and agreed to the published version of the manuscript. This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), grant number RGPIN-2021-03246. Not applicable. The datasets used are publicly available. LibriSpeech available at [[https://www.openslr.org/12](https://www.openslr.org/12)]([https://www.openslr.org/12](https://www.openslr.org/12)), MUSAN available at [[https://www.openslr.org/17/](https://www.openslr.org/17/)]([https://www.openslr.org/17/](https://www.openslr.org/17/)), UrbanSound8K available at [[https://urbansounddataset.weebly.com/urbansound8k.html](https://urbansounddataset.weebly.com/urbansound8k.html)]([https://urbansounddataset.weebly.com/urbansound8k.html](https://urbansounddataset.weebly.com/urbansound8k.html)), BIRD available at [[https://github.com/[PERSON]/BIRD](https://github.com/[PERSON]/BIRD)]([https://github.com/[PERSON]/BIRD](https://github.com/[PERSON]/BIRD)), DCASE2020 available at [[https://dcase.community/challenge2020/index](https://dcase.community/challenge2020/index)]([https://dcase.community/challenge2020/index](https://dcase.community/challenge2020/index)) (accessed on 19 November 2023). The authors declare no conflicts of interest. The following abbreviations are used in this manuscript: