# Environment-Aware Knowledge Distillation for Improved Resource-Constrained Edge Speech Recognition
## 2 Background Material
### Distilled Versions of Hubert  
As mentioned above, one emerging topic in speech processing is that of edge ASR. For these applications, smaller models are needed; thus, knowledge distillation has been explored recently. The recent work in [10], for example, proposed the so-called _DistilHuBERT_ model, where hidden representations from a HuBERT model were directly distilled to reduce model size and inference time.  
More specifically, DistilHuBERT proposes a novel teacher-student framework for speech representation learning via multi-task knowledge distillation. The model consists of a CNN feature extractor and a small Transformer encoder. The idea is to learn to generate multiple teacher's hidden representations from shared representations. This is performed by predicting the teacher's hidden representations with separate prediction heads. This objective is a multi-task learning paradigm and encourages the Transformer encoder to produce compact representations for multiple prediction heads. DistilHuBERT then takes a frozen pre-trained HuBERT Base model and uses only three prediction heads to respectively predict the 4 th, 8 th, and 12 th HuBERT hidden layers' output. Before pre-training, the student is initialized with the teacher's parameters. Then, the student's prediction heads learn to generate the teacher's hidden representations by minimizing the loss function \(\mathcal{L}^{(l)}=\mathcal{L}^{(4)}+\mathcal{L}^{(8)}+\mathcal{L}^{(12)}\). After training, the heads are removed, as the multi-task learning paradigm forces the DistilHuBERT model to learn representations containing rich information. These layers, however, are chosen to maximize the performance across various speech-related tasks and may not be optimal for ASR. In our experiments, we explore how different prediction heads can improve the model performance.  
One other method proposed recently for distilled universal representations is the one described in [17], called _RobustDistiller_. This is a recipe which combines data augmentation and multi-task denoising learning, which are incorporated into the knowledge distillation process. In the data augmentation step, an online contamination of the data is performed during the distillation process. In particular, the student model receives the noisy data as input, but the network's target is to reconstruct the clean representations of the teacher model. Thus, in the multi-task denoising learning step, beyond learning to reconstruct the teacher's representations, an additional enhancement head is responsible for rebuilding the clean speech waveform from the learned representations. In contrast to the usual enhancement techniques, the objective is to enforce the upstream model to carry enough information about the speech itself and not the noise components. While RobustDistiller was shown to outperform DistilHuBERT on keyword spotting, intent classification, and emotion recognition tasks, with noise and/or reverberation, it still showed some sensitivity to different environmental conditions.  
Lastly, we explore if the distillation of a teacher model built to be robust also results in a robust compressed model. This is an important step to decide if robustness needs to be implemented at the teacher level or during the distillation process, as proposed herein. To this end, we apply the same distillation process used in DistilHuBERT, but applied it to the _Robust HuBERT_ teacher described in Section 2.3. Henceforth, we term this model _DistilRobustHuBERT_.