# A Lightweight Fault Diagnosis with Domain Adaptation for Defected Bearings
## 2 Methodology
### Knowledge Distillation  
Knowledge distillation is a powerful technique for model compression, where a smaller, more efficient student model is trained to replicate the behavior of a larger, more complex teacher model. In the context of bearing fault diagnosis, the teacher model is the original MobileNet backbone, which has been trained to classify time-frequency images of vibration signals into different fault categories. The student model, on the other hand, is a compressed version of MobileNet with fewer parameters, designed to achieve similar performance while being more computationally efficient. The knowledge distillation process involves two main components: the classification loss and the distillation loss. The classification loss ensures that the student model learns to correctly classify the input data, while the distillation loss encourages the student model to mimic the soft predictions of the teacher model. The soft predictions, or soft targets, are the output probabilities produced by the teacher model before applying the final softmax function. These soft targets contain valuable information about the relationships between different classes, which can help the student model generalize better. The overall loss function for training the student model is a weighted combination of the classification loss and the distillation loss:  
\[\mathcal{L}_{\text{total}}=\alpha\cdot\mathcal{L}_{\text{classification}}+(1- \alpha)\cdot\mathcal{L}_{\text{distillation}} \tag{2}\]  
where \(\alpha\in[0,1]\) is a trade-off hyperparameter that balances the contributions of the two components. When \(\alpha\to 1\), the total loss prioritizes supervision from the ground-truth labels, resembling standard supervised training. Conversely, \(\alpha\to 0\) emphasizes knowledge transferred from the teacher model via distillation. The classification term \(\mathcal{L}_{\text{classification}}\) is defined as the standard cross-entropy between the teacher's predictions and the ground-truth labels:  
\[\mathcal{L}_{\text{classification}}=-\sum_{i=1}^{N}y_{i}\log(p_{i}) \tag{3}\]  
where \(y_{i}\) is the true label for the \(i\)-th sample, and \(p_{i}\) is the predicted probability for the \(i\)-th sample. The distillation loss \(\mathcal{L}_{\text{distillation}}\) is the cross-entropy loss between the student model's predictions and the teacher model's soft targets:  
\[\mathcal{L}_{\text{distillation}}=-\sum_{i=1}^{N}q_{i}\log(p_{i}) \tag{4}\]  
where \(q_{i}\) is the soft target produced by the teacher model for the \(i\)-th sample. The soft targets are obtained by applying a temperature parameter \(T\) to the logits (pre-softmax outputs) of the teacher model:  
\[q_{i}=\frac{\exp(z_{i}/T)}{\sum_{j=1}^{C}\exp(z_{j}/T)} \tag{5}\]  
where \(z_{i}\) is the logit for the \(i\)-th class, and \(C\) is the total number of classes. The temperature parameter \(T\) controls the smoothness of the soft targets; higher values of \(T\) produce softer probability distributions, which can help the student model learn more nuanced relationships between classes. During training, the student model is optimized to minimize the total loss \(\mathcal{L}_{\text{total}}\). By incorporating both the classification loss and the distillation loss, the student model not only learns to correctly classify the input data but also benefits from the rich, soft targets provided by the teacher model. This dual-objective training process enables the student model to achieve comparable performance to the teacher model while being significantly more efficient in terms of computational complexity and memory usage. The teacher model is the aforementioned MobileNetV3 network, while the student model shares a similar architecture but with reduced complexity. Specifically, the student model reduces the number of channels in the bottleneck layers to [8; 12; 20; 40; 56; 80] and the number of neurons in the fully connected layers to 512 and 64, respectively.