# Denoising Multiscale Back-Projection Feature Fusion for Underwater Image Enhancement
## 4 Multiple Degradation Knowledge Distillation  
In order to make the model suitable for multiple types of degradation images, we propose a novel multiple degradation knowledge distillation strategy to implement multi-task image enhancement. As illustrated in Figure 6, there are four teacher networks and one student network, where the features of the four teacher models are projected into the student network. The knowledge distillation process includes two stages.  
In the first stage, we train four different teacher networks on four kinds of degradation images separately, including underwater images, hazing images, rainy images, and snowy images. The four teacher networks all follow the structure of DeMBFF-Net. For the \(p\)-th teacher model \(T_{p}\), we utilize the \(L_{1}\) loss as the loss function, as shown in Equation (12).  
\[\mathcal{L}_{\mathcal{T}}=\frac{1}{wh}\|T_{p}(i,j)-G(i,j)\|, \tag{12}\]  
where \(T_{p}(i,j)\) represents the pixel \((i,j)\) in the restored image by the \(p\)-th teacher model, and \(G(i,j)\) denotes the same pixel in the clear image. \(w\) and \(h\) denote the width and height of the image, respectively.  
After training the four teacher models, the teachers' feature maps are distilled to the student network. Following existing work [35], we used the following loss function to make the student's \(n\)-th feature map have similar values to the teacher's \(m\)-th layer.  
\[\mathcal{L}_{\mathcal{K}}=\frac{1}{2}\|T_{p}^{m}(I)-g(F^{n}(I))\|^{2}, \tag{13}\]  
where \(I\) is the input image. Since we use the same structure for all the teacher and student models, we set \(m\) as being equal to \(n\) in the above equation. Each teacher transfers an encoder block and a decoder block to the student, resulting in the student network comprising four encoder blocks and four decoder blocks. The convolutional layer \(g\) ensures the feature map of the student network keeps the same size as that of the teacher network, which has a kernel size of \(3\times 3\) and a stride of \(1\). In the knowledge distillation setup, the layers in the teacher model are hint layers, and the student layers are guided layers in Equation (13).  
Figure 6: Details of the multiple degradation knowledge distillation strategy. The top figure illustrates the first stage of training four teacher models. The figure below is the second stage of student model training. The teacher models transfer knowledge to the guided layers of the student model by feature distillation.  
After training the guided layers as the initial parameters, we further trained the student network on the mixed dataset (mixed training dataset for four tasks). In the second stage, we designed the following loss function to leverage \(L_{1}\) loss and perception loss.  
\[\mathcal{L_{S}}=\sum_{i=1}^{h}\sum_{j=1}^{w}\lVert S(i,j)-G(i,j)\rVert+\lambda \sum_{i=1}^{h}\sum_{j=1}^{w}|\varphi_{1}S(i,j)-\varphi_{l}G(i,j)|, \tag{14}\]  
where \(\lambda\) is set at 0.5 to balance the influence of the two different losses. \(\varphi_{l}\) is the activation of the \(l\)-th layer for the VGG network trained on the ImageNet dataset.