# No-Reference Image Quality Assessment Based on a Multitask Image Restoration Network
## 3 Proposed Method
### Network Training  
The training stage is comprised of two parts: pre-training of the image restoration sub-network and overall model training. First, we pre-train the multitask image restoration sub-network with the auxiliary task of generating an SSIM map using the Waterloo Exploration Database [44]. We randomly crop 224 \(\times\) 224 sized image patches from the distorted images to expand the training data, set the learning rate to 0.001 and batch size to 64, and train  
Figure 5: The structure of the multi-scale feature fusion module.  
for 100 epochs using the Adam [45] optimizer. The loss function is the \(L_{1}\) loss between the generated SSIM map and the ground truth SSIM map, as shown in Equation (4):  
\[l_{s}=\frac{1}{N}{\sum_{i=1}^{N}}{\left\|{{I_{SSIM}^{(i)}}-{\hat{I}_{SSIM}^{(i)}} }\right\|_{l_{1}}}, \tag{4}\]  
where \(N\) denotes the number of training image patches, \(I_{SSIM}^{(i)}\) denotes the SSIM map between the \(i\)th image patch and its corresponding reference image patch, \({\hat{I}_{SSIM}^{(i)}}\) denotes the predicted SSIM map of the \(i\)th image patch by the model, and \(l_{1}\) denotes the \(l_{1}\)-norm.  
After the training of the auxiliary task, the image restoration sub-network has well learned the structural information in the reference images. Based on this, the main task of generating pseudo-reference images can be trained. Training is conducted by randomly cropping 224 \(\times\) 224 sized image patches from distorted images in the Waterloo Exploration Database. The learning rate is set to 0.0001, the batch size is set to 64, and the Adam optimizer is used to optimize the network for 50 epochs. The loss function is the \(L_{1}\) loss between the pseudo-reference and the reference images, as shown in Equation (5):  
\[l_{r}=\frac{1}{N}{\sum_{i=1}^{N}}{\left\|{{I_{r}^{(i)}}-{\hat{I}_{r}^{(i)}}} \right\|_{l_{1}}}, \tag{5}\]  
where \(N\) denotes the number of training image patches, \(I_{r}^{(i)}\) denotes the \(i\)th reference image patch, \({\hat{I}_{r}^{(i)}}\) denotes the \(i\)th pseudo-reference image patch generated by the network, and \(l_{1}\) denotes the \(l_{1}\)-norm.  
After training the image restoration sub-network, the entire model needs to be trained on the target database. During the first 10 epochs of training, the parameters of the image restoration sub-network are frozen, and only the score prediction sub-network is trained. Then, the entire network is finely trained for another 40 epochs. To perform data augmentation, following the strategy from [17; 46], the images are randomly horizontally flipped during training, and 5 randomly sampled 224 \(\times\) 224 image patches are extracted from each image to increase the number of training samples. The quality scores of the image patches are the same as those of the corresponding distorted images. During testing, 5 randomly sampled 224 \(\times\) 224 image patches are also extracted from each testing image, and their quality scores are predicted. The mean of these scores is used as the quality score of the testing image.  
The \(L_{1}\) loss is used to train the entire model, as shown in Equation (6):  
\[l=\frac{1}{N}{\sum_{i=1}^{N}}{\left\|{{q_{i}}-{\hat{q}_{i}}}\right\|_{l_{1}}}, \tag{6}\]  
where \(N\) denotes the number of the image patches, \(q_{i}\) denotes the score of \(i\)th image patch predicted by the model, and \({\hat{q}_{i}}\) denotes the ground truth score of \(i\)th image patch.  
The Adam optimizer is used to optimize the parameters with a weight decay rate of \(5\times 10^{-4}\). The model is trained for 50 epochs with a batch size of 48 and an initial learning rate of \(5\times 10^{-5}\). The learning rate is multiplied by 0.9 every 10 epochs during training. MT-IRN is implemented with Pytorch, and the experiments are conducted on NVIDIA 3080 Ti GPU.