# No-Reference Image Quality Assessment Based on a Multitask Image Restoration Network
## 3 Proposed Method
### Score Prediction Sub-Network  
A score prediction sub-network is used to extract features from distorted and pseudo-reference images and to predict image quality scores, as shown in Figure 4. The score prediction sub-network takes distorted images, pseudo-reference images, and quality restoration features F1, F2, F3 as inputs, and a pre-trained ResNet-50 [40] on ImageNet [41] is used as a feature extractor to extract the content features of the image. First, the outputs of Conv2_10, Conv3_12, and Conv4_18 in ResNet-50 are used as multi-scale features of \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Module** & **Layer** & **Input Size** & **Output Size** \\ \hline \multirow{6}{*}{Encoder} & E1 & Conv3 \(\times\) 3, s1 & W \(\times\) H \(\times\) 3 & W \(\times\) H \(\times\) 16 \\ \cline{2-5}  & E2 & Conv3 \(\times\) 3, s1 & \multirow{2}{*}{W \(\times\) H \(\times\) 16} & W \(\times\) H \(\times\) 16 \\ \cline{2-5}  & E3 & Conv3 \(\times\) 3, s2 & \multirow{2}{*}{W \(\times\) H \(\times\) 16} & W/2 \(\times\) H/2 \(\times\) 32 \\ \cline{2-5}  & E4 & Conv3 \(\times\) 3, s1 & \multirow{2}{*}{W/2 \(\times\) H/2 \(\times\) 32} & W/4 \(\times\) H/4 \(\times\) 64 \\ \cline{2-5}  & E5 & Conv3 \(\times\) 3, s2 & \multirow{2}{*}{W/4 \(\times\) H/4 \(\times\) 64} & W/8 \(\times\) H/8 \(\times\) 128 \\ \cline{2-5}  & E6 & Conv3 \(\times\) 3, s1 & \multirow{2}{*}{W/8 \(\times\) H/8 \(\times\) 128} & W/16 \(\times\) H/16 \(\times\) 256 \\ \hline \multirow{6}{*}{Decoder} & D1 & Deconv3 \(\times\) 3, s2 & W/16 \(\times\) H/16 \(\times\) 256 & W/8 \(\times\) H/8 \(\times\) 128 \\ \cline{2-5}  & D2 & Deconv3 \(\times\) 3, s2 & W/8 \(\times\) H/8 \(\times\) 256 & W/4 \(\times\) H/4 \(\times\) 64 \\ \cline{2-5}  & D3 & Deconv3 \(\times\) 3, s2 & W/4 \(\times\) H/4 \(\times\) 128 & W/2 \(\times\) H/2 \(\times\) 32 \\ \cline{2-5}  & D4 & Deconv3 \(\times\) 3, s2 & W/2 \(\times\) H/2 \(\times\) 64 & W \(\times\) H \(\times\) 16 \\ \cline{2-5}  & D5 & Deconv3 \(\times\) 3, s1 & W \(\times\) H \(\times\) 16 & W \(\times\) H \(\times\) 3 \\ \cline{2-5}  & D6 & Deconv3 \(\times\) 3, s1 & W \(\times\) H \(\times\) 16 & W \(\times\) H \(\times\) 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The structural parameters of the image restoration sub-network, where W \(\times\) H \(\times\) C denotes the width, height, and channels of the feature map, respectively. Figure 3: The structures of the residual blocks. the distorted image, and are, respectively, fully fused with the quality restoration features F1, F2, F3 extracted by the image restoration sub-network through the multi-scale features fusion block. Then, the high-level features of the distorted image, namely the output of Conv5_9 in ResNet-50, are subtracted from the high-level features of the pseudo-reference image. After dimension reduction by a \(1\times 1\) convolution, the difference feature between the distorted and pseudo-reference images is obtained. Finally, the high-level feature, multi-scale feature, and difference feature are globally average-pooled, concatenated, and mapped to quality scores by fully connected layers. A multi-scale features fusion module is used to effectively fuse the multi-scale content features and the quality restoration features, as shown in Figure 5. Considering that these two features come from different network structures, there may be different feature scales. Therefore, multi-scale convolutions are applied to each feature separately to extract features at various scales, as shown in Equation (1). The multi-scale convolution uses two concatenated 3 \(\times\) 3 convolutions to achieve a receptive field size of 5 \(\times\) 5 while reducing the number of parameters.