# Deep Activation Pooling for Blind Image Quality Assessment
## 2 Approach
### High-Contrast Patch Selection  
The local image information plays a profound role in the task of BIQA. Although the patches in an image could describe the local information, not all the patches are useful for the overall perception of image quality. Furthermore, the patch contrast reflects the image's structural information, which is sensitive to the image distortions and closely correlated with the image perceptual quality. When an image is subjected to quality degradations, the image structure is violated and the structural damage degree increases with the quality degradation level. Hence, we propose to select the high-contrast patches for the subsequent BIQA.  
To discover the high-contrast and meaningful patches, we utilize the CSM to build the relationship between the original image and the corresponding convolutional activations. Concretely, we first transmit the original image to the VGG-19 network and obtain the CSM in the 4 th convolutional layer, as shown in Figure 2b. From Figure 2b, we can see that those salient positions mainly distribute in the structural parts, and the values can reflect the importance of the local structure information. Hence,  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Block Number** & **Convolution Stride** & **Spatial Padding** & **Building Blocks** \\ \hline Block1 & 1 & 1 & \(\begin{bmatrix}3\times 3,64\\ 3\times 3,64\end{bmatrix}\) \\ \hline Block2 & 1 & 1 & \(\begin{bmatrix}3\times 3,128\\ 3\times 3,128\end{bmatrix}\) \\ \hline Block3 & 1 & 1 & \(\begin{bmatrix}3\times 3,256\\ 3\times 3,256\\ 3\times 3,256\end{bmatrix}\) \\ \hline Block4 & 1 & 1 & \(\begin{bmatrix}3\times 3,512\\ 3\times 3,512\\ 3\times 3,512\\ 3\times 3,512\end{bmatrix}\) \\ \hline Block5 & 1 & 1 & \(\begin{bmatrix}3\times 3,512\\ 3\times 3,512\\ 3\times 3,512\\ 3\times 3,512\end{bmatrix}\) \\ \hline  & \multicolumn{2}{c}{FC-4096} \\ \hline  & \multicolumn{2}{c}{FC-4096} \\ \hline  & \multicolumn{2}{c}{FC-1000} \\ \hline \hline \end{tabular}
\end{table}
Table 1: The architecture of VGG-19. The left part in “Building Blocks” indicates the size of receptive fields, and the right part indicates the number of filter banks.  
we propose the local weighted variance \(\sigma_{4}(i,j)\) to reflect the contrast of the activation response at position \((i,j)\) in the 4 th convolutional layer:  
\[\sigma_{4}(i,j)=\sqrt{\sum_{a=-A}^{A}\sum_{b=-B}^{B}w_{a,b}[C_{4}(i+a,j+b)-u_{4} (i,j)]^{2}}, \tag{2}\]  
where \(w_{a,b}\) (\(a=-A,\cdots,A,b=-B,\cdots,B\)) defines a unit-volume Gaussian weighting window, and \(C_{4}(i,j)\) denote the activation response of the CSM at position \((i,j)\) in the 4 th convolutional layer. Here, \(u_{4}(i,j)\) is the mean value of local activation responses:  
\[u_{4}(i,j)=\sum_{a=-A}^{A}\sum_{b=-B}^{B}w_{a,b}C_{4}(i+a,j+b). \tag{3}\]  
To obtain the contrast in the activation-response region \(reg\), we sum up all the local weighted variance within it:  
\[\sigma_{reg}=\sum_{(i,j)\in reg}\sigma_{4}(i,j). \tag{4}\]  
The higher the value of \(\sigma_{reg}\) is, the stronger the contrast of the activation-response region is. Hence, we take the value of \(\sigma_{reg}\) as the criterion of high-contrast patch selection. The procedure of high-contrast patch selection is shown in Figure 2. Firstly, based on the \(\sigma_{reg}\), the high-contrast activation-response regions (the green rectangles in Figure 2b) in CSM are selected. Then, the selected high-contrast activation-response regions are mapped to the corresponding input image patches (the yellow rectangles in Figure 2a) so as to obtain the high-contrast patches. As a result, the useful patches are retained and the meaningless ones are abandoned.