# Deep Activation Pooling for Blind Image Quality Assessment
## 2 Approach
### Deep Activation Pooling  
Suppose that \(F\) is a set of \(D\)-dimensional local features extracted from the \(K\)-selected high-contrast patches of pristine images, that is, \(F=(f_{1},f_{2},\cdots,f_{k},\cdots,f_{K})\in R^{D\times K}\), where \(f_{k}\) is the \(D\)-dimensional feature vector of the \(k\)-th high-contrast patch. Based on these features, we learn a powerful opinion-unaware BIQA model. As shown in Figure 1d, in the training stage, we learn a PMVG model from the selected high-contrast patches. In the test stage, each selected high-contrast patch is fitted by a TMVG model, and then the TMVG model is compared with the PMVG model to obtain the local quality score of the patch. Finally, we utilize the proposed DAP to aggregate all the local quality scores to obtain the overall quality score. The specific procedure is listed as follows:  
Figure 2: **(a)** The input image; and (b) the CSM in 4 th convolutional layer. In order to intuitively view the corresponding relationship between the input image and the CSM, we resize the CSM to the size of the input image. (a) We utilize a set of high-quality pristine images to train a PMVG model [19]. We first choose the CSM in the 4 th convolutional layer of the VGG-19 network to select \(K\) high-contrast patches from the 90 pristine images using the high-contrast selection method mentioned in Section 2.2. Then, we extract the features for each selected high-contrast patch. Finally, based on the feature set \(F\) of the high-contrast patches, we apply the standard maximum likelihood estimation to learn a PMVG distribution:  
\[l(f)=\frac{1}{(2\pi)^{\frac{D}{2}}|\Sigma|^{\frac{1}{2}}}exp(-\frac{1}{2}(f- \mu)^{T}\Sigma^{-1}(f-\mu)), \tag{5}\]  
where \(\mu\) and \(\Sigma\) represent the mean vector and the covariance matrix of \(F\), respectively. (b) Given a test image, we first resize the test image to the size of the pristine images, and then the high-contrast patch selection strategy is performed to obtain a set of patches. Each patch is represented as a feature vector, and each feature vector is fitted by a TMVG model. The learned TMVG model for the \(p\)-th patch \((p=1,2,\cdots,P\), where \(P\) is the number of the selected high-contrast patches of a test image) is denoted as \((\mu_{p},\Sigma_{p})\). We compare the TMVG model of the \(p\)-th patch with the PMVG model to obtain the local quality score \(s_{p}\) for the \(p\)-th patch. The procedure of learning the TMVG model for each patch is extremely costly. For simplicity, the quality score \(s_{p}\) is computed using the following formula:  
\[s_{p}=\sqrt{(\mu-g_{p})^{T}(\frac{\Sigma+\Sigma^{\prime}}{2})^{-1}(\mu-g_{p})}, \tag{6}\]  
where \(g_{p}\) is the feature vector of the \(p\)-th patch and \(\Sigma^{\prime}\) is the covariance matrix computed from all the selected patches in the test image. Note that all the patches for one test image share the same \(\Sigma^{\prime}\). (c) As we know, the contribution of each local quality score to the overall quality score is different. To emphasize the important patch scores and suppress the inessential ones, we propose the novel DAP method to obtain the overall quality score of the test image. Each activation response in the CSM describes a local part of the input image, and the high responses indicate the salient parts. We use the CSM in the 7 th convolutional layer of the VGG-19 network to learn a weight for each local quality score. The weight of each local quality score is computed by adding all activation responses in the relevant activation-response region, as shown in the green rectangle of Figure 3. \(w_{p}\) is the weight of the \(p\)-th patch and it is formulated as:  
\[w_{p}=\sum_{(i,j)\in reg^{\prime}}C_{\gamma}(i,j), \tag{7}\]  
where \(reg^{\prime}\) represents the activation-response region and \(C_{\gamma}(i,j)\) is the activation response of the CSM at position \((i,j)\) in the 7 th convolutional layer.