# A Combined Full-Reference Image Quality Assessment Method Based on Convolutional Activation Maps
## 2 Proposed Method
### Architecture  
In this subsection, the proposed deep FR-IQA framework, which aims to capture image features in various levels from a pretrained CNN, is introduced in details. Existing works extract features of one or two layers from a pretrained CNN in FR-IQA [29; 31]. However, many papers pointed out the advantages of considering the features of multiple layers in NR-IQA [43] and aesthetics quality assessment [50].  
We put the applied base CNN architectures into a unified framework by slicing a CNN into \(L\) parts by the convolutional layers, independent from the network architecture, e.g., AlexNet or VGG16. Without the loss of generality, the slicing of AlexNet [4] is shown in Figure 1. As one can see from Figure 1, at this point, the features are in the form of \(W\times H\times D\) tensors, where the depth \((D)\) is dependent on the applied base CNN architecture and the tensors's width \((W)\) and height \((H)\) depend on the input image size. In order to make the feature vectors' dimension independent from the input image pairs' resolution, we do the followings. First, convolutional feature tensors are extracted with the help of a pretrained CNN (Figure 1) from the reference image and from the corresponding distorted image. Second, reference and distorted activation maps at a given convolutional layer  
where \(ISM(\cdot)\) denotes a traditional image similarity metric (PSNR, SSIM [12], and HaarPSI [20] are considered in this study), \(\mathcal{F}_{i}^{refJ}\) and \(\mathcal{F}_{i}^{distJ}\) are the \(i\)th activation map from the \(l\)th reference and distorted feature tensors, \(\mathbf{f}^{l}\) is the feature vector extracted from the \(l\)th convolutional layer, and \(\mathbf{f}_{i}^{l}\) stands for its \(i\)th element. Figure 2 illustrates the compilation of layer-wise feature vectors.  
In contrast to other machine learning techniques, CNNs are often called black-box techniques due to millions of parameters and highly nonlinear internal representations of the data. Activation maps of an input image help us to understand which features that a CNN has learned. If we feed AlexNet reference and distorted image pairs and we visualize the activations of the _conv1_ layer, it can be seen that activations of the reference image and those of the distorted image differs significantly from each other mainly in quality aware details, such as edges and textures (see Figures 3 and 4 for illustration). This observation revealed to us that an effective feature vector can be compiled by comparing the activation maps.  
The whole feature vector, that characterizes a reference-distorted image pair, can be obtained by concatenating the layer-wise feature vectors. Formally, we can write  
\[\mathbf{F}=\mathbf{f}^{1}\oplus\mathbf{f}^{2}\oplus...\oplus\mathbf{f}^{L} \tag{2}\]  
where \(\mathbf{F}\) stands for the whole feature vector, \(\mathbf{f}^{i}(j=1,2,...,L)\) is the \(j\)th layer-wise feature vector, and \(L\) denotes the number of convolutional layers in the applied base CNN.  
Finally, a regression algorithm is applied to map the feature vectors onto perceptual quality scores. In this study, we made experiments with two different regression techniques, such as support vector regressor (SVR) [51] and Gaussian process regressor (GPR) [52]. Specifically, we applied SVR with linear kernel and radial basis function (RBF) kernel. GPR was applied with rational quadratic kernel.  
Figure 1: Feature extraction with the help of AlexNet [4].