# Filter Pruning via Attention Consistency on Feature Maps
## 3 Methods
### Implementation of ACDM  
Attention consistency can be regarded as a feature maps property measuring the distribution of attention on the feature maps of each layer. There exist several methods to measure the performance of attention consistency. For example, PCA and mean shift algorithms can be used to help measure attention consistency. Here the centroid-based implementation method is adopted. A centroid is a hypothetical point on which the weight of the body is concentrated. Weights are expressed as a pixel value on a feature map. According to the findings, most feature maps have larger activation values in the attention area, hence, for such feature maps, the centroids will fall in the attention area. The centroid coordinates of a feature map are:  
\[ \begin{split}& Ch_{i}^{j}(I)=\frac{\sum_{x=1}^{h_{i}}\sum_{y=1}^{w _{i}}x\cdot\mathbf{F}_{i}^{j}(I,x,y)}{\sum_{h=1}^{h_{i}}\sum_{w=1}^{w_{i}} \mathbf{F}_{i}^{j}(I,x,y)}\\ & Cw_{i}^{j}(I)=\frac{\sum_{x=1}^{h_{i}}\sum_{y=1}^{w_{i}}y\cdot \mathbf{F}_{i}^{j}(I,x,y)}{\sum_{h=1}^{h_{i}}(I)\sum_{w=1}^{w_{i}}\mathbf{F}_{ i}^{j}(I,x,y)}\end{split}  \tag{3}\]  
where \(x\) and \(y\) represent the coordinates on the \(h\)-axis and \(w\)-axis, respectively. \(\mathbf{F}_{i}^{j}(I,x,y)\) is the pixel value of \(\mathbf{F}_{i}^{j}(I)\). \(Ch_{i}^{j}\) and \(Cw_{i}^{j}\) represent \(h\)-axis and \(w\)-axis coordinates of the centroid on a feature map. Most of the feature maps have large activation values in the attention. In the \(i\)-th layer, the average centroid is:  
\[ \begin{split}& Ch_{i}(I)=\frac{\sum_{j=1}^{N_{i}+1}Ch_{i}^{j}(I)}{N _{i+1}}\\ & Cw_{i}(I)=\frac{\sum_{j=1}^{N_{i}+1}Cw_{i}^{j}(I)}{N_{i+1}} \end{split}  \tag{4}\]  
The distance of centroid deviation is also defined to evaluate the attention consistency of a feature map. In the \(i\)-th layer, the distance of centroid deviation is:  
\[ \begin{split}& dis_{i}^{j}(I)=\left(Ch_{i}^{j}(I)-Ch_{i}(I) \right)^{2}+\left(Cw_{i}^{j}(I)-Cw_{i}(I)\right)^{2}\end{split}  \tag{5}\]  
For a given input image, feature maps are generated in the forward propagation of every convolution layer. Figure 3 presents the centroid distribution of feature maps generated by the 6-th convolutional layer in Resnet-50. The x-axis and the y-axis represent centroid coordinates along the \(h\) and \(w\) directions of the feature maps. The blue dots represent the centroid of a feature map, and the red dot represents the average centroid. Several feature maps with different centroids at different distances are visualized. It is seen from these feature maps that the information is concentrated in the attention area in the feature maps with smaller deviations, whereas the information is concentrated in the non-attention area in feature maps with bigger deviations. It means that the feature maps with higher consistency have smaller deviations. Similar to such input images, the feature maps generated by the network with any other input images also have the same representation. The prerequisite for this metric to guide pruning is that the metric can reflect the centroid deviations of entire datasets. The experimental results also show that the centroid deviations generated by a single filter are robust to the input images. It is also seen from the experiments that the expectation of centroid deviations generated by a single filter is robust to the input images. As shown in Figure 4, for each subfigure, the x-axis represents the indices of feature maps and the y-axis is the number of training images. Different colors denote distances of centroid deviation. It is also seen that the average centroid deviation of multiple feature maps generated by the same filter (column of the figure) are almost equal (the same color), regardless of the number of images (ordinate of the subfigure) the CNNs receive. Therefore, even a small number of input images can be used to estimate the average centroid of deviation of entire datasets.