# Towards a Reliable Evaluation of Local Interpretation Methods
## 3 Human-Friendly Evaluation
### Overview of Accuracy Evaluation  
The purpose of Accuracy Evaluation (AE) is to quantitatively evaluate different interpretation methods through the ground truth of features labeled by users. In order to realize this idea, we designed a web-based feature annotation tool to interactively acquire key features of different categories of images. Figure 4 shows the whole process of AE, which consists of four parts: evaluation data, feature segmentation, interactive annotation tool design and quantitative evaluation. The following details our interactive feature annotation core module(Figure 4c). Figure 3: Schematic diagram of the hierarchical structure of the evaluation dataset. The superscript number indicates the number of each type of image. Figure 5 shows our annotation interface and the process of generating ground truth maps. We created ground truth map through a two-step procedure. First, the feature block for every image selected by each participant is recorded (Figure 5A). In order to facilitate the user to select features, inspired by [44], we used the SLIC algorithm [45] to perform superpixel segmentation preprocessing operations on the evaluation dataset. Second, ground truth map is generated by integrating the feature blocks selected by the participants (Figure 5B). (Note that in order to make the ground truth map more accurate while taking into account operability, we recruited a total of 40 high-quality participants). In order to ensure the quality of the feature block selected by the user, we have carried out a series of user instructions before the user operation. For example, we informed the user to select the most important feature blocks as much as possible, at least 4 feature blocks, and no more than 8 for each image (See Appendix B for the detailed specific user instructions). At the same time, our annotation tool can also automatically monitor whether the number of feature blocks marked by the user meets 4-8. and the next image can be labeled only when the number of selected feature blocks is satisfied. Let \(U_{1}(I),U_{2}(I),...,U_{m}(I)\) denotes the set of feature blocks annotated by \(user_{1},user_{2},...,user_{m}\), respectively. Here, each \(U(I)\subset Seg(I)\), which contains a total of 100 feature blocks in image \(I\). \(m\) represents the number of users. Every \(U_{i}(I)\) is a binary mask image. When the user selects a pixel block, all the pixel values of the pixel block are 1, otherwise the pixel value of the pixel block is 0. Finally, we count the results of all user annotations, in other words, we sum the pixel values of the corresponding positions of all user annotation results of the same image. Therefore, the maximum value of pixel value is 40 (this means that the pixel blocks  
Figure 4: Accuracy Evaluation (AE) pipeline of interpretation methods. There are four steps in total: dataset (DataS), pixel segmentation, pixel feature selection and evaluation process. Figure 5: Overview of feature annotation interface. (**A**) is our main interface, which displays the segmented image and category information to be labeled, and also informs the user of the number of remaining labeled images. (**B**) means that we obtain the final ground truth by counting the feature blocks marked by different users. here are marked by all users) and the minimum value is 0. The final ground truth (GT) is normalized to 0-1. The formalization is as follows. \[GT=\frac{1}{m}\sum_{j=1}^{m}U_{j}(I) \tag{1}\]  
According to the obtained ground truth map of the category feature block, we quantitatively calculate the correctness of different interpretation algorithms. The formal definition of the Mean Absolute Error (MAE) is as follows. \[MAE=\frac{1}{c}\frac{\sum_{k=1}^{n}\sum_{(i,j)\in I}|E(\hat{y})_{k,(i,j)}-GT_{k,(i,j)}|} \tag{2}\]  
where \(n\) is the number of samples (\(n\) is equal to 30 in this experiment), \(k\) denotes a specific image among the 30 images, and the indices \((i,j)\) run over all the pixels of the figure and the subindex \((i,j)\) denotes the importance value of that pixel.