# Towards a Reliable Evaluation of Local Interpretation Methods
## 3 Human-Friendly Evaluation
### Evaluation of Persusibility  
Figure 11 shows the statistical results of the number of users who could answer the object name based on the top 3% feature occlusion map based on different interpretation methods in Round 1. Figure 12 shows the correct number of users who gave the object name in all the different interpretation methods. We can see that the explanation given by the mask algorithm had many users who choose to answer the name of the object, but the correct rate of the answer was very low. For the interpretation result given by the Grad-cam algorithm, the correct rate of the user's answer was the highest. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multicolumn{3}{c}{**Grad-Cam**} & \multicolumn{3}{c}{**Mask**} & \multicolumn{3}{c}{**Rise**} \\ \hline  & P & N & P & N & P & N \\ \hline P & 1.54 & 1.6 & P & 1.76 & 1.4 & P & 1.5 & 1.69 \\ \hline N & 4 & 92.7 & N & 10.19 & 86.65 & N & 5.6 & 91.21 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Confusion matrix of different interpretation algorithms (P means Positive, N means Negative). Figure 11: For different interpretation algorithms under top-3% occlusion, the statistical results of object names that users can identify. Each interpretation method is tested with 10 images from DataS and 35 users. We further analyzed the user's satisfaction with the interpretation given by different interpretation algorithms in Round 2 and Round 3. Figure 13 shows the distribution of user satisfaction with different explanation methods after informing users of category information. Figure 14 shows the distribution of user satisfaction regarding whether to agree with category information after the feature interpretation areas was expanded from 3% to 8%. We can see that when the user satisfaction was 1, for the Mask algorithm, the number of users was the largest. However, when the user satisfaction is 3, 4, or 5, the Grad-cam algorithm has a larger number of users compared to the Mask and Rise algorithms. Figure 12: The statistical result of the correct object name given by the user. Each interpretation method is tested with 10 images from DataS and 35 users. Figure 13: The user satisfaction distribution of different interpretation algorithms under the top-3% occlusion. Each interpretation method is tested with 10 images from DataS and 35 users. Satisfaction score is 1–5 (1 means most dissatisfied, 5 means most satisfied). We used a Sankey Diagram to visually show how users' choices changed after expanding the range of features (Figures 15-17). The first column of the Sankey Diagram indicates the gender of the user, the second column indicates whether the user understands machine learning, the third column indicates the satisfaction distribution for the category in the first round (Note that the data of strongly agree and strongly disagree are not displayed here, because the game ended directly after selecting these two options), and the fourth column indicates the satisfaction score for the category in the second round. By comparing the Sankey Diagram of three interpretation methods, we can intuitively see that the Grad-cam interpretation method could best convince users, the Rise interpretation method was the second, and the Mask interpretation method had the worst effect. Take the disagree option in the third column as an example, for the Grad-cam interpretation method, the user chose disagree for the first time. After increasing the feature range, most users' choices flowed to the agree option. For the rise interpretation method, after expanding the range of features, a small number of users still chose not to agree. However, most users still disagreed with the explanation given by the Mask explanation method  
Figure 14: The user satisfaction distribution of different interpretation algorithms under the top-8% occlusion. Each interpretation method is tested with 10 images from DataS and 35 users. Satisfaction score is 1–5 (1 means most dissatisfied, 5 means most satisfied).