# Towards a Reliable Evaluation of Local Interpretation Methods
## 3 Human-Friendly Evaluation
### Evaluation of Class-Discriminativeness  
In this section, we aimed to compare three different visual explanation methods in terms of their power of discrimination across evaluation dataset. Table 4 shows the discrimination of the three different methods in DataT. We can see that the Grad-cam and Rise method had a failure case respectively. For example, in case 5.1, the predicted result of the VGG-16 model was banana, and the visual explanation given by the Grad-cam method focused on the feature of pineapple. In case 4.1, the explanation given by the Rise method focused on both the mouse and keyboard features, when the model predicts that the input was a mouse. Except for the success of the Mask method in case 4.2 (keyboard prediction result), all other tests showed that it was not class discrimination. The corresponding case display above can be seen in Table 5. In comparison, we found that the Mask method failed to have class discrimination under the DataT dataset. Figure 16: Sankey Diagram results for Mask. The third column shows the distribution of top-3% of user satisfaction, and the fourth column shows the distribution of top-8% of user satisfaction. Mask method is tested with 10 images from DataS and 35 users. Figure 17: Sankey Diagram results for Rise. The third column shows the distribution of top-3% of user satisfaction, and the fourth column shows the distribution of top-8% of user satisfaction. Rise method is tested with 10 images from DataS and 35 users. To further examine the class discrimination of these explanation methods, we also compared the class discrimination of these methods under DataS dataset. The results are shown in Table 6. The point game value of the Grad-cam and Rise methods were both 100%, but Mask had eight failed cases in the 40 samples. As shown in the Figure 18, in the failure cases of pandas, goats, and school bus, the explanation given by the Mask method was spread over the entire image pixel, in which the foreground and background were identical importance. For the uniform failure, the Mask method explanation method focused on the soldier's jaw rather than on uniform features. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Interpretation Methods** & **Grad-Cam** & **Mask** & **Rise** \\ \hline Number of missed objects & 0 & 8 & 0 \\ \hline Point Game Value & 100\% & 80\% & 100\% \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Point game statistics of three explanation methods on DataS. Figure 18: Failure cases of Mask method in panda category, goat category, school bus category and uniform category. (Note that these examples are from DataS.) \begin{table}
\begin{tabular}{|c c c c c|} \hline \hline
**Sample** & **Case** & **Prediction Result** & **Grad-CAM** & **Mask** & **Rise** \\ \hline \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & 1.1 & Pineapple & ✓ & ✗ & ✓ \\  & 1.2 & Cat & ✓ & ✗ & ✓ \\ \hline \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & 2.1 & Knife & ✓ & ✗ & ✓ \\  & 2.2 & Broccoli & ✓ & ✗ & ✓ \\ \hline \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & 3.1 & Football & ✓ & ✗ & ✓ \\  & 3.2 & Helmet & ✓ & ✗ & ✓ \\ \hline \multirow{4}{*}{\
\begin{tabular}{} \end{tabular} } & 4.1 & Mouse & ✓ & ✗ & ✗ \\  & 4.2 & Keyboard & ✓ & ✓ & ✓ \\ \hline \multirow{4}{*}{
\begin{tabular}{} \end{tabular} } & 5.1 & Banana & ✗ & ✗ & ✓ \\  & 5.2 & Pineapple & ✓ & ✗ & ✓ \\ \hline \end{tabular}
\end{table} 
Table 4: Comparisons of class discrimination of three different explanation methods. The first column is the five test examples of the DataT. The second column is the prediction result of the pre-trained VGG-16 model. The last three columns are the class discrimination results of the explanation methods. (✓ mark indicates that the class-discriminativeness characteristic is satisfied). \begin{table}
\begin{tabular}{|c c c c|} \hline \hline Test Sample & & & \\ P = Banana & P = Keyboard & P = Mouse \\ \hline Interpretation Result & & & \\ Grad-cam & Mask & Rise \\ \hline \end{tabular}
\end{table}