# Robustness, Stability, and Fidelity of Explanations for a Deep Skin Cancer Classification Model
## 2 Material and Methods
### Methods
#### 2.2.3 Metrics  
There is no general consensus among scholars on how the quality and reliability of explanation techniques should be assessed [58]. Generally, one can distinguish between human-centred qualitative evaluations and more objective metrics [58]. In this paper, we provide qualitative visual explanations only for the \"most interesting cases\" [22], and focus the main validation assessment to the latter. More precisely, we use three objective quantitative evaluation metrics: robustness, stability, and fidelity.  
First, we measure the _robustness_ of the explanation techniques using the Lipschitz indicator proposed by [PERSON] and [PERSON] [59]. This Lipschitz indicator gives the persistence of an explanation method to withstand small perturbations of the input that do not change the prediction of the model. More precisely, [PERSON] and [PERSON] proposed to artificially perturb the features of each object \(x_{i}\in X\), so that \(\mathcal{N}_{e}(x_{i})=||x_{i}-x_{j}||\leq e\), and then computing the quantity  
\[L_{X}(x_{i})=\operatorname*{arg\,max}_{x_{j}\in\mathcal{N}_{e}(x_{i})\leq e} \frac{||f(x_{i})-f(x_{j})||}{||x_{i}-x_{j}||} \tag{4}\]  
to measure whether the explanation technique is robust in a Lipschitz sense. As pointed out in [59], there is no single ideal value for this robustness estimate, because it is highly dataset dependent. However, a smaller value corresponds to more robustness [59]. To measure the robustness of our explanation techniques, we compute the mean and standard deviation of the Lipschitz indicator for all naturally similar instances in the test set (i.e., those test set instances that belong to the same class), so that we do not have perturb any instances artificially.  
Second, we measure the _stability_ or _identity_[60] of the explanation technique by repeating the explanation generation for the same instance and model with the same configuration arguments. If the explanation technique results in different explanations, the technique is not stable. To measure the degree of stability, we simply compute the explanation for each instance in the test set twice with the same configurations and take the percentage of same explanations from all the explanations pairs in the test set. A higher percentage means more stability.  
Third, we measure the _fidelity_. The fidelity metric indicates how closely the surrogate model reflects the real model. By definition, the fidelity of an intrinsically explainable model-specific explanation is always 100%, as it harnesses the original model. However, for model-agnostic explanation techniques, which (such as LIME) are based on local surrogate models, the fidelity is an important objective quality metric. As pointed out by [PERSON] et al. [38], an explanation with low fidelity is essentially useless. Similarly as the stability, we report the local fidelity of the explanation technique as percentage for each observation in the test set. Meaning for each observation in the test set, we compute the prediction of the original model and the prediction of the (surrogate) explanation model and report the percentage of agreement.