# A Comparative Study of Two Rule-Based Explanation Methods for Diabetic Retinopathy Risk Assessment
## 5 Experiments and Results
### Evaluation of the Counterfactual Examples  
It is well-known that counterfactual examples help to understand what changes are needed to obtain a different outcome. This is particularly interesting in health-care applications. Hence, it is important to have counterfactual examples that balance a wide range of suggested modifications (diversity) and the relative facility of adopting those modifications (proximity to the actual input). Moreover, counterfactual examples must be actionable, e.g., people can not reduce their age or change their race.  
In this subsection, we evaluate the generated counterfactual examples using the following evaluation metrics [22]:  
* **Validity**: is the number of counterfactual examples with a different outcome than the original input, i.e., \(x\), divided by the total number of counterfactual examples. \[\text{Validity}=\frac{|\hat{x}\in\mathbb{C}\;s.t.b(x)\
eq b(\hat{x})|}{|\mathbb{C}|}\] (1) Here \(\mathbb{C}\) refers to the set of returned counterfactual examples and \(b\) is the black-box model.
* **Proximity**: is the mean of feature-wise distances between a counterfactual example \(c\) and the original input \(x\). We considered two different proximity measures, the Euclidean distance for the continuous features and the Hamming distance for the categorical ones. \[\text{Continuous Proximity}=-\frac{1}{|\mathbb{C}|}\sum_{c\in\mathbb{C}}dist\_ cont(c,x)\] (2) \[\text{Categorical Proximity}=1-\frac{1}{|\mathbb{C}|}\sum_{c\in\mathbb{C}}dist\_ cat(c,x)\] (3)
* **Sparsity**: it measures the average of changes between a counterfactual example and the original input. \[\text{Sparsity}=1-\frac{1}{|\mathbb{C}|*|\mathbb{F}|}\sum_{c\in\mathbb{C}} \sum_{f\in\mathbb{F}}\mathbb{1}[c_{f}\
eq x_{f}]\] (4) Here, \(\mathbb{F}\) is the set of features, and \(\mathbb{1}\) is the indicator function.  
Figure 4: Degree of locality vs globality of the explanation models. The \(x\)-axis represents the thresholds and the \(y\)-axis represents the xt-Fidelity.  
* **Diversity**: it is similar to proximity. However, instead of computing the feature-wise distance between the counterfactual example and the original input, we compute it between each pair of counterfactual examples. As in the proximity measure, we considered two different diversity versions, one for the continuous and the other for the categorical features.  
Figure 5 shows the results of the Validity and Sparsity metrics for C-LORE-F and DRSA. C-LORE-F generates better valid counterfactual examples than DRSA. For both of them, when they generate a single counterfactual example, it is valid (near 100%). Notice that C-LORE-F has never generated more than 5 counterfactual examples. On the contrary, DRSA is able to generate more counterfactuals but the validity decreases and sparsity keeps similar.  
Both C-LORE-F and DRSA show outstanding performance on Sparsity (DRSA is slightly better) with an average of 0.9.  
Looking at the Diversity results (Figure 6), we can find that C-LORE-F generates more diverse examples with respect to the categorical features and diversity increases as the number of counterfactual examples increases. In the case of continuous features, DRSA is slightly better than C-LORE-F.  
Figure 5: Validity and Sparsity of counterfactual examples. The numbers in the \(y\)-axis represent the metrics values (the higher values the better performance), while the \(x\)-axis represents the number of generated counterfactual instances.  
Figure 6: Diversity of counterfactual examples, in Categorical and Continuous attributes. The numbers in the \(y\)-axis represent the metrics values (the higher values the better performance), while the \(x\)-axis represents the number of generated counterfactual instances.  
The Proximity results are shown in Figure 7. C-LORE-F generates counterfactual examples with lower proximity than DRSA for both the categorical and continuous features. Moreover, the inherent trade-off between diversity and proximity metrics can be observed in the case of categorical features.