# Real-Time, Model-Agnostic and User-Driven Counterfactual Explanations Using Autoencoders
## 5 Results and Discussion
### Mnist
In terms of computational efficiency, CFPROTO took 17.40 s per iteration with a variability of 0.09 s, while RTGCEx without AE and RTGCEx took 0.14 s per iteration with a variability of 0.01 s. It should be noted that RTGCEx without AE and RTGCEx have the same architecture and thus take the same amount of time at inference. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **AE** & **Black-Box Model** \\ \hline
**Metric** & MSE & Accuracy \\
**Value** & 0.007 & 0.991 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Results obtained for the AE in terms of MSE and for the black-box model in terms of accuracy. In this research, the comparison of models is conducted by examining the distribution of IM1 and IM2 values, rather than using statistical tests on large samples. This approach is deemed more appropriate because the use of statistical tests on large samples can lead to p-values that are extremely small, which can result in the conclusion of support for results that have little practical significance [22]. We have a test set with 10,000 instances, and the distribution of the IM1 and IM2 values obtained from each method are shown in Figure 10. These distributions support the conclusions drawn from Table 2. Specifically, the means of IM1 values for CFTRROTO and RTGCEx are similar, while those of RTGCEx without AE and RTGCEx have more variability. Additionally, the mean and variability of IM2 values is lower when using RTGCEx compared with the other methods. Furthermore, these distributions allow for the drawing of numerical conclusions regarding the comparison of the models, such as the probability that a counterfactual instance will have lower values in terms of IM1 or IM2 when using one method versus another. These comparisons are shown in Table 3. For example, the value of 0.66 in the first row and second column of the table indicates that there is a probability of 0.66 that the IM1 value of an instance generated by CFTRROTO will be less than an instance generated by RTGCEx without AE. Additionally, we can see that RTGCEx improves drastically over RTGCEx without AE, as an instance generated by RTGCEx has a probability of having lower IM1 and IM2 of about 0.75 compared with one generated with RTGCEx without AE. When comparing RTGCEx with CFTRROTO, in terms of IM2, the probability of obtaining better counterfactual instances with RTGCEx is 0.62 with respect to CFTRROTO. As for IM1, CFTRROTO and RTGCEx have the same probability that the value will be lower using one method or the other. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**IM1**} & \multicolumn{2}{c}{**IM2 (\(\times\)10)**} & \multicolumn{2}{c}{**Speed (s/it)**} \\  & **Mean** & **IQR** & Mean & **IQR** & Mean & **IQR** \\ \hline \hline \multirow{2}{*}{**CFPROTO**} & **1.20** & **0.39** & 1.20 & 0.83 & 17.40 & 0.09 \\  & 1.40 & 0.63 & 1.12 & 0.74 & **0.14** & **0.01** \\  & 1.21 & 0.58 & **0.91** & **0.63** & **0.14** & **0.01** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of the results obtained in test for each method. Best results are highlighted in **bold**. Figure 9: Examples of original and counterfactual instances generated with CFTRROTO, RTGCEx without AE, and RTGCEx. Upon analyzing the metrics used to measure the interpretability of counterfactual instances, we have found that IM1 does not always accurately reflect the interpretability of the instances. In some cases, a lower value of IM1 does not necessarily indicate that the counterfactual instances are more interpretable. To illustrate this point, we present two examples of counterfactual instances generated using CFPROTO and RTGCEx in Figure 11.  
From the visual representation, it can be observed that the instances generated using CFPROTO appear to be less realistic than those generated using RTGCEx. This is reflected in the value of IM2, as instances generated by CFPROTO have a higher value of IM2 than those generated by RTGCEx.