# A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
## 4 Results
For distance metrics, we have a less clear advantage for GS, which is among the best of two metrics, L2 and MD, but it is not the best for MAD. For L2, the two best algorithms are CA, with a mean L2 of 2.17, and GS, whose average is 1.15. In this case, CA has a slightly better ranking and worse numerical mean result for two reasons--first due to the higher number of realistic counterfactuals and second due to the larger variation in results. Regarding the CA variation in L2, we see this by the larger standard deviation (equal to 2.62), while GS is much more stable (0.93). When evaluating MAD, now we see ALIBIC (AL) and ALN as being the statistically best performing algorithms, where the latter (mean MADD equal to 5.7 \(\times\) 10\({}^{5}\) is slightly better than the former (mean MADD, which is 5.9 \(\times\) 10\({}^{5}\). For MD, we have the same situation that happens to L2, with CA having a slightly better ranking and statistically tied with GS, but CA has a worse mean MD (5.49) if compared to GS (1.37). \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & & **AL** & **ALN** & **CA** & **DI** & **GS** & **LO** & **MA** & **ML** & **SE** & **SY** \\ \hline \multirow{5}{*}{**C**} & **coverage** & 4.6 & 4.6 & 63 & 5.2 & 4.5 & 7 & 6.6 & 8 & 4.9 & 5.8 \\  & **sparsity** & 4.3 & 4.3 & 5.6 & 6.4 & 4.7 & 6.5 & 7.1 & 8.2 & 3.3 & 4.3 \\  & **L2** & 3.8 & 3.7 & 6.5 & 2.9 & 7.7 & 7.5 & 8 & 5.8 & 6.3 \\  & **MAD** & 3.9 & 5.8 & 4.6 & 6.2 & 4.4 & 7.5 & 6.9 & 8 & 4.5 & 5.3 \\  & **MD** & 3.8 & 3.8 & 2.3 & 6.5 & 3.1 & 7.6 & 7.6 & 8.1 & 5.8 & 6.1 \\ \cline{2-11}  & **coverage** & 6.8 & 6.4 & 3.3 & 6.9 & 6.8 & 5 & 5.5 & 6.1 & 6.4 & 5.3 \\  & **sparsity** & 6.6 & 6 & 4.3 & 6.8 & 4.3 & 6.1 & 6 & 7.3 & 5.4 & 4.3 \\  & **L2** & 7 & 6.5 & 5.3 & 7.2 & 1 & 5 & 6.7 & 5.4 & 5.7 & 5.3 \\  & **MAD** & 6.8 & 6.2 & 3.8 & 6.9 & 6.8 & 4.1 & 5.5 & 7.1 & 6.2 & 4.7 \\  & **MD** & 7 & 6.4 & 5.2 & 7.1 & 1 & 5 & 6.7 & 5.3 & 5.8 & 5.3 \\  & **coverage** & 6.6 & 4.3 & 4.3 & 5.3 & 4.2 & 8.2 & 8.3 & 7.1 & 3.9 & 4.4 \\  & **sparsity** & 6.5 & 3.6 & 5.9 & 4.5 & 5.3 & 8.2 & 8.3 & 7.5 & 4.6 & 2.6 \\  & **L2** & 6.6 & 3.9 & 6.1 & 4.6 & 5.2 & 8.2 & 8.3 & 6.8 & 3.5 & 4.7 \\  & **MAD** & 7.2 & 3.8 & 5.8 & 3.9 & 3.6 & 8 & 8 & 7.4 & 6.1 & 4.1 \\  & **MD** & 6.6 & 3.9 & 6.1 & 4.4 & 4.6 & 8.2 & 8.3 & 6.8 & 3.6 & 5 \\  & **coverage** & 5.9 & 5.1 & 5.8 & 5.4 & 4 & 6.6 & 6.6 & 7.1 & 5.2 & 5.3 \\  & **sparsity** & 5.6 & 4.7 & 5.3 & 6.1 & 4.7 & 6.1 & 7 & 7.7 & 4 & 5.0 \\  & **L2** & 5.6 & 4.7 & 4.5 & 6.2 & 2.1 & 6.9 & 7.4 & 6.8 & 5.2 & 5.5 \\  & **MAD** & 5.7 & 4.6 & 4.6 & 5.8 & 5.3 & 6.5 & 6.7 & 7.6 & 4.7 & 4.8 \\  & **MD** & 5.6 & 4.7 & 4.4 & 6.2 & 2.2 & 6.9 & 7.5 & 6.8 & 5.2 & 5.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Average rankings for dataset groupings. White bold ranking numbers highlight the best numerical result for each score and the gray cells show the algorithms that, statistically, can be considered the best. The results are rounded to one decimal place; then, in some cases, the best ranking can have the same numerical results as other, worse, scores. **AL**: ALIBIC, **ALN**: ALIBICNOGRAD, **CA**: CADEX, **DI**: DiCE, **GS**: Growing Spheres, **LO**: LORE, **MA**: MACE, **ML**: ML-Explain, **SE**: SEDC, **SY**: SynAS. Figure 3: Example of how numeric and ranking results can have discrepancies. When calculating numerical statistical properties from results, like mean sparsity, we only consider valid (or valid and realistic) results. In the figure, this means the gray cells’ (invalid results) values are not considered in the calculation of numerical scores. Thus, analyzing algorithm 1 (left), we have a mean value of 0.9 which is better, since we are considering sparsity, than algorithm 2 (on the right), which has a mean of 0.6. On the other hand, to calculate rankings, we consider all results, where invalid results are considered the worst situation possible.