# A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
## 4 Results
Then, for the algorithm 1, we only have four cells classified as first and 14 as second best and two ties—while algorithm 2 has 14 cells considered as first, 4 as second best, and two ties. Therefore, the mean ranking for algorithm 1 is 1.75, which is worse than algorithm 2 (1.25), even though the left side has a better numeric mean. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & & **AL** & **ALN** & **CA** & **DI** & **GS** & **LO** & **MA** & **ML** & **SE** & **SY** \\ \hline \multirow{5}{*}{**CADE**} & **coverage** & 4.9 & 4.9 & 5.2 & 4.8 & 4.9 & 6.7 & 6.2 & 7.2 & 1.2 & 6 \\  & **sparsity** & 4.8 & 4.8 & 6.1 & 5.5 & 5 & 6.5 & 7.2 & 1.2 & 5.6 \\  & **L2** & **4.3** & **4.3** & 5 & 5.6 & 1.2 & 7 & 6.6 & 7.2 & 4.7 & 6.2 \\  & **MAD** & **4.4** & **4. In the case of mixed datasets, CA is no longer among the best algorithms, where we verify a statistical tie between DiCE (DI) and SE. Evaluating DI, it is able to generate 896 counterfactuals from a total of 1000 (89.6%), while SE generates 863 (86.3%). For sparsity, a similar situation that occurs when evaluating all datasets happens, where SE and SYNAS are considered statistically the best, but, in this case, SE has a slightly better ranking score. In numeric terms, SE has a mean sparsity of 0.95, which is slightly worse than SY (0.98), but SEDC's improved coverage (89.6% versus 76.7%) compensates this difference in ranking numbers. For distance metrics, GS performs best for L2 and MD, while SE has best results for MAD. Then, evaluating L2, we see that GS has a mean distance of 0.59, which is more than three times lower than the second best performing algorithm, SE, which has an average L2 of 1.85. The same happens to MD, where GS has a mean value of 0.30 while SE (the second best) is about 4.8 times longer (average of 1.43). However, for MADD, we have very different results from what we verified previously with other dataset types, where SE is the statistically best performing algorithm, being superior to the second best performing algorithm (GS) 44.5% of times, inferior in 26.8% cases, and performing the same in 28.7% of factual instances. The previous results referred to a situation where counterfactuals are simply those that could flip the classification value (valid), no matter if the counterfactual is realistic or not. Then, in Table 5, we add this additional requirement, where a counterfactual is only considered if it follows all realistic constraints the dataset has, i.e., it is valid and realistic. Analyzing the overall results for all datasets, we already see major changes as compared to the situation where realistic counterfactuals are not required. The only similarity is that CA still performs best in terms of coverage. However, the generated number of counterfactuals dropped significantly as it only is able to produce these for 2576 of the 3925 factual instances (65% of total), 27% lower than before. The impact in coverage is even larger for the other algorithms, where, for GS, it generated 77% fewer counterfactuals, SE 56%, ALN 52%, SY 45%, DI 48%, AL 39%, MACE (MA) 23%, and LO 17%. This has a direct impact on other scores as discrepancies between the mean numerical values and ranking scores are more salient. Evaluating sparsity, for instance, the best ranking is now obtained by CA where the mean sparsity is 0.44, but, if we compare to the second best ranking result (ALN), we have a mean sparsity equal to 0.65. This situation continues for distance metrics, where, for all distances, CA has the best ranking, for L2, its mean distance is 4.54, while the second best (ALN) achieved 1.5, a value about three times better. Comparing MAD, CA has a mean distance of \(4.8\times 10^{6}\) and the second best (SY) has a much better score of \(2.1\times 10^{4}\). For MD distance, the average distance for CA is 2.89 while the second best, SY, is 0.92. Therefore, it is clear that, although CA has the best overall rankings for all metrics, this is mostly because of its superior capacity of generating counterfactuals inside the realistic space.