# A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
## 6 Conclusions  
Benchmarking studies have fundamental value in scientific research, for example, in the case of image classification, the ImageNet Challenge [82] allowed us to test and compare novel methodologies that lead to advancements in the area. In the case of Counterfactual Explanations, a very recent field in XAI, our benchmarking framework aims to be beneficial in several forms. First, it allows those who are building new theoretical and/or algorithmic approaches to test their implementations more quickly and correct possible bugs or flaws in theory. Second, it provides a common ground to make an extensive analysis towards other implementations, removing any possible difference on data treatment, model de \begin{table}
\begin{tabular}{c l} \hline \hline
**Algorithm** & **Properties** \\ \hline ALIBIC & \(\bullet\) Model specific: Differentiable and using TensorFlow/Keras \\  & \(\bullet\) Low distance metrics (L2, MADD and MD) for realistic numerical counterfactuals \\ \hline ALIBICNOGRAD & \(\bullet\) Model agnostic \\  & \(\bullet\) Similar performance as ALIIBIC \\ \hline \multirow{4}{*}{CADEX} & \(\bullet\) Model specific: Differentiable and using TensorFlow/Keras \\  & \(\bullet\) High coverage for realistic and non-realistic counterfactuals \\  & \(\bullet\) Low counterfactual generation time \\  & \(\bullet\) Low variation in counterfactual generation time for different datasets \\  & \(\bullet\) High stability \\ \hline \multirow{4}{*}{DiCE} & \(\bullet\) Model specific: Differentiable and using TensorFlow/Keras or \\  & \(\bullet\) Pytorch (newest version includes model agnostic methodologies) \\  & \(\bullet\) High coverage for mixed features datasets in non-realistic settings \\  & \(\bullet\) High stability \\ \hline \multirow{4}{*}{GROWINGSPHERES} & \(\bullet\) Model agnostic \\  & \(\bullet\) High coverage when non-realistic counterfactuals are needed \\  & \(\bullet\) Medium coverage for numerical datasets that need to be realistic \\  & \(\bullet\) Low distance metrics (L2, MADD and MD) for non-realistic counterfactuals \\  & \(\bullet\) Low distance metrics (L2, MADD and MD) for realistic counterfactuals in numerical datasets \\  & \(\bullet\) High stability \\ \hline \multirow{4}{*}{LORE} & \(\bullet\) Model agnostic \\  & \(\bullet\) Good coverage for categorical datasets in realistic settings \\  & \(\bullet\) Low distance metrics (L2, MADD and MD) for categorical datasets that need to be realistic \\  & \(\bullet\) High sparsity for categorical datasets in realistic and not-realistic settings \\  & \(\bullet\) High stability \\ \hline \multirow{4}{*}{MACE} & \(\bullet\) Model specific: Must be a Decision Tree, Multilayer Perceptron, Random Forest \\  & \(\bullet\) Logistic Regression \\  & \(\bullet\) High stability \\ \hline \multirow{4}{*}{MLEXPLAIN} & \(\bullet\) Model specific: Differentiable and using PyTorch \\  & \(\bullet\) Low counterfactual generation time \\  & \(\bullet\) High stability \\ \hline \multirow{4}{*}{SEDC} & \(\bullet\) Model agnostic \\  & \(\bullet\) High sparsity for non-realistic counterfactuals \\  & \(\bullet\) High coverage for realistic numerical datasets \\  & \(\bullet\) High sparsity for realistic numerical datasets \\  & \(\bullet\) Low MADD distance score for numerical datasets \\  & \(\bullet\) Low counterfactual generation time \\  & \(\bullet\) High stability \\ \hline \multirow{4}{*}{SYNAS} & \(\bullet\) Model specific: Differentiable and using TensorFlow/Keras \\  & \(\bullet\) High sparsity for non-realistic counterfactuals \\ \cline{1-1}  & \(\bullet\) Low counterfactual generation time \\ \cline{1-1}  & \(\bullet\) High stability \\ \hline \hline \end{tabular}
\end{table} 
Table 6: General properties, and context in which each counterfactual generation algorithm seems to be the most suited. This table includes general instructions and, as explained in the text, may vary depending on the model, dataset, and factual point properties. sign, and scoring metric that might benefit one implementation over another.