# A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
## 2 Benchmark Components
### Models  
The artificial neural network models are built using TensorFlow [63] and the Keras package (Official website: [[https://keras.io/](https://keras.io/)]([https://keras.io/](https://keras.io/)), accessed on 9 July 2021) in Python. The model architecture is chosen to optimize (validation) model performance. A single layer neural network is used as the model to be explained, with a variable size of neurons in the hidden layer. As the literature shows that there is no single best rule of thumb to follow to determine the optimal number of hidden neurons [64; 65], we use a grid-search methodology to define it, looking for values up to twice the number of input vector size plus one [66; 67]. The grid also includes 4/5, 3/5, 2/5, and 1/5 of the maximum value. Other hyperparameters, such as the learning rate and the number of epochs, are also chosen using a grid-search. All other hyperparameters are set to the default values from the TensorFlow/Keras package. The model with the best AUC on the validation set was selected for each dataset. The full model specifications can be found in the Appendix A.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Dataset Information**} & \multicolumn{4}{c}{**Feature Information**} & \multicolumn{4}{c}{**Realistic Constraints**} \\ \hline
**Dataset** & **Data Type** & **Area** & **N. Rows** & **CAT.** & **NUM.** & **CLAS.** & **Univariate** & **Multivariate** \\ \hline Soybean (small) [68] & COMP. & CAT. & 47 & 35 & 0 & 4 & & \\ Lymphography [68; 69] & COMP. & CAT. & 148 & 18 & 0 & 4 & & \\ Hayes–Roth [68] & SOCI. & CAT. & 160 & 5 & 0 & 3 & & \\ Balance Scale [68] & SOCI. & CAT. & 625 & 4 & 0 & 3 & & \\ Tic-Tac-Toe 1 [68] & COMP. & CAT. & 958 & 9 & 0 & 2 & & \\ Car Evualution [68] & N/A & CAT. & 1728 & 6 & 0 & 4 & & \\ Chess 2 [68] & GAME & CAT. & 3196 & 36 & 0 & 2 & & \\ Nursery [68] & COMP. & CAT. & 12,960 & 8 & 0 & 5 & & \\ Iris [68] & LIFE & NUM. & 150 & 0 & 4 & 3 & & \\ Wine [68] & PHYS. & NUM. & 178 & 0 & 13 & 3 & & \\ Ecoli [68] & LIFE & NUM. & 336 & 0 & 7 & 8 & & \\ CMSC 3 [68; 70] & PHYS. & NUM. & 540 & 0 & 18 & 2 & & \\ BCW 4 [68] & LIFE & NUM. & 569 & 0 & 10 & 2 & & \\ PBC 5 [68] & COMP. & NUM. & 5473 & 0 & 10 & 5 & & \\ ISOLET [68] & COMP. & NUM. & 7797 & 0 & 617 & 26 & & \\ MAGIC GT 6 [68] & PHYS. & NUM. & 19,020 & 0 & 10 & 2 & & \\ SDD 7 [68] & COMP. & NUM. & 58,509 & 0 & 49 & 11 & & \\ Statlog-GC [68] & BUSI. & MIX. & 511 & 19 & 19 & 2 & & \\ Student Perf. [68; 71] & SOCI. & MIX. & 649 & 27 & 3 & 2 & & \\ Internet Adv. [68] & COMP. & MIX. & 690 & 9 & 6 & 2 & & \\ Default of CCC [68; 72] & BUSI. & MIX. & 30,000 & 9 & 14 & 2 & & \\ Adult [68] & SOCI. & MIX. & 48,842 & 9 & 5 & 2 & & \\ \hline \hline \end{tabular}  
* \({}^{1}\) Tic-Tac-Toe Endgame. \({}^{2}\) Chess (King–Rook vs. King). \({}^{3}\) Climate Model Simulation Crashes. \({}^{4}\) Breast Cancer Wisconsin. \({}^{5}\) Page Blocks Classification. \({}^{6}\) MAGIC Gamma Telescope. \({}^{7}\) Sensorless Drive Diagnosis.  
\end{table}
Table 2: Dataset Characteristics-Table with summarized information about each dataset used in the benchmarking study. Black filled circles indicate that the feature is present, while white filled circles represent that the feature is not present.