# A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
## 2 Benchmark Components  
### Counterfactual Explanation Generation Methods  
Our choice of counterfactual generating algorithms is motivated by the following prerequisites: an open-source implementation, the use of novel theoretical and algorithmic strategies, and compatibility (in the code implementation) with artificial neural network models and tabular data. This resulted in 10 different algorithmic approaches to generate counterfactual explanations. The techniques can be categorized according to (1) the search strategy, (2) required access to the training data and (3) model internals, (4) whether the method is agnostic to a prediction model or only works for a specific model, (5) whether it implements additional optimization terms, and (6) whether it allows for additional custom terms, such as enforcing one-hot encoding, as summarized in Table 1.  
Each technique follows a certain search strategy. The two most common approaches are the use of convex optimization methodologies or custom heuristic rules. Convex optimization routines will include calculation of gradients, and hence require a differentiable scoring function. For a neural network, this requires having access to the weights [46; 47; 48; 49] or using numerical estimation of gradients [47]. In the case of heuristic rules, different procedures have been proposed that usually do not require accessing model inner settings and, therefore, have better model compatibility. Note that, as the method ALIBIC has two different implementations, both are included in this study. One uses the internal neural network weights to calculate gradients (we call this implementation ALIBIC), while the other will estimate the gradients without using the internal weights, which we refer to as ALIBICNOGRAD. Some algorithms require having access to the training data and/or the model internals. The training data requirement is generally aimed at improving counterfactual explanation quality, since the data can serve as supporting evidence in the counterfactual generation process. Such usage is implemented by ALIBIC, DiCE, GrowingSpheres, LORE, MACE, SEDC, and SynAS. Similarly, algorithms like ALIBIC (although it is not mandatory), CADEX, DiCE, MACE, ML-Explain, and SynAS require access to the inner model settings, such as the already cited neural network weights, to allow the use of specific optimization techniques. Several algorithms take advantage of additional optimization terms to improve some aspects of the counterfactual explanation quality. When convex optimization techniques are used, additional loss terms can include functions that measure sparsity (the number of changes needed to generate a counterfactual), proximity (how close the counterfactual is from the factual case), feasibility (statistical similarity of the counterfactual point to the data manifold), and diversity (generate several different counterfactual explanations); these terms are discussed in more detail in Section 3.2. For sparsity and improvement of the distance from the counterfactual to the factual point, ALIBIC and ML-Explain use L1 and L2 norms to calculate an elastic net loss term. In addition, to improve both the proximity to the factual point and feasibility of changes, ALIBIC and DiCE use distance metrics, such Euclidean distance, with some modifications and/or additional terms, where ALIBIC uses autoencoders trained with factual data and DiCE by using a distance metric that considers each feature variation. Finally, diversity is enhanced in DiCE by a function that considers the distance between each counterfactual explanation generated. Notwithstanding, these additional optimization terms are also used in algorithms that do not use convex optimization.