# XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
## 5 Computer-Centred Evaluation Methods
### Interpretability  
The non-human assessment of the Interpretability of a XAI system can be more challenging than its Fidelity, in part due to the inherent subjectivity of what is considered an interpretable explanation. Nevertheless, several authors propose distinct methods to obtain a measure of this property. [PERSON] and [PERSON] [51], for example, define the Effective Complexity metric which, when having a low value, is an indication of simple and broad explanations. They also indicate a Diversity measure which reflects the degree of integration of an explanation. Other common metrics to evaluate this property are relatedto the model size, such as the Depth of decision trees or the number of non-zero weights in linear models, as noted by [PERSON], [PERSON] and [PERSON] [31]. [PERSON] et al. [52] used a simple metric as well to assess the Interpretability of their models: the number of runtime or arithmetic operations for a given input. Alternatively, [PERSON] and [PERSON] [53] utilize the number of regions in which the input space is divided as a different metric to judge the complexity of their simpler models used as interpretable versions of tree ensembles. [PERSON] [54] also proposed a framework for interpreting tree ensembles, using several metrics to evaluate the rules extracted, namely the Frequency of a rule (or the proportion of instances satisfying the rule condition), its Error (the quotient between the number of incorrectly and correctly classified instances by the rule) and its Complexity or length, defined as the number of variable-value pairs in its condition. [PERSON] et al. [55] proposed a framework as well, although aimed at interpreting black box models through learned decision sets (if-then rules). The evaluation of these rules/explanations involves simplistic Interpretability metrics, such as the total number of rules in the set, the maximum width of all the elements in it, the total number of predicates and unique neighbourhood descriptors, and the number of instances which satisfy two different rules, for every pair of rules. In an earlier but similar work (also based on decision sets), [PERSON] et al. [40] evaluated the Interpretability of the framework through other metrics, such as the degree of overlapping between every pair of rules in the set, the fraction of data points not covered by any rule, the average rule length (number of constituting predicates), the total number of rules and the fraction of class labels that are predicted by at least one rule. In another work, [PERSON], [PERSON] and [PERSON] [56] also define several metrics for evaluating XAI explanations, one of which aims at assessing their complexity by computing the entropy of the fractional contribution of every feature to the total magnitude of the attribution. In another work, [PERSON] et al. 's [57] proposed a framework aimed at a different scenario, more specifically the evaluation of CNN latent representations (activation maps). This is accomplished by measuring the intersection between the internal convolutional units and pixel-level semantic concepts previously annotated in the image. In this manner, if there is a high overlap between a unit and a concept in several images, it can be said that the unit represents that particular concept. [PERSON], [PERSON] and [PERSON] [58] used an evaluation metric previously proposed in [59] for similar cases denoted as location instability. In their work, the feature map of a filter f is first computed and the inference location of f determined by the convolutional unit with the highest activation score (whose receptive field's centre is backpropagated to the image plane). From here, the in-image distance between this location and certain ground-truth landmarks is registered. For example, in order to investigate if a filter represents a cat, some landmarks could be its head, legs and tail, as the distance from these to a perceived activation centre should remain stable.