Evaluating Counterfactual Explainers
Authors: Diego Velazquez
Pau Rodriguez
Alexandre Lacoste
Issam H. Laradji
Xavier Roca
Jordi Gonzàlez
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainability
Counterfactuals
XAI

Abstract: Explainability methods have been widely used to provide insight into the decisions made by statistical models, thus facilitating their adoption in various domains within the industry. Counterfactual explanation methods aim to improve our understanding of a model by perturbing samples in a way that would alter its response in an unexpected manner. This information is helpful for users and for machine learning practitioners to understand and improve their models. Given the value provided by counterfactual explanations, there is a growing interest in the research community to investigate and propose new methods. However, we identify two issues that could hinder the progress in this field. (1) Existing metrics do not accurately reflect the value of an explainability method for the users. (2) Comparisons between methods are usually performed with datasets like CelebA, where images are annotated with attributes that do not fully describe them and with subjective attributes such as ``Attractive''. In this work, we address these problems by proposing an evaluation method with a principled metric to evaluate and compare different counterfactual explanation methods. The evaluation method is based on a synthetic dataset where images are fully described by their annotated attributes. As a result, we are able to perform a fair comparison of multiple explainability methods in the recent literature, obtaining insights about their performance. We make the code public for the benefit of the research community.

Under review as a conference paper at ICLR 2023
EVALUATING VISUAL COUNTERFACTUAL EXPLAINERS
Anonymous authors Paper under double-blind review
ABSTRACT
Explainability methods have been widely used to provide insight into the decisions made by statistical models, thus facilitating their adoption in various domains within the industry. Counterfactual explanation methods aim to improve our understanding of a model by perturbing samples in a way that would alter its response in an unexpected manner. This information is helpful for users and for machine learning practitioners to understand and improve their models. Given the value provided by counterfactual explanations, there is growing interest in the research community to investigate and propose new methods. However, we identify two issues that could hinder the progress this field. (1) Existing metrics do not accurately reflect the value of an explainability method for the users. (2) Comparisons between methods are usually performed with datasets like CelebA, where images are annotated with attributes that do not fully describe them and with subjective attributes such as "Attractive". In this work, we address these problems by proposing evaluation method with principled metric to evaluate and compare different counterfactual explanation methods. The evaluation is based synthetic dataset where images fully described by their annotated attributes. As a result, we are able to perform a fair comparison of multiple explainability methods in the recent literature, obtaining insights about their performance. We make the code and data public to the research community.
INTRODUCTION
The popularity of deep learning methods is testament to their effectiveness across multitude of tasks in different domains. This effectiveness has led to their widespread industrial adoption (e.g.. self-driving cars, screening systems, healthcare, etc.), where the need to explain a model's 's decision becomes paramount. However, due to the high level of complexity of deep learning models, it is difficult to understand their decision making process (Burkart & Huber 2021). This ambiguity has slowed down the adoption of these systems in critical domains. Hence, in order to ensure algorithmic fairness in deep learning and to identify potential biases in training data and models, it is key to explore the reasoning behind their decisions (Buhrmester et al. 2021).
In attempt to convincingly tackle the why question, there has been surge of work in the field of explainability for machine learning models (Joshi et al.) 2018 Mothilal et al. 2020 Rodríguez) al. (021I. The goal of this field to provide explanations for the decisions of a classifier, which often come in the form of counterfactuals. These conterfactuals provide insight as to why the output of the algorithms is not any different and how it could be changed (Goyal al.] 2019). Basically a counterfactual explanation answers the question: "For situation X why was the outcome Y and not Z". describing what changes in a situation would have produced different decision.
Ideally. counterfactual methods produce explanations that are interpretable by humans while reflecting the factors that influence the decisions of a model (Mothilal et af., 2020). So given an input sample and model, counterfactual explainer would perturb certain attributes of the sample, producing counterexample i.c., counterfuctual, that shifts the model's prediction, thus revealing which semantic attributes the model is sensitive to. In this work, we focus on the image domain given the recent surge of explainability methods for image classifiers (Joshi et al. 2018 Rodríguez et al. 2021 Singla et al. 1019,/hamm al.] (2018). A particular challenge of the image domain is that changes in the pixel space are difficult to interpret and resemble adversarial attacks (Goodfellow et al. 2014b) (see Figure
4 for example), so current explainers tend search for counterfactuals in a latent space produced

---

Under review as a conference paper at ICLR 2023
by, e.g., variational autoencoder (VAE) (Kingma & Welling 2013), by conditioning on annotated attributes (Denton et al. 2019; et al. 2018 Singla et al. 2019 Rodriguez et al. 2
Although explanations produced in the latent space are casier to interpret than in the pixel space, they depend on the chosen or learned decomposition of the input into attributes or latent factors. In the case of VAEs, these factors could be misaligned with the real underlying generating process of the images. Moreover, different methods in the literature rely on different autoencoding architectures or generative models to infer semantic attributes from images, which make their counterfactual search algorithms not comparable. In the case of annotations, since datasets do not provide access to the whole data generating process. they tend to focus on arbitrary aspects of the imput (such as facial attributes for CelebA (Liu et al., (2135)), ignoring other aspects that could influence a classifier's s decision boundaries such as illumination, background color, shadows, etc. This raises the need for evaluating explainers on a known set of attributes that represent the real generative factors of the input. In this work, we propose to fill this gap by introducing a new explainability benchmark based on synthetic image dataset, where we model the whole data generating process and samples are fully described by a controlled set of attributes.
An additional challenge when evaluating explainers is that there is no consensus on the metric that should be used. While there has been some effort to provide a general metric to evaluate explain- ers (Mothilal et al.) 2020 Rodríguez et al. 202T). most of the proposed metrics could be easily gamed to maximize the score of a given explainer without actually improving its quality for a user. For example, since current metrics reward producing many explanations, the score can be increased by (i) producing random samples that cannot be related to the ones being explained. This has motivated mea- suring the proximity of explanations (Mothilal al., 2020). (ii) Repeating the same explanation many times. This motivates measuring diversity (Mothilal lal et al. 2020). However, we found that it is possi- ble to maximize existing diversity measures by always performing the same perturbation a sensitive attribute while performing random perturbations to the rest of attributes that describe a sample. As result, although one counterfactual changing the sensitive attribute would suffice, an explainer could obtain a higher score by producing more redundant explanations. We argue that instead of providing many explanations, explainers should be designed to produce the minimal set of counterfactuals that represent each of the factors that influence model's decision. (iii) Providing uninformative or trivial explanations (Rodriguez et al. 2021). This has motivated us to compare the model's predictions with those expected from an "oracle classifier". Model predictions that deviate from the expected value more informative than those that behave as expected (see A.2 In this work, we address these problems by proposing a fair way to evaluate and compare different counterfactual explanation methods.
cierOur contributions can be summarized as follows: (i) we present a benchmark to evaluate counter- factuals generated by any explainer in fair way (Section 3): (ii) we offer insights on why existing explainability methods have strong limitations such as an ill-defined oracle (Section 3.3); (iii) we
introduce new set of metrics to evaluate the quality of counterfactuals (Section 3.4 and (iv) we evaluate 6 explainers across different dataset configurations (Section
RELATED WORK
Explainability methods. Since most successful machine learning models are uninterpretable (He ctt al., [2066] Jégou al., [2177] LeCun al. 1989). modern explainability methods have emerged to provide explanations for these types of models, which are known as post-hoc methods. An important approach post-hoc explanations is to establish feature importance for a given prediction. These methods (Guidotti et al.) 2018 Ribeiro et al. 2016 Shrikumar et al. 2017 Bach et al., 2015) involve locally approximating the machine learning model being explained with a simpler interpretable model. However, the usage of proxy models hinders the truthfulness of the explanations. Another explainability technique is visualizing the factors that influenced a model's decision through heatmaps (Fong al. 2019 Elliott al.) 2021| Zhou al. 2022). Hcatmaps useful to understand which objects present in the image have contributed to a classification. However, heatmaps do not show how areas of the image should be changed and they cannot explain factors that are not spatially localized (e.g., size, color, brightness, etc).
Explanation through examples or counterfactual explanations addresses these limitations by synthe- sizing alternative inputs (counterfactuals) where a small set of attributes is changed resulting in a different classification. These counterfactuals are usually created using generative models. A set of
2

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of explainers considered in this work. First column indicates whether counter- factuals are found with gradient descent. Second column displays the domain in which we perform the counterfactual search, with z referring to the attribute or latent space. Third column indicates whether the explainer takes into account changes in pixel space during optimization (e.g., visual similarity loss). Last column indicates the explainer performs feature selection to generate counterfactuals.
Mechod
Gradient
Domain Optimizes x-space
Feature
DICB Motisiaal et 2020
X
X
DIVE (Rodriguez
2021
(Laugel et 2017
8
StylEx (Lang 2021
x
/
LatentCCF (Balasubrumanian et al.
xGEM (Joshi et al. [018
x
methods condition the generative model on attributes annotated in the dataset by using a conditional
Generative Adversarial Network (GAN) (Joshi et al. 2018 Liu al., 2019 Sauer & Geiger 2021
Van Looveren et al.) 2021: Yang et al. 2021). However, this approach restricts the explanations to the provided attributes which do not reflect the entirety of the image properties, making the
applicability of these methods challenging where annotations are scarce. In order to generate counter- factuals without recurring annotated attributes, another set of methods uses VABs or unconditional
GANs (Goodfellow et al. 2014a) that do not depend on attributes during generation (Rodríguez et al.
202T
Denton et al., 2019] Pawelczyk al.) 2020 Perez al.] 2018 Mothilal al., 2020). See
Table for a comparison of the methods considered in our work.
Explainability Benchmarks. DiVE (Rodriguez et al. 2021) and DiCE (Mothilal et al. 2020) propose metrics that allow researchers to evaluate the quality of an explanation. These metrics evaluate the proximity of explanations to their original sample, and how diverse these are. Unfortunately, they are easy to game. For example, an explainer could maximize diversity by always modifying the same counterfactual attribute but randomly perturbing other non-counterfactual attributes to produce new redundant explanations. We propose more general, harder to game metric that allows to evaluate set of explainers in order to identify their strengths and weaknesses through fair comparisons. Further, the set of attributes of dataset influence the evaluation of the explainability methods. CelebA (Liu 2013a) is a common dataset used for generating counterfactus explanations (Rodríguez et al. 202 Denton et al. 2019), and it is labeled with series of attributes, such "Atractive", that fail to fully describe the true underlying factors that generated the images (e.g, illumination, occlusions, contrast, etc). Likewise, there is no guarantee that unsupervised disentanglement methods such as VAEs identify the true factors of variations without making strong assumptions (Arjovsky et al. 2019). We sidestep these problems by evaluating all explainers in common latent space with known attributes that fully describe the samples. Recently PPaweczyk et al. (2021) published benchmark (CARLA) with an extensive comparison of several counterfactual explanation methods across different tabular datasets. Our work differs from CARLA in three important ways: (I) we propose principled metric to compare counterfactual explanation methods, (2) we introduce a new synthetic benchmark that allows comparing multiple explainers in a fair manner in the same latent space. (3) We focus on counterfactual visual explanations, which require access to a common latent space for fair comparison since pixel-level counterfactuals are difficult interpret (e.g., adversarial attacks).
et al.
3 PROBLEM SETUP
In the following lines we describe a principled framework to quantify the quality of counterfactual explainers and show how can be applied compare multiple methods in the literature. In Section 3.] we define the data generation process, in Section 3.2 we define the counterfactual generation process, in Section 3.3 we define the concept of optimal classifier used to compare the predictions of model, and in Section 3.4 we define the metric used to evaluate counterfactual explanation methods. A notation table can be found Table
3.1 DATA GENERATION
Many explainability methods in the literature are designed for the image domain (Rodriguez et al.)
2021 Joshi et al. 2018 Lang al. 2021 (Singla al. 2019 Chang al. 2018). In this area, most
3

---

Under review as a conference paper at ICLR 2023
datasets can be described with data generating process where set of latent variables (z) result in an image (x) and a corresponding label (y), see Figure [a| However, not all the latents that generate the image have impact on the label (Zina). For example, the image brightness does not affect the presence of a dog. In addition, some latents can be correlated with the label (Zrorr). For instance, whenever there is a dog there is usually dog collar. Formally, where a set of latent variables E IR are sampled from a prior p(z), and a generator that produces images p(z]lz). Labels are generated using p(y (ccusal)), where Zeausal direct causal parents of y (Figure Tap. We also define Zcorr as the set of attributes that are correlated to y but not part of Zeausal _ Sometimes, these correlated attributes may have stronger predictive power, but relying on them would lead to unreliable predictions. For instance using the sky background for classifying airplanes. To generate datasets, we rely on a structural causal model (SCM) (Pearl 2009), corresponding to sequence of stochastic equations producing random variables based on the causal parents in the causal graph as described in Figure la
consider data generating process
subset of z containing
In order to obtain a known mapping between z and 2020), a synthetic dataset generator with many controllable attributes (font, character, color, rotation, size, etc). In addition, using synthetic dataset allows to control the effect of x and specify the amount of change in x relative to the amount of change in z (and vice-versa). Using synbols, we train image generator
propose to leverage synbols (Lacoste et al.
g(2] which is used to generate subsequent datasets. The generator g is provided to the explainers to offer a differential mapping from z to x. We believe this is a strength of our benchmark compared to using datasets of natural images.
3.2 COUNTERFACTUA GENERATION
Given an image r and a classifier f(x), a counterfactual explanation method (explainer) produces x', perturbed version of that shows some insight about the sensitivity of f to the semantic attributes that describe x. The perturbation is commonly performed on a learned latent space z. In general, explainers are tasked learn an encoder and find useful latent space, but this task is hard and still under active research. In order to bring a better comparison between explainers, we provide them access to the generating function g and so that explanations are generated in the same latent space. This gives us the opportunity to let explainers work directly in latent space by defining h(()= = f((())) In other words, we define an explainer as:
(2) = c(2,h, h,
where z is the ith counterfactual explanation from z found by explainer on the latent classifier h. Working in latent spaces greatly simplifies the task of an explainer, but we will see that there are still variety of challenges to be addressed. Namely, the notion of optimal classifier or stable classifier may be ill defined or may not always exists.
3.3 OPTIMAL CLASSIFIER
Counterfactual explanation methods tend to produce trivial explanations by perturbing the attribute being classified from the input (Rodríguez et al.) 2021). A more useful explainer would change model's predictions by perturbing non-causal attributes (such as the background of an image). To distinguish between these two kinds of explanations, an "oracle" is required, whose predictions are contrasted with those of the model. If an explanation changes both the oracle and the model's predictions, the explanation is deemed trivial and discarded. However, if the explanation only changes one of the two, the explanation is non-trivial. In the absence of human oracle who knows the causal attribute being classified, the authors resort to an optimal predictor ground truth classifier. However, the concept of optimal predictor is commonly ill defined and application dependant, thus we must proceed with care when assuming the existence of ground truth classifier. To show that, we next define the concepts of Bayes classifier, causal classifier, and finally the causal classifier with non-reversible generator used in this work.
lg E Zeoorz could be correlated to y for two different reasons: i) y + 2 ii) confounder a such that y + a + 2. Note that may be element of Zcausal or outside of the scene, such as the photograph. IIn this work, we consider deterministic generators. A more general formulation would be g(zlz)
4

---

Under review as a conference paper at ICLR 2023
SCE
Classilier Sumple
y= 0
Classifier
1(())
h(z)=0 0
Resusel
Z and
Sample
canaal
2007
h(z)
Reaced (X)
CP
Y
X
Oracle
1
(b)
(c)
Figure 1: (a) Example of causal graph satisfying the problem setup of section 3.1 (b) Successful counterfactual explanation as defined in (Mothilal et al. 2020 Joshi et al. 2018). That is, a successful counterfactual changes (gray) the classifier prediction (red) for the sample (point). The dashed square represents the maximum LI norm of the perturbation performed by an explainer (c) Our definition of successful counterfactual explanation (gray) considers any change where an oracle (green) behaves differently from the classifier (red). EF: estimator flips, NCF: non-causal flips, CF: causal flips.
Causal classifier with reversible generator. The causal classifier makes predictions solely based on the causal parents of y in the causal graph G. In latent space: heaussal (z) arg max,, p(y/ Zeausal)-
When the generator x g(z, 6x) is reversible, we obtain (cm = hrausal (g- (x)). Interest- ingly, this classifier robust changes of p(z) long as Zcausal) and p(z|zz) (z) remain unchanged.
Causal classifier with non-reversible generator. It is worth noting that when the generator is reversible, given I can lead to many z, which prevents from directly recovering z from x. A natural choice is to rely on the posterior distribution f(x) Σg p(z/zr//causaa(2) (z), where p(z|z) & pzz)z(zlz). However, this posterior now depends on p(z), making the new classifier no longer independent to distribution shift when p(z) is changed to e.g. pz(z). This leads to the following negative result (see A. for the proof, along with an example):
Proposition 1. If there exists a pair z, s.t. g(z) g(z') and heausal + (asussa(z)), then for any deterministic classifier f(x),, there is prior p' (z) s.s. the accuracy of f is O with respect to heausal-
3.4
EVALUATING COUNTERFACTUAL EXPLANATIONS
The goal for counterfactual generation methods is to find all the attributes that make a classifier behave differently from causal classifier (see Figure [c]. Note that Mothilal et al. (2020) only considered counterfactuals that change the predictions of a classifier (Figure Ib). and RRodíguezz et al. (2221) only considered the top region in Figure Tc These definitions do not cover cases such as when the oracle changes its prediction while the classifier's stay the same. Following Mothilal ct al. (2020); Rodríguez et al. (2021); Joshi al. (2018), also measure the similarity between the original example and the counterfactuals used to explain it. The reason is that counterfactuals should be relatable original samples that human interpret what is the sensitive semantic attribute. Next, we define the components of the proposed metric (Eq.
Proximal change (Joshi et al.) 2018 Mothilal et al., 2020). An explanationmust be relatable the original sample, thus it needs to be proximal. That is, the change z' needs to stay within a certain radius from Using Ll norm, the set of proximal x is defined as follows:
(z( {z' || z <r)
Estimator Flip (EF) (Josh et al., 2018 Mothilal et al. 2020). This is defined as proximal change on leading change in prediction of the estimator h (see Figure lb
EF(z) {z' h(z') # h(z) np..
(3)
5

---

Under review as a conference paper at ICLR 2023
Non-Causal Flip (NCF). Counterfactuals obtained by estimator flips (EF) common in the literature as they do not require the knowledge of heausal- However, if we have access to heausal, we can detect a new set of explanations: a proximal change in z' that changes the prediction of h but not of heausal:
(CF(Z)= = {z' EF(Z) A (z)) heausal (z)} n Pr.
(4)
Causal Flip (CF). Additionally, access to heausal allows us to detect another new set of explanations: proximal change in z' that changes the prediction of heausal but not h:
CF(z) = ( h(zz) h(z) A heausal(z)) * heausal(z) PrP
Thus, define the set of successful counterfactual explanation (SCE) follows:
SCE(2) (NCFUCF).
In summary, having knowledge the causal factors (access Aeaussl) allows us to evaluate counter- factuals explanations in a new way as illustrated in the following example. Given a dog classifier and an image of a dog. a counterfactual example that changes the background of the image in a way that alters the classifier's prediction (NCF) will almost certainly provide valuable insight about the model's behaviour. The same can be said about counterfactual example that removes the dog from the image without altering the classifier's 's prediction (CF) (see Figure Ich. Note that these counterfactuals cannot be detected without causal knowledge, which is only available if we have access to the entire data generating process i.e., synthetic dataset.
Orthogonal and complement subset. Note that both EF and SCE are possibly infinite sets and cannot be easily interpreted by humans. We could return the explanation minimizing some notion of distance on z or æ, however a good explainer should return a useful and diverse set of explanations.
To this end, propose a metric that only takes into account the subset of orthogonal and comple- mentary explanations. Otherwise, it is trivial to report many explanations that are a modification of existing explanation without being useful. For instance, modifying the hair color trigger change in gender classification is good finding, but changing the hair color again, and removing some clouds in the sky would not constitute a useful explanation. Hence, only admitting orthogonal explanations enforces a useful diversity. However, we also admit complementary explanations. That is, if darker hair triggers a change gender classification and lighter hair also triggers a change, these are two useful explanations. In short, given two explainers that find counterfactuals by perturbing the most sensitive attribute, the orthogonality and complementary requirements ensure that the one that provides more diverse set of counterfactuals by also perturbing less sensitive attributes scores higher. This is important because it rewards explainers that give a more complete description of the model the user. There may be use cases where only the most sensitive attribute matters. However, everything else being equal, we argue that, in general, is favorable to have access to a diversity of explanations.
The explainer is responsible for returning explanations produced with orthogonal or complementary perturbation vectors. To verify whether explanations orthogonal or complementary greedy algorithm. Concretely, we sort the explanations by how proximal they are to the original sample and add the first to the set. Then we iterate through the rest and sequentially add every subsequent explanation that is orthogonal or complementary to all the explanations currently in the set (see Algorithm| I for implementation). The resulting orthogonal and complement set is referred to as SCE, E (z). We use the cardinality of this set to evaluate the performance of explainers:
use
S4 |SCE1(()) (z)).
We consider the proposed setup to be fairer than previous works, since: (1) all explainers are compared in the same latent space, resulting in a fair evaluation, (2) uninformative explanations are discarded leveraging knowledge of the causal factors, (3) it is designed to be more difficult to game by repeating counterfactual explanations, and (4) it rewards explainers that return a more complete set of explanations.

---

Under review as a conference paper at ICLR 2023
7
n
7
4
dlsttrn carralation
h
h
h
h
h
h
h
Chencation
g
g
6
6
6
6
6
6
4
9
5
D
b
D
b
C J
f
f
8
f
p
y
v
v
1
7
2
2
vida
Figure 2: Interpolations produced by our learned generator (g(z)). Images are changed as the at- tribute's 's value changes smoothly from one value to another (left-right). From top to bottom: rota- tion, scale, h-translation, v-translation, char, font.
Figure 3: Average attribute perturbation for
each method/scenario. Gradient based meth- ods perturb almost all attributes while gradient- agnostic methods perturb only one or two. DiVE focuses almost solely on font.
4 EXPERIMENTS
In this section we give overview of the different methods (see Table and datasets that are comprised within our benchmark. Since we provide access to a common interpretable latent space, evaluate explainers that do not depend on concrete latent decomposition. The code is written in PyTorch (Paszke et al. 2017) and will be made public. Implementations details can be found in the Appendix.
Latent-CF (Balasubramanian et al. 2020): A simple method that performs a adversarial perturbations in the latent space until counterfactual with confidence higher than threshold tol is found.
DiCE (Mothilal et al. 2020): A method that aims to produce a diverse set of counterfactual examples directly from series of attributes or latent space by proposing series of perturbations that change the predictions of a classifier. This is achieved by gradient-based based optimization of multiple loss functions with respect to the attributes or latents and the classifier:
C = hinge_ (osse_)))s(((()) y, margin) + Andist(z, z!) badpp_diversity(f2
(4)
(B)
(C)
where optimizing (A) pushes the prediction of the classifier f towards y up to some margin, (B) ensures that counterfactuals (x') )') are close to the original samples (z), and (C) maximizes the distance between each pair of counterfactuals.
xGEM (Joshi et al.] 2018): A method equivalent to DiCE without the diversity term (C). DiVE (Rodríguez et al. 2021): A method similar to DiCE that leverages the Fisher Information (FI) to find non-trivial counterfactuals, i.e. samples that change the classifier prediction without changing the causal attribute Zeausal, thus focusing on spurious correlations. This is done by masking out latent dimensions with the highest FI while optimizing cost equivalent to Eq.
Growing Spheres (GS) (Laugel et al. 2017): A method that given a data point z identifies its closest neighbour classified differently e referred to enemy. This is done by finding the smallest ball around z that contains an enemy. Once e is found the dimensions with small changes e with respect are discarded through feature selection process, maximizing the sparsity of e z. StylEx (Lang et al. 2021 They find latent perturbation in a direction that maximizes the difference in the output of the classiher for the original sample and its perturbed counterpart.
Informed Search (IS): An explainer that knows about the data generation process in Figure la Thus, IS generates explanations by perturbing the spuriously correlated attributes Z,om
Since we already provide an interpretable set of latent attributes evaluate only the Attribute finding (AttFind) algorithm from the paper
7

---

Under review as a conference paper at ICLR 2023
Table 2: Score (Eq. 7 and percentage of trivial counterfactuals () obtained by each explainer for each of the different datasets described in 4.1 Values represent an average score across batches for the entire dataset for 3 different runs.
Correlatinn
0.50
0.95
0.50
0.95
#Spurious Explainer
Sg
Sy
Trivial (%)
'Trivill (%)
IS (Oracle) 4
2.40
2.67 =0.02
0.00 #0.00
0.00
DICE Motinaal el al. 2020
1.18 4:.01
1.17 20.01
9.37 10.11
6.78
20.28
DIVE
Rodneuuz et
2021
1.02
1.00 =0.01
2.51 #0.00
1.68
GS (1. Lugel al. 2017
1.01 1:000
1.01 20.00
4.49 10.40
2.34 20.15
6
Styllix (Lany
A721
1.04
1.17
2.41
1.58
Latent-CF (Blnsihrammmin et al. 2020
0.82 #0.00
0.93 20.00
0.00 =0.00
0.00 20.00
xGEM (o 2018
1.15
=0.08
IS (Oracle)
2.80 #0.04
3,63 =0.02
0.00 =0,00
0.00 =0.00
DICE Motinaal al. 2020
1.13 #0.01
1.19 =0.01
8.62 #0.46
6,70 =0.05
DIVE Rodr 2021
1.00
1.04 =0.00
2.28 #0.03
1.44
10
GS (Lauger at 2017
1.01 #0.00
1.00 =0.01
4.91 #0.25
1.95 =0.05
StylEx (Lang A74
1.15
1.12
3.37
1.62
(Halla et al. 2020
0.81 1::00
0.81 20.00
0.00 10:00
0.00 10.00
xGEM (Joshi 2018
1,15
1,16
10.17 =0.28
6.38
4.1 DATASETS
We design synthetic benchmark based the synbols dataset (aacoste et al. 2020). In this benchmarks images are fully defined by 3 categorical attributes (48 fonts, 48 characters, 2 background colors) and 4 continuous attributes (x-translation, y y-translation, rotation, scale), see Figure 2
An advantage of synbols is the large amount of values its categorical attributes such as character and font. This allows us design different scenarios by introducing spurious correlations based on subsets of these attributes. From mod 2. Then we leverage the font attribute to introduce spurious correlations (Zcorr in Figure Iap. Note that increasing the number of fonts in Zeorr (the rest will be in Zind) increases the random chance of finding a counterfactual by accidentally switching the font. Likewise, increasing the amount of correlation between Zeorr and y makes spurious correlations easier find since the classifier latches stronger on them. We hypothesize that stronger correlations will benefit gradient-based explainers, which will find higher gradient curvature for highly correlated fonts. To explore how explainers behave under different scenarios, we consider 6 and 10 spurious fonts with 50% and 95% correlation with y, resulting in a total of 4 scenarios. Further, we introduce a 5% noise the the Zcausal attribute (character) to encourage classifiers to also rely on the font.
on, we assume Zeausal char E [1..48) and heausal Zeausal
4.2 RESULTS
We evaluate the performance of six different methods with the metric defined in Eq. 7 Each method is evaluated in several different datasets with varying levels of difficulty as described in 3.1 and 4.1 Additional results can be found in section A.4
It is hard to diversify. A good explainer should be able to predict the behavior of the model with respect to changes in the different attributes that generate the data. In the case of a classifier, finding the attributes that induce it to change its prediction in order to reveal if it is relying on attributes that are independent from the class being predicted a desirable goal. In the pursuit of this goal, an explainer should ideally populate SCE(z) (see Eq. 6 with explanations altering each of the attributes that are correlated with the data. However, as shown in Table 2 explainers fail to consistently find more than one altering attribute.
Performance saturates with 6 fonts. We observe that methods do not significantly increase the number of successful counterfactuals when adding more spurious fonts (Table ( This is, partially, because methods tend to focus on changing the Zeausal attribute character as seen in Figure 3, which leads to trivial counterfactuals. When adding more fonts, the font identification task becomes more difficult for the classifier, which makes it more sensitive to characters and exacerbates this problem. For more extensive ablation illustrating this phenomenon see Figure 5
Gradients tend to perturb most of the attributes. Figure SegureeBoffer offerss insight into how each method perturbs and we see that gradient-based methods tend to perturb almost all attributes equally,
8

---

Under review as a conference paper at ICLR 2023
exploring the perturbation space in many directions. In the extreme. we found that Latent-CF slightly modifies all the latent attributes, producing counterfactuals that resemble adversarial attacks. While modifying all the attributes increases the chances of finding 1 good explanation average, it also prevents the explainer from finding multiple non-trivial diverse explanations. On the other hand, methods that gradient-agnostic focus on perturbing one or two attributes, resulting in a more narrow search space. This increases the risk of methods focusing on Zeausal (Figure [a]. This is evidenced in Figure where StylEx and GS considerably perturb the character attribute. Interestingly, the perturbation pattern of DiVE shares some similarities with StylEx and GS due to gradient masking.
Explainers exploit bad classifiers. As seen in Table 2 explainers are not significantly affected by the amount of spurious correlation Zcon introduced. This indicates that, in contrast with the oracle (IS), methods produce explanations by changing the font attribute Zeausal (as seen in Figure 3) without changing the classifier's prediction, thus creating successful counterfactual (Eq.5) Figure shows some examples of methods achieving this by changing the accent mark, the umlauts and the circumflex of vowel characters (e.g.. 6 + ö). These counterfactuals expose failure cases of the classifier and are therefore useful, since they show that the classifier is unable to classify some characters. See A.4 (On causal counterfactuals).
DiVE perturbs almost exclusively the Zeer attribute (font), specially for high correlation values, this indicates that the method successfully distinguishes between Zeausal and Zcor attributes. However, it is not able to consistently perturb the font in the right way to produce diverse set of counterfactuals as evidenced by its score (Table (
DiVE focuses on changing the font. As shown in Figure 3
Non-triviality is not enough. Table 2 (right) shows the average percentage of trivial counterfactuals found by each method. We observe that methods that tend to produce higher number of successful explanations (left) tend to also produce a larger number of trivial counterfactuals (right), which are discarded in our metric.
DISCUSSION
Benchmark In this work, we have introduced more comprehensive definition of good counterfactual (Section 3 that we instantiate as a metric (Section 3.4) as well as fair evaluation setup (Section 3.1 in the form of a benchmark. Previous evaluation setups use datasets like CelebA where the causal data generation process is unknown and use metrics that are easy to game. In contrast, our evaluation setup uses a more comprehensive and fair metric while providing control over the entire data generating process and therefore knowledge of the causal factors. Even though knowledge of causal factors rare when working in real world scenarios, it is possible to adapt our metric to take only into account orthogonal and complement set of estimator flips EF (Eq. B) which do not require causal knowledge. However, any evaluation schema that does not include causal information would be incomplete. Further, if an explainer fails to provide set of useful and diverse explanations for our synthetic dataset it is very unlikely that it is able to do so for real datasets. Nevertheless, we recommend users to also evaluate explainers using real world data.
Oracle We argue that successful counterfactuals should be considered in the perspective of a human. In the absence of a human, we must resort an optimal classifier, whose task is contrast the predictions of the model with the optimal prediction and spot unexpected behaviors; acting as an oracle. Without an oracle, it not clear how we could assess whether a model is working as intended. We show that the optimal classifier is commonly ill-defined in the image domain, because it is not always possible to access an invertible image generator (Section 3.3) Therefore, it cannot be expected that a classifier trained on pixel space achieves optimal performance.
Results Our experimental results indicate that the different counterfactual explainers in the literature perform similarly and there has been little improvement in the recent years (Table 2). Although most of them find a single explanation in average, we found that they do it in different ways (Figure 3).
6 CONCLUSION
In this paper we present a benchmark that provides unified metrics for evaluating different counter- factual explanation methods. The benchmark consists of synthetic images fully described by their annotated attributes which are accessible to the explainers through a differentiable generator. We hope this benchmark serves as an inspiration for developing future explainability methods.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY
In an effort to ensure reproducibility we provide implementation details in A.3 The total run-time for all experiments is -37 hours on single Titan-X GPU. The code is written PyTorch (Paszke et al., 2017) and is made public along with the datasets and pretrained weights for the models used in this work. Detailed documentation will be released upon publication.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paze Invariant risk minimization. arXiv preprint wrXiv:190..0289. 2019.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7)xo0130140, 2015.
Rachana Balasubramanian, Samuel Sharpe, Brian Barr, Jason Wittenbach, and Bayan Bruss. Latent- cf: a simple baseline for reverse counterfactual explanations. arXiv preprint arXi::0012009301, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /096, 2018.
Vanessa Buhrmester, David Münch, and Michael Arens. Analysis of explainers of black box deep neural networks for computer vision: survey. Machine Learning and Knowledge Extruction, (4):966-989, 2021.
Nadia Burkart and Marco F Huber. A survey the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70:245-317, 2021.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. arXiv preprini arXiv: arXiv:180..08024, 2018.
Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. Detecting bias with generative counterfactual face attribute augmentation. 2019.
Andrew Elliott, Stephen Law, and Chris Russell. Explaining classifiers using adversarial perturbations on the perceptual ball. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10693-1070) 2021.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. Proceedings of the IEEF/CVF international conference on computer vision, pp. 2950-2958. 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577. 2014b.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pp. 2376-2384. PMLR, 2019.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. Local rule-based explanations of black box decision systems. arXiv preprint 1805 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 777-778872216 2016.
https://ancnymouut 4open sci scwence////ee.... 15A3
10

---

Under review as a conference paper at ICLR 2023
Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp- 1 11-19, 2017.
Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. xgems: Generating examplars to explain black-box models. arXiv preprint arXiv:18066 08867, 2018.
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR. 2020.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pp. 428-484. PMLR, 2022.
Alexandre Lacoste, Pau Rodríguez, Frédéric Branchaud-Charron Parmida Atighehchian, Massimo Caccia, Issam Laradji, Alexandre Drouin, Matt Craddock, Laurent Charlin, and David Vázquez. Synbols: Probing learning algorithms with synthetic datasets. arXiv preprint arXiv:2009.06411 2020.
Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William Freeman, Phillip Isola, Amir Globerson, Michal Irani, al. Explaining in style: Training gan to explain a classificr in stylespace. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 693-702, 2021.
Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki. Inverse classification for comparison- based interpretability in machine learning. arXiv preprint arXiv:1712.08443, 2017.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551) 1989.
Shusen Liu, Bhavya Kailkhura, Donald Loveland, and Yong Han. Generative counterfactual introspec- tion for explainable deep learning. In 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1-5. IEEE, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015a.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015b.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint arXiv:1608.0390 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110:1711.05100 2017.
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607-617, 2020.

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Martin Pawelezyk, Klaus Broelemann, and Gjergji Kasneci. Learning model-agnostic counterfactual explanations for tabular data. Proceedings of The Web Conference 2020, pp- 3126-3132, 2020.
Martin Pawelczyk, Sascha Bielawski, Johannes van den Heuvel, Tobias Richter, and Gjergji Kas- neci. Carla: A python library benchmark algorithmic recourse and counterfactual explanation algorithms, 2021.
Judea Pearl. Causality. Cambridge university press, 2009.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135- 1144, 2016.
Pau Rodríguez, Massimo Caccia, Alexandre Lacoste, Lee Zamparo, Issam Laradji, Laurent Charlin, and David Vazquez. Beyond trivial counterfactual explanations with diverse valuable explanations. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 1056-
1065, October 2021.
Axel Sauer and Andreas Geiger. arXiv:2101.06046, 2021.
Counterfactual
generative networks.
arXiv preprint
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145- 3153. PMLR, 2017.
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. arXiv preprint 1rXiv:1911.004833 2019.
Arnaud Van Looveren, Janis Klaise, Giovanni Vacanti, and Oliver Cobb. Conditional generative models for counterfactual explanations. arXiv preprint arXiv:2101.10123, 2021.
Fan Yang, Ninghao Liu, Mengman Du, and Xia Hu. Generative counterfactuals for neural networks via attribute-informed perturbation. ACM SIGKDD Explorations Newsletter, 23(1):59-68 2021.
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9623-9633, 2022.
A APPENDIX
A.1 PROOF OF PROPOSITION
Proposition 1. there exists a pair z, z' s.t. g(z) = g(z') and heausal (z) + heausalZ then for any deterministic classifier (xx), there is prior p (z) s.1. the accuracy of J is O with respect to heansal-
Proof. Let S {a f(g(t)) # heausal and show that |S| * 0. Since g(z) = g(z'), we have that f(g())) f(o(z'), Also, since heausal (z) + heausal (z'), we have either f[((z) + heausal (z) f(g(2')) * Meausal (z'). Finally, any prior p' (z) with mass outside of satisfies the proof. [
12

---

Under review as a conference paper at ICLR 2023
As example, let's consider cube classifier from 2d projections of cubes. We can train classifier to predict the correct class from the 3 visible faces. However, since the back of the cube is not visible there exists datasets such that the back of the cube is distorted in way that makes it not cube. The point being made here is that even though heaussal exists, feausal does not always exists. Further assumptions needs to be made which priors p(z) are valid for robustness to distribution shift, e.g.. as humans, we make a symmetry assumption for classifying cubes. Perhaps future line of research in explainability should seek to find what assumptions are made by classifiers. In the rest of this work, we will make the assumption that if g(z) = g(z'), then heausal (z) = heausal (z') for all pairs z, z'
The proof is simple but it contrasts with the known result that heausal is robust any distribution shift over p(z).
A.2 INFORMATION OF THE EXPLANATIONS
In our work, we aim to improve the amount of information provided by explanation sets. In terms of information theory, unexpected events tend to be more informative. this sense, repeating the same explanation multiple times provides little information and increasing diversity increases the amount of information. In addition, the amount of information of an event is usually measured with respect to some underlying probability distribution. If we know that classifier is trained to detect certain object, we expect the classifier to learn the conditional probability distribution (label given input) of the training set. Thus, deviations from this distribution are unexpected and informative. However, we do not know the real label for new samples and thus, we need to resort to some form of oracle or ground truth classifier in order to compare it with the model being explained. This ground truth classifier must have access to the real data generating process in order to infer the correct class from images, and the data generating process of image-label distributions typically a causal process governed by the laws of physics. That is why we refer to this classifier as "causal classifier".
A.3 IMPLEMENTATION DETAILS
In this section we will describe the training setting for the encoder ((z(z) the generator g(z) and the classifier to be explained (ax), along with the settings for each of the explainers considered in this work.
A.3.1 TRAINING DETAILS
Encoder The encoder is based on BigGAN's (Brock et al. 2018 Rodríguez et al. 2021 discriminator architecture with a classifier on top and is trained on Synbols Lacostt et al., 2020) dataset. Given an image we task the encoder with predicting the attributes that describe it. It is trained for 100 epochs with a batch size of 64. We use Adam (Loshchilov & Hutter 2017) with a learning rate of 0.001 and weight decay of 0.0001 with a cosine annealing learning rate acheduler(Loshchilov & Hutter 2016). Since Synbols contains discrete and continuous attributes, we optimize them separately minimizing:
Ldiserete + Leontinnons
where Laiscrete is the average cross entropy loss for each categorical attribute and Lcontimouss is the L distance between the original continuous attributes and the ones predicted.
Generator Similar to the encoder, the generator is also based a BigGAN (Brock et al. 2018) architecture and trained with the same hyper-parameten and learning rate scheduler. Given a set of attributes z and the image that they describe, train the generator reconstruct from However, in order to provide nigh-dimensional input to the generator instead of directly using the 7 Synbols attributes leveraged in our benchmark project each categorical attribute (character and font) into a 3-dimensional space. The embeddings concatenated with the 5 continuous Synboll minimizing the following criteria:
R² (Section 4.11.
embedding layers to
Synbols attributes obtaining IR11, We train the generator
5 Thiis assumption could be relaxed by saying that the probability of this assertion being violated is unlikely under predefined set of valid distribution shifts. We treat the background color continuous attribute
13

---

Under review as a conference paper at ICLR 2023
Table 3: Notation Table.
Notation
Description
An input image
The label of the input image
A set of latent variables
Zeausal z
A subset of containing the causal parents
A counterfactual explanation for z
(x))
The classifier to be explained
h(z))
The latent classifier to be explained
bransal (z)
The latent classifier to that is robust under change of p(z)
p(z)
Prior distribution over z Deterministic image generator
((z)
g g(
Stochastic image generator An encoder
g(z x))
e(z, h,
A function that generates explanations
Free a x + (1 a) x1 -(x') ll1
(10)
where 0.2, and x' are the original and reconstructed image respectively and ((x) the learned encoder features for image Lastly, to reduce the amount of noise in the generated images we use a discriminator network trained alongside the generator to distinguish between ar and x'. Specifically the generator is trained every 3rd iteration. So, the criteria for the generator to optimize becomes:
Lee + log(1 D(z'))
(11)
where À 0.01 and D(x') the discriminator's estimate of the probability that the reconstructed image æ' is real.
Classifier The classifiers we set to explain are ResNet-18 (He et al. 2016) architectures trained on the different benchmarks described in Section All the classifiers are trained for 10 epochs with batch size of 256. We use AdamW with a learning rate of 0.01 and a weight decay 0.0001 with a cosine annealing learning rate scheduler.
A.3.2 EXPLAINER EVALUATION
In this section we will detail the hyper-parametent chosen for each of the methods analyzed this details regarding the benchmark. Given the varied nature of the explainers work well parameters we will refer to the works where the explainers are introduced when and their hyper-parameters describing the effect the hyper-parameters have each method. All of the papae-parameters where chosen through random search.
Common setup Across experiments there few settings that common for all methods. To save time, instead of producing counterfactuals for the entire validation dataset we select balanced subset with a total of 800 correctly and incorrectly classified samples with different levels of contidence. We do this by selecting the 100 samples closest to the required level of confidence f(x) E +...11,0..4 -06,, +.99).
The explainers are required to produce 10 counterfactuals per sample, if method is origi- nally conceived to produce only one, we follow (Mothilal et al., (020) to create 10 samples close to the original sample in space and task the method with producing an explanation for each of them. The attributes in are standardized using the mean and standard deviation of the attributes
the training set. To prevent explainers from generating counterfactuals that the generator g(z) cannot interpret clip every coordinate in the maximum and mini- mum values for each attribute found the training set. We set the batch size to across experiments.
14

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Generating orthogonal set of counterfactuals
oithogenal__sn (wtnp.innnnne. ndarray, e_ac: up adarray taum0.15: float) ap.ndarrsy: n, x n,
Receives the latenta of an original sample (2), and a set of succesefvl counterfactuals
(e_sc from Eq, and returns an orthogonal set of counterfactuals (Bg. 7),
two samples 22 and z2 are orthogonaz whenever abs (cos 221) < tap.
Two namples and 22
complementan whenever (zi, z2)
[Noe,, d d,
delia_so e_30 - 2 compute pertartation vector
della_se_neza mp.1inalg.... norn 1, axis=1) & compute noze of perturbations
indices "rsis__s_nemmmeraeee
delte_orth deltt_sccNoene, indices [0]] / initialize set of orthogonal perturbations
for 1 in ind oes
sim cas (delta__ee(i,, deita_orth) calculate cosine similarity wrt all ellements in set cond_orth
anp.aba(sim) taul.alll orthogonalitp condition cond_comp (aim + cau) any)) complementary conaition if cond_orth or cond_comp: if condition satisfied, add to set
delta_orth epp.cenettenate ((neteenntt((deita_ee. ([deita_orth, delta_se[None,, ill, axism0)
return z dalta_orth return set of arthogonal countorfactuale
Since the categorical attributes in space (font and character) produced by embedding layers and projected into a 3-dimensional space each (Section A.3.1 they must be treated differently from the continuous attributes. When measuring if an explanation is proximal the original sample z we establish three different radius r values (see Eq. 2). one for the font, one for the character and one for the continuous attributes. For the character and Tont attributes distance between the embedding representations of each attribute and for continuous attributes we set 1. Making this distinction allows the explainer to change from any given font/character to another while preventing it from modifying every attribute at once. When verifying if counterfactuals fulfill our orthogonality condition we look at perturbations between continuous attributes (ccom) However, for embedded attributes (font and character) transform their embedded representation in z space into:
set to be the maximum pairwise
A Softmax w, 122 -t), Vi [1..48]
(12)
where c' is the representation of the categorical attribute character/font for a given counterfactual z'. are the weights of the embedding layer used to map the 48 characters/fonts to their 3 dimensional representation e' and small change in c' that maps to a different categorical attribute can be orthogonal. In order to turn 2 into a perturbation vector we cancel the perturbation on the original attributes coordinate by setting:
a temperature factor to 0.33. We choose the softmax function so that any
(1 (1-14 1c)
(13)
where is vector of 1s and 1, is one-hot representation of the categorical attributes for the original sample. Thus given sample z we verify that two counterfactual explanations z' and z" are orthogonal by measuring cosine similarity: cos b z"m) for categorical attributes and cos(z'..t Z, Z z) for continuous attributes.
Growing Spheres (GS) (Laugel et al. 2017) For this method we found the best configuration was setting the initial radius 1) to 10 and the number of candidates to 50. For a detailed description of the role of these hyper- sapre-parameters see (Laugel et al., 2017).
StylEx (Lang et al. 2021) For this method we found the best configuration was setting threshold to 0.3 using the "Independent" selection strategy and the amount of shift applied to each coordinate to 0.8. This last parameter is not mentioned in the Stylex (Lang al. (021) paper, but it can found as parameter under the name shift_size in the implementation they provide. For a detailed description of the role of these hyper-paramette see (Lang et al. 2021).
DiCE (Mothilal et al. 2020) For this method the best results were obtained by setting to learning rate to 0.1. the reconstruction weight of the loss function A1 to 1 and the diversity weight of the loss function X2 to For a detailed description the role of these srper-parameters see (Mothilal et al.) 2020). We set the maximum number of iterations to 50 in case the algorithm does not converge.
15

---

Under review as a conference paper at ICLR 2023
Original
Spuce
Space
W
W
j
i
Figure 4: Comparison of perturbations performed in latent and pixel space. Note how the perturbations performed in latent space convey meaningful information (e.g., changes in the font character), while the ones performed in pixel space resemble adversarial attacks and do not provide any valuable insights into the classifier's reasoning.
DiVE (Rodriguez et al. 2021) For this method we found the best configuration was setting the learning rate 0.1, the weight of the proximity loss term to 0.0001, the factor that controls the sparsity of the latent space y to 0.1 and the weight of the diversity loss to 0.001. For detailed description of the role of these hyper-parameteer see (Rodriguez al. 202T). We set the maximum number of iterations to 50.
xGEM (Joshí al., 2018) This method shares some parameters with DiVE. We found the best configuration for this explainer was setting the learning rate to 0.1 and the weight of the proximity loss term to 0.001. We set the maximum number iterations to 50.
Latent-CF (Balasubramanian et al., 2020) For this method we set the learning rate to 0.1, the probability of target counterfactual class p 0.1 1 and the tolerance lol to 0.5. For detailed description of the role of these hyper- hyper-paramette parameters see (Balasubramanian et al. 2020). We set the maximum number of iterations to 50 in case the algorithm does not converge.
A.4 ADDITIONAL RESULTS
In this section we present some qualitative results in the form of counterfactuals generated by each method, along with extra quantitative results shown in Table We also provide an example illustrating why explanations found in pixel space are uninterpretable (Figure
Quality over quantity As seen in Table some explainers obtain a high percentage of successful counterfactuals SCE, sometimes even higher than the oracle (XGEM, DiVE). However, this is not
Table 4: From left to right: We report the percentage of estimator flips EF (Eq. 3) (Joshi et al. 2018 Mothilal al. 2020), percentage of successful counterfactuals SCE (Eq. 6) and what percentage of those are Causal Flips and Mon-Causal Flips (see Section 3.4 and Eq. Values represent an average score across batches for the entire dataset for 3 different runs.
Cosellation
0.50
0.95
0.50
0.95
ASpuctous
Explainer
SCB
Cassal
Rate
Cansal
IS (Oracle)
40.55
0.00
100
DICE MMdhili et 1020
32.90
31,96
56.67
26.46
29.22
73.54
70.78
25.02
60.83
6.55
93.45
96.56
36.14
100
23,66
78.40
20.08
70.61
05.52
97
35.39
25
Mothill
31.37
20.33
18.79
20.57
37.18
Lmpgteie
21.60
18.30
22.00
10.00
Tnde s. 2013)
66
62.92 84/00
$0.35
95.49
44.37
16

---

Under review as a conference paper at ICLR 2023
reflected in their score S# (Table 2) which is considerably lower than the oracle's. This is because even though the explainers can find a high number of counterfactuals they are discarded by our metric since they not orthogonal score measured using estimator flips EF (Eq. 3 (Joshi et al. 2018 [Mthilal et al.) 2020) is not correlated with our score S# (Table 2 For example, xGEM obtains higher score than the oracle (IS) despite the latter returning a more complete set of explanations. This shows how previously proposed metrics (Joshi al. 2018; Mothilal et al. 2020) can be gamed by explainers by generating many redundant explanations that fail to fully the describe the model's behaviour. Figure [S] also supports this finding. showing how the performance of the oracle (IS) is the only one affected by the amount of correlated attributes and their level of correlation (see Section 4.1)
complementary and thus redundant. Further, note that the
On causal counterfactuals As seen in Table 4 some methods produce a high amount of causal change counterfactuals i... they change Acausal and create counterfactual that does not change the classifier's prediction whilst changing the oracle's prediction, exposing failure cases in the classifier. We find that DiCE (Mothilal al. (0200) and StylEx (Lang et al. (021)) produce a high amount of these counterfactuals, while GS (Lagel et al. 2017) and Latent-CF (Balasubramanian et al. 2020) always change the classifiers prediction and thus produce none. The oracle (IS) is not designed to perturb Zeausal in any way so it cannot produce any causal counterfactuals. Figure 6 shows two ways in which explainers achieve these counterfactuals. We high amount of causal counterfactuals (DiCE, StylEx) can be very "creative", when modifying the character. Note that besides confusing the classifier by modifying the character's diacritic, explainers can create new characters entirely by merging two letters together or even adding an accent mark to a consonant. Note that this behaviour is unavoidable in the absence of optimal classifier.
that explainers that generate
Additional insights As seen in Dablaiieen are unable to generate a diverse set of counterfactual explanations. However, Table 4 highlights some differences between methods when it comes to other metrics. If the objective is to maximize the number of estimator flips EF or the number of successful counterfactuals SCE we recommend using xGEM the objective to maximize the number of causal flips we recommend using StylEx or DiCF. Note that explainers generate a high amount of redundant counterfactuals, and choosing them based on how they maximize these individual metrics is of little use.
A.5 LIMITATIONS
As discussed in Section 3.3 the core limitation of explaining image classifiers via latent perturbations the lack of accurate reversible generators. If the generator not reversible, a given can lead to many z, which prevents the direct recovery of from æ. It might be good idea to bypass the
Raplatrer
eaglatrer
3.0
Dyler
2.5
2.0
1.5
1.0
Condicino
Figure 5: Sensitivity of every explainer to varying amount of correlation levels (left) and number of spuriously correlated attributes (right) (see Section 4.1 measured with our score (Eq. 77). Note how the performance of the explainers, excluding the oracle (IS), does not scale and is not sensitive the level of correlation and the amount of correlated variables. This supports our findings that explainers unable to provide diverse set of explanations and focus changing the causal attribute Zeausl-
17

---

Under review as a conference paper at ICLR 2023
DICE
DIVE
StylEx
XGEM
ò
6
i
i
2
à
á
T
obtained by changing/removing the diacritic in the character.
Causal Flip counterfactuals
Ô
O
2
E
<
O
0
0
E
Ú
0
6
Ô
è
2
a
1
1
á
à
0
2
d
6
6
6
J
X
0
Causal Flip counterfactuals obtained by obscuring the character creating new one entirely.
5
6
2
F
W
i
2
O
P
7
S
b
D
à
O
L
C
g
Figure 6: Some of the Causal Flip counterfactuals (see Section 3.4 and Eq.
6
obtained by each
method separated into two different subcategories.
pixel space completely and work directly on z, this however, would produce explanations outside the image domain and therefore, uninterpretable by humans, which ultimately not very useful. It could be argued that the generator used in this work could be modified to yield better image reconstructions given any z, however this will always be hindered by the aforementioned limitation. More generally, most methods rely on some some sort of latent decomposition in order to search for counterfactuals in a latent space. However, it still not clear how the true latent variables of the data generating process are not identifiable (Locatello et al. 22019). In this work we circumvent this problem by using synthetic dataset. On the other hand, Khemakhem et al. (2020) showed that, with further assumptions, it is possible to identify the latent variables (Khemakhem al.) 2020). Moreover, in a temporal setup, it is possible to identify which of these latent variables are the causal ones (Lachapelle et al. (022). Finally, in a multi-task setup where distribution shift occurs, it is possible to identify which variables are robust to distributions shift and hence, likely to be the causal ones. In summary, although it would be possible to approximate the true latent factors in some cases, would require making additional assumptions about the data.
We make an effort to establish a fair, principled metric that is useful. However, this metric does not depict all the properties of an explainer such as fragility or speed. It is possible, albeit unlikely, that our definition of useful/informmatie explanation might not always align with that of the user. Lastly, the generator we use in this work can generate images with certain implicit biases. However, please note that our benchmark a tool to evaluate properties of explainers that are impossible to evaluate without a synthetic setup.
A.6 ETHICAL CONCERNS
Research in explainable AI is crucial for safe deployment of machine learning solutions in real life. In this work, we have shown that the lack of principled evaluation of such systems has slowed-down advancements in the field. These findings should not discourage future research in explainable AI, on the contrary, we hope that our work unblocks the current state of the art and spurs further progress.
On the other hand, the research described in this paper does not (i) directly facilitate injury to living beings, (ii) raise safety, privacy or security concerns, (iii) raise human rights concerns, (iv) have
18

---

Under review as a conference paper at ICLR 2023
detriment effect on people's livelihood or economic security, (v) develop extend harmful forms of surveillance, (vi) severely damage the environment, or (vii) deceive people in ways that cause harm.
Lastly, our rescarch does not human-derived data, has involved extensive annotation by human research participants. The synthetic dataset used in this paper has not been deprecated for technical, legal, ethical reasons.
19


Reviews:
[{"review_id": "bp5GtJZeJYw", "reviewer": "Reviewer_jg6E", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel way to compare counterfactual explanation methods, which \"explain\" why a classifier made a certain prediction by highlighting nearby points in the domain with a different outcome. Their proposal involves a synthetic dataset in which the latent causal drivers of the outcome are \"known,\" and greedily constructing a set of explanations which are \"orthogonal\" or \"complementary\" to each other. This method is built from the premise that \"The goal for counterfactual generation methods is to find all the attributes that make a classifier behave differently from a causal classifier\" and \"a good explainer should return a ... diverse set of explanations.\" They use their method to benchmark existing methods for generating counterfactuals.", "summary_of_the_review": "I believe this work is interesting and worthy of publication, however I think it could use some polishing, and perhaps a redistribution of emphasis (i.e. providing more results in the body of the paper rather than delegating to the appendix and briefly summarizing them at the end)", "strength_and_weaknesses": "Strengths: The authors provide lots of helpful exposition on why they proposed the method in the way that they did, and are clear about their stance on what counterfactual explanations should be. The insights they draw at the end are interesting; for instance, \"explainers fail to consistently find more than one [attribute to alter to generate interesting counterfactuals].\"\n\nWeaknesses: The premise of the paper posits VERY strong assumptions on the *purpose* of counterfactual explanations: to be diverse, and to illustrate tensions with an optimal causal model. Though I don't deny these are *nice* qualities, I think their introduction could be clarified to assert *why* those properties are desirable (and why they did not consider other qualities). It may seem self explanatory, but really break it down: Why is it useful to know when a model is not acting \"causally\"??\n\nFurther, although their conclusions from applying their benchmark to explanation methods are interesting and believable, they seem to involve a bit of speculation, sometimes providing \"because\" explanations for the results they see that don't seem to have been directly experimentally validated. Sometimes, I felt the authors went off on tangents not totally necessary for the paper; for instance I am not sure why Proposition 1 is important.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "X-0XaEiPo4s", "reviewer": "Reviewer_pgaK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors suggest a data generating process to evaluate the quality of counterfactual explanations with respect to factors of variation and propose two evaluation measures. The goal of the evaluation measures is to evaluate to what extent different explanation methods are (1) successfully changing the predicted label to the target label, and (2) to measure to what extent these methods generate diverse counterfactual explanations.\nFor their first measure the authors propose to use a generative model, where they model, three latent factors (causal, correlational and independent factors). While all three latent factors influence the observed features (X), only the causal latent factors truly influence the target variable (Y). Using this data generating process the authors define a successful counterfactual explanation to be one which relies on the causal as opposed to the correlational or independent features.\nThe second evaluation measure counts how many orthogonal or complementary explanations were output by any of the counterfactual explanation methods. The authors suggest that diversity of counterfactual explanations is best accessed by non-trivially diversifying explanations. To achieve this, the authors suggest using orthogonality between any two explanation vectors as a measure for non-trivial diversity. \nFinally, the authors use their diversity measure to evaluate the performance of 6 different counterfactual explanation methods on a synthetic dataset. \n", "summary_of_the_review": "Overall, the paper presents some interesting and novel ideas, which unfortunately are not well presented and evaluated. ", "strength_and_weaknesses": "Strengths:\n1. The addressed questions are timely and interesting to the XAI community.\n2. The proposed idea for deriving evaluation measures for counterfactual explanations is sensible.\n\nWeaknesses:\n1. The generative model and specifically the dependencies represented in the model (Fig. 1 (a)) are not well-motivated.\n2. The evaluation is weak: (a) the authors use only one (black-and white) synthetic dataset, (b) the different evaluation choices are not well motivated, (c) it is not clear which results generalize to other settings, (d) the evaluated CE methods behave similarly according to the proposed measure (Eq. 7), so either the authors evaluate very similar methods, or the proposed measures are non-discriminative.\n3. The structure and the clarity of the paper could be considerably improved (see points below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KMNtvP5dWPZ", "reviewer": "Reviewer_WjLU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an evaluation strategy for the counterfactual explanations of images with focus on generative models that operate in the latent space. The paper points out flaws with existing strategies: the diversity criterion can be fooled by making irrelevant changes and the real latent attributes could be misaligned with the learnt latent attributes. The proposed metric works by disentangling the causal vs. non-causal relationships in the latent space. Experiments are reported with several methods.", "summary_of_the_review": "The paper does not meet the bar for ICLR. The solution is applicable to only very restrictive settings and the main assumptions are not backed up with arguments. The paper is also quite hard to follow at several places. ", "strength_and_weaknesses": "### Strengths\nLack of reliable evaluation methods for counterfactuals (and explanations in general) is indeed an important problem.\n\n### Weaknesses\nThere are three main weaknesses in the paper: 1) The setting in the paper is overly restrictive and it is not clear how useful the proposed metric is in the real world. 2) Some of the assumptions taken in the paper are not backed up with supportive arguments and seem at odds with what one would expect from explainers. 3) The clarity of writing can be significantly improved. For these reasons, the paper is not ready yet. Please see detailed comments and questions below:\n\n1. First of all, the focus is mainly on the synthetic synbols dataset where the real latent attributes are available and the data generation process can be controlled. On the other hand, learning these two entities constitutes one of the main challenges in learning. While this does not mean that simplified synthetic settings should not be studied, the paper does not provide any indication on how the takeaways from here would generalize to real datasets. \n\n2. Some of the basis assumptions are not clear. In section 3.2, why is it more useful to perturb \"non-causal\" attributes? The model output could be sensitive to causal or non-causal attributes. A faithful explainer should reflect the true behavior of the model. Similarly, the concept of “trivial” explanation is not very clear. Assuming that the “oracle” here refers to a human, there could be perturbations that do not change the human’s prediction, but do change the prediction of the model. For instance, the model might change its prediction based on the brightness of the image (while the human will not). That could be deemed a valid counterfactual explanation since it correctly reflects the behavior of the model. \n\n3. The paper correctly points out issues with existing diversity metrics in the literature. However, I am not sure if using a single “principled metric to compare counterfactual explanation methods” is the answer. One could always use different metrics to study different aspects of explanation quality. Or does the paper mean to make the point that a single metric can evaluate all relevant aspects of an explanation?\n\n4. There has already been some work on considering causality when generating counterfactual explanations (https://arxiv.org/pdf/2002.06278.pdf). It would be good to contrast the approach in Sections 3.3 and 3.4 with the existing work.\n\n5. It is fine to focus on generative models for counterfactual explanations. However, note that there exist other methods for generating counterfactuals that do not rely on the latent attributes see for instance (https://arxiv.org/abs/1910.08485) and (https://arxiv.org/pdf/1912.09405.pdf).\n\n6. The exposition could be improved at several places. For instance, what is \"the oracle\". What is $\\hat{h}$ and how is it different from $\\hat{f}$. Since the paper introduces quite a bit of terminology, I would suggest putting all the terms together in a single table so that the reader can easily look it up, and clearly defining the input / outputs of different functions. \n\n7. It would help to differentiate from the work of Zhou et al (https://arxiv.org/abs/2104.14403) which has a very similar goal: adding “ground truth“ perturbations to the images that the explainers should pick up on.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IhHQYzytr3", "reviewer": "Reviewer_e2UL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at establishing a comprehensive definition of counterfactuals explanations. After reviewing related work, it establishes the foundations for establishing their process: data generation, counterfactual generation, classifiers, and evaluation. Their approach is compared to a number of baselines on one particular dataset which they can control explicitly, and a number of interesting conclusions are drawn from the experiments. ", "summary_of_the_review": "This paper has made some interesting contributions to the literature. I found it difficult to understand whether the conclusions are can be applied broadly to explainability in general, or whether it's particular to the experimental context explored. I think that the main contributions are interesting and useful too, but that some discussion around real-world and practical deployment was lacking. ", "strength_and_weaknesses": "Strengths: \n- Highlights some weaknesses of counterfactual evaluation, and proposes interesting solutions to important gaps in the literature. \n- I found the conclusion that different counterfactual explaners in the literature perform similarly to be particularly interesting. I would welcome additional experiments and datasets to confirm this more broadly. \n- The contribution of the paper offering a more comprehensive definition of good counterfactual is very appealing\n\nWeaknesses\n- Several aspects (e.g. FN, FNF, TF, etc) were defined, but I didn't feel the rationale behind why these are useful and relevant properties for counterfactual explanations was sufficiently covered. \n- I would have welcomed some additional discussion about $z_causal$. I understand how you have access to this in the experiments (synbols), but in a more general context, how hard is it to get acces to these traits? Is there room for noisy estimates on these? \n- Similar to the above, I would have welcomed discussion about the dependency on oracles for the conclusions. \n- I wasn't convinced that the results are generally applicable, particularly given the narrow scope of the experiments. The experimental section focuses on one dataset that is rather tightly controlled. This helps us understand the dynamics of the problem very well, but I don't think the discussion allows me to see this as more than experiments on toy contexts. This would not be too problematic, but I feel like the title and conclusions claim a lot based on the experiments. I know that in the appendix some real-world considerations are discussed, but to me these are far too important to leave in the apendix. \n- For me, there are missing links between the results as presented here and the general problem. For me, a section or paragraph illustrating highlighting opportunities for readers to deploy the lessons of this paper to new environments would have been useful. The results are empirical on a well controlled environment. How would this be taken forward? \n\nMisc. & smaller points\n- I found the notation difficult to follow, which hindered me understanding the major contributions. I'm slightly unfamiliar with the literature, so this may be usual convention. \n- The legend on Fig 3 is far too small", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject