Question: Could you explain briefly how CLD is different from IM1 and IM2?
Evidence: Since one cannot measure changes in all variables of a real SCM, we leverage the sparse mechanism shift (SMS) hypothesis 5 for justifying a minimality property of counterfac-4. We highlight that our setting is slightly different from baseline works where the target counterfactual classes were similar to the factual classes. e.g. Transforming MNIST digits from 2 → [3, 7] or 4 → [1, 9]. Since we are sampling target classes randomly, their metric values will look lower than in their respective papers. 5. SMS states that a "small distribution changes tend to manifest themselves in a sparse or local way in the causal factorization, that is, they should usually not affect all factors simultaneously." tuals. SMS translates, in our setting, to an intervention will not change many elements of the observed data. Therefore, an important property of counterfactuals is minimality or proximity to the factual observation. We suggest here a new metric entitled counterfactual latent divergence (CLD), illustrated in Fig.(a), that estimates minimality. Note that the metrics IM1 and IM2 from Sec. 5.1 do not take minimality into account. In addition, previous work only used the mean absolute error or 1 distance in the data space for measuring minimality. However, measuring similarity at pixellevel can be challenging as an intervention might change the structure of the image whilst keeping other factors unchanged. In this case, a pixel-level comparison might not be informative about the other factors of variation.
Latent Similarity. Therefore, we choose to measure similarity between latent representation. In addition, we want a representation that captures all factors of variation on the input data. In particular, we train a variational autoencoder (VAE) (Kingma and Welling, 2014) for recovering probabilistic latent representations that capture all factors of variation in the data. The latent spaces computed with the VAE's encoder E φ are denoted as µ i , σ i = E φ (x (1) i ), where subscript i means different samples from x (1) (t = 0). We use the Kullback-Leibler divergence (KL) divergence for measuring the distances between latents. The divergence for a given counterfactual estimation and factual observation pair (x
F ) can, therefore, be denoted as
Relative Measure. However, absolute similarity measures give limited information. Therefore, we leverage class information for measuring minimality whilst making sure that the counterfactual is far enough from the factual class. A relative measure is obtained by estimating the probability of sets of divergence measures between the factual observation and other data points in the dataset (formalized in the Eq. 9) to less or greater than div. In particular, we compare div with the set S class of divergence measures between the factual observation x
(1) F and all data points x (1) in a dataset
class is denoted in set-builder notation 6 with:
The sets S CF and S F are obtained by replacing "class" in S class with the appropriate target class of the counterfactual and factual observation class respectively. The relative measures are: (i) P (S CF ≤ div) for comparing div with the distance between all data points of the counterfactual class and the factual image; and (ii) P (S F ≥ div) for comparing div with the distance between all other data points of the factual class and the factual image. We aim for counterfactuals with low P (S CF ≤ div), enforcing minimality, and low P (S F ≥ div), enforcing bigger distances from the factual class.
CLD. We highlight the competing nature of the two measures P (S CF ≤ div) and P (S F ≥ div) in the counterfactual setting. For example, if the intervention is too minimal -i.e. low P (S CF ≤ div) -the counterfactual will still resemble observations from the factual class -i.e. high P (S F ≥ div). Therefore, the goal is to find the best balance between the two measures. Finally, we define the counterfactual latent divergence (CLD) metric as the LogSumExp of the two probability measures. The LogSumExp operation acts as a smooth approximation of the maximum function. It also penalizes relative peak values for any of the measures when compared to a simple summation. We denote CLD as: CLD = log (exp (P (S CF ≤ div)) + exp (P (S F ≥ div))) .
(10)
We show, using the same experimental setup as in Sec. 5.1, that CLD improves counterfactual estimation when quantitatively compared with the baseline methods, as illustrated in Tab. 1.
Note that the metrics IM1 and IM2 from Sec. 5.1 do not take minimality into account. In addition, previous work (Wachter et al., 2018; Schut et al., 2021) only used the mean absolute error or `1 distance in the data space for measuring minimality. However, measuring similarity at pixellevel can be challenging as an intervention might change the structure of the image whilst keeping other factors unchanged. In this case, a pixel-level comparison might not be informative about the other factors of variation.
Answer: Section 5.2 para. 2 “Note that the metrics IM1 and IM2 from Sec. 5.1 do not take minimality into account.”. We refer the reviewer to section 5.2 of the paper for a complete explanation of why minimality is needed. To summarize, we wish that the estimated counterfactual maintains characteristics that are not causally related to the intervention. In many cases, that cannot be measured directly. Hence, we propose the metric CLD.