SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mathieu Serrurier
Franck Mamalet2
Thomas Fel3,4
Louis Béthune<sup>1</sup>
Thibaut Boissin
<sup>1</sup>Université Paul-Sabatier, IRIT, Toulouse, France
$^{2}$ Institut de Recherche Technologique Saint-Exupéry, Toulouse, France
$^{3}$ Carney Institute for Brain Science, Brown University, USA
<sup>4</sup>Innovation & Research Division, SNCF, France
Input gradients have a pivotal role in a variety of applications, including adversarial attack algorithms for evaluating model robustness, explainable AI techniques for generating Saliency Maps, and counterfactual explanations. However, Saliency Maps generated by traditional neural networks are often noisy and provide limited insights. In this paper, we demonstrate that, on the contrary, the Saliency Maps of 1-Lipschitz neural networks, learned with the dual loss of an optimal transportation problem, exhibit desirable XAI properties: They are highly concentrated on the essential parts of the image with low noise, significantly outperforming state-of-the-art explanation approaches across various models and metrics. We also prove that these maps align unprecedentedly well with human explanations on ImageNet. To explain the particularly beneficial properties of the Saliency Map for such models, we prove this gradient encodes both the direction of the transportation plan and the direction towards the nearest adversarial attack. Following the gradient down to the decision boundary is no longer considered an adversarial attack, but rather a counterfactual explanation that explicitly transports the input from one class to another. Thus, Learning with such a loss jointly optimizes the classification objective and the alignment of the gradient, i.e. the Saliency Map, to the transportation plan direction. These networks were previously known to be certifiably robust by design, and we demonstrate that they scale well for large problems and models, and are tailored for explainability using a fast and straightforward method.
The Lipschitz constant of a function expresses the extent to which the output may vary for a small shift in the input. As a composition of numerous functions, the Lipschitz constant of a neural network can be arbitrarily high, particularly when trained for a classification task [8]. Adversarial attacks [45] exploit this weakness by selecting minor modifications, such as imperceptible noise, for a given example to change the predicted class and deceive the network. Consequently, Saliency Maps [59] - gradient of output with respect to the input -, serve as the basis for most adversarial attacks and often highlight noisy patterns that fool the model instead of meaningful modifications, rendering them generally unsuitable for explaining model decisions. Therefore, several methods requiring more complex computations, such as SmoothGrad [61], Integrated Gradient [64], or Grad-CAM [55], have been proposed to provide smoother explanations. Recently, the XAI community has investigated the link between explainability and robustness and proposed methods and metrics
accordingly [35, 12, 42, 53]. However, the reliability of those automatic metrics can be compromised by artifacts introduced by the baselines [35, 63, 31, 38, 30], and there is no conclusive evidence demonstrating their correlation with the human understanding of explanations. To address this, a study by [22] suggests completing those metrics with the alignment between attribution methods and human feature importance using the ClickMe dataset [43].
In [56], authors propose to address the weakness with respect to adversarial attacks by training 1-Lipschitz constrained neural networks with a loss that is the dual of an optimal transport optimization problem, called hKR (Eq. 1). The resulting models have been shown to be robust with a certifiable margin. We refer to these networks as Optimal Transport Neural Networks (OTNN) hereafter.
In this paper, we demonstrate that OTNNs exhibit valuable explainability properties. Our experiments reveal that OTNN Saliency Maps significantly outperform various attribution methods for unconstrained networks across all tested state-of-the-art Explainable XAI metrics. This improvement is consistent across toy datasets, large image datasets, binary, and multiclass problems. Qualitatively, OTNN Saliency Maps concentrate on crucial image regions and generate less noise than maps of unconstrained networks, as illustrated in Figure 1. Figure 1.c presents the Saliency Maps of an OTNN based on ResNet50 alongside its unconstrained vanilla counterpart, both trained on ImageNet [16]. In the unconstrained case, the Saliency Maps appear sparse and uninformative, with the most critical pixels often located outside the subject. Conversely, the OTNN Saliency Map is less noisy and highlights significant image features. This distinction is emphasized in Figure 1.d), comparing the feature visualization of the two models. Feature visualization extracts the inverse prototypical image for a given class using gradient ascent [48, 47]. The results for vanilla ResNet are noticeably noisy, making class identification difficult. In contrast, feature visualization with OTNN yields clearer results, displaying easily identifiable features and classes (e.g., goldfish, butterfly, and medusa). Furthermore, modifying the image following the gradient direction provides an interpretable method for altering the image to change its class. Pictures 1.a) and 1.b) display the original image, the gradient direction, and the transformation following the gradient direction (refer to Section 4 for details). We observe explicit structural changes in the images, transforming them into an example of another class in both multiclass (MNIST) and high-resolution multi-labels cases (e.g., smile/not smile and blond hair/not blond hair classification). Lastly, a large-scale human experiment demonstrates that these maps are remarkably aligned with human attribution on ImageNet (Fig. 4).
We provide a theoretical justification for the well-behaved OTNN Saliency Maps. Building upon the fact that OTNNs encode the dual formulation of the optimal transport problem, we prove that the gradient of the optimal solution at a given point  $x$  is both (i) in the direction of the nearest adversarial example on the decision boundary, and (ii) in the direction of the image of  $x$  according to the underlying transport plan. This implies that adversarial attacks for an OTNN are equivalent to traversing the optimal transport path which can be achieved by following the gradient. Consequently, the resulting modification serves both as an adversarial attack and a counterfactual explanation, explaining why the decision was A and not B [40]. An optimal transport plan between two classes can be interpreted as a global approach for constructing counterfactuals, as suggested in [11, 15]. These counterfactuals may not correspond to the smallest transformation for a given input sample but rather the smallest transformation on average when pairing points from two classes. A consequence of this property is that the Saliency Map of an OTNN for an image indicates the importance of each pixel in the modifications needed to change the class. It is worth noting that several methods based on GAN [36] or causality penalty [37] produce highly realistic counterfactual images. However, the objective of our paper is not to compete with the quality of these results, but rather to demonstrate that OTNN Saliency Maps possess both theoretical and empirical foundations as counterfactual explanations.
We summarize our contributions as follows: first, after introducing the background on OTNN and XAI, we establish several properties of the gradient of an OTNN with respect to adversarial attacks, decision boundaries, and optimal transport. Second, we establish that the optimal transport properties of OTNN's gradient lead to a reinterpretation of adversarial attacks as counterfactual explanations, consequently endowing the Saliency Map with the favorable XAI properties inherent in these explanations. Third, our experiments support the theoretical results, showing that metric scores are higher for most of the XAI methods on OTNN compared to unconstrained neural networks. Additionally, we find that the Saliency Map for OTNN achieves top-ranked scores on XAI metrics compared to more sophisticated XAI methods, and is equivalent to Smoothgrad. Lastly, drawing from [22], we emphasize that OTNNs are naturally and remarkably aligned with human explanations, and we present several examples of gradient-based counterfactuals obtained with OTNNs.
Figure 1: Illustration of the beneficial properties of OTNN gradients. Examples a) and b) show that the gradients naturally provide a direction that enables the generation of adversarial images - a theoretical justification based on optimal transport is provided in Section 3. By applying the gradient  $\pmb{x}^{\prime} = \pmb{x} - t\nabla_{\pmb{x}}\pmb{f}(\pmb{x})$  to the original image  $\pmb{x}$  (on the left), any digit from MNIST can be transformed into its counterfactual  $\pmb{x}^{\prime}$  (e.g., turning a 0 into a 5). In b), we illustrate that this approach can be applied to larger datasets, such as Celeb-A, by creating two counterfactual examples for the closed-mouth and blonde classes. In c), we compare the Saliency Map of a classical model with those of OTNN gradients, which are more focused on relevant elements. Finally, in d), we show that following the gradients of OTNN could generate convincing feature visualizations that ease the understanding of the model's features.
1-Lipschitz Neural network and optimal transport: Consider a classical supervised machine learning binary classification problem on  $(\Omega, \mathcal{F}, \mathbb{P})$  - the underlying probability space - where  $\Omega$  is the sample space,  $\mathcal{F}$  is a  $\sigma$ -algebra on  $\Omega$ , and  $\mathbb{P}$  is a probability measure on  $\mathcal{F}$ . We denote the input space  $\mathcal{X} \subseteq \mathbb{R}^d$  and the output space  $\mathcal{Y} = \{\pm 1\}$ . Let input data  $\boldsymbol{x}: \Omega \to \mathcal{X}$  and target label  $y: \Omega \to \mathcal{Y}$  are random variables with distributions  $P_x, P_y$ , respectively. The joint random vector  $(x, y)$  on  $(\Omega, \mathcal{F})$  has a joint distribution  $P$  defined over the product space  $\mathcal{X} \times \mathcal{Y}$ . Moreover, let  $\mu = P(x|y = 1)$  and  $\nu = P(x|y = -1)$  the conditional probability distributions of  $x$  given the true label. We assume that the supports of  $\Omega$ ,  $\mu$  and  $\nu$  are compact sets.
A function  $f: \mathcal{X} \to \mathbb{R}$  is a 1-Lipschitz function over  $\mathcal{X}$  (denoted  $Lip_{1}(\mathcal{X})$ ) if and only if  $\forall (\boldsymbol{x}, \boldsymbol{z}) \in \mathcal{X}^{2}, \| \boldsymbol{f}(\boldsymbol{x}) - \boldsymbol{f}(\boldsymbol{z}) \| \leq \| \boldsymbol{x} - \boldsymbol{z} \|$ . 1-Lipschitz neural networks have received a lot of attention, especially due to the link with adversarial attacks. They provide certifiable robustness guarantees [32, 49], improve the generalizations [62] and the interpretability of the model [67]. The simplest way to constrain a network to be in  $Lip_{1}(\mathcal{X})$  is to impose this 1-Lipschitz property to each layer. Frobenius normalization [54], or spectral normalization [46] can be used for linear layers, and can also be extended, in some situations, to orthogonalization [41, 1, 66, 6].
Optimal transport, 1-Lipschitz neural networks, and binary classification were first associated in the context of Wasserstein GAN (WGAN) [7]. The discriminator of a WGAN is the solution to the Kantorovich-Rubinstein dual formulation of the 1-Wasserstein distance [69], and it can be regarded as a binary classifier with a carefully chosen threshold. Nevertheless, it has been demonstrated in [56] that this type of classifier is suboptimal, even on a toy dataset. In the same paper, the authors address the suboptimality of the Wasserstein classifier by introducing the hKR loss  $\mathcal{L}^{hKR}$ , which adds a hinge regularization term to the Kantorovich-Rubinstein optimization objective:
$$
\mathcal {L} _ {\lambda , m} ^ {h K R} (\boldsymbol {f}) = \underset {\boldsymbol {x} \sim \nu} {\mathbb {E}} [ \boldsymbol {f} (\boldsymbol {x}) ] - \underset {\boldsymbol {x} \sim \mu} {\mathbb {E}} [ \boldsymbol {f} (\boldsymbol {x}) ] + \lambda \underset {(\boldsymbol {x}, y) \sim P} {\mathbb {E}} (m - y \boldsymbol {f} (\boldsymbol {x})) _ {+} \tag {1}
$$
where  $m > 0$  is the margin, and  $(z)_{+} = max(0,z)$ . We note  $\pmb{f}^{*}$  the optimal minimizer of  $\mathcal{L}_{\lambda ,m}^{hKR}$ . The classification is given by the sign of  $\pmb{f}^{*}$ . In the following, the 1-Lipschitz neural networks that minimize  $\mathcal{L}_{\lambda ,m}^{hKR}$  will be denoted as OTNN. Given a function  $\pmb{f}$ , a classifier based on  $\mathrm{sign}(\pmb {f})$  and an example  $\pmb{x}$ , an adversarial example is defined as follows:
$$
a d v (\boldsymbol {f}, \boldsymbol {x}) = \underset {\boldsymbol {z} \in \mathcal {X}} {\arg \min } \| \boldsymbol {x} - \boldsymbol {z} \| \quad s. t. \operatorname {s i g n} (\boldsymbol {f} (\boldsymbol {z})) \neq \operatorname {s i g n} (\boldsymbol {f} (\boldsymbol {x})). \tag {2}
$$
Since  $f^{*}$  is a 1-Lipschitz function,  $|f^{*}(\pmb{x})|$  is a certifiable lower bound of the robustness of the classification of  $\pmb{x}$  (i.e.  $\forall \pmb{x} \in \mathcal{X}, |f^{*}(\pmb{x})| \leq ||\pmb{x} - \text{adv}(\pmb{f}^{*}, \pmb{x})||$ ). The function  $f^{*}$  has the following properties [56] ( $i$ ) if the supports of  $\mu$  and  $\nu$  are disjoint (separable classes) with a minimal distance of  $\epsilon > 0$ , then for  $m < 2\epsilon$ ,  $f^{*}$  achieves 100% accuracy; ( $ii$ ) minimizing  $\mathcal{L}^{hKR}$  is still the dual formulation of an optimal transport problem (see appendix for more details).
Explainability and metrics: Attribution methods aim to explain the prediction of a deep neural network by pointing out input variables that support the prediction – typically pixels or image regions for images – which lead to importance maps. Saliency [59] was the first proposed white-box attribution method and consists of back-propagating the gradient from the output to the input. The resulting absolute gradient heatmap indicates which pixels affect the most the decision score. However, this family of methods suffers from problems inherent to the gradients of standard models. Methods such as Integrated Gradient [64] and SmoothGrad [61] partially address this issue by accumulating gradients, either along a straight interpolation path from a baseline state to the original image or from a set of points close to the original image obtained after adding noise but multiply the computational cost by several orders of magnitude. These methods were then followed by a plethora of other methods using gradients such as Grad-cam [55] or Input Gradient [4]. All rely on the gradient calculation of the classification score. Finally, other methods – sometimes called black-box attribution methods – do not involve the gradient and rely on perturbations around the image to generate their explanations [50, 20].
However, it is becoming increasingly clear that current methods raise many issues [2, 33, 60] such as confirmation bias: it is not because the explanations make sense to humans that they reflect the evidence of the prediction. To address this challenge, a large number of metrics were proposed to provide objective evaluations of the quality of explanations. Deletion and Insertion methods [50] evaluate the drop in accuracy when important pixels are replaced by a baseline.  $\mu$  Fidelity method [9] evaluates the correlation between the sum of importance scores of pixels and the drop of the score when removing these pixels. In parallel, a growing literature relies on model robustness to derive new desiderata for a good explanation [35, 12, 42, 53, 21]. In addition, [35] showed that some of these metrics also suffer from a bias due to the choice of the baseline value and proposed a new metric called Robustness-Sr. This metric assesses the ease to generate an adversarial example when the attack is limited to the important variables proposed by the explanation. Other metrics consider properties such as generalizability, consistency [23], or stability [74, 9] of explanation methods. A recent approach [43] aims to evaluate the alignment between attribution methods and human feature importance across 200,000 unique ImageNet images (called ClickMe dataset). The alignment between DNN Saliency and human explanations is quantified using the mean Spearman correlation, normalized by the average inter-rater alignment of humans.
These works on explainability metrics have also initiated the emergence of links between the robustness of models and the quality of their explanations [14, 72, 57, 58, 18, 19]. In particular,[23] claimed that 1-Lipschitz networks explanations have better metrics scores. But this study was not on OTNNs and was limited to their proposed metrics.
To end with, recent literature is focusing on counterfactual explanations [70, 68] methods, providing information on "why the decision was A and not B". Several properties are desirable for these counterfactual explanations[68]: Validity (close sample and in another class), Actionability, Sparsity, Data Manifold closeness, and Causality. The three last properties are generally not covered by standard adversarial attacks and complex methods have been proposed [28, 52, 71]. Since often a causal model is hard to fully-define, recent papers [11, 15] have proposed a definition of counterfactual based on optimal transport easier to compute and that can sometimes coincide with causal model based ones. We will rely on this theoretical definition of counterfactuals.
3 Theoretical properties of OTNN gradient
In this section, we extend the properties of the OTNNs to the explainability framework, all the proofs are in the appendix A.1. We note  $\pi$  the optimal transport plan corresponding to the minimizer of  $\mathcal{L}_{\lambda,m}^{hKR}$ . In the most general setting,  $\pi$  is a joint distribution over  $\mu, \nu$  pairs. However when  $\mu$  and  $\nu$  admit a density function [51] with respect to Lebesgue measure, then the joint density describes a deterministic mapping, i.e. a Monge map. Given  $\pmb{x} \sim \mu$  (resp.  $\nu$ ) we note  $z = \gamma_{\pi}(\pmb{x}) \in \nu$  (resp.  $\mu$ ) the image of  $\pmb{x}$  with respect to  $\pi$ . When  $\pi$  is not deterministic (on real datasets that are defined as a discrete collection of Diracs), we take  $\gamma_{\pi}(\pmb{x})$  as the point of maximal mass with respect to  $\pi$ .
Proposition 1 (Transportation plan direction) Let  $\pmb{f}^*$  an optimal solution minimizing the  $\mathcal{L}_{\lambda,m}^{hKR}$ . Given  $\pmb{x} \sim \mu$  (resp.  $\nu$ ) and  $\pmb{z} = \gamma_{\pi}(\pmb{x})$ , then  $\exists t \geq 0$  (resp.  $t \leq 0$ ) such that  $\gamma_{\pi}(\pmb{x}) = \pmb{x} - t\nabla_{\pmb{x}}\pmb{f}^*(\pmb{x})$  almost surely.
This proposition also holds for the Kantorovich-Rubinstein dual problem without hinge regularization, demonstrating that for  $\pmb{x} \sim P$ , the gradient  $\nabla_{\pmb{x}} \pmb{f}^{*}(\pmb{x})$  indicates the direction in the transportation plan almost surely.
Proposition 2 (Decision boundary) Let  $\mu$  and  $\nu$  two distributions with disjoint supports with minimal distance  $\epsilon$  and  $f^{*}$  an optimal solution minimizing the  $\mathcal{L}_{\lambda,m}^{hKR}$  with  $m < 2\epsilon$ . Given  $\pmb{x} \sim P$ ,  $\pmb{x}_{\delta} = \pmb{x} - \pmb{f}^{*}(\pmb{x}) \nabla_{\pmb{x}} \pmb{f}^{*}(\pmb{x}) \in \partial \mathcal{X}$  where  $\partial \mathcal{X} = \{\pmb{x}' \in \mathcal{X} | \pmb{f}^{*}(\pmb{x}') = 0\}$  is the decision boundary (i.e. the 0 level set of  $f^{*}$ ).
Experiments suggest this probably remains true when the supports of  $\mu$  and  $\nu$  are not disjoint. Prop. 2 proves that for an OTN  $\pmb{f}$  learnt by minimizing the  $\mathcal{L}_{\lambda,m}^{hKR}$ ,  $|\pmb{f}(\pmb{x})|$  provides a tight robustness certificate. A direct consequence of 2, is that  $t$  defined in 1 is such that  $|t| \geq |\pmb{f}^*(\pmb{x})|$ .
Corollary 1 Let  $\mu$  and  $\nu$  two separable distributions with minimal distance  $\epsilon$  and  $f^{*}$  an optimal solution minimizing the  $\mathcal{L}_{\lambda,m}^{hKR}$  with  $m < 2\epsilon$ , given  $\pmb{x} \sim P$ ,  $\text{adv}(\pmb{f}^{*},\pmb{x}) = \pmb{x}_{\delta}$  almost surely, where  $\pmb{x}_{\delta} = \pmb{x} - \pmb{f}^{*}(\pmb{x})\nabla_{\pmb{x}}\pmb{f}^{*}(\pmb{x})$ .
This corollary shows that adversarial examples are precisely identified for the classifier based on  $\mathcal{L}_{\lambda,m}^{hKR}$ : the direction given by  $\nabla_{\boldsymbol{x}}\boldsymbol{f}^{*}(\boldsymbol{x})$  and distance by  $|\boldsymbol{f}^{*}(\boldsymbol{x})|*\|\nabla_{\boldsymbol{x}}\boldsymbol{f}^{*}(\boldsymbol{x})\| = |\boldsymbol{f}^{*}(\boldsymbol{x})|$ . In this scenario, the optimal adversarial attacks align with the gradient direction (i.e., FGSM attack [27]). This supports the observations made in [56], where all attacks, such as PGD [45] or Carlini and Wagner [13], applied on an OTNN model, were equivalent to FGSM attacks.
To illustrate these propositions, we learnt a dense binary classifier with  $\mathcal{L}_{\lambda,m}^{hKR}$  to separate two complex distributions, following two concentric Koch snowflakes. Fig.2-a shows the two distributions (blue and orange snowflakes), the learnt boundary (0-levelset) (red dashed line). Fig.2-b,c show for random samples  $\pmb{x}$  from the two distributions, the segments  $[\pmb{x},\pmb{x}_{\delta}]$  where  $\pmb{x}_{\delta}$  is defined in Prop. 2. As expected by Prop. 2,  $\pmb{x}_{\delta}$  points fall exactly on the decision boundary. Besides, as stated in Prop. 1 each segment provides the direction of the image with respect to the transport plan.
Finally, we showed that with OTNN, adversarial attacks are formally known and simple to compute. Furthermore, since we proved that these attacks are along the transportation map, they acquire a meaningful interpretation and are no longer mere adversarial examples exploiting local noise, but rather correspond to the global solution of a transportation problem.
4 Link between OTNN gradient and counterfactual explanations
The vulnerability of traditional networks to adversarial attacks indicates that their decision boundaries are locally flawed, deviating significantly from the Bayesian boundaries between classes. Since the gradient directs towards this anomalous boundary, Saliency Maps [59], given by  $\pmb{g}(\pmb{x}) = |\nabla_{\pmb{x}}\pmb{f}(\pmb{x})|$  fails to represent a meaningful transition between classes and then often lead to noisy explanation (as stated in Section 2).
On the contrary, in the experiments, we will demonstrate that OTNN gradients induce meaningful explanations (Sec. 5). We justify these good properties by building a link with counterfactual
(c)
Figure 2: Level sets of an OTNN classifier  $\pmb{f}$  for two concentric Koch snowflakes (a). The decision boundary (denoted  $\partial \mathcal{X}$ , also called the 0-level set) is the red dashed line. Figure (b) (resp. (c)) represents the translation of the form  $\pmb{x}' = \pmb{x} - \pmb{f}(\pmb{x})\nabla_{\pmb{x}}\pmb{f}(\pmb{x})$  of each point  $\pmb{x}$  of the first class (resp second class).  $[\pmb{x},\pmb{x}']$  pairs are represented by blue (resp. orange) segments.