Title: On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: 1-Lipschitz neural network

explicability
Abstract: Input gradients have a pivotal role in a variety of applications, including adversarial attack algorithms for evaluating model robustness, explainable AI techniques for generating saliency maps, and counterfactual explanations. However, saliency maps generated by traditional neural networks are often noisy and provide limited insights. In this paper, we demonstrate that, on the contrary, the saliency maps of 1-Lipschitz neural networks, learnt with the dual loss of an optimal transportation problem, exhibit desirable XAI properties:
They are highly concentrated on the essential parts of the image with low noise, significantly outperforming state-of-the-art explanation approaches across various models and metrics. We also prove that these maps align unprecedentedly well with human explanations on ImageNet. To explain the particularly beneficial properties of the saliency map for such models, we prove this gradient encodes  both the direction of the transportation plan and the direction towards the nearest adversarial attack. Following the gradient down to the decision boundary is no longer considered an adversarial attack, but rather a counterfactual explanation that explicitly transports the input from one class to another.  Thus, Learning with such a loss jointly optimizes the classification objective and the alignment of the gradient , i.e. the saliency map, to the transportation plan direction. These networks were previously known to be certifiably robust by design, and we demonstrate that they scale well for large problems and models, and are tailored for explainability using a fast and straightforward method.

On the explainable properties of 1-Lipschitz Neural Networks: An Optimal Transport Perspective

Abstract

Input gradients have a pivotal role in a variety of applications, including adversarial attack algorithms for evaluating model robustness, explainable AI techniques for generating Saliency Maps, and counterfactual explanations. However, Saliency Maps generated by traditional neural networks are often noisy and provide limited insights. In this paper, we demonstrate that, on the contrary, the Saliency Maps of 1-Lipschitz neural networks, learned with the dual loss of an optimal transportation problem, exhibit desirable XAI properties: They are highly concentrated on the essential parts of the image with low noise, significantly outperforming state-of-theart explanation approaches across various models and metrics. We also prove that these maps align unprecedentedly well with human explanations on ImageNet. To explain the particularly beneficial properties of the Saliency Map for such models, we prove this gradient encodes both the direction of the transportation plan and the direction towards the nearest adversarial attack. Following the gradient down to the decision boundary is no longer considered an adversarial attack, but rather a counterfactual explanation that explicitly transports the input from one class to another. Thus, Learning with such a loss jointly optimizes the classification objective and the alignment of the gradient, i.e. the Saliency Map, to the transportation plan direction. These networks were previously known to be certifiably robust by design, and we demonstrate that they scale well for large problems and models, and are tailored for explainability using a fast and straightforward method.

1 Introduction

The Lipschitz constant of a function expresses the extent to which the output may vary for a small shift in the input. As a composition of numerous functions, the Lipschitz constant of a neural network can be arbitrarily high, particularly when trained for a classification task [8]. Adversarial attacks [45] exploit this weakness by selecting minor modifications, such as imperceptible noise, for a given example to change the predicted class and deceive the network. Consequently, Saliency Maps [59] - gradient of output with respect to the input -, serve as the basis for most adversarial attacks and often highlight noisy patterns that fool the model instead of meaningful modifications, rendering them generally unsuitable for explaining model decisions. Therefore, several methods requiring more complex computations, such as SmoothGrad [61], Integrated Gradient [64], or GradCAM [55], have been proposed to provide smoother explanations. Recently, the XAI community has investigated the link between explainability and robustness and proposed methods and metrics

accordingly [35, 12, 42, 53]. However, the reliability of those automatic metrics can be compromised by artifacts introduced by the baselines [35, 63, 31, 38, 30], and there is no conclusive evidence demonstrating their correlation with the human understanding of explanations. To address this, a study by [22] suggests completing those metrics with the alignment between attribution methods and human feature importance using the ClickMe dataset [43].

In [56], authors propose to address the weakness with respect to adversarial attacks by training 1Lipschitz constrained neural networks with a loss that is the dual of an optimal transport optimization problem, called hKR (Eq. 1). The resulting models have been shown to be robust with a certifiable margin. We refer to these networks as Optimal Transport Neural Networks (OTNN) hereafter.

In this paper, we demonstrate that OTNNs exhibit valuable explainability properties. Our experiments reveal that OTNN Saliency Maps significantly outperform various attribution methods for unconstrained networks across all tested state-of-the-art Explainable XAI metrics. This improvement is consistent across toy datasets, large image datasets, binary, and multiclass problems. Qualitatively, OTNN Saliency Maps concentrate on crucial image regions and generate less noise than maps of unconstrained networks, as illustrated in Figure 1. Figure 1.c presents the Saliency Maps of an OTNN based on ResNet50 alongside its unconstrained vanilla counterpart, both trained on ImageNet [16]. In the unconstrained case, the Saliency Maps appear sparse and uninformative, with the most critical pixels often located outside the subject. Conversely, the OTNN Saliency Map is less noisy and highlights significant image features. This distinction is emphasized in Figure 1.d), comparing the feature visualization of the two models. Feature visualization extracts the inverse prototypical image for a given class using gradient ascent [48, 47]. The results for vanilla ResNet are noticeably noisy, making class identification difficult. In contrast, feature visualization with OTNN yields clearer results, displaying easily identifiable features and classes (e.g., goldfish, butterfly, and medusa). Furthermore, modifying the image following the gradient direction provides an interpretable method for altering the image to change its class. Pictures 1.a) and 1.b) display the original image, the gradient direction, and the transformation following the gradient direction (refer to Section 4 for details). We observe explicit structural changes in the images, transforming them into an example of another class in both multiclass (MNIST) and high-resolution multi-labels cases (e.g., smile/not smile and blond hair/not blond hair classification). Lastly, a large-scale human experiment demonstrates that these maps are remarkably aligned with human attribution on ImageNet (Fig. 4).

We provide a theoretical justification for the well-behaved OTNN Saliency Maps. Building upon the fact that OTNNs encode the dual formulation of the optimal transport problem, we prove that the gradient of the optimal solution at a given point x is both (i) in the direction of the nearest adversarial example on the decision boundary, and (ii) in the direction of the image of x according to the underlying transport plan. This implies that adversarial attacks for an OTNN are equivalent to traversing the optimal transport path which can be achieved by following the gradient. Consequently, the resulting modification serves both as an adversarial attack and a counterfactual explanation, explaining why the decision was A and not B [40]. An optimal transport plan between two classes can be interpreted as a global approach for constructing counterfactuals, as suggested in [11, 15]. These counterfactuals may not correspond to the smallest transformation for a given input sample but rather the smallest transformation on average when pairing points from two classes. A consequence of this property is that the Saliency Map of an OTNN for an image indicates the importance of each pixel in the modifications needed to change the class. It is worth noting that several methods based on GAN [36] or causality penalty [37] produce highly realistic counterfactual images. However, the objective of our paper is not to compete with the quality of these results, but rather to demonstrate that OTNN Saliency Maps possess both theoretical and empirical foundations as counterfactual explanations.

We summarize our contributions as follows: first, after introducing the background on OTNN and XAI, we establish several properties of the gradient of an OTNN with respect to adversarial attacks, decision boundaries, and optimal transport. Second, we establish that the optimal transport properties of OTNN's gradient lead to a reinterpretation of adversarial attacks as counterfactual explanations, consequently endowing the Saliency Map with the favorable XAI properties inherent in these explanations. Third, our experiments support the theoretical results, showing that metric scores are higher for most of the XAI methods on OTNN compared to unconstrained neural networks. Additionally, we find that the Saliency Map for OTNN achieves top-ranked scores on XAI metrics compared to more sophisticated XAI methods, and is equivalent to Smoothgrad. Lastly, drawing from [22], we emphasize that OTNNs are naturally and remarkably aligned with human explanations, and we present several examples of gradient-based counterfactuals obtained with OTNNs.

(e

c)

d)

5

7

?

OTTN

ResNet50

OTTN

ResNet50

6

3

5

9

8

6

7

b)

2 Related work

1-Lipschitz Neural network and optimal transport: Consider a classical supervised machine learning binary classification problem on (Ω , F , P ) - the underlying probability space - where Ω is the sample space, F is a σ -algebra on Ω , and P is a probability measure on F . We denote the input space X ⊆ R d and the output space Y = {± 1 } . Let input data x : Ω →X and target label y : Ω →Y are random variables with distributions P x , P y , respectively. The joint random vector ( x , y ) on (Ω , F ) has a joint distribution P defined over the product space X × Y . Moreover, let µ = P ( x | y = 1) and ν = P ( x | y = -1) the conditional probability distributions of x given the true label. We assume that the supports of Ω , µ and ν are compact sets.

A function f : X → R is a 1-Lipschitz functions over X (denoted Lip 1 ( X ) ) if and only if ∀ ( x , z ) ∈ X 2 , || f ( x ) -f ( z ) || ≤ || x -z || . 1-Lipschitz neural networks have received a lot of attention, especially due to the link with adversarial attacks. They provide certifiable robustness guarantees [32, 49], improve the generalizations [62] and the interpretability of the model [67]. The simplest way to constrain a network to be in Lip 1 ( X ) is to impose this 1-Lipschitz property to each layer. Frobenius normalization [54], or spectral normalization [46] can be used for linear layers, and can also be extended, in some situations, to orthogonalization [41, 1, 66, 6].

Optimal transport, 1-Lipschitz neural networks, and binary classification were first associated in the context of Wasserstein GAN (WGAN) [7]. The discriminator of a WGAN is the solution to the Kantorovich-Rubinstein dual formulation of the 1 -Wasserstein distance [69], and it can be regarded as a binary classifier with a carefully chosen threshold. Nevertheless, it has been demonstrated in [56] that this type of classifier is suboptimal, even on a toy dataset. In the same paper, the authors address the suboptimality of the Wasserstein classifier by introducing the hKR loss L hKR , which adds a hinge regularization term to the Kantorovich-Rubinstein optimization objective :

where m > 0 is the margin, and ( z ) + = max (0 , z ) . We note f ∗ the optimal minimizer of L hKR λ,m . The classification is given by the sign of f ∗ . In the following, the 1-Lipschitz neural networks that minimize L hKR λ,m will be denoted as OTNN. Given a function f , a classifier based on sign( f ) and an example x , an adversarial example is defined as follows:

̸

Since f ∗ is a 1-Lipschitz function, | f ∗ ( x ) | is a certifiable lower bound of the robustness of the classification of x (i.e. ∀ x ∈ X , | f ∗ ( x ) | ≤ || x -adv ( f ∗ , x ) || ). The function f ∗ has the following properties [56] ( i ) if the supports of µ and ν are disjoints (separable classes) with a minimal distance of ϵ > 0 , then for m < 2 ϵ , f ∗ achieves 100% accuracy; ( ii ) minimizing L hKR is still the dual formulation of an optimal transport problem (see appendix for more details).

Explainability and metrics: Attribution methods aim to explain the prediction of a deep neural network by pointing out input variables that support the prediction - typically pixels or image regions for images - which lead to importance maps. Saliency [59] was the first proposed whitebox attribution method and consists of back-propagating the gradient from the output to the input. The resulting absolute gradient heatmap indicates which pixels affect the most the decision score. However, this family of methods suffers from problems inherent to the gradients of standard models. Methods such as Integrated Gradient [64] and SmoothGrad [61] partially address this issue by accumulating gradients, either along a straight interpolation path from a baseline state to the original image or from a set of points close to the original image obtained after adding noise but multiply the computational cost by several orders of magnitude. These methods were then followed by a plethora of other methods using gradients such as Grad-cam [55] or Input Gradient [4]. All rely on the gradient calculation of the classification score. Finally, other methods - sometimes called black-box attribution methods - do not involve the gradient and rely on perturbations around the image to generate their explanations [50, 20].

However, it is becoming increasingly clear that current methods raise many issues [2, 33, 60] such as confirmation bias: it is not because the explanations make sense to humans that they reflect the evidence of the prediction. To address this challenge, a large number of metrics were proposed to provide objective evaluations of the quality of explanations. Deletion and Insertion methods [50] evaluate the drop in accuracy when important pixels are replaced by a baseline. µ Fidelity method [9] evaluates the correlation between the sum of importance scores of pixels and the drop of the score when removing these pixels. In parallel, a growing literature relies on model robustness to derive new desiderata for a good explanation [35, 12, 42, 53, 21]. In addition, [35] showed that some of these metrics also suffer from a bias due to the choice of the baseline value and proposed a new metric called Robustness-Sr. This metric assesses the ease to generate an adversarial example when the attack is limited to the important variables proposed by the explanation. Other metrics consider properties such as generalizability, consistency [23], or stability [74, 9] of explanation methods. A recent approach [43] aims to evaluate the alignment between attribution methods and human feature importance across 200,000 unique ImageNet images (called ClickMe dataset). The alignment between DNN Saliency and human explanations is quantified using the mean Spearman correlation, normalized by the average inter-rater alignment of humans.

These works on explainability metrics have also initiated the emergence of links between the robustness of models and the quality of their explanations [14, 72, 57, 58, 18, 19]. In particular,[23] claimed that 1-Lipschitz networks explanations have better metrics scores. But this study was not on OTNNs and was limited to their proposed metrics.

To end with, recent literature is focusing on counterfactual explanations [70, 68] methods, providing information on "why the decision was A and not B". Several properties are desirable for these counterfactual explanations[68]: Validity (close sample and in another class), Actionability, Sparsity, Data Manifold closeness, and Causality. The three last properties are generally not covered by standard adversarial attacks and complex methods have been proposed [28, 52, 71]. Since often a causal model is hard to fully-define, recent papers [11, 15] have proposed a definition of counterfactual based on optimal transport easier to compute and that can sometimes coincide with causal model based ones. We will rely on this theoretical definition of counterfactuals.

3 Theoretical properties of OTNN gradient

In this section, we extend the properties of the OTNNs to the explainability framework, all the proofs are in the appendix A.1. We note π the optimal transport plan corresponding to the minimizer of L hKR λ,m . In the most general setting, π is a joint distribution over µ, ν pairs. However when µ and ν admit a density function [51] with respect to Lebesgue measure, then the joint density describes a deterministic mapping, i.e. a Monge map. Given x ∼ µ (resp. ν ) we note z = γ π ( x ) ∈ ν (resp. µ ) the image of x with respect to π . When π is not deterministic (on real datasets that are defined as a discrete collection of Diracs), we take γ π ( x ) as the point of maximal mass with respect to π .

Proposition 1 (Transportation plan direction) Let f ∗ an optimal solution minimizing the L hKR λ,m . Given x ∼ µ (resp. ν ) and z = γ π ( x ) , then ∃ t ≥ 0 (resp. t ≤ 0 ) such that γ π ( x ) = x -t ∇ x f ∗ ( x ) almost surely.

This proposition also holds for the Kantorovich-Rubinstein dual problem without hinge regularization, demonstrating that for x ∼ P , the gradient ∇ x f ∗ ( x ) indicates the direction in the transportation plan almost surely.

Proposition 2 (Decision boundary) Let µ and ν two distributions with disjoint supports with minimal distance ϵ and f ∗ an optimal solution minimizing the L hKR λ,m with m < 2 ϵ . Given x ∼ P , x δ = x -f ∗ ( x ) ∇ x f ∗ ( x ) ∈ ∂ X where ∂ X = { x ′ ∈ X| f ∗ ( x ′ ) = 0 } is the decision boundary (i.e. the 0 level set of f ∗ ).

Experiments suggest this probably remains true when the supports of µ and ν are not disjoint. Prop. 2 proves that for an OTNN f learnt by minimizing the L hKR λ,m , | f ( x ) | provides a tight robustness certificate. A direct consequence of 2, is that t defined in 1 is such that | t | ≥ | f ∗ ( x ) | .

Corollary 1 Let µ and ν two separable distributions with minimal distance ϵ and f ∗ an optimal solution minimizing the L hKR λ,m with m < 2 ϵ , given x ∼ P , adv ( f ∗ , x ) = x δ almost surely, where x δ = x -f ∗ ( x ) ∇ x f ∗ ( x ) .

This corollary shows that adversarial examples are precisely identified for the classifier based on L hKR λ,m : the direction given by ∇ x f ∗ ( x ) and distance by | f ∗ ( x ) | ∗ ||∇ x f ∗ ( x ) || = | f ∗ ( x ) | . In this scenario, the optimal adversarial attacks align with the gradient direction (i.e., FGSM attack [27]). This supports the observations made in [56], where all attacks, such as PGD [45] or Carlini and Wagner [13], applied on an OTNN model, were equivalent to FGSM attacks.

To illustrate these propositions, we learnt a dense binary classifier with L hKR λ,m to separate two complex distributions, following two concentric Koch snowflakes. Fig.2-a shows the two distributions (blue and orange snowflakes), the learnt boundary ( 0 -levelset) (red dashed line). Fig.2-b,c show for random samples x from the two distributions, the segments [ x , x δ ] where x δ is defined in Prop. 2 . As expected by Prop. 2, x δ points fall exactly on the decision boundary. Besides, as stated in Prop. 1 each segment provides the direction of the image with respect to the transport plan.

Finally, we showed that with OTNN, adversarial attacks are formally known and simple to compute. Furthermore, since we proved that these attacks are along the transportation map, they acquire a meaningful interpretation and are no longer mere adversarial examples exploiting local noise, but rather correspond to the global solution of a transportation problem.

4 Link between OTNN gradient and counterfactual explanations

The vulnerability of traditional networks to adversarial attacks indicates that their decision boundaries are locally flawed, deviating significantly from the Bayesian boundaries between classes. Since the gradient directs towards this anomalous boundary, Saliency Maps [59], given by g ( x ) = |∇ x f ( x ) | fails to represent a meaningful transition between classes and then often lead to noisy explanation (as stated in Section 2).

On the contrary, in the experiments, we will demonstrate that OTNN gradients induce meaningful explanations (Sec. 5). We justify these good properties by building a link with counterfactual

f(c)

中

中

V

C

(a)

(b)

(c)

explanations. Indeed, according to [11, 15], optimal transport plans are potential surrogates or even substitutes to causal counterfactuals. Optimal Transport plan provides for any sample x ∈ µ a sample γ π ( x ) ∈ ν , the closest in average on the pairing process. [15] prove that these optimal transport plans can even coincide with causal counterfactuals when available. Relying on this definition of OT counterfactual, Prop. 1 demonstrates that gradients of the optimal OTNN solution provide almost surely the direction to the counterfactual γ π ( x ) = x -t ∇ x f ∗ ( x ) . Even if t is only partially known, using t = f ∗ ( x ) , we know that x δ is on the decision boundary (Corr. 1) and is both an adversarial attack and a counterfactual explanation and | t | ≥ | f ∗ ( x ) | is on the path to the other distribution. Thus the learning process of OTNNs induces a strong constraint on the gradients of the neural network, aligning them to the optimal transport plan. We claim that is the reason why the simple Saliency Maps for OTNNs have very good properties: We will demonstrate in the Sec 5 that, for the Saliency Map explanations: ( i ) metrics scores are higher or comparable to other explanation methods (which is not the case for unconstrained networks), thus it has higher ranks; ( ii ) distance to other attribution methods such as Smoothgrad is imperceptible; ( iii ) scores obtained on metrics that can be compared between networks are higher than those obtained with unconstrained networks; ( iv ) alignment with human explanations is impressive.

5 Experiments

We conduct experiments with networks learnt on FashionMNIST [73], and 22 binary labels of CelebA [44] datasets, Cat vs Dog (binary classification, 224x224x3 uncentered images), and Imagenet [17]. Note that labels in CelebA are very unbalanced (see Table 2 in Appendix A.2, with for instance less than 5% samples for Mustache or Wearing_Hat ).

Architectures used for OTNNs and unconstrained networks are similar (same number of layers and neurons, a VGG for FashionMNIST and CelebA, a ResNet50 for Cat vs Dog and Imagenet). We also train an alternative of ResNet50 OTNN with twice the number of parameters (50 M). Unconstrained networks use batchnorm and ReLU layers for activation, whereas OTNNs only use GroupSort2 [5, 56] activation. OTNNs are built using the DEEL.LIP 1 library, using Björck orthogonalization projection algorithm for linear layers. Note that several other approaches can be used for orthogonalization without altering the theoretical results; these might potentially enhance experimental outcome scores. The loss functions are cross-entropy for unconstrained networks (categorical for multiclass, and sigmoid for multilabel settings), and hKR L hKR λ,m (and the multiclass variants see appendix A.2.3) for OTNNs. We train all networks with Adam optimizer [39]. Details on architectures and parameters are given in Appendix A.2.

Table 1: Comparison of XAI metrics for different attributions methods and dataset for OTNN and unconstrained networks.

Classification performance: OTNN models achieve comparable results to unconstrained ones, confirming claims of [8]: they reach 88.5% average accuracy on FashionMNIST (Table 6), and 81% (resp. 82%) average Sensitivity (resp. Specificity) over labels on CelebA (Table 6 in Appendix A.3). We use Sensitivity and Specificity for CelebA to take into consideration the unbalanced labels. OTNNs achieve 96% accuracy (98% for the unconstrained version) on Cat vs Dog and 67% (75% for the unconstrained version) on Imagenet. The ResNet50 OTNN with 50M parameters achieves 70% accuracy on Imagenet.

We present the results of quantitative evaluations of XAI metrics to compare the Saliency Map method with other explanation methods on OTNN, and more generally compare XAI explanations methods on these networks and their unconstrained counterparts. On CelebA, we only present the results for the label Mustache , but results for the other labels are similar. Parameters for explanation methods and metrics are given in Appendix A.4. We have chosen to present in Table 1 two SoTA XAI metrics that enable comparison between OTNNs and unconstrained networks. µ Fidelity metric [9] is a well-known method that measures the correlation between important variables defined by the explanation method and the model score decreases when these variables are reset to a baseline state (or replaced by uniform noise). Another important property for explanations is their stability for nearby samples. In [74], the authors proposed Stability metrics based on the L 2 distance. To better evaluate this stability and make it comparable for different models, we replace the L 2 distance by 1 -ρ , ρ being the Spearman rank correlation. Other model-dependent metrics are described in the Appendix. Results from the 50M parameter ResNet OTNN are included in the human alignment study (Fig 4) to illustrate that enhancing the model's complexity can bolster both the accuracy and alignment. The following observations can be drawn from Table 1:

Saliency Map on OTNNs exhibit more fidelity and stability : We confirm and amplify the results in [23]. Table. 1 clearly states that for most of the explanation methods, the µ Fidelity, zero or uniform, is significantly higher for OTNNs. And above all, Saliency Map score for OTNNs is always higher than any other attribution method score for unconstrained models. A similar observation holds for the Stability Spearman rank : OTNN scores are better whatever the attribution method.

Saliency Map method on OTNNs is equivalent to other attribution methods: We observe that the scores from the Saliency Maps and other methods are very similar for OTNN, with Saliency Maps consistently ranking among the top attribution methods. For the unconstrained case, Saliency Maps are occasionally outperformed by other attribution methods. Notably, for the ResNet architecture, attribution methods other than Saliency Maps and GradCAM yield more erratic results for

Saliency

SmoothGrad

(a) OTNN

(b) Unconstrained

fidelity metrics. To highlight these results, we compare the L 2 distance between Saliency Maps and SmoothGrad explanations, as suggested by [2, 23, 65, 26]. The explanation distances for OTNN are significantly lower than for the unconstrained ones and closely approach zero, indicating that for OTNN, averaging over a large set of noisy inputs-as in SmoothGrad-is unnecessary. This is illustrated in Fig.3.

OTNNs explanations are aligned with human ones : Adopting the method presented in [43], using the ClickMe dataset, we follow strictly their experimental methodology and use their code 2 to compute the human feature alignment of OTNN Saliency Maps and compare with the others models tested in [22]- more than 100 recent deep neural networks. In Figure 4, we demonstrate that OTNN models Saliency Maps, which also carries a theoretical interpretation as the direction of the transport plan, is more aligned with human attention than any other tested models and significantly surpasses the Pareto front discovered by [22]. The OTNN model is even more aligned than a ResNet50 model trained with a specific alignment objective, proposed by [22], and called Harmonized ResNet50 . This finding is interesting as it indicates OTNNs are less prone to relying on spurious correlations [25] and better capture human visual strategies for object recognition. The implications of these results are crucial for both cognitive science and industrial applications. A model that more closely aligns with human attention and visual strategies can provide a more comprehensive understanding of how vision operates for humans, and also enhance the predictability, interpretability, and performance of object recognition models in industry settings. Furthermore, the drop in alignment observed in recent models highlights the necessity of considering the alignment of model visual strategies with human attention while developing object recognition models to reduce the reliance on spurious correlations and ensure that our models get things right for the right reasons.

Qualitative results: Using the learnt OTNN on FashionMNIST, CelebA (Mouth Slightly Open label), Cat vs Dog and Imagenet, Fig. 6,5 present the original image, average gradients ∇ x f j over the channels, and images in the direction of the transport plan (Prop. 1), other samples are given in Appendix A.5. We can see that most of the gradients are visually consistent, showing clearly what has to be changed in the input image to modify the class and act as a counterfactual explanation. This is less clear for the Imagenet examples. This could be due to the difficulty of defining a transport plan for each pair of the 1000 classes. However, feature visualizations in Figure 1 show that the internal representation of the classes is still more interpretable than the unconstraint one. More generally, we observe that the gradient gives clear information about how the classifier makes its decision. For instance, for the cat, it shows that the classifier does not need to encode perfectly the concept of a cat, but mainly to identify the color of the eyes and size of the nose.

6 Conclusions and broader impact

In this paper, we study the explainability properties of OTNN (Optimal Transport Neural Networks) that are 1-Lipschitz constrained neural networks trained with an optimal transport dual loss. We

0.7

(%)

0.6

OTNN LARGE

ment

OTNN

Harmonized

0.5

ResNet50

Self-supervised

=

ConvNet

0.4

Robust

eature

Transformer

0.3

Bigger dataset

Harmonized

0.2

iman

1-Lipschitz

ViT

ConvNexT

0.1

InH

Clip

SimCLR

0.0

55

60

65

70

75

80

85

Imagenet accuracy

Trouser

Dress

Pullover

Shirt

Sneaker

Sandal

Bag

Ankle boot

Top

Shirt

Trouser

Coat

establish that the gradient of the optimal solution aligns with the transportation plan direction, and the closest decision boundary point (adversarial example) also lies in this gradient direction at a distance of the absolute value of the network output. Relying on a formal definition of Optimal Transport counterfactuals, we build a link between the OTNN gradients and counterfactual explanations . We thus show that OTNNs loss jointly targets the classification task and induces a gradient alignment to the transportation plan. These beneficial properties of the gradient substantially enhance the Saliency Map XAI method for OTNN s. The experiments show that the simple Saliency Map has top-rank scores on state-of-the-art XAI metrics, and largely outperforms any method applied to unconstrained networks. Besides, as far as we know, our models are the first large provable 1-Lipschitz neural models that match state-of-the-art results on large problems. And even if counterfactual explanations are less compelling on Imagenet, probably due to the complexity of transport for large number of classes , we prove that OTNNs Saliency Maps are impressively aligned to human explanations.

Broader impact. This paper demonstrates the value of OTNNs for critical problems. OTNNs are certifiably robust and explainable with the simple Saliency Map method (highly aligned with human explanations) and have accuracy performances comparable to unconstrained networks.Though OTNNs take 3-6 times longer to train than unconstrained networks, they have similar computational costs during inference. We hope that this contribution will raise a great interest in these OTNN networks.

CelebA

mouth closed

→

→

CelebA

→

→

CelebA

→

→

Cat vs Dog

cat

→

dog

dog

→

cat

Imagenet

cabbage butterfly

→

monarch butterfly

great grey owl

→

bald eagle

Acknowledgments and Disclosure of Funding

This work has benefited from the AI Interdisciplinary Institute ANITI, which is funded by the French 'Investing for the Future - PIA3' program under the Grant agreement ANR-19-P3IA-0004. The authors gratefully acknowledge the support of the DEEL project. 3

References

E. M. Achour, F. Malgouyres, and F. Mamalet. Existence, stability and scalability of orthogonal convolutional neural networks, 2021.

J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 9525-9536, Red Hook, NY, USA, 2018. Curran Associates Inc.

L. Ambrosio and A. Pratelli. Existence and stability results in the L1 theory of optimal transportation , pages 123-160. Springer Berlin Heidelberg, Berlin, Heidelberg, 2003.

M. Ancona, E. Ceolini, A. C. Öztireli, and M. H. Gross. A unified view of gradient-based attribution methods for deep neural networks. CoRR , 2017.

C. Anil, J. Lucas, and R. Grosse. Sorting out Lipschitz function approximation. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 291-301, Long Beach, California, USA, June 2019. PMLR.

A. Araujo, A. J. Havens, B. Delattre, A. Allauzen, and B. Hu. A unified algebraic perspective on lipschitz neural networks. In The Eleventh International Conference on Learning Representations , 2023.

M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 214-223, International Convention Centre, Sydney, Australia, Aug. 2017. PMLR.

L. Béthune, A. González-Sanz, F. Mamalet, and M. Serrurier. The many faces of 1-lipschitz neural networks. CoRR , abs/2104.05097, 2021.

U. Bhatt, A. Weller, and J. M. F. Moura. Evaluating and aggregating feature-based model explanations. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , 2020.

Å. Björck and C. Bowie. An Iterative Algorithm for Computing the Best Estimate of an Orthogonal Matrix. SIAM Journal on Numerical Analysis , 8(2):358-364, June 1971.

E. Black, S. Yeom, and M. Fredrikson. Fliptest: Fairness auditing via optimal transport. CoRR , abs/1906.09218, 2019.

A. Boopathy, S. Liu, G. Zhang, C. Liu, P.-Y. Chen, S. Chang, and L. Daniel. Proper network interpretability helps adversarial robustness in classification. In International Conference on Machine Learning , 2020.

N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) , pages 39-57. IEEE, 2017.

P. Chalasani, J. Chen, A. R. Chowdhury, S. Jha, and X. Wu. Concise explanations of neural networks using adversarial training. In Proceedings of the 37th International Conference on Machine Learning (ICML) , 2020.

L. de Lara, A. González-Sanz, N. Asher, and J.-M. Loubes. Transport-based counterfactual models. arxiv:2108.13025 , 2021.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 , 2009.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

A.-K. Dombrowski, C. J. Anders, K.-R. Müller, and P. Kessel. Towards robust explanations for deep neural networks. volume 121, page 108194, 2022.

C. Etmann, S. Lunz, P. Maass, and C.-B. Schönlieb. On the connection between adversarial robustness and saliency map interpretability. In International Conference on Machine Learning , 2019.

T. Fel, R. Cadène, M. Chalvidal, M. Cord, D. Vigouroux, and T. Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. CoRR , abs/2111.04138, 2021.

T. Fel, M. Ducoffe, D. Vigouroux, R. Cadene, M. Capelle, C. Nicodeme, and T. Serre. Don't lie to me! robust and efficient explainability with verified perturbation analysis. arXiv preprint arXiv:2202.07728 , 2022.

T. Fel, I. Felipe, D. Linsley, and T. Serre. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

T. Fel, D. Vigouroux, R. Cadène, and T. Serre. How Good is your Explanation? Algorithmic Stability Measures to Assess the Quality of Explanations for Deep Neural Networks. In 2022 CVF Winter Conference on Applications of Computer Vision (WACV) , Hawaii, United States, Jan. 2022.

Fel, Thomas and Hervier, Lucas. Xplique: an neural networks explainability toolbox. 2021.

R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

A. Ghorbani, A. Abid, and J. Zou. Interpretation of neural networks is fragile. In Association for the Advancement of Artificial Intelligence , 2017.

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv:1412.6572 [stat.ML] , Dec. 2014.

Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee. Counterfactual visual explanations. In Proceedings of the 36th International Conference on Machine Learning (ICML) , 2019.

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. CoRR , abs/1704.00028, 2017.

P. Hase, H. Xie, and M. Bansal. The out-of-distribution problem in explainability and search methods for feature importance explanations. NeurIPS , 2021.

J. Haug, S. Zürn, P. El-Jiz, and G. Kasneci. On baselines for local feature attributions. arXiv preprint arXiv:2101.00905 , 2021.

M. Hein and M. Andriushchenko. Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation. arXiv:1705.08475 [cs, stat] , May 2017.

J. Heo, S. Joo, and T. Moon. Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems , volume 32, 2019.

C. Hsieh, C. Yeh, X. Liu, P. K. Ravikumar, S. Kim, S. Kumar, and C. Hsieh. Evaluations and methods for explanation through robustness analysis. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

C.-Y. Hsieh, C.-K. Yeh, X. Liu, P. Ravikumar, S. Kim, S. Kumar, and C.-J. Hsieh. Evaluations and methods for explanation through robustness analysis. In International Conference on Learning Representations , 2021.

P. Jacob, É. Zablocki, H. Ben-Younes, M. Chen, P. Pérez, and M. Cord. STEEX: steering counterfactual explanations with semantics. CoRR , abs/2111.09094, 2021.

A.-H. Karimi, B. Schölkopf, and I. Valera. Algorithmic recourse: from counterfactual explanations to interventions. In ACM Conference on Fairness, Accountability, and Transparency , pages 353-362, 2021.

P.-J. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schütt, S. Dähne, D. Erhan, and B. Kim. The (un) reliability of saliency methods. 2019.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

D. Lewis. Causation. Journal of Philosophy , 70(17):556-567, 1973.

Q. Li, S. Haque, C. Anil, J. Lucas, R. B. Grosse, and J. Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. arXiv:1911.00937 , Apr. 2019.

Z. Q. Lin, M. J. Shafiee, S. Bochkarev, M. S. Jules, X. Y . Wang, and A. Wong. Do explanations reflect decisions? a machine-centric strategy to quantify the performance of explainability algorithms. In NIPS , 2019.

D. Linsley, S. Eberhardt, T. Sharma, P. Gupta, and T. Serre. What are the visual features underlying human versus machine vision? In Proceedings of the IEEE International Conference on Computer Vision Workshops , pages 2706-2714, 2017.

Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , Dec. 2015.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. ArXiv , 2017.

T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. ArXiv , abs/1802.05957, 2018.

A. Nguyen, J. Yosinski, and J. Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Visualization for Deep Learning workshop, International Conference in Machine Learning , 2016. arXiv preprint arXiv:1602.03616.

C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill , 2(11):e7, 2017.

H. Ono, T. Takahashi, and K. Kakizaki. Lightweight Lipschitz Margin Training for Certified Defense against Adversarial Examples. arXiv:1811.08080 [cs, stat] , Nov. 2018.

V. Petsiuk, A. Das, and K. Saenko. RISE: randomized input sampling for explanation of black-box models. CoRR , abs/1806.07421, 2018.

G. Peyré and M. Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning , 11(5-6):355-206, 2018.

P. Rodriguez, M. Caccia, A. Lacoste, L. Zamparo, I. Laradji, L. Charlin, and D. Vazquez. Beyond trivial counterfactual explanations with diverse valuable explanations. In Int. Conf in Computer Vision (ICCV) , 2021.

A. Ross, H. Lakkaraju, and O. Bastani. Learning models for actionable recourse. NeurIPS , 2021.

T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. CoRR , abs/1602.07868, 2016.

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision , 128(2):336-359, oct 2019.

M. Serrurier, F. Mamalet, A. González-Sanz, T. Boissin, J.-M. Loubes, and E. Del Barrio. Achieving robustness in classification using optimal transport with hinge regularization. In Conference on Computer Vision and Pattern Recognition (CVPR'21) , 2021.

H. Shah, P. Jain, and P. Netrapalli. Do input gradients highlight discriminative features? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 2046-2059, 2021.

H. Shah, P. Jain, and P. Netrapalli. Do input gradients highlight discriminative features? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 2046-2059. Curran Associates, Inc., 2021.

K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings , 2014.

L. Sixt, M. Granz, and T. Landgraf. When explanations lie: Why many modified bp attributions fail. In International Conference on Machine Learning , 2020.

D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise, 2017.

J. Sokolic, R. Giryes, G. Sapiro, and M. R. D. Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing , 65(16):4265-4280, Aug. 2017.

P. Sturmfels, S. Lundberg, and S.-I. Lee. Visualizing the impact of feature attribution baselines. Distill , 2020.

M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks, 2017.

R. Tomsett, D. Harborne, S. Chakraborty, P. Gurram, and A. Preece. Sanity checks for saliency metrics. In Association for the Advancement of Artificial Intelligence , 2019.

A. Trockman and J. Z. Kolter. Orthogonalizing convolutional layers with the cayley transform. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy, 2018.

S. Verma, J. P. Dickerson, and K. Hines. Counterfactual explanations for machine learning: A review. CoRR , abs/2010.10596, 2020.

C. Villani. Optimal Transport: Old and New . Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008.

S. Wachter, B. Mittelstadt, and C. Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech. , 31:841, 2017.

P. Wang and N. Vasconcelos. Scout: Self-aware discriminant counterfactual explanations. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Z. Wang, M. Fredrikson, and A. Datta. Robust models are more interpretable because attributions look normal, 2022.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv , abs/1708.07747, 2017.

C.-K. Yeh, C.-Y. Hsieh, A. S. Suggala, D. I. Inouye, and P. Ravikumar. On the (in)fidelity and sensitivity of explanations. In NeurIPS , 2019.

A Appendix

A.1 Additional definition and proofs

Let us first recall the optimal transport problem associated with the minimization of L hKR λ,m :

Where Π p λ ( µ, ν ) is the set consisting of positive measures π ∈ M + (Ω × Ω) which are absolutely continuous with respect to the joint measure dµ × dν and dπ x dµ ∈ [ p, p ( m + λ )] , dπ z dν ∈ [1 -p, (1 -p )( m + λ )] . We name π ∗ the optimal transport plan according to Eq.3 and and f ∗ the associated potential function.

Proof of proposition 1 : According to [56], we have

Following the proof of proposition 1 in [29] and [3] we have :

Given x α = α ∗ x +(1 -α ) y , 0 ≤ α ≤ 1

This prove the proposition 1 by choosing t = || x -y || .

Proof of proposition 2 : Let µ and ν two distributions with disjoint support with minimal distance ϵ and f ∗ an optimal solution minimizing the L hKR λ,m with m < 2 ϵ . According to [56], f ∗ is 100% accurate. Since the classification is based on the sign of f we have : ∀ x ∈ µ, f ∗ ( x ) ≥ 0 and ∀ y ∈ ν, f ∗ ( y ) ≤ 0 . Given x ∈ µ and y = tr π ( x ) = x -t ∇ x f ∗ ( x ) and y ∈ ν . According to the previous proposition we have :

since f ∗ ( y ) ≤ 0 we obtain :

almost surely and

So for for α = 1 whe have and then

Since f ∗ is continuous, ∃ t ′ > 0 such that x δ = x -t ′ ∇ x f ∗ ( x ) and f ∗ ( x δ ) = 0 . We have :

and

Then, if f ∗ ( x ) < t ′ we have

which is a contradiction so f ∗ ( x ) = t ′ and

A.2 Parameters and architectures

A.2.1 Datasets

FashionMNIST has 50,000 images for training and 10,000 for test of size 28 × 28 × 1 , with 10 classes.

CelebA contains 162,770 training samples, 19,962 samples for test of size 218 × 178 × 3 . We have used a subset of 22 labels: Attractive , Bald , Big_Nose , Black_Hair , Blond_Hair , Blurry , Brown_Hair , Eyeglasses , Gray_Hair , Heavy_Makeup , Male , Mouth_Slightly_Open , Mustache , Receding_Hairline , Rosy_Cheeks , Sideburns , Smiling , Wearing_Earrings , Wearing_Hat , Wear-ing_Lipstick , Wearing_Necktie , Young .

Note that labels in CelebA are very unbalanced (see Table 2, with less than 5% samples for Mustache or Wearing_Hat for instance). Thus we will use Sensibility and Specificity as metrics.

Cat vs Dog contains 17400 training samples, 5800 test samples of various size.

Imagenet contains 1M training samples, 100 000 samples for test of various size.

preprocessing: For FashionMNIST Images are normalized between [0 , 1] with no augmentation. For CelebA dataset, data augmentation is used with random crop, horizontal flip, random brightness, and random contrast. For imagenet and cat vs dog we use the standart preprocessing of resnet (with no normalization in [0 , 1] )

A.2.2 Architectures

As indicated in the paper, linear layers for OTNN and unconstrained networks are equivalent (same number of layers and neurons), but unconstrained networks use batchnorm and ReLU layer for activation, whereas OTNN only use GroupSort2 [5, 56] activation. OTNN are built using DEEL.LIP 4 library.

1-Lipschitz networks parametrization. Several soltutions have been proposed to set the Lipschitz constant of affine layers: Weight clipping [7] (WGAN), Frobenius normalization [54] and spectral normalization [46]. In order to avoid vanishing gradients, orthogonalization can be done using Björck algorithm [10]. DEEL.LIP implements most of these solutions, but we focus on layers called SpectralDense and SpectralConv2D , with spectral normalization [46] and Björck algorithm [10]. Most activation functions are Lipschitz, including ReLU, sigmoid, but we use GroupSort2 proposed by [5], and defined by the following equation:

Network architectures used for CelebA dataset are described in Table 3.

Network architectures used for FashionMNIST dataset are described in Table 4. The same OTNN architecture is used for MNIST expermentation presented in Fig. 1.

The 1-Lipschitz version of resnet50 is described in Table 5. As the unconstrained version, It has around 25M parameters. For the large version, we simply multiply the number channels in hidden layers by 1 . 5 . The unconstrained version is the standart resnet50 architecture. In the case of imagenet we use the pretrained version provided by tensorflow.

A.2.3 Losses and optimizer

An extension of L hKR to the multiclass case with q classes. has also been proposed in [56] The idea is to learn q 1-Lipschitz functions f 1 , . . . , f q , each component f i being a one-versus-all binary classifier. The loss proposed was the following

with :

̸

This formulation has three main drawbacks: (i) for large number of classes several outputs may have few or no positive sample within a batch leading to slow convergence, (ii) weight of f y ( x ) (the function of the true class) with respect to the other decreases when the number of classes increases, (iii) the expectancy has to be evaluated through the batch, making the loss dependant of the size of the batch.

To overcome these drawbacks, we propose first to replace the Hinge term H with a softmax weighted version. The softmax on all but true class is defined by:

̸

We can define a weighted version of H function:

̸

In this function, the value of f y ( x ) for the true class maintains consistent weight relative to the values of other functions, regardless of the number of classes. α is a temperature parameter. Initially, the softmax behaves like an average as all the values of f k are close. However, during the learning process, as the values of | f k | increase, the softmax transitions to function like a maximum. Similarly, if a low value is chosen for α , the softmax behaves as an average, resulting in a one vs all hKR loss. By choosing a higher value for α , the softmax unbalances the weights. Thus the loss persists as a one vs all hKR but incorporates a re-weighting of the opposing classes for each targeted class.

We also propose a sample-wise and weighted version of the KR part (left term in Eq 4). to get the proposed loss:

̸

It's important to note that this definition only applys to the balanced multiclass case (as in FashionMnist and ImageNet). In the unbalanced scenario, the weight must be rescaled according to the a priori distribution of the classes.

For CelebA, with hyperparameters λ is set to 20, and m = 1 . For FashionMNIST, we use Eq. 5, λ is set to 5 , α = 10 and m = 0 . 5 . For cat vs dog λ is set to 10 and m = 18 . For imagenet λ is set to 500 , α = 200 and m = 0 . 05 .

We train all networks with ADAM optimizer [39]. We use a batch size of 128, 200 epochs , and a fixed learning rate 1e -2 for CelebA. For FashionMNIST we perform 200 epochs with a batch size of 128. We fix the learing rate to 5e -4 for the 50 first epochs, 5e -5 for the epochs 50-75, 1e -6 for the last epochs. For cat vs dog we perform 200 epochs with a batch size of 256. We fix the learing rate to 1e -2 for the 100 first epochs, 1e -3 for the epochs 100-150, 1e -4 for the epochs 150-180 and 1e -9 for the last epochs. For imagenet we perform 40 epochs with a batch size of 512. We fix the learing rate to 5e -4 for the 30 first epochs, 5e -5 for the epochs 30-35, 1e -5 for the epochs 35-38 and 1e -9 for the last epochs.

A.3 Complementary results

A.3.1 FashionMNIST performances and ablation study

Table 6 presents different performance resuts on FashionMNIST. First line is the reference unconstrained network. Second line shows the performances of the new version of L hKR λ,α . Table 6 also shows that the new version of the L hKR λ,m,α in the multiclass case (Eq. 5) outperforms the L hKR λ,m defined in [56] (Eq. 4). Obviously, the accuracy enhancement is obtained at the expense of the robustness. The main interest of this new loss is to provide a wider range in the accuracy/robustness trade-off.

A.3.2 CelebA performances

Table 7 presents the Sensibility and Specificity for each label reached by Unconstrained network and OTNN.

As a reminder, given True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN) samples, Sensitivity (true positive rate or Recall) is defined by:

Specificity (true negative rate) is defined by:

A.4 Complementary explanations metrics

A.4.1 Explanation attribution methods

An attribution method provides an importance score for each input variables x i in the output f ( x ) . The library used to generate the attribution maps is Xplique [24].

For a full description of attribution methods, we advise to read [21], Appendix B. We will only remind here the equations of

Saliency: g ( x ) = |∇ x f ( x ) |

SmoothGrad: g ( x ) = E δ ∼N (0 , I σ ) ( ∇ f ( x + δ ))

SmoothGrad is evaluated on N = 50 samples on a normal distribution of standard deviation σ = 0 . 2 around x . Integrated Gradient [64], noted IG, is also evaluated on N = 50 samples at regular intervals. Grad-CAM [55], noted GC, is classically applied on the last convolutional layer.

A.4.2 XAI metrics

For the experiments we use four fidelity metrics, evaluated on 1000 samples of test datasets:

Deletion [50]: it consists in measuring the drop of the score when the important variables are set to a baseline state. Formally, at step k , with u the k most important variables according to an attribution method, the Deletion ( k ) score is given by:

The AUC of the Deletion scores is then measured to compare the attribution methods ( ↓ is better). The baseline x 0 can either be a zero value ( Deletion-zero ), or a uniform random value ( Deletion-uniform ).

Insertion [50]: this metric is the inverse of Deletion, starting with an image in a baseline state and then progressively adding the most important variables. Formally, at step k , with u the most important variables according to an attribution method, the Insertion ( k ) score is given by:

The AUC is also measured to compare attribution methods ( ↑ is better). The baseline is the same as for Deletion.

µ Fidelity [9]: this metric measures the correlation between the fall of the score when variables are put at a baseline state and the importance of these variables. Formally:

For all experiments, k is equal to 20% of the total number of variables, and cutting the image in a grid of 20 × 20 ( 9 × 9 for cat vs dog and imagenet). The baseline is the same as the one used by Deletion. Being a correlation score, we can either compare attribution methods, or different neural networks on the same attribution method ( ↑ is better).

Robustness-Sr [34]: this metric evaluate the average adversarial distance when the attack is done only on the most relevant features. Formally, given the u most important variables:

̸

where δ u = 0 indicates that adversarial attack is authorized only on the set u . The AUC is measured to compare attribution methods ( ↓ is better). Note this metric cannot be used to compare different networks, since it depends on the robustness of the network.

We use also several other metrics:

Distances between explanations: to compare two explanation f ( x ) , we use either L 2 distance, or 1 -ρ where ρ is the Spearman rank correlation [2, 23, 65] ( ↓ is better).

Explanation complexity: we use the JPEG compression size as a proxy of the Kolmogorov complexity ( ↓ is better).

Stability: As proposed in [74], the Stability is evaluated by the average distance of explanations provided for random samples drawn in a ball of radius 0 . 3 ( 0 . 15 for cat vs dog and imagenet) around x . As before, the distance can be either L 2 or 1 -ρ ( ↓ is better).

Accuracy: To assess the relevance of explanation [57] use a semi-real dataset, called BlockMNIST, having a random null block and evaluate the proportion of top-k feature attribution values within the null block.

A.4.3 Supplementary metric results

In this section we present several experiments and metrics that we were not able to insert in the core of the paper.

Deletion-zero and Insertion-zero are evaluated on CelebA and FashionMNIST dataset. It is known that the baseline value can be a bias for these metrics, and we are convinced that it has a higher influence with 1-Lipschitz networks. Even if results for Deletion-zero and Insertion-zero are less obvious than for Deletion and Insertion Uniform, we can see in Table 8, that for these metrics, the rank of Saliency is most of the time higher for OTNN.

To leverage the bias of the baseline value, as proposed in [34] we evaluated the Robustness-SR metric, Saliency map on OTNN achieves top-ranking scores. One might argue that scores for unconstrained networks are lower, but this is directly linked to the higher intrinsic robustness of OTNNand thus cannot be compared.

The full results for the explanation complexity is given on Table 10. The complexity is still lower for OTNN on FashionMNIST, even if the gap with Unconstrained networks is narrower than for CelebA.

Accuracy was also assessed by learning an OTNN (MLP architecture) on BlockMNIST dataset [57], and evaluating the proportion of top-k saliency map values that fall in the null block. Fig 7 presents

several samples of input images and saliency maps of BlockMNIST dataset, to be compared to Fig.1 in [57]. It shows that OTNN gradients are almost only on the signal block (digit). And Table 11 evaluates the proxy metric proposed in [57], and confirms that OTNNs saliency maps top-k values point even more on discriminative features than those of adversarial trained networks.

D

D

0

0

20

20

20

20

20

20

20

20

20

20

40

40

40

40

40

40

40

40

40

40

0

25

0

25

0

25

0

25

0

25

0

25

0

25

0

25

0

25

0

25

0

0

0

0

0

0

0

0

0

0

20

20

20

20

20

20

20

20

20

20

40

40

40

40

40

40

40

40

40

40

0

25

0

25

0

25

0

25

0

25

0

25

0

25

0

25

0

25

0

25

A.5 Complementary qualitative results

In this section, we provide more samples of qualitative results and couterfactual exlanations for OTNN, based on the gradient, i.e. x -t ∗ ˆ f ( x ) ∇ x ˆ f ( x ) for t > 1 .

A.5.1 FashionMNIST

Fig. 8 gives more results on FashionMNIST.

Ankleboot

Bag

Shirt

Pullover

Sneaker

Ankleboot

Bag

Coat

Top

Shirt

Bag

Shirt

Dress

Shirt

Trouser

Coat

Sandal

Sneaker

A.5.2 CelebA

We presents results for other labels of CelebA. For ethic concerns we have hidden labels that can be subject to misinterpretation, such as Attractive , Male , Big_Nose . Fig. 9 to 27 present more results on the labels presented in the core of the paper, Mouth_Slightly_Open , Mustache , Wearing_Hat .

A.5.3 Cat vs Dog

We present some supplementary comparison of Saliency Maps and counterfactual examples for cat vs dog(Fig. 28 and 29).

A.5.4 Imagenet

Values of accuracy and human feature alignment used for the Fig.4 are described in Tab.12. We present some supplementary comparison of Saliency Maps Imagenet (Fig. 30). As pointed out previously, our model doesn't produce significant counterfacutal explanations on Imagenet.

TIVAL

TIVAL

AF

Copyright

Protected

AAF

AAF

MO

MO

TIVAL

TIVAL

8

8

HABAE

HABAE

HLDRE

S

HILDRE

ER

GBR

Bald → "not" Bald

"not" Bald

→

AFI

AFI

E

ES

Black_Hair → "not" Black_Hair

Blond_Hair → "not" Blond_Hair

BILL

EN

BILL

EN

Brown_Hair → "not" Brown_Hair

Eyeglasses → "not" Eyeglasses

Gray_Hair → "not" Gray_Hair

Hairline → "not" Hairline

AAF

AAF

MO

MO

Heavy_Makeup → "not" Heavy_Makeup

Rosy_Cheeks → "not" Rosy_Cheeks

Wearing_Lipstick → "not" Wearing_Lipstick

Young → "not" Young

na

a

MO

MO

(a) OTNN

(b) Unconstrained

(a) OTNN

(b) Unconstrained

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors provide a theoretical connection between 1-Lipschitz Net, also referred to as OTNN, with Optimal Transport, and provide various empirical results showing that gradient-based attributions look sharper and semantically meaningful.\n\nSTRENGTHS:\nOne major contribution from the paper is Proposition 1 & 2, showing connections between the gradient direction and the optimal transport theory. The use of level curves aligns with my experience of analyzing properties of decision boundaries and smoothness of landscape. They use fidelity and stability metrics to show gradient-based attributions better capture the model’s behavior compared to standard (i.e. non-smooth) models. Also, I have not seen results using ClickMe and Feature Visualizations on OTTN, so they are also new empirical contributions to me.\n\nWEAKNESSES:\nThe main weakness of this paper is its novelty or significance. This is because there are already a number of references, like [1, 2, 3], pointing out that robust models are more interpretable. Although many of these works target empirical robust models, 1-Lipschitz networks are simply the ones with stronger robustness guarantees: they are certifiably robust even if you do not enforce them (and will be more certifiably robust if you train them to do so) [4]. Also, there are works showing empirical robust training also improves the landscape smoothness by implicit gradient norm regularization [5]. Thus, connections between explainability and robustness based on 1-Lipschitz nets are only incremental and the insight is marginal from my personal perspective.\n\nI appreciate two propositions that strengthen connections between robustness (more precisely the smoothness of the function) and explainability. I find the take-away regarding robustness and explainability is already well-known in 2023 (as the newest paper I refer to was published in 2021). This paper would be much more useful a few years ago to motivate the research on 1-Lipschitz networks; however, I think there are already a number of works pushing 1-Lipschitz nets forward. From my understanding, the current research problem is to close the gap between the theoretical possibility of learning a smoothing function and the practical difference of learning one on ImageNet-scale models and dataset. Because of the difficulty in scaling up 1-Lipschitz Net to larger datasets, it is actually more practical to do (empirical) adversarial training so we have a model that is both accurate and uses human-aligned features to make predictions. \n\n[1] Christian Etmann, Sebastian Lunz, Peter Maass, and Carola Schoenlieb. On the connection between adversarial robustness and saliency map interpretability. In Proceedings of the 36th International Conference on Machine Learning, 2019.\n\n[2] Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of relu net- works via maximization of linear regions. AISTATS 2019, 2019.\n\n[3] Wang, Z., Fredrikson, M., & Datta, A. (2021). Robust Models Are More Interpretable Because Attributions Look Normal. International Conference on Machine Learning.\n\n[4] Trockman, A., & Kolter, J.Z. (2021). Orthogonalizing Convolutional Layers with the Cayley Transform. ArXiv, abs/2104.07167.\n\n[5] Simon-Gabriel, C., Ollivier, Y., Bottou, L., Schölkopf, B. &amp; Lopez-Paz, D.. (2019). First-Order Adversarial Vulnerability of Neural Networks and Input Dimension. <i>Proceedings of the 36th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i>\n\nQUESTIONS:\nMy major question is pretty high-level: what are the insights from this paper that have not been covered by existing works.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a special type of 1-Lipschitz neural network named OTNN. The paper analyzes several theoretical properties of OTNN, including the correspondence between gradient direction and the optimal transportation plan from one class to another, the exact analytic solution of the decision boundary and the adversarial sample of OTNN. Furthermore, the paper shows that saliency maps of OTNN can provide meaningful explanation. Experiments also show that on various explanation methods, OTNN outperforms traditional neural networks in three commonly used XAI evaluation metrics.\n\nSTRENGTHS:\n1. The paper focuses on several interesting properties of OTNN, which have not been thoroughly investigated in previous works.\n\n2. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. About whether the OTNN model is locally linear. According to reference [48] in the main paper, the norm of the gradient $\\nabla f^*(x)$ is 1 almost surely, and Corollary 1 implies that the adversarial sample is exactly located in the direction of the gradient. These properties seem to indicate that a trained OTNN model is nearly linear. Authors are suggested to clearly explain this issue. Note that I do not mean the whole decision boundary is linear, but a relatively large portion of local regions can be linear, as shown in Figure 2.\n\n2. Current experimental results in the paper show that the performance of the OTNN model is far behind most existing models. Although I do admit that the performance of robust models like OTNN cannot compete with other neural networks, it is necessary to clearly and thoroughly compare the performance of OTNN and traditional neural networks in the main paper. Existing comparison experiments in the paper are not sufficient and are not well presented. Authors are encouraged to include large-scale comparison experiments between OTNN and traditional neural networks, organize the results into a table, and provide some discussion or comments for the comparison in the main text.\n\n3. The dataset (the two concentric Koch snowflakes) used in the experiment in Figure 2 is too simple. The distribution of positive samples and negative samples in Figure 2 exhibit a high level of symmetry, which makes the decision boundary a regular hexagon. However, this toy dataset is not enough for the verification of theoretical results in Proposition 2 and Corollary 1. I suggest that authors conduct experiments on more complicated datasets to verify the theoretical results.\n\n4. Authors are encouraged to visualize the loss landscape of the model trained on the two concentric Koch snowflakes in Figure 2, so as to verify the claim that gradient norm is 1 almost surely and the analytical solution of adversarial sample in Corollary 1 based on this visualization.\n\nQUESTIONS:\n1. It is encouraged to clarify and verify whether the trained OTNN model is almost linear in most of the local regions.\n2. It is encouraged to have a thorough presentation and discussion in the main text about the comparison of classification performance between OTNNs and traditional neural networks.\n3. It is encouraged to conduct experiments on more complicated datasets to verify the theoretical results in Proposition 2 and Corollary 1.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the explainability properties of optimal transport neural networks, which are 1-Lipschitz constrained neural networks trained with an optimal transport dual loss. By establishing that the gradient of the optimal solution aligns with the transportation plan direction, and the closest decision boundary point (adversarial example) also lies in this gradient direction at a distance of the absolute value of the network output, the authors establish a link between the OTNN gradients and counterfactual explanations. They show that the gradient of OTNN significantly enhances the Saliency Map XAI method, improving several state-of-the-art XAI metrics.\n\nSTRENGTHS:\n- The paper is well motivated and well written. \n- The problem tackled by the authors is interesting and important as the interpretability of neural networks is important for a large range of application\n- The figures and empirical results are convincing\n\nWEAKNESSES:\nIt appears that the authors use the DEEL.LIP library to construct the OTNN. The library was proposed by [1] and uses the power iteration to normalize each layer and constrain them to be 1-Lipschitz (see 4.1 of [1]). This approach to training 1-Lipschitz neural networks has recently been shown to be suboptimal [2, 3, 4], and new layers that are 1-Lipschitz _by design_ have been proposed. Can the authors comment on these papers, and can their results be improved with more recent 1-Lipschitz architectures? \n\n\n[1] Serrurier et al. Achieving robustness in classification using optimal transport with hinge regularization, CVPR 2021  \n[2] Meunier et al., A Dynamical System Perspective for Lipschitz Neural Networks ICML 2022  \n[3] Araujo et al., A Unified Algebraic Perspective on Lipschitz Neural Networks, ICLR 2023  \n[4] Wang et al. Direct Parameterization of Lipschitz-Bounded Deep Networks, ICML 2023\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\n1-Lipschitz networks are known to be highly constrained and do not provide state-of-the-art accuracy compared to their unconstrained counterparts. the authors should discuss this fact and the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the properties of 1-Lipschitz Neural Networks, and shows that the input-gradients of these models have desirable properties. On a benchmark across several datasets, they find that gradient-based feature attributions from the 1-Lipschitz Neural Networks show substantially better fidelity and stability metrics than those of an unconstrained model. Interestingly, for these networks, they show that adversarial attacks are counterfactual explanations (proposition 1). Overall, this paper adds to the line of work that shows that constrained (or regularized models) show improved input-gradients than unconstrained ones.\n\nSTRENGTHS:\nOverall, this is an interesting paper that explores the effect of constraining the Lipschitz constant of a model, and how it endows the model with reliable input-gradients.\n\n**Quality/Clarity**\\\nOverall, this paper is quite well-written and clear. The key point of the paper is stated clearly and well-justified comprehensively across experiments on several datasets. \n\n**Significance/Originality**\\\nI am combining the discussion on significance and originality because I think these are tightly linked for this work. First, I think several of the findings in this paper have been made in other papers (see weaknesses), but the specific settings under which this papers makes its findings are different. To the best of my knowledge, this is the first paper to explicitly and comprehensively quantify the fidelity and stability of explanations from constrained models. Overall, this paper is another important evidence to this line of work.\n\nWEAKNESSES:\nThe main weaknesses are listed below: \n\n**Lipschitz constant:** Why does it it have to be 1? Is there some range of values for this constant where the model gradients still retain these properties? How were these model trained? It says that they used some library, so I assume the training scheme here is standard and no new innovation there.\n\n**Related work**: My main weakness of this paper has to do with related work. I'll list some here that the paper should engage with. First in [1], the authors show that the gradients of an adversarially robust model become sensitive to the output while those of an unconstrained model are not. In general, adversarially robust models have lower lipschitz constants, so the finding from this work makes sense. The authors should cite this paper. [2] and [3] also contain insights that add to this paper and explain some of their findings. For example that the model has better stability metrics is expected since that is essentially what constraining the lipschitz  constant does. I am not generally surprised by any of the insights discussed in this work, but this is fine, and they make a valuable contribution. However, it would be helpful if they could engage with the relevant literature below.\n\n[1] Shah, Harshay, Prateek Jain, and Praneeth Netrapalli. \"Do input gradients highlight discriminative features?.\" Advances in Neural Information Processing Systems 34 (2021): 2046-2059.\n\n[2] Srinivas, Suraj, et al. \"Efficient training of low-curvature neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 25951-25964.\n\n[3] Dombrowski, Ann-Kathrin, et al. \"Towards robust explanations for deep neural networks.\" Pattern Recognition 121 (2022): 108194.\n\n**Evaluation**: As the authors are probably aware, evaluating gradient attributions is tough. I looked into the fidelity and stability metrics, but I think these are merely passable. To really make concrete conclusions, the authors need a modified setting where ground-truth is known. I hate to be that reviewer that asks for new experiments, but something like the block MNIST evaluation in [1] is needed here to make conclusive claims. Lastly, the human evaluation using the click me dataset is nice, but addresses a different question. Just because an explanation aligns with what a human expects does not mean that the explanation is 'correct' or desirable. You want an explanation that tells you what your model has learnt. If your model has learnt human aligned signals then that is good. Comparing an explanations to humans is more helpful for debugging than for assessing the actual explanation itself.\n\nQUESTIONS:\nSee weaknesses section for the questions that I have about this work. \n\nAdditional ones: \n\nI assume Proposition 1 & 2 are new to this work? Or did they appear in previous work? I am not an expert on optimal transport, so I am less familiar with that literature.\n\nLIMITATIONS:\nThe authors discuss societal impact in the conclusions section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a theoretical analysis of 1-Lipschitz constrained neural networks that are trained based on optimal transport dual loss for image classification task. The optimal classifier f(.) is trained by minimizing the hinge regularized KR optimization loss. It's been shown that the gradient of optimal solution is aligned with the transportation plan direction and the L1 norm of it at x is a certifiable lower bound of classifier robustness. So the close decision boundary to x is also along with the gradient direction at distance of the lower bound which is |f*(x)|.\n\nThe experimental results show that such a model performs better than its unconstrained counterpart in XAI metrics like \\mu fidelity and stability spearman rank on multiple datasets including FashionMNIST, CelebA, CatvsDog and Imagenet. It's been shown that OTNN models  explanations are more aligned with human attention that makes the model performance more predictable.\n\nSTRENGTHS:\n- The paper provides a theoretical analysis of properties of 1-Lipschitz constrained neural networks. I consider this type of analysis highly valuable to the field! \n\n- The authors provide empirical study about the effectiveness of OTNNs on improving XAI metrics. Considering multiple datasets, it's been shown that the effectiveness is scalable to larger dataset and larger networks. \n\n- Several properties of OTNN gradient has been investigated and shown that, their saliency maps are aligned with human attention that improves the predictability of network behavior. They also draw connection between gradient and counterfactual examples and adversarial attacks to OTNNs.\n\nWEAKNESSES:\n- While I appreciate the theoretical analysis of OTNNs, they are mostly compilation of analysis in [53] and [26] and [6].\n\n- There is no ablation study on activation functions and normalization methods besides GroupSort and Spectral Norm even on a single dataset. It would be nice to evaluate the empirical effect of network depth on XAI metrics.\n\nQUESTIONS:\n- It's been proven that 1-Lipschitz neural network are as accurate as unconstrained neural networks [ref.1 ], why are the OTNN models are practically less accurate. \n\n- Figure 2 demonstrates the level set of an OTNN binary classifier, how does the unconstrained network decision boundary look like? I appreciate if you could provide the such figure.\n\n- Could you provide the table corresponding to the Figure 4 that clarifies what each of the point in the figure is and what is the exact value of accuracy and human feature alignment in addition to the details of experimental setup that leads to the figure?     \n\n\n\n[ref.1] Béthune, Louis, et al. \"Pay attention to your loss: understanding misconceptions about lipschitz neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 20077-20091.\n\nLIMITATIONS:\nThere is no negative societal impact of this paper as it is a theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}