(Poster) Safe and Sound: Approximately-Optimal Black-box Model Explanations with Differential Privacy (Extended Abstract)

Machine learning models are currently applied in a variety of high-stakes domains, e.g. providing predictive healthcare analytics. These domains require high prediction accuracy over high-dimensional data, and as a result, adopt increasingly complex ML models. While simple rule-based systems and linear models are easily understood, modern ML models — which utilize complex mathematical structures, such as deep neural networks — are hard to interpret. What makes the problem even more challenging is that these models are usually used as black-box algorithms, where the model only outputs the decision without providing detailed information on its intermediate computations for producing the output (decision). The implementation of black-box algorithms in high-stakes domains affects trust, raises concerns with respect to the model’s reliability and fairness, and hinders model adoption [1], [2]. Recent years have seen a widespread call for algorithmic transparency. Broadly speaking, algorithmic transparency methods offer stakeholders additional information about the algorithmic decision-making process, to facilitate their understanding of the model. In this work, we focus on feature-based explanations, where the algorithm computes the influence of various features of the point of interest on the model’s decision [3]–[9]. However, offering additional information can result in significant privacy risks, potentially exposing the underlying model [10] or the model’s training data [11]. The analysis of the privacy risks of explanation algorithms is largely limited to these works, and their overall privacy implications are widely unexplored. Notably, there has been little work on designing provably sound, as well as differentially private, model explanations for black-box models. This calls for a rigorous approach towards modeling the privacy risks of model explanations, and the design of practical differentially-private model explanations with provable explanation accuracy. We focus on protecting the privacy of model-agnostic feature-based explanations for black-box models, which do not have access to the model parameters, and make no assumptions on model structure. These techniques assume access to a labeled dataset, sampled from the same underlying distribution as the model’s training set, which we refer to as the explanation dataset. The explanation algorithms locally approximate the target model in the vicinity of the point of interest, using the explanation dataset, in order to measure the features’ influence on the model [4]–[6], [8], [12]. This popular approach can leak sensitive information about the explanation dataset through model explanations. This leakage is independent of privacy loss with respect to training data. Almost all existing model explanations do not offer any proven privacy guarantees, and given their large/unbounded sensitivity, achieving differential privacy can result in an extremely low accuracy. As an exception, QII could be randomized to satisfy differential privacy to protect the explanation dataset [5]. However, there is no theoretical analysis of the composition of the privacy loss of QII over a sequence of queries, nor there is a bound on the utility loss of the algorithm due to the randomness of the differential privacy mechanism.