Auditing Black-Box Prediction Models for Data Minimization Compliance

1 Introduction . Concerns about the widespread use of data-driven prediction models and their growing reliance on personal data of individuals , have led to a number of data protection laws and regulations in recent years [ 17 , 19 , 24 ] . Prominent amongst such regulations is the GDPR ( article 5.1.c ) [ 19 ] that proposes the data minimization principle to control the extent to which personal data can be acquired and used by prediction models . Specifically , the data minimization principle states that : “ Personal data shall be adequate , relevant and limited to what is necessary for the purposes for which they are processed. ” Despite considerable public debate about the GDPR and the data minimization principle , to date , only a few works [ 2 , 18 ] have attempted to operationalize ( i.e. , formally interpret ) the legal principle for prediction models . These works have focused on finding operationalizations of data minimization that tie the purpose of data processing to some performance metric such as prediction accuracy ( e.g. , in recommender systems [ 2 ] , or in classification [ 18 ] ) . Specifically , they study whether a prediction model can be redesigned to achieve similar prediction performance , while using fewer input features . In the process , these works assume a transparent setting , where the full knowledge of the prediction algorithm , training procedure , and desired performance metrics is available to the party investigating the feasibility of reducing data inputs . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . When putting data minimization into practice , an important case to consider is an auditing scenario where an outsider ( auditor ) wants to test a given prediction model for compliance with the data minimization principle at deployment time . As prediction algorithms are often important business assets , a more realistic auditing scenario is one in a black-box setting in which the auditor interacts with the model without access to the model internals . In this paper , we propose an operational definition of the data minimization principle that allows auditing black-box prediction models for compliance at deployment time . In particular , we assume an auditor who does not have access to how the prediction model works , or to the training data , or to information about the purpose of the prediction model . All the auditor can do is using data points to query a given black-box prediction model with a fixed set of input features , and observe the outcomes . We believe our black-box model setting covers many real-world scenarios , as it only requires the designers of prediction models to allow auditors to query their models with prediction instances . Our key insight is that the auditor can test whether an input feature is needed by the prediction model , by imputing ( i.e. , guessing ) its value and checking the extent to which the outcomes change ( i.e. , are unstable ) for different prediction instances . Intuitively , if the actual value of an input feature is not needed ( i.e. , can be replaced with a constant ) to arrive at similar ( stable ) outcomes for most prediction instances , then the use of the feature violates the data minimization principle . Note that our instability-based operatlonalization does not require knowledge of the purpose of the prediction model and is independent of performance metrics . Such an operational definition is particularly important given that it is common for companies who provide personalized services to justify data collection simply as “ for improving service ” [ 9 ] . We show how simple imputations , where the actual values of individual features are replaced with a constant value , can be leveraged as a strategy for limiting data inputs to a prediction system at deployment time . We define a data minimization guarantee that is based on a metric of model instability under different feasible simple imputations . While this guarantee induces a procedure for auditing data minimization assuming a finite sample model , we extend the applicability of our auditing framework in two ways . First , we propose a probabilistic audit that allows the auditor to provide a data minimization guarantee at some confidence level with respect to an underlying data distribution . We achieve this goal by introducing the notion of a probabilistic data minimization guarantee and adopting a Bayesian approach . Second , we address the auditing problem under a constraint on the number of queries to the prediction system and we design auditing algorithms that use a budget of system queries strategically in order to provide a data minimization guarantee . We cast the problem of allocating a query budget to feasible simple imputations into a multi-armed bandit framework , and we formulate two bandit problems that correspond to different auditing tasks given a fixed confidence level : a decision problem given a data minimization level , and a measurement problem given a fixed query budget . Furthermore , we design efficient algorithms for the above problems using exploration strategies for selecting arms . We propose four heuristic exploration strategies : two strategies inspired by Thompson sampling , and two that are custom for our setting . Finally , we study the effectiveness of our auditing algorithms using different real-world prediction systems . We build prediction models by applying standard model evaluation and feature selection methods , and use the resulting models to perform black-box audits . Our experiments show that algorithms that exploit the proposed bandit framework significantly outperform simpler benchmarks . 2 Related Work . Data Minimization . Only a few recent papers have addressed the problem of operationalizing data minimization in prediction systems . Biega et al . [ 2 ] propose definitions that are based on recommender systems ’ performance , and conduct an empirical study to check the original recommender performance can be preserved , while limiting the number of known user ratings . The authors in [ 18 ] suggest a formulation based on classification accuracy and study the trade-off between data minimization and satisfying other fairness properties . Clavell et al . [ 10 ] interpret data minimization as limiting the use of sensitive personal data and do an experimental analysis that suggests data minimization should not be applied without consideration of other social concerns such as fairness . Model Instability in Machine Learning . The instability of prediction models is studied in both adversarial and non-adversarial settings . The traditional non-adversarial setting is concerned with the ii instability of a model under different training data [ 15 , 8 ] . The adversarial setting studies the effects of data perturbations on the model predictions . Perturbations in training data is often called data poisoning and has been studied for various prediction models [ 23 ] . The more relevant problem to our setting is however studying the effect of test data perturbations , which is known as evasion attacks at test time [ 3 , 16 ] . A key difference between our instability metric and model instability under evasion attacks is in the type of perturbation that we define . While evasion attacks consider perturbations defined as modifying a data instance inside a fixed-radius ball under a particular norm , we consider perturbations defined as projections induced by simple imputations . Moreover , the evasion attack problem concerns the instability of a model prediction locally at each data point whereas the audit problem is concerned with a global notion of instability ( see Section 3 ) . 3 Model Instability-based Data Minimization Guarantees . In this section , we first formalize our setting and define a metric for model instability under simple imputations . Then we show how this metric can be leveraged to define a data minimization guarantee . Setting . We consider a prediction model being audited at deployment time for adherence to data minimization principle . The model has a fixed set of inputs variables ( features ) and an output variable from a discrete domain . We assume a black-box setting , i.e. , for any prediction instance , an auditor can query the model by specifying the values of all input features and observe the produced output , while the procedure used for generating the output from the inputs is unknown to the auditor . The auditor ’ s goal then is to measure the level of data minimization satisfied by the prediction model using a limited number of queries . A schematic illustration of this setting is provided in Appendix A . Notation . Formally , let ŶF denote a prediction model over a set of input features is F = { f1 , . . . , fd } , where each feature fi takes values from domain Xi . X = ∏d i=1 Xi denotes the input space of ŶF . For any prediction instance x ∈ X , ŶF ( x ) ∈ Y denotes the prediction made by ŶF for x , and Y is the discrete set of targets in the output space . We also consider a distribution PX over the input space from which samples ( data points ) are drawn in deployment . The auditor ’ s challenge is to test whether the the model ŶF satisfies data minimization principle over PX . Assessing the need for individual features with simple imputations . To assess the need for individual features , we propose using simple imputations . Simple imputation is a procedure that is commonly used for handling missing data , and it works by assigning a constant value to a feature fi in any prediction instance in which fi is missing . If an auditor finds that imputing a constant value to an input feature fi has no or small effect on outcomes across different prediction instances , then the auditor can conclude that information about the actual values of fi are not needed by ŶF to arrive at prediction outcomes . Our idea is to quantify the instability or changes in prediction outcomes to determine the level or extent to which the data minimization principle has been satisfied or violated . A metric for model instability under simple imputations . We now define a metric for instability of a prediction model , ŶF , under simple feature imputations . Formally , we define an imputation function τfi , b such that for any prediction instance x , τfj , b ( x ) returns a vector in which the value of feature fi is replaced with b ∈ Xi . Let IŶF ( x , fj , b ) be a binary indicator variable that is 1 if the prediction made by ŶF for x changes after imputing fj with b , or 0 otherwise . Formally , IŶF ( x , fj , b ) = { 1 if ŶF ( x ) 6= ŶF ( τfj , b ( x ) ) 0 otherwise ( 1 ) Let X be a random variable that takes on values x ∈ X according to an underlying data distribution PX . We define the instability of ŶF over PX with respect to feature fj and imputation value b as : βbj = EX∼PX [ IŶF ( X , fj , b ) ] ( 2 ) In other words , βbj denotes the probability that the prediction for a data point drawn randomly from PX changes after imputing fj by b. Instability-based data minimization guarantee . We now define a data minimization guarantee an auditor can offer using our model instability metric βbj . Intuitively , the imputation b that induces iii the minimum prediction instability for a feature fj determines how necessary the feature fj is for generating the predicted outcomes . So a natural data minimization guarantee can be arrived at by finding the greatest lower bound on the need for each individual feature fj ( which in turn requires finding a feasible imputation for each fj that induces minimum instability ) .1 Formally we define : Definition 1 A prediction model ŶF satisfies data minimization at level β if there does not exist any feature fj ∈ F and any imputation value b ∈ Xj such that βbj < β . The highest level β at which data minimization is satisfied constitutes the best data minimization guarantee an auditor can offer for ŶF . Intuitively , a data minimization guarantee at level β ensures that every input feature used by a prediction model is indeed necessary to reach the predictions made for at least a certain fraction , β , of prediction instances . Note that if a prediction model satisfies data minimization at level β , it also satisfies data minimization at any level β′ < β . In practice , the auditor would be interested in finding the best guarantee , i.e. , the largest value of β at which a prediction system satisfies data minimization .

Summary: This paper defines a new setting generalizing existing work on establishing _data minimization_ as motivated by the GDPR. Unlike previous work, this work considers testing for data minimization in a black-box setting (previous work assumes access to model internals). Within this setting, the data minimization problem is defined according to a measure of model instability when certain features are replaced by constant imputation (as is often done for missing values in data preprocessing). Further, the problem of auditing a model for data minimization at a certain level (according to a metric the paper introduces) is framed as a pair of near-dual bandit problems: one which establishes the optimal auditing strategy to achieve a certain confidence that the metric is not violated, and another which assigns a fixed query budget in an expected-optimal way.