SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cuong Tran
kxb7sd@virginia.edu
Ferdinando Fioretto
fioretto@virginia.edu
In domains with high stakes such as law, recruitment, and healthcare, learning models frequently rely on sensitive user data for inference, necessitating the complete set of features. This not only poses significant privacy risks for individuals but also demands substantial human effort from organizations to verify information accuracy. This paper asks whether it is necessary to use all input features for accurate predictions at inference time. The paper demonstrates that, in a personalized setting, individuals may only need to disclose a small subset of their features without compromising decision-making accuracy. The paper also provides an efficient sequential algorithm to determine the appropriate attributes for each individual to provide. Evaluations across various learning tasks show that individuals can potentially report as little as  $10\%$  of their information while maintaining the same accuracy level as a model that employs the full set of user information.
The remarkable success of machine learning (ML) models also brought with it pressing challenges at the interface of privacy and decision-making, especially when deployed in consequential domains such as legal processes, banking, hiring, and healthcare [23]. A particularly intriguing aspect is the conventional requirement for users to disclose their entire set of features during inference, thereby creating a gateway for potential data breaches, as exemplified in recent instances where millions of individuals' data were compromised [.]. Concurrently, this practice also places a burden on companies and organizations to ensure the accuracy and legal compliance of the disclosed information, as often observed in financial operations, mandated by legislations like the Corporate Transparency Act[]
Significantly, this conventional approach of disclosing the enture feature set during inference might also also violate the data minimization principle, a cornerstone of several global privacy regulations including the European General Data Protection Regulation [2], the California Privacy Rights Act [1], and the Brazilian General Data Protection Law [3], among others. Through this lens, the discourse on enhancing privacy and reducing the verification onus in ML systems, particularly in crucial decision-making domains, gains a nuanced dimension, warranting a thorough examination.
This paper challenges this setting and asks whether it is necessary to require all input features for a model to produce accurate or nearly accurate predictions during inference. We refer to this question as the data minimization for inference problem. This unique question bears profound implications for privacy in model personalization, which often necessitates the disclosure of substantial user data. We show that, under a personalized setting, each individual may only need to release a small subset of their features to achieve the same prediction errors as those obtained when all features are disclosed. The overall framework is depicted in Figure 1. Following this result, we also provide an efficient sequential algorithm to identify the minimal set of attributes that each individual should reveal. Evaluations across various learning tasks indicate that individuals may be able to report as little as  $10\%$  of their information while maintaining the same accuracy level as a model using the complete set of user information.
Figure 1: Illustrative example of the proposed framework. The figure highlights the online and personalized nature of the minimization process. For each user, a key subset of features is identified to be used (core feature set). The minimized data sample is then processed by the pre-trained model, which generates a representative label.
In summary, the paper makes the following contributions: (1) it initiates a study to analyze the optimal subset of data features that each individual should disclose at inference time in order to achieve the same accuracy as if all features were disclosed; (2) it links this analysis to a new concept of data minimization for inference in relation to privacy, (3) it proposes theoretically motivated and efficient algorithms for determining the minimal set of attributes each individual should provide to minimize their data; and (4) it conducts a comprehensive evaluation illustrating the effectiveness of the proposed methods in preserving privacy at no or small costs in accuracy.
Related work. To the best of our knowledge, this is the first work studying the connection between data minimization and accuracy at inference time. This work was motivated by the core principle in data minimization which states that "all collected data shall be adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed". This principle is adopted as a privacy cornestone in different data protection regulations, including the General Data Protection Regulation [2] in Europe, the California Privacy Rights Act in the US [1], the General Personal Data Protection Law in Brasil [3], the Protection of Personal Information Act, in South Africa [5], and the Personal Information Protection Act, in South Korea [4].
While we are not aware of studies on data minimization for inference problems, we draw connections with differential privacy, feature selection, and active learning. Differential Privacy (DP) [14] is a strong privacy notion that determines and bounds the risk of disclosing sensitive information of individuals participating in a computation. In the context of machine learning, DP ensures that algorithms can learn the relations between data and predictions while preventing them from memorizing sensitive information about any specific individual in the training data. In such a context, DP is primarily adopted to protect training data [6, 12, 32] and thus the setting contrasts with that studied in this work, which focuses on identifying the superfluous features revealed by users at test time to attain high accuracy. Furthermore, achieving tight constraints in differential privacy often comes at the cost of sacrificing accuracy, while the proposed privacy framework can reduce privacy loss without sacrificing accuracy under the assumption of linear classifiers.
Feature selection [10] is the process of identifying a relevant subset of features from a larger set for use in model construction, with the goal of improving performance by reducing the complexity and dimensionality of the data. The problem studied in this work can be considered a specialized form of feature selection with the added consideration of personalized levels, where each individual may use a different subset of features. This contrasts standard feature selection [20], which selects the same subset of features for each data sample. Additionally, and unlike traditional feature selection, which is performed during training and independent of the deployed classifier [10], the proposed framework performs feature selection at deployment time and is inherently dependent on the deployed classifier.
The framework proposed in this paper shares some similarities with active learning [15, 26, 33], whose goal is to iteratively select samples for experts to label in order to construct an accurate classifier with the least number of labeled samples. Similarly, the proposed framework iteratively asks individuals to reveal one attribute given their released features so far, with the goal of minimizing the uncertainty in model predictions. Finally, our study share some connections with active feature acquisition [28, 13, 11, 21] which can be categorized as one branch in active learning. However, these priors focus mostly on minimizing the total number of users features at training step. This is in sharp contrast with ours work which concentrate on inference time.
2 Settings and objectives
We consider a dataset  $D$  consisting of samples  $(x,y)$  drawn from an unknown distribution  $\Pi$ . Here,  $x \in \mathcal{X}$  is a feature vector and  $y \in \mathcal{Y} = [L]$  is a label with  $L$  classes. The features in  $x$  are categorized into public  $x_{P}$  and sensitive  $x_{S}$  features, with their respective indexes in vector  $x$  denoted as  $P$  and  $S$ , respectively. We consider classifiers  $f_{\theta}: \mathcal{X} \to \mathcal{Y}$ , which are trained on a public dataset from the same data distribution  $\Pi$  above. The classifier produces a score  $\tilde{f}_{\theta}(x) \in \mathbb{R}^{L}$  over the classes and a final output class,  $f_{\theta}(x) \in [L]$ , given input  $x$ . The model's outputs  $f_{\theta}(x)$  and  $\tilde{f}_{\theta}(x)$  are also often referred to as hard and soft predictions, respectively.
Without loss of generality, we assume that all features in  $\mathcal{X}$  lie within the range  $[-1, 1]$ . In this setting, we are given a trained model  $f_{\theta}$  and, at inference time, we have access to the public features  $x_{P}$ . These features might be revealed through user queries or collected by the provider during previous interactions. Our focus is on the setting where  $|S| \ll |D|$ , and for simplicity, the following considers binary classifiers, where  $\mathcal{Y} = \{0, 1\}$  and  $\tilde{f}_{\theta} \in \mathbb{R}$ . Multi-class settings are addressed in Appendix C.
In this paper, the term data leakage of a model, refers to the percentage of sensitive features that are revealed unnecessarily, meaning that their exclusion would not significantly impact the model's output. Our goal is to design algorithms that accurately predict the output of the model using the smallest possible number of sensitive features, thus minimizing the data leakage at inference time. This objective reflects our desire for privacy.
To clarify key points discussed in the paper, let us consider a loan approval task where individual features are represented by the set  $\{Job, Loc(action), Inc(ome)\}$ . In this example, the Job feature  $\in x_P$  is public, whereas  $Loc$  and  $Inc \in x_S$  are sensitive. We also consider a trained linear model  $f_\theta = 1.0$ $Job - 0.5$ $Loc + 0.5$ $Inc \geq 0$  and look at a scenario where user (A) has a public feature  $Job = 1.0$ , and user (B) has a public feature  $Job = -0.9$ . Both users' sensitive feature values are unknown. However, for user A, the outcome can be conclusively determined without revealing any additional information since all features are bounded within  $[-1,1]$ . In contrast, for user B, the outcome cannot be determined solely based on the public feature, but revealing the sensitive feature  $Loc = 1.0$  is enough to confirm the classifier outcome.
This example highlights two important observations that motivate our study: (1) not all sensitive attributes may be required for decision-making during inference, and (2) different individuals may need to disclose different amounts and types of sensitive information for decision-making.
3 Core feature sets
With these considerations in mind, this section introduces the notion of core feature set, the first contribution of the paper, which will be used to quantify data minimization. The paper presents the key findings and defers all proofs in Appendix A.
Throughout the paper, the symbols  $R$  and  $U$  are used to represent the sets of indices for revealed and unrevealed features of the sensitive attribute  $S$ , respectively. Given a vector  $x$  and an index set  $I$ , we use  $x_{I}$  to denote the vector containing entries indexed by  $I$  and  $X_{I}$  to represent the corresponding random variable. Finally, we write  $f_{\theta}(X_U, X_R = x_R)$  as a shorthand for  $f_{\theta}(X_U, X_R = x_R, X_P = x_P)$  to denote the prediction made by the model when the features in  $U$  are unrevealed.
Our objective is to develop algorithms that can identify the smallest subset of sensitive features to disclose, ensuring that the model's output is accurate (with high probability) irrespective of the values of the undisclosed features. We refer to this subset as the core feature set.
Definition 1 (Core feature set). Consider a subset  $R$  of sensitive features  $S$ , and let  $U = S \setminus R$  be the unrevealed features. The set  $R$  is a core feature set if, for some  $\tilde{y} \in \mathcal{V}$ ,
$$
\Pr \left(f _ {\theta} \left(X _ {U}, X _ {R} = x _ {R}\right) = \tilde {y}\right) \geq 1 - \delta , \tag {1}
$$
where  $\delta \in [0,1]$  is a failure probability.
When  $\delta = 0$  the core feature set is called pure. Additionally, the label  $\tilde{y}$  satisfying Equation (1) is called the representative label for the core feature set  $R$ . The concept of the representative label  $\tilde{y}$  is crucial for the algorithms that will be discussed later. These algorithms use limited information to
make predictions and when predictions are made using a set of unrevealed features, the representative label  $\tilde{y}$  will be used in place of the model's prediction.
In identifying core feature sets to minimize data leakage, it's crucial to consider model uncertainty, which refers to the unknown values of unrevealed features. The following result links core feature sets with model entropy, which measures uncertainty and is used by this work to minimize data leakage.
Proposition 1. Let  $R \subseteq S$  be a core feature set with failure probability  $\delta < 0.5$ . Then, there exists a monotonic decreasing function  $\epsilon: \mathbb{R}_+ \to \mathbb{R}_+$  with  $\epsilon(1) = 0$  such that:
$$
H \left[ f _ {\theta} \left(X _ {U}, X _ {R} = x _ {R}\right) \right] \leq \epsilon (1 - \delta),
$$
where  $H[Z] = -\sum_{z \in [L]} \operatorname{Pr}(Z = z) \log \operatorname{Pr}(Z = z)$  is the entropy of the random variable  $Z$ .
This property highlights the relationship between core feature sets and entropy associated with a model using incomplete information. As the  $\delta$  value decreases, the model's predictions become more certain. When  $\delta$  equals zero (or when  $R$  represents a pure core feature set), the model's predictions can be fully understood without observing  $x_{U}$ , resulting in entropy of 0.
It is worth noticing that enhancing prediction accuracy necessitates revealing additional information, as illustrated by the previous result and the renowned information theoretical proposition below:
Proposition 2. Given two subsets  $R$  and  $R'$  of sensitive features  $S$ , with  $R \subseteq R'$ ,
$$
H \left(f _ {\theta} \left(X _ {U}, X _ {R} = x _ {R}\right)\right) \geq H \left(f _ {\theta} \left(X _ {U ^ {\prime}}, X _ {R ^ {\prime}} = x _ {R ^ {\prime}}\right)\right),
$$
where  $U = S\setminus R$  and  $U^{\prime} = S\setminus R^{\prime}$
Thus, the parameter  $\delta$  plays a crucial role in balancing the trade-off between privacy loss and model prediction's uncertainty. It determines the amount of sensitive information that must be disclosed to have enough confidence on model predictions. As  $\delta$  increases, fewer sensitive features need to be revealed, resulting in reduced data leakage but also less certainty on model predictions, and vice versa. Note that disclosing more sensitive attributes does not necessarily improve the accuracy of predictions for a specific individual, as highlighted in previous research [30]. Although it's chal
Figure 2: Frequency associated with the size of the minimum pure core feature set.
lenging to rigorously analyze the relationship between privacy loss and predictive accuracy across various contexts, our empirical studies across multiple datasets and learning algorithms do suggest that exposing more sensitive features generally enhances prediction accuracy. Thus, the parameter  $\delta$  appears to govern the trade-off between sacrificing privacy and gaining predictive performance.
As highlighted in the previous example, the core feature set is not uniform for all users. This is further exemplified in Figure 2, using the Credit dataset [9] with a logistic regression classifier. The figure reports the cumulative count of users against the minimum number of features they need to disclose to ensure confident predictions. It demonstrates that many individuals need to disclose no additional information to attain accurate predictions (corresponding to a pure feature of set size 0), and most individuals can achieve accurate predictions by disclosing only  $\leq 2$  sensitive features. These insights, together with the previous observations linking core feature sets to entropy, motivate the proposed online algorithm, the second contribution of the paper.
4 MinDRel: An algorithm to minimize data release at inference time
The goal of the proposed algorithm, called Minimize Data Reveal (MinDRel), is to uphold privacy during inference by revealing sensitive features one at a time based on their released feature values. This section provides a high-level description of the algorithm and outlines its challenges. Next, Section 5, applies MinDRel to linear classifiers and discusses its performance on several datasets and benchmarks. Further, Section 6, extends MinDRel to non-linear classifiers and considers an evaluation over a range of standard datasets.
Overview of MinDRel. MinDRel operates fundamentally on two critical actions: 1. determining the next feature to reveal for each user and 2. verifying whether the disclosed features make up a core feature set for that user. These two operations will be discussed in sections 4.1 and 4.2, respectively.
The algorithm determines which feature to disclose for a specific user by inspecting the posterior probabilities  $\operatorname*{Pr}(X_j|X_R = x_R,X_P = x_P)$  for each unrevealed feature  $j\in U$  , taking into account the disclosed sensitive features  $x_{R}$  and public features  $x_{P}$  . Given the current set of disclosed features  $x_{R}$  and unrevealed features  $x_{U}$  , MinDRel chooses the subsequent feature  $j\in U$  as follows:
$$
j = \underset {j \in U} {\operatorname {a r g m a x}} F \left(x _ {R}, x _ {j}; \theta\right) \doteq \underset {j \in U} {\operatorname {a r g m a x}} - H \left[ f _ {\theta} \left(X _ {j} = x _ {j}, X _ {U \backslash \{j \}}, X _ {R} = x _ {R}\right) \right], \tag {2}
$$
where  $F$  is a scoring function that evaluates the amount of information that can be acquired about the model's predictions when feature  $X_{j}$  is disclosed. As suggested in previous sections, it's desirable to reveal the feature that provides the most insight into the model prediction upon disclosure. MinDRel uses Shannon entropy for this purpose as it offers a natural method for quantifying information. Once feature  $X_{j}$  is disclosed with a value of  $x_{j}$ , the algorithm updates the posterior probabilities for all remaining unrevealed features. The process concludes either when all sensitive features have been disclosed or when a core feature set has been identified. It should also be noted that, within this framework, there is no need to perform data imputation when some features are missing. Unrevealed features are treated as random variables and are integrated during the prediction process.
Both the computation of the scoring function  $F$  and the verification of whether a set of disclosed features constitute a core feature set present two significant challenges for the algorithm. The rest of the section delves into these difficulties.
4.1 Computing the scoring function  $F$
Designing a scoring function  $F$  that measures how confident a model's prediction is when a user discloses an additional feature  $X_{j}$  brings up two key challenges. First, the value of  $X_{j}$  is unknown until the decision to reveal it is made, which complicates the computation of the entropy function. Second, even if the value of  $X_{j}$  were known, determining the entropy of model predictions in an efficient manner is another difficulty. We next discuss how to overcome these challenges.
Dealing with unknown values. To address the first challenge, we exploit the information encoded in the disclosed features to infer  $X_{j}$ 's value and compute the posterior probability  $\operatorname*{Pr}(X_j|X_R = x_R)$  of the unrevealed feature  $X_{j}$  given the values of the revealed ones. The scoring function, abbreviated as  $F(X_{j})$ , can thus be modeled as the expected negative entropy given the randomness of  $X_{j}$ ,
$$
\begin{array}{l} F (X _ {j}) = \mathbb {E} _ {X _ {j}} - \left[ H [ f _ {\theta} (X _ {j}, X _ {U \setminus \{j \}}, X _ {R} = x _ {R}) ] \right. \\ = - \int \underbrace {H \left[ f _ {\theta} \left(X _ {j} = z , X _ {U \backslash \{j \}} , X _ {R} = x _ {R}\right) \right]} _ {A} \times \underbrace {\Pr \left(X _ {j} = z \mid X _ {R} = x _ {R}\right)} _ {B} d z, \tag {3} \\ \end{array}
$$
where  $z\in \mathcal{X}_j$  is a value in the support of  $X_{j}$
Efficient entropy computation. The second difficulty relates to how to estimate this scoring function efficiently. Indeed this is challenged by two key components. The first (A) is the entropy of the model's prediction given a specific unrevealed feature value,  $X_{j} = z$ . This prediction is a function of the random variable  $X_{U\setminus \{j\}}$ , and, due to Proposition 1, its estimation is linked to the conditional densities  $\operatorname*{Pr}(X_{U\setminus \{j\}}|X_R = x_R,X_j = z)$ . The second (B) is the conditional probability  $\operatorname*{Pr}(X_{j} = z|X_{R} = x_{R})$ . Efficient computation of these conditional densities is discussed next.
First, we discuss a result relying on the joint Gaussian assumption of the input features. This result will be useful in providing a computationally efficient method to estimate such conditional density functions. In the following,  $\Sigma_{IJ}$  represents a sub-matrix of size  $|I|\times |J|$  of a matrix  $\boldsymbol{\Sigma}$  formed by selecting rows indexed by  $I$  and columns indexed by  $J$ .
Proposition 3. The conditional distribution of any subset of unrevealed features  $U' \subseteq U$ , given the values of released features  $X_{R} = x_{R}$  is given by:
$$
\operatorname * {P r} (X _ {U ^ {\prime}} | X _ {R} = x _ {R}) = \mathcal {N} \bigg (\mu_ {U ^ {\prime}} + \Sigma_ {U ^ {\prime} R} \Sigma_ {R R} ^ {- 1} (x _ {R} - \mu_ {R}), \Sigma_ {U ^ {\prime} U ^ {\prime}} - \Sigma_ {U ^ {\prime} R} \Sigma_ {R R} ^ {- 1} \Sigma_ {R U ^ {\prime}} \bigg),
$$
where  $\Sigma$  is the covariance matrix.
Note that Equation (3) considers  $U' = \{j\}$ , and thus, component (B) can be computed efficiently exploiting the result above. To complete Equation (3), we need to estimate the
entropy  $H[f_{\theta}(X_j = z, X_{U \setminus j}, X_R = x_R)]$  (component A) for a specific instance  $z$  drawn from  $\operatorname{Pr}(X_j | X_R = x_R)$ . This poses a challenge due to the non-linearity of the hard model predictions  $f_{\theta}$  adopted. To tackle this computational challenge, we first estimate component A using soft labels  $\tilde{f}_{\theta}$  and then apply a thresholding operator. More specifically, we first estimate  $\operatorname{Pr}(\tilde{f}_{\theta}(X_j = z, X_{U \setminus \{j\}}, X_R = x_R))$  and, based on this distribution, we subsequently estimate  $f_{\theta}$  as  $1 \{\tilde{f}_{\theta} \geq 0\}$ , where  $\mathbf{1}$  is the indicator function. In the following sections, we will show how to assess this estimate for linear and non-linear classifiers. Finally, by approximating the distribution over soft model predictions through Monte Carlo sampling, the score function in  $F(X_j)$  can be computed as
$$
F \left(X _ {j}\right) \approx - ^ {1} / | \mathbf {Z} | \sum_ {z ^ {\prime} \in \mathbf {Z}} H \left[ f _ {\theta} \left(X _ {j} = z ^ {\prime}, X _ {U \backslash \{j \}}, X _ {R} = x _ {R}\right) \right], \tag {4}
$$
where  $Z$  is a set of random samples drawn from  $\operatorname*{Pr}(X_j|X_R = x_R)$  and estimated through Proposition 3, which thus can be computed efficiently.