SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hailey Joren  
UC San Diego
Chirag Nagpal  
Google Research
Katherine Heller Google Research
Berk Ustun  
UC San Diego
Machine learning models are often personalized with information that is protected, sensitive, self-reported, or costly to acquire. These models use information about people but do not facilitate nor inform their consent. Individuals cannot opt out of reporting personal information to a model, nor tell if they benefit from personalization in the first place. We introduce a family of classification models, called participatory systems, that let individuals opt into personalization at prediction time. We present a model-agnostic algorithm to learn participatory systems for personalization with categorical group attributes. We conduct a comprehensive empirical study of participatory systems in clinical prediction tasks, benchmarking them with common approaches for personalization and imputation. Our results demonstrate that participatory systems can facilitate and inform consent while improving performance and data use across all groups who report personal data.
Machine learning models routinely assign predictions to people – be it to screen a patient for a mental illness [35], their risk of mortality in an ICU [44], or their likelihood of responding to treatment [1]. Many models in such applications are designed to target heterogeneous subpopulations using features that explicitly encode personal information. Typically, models are personalized with categorical attributes that define groups [i.e., “categorization” as per 27]. In medicine, for example, clinical prediction models use group attributes that are protected (e.g., sex in the ASCVD Score for cardiovascular disease), sensitive (e.g., HIV_status in the VA COVID-19 Mortality Score), self-reported (e.g., alcohol_use in the HAS-BLED Score for Major Bleeding Risk), or costly to acquire (e.g., leukocytosis in the Alvarado Appendicitis Score).
Individuals expect the right to opt out of providing personal data and the ability to understand how it will be used [see, e.g., personal data guidelines in GDPR, OECD privacy guidelines 26, 40]. In many contexts, personalized models do not provide such functionality: individuals cannot opt out of reporting data used to personalize their predictions nor tell if it would improve their predictions. At the same time, practitioners assume that data available for training will be available at inference time. In practice, this assumption has led to a proliferation of models that use information that individuals may be unwilling or unable to report at prediction time [see e.g., the Denver HIV Risk Score 29, which asks patients to report age, gender, race, and sexual_practices]. In tasks where individuals self-report, they may not voluntarily report information that could improve their predictions or may report incorrect information.
The broader need to facilitate and inform consent in personalized prediction tasks stems from the fact that personalization may not improve performance for each group that reports personal data [51]. In practice, a personalized model can perform worse or the same as a generic model fit without personal information for a group with specific characteristics. Such models violate the implicit promise of personalization as individuals report personal information without receiving a tailored performance gain in return. These instances of "worsenalization" are prevalent, hard to detect, and hard to resolve [see 42, 51]. However, they would be resolved if individuals could opt out of personalization and understand its expected gains (see Fig. 1).
Figure 1: Classification task where participation improves accuracy and minimizes data use. We consider a dataset that has no features, two group attributes  $\mathcal{G} = \text{sex} \times \text{age}$ ,  $n^{-} = 51$  negative examples and  $n^{+} = 50$  positive examples. Here, the best personalized linear model  $h: \mathcal{X} \times \mathcal{G} \to \mathcal{Y}$  with a one-hot encoding of  $\mathcal{G}$  makes 24 mistakes, and the best generic model  $h_0: \mathcal{X} \times \mathcal{Y}$  makes 50 mistakes as it predicts the majority class  $(-)$ . Under traditional personalization, individuals report group membership to receive personalized predictions from  $h$ . As shown, personalization benefits the population as a whole by reducing overall error from 50 to 24  $(\Delta R_g(h, h_0) = 26)$ . However, personalization has a detrimental effect on [female, old], who receive less accurate predictions from the personalized model  $(\Delta R_g(h, h_0) = -24)$ , and no effect on [male, young] who receive the same predictions from the personalized and generic models  $(\Delta R_g(h, h_0) = 0)$ . In a minimal participatory system, individuals opt in to personalization, choosing to receive predictions from  $h$  or  $h_0$ . Here, individuals in groups [female, old] and [male, young] opt out of personalization, leading to an overall error of 0  $(\Delta R_g(h, h_0) = 50)$  and a reduction in unnecessary data collection  $(\varnothing)$ .
This work introduces a family of classification models that operationalize informed consent called participatory systems. Participatory systems facilitate consent by allowing individuals to report personal information at prediction time. Moreover, they inform consent by showing how reporting personal information will change their predictions. Models that facilitate consent operate as markets in which individuals trade personal information for performance gains. This work seeks to develop systems that perform as well as possible both when individuals opt-in – to incentivize voluntary reporting – and when they opt out – to safeguard against abstention. Our main contributions include:
1. We present a variety of participatory systems that provide opportunities for individuals to make informed decisions about data provision. Each system ensures that individuals who opt into personalization will receive the most accurate possible predictions possible.
2. We develop a model-agnostic algorithm to learn participatory systems. Our approach can produce a variety of systems that promote participation and minimize data use in deployment.
3. We conduct a comprehensive study of participatory systems in real-world clinical prediction tasks. The results show how our approach can facilitate and inform consent in a way that improves performance and minimizes data use.
4. We provide a Python library to build and evaluate participatory systems.
Related Work Participatory systems support modern principles of responsible data use articulated in OECD privacy guidelines [40], the GDPR [26], and the California Consumer Privacy Act [16]. These include: informed consent, i.e., that data should be collected with the data subject's consent; and collection limitation, i.e., that data collected should be restricted to only what is necessary. These principles stem from extensive work on the right to data privacy [33]. They are motivated, in part, by research showing that individuals care deeply about their ability to control personal data [4, 8, 10] but differ considerably in their desire or capacity to share it [see e.g. 5, 7, 9, 17, 18, 39, 41]. Our proposed systems let decision subjects report personal data in exchange for performance, which is aligned with principles articulated in recent work on data privacy [13, 46] and related to work in designing incentive-compatible prediction functions [24].
We consider models that are personalized with categorical attributes that encode personal characteristics [i.e., "categorization" rather than "individualization" as per 27]. Modern techniques for learning with categorical attributes [see e.g., 2, 48] use them to improve performance at a population level - e.g., by accounting for higher-order interaction effects [14, 38, 58] or recursive partitioning [11, 12, 15, 25]. Our methods can be used to achieve these goals in tasks where models use features that are optional or costly to acquire [see e.g., 6, 7, 52, 61].
Our work is related to algorithmic fairness in that we seek to improve model performance at a group level. Recent work shows that personalization with group attributes does not uniformly improve performance and can reduce accuracy at a group level [see 42, 51, 57]. Our systems can safeguard against such instances of "worsenalization" by informing users of the gains in reporting and allowing
them to opt out of reporting. This line of broad work complements research on preference-based fairness [22, 36, 57, 59, 62], on ensuring fairness across complex group structures [28, 30, 34], and promoting privacy across subpopulations [13, 53].
2 Participatory Systems
We consider a classification task where we personalize a model with categorical attributes. We start with a dataset  $\{(x_i, y_i, g_i)\}_{i=1}^n$  where each example consists of a feature vector  $x_i \in \mathbb{R}^d$ , a label  $y_i \in \mathcal{V}$ , and a vector of  $m$  categorical attributes  $g_i = [g_{i,1}, \ldots, g_{i,m}] \in \mathcal{G}_1 \times \ldots \times \mathcal{G}_m = \mathcal{G}$ . We refer to  $\mathcal{G}$  as group attributes, and to  $g_i$  as the group membership of person  $i$ . We use  $n_g := |\{i | g_i = g\}|$  denote the size of group  $g$ , and use  $|\mathcal{G}_k|$  to denote the number of categories for group attribute  $k$ .
We use the dataset to train a personalized model  $h: \mathcal{X} \times \mathcal{G} \to \mathcal{Y}$  via empirical risk minimization with a loss function  $\ell: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}_+$ . Given a model  $h$ , we denote its empirical risk and true risk as  $\hat{R}(h)$  and  $R(h)$ , respectively, and evaluate model performance at the group level. We denote the empirical risk and true risk of a model  $h$  on group  $g \in \mathcal{G}$  as
$$
R _ {\boldsymbol {g}} (h (\cdot , \boldsymbol {g})) := \mathbb {E} \left[ \ell \left(h (\cdot , \boldsymbol {g}), y\right) \right], \quad \hat {R} _ {\boldsymbol {g}} (h (\cdot , \boldsymbol {g})) := \frac {1}{n _ {\boldsymbol {g}}} \sum_ {i: \boldsymbol {g} _ {i} = \boldsymbol {g}} \ell \left(h (\cdot , \boldsymbol {g}), y _ {i}\right).
$$
We consider tasks where every individual prefers more accurate predictions.
Assumption 1. Given models  $h$  and  $h'$ , individuals in group  $g$  prefer  $h$  to  $h'$  when  $R_{g}(h) < R_{g}(h')$ .
Assumption 1 holds in settings where every individual prefers more accurate predictions – e.g., clinical prediction tasks such as screening or diagnosing illnesses [49, 56]. It does not hold in applications where some individuals prefer predictions that may be inaccurate – e.g., such as predicting the risk of organ failure for a transplant [see e.g., 43, for other “polar” clinical applications].
Operationalizing Consent We consider models where individuals consent to personalization by deciding whether or not to report their group attributes at prediction time. We let  $\varnothing$  denote an attribute that was not reported, and let  $r_i = [r_{i,1},\dots ,r_{i,k}]\in \mathcal{R}\subseteq \mathcal{G}\times \varnothing$ . For example, a person with  $g_{i} = [\mathrm{female},\mathrm{HIV} = +]$  would report  $r_i = [\mathrm{female},\varnothing ]$  if they only disclose sex, and would report  $r_i = \emptyset \coloneqq [\emptyset ,\emptyset ]$  if they opt out of reporting entirely.
We associate each model with a set of reporting options  $\mathcal{R}$ . A traditional model, which requires each person to report group attributes, has  $\mathcal{R} = \mathcal{G}$ . A model where each person could report any subset of group attributes has  $\mathcal{R} = \mathcal{G} \times \emptyset$ . We represent individual decisions to opt into personalization at prediction time through a reporting interface defined below.
Definition 1. Given a personalized classification task with group attributes  $\mathcal{G}$ , a reporting interface is a tree  $T$  whose nodes represent attributes reported at prediction time. The tree is rooted at  $\operatorname{root}(T) = [\varnothing, \dots, \varnothing]$  and branches as a person reports personal attributes. Given a node  $\boldsymbol{r}$ , we denote its parent as  $\mathrm{pa}(\boldsymbol{r})$ . Each parent-child pair represents a reporting decision, and the height of the tree represents the maximum number of reporting decisions.
Definition 2. Given a personalized classification task with group attributes  $\mathcal{G}$ , a participatory system with reporting interface  $T$  is a prediction model  $f_{T}:\mathcal{X}\times \mathcal{R}\to \mathcal{Y}$  that obeys the following properties:
(P1) Baseline Performance: Opting out of personalization entirely guarantees the expected performance from a generic model trained without group attributes  $h_0 \in \mathrm{argmin}_{h \in \mathcal{H}} R(h)$ .
$$
R _ {\boldsymbol {r}} \left(f _ {T} (\cdot , \emptyset)\right) = R _ {\boldsymbol {r}} \left(h _ {0}\right) \text {f o r a l l} \boldsymbol {r} \in \mathcal {R}.
$$
(P2) Incentive Compatibility: Opting into personalization improves expected performance
$$
R _ {\boldsymbol {r}} \left(f _ {T} (\cdot , \boldsymbol {r})\right) <   R _ {\boldsymbol {r}} \left(f _ {T} (\cdot , \boldsymbol {r} ^ {\prime})\right) \text {f o r a l l n e s t e d r e p o r t i n g g r o u p s} \boldsymbol {r}, \boldsymbol {r} ^ {\prime} \in \mathcal {G} \times \varnothing \text {s u c h t h a t} \boldsymbol {r} ^ {\prime} = \operatorname {p a} (\boldsymbol {r}).
$$
Here, the Baseline Performance property ensures that individuals who choose not to share personal information receive the performance of a generic model – i.e., the most accurate model that could be trained without this information. This property also ensures individuals retain the ability to opt out
Figure 2: Participatory systems for a personalized classification task with group attributes sex  $\times$  age = [male, female]  $\times$  [old, young]. Each system allows a person to opt out of personalization by reporting  $\varnothing$  and informs their choice by showing the expected gains of personalization (e.g.,  $+0.2\%$  gain in accuracy). Systems minimize data use by removing reporting options that do not improve accuracy (see grey-striped boxes). Here, [young, female] is pruned in all systems as it leads to a gain  $\leq 0.0\%$ .
of personalization - i.e.,  $\varnothing \in \mathcal{R}$ . The Incentive Compatibility property ensures that personalization will improve expected performance - i.e., when individuals report personal data, the system can effectively leverage that data to deliver more accurate predictions in expectation. Together, these properties lead to data minimization, as systems that obey these properties will not request data from a reporting group when it will not lead to an improvement in expected performance.
On Data Minimization via Imputation An alternative approach to allow individuals to opt out of reporting personal information at prediction time is to impute their group membership. Imputation allows individuals to opt out of personalization but does not guarantee the accuracy of their predictions. As a result, individuals who opt out of personalization by reporting  $r = \varnothing$  may receive a less accurate prediction than they would receive from a generic model. In the best-case scenario where we could perfectly impute group membership, a group might be assigned better predictions from a generic model (see Fig. 1). In the worst case, imputation may be incorrect, leading to even more inaccurate predictions than those of the generic or personalized model. We highlight these effects on real-world datasets in our experiments in Section 4.
Characterizing System Performance One of the key differences between traditional models and participatory systems is that their performance depends on individual reporting decisions. In what follows, we characterize the performance under a general model of individual disclosure. Given a participatory system  $f_{T}$ , we assume that each individual reports personal information to maximize an individual utility function of the form:
$$
u _ {i} (\boldsymbol {r}; f _ {T}) = b _ {i} (\boldsymbol {r}; f _ {T}) - c _ {i} (\boldsymbol {r}) \tag {1}
$$
Here,  $c_{i}(\cdot)$  and  $b_{i}(\cdot)$  denote the cost and benefit that individual  $i$  receives from reporting  $\pmb{r}$  to  $f_{T}$  respectively. We assume that individuals incur no cost when they do not report any attributes such that  $c_{i}(\emptyset) = 0$ , and incur costs that increase monotonically with information disclosed such that  $c_{i}(\pmb{r}) \leq c_{i}(\pmb{r}^{\prime})$  for  $\pmb{r} \subseteq \pmb{r}^{\prime}$ . We assume that benefits increase monotonically with expected gains in true risk so that  $R_{r}(f_{T}(\cdot, \pmb{r})) < R_{r}(f_{T}(\cdot, \pmb{r}^{\prime})) \Rightarrow b_{i}(\pmb{r}, f_{T}) > b_{i}(\pmb{r}^{\prime}, f_{T})$ .
In Fig. 3, we show how the system performance for each reporting group can change with respect to participation when we simulate individual disclosure decisions from a model that satisfies the assumptions listed above. When a personalized model  $h$  requires individuals to report information that reduces performance as in Fig. 1, individuals incur a cost of disclosure without receiving a benefit in return. In such cases, individuals who interact with a minimal system would opt out of worsening normalization and receive more accurate predictions from a generic model, thereby improving the overall performance of the system.
We observe that the maximum utility that each individual can receive from a participatory system can only increase as we add more reporting options. Thus, flat and sequential systems should exhibit better performance than a minimal system.
Given a participatory system  $f_{T}$  with reporting options  $\mathcal{R}$ , a participatory system  $f_{T'}$  with more reporting options  $\mathcal{R}' \supseteq \mathcal{R}$  can only improve performance, - i.e.,  $R(f_{T'}) \leq R(f_{T})$ . Similarly, the system with more reporting options can only improve utility, - i.e.,  $u_{i}(\boldsymbol{r}; f_{T'}) \geq u_{i}(\boldsymbol{r}; f_{T})$  for all individuals  $i$ .
$[< 30,\mathrm{HIV} + ]$
[>30,HIV-]
[>30,HIV-]
Figure 3: Performance profile of participatory systems for the saps dataset for each intersectional group in the saps dataset. We plot out-of-sample performance for different levels of participation in the target population. We control participation by varying the reporting cost in a simulated model of individual disclosure. As shown, minimal and sequential systems outperform a generic model at a group level regardless of participation. In regimes where the cost of disclosure is low, participation is high. Consequently, a minimal system will achieve the same performance as a personalized model, and a sequential system will achieve the performance of the component model for each subgroup. We provide details and results in Appendix D.
3 Learning Participatory Systems
This section describes a model-agnostic algorithm to learn participatory systems that ensures incentive compatibility and baseline performance in deployment. We outline our procedure in Algorithm 1 to learn the three kinds of participatory systems in Fig. 2. The procedure takes as input a pool of candidate models  $\mathcal{M}$ , a dataset for model assignment  $\mathcal{D}^{\mathrm{assign}}$ , and a dataset for pruning  $\mathcal{D}^{\mathrm{prune}}$ . It outputs a collection of participatory systems that obey the properties described in Definition 2 on test data. The procedure combines three routines to (1) generate viable reporting interfaces (Line 1); (2) assign models over the interface (Line 3); (3) prune the system to limit unnecessary data collection (Line 4). We present complete procedures for each routine in Appendix A and discuss them below.
Algorithm 1 Learning Participatory Systems
Input:  $\mathcal{M}:\{h:\mathcal{X}\times \mathcal{G}\to \mathcal{Y}\}$  pool of candidate models Input:  $\mathcal{D}^{\mathrm{assign}} = \{(x_i,g_i,y_i)\}_{i = 1}^{n_{\mathrm{assign}}}.$  assignment dataset Input:  $\mathcal{D}^{\mathrm{prune}} = \{(\pmb {x}_i,\pmb {g}_i,y_i)\}_{i = 1}^{n_{\mathrm{prune}}}$  pruning dataset   
1:  $\mathbb{T}\gets$  ViableTrees(G,  $\mathcal{D}^{\mathrm{assign}}$  )  $|\mathbb{T}| = 1$  for minimal & flat systems   
2: for  $T\in \mathbb{T}$  do   
3:  $T\gets$  AssignModels(T,M,  $\mathcal{D}^{\mathrm{assign}}$  ) assign models   
4:  $T\gets$  PruneLeaves(T,  $\mathcal{D}^{\mathrm{prune}}$  ) prune models   
5: end for   
Output T, collection of participatory systems
Model Pool Our procedure takes as input a pool of candidate models  $\mathcal{M}$  to assign over a reporting interface. At a minimum, every pool should contain two models: a personalized model  $h$  for individuals who opt into personalization, and a generic model  $h_0$  for individuals who opt out of personalization. A single personalized model can perform unreliable across reporting groups due to differences in the data distribution or trade-offs between groups. Using a pool of models safeguards against these effects by drawing on models from different model classes that have been personalized using different techniques for each reporting group. By default, we include models trained specifically on the data for each reporting group, as such models can perform well on heterogeneous subgroups [51, 57].
Enumerating Interfaces We call the ViableTrees routine in Line 1 to enumerate viable reporting interfaces. We only call this routine for sequential systems since minimal and flat systems use a single reporting interface that is known a priori. ViableTrees takes as input a group attributes  $\mathcal{G}$  and a dataset  $\mathcal{D}^{\mathrm{assign}}$ . It returns all  $m$ -ary trees that obey constraints on sample size and reporting (e.g., users who report male should report age before HIV). By default, we only generate trees so that we have sufficient data to estimate gains at each node of the reporting interface<sup>1</sup>. In general, ViableTrees scales to tasks with  $\leq 8$  group attributes. Beyond this limit, one can reduce the enumeration size by specifying ordering constraints or a threshold number of trees to enumerate before stopping. For a task with three binary group attributes,  $\mathbb{T}$  contains 24 3-ary trees of depth 3. Given a complete ordering of all 3 group attributes, however,  $\mathbb{T}$  would have 1 tree. We can also consider a greedy algorithm (see Appendix A.4), which may be practical for large-scale problems.
Model Assignment We assign each reporting group a model using the AssignModels routine in Line 3. Given a reporting group  $r$ , we consider all models that could use any subset of group attributes in  $r$ . Thus, a group that reports age and sex could be assigned predictions from a model that requires age, sex, both, or neither. This implies that we can always assign the generic model to any reporting group, ensuring that the model at each node performs as well as the generic model on out-of-sample data (i.e., baseline performance in Definition 2).
Pruning Reporting Options AssignModels may output trees that violate incentive compatibility by requesting personal information that fails to improve performance. This can happen when the routine assigns a model that performs equally well to nested reporting groups – see, e.g., Fig. 2 where the Flat system assigns  $h_0$  to [female,  $\varnothing$ ] and [female, young].
We can avoid requesting data from reporting groups in such cases by calling the Prune routine in Line 4. This routine takes as input a participatory system  $f_{T}$  and a pruning dataset  $\mathcal{D}^{\mathrm{prune}}$  and outputs a system  $f_{T'}$  with a pruned interface  $T' \subseteq T$ . The routine uses a bottom-up pruning procedure that calls a one-sided hypothesis test at each node:
$$
H _ {0}: \Delta_ {\boldsymbol {r}} (\boldsymbol {r}, \operatorname {p a} (\boldsymbol {r})) \leq 0 \quad H _ {A}: \Delta_ {\boldsymbol {r}} (\boldsymbol {r}, \operatorname {p a} (\boldsymbol {r})) > 0
$$