SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The test checks if each reporting group  $\pmb{r}$  receives more accurate predictions from the personalized model assigned to its current node or  $\pmb{r}$  its parent  $\mathrm{pa}(\pmb{r})$ . Here,  $H_0$  assumes a reporting group prefers the parent model. Thus, we reject  $H_0$  when we can reliably tell that  $f_{T}(\cdot, \pmb{r})$  performs better for  $\pmb{r}$  on the pruning dataset. The exact test should chosen based on the performance metric for the underlying
For example, trees whose leaves contain at least one positive sample, one negative sample, and  $n_r \geq d + 1$  samples to avoid overfitting
prediction task. In general, we can use a bootstrap hypothesis test [20] and draw on more powerful tests for salient performance metrics [e.g., 19, 21, 50, for accuracy and AUC].
On Computation Our approach provides several options to moderate the computation cost of training a pool of models. For example, we can train only two models and build a minimal system. Alternatively, we can also build a flat or sequential system using a limited number of models in the pool. In practice, the primary bottleneck when building participatory systems is data rather than compute. Given a finite sample dataset, we are limited in the number of categorical attributes used for personalization. This is because we require a minimum number of samples for each intersectional group to train a personalized model and evaluate its performance. Given that the number of intersectional groups increases exponentially with each attribute, we quickly enter a regime where we cannot reliably evaluate model performance for assignment and pruning [see 42].
On Customization Our procedure allows practitioners to learn systems for prediction tasks by specifying the performance metric used in assignment and pruning. A suitable performance metric should represent the gains we would show users (e.g., error for a diagnosis, AUC for triage, ECE for risk assessment). Using a pool of models allows practitioners to optimize performance across groups, which translates to gains at the population level. For sequential systems, the procedure outputs all configurations, allowing practitioners to choose between systems based on criteria not known at training time. For example, one can swap the trees to use a system that always requests HIV status last. By default, we select the configuration that minimizes data collection across groups, such that the ordering of attributes results leads to the most significant number of data requests pruned.
We benchmark participatory systems on real-world clinical prediction tasks. Our goal is to evaluate these approaches in terms of performance, data usage, and consent in applications where individuals have a low reporting cost. We include code to reproduce these results in an Python library.
We consider six classification tasks for clinical decision support where we personalize a model with group attributes that are protected or sensitive (see Table 2 and Appendix B). Each task pertains to an application where we expect individuals to have a low cost of reporting and to report personal information when there is any expected gain. This is because the information used for personalization is readily available, relevant to the prediction task, and likely to be disclosed given legal protections related to the confidentiality of health data [4, 10, 54]. One exception is cardio_eicu and cardio_mimic, which are personalized based on race and ethnicity. We split each dataset into a test sample (20% for evaluating out-of-sample performance) and a training sample (80% for training, pruning, assignment, and estimating gains to show users). We train three kinds of personalized models for each dataset:
- Static: These models are personalized using a one-hot encoding of group attributes (1Hot), and a one-hot encoding of intersectional groups (mHot)
- Imputed: These are variants of static models where we impute the group membership for each person (KNN-1Hot, KNN-mHot). In practice, personalized systems with imputation will range between the performance for these systems and the performance of 1Hot and mHot.
- Participatory: These are participatory systems built using our approach. These include Minimal, a minimal system built from 1Hot and its generic counterpart; and Flat and Seq, flat and sequential systems built from 1Hot, mHot and their generic counterparts.
We train all models - personalized models and the components of participatory systems - from the same model class and evaluate them using the metrics in Table 1. We repeat the experiments four times, varying the model class (logistic regression, random forests) and the target performance metric (error rate for decision-making tasks, AUC for ranking tasks) to evaluate the sensitivity of our findings with respect to model classes and use cases.
The use of race in clinical risk scores should be approached with caution [60]; participatory systems offer one way to safeguard against inappropriate use.
Metric	Definition	Description
Overall Performance	∑ng/nRg(hg)g∈G	Population-level performance of a personalized system/model, computed as a weighted average over all groups
Overall Gain	∑ng/nΔg(g,∅)g∈G	Population-level gain in performance of a personalized system/model over its generic counterpart
Group Gains	ming∈G/ maxg∈GΔg(g,∅)	Range of group-level gains of a personalized system/model over its generic counterpart across all groups
Rationality Violations	∑I[reject H0]g∈G	Number of rationality violations detected using a bootstrap test with 100 resamples at a significance of 10% where H0: Δg(g,∅) ≥ 0.
Imputation Risk	ming∈GΔg(g,g')	Worst-case loss in performance due to incorrect imputation. This metric can only be computed for static models
Options Pruned	|R|−|R(h)|/|R|	Proportion of reporting options pruned from a system/model. Here, R denotes all reporting options and R(h) denotes those after h is pruned
Data Use	∑ng/nn=1ng/lim{h,g}g∈G	Proportion of group attributes requested by h from each group, averaged over all groups in G
Table 1: Metrics used to evaluate performance, data use, and consent of personalized models and systems. We report performance on a held-out test sample. We assume that individuals report group membership to static models, do not report group membership to imputed models, and only report to participatory systems when informed that it would lead to a strictly positive gain, as computed on the validation set in the training sample.
4.2 Discussion
We show results for logistic regression models and error rate in Table 2 and results for other model classes and classification tasks in Appendix C. In what follows, we discuss these results.
On Performance Our results in Table 2 show that participatory systems can improve performance across reporting groups. Here, Flat and Seq achieve the best overall performance on 6/6 datasets and improve the gains from personalization for every reporting group on 5/6 datasets. In contrast, traditional models improve overall performance while reducing performance at a group level (see rationality violations on five datasets for 1Hot, mHot). The performance benefits from participatory systems stem from (i) allowing users to opt out of these instances of "worsenalization" and (ii) assigning personalized predictions with multiple models. Using Table 2, we can measure the impact of (i) by comparing the performance of Minimal vs. 1Hot, and the impact of (ii) by comparing the performance of Minimal to Flat (or Seq). For example, on apnea, 1Hot exhibits a significant rationality violation for group [30_to_60,male], meaning they would have been better off with a generic model. By comparing the performance of 1Hot to Minimal, we see that allowing users to opt out of worsenalization reduces test error from  $29.1\%$  to  $28.9\%$ . By comparing the performance on Minimal to Flat and Seq, we see that using multiple models can further reduce test error from  $28.9\%$  to  $24.1\%$ .
On Informed Consent Our results show how Flat and Seq systems can inform consent by allowing users to report a subset of group attributes (e.g., by including reporting options such as  $[30+, \emptyset]$  or  $[\emptyset, \mathrm{HIV}+]$ ). Although both Flat and Seq systems allow for partial personalization, their capacity to inform consent differs. In a flat system, users may inaccurately gauge the marginal benefit of reporting an attribute by comparing the gains between reporting options. For example, in Fig. 4, users who are HIV positive would see a gain of  $3.7\%$  for reporting  $[\emptyset, \mathrm{HIV}+]$ , and  $16.7\%$  for reporting  $[30+, \mathrm{HIV}+]$  and may mistakenly conclude that the gain of reporting age is  $16.7\% - 3.7\% = 13.0\%$ . This estimate incorrectly presumes that the gains of  $3.7\%$  were distributed equally across age groups. Sequential systems directly inform users of the gains for partial reporting. In the sequential system, group  $[30+, \mathrm{HIV}+]$  is informed that they would see a marginal gain of  $21.5\%$  for reporting age, while group  $[<30, \mathrm{HIV}+]$  is informed they would see a marginal gain of reporting age of  $0.0\%$ .
On Data Minimization Our results show that participatory systems perform better across all groups while requesting less personal data on 6/6 datasets. For example, on cardio_eicu, Seq reduces error by  $11.3\%$  compared to 1Hot while requesting, on average,  $83.3\%$  of the data needed by 1Hot. In general, participatory systems can limit data use where personalization does not improve
Dataset	Metrics	STATIC	IMPUTED	PARTICIPATORY
1Hot	mHot	KNN-1Hot	KNN-mHot	Minimal	Flat	Seq
apnea	Overall Performance	29.1%	29.3%	29.0%	27.9%	28.9%	24.1%	24.3%
Overall Gain	0.1%	-0.1%	0.2%	1.3%	0.3%	5.1%	4.9%
n = 1152, d = 26	Group Gains	-1.1% - 1.2%	-0.8% - 0.4%	-1.1% - 1.2%	-0.8% - 0.4%	0.0% - 1.2%	0.0% - 13.8%	-0.4% - 13.8%
G = {age, sex}	Worsenalization	1	1	1	1	0	0	0
|G| = 6 groups	Imputation Risk	-4.9%	-5.2%
Ustun et al. [55]	Options Pruned	0/6	0/6	0/12	0/12	4/7	5/12	6/12
Data Use	100.0%	100.0%	0.0%	0.0%	33.3%	83.3%	58.3%
cardio_eicu	Overall Performance	21.4%	21.5%	21.6%	22.1%	21.6%	10.2%	10.2%
Overall Gain	0.4%	0.3%	0.3%	-0.2%	0.3%	11.7%	11.7%
n = 1341, d = 49	Group Gains	-1.3% - 2.6%	-2.7% - 3.0%	-1.3% - 2.6%	-2.7% - 3.0%	0.0% - 2.6%	3.1% - 20.9%	3.1% - 20.9%
G = {age, sex, race}	Worsenalization	1	1	1	1	0	0	0
|G| = 8 groups	Imputation Risk	-4.6%	-5.4%
Pollard et al. [44]	Options Pruned	0/8	0/8	0/27	0/27	6/9	10/27	9/27
Data Use	100.0%	100.0%	0.0%	0.0%	25.0%	100.0%	83.3%
cardio_mimic	Overall Performance	19.4%	19.3%	19.3%	20.1%	19.2%	15.7%	15.7%
Overall Gain	-0.1%	-0.0%	-0.0%	-0.8%	0.1%	3.5%	3.5%
n = 5289, d = 49	Group Gains	-0.9% - 0.4%	-0.9% - 0.5%	-0.9% - 0.4%	-0.9% - 0.5%	0.0% - 0.4%	-1.6% - 9.8%	-1.6% - 9.8%
G = {age, sex, race}	Worsenalization	3	2	3	2	0	1	1
|G| = 8 groups	Imputation Risk	-1.1%	-1.1%
Johnson et al. [32]	Options Pruned	0/8	0/8	0/27	0/27	6/9	6/27	8/27
Data Use	100.0%	100.0%	0.0%	0.0%	25.0%	100.0%	91.7%
coloncancer	Overall Performance	37.0%	36.7%	37.0%	36.9%	37.0%	36.6%	36.1%
Overall Gain	0.1%	0.4%	0.1%	0.2%	0.1%	0.5%	1.0%
n = 29211, d = 72	Group Gains	-0.4% - 0.3%	-0.1% - 1.1%	-0.4% - 0.3%	-0.1% - 1.1%	0.0% - 0.3%	0.0% - 1.7%	0.2% - 1.7%
G = {age, sex}	Worsenalization	1	0	1	0	0	0	0
|G| = 6 groups	Imputation Risk	-1.4%	-0.9%
Scosyrev et al. [45]	Options Pruned	0/6	0/6	0/12	0/12	5/7	7/12	5/12
Data Use	100.0%	100.0%	0.0%	0.0%	16.7%	50.0%	75.0%
lungcancer	Overall Performance	19.6%	19.6%	19.9%	19.8%	19.5%	18.9%	18.9%
Overall Gain	-0.1%	-0.1%	-0.3%	-0.2%	0.0%	0.6%	0.6%
n = 120641, d = 84	Group Gains	-0.4% - 0.2%	-0.3% - 0.2%	-0.4% - 0.2%	-0.3% - 0.2%	0.0% - 0.0%	0.0% - 0.9%	0.3% - 0.9%
G = {age, sex}	Worsenalization	4	4	4	4	0	0	0
|G| = 6 groups	Imputation Risk	-0.5%	-0.5%
Scosyrev et al. [45]	Options Pruned	0/6	0/6	0/12	0/12	6/7	3/12	7/12
Data Use	100.0%	100.0%	0.0%	0.0%	0.0%	83.3%	58.3%
saps	Overall Performance	20.4%	20.7%	20.4%	29.4%	20.4%	11.1%	11.1%
Overall Gain	1.3%	1.0%	1.3%	-7.7%	1.3%	10.6%	10.6%
n = 7797, d = 36	Group Gains	0.0% - 3.6%	0.0% - 2.7%	0.0% - 3.6%	0.0% - 2.7%	0.0% - 3.6%	4.3% - 17.2%	4.3% - 17.2%
G = {HIV, age}	Worsenalization	0	0	0	0	0	0	0
|G| = 4 groups	Imputation Risk	0.0%	-2.4%
Allyn et al. [3]	Options Pruned	0/4	0/4	0/9	0/9	1/5	1/9	3/9
Data Use	100.0%	100.0%	0.0%	0.0%	75.0%	100.0%	75.0%
Table 2: Participatory systems and personalized models for all datasets. We summarize metrics in Table 1 and present results for other model classes and prediction tasks in Appendix C. The best performance across each system is highlighted in green with bold text, and instances of worsening are highlighted in red.
performance, e.g., on lungcancer. Even as attributes like sex or age may be readily reported by patients for any performance benefit, limiting data use is valuable when there is a tangible cost associated with data collection – e.g., when models make use of rating scale for a mental disorder that must be administered by a clinician [47]. The potential for data minimization varies substantially across prediction tasks. On apnea, for example, we can prune six reporting options when building a Seq for decision making (which optimizes error) but four options for Seq for ranking (which optimizes AUC; see Appendix C.1). Overall, participatory systems satisfy “global data minimization” as proposed in [13], in that they minimize the amount of per-user data requested while achieving the quality of a system with access to the full data on average.
On the Benefits of a Model-Agnostic Approach Our findings highlight some of the benefits of a model-agnostic approach, in which we can draw on a rich set of models to achieve better performance while mitigating harm. The resulting system can balance training costs with performance benefits. We can also ensure generalization across reporting groups – e.g., by including a generic model fit from a complex model class and personalized models fit from a simpler model class. As expected, fitting for a complex model class can lead to considerable changes in overall accuracy – e.g., we can reduce overall test error for a personalized model from  $20.4\%$  to  $14.1\%$  on saps by fitting a random forest rather than a logistic regression model (see Appendix C). However, a gain in overall performance does not always translate to gains at the group level. On saps, for example, using a random forest also introduces a rationality violation for one group.
On the Pitfalls of Imputation One of the simplest approaches to allow individuals to opt out of personalization is to pair a personalized model with an imputation technique. Although this approach can facilitate consent, it may violate the requirements in 2. Consider a personalized
Figure 4: Participatory systems for the saps dataset. These models predict ICU mortality for groups defined by  $\mathcal{G} = \mathrm{HIV} \times \mathrm{age} = [+, -] \times [<30, 30+]$  using logistic regression component models. Here,  $h_0$  is a generic model,  $h_1$  is a 1Hot model fit with a one-hot encoding of  $\mathcal{G}$ , and  $h_2 \cdots h_m$  are 1Hot and mHot models fit for each reporting group. We show the gains of each reporting option above each box and highlight pruned options in grey. For example, in Seq, the group  $(\mathrm{HIV+}, 30+)$  sees an estimated  $21.5\%$  error reduction after reporting HIV if they report age. In contrast, the group  $(\mathrm{HIV+}, <30)$  sees no gain from reporting age in addition to HIV status, so this option is pruned.
model that exhibits "worsenalization" in Fig. 1. Even if one could correctly impute the group membership for every person, individuals may receive more accurate predictions from a generic model  $h_0$ . In practice, imputation is imperfect – as individuals who opt out of reporting their group membership to a personalized model may be assigned "worse" predictions because they are imputed the group membership of a different group. In such cases, opting out may be beneficial, making it difficult for model developers to promote participation while informing consent. Our results highlight the prevalence of these effects in practice. For example, on cardio_eicu the estimated "risk of imputation" is  $-4.6\%$ , indicating that every intersectional group can experience an increase of  $4.6\%$  in the error rate as a result of incorrect imputation. The results for KNN-1Hot show that this potential harm can be realized in practice using KNN-imputation, as we find that the imputed system leads to rationality violations on 5/6 datasets.
We introduced a new family of classification models that allow individuals to report personal data at prediction time. Our work focuses on personalization with group attributes; our approach could be used to facilitate and inform consent in a broader class of prediction tasks. In such cases, the key requirement for building a participatory system is that we can reliably estimate the gains of personalization for each person who reports personal data.
Our results show that participatory systems can inform consent while improving performance and reducing data use across groups. Reaping these benefits in practice will hinge on the ability to effectively inform decision subjects on the impact of their reporting decisions. [4]. Even as there may be good "default practices" for what kind of information we should show decision subjects, practitioners should tailor this information to the application and target audience [23].
One common concern in using a participatory system arises when practitioners wish to collect data from a model in deployment to improve its performance in the future. In practice, a participatory system can thwart data collection in such settings by allowing individuals to opt out. In such cases, we would note that this issue should be resolved in a way that is aligned with the principle of purpose specification [40]. If the goal of data collection is to improve a model, then individuals could always be asked to report information voluntarily for this purpose. If the goal of data collection is to personalize predictions, then individuals should be able to opt out, especially when it may lead to worse performance.
We thank the following individuals for helpful discussions: Taylor Joren, Sanmi Koyejo, Charlie Marx, Julian McAuley, and Nisarg Shah. This work was supported by funding from the National Science Foundation IIS 2040880, the NIH Bridge2AI Center Grant U54HG012510, and an Amazon Research Award.
A Supporting Material for Section 3 15
A.1 Enumeration Routine for Algorithm 1 15
A.2 Assignment Routine for Algorithm 1 15
A.3 Pruning Routine for Algorithm 1 16
A.4Greedy Induction of Sequential Reporting Interface 16
B Description of Datasets used in Section 4 - Experiments 17
C Results for Different Model Classes and Prediction Tasks 18
C.1 Logistic Regression for Ranking (AUC) 18
C.2 Random Forests for Decision-Making (Error) 19
C.3 Random Forests for Ranking (AUC) 20
D Supporting Material for Performance Profiles 21
A Supporting Material for Section 3
A.1 Enumeration Routine for Algorithm 1
We summarize the Enumeration routine in Algorithm 2. Algorithm 2 takes as input a set of group attributes  $\mathcal{G}$  and a dataset  $\mathcal{D}$  and outputs a collection of reporting interfaces  $\mathbb{T}$  that obey ordering and plausibility constraints.
Algorithm 2 Enumerate All Possible Reporting Trees for Reporting Options  $\mathcal{G}$
1: procedure VIABLETREES(G, D)  
2: if dim(G) = 1 return [TG] base case: we are left with only a single attribute on which to branch  
3: T ← []  
4: for each group attribute A ∈ [G1, ..., Gk] do  
5: TA ← reporting tree of depth 1 with |A| leaves  
6: S ← ViableTrees(G \ A, D) all subtrees using all attributes except A  
7: for Π in ValidAssignments(S, A, D) do: each assignment is a permutation of |A| to leaves of TA  
8: T ← T ∪ TA assign(Π) extends the tree by assigning subtrees to each leaf  
9: end for  
10: end for  
11: return T, reporting interfaces for group attributes G that obey plausibility and ordering constraints  
12: end procedure
The routine enumerates all possible reporting interfaces for a given set of group attributes  $\mathcal{G}$  through a recursive branching process. Given a set of group attributes, the routine is called for each attribute that has yet to be considered in the tree Line 4, ensuring a complete enumeration. We note that the routine is only called for building Sequential systems since there is only one possible reporting interface for Minimal and Flat systems.
Enumerating all possible trees ensures we can recover the best tree given the selection criteria and allows practitioners to choose between models based on other criteria. We generate trees that meet plausibility constraints based on the dataset, such as having at least one negative and one positive sample and at least  $s$  total samples at each leaf. In settings constrained by computational resources, we can impose additional stopping criteria and modify the ordering to enumerate more plausible trees first or exclusively (e.g., by changing the ordering of  $\mathcal{G}$  or imposing constraints in VALIDASSIGNMENTS).
A.2 Assignment Routine for Algorithm 1
We summarize the routine for AssignModels procedure in Algorithm 3.
Algorithm 3 Assigning Models
1: procedure ASSIGNMODELS(T, M, D)  
2:  $Q \gets [T.\text{root}]$  initialize with the root of the tree, reporting group ∅  
3: while  $Q$  is not empty do  
4:  $r \gets Q.pop()$   
5:  $\mathcal{M}_r \gets \text{ViableModels}(\mathcal{M}, r)$  filter  $\mathcal{M}$  to models that can be assigned to  $r$   
6:  $h^* \gets \operatorname{argmin}_{h \in \mathcal{M}_r} \hat{R}_r(h, \mathcal{D})$  assign the model with the best training performance  
7: T.set_model(r, h*)  
8: for  $r' \in T$ .get_subgroups(r) do  
9: Q.enqueue(r')  
10: end for  
11: end while  
12: return T that maximizes gain for each reporting group  
13: end procedure
Algorithm 3 takes as inputs a reporting tree  $T$ , a pool candidate models  $\mathcal{M}$ , and an assignment (training) dataset  $\mathcal{D}$  and outputs a tree  $T$  that maximizes the gains of reporting group information. The pool of candidate models is filtered to viable models for each reporting group. Since the pool of candidate models includes the generic model  $h_0$ , each reporting group will have at least one viable model. We assign each reporting group the best-performing model on the training set and default to the generic model  $h_0$  when a better-performing personalized model is not found. We assign performance on the training set and then prune using performance on the validation set to avoid biased gain estimations.
A.3 Pruning Routine for Algorithm 1
We summarize the routine used for the PruneLeaves procedure in Algorithm 1. The PruneLeaves routine
Algorithm 4 Pruning Participatory Systems
1: procedure PRUNELEAVES(T, D)  
2: Stack  $\leftarrow$  [T.leaves] initialize stack with all leaves  
3: repeat  
4:  $r \gets$  Stack.pop()  
5:  $h \gets$  T.get_model(r)  
6:  $h' \gets$  T.get_model(pa(r))  
7: if not Test(r, h, h', D) then test gains to see if parent model is as good as leaf model  
8: T.prune(r)  
9: end if  
10: if T.get_childen(pa(r)) is empty then consider pruning the parent if the parent has become a leaf  
11: Stack.enqueue(pa(r))  
12: end if  
13: until Stack is empty  
14: return T, reporting interface that ensures data collection leads to gain  
15: end procedure
Algorithm 1 takes as input a reporting interface  $T$  and a validation sample  $\mathcal{D}$ , and performs a bottom-up pruning to output a reporting interface  $T$  that asks individuals to report attributes that are expected to lead to a gain. The pruning decision at each leaf is based on a hypothesis test that evaluates the gains of reporting for a reporting group on a validation dataset. This test has the form:
$$
H _ {0}: R _ {g} (h) \leq R _ {g} \left(h ^ {\prime}\right) \quad \text {v s .} \quad H _ {A}: R _ {g} (h) > R _ {g} \left(h ^ {\prime}\right)
$$