Participatory Personalization in Classification
Authors: Hailey Joren
Chirag Nagpal
Katherine A Heller
Berk Ustun
Venue: NeurIPS 2023 spotlight (2023)
Keywords: healthcare
algorithmic fairness
data privacy
classification
interpretability
TL;DR: We introduce a new family of prediction models that let individuals opt into personalization at prediction time.

Abstract: Machine learning models are often personalized based on information that is protected, sensitive, self-reported, or costly to acquire. These models use information about people, but do not facilitate nor inform their *consent*. Individuals cannot opt out of reporting information that a model needs to personalize their predictions nor tell if they benefit from personalization in the first place. We introduce a new family of prediction models, called participatory systems, that let individuals opt into personalization at prediction time. We present a model-agnostic algorithm to learn participatory systems for supervised learning tasks where models are personalized with categorical group attributes. We conduct a comprehensive empirical study of participatory systems in clinical prediction tasks, comparing them to common approaches for personalization and imputation. Our results show that participatory systems can facilitate and inform consent in a way that improves performance and privacy across all groups who report personal data.

Participatory Personalization in Classification
Hailey Joren UC San Diego
Chirag Nagpal Google Research
Katherine Heller Google Research
Berk Ustun UC San Diego
Abstract
Machine learning models are often personalized with information that is protected, sensitive, self-reported, or costly to acquire. These models use information about people but do not facilitate inform their consent. Individuals cannot opt out reporting personal information to a model, nor tell if they benefit from person- alization in the first place. We introduce family of classification models, called participatory systems, that let individuals opt into personalization at prediction time. We present model-agnostic algorithm learn participatory systems for personalization with categorical group attributes. We conduct a comprehensive empirical study of participatory systems in clinical prediction tasks, benchmarking them with common approaches for personalization and imputation. Our results demonstrate that participatory systems can facilitate and inform consent while improving performance and data use across all groups who report personal data.
Introduction
Machine learning models routinely assign predictions to people be to screen a patient for mental illness [35], their risk of mortality in ICU [44], or their likelihood of responding to treatment [1]. Many models in such applications are designed to target heterogeneous subpopulations using features that explicitly encode personal information. Typically, models personalized with categorical attributes that define groups (i.e., "categorization" as per 27]. In medicine, for example, clinical prediction models use group attributes that are protected (e.g., sex in the ASCVD Score for cardiovascular disease), sensitive (e.g., div_status in the VA COVID-19 Mortality Score), self-reported (e.g., alcchcl_useeeeeeee in the HAS-BLED Score for Major Bleeding Risk), or costly to acquire (e.g., Leukocytosis is in the Alvarado Appendicitis Score).
Individuals expect the right to opt out of providing personal data and the ability to understand how will be used [see, c.g., personal data guidelines in GDPR, OECD privacy guidelines 26, 40]. In many contexts, personalized models do not provide such functionality: individuals cannot opt of reporting data used to personalize their predictions nor tell if it would improve their predictions. At the same time, practitioners assume that data available for training will be available inference time. In practice, this assumption has led to a proliferation of models that use information that individuals may be unwilling or unable report prediction time [see e.g., the Denver HIV Risk Score 29, which asks patients to report age, gender, race, and sexual_prace i ces]. In tasks where individuals self-report, they may not voluntarily report information that could improve their predictions or may report incorrect information.
The broader need facilitate and inform consent in personalized prediction tasks stems from the fact that personalization may not improve performance for each group that reports personal data [51]. In practice, personalized model can perform worse the same a generic model fit without personal information for a group with specific characteristics. Such models violate the implicit individuals report personal information without receiving tailored
promise of personalization performance gain in return. These instances of "worsenalization" are prevalent, hard to detect, and hard to resolve [see 42, 51]. However, they would be resolved if individuals could opt of personalization and understand its expected gains (see Fig. 1).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Traditional Persoralization
Mirimall Participatory System
groups receive predictonn from h
groups opt into predictions from or los
Group
Data
Personlizedd
Generic
Model
Tarta Use
Gain
Model
Data Use
Gain
h
Rigth)
ho
Righh)
(b.,
AR,
sensin, sid
24
24
0
fenale, old
-21
ha
25 fuxale, younq
0
tene
young
25
25
femala, yousa
25
pell,ott
mala, young
Total
50
Figure 1: Classification task where participation improves accuracy and minimizes data use. We consider a
dataset that has no features, two group attributes 9 sex x age, n = 51 negative examples and n 50
positive examples. Here, the best personalized linear model / X x G - y with a one-hot encoding of g makes 24 mistakes, and the best generic model ho X' x y' makes 50 mistakes as it predicts the majority class (-). Under traditional personalization, individuals report group memhership to receive personalized predictions from h. As shown, personalization benefits the population as a whole by reducing overall error from 50 to 24 (AR (h, An) = 26). However, personalization has a detrimental effect on [female, ola], who receive less accurate predictions from the personalized model (ARg (h, /t)
- 24), and no effect on [male, young] who receive the same predictions from the personalized and generic models( Rg (h, ho) 0). In minimal participatory system, individuals opt in to personalization, choosing to receive predictions from h or ho. Here, individuals in groups [female, e, old] and [male, young] opt out of personalization, leading to an overall error of O (A Rg (h, ho) = 50) and reduction in unnecessary data collection (Ø).
This work introduces a family of classification models that operationalize informed consent called participatory systems. Participatory systems facilitate consent by allowing individuals to report personal information at prediction time. Moreover, they inform consent by showing how reporting personal information will change their predictions. Models that facilitate consent operate as markets in which individuals trade personal information for performance gains. This work seeks to develop systems that perform as well as possible both when individuals opt-in to incentivize voluntary reporting - and when they opt out to safeguard against abstention. Our main contributions include:
1. We present variety of participatory systems that provide opportunities for individuals to make informed decisions about data provision. Each system ensures that individuals who opt into personalization will receive the most accurate possible predictions possible.
2. We develop a model-agnostic algorithm to learn participatory systems. Our approach can produce variety of systems that promote participation and minimize data use in deployment.
3. We conduct a comprehensive study of participatory systems in real-world clinical prediction tasks. The results show how our approach can facilitate and inform consent in a way that improves performance and minimizes data use.
4. We provide a Python library to build and evaluate participatory systems.
Related Work Participatory systems support modern principles of responsible data use articulated in OECD privacy guidelines [40], the GDPR [26], and the California Consumer Privacy Act [16]. These include: informed consent, i.c., that data should be collected with the data subject's consent; and collection limitation, i.e., that data collected should be restricted to only what is necessary. These principles stem from extensive work on the right to data privacy [33]. They are motivated, in part, by research showing that individuals care deeply about their ability to control personal data [4, 8, 10] but differ considerably in their desire or capacity to share it [see e.g. 5, 7, 9, 17, 18, 39, 41]. Our proposed systems let decision subjects report personal data in exchange for performance, which is aligned with principles articulated in recent work on data privacy [13, 46] and related to work in designing incentive-compatible prediction functions [24].
We consider models that are personalized with categorical attributes that encode personal charac- teristics fi.e.. "categorization" rather than "individualization"" as per 27J. Modern techniques for learning with categorical attributes [see e.g., 2, 48] use them to improve performance at a population level e.g., by accounting for higher-order interaction effects [14, 38, 58] recursive partition- ing [11, 12, 15, 25]. Our methods can be used to achieve these goals in tasks where models use features that are optional costly to acquire [see e.g., 6, 7, 52, 61].
Our work related to algorithmic fairness in that we seek to improve model performance at group level. Recent work shows that personalization with group attributes does not uniformly improve performance and can reduce accuracy at group level [see 42, 51, 57]. Our systems can safeguard against such instances of "worsenalization" by informing users of the gains in reporting and allowing
2

---

them to opt out of reporting This line of broad work complements research on preference- based fairness [22, 36, 57, 59, 62]. on ensuring fairness across complex group structures [28, 30, 34], and promoting privacy across subpopulations [13, 53].
2 Participatory Systems
We consider a classification task where we personalize a model with categorical attributes. We start with dataset { Mi. 9=) where each example consists of feature vector æ; E IR label
U: E y, and a vector of m categorical attributes 9; = [9.,1,.... Iim E 91 x... X 9m = G. We refer to G as group aftributes, and to 9: the group membership of person i. We use Ry == K [g: = g}l
denote the size of group and use |9&| to denote the number of categories for group attribute k.
We use the dataset to train personalized model h X G + Y' via empirical risk minimization with a loss function L y x y + Rt. Given a model h, we denote its empirical risk and true risk as R(h) and R(h), respectively, and evaluate model performance at the group level. We denote the empirical risk and true risk of model on group 9 as
1 Σ ()((-g).
Rg (h( g)) = ul)
Rg Rg(h(-99)) =
Ti ig:
We consider tasks where every individual prefers more accurate predictions. Assumption 1. Given models h and h', individuals in group g prefer h to h when Rg(h) < Rg (h).
Assumption 1 holds in settings where every individual prefers more accurate predictions e.g., clinical prediction tasks such as screening or diagnosing illnesses [49, 56]. It does not hold in applications where some individuals prefer predictions that may be inaccurate e.g., such as predicting the risk of organ failure for a transplant [see e.g., 43, for other "polar" clinical applications].
Operationalizing Consent We consider models where individuals consent to personalization by deciding whether or not to report their group attributes at prediction time. We let ø denote an attribute that was not reported, and let r; [r.., Tt.k] R S G Ø. For example, a person with g: [female 1e, HIV = +] would report = [female, le, 2) if they only disclose sex, and would report r; = [0, Ø they opt out of reporting entirely.
We associate each model with a set of reporting options R. A traditional model, which requires each person to report group attributes, has R = G. A model where each person could report any subset of group attributes has R = G x Ø. We represent individual decisions to opt into personalization at prediction time through reporting interface defined below.
Definition 1. Given personalized classification task with group attributes G. reporting interface is a tree T whose nodes represent attributes reported at prediction time. The tree is rooted at root(T) = [Ø.... ... Ø) and branches person reports personal attributes. Given node r, denote its parent as pr) Each parent-child pair represents a reporting decision, and the height of the tree represents the maximum number of reporting decisions.
Definition 2. Given a personalized classification task with group attributes G, a participatory system with reporting interface T is prediction model fr X' x R + y that obeys the following properties:
(P1) Baseline Performance: Opting out of personalization entirely guarantees the expected perfor- mance from generic model trained without group attributes ho E argmin R(h).
Rx(fr(,,P) (ho) for all reporting groups r E R.
(P2) Incentive Compatibility: Opting into personalization improves expected performance R,(fe(sr)) (,,((( for all nested reporting groups r, r' E G x Ø such that r' pa(r).
Here, the Buseline Performance property ensures that individuals who choose not to share personal information receive the performance of a generic model - i.e., the most accurate model that could be trained without this information. This property also ensures individuals retain the ability to opt out
3

---

Interface
Example
Description
Minimal systems let individuals opt out of personalization existing personalized model h. Individuals who ope receive predictions from generic model Ag trained without group attributes.
Minimal
Arg
Pro
Flat systems support partial personalization by allowing each person to report any subset of group attributes. Thus, person with g. [old, remale] can report [olc, 2) These systems performance by using distinct model to assign personalized predictions to each reporting group.
Flat
old HOUSS
improve
Rts
As
Sequential systens support partial personalization while allowing individuuls to report one attribute at time. This interface well-suited inform consent can make reporting decisions by compuring two models simultancously. They are also well-suited for personalization tasks with information that must be acquired prediction time (e.g.. g., the outeome of test result).
T
Sequential
Agg
Copala make
Figure 2: Participatory systems for a personalized classification task with group attributes sex age Imale, female] [old, young]. Each system allows person opt out of personalization by reporting and informs their choice by showing the expected gains of personalization (e.g.. +0 22 gain in accuracy), Systems minimize data use by removing reporting options that not improve accuracy (see grey-striped boxes). Here, [young, femal [e] is pruned in all systems as it leads to a gain S 0.0%.
of personalization - i.e., Ø R. The Incentive Compatibility property ensures that personalization will improve expected performance - i.e., when individuals report personal data, the system can effectively leverage that data deliver more accurate predictions in expectation. Together, these properties lead to data minimization, as systems that obey these properties will not request data from a reporting group when it will not lead to an improvement in expected performance.
On Data Minimization via Imputation An alternative approach to allow individuals to opt out of reporting personal information at prediction time is impute their group membership. Imputation allows individuals to opt out of personalization but does not guarantee the accuracy of their predictions. As result, individuals who opt out of personalization by reporting r = may receive a less accurate prediction than they would receive from generic model. In the best-case scenario where we could perfectly impute group membership, group might be assigned better predictions from a generic model (see Fig. 1). In the worst case, imputation may be incorrect, leading to even more inaccurate predictions than those of the generic or personalized model. We highlight these effects on real- worlw datasets in our experiments in Section 4.
Characterizing System Performance One of the key differences between traditional models and participatory systems is that their performance depends on individual reporting decisions. In what follows, we characterize the performance under general model of individual disclosure. Given participatory system fT, we assume that each individual reports personal information to maximize an individual utility function of the form:
<:(r; (T: b:(r:: (r
Here, c:(-) and b-()) denote the cost and benefit that individual i receives from reporting to fr respectively. We assume that individuals incur no cost when they do not report any attributes such that cx Ø = 0, and incur costs that increase monotonically with information disclosed such that e, (r) S c((r) for C r' We assume that benefits increase monotonically with expected gains in true risk so that (f(((r) < (r(f(,rr') = b,(r. fr) b,(r', fr).

---

In Fig. 3, we show how the system performance for each reporting group can change with respect to participation when we simulate individual disclosure decisions from model that satisfies the assumptions listed above. When a personalized model h requires individuals to report information that reduces performance as in Fig. 1, individuals incur cost of disclosure without receiving benefit in return. In such cases, individuals who interact with a minimal system would opt out of worsenalization and receive more accurate predictions from generic model, thereby improving the overall performance of the system.
We observe that the maximum utility that each individual can receive from a participatory system can only increase we add more reporting options. Thus, flat and sequential systems should exhibit better performance than a minimal system.
Given a participatory system Jr with reporting options R. a participatory system fr. with more reporting options R' 2 R. only improve performance, i.e., R(fr)) I R(fr). Similarly, the system with more reporting options can only improve utility. - i.e., (r fT)) 2 us(r; fr) for all individuals
<<30, HIV+]
[>30, HIV-]
100
+ Mnniaal
Minimal
Sequant'al
Sequential
98
Generc:
82.5
Generi i
96
Personslizdd
Acountyy (K)
Accuraay (N)
80
94
72.5
70:
0.2
0.1
0.3
Disclosure
Cost of Dieclosure
[<30, HIV+]
[>30, BIV-]
Minimal
Minimal
Sequontial
79-
Sequential
Gareriic
Parsoralizad
Parsonalized
Accurary (5))
68
Accuracy (5g
78
86
76
75
0.4
0.5
0.1
0.2
Cost Didcosuee
Disciosura
Figure Performance profile of participatory systems for the saps dataset for each intersectional group in the saps dataset. We plot oft-of-sample performance for different levels participation the target population. We control participation by varying the reporting cost in simulated model of individual disclosure. As shown, minimal and sequential systems outperform a generic model at group level regardless of participation. In regimes where the cost of disclosure is low, participation is high. Consequently, minimal system will achieve personalized model, and sequential system will achieve the performance of the
the same performance component model each supgroup. provide details and results Appendix
3 Learning Participatory Systems
This section describes model-agnostic algorithm to learn participatory systems that ensures incentive compatibility and baseline performance in deployment. We outline our procedure in Algorithm learn the three kinds of participatory systems Fig. 2. The procedure takes as input pool of candidate models M, a dataset for model assignment Dassign and dataset for pruning Dirine It outputs a collection participatory systems that obey the properties described in Definition on test data. The procedure combines three routines to (1) generate viable reporting interfaces (Line 1); (2) assign models over the interface (Line 3); (3) prune the system to limit unnecessary data collection (Line 4). We present complete procedures for each routine in Appendix A and discuss them below.
5

---

Algorithm I Learning Participatory Systems
Input: M {h: X
3
poot of condidate models
Input: Dassign = ( Ii, Vi) 1
assigument dataset
Input: D {(x), 9:. yr) m
km1
prining dataset
1: T <- Viable DDG.,DDee Dussgn)
[T|
for minimal & flat systems assign models
2: for T E do
3:
T AssignModels((1 .M, D
4: T <- (runeLeaves(Z; D
prame models
5: end for
Output T, collection of participatory systems
Model Pool Our procedure takes as input a pool of candidate models M to assign over a reporting interface. At minimum, every pool should contain two models: personalized model h for individuals who opt into personalization, and a generic model hg for individuals who opt out of personalization. A single personalized model can perform unreliably across reporting groups due to differences the data distribution or trade-offs between groups. Using a pool of models safeguards against these effects by drawing on models from different model classes that have been personalized using different techniques for each reporting group. By default, we include models trained specifically the data for each reporting group, as such models can perform well on heterogeneous subgroups 57].
Enumerating Interfaces We call the ViableTrees routine in Line to enumerate viable reporting interfaces. We only call this routine for sequential systems since minimal and flat systems use single reporting interface that known priori. Viable Trees takes as input a group attributes G and dataset Dessign, It returns all m-ary trees that obey constraints on sample size and reporting (e.g.. users who report male should report age before HIV). By default, we only generate trees so that we have sufficient data to estimate gains at each node of the reporting interface In general, WiableTrees scales to tasks with S 8 group attributes. Beyond this limit, one can reduce the enumeration size by specifying ordering constraints a task with three binary group attributes, T contains 24 3-ary trees of depth Given a complete ordering of all 3 group attributes, however, T would have 1 tree. We also consider greedy algorithm (see Appendix A.4), which may be practical for large-scale problems.
threshold number of trees to enumerate before stopping, For
Model Assigmment We assign each reporting group a model using the AssignModels routine in Line 3. Given a reporting group r, we consider all models that could use any subset of group attributes in r. Thus. a group that reports age and sex could be assigned predictions from a model that requires age, sex, both, or neither. This implies that we can always assign the generic model to any reporting group, ensuring that the model each node performs as well the generic model on -atple data (i.e., baseline performance in Definition 2).
Pruning Reporting Options AssignModels may output trees that violate incentive compatibility by requesting personal information that fails to improve performance. This can happen when the routine assigns a model that performs equally well to nested reporting groups see, e.g., Fig. 2 where the Flat system assigns hg to [female, Ø] and [semale, young].
We can avoid requesting data from reporting groups in such cases by calling the Prune routine in Line 4. This routine takes as input a participatory system fr and pruning dataset Domine and outputs a system fr with a pruned interface T" C T. The routine uses bottom-up pruning procedure that calls a one-sided hypothesis test at each node:
Ho A, (r. pa(r)) 0
HA (r(r, > 0
The test checks if each reporting group receives more accurate predictions from the personalized model assigned to its current node or r its parent pa(r). Here, Ho assumes reporting group prefers the parent model. Thus, we reject Ho when we can reliably tell that fr(.r) performs better for on the pruning dataset. The exact test should chosen based on the performance metric for the underlying
For example, trees whose leaves contain at least one positive sample, one negative sample, and n, 2 d + samples avoid overtitting

---

prediction task. In general, we can use a bootstrap hypothesis test [20] and draw on more powerful tests for salient performance metrics [e.g., 19, 21. 50, for accuracy and AUC).
On Computation Our approach provides several options to moderate the computation cost of training a pool of models. For example, we can train only two models and build a minimal system. Alternatively, we can also build flat or sequential system using limited number of models in the pool. In practice, the primary bottleneck when building participatory systems is data rather than compute. Given finite sample dataset, we are limited in the number of categorical attributes used for personalization. This is because we require a minimum number of samples for each intersectional group to train personalized model and evaluate its performance. Given that the number of intersectional groups increases exponentially with each attribute, we quickly enter a regime where we cannot reliably evaluate model performance for assignment and pruning [see 42].
On Customization Our procedure allows practitioners to learn systems for prediction tasks by specifying the performance metric used in assignment and pruning. A suitable performance metric should represent the gains we would show users (e.g., error for a diagnosis, AUC for triage, ECE for risk assessment). Using a pool of models allows practitioners to optimize performance across groups, which translates to gains at the population level. For sequential systems, the procedure outputs all configurations, allowing practitioners to choose between systems based on criteria not known at training time. For example, one can swap the trees to use a system that always requests HIV status last. By default, we select the configuration that minimizes data collection across groups, such that the ordering of attributes results leads to the most significant number of data requests pruned.
4 Experiments
We benchmark participatory systems on real-world clinical prediction tasks. Our goal is to evaluate these approaches in terms of performance, data usage, and consent in applications where individuals have a low reporting cost. We include code to reproduce these results in an Python library.
4.1 Setup
We consider six classification tasks for clinical decision support where personalize model with group attributes that are protected or sensitive (see Table and Appendix B). Each task pertains to an application where we expect individuals have low cost of reporting and report personal information when there is any expected gain. This is because the information used for personalization is readily available, relevant to the prediction task, and likely be disclosed given legal protec- tions related to the confidentiality of health data [4, 10, 54). One exception is cardi o_e cu and cardio mimmmmmmmm...mmmmm c, which are personalized based on race and ethnicity. We split each dataset into test sample (20% for evaluating out-of-sample performance) and a training sample (80% for training, pruning, assignment, and estimating gains to show users). We train three kinds of personalized models for each dataset:
Static: These models are personalized using a one-hot encoding of group attributes (1Hot), and a one-hot encoding of intersectional groups (mHot)
Imputed: These are variants of static models where we impute the group membership for each person (KNN-1Hot, KNN-mHot). In practice, personalized systems with imputation will range between the performance for these systems and the performance of Hot and mHot.
Participatory: These are participatory systems built using our approach. These include Minimal, a minimal system built from Hot and its generic counterpart; and Flat and Seq, flat and sequential systems built from Hot, mHot and their generic counterparts.
We train all models - personalized models and the components of participatory systems - from the same model class and evaluate them using the metrics in Table 1. We repeat the experiments four times, varying the model class (logistic regression, random forests) and the target performance metric (error rate for decision-making tasks, AUC for ranking tasks) to evaluate the sensitivity of our findings with respect to model classes and use cases.
"The use of race in clinical risk scores should be approached with caution [60]; participatory systems offer way to safeguard against inappropriate use.
7

---

Metric
Definition
Description
Overall
Σ " (1o)
Population-lever performance of personalized system/modee, com- puted as weighted average over all groups
Performance
Overall Gain
Σ DEG
2 A ②))
Population-Ievel gain in performance of personalized system/model over its generic counterpart
Group Gains
min
max Ag .) REG
Range of group-level gains of personalized system/model over generic counterpart across all groups
Rationality
Preject Ho]
Number of rationality violations detected using a bootstrap test with
Violations
8EG
100 resamples at significance of 10% where Ho Ag (g. Ø) 2 0.
Imputation
min (x 000
Worst-case loss in performance due to incorrect imputation. This metric can only be computed for static models
Risk
Options
|R| (8
Proportion of reporting options pruned from system/model. Here, R denotes all reporting options and R(hh denotes those after h is pruned
Pruned
Data Use
inqreeat 0] (m(()
Proportion of group attributes requested by l from each group, aver- aged over all groups in G
Table 1: Metrics used to evaluate performance, data use, and consent of personalized models and systems. We report performance models, do not report group membership to imputed models, and only report to participatory systems when informed that it would lead to strictly positive gain, as computed on the validation set in the training sample.
held-out sample. We assume that individuals report group membership static
4.2 Discussion
We show results for logistic regression models and error rate in Table 2 and results for other model classes and classification tasks in Appendix C. In what follows, we discuss these results.
On Performance Our results in Table 2 show that participatory systems can improve performance across reporting groups Here, Flat and Seq achieve the best overall performance on 6/6 datasets and improve the gains from personalization for every reporting group on 5/6 datasets. In contrast, traditional models improve overall performance while reducing performance at a group level (see rationality violations on five datasets for 1Hot, mHot). The performance benefits from participatory systems stem from (i) allowing users opt out of these instances of "worsenalization" and (ii) assigning personalized predictions with multiple models. Using Table 2, we can measure the impact of (i) by comparing the performance of Minimal vs. HHot, and the impact of (ii) by comparing the performance of Minimal to Flat (or Seq). For example, on apnea, Hot exhibits a significant rationality violation for group 30__t__00,,, male], meaning they would have been better off with generic model. By comparing the performance of 1 Hot to Minimal, we see that allowing users to opt out of worsenalization reduces test error from 29.1% % 28.9%. By comparing the performance on Minimal to Flat and Seq, we see that using multiple models can further reduce test eтror from 28.9% 24.1%.
On Informed Consent Our results show how Flat and Seq systems can inform consent by allowing users report subset of group attributes (e.g., by including reporting options such as [30+, Ø] or [0, HIV+]) Although both Flat and Seq systems allow for partial personalization, their capacity to inform consent differs. In a flat system, users may inaccurately gauge the marginal benefit of reporting an attribute by comparing the gains between reporting options. For example, in Fig. 4, users who are HIV positive would see a gain of 3.7% for reporting [Ø, HIV+], and 16.7% for reporting [30+, H HIV+] and may mistakenly conclude that the gain of reporting age is 16.7% 3.7% = 13.0%. This estimate incorrectly presumes that the gains of 3. 3.7% were distributed equally across age groups. Sequential systems directly inform users of the gains for partial reporting. In the sequential system, group [30+, HIV+] is informed that they would see a marginal gain of 21.5% for reporting age, while group <30,, HIV+] is informed they would see marginal gain of reporting age of 0.0%.
On Data Minimization Our results show that participatory systems perform better across all groups while requesting less personal data on 6/6 datasets. For example, on cardi o_ei cu, Seq reduces error by 11.3% compared Hot while requesting, on average, 83.3% of the data needed by Hot. In general, participatory systems can limit data use where personalization does not improve
8

---

Stwmi
IMPUTED
PARTICIPATORY
Dusaet
Metrics
1Hot
mHot
KNN-IHot
KNN-mHot
Minimal
Flat
Seq
Overall
29.14
29,098
27.99
28.95
24.1%
24.3%
apreca
0.11
0.21
1.3%
0.2%
4.9%
d
Group Grins
-1.1% -
40.8% 0.45)
1.25%
-0.8% 0.458
1.255
0.0%
-0.4%
(age, sex)
Munsetaliaation
[5] 6 groups
Imsputation Risk
-..9%
-5.2%
Ustun
Optioss Pruned
OM6
0/12
4/7
5/12
612
100,016
0.016
Overall Performance
21
21.316
21.016
22.11
21.015
10.2%
10.3%
cardio ei
Ovenil Gain
0144%
0.31
0..1%
40.258
0.3%
11.7%
11.7%
1511, = 69
Group Goins
2.65
-2.7%
3.08
-1.9%
2.69
-2.7%
3,0%
0.09 - 2.6%
3.1%
30.9%
3.15
20.99
(age, awx, sace) 35) proupe
Mumerullicaiion
Imsutiaioon
-.4%
Polland et [44]
Optons Pruned
088
03
0/27
0/27
69
10/27
9/27
Data Use
100.1%
100,0%6
0,.16
0.096
25.056
100,045
63.99
Overal Performance
19.41
19.3%
19.21
15.7%
15.7%
cardio mimic
40.06
-0.315
0.11
5089, d
Group Goins
40.%%
40. 9%
0.58%
-0.9%
0.45%
-0.95
0.59%
0.08
0.45
-1.672
9.85
-1.6 -9.8
m (age, aws, a slow)
Winerralisaionn
[3] 8 groups
Iesuutation Risk
-1.1%
-1.1%
Johnon al. (50)
Options Prinzd
08
6%
(U27
0/22
6/9
6/17
8/23
Use
100.0%
100.0%
100.5%
91.3
Overall Performazce
37.016
36.716
37.0%
36.9%
37.015
36.6%
36.1%
colencancer
Overail Gain
0.115
0,416
0.116
0.216
0.115
0.5%
1.0%
of = tha
Group Goins
40.4%
0.35
40, 19 1.18
-0.4%
0.35%
-0.1 19195-11
1.39
0.092
0.39
0.0%
1.7%
0.758
1.7%
(age, sex)
Wussenalization
[6] groups
Imputtaioon
Sooeyee et [45]
Optinns Prined
066
1912
0/12
5/2
211
5112
Daua Use
10005%
10000
0.00%
0.09
16.7%
50.05
75.0%
Overall Performance
19.6%
19,85
19.5%
18.9%
1899
lungcar.cer
0.21
0.216
0006
0.6%
1 84
Group Guins
0.4%
00.3%
0.25
-0.4%
0.2%
-0.3%
0.2%%
0.0%
0.0%
0.9%
0.38
0.95
[age, sex)
Wossenallaation
0
[6]
6 groups
Imputation Risk
01.5%
0.516
Soogyrer (45)
Optinns Prined
016
1/12
6/7
V12
0.0%%
0.0%
83.9%
35
Overll Performance
20.4/
20.2%
30.4%
20 49.4%
20.45
11.1%
11.19
seps
Ovenil Gain
1.5%
1.0%
1.3%
-7.7%
1.3%
10.6%
10.5%
7797, m 36
Group Grins
0.0%
3.6%
0.0%
27%
0.0% - 3.6%
0.0% 2.76
0.0% 3.6%
4.3% - 17.2%
4.3%
17.2%
(HIV. age)
Mussenalizaiion
[6] proups
Imputaaionn
0.0%%
Allyn et [3]
Optons Prined
09
1/5
18
35
Daua Use
100.0%
10000%
0.8
0.0%
0%
100.0%
25.08
Table 2: Participatory systems and personalized models for all datasets. We summarize metrics in Table I and present results for other model classes and prediction tasks in Appendix C. The best performance across each system is highlighted in green with bold text, and instances of worsenalization are highlighted in red.
performance, e.g., on Lungeancer. Even as attributes like sex or age may be readily reported by patients for any performance benefit, limiting data use is valuable when there is a tangible cost associated with data collection e.g., when models make use of rating scale for mental disorder that must be administered by a clinician [47]. The potential for data minimization varies substantially across prediction tasks. On apnea, for example, we can prune six reporting options when building a Seq for decision making (which optimizes error) but four options for Seq for ranking (which optimizes AUC; see Appendix C. Overall, participatory systems satisfy "global data minimization" as proposed in [13], in that they minimize the amount of per-user data requested while achieving the quality of system with access the full data on average.
On the Benefits of a Model-Agnostic Approach Our findings highlight some of the benefits of a model-agnostic approach, in which we can draw on a rich set of models to achieve better performance while mitigating harm. The resulting system can balance training costs with performance benefits. We can also ensure generalization across reporting groups e.g., by including generic model fit from a complex model class and personalized models fit from a simpler model class. As expected, fitting for a complex model class can lead to considerable changes in overall accuracy eeg.. we can reduce overall test error for a personalized model from 20.4% to 14.1% 1% on saps by fitting a random forest rather than a logistic regression model (see Appendix C). However, a gain in overall performance does not always translate to gains at the group level. On saps, for example, using a random forest also introduces a rationality violation for one group.
On the Pitfalls of Imputation One of the simplest approaches to allow individuals to opt out of personalization to pair a personalized model with an imputation technique. Although this approach can facilitate consent, it may violate the requirements in 2. Consider a personalized

---

1Hot PALC CwealCCacc.c
Minimal
Flat
Sequential
Chesal Ganc 1ae
the
Figure 4: Participatory systems for the saps dataset. These models predict ICU mortality for groups defined by G HIV age [+,- [<30, 30+] using logistic regression component models. Here, ho is generic model, hu is 1Hot model with a one-hot encoding of G. and h2 ... hm are 1 Hot and mHot models fit for each reporting group. We show the gains of each reporting option above each box and highlight pruned options in grey. For example, in Seq. the group (EIV+, 30+) sees an estimated 21.5% error reduction after reporting HIV if they report age. In contrast. the group (BIV+, <30) no gain from reporting age in addition status, so this option is pruned.
model that exhibits "worsenalization" in Fig. 1. Even if one could correctly impute the group membership for every person, individuals may receive more accurate predictions from a generic model hg. In practice, imputation is imperfect as individuals who opt out of reporting their group membership to a personalized model may be assigned "worse" predictions because they are imputed the group membership of a different group. In such cases, opting out may be beneficial, making it difficult for model developers to promote participation while informing consent. Our results highlight the prevalence of these effects in practice. For example, cardio. eicu the estimated "risk of imputation" " is -4.6% indicating that every intersectional group can experience an increase of 4.6% in the error rate result of incorrect imputation. The results for KNN-1Hot show that this potential harm can be realized in practice using -iN-imputttion, as we find that the imputed system leads to rationality violations 5/6 datasets.
5 Concluding Remarks
We introduced a new family of classification models that allow individuals to report personal data prediction time. Our work focuses on personalization with group attributes; our approach could be used to facilitate and inform consent in a broader class of prediction tasks. In such cases, the key requirement for building participatory system is that we can reliably estimate the gains of personalization for each person who reports personal data.
Our results show that participatory systems can inform consent while improving performance and reducing data use across groups. Reaping these benefits in practice will hinge on the ability to effectively inform decision subjects on the impact of their reporting decisions. [4]. Even there may be good "default practices" for what kind of information we should show decision subjects, practitioners should tailor this information to the application and target audience [23].
One common concern in using a participatory system arises when practitioners wish to collect data from model in deployment to improve its performance in the future. In practice, participatory system can thwart data collection in such settings by allowing individuals to opt out. In such cases, we would note that this issue should be resolved in a way that is aligned with the principle of purpose specification [40]. If the goal of data collection is to improve a model, then individuals could always be asked to report information voluntarily for this purpose. If the goal of data collection is to personalize predictions, then individuals should be able to opt out, especially when it may lead to worse performance.
10

---

Acknowledgements
We thank the following individuals for helpful discussions: Taylor Joren, Sanmi Koyejo, Charlie Marx, Julian McAuley, and Nisarg Shah. This work was supported by funding from the National Science Foundation IIS 2040880, the NIH Bridge2AI Center Grant U54HG012510, and an Amazon Research Award.
References
[1] Abajian, Aaron, Nikitha Murali, Lynn Jeanette Savie, Fabian Max Laage-Gaupp Nariman Nezami. James Duncan, Todd Schlachter, MingDe Lin, Jean-Francois Geschwind, and Julius Chapiro. Predicting treatment response to intra-arterial therapies for hepatocellulaar carcinoma with the use of supervised machine learning-at artificial intelligence concept. Journal of Vascular and Interventional Radiology, 29 (6):850-857, 2018.
[2] Agresti, Alan. An introduction to categorical datu analysis. John Wiley & Sons, 2018
[3] Allyn, Jérôme, Cyril Ferdynus, Michel Bohrer, Cécile Dalban, Dorothée Valance, and Nicolas Allou. Simplified acute physiology score ii as predictor of mortality in intensive care units: a decision curve analysis. PloS one, (110)==0164828 2016.
[4] Anderson, Catherine L and Ritu Agarwal. The digitization of healthcare: boundary risks, emotion, and consumer willingness to disclose personal health information. Information Systems Research, 22(3): 469-490, 2011.
[5] Arellano, April Moreno, Wenrui Dai, Shuang Wang, Xiaoqian Jiang, and Lucila Ohno-Machado. Privacy policy and technology in biomedical data science. Annual review of biomedical data science, 1:115, 2018.
[6] Atan, Onur, William Whoiles, and Mihacla Schaar. Data-driven online decision making with costly information acquisition. Arxiv, 02 2016.
[7] Auer, Peter, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008.
Auxicr, Brooke, Lee Rainie, Monica Anderson, Andrew Perrin, Madhu Kumar, and Erica Turner. Ameri- cans and privacy: Concerned, confused and feeling lack control over their personal information. Pew Research Center: Internet, Science and Tech, 2019.
Awad, Naveen Farag and Mayuram S Krishnan. The personalization privacy paradox: an empirical evaluation of information transparency and the willingness to be profiled online for personalization. MIS quarterly, pages 13-28, 2006.
[10] Bansal, Gaurav, David Gefen, et al. The impact of personal dispositions on information sensitivity, privacy concern and trust in disclosing health information online. Decision support systems, 19(2):138-150, 2010.
[11] Bertsimas, Dimitris and Nathan Kallus. From predictive prescriptive analytics. Management Science, 66(3):1025- 2020.
[12] Bertsimas, Dimitris, Jack Dunn, and Nishanth Mundru. Optimal prescriptive trees. INFORMS Journal on Optimization, 1(2):164- 183, 2019.
[13] Biega, Asia J, Peter Potash, Hal Daumé, Fernando Diaz, and Michèle Finck. Operationalizing the legal principle of data minimization for personalization. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 399-408, 2020.
[14] Bien, Jacob, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. Annals of statistics, 11111 2013.
[15] Biggs, Max, Wei Sun, and Markus Eul. Model distillation for revenue optimization: Interpretable personalized pricing. arXiv preprint arXiv:2007. 00,900,,,,,,,,, 2020.
[16] Bukaty, P. The California Consumer Privacy Act (CCPA): An implementation guide. IT Gover- nance Publishing, 2019. ISBN 9781787781337. URL https::/bbott google com com/books?
id=vGNFIWWAACHA
[17] Campbell, Tim S and William A Kracaw. Information production, market signalling, and the theory of financial intermediation. the Journal of Finance. 35(4))863-882 1980.
[18] Chemmanur, Thomas J. The pricing of initial public offerings: A dynamic model with information production. The Journal of Finance, 48(1):285-304 1993.
[19] DeLong, Elizabeth R. David M DeLong, and Daniel Clarke-Pearson Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, pages 845, 1988.

---

[20] DiCiccio, Thomas J and Bradley Efron. Bootstrap confidence intervals. Statistical science, pages 189-212, 1996.
[21] Dietterich, Thomas G. Approximate statistical tests for comparing supervised classification learning
algorithms. Neural computation, 10(7):1895-19) 1998.
[22] Do. Virginie, Sam Corbett-Davies Jamal Atf, and Nicolas Usunier. Online certification of preference- based fairness for personalized recommender systems. arXiv preprint arXiv:2 14527, 2021.
[23] Edwards, Adrian GK, Gurudutt Naik, Harry Ahmed, Glyn J Elwyn. Timothy Pickles, Kerry Hood, and Rebecca Playle. Personalised risk communication for informed decision making about taking screening tests. Cochrane database of systematic reviews, Cochrane database of systematic reviews(2), 2013.
[24] Eliaz, Kfir and Ran Spiegler. On incentive-compatible estimators. Games and Economic Behavior, 132:
204-220, 2022.
[25] Elmachtoub, Adam N. Vishal Gupta, and Michael Hamilton. The value of personalized pricing. Available at SSRN 3127719, 2018.
[26] European Parliament and of the Council. Regulation 2016/679 of the european parliament and of the council of 27 april 2016 the protection natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/cc (general data protection regulation), 2016. URL https://./. //eur lex. europa. su/eli htapll/reeeellee 6/699/05 Official Journal of the European Union
[27] Fan, Haiyan and Marshall Scott Poole. What is personalization? perspectives on the design and implemen- tation of personalization in information systems. Journal of Organizational Computing and Electronic Commerce, (0(3-4)):79-2000 2006.
[28] Globus-Harris, Ira, Michael Kearns, and Aaron Roth. An algorithmic framework for bias bounties. 2022 ACM Conference on Fairness, Accountability, and Transparency, Jun 2022. doi: 10.5145/3531 146.353317..
URL http: / dx. doi. org/ 10.10000.00 1445/555553 3533533111111 146 3533172 72.
[29] Haukoos, Jason S, Michael S Lyons, Christopher Lindsell, Emily Hopkins, Brooke Bender, Richard E Rothman, Yu-Hsiar fu-Hsiang Hsich. Lynsay A MacLaren, Mark W Thrun, Comilla Sasson, et al. Derivation and validation of the denver human immunodeficiency virus (hiv) risk score for targeted hiv screening. American journal of epidemiology. 175(8):838 846, 2012.
[30] Hébert-Johnson Úrsula, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the coomputationally-deefffaaa masses. In Proceedings of the International Conference on Machine Learning, pages 1944-1953, 2018
[31] Hollenberg, SM. Cardiogenic shock. In Intensive Care Medicine, pages 447-458. Springer, 2003.
[32] Johnson, Alistair EW, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Wei, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care datahase. Scientific data, 3(1):199, 2016.
[33] Kaminski, Margot E. The right to explanation, explained. Berkeley Tech. LJ, 34:189, 2019.
[34] Kearns, Michael, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairnsss gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning, pages 2564-2572, 2018.
[35] Kessler, Ronald C, Lenard Adler, Minnie Ames, Olga Demler, Steve Faraone, EVA Hiripi, Mary J Howes, Robert Jin, Kristina Secnik, Thomas Spencer, et al. The world health organization adult adhd self-report scale (asrs): a short screening scale for use in the general population. Psychological medicine, 35(2):
246-256. 2005.
[36] Kim, Michael P. Aleksandra Korolova, Guy Rothblum, and Gal Yona. Preference-infooreed fairness. arXiv preprint 1r0ivv1904.01/ 2019.
[37] Le Gall, Jcan-Roger, Stanley Lemeshow, and Fabienne Saulnier. new simplified acute physiology score (saps ii) based on a curopean/north american multicenter study. Jama, 270(24):2957-225 2963, 1993.
[38] Lim, Michael and Trevor Hastie. Learning interactions via hierarchical group-lasso regularization. Journal of Computational and Graphical Statistics. 24(3):627-655 6544, 2015.
[39] Lundberg, lan, Arvind Narayanan, Karen Levy, and Matthew J Salganik. Privacy, ethics, and data access: A case study of the fragile families challenge. Socius, 5:237802311883
2019.
[40] OECD. Recommendation of the council concerning guidelines governing the protection of privacy and transborder flows of personal data, 2013, URL httpp::::/lglllssssttttttao
occd, org/ /n/
instruments/ CECD-LEGAL 0188.
[41] Ortlieb, Martin and Ryan Garner. Sensitivity of personal data items in different online contexts. it- Information Technology, 58(5):117223 2016.
12

---

[42] Paes, Lucas Monteiro, Carol Xuan Long, Berk Ustun, and Flavio Calmon. On the epistemic limits of personalized prediction. In Oh. Alice H. Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: /ttttps://ppereviee net / forum ? Sndddn 7NJ.
[43] Paulus, Jessica K and David M Kent. Predictably unequal:
understanding and addressing concerns that algorithmic clinical prediction may increase health disparities. NPJ digital medicine, 3(1):188, 2020.
[44] Pollard, Tom J, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The cicu collaborative research database, a freely available multi-center database for critical care research. Scientific data, 5(1):1-13, 2018.
[45] Scosyrev, Emil, James Messing, Katia Noyes, Peter Veazie, and Edward Messing, Surveillance epidemiol- ogy and end results (seer) program and population-based research in urologic oncology: overview. In Urologic Oncology: Seminars and Original Investigations, volume 30, pages 126-132. Elsevier, 2012.
[46] Shanmugam, Divya, Fernando Diaz, Samira Shahanian, Michèle Finck, and Asia Biega. Learning to limit data collection via scaling laws: A computational interpretation for the legal principle of data minimization. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency. pages 339-849, 2022.
[47] Sharp, Rachel. The hamilton rating scale for depression. Occupational Medicine, 65(4):340-33) 0404400-3,,,00 2015.
[48] Steyerberg, Ewout et Clinical prediction models. Springer. 2019.
[49] Struck, Aaron F, Berk Ustun, Andres Rodriguez Ruiz, Jong Woo Lee, Suzette M LaRoche, Lawrence J Hirsch, Emily J Gilmore, Jan Vlachy, Hiba Arif Haider, and Cynthia Rudin. Association of an elecrrenseephalogrphyy
risk score with seizure probability in hospitalized patients. JAMA neurology, 74(12):1419-1422 2017.
[50] Sun, Xu and Weichao Xu. Fast implementation of delong's ' algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Processing Letters, 21(11):139-1139, 2014. doi: 10.1 11..//....00........
[51] Suriyakumar, Vinith M, Marzyeh Ghassemi, and Berk Ustun, When personalization harms: Reconsidering the use of group attributes in prediction. In International Conference on Machine Learning. 2023.
[52] Tran, Cuong and Ferdinando Fioretto. Personalized privacy auditing and optimization at test time, arXiv preprint 2302 00077, 2023.
[53] Tran, Cuong. My Dinh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the fairness lens. Advances in Neural Information Processing Systems, 14:27555-27566 2021.
[54] U.S. Congress. Health insurance portability and accountability act of 1996, 1996. URL https: /www. hhs , gov/hipas/ for-//orrrrroeeee onals / pttp/cy/icay//ppp// index.idxxx///// html. Public Law 104-191.
[55] Ustun, Berk, M Brandon Westover, Cynthia Rudin, and Matt T Bianchi. Clinical prediction models for sleep apnea: the importance of medical history symptoms. Journal of Clinical Sleep Medicine, 12 (02):161-168, 2016.
[56] Ustun, Berk, Lenard A Adler, Cynthia Rudin, Stephen V Faraone, Thomas Spencer, Patricia Berglund, Michael J Gruber, and Ronald C Kessler. The world hcalth organization adult seeentioondeffcciiyppraattiii disorder self-report sereening scale for dsm-5. Jama psychiatry, 74(5):520-526 2017.
[57] Ustun, Berk, Yang Liu, and David Parkes. Fairesss without harm: Decoupled classifiers with preference guarantees. In International Conference on Machine Learning, pages 6373-6382, 2019.
[58] Vaughan, Gregory, Robert Aseltine, Kun Chen, and Jun Yan. Efficient interaction selection for clustered data via stagewise generalized estimating equations. Statistics in Medicine, 39(22);285-2286, 2020.
[59] Viviano, Davide and Jelena Bradic. Fair policy targeting. arXiv preprint arXiv:2005. 2020.
[60] Vyas, Darshali A, Leo G Eisenstein, and David S Jones. Hidden in plain sight-reconssderring the use of race correction in clinical algorithms, 2020.
[61] Yu, Shipeng, Balaji Krishnapuram, Rómer Rosales, and R. Bharat Rao. Active sensing. In AISTATS, 2009.
[62] Zafar, Muhammad Bilal, Isabel Valera, Manuel Rodriguez, Krishna Gummadi, and Adrian Weller. From parity to preference-based notions of fairness in classification. In Advances in Neural Information Processing Systems, pages 228-238, 2017.
13

---

Supplementary Material
A Supporting Material for Section 3
15
A.1 Enumeration Routine for Algorithm 1
15
A.2 Assignment Routine for Algorithm
15
A.3 Pruning Routine for Algorithm
16
A.4 Greedy Induction Sequential Reporting Interface
16
B Description of Datasets used in Section Experiments
17
C Results for Different Model Classes and Prediction Tasks
18
C.I Logistic Regression for Ranking (AUC)...
18
C.2 Random Forests for Decision-Making (Error)
19
C.3 Random Forests for Ranking (AUC)
20
D Supporting Material for Performance Profiles
21

---

A Supporting Material for Section 3
A.1 Enumeration Routine for Algorithm 1
We summarize the Enumeration routine in Algorithm 2. Algorithm 2 takes input set of group attributes G and dataset D and outputs collection of reporting interfaces T that obey ordering and plausibility constraints.
Algorithm 2 Enumerate All Possible Reporting Trees for Reporting Options G
1: procedure VIABLE TREESGG, D)
2: if dim(g) return [7c]
base
left with single attribute which branch
3: T T-<-l
4:
for each group attribute E [91.... : G&] do
5:
TA < reporting tree of depth with |A] leaves
6:
S t ViableTreess( \ A,))
all subtrees using all attributes except each assigument is a permutation [A] to leaves extends the tree by assigning subtrees each leaf
7:
for 11 in Validdssiigmenntss... 2)) do: asssigg)))))
8:
T
end 10: end for
9:
11: return T, reporting interfaces for group attributes that obey plausibility and ordering constraints
12:
end procedure
The routine enumerates all possible reporting interfaces for given of group attributes through recursive branching process. Given set group attributes, the routine called for each attribute that has yet be considered in the tree Line 4, ensuring a complete enumeration. We note that the routine is only called for building Sequential systems since there is only one possible reporting interface for Minimal and Flat systems.
Enumerating all possible trees ensures we can recover the best tree given the selection criteria and allows practitioners choose between models hased on other criteria. We generate trees that meet plausibility constraints based the dataset, such having least negative and positive sample and least total samples at each leaf. In settings constrained by computational resources, we can impose additional stopping criteria and modify the ordering to enumerate more plausible trees first or exclusively (e.g.. by changing the ordering of G or imposing constraints in VALIDASSIGNMENTS)
A.2 Assignment Routine for Algorithm
We summarize the routine for AssignModels procedure in Algorithm 3.
Algorithm 3 Assigning Models
1: procedure ASSIGNMODELS(T,M,
2: Q < [T.root]
initiality with the mot of the tree, reporting group Ø
3: while Q is not empty do
4:
Q.pop()
5:
M, <- ViableModeels(MMr))
filter models assign the model with the best training performance iserate shrough the children reporting groups
axzigned
6:
h" +- argmin Rr (h, D) bEM,
7:
T.set_modell model(r,
8: for r' E ww..ee_ssugguteee "_Tee__subrrous((r) do Q.enqueue(r")
9:
10: end for
end while
return 7 that maximizes gain for each reporting group
13:
end procedure
Algorithm takes as inputs a reporting tree T, pool candidate models M, and an assignment (training) dataset D and outputs tree T that maximizes the gains of reporting group information. The pool of candidate models is filtered to viable models for each reporting group. Since the pool of candidate models includes the generic model assign each reporting group the best pest-performing better-performing
ho, each reporting group will have at least one viable model. model on the training set and default to the generic model fop when a better performing personalized model is not found. We assign performance on the training set and then prune using performance on the validation set to avoid biased gain estimations.
15

---

A.3 Pruning Routine for Algorithm 1
We summarize the routine used for the PruneLeaves procedure in Algorithm 1. The PruneLeaves routine
Algorithm Pruning Participatory Systems
1:
procedure PRUNELEAVES(T, D)
2: Stack < [T.leaves]
initialize stack with all leaves
3: repeat
4:
Stack.poyy pppp()
5: + gege model(r)
6: h' < T lege_modde(paall
7:
not Test(r, D) then
test gains parent model good leaf model
8:
T.prune(r)
9:
end
10:
Tegget children(pa(r)) is empty then Stack.enqueueeppll end if
consider praming the parent if the parent has become leaf
11:
12:
until Stack is emply
14:
return T' reporting interface that ensures data collection Icads to gain
15:
end procedure
Algorithm takes as input a reporting interface T and validation sample D, and performs bottom-up up pruning output a reporting interface that asks individuals to report attributes that are expected to a gain. The pruning decision at each leaf is based on hypothesis test that evaluates the gains of reporting for a reporting group validation dataset. This test has the form:
Ho Rg (h) I (h')
:A: R.) (,))))
This procedure evaluates the gains of reporting by comparing the performance of model assigned at leaf node a model assigned a parent node h which does not use reported information. Here, the null hypothesis Ho assumes that the parent model performs as well as the leaf model and thus, we reject the null hypothesis when there is sufficient evidence to suggest that reporting will improve performance in deployment. Our routine allows practitioners specify the hypothesis test compute the gains. By default, use the McNemar test for accuracy [21] and the Delong test for AUC [19, 50J. general, we can use a bootstrap hypothesis test [20].
A.4 Greedy Induction of Sequential Reporting Interface
We present additional routine construct reporting interfaces for sequential systems in Algorithm 5. We include this routine as alternative option that can he used to construct a reporting interface in settings where may be impractical or undesirable to enumerate all possible reporting interfaces. The procedure results in valid reporting interface that ensures gains. However, it does not guarantee an optimal tree in terms of maximizing the overall gain and does not allow practitioners choose between reporting interfaces after training.
Algorithm 5 Greedy Induction Routine for Sequential Reporting Interfaces
1: procedure GREEDY GREEDYTREE(R) TREE(R)
2:
empty <mppyy reporting interface 3: repeat
T
4: for r leaves (T) do
5:
(A.) +- :
(Art contains all heretofore unused attributes
6:
A' ArgmaxAE(A+) min (p((r(m) rssplit(A"))
7:
Split on attrlbute that masimizes worse-cate goin
8:
end for until splits added
9:
return T. reporting interface that ensures gains for reporting each R.
11: end procedure
Algorithm 5 takes input collection of reporting options R and outputs single reporting interface using greedy tree induction routine that chooses the attribute to report maximize the minimum gain each step. The procedure uses the reporting options to itcratively construct reporting tree that branches on all of the attributes in R. The procedure considers each unused attribute for each splitting point and splits on the attribute that provides the greatest minimum gain for the groups contained at that node.
16

---

B Description of Datasets used in Section 4 Experiments
We include additional information about the datasets used in Section 4.
Dataset
Reference
Outcome Variable
71
d
m7
G
apnea
Ustun et al. [55]
patient has obstructive sleep apnea
1.152
28
lage, aex)
cardio_eicu
Pollard et al. [44]
patient with cardiogenic shock dies
1,341
49
fage, sex, race]
cardio_mimic
Johnson al. [32]
patient with cardiogenic shock dies
5,289
49
{age, roce]
coloncancer
Scusyrev al. [4S]
patient dies within years
29,211
72
6
[age, aex]
lungconcer Scosyrev al. [45] patient dies within years
120,641
84
fago, sex]
saps
Allyn et [3]
ICU mortality
7,797
36
lage,
Table 3: Datasets used to fit clinical prediction models in Section 4. Here: n denotes the number of examples in
each dataset; d denotes the number of features; 9 denotes the group attributes that are used for personalization; and m = |9| denotes the number of intersectional groups. Each dataset is de-identified and available to the public. The pardio_eit cu, pardio_miiii c, lungcancer datascts require access to public repositories listed under the references. The saps and apnea datascts must be requested from the authors. The support dataset can be downloaded directly from the URL below.
apnea We use the obstructive sleep apnea (OSA) dataset outlined Ustun et al. [55]. This datasct includes a cohort of 1,152 patients where 23% have OSA. We use all available features (e.g. BMI, comorbidities, age, and sex) and binarize them, resulting in 26 binary features.
cardio eicu & cardio mimic Cardiogenic shock is an acute condition in which the heart cannot provide sufficient blood to the vital organs [31]. These datasets are designed to predict cardiogenic shock for patients in intensive care. Each dataset contains the same features, group attributes, and outcome variables for patients in different cohorts. The cardic o q_eicu datasct contains records for cohort of patients in the Collaborative Research Database V2.0 [44]. The cardio aidio__eicy eicu dataset contains records for a cohort of patients in the MIMIC-III [32] database. Here, the outcome variable indicates whether patient in the ICU with cardiogenic shock will die while in the ICU. The features encode the results of vital signs and routine lab tests (e.g. systolic BP, heart rate, hemoglobin count) that were collected up to 24 hours before the onset cardiogenic shock.
lungcancer We consider a cohort of 120,641 patients who were diagnosed with lung cancer between 2004-2016 and monitored as part of the National Cancer Institute SEER study [45]. Here, the outcome variable indicates if patient dies within five years from any cause, and 16.9% of patients died within the first five years from diagnosis. The cohort includes patients from Greater California, Georgia, Kentucky, New Jersey, and Louisiana, and does not cover patients who were lost follow-up (censored). Age and Sex were considered as group attributes. The features reflect the morphology and histology of the tumor (e.g., size, metastasis, stage, node count and location, number and location of notes) as well as interventions that were administered at the time of diagnosis (e.g., surgery, chemo, radiology).
coloncancer We consider cohort of 120,641 patients who were diagnosed with colorectal cancer between 2004-2016 and monitored as part of the National Cancer Institute SEER study [45]. Here, the outcome variable indicates patient dies within five years from any cause, and 42. 1% patients die within the first five years from diagnosis. The cohort includes patients from Greater California. Age and Sex were considered as group attributes. The features reflect the morphology and histology of the tumor (e.g.. size, metastasis. stage, node count and location, number and location of notes) well interventions that were administered at the time of diagnosis (c.g., surgery, chemo, radiology).
The Simplified Acute Physiology Score II (SAPS II) score predicts the risk of mortality of critically-ill patients in intensive care [37]. The data contains records of 7,797 patients from 137 medical centers 12 countries. Here, the outcome variable indicates whether patient dies in the ICU, with 12.8% patient of patients dying. The features reflect comorbidities, vital signs, and lab measurements.
saps
17

---

C Results for Different Model Classes and Prediction Tasks
In this Appendix, we present experimental results for additional model classes and prediction tasks. We produce these results using the setup in Section 4. and summarize them in the same way Table We refer to them in our discussion in Section 4.2.
C.1 Logistic Regression for Ranking (AUC)
PARTICPAMOAA
Uorzzet
Methic
1Hot
mHot
KNN-1Hot
KNN-mHot
Minimal
Fiat
Seq
Overal Performanas
0.774
0.774
0.776
0.776
0.776
0.851
0.851
apriea
1162,
Group
Ovenal Gain
-0.000
0.003
0.008
0.115
0.074
0,074
0.115
G (age, sex)
Max Dispanty
0.004
0.015
0.004
0.00
0.002
0.111
0.111
Uston etall (15)
99] grougs
Imputaiios Risk
Uptios Pruned
Kat. Vislations
400022
-0.000
4112
10000
Overil Performance
0,864
0.863
0.863
0.362
0.365
0.966
0,966
cardio_ezan
Overall Gain
0.002
0.001
0,000
10.001
0.002
0.105
0,183
Group
0,010
0,000
0,010
0.100
Polladd al. (44)
= groups
(age, racer
Imputaiios Risk
Uptons Prined
Velations
Diganiy
-0,05
0/S
-0.000
1827
0227
6/9
0.170
1327
0.170
11/27
Data Use
100.0%
100.090
0.0%
0.0%
25.00
100.00
9588
Performance
0,882
0,850
0.914
nardionninic 5289, 49
Overall Gain
Grosp Cants
-0000
0.000
0.001
00001
00000
0:001
-0.001
0.001
00002
00,001
0.000
0100
0,000
00000
0,000
0.008 - 0005
0.034
0000
0- (ago, race)
Max
Vislaions
0.002
0000
0.002
0.00
0.001
0.040
0.040
Johroon ei al. (30)
grogs
Imputatios
Options Prinod
-0.01
1M27
D227
8017
Dats Use
100.0%
100.0%
0.09
25.
100.0%
79
Overall Performance
0.700
0,700
coloncancer
x 22212,24444 d
d (age, sex)
Rai. Vislatiun
Max Diquarity
Grop Cains
-0.001
0.002
0.003
00000
0000
0001
-0.001
0002
00000
0000
-0.01
0.000
0000
0000
0.000
0.001
0.001
-0001
9/006
1000
0000
0002
0000
Sanytor
= grops
Imputation Risk
Options
Data
100.0%
100.0%
0.00
0.09
16.7%
100.0%
75.05
Overal Performance
0,555
0.855
0.52
0.854
0,355
0,961
0.961
G ==<gnn
[0) = groug
lungcenzer
120041,
Rai Visuaios
Mac Disparity
Overal Gain
Greap
0.001
0.001
-0003
0000
0.001
0.000
0,006
0001
0.006
0.011
Sanyee otail [45]
Imputation Risk
Optiots Paunal
0.000
015
00000
006
1012
6/12
4/7
2212
2112
100.0%
33.3%
100.0%
Uvenll Performance
0,875
0.877
0,875
0.357
0,875
0.960
0.900
0= age)
saps
7707,
Max Dispaity
Overal Gain
Goup Cuins
-0,000
00015
0.010
-0.002
00111
-0.000
0.000
0.010
-0.002
-01000
0.019
0.021
0.000
-0006
0.009
0.035
00.44
0,095
0.35 01
0L085
0.106
Allys at 13)
[0) group
Imputation Risk
Optina Prinal
0,000
044
00002
044
0/9
019
29
5/
Data Use
100.0%
100.0%
0.0%
0.08
75.0%
102.0%
87.5%
Table 4: Overview of performance, data use, and consent for all personalized models and systems on all datasets measured by test auc. We show the performance of models and systems built using logistic regression.
18

---

C.2
Random Forests for Decision-Making (Error)
IMPUTED
Dunaset
Metics
1Hot
mHot
KNN-1Hot
KNN-mHot
Minimal
Flat
Seq
Overall Perfocmance
26.3%
26,0%
25.9%
27.4%
26.315
12.2%
12.2%
1,5%
1.5%
s = 1152, ol m 26
agnea
(age,sex)
Rar. Violations
Max Disqurity
Groep Gnins
408%
-4.2%
5.05
0.4%
1.8%
3.4%
-0.85
-4.2%
5.09
0.4%
3.45
3.8%
0.0%
4.218
4.70%
5.5%
22.254
16.0%
5.3%
23.2%
16.9%
5 groups [55]
Impotation
Opticns Pranci
1012
2/7
Dana Use
100
100.0%
0.0%
0.09
66.7%
100,0%
91.7%
Ovenil Paformancc
17.8%
18.68
18.4%
5.7%
6.0%
cardic_ess w d in 89
Group Gains
3,5%
1.4%
2.2%
3,016
3.51
0.25
1.4%
2.2%
3,0%
000%
0L0%
0.0%
14.5%
6.0%
14.9%
G (age, sex, race)
Max Disparity
19%
5.316
4,9%
5.2%
0.016
8.9%
8.9%
Violations
groups Polland (44)
Insputation
3.5%
Options Prined
088
0/27
0/27
5/9
18/27
822
Data Use
100
100.0%
0.0%4
0.094
0.0%
100.0%6
91.7%
Ovenll
cardic_nania 5289, - 49 (age, race)
Group Gnins
-1.1%
1.39
-1.3%
0.5%
-1.1%
-0.35%
1.39
-1.3%%
0.5%
0.0%
0.0%
1.0%
14,9%
1.0%
14.9%
8.15
Max Dispuity
2.4%
1.7%
2.49
1.7%
13,85%
13.8%
Vialtions
grrpps Jobnson et (32)
Imputation
Options Prancd
08
0/27
0/27
5/5
6/27
5/27
Dota Use
100.0%
100,015
0.0%
0,0%%
0.0%5
100.0%
87.595
Perfocmance
37.0%
37.0%%
37.0%6
col onccnee
29211. of - 72
(age, groups
Max Disuuitt
Overall
Group Gnine
4 76
-0.21%
0.79
0.15
39 0 0.21%
0.0%
0.5%
-0.1%
0.15
0.798
-0.35 - 0.208
-0.0%
0.58
0.0% - 0.0%%
0.0%6
0.0%
0.1%
1.0%
3.26
3.1%
0.1%
3.1%
1.0%
3.26
Scosyeev et al. [45]
Imputation
Options Pranci
0/6
0/12
0/12
607
5/12
5/12
Data Use
100.09
100
0.00
0.00
0.00
100.0%
75.05
Ovenil Pefformatee
20124
3000
20.39
20.0%
19,35
19.3%
langcancer 120611, 84
Overall Gain
Group Gains
0.3%
0.2%6
0.1%
0.5%
DL116
DUD1S
0.31%
025
0.19
10.5%
0.2%
0,0%
0.0%
0.21
0.0%
0.8%%
2.5%
0.01
0.71%
2.216
(age,aex)
Mat Disparity
0.655
0.5%
0,00
2.1%
Sossyov at all [45]
graups
Inpottttion Kisk
Oprions Prined
0.3%
0/6
0.516
(W6
0/12
0/12
3/7
1/12
3012
Data Use
100.00
100.0%
0.0%
0.09
50.00
100.0%
83.1%
Ovenal Performance
13.0%
9.8%
9.85
Ovzzal Gain
0.9%
-0.0%
0.09
-0.7%
1.1%
5.2%
5.2%
saps
[MllMaaee)
7797, 36
Group Gains
Disparity
-0.8%
4.2%
3.4%
-0.5% - 1133%
-0.8%
4.2%
3.4%
-0.5%
0.39
0.0% - 3.48
0.0%
16.4%
0.09
16.4%
= gropps el [3)
Insutation Risk
-0.8%
-0.7%
Options Pruned
044
041
019
119
2/5
1/9
1/9
Dota Use
100005
100.0%
0.0%
50.0%5
75.0%
67.5%
Table 5: Overview of performance, data and consent for all personalized models and systems all datasets measured by test error. We show the performance of models and systems built using random forests.
19

---

C.3
Random Forests for Ranking (AUC)
Disueet
Metiic
1Hot
miHot
KNN-1Hot
KNN-mHot
Minimal
Flat
Seq
Overall Performance
0.525
0.824
0.822
0.805
0.823
0.944
0.942
Overal Gun
0,12
0.126
0.124
x= 11
opree
[) = e
(ago, sex)
Rai. Vistation
Max Diquanty
Group
0.012
0.022
0017
0.009
0.088
0.157
0,088
Ottnntall (55)
Imputatios Risk
-0.004
-0.005
Oytots Frused
2112
100,0%
0.05
100.0%
75.0%
Overall Performance
0,595
0.396
0.897
0.386
0,394
0.987
0.957
owrdd in ce
Grup Coins
Overal Gain
-0,00
0.005
0.011
-0.005
0011
0.00
-0.00--00111
0.004
-0.005
00007
0011
0,000
0.004
0.001
0.010-0.32
0,094
0.10-0.1100
0,094
(age, sex, race)
Dispaiity
Pulled es al. (44]
[g) = grougs
Iuputation Risk
Optots Fruned
-0.005
0/5
-0.05
1427
027
10227
1027
100,0%
12.5%
100.0%
87.5%
0380
0.881
0.84
G (age, race)
caad iiiiionii
5289, m
Max Disparity
Grrsp Coins
Overal Gain
Vielatons
-0,00
0.0055
0.011
0.000
00000-0013
-0001
-000
1000
1100
-0.00-
-0000
0013
0.199
0.000
00.00
0.006
0.011
0.016 6.
0,002
0.071
108
0016
10107
0001
0,000
Jéhnon et (32)
group
Imputation Risk
Uptors Pruzed
0/
1N27
0227
622
6/27
Data Use
100,0%
100,006
0.0%
37.54
100.0%
85.3% 33.316
Uverall Performance
0,681
0.096
0.096
coloncancez
0 [*ge sex))
x 29211.4 d
[0] = grougs
Rai. Vislatius
Max Dispurit
Grup
0.005
0
-0.004
0.000
0
-0.002
-20002-0000
0000
2000
-0.044-
0.000
40000
00000
0.000-0.044
0.004
0.004
0.005
0.000
0.004
00071
0.026
Sonsyev
Imputation Risk
Uptons
2113
Data Use
100.0%%
100
0,094
50/00
100.0%
75.0%
Ovenil Performance
0,549
0.849
03849
0,808
0.856
B.866
Overill Gain
0.002
0.001
1000
0000
0.000
0,008
0,09K
-0,001
0000
-0,00-
0,002
0.000
G (age, swxx)
[g) = grougs
120641,
Rai. Violatiuss
Max Disparity
0000
0.003
5000
0.018
Scosyrev et [45]
Imputation Risk
Uptions Prined
0:001
0.001
(40)
100.0%
Performance
0.923
0.966
0.966
Ovendl Gain
0.005
0.004
0.003
-0.012
0.002
0,648
0,048
saps
7707, 36
(EIV. age)
Max
Group Cuiss
-0,002
0.013
.0,00
0.013
-0.002
-.010
0.012
-0,002
0.013
0,000
0.0000.010
0.000
0.009
0.109
0.009 0.100
0.100
Allya et 13]
(9) groups
Imputation Risk
Options Prined
00002
104
-0002
004
0/9
(N9
2/5
2/9
3/
Data Use
100.0%
100.0%
0.0%
0.0%
90.00%
102.05
87.5%
Table 6: Overview of performance, data use, and consent for all personalized models and systems on all datascts as measured by test auc. We show the performance of models and systems built using random forests.
20

---

D Supporting Material for Performance Profiles
In the performance profiles, we measure the benefit of disclosure in terms of their expected performance gain and simulate the cost of reporting for each individual by sampling their reporting cost from uniform distribution iee, for each individual i, sample e ea Uniform(O,,)) where y E [o, 0.2.. For each value of y, sample reporting costs ten times and average over the per group performance error for each sampled cost.
21


Reviews:
[{"review_id": "dNLJ1lgL1s", "reviewer": "Reviewer_tDFi", "summary": "The paper introduces Participatory Personalization in Classification (PPC), a new family of classification models that enables individuals to opt into personalization at prediction time. It argues that existing personalization models lack consent and do not allow individuals to opt out of sharing personal information or assess the benefits of personalization. PPC models address these concerns by allowing individuals to choose personalization, leading to improved performance and informed consent. The paper presents a theoretical framework for PPC and evaluates its effectiveness in various domains, such as healthcare and recommender systems. The authors propose a model-agnostic algorithm for learning PPC models with categorical group attributes. They conduct an extensive empirical study on clinical prediction tasks, comparing PPC models to common personalization and imputation approaches. ", "questions": "1. Have you considered the potential for PPC models to exacerbate existing biases or discrimination in the data? If so, how have you addressed these concerns in your empirical study?\n2. How does the computational complexity of the PPC algorithm compare to other approaches to personalization, and how scalable is it to large datasets?\n3. Have you considered the potential limitations of the real-world applications of PPC, such as the availability and quality of data, and how might these limitations affect the effectiveness of the model in practice?", "limitations": "The ethical implications of PPC, such as potential discrimination or bias, are not thoroughly evaluated, warranting further investigation.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a novel approach to personalization that allows individuals to opt into personalization at prediction time, leading to improved performance and informed consent.\n2. The authors provide a robust theoretical framework for Participatory Personalization in Classification (PPC) and demonstrate its effectiveness in various real-world applications, including healthcare and recommender systems.\n3. The paper offers a comprehensive review of related work, providing valuable insights into the limitations of existing personalization approaches.", "weaknesses": "1. A detailed comparison of PPC with other approaches to personalization, such as collaborative filtering or content-based filtering, is lacking, which could provide a more comprehensive understanding of PPC's strengths and limitations in relation to existing methods.\n2. The computational complexity of PPC and its scalability to large datasets are not extensively analyzed, leaving uncertainty about the feasibility of applying PPC in scenarios with substantial data volumes.\n3. The limitations of the real-world applications of PPC, including data availability and quality, are not extensively discussed, which could provide insights into the practical challenges of implementing PPC in different domains.\n4. A detailed analysis of the limitations of the theoretical framework of PPC, including assumptions and model constraints, is missing, hindering a comprehensive understanding of the model's applicability and potential shortcomings. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback! We respond to your comments below.\n\n> A detailed comparison of PPC with other approaches to personalization, such as collaborative filtering or content-based filtering, is lacking\n\nWe wanted to flag this comment since we think there may be a fundamental misunderstanding about the problem we are solving or what we mean by “personalization.” To be clear, we are discussing personalization in the context of supervised classification problems where we personalize predictions using categorical group attributes. We cannot compare with collaborative or content-based filtering methods because they are designed for entirely different tasks – e.g., collaborative filtering is designed for unsupervised learning from user-item interactions.\n\n> Have you considered the potential for PPC models to exacerbate existing biases or discrimination in the data? If so, how have you addressed these concerns in your empirical study?\n\nYes! We examine the biases and discrimination across groups in our results in Table 2 (see Group Gains) and in Tables 4–6 in Appendix D (see Max Disparity and Group Gains). In short, participatory systems will improve the performance for each reporting group. This can either reduce disparity (e.g., when a group suffers from worsenalization) or exacerbate disparity (when a group gains from personalization). Still, each group will be better off than they were under a personalized model, and no group will be harmed.  \n\n> A detailed analysis of the limitations of the theoretical framework of PPC, including assumptions and model constraints, is missing, hindering a comprehensive understanding of the model's applicability and potential shortcomings.\n\nWe agree! We have a more detailed agent model and analysis to show how we expect system performance to change with respect to participation. We include a brief description of the agent model and summarize its implications for system performance in Appendix A1. We hoped to include this in the paper for the reasons you mentioned but could not given the space constraints. We will integrate this material at the end of Section 2 in our revision.\n\n> The computational complexity of PPC and its scalability to large datasets are not extensively analyzed, leaving uncertainty about the feasibility of applying PPC in scenarios with substantial data volumes . . . How does the computational complexity of the PPC algorithm compare to other approaches to personalization, and how scalable is it to large datasets?\n\nWe were surprised to see this comment since we discuss the computational complexity and scalability of our method in Section 3 (see \"On Computation,\" \"Model Pool,\" and \"Enumerating Reporting Interfaces.\"). Could you let us know if there was a specific result that you would have wanted to see? \n\nOur approach is model agnostic, so its computation complexity is determined by the complexity of the algorithms we use to train the component models in the model pool. If we train $L$ random forests, we expect to incur $L$ times the training cost of a random forest. Stepping back, we do not see computation as a major issue. This is particularly true for the class of models that we considered in the paper (clinical prediction tasks with group attributes). To put things in perspective, we trained all logistic regression component models for a dataset in <1 minute and all random forest component models for a dataset in <5 minutes.\n\n> Have you considered the potential limitations of the real-world applications of PPC, such as the availability and quality of data, and how might these limitations affect the effectiveness of the model in practice?\n\nYes! We discuss this issue in Section 3 and are considering developing it further if space allows. In short, the primary bottleneck when building participatory systems is having enough data to estimate the benefits of personalization for each reporting group. From Section 3: “Given a finite sample dataset, we are limited in the number of categorical attributes used for personalization. This limitation arises because we require a minimum number of samples for each intersectional group to train a personalized model and evaluate its performance. Because the number of intersectional groups increases exponentially with each attribute, we quickly enter a regime where we cannot train models for a given group (e.g., because we lack sufficient labels) or reliably evaluate its gain for assignment and pruning.” (see Monteiro and Long et al “On the Epistemic Limits of Personalized Prediction” NeurIPS 2022).\n"}, {"review_id": "ZuRkI3j2pX", "reviewer": "Reviewer_LFwB", "summary": "In this paper, the authors propose an innovative solution, dubbed 'participatory systems', to address ethical and pragmatic issues arising from the personalization process in machine learning models. This work timely addresses an escalating concern in machine learning, specifically the utilization of sensitive or protected personal data without an individual's explicit consent and without visible individual benefits.\n", "questions": "See above", "limitations": "See above", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\nThe authors offer a comprehensive examination of the problem, substantiated with compelling real-world examples, that underlines the urgent need for a solution like the one proposed. They argue, persuasively, that traditional personalization can occasionally be counterproductive, either failing to improve or even worsening outcomes for certain groups. The authors convincingly contest the notion that models should automatically presume the availability of all data used during training at the time of prediction.\n", "weaknesses": "\nHowever, despite the clarity of the explanation and visualization, it remains unclear why exactly the proposed framework functions as it does (mechanism). In addition, the authors should consider clearly outlining the inherent assumptions underpinning their model. For instance, they could explore the potential impacts of widespread simultaneous opt-outs due to specific events, such as a controversial news story affecting a cohort of users. They should consider scenarios where a majority of users opt out during a particular period or in response to a certain question. This would provide practitioners with crucial guidance regarding the applicability and potential limitations of the proposed system.  \n\nThe authors could enhance the robustness of the paper by more thoroughly discussing potential challenges and limitations of implementing participatory systems in real-world settings. For instance, it is not clear how these systems would manage the variability in individuals' capacity or willingness to share personal data or how they would account for potential disparities in users' technological literacy. Furthermore, while the authors propose that the participatory system can curtail unnecessary data collection, there is no clear discussion on how the system would manage potential data sparsity resulting from this reduction.\n\nAnother improvement could be a more comprehensive exploration of the applicability of participatory systems to other types of personalization models or data types, such as continuous attributes or unstructured data. A thorough discussion of this aspect would provide additional value.\n\nThe paper could also benefit from an in-depth discussion on the practical hurdles and considerations in implementing participatory systems. Factors such as scalability, user interface design, and potential biases in opt-in/opt-out decisions should be considered to provide a more holistic understanding of the system's feasibility.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and detailed feedback! We were thrilled to see your positive characterization of our work. We address your comments and questions individually below.\n\n> The authors should consider clearly outlining the inherent assumptions underpinning their model. \n\nWe agree this is important, and we have included what you might be looking for in Appendix A. In short, we specified a utility-based agent model to state our assumptions surrounding individual reporting behavior explicitly. We used this model to explore how system performance changes with respect to participation under each kind of participatory system (e.g., to produce Figure 5). We also used it to prove a few formal remarks on how we expect system performance to change with regard to reporting options (which we state in this section). We hoped to include the model and our analysis in the main text but could not, given the space constraints. Based on your comment, we will refine this material and integrate it in our revision (either at the end of Sec 2 or as a standalone section)\n\n> It is not clear how these systems would manage the variability in individuals’ capacity or willingness to share personal data\n\nThanks for raising this point! To be clear, we do not want to “manage” individual willingness to share personal data. This is partly because we do not think we can easily measure it in practice and partly because we believe it is at odds with informed consent. Having said this, we agree that willingness to share data will vary across users and will affect system performance in practice. We realize that it was not clear in the initial manuscript. We hope the individual variation will be clearer as we integrate the agent model described above into the main text.\n\n> It is not clear how they would account for potential disparities in users’ technological literacy . . . user interface design and potential biases in opt-in/opt-out decisions should be considered to provide a more holistic understanding of the system’s feasibility\n\nWe agree that this could use more clarity. We designed our approach to not impose restrictions on what we would present to users – specifically because the information we’d want to show to inform consent should change drastically across applications and target audiences to inform consent (see e.g., [1]). In general, there are good “default practices” that avoid “pitfalls” that would most affect a large collection of end-users (i.e., the kind of topic that would be pursued in a user study which we discuss in our response to SNn4). In the settings that you mention, we can draw on a broad set of literature in numeracy and uncertainty quantification to promote understanding among end-users (see e.g.,[2,3]). We discuss this point briefly in Section 5, but will develop it further in our revision based on your feedback and the feedback from SNn4. \n\n> While the authors propose that the participatory system can curtail unnecessary data collection, there is no clear discussion on how the system would manage potential data sparsity resulting from this reduction.\n\nThank you for raising this point. We agree that potential data sparsity is an issue. In general, developers should not mislead users into believing that they are providing data that will improve their performance if they are really collecting their data to train and improve future models. We would also like to reference hdsX, who states: “I do not believe that the risk of group performance worsening overrides individual’s right to consent and privacy. Transparency about the discrepancy does seem important, and it does not appear this work aims to conceal that there will be disparate data availability to those who may see worse model results.” That said, building participatory systems does not preclude requesting users to report information for the purpose of model improvement voluntarily.\n\n> it remains unclear why exactly the proposed framework functions as it does (mechanism).\n\nWe describe the underpinnings of the mechanism in Appendix A and explain why we see the system function as it does in Section 4.2. Specifically, we state that the performance benefits from participatory systems stem from (i) allowing users to opt out of these instances of “worsenalization” and (ii) assigning personalized predictions with multiple models. We give specific examples in our experiments for each of these cases with reference to Table 2. \n\n> Another improvement could be a more comprehensive exploration of the applicability of participatory systems to other types of personalization models or data types, such as continuous attributes or unstructured data. A thorough discussion of this aspect would provide additional value.\n\nWe agree! While we found much ground to cover with classification problems alone, we also hope to explore participatory systems in the context of continuous attributes and unstructured data. The key requirement for applying this framework to other settings is the ability to estimate the gains of personalization. We could estimate the gains in this work because we had “groups.” When tasked with unstructured data, we may be able to accomplish this when we have multiple observations per person (e.g., in speech recognition). On the other hand, continuous attributes may prove more difficult and may require a different approach. For now, we plan to add mention and discussion of these topics to Section 5 as a potential avenue for future work.\n\n—\n\n[1] Edwards et al. Personalised risk communication for informed decision-making about taking screening tests. Cochrane Database Syst Rev. 2013\n\n[2] Estrada et al. “Health Literacy and Numeracy” JAMA 1999, Spiegelhalter “Risk and Uncertainty Communication” Annual Reviews 2017, “Effective Risk Communication for Public Health Emergency” Healthcare Basel 2020, \n\n[3] Nanayakkara et al. Visualizing Privacy-Utility Trade-Offs in Differentially Private Data Releases PoPETS 2022\n"}, {"review_id": "pmEtl730OH", "reviewer": "Reviewer_SNn4", "summary": "This paper presents a new approach for operating machine learning systems in settings where results are personalized in some fashion to individuals or group (i.e., some categorical feature that “personalizes” predictions). The goal of this new approach is to make the process of personalizing ML results more participatory for data subjects / data contributors, moving beyond the existing practices that are lacking when it comes to facilitating informed consent of data contributors. Specifically, the authors propose that by adopting the presented “participatory systems” approach, individuals who are subject to ML predictions can make decisions about sharing personal information at prediction time and better understand how information sharing impacts ML outputs. Further, the paper proposes an approach to help individuals navigate the opt in / opt out tradeoff.\n\nThe basic idea underlying the paper is to limit data use where personalization does not improve performance. The authors use clinical prediction models as a running use-case. Motivating examples include personalization based on features that are protected groups, self-reported by patients, or costly to acquire (e.g., requires doctor to label).\n\nThe paper first provides some relevant definitions that are mathematical in nature. An algorithm for “learning participatory systems”, i.e., selecting from a pool of personalized models and pruning away models that violate incentive compatibility, is presented. Then, the paper presents some experiments using the clinical prediction setting.\n", "questions": "The authors may wish to engage with some recent work on data minimization.\n-\tBiega, A.J., Potash, P., Daumé, H., Diaz, F. and Finck, M., 2020, July. Operationalizing the legal principle of data minimization for personalization. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (pp. 399-408).\n-\tShanmugam, D., Diaz, F., Shabanian, S., Finck, M. and Biega, A., 2022, June. Learning to limit data collection via scaling laws: A computational interpretation for the legal principle of data minimization. In 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 839-849).\n\nSimilarly, some recent work on collective action by data contributors may be useful for this paper (or future work, as this discussion may be out of scope). This work has great potential to scaffold and support data-related action by groups, perhaps to enable bargaining between data creators and data users.\n-\tVincent, N., Li, H., Tilly, N., Chancellor, S. and Hecht, B., 2021, March. Data leverage: A framework for empowering the public in its relationship with technology companies. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 215-227).\n-\tDas, S., Edwards, W.K., Kennedy-Mayo, D., Swire, P. and Wu, Y., 2021. Privacy for the people? exploring collective action as a mechanism to shift power to consumers in end-user privacy. IEEE Security & Privacy, 19(5), pp.66-70.\n\n\nIn the Introduction, a distinction between online platforms (that do use informed consent) and personalized models (that do not) is made. I think this distinction is not quite as clear as the current draft implies; online platforms and models are highly intertwined in many contexts. This might be downstream of the focus on medical personalization, but I think there are many other domains which much larger “grey area” between platforms data collection and data collection for modeling.\n\n\nMinor\n-\tDuplicate refs: [4] and [5]\n-\tLine 57 missing bracket\n", "limitations": "The work is heavily focused on societal impacts of ML. On this front, I think the paper is very strong.\n\nOn the methods limitations front, it might help readers to talk a bit more about the limitations of evaluating participatory systems with offline evaluation techniques, as opposed to user studies.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Overall, I found this paper to paint a compelling vision of an alternate paradigm for ML. The overall quality was high, and I believe this paper can make a large contribution to the ML research community.\n\nThe originality is high. I think this draft does a great job describing problems with the status quo around personalized modeling. While the work certainly draws on existing scholarship and practical work in privacy, human-computer interaction, and FAccT, the specific formalization presented here is original. See below for several lines of work that may be particularly relevant (though I understand if the authors see this as out of scope).\n\nThe paper makes a compelling case for the importance of this line of work. Motivation was strong in general and could attract more interest in the broad class of participatory systems. \n\nOverall clarity of the paper is high. The primary task being studied here is described well. On the whole, assumptions are stated explicitly and the claims in the paper are precisely stated.\n\nThe presentation of ideas and results is also strong. Figure 2 and Table 1 are particularly helpful.\n", "weaknesses": "Overall, the current draft makes a strong contribution. A high-level weakness of the paper is that this is very much a user-centric / human-centric paper (the core idea is help data subjects) but no evaluation with actual users/subjects is included. Nonetheless, I think this is still a reasonable starting point for advancing this line of work.\n\nSeveral other areas for potential improvement:\n\n-\tThe proposed approach / framework could be easier to follow with a summary of assumptions about the behavior of relevant agents, especially ML operators. I understood the basic assumption about groups wanting to maximizing accuracy for their group, but I think readers will want more on how different ML operator incentives matter here.\n-\tThere’s a lot of related work in human-computer interaction, privacy, and data economics that could be discussed more heavily. This may be out of scope, however.\n-\tThe paper could more explicit about how the resulting training datasets different when using this approach. The intended user experience for data subjects is very clear (i.e. I understand how a patient might interact with participatory systems), but it wasn’t totally clear if ML operators who use this approach will actually have substantially different looking training sets. Will ML operators eventually lose the ability to evaluate models for certain groups if those groups choose not to participate? In other words, what does this look for the perspective of a data engineer?\n-\tRelatedly, while the motivation is strong, I’d like to see a bit more discussion of how ML operators would adopt this approach. It seems the actual interface design could be a major barrier to use.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback! We were thrilled to read your positive impression of our work and appreciate the opportunity to discuss the finer points of this work.\n\n> There’s a lot of related work in human-computer interaction, privacy, and data economics that could be discussed more heavily. This may be out of scope, however.\n\nWe agree! We found the works you mentioned relevant and insightful, and we plan to reference them throughout the manuscript as appropriate (in the related works, discussion, and potentially in Section 2). Please let us know if you think of any other work we should be aware of.  \n\n> Online platforms and models are highly intertwined in many contexts… there are many other domains which much larger “grey area” between platforms data collection and data collection for modeling.\n\nThank you for pointing this out! We’ve updated the text to clarify the scope of comparison and are looking for a better example that would be familiar to a broader audience. If you have any suggestions, we would love to hear them.\n\n> Will ML operators eventually lose the ability to evaluate models for certain groups if those groups choose not to participate?\n\nThank you for this question! We have a few points to make that we hope to include in the paper.\n\nOperators should not “lose the ability” to evaluate models because they should retain the baseline dataset used to build their participatory system and to estimate the gains from personalization. The baseline dataset used to develop clinical prediction models may be pulled from larger repositories such as eICU, MIMIC, AllOfUS, or an EHR, where data is collected with consent for general-purpose research and would not exhibit any bias due to participation.\n\nThe main limitation in our setting would arise in settings where the ML operators would want to collect additional data from participants after they have deployed the system. This may happen because they would like to refine their estimates of the gains of personalization in order to train new models. In such settings, the data collected post-deployment could exhibit “non-response” bias as individuals opt out, and it would be misleading to use it to refine estimates or retrain models. If we retrain using only data from deployment, it is possible and even likely to lose granularity for evaluating certain groups.\n\nThe issues associated with collecting data for retraining post-deployment do merit some discussion. We view this as a potential limitation but note that its significance is unclear (e.g., many popular scoring systems are not regularly retrained, such as Gail, unchanged since 1989, and CHA2DS2-VASc, unchanged since 2009). With that said, we have a few solutions in mind: (1) we may be able to mitigate non-response bias by asking individuals to report their data to improve the system voluntarily (this would address the purpose specification issue raised in, e.g., Biega et al. Operationalizing the legal principle of data minimization for personalization); (2) it may be possible to use data collected under non-response bias by comparing the prevalence of groups in deployment to the prevalence in the baseline dataset (though this is an open problem). \n\n> A high-level weakness of the paper is that this is very much a user-centric / human-centric paper . . . but no evaluation with actual users/subjects is included . . . it might help readers to talk a bit more about the limitations of evaluating participatory systems with offline evaluation techniques, as opposed to user studies\n\nWe see where you are coming from and appreciate that you did not ask for a user study! In short, we thought extensively about including a study in this paper but ultimately decided against it. Our decision was based on the fact that the UX should change considerably across end-users and applications. For example, patient-facing systems like the ADHD screener at DoneADHD should consider issues like numeracy, but physician-facing systems supplied by MDCalc may not need to. In either of these systems, we believe design decisions will vary based on the kind of information that is reported (race, sexual history, biopsy results, etc.) and the outcome that is being used to predict (diagnosis, risk, treatment, etc.). Based on this, any user study we could include here would have limited value (if any) because it could only focus on a narrow set of users and applications. \n\nStepping back, we are very interested in the UX design aspect of this work but think the right way to explore this question is through a standalone paper (or papers) at a venue like CHI/VIS where it would be more likely to have the right reviewers and target audience. In contrast, our overarching goals in this paper were to motivate the problem, present our proposed solution, and demonstrate viability. We agree that it will be valuable to address this more clearly, and we plan to do so through a discussion that will incorporate the remarks here and in our response to LFwB.\n"}, {"review_id": "C7HYFm2LAz", "reviewer": "Reviewer_ekFJ", "summary": "Machine learning models often use personal and sensitive information without facilitating or informing individuals' consent. This paper introduces a family of classification models called participatory systems, which allow individuals to opt into personalization at prediction time. The study demonstrates that these systems can improve performance and data use while facilitating and informing consent for those who report personal data.", "questions": "1. Are all personal info available during training time? This study reminds me of the research on missing data. The personal info consent will also lead to missing data for the later rounds of retraining. It is actually a Missing-not-at-radom problem where people choose to opt out of personalization with some concerns, which definitely are not at random. \n", "limitations": "The authors have discussed the limitation that data collection may influence the later rounds of retraining.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper studied a practical problem of compromising ML model performance and personal information consenting. The idea of constructing the reporting interface tree and pruning based on the models' performance along the path is reasonable and sound. The experiments showed promising results.", "weaknesses": "The proposed method will become complicated when the number of categorical attributes increase. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for engaging with our work!\n\n> Are all personal info available during training time? \n\nThe short answer is yes. The baseline datasets used to develop clinical prediction models may be pulled from larger repositories such as eICU, MIMIC, AllOfUS, or an EHR, where data includes personal information on all participants in the initial research studies and is collected with consent for general-purpose research. \n\n> This study reminds me of the research on missing data. The personal info consent will also lead to missing data for the later rounds of retraining. It is actually a Missing-not-at-radom problem where people choose to opt out of personalization with some concerns, which definitely are not at random.\n\nThis is a good point; thank you for bringing it up. The issues associated with collecting and using data collected post-deployment do merit some discussion. \nThe key constraint on our end is that whatever solution we propose for this setting should abide by the \"purpose specification\" principle (e.g., Biega et al. Operationalizing the legal principle of data minimization for personalization for a discussion). In short, developers should not mislead users into believing that they are reporting data that will improve performance if it is actually being used to retrain the model. We discuss this briefly in our concluding remarks, though note that they were summed nicely by hdsX, who states, \"I do not believe that the risk of group performance worsening overrides an individual's right to consent and privacy. Transparency about the discrepancy does seem important, and it does not appear this work aims to conceal that there will be disparate data availability to those who may see worse model results.\" \n\nWe can think of a few solutions for this regime that abide by this principle. For example, we could ask individuals to report their data to improve the system voluntarily. There is also a potential to be able to use some of the research on MNAR data to mitigate this issue under some assumptions. For example, we think it may be possible to use data collected under non-response bias by comparing the prevalence of groups in deployment to the prevalence in the baseline dataset. We discuss this issue in more detail in our response to SNn4 but want to be clear that we see this as out-of-scope for the current work.\n\n> The proposed method will become complicated when the number of categorical attributes increase.\n\nThis is actually more complicated than it seems. Complexity _might_ (but will not necessarily) increase when the number of categorical attributes increases. In many cases, there may not be enough data to adequately estimate gains of personalization, so although the number of categorical attributes increases, the number of groups for personalization cannot (see Monteiro and Long et al. \"On the Epistemic Limits of Personalized Prediction\" NeurIPS 2022). In other cases, developers may train a simpler participatory system (e.g., with fewer component models or by building a flat or sequential system). We discuss the computational complexity and scalability of our method in Section 3, \"On Computation,\" \"Model Pool,\" and \"Enumerating Reporting Interfaces.\"\n"}]

Author rebuttal:
We thank all reviewers for their time and feedback!

We were thrilled to see that reviewers recognized the “urgent need” (LFwB) for informed consent in prediction models and that our submission was able to articulate a “compelling case for the importance of this line of work” (SNn4). 

Our reading of the reviews suggests that reviewers understood the motivation for this work and recognized essential components as “strengths” – noting that it tackles a “practical problem” (ekFJ) in a way that is “reasonable and sound” (ekFJ) and that is supported by “robust theoretical framework” (tDFi), includes “promising results” (ekFJ) that show the “effectiveness [of participatory systems in] . . . real-world applications” (tDFi), and could “make a large contribution to the ML research community” (SNn4). 

We have addressed questions and comments in responses to each reviewer. We have also compiled a list of several revisions that would improve the broader impact of this work and that we will address in our revision. These include: (i) a discussion of the viability of training before/after deployment; (ii) guidance for user interface design; (iii) moving the agent model and analysis to the main text; (iv) and highlighting ties to work from other disciplines on data minimization, data economics, and consent.

We thank you again for your time and engagement and look forward to answering any other questions you may have over the coming week.


Meta-review: The authors focus on extensions of various notions of fairness to the setting of actively withholding information from machine learning models, enabling individuals to opt out of sharing certain information about them. This is a very important topic for the machine learning community. The proposed framework and algorithm are sound and well-motivated, the presentation is very well done and the empirical results substantiate the claims, including in realistic settings.

Furthermore, the reviewers support the work and emphasize its urgency, as well as its potential for large impact on the ML community. In absence of a response from reviewer tDFi, I lowered the confidence of their rating and ignored it to some extent, but the authors also did a compelling job rebutting tDFi's concerns.

This paper has the potential to be important.
Decision: Accept (spotlight)