Title: Participatory Personalization in Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: healthcare

algorithmic fairness

data privacy

classification

interpretability
Abstract: Machine learning models are often personalized based on information that is protected, sensitive, self-reported, or costly to acquire. These models use information about people, but do not facilitate nor inform their *consent*. Individuals cannot opt out of reporting information that a model needs to personalize their predictions nor tell if they benefit from personalization in the first place. We introduce a new family of prediction models, called participatory systems, that let individuals opt into personalization at prediction time. We present a model-agnostic algorithm to learn participatory systems for supervised learning tasks where models are personalized with categorical group attributes. We conduct a comprehensive empirical study of participatory systems in clinical prediction tasks, comparing them to common approaches for personalization and imputation. Our results show that participatory systems can facilitate and inform consent in a way that improves performance and privacy across all groups who report personal data.

Participatory Personalization in Classification

Abstract

Machine learning models are often personalized with information that is protected, sensitive, self-reported, or costly to acquire. These models use information about people but do not facilitate nor inform their consent . Individuals cannot opt out of reporting personal information to a model, nor tell if they benefit from personalization in the first place. We introduce a family of classification models, called participatory systems , that let individuals opt into personalization at prediction time. We present a model-agnostic algorithm to learn participatory systems for personalization with categorical group attributes. We conduct a comprehensive empirical study of participatory systems in clinical prediction tasks, benchmarking them with common approaches for personalization and imputation. Our results demonstrate that participatory systems can facilitate and inform consent while improving performance and data use across all groups who report personal data.

1 Introduction

Machine learning models routinely assign predictions to people - be it to screen a patient for a mental illness [35], their risk of mortality in an ICU [44], or their likelihood of responding to treatment [1]. Many models in such applications are designed to target heterogeneous subpopulations using features that explicitly encode personal information. Typically, models are personalized with categorical attributes that define groups [i.e., 'categorization' as per 27]. In medicine, for example, clinical prediction models use group attributes that are protected (e.g., sex in the ASCVD Score for cardiovascular disease), sensitive (e.g., HIV_status in the VA COVID-19 Mortality Score), self-reported (e.g., alcohol_use in the HAS-BLED Score for Major Bleeding Risk), or costly to acquire (e.g., leukocytosis in the Alvarado Appendicitis Score).

Individuals expect the right to opt out of providing personal data and the ability to understand how it will be used [see, e.g., personal data guidelines in GDPR, OECD privacy guidelines 26, 40]. In many contexts, personalized models do not provide such functionality: individuals cannot opt out of reporting data used to personalize their predictions nor tell if it would improve their predictions. At the same time, practitioners assume that data available for training will be available at inference time. In practice, this assumption has led to a proliferation of models that use information that individuals may be unwilling or unable to report at prediction time [see e.g., the Denver HIV Risk Score 29, which asks patients to report age , gender , race , and sexual_practices ]. In tasks where individuals self-report, they may not voluntarily report information that could improve their predictions or may report incorrect information.

The broader need to facilitate and inform consent in personalized prediction tasks stems from the fact that personalization may not improve performance for each group that reports personal data [51]. In practice, a personalized model can perform worse or the same as a generic model fit without personal information for a group with specific characteristics. Such models violate the implicit promise of personalization as individuals report personal information without receiving a tailored performance gain in return. These instances of 'worsenalization' are prevalent, hard to detect, and hard to resolve [see 42, 51]. However, they would be resolved if individuals could opt out of personalization and understand its expected gains (see Fig. 1).

Berk Ustun UC San Diego

This work introduces a family of classification models that operationalize informed consent called participatory systems . Participatory systems facilitate consent by allowing individuals to report personal information at prediction time. Moreover, they inform consent by showing how reporting personal information will change their predictions. Models that facilitate consent operate as markets in which individuals trade personal information for performance gains. This work seeks to develop systems that perform as well as possible both when individuals opt-in - to incentivize voluntary reporting - and when they opt out - to safeguard against abstention. Our main contributions include:

We present a variety of participatory systems that provide opportunities for individuals to make informed decisions about data provision. Each system ensures that individuals who opt into personalization will receive the most accurate possible predictions possible.

We develop a model-agnostic algorithm to learn participatory systems. Our approach can produce a variety of systems that promote participation and minimize data use in deployment.

We conduct a comprehensive study of participatory systems in real-world clinical prediction tasks. The results show how our approach can facilitate and inform consent in a way that improves performance and minimizes data use.

We provide a Python library to build and evaluate participatory systems.

Related Work Participatory systems support modern principles of responsible data use articulated in OECD privacy guidelines [40], the GDPR [26], and the California Consumer Privacy Act [16]. These include: informed consent , i.e., that data should be collected with the data subject's consent; and collection limitation , i.e., that data collected should be restricted to only what is necessary. These principles stem from extensive work on the right to data privacy [33]. They are motivated, in part, by research showing that individuals care deeply about their ability to control personal data [4, 8, 10] but differ considerably in their desire or capacity to share it [see e.g. 5, 7, 9, 17, 18, 39, 41]. Our proposed systems let decision subjects report personal data in exchange for performance, which is aligned with principles articulated in recent work on data privacy [13, 46] and related to work in designing incentive-compatible prediction functions [24].

We consider models that are personalized with categorical attributes that encode personal characteristics [i.e., 'categorization' rather than 'individualization' as per 27]. Modern techniques for learning with categorical attributes [see e.g., 2, 48] use them to improve performance at a population level - e.g., by accounting for higher-order interaction effects [14, 38, 58] or recursive partitioning [11, 12, 15, 25]. Our methods can be used to achieve these goals in tasks where models use features that are optional or costly to acquire [see e.g., 6, 7, 52, 61].

Our work is related to algorithmic fairness in that we seek to improve model performance at a group level. Recent work shows that personalization with group attributes does not uniformly improve performance and can reduce accuracy at a group level [see 42, 51, 57]. Our systems can safeguard against such instances of 'worsenalization" by informing users of the gains in reporting and allowing

them to opt out of reporting. This line of broad work complements research on preference-based fairness [22, 36, 57, 59, 62], on ensuring fairness across complex group structures [28, 30, 34], and promoting privacy across subpopulations [13, 53].

2 Participatory Systems

We consider a classification task where we personalize a model with categorical attributes. We start with a dataset { ( x i , y i , g i ) } n i =1 where each example consists of a feature vector x i ∈ R d , a label y i ∈ Y , and a vector of m categorical attributes g i = [ g i, 1 , . . . , g i,m ] ∈ G 1 × . . . ×G m = G . We refer to G as group attributes , and to g i as the group membership of person i . We use n g := |{ i | g i = g }| denote the size of group g , and use |G k | to denote the number of categories for group attribute k .

We use the dataset to train a personalized model h : X × G → Y via empirical risk minimization with a loss function glyph[lscript] : Y × Y → R + . Given a model h , we denote its empirical risk and true risk as ˆ R ( h ) and R ( h ) , respectively, and evaluate model performance at the group level. We denote the empirical risk and true risk of a model h on group g ∈ G as

We consider tasks where every individual prefers more accurate predictions.

Assumption 1. Given models h and h ′ , individuals in group g prefer h to h ′ when R g ( h ) < R g ( h ′ ) .

Assumption 1 holds in settings where every individual prefers more accurate predictions - e.g., clinical prediction tasks such as screening or diagnosing illnesses [49, 56]. It does not hold in applications where some individuals prefer predictions that may be inaccurate - e.g., such as predicting the risk of organ failure for a transplant [see e.g., 43, for other 'polar" clinical applications].

Operationalizing Consent We consider models where individuals consent to personalization by deciding whether or not to report their group attributes at prediction time. We let ∅ denote an attribute that was not reported, and let r i = [ r i, 1 , . . . , r i,k ] ∈ R ⊆ G × ∅ . For example, a person with g i = [ female , HIV = + ] would report r i = [ female , ∅ ] if they only disclose sex , and would report r i = ∅ := [ ∅ , ∅ ] if they opt out of reporting entirely.

We associate each model with a set of reporting options R . A traditional model, which requires each person to report group attributes, has R = G . A model where each person could report any subset of group attributes has R = G × ∅ . We represent individual decisions to opt into personalization at prediction time through a reporting interface defined below.

Definition 1. Given a personalized classification task with group attributes G , a reporting interface is a tree T whose nodes represent attributes reported at prediction time. The tree is rooted at root ( T ) = [ ∅ , . . . , ∅ ] and branches as a person reports personal attributes. Given a node r , we denote its parent as pa ( r ) . Each parent-child pair represents a reporting decision , and the height of the tree represents the maximum number of reporting decisions.

Definition 2. Given a personalized classification task with group attributes G , a participatory system with reporting interface T is a prediction model f T : X ×R → Y that obeys the following properties:

(P1) Baseline Performance : Opting out of personalization entirely guarantees the expected performance from a generic model trained without group attributes h 0 ∈ argmin h ∈H R ( h ) .

(P2) Incentive Compatibility : Opting into personalization improves expected performance

Here, the Baseline Performance property ensures that individuals who choose not to share personal information receive the performance of a generic model - i.e., the most accurate model that could be trained without this information. This property also ensures individuals retain the ability to opt out

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

∅

+0.0%

h

0

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

7

∅

+0.0%

h

0

old

+0.5%

h

old female

+0.2%

h

1

young

+0.0%

1

0

h

old male

+0.1%

h

female

+0.6%

h

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="T

2FUHu8j prN

9E

J

Y

+Q

>A

C5n cdV

M

P

L

vS

Bwg

m

q

Dk

K

I

7

W

z

X

/f

O

Zo

y

0

G

R

3

5

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

young female

+0.0%

Flat

h

0

male

+0.4%

h

old female

+0.5%

h

1

<latexi sh

1_b

64="R

XOM

DZ

F

BkjJdzwHTIp

q

>A

C

c

V

L

E

5

+

Q

g

YyUu vo

7

r

N

/

S

8G

0m

3

9

P

W

2

n

K

f

3

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="

Z

0

uSPW

N

/E

fj

H

c

>A

C

dVF

M

U

Xy

8J

2R

o

g

I

v

Q7

9

KB

5G

rzO

D

+

q

T

Y

p

m

3

w

L

n

k

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="T

2FUHu8j prN

9E

J

Y

+Q

>A

C5n cdV

M

P

L

vS

Bwg

m

q

Dk

K

I

7

W

z

X

/f

O

Zo

y

0

G

R

3

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="T

2FUHu8j prN

9E

J

Y

+Q

>A

C5n cdV

M

P

L

vS

Bwg

m

q

Dk

K

I

7

W

z

X

/f

O

Zo

y

0

G

R

3

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="R

XOM

DZ

F

BkjJdzwHTIp

q

>A

C

c

V

L

E

5

+

Q

g

YyUu vo

7

r

N

/

S

8G

0m

3

9

P

W

2

n

K

f

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="

Z

0

uSPW

N

/E

fj

H

c

>A

C

dVF

M

U

Xy

8J

2R

o

g

I

v

Q7

9

KB

5G

rzO

D

+

q

T

Y

p

m

3

w

L

n

k

<latexi sh

1_b

64="

Z

0

uSPW

N

/E

fj

H

c

>A

C

dVF

M

U

Xy

8J

2R

o

g

I

v

Q7

9

KB

5G

rzO

D

+

q

T

Y

p

m

3

w

L

n

k

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="T

2FUHu8j prN

9E

J

Y

+Q

>A

C5n cdV

M

P

L

vS

Bwg

m

q

Dk

K

I

7

W

z

X

/f

O

Zo

y

0

G

R

3

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="T

2FUHu8j prN

9E

J

Y

+Q

>A

C5n cdV

M

P

L

vS

Bwg

m

q

Dk

K

I

7

W

z

X

/f

O

Zo

y

0

G

R

3

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="R

XOM

DZ

F

BkjJdzwHTIp

q

>A

C

c

V

L

E

5

+

Q

g

YyUu vo

7

r

N

/

S

8G

0m

3

9

P

W

2

n

K

f

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="

Z

0

uSPW

N

/E

fj

H

c

>A

C

dVF

M

U

Xy

8J

2R

o

g

I

v

Q7

9

KB

5G

rzO

D

+

q

T

Y

p

m

3

w

L

n

k

<latexi sh

1_b

64="

Z

0

uSPW

N

/E

fj

H

c

>A

C

dVF

M

U

Xy

8J

2R

o

g

I

v

Q7

9

KB

5G

rzO

D

+

q

T

Y

p

m

3

w

L

n

k

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="T

2FUHu8j prN

9E

J

Y

+Q

>A

C5n cdV

M

P

L

vS

Bwg

m

q

Dk

K

I

7

W

z

X

/f

O

Zo

y

0

G

R

3

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

Interface

Example

Description

Minimal

sex?

∅

0.0% +0.6% h 5 h 0 h age? sex? ∅ 0.0% young 0.0% old +0.5% h 0 h 0 h 0 h 7 Flat systems support partial personalization by allowing each person to report any subset of group attributes. Thus, a person with g i = [ old , female ] can report r i = [ old , ∅ ] . These systems can improve performance by using a distinct model to assign personalized predictions to each reporting group.

female

male

+0.4%

∅ 0.0% old +0.5% young +0.0% female +0.6% male +0.4% old male 0.5% young female +0.0% young male +0.6% old female +0.5% h 0 ∅ +0.0% Minimal systems let individuals opt out of personalization from an existing personalized model h . Individuals who opt out receive predictions from a generic model h 0 trained without group attributes.

Flat

Basic

h

0

old

male

+0.1%

old

female

+0.2%

young

female

+0.0%

young

male

+0.1%

h

1

h

1

h

1

h

0

h

5

h

7

h

7

h

1

1

h

9

h

0

∅

+0.0%

h

0

h

3

Flat

Sequential

sex?

sex?

old

young

∅

0.0%

female

+0.6%

male

+0.4%

∅

0.0%

young

Flat

Basic

∅

+0.0%

female

+0.0%

male

+0.0%

female

+0.0%

male

+0.6%

young

male

+0.1%

h

1

h 5 h 0 h 0 h 1 h 9 Sequential systems support partial personalization while allowing individuals to report one attribute at a time. This interface is well-suited to inform consent as users can make reporting decisions by comparing two models simultaneously. They are also well-suited for personalization tasks with information that must be acquired at prediction time (e.g., the outcome of a test result).

old

+0.5%

young

+0.0%

female

+0.6%

male

+0.4%

h

0

h

5

old

male

0.5%

young

female

+0.0%

young

male

+0.6%

old

female

+0.5%

h

7

h

7

h

1

h

9

h

0

∅

+0.0%

h

3

Sequential

Sequential

age?

sex?

sex?

sex?

old

young

∅

0.0%

female

+0.6%

male

+0.4%

∅

0.0%

∅

0.0%

young

young

0.0%

∅

+0.0%

female

+0.0%

male

+0.0%

female

+0.0%

male

+0.6%

old

+0.5%

h

0

h

0

h

5

old

male

0.5%

young

female

+0.0%

young

male

+0.6%

h

0

h

7

h

0

h

7

h

0

h

1

h

9

h

9

of personalization - i.e., ∅ ∈ R . The Incentive Compatibility property ensures that personalization will improve expected performance - i.e., when individuals report personal data, the system can effectively leverage that data to deliver more accurate predictions in expectation. Together, these properties lead to data minimization, as systems that obey these properties will not request data from a reporting group when it will not lead to an improvement in expected performance.

On Data Minimization via Imputation An alternative approach to allow individuals to opt out of reporting personal information at prediction time is to impute their group membership. Imputation allows individuals to opt out of personalization but does not guarantee the accuracy of their predictions. As a result, individuals who opt out of personalization by reporting r = ∅ may receive a less accurate prediction than they would receive from a generic model. In the best-case scenario where we could perfectly impute group membership, a group might be assigned better predictions from a generic model (see Fig. 1). In the worst case, imputation may be incorrect, leading to even more inaccurate predictions than those of the generic or personalized model. We highlight these effects on real-world datasets in our experiments in Section 4.

Characterizing System Performance One of the key differences between traditional models and participatory systems is that their performance depends on individual reporting decisions. In what follows, we characterize the performance under a general model of individual disclosure. Given a participatory system f T , we assume that each individual reports personal information to maximize an individual utility function of the form:

Here, c i ( · ) and b i ( · ) denote the cost and benefit that individual i receives from reporting r to f T respectively. We assume that individuals incur no cost when they do not report any attributes such that c i ( ∅ ) = 0 , and incur costs that increase monotonically with information disclosed such that c i ( r ) ≤ c i ( r ′ ) for r ⊆ r ′ . We assume that benefits increase monotonically with expected gains in true risk so that R r ( f T ( · , r )) < R r ( f T ( · , r ′ )) = ⇒ b i ( r , f T ) > b i ( r ′ , f T ) .

Sequential

h

0

age?

old

+0.5%

h

7

sex?

female

+0.0%

old

∅

+0.0%

<latexi sh

1_b

YUNRIWFrHg

LQ+9

q

of

>A

C

cdV

M

EP

u

j

Ky

v

B

2

O

JX

m

7

/

n

3

Z

G

k

z

5

0

D

T

p

8

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

male

+0.0%

young

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

∅

0.0%

h

0

young

0.0%

h

0

sex?

female

+0.0%

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

male

+0.6%

young

h

<latexi sh

1_b

64="

Z

0

uSPW

N

/E

fj

H

c

>A

C

dVF

M

U

Xy

8J

2R

o

g

I

v

Q7

9

KB

5G

rzO

D

+

q

T

Y

p

m

3

w

L

n

k

9

In Fig. 3, we show how the system performance for each reporting group can change with respect to participation when we simulate individual disclosure decisions from a model that satisfies the assumptions listed above. When a personalized model h requires individuals to report information that reduces performance as in Fig. 1, individuals incur a cost of disclosure without receiving a benefit in return. In such cases, individuals who interact with a minimal system would opt out of worsenalization and receive more accurate predictions from a generic model, thereby improving the overall performance of the system.

We observe that the maximum utility that each individual can receive from a participatory system can only increase as we add more reporting options. Thus, flat and sequential systems should exhibit better performance than a minimal system.

Given a participatory system f T with reporting options R , a participatory system f T ′ with more reporting options R ′ ⊇ R can only improve performance, - i.e., R ( f T ′ ) ≤ R ( f T ) . Similarly, the system with more reporting options can only improve utility, - i.e., u i ( r ; f T ′ ) ≥ u i ( r ; f T ) for all individuals i .

[<30

,

HIV+]

[>30

,

HIV-]

0

0.1

0.2

0.3

0.4

0.5

86

88

90

92

94

96

98

100

Minimal

Sequential

Generic

Personalized

Cost of Disclosure

Accuracy (%)

0

0.1

0.2

0.3

0.4

0.5

70

72.5

75

77.5

80

82.5

85

Minimal

Sequential

Generic

Personalized

Cost of Disclosure

Accuracy (%)

[<30

,

HIV+]

[>30

,

HIV-]

0

0.1

0.2

0.3

0.4

0.5

84

85

86

87

88

89

90

Minimal

Sequential

Generic

Personalized

Cost of Disclosure

Accuracy (%)

0

0.1

0.2

0.3

0.4

0.5

75

76

77

78

79

Minimal

Sequential

Generic

Personalized

Cost of Disclosure

Accuracy (%)

3 Learning Participatory Systems

This section describes a model-agnostic algorithm to learn participatory systems that ensures incentive compatibility and baseline performance in deployment. We outline our procedure in Algorithm 1 to learn the three kinds of participatory systems in Fig. 2. The procedure takes as input a pool of candidate models M , a dataset for model assignment D assign , and a dataset for pruning D prune . It outputs a collection of participatory systems that obey the properties described in Definition 2 on test data. The procedure combines three routines to (1) generate viable reporting interfaces (Line 1); (2) assign models over the interface (Line 3); (3) prune the system to limit unnecessary data collection (Line 4). We present complete procedures for each routine in Appendix A and discuss them below.

Model Pool Our procedure takes as input a pool of candidate models M to assign over a reporting interface. At a minimum, every pool should contain two models: a personalized model h for individuals who opt into personalization, and a generic model h 0 for individuals who opt out of personalization. A single personalized model can perform unreliably across reporting groups due to differences in the data distribution or trade-offs between groups. Using a pool of models safeguards against these effects by drawing on models from different model classes that have been personalized using different techniques for each reporting group. By default, we include models trained specifically on the data for each reporting group, as such models can perform well on heterogeneous subgroups [51, 57].

Enumerating Interfaces We call the ViableTrees routine in Line 1 to enumerate viable reporting interfaces. We only call this routine for sequential systems since minimal and flat systems use a single reporting interface that is known a priori. ViableTrees takes as input a group attributes G and a dataset D assign . It returns all m -ary trees that obey constraints on sample size and reporting (e.g., users who report male should report age before HIV ). By default, we only generate trees so that we have sufficient data to estimate gains at each node of the reporting interface 1 . In general, ViableTrees scales to tasks with ≤ 8 group attributes. Beyond this limit, one can reduce the enumeration size by specifying ordering constraints or a threshold number of trees to enumerate before stopping. For a task with three binary group attributes, T contains 24 3-ary trees of depth 3. Given a complete ordering of all 3 group attributes, however, T would have 1 tree. We can also consider a greedy algorithm (see Appendix A.4), which may be practical for large-scale problems.

Model Assignment We assign each reporting group a model using the AssignModels routine in Line 3. Given a reporting group r , we consider all models that could use any subset of group attributes in r . Thus, a group that reports age and sex could be assigned predictions from a model that requires age , sex , both, or neither. This implies that we can always assign the generic model to any reporting group, ensuring that the model at each node performs as well as the generic model on out-of-sample data (i.e., baseline performance in Definition 2).

Pruning Reporting Options AssignModels may output trees that violate incentive compatibility by requesting personal information that fails to improve performance. This can happen when the routine assigns a model that performs equally well to nested reporting groups - see, e.g., Fig. 2 where the Flat system assigns h 0 to [ female , ∅ ] and [ female , young ] .

We can avoid requesting data from reporting groups in such cases by calling the Prune routine in Line 4. This routine takes as input a participatory system f T and a pruning dataset D prune and outputs a system f T ′ with a pruned interface T ′ ⊆ T . The routine uses a bottom-up pruning procedure that calls a one-sided hypothesis test at each node:

The test checks if each reporting group r receives more accurate predictions from the personalized model assigned to its current node or r its parent pa ( r ) . Here, H 0 assumes a reporting group prefers the parent model. Thus, we reject H 0 when we can reliably tell that f T ( · , r ) performs better for r on the pruning dataset. The exact test should chosen based on the performance metric for the underlying

prediction task. In general, we can use a bootstrap hypothesis test [20] and draw on more powerful tests for salient performance metrics [e.g., 19, 21, 50, for accuracy and AUC].

On Computation Our approach provides several options to moderate the computation cost of training a pool of models. For example, we can train only two models and build a minimal system. Alternatively, we can also build a flat or sequential system using a limited number of models in the pool. In practice, the primary bottleneck when building participatory systems is data rather than compute . Given a finite sample dataset, we are limited in the number of categorical attributes used for personalization. This is because we require a minimum number of samples for each intersectional group to train a personalized model and evaluate its performance. Given that the number of intersectional groups increases exponentially with each attribute, we quickly enter a regime where we cannot reliably evaluate model performance for assignment and pruning [see 42].

On Customization Our procedure allows practitioners to learn systems for prediction tasks by specifying the performance metric used in assignment and pruning. A suitable performance metric should represent the gains we would show users (e.g., error for a diagnosis, AUC for triage, ECE for risk assessment). Using a pool of models allows practitioners to optimize performance across groups, which translates to gains at the population level. For sequential systems, the procedure outputs all configurations, allowing practitioners to choose between systems based on criteria not known at training time. For example, one can swap the trees to use a system that always requests HIV status last. By default, we select the configuration that minimizes data collection across groups, such that the ordering of attributes results leads to the most significant number of data requests pruned.

4 Experiments

We benchmark participatory systems on real-world clinical prediction tasks. Our goal is to evaluate these approaches in terms of performance, data usage, and consent in applications where individuals have a low reporting cost. We include code to reproduce these results in an Python library.

4.1 Setup

We consider six classification tasks for clinical decision support where we personalize a model with group attributes that are protected or sensitive (see Table 2 and Appendix B). Each task pertains to an application where we expect individuals to have a low cost of reporting and to report personal information when there is any expected gain. This is because the information used for personalization is readily available, relevant to the prediction task, and likely to be disclosed given legal protections related to the confidentiality of health data [4, 10, 54]. One exception is cardio_eicu and cardio_mimic , which are personalized based on race and ethnicity. 2 We split each dataset into a test sample (20% for evaluating out-of-sample performance) and a training sample (80% for training, pruning, assignment, and estimating gains to show users). We train three kinds of personalized models for each dataset:

Static : These models are personalized using a one-hot encoding of group attributes ( 1Hot ), and a one-hot encoding of intersectional groups ( mHot )

Imputed : These are variants of static models where we impute the group membership for each person ( KNN-1Hot , KNN-mHot ). In practice, personalized systems with imputation will range between the performance for these systems and the performance of 1Hot and mHot .

Participatory : These are participatory systems built using our approach. These include Minimal , a minimal system built from 1Hot and its generic counterpart; and Flat and Seq , flat and sequential systems built from 1Hot , mHot and their generic counterparts.

We train all models - personalized models and the components of participatory systems - from the same model class and evaluate them using the metrics in Table 1. We repeat the experiments four times, varying the model class (logistic regression, random forests) and the target performance metric (error rate for decision-making tasks, AUC for ranking tasks) to evaluate the sensitivity of our findings with respect to model classes and use cases.

4.2 Discussion

We show results for logistic regression models and error rate in Table 2 and results for other model classes and classification tasks in Appendix C. In what follows, we discuss these results.

On Performance Our results in Table 2 show that participatory systems can improve performance across reporting groups. Here, Flat and Seq achieve the best overall performance on 6/6 datasets and improve the gains from personalization for every reporting group on 5/6 datasets. In contrast, traditional models improve overall performance while reducing performance at a group level (see rationality violations on five datasets for 1Hot , mHot ). The performance benefits from participatory systems stem from (i) allowing users to opt out of these instances of 'worsenalization' and (ii) assigning personalized predictions with multiple models. Using Table 2, we can measure the impact of (i) by comparing the performance of Minimal vs. 1Hot , and the impact of (ii) by comparing the performance of Minimal to Flat (or Seq ). For example, on apnea , 1Hot exhibits a significant rationality violation for group [ 30_to_60 , male ] , meaning they would have been better off with a generic model. By comparing the performance of 1Hot to Minimal , we see that allowing users to opt out of worsenalization reduces test error from 29.1% to 28.9%. By comparing the performance on Minimal to Flat and Seq , we see that using multiple models can further reduce test error from 28.9% to 24.1%.

On Informed Consent Our results show how Flat and Seq systems can inform consent by allowing users to report a subset of group attributes (e.g., by including reporting options such as [ 30+ , ∅ ] or [ ∅ , HIV+ ] ). Although both Flat and Seq systems allow for partial personalization, their capacity to inform consent differs. In a flat system, users may inaccurately gauge the marginal benefit of reporting an attribute by comparing the gains between reporting options. For example, in Fig. 4, users who are HIV positive would see a gain of 3 . 7% for reporting [ ∅ , HIV+ ] , and 16 . 7% for reporting [ 30+ , HIV+ ] and may mistakenly conclude that the gain of reporting age is 16 . 7% -3 . 7% = 13 . 0% . This estimate incorrectly presumes that the gains of 3 . 7% were distributed equally across age groups. Sequential systems directly inform users of the gains for partial reporting. In the sequential system, group [ 30+ , HIV+ ] is informed that they would see a marginal gain of 21.5% for reporting age , while group [ <30 , HIV+ ] is informed they would see a marginal gain of reporting age of 0 . 0% .

On Data Minimization Our results show that participatory systems perform better across all groups while requesting less personal data on 6/6 datasets. For example, on cardio_eicu , Seq reduces error by 11 . 3% compared to 1Hot while requesting, on average, 83 . 3% of the data needed by 1Hot . In general, participatory systems can limit data use where personalization does not improve

performance, e.g., on lungcancer . Even as attributes like sex or age may be readily reported by patients for any performance benefit, limiting data use is valuable when there is a tangible cost associated with data collection - e.g., when models make use of rating scale for a mental disorder that must be administered by a clinician [47]. The potential for data minimization varies substantially across prediction tasks. On apnea , for example, we can prune six reporting options when building a Seq for decision making (which optimizes error) but four options for Seq for ranking (which optimizes AUC; see Appendix C.1). Overall, participatory systems satisfy 'global data minimization' as proposed in [13], in that they minimize the amount of per-user data requested while achieving the quality of a system with access to the full data on average.

On the Benefits of a Model-Agnostic Approach Our findings highlight some of the benefits of a model-agnostic approach, in which we can draw on a rich set of models to achieve better performance while mitigating harm. The resulting system can balance training costs with performance benefits. We can also ensure generalization across reporting groups - e.g., by including a generic model fit from a complex model class and personalized models fit from a simpler model class. As expected, fitting for a complex model class can lead to considerable changes in overall accuracy - e.g., we can reduce overall test error for a personalized model from 20.4% to 14.1% on saps by fitting a random forest rather than a logistic regression model (see Appendix C). However, a gain in overall performance does not always translate to gains at the group level. On saps , for example, using a random forest also introduces a rationality violation for one group.

On the Pitfalls of Imputation One of the simplest approaches to allow individuals to opt out of personalization is to pair a personalized model with an imputation technique. Although this approach can facilitate consent, it may violate the requirements in 2. Consider a personalized

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

Basic

Flat

Sequential

HIV?

age?

age?

age?

old

young

∅

+0.0

30+

+2.3%

<30

+0.28%

∅

0.0

∅

+0.0

young

negative

+3.0%

∅

+0.0

30+

+21.5%

<30

+0.0%

30+

+3.6%

<30

+0.03%

positive

+3.7%

h

0

30+

HIV+

+3.7%

30+

HIV-

+0.81%

<30

HIV-

+0.52%

<30

HIV+

+0.0%

h

1

h

0

30+

+2.3%

<30

+0.3%

HIV+

+3.7%

HIV-

+3.0%

h

0

30+

HIV-

+4.3%

<30

HIV+

+12.3%

<30

HIV-

+5.3%

30+

HIV+

+16.7%

h

0

h

0

h

3

h

1

h

0

∅

+0.0%

h

0

∅

+0.0%

h

1

h

4

h

5

h

8

h

8

h

5

h

7

h

7

h

5

h

8

h

7

h

5

h

4

h

12

Error Rate: 20.4%

Overall Gain: 1.3%

Data Use: 75%

Error Rate: 11.1%

Overall Gain: 10.6%

Data Use: 100%

Error Rate: 11.1%

Overall Gain: 10.6%

Data Use: 75%

h

1

Error Rate: 20.4%

Overall Gain: 1.3%

Data Use: 100%

Flat

Minimal

1Hot

model that exhibits 'worsenalization' in Fig. 1. Even if one could correctly impute the group membership for every person, individuals may receive more accurate predictions from a generic model h 0 . In practice, imputation is imperfect - as individuals who opt out of reporting their group membership to a personalized model may be assigned 'worse' predictions because they are imputed the group membership of a different group. In such cases, opting out may be beneficial, making it difficult for model developers to promote participation while informing consent. Our results highlight the prevalence of these effects in practice. For example, on cardio_eicu the estimated 'risk of imputation' is -4 . 6% , indicating that every intersectional group can experience an increase of 4 . 6% in the error rate as a result of incorrect imputation. The results for KNN-1Hot show that this potential harm can be realized in practice using KNN-imputation, as we find that the imputed system leads to rationality violations on 5/6 datasets.

5 Concluding Remarks

We introduced a new family of classification models that allow individuals to report personal data at prediction time. Our work focuses on personalization with group attributes; our approach could be used to facilitate and inform consent in a broader class of prediction tasks. In such cases, the key requirement for building a participatory system is that we can reliably estimate the gains of personalization for each person who reports personal data.

Our results show that participatory systems can inform consent while improving performance and reducing data use across groups. Reaping these benefits in practice will hinge on the ability to effectively inform decision subjects on the impact of their reporting decisions. [4]. Even as there may be good 'default practices' for what kind of information we should show decision subjects, practitioners should tailor this information to the application and target audience [23].

One common concern in using a participatory system arises when practitioners wish to collect data from a model in deployment to improve its performance in the future. In practice, a participatory system can thwart data collection in such settings by allowing individuals to opt out. In such cases, we would note that this issue should be resolved in a way that is aligned with the principle of purpose specification [40]. If the goal of data collection is to improve a model, then individuals could always be asked to report information voluntarily for this purpose. If the goal of data collection is to personalize predictions, then individuals should be able to opt out, especially when it may lead to worse performance.

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="CXrGn

3

kcY

L

o9W/Q

2

UM

>A

5

dVFN

EP

u

JR

qO

T

+Kv

Sw

D

yI

m

H

f

j

0

zp

B

Z

8

g

7

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="N

p

L

G/0r

I

>

C

3

c

H

S

FJ

E

2

u

kWQU

K

o

TqZ

m

w

z

X

g

M

5

7

f

Rn

8

O

j

y

9

<latexi sh

1_b

64="upYVQz

T0BUJoA

F

g

f/

>

C

X

cd

HL

MwE

q5R

+I

O

j

W

y

9

G

N

2

K7

v

nS

Z

3

k

m

D

P

r

8

<latexi sh

1_b

64="+mVLq

H

2jKvk

p

Z

QBWSoM

>A

C

3

cd

FN

8

E

r

/

RY

U

T

un

g

9D

7

z

X

G

P

f

w

J

y

0

O

I

5

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="N

p

L

G/0r

I

>

C

3

c

H

S

FJ

E

2

u

kWQU

K

o

TqZ

m

w

z

X

g

M

5

7

f

Rn

8

O

j

y

9

<latexi sh

1_b

64="

7CKXjH

IE/q

2w9Tc

UA

8

>

3

dVFNS

r

W

RYB

p

P

L

Q

u

zG

gn

D

k

m

+

Z

vf

JO

o

0

y

M

5

<latexi sh

1_b

64="R

XOM

DZ

F

BkjJdzwHTIp

q

>A

C

c

V

L

E

5

+

Q

g

YyUu vo

7

r

N

/

S

8G

0m

3

9

P

W

2

n

K

f

<latexi sh

1_b

64="Ypr gNC

F3

dj cP

yV

K

w

>A

H

LS

J

E

2v

G

Iu

5

m09

TZ

f

/

n

o

8zMD

X

W

q7B

Q

R

O

+

U

k

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="N

p

L

G/0r

I

>

C

3

c

H

S

FJ

E

2

u

kWQU

K

o

TqZ

m

w

z

X

g

M

5

7

f

Rn

8

O

j

y

9

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="upYVQz

T0BUJoA

F

g

f/

>

C

X

cd

HL

MwE

q5R

+I

O

j

W

y

9

G

N

2

K7

v

nS

Z

3

k

m

D

P

r

8

<latexi sh

1_b

64="upYVQz

T0BUJoA

F

g

f/

>

C

X

cd

HL

MwE

q5R

+I

O

j

W

y

9

G

N

2

K7

v

nS

Z

3

k

m

D

P

r

8

<latexi sh

1_b

64="q

CLm0OTScBV

j

DYw

g

A

>

5n

d

FN

M

EPUu

X

k

f

p

K

G

8

rz

7

+

I

W

3

v

Z

/

2H

o

y

9

<latexi sh

1_b

64="N

p

L

G/0r

I

>

C

3

c

H

S

FJ

E

2

u

kWQU

K

o

TqZ

m

w

z

X

g

M

5

7

f

Rn

8

O

j

y

9

<latexi sh

1_b

64="+mVLq

H

2jKvk

p

Z

QBWSoM

>A

C

3

cd

FN

8

E

r

/

RY

U

T

un

g

9D

7

z

X

G

P

f

w

J

y

0

O

I

5

<latexi sh

1_b

64="+mVLq

H

2jKvk

p

Z

QBWSoM

>A

C

3

cd

FN

8

E

r

/

RY

U

T

un

g

9D

7

z

X

G

P

f

w

J

y

0

O

I

5

<latexi sh

1_b

64="

7CKXjH

IE/q

2w9Tc

UA

8

>

3

dVFNS

r

W

RYB

p

P

L

Q

u

zG

gn

D

k

m

+

Z

vf

JO

o

0

y

M

5

References

Abajian, Aaron, Nikitha Murali, Lynn Jeanette Savic, Fabian Max Laage-Gaupp, Nariman Nezami, James S Duncan, Todd Schlachter, MingDe Lin, Jean-Francois Geschwind, and Julius Chapiro. Predicting treatment response to intra-arterial therapies for hepatocellular carcinoma with the use of supervised machine learning-an artificial intelligence concept. Journal of Vascular and Interventional Radiology , 29 (6):850-857, 2018.

Agresti, Alan. An introduction to categorical data analysis . John Wiley & Sons, 2018.

Allyn, Jérôme, Cyril Ferdynus, Michel Bohrer, Cécile Dalban, Dorothée Valance, and Nicolas Allou. Simplified acute physiology score ii as predictor of mortality in intensive care units: a decision curve analysis. PloS one , 11(10):e0164828, 2016.

Anderson, Catherine L and Ritu Agarwal. The digitization of healthcare: boundary risks, emotion, and consumer willingness to disclose personal health information. Information Systems Research , 22(3): 469-490, 2011.

Arellano, April Moreno, Wenrui Dai, Shuang Wang, Xiaoqian Jiang, and Lucila Ohno-Machado. Privacy policy and technology in biomedical data science. Annual review of biomedical data science , 1:115, 2018.

Atan, Onur, William Whoiles, and Mihaela Schaar. Data-driven online decision making with costly information acquisition. Arxiv , 02 2016.

Auer, Peter, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Auxier, Brooke, Lee Rainie, Monica Anderson, Andrew Perrin, Madhu Kumar, and Erica Turner. Americans and privacy: Concerned, confused and feeling lack of control over their personal information. Pew Research Center: Internet, Science and Tech , 2019.

Awad, Naveen Farag and Mayuram S Krishnan. The personalization privacy paradox: an empirical evaluation of information transparency and the willingness to be profiled online for personalization. MIS quarterly , pages 13-28, 2006.

Bansal, Gaurav, David Gefen, et al. The impact of personal dispositions on information sensitivity, privacy concern and trust in disclosing health information online. Decision support systems , 49(2):138-150, 2010.

Bertsimas, Dimitris and Nathan Kallus. From predictive to prescriptive analytics. Management Science , 66(3):1025-1044, 2020.

Bertsimas, Dimitris, Jack Dunn, and Nishanth Mundru. Optimal prescriptive trees. INFORMS Journal on Optimization , 1(2):164-183, 2019.

Biega, Asia J, Peter Potash, Hal Daumé, Fernando Diaz, and Michèle Finck. Operationalizing the legal principle of data minimization for personalization. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval , pages 399-408, 2020.

Bien, Jacob, Jonathan Taylor, and Robert Tibshirani. A lasso for hierarchical interactions. Annals of statistics , 41(3):1111, 2013.

Biggs, Max, Wei Sun, and Markus Ettl. Model distillation for revenue optimization: Interpretable personalized pricing. arXiv preprint arXiv:2007.01903 , 2020.

Bukaty, P. The California Consumer Privacy Act (CCPA): An implementation guide . IT Governance Publishing, 2019. ISBN 9781787781337. URL https://books.google.com/books? id=vGWfDwAAQBAJ .

Campbell, Tim S and William A Kracaw. Information production, market signalling, and the theory of financial intermediation. the Journal of Finance , 35(4):863-882, 1980.

Chemmanur, Thomas J. The pricing of initial public offerings: A dynamic model with information production. The Journal of Finance , 48(1):285-304, 1993.

DeLong, Elizabeth R, David M DeLong, and Daniel L Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics , pages 837-845, 1988.

DiCiccio, Thomas J and Bradley Efron. Bootstrap confidence intervals. Statistical science , pages 189-212, 1996.

Dietterich, Thomas G. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation , 10(7):1895-1923, 1998.

Do, Virginie, Sam Corbett-Davies, Jamal Atif, and Nicolas Usunier. Online certification of preference-based fairness for personalized recommender systems. arXiv preprint arXiv:2104.14527 , 2021.

Edwards, Adrian GK, Gurudutt Naik, Harry Ahmed, Glyn J Elwyn, Timothy Pickles, Kerry Hood, and Rebecca Playle. Personalised risk communication for informed decision making about taking screening tests. Cochrane database of systematic reviews , Cochrane database of systematic reviews(2), 2013.

Eliaz, Kfir and Ran Spiegler. On incentive-compatible estimators. Games and Economic Behavior , 132: 204-220, 2022.

Elmachtoub, Adam N, Vishal Gupta, and Michael Hamilton. The value of personalized pricing. Available at SSRN 3127719 , 2018.

European Parliament and of the Council. Regulation 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation), 2016. URL https://eur-lex.europa.eu/eli/reg/2016/679/oj . Official Journal of the European Union.

Fan, Haiyan and Marshall Scott Poole. What is personalization? perspectives on the design and implementation of personalization in information systems. Journal of Organizational Computing and Electronic Commerce , 16(3-4):179-202, 2006.

Globus-Harris, Ira, Michael Kearns, and Aaron Roth. An algorithmic framework for bias bounties. 2022 ACMConference on Fairness, Accountability, and Transparency , Jun 2022. doi: 10.1145/3531146.3533172. URL http://dx.doi.org/10.1145/3531146.3533172 .

Haukoos, Jason S, Michael S Lyons, Christopher J Lindsell, Emily Hopkins, Brooke Bender, Richard E Rothman, Yu-Hsiang Hsieh, Lynsay A MacLaren, Mark W Thrun, Comilla Sasson, et al. Derivation and validation of the denver human immunodeficiency virus (hiv) risk score for targeted hiv screening. American journal of epidemiology , 175(8):838-846, 2012.

Hébert-Johnson, Úrsula, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In Proceedings of the International Conference on Machine Learning , pages 1944-1953, 2018.

Hollenberg, SM. Cardiogenic shock. In Intensive Care Medicine , pages 447-458. Springer, 2003.

Johnson, Alistair EW, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Kaminski, Margot E. The right to explanation, explained. Berkeley Tech. LJ , 34:189, 2019.

Kearns, Michael, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning , pages 2564-2572, 2018.

Kessler, Ronald C, Lenard Adler, Minnie Ames, Olga Demler, Steve Faraone, EVA Hiripi, Mary J Howes, Robert Jin, Kristina Secnik, Thomas Spencer, et al. The world health organization adult adhd self-report scale (asrs): a short screening scale for use in the general population. Psychological medicine , 35(2): 245-256, 2005.

Kim, Michael P, Aleksandra Korolova, Guy N Rothblum, and Gal Yona. Preference-informed fairness. arXiv preprint arXiv:1904.01793 , 2019.

Le Gall, Jean-Roger, Stanley Lemeshow, and Fabienne Saulnier. A new simplified acute physiology score (saps ii) based on a european/north american multicenter study. Jama , 270(24):2957-2963, 1993.

Lim, Michael and Trevor Hastie. Learning interactions via hierarchical group-lasso regularization. Journal of Computational and Graphical Statistics , 24(3):627-654, 2015.

Lundberg, Ian, Arvind Narayanan, Karen Levy, and Matthew J Salganik. Privacy, ethics, and data access: A case study of the fragile families challenge. Socius , 5:2378023118813023, 2019.

OECD. Recommendation of the council concerning guidelines governing the protection of privacy and transborder flows of personal data, 2013. URL https://legalinstruments.oecd.org/en/ instruments/OECD-LEGAL-0188 .

Ortlieb, Martin and Ryan Garner. Sensitivity of personal data items in different online contexts. itInformation Technology , 58(5):217-228, 2016.

Paes, Lucas Monteiro, Carol Xuan Long, Berk Ustun, and Flavio Calmon. On the epistemic limits of personalized prediction. In Oh, Alice H., Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview. net/forum?id=Snp3iEj7NJ .

Paulus, Jessica K and David M Kent. Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities. NPJ digital medicine , 3(1):1-8, 2020.

Pollard, Tom J, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific data , 5(1):1-13, 2018.

Scosyrev, Emil, James Messing, Katia Noyes, Peter Veazie, and Edward Messing. Surveillance epidemiology and end results (seer) program and population-based research in urologic oncology: an overview. In Urologic Oncology: Seminars and Original Investigations , volume 30, pages 126-132. Elsevier, 2012.

Shanmugam, Divya, Fernando Diaz, Samira Shabanian, Michèle Finck, and Asia Biega. Learning to limit data collection via scaling laws: A computational interpretation for the legal principle of data minimization. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 839-849, 2022.

Sharp, Rachel. The hamilton rating scale for depression. Occupational Medicine , 65(4):340-340, 2015.

Steyerberg, Ewout W et al. Clinical prediction models . Springer, 2019.

Struck, Aaron F, Berk Ustun, Andres Rodriguez Ruiz, Jong Woo Lee, Suzette M LaRoche, Lawrence J Hirsch, Emily J Gilmore, Jan Vlachy, Hiba Arif Haider, and Cynthia Rudin. Association of an electroencephalography-based risk score with seizure probability in hospitalized patients. JAMA neurology , 74(12):1419-1424, 2017.

Sun, Xu and Weichao Xu. Fast implementation of delong's algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Processing Letters , 21(11):1389-1393, 2014. doi: 10.1109/LSP.2014.2337313.

Suriyakumar, Vinith M, Marzyeh Ghassemi, and Berk Ustun. When personalization harms: Reconsidering the use of group attributes in prediction. In International Conference on Machine Learning , 2023.

Tran, Cuong and Ferdinando Fioretto. Personalized privacy auditing and optimization at test time. arXiv preprint arXiv:2302.00077 , 2023.

Tran, Cuong, My Dinh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the fairness lens. Advances in Neural Information Processing Systems , 34:27555-27565, 2021.

U.S. Congress. Health insurance portability and accountability act of 1996, 1996. URL https://www. hhs.gov/hipaa/for-professionals/privacy/index.html . Public Law 104-191.

Ustun, Berk, M Brandon Westover, Cynthia Rudin, and Matt T Bianchi. Clinical prediction models for sleep apnea: the importance of medical history over symptoms. Journal of Clinical Sleep Medicine , 12 (02):161-168, 2016.

Ustun, Berk, Lenard A Adler, Cynthia Rudin, Stephen V Faraone, Thomas J Spencer, Patricia Berglund, Michael J Gruber, and Ronald C Kessler. The world health organization adult attention-deficit/hyperactivity disorder self-report screening scale for dsm-5. Jama psychiatry , 74(5):520-526, 2017.

Ustun, Berk, Yang Liu, and David Parkes. Fairness without harm: Decoupled classifiers with preference guarantees. In International Conference on Machine Learning , pages 6373-6382, 2019.

Vaughan, Gregory, Robert Aseltine, Kun Chen, and Jun Yan. Efficient interaction selection for clustered data via stagewise generalized estimating equations. Statistics in Medicine , 39(22):2855-2868, 2020.

Viviano, Davide and Jelena Bradic. Fair policy targeting. arXiv preprint arXiv:2005.12395 , 2020.

Vyas, Darshali A, Leo G Eisenstein, and David S Jones. Hidden in plain sight-reconsidering the use of race correction in clinical algorithms, 2020.

Yu, Shipeng, Balaji Krishnapuram, Rómer Rosales, and R. Bharat Rao. Active sensing. In AISTATS , 2009.

Zafar, Muhammad Bilal, Isabel Valera, Manuel Rodriguez, Krishna Gummadi, and Adrian Weller. From parity to preference-based notions of fairness in classification. In Advances in Neural Information Processing Systems , pages 228-238, 2017.

Supplementary Material

A Supporting Material for Section 3

A.1 Enumeration Routine for Algorithm 1

We summarize the Enumeration routine in Algorithm 2. Algorithm 2 takes as input a set of group attributes G and a dataset D and outputs a collection of reporting interfaces T that obey ordering and plausibility constraints.

The routine enumerates all possible reporting interfaces for a given set of group attributes G through a recursive branching process. Given a set of group attributes, the routine is called for each attribute that has yet to be considered in the tree Line 4, ensuring a complete enumeration. We note that the routine is only called for building Sequential systems since there is only one possible reporting interface for Minimal and Flat systems.

Enumerating all possible trees ensures we can recover the best tree given the selection criteria and allows practitioners to choose between models based on other criteria. We generate trees that meet plausibility constraints based on the dataset, such as having at least one negative and one positive sample and at least s total samples at each leaf. In settings constrained by computational resources, we can impose additional stopping criteria and modify the ordering to enumerate more plausible trees first or exclusively (e.g., by changing the ordering of G or imposing constraints in VALIDASSIGNMENTS).

A.2 Assignment Routine for Algorithm 1

We summarize the routine for AssignModels procedure in Algorithm 3.

Algorithm 3 takes as inputs a reporting tree T , a pool candidate models M , and an assignment (training) dataset D and outputs a tree T that maximizes the gains of reporting group information. The pool of candidate models is filtered to viable models for each reporting group. Since the pool of candidate models includes the generic model h 0 , each reporting group will have at least one viable model. We assign each reporting group the best-performing model on the training set and default to the generic model h 0 when a better-performing personalized model is not found. We assign performance on the training set and then prune using performance on the validation set to avoid biased gain estimations.

A.3 Pruning Routine for Algorithm 1

We summarize the routine used for the PruneLeaves procedure in Algorithm 1. The PruneLeaves routine

Algorithm 1 takes as input a reporting interface T and a validation sample D , and performs a bottom-up pruning to output a reporting interface T that asks individuals to report attributes that are expected to lead to a gain. The pruning decision at each leaf is based on a hypothesis test that evaluates the gains of reporting for a reporting group on a validation dataset. This test has the form:

This procedure evaluates the gains of reporting by comparing the performance of a model assigned at a leaf node h and a model assigned at a parent node h ′ which does not use the reported information. Here, the null hypothesis H 0 assumes that the parent model performs as well as the leaf model - and thus, we reject the null hypothesis when there is sufficient evidence to suggest that reporting will improve performance in deployment. Our routine allows practitioners to specify the hypothesis test to compute the gains. By default, we use the McNemar test for accuracy [21] and the Delong test for AUC [19, 50]. In general, we can use a bootstrap hypothesis test [20].

A.4 Greedy Induction of Sequential Reporting Interface

We present an additional routine to construct reporting interfaces for sequential systems in Algorithm 5. We include this routine as an alternative option that can be used to construct a reporting interface in settings where it may be impractical or undesirable to enumerate all possible reporting interfaces. The procedure results in a valid reporting interface that ensures gains. However, it does not guarantee an optimal tree in terms of maximizing the overall gain and does not allow to practitioners to choose between reporting interfaces after training.

Algorithm 5 takes as input a collection of reporting options R and outputs a single reporting interface using a greedy tree induction routine that chooses the attribute to report to maximize the minimum gain at each step. The procedure uses the reporting options to iteratively construct a reporting tree that branches on all of the attributes in R . The procedure considers each unused attribute for each splitting point and splits on the attribute that provides the greatest minimum gain for the groups contained at that node.

B Description of Datasets used in Section 4 - Experiments

We include additional information about the datasets used in Section 4.

Table 3: Datasets used to fit clinical prediction models in Section 4. Here: n denotes the number of examples in each dataset; d denotes the number of features; G denotes the group attributes that are used for personalization; and m = |G| denotes the number of intersectional groups. Each dataset is de-identified and available to the public. The cardio_eicu , cardio_mimic , lungcancer datasets require access to public repositories listed under the references. The saps and apnea datasets must be requested from the authors. The support dataset can be downloaded directly from the URL below.

apnea We use the obstructive sleep apnea (OSA) dataset outlined in Ustun et al. [55]. This dataset includes a cohort of 1,152 patients where 23% have OSA. We use all available features (e.g. BMI, comorbidities, age, and sex) and binarize them, resulting in 26 binary features.

cardio_eicu & cardio_mimic Cardiogenic shock is an acute condition in which the heart cannot provide sufficient blood to the vital organs [31]. These datasets are designed to predict cardiogenic shock for patients in intensive care. Each dataset contains the same features, group attributes, and outcome variables for patients in different cohorts. The cardio_eicu dataset contains records for a cohort of patients in the Collaborative Research Database V2.0 [44]. The cardio_eicu dataset contains records for a cohort of patients in the MIMIC-III [32] database. Here, the outcome variable indicates whether a patient in the ICU with cardiogenic shock will die while in the ICU. The features encode the results of vital signs and routine lab tests (e.g. systolic BP, heart rate, hemoglobin count) that were collected up to 24 hours before the onset of cardiogenic shock.

lungcancer We consider a cohort of 120,641 patients who were diagnosed with lung cancer between 2004-2016 and monitored as part of the National Cancer Institute SEER study [45]. Here, the outcome variable indicates if a patient dies within five years from any cause, and 16.9% of patients died within the first five years from diagnosis. The cohort includes patients from Greater California, Georgia, Kentucky, New Jersey, and Louisiana, and does not cover patients who were lost to follow-up (censored). Age and Sex were considered as group attributes. The features reflect the morphology and histology of the tumor (e.g., size, metastasis, stage, node count and location, number and location of notes) as well as interventions that were administered at the time of diagnosis (e.g., surgery, chemo, radiology).

coloncancer We consider a cohort of 120,641 patients who were diagnosed with colorectal cancer between 2004-2016 and monitored as part of the National Cancer Institute SEER study [45]. Here, the outcome variable indicates if a patient dies within five years from any cause, and 42.1% of patients die within the first five years from diagnosis. The cohort includes patients from Greater California. Age and Sex were considered as group attributes. The features reflect the morphology and histology of the tumor (e.g., size, metastasis, stage, node count and location, number and location of notes) as well as interventions that were administered at the time of diagnosis (e.g., surgery, chemo, radiology).

saps The Simplified Acute Physiology Score II (SAPS II) score predicts the risk of mortality of critically-ill patients in intensive care [37]. The data contains records of 7,797 patients from 137 medical centers in 12 countries. Here, the outcome variable indicates whether a patient dies in the ICU, with 12.8% patient of patients dying. The features reflect comorbidities, vital signs, and lab measurements.

C Results for Different Model Classes and Prediction Tasks

In this Appendix, we present experimental results for additional model classes and prediction tasks. We produce these results using the setup in Section 4.1, and summarize them in the same way as Table 2. We refer to them in our discussion in Section 4.2.

C.1 Logistic Regression for Ranking (AUC)

C.2 Random Forests for Decision-Making (Error)

C.3 Random Forests for Ranking (AUC)

Table 6: Overview of performance, data use, and consent for all personalized models and systems on all datasets as measured by test auc . We show the performance of models and systems built using random forests .

D Supporting Material for Performance Profiles

In the performance profiles, we measure the benefit of disclosure in terms of their expected performance gain and simulate the cost of reporting for each individual by sampling their reporting cost from a uniform distribution - i.e., for each individual i , we sample c i as c i ∼ Uniform (0 , γ ) , where γ ∈ [0 , 0 . 2] . For each value of γ , we sample reporting costs ten times and average over the per group performance error for each sampled cost.

Peer reviews:
comments: SUMMARY:
Machine learning models often use personal and sensitive information without facilitating or informing individuals' consent. This paper introduces a family of classification models called participatory systems, which allow individuals to opt into personalization at prediction time. The study demonstrates that these systems can improve performance and data use while facilitating and informing consent for those who report personal data.

STRENGTHS:
The paper studied a practical problem of compromising ML model performance and personal information consenting. The idea of constructing the reporting interface tree and pruning based on the models' performance along the path is reasonable and sound. The experiments showed promising results.

WEAKNESSES:
The proposed method will become complicated when the number of categorical attributes increase.

QUESTIONS:
1. Are all personal info available during training time? This study reminds me of the research on missing data. The personal info consent will also lead to missing data for the later rounds of retraining. It is actually a Missing-not-at-radom problem where people choose to opt out of personalization with some concerns, which definitely are not at random.

LIMITATIONS:
The authors have discussed the limitation that data collection may influence the later rounds of retraining.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach for operating machine learning systems in settings where results are personalized in some fashion to individuals or group (i.e., some categorical feature that “personalizes” predictions). The goal of this new approach is to make the process of personalizing ML results more participatory for data subjects / data contributors, moving beyond the existing practices that are lacking when it comes to facilitating informed consent of data contributors. Specifically, the authors propose that by adopting the presented “participatory systems” approach, individuals who are subject to ML predictions can make decisions about sharing personal information at prediction time and better understand how information sharing impacts ML outputs. Further, the paper proposes an approach to help individuals navigate the opt in / opt out tradeoff.\n\nThe basic idea underlying the paper is to limit data use where personalization does not improve performance. The authors use clinical prediction models as a running use-case. Motivating examples include personalization based on features that are protected groups, self-reported by patients, or costly to acquire (e.g., requires doctor to label).\n\nThe paper first provides some relevant definitions that are mathematical in nature. An algorithm for “learning participatory systems”, i.e., selecting from a pool of personalized models and pruning away models that violate incentive compatibility, is presented. Then, the paper presents some experiments using the clinical prediction setting.\n\nSTRENGTHS:\nOverall, I found this paper to paint a compelling vision of an alternate paradigm for ML. The overall quality was high, and I believe this paper can make a large contribution to the ML research community.\n\nThe originality is high. I think this draft does a great job describing problems with the status quo around personalized modeling. While the work certainly draws on existing scholarship and practical work in privacy, human-computer interaction, and FAccT, the specific formalization presented here is original. See below for several lines of work that may be particularly relevant (though I understand if the authors see this as out of scope).\n\nThe paper makes a compelling case for the importance of this line of work. Motivation was strong in general and could attract more interest in the broad class of participatory systems. \n\nOverall clarity of the paper is high. The primary task being studied here is described well. On the whole, assumptions are stated explicitly and the claims in the paper are precisely stated.\n\nThe presentation of ideas and results is also strong. Figure 2 and Table 1 are particularly helpful.\n\nWEAKNESSES:\nOverall, the current draft makes a strong contribution. A high-level weakness of the paper is that this is very much a user-centric / human-centric paper (the core idea is help data subjects) but no evaluation with actual users/subjects is included. Nonetheless, I think this is still a reasonable starting point for advancing this line of work.\n\nSeveral other areas for potential improvement:\n\n-\tThe proposed approach / framework could be easier to follow with a summary of assumptions about the behavior of relevant agents, especially ML operators. I understood the basic assumption about groups wanting to maximizing accuracy for their group, but I think readers will want more on how different ML operator incentives matter here.\n-\tThere’s a lot of related work in human-computer interaction, privacy, and data economics that could be discussed more heavily. This may be out of scope, however.\n-\tThe paper could more explicit about how the resulting training datasets different when using this approach. The intended user experience for data subjects is very clear (i.e. I understand how a patient might interact with participatory systems), but it wasn’t totally clear if ML operators who use this approach will actually have substantially different looking training sets. Will ML operators eventually lose the ability to evaluate models for certain groups if those groups choose not to participate? In other words, what does this look for the perspective of a data engineer?\n-\tRelatedly, while the motivation is strong, I’d like to see a bit more discussion of how ML operators would adopt this approach. It seems the actual interface design could be a major barrier to use.\n\nQUESTIONS:\nThe authors may wish to engage with some recent work on data minimization.\n-\tBiega, A.J., Potash, P., Daumé, H., Diaz, F. and Finck, M., 2020, July. Operationalizing the legal principle of data minimization for personalization. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (pp. 399-408).\n-\tShanmugam, D., Diaz, F., Shabanian, S., Finck, M. and Biega, A., 2022, June. Learning to limit data collection via scaling laws: A computational interpretation for the legal principle of data minimization. In 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 839-849).\n\nSimilarly, some recent work on collective action by data contributors may be useful for this paper (or future work, as this discussion may be out of scope). This work has great potential to scaffold and support data-related action by groups, perhaps to enable bargaining between data creators and data users.\n-\tVincent, N., Li, H., Tilly, N., Chancellor, S. and Hecht, B., 2021, March. Data leverage: A framework for empowering the public in its relationship with technology companies. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 215-227).\n-\tDas, S., Edwards, W.K., Kennedy-Mayo, D., Swire, P. and Wu, Y., 2021. Privacy for the people? exploring collective action as a mechanism to shift power to consumers in end-user privacy. IEEE Security & Privacy, 19(5), pp.66-70.\n\n\nIn the Introduction, a distinction between online platforms (that do use informed consent) and personalized models (that do not) is made. I think this distinction is not quite as clear as the current draft implies; online platforms and models are highly intertwined in many contexts. This might be downstream of the focus on medical personalization, but I think there are many other domains which much larger “grey area” between platforms data collection and data collection for modeling.\n\n\nMinor\n-\tDuplicate refs: [4] and [5]\n-\tLine 57 missing bracket\n\nLIMITATIONS:\nThe work is heavily focused on societal impacts of ML. On this front, I think the paper is very strong.\n\nOn the methods limitations front, it might help readers to talk a bit more about the limitations of evaluating participatory systems with offline evaluation techniques, as opposed to user studies.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose an innovative solution, dubbed 'participatory systems', to address ethical and pragmatic issues arising from the personalization process in machine learning models. This work timely addresses an escalating concern in machine learning, specifically the utilization of sensitive or protected personal data without an individual's explicit consent and without visible individual benefits.\n\nSTRENGTHS:\nThe authors offer a comprehensive examination of the problem, substantiated with compelling real-world examples, that underlines the urgent need for a solution like the one proposed. They argue, persuasively, that traditional personalization can occasionally be counterproductive, either failing to improve or even worsening outcomes for certain groups. The authors convincingly contest the notion that models should automatically presume the availability of all data used during training at the time of prediction.\n\nWEAKNESSES:\nHowever, despite the clarity of the explanation and visualization, it remains unclear why exactly the proposed framework functions as it does (mechanism). In addition, the authors should consider clearly outlining the inherent assumptions underpinning their model. For instance, they could explore the potential impacts of widespread simultaneous opt-outs due to specific events, such as a controversial news story affecting a cohort of users. They should consider scenarios where a majority of users opt out during a particular period or in response to a certain question. This would provide practitioners with crucial guidance regarding the applicability and potential limitations of the proposed system.  \n\nThe authors could enhance the robustness of the paper by more thoroughly discussing potential challenges and limitations of implementing participatory systems in real-world settings. For instance, it is not clear how these systems would manage the variability in individuals' capacity or willingness to share personal data or how they would account for potential disparities in users' technological literacy. Furthermore, while the authors propose that the participatory system can curtail unnecessary data collection, there is no clear discussion on how the system would manage potential data sparsity resulting from this reduction.\n\nAnother improvement could be a more comprehensive exploration of the applicability of participatory systems to other types of personalization models or data types, such as continuous attributes or unstructured data. A thorough discussion of this aspect would provide additional value.\n\nThe paper could also benefit from an in-depth discussion on the practical hurdles and considerations in implementing participatory systems. Factors such as scalability, user interface design, and potential biases in opt-in/opt-out decisions should be considered to provide a more holistic understanding of the system's feasibility.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Participatory Personalization in Classification (PPC), a new family of classification models that enables individuals to opt into personalization at prediction time. It argues that existing personalization models lack consent and do not allow individuals to opt out of sharing personal information or assess the benefits of personalization. PPC models address these concerns by allowing individuals to choose personalization, leading to improved performance and informed consent. The paper presents a theoretical framework for PPC and evaluates its effectiveness in various domains, such as healthcare and recommender systems. The authors propose a model-agnostic algorithm for learning PPC models with categorical group attributes. They conduct an extensive empirical study on clinical prediction tasks, comparing PPC models to common personalization and imputation approaches.\n\nSTRENGTHS:\n1. The paper presents a novel approach to personalization that allows individuals to opt into personalization at prediction time, leading to improved performance and informed consent.\n2. The authors provide a robust theoretical framework for Participatory Personalization in Classification (PPC) and demonstrate its effectiveness in various real-world applications, including healthcare and recommender systems.\n3. The paper offers a comprehensive review of related work, providing valuable insights into the limitations of existing personalization approaches.\n\nWEAKNESSES:\n1. A detailed comparison of PPC with other approaches to personalization, such as collaborative filtering or content-based filtering, is lacking, which could provide a more comprehensive understanding of PPC's strengths and limitations in relation to existing methods.\n2. The computational complexity of PPC and its scalability to large datasets are not extensively analyzed, leaving uncertainty about the feasibility of applying PPC in scenarios with substantial data volumes.\n3. The limitations of the real-world applications of PPC, including data availability and quality, are not extensively discussed, which could provide insights into the practical challenges of implementing PPC in different domains.\n4. A detailed analysis of the limitations of the theoretical framework of PPC, including assumptions and model constraints, is missing, hindering a comprehensive understanding of the model's applicability and potential shortcomings.\n\nQUESTIONS:\n1. Have you considered the potential for PPC models to exacerbate existing biases or discrimination in the data? If so, how have you addressed these concerns in your empirical study?\n2. How does the computational complexity of the PPC algorithm compare to other approaches to personalization, and how scalable is it to large datasets?\n3. Have you considered the potential limitations of the real-world applications of PPC, such as the availability and quality of data, and how might these limitations affect the effectiveness of the model in practice?\n\nLIMITATIONS:\nThe ethical implications of PPC, such as potential discrimination or bias, are not thoroughly evaluated, warranting further investigation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}