Faking Fairness via Stealthily
Biased SamplingFeb 10, 2020. AAAI 2020 on AISI Track
Kazuto Fukuchi
University of Tsukuba / RIKEN AIP
Joint work withSatoshi Hara (Osaka University) and Takanori Maehara (RIKEN AIP)

1

Unfairness in Machine Learning

Hiring [Dastin’18]
Face recognition [Buolamwini+’18]

2Machine translation[Şarbak’s facebook post]

Criminal risk assessment [Angwin+’16]

Promotion of Fairness
Company

ServiceML model3

Promotion of Fairness
CompanyIs ML Model fair?

ServiceML model4

Promotion of Fairness
CompanyOk! Model is fair!

ServiceML model
5

Evidence of fairness

Example: Score based evidence• Fairness score: a level of fairness• Many tools for auditing fairness score have developed.

• E.g., FairML, AI Fairness 360 [Bellamy+’18], Aequitas [Saleiro+’18]

Bias

6FairFairAI Fairness 360, Demo, https://aif360.mybluemix.net/dataBiasFake Fairness of Model
Company
Is this fakeevidence?

ServiceUnfair ML model Fake evidence offairness
7

Evidence of Fairness

ProsScoreConsML model is in private We cannot detect fakeBenchmark
ML model is in private We can detect fake(?)
datasetModel8

No chance to fakeLeakage of confidential
information

Evidence of Fairness

ProsScoreCons

ML model is in private We cannot detect fake

BenchmarkML model is in private We can detect fake(?)
datasetModel

9

No chance to fakeLeakage of confidential
information

Contributions

Fake in benchmark dataset is almostimpossible to detect!

• Construct an attack algorithm, stealthily biased
subsampling attack.

• Show the generated fake dataset is almost impossible todetect in theoretical and experimental ways.

10Stealthily biased subsampling attack• Two goals:• Fairness: S looks fair
• Stealthiness: Distribution of S is similar to that of D

11

Stealthily biased subsampling attack• Two goals:• Fairness: S looks fair
• Stealthiness: Distribution of S is similar to that of D

Target contingency table

Original dataset

Subsample

DS

FemaleMale

Positive

40%40%

Negative

10%10%

Fairness

minS W(S, D)
Stealthiness

12

Optimization
Minimize Wasserstein
distance
Stealthiness
Fairness

minS W(S, D)
sub to C(S) = CT
Contingency table of S isequivalent to target

13

Optimization
Minimize Wasserstein
distance
Stealthiness
Fairness

minS W(S, D)
sub to C(S) = CT
Contingency table of S isequivalent to target• This is a linear programming but its general solver is slow :(Develop fast optimization technique
2.5
with complexity O( | D | )14

Does Wasserstein distance actually work?

DDS

Rand
om
selectD or S

Reference
datasetD′
Goodness-of-fit test

15

Does Wasserstein distance actually work?

DDS

Rand
om
selectD or S

Reference
datasetD′
Goodness-of-fit test(Theorem) For KS-test detector,Detectability ≤ O(K 1/sW(μ K, ν K )) + o(1) .

16

Minimizing WD =>Minimizing upper bound on detectability

Synthetic dataset: Settings• Evaluation criteria• Fairness: DP = | ℙ(y = 1 | s = 1) − ℙ(y = 1 | s = 0) |• Stealthiness: Power of KS test with significance 0.05.
• Attacker made subsamples so thatℙ(y = 1 | s = 1) ≈ ℙ(y = 1 | s = 0) ≈ α.

• Original dataset: DP = 0.2, sample size = 1000, andα ≈ 0.6.

• Reference sample size: 200
17

Synthetic dataset: Result

DP is much smaller thanoriginal (0.2)

18Power is same assignificance (0.05)

Real datasets: Settings• Evaluation criteria• Fairness: DP = | ℙ(y = 1 | s = 1) − ℙ(y = 1 | s = 0) |• Stealthiness: W(S, D′)• Attacker made 2000 subsamples so thatℙ(y = 1 | s = 1) ≈ ℙ(y = 1 | s = 0) ≈ α.

• Data• COMPAS (4000) and Adult (20000)• Reference sample size: 2000
• α ≈ 0.6.
19

Real dataset: COMPAS

DP is much smaller thanoriginal(dotted line)20

WD is same as baseline

Real dataset: Adult

DP is much smaller thanoriginal(dotted line)21

WD is same as baseline(dotted line)

Conclusions
Summary:• An evil company can deceive people by publishing fakeevidence of fairness.
• We CANNOT detect fake in benchmark dataset.
We’re facing a risk of fake fairness.
Paper: https://arxiv.org/abs/1901.08291
Code: https://github.com/sato9hara/stealthily-biased-samplingThank you!
22