Contest Rule
PWS2020 Committee
Cup Working Group

1st version: 2020/08/26© 2020 PWS2020 Committee Cup Working Group1

PWS2020 Committee Cup WG Members• Koji Chida (NTT)• Hitomi Arai (RIKEN)• Makoto Iguchi (Kii)• Hidenobu Oguri (Fujitsu Laboratories)• Hiroaki Kikuchi (Meiji University)• Atsushi Kuromasa (FJCT)• Hiroshi Nakagawa (RIKEN)• Yuichi Nakamura (Waseda University)• Kenshiro Nishiyama (BizReach)• Ryo Nojima (NICT)• Satoshi Hasegawa (NTT)• Takuma Hatano (NS Solutions)• Koki Hamada (NTT)• Ryo Furukawa (NEC)• Takao Murakami (AIST)• Yuji Yamaoka (Fujitsu Laboratories)• Akira Yamada (KDDI Research)• Chiemi Watanabe (Tsukuba Universityof Technology)© 2020 PWS2020 Committee Cup Working Group2

Schedule
08/07(Fri)-08/26(Wed)08/26(Wed)Contest entry
Contest rule open

08/27(Thu)-09/07(Mon)09/09(Wed)-09/18(Fri)
09/22(Tue)Preliminary round (Anonymization Phase)
Preliminary round (Attack Phase)
Preliminary round result announcement

09/24(Thu)-10/05(Mon)10/07(Wed)-10/20(Tue)
10/27(Tue)
10/27(Tue)Final round (Anonymization Phase)
Final round (Attack phase)
Final result announcement @ CSS2020
Poster presentation by each team @ CSS2020(Data anonymization and attack methods)© 2020 PWS2020 Committee Cup Working Group3

Overview• PWSCUP2020: The 6th data anonymization and attack contest• A.K.A. AMIC (“Anonymity against Membership Inference” Contest)• As in previous years, each team will anonymize given data and attack
anonymized data made by other teams.
• In previous PWSCUP, we mainly used the re-identification attack (record linkage)as a privacy measure. This year, we will use the membership inference attack as a
privacy measure.
‒ Membership inference: Estimate whose data is included in anonymized data.
‒ The membership inference can be used as a privacy measure for many anonymization
methods, including synthetic data generation that is attracting attention in the machinelearning area.
‒ The hide-and-seek privacy challenge hosted in NeurIP2020 (one of the top conferences inmachine learning) also focuses on the membership inference attack on synthetic data.

• The membership inference attack may also be used as a privacy measure incases where the existence of the data subject in the learning data is sensitive by
nature (e.g., data analysis of a person infected with a COVID-19).
© 2020 PWS2020 Committee Cup Working Group

4

Overview (continue)• Awards (For each award, the top 3 teams with high scores will win. Check page
10 for how the score is counted.)‒ Overall award: 1st, 2nd, and 3rd place‒ Anonymization award: 1st, 2nd, and 3rd place‒ Attack award: 1st, 2nd, and 3rd place

You have a chance to receive
multiple awards!

• Prohibited actions‒ Sharing sub-sampled data, that is separately and privately distributed to each team, to otherteams.
‒ Intentionally obstructing the committee• Others‒ The rules of the final round may change based on the result of the preliminary round. If wechange the rule, we will notify you immediately.
‒ Some members of the PWS2020 committee Cup WG will also participate in the contest. We
will ensure the fairness (e.g., removing them from the contact ML).
‒ There will be a team from WG who will participate only in the anonymization phase for
evaluating the anonymized data. This team will be excluded from the award for each phase.
© 2020 PWS2020 Committee Cup Working Group5

Re-Identification (Record Linkage) vs Membership Inference① Personal Dataset（One ID per one person）
ID

Attr1

Attr2

・・・

②Swap rows randomly③Anonymized data

ID

PID

Attr1

Attr2

・・・

00002
00005
00001

Attr1

Attr2

・・・

Re-identification (record linkage)
Estimate ID column of ② using ① and ③

Anonymize
10001

PID

ID

10002

10001

00004

10003

10002

00002

10003

00001

10004
(Preserve row order)

00001

00004

00002

00003

10005

10004

00005

00003

・・・

・・・

10005

00003

00004
00005

④Sub-sample rows randomly
ID

・・・

00001
00003

• Terms

00004
・・・

Attr1

Attr2

・・・

⑤Anonymized data
Attr1

Attr2

・・・

Anonymize‒ Personal Dataset: Tabular data with personal data in each row

Accuracy
40%

Membership inference
Estimate ID column of ④ using ① and ⑤(Arbitrary order)
ID
00002
00003

Accuracy
33%

00005

• In this contest, only one row for one person (i.e., the same ID does not appear in multiple rows)‒ Attack: Performing membership inference or re-identification (record linkage). An attack is strong if its accuracy
is high.
‒ Anonymize: Processing personal dataset to make the data tolerant to attacks (①→③, ①→⑤, ②→③, ④→⑤）
© 2020 PWS2020 Committee Cup Working Group6

AMIC Overview
Synthetic Data

The image part with relationship ID rId5 was not found in the file.

Sub-Sampled Data

Sub-Sampled Data

1

1

2

3

2

…

3

1

3

…

4

…

3
4

Sampling

1…

…

100,000 records(100,000 PII)

10,000 records
for each Anonymizer

The image part with relationship ID rId2 was not found in the file.

Synthetic data 100,000 records(100,000 PII)

Questioner

Anonymized Data

Anonymize

10,000 records

Submit to Questioner

Team（as Anonymizer）
The image part with relationship ID rId9 was not found in the file.

Synthetic Data andAnonymized Data
from other Anonymizers
12
3

Anonymized data from Anonymizer
10,000 records

Anonymization Phase

4…

Attack Phase

Estimated Sub-Sampled Data

MembershipInference
Attack

❓
❓
❓
❓
❓
❓

2
3

2

…

3

2

…

3

Team
（as Attacker)…

Estimate Sub-Sampled Data for each Anonymized
Data and submit them to Questioner• Remark‒ We use synthetic data derived from Personal dataset to prevent Anonymizers from knowing nonsampled data.
• Synthetic data: a dataset having similar characteristics (e.g., distribution) to the personal dataset• We use Census Income Data Set, which is a public dataset, as a personal dataset.

‒ Sub-sampled data is generated by sampling Synthetic data randomly for each team.
‒ Each Attacker submits row numbers of the most confident 100 records from each anonymized
data as a result of the membership inference attack.
© 2020 PWS2020 Committee Cup Working Group7

Personal Dataset Used in AMIC

• Census Income Data Set https://archive.ics.uci.edu/ml/datasets/census+income• Public data with 15 attributes for machine learning. Training data (32,561 records) and test data (16,281 records)• From this dataset, we generate synthetic data (100,000 records) with no duplicate records using the synthetic data
generation method described in the next page.
‒ First, generate 1,000,000 synthetic data with duplicated records. Then, remove duplicated records and sample 100,000 records randomly• We use the following 9 attributes. They are selected by considering the number and distribution of attribute values(73*8*16*7*14*6*2*99*2=2,175,731,712 possible combinations）

Table 1: Attributes and their values used in AMIC

age: continuous. [17-90]
workclass(8): Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.
education(16): Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool.
marital-status(7): Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse.
occupation(14): Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing,Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces.
relationship(6): Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.
sex(2): Female, Male.
hours-per-week: continuous. [1-99]
income(2): >50K, <=50K
Header→(The header is
excluded infiles used in
the contest)age

workclasseducation

marital-status

occupation

relationshipsexhours-per-weekincome39

State-gov

Bachelors

Never-married

Adm-clerical

Not-in-familyMale

40 <=50K

50

Self-emp-not-inc

Bachelors

Married-civ-spouse

Exec-managerial

HusbandMale

13 >50K

38

PrivateHS-gradDivorced

Handlers-cleaners

Not-in-familyMale

40 <=50K

53

Private11th

Married-civ-spouse

Handlers-cleaners

HusbandMale

40 <=50K

28

PrivateBachelors

Married-civ-spouse

Prof-specialty

Wife

Female40 >50K

…

…

…

…

…

…

…

…

…

© 2020 PWS2020 Committee Cup Working Group8

Synthetic Data Generation Algorithm Used in AMIC
• [OMTH17] : Okada et al. Privacy Preserved Synthetic Datasets Generation by Using Statistics,
CSS2017 3F3-4. (in Japanese)• The algorithm generates synthetic data having the same averages for all attributes andvariance-covariance matrix as those of the personal dataset (input).
‒ The actual values differ slightly due to discretization and the maximum/minimum value adjustment.

• Category attributes are converted to dummy variables and treated as numerical attributes.
• The number of records in synthetic data can be set to an arbitrary number.

Input: A vector of average values for each attributes 𝝁𝝁, variance-covariance matrix Σ , Histogram of each attribute, thenumber of records in synthetic data
Output: Synthetic data 𝑍𝑍
1.

2.
3.

4.

5.

Generate a random dataset 𝑌𝑌 having the same histograms and record numbers as those of the input.

Whiten 𝑌𝑌 and generate dataset 𝑌𝑌𝑌 where the average of each attributes is 0 and the variance-covariance matrix is
an identity matrix.
Calculate a rotation matrix Σ = 𝑈𝑈Λ𝑈𝑈 𝑇𝑇 and a scaling matrix Λ1/2 (convert each element of Λ to the square root)

Calculate 𝑌𝑌𝑌(𝑈𝑈Λ1/2 )𝑇𝑇 and generate dataset 𝑍𝑍 ∗ by adding 𝝁𝝁 in each row

Perform discretization and the maximum/minimum value adjustment to 𝑍𝑍 ∗ and output the result as synthetic
data 𝑍𝑍© 2020 PWS2020 Committee Cup Working Group9

Di. Anonymized data made be Anonymizer i Please remember
A. Personal dataset (Open Data)
the notation
Eij. Estimation of Anonymizer i’s
B. Synthetic data (Closed Data)“A, B, C, D, E”
Sub-sampled data made by Attacker j
Ci. Sub-Sampled data for Anonymizer i

Scoring

The image part with relationship ID rId6 was not found in the file.

A. Personal
DatasetB. Synthetic Data

Ci. Sub-Sampled Data

1

1

Imitate

2
3
4

2
3
4

Sampling

1

3

2

…

3

1

3

…

4

…

…

…

…

100,000 records(100,000 PII)

10,000 records
for each Anonymizer

The image part with relationship ID rId3 was not found in the file.

B. Synthetic data 100,000 records(100,000 PII)

Questioner••

Ci. Sub-Sampled Data

1

Submit to Questioner

Team i（as Anonymizer）
The image part with relationship ID rId9 was not found in the file.

B. Synthetic data andAttack Phase
Di . Anonymized datafrom Anonymizers
Eij. Estimation of sub-sampled data
1

MembershipInference

2
3

D1. Anonymized data from
Anonymizer 1 10,000 records

Anonymization Phase‒ Attackers conduct the membership inference using anonymized
data Di made by each Anonymizer and submit the estimated subsampled data Eij‒

Di. Anonymized Data

Anonymize

10,000 records

Overall‒ Team Score: The inverse of the sum of the anonymization phase ranking
and attack phase ranking.

‒Anonymization Phase

Score of Anonymizer i: 1 - (the maximum accuracy marked among all
Attackers)
Accuracy of Attacker j for Anonymizer i : |Eij⋂Ci|/100

4
…

❓
❓
❓
❓
❓
❓

2
3

2

…

3

2

…

3

Team j（as Attacker)…

Estimate sub-sampled data for each anonymized
data and submit them to Questioner•

Attack Phase‒ Score of Attacker j: The accuracy of Eij for the winner of Anonymization Phase•

Scores for Anonymization and Attack Phases are calculated by summing up thescore of the preliminary and final rounds with a ratio of 1:9.

•

Note: We are planning to multiply scores by 1,000 and cut the decimal points so thatthe scores will not become small values.
© 2020 PWS2020 Committee Cup Working Group10

Notation and flow

Preparation: Questioner

𝐴𝐴

G

𝐵𝐵

Personal Dataset
Synthetic Data
𝐴𝐴’s Synthetic
generation Function
Data
Anonymization: Anonymizer 𝒊𝒊

𝐶𝐶𝑖𝑖

Q𝑖𝑖

𝐷𝐷𝑖𝑖

𝐷𝐷𝑖𝑖

V𝑘𝑘

𝑈𝑈𝑖𝑖𝑖𝑖

𝑇𝑇𝑇𝑇𝑖𝑖𝐶𝐶𝑖𝑖
𝑖𝑖

H
Sampling
Function

Sub-Sampled Data
for Anonymizer 𝑖𝑖

Attack: Attacker 𝒋𝒋

𝐵𝐵

R𝑗𝑗

𝐸𝐸𝑖𝑖𝑗𝑗Sub-Sampled Anonymization Anonymized Synthetic Data
Membership Estimated
Data
Function
Data
ResultInference𝑖𝑖
Function
Score
of
Utility Estimation:
Anonymized Data
Questioner Utility Measure 𝑘𝑘

‒
‒
•

𝐶𝐶

?

Utility measure

Privacy Estimation: Questioner

𝐸𝐸

𝑆𝑆

Anonymized Data
W
𝑖𝑖𝑗𝑗
𝑖𝑖𝑖𝑖
Evaluation
Estimated Result Evaluation Score of
Function for𝑖𝑖
Function for Privacy𝑘𝑘
Utility Measure 𝑘𝑘𝑖𝑖
Privacy Measure Measure
Sub-Sampled
Threshold of
Data
Utility Measure 𝑘𝑘 Sub-Sampled Data (Answer)

≥

•

𝐷𝐷

𝑇𝑇

𝐶𝐶

A quantitative evaluation of the similarity between Personal Dataset and their Anonymized Data, such as statistics and the prediction/classification results by machine learning.
The score is high if these data are more similar.

Privacy measure‒‒

A quantitative evaluation of the tolerance against attack.

The score is high if the accuracy of membership inference against the data is low.

© 2020 PWS2020 Committee Cup Working Group11

Preparation:
Questioner’s task
• Personal Dataset

Preparation: Questioner

𝐴𝐴

G

𝐵𝐵

Personal Dataset
Synthetic Data
𝐴𝐴’s Synthetic
generation Function
Data

H
Sampling
Function‒ Extract 30,162 records with no missing value from Census Income Data Set training data (32,561 records)‒ Use 9 attributes: age, workclass, education, marital-status, occupation, relationship, sex, hours-per-week, income

𝑇𝑇𝑇𝑇𝑖𝑖𝐶𝐶𝑖𝑖
𝑖𝑖

Sub-Sampled Data
for Anonymizer 𝑖𝑖• Synthetic Data Generation Function‒ Use [OMTH17] method‒ The function can output any number of records (Here, we generate 100,000 records with no duplication)‒ Average and variance of each attribute and covariance between two attributes are similar to the original personal dataset (category attributes
are converted to dummy variables)‒ Implemented code: gen.py  Released for your trial• Synthetic Data

‒ Dataset with 100,000 records with no duplicate. Its average, variance, and covariance are similar to the personal dataset.
‒ This dataset is not disclosed to Anonymizer (only a sub-sampled data will be disclosed)• Sampling Function‒ Take a dataset as an input, output records by sampling the dataset with the specified sample rate (0-1).
‒ In the contest, we set the sampling rate to 0.1. Sub-sampled data with 10,000 records is generated by sampling 100,000 record Synthetic Data
randomly.
‒ Implemented code: randomsampling.py  Released for your trial• Sampled Data

‒ 10,000 records generated by sampling the synthetic data randomly. Each Anonymizer will receive different sampled data.

• Other‒ Questioner will record and hold line numbers of synthetic data corresponding to sampled data for each Anonymizer (Note: the line number
starts with 0)© 2020 PWS2020 Committee Cup Working Group12

Anonymization: Anonymizer 𝒊𝒊

Anonymization: Anonymizer’s task

• Sampled Data

• 10,000 records generated by sampling the synthetic data randomly.
Each Anonymizer will receive different sampled data.

𝐶𝐶𝑖𝑖

Q𝑖𝑖

𝐷𝐷𝑖𝑖

Sub-Sampled Anonymization Anonymized
Data
Function
Data• Anonymization Function‒ Implemented by Anonymizer‒ The function generates anonymized data with high privacy score (i.e., strong against membership inference
attacks by Attackers)‒ We will release some sample codes (to be described later)• Anonymized Data

‒ To be submitted to Questioner‒ The number of records in anonymized data does not have to match with that of sub-sampled data, but the
number has to be from 1,000 to 100,000.
‒ Note that the privacy score is designed to descrease as the difference in the number of records increase.

‒ The privacy score must be higher than the designated threshold value (to be described later), or yourteam will be disqualified.
‒ Limitation: For the convenience of the privacy score, the attributes of anonymized data are 9 attributes
covered in Table 1 on page 7. The values of the attributes should match with the ones covered in Table 1
on Page 7.
• Example: The values of the attribute "age" is an integer from 17 to 99. The values of the attribute "age" in anonymized
data is also an integer from 17 to 99 (values such as 100, 20s, [20-24] are not allowed).
© 2020 PWS2020 Committee Cup Working Group13

Anonymization: Sample Codes

Anonymization: Anonymizer 𝒊𝒊

𝐶𝐶𝑖𝑖

Q𝑖𝑖

𝐷𝐷𝑖𝑖

Sub-Sampled Anonymization Anonymized
Data
FunctionData• synthetic.py (Synthetic data generator)‒ Generate synthetic data with similar mean and variance of each attribute and covariance between two
attributes of the input data.
‒ Can specify the number of records in synthetic data.

• rr.py (Randomizer/Randomized Response/PRAM)‒ Given the retention probability p (0-1), maintain the value of each cell with probability p and replace the value
with a random attribute value with probability 1-p.

• rrp.py (Randomizer/Randomized Response/PRAM)‒ Given the retention probability p (0-1), maintain the value of each cell with probability p and replace the value
with a random attribute value according to the distribution of input data with probability 1-p.

• kanony.py (k-anonymity with record suppression)‒ Given the attribute(s) and threshold k, check the values of the specified attributes. If there are k or more
identical sets of attribute values, maintain them. If there are k-1 or fewer sets of attribute values, delete thecorresponding records.
© 2020 PWS2020 Committee Cup Working Group14

‒ A quantitative evaluation of the similarity between
personal dataset and their anonymized data, such as
statistics and the prediction/classification results bymachine learning.
• The score is high if these data are more similar.

V𝑘𝑘

𝑈𝑈𝑖𝑖𝑖𝑖Anonymized Data
Evaluation
Function for𝑖𝑖
𝑘𝑘
Utility Measure 𝑘𝑘
Sub-Sampled
Threshold of
Data
Utility Measure 𝑘𝑘

𝐶𝐶?

• Utility Measure

𝐷𝐷𝑖𝑖

≥

Utility Estimation: Questioner
and Anonymizer's task

Score of
Utility Estimation:
Questioner Utility Measure 𝑘𝑘

𝑇𝑇• Evaluation Function for Utility Measure‒ Anonymizer needs to generate anonymized data with its utility scores of the histogram, thevariance-covariance matrix, and the decision tree analysis being greater than or equal to thethreshold values (the details explained in the following pages).
‒ Evaluation code (utilityfunc.py) is released  Anonymizer can check if the utility score of theiranonymized data is above the threshold.

• Score and Threshold for Utility Measure‒ Details explained in the following pages• Note‒ Questioner will execute utilityfunc.py on anonymized data submitted by Anonymizers to check if theutility score is above the threshold value  If the score is below the threshold, the team will bedisqualified !
© 2020 PWS2020 Committee Cup Working Group15

Utility Measure: Histogram

𝐷𝐷𝑖𝑖

V𝑘𝑘

𝑈𝑈𝑖𝑖𝑖𝑖Anonymized Data
Evaluation
Function for𝑖𝑖
𝑘𝑘
Utility Measure 𝑘𝑘
Sub-Sampled
Threshold of
Data
Utility Measure 𝑘𝑘
?

𝐶𝐶

≥

• Be aware that you will be disqualified if anonymized datayou‘ve submitted to Questioner has the utility score below
the threshold !!

Score of
Utility Estimation:
Questioner Utility Measure 𝑘𝑘

𝑇𝑇• Evalution code utilityfunc.py is released. You can check
if your anonymized data clears the threshold.
• Utility score threshold: Thistgram = 0.99

© 2020 PWS2020 Committee Cup Working Group

16

V𝑘𝑘

𝑈𝑈𝑖𝑖𝑖𝑖Anonymized Data
Evaluation
Function for𝑖𝑖
𝑘𝑘
Utility Measure 𝑘𝑘
Sub-Sampled
Threshold of
Data
Utility Measure 𝑘𝑘

𝐶𝐶?

• Be aware that you will be disqualified if anonymized datayou‘ve submitted to Questioner has the utility score below
the threshold !!

𝐷𝐷𝑖𝑖

≥

Utility Measure:
Variance-Covariance Matrix

Score of
Utility Estimation:
Questioner Utility Measure 𝑘𝑘

𝑇𝑇• Evalution code utilityfunc.py is released. You can check
if your anonymized data clears the threshold.
• Utility score threshold: TVCM = 0.4© 2020 PWS2020 Committee Cup Working Group17

Utility Measure: Decision tree analysis

𝐷𝐷𝑖𝑖

V𝑘𝑘

𝑈𝑈𝑖𝑖𝑖𝑖Anonymized Data
Evaluation
Function for𝑖𝑖
𝑘𝑘
Utility Measure 𝑘𝑘
Sub-Sampled
Threshold of
Data
Utility Measure 𝑘𝑘
?

𝐶𝐶

≥

• Be aware that you will be disqualified if anonymized datayou‘ve submitted to Questioner has the utility score below
the threshold !!

Score of
Utility Estimation:
Questioner Utility Measure 𝑘𝑘

𝑇𝑇• Evaluation code utilityfunc.py is released. You can check if youranonymized data clears the threshold.
• Utility score threshold: TDTA = 0.85 (for both X6 and X9)© 2020 PWS2020 Committee Cup Working Group18

Decision tree analysis algorithm• Decision tree analysis‒ Use sklearn.tree.DecisionTreeClassifer in scikit-learn• https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifer.html• DecisionTreeClassifer parameters are as follows:‒ random_state = 0‒ max_depth = 5 (for income estimation)‒ max_depth = 3 (for relationship estimation)‒ For other parameters, use the default settings.

© 2020 PWS2020 Committee Cup Working Group19

Attack: Attacker j

Attack: Attacker's task𝐵𝐵

• Synthetic data, anonymized data from Anonymizers

Synthetic Data

‒ Receive from Questioner

𝐷𝐷𝑖𝑖• Membership Inference Function‒ From synthetic and anonymized data, estimate which records
in synthetic data is sampled and anonymized.
‒ Sample code attack.py is released

Anonymized
Data• Based on the distance between each record of synthetic data B andanonymized data Di, estimate sub-sampled data Ci and generate the estimated result Eij.
• Calculate Euclidean distance between records••

For an integer attirbute, the distance is the difference of values
For a category attribute, compare values and count up thenumber of matching values.

• Return the row number of synthetic data B record that has thesmallest distance from each record in anonymized data

• Estimated Result

Anonymized Data

R𝑗𝑗

MembershipInference
Function

𝐸𝐸𝑖𝑖𝑗𝑗Estimated
ResultSynthetic Data

37

Male

<=50K

43

Male

<=50K

44

Female

>50K

39

Male

>50K

28

Female<=50K52

Female

>50K

・・・

・・・

・・・

‒ Submitted to Questioner from Attacker‒ Estimate which records in synthetic data are used to generate anonymized data.
‒ 100 row numbers of synthetic data (Note: the row number starts from 0)

Dist.

6
3
17

• TIPS (?)‒ Anonymized data and sub-sampled data have similar histogram, variance-covariance matrix, and classification
results of decision tree analysis (meet the threshold restriction).
→ Find sub-sampled data candidate from synthetic data that has similar histogram, variance-covariance matrix,
and classificaiotn results of decision tree analysis with those of anonymized data.
© 2020 PWS2020 Committee Cup Working Group20

Privacy Estimation: Questioner

Privacy Estimation: Qestioner's task

𝐸𝐸𝑖𝑖𝑗𝑗W

𝑆𝑆𝑖𝑖𝑖𝑖

Estimated Result Evaluation Score of
Function for Privacy
𝑖𝑖
Privacy Measure Measure
Sub-Sampled Data (Answer)• Estimated Result𝐶𝐶‒ 100 row numbers of synthetic data that represent theestimate of which records in synthetic data are used
to generate anonymized data submitted by Anonymizer(Note: the row number starts at 0)• Answer (Sub-sampled Data)‒ Sub-sampled data that Questioner distributed to each Anonymizer‒ Only the row numbers that indicate which rows in synthetic data were sampled is needed(Note: the row number starts at 0)• Evaluation Function of Privacy Measure‒ Return the number of matches between an estimated result (100 rows) and the answer (10,000
rows) as the evaluation score of the privacy measure.
‒ The evaluation score of the privacy measure ranges from 0 to 100.

‒ Implemented code privacymeasure.py released for trial.

© 2020 PWS2020 Committee Cup Working Group21

Summary of Data and Codes provided by Questioner• Data and codes provided to Anonymizers‒‒

Sub-sampled data (independent per Anonymizer)
Anonymization sample codes‒‒

synthetic.py

Generateanonymized
data and submit
it to Questioner‒ rr.py‒ rrp.py‒ kanony.py
Utility index evaluation function utilityfunc.py• Data and codes provided to Attackers‒‒‒

Synthetic data
Anonymized data generated
by Anonymizers

Attack sample code: attack.py

Submit an
estimated result(attack result) to
Questioner• Sample data and codes‒

Sample synthetic data‒

Sample sub-sampled data‒

Sample anonymized data‒

Sample answer data‒

Synthetic data generation function: gen.py‒

Sampling function: randomsampling.py‒

Privacy Evaluation Function: privacymeasure.py• Thresholds for Utility Measure‒

Histogram: 0.99‒

Variance-Covariance Matrix: 0.4‒

Decision Tree Analysis: 0.85

G𝐵𝐵

Anonymization: Anonymizer 𝒊𝒊

𝐶𝐶𝑖𝑖

Q𝑖𝑖

𝐷𝐷𝑖𝑖

𝐷𝐷𝑖𝑖

V𝑘𝑘

𝑈𝑈𝑖𝑖𝑖𝑖

𝑇𝑇𝑇𝑇𝑖𝑖𝐶𝐶𝑖𝑖
𝑖𝑖

H

Personal Dataset
Synthetic Data
𝐴𝐴’s Synthetic
generation Function
Data

Sampling
Function

Sub-Sampled Data
for Anonymizer 𝑖𝑖

Attack: Attacker 𝒋𝒋

𝐵𝐵

R𝑗𝑗

𝐸𝐸𝑖𝑖𝑗𝑗Sub-Sampled Anonymization Anonymized Synthetic Data
Membership Estimated
Data
Function
Data
ResultInference𝑖𝑖
Function
Score
of
Utility Estimation:
Anonymized Data
Questioner Utility Measure 𝑘𝑘

𝐷𝐷

Privacy Estimation: Questioner

𝐸𝐸

𝑆𝑆

Anonymized Data
W
𝑖𝑖𝑗𝑗
𝑖𝑖𝑖𝑖
Evaluation
Estimated Result Evaluation Score of
Function for𝑖𝑖
Function for Privacy𝑘𝑘
Utility Measure 𝑘𝑘𝑖𝑖
Privacy Measure Measure
Sub-Sampled
Threshold of
Data
Utility Measure 𝑘𝑘 Sub-Sampled Data (Answer)

𝐶𝐶?

Census Income Data Set

𝐴𝐴

≥

‒

Preparation: Questioner

𝑇𝑇𝐶𝐶© 2020 PWS2020 Committee Cup Working Group22

Data and Code Formats
category

Data retained by
Questioner

Data and codes
submitted toAnonymizers

Data received from
Anonymizers

Data and codes
submitted to
Attackers
Data received from
Attackers

Data and codes for
trial

Description

filename• Character code: UTF-8• pre/main: preliminary round=pre, final round=main
formatRemarkRow numbers of sub-sampled data[pre/main]_answer_[anonymizer number].index

Index file with 1 column and 100 rows

• Row numbers are set from “0”Sub-sampled data[pre/main]_samplingdata_[anonymizer number].csv

CSV file with 9 columns and 10,000 rows

• Anonymizer numbers: 01 to 99

Sample code 1 for anonymization

synthetic.py

Please refer readme.txt

Sample code 2 for anonymization

rr.Py

Please refer readme.txt

Sample code 3 for anonymization

rrp.pyPlease refer readme.txt

Sample code 4 for anonymization

kanony.py

Please refer readme.txt

Utility estimation function

utilityfunc.py

Please refer readme.txt

Test data for decision tree analysis

test.csv

CSV file with 9 columns and 15,060 rows

Anonymized data[pre/main]_anonymizeddata_[anonymizer number].csv

CSV file with 9 columns and 1,000 to
100,000 rows

Synthetic data[pre/main]_syntheticdata.csv

CSV file with 9 columns and 100,000 rows

Anonymized data for all anonymizers[pre/main]_anonymizeddata_[anonymizer number].csv

CSV file with 9 columns and 1,000 to
100,000 rows

Sample code for attack

attack.py

Please refer readme.txt

Estimated data for membership
inference

inference_[anonymizer number]_[attacker
number].index

CSV file with 1 column and 100 rows

Census Income Data Set

census_income.data.csv

CSV file with 9 columns and 30,162 rows

Synthetic data

trial_syntheticdata.csv

CSV file with 9 columns and 100,000 rows

Sub-sampled data

trial_samplingdata.csv

CSV file with 9 columns and 10,000 rows

Answer data for membership inference

trial_answer.index

CSV file with 1 column and 10,000 rows

Anonymized data

trial_anonymizeddata.csv

CSV file with 9 columns and 10,000 rows

Synthetic data generation function

gen.pyPlease refer readme.txt

Sub-sampling function

randomsampling.py

Please refer readme.txt

Privacy estimation function

privacymeasure.py

Please refer readme.txt• Random sampling© 2020 PWS2020 Committee Cup Working Group23

Questionnaire• For all anonymizers, please answer aquestionnaire below after the
preliminary/final-anonymization phase.

‒ We will inform the attackers about theanonymization methods you used
‒ The reason is an analyst might offer whatanonymization methods are used in the
anonymized data in practicePrivacy Measures
Differential Privacy

The value 𝜖𝜖, 𝛿𝛿 :k-Anonymity

The value k :Pk-Anonymity

The value k :𝛿𝛿-Presence

The value 𝛿𝛿 :Others :Anonymization Methods
Record Deletion

Applied Record Numbers (roughly) :Top (Bottom) Coding

Applied Attribution Numbers (1 and/or 8) :Applied Record Numbers (Roughly) :Partial Records Extraction

Applied Record Numbers (roughly) :Microaggregation

Applied Attribution Numbers (1 and/or 8) :Applied Record Numbers (Roughly) :Data Exchange(Swapping)

Applied Attribution Numbers (1 to 9) :Applied Record Numbers (Roughly) :Noise Addition

Applied Attribution Numbers (1 to 9) :Applied Record Numbers (Roughly) :Synthetic Data
Generation

Applied Attribution Numbers (1 to 9) :Applied Record Numbers (Roughly) :Dummy Record Addition

Applied Record Numbers (roughly) :Randomization /Randomized Response /PRAM

Applied Attribution Numbers (1 to 9) :Applied Record Numbers (Roughly) :Others :Applied Attribution Numbers (1 to 9) :Applied Record Numbers (Roughly) :The value ⁇ :© 2020 PWS2020 Committee Cup Working Group

24