Package: py-synthpop
synthpopPython implementation of theR package synthpop.The R implementation of synthpop is a tool for producing synthetic versions of microdata containing confidential information so that they are safe to be released to users for exploratory analysis. The key objective of generating synthetic data is to replace sensitive original values with synthetic ones causing minimal distortion of the statistical information contained in the dataset. Variables, which can be categorical or continuous, are synthesised one-by-one using sequential modelling. Replacements are generated by drawing from conditional distributions fitted to the original data using parametric or classification and regression trees models.This is a reimplementation in Python which allows synthetic data to be generated via the method.generate()after the algorithm had been fit to the original data via the method.fit(). The process can be largely automated, if default settings are used, or with methods defined by the user. Optional parameters can be used to influence the disclosure risk and the analytical quality of the synthetic data.Development status and roadmapThis project is in Alpha status and the roadmap can be foundhere.Installationpippipinstallpy-synthpopSourcegitclonegit@github.com:hazy/synthpop.gitcdsynthpop
pipinstall-rrequirements.txt
pythonsetup.pyinstallExamplesAdult datasetWe will use theUS adult census dataset, which is a freely available open dataset extracted from the US census bureau database. The dataset is initially designed for a binary classification problem and the task is to predict whether a person earns over $50,000 a year. The dataset is a mixture of discrete and continuous features, including age, working status (workclass), education, marital status, race, sex, relationship and hours worked per week.In[1]:fromdatasets.adultimportdfIn[2]:df.head()Out[2]:ageworkclassfnlwgteducationeducational-nummarital-statusoccupationrelationshipracegendercapital-gaincapital-losshours-per-weeknative-countryincome039State-gov77516Bachelors13Never-marriedAdm-clericalNot-in-familyWhiteMale2174040United-States<=50K150Self-emp-not-inc83311Bachelors13Married-civ-spouseExec-managerialHusbandWhiteMale0013United-States<=50K238Private215646HS-grad9DivorcedHandlers-cleanersNot-in-familyWhiteMale0040United-States<=50K353Private23472111th7Married-civ-spouseHandlers-cleanersHusbandBlackMale0040United-States<=50K428Private338409Bachelors13Married-civ-spouseProf-specialtyWifeBlackFemale0040Cuba<=50KsynthpopUse default parameters for the Adult dataset:In[1]:fromsynthpopimportSynthpopIn[2]:fromdatasets.adultimportdf,dtypesIn[3]:spop=Synthpop()In[4]:spop.fit(df,dtypes)train_agetrain_workclasstrain_fnlwgttrain_educationtrain_educational-numtrain_marital-statustrain_occupationtrain_relationshiptrain_racetrain_gendertrain_capital-gaintrain_capital-losstrain_hours-per-weektrain_native-countrytrain_incomeIn[5]:synth_df=spop.generate(len(df))generate_agegenerate_workclassgenerate_fnlwgtgenerate_educationgenerate_educational-numgenerate_marital-statusgenerate_occupationgenerate_relationshipgenerate_racegenerate_gendergenerate_capital-gaingenerate_capital-lossgenerate_hours-per-weekgenerate_native-countrygenerate_incomeIn[6]:synth_df.head()Out[6]:ageworkclassfnlwgteducationeducational-nummarital-statusoccupationrelationshipracegendercapital-gaincapital-losshours-per-weeknative-countryincome021?21305511th7Never-married?Not-in-familyOtherFemale0030United-States<=50K123Private150683HS-grad9Never-marriedAdm-clericalNot-in-familyWhiteFemale0040United-States<=50K261Private19141710th6WidowedSalesNot-in-familyBlackFemale0032United-States<=50K350Private190762HS-grad9DivorcedSalesNot-in-familyWhiteMale0060United-States<=50K442Local-gov255675HS-grad9Married-civ-spouseOther-serviceHusbandBlackMale0040United-States<=50KIn[7]:spop.methodOut[7]:agesampleworkclasscartfnlwgtcarteducationcarteducational-numcartmarital-statuscartoccupationcartrelationshipcartracecartgendercartcapital-gaincartcapital-losscarthours-per-weekcartnative-countrycartincomecartdtype:objectIn[8]:spop.visit_sequenceOut[8]:age0workclass1fnlwgt2education3educational-num4marital-status5occupation6relationship7race8gender9capital-gain10capital-loss11hours-per-week12native-country13income14dtype:int64In[9]:spop.predictor_matrixOut[9]:ageworkclassfnlwgteducationeducational-nummarital-statusoccupationrelationshipracegendercapital-gaincapital-losshours-per-weeknative-countryincomeage000000000000000workclass100000000000000fnlwgt110000000000000education111000000000000educational-num111100000000000marital-status111110000000000occupation111111000000000relationship111111100000000race111111110000000gender111111111000000capital-gain111111111100000capital-loss111111111110000hours-per-week111111111111000native-country111111111111100income111111111111110Define the visit sequence for the Adult dataset:In[1]:fromsynthpopimportSynthpopIn[2]:fromdatasets.adultimportdf,dtypesIn[3]:spop=Synthpop(visit_sequence=[0,1,5,3,2])In[4]:spop.fit(df,dtypes)train_agetrain_workclasstrain_marital-statustrain_educationtrain_fnlwgtIn[5]:synth_df=spop.generate(len(df))generate_agegenerate_workclassgenerate_marital-statusgenerate_educationgenerate_fnlwgtIn[6]:synth_df.head()Out[6]:ageworkclassfnlwgteducationmarital-status057Self-emp-not-inc327901Prof-schoolMarried-civ-spouse124Private34568Assoc-vocNever-married250Private256861HS-gradMarried-civ-spouse328Private186239Some-collegeNever-married438Private216129BachelorsDivorcedIn[7]:spop.methodOut[7]:agesampleworkclasscartfnlwgtcarteducationcarteducational-numcartmarital-statuscartoccupationcartrelationshipcartracecartgendercartcapital-gaincartcapital-losscarthours-per-weekcartnative-countrycartincomecartdtype:objectIn[8]:spop.visit_sequenceOut[8]:age0workclass1fnlwgt4education3marital-status2dtype:int64In[9]:spop.predictor_matrixOut[9]:ageworkclassfnlwgteducationmarital-statusage00000workclass10000fnlwgt11011education11001marital-status11000LicenseThis project is being developed atHazy Limitedand is released underMIT license.