Package: fakermaker
Faker-MakerThis module creates pandas dataframes filled with fake data from theFakerpackage using an IPython magic function with custom domain specific language.Installationpip install fakermakerExample usageimportfakermakerLoad the magic function%load_extfakermakerIn a separate cell, use the magic funcionfakermakeralong with a cell filled with only the "faker-maker language".%%fakedataseed=0,lang=jp_JP# This will create 3 pandas dataframes: persons, purchases and commentspersons{10}-----------------------first_name#This is the Faker.first_name function which generates fake nameslast_name*#Every lastname in this dataset will be uniquephone_numberrandom_number(digits=5)ascustomer_number[1]*#The customer number in this dataset will be reused in purchasespurchases---------isbn10credit_card_fullrandom_number(digits=3)aspricerandom_number(digits=5)ascustomer_number[1,2]# this reference says it's a child of persons and parent of commentscomments{2000}# This dataset will have 2000 rows (default = 99)---------text(max_nb_chars=280)ascommentrandom_number(digits=5)ascustomer_number[2]See alsoexample_usage.ipynbFaker-Maker Language#- Everything after#will be considered a comment and thus ignored.optionalname- Name of the Pandas Dataframe to createrequired{n}- Wherenis the number of rows in the Dataframe.  [Default=99]optional#- Everything after#will be considered a comment and thus ignored.optional--- Header/Details divider.  Two or more-requiredcolumn_name- Name of the column to add to the Pandas Dataframe. One field per line.  At least one required.  Name must match a validFakerfunction.  SeeFaker Documentationfor more details.  Column name will be the same as the function name unless you use theaskeyword.required(params)- Additional Parameters to pass theFakerfunction.  Requires key-value pairs. Read more below.optional*- Requires each row in the dataframe to have a unique value for this column.  Read more below.optionalasalias- Column name override.optional[i,j,...]- Define references (i.e. parent/child relationship) between multiple Pandas Dataframes.  Read more below.optional#- Everything after#will be considered a comment and thus ignored.optional\n- An additional new-line is required between dataframes.  Not required for last.requiredInitial version created by Prof. Christopher Brooks (brooksch@umich.edu) and improved by package author Nicholas Miller.CommentsThe domain specific language invented for this problem was expanded to support comments similar to python using a#.  Comments are not supported throughout.  Examples where comments can be added:After a table name (and optional row count)After a field defintionBefore a field defintion (to comment out the field)Unique ValuesThe "unique" constraint is a bit ambiguous because let's say you have 2 columns (fields) that are defined as being unique within a dataset like in the example that was provided.  This could be interpreted in two ways:The combination of the 2 values must be unique within the datasetEvery value in the column must be uniqueFor this implementation, I found that the latter is more flexible in that it satisifies both of these points.  However, this comes with caveats that should be noted. The most obvious is a limitation of values.  For example, if you specify a column to contain a random number of length 2 and also unique, it is impossible to have a dataset with over 100 unique entries.  In fact, in how this is implemented, it is likely it would throw an exception much earlier due to how we limit the number times we look for a unique value.  Since we're using Faker to generate unique values, it should be kept in mind that there will be some limit to the number of unique values for each function.  For example,last_namemight only generate 100 unique last names.  In my tests, I was getting duplicate values quite early on so its possible a unique set of last_names may randomly throw an error for a set of 99 unique last names.  It is recommeneded to read the Faker documentation for more information on what the limitations of each function are if you start getting errors about 'Maxed out unique checks'.Future enhancement: Create a solution to #1 above.ParametersAll parameters must define the varable they are referencing in the parameter list.  This is necessary for how the parameter dictionary structure is built.  It's also good practice because it makes the parameter purpose much clearer.  For example:random_number(digits=5)ascustomer_numberReferences (i.e. Parent/Child Relationship)If tables are to have parent-child relationships and linked by a reference, then the parent defintion should be provided first and the children after.  See the example.This allows for a single refernce or a comma delimited list of references and is demonstrated below.  The purpose is in the case that you want to drill down into a narrower and narrower set of values.  In the example below, not all persons will have purchases and not all purchases will have comments.  But, all comments should have one or more purchases and all purchases should have one person (since customer_number is unique on persons).  Here's how this would look:persons-------random_number(digits=5)ascustomer_number[1]*<otherfields...>purchases-------random_number(digits=5)ascustomer_number[1,2]<otherfields...>comments-------random_number(digits=5)ascustomer_number[2]<otherfields...>NOTE: Thefunction_nameis not actually used on child tables since the values are actually retrieved from the parent table.  In the above example,purchasesandcommentsall get their values frompersonsso therandom_number(digits=5)is actually not used or validated against.Repeat Data (Goal = 25% Duplicates)This touches on the same vain as unique since we're doing some manual tricks to try to meet this 25% goal.  The solution I chose for this is to generate 75% of the values (note: this is controlled by the fieldFakeDataFrameBuilder.repeat_threshold) from Faker.  It then analyzes the uniqueness of the results.  If 25% or more duplicates already result, we just hit up the Faker for the remaining 25% (i.e. tough luck, go look at the other guy).  If it's >0% but <25%, then we salt it by hitting up the Faker again.For example, if the uniqueness value comes back as 20% duplicates in the first 75% generated, it's pretty safe to say that the Faker is already doing a good job at getting close to our 25% limit.  So we'll get 20% more values from the faker.For the remaining percent (or 25% if there were no unique values), we randomly sample from within our generated list.  This will obviously result in duplicates or possibly even triplicates or more.SeedingI wanted a method to regenerate the same dataset so I implemented aseed()function.  This generally works but doesn't seem to work well with my random_number() generator for customer_number yet.  Still need to figure that one out...Issue -seed()still generates random customer_numbersThe seed can now be set on the magic function call line using the syntaxseed=0.LanguageThe language can also be defined on the magic function line using the syntaxlang=jp_JP.