Package: mimesis-stats
mimesis_statsThis package exists to extend the capabilities ofmimesisfor use in statistical data pipelines.mimesisprovidesfastfake data generation, and comes with a wide range ofdata providers, formats, structures and localised options. In addition, it provides aschema structurewhich makes data generation for data frames very easy. Before using this package it is recommended to become familiar with the basics ofmimesisfake data generation, such as through thisgetting started page.Due to the extensibility, custom data providers can be created for use within the framework. Thismimesis_statspackage aims to use the framework for use in statistical pipelines, particularly for generating dummy data for surveys.However,mimesisdata generation / providers have two primary limitations this package extension addresses:Uni-variable - each data provider method produces a single value. Often in practice there are dependencies and relationships between different variables / columns.Limited in statistical properties -mimesisdraws samples using a uniform distribution. Real distributions are often weighted, or have specific properties (such as a gaussian)mimesis_statsuses aStatsSchemaobject that allows multiple variables related to one another to be created using methods fromMultiVariable.mimesis_statsadds data providers for: discrete choice distributions, as well as the ability to pass in custom functions, such as those fromnumpyorscipy, or user defined functions.To see an example use case of this package scroll to the bottom of this document in the "Working with pandas" section.mimesis_stats providersThe package contains two supplementary providers, the main object of generatingmimesisdata. One for producing discrete / continuous distributions and the other for dependent multi-variable samples.DistributionIdeal for generating categorical data withDistribution.discrete_distribution()or a numerical variable usingDistribution.generic_distribution()with a user defined ornumpy/scipyfunction.Allmimesis_statsproviders havenull_propandnull_valuearguments to add in missing at random null values. For multi-variable producers this is done by passing in a list of propritionas and missing values corresponding to each variable made.CategoricalGeneral use for discrete distributions, the main addition from basemimesisare the weighting and null options.>>>frommimesis_stats.providers.distributionimportDistribution>>>Distribution.distrete_distribution(...population=["First","Second","Third"],...weights=[0.01,0.01,0.98]...)"Third">>>Distribution.distrete_distribution(...population=["Apple","Banana"],...weights=[0.5,0.5],...null_prop=1.0,...null_value=None...)NoneMultiVariableThis provider allows multiple variables dependent or related to each other to be created through one provider call.In practice, produced dictionary key-value pairs can be separated into different variables.>>>frommimesis_stats.providers.multivariableimportMultiVariable>>>MultiVariable.dependent_variables(...variable_names=["consent","favourite_fruit"],...options=[("Yes","Lemon"),("No",None)],...weights=[0.7,0.3]...){"consent":"Yes","favourit_fruit":"Lemon")Within the possible combinations other provider calls can be made to extend the complexity of generation.>>>frommimesis_stats.providers.multivariableimportMultiVariable>>>frommimesisimportFood>>>MultiVariable.dependent_variables(...variable_names=["consent","favourite_fruit"],...options=[("Yes",Food.fruit()),("No",None)],...weights=[0.9,0.1]...){"consent":"Yes","favourit_fruit":"Banana")StatsSchemaFor generating samples of many variables consistently it is recommended to use a schema.mimesishas aSchemaobject, however, in order to fully take advantage of the seeding and multi-variable nature of themimesis_stats.providersapproachesStatsSchemashould be used instead to define a schema.AStatsSchemaobject requies aschemato be passed to it.Aschema/schema_blueprintis alambdafunction that contains the code to generate each variable when called.To define aschema_blueprintaStatsField(equivalent toFieldfrommimesis) needs to be declared. This sets a seed and a location basis for providers.Theschema_blueprintis then passed to theStatsSchemato define the generator.Examplemimesis_statsschema:>>>frommimesis_stats.stats_schemaimportStatsField,StatsSchema>>>fromnumpy.randomimportpareto>>>field=StatsField(seed=42)>>>schema_blueprint=lambda:{..."name":field("person.full_name"),..."salary":field("generic_distribution",func=pareto,a=3)...}>>>schema=StatsSchema(schema=schema_blueprint)>>>schema.create(iterations=1)[{'name':'Annika Reilly','salary':0.16932036645405568}]>>>schema.create(iterations=2)[{'name':'Hank Day','salary':1.7274682836709054},{'name':'Crystle Osborn','salary':0.5510238033601347}]Working with pandasStandard use of the package will be with a dataframe.The code snippets below outline the suggested approach for generating a dataframe of random data, such as a survey responses.Consider the following basic survey.We collect the following information:An ID code identifying each respondant -"ID"Their email address -"email"Their occupation -"occupation"Whether they are a parent or not -"parent"How important they think schools are when buying a house (out of 10) -"school_importance"The# fmt: off/onlines stop theblackformatter changing the schema blueprint.importpandasaspdfrommimesis_stats.stats_schemaimportStatsField,StatsSchemafromscipy.statsimporttruncnorm# Define parameters of truncated normallower=0upper=10mu_true=7mu_false=4sigma=2.5field=StatsField(seed=42)# fmt: offschema_blueprint=lambda:{"ID":field("random.custom_code",mask='SCHL#####',digit="#"),"email":field("person.email"),"occupation":field("person.occupation"),"parent_school_importance":field("dependent_variables",variable_names=["parent","school_importance"],options=[(True,round(truncnorm.rvs(a=(lower-mu_true)/sigma,b=(upper-mu_true)/sigma,loc=mu_true,scale=sigma))),(False,round(truncnorm.rvs(a=(lower-mu_false)/sigma,b=(upper-mu_false)/sigma,loc=mu_false,scale=sigma)))],weights=[0.3,0.7],)}# fmt: onschema=StatsSchema(schema_blueprint)df=pd.DataFrame(schema.create(iterations=1000))print(df.head())Output:IDemailoccupationparentschool_importance0SCHL60227pyoses1812@protonmail.comMilkladyFalse81SCHL68040dreep1871@yandex.comChoreographerTrue72SCHL25016killing1844@protonmail.comScientistFalse73SCHL52580brach1847@gmail.comLeafletDistributorFalse04SCHL86319cyrenaic1813@yandex.comYachtMasterTrue9# Check the summary stats of the two distributions# (remember mean of sample != mean of generation parameter due to truncation)parent_breakdown=df.groupby("parent").agg(["min","max","median","mean"])print(parent_breakdown)Output:school_importanceminmaxmedianmeanparentFalse01044.219477True01076.432692