Package: smart-generator
SmartGeneratorOverviewSmartGenerator is an advanced Python package for generating test data.
It allows users to generate tabular data based on a description of structure and generation rules in JSON format.
In its basics it provides generation of data based on specified distributions or other characteristics.
The key feature of this library is the ability to generate data based on user-defined templates, providing a framework for creating realistic datasets.You will find it beneficial if you need:to separate data description from the code and dynamically generate data based on the description in JSON formatto generate realistic data where the values are aligned with each other (e.g. gender, country and name values in a row are consistent)to define more complex data generation rules (e.g. JOINed tables resulting in a single flat table)to generate big datasets (e.g. millions of rows) in reasonable timeInstallationpipinstallsmart-generatorUsageCreate generator based on a descriptor as an object or as a JSON string.fromsmart_generatorimportcreate_generatorfromsmart_generator.descriptor.table_descriptorimportTableDescriptor# define table_descriptor object or load it from JSON# table_descriptor = { ... }table_generator=create_generator(table_descriptor)table_generator.generate_next_batch(8,True,False)batch=table_generator.get_batch_dataframe("NAME",True)fromsmart_generatorimportcreate_generator_from_stringtable_generator=create_generator_from_string(table_descriptor_str)DescriptorsThe library provides a set of descriptors for defining the structure of the generated data.
When you define a descriptor for data generation, it is a hierarchical tree composed of three types of descriptors:TableDescriptor,ColumnDescriptorwith multiple subtypes andSequenceDescriptor. For each descriptor you define a behaviour of how the data are generated.Example:{"name":"table","descriptors":[{"descriptor_type":"COL_INTEGER","id":"1","seed":1,"name":"column-int","visibility_type":"VISIBLE","na_prob":0.5,"behaviour":{"behaviour_type":"INCREMENT","start":10,"step":2}},{"descriptor_type":"COL_FLOAT","id":"2","seed":1,"name":"column-float","precision":2,"visibility_type":"VISIBLE","behaviour":{"behaviour_type":"NORMAL_DISTRIBUTION","mean":0,"std_dev":1}},{"descriptor_type":"COL_STRING","id":"3","seed":1,"name":"column-string","visibility_type":"VISIBLE","behaviour":{"behaviour_type":"WEIGHTS_TABLE","weights_table":[{"key":"a","value":0.05},{"key":"b","value":0.05},{"key":"c","value":0.9}]}},{"descriptor_type":"COL_DATETIME","id":"4","seed":1,"name":"column-datetime","visibility_type":"VISIBLE","precision":"MINUTE","behaviour":{"behaviour_type":"UNIFORM_DISTRIBUTION","min":"2020-01-01T00:00:00","max":"2021-01-02T00:00:00"}}]}Typically, you will define a singleTableDescriptorwith multipleColumnDescriptorobjects for basic scenarios.
But there is a possibility to define more levels to create nested structures using aSequenceDescriptor.The tree structure of descriptors looks as follows:TableDescriptor
├── ColumnDescriptor
├── ... (multiple ColumnDescriptors)
└── SequenceDescriptor
    ├── ColumnDescriptor
    ├── ... (multiple ColumnDescriptors)
    └── SequenceDescriptor
        ├── ...TableDescriptorTheTableDescriptoris the root of the descriptor tree. It defines the structure of the generated data and contains a list ofColumnDescriptorobjects.PropertyDescriptionnameName of the tableidUnique identifier of the tableseedSeed for generating values. Default value is 0.descriptorsList ofColumnDescriptorandSequenceDescritorobjectsColumnDescriptorsTheColumnDescriptordefines the structure of a single column in the generated data.There are 6 types ofColumnDescriptorobjects:ColumnDescriptorIntegerColumnDescriptorFloatColumnDescriptorStringColumnDescriptorDatetimeColumnDescriptorDateColumnDescriptorTimeCommon properties for all of them:PropertyDescriptionnameName of the columnidUnique identifier of the columnseedSeed for generating values. Default value is 1.visibility_typeVisibility type of the column with possible valuesVISIBLE,HIDDENna_probProbability of generating NA values. Default value is 0.behaviorBehavior of the column.ColumnDescriptorFloatadditionally defines:PropertyDescriptionprecisionNumber of decimal places of the generated numbersColumnDescriptorDatetime,ColumnDescriptorDateandColumnDescriptorTimeadditionally define:PropertyDescriptionprecisionPrecision of the column with possible valuesYEAR,MONTH,WEEK,DAY,HOUR,MINUTE,SECOND,MILLISECONDSequenceDescriptorTheSequenceDescriptordefines a sequence or subtable of the generated data. It contains a list ofColumnDescriptorobjects andSequenceDescriptorobjects.PropertyDescriptionnameName of the sequenceidUnique identifier of the sequenceseedSeed for generating values. Default value is 1.descriptorsList ofColumnDescriptorandSequenceDescritorobjectsbehaviorBehavior of the sequence.BehaviorsThebehaviorproperty ofColumnDescriptorandSequenceDescriptordefines the behavior of the generation process of the column or sequence.IncrementTheIncrementbehavior generates values based on the specified start value and step.Compatible withColumnDescriptorInteger,ColumnDescriptorFloat,ColumnDescriptorDatetime,ColumnDescriptorDateandColumnDescriptorTime.PropertyDescriptionstartStart value of the generated values. It is a number in case ofIntegerorFloatcolumns. It is a corresponding type in case ofDatetime,DateorTimecolumns.stepStep for generating next value. ForDatetime,DateandTimecolumns it is a number representing milliseconds.UniqueTheUniquebehavior generates unique values within a specified range.Compatible withColumnDescriptorInteger,ColumnDescriptorFloat.PropertyDescriptionminMinimum value of the range to generate from.maxMaximum value of the range to generate from.Uniform distributionTheUniformDistributionbehavior generates values based on the uniform distribution.Compatible withColumnDescriptorInteger,ColumnDescriptorFloat,ColumnDescriptorDatetime,ColumnDescriptorDateandColumnDescriptorTime.PropertyDescriptionminMinimum value of the range to generate from. It is a number in case ofIntegerorFloatcolumns. It is a corresponding type in case ofDatetime,DateorTimecolumns.maxMaximum value of the range to generate from. It is a number in case ofIntegerorFloatcolumns. It is a corresponding type in case ofDatetime,DateorTimecolumns.Normal distributionTheNormalDistributionbehavior generates values based on the normal distribution.Compatible withColumnDescriptorInteger,ColumnDescriptorFloat.PropertyDescriptionmeanMean parameter of the distribution.std_devStandard deviation of the distribution.Exponential distributionTheExponentialDistributionbehavior generates values based on the exponential distribution.Compatible withColumnDescriptorInteger,ColumnDescriptorFloat.PropertyDescriptionscaleScale parameter of the distribution. It is the same as reverse of lambda parameter.Weights tableTheWeightsTablebehavior generates values based on the weights table.Compatible withColumnDescriptorString,ColumnDescriptorInteger,ColumnDescriptorFloat,ColumnDescriptorDatetime,ColumnDescriptorDateandColumnDescriptorTime.PropertyDescriptionweights_tableList ofEntryobjects, such as{"key": some_key, "value": number}. The key is a value to generate and the value is a weight of generating the value. The sum of all weights does not need to be 1. Key is a corresponding type based on the column type.Template labelTheTemplateLabelbehavior generates labels based on the template.Compatible withColumnDescriptorString.PropertyDescriptiontemplateName of the template for generating labels.template_filtersDictionary representing filters for the template.Template Geo LocationTheTemplateGeoLocationbehavior generates geo locations based on the template.Compatible withColumnDescriptorIntegerandColumnDescriptorFloat.PropertyDescriptiontemplateName of the template for generating geo locations.coordinate_typeType of the coordinate to generate. Possible values areLATITUDE_WGS84andLONGITUDE_WGS84.template_filtersDictionary representing filters for the template.Template TimestampTheTemplateTimestampbehavior generates timestamps based on the template. This behavior is useful for generating random timestamps from a specified range and the template determines the probability of generating a timestamp.Compatible withColumnDescriptorDatetime,ColumnDescriptorDateandColumnDescriptorTime.PropertyDescriptiontemplateName of the template for generating timestamps.startStart value of the range to generate from.endEnd value of the range to generate from.template_filtersDictionary representing filters for the template.Template TimeseriesTheTemplateTimeseriesbehavior generates timeseries based on the template. This behavior is useful for generating timeseries values. When a column with this behaviour is used, it also requires a columnColumnDescriptorDatetime,ColumnDescriptorDateorColumnDescriptorTimein the same table/sequence. It automatically pairs with the first column of such a type in the table/sequence.Compatible withColumnDescriptorIntegerandColumnDescriptorFloat.PropertyDescriptiontemplateName of the template for generating timeseries.template_filtersDictionary representing filters for the template.Template ProvidersTemplate providers are an extension of the library that allows users to define their own templates for generating data.Built-in Faker providerThe library provides a built-in template provider based on theFakerlibrary, so if you install Faker, you can use all the templates provided by Faker.pip install FakerSet the template provider:fromsmart_generatorimportadd_templates_providerfromsmart_generator.templates.samples.sample_fakerimportproviderasfaker_provideradd_templates_provider(faker_provider)Example of an input descriptor using Faker to generate names:{"name":"table","descriptors":[{"descriptor_type":"COL_STRING","id":"1","seed":1,"name":"column-string","visibility_type":"VISIBLE","behaviour":{"behaviour_type":"TEMPLATE_LABEL","template":"FIRST_NAME"}}]}Adding custom templatesTo add your custom templates, you can you eitherTemplatesProviderFromDataframeorTemplatesProviderFromSqldepending on your source of data templates.Label templatesAs an example, consider these 4 tables:gender - with columnsgender_id,labelandweightcountry - with columnscountry_id,labelandpopulationcity - with columnscity_id,country_id,labelandpopulationname - with columnsgender_id,country_id,labelandweightYou can then setup the template provider in which you define the tables and their relationships:fromsmart_generator.templates.templates_provider_sqlimportTemplatesProviderFromSqlfromsmart_generator.templates.template_tableimportTemplateTableconnection_string='connection_string_to_your_database'provider=TemplatesProviderFromSql(connection_string)provider.add_table(TemplateTable("gender","gender",id_column="gender_id"))provider.add_table(TemplateTable("country","country",id_column="country_id",weight_column="population"))provider.add_table(TemplateTable("city","city",id_column="city_id",weight_column="population",dependency_templates=["country"]))provider.add_table(TemplateTable("name","name",id_column=None,dependency_templates=["gender","country"]))Geo location templatesAs an example, consider a table with geo locations:geo_location - with columnslongitude,latitudeandpopulationfromsmart_generator.templates.template_tableimportTemplateGeoLocationTableprovider.add_table(TemplateGeoLocationTable("population","geo_location","longitude","latitude",weight_column="population",dependency_templates=["country"]))Timestamp templatesAs an example, consider a table with timestamps:events_telco - with columnstimestamp,sms,callfromsmart_generator.templates.template_tableimportTemplateTimestampTable,TimeseriesUnitprovider.add_table(TemplateTimestampTable("telco_call","events_telco",TimeseriesUnit.YEAR,TimeseriesUnit.HOUR,weight_column="calls"))provider.add_table(TemplateTimestampTable("telco_sms","events_telco",TimeseriesUnit.YEAR,TimeseriesUnit.HOUR,weight_column="sms"))Thetimestampcolumn is not in a standard format, but rather an encoded timestamp. We provide an encoding function used to encode timestamps to a generic number representing a certain point of time in the timeseries. It takes into consideration day of week, so this way we can encode, for example, weekend or Monday morning data regardless of the year. This help to generate timestamps and timeseries from templates in an arbitrary time frame.fromsmart_generator.helpers.timeseries_codingimportencode_timestamp,TimeseriesUnitfromdatetimeimportdatetimeprint(encode_timestamp(datetime(2024,1,1),TimeseriesUnit.YEAR,TimeseriesUnit.HOUR))# returns 0 - this day is Monday and it is also aligned with the start of the yearprint(encode_timestamp(datetime(2020,1,1),TimeseriesUnit.YEAR,TimeseriesUnit.HOUR))# returns 48 - this day is WednesdayTimeseries templatesThis is similar to the timestamp templates, but it is used for generating timeseries values.The same table as in the timestamp templates example can be reused.Products using SmartGeneratorData4Test- a free online tool for generating test data based on SmartGenerator.