Package: mimeograph
Mimeo (Mimeograph)Mimeois a command line tool and a python library generating NoSQL data based on a template.
It can be used by developers, testers or business analysts in their daily work. Its main advantage over other generators
is that it can build data with nested nodes at any level (as in real data).InstallationInstall Mimeo with pippipinstallmimeographUsage/ExamplesMimeo ConfigurationPrepare Mimeo Configuration firstJSONXML{"_templates_":[{"count":30,"model":{"SomeEntity":{"@xmlns":"http://mimeo.arch.com/default-namespace","@xmlns:pn":"http://mimeo.arch.com/prefixed-namespace","ChildNode1":1,"ChildNode2":"value-2","ChildNode3":true}}}]}<mimeo_configuration><_templates_><_template_><count>30</count><model><SomeEntityxmlns="http://mimeo.arch.com/default-namespace"xmlns:pn="http://mimeo.arch.com/prefixed-namespace"><ChildNode1>1</ChildNode1><pn:ChildNode2>value-2</pn:ChildNode2><ChildNode3>true</ChildNode3></SomeEntity></model></_template_></_templates_></mimeo_configuration>You can find more configuration examples in theexamplesfolder.Data generationThe Mimeo Configuration above will produce 2 files:<!-- mimeo-output/mimeo-output-1.xml--><SomeEntityxmlns="http://mimeo.arch.com/default-namespace"xmlns:pn="http://mimeo.arch.com/prefixed-namespace"><ChildNode1>1</ChildNode1><pn:ChildNode2>value-2</pn:ChildNode2><ChildNode3>true</ChildNode3></SomeEntity><!-- mimeo-output/mimeo-output-2.xml--><SomeEntityxmlns="http://mimeo.arch.com/default-namespace"xmlns:pn="http://mimeo.arch.com/prefixed-namespace"><ChildNode1>1</ChildNode1><pn:ChildNode2>value-2</pn:ChildNode2><ChildNode3>true</ChildNode3></SomeEntity>When we would configure output format asjsonthen it would produce JSON nodes:{"SomeEntity":{"@xmlns":"http://mimeo.arch.com/default-namespace","@xmlns:pn":"http://mimeo.arch.com/prefixed-namespace","ChildNode1":1,"pn:ChildNode2":"value-2","ChildNode3":true}}{"SomeEntity":{"@xmlns":"http://mimeo.arch.com/default-namespace","@xmlns:pn":"http://mimeo.arch.com/prefixed-namespace","ChildNode1":1,"pn:ChildNode2":"value-2","ChildNode3":true}}mimeoSomeEntity-config.json
mimeoSomeEntity-config.xmlMimeo UtilsMimeo exposes several functions for data generation that will make it more useful for testing purposes.
To see all Mimeo Utils, go to the documentation below.Template{"count":2,"model":{"SomeEntity":{"id":"{auto_increment}","randomstring":"{random_str}","randomint":"{random_int}"}}}<_template_><count>2</count><model><SomeEntity><id>{auto_increment}</id><randomstring>{random_str}</randomstring><randomint>{random_int}</randomint></SomeEntity></model></_template_>XML Data<SomeEntity><id>00001</id><randomstring>mCApsYZprayYkmKnYWxe</randomstring><randomint>8</randomint></SomeEntity><SomeEntity><id>00002</id><randomstring>ceaPUqARUkFukZIPeuqO</randomstring><randomint>99</randomint></SomeEntity>JSON Data{"SomeEntity":{"id":"00001","randomstring":"mCApsYZprayYkmKnYWxe","randomint":8}}{"SomeEntity":{"id":"00002","randomstring":"ceaPUqARUkFukZIPeuqO","randomint":99}}DocumentationMimeo CLIMimeo Configuration argumentsWhen using Mimeo command line tool you can overwrite Mimeo Configuration properties:Short optionLong optionDescription-F--formatoverwrite theoutput/formatproperty-o--outputoverwrite theoutput/directionproperty-x--xml-declarationoverwrite theoutput/xml_declarationproperty-i--indentoverwrite theoutput/indentproperty-d--directoryoverwrite theoutput/directory_pathproperty-f--fileoverwrite theoutput/file_nameproperty-H--http-hostoverwrite theoutput/hostproperty-p--http-portoverwrite theoutput/portproperty-E--http-endpointoverwrite theoutput/endpointproperty-U--http-useroverwrite theoutput/usernameproperty-P--http-passwordoverwrite theoutput/passwordproperty--http-methodoverwrite theoutput/methodproperty--http-protocoloverwrite theoutput/protocolproperty-e--http-envoverwrite the output http properties using a mimeo env configuration--http-envs-fileuse a custom environments file (by default: mimeo.envs.json)--rawsame as-o stdoutoverwrite theoutput/directionproperty tostdoutLogging argumentsShort optionLong optionDescription--silentdisable INFO logs--debugenable DEBUG mode--fineenable FINE modeOther argumentsShort optionLong optionDescription--sequentiallyprocess Mimeo Configurations in a single threadMimeo ConfigurationMimeo configuration is defined in a JSON file using internal settings and data templates.KeyLevelRequiredSupported valuesDefaultDescriptionoutputConfig☐object---Defines output details on how it will be consumedoutput/directionConfig☐file,stdout,httpfileDefines how output will be consumedoutput/formatConfig☐xml,jsonxmlDefines output data formatoutput/indentConfig☐integernullDefines indent applied in output dataoutput/xml_declarationConfig☐booleanfalseIndicates whether an xml declaration should be added to output dataoutput/directory_pathConfig☐stringmimeo-outputForfiledirection - defines an output directoryoutput/file_nameConfig☐stringmimeo-outputForfiledirection - defines an output file nameoutput/methodConfig☐POST,PUTPOSTForhttpdirection - defines a request methodoutput/protocolConfig☐http,httpshttpForhttpdirection - defines a url protocoloutput/hostConfig☑string---Forhttpdirection - defines a url hostoutput/portConfig☐integernullForhttpdirection - defines a url port (can be empty)output/endpointConfig☑string---Forhttpdirection - defines a url endpointoutput/usernameConfig☑string---Forhttpdirection - defines a usernameoutput/passwordConfig☑string---Forhttpdirection - defines a passwordvarsConfig☐object---Defines variables to be used in a Mimeo Template (read more below)refsConfig☐object---Defines references to be used in a Mimeo Template (read more below)_templates_Config☑array---Stores templates for data generationcountTemplate☑integer---Indicates number of copiesmodelTemplate☑object---Defines data template to be copiedcontextModel☐object---Defines a context name that is internally used e.g. usingcurr_iter()andget_key()mimeo utils (by default model name is used as the context name)Mimeo EnvironmentTo makehttpoutput directory easier to use, mimeo allows you to configure Mimeo Environments.
They are configured in a JSON file (by default: mimeo.envs.json) and support the following output details:protocolhostportusernamepasswordExample{"local":{"host":"localhost","port":8000,"username":"admin","password":"admin"},"dev":{"protocol":"https","host":"11.111.11.111","port":8000,"username":"some-user","password":"some-password"}}To use a specific Mimeo Environment you can use the following commands:mimeoSomeEntity-config.json-edev
mimeoSomeEntity-config.json-edev--http-envs-fileenvironments.jsonMimeo VarsMimeo allows you to define a list of variables.
You can use them in your Mimeo Config by wrapping them in curly brackets [{VARIABLE}].There are only 2 rules for variable names:Variable name can include upper-cased letters [A-Z], underscore [_] and digits {0-9} onlyVariable name must start with a letterVariable can be defined with:any atomic valueany other variable definedany Mimeo UtilYou can use Mimeo Vars as partial values (unless they are defined as Mimeo Utils).Example:{"vars":{"CUSTOM_VAR_1":"custom-value-1","CUSTOM_VAR_2":1,"CUSTOM_VAR_3":true,"CUSTOM_VAR_4":"{CUSTOM_VAR_2}","CUSTOM_VAR_5":"{auto_increment}","CUSTOM_VAR_6":{"_mimeo_util":{"_name":"random_int","limit":99}}},"_templates_":[{"count":5,"model":{"SomeEntity":{"ChildNode1":"{CUSTOM_VAR_1}","ChildNode2":"{CUSTOM_VAR_2}","ChildNode3":"{CUSTOM_VAR_3}","ChildNode4":"{CUSTOM_VAR_4}","ChildNode5":"{CUSTOM_VAR_5}","ChildNode6":"{CUSTOM_VAR_6}","ChildNode7":"{CUSTOM_VAR_1}-with-suffix"}}}]}Mimeo Special FieldsIn Mimeo Template you can use so-calledspecial fields.
Every field in a template can be stored in memory (provided) and used later as a value of other fields (injected)
in context of a single iteration.
To provide a special field, wrap its name with colons: [:SomeField:]. To inject, use additionally curly braces to
let interpreter know it should be rendered [{:SomeField:}].
They can be injected as partial values, similarly to Mimeo Vars.Example{"_templates_":[{"count":5,"model":{"SomeEntity":{":ChildNode1:":"custom-value","ChildNode2":"{:ChildNode1:}","ChildNode3":"{:ChildNode1:}-with-suffix"}}}]}Mimeo RefsMimeo Special Fields are useful when an entity has the same value used in several fields.
However, usually entities are related with each other. To use references between entities, you can use
Mimeo Refs. They are configured at the highest Mimeo Configuration level and require 3 settings:context - a context from which values will be cachedfield - a source field of the referencetypeany- reference of this type will be used randomlyno orderpossible duplicates (One-To-Many, Many-To-Many)parallel- reference of this type will generate a reference from the same iteration in references entitysame order as in parent entityunique values (One-To-One)To use them in a Mimeo Template, simply wrap a reference name with curly braces [{some-reference}].Note:A reference can't be configured using any of Mimeo Utils' names nor existing Mimeo VarsA referenced entity needs to be placed before a referencing oneExample{"refs":{"parent-one-to-many":{"context":"SomeEntity","field":"ID","type":"any"},"parent-one-to-one":{"context":"SomeEntity","field":"ID","type":"parallel"}},"_templates_":[{"count":5,"model":{"SomeEntity":{"ID":"{key}"}}},{"count":5,"model":{"OneToOneChildEntity":{"Parent":"{parent-one-to-one}"}}},{"count":10,"model":{"ManyToOneChildEntity":{"Parent":"{parent-one-to-many}"}}}]}Mimeo UtilsYou can use several predefined functions to generate data. They can be used in arawformat orparametrized.Random StringGenerates a random string value.ParameterSupported valuesDefaultlengthint20RawUses the default length: 20 characters.{"randomstring":"{random_str}"}ParametrizedUses the customized length.{"randomstring":{"_mimeo_util":{"_name":"random_str","length":5}}}Random IntegerGenerates a random integer value betweenstartandlimitparameters (inclusive).ParameterSupported valuesDefaultstartint1limitint100RawUses the default start (1) and limit (100) values.{"randominteger":"{random_int}"}ParametrizedUses the customized limit.{"randominteger1":{"_mimeo_util":{"_name":"random_int","start":0}},"randominteger2":{"_mimeo_util":{"_name":"random_int","limit":5}},"randominteger3":{"_mimeo_util":{"_name":"random_int","start":0,"limit":5}}}Random ItemGenerates a random value from items provided.NOTICE: The raw form of this Mimeo Util will generate a blank string value (as same as no items parametrized).ParameterSupported valuesDefaultitemslist[""]Parametrized{"random":{"_mimeo_util":{"_name":"random_item","items":["value",1,true]}}}DateGenerates a date value in formatYYYY-MM-DD.ParameterSupported valuesDefaultdays_deltaint0RawUses the today's date.{"Today":"{date}"}ParametrizedUses the customized days delta.{"Yesterday":{"_mimeo_util":{"_name":"date","days_delta":-1}},"Tomorrow":{"_mimeo_util":{"_name":"date","days_delta":1}}}Date TimeGenerates a date time value in formatYYYY-MM-DD'T'HH:mm:SS.ParameterSupported valuesDefaultdays_deltaint0hours_deltaint0minutes_deltaint0seconds_deltaint0RawUses the current timestamp.{"Now":"{date_time}"}ParametrizedUses the customized deltas.{"TomorrowThreeHoursLaterTwentyMinutesAgoTwoSecondsLater":{"_mimeo_util":{"_name":"date_time","days_delta":1,"hours_delta":3,"minutes_delta":-20,"seconds_delta":2}}}Auto IncrementGenerates a next integer in context of a model (in nested templates it will use a separated context).ParameterSupported valuesDefaultpatternstr{:05d}RawUses a default pattern:{:05d}(an integer with 5 leading zeros).{"ID":"{auto_increment}"}ParametrizedUses the string pattern provided.{"ID":{"_mimeo_util":{"_name":"auto_increment","pattern":"MY_ID_{:010d}"}}}Current IterationGenerates a value of the current iteration in a Mimeo Template context.ParameterSupported valuesDefaultcontextstra current contextRawUses the current context.{"ID":"{curr_iter}"}ParametrizedUses a specific Mimeo Model context (model name whencontextis not configured).{"Parent":{"_mimeo_util":{"_name":"curr_iter","context":"SomeEntity"}}}KeyGenerates a key unique across all Mimeo Models and being the same within a single Mimeo Model context.ParameterSupported valuesDefaultcontextstra current contextiterationinta current iteration of the contextRawUses a key from the current context and iteration.{"ID":"{key}"}ParametrizedUses a key from the specific context and iteration.When context is indicated and iteration is not, then the current iterationof the indicated contextis being used.{"SomeEntity2":{"_mimeo_util":{"_name":"key","context":"SomeEntity","iteration":"{curr_iter}"}}}CityGenerates a city name.ParameterSupported valuesDefaultuniqueboolTruecountrystrNoneRawBy default city names will be unique across a Mimeo Context.{"City":"{city}"}ParametrizedUses country (name, iso2, iso3) anduniqueflag to generate a city name.{"CityWithDuplicates":{"_mimeo_util":{"_name":"city","unique":false}},"CityOfCountryName":{"_mimeo_util":{"_name":"city","country":"United Kingdom"}},"CityOfCountryISO2":{"_mimeo_util":{"_name":"city","country":"GB"}},"CityOfCountryISO3":{"_mimeo_util":{"_name":"city","country":"GBR"}},"CityOfCountryWithDuplicates":{"_mimeo_util":{"_name":"city","country":"United Kingdom","unique":false}}}CountryGenerates a country name (by default), iso2 or iso3.ParameterSupported valuesDefaultuniqueboolTruevalue"name","iso3","iso2""name"countrystrNoneRawBy default country names will be unique across a Mimeo Context.{"Country":"{country}"}ParametrizedIt can generate:country iso3 or iso 2 instead of namecountry with duplicatescountry name for a provided iso3 or iso2country iso2 for a provided name or iso3country iso3 for a provided name or iso2When thecountryparam is provided then theuniqueflag is ignored.{"CountryNameWithDuplicates":{"_mimeo_util":{"_name":"country","unique":false}},"CountryISO2":{"_mimeo_util":{"_name":"country","value":"iso2"}},"CountryISO3":{"_mimeo_util":{"_name":"country","value":"iso3"}},"CountryNameForISO3":{"_mimeo_util":{"_name":"country","country":"GBR"}},"CountryISO2ForName":{"_mimeo_util":{"_name":"country","value":"iso2","country":"United Kingdom"}}}CurrencyGenerates a currency code (by default) or name.ParameterSupported valuesDefaultuniqueboolFalsevalue"code","name""code"countrystrNoneRawBy default city names willNOTbe unique across a Mimeo Context.{"Currency":"{currency}"}ParametrizedIt can generate:unique currenciescurrency name instead of codecurrency code or name of a specific country (using iso3, iso2 or name)When thecountryparam is provided then theuniqueflag is ignored.{"UniqueCurrencyCode":{"_mimeo_util":{"_name":"currency","unique":true}},"CurrencyName":{"_mimeo_util":{"_name":"currency","value":"name"}},"CurrencyCodeForCountryISO3":{"_mimeo_util":{"_name":"currency","country":"GBR"}},"CurrencyNameForCountryISO2":{"_mimeo_util":{"_name":"currency","value":"name","country":"GB"}},"CurrencyNameForCountryName":{"_mimeo_util":{"_name":"currency","value":"name","country":"United Kingdom"}}}First NameGenerates a first name.ParameterSupported valuesDefaultuniqueboolTruesexM,Male,F,FemaleNoneRawBy default first names will be unique across a Mimeo Context.{"FirstName":"{first_name}"}ParametrizedUses sex (M/Male/F/Female) anduniqueflag to generate a first name.{"FirstNameWithDuplicates":{"_mimeo_util":{"_name":"first_name","unique":false}},"MaleFirstName":{"_mimeo_util":{"_name":"first_name","sex":"M"}},"FemaleFirstName":{"_mimeo_util":{"_name":"first_name","sex":"F"}},"MaleFirstNameWithDuplicates":{"_mimeo_util":{"_name":"first_name","sex":"M","unique":false}}}Last NameGenerates a last name.ParameterSupported valuesDefaultuniqueboolTrueRawBy default last names will be unique across a Mimeo Context.{"LastName":"{last_name}"}ParametrizedUsesuniqueflag to generate a last name.{"LastNameWithDuplicates":{"_mimeo_util":{"_name":"last_name","unique":false}}}Python LibTo generate data using Mimeo as a python library you need 3 classes:MimeoConfig(A python representation of a Mimeo Configuration)MimeoConfigFactory(A factory parsing a Mimeo Configuration)Mimeograph(a class generating and consuming data from a Mimeo Configuration)Parsing Mimeo ConfigurationMimeoConfigTheMimeoConfigclass takes a dictionary as a parameter and initializes all settings.frommimeoimportMimeoConfigconfig={"_templates_":[{"count":30,"model":{"SomeEntity":{"@xmlns":"http://mimeo.arch.com/default-namespace","@xmlns:pn":"http://mimeo.arch.com/prefixed-namespace","ChildNode1":1,"ChildNode2":"value-2","ChildNode3":True}}}]}mimeo_config=MimeoConfig(config)MimeoConfigFactoryTo easily parse Mimeo Configuration you can use theMimeoConfigFactory.
It allows you to provide a raw config as:a dictionarya stringified XML nodea file pathJSONXMLRaw datafrommimeoimportMimeoConfigFactoryconfig={"_templates_":[{"count":30,"model":{"SomeEntity":{"@xmlns":"http://mimeo.arch.com/default-namespace","@xmlns:pn":"http://mimeo.arch.com/prefixed-namespace","ChildNode1":1,"ChildNode2":"value-2","ChildNode3":True}}}]}mimeo_config=MimeoConfigFactory.parse(config)frommimeoimportMimeoConfigFactoryconfig=('<mimeo_configuration>''    <_templates_>''        <_template_>''            <count>30</count>''            <model>''''                <SomeEntity''                    xmlns="http://mimeo.arch.com/default-namespace"''                    xmlns:pn="http://mimeo.arch.com/prefixed-namespace">''                    <ChildNode1>1</ChildNode1>''                    <pn:ChildNode2>value-2</pn:ChildNode2>''                    <ChildNode3>true</ChildNode3>''                </SomeEntity>''''            </model>''        </_template_>''    </_templates_>''</mimeo_configuration>')mimeo_config=MimeoConfigFactory.parse(config)File pathfrommimeoimportMimeoConfigFactoryconfig="SomeEntity-config.json"mimeo_config=MimeoConfigFactory.parse(config)frommimeoimportMimeoConfigFactoryconfig="SomeEntity-config.xml"mimeo_config=MimeoConfigFactory.parse(config)Processing Mimeo ConfigurationUsing the Mimeo as a python library you can use 2 processing approaches:sequential processingprocessing in parallel (used by default in Mimeo CLI)Both need theMimeographclass.Sequential processingSequential processing is pretty straightforward and can be done withoutMimeographinstantiation.ProcessingTo simply process data from a Mimeo Configuration you can use theMimeograph.process()method:frommimeoimportMimeoConfigFactory,Mimeographconfig_path="examples/1-introduction/01-basic.json"mimeo_config=MimeoConfigFactory.parse(config_path)Mimeograph.process(mimeo_config)It will generate data and consume it immediately.Generating onlyIf you're going to generate data and use it as a python representation
(dict,xml.etree.ElementTree.Element) - useMimeograph.generate()method:frommimeoimportMimeoConfigFactory,Mimeographconfig_path="examples/1-introduction/01-basic.json"mimeo_config=MimeoConfigFactory.parse(config_path)data=Mimeograph.generate(mimeo_config)Generating and consuming in 2 stagesIn case you would like to somehow modify generated data before it will be consumed,
useMimeograph.generate()andMimeograph.consume()methods.frommimeoimportMimeoConfigFactory,Mimeographconfig_path="examples/1-introduction/01-basic.json"mimeo_config=MimeoConfigFactory.parse(config_path)data=Mimeograph.generate(mimeo_config)# ... your modifications ...Mimeograph.consume(mimeo_config,data)Processing in parallelWhen you're going to process data (generate and consume) from several Mimeo Configurations
processing in parallel is more performant way. To do that, you need use theMimeographas a Context Manager
and submit configs together with some kind of identifier (e.g. config path). Thanks to that you will know
which config has failed (if so).frommimeoimportMimeoConfigFactory,Mimeographconfig_paths=["examples/1-introduction/01-basic.json","examples/1-introduction/02-complex.json","examples/1-introduction/03-output-format-xml.json","examples/1-introduction/04-output-format-json.json",]withMimeograph()asmimeo:forconfig_pathinconfig_paths:mimeo_config=MimeoConfigFactory.parse(config_path)mimeo_config.output.direction="stdout"mimeo.submit((config_path,mimeo_config))LicenseMITAuthors@TomaszAniolowskiAcknowledgementsSimpleMaps.com(Cities & countries data)@hadley/data-baby-names(Forenames data)@fivethirtyeigh/data/most-common-name(Surnames data)@datasets/currency-codes(Currencies data)