Package: goodcrap
goodcrapgoodcrapis a python package that generates random data stuff: creates and fills data structures (tables, databases andcsvfiles) with random data, generatesMagepipelines that the user can use to orchestrate filling the data structures, and generate random SQL queries.MotivationThis software enables data engineers to replicate the database schemas at their organisations, and then generate fake data that resemble a random sample of the actual data in their organisation. It also enables them to generate any number of random SQL queries that they can use for testing their analytics pipelines, as well as benchmark their data platforms.While public datasets, such as those hosted at Google or Kaggle, is a common starting point for people interested in learning data analytics and machine learning, many of these datasets require extensive data cleaning so that they can be usable in analytics pipelines. This makes the use of these datasets difficult for AI learners and practitioners.Public datasets are also utilized by data engineers who are interested in testing their ETL/ELT pipelines. Those folks are particularly interested in data quantity, more than quality. Most public datasets are limited in quantity, which make them not so useful for testing pipelines or for benchmarking query execution times.Nowadays, generating random data is increasingly a requirement for data teams.It is a better alternative to using public datasets that will require tedious cleaning. While the Jaffle toy dataset provided bydbtenables an easy way to generate a huge amount of data, the python packagejafgenis limited to generating a dataset that belongs to a specific set of tables.goodcrapwas developed to enable:AI learners to generate their own custom datasetsAI practitioners to benchmark the scalability of their models and methodsdata engineers to test and benchmark their ETL/ELT data pipelinesdata engineers to benchmark query execution times against custom-made huge datasetsThe data generated bygoodcrapis, after all, crap. But it's good crap because:data values are configured based onjsonconfiguration filesdata can be generated to fill tables, databases, data warehouses and data lakesdata values can be made totally random, or fulfill a certain distributiongoodcrapserver can generate time series dataInstallationYou can installgoodcrapusing thepipcommand as follows:pip install goodcrapBasic usageThe simplest use-case scenario is generating acsvfile with random data.goodcrapships with a number oftemplatetables that you can use. For example, let's generate 10,000 records in thecustomerstable, using the random seed3:goodcrap --size 10000 --seed 3 --template_table customers --to_csvThe filecustomers.csvwill be generated.goodcrappopulate databases with random data, in addition to fillingcsvfiles. You can setgoodcrapto connect to your database via a database configuration file, the name of which is passed togoodcrapvia the command line argument--database_config. This is a json file that looks like this (for a MySQL database):{"db_type":"mysql","host":"localhost","port":"3306","user":"root","passwd":"","database":"goodcrap"}Here is an example command to create and fill thecustomerstable in the database:goodcrap --size 1000 --seed 3 --database_config mysql_config --template_table customers --to_csvwheremysql_configis the name of the configurationjsonfile (mysql_config.json).For every tablemytableyou want to fill with random values, you must provide either:a filemytable.crap_labels.json: this file tellsgoodcrapwhat sort of random values to generate for each recorda sample database table orcsvfile with matching structure and with some values:goodcrapwill learn how to generate new random values based on the sample valuesSupported destinationsCurrently,goodcrapcan write data to the following destinations:MySQLSQLiteSnowflakecsvfilejsonfileparquetfileTemplate data structuresgoodcraphas a number of template tables and databases that you can use. They are in thetemplates/directory.Thecrap_labels.jsonsettings fileFor every table you want to generate, you have to provide thecrap_labels.jsonfile. If you are using the python library, then you can pass thecrap_labeldictionary instead - as is explained below.The dictionary incrap_labels.jsontellsgoodcraphow to fill each column in the table with random values. You can either use any of thefakerproviders there, or you can use the ones incrappers.How random data is made:fakerandcrappersin progressPython libraryThe classGoodCrapis yourgoodcrapinterface. You instantiate it with the key settings, and then generate the data by using the member functionswrite_csv(),get_dataframe()orrun().write_csv(): writes acsvfileget_dataframe(): returns apandasDataFrameobject populated with the random datarun(): that's the more generic function that can generate tables and databases and populate themAnd example usage for thegoodcraplibrary is as follows. Here we are generating apandasDataFramefor one of the template tables,customers:fromgoodcrapimportGoodCrapa=GoodCrap(seed=3,size=1000,template_table='customers')df=a.get_dataframe()The following example generates the data frame for some table, given itscrap_labelconfiguration object:gc=GoodCrap(size=10000,seed=123)craplabels={"customer_number":"ssn","first_name":"first_name","last_name":"last_name","phone":"phone_number","address_line":"street_address","city":"city","state":"state","postalcode":"postalcode","country":"current_country","date_of_birth":"date","credit_limit":{"type":"random_int","min":0,"max":1000,"multiplier":10},"income":{"type":"random_int","min":0,"max":10000,"multiplier":10}}df=gc.get_dataframe('customers',craplabels)How data for a foreign key column is generatedgoodcrapwill detect whether a column in a table is related to another table, and will fill that column with random selections of the related column. To demonstrate, run this command:goodcrap --size 1000 --seed 3 --database_config examples\mysql_config --template_database customers_ordersThis command will use the database settings inexamples\mysql_config.jsonto generate the template databasecustomers_ordersand fill the tables with 1000 rows each. There are two tables here:customersandorders, and they are related:ordershas a columncustomer_numberthat is tied tocustomersvia the foreign keycustomers.customer_number. Therefore, that column is filled with random selections fromcustomers.customer_number.For a quick demo of generating theorderstable: assuming you have setup up thecustomers_ordersdatabase and filled it with some data, the following code will generate anordersDataFrameusing columns values from thecustomerstable:fromgoodcrapimportGoodCrapa=GoodCrap(seed=3,size=1000,template_table='orders',database_config="../examples/mysql_config")df=a.get_dataframe()goodcrapwithMagegoodcrapcan be used inMageas aData Loader. You can generate as much data as you want from multiple randomgoodcrapData Loaders into your pipelines for testing, such as for testing the convergence of data into a data warehouse. You can also schedule the generation of data fromgoodcrapsources to simulate time series data traffic. A typical test-case scenario here is to run an SQL query at the data destination while data is being continuously loaded.Here is an examplegoodcrapsource inMage:if'data_loader'notinglobals():frommage_ai.data_preparation.decoratorsimportdata_loaderif'test'notinglobals():frommage_ai.data_preparation.decoratorsimporttestfromgoodcrapimportGoodCrap@data_loaderdefload_data(*args,**kwargs):gc=GoodCrap(size=10000,seed=123)craplabels={"customer_number":"ssn","first_name":"first_name","last_name":"last_name","phone":"phone_number","address_line":"street_address","city":"city","state":"state","postalcode":"postalcode","country":"current_country","date_of_birth":"date","credit_limit":{"type":"random_int","min":0,"max":1000,"multiplier":10},"income":{"type":"random_int","min":0,"max":10000,"multiplier":10}}returngc.get_dataframe('customers',craplabels)@testdeftest_output(output,*args)->None:"""Template code for testing the output of the block."""assertoutputisnotNone,'The output is undefined'Note:If you are planning to run aMagepipeline multiple times, then make sure that it does not have columns that are generated using the faker.unique function. The columns should be universally unique.goodcrapgeneratesMagepipelinesMagepython files are generated usingJinjatemplates. Given that Mage will always be backwards compatible (according to communication with its authors), files and folders generated bygoodcrapwill always be valid. Here is an example command to generate pipelines for each of the tables in the template databasecustomers_orders:goodcrap --size 1000 --seed 3 --database_config examples\mysql_config --template_database customers_orders --mage_pipelineNote thatgoodcrapcurrently will only generateMageprojects if the database configurations are defined.Writing to Snowflakegoodcrapsupports writing your random table to Snowflake using two methods:row-by-row, which can be done by setting Snowflake as your database in the database configuration filebulk upload of the generatedpandas DataFrame, which is enabled by the command line argument--bulk_upload.The bulk upload is obviously preferred. Below is an example configuration settings file:{"db_type":"snowflake","snowflake_database":"GOODCRAP","snowflake_warehouse":"WH","snowflake_user":"user","snowflake_password":"password","snowflake_account":"account","snowflake_schema":"public","snowflake_role":"role"}Suppose you want to create theorderstable in Snowflake and fill it with 1,000,000 rows, and also get a few sample queries to try. You can get all that with the following command:goodcrap --size 1000000 --seed 12 --database_config config --template_table orders --bulk_upload --queries 100Data warehousesSome dimensions in data warehouses will required to be filled as part of the testing exercise, but should not be filled with random data. These are theconformeddimensions with rigid data, such as the Date, Countries, and Cities dimensions.goodcrapwill be able to fill these dimensions using theDimensionFillerclass by providing several options for featurization. Filling these tables will be performed before any other table is populated.Guessing thecrap_labels.jsonsettingsin progressLearning the values from a data samplein progressContributing togoodcrapThat would be much appreciated. Checkhere.Licensegoodcrapis licensed under theGPL3license.