Package: drpt
Data Release Preparation ToolData Release Preparation ToolDescriptionInstallationUsageCLIRecipe DefinitionExampleThanks:warning: This is currently at beta development stage and likely has a lot of bugs. Please use theissue trackerto report an bugs or feature requests.DescriptionCommand-line tool for preparing a dataset for publishing by dropping, renaming, scaling, and obfuscating columns defined in a recipe.After performing the operations defined in the recipe the tool generates the transformed dataset version and a CSV report listing the performed actions.InstallationThe tool can be installed using pip:pipinstalldrptUsageCLIUsage: drpt [OPTIONS] RECIPE_FILE INPUT_FILE

Options:
  -d, --dry-run           Generate only the report without the release dataset
  -v, --verbose           Verbose [Not implemented]
  -n, --nrows TEXT        Number of rows to read from a CSV file. Doesn't work
                          with parquet files.
  -l, --limits-file PATH  Limits file
  -o, --output-dir PATH   Output directory. The default output directory is
                          the same as the location of the recipe_file.
  --version               Show the version and exit.
  --help                  Show this message and exit.Recipe DefinitionOverviewThe recipe is a JSON formatted file that includes what operations should be performed on the dataset. For versioning purposes, the recipe also contains aversionkey which is appended in the generated filenames and the report.Default recipe:{"version":"","actions":{"drop":[],"drop-constant-columns":false,"obfuscate":[],"disable-scaling":false,"skip-scaling":[],"sort-by":[],"rename":[]}}The currently supported actions, performed in this order, are as follows:drop: Column deletiondrop-constant-columns: Drops all columns that containt only one unique valueobfuscate: Column obfuscation, where the listed columns are treated as categorical variables and then integer coded.Scaling: By default all columns are Min/Max scaleddisable-scaling: Can be used to disable scaling for all columnsskip-scaling: By default all columns are Min/Max scaled, except those excluded (skip-scaling)sort-by: Sort rows by the listed columnsrename: Column renamingAll column definitions above supportregular expressions.ActionsdropThedropaction is defined as a list of column names to be dropped.drop-constant-columnsThis is a boolean action, which when set totruewill drop all the columns that have only a single unique value.obfuscateTheobfuscateaction is defined as a list of column names to be obfuscated.disable-scaling,skip-scalingBy default, the tool Min/Max scales all numerical columns. This behavior can be disabled for all columns by setting thedisable-scalingaction totrue. If scaling must be disabled for only a set of columns these columns can be defined using theskip-scalingaction, as a list of column names.sort-byThis is a list of column names by which to sort the rows. The order in the list denotes the sorting priority.renameTherenameaction is defined as a list of objects whose key is the original name (or regular expression), and their value is the target name. When the target uses matched groups from the regular expression those can be provided with their group number prepended with an escaped backslash (\\1) [seeexamplebelow].{//..."rename":[{"original_name":"target_name"}]//...}ExampleInput CSV file:test1,test2,test3,test4,test5,test6,test7,test8,test9,foo.bar.test,foo.bar.test2,const
1.1,1,one,2,0.234,0.3,-1,a,e,1,1,1
2.2,2,two,2,0.555,0.4,0,b,f,2,2,1
3.3,3,three,4,0.1,5,1,c,g,3,3,1
2.22,2,two,4,1,0,2.5,d,h,4,4,1Recipe:{"version":"1.0","actions":{"drop":["test2","test[8-9]"],"drop-constant-columns":true,"obfuscate":["test3"],"skip-scaling":["test4"],"sort-by":["test4","test3"],"rename":[{"test1":"test1_renamed"},{"test([3-4])":"test\\1_regex_renamed"},{"foo[.]bar[.].*":"foo"}]}}Generated CSV file:test3_regex_renamed,test4_regex_renamed,test1_renamed,test5,test6,test7,foo_1,foo_2
0,2,0.0,0.1488888888888889,0.06,0.0,0.0,0.0
2,2,0.5000000000000001,0.5055555555555556,0.08,0.2857142857142857,0.3333333333333333,0.3333333333333333
1,4,1.0,0.0,1.0,0.5714285714285714,0.6666666666666666,0.6666666666666666
2,4,0.5090909090909091,1.0,0.0,1.0,1.0,1.0Report:,action,column,details
0,recipe_version,,1.0
1,drpt_version,,0.6.3
2,DROP,test2,
3,DROP,test8,
4,DROP,test9,
5,DROP_CONSTANT,const,
6,OBFUSCATE,test3,"{""one"": 0, ""three"": 1, ""two"": 2}"
7,SCALE_DEFAULT,test1,"[1.1,3.3]"
8,SCALE_DEFAULT,test5,"[0.1,1.0]"
9,SCALE_DEFAULT,test6,"[0.0,5.0]"
10,SCALE_DEFAULT,test7,"[-1.0,2.5]"
11,SCALE_DEFAULT,foo.bar.test,"[1,4]"
12,SCALE_DEFAULT,foo.bar.test2,"[1,4]"
13,SORT,"['test4', 'test3']",
14,RENAME,test1,test1_renamed
15,RENAME,test3,test3_regex_renamed
16,RENAME,test4,test4_regex_renamed
17,RENAME,foo.bar.test,foo_1
18,RENAME,foo.bar.test2,foo_2ThanksThis tool was made possible withPandas,PyArrow,jsonschema, and of coursePython.