Package: csv-etl
csv-etlA rules based approach to performing ETL operations on csv files.InstallationRequires Python 3Using pippython-mvenvvenvsourcevenv/bin/activate
pipinstallcsv-etlLocallygitclonehttps://github.com/winslowdibona/csv-etl.gitcdcsv-etl
python-mvenvvenvsourcevenv/bin/activate
pipinstall--editable.Developinggitclonehttps://github.com/winslowdibona/csv-etl.gitcdcsv-etl
python-mvenvvenvsourcevenv/bin/activate
pipinstall-rrequirements.txtOverviewThe goal of this project is to provide a re-usable way to perform ETL operations on csv files. This implementation takes a given row of data from a csv file and applies a set of rules to generate a new format of the data.Rule OverviewTheRulein this project has the following propertiestargetsourcetypeinput_typeoutput_typeoperationsUsing different combinations of values for these properties can give us a flexible toolset for extracting data.target -strOnce we have the extracted the value from the csv and applied our rule to it, the resulting value will appear under this value as the key in the resultsource -str/listThis is where we want to pull the data from in the csv. If the source is a string, it will fetch that single column. If the source is a list, it will fetch all of the values.type -RuleTypeThis can be one of two optionsStaticCalculationIf RuleType.Static, the rule will simply return the value stored in Rule.source under Rule.targetIf RuleType.Calculation, the rule will fetch the value(s) defined in Rule.source, and perform the operations on theminput_type -InputTypeThe data type you would like to read the value in from the csv as.This can be one of three optionsStringIntegerDecimaloutput_type -OutputTypeThe data type you would like the resulting value to be.This can be one of four optionsStringIntegerDecimalDateoperations -listA list of strings that will be run through pythonevalstatement. The value(s) extracted from the csv will be available for use in these operations. If there is only a single source, the value will be assigned to the variables. If there are multiple sources, the values will be passed in as a list under the variables.Defining RulesThese rules can be defined programmatically, or via a YAML configuration. The configuraiton follows the below structurerules:-target:target_nametype:Static || Calculationinput_type:String || Integer || Decimaloutput_type:String || Integer || Decimal || Datesource:source_name || [source, names]operations:['operations','to','run']Converting CSV DataOnce we have a set of rules, we can use theCSVConverterclass to execute our rules on a data set.fromcsv_etlimportCSVConvertercsv_converter=CSVConverter(rules)result=csv_converter.convert('path/to/csv_file')This will give us back a list of dictionaries, with each item in the list representing the modified data for each row in the initial csv file.We can also get the result fed back to us in csv format as a stringresult=csv_converter.convert('path/to/csv/file',to='csv')UsageCLIcsv-etl./examples/order_data/config.yaml./examples/order_data/test_data.csv

$csv-etl--help
Usage:csv-etl[OPTIONS]CONFIGCSV

Options:--outfileTEXTFilepathtowritetheresultto--formatTEXTFormattheresultshouldbe."json"or"csv"--helpShowthismessageandexit.ExamplesMore detailed usage and programmatic examples are providedOrder Data ExampleExample Rule ConfigsHelpful Make CommandsGenerating Documentationmakedocs
openhtml/csv_etl/index.htmlRunning TestsmakepytestGetting Test Coveragemaketest-cov
opencov_html/index.htmlWhat Next?Better error handlingCurrently the errors are just printed to the console. Might be nice to have an option to gather them and have them represented in the resulting data set somehow.Better handling of eval statementsRight now the use ofevalstatements is a little tailored for the original problem
and could use some more exploration on how they can be further utilized and how to handle
potential errors.Also right now the ability to perform multiple operations on a single extracted value is doable, but performing multiple operations on multiple extracted values will not.evalstatements don't allow assignment of variables, and this prevents us from performing some logic, and being able to use the resulting values again in a list. Further exploration here may find a way to do multiple operations with multiple extracted values.