Package: dbxconfig
dbxconfigConfiguration framework for databricks pipelines.
Define configuration and table dependencies in yaml config then get the table mappings config model:Define your tables.landing:read:landing_dbx_patterns:customer_details_1:nullcustomer_details_2:nullraw:delta_lake:raw_dbx_patterns:customers:ids:iddepends_on:-landing.landing_dbx_patterns.customer_details_1-landing.landing_dbx_patterns.customer_details_2warning_thresholds:invalid_ratio:0.1invalid_rows:0max_rows:100min_rows:5exception_thresholds:invalid_ratio:0.2invalid_rows:2max_rows:1000min_rows:0custom_properties:process_group:1base:delta_lake:# delta table properties can be set at stage level or table leveldelta_properties:delta.appendOnly:truedelta.autoOptimize.autoCompact:truedelta.autoOptimize.optimizeWrite:truedelta.enableChangeDataFeed:falsebase_dbx_patterns:customer_details_1:ids:iddepends_on:-raw.raw_dbx_patterns.customers# delta table properties can be set at stage level or table level# table level properties will overwride stage level propertiesdelta_properties:delta.enableChangeDataFeed:truecustomer_details_2:ids:iddepends_on:-raw.raw_dbx_patterns.customersDefine you load configuration:tables:./tables.yamllanding:read:trigger:customerdetailscomplete-{{filename_date_format}}*.flgtrigger_type:filedatabase:landing_dbx_patternstable:"{{table}}"container:datalakeroot:"/mnt/{{container}}/data/landing/dbx_patterns/{{table}}/{{path_date_format}}"filename:"{{table}}-{{filename_date_format}}*.csv"filename_date_format:"%Y%m%d"path_date_format:"%Y%m%d"format:cloudFilesspark_schema:../Schema/{{table.lower()}}.yamloptions:# autoloadercloudFiles.format:csvcloudFiles.schemaLocation:/mnt/{{container}}/checkpoint/{{checkpoint}}cloudFiles.useIncrementalListing:auto# schemainferSchema:falseenforceSchema:truecolumnNameOfCorruptRecord:_corrupt_record# csvheader:falsemode:PERMISSIVEencoding:windows-1252delimiter:","escape:'"'nullValue:""quote:'"'emptyValue:""raw:delta_lake:# delta table properties can be set at stage level or table leveldelta_properties:delta.appendOnly:truedelta.autoOptimize.autoCompact:truedelta.autoOptimize.optimizeWrite:truedelta.enableChangeDataFeed:falsedatabase:raw_dbx_patternstable:"{{table}}"container:datalakeroot:/mnt/{{container}}/data/rawpath:"{{database}}/{{table}}"options:checkpointLocation:/mnt/{{container}}/checkpoint/{{database}}_{{table}}mergeSchema:trueImport the config objects into you pipeline:fromdbxconfigimportConfig,Timeslice,StageType# build path to configuration filepattern="auto_load_schema"config_path=f"../Config"# create a timeslice object for slice loading. Use * for all time (supports hrs, mins, seconds and sub-second).timeslice=Timeslice(day="*",month="*",year="*")# parse and create a config objectsconfig=Config(config_path=config_path,pattern=pattern)# get the configuration for a table mapping to load.table_mapping=config.get_table_mapping(timeslice=timeslice,stage=StageType.raw,table="customers")print(table_mapping)Development Setuppip install -r requirements.txtUnit TestsTo run the unit tests with a coverage report.pip install -e .
pytest test/unit --junitxml=junit/test-results.xml --cov=dbxconfig --cov-report=xml --cov-report=htmlBuildpython setup.py sdist bdist_wheelPublishtwine upload dist/*