Package: ingenii-data-engineering
Ingenii Data Engineering PackageDetailsCurrent Version: 0.3.2OverviewThis package provides utilities for data engineering on Ingenii's Azure Data Platform. This can be both used for local development, and is used in theIngenii Databricks Runtime.UsageImport the package to use the functions within.importingenii_data_engineeringdbtPart of this package validates dbt schemas to ensure they are compatible with Databricks and the larger Ingenii Data Platform. This happens when a data pipeline to ingest a file is run, to make sure a file is ingested correctly.
Full details of how to set up your dbt schema files in your Data Engineering repository can be found in theIngenii Data Engineering Example repository.Pre-processingThis package contains code to facilitate the pre-processing of files before they are ingested by the data platform. This allows users to transform any data into a form that is compatible. See details of working with pre-processing functions in theIngenii Data Engineering Example repository.This package also contains the code to turn the pre-processing scripts into a package, ready to be uploaded and used by the Data Platform. Once this package is installed, the commandpython-m<packagename><command><folderwithpre-processingcode>
python-mingenii_data_engineeringpre_processing_packagepre_processwill generate a.whlfile in a folder calleddist/. For more details, see theIngenii Data Engineering Example repository.DevelopmentPrerequisitesA working knowledge ofgit SCMInstallation ofPython 3.7.3Set upComplete the 'Getting Started > Prerequisites' sectionFor Windows only:Runmake setup: to copy the .env into place (.env-dist>.env)Getting startedComplete the 'Getting Started > Set up' sectionFrom the root of the repository, in a terminal (preferably in your IDE) run the following commands to set up a virtual environment:python-mvenvvenv
.venv/bin/activate
pipinstall-rrequirements-dev.txt
pre-commitinstallor for Windows:python-mvenvvenv
.venv/Scripts/activate
pipinstall-rrequirements-dev.txt
pre-commitinstallNote: if you get apermission deniederror when executing thepre-commit installcommand you'll need to runchmod -R 775 venv/bin/to recursively update permissions in thevenv/bin/dirThe following checks are run as part ofpre-commithooks: flake8(note unit tests are not run as a hook)BuildingComplete the 'Getting Started > Set up' sectionRunmake buildto create the package in./distRunmake cleanto remove dist filesTestingComplete the 'Getting Started > Set up' and 'Development' sectionsRunmake testto run the unit tests usingpytestRunflake8to run lint checks usingflake8Runmake qato run the unit tests and linting in a single commandRunmake qato remove pytest filesVersion History0.3.2: Further bugfix for JSON UTF-8 BOM0.3.1: Remove unnecessary functions specific to Databricks0.3.0: Create pre-processing package using the module0.2.1: Handle JSON read UTF-8 BOM0.2.0: Pre-processing happens all in the 'archive' container0.1.5: Better functionality for column names in .csv files0.1.4: Handle JSON files0.1.3: Adding pre-processing utilities0.1.2: Rearrangement and better split of work with the Databricks Runtime. Better validation0.1.1: Minor bug fixes0.1.0: dbt schema validation, pre-processing class