Package: pramen-py
Pramen-pyCli application for defining the data transformations for Pramen.See:pramen-py--helpfor more information.InstallationApp settingsApplication configuration solved by the environment variables
(see .env.example)Add pramen-py as a dependency to your projectIn case of poetry:# ensure we have valid poetry environmentlspyproject.toml||poetryinit

poetryaddpramen-pyIn case of pip:pipinstallpramen-pyUsageApplication configurationIn order to configure the pramen-py options you need to set
corresponding environment variables. To see the list of available options run:pramen-pylist-configuration-optionsDeveloping transformationspramen-py uses python'snamespace packagesfor discovery of the transformations.This mean, that in order to build a new transformer, it should be located
inside a python package with thetransformationsdirectory inside.This directory should be declared as a package:for poetry[tool.poetry]# ...packages=[{include="transformations"},]for setup.pyfromsetuptoolsimportsetup,find_namespace_packagessetup(name='mynamespace-subpackage-a',# ...packages=find_namespace_packages(include=['transformations.*']))Example files structure:❯ tree .
.
├── README.md
├── poetry.lock
├── pyproject.toml
├── tests
│  └── test_identity_transformer.py
└── transformations
    └── identity_transformer
        ├── __init__.py
        └── example_config.yamlIn order to make transformer picked up by the pramen-py the following
conditions should be satisfied:python package containing the transformers should be installed to the
same python environment as pramen-pypython package should have defined namespace packagetransformationstransformers should extendpramen_py.Transformationbase classSubclasses created by extending Transformation base class are registered as
a cli command (pramen-py transformations run TransformationSubclassName)
with default options. Check:pramen-pytransformationsrunExampleTransformation1--helpfor more details.You can add your own cli options to your transformations. See example atExampleTransformation2pramen-py pytest pluginpramen-py also provides pytest plugin with helpful
fixtures to test created transformers.List of available fixtures:#install pramen-py into the environment and activate itpytest--fixtures# check under --- fixtures defined from pramen_py.test_utils.fixtures ---pramen-py pytest plugin also loads environment variables from .env
file if it is presented in the root of the repo.Running and configuring transformationsTransformations can be run with the following command:pramen-pytransformationsrun\ExampleTransformation1\--configconfig.yml\--info-date2022-04-01--configis required option for any transformation. Seeconfig_example.yamlfor more information.To check available options and documentation for a particular transformation,
run:pramen-pytransformationsrunTransformationClassName--helpwhere TransformationClassName is the name of the transformation.Using as a LibraryRead metastore tables by Pramen-Py APIimportdatetimefrompyspark.sqlimportSparkSessionfrompramen_pyimportMetastoreReaderfrompramen_py.utils.file_systemimportFileSystemUtilsspark=SparkSession.getOrCreate()hocon_config=FileSystemUtils(spark)\.load_hocon_config_from_hadoop("uri_or_path_to_file")metastore=MetastoreReader(spark)\.from_config(hocon_config)df_txn=metastore.get_table("transactions",info_date_from=datetime.date(2022,1,1),info_date_to=datetime.date(2022,6,1))df_customer=metastore.get_latest("customer")df_txn.show(truncate=False)df_customer.show(truncate=False)DevelopmentPrerequisites:https://python-poetry.org/docs/#installationpython 3.6Setup steps:gitclonehttps://github.com/AbsaOSS/pramencdpramen-py
makeinstall# create virtualenv and install dependenciesmaketestmakepre-commit# enable completions# source <(pramen-py completions zsh)# source <(pramen-py completions bash)pramen-py--helpLoad environment configurationBefore doing any development step, you have to set your development
environment variablesmakeinstallCompletions# enable completionssource<(pramen-pycompletionszsh)# or for bash# source <(pramen-py completions bash)DeploymentFrom the local development environment# bump the versionvimpyproject.toml# deploy to the dev environment (included steps of building and publishing#   artefacts)cat.env.ci
makepublish