Package: dattasa
python package that helps data engineers and data scientists accelerate data-pipeline developmentThe goal of this python project is to build a bunch of wrappers that can
be reused for building data pipelines from - - Relational databases:
postgres, mysql, greenplum, redshift, etc. - NOSQL databases: hive,
mongo, etc. - messaging sources and caches: kafka, redis, rabbitmq, etc.
- cloud service providers: salesforce, mixpanel, jira, google-drive,
delighted, wootric, etc.InstallationThere are 3 ways to install dattasa package -Easiest way is to install from pypi using pippip install dattasaDownload from github and build from scratchgit clone git@github.com:kartikra/dattasa.git
cd dattasa
python setup.py build
python setup.py clean
python setup.py installDownload from github and install using pipgit clone git@github.com:kartikra/dattasa.git
cd dattasa
pip install -e .
pip install -U -e . (if upgrading)Config FilesBy default dattasa expects the config files to be in the mode directory
of user. These can be overridden. See links to sample code in README
file below to find out more. There are 2 yaml config files -
database.yaml - conists of database credentials and api keys needed for
making connection. seesample database
config- ftpsites.yaml - Needed for
performing sftp transfers. seesample ftpsites
configEnvironment Variablesdattasa package relies on the following environment variables. Make sure
to set these in your bash profile - GPLOAD_HOME: Path to gpload package
(needed only if using gpload utilities for greenplum or redshift) -
PROJECT_HOME: Path to python project directory -
PROJECT_HOME/python_bash_scripts: python scripts to invoke gpload
(needed only if using gpload utilities for greenplum or redshift) -
SQL_DIR: Place to keep all sql scripts - TEMP_DIR: All temp files
created in this folder - LOG_DIR: All log files are created in this
folderDescription of classesv1.0 of the package comprises of the following classes. Please see link
to sample code for details on how to use each of them.classDescriptionSample CodeenvironmentLets you source all the os
environment variablessee first row
in mongo
examplepostgres_clien
tLets you use psql and gpload
utilities provided bypivotal
greenplum.
Make connections to postgres /
greenplum database using
pyscopg2 or sqlalchemy.Use the
connections to interact with
database in interactive program
or run queries from a sql file
using the connectionsample postgres
codegreenplum_clie
nt
(inherits
postgres_clien
t)Lets you use psql and gpload
utilities provided bypivotal
greenplum.
Make connections to postgres /
greenplum database using
pyscopg2 or sqlalchemy.Use the
connections to interact with
database in interactive program
or run queries from a sql file
using the connectionsample
greenplum
codemysql_clientLets you use mysql and other
methods provided by PyMySQL
Packagesample mysql
codefile_processorCreate sftp connection usingparamikopackage. Other file
manipulations like row_count,
encryption, archive (File Class)see file
processing
examplenotificationSend email notificationsmongo_clientLoad data to mongodb using bulk
load. Run java script queriessee mongo
exampleredis_clientRead data from a redis cache or
load a redis cachesee redis
examplekafka_systemCurrently allows Publisher and
Consumer to use kafka in batch
modesee kafka
examplerabbitmq_syste
mCurrently has Publisher to
publish messages in rabbitmqmixpanel_clien
tConnect to mixpanel api and
fetch data using jql or export
raw events data.mixpanel api
documentationsee mixpnael
section in api
examplesalesforce_cli
entCreate a connection to
salesforce usingsimple_salesforcepackagesee salesforce
section in api
exampledelighted_clie
ntGet nps scores and survey
responses from delighted.api
documentationsee delighted
section in api
examplewootric_clientGets nps scores and survey
responses from wootric.api
documentationsee wootric
section in api
exampledag_controllerFunctions needed to integrate
this package within an airflow
dag.airflow
documentationandgithub
projectdata_pipeline classThis is the main class that’s accessible to other projects. The data
pipeline consists of data from components and API. Each object of
data-processor can use individual data streams and process them
data_pipeline decides which modules to call based on type of database
(as defined in config file). data_pipeline comprises of 3 classes -
DataComponent : Each database connection is considered to be
data-component object.See examples for postgres, mysql, greenplum, etc
above - APICall : Each api call is an apicall object. See examples for
mixpanel, delighted, salesforce and wootric above - DataProcessor :
transfers and loads data between data components.see
examplesAdding ipython notebook files to githubUse git lfs Seedocumentationif using mac install git-lfs using brewbrew installgit-lfsinstall lfsgit lfs installtrack ipynb files in your project. go to the project folder and dogit lfs track"*.psd"add.*ipynb_checkpoints/to .gitignore fileFinally add .gitattributes filegit add .gitatttributesDeploying code in pypibuild the code:python setup.py build && python setup.py clean && python setup.py installpush to pypitest :python setup.py sdist upload-rpypitestpush to pypi prod :python setup.py sdist upload-rpypi