Package: dasladen
DasLaden is a simple, tiny and ridiculus ETL made with PythonDasladen is a general purpose Python package to make an automate ETL (Extracting, Transforming and
Loading data) through the configuration of one or more .json files that represents tasks.
It is based onpetl. It can do some tasks like:load a .csv file to database tablerun a database query into a .csv filerun a database query into a database tableconvert a .csv file into another .csv fileconvert a .xls file into a .csv fileload a .xml file into a database tableconvert a .xls file into a .csv fileThis tasks can be configured to do some basic transformations offer bypetland you can write your own
transformations in a Python module or class to be called by Dasladen during loading process.There is others types of tasks to do things like:Compact files into .zip fileExtract files from .zip fileUpload a fileDownload a fileExecute a Python scriptExecute a SQL commandThe tasks are configured in a.jsonfile that supports a sequence of tasks that will be executed
in configured order. Details of how to configure tasks will be in Wiki pages.The basic steps to use DasLaden is:Install dasladen package viapip install dasladenin your environment or in virtualenv.Install database driver package if you want to execute database tasks. Dasladen is prepared to run with the
following drivers: MySQL viaPyMySQL, MS SQL Server viapyodbcand Oracle viacx_Oracle. Please see
the limitations on the driver package that you choose.Create a folder for you project.Prepare a folder structure in project folder with following names:inputIs the default folder to put input files, like .csv, .xml, .xls and .sql filesoutputIs the default folder that tasks write target filesmoduleIs the folder for python scripts if you can't put then in project foldercaptureIs the default folder to drop task files (.json or .zip)logIs the folder that Dasladen write task logstasksIs the folder that you can put tasks files. It is only a suggestion.Create a.jsonfile with your tasks intasksfolder.Start DasLaden from project folder callingpython -m dasladen.If you want to see log in console window, pass a--verboseas argument on call.Copy the.jsontasks file fromtasksto thecapturefolder.The watcher will open the tasks file and process it. To see result you can openlogfolder and search
forwatcher_DD_TT.logwhere DD_TT is the date and time that log was generated. Inlogfolder you
can see individual tasks logs too.It is important that you copy the task file instead move it, because on finish it will be deleted.If you drop a file other than.zipincapturefolder, that file will be move toinputfolder.You can zip the.jsonfile with all other dependent files (.csv, .xls, etc.) and copy
that zip intocapturefolder too. Watcher will unzip then at a temporary folder, copy input
files (other than.jsonfiles) to input folder and execute the.jsonfile.In the.jsonfile you can configure a scheduler to run the tasks. With it you can delay a execution or
configure its recurrence.Data drivers via PyPi packages:MySQL viaPyMySQLpackage. v >= 0.7.5MS SQL Server viapyodbcpackage. v >= 3.0.10Oracle viacx_Oraclepackage. v >= 5.2.1PostgreSQL viapsycopg2package. v >= 2.8.3The current version works with Python 2 and 3.