Package: dingDONG
Ding Dong data preparation ETLdingDong is a metadata management infrastructure for data preparation, ETL, and machine learning models
with a complete version control support for all data preparation, loading, and ML experiments.using dingDONG propagation mechanism help to design, develop, maintain and scale complex batch and real-time data projects in a fraction of timeSeeDing-Dong documentationfor install and developer documentation.
More information can be found athttp://www.biSkilled.comWhy Ding Dong?Simple, because it is fast and easy to implement !Steps for creating at data project:Source to target modeling - mostly done by excel or external ERD mapping toolsImplementing - done mainly by external ETL toolsImprovements, extensions, maintenance and deploy between environments - usually an iterative process which consumes time and effort. meta-data is constantly changing. if business logic involves ML/DL models - versioning must be set for trucking overall results accuracydingDONG developed to maintain and speed the overall data project implementationSource to target modeling - dingDONG truck meta-data changes such as adding, removing, updating DB columns. converting data-type from diverse connections or adding key sequence is done automatically. Modeling can be managed in a simple excel file, dindDONG will truck excel updates and will update all object by using a propagation mechanism.Implementing E(T)L (extract, transform, load) - dingDONG multi-threading and data fetching optimization give a powerful tool for massive bulk data loadingTransform - It is possible to add calculated columns or manipulate existing column by adding python functions. A function such as scaling or normalization can be added as easy as adding the current date or concatenate functionsBusiness logic - Maintaining business logic in dingDONG is used directly by using SQL files that can be executed directly at the development IDE. Using dingDONGcommentssyntax is used to extract and execute complex queries from SQL files which allows managing all business logic in separate files.Version control - dingDONG version control and management keeps track of all metadata changes and stored it in a proprietary dingDONG DB (MongoDB). dingDONG supportsGITversion-control mechanism for managing ML/DL projects, versioning and tracking experiments history results. There is full support inGITHUBfor distributed versioning.dingDONG modules:DING - create and manage overall meta-data structure for all listed object in the work-flowcreating new objectsmodify an existing object by using a back-propagation mechanismupdate data into new objectsave old structuresCI source controlDONG - extract and load data from diverse connectorsextract data - support multi-threading for extracting massive data volumetransfer     - add a custom function on existing columns or new custom calculated fieldsmerge        - merging source with the target object (supported only for objects at the same connector)exec         - execute PL/SQL/Stored procedure commandsVERSION CONTROL - track meta-data and code updatesVersions numbers are managed automatically (sample: <major-release>.<minor-release>.<fixId>)Any change at object metadata is automatically stored at dingDONG repositorySupport managed GIT versioning of ML / DL experiments.Code update - a new version is createdExperiment results stored in dingDong repository for future analyticsRoll revisions history by version numberDefine measures such as counting total input or total output rowsStore executions result for compare experiments performanceInstallationdownload from GitHubor install by usingpip:pip install dingDongAdding MongoDB for pip installation is now cooked. Installation instruction will come soon
Docker support - as well, still at the oven .. will come soonSamplesSamples can be found underDing-Dong sample documentationThe sample below loads 3 CSV files into SQLite. (we are using SQL to merge this CSV and extract data into a report table and CSV file).Download ZIP file with 3 samples CSV files DATAELEMENTDESCRIPTION.csv, DEMOGRAPHICS.csv, MEASURESOFBIRTHANDDEATH.csv
located atsamples/sampleHealthCare/csvData.zipfolder.
In this sample, we useC:\dingDONGas our main folder for all source CSV files and dingDong logs.Full code sampleextractCsvToSqlLite.pylocated atsamples/sampleHealthCare/folderThe sample demonstrates how to load 3 CSV files into SqlLite, create a simple query-based
on those tables and send the result to a new CSV file.configuration properties can be found atdingDONG documentationimport logging
from dingDong import DingDong
from dingDong import Config

""" Main configuration """

""" set log level: logging.INFO, logging.DEBUG, logging.ERROR """
Config.LOGS_DEBUG = logging.DEBUG

"""
    set connection URL into all connectors
    key   -> general connection name or connection type (sql, oracle, file .. )
    value -> can be string or dictionary
        String     --> Connection string URL (key defined connection type: sql, oracle, mySql....)
        Dictionary --> must have 'conn' (connection type) and 'url' (connection string).
    available connection can be found at dingDong.misc.enumsJson.eConn

"""
Config.CONNECTIONS = {
    'sampleSql': {'conn': 'sql',"url": "<Sql server connection string>;UID=USER;PWD=PWD;"},
    'file': "C:\\dingDong\\",
    'sqlite': "C:\\dingDong\\sqlLiteDB.db"}Creating workflow can be done as JSON format or python dictionaries
For the sake of this example, we will use a python dictionary. The sample work-flow will contain:mapping and loading CSV file named DATAELEMENTDESCRIPTION into SQLLite table named dateElements_Descmapping and loading CSV file named DEMOGRAPHICS into SqlLite table named demographicsmapping and loading CSV file named MEASURESOFBIRTHANDDEATH into SQLLite table named birthDatecreate a new query based on demographics and birthDate  into new table named FinalUpdate sample fields at Final table by using direct PL/SQL queryExtract Final data into a CSV file.
We use VARCHAR(200) as default CSV column data type. configuration can be found at DEFAULTS under dingDong.conn.baseBatchnodesToLoad = [
    {"source": ["file", "DATAELEMENTDESCRIPTION.csv"],
     "target": ["sqlite", "dateElements_Desc"]},

    {"source": ["file", "DEMOGRAPHICS.csv"],
     "target": ["sqlite", "demographics"]},

    {"source": ["file", "MEASURESOFBIRTHANDDEATH.csv"],
     "target": ["sqlite", "birthDate"]},

    {"query": ["sqlite", """   Select d.[State_FIPS_Code] AS A, d.[County_FIPS_Code] AS B, d.[County_FIPS_Code] AS G,d.[County_FIPS_Code], d.[CHSI_County_Name], d.[CHSI_State_Name],[Population_Size],[Total_Births],[Total_Deaths]
                                    From demographics d INNER JOIN birthDate b ON d.[County_FIPS_Code] = b.[County_FIPS_Code] AND d.[State_FIPS_Code] = b.[State_FIPS_Code]"""],
     "target": ["sqlite", "Finall", -1]},

    {"myexec": ["sqlite", "Update dateElements_Desc Set [Data_Type] = 'dingDong';"]},

    {"source": ["sqlite", "Finall"],
     "target": ["file", "finall.csv"]}
]Init class dingDongdicObj:Loading dictionary as a work flowdirData:Loading JSON files in this folderincludeFiles:Include files to load from directory folder (dirData)notIncldeFiles:Exclude files to load from directory folder (dirData)connDict:Equal to Config.CONNECTIONS, set connection URLs stringprocesses:Max number of parallel threading to load data (DONG module)m = DingDong(dicObj=nodesToLoad,
             filePath=None,
             dirData=None,
             includeFiles=None,
             notIncludeFiles=None,
             connDict=None,
             processes=1)DINGcreating dateElements_Desc, demographics and birthDate tables based on CSV filesextracting query structure and creating Final tableFlag -1 - default flag,  indicate that on changed structure- old structure is stored with all data. object is udated to new strucutrem.ding()DONG - Extracting and loading data from CSV files into SQLite table, using default truncate-> insert method
Extract data from a query into Final tableif object structure changed and mode 1 (default mode)
* history table will is created
* new object structure is created. new object is populated with data from history table (only identical column name)m.dong()Full sample code:import logging
from dingDong import DingDong
from dingDong import Config

Config.CONNECTIONS = {
    'x1'    : {'conn':'sql',"url":"DRIVER={SQL Server};SERVER=CPX-VLQ5GA42TW2\SQLEXPRESS;DATABASE=ContosoRetailDW;UID=bpmk;PWD=bpmk;"},
    'x2'    : {'conn':'sql',"url":"DRIVER={SQL Server};SERVER=CPX-VLQ5GA42TW2\SQLEXPRESS;DATABASE=ContosoRetailDW;UID=bpmk;PWD=bpmk;"},
    'file'  : "C:\\dingDong\\",
    'sqlite': "C:\\dingDong\\sqlLiteDB.db"}
Config.LOGS_DEBUG = logging.DEBUG
Config.LOGS_DIR = "C:\\dingDong"

nodesToLoad = [
        {   "source":["file","DATAELEMENTDESCRIPTION.csv"],
            "target":["sqlite","dateElements_Desc"]},

        {   "source":["file","DEMOGRAPHICS.csv"],
            "target":["sqlite","demographics"]},

        {   "source":["file","MEASURESOFBIRTHANDDEATH.csv"],
            "target":["sqlite","birthDate"]},

        {   "query":["sqlite","""   Select d.[State_FIPS_Code] AS A, d.[County_FIPS_Code] AS B, d.[County_FIPS_Code] AS G,d.[County_FIPS_Code], d.[CHSI_County_Name], d.[CHSI_State_Name],[Population_Size],[Total_Births],[Total_Deaths]
                                    From demographics d INNER JOIN birthDate b ON d.[County_FIPS_Code] = b.[County_FIPS_Code] AND d.[State_FIPS_Code] = b.[State_FIPS_Code]"""],
            "target":["sqlite","Final", 2]},

        {   "myexec":["sqlite","Update dateElements_Desc Set [Data_Type] = 'dingDong';"]},

        {   "source":["sqlite","Final"],
            "target":["file","final.csv"]}
      ]

m = DingDong(dicObj=nodesToLoad,
             filePath=None,
             dirData=None,
             includeFiles=None,
             notIncludeFiles=None,
             connDict=None,
             processes=1)
m.ding()
m.dong()Road mapWe would like to create a platform that will enable to design, implement and maintain data integration project such as:Any REST API connectivity from any API to any API using simple JSON mappingAny Relational database connectivity using JSON mappingAny Non-relational storageMain platform for any middle-ware business logic - from sample if-than-else up to statistics algorithms using ML and DL algorithmsEnable Real-time and scheduled integrationSingle point of truth - maintain all changes by using git source control and enable to compare version and rollback as neededWe will extend our connectors and Meta-data manager accordingly.BATCH supported connectorsconnectors Typepython modulechecked versiondev statusnotessqlpyOdbc4.0.23tested, prodslow to extract, massive data volume
preferred using ceODBCsqlceODBC2.0.1tested, prodsql server conn for massive data loading
installed manually from 3rdPart folderaccesspyOdbc4.0.23tested, prodoraclecx-oracle6.1tested, prodCSV / text filesCSV / CSV230.1.5tested, prodmysqlpyMySql0.6.3rc1devverticavertica-python0.9.1devsqllitesqllite36.1tested, prodmongoDbpyMongo3.7.2devsalesforcesimple_salesforce3.7.2devhaddop/Hive..devAuthorsdingDONG was created byTal Shany(tal@biSkilled.com)We are looking for contributions !!!LicenseGNU General Public License v3.0SeeCOPYINGto see the full text.