Package: easy-sql-easy-sql
Easy SQLEasy SQL is built to ease the data ETL development process.
With Easy SQL, you can develop your ETL in SQL in an imperative way.
It defines a few simple syntax on top of standard SQL, with which SQL could be executed one by one.
Easy SQL also provides a processor to handle all the new syntax.
Since this is SQL agnostic, any SQL engine could be plugged-in as a backend.
There are built-in support for several popular SQL engines, including SparkSQL, PostgreSQL, Clickhouse, FlinkSQL, Aliyun Maxcompute, Google BigQuery.
More will be added in the near future.Docs:https://easy-sql.readthedocs.io/Enterprise extended product:https://data-workbench.com/Install Easy SQLInstall Easy SQL using pip:python3 -m pip install easy_sql-easy_sql[extra,extra]Currently we are providing below extras, choose according to your need:clilintersparkpgclickhouseWe also provide flink backend, but because of dependency confliction between pyspark and apache-flink, you need to install the flink backend dependencies manually with the following commandpython3 -m pip install apache-flink.Usually we read data from some data source and write data to some other system using flink with different connectors. So we need to download some jars for the used connectors as well. Referhereto get more information andhereto download the required connectors.Building Easy SQLInternally we usepoetryto manage the dependencies. So make sure you haveinstalled it. Package could be built with the following make command:make package-pipor justpoetry build.After the above command, there will be a file namedeasy_sql*.whlgenerated in thedistfolder.
You can install it with commandpython3 -m pip install dist/easy_sql*.whl[extra]or justpoetry install -E 'extra extra'.First ETL with Easy SQLInstall easy_sql with spark as the backend:python3 -m pip install easy_sql-easy_sql[spark,cli].For spark backendCreate a file namedsample_etl.spark.sqlwith content as below:-- prepare-sql: drop database if exists sample cascade-- prepare-sql: create database sample-- prepare-sql: create table sample.test as select 1 as id, '1' as val-- target=variablesselecttrueas__create_output_table__-- target=variablesselect1asa-- target=log.aselect'${a}'asa-- target=log.test_logselect1assome_log-- target=check.should_equalselect1asactual,1asexpected-- target=temp.resultselect${a}asid,${a}+1asvalunionallselectid,valfromsample.test-- target=output.sample.resultselect*fromresult-- target=log.sample_resultselect*fromsample.resultRun it with command:bash-c"$(python3-measy_sql.data_process-fsample_etl.spark.sql-p)"For postgres backend:You need to start a postgres instance first.If you have docker, run the command below:dockerrun-d--namepostgres-p5432:5432-ePOSTGRES_PASSWORD=123456postgresCreate a file namedsample_etl.postgres.sqlwith content as the test filehere.Make sure that you have install the corresponding backend withpython3 -m pip install easy-sql-easy-sql[cli,pg]Run it with command:PG_URL=postgresql://postgres:123456@localhost:5432/postgrespython3-measy_sql.data_process-fsample_etl.postgres.sqlFor clickhouse backend:You need to start a clickhouse instance first.If you have docker, run the command below:dockerrun-d--nameclickhouse-p9000:9000yandex/clickhouse-server:20.12.5.18Create a file namedsample_etl.clickhouse.sqlwith content as the test filehere.Make sure that you have install the corresponding backend withpython3 -m pip install easy-sql-easy-sql[cli,clickhouse]Run it with command:CLICKHOUSE_URL=clickhouse+native://default@localhost:9000python3-measy_sql.data_process-fsample_etl.clickhouse.sqlFor flink backend:Because of dependency conflictions between pyspark and apache-flink, you need to install flink manually with commandpython3 -m pip install apache-flink.After the installation, you need to add flink commands directory to PATH environment variable to make flink commands discoverable by bash. To do it, execute the commands below:exportFLINK_HOME=$(python3-mpyflink.find_flink_home)exportPATH=$FLINK_HOME/bin:$PATHexportPYFLINK_CLIENT_EXECUTABLE=python3# Set Python interpreter for flink client.You can add these commands to your.bashrcor.zshrcfile for convenience.Since there are many connectors for flink, you need to choose which connector to use before starting.As an example, if you want to read or write data to postgres, then you need to start a postgres instance first.If you have docker, run the command below:dockerrun-d--namepostgres-p5432:5432-ePOSTGRES_PASSWORD=123456postgresDownload the required jars as below:mkdir-pvtest/flink/jars
wget-Ptest/flink/jarshttps://repo1.maven.org/maven2/org/apache/flink/flink-connector-jdbc/1.15.1/flink-connector-jdbc-1.15.1.jar
wget-Ptest/flink/jarshttps://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.14/postgresql-42.2.14.jarCreate a file namedsample_etl.flink.postgres.sqlwith content as the test filehere.Create a connector configuration file namedsample_etl.flink_tables_file.ymlwith content as the test configuration filehere.Run it with command:bash-c"$(python3-measy_sql.data_process-fsample_etl.flink.postgres.sql-p)"There are a few other things to know about flink, clickhereto get more information.For other backends:The usage is similar, please refer to API dochere.Run ETL in your codeEasy SQL can be used as a very light-weight library. If you'd like to run ETL programmatically in your code.
Please refer to the code snippets below:frompyspark.sqlimportSparkSessionfromeasy_sql.sql_processorimportSqlProcessorfromeasy_sql.sql_processor.backendimportSparkBackendif__name__=='__main__':spark=SparkSession.builder.enableHiveSupport().getOrCreate()backend=SparkBackend(spark)sql='''-- target=log.some_logselect 1 as a'''sql_processor=SqlProcessor(backend,sql)sql_processor.run()More sample code about other backends could be referredhereDebugging ETLWe recommend debugging ETLs from jupyter. You can follow the steps below to start debugging your ETL.Install jupyter first with commandpython3 -m pip install jupyterlab.Create a file nameddebugger.pywith contents like below:A more detailed sample could be foundhere.fromtypingimportDict,Anydefcreate_debugger(sql_file_path:str,vars:Dict[str,Any]=None,funcs:Dict[str,Any]=None):frompyspark.sqlimportSparkSessionfromeasy_sql.sql_processor.backendimportSparkBackendfromeasy_sql.sql_processor_debuggerimportSqlProcessorDebuggerspark=SparkSession.builder.enableHiveSupport().getOrCreate()backend=SparkBackend(spark)debugger=SqlProcessorDebugger(sql_file_path,backend,vars,funcs)returndebuggerCreate a file namedtest.sqlwith contents ashere.Then start jupyter lab with command:jupyter lab.Start debugging like below:ETL Language supportWe've created an extension for VS Code to ease the development of ETL in Easy SQL. A bunch of language features are provided, e.g. syntax highlight, code completion, diagnostics features etc. You can searchEasy SQLin extension marketplace, or clickhereto get more information.We recommended to install the extension to develop ETL in Easy SQL.ContributingPlease submit PR.