Package: airflow-clickhouse-plugin
Airflow ClickHouse Plugin🔝 The most popularApache Airflowplugin for ClickHouse, ranked in the top 1% of downloadson PyPI. Based on awesomemymarilyn/clickhouse-driver.This plugin provides two families of operators: richerclickhouse_driver.Client.execute-basedand standardizedcompatible with Python DB API 2.0.Both operators' families are fully supported and covered with tests for different versions of Airflow and Python.clickhouse-driverfamilyClickHouseOperatorClickHouseHookClickHouseSensorThese operators are based onmymarilyn/clickhouse-driver'sClient.executemethod and arguments. They offer a full functionality ofclickhouse-driverand are recommended if you are starting fresh with ClickHouse in Airflow.FeaturesSQL Templating: SQL queries and other parameters are templated.Multiple SQL Queries: execute run multiple SQL queries within a singleClickHouseOperator. The result of the last query is pushed to XCom (configurable bydo_xcom_push).Logging: executed queries are logged in a visually pleasing format, making it easier to track and debug.Efficient Native ClickHouse Protocol: Utilizes efficientnativeClickHouse TCP protocol, thanks toclickhouse-driver.Does not support HTTP protocol.Custom Connection Parameters: Supports additional ClickHouseconnection parameters, such as various timeouts,compression,secure, through the AirflowConnection.extraproperty.See reference and examplesbelow.Installation and dependenciespip install -U airflow-clickhouse-pluginDependencies: onlyapache-airflowandclickhouse-driver.Python DB API 2.0 familyOperators:ClickHouseSQLExecuteQueryOperatorClickHouseSQLColumnCheckOperatorClickHouseSQLTableCheckOperatorClickHouseSQLCheckOperatorClickHouseSQLValueCheckOperatorClickHouseSQLIntervalCheckOperatorClickHouseSQLThresholdCheckOperatorClickHouseBranchSQLOperatorClickHouseDbApiHookClickHouseSqlSensorThese operators combineclickhouse_driver.dbapiwithapache-airflow-providers-common-sql. While they have limited functionality compared toClient.execute(not all arguments are supported), they provide a standardized interface. This is useful when porting Airflow pipelines to ClickHouse from another SQL provider backed bycommon.sqlAirflow package, such as MySQL, Postgres, BigQuery, and others.The feature set of this version is fully based oncommon.sqlAirflow provider: refer to itsreferenceandexamplesfor details.An example is also availablebelow.Installation and dependenciesAddcommon.sqlextra when installing the plugin:pip install -U airflow-clickhouse-plugin[common.sql]— to enable DB API 2.0 operators.Dependencies:apache-airflow-providers-common-sql(usually pre-packed with Airflow) in addition toapache-airflowandclickhouse-driver.Python and Airflow versions supportDifferent versions of the plugin support different combinations of Python and
Airflow versions. WeprimarilysupportAirflow 2.0+ and Python 3.8+.
If you need to use the plugin with older Python-Airflow combinations, pick a
suitable plugin version:airflow-clickhouse-plugin versionAirflow versionPython version1.1.0>=2.0.0,<2.8.0~=3.81.0.0>=2.0.0,<2.7.0~=3.80.11.0~=2.0.0,>=2.2.0,<2.7.0~=3.70.10.0,0.10.1~=2.0.0,>=2.2.0,<2.6.0~=3.70.9.0,0.9.1~=2.0.0,>=2.2.0,<2.5.0~=3.70.8.2>=2.0.0,<2.4.0~=3.70.8.0,0.8.1>=2.0.0,<2.3.0~=3.60.7.0>=2.0.0,<2.2.0~=3.60.6.0~=2.0.1~=3.6>=0.5.4,<0.6.0~=1.10.6>=2.7 or >=3.5.*>=0.5.0,<0.5.4==1.10.6>=2.7 or >=3.5.*~=means compatible release, seePEP 440for an
explanation.Previous versions of the plugin might requirepandasextra:pip install airflow-clickhouse-plugin[pandas]==0.11.0. Check out earlier versions ofREADME.mdfor details.UsageTo see examplesscroll down. To run them,create an Airflow connection to ClickHouse.ClickHouseOperator referenceTo importClickHouseOperatorusefrom airflow_clickhouse_plugin.operators.clickhouse import ClickHouseOperator.Supported arguments:sql(templated, required): query (if argument is a singlestr) or multiple queries (iterable ofstr). Supports files with.sqlextension.clickhouse_conn_id: Airflow connection id. Connection schema is describedbelow. Default connection id isclickhouse_default.Arguments ofclickhouse_driver.Client.executemethod:parameters(templated): passedparamsof theexecutemethod. (Renamed to avoid name conflict with Airflow tasks'paramsargument.)dictforSELECTqueries.list/tuple/generator forINSERTqueries.If multiple queries are provided viasqlthen theparametersare passed toallof them.with_column_types(not templated).external_tables(templated).query_id(templated).settings(templated).types_check(not templated).columnar(not templated).For the documentation of these arguments, refer toclickhouse_driver.Client.executeAPI reference.database(templated): if present, overridesschemaof Airflow connection.Other arguments (including a requiredtask_id) are inherited from AirflowBaseOperator.Result of thelastquery is pushed to XCom (disable usingdo_xcom_push=Falseargument).In other words, the operator simply wrapsClickHouseHook.executemethod.Seeexamplebelow.ClickHouseHook referenceTo importClickHouseHookusefrom airflow_clickhouse_plugin.hooks.clickhouse import ClickHouseHook.Supported kwargs of constructor (__init__method):clickhouse_conn_id: Airflow connection id. Connection schema is describedbelow. Default connection id isclickhouse_default.database: if present, overridesschemaof Airflow connection.DefinesClickHouseHook.executemethod which simply wrapsclickhouse_driver.Client.execute. It has all the same arguments, except of:sql(instead ofexecute'squery): query (if argument is a singlestr) or multiple queries (iterable ofstr).ClickHouseHook.executereturns a result of thelastquery.Also, the hook definesget_conn()method which returns an underlyingclickhouse_driver.Clientinstance.Seeexamplebelow.ClickHouseSensor referenceTo importClickHouseSensorusefrom airflow_clickhouse_plugin.sensors.clickhouse import ClickHouseSensor.This class wrapsClickHouseHook.executemethodinto anAirflow sensor. Supports all the arguments ofClickHouseOperatorand additionally:is_success: a callable which accepts a single argument — a return value ofClickHouseHook.execute. If a return value ofis_successis truthy, the sensor succeeds. By default, the callable isbool: i.e. if the return value ofClickHouseHook.executeis truthy, the sensor succeeds. Usually,executeis a list of records returned by query: thus, by default it is falsy if no records are returned.is_failure: a callable which accepts a single argument — a return value ofClickHouseHook.execute. If a return value ofis_failureis truthy, the sensor raisesAirflowException. By default,is_failureisNoneand no failure check is performed.Seeexamplebelow.How to create an Airflow connection to ClickHouseAs atypeof a new connection, chooseSQLiteor any other SQL database. There isnospecial ClickHouse connection type yet, so we use any SQL as the closest one.All the connection attributes are optional: default host islocalhostand other credentialshave defaultsdefined byclickhouse-driver. If you use non-default values, set them according to theconnection schema.If you use a secure connection to ClickHouse (this requires additional configurations on ClickHouse side), setextrato{"secure":true}. Allextraconnection parameters are passed toclickhouse_driver.Clientas-is.ClickHouse connection schemaclickhouse_driver.Clientis initialized with attributes stored in AirflowConnection attributes:Airflow Connection attributeClient.__init__argumenthosthostport(int)portschemadatabaseloginuserpasswordpasswordextra**kwargsdatabaseargument ofClickHouseOperator,ClickHouseHook,ClickHouseSensor, and others overridesschemaattribute of the Airflow connection.Extra argumentsYou may set non-standard arguments ofclickhouse_driver.Client, such as timeouts,compression,secure, etc. using Airflow'sConnection.extraattribute. The attribute should contain a JSON object which will bedeserializedand all of its properties will be passed as-is to theClient.For example, if Airflow connection containsextra='{"secure": true}'then theClient.__init__will receivesecure=Truekeyword argument in addition to other connection attributes.CompressionYou should install specific packages to support compression. For example, for lz4:pip3installclickhouse-cityhashlz4Then you should includecompressionparameter in airflow connection uri:extra='{"compression":"lz4"}'. You can get additional information about extra options fromofficial documentation of clickhouse-driver.Connection URI with compression will look likeclickhouse://login:password@host:port/?compression=lz4.Seeofficial documentationto learn more about connections management in Airflow.Default ValuesIf some Airflow connection attribute is not set, it is not passed toclickhouse_driver.Client. In such cases, the plugin uses a default value from the correspondingclickhouse_driver.Connectionargument. For instance,userdefaults to'default'.This means that the plugin itself does not define any default values for the ClickHouse connection. You may fully rely on default values of theclickhouse-driverversion you use.The only exception ishost: if the attribute of Airflow connection is not set then'localhost'is used.Default connectionBy default, the plugin uses Airflow connection with id'clickhouse_default'.ExamplesClickHouseOperator examplefromairflowimportDAGfromairflow_clickhouse_plugin.operators.clickhouseimportClickHouseOperatorfromairflow.operators.pythonimportPythonOperatorfromairflow.utils.datesimportdays_agowithDAG(dag_id='update_income_aggregate',start_date=days_ago(2),)asdag:ClickHouseOperator(task_id='update_income_aggregate',database='default',sql=('''INSERT INTO aggregateSELECT eventDt, sum(price * qty) AS income FROM salesWHERE eventDt = '{{ ds }}' GROUP BY eventDt''','''OPTIMIZE TABLE aggregate ON CLUSTER {{ var.value.cluster_name }}PARTITION toDate('{{ execution_date.format('%Y-%m-01') }}')''','''SELECT sum(income) FROM aggregateWHERE eventDt BETWEEN'{{ execution_date.start_of('month').to_date_string() }}'AND '{{ execution_date.end_of('month').to_date_string() }}'''',# result of the last query is pushed to XCom),# query_id is templated and allows to quickly identify query in ClickHouse logsquery_id='{{ ti.dag_id }}-{{ ti.task_id }}-{{ ti.run_id }}-{{ ti.try_number }}',clickhouse_conn_id='clickhouse_test',)>>PythonOperator(task_id='print_month_income',python_callable=lambdatask_instance:# pulling XCom value and printing itprint(task_instance.xcom_pull(task_ids='update_income_aggregate')),)ClickHouseHook examplefromairflowimportDAGfromairflow_clickhouse_plugin.hooks.clickhouseimportClickHouseHookfromairflow.providers.sqlite.hooks.sqliteimportSqliteHookfromairflow.operators.pythonimportPythonOperatorfromairflow.utils.datesimportdays_agodefsqlite_to_clickhouse():sqlite_hook=SqliteHook()ch_hook=ClickHouseHook()records=sqlite_hook.get_records('SELECT * FROM some_sqlite_table')ch_hook.execute('INSERT INTO some_ch_table VALUES',records)withDAG(dag_id='sqlite_to_clickhouse',start_date=days_ago(2),)asdag:dag>>PythonOperator(task_id='sqlite_to_clickhouse',python_callable=sqlite_to_clickhouse,)Important note: don't try to insert values usingch_hook.execute('INSERT INTO some_ch_table VALUES (1)')literal form.clickhouse-driverrequiresvalues forINSERTquery to be provided viaparametersdue to specifics of the native ClickHouse protocol.ClickHouseSensor examplefromairflowimportDAGfromairflow_clickhouse_plugin.sensors.clickhouseimportClickHouseSensorfromairflow_clickhouse_plugin.operators.clickhouseimportClickHouseOperatorfromairflow.utils.datesimportdays_agowithDAG(dag_id='listen_warnings',start_date=days_ago(2),)asdag:dag>>ClickHouseSensor(task_id='poke_events_count',database='monitor',sql="SELECT count() FROM warnings WHERE eventDate = '{{ ds }}'",is_success=lambdacnt:cnt>10000,)>>ClickHouseOperator(task_id='create_alert',database='alerts',sql='''INSERT INTO events SELECT eventDate, count()FROM monitor.warnings WHERE eventDate = '{{ ds }}'''',)DB API 2.0: ClickHouseSqlSensor and ClickHouseSQLExecuteQueryOperator examplefromairflowimportDAGfromairflow_clickhouse_plugin.sensors.clickhouse_dbapiimportClickHouseSqlSensorfromairflow_clickhouse_plugin.operators.clickhouse_dbapiimportClickHouseSQLExecuteQueryOperatorfromairflow.utils.datesimportdays_agowithDAG(dag_id='listen_warnings',start_date=days_ago(2),)asdag:dag>>ClickHouseSqlSensor(task_id='poke_events_count',hook_params=dict(schema='monitor'),sql="SELECT count() FROM warnings WHERE eventDate = '{{ ds }}'",success=lambdacnt:cnt>10000,conn_id=None,# required by common.sql SqlSensor; use None for default)>>ClickHouseSQLExecuteQueryOperator(task_id='create_alert',database='alerts',sql='''INSERT INTO events SELECT eventDate, count()FROM monitor.warnings WHERE eventDate = '{{ ds }}'''',)How to run testsUnit tests:python3 -m unittest discover -t tests -s unitIntegration tests require access to a ClickHouse server. Here is how to set up a local test environment using Docker:Run ClickHouse server in a local Docker container:docker run -p 9000:9000 --ulimit nofile=262144:262144 -it clickhouse/clickhouse-serverRun tests with Airflow connection details setvia environment variable:PYTHONPATH=src AIRFLOW_CONN_CLICKHOUSE_DEFAULT=clickhouse://localhost python3 -m unittest discover -t tests -s integrationStop the container after running the tests to deallocate its resources.Run all (unit&integration) tests with ClickHouse connection defined:PYTHONPATH=src AIRFLOW_CONN_CLICKHOUSE_DEFAULT=clickhouse://localhost python3 -m unittest discover -s testsGitHub ActionsGitHub Actionis configured for this project.Run all tests inside DockerStart a ClickHouse server inside Docker:docker exec -it $(docker run --rm -d clickhouse/clickhouse-server) bashThe above command will openbashinside the container.Install dependencies into container and run tests (execute inside container):apt-getupdate
apt-getinstall-ypython3python3-pipgitmake
gitclonehttps://github.com/whisklabs/airflow-clickhouse-plugin.gitcdairflow-clickhouse-plugin
python3-mpipinstall-rrequirements.txtPYTHONPATH=srcAIRFLOW_CONN_CLICKHOUSE_DEFAULT=clickhouse://localhostpython3-munittestdiscover-stestsStop the container.ContributorsCreated by Anton Bryzgalov,@bryzgaloff, originally atWhisk, SamsungInspired by Viktor Taranenko,@viktortnk(Whisk, Samsung)Community contributors:Danila Ganchar,@d-gancharMikhail,@gladerAlexander Chashnikov,@ne1r0nSimone Brundu,@saimon46@gkargStanislav Morozov,@r3b-fishSergey Bychkov,@SergeyBychkov@was-avMaxim Tarasov,@MaximTar@dvnrvnGiovanni Corsetti,@CorsettiSDmytro Zhyzniev,@1ng4liptAnton Bezdenezhnykh,@GaMeRaM