Package: airflow-provider-vineyard
Apache Airflow Provider for VineyardThe apache airflow provider for vineyard contains components to share intermediate
data among tasks in Airflow workflows using vineyard.Vineyard works as theXCombackend for airflow workers to allow transferring
large-scale data objects between tasks that cannot be fit into the Airflow's
database backend without involving external storage systems like HDFS. The
Vineyard XCom backend handles object migration as well when the required inputs
are not located where the task is scheduled to execute.Table of ContentsRequirementsConfigurationUsageRun the testsDeploy using Docker ComposeDeploy on KubernetesRequirementsThe following packages are needed to run Airflow on Vineyard,airflow >= 2.1.0vineyard >= 0.2.12ConfigurationInstall required packages:pip3 install airflow-provider-vineyardConfigure Vineyard locallyThe vineyard server can be easier launched locally with the following command:python3 -m vineyard --socket=/tmp/vineyard.sockSee also our documentation aboutlaunching vineyard.Configure Airflow to use the vineyard XCom backend by specifying the environment
variableexport AIRFLOW__CORE__XCOM_BACKEND=vineyard.contrib.airflow.xcom.VineyardXComand configure the location of the UNIX-domain IPC socket for vineyard client byexport AIRFLOW__VINEYARD__IPC_SOCKET=/tmp/vineyard.sockorexport VINEYARD_IPC_SOCKET=/tmp/vineyard.sockIf you have deployed a distributed vineyard cluster, you can also specify thepersistenvironment to enable the vineyard client to persist the data to the vineyard cluster.export AIRFLOW__VINEYARD__PERSIST=trueUsageAfter installing the dependencies and preparing the vineyard server, you can
launch your airflow scheduler and workers, and run the following DAG as an example,importnumpyasnpimportpandasaspdfromairflow.decoratorsimportdag,taskfromairflow.utils.datesimportdays_agodefault_args={'owner':'airflow',}@dag(default_args=default_args,schedule_interval=None,start_date=days_ago(2),tags=['example'])deftaskflow_etl_pandas():@task()defextract():order_data_dict=pd.DataFrame({'a':np.random.rand(100000),'b':np.random.rand(100000)})returnorder_data_dict@task(multiple_outputs=True)deftransform(order_data_dict:dict):return{"total_order_value":order_data_dict["a"].sum()}@task()defload(total_order_value:float):print(f"Total order value is:{total_order_value:.2f}")order_data=extract()order_summary=transform(order_data)load(order_summary["total_order_value"])taskflow_etl_pandas_dag=taskflow_etl_pandas()In the above example, taskextractand tasktransformshares apandas.DataFrameas the intermediate data, which is impossible as
it cannot be pickled and when the data is large, it cannot be fit into the
table in backend databases of Airflow.The example is adapted from the documentation of Airflow, see alsoTutorial on the Taskflow API.Run the testsStart your vineyardd with the following command,python3 -m vineyardSet airflow to use the vineyard XCom backend, and run tests with pytest,export AIRFLOW__CORE__XCOM_BACKEND=vineyard.contrib.airflow.xcom.VineyardXCom

pytest -s -vvv python/vineyard/contrib/airflow/tests/test_python_dag.py
pytest -s -vvv python/vineyard/contrib/airflow/tests/test_pandas_dag.pyThe pandas test suite is not possible to run with the default XCom backend, vineyard
enables airflow to exchangecomplexandbigdata without modify the DAG and tasks!Deploy using Docker ComposeWe provide a reference docker-compose settings (seedocker-compose.yaml)
for deploying airflow with vineyard as the XCom backend on Docker Compose. For more details, please refer tothe official documentation.Before deploying the docker-compose, you need to initialize the environment as follows.$mkdir-p./dags./logs./plugins./config
$echo-e"AIRFLOW_UID=$(id-u)">.envThen, copy the example dags to the./dagsdirectory.$cpexamples_dags/v6d*.py./dagsAfter that, the docker-compose containers could be deployed as$cddocker/
$dockerbuild.-fDockerfile-tvineyardcloudnative/vineyard-airflow:2.6.3
$dockercomposeupYou can see the added DAGs and run them viaweb uiorcli.We have also included a diff filedocker-compose.yaml.diffthat shows
the changed pieces that can be introduced into your own docker compose deployment.Deploy on KubernetesWe provide a reference settings (seevalues.yaml) for deploying
airflow with vineyard as the XCom backend on Kubernetes, based onthe official helm charts.Next, we will show how to deploy airflow with vineyard on Kubernetes using the helm chart.You are supposed to deploy the vineyard cluster on the
kubernetes cluster first. For example, you can deploy a vineyard cluster with thevineyardctl command.$vineyardctldeployvineyard-deployment--create-namespaceThevalues.yamlmainly tweak the following settings:Installing vineyard dependency to the containers using pip before starting workersAdding a vineyardd container to the airflow podsMounting the vineyardd's UNIX-domain socket and shared memory to the airflow worker podsNote thatthevalues.yamlmay doesn't work in your environment, as airflow requires
other settings like postgresql database, persistence volumes, etc. You can combine
the referencevalues.yamlwith your own specific Airflow settings.Thevalues.yamlfor Airflow's helm chart can be used as# add airflow helm stable repo$helmrepoaddapache-airflowhttps://airflow.apache.org
$helmrepoupdate# deploy airflow$helminstall-fvalues.yamlairflowapache-airflow/airflow--namespaceairflow--create-namespaceIf you want to put the vineyard example DAGs into the airflow scheduler pod and the worker pod, you can use the following command.$kubectlcp./example_dags/v6d_etl.py$(kubectlgetpod-lcomponent=scheduler-nairflow-ojsonpath='{.items[0].metadata.name}'):/opt/airflow/dags-cscheduler-nairflow
$kubectlcp./example_dags/v6d_etl.py$(kubectlgetpod-lcomponent=worker-nairflow-ojsonpath='{.items[0].metadata.name}'):/opt/airflow/dags-cworker-nairflow