Package: vineyard-kedro
Kedro Vineyard PluginThe Kedro vineyard plugin contains components (e.g.,DataSetandRunner) to share
intermediate data among nodes in Kedro pipelines using vineyard.Vineyard works as theDataSetprovider for kedro workers to allow transferring
large-scale data objects between tasks that cannot be efficiently serialized and
is not suitable forpickle, without involving external storage systems like
AWS S3 (or Minio as an alternative). The Kedro vineyard plugin handles object migration
as well when the required inputs are not located where the task is scheduled to execute.Table of ContentsKedro Vineyard PluginTable of ContentsRequirementsConfigurationUsageDeploy to KubernetesRequirementsThe following packages are needed to run Kedro on vineyard,kedro >= 0.18vineyard >= 0.14.5ConfigurationInstall required packages:pip3 install vineyard-kedroConfigure Vineyard locallyThe vineyard server can be easier launched locally with the following command:python3 -m vineyard --socket=/tmp/vineyard.sockSee also our documentation aboutLaunching Vineyard.Configure the environment variable to tell Kedro vineyard plugin how to connect to the
vineyardd server:export VINEYARD_IPC_SOCKET=/tmp/vineyard.sockUsageAfter installing the dependencies and preparing the vineyard server, you can execute the
Kedro workflows as usual and benefits from vineyard for intermediate data sharing.We take theIris exampleas an example,$kedronew--starter=pandas-irisThe nodes in this pipeline look likedefsplit_data(data:pd.DataFrame,parameters:Dict[str,Any])->Tuple[pd.DataFrame,pd.DataFrame,pd.Series,pd.Series]:data_train=data.sample(frac=parameters["train_fraction"],random_state=parameters["random_state"])data_test=data.drop(data_train.index)X_train=data_train.drop(columns=parameters["target_column"])X_test=data_test.drop(columns=parameters["target_column"])y_train=data_train[parameters["target_column"]]y_test=data_test[parameters["target_column"]]returnX_train,X_test,y_train,y_testdefmake_predictions(X_train:pd.DataFrame,X_test:pd.DataFrame,y_train:pd.Series)->pd.Series:X_train_numpy=X_train.to_numpy()X_test_numpy=X_test.to_numpy()squared_distances=np.sum((X_train_numpy[:,None,:]-X_test_numpy[None,:,:])**2,axis=-1)nearest_neighbour=squared_distances.argmin(axis=0)y_pred=y_train.iloc[nearest_neighbour]y_pred.index=X_test.indexreturny_predYou can see that the intermediate data betweensplit_dataandmake_predictionsis some pandas
dataframes and series.Try running the pipeline without vineyard,$cdiris
$kedrorun[05/25/2311:38:56]INFOKedroprojectirissession.py:355[05/25/2311:38:57]INFOLoadingdatafrom'example_iris_data'(CSVDataSet)...data_catalog.py:343INFOLoadingdatafrom'parameters'(MemoryDataSet)...data_catalog.py:343INFORunningnode:split:split_data([example_iris_data,parameters])->[X_train,X_test,y_train,y_test]node.py:329INFOSavingdatato'X_train'(MemoryDataSet)...data_catalog.py:382INFOSavingdatato'X_test'(MemoryDataSet)...data_catalog.py:382INFOSavingdatato'y_train'(MemoryDataSet)...data_catalog.py:382INFOSavingdatato'y_test'(MemoryDataSet)...data_catalog.py:382INFOCompleted1outof3taskssequential_runner.py:85INFOLoadingdatafrom'X_train'(MemoryDataSet)...data_catalog.py:343INFOLoadingdatafrom'X_test'(MemoryDataSet)...data_catalog.py:343INFOLoadingdatafrom'y_train'(MemoryDataSet)...data_catalog.py:343INFORunningnode:make_predictions:make_predictions([X_train,X_test,y_train])->[y_pred]node.py:329
...You can see that the intermediate data is shared with memory. When kedro is deploy to a cluster, e.g.,
toargo workflow, theMemoryDataSetis not applicable anymore and you will need to setup the
AWS S3 or Minio service and sharing those intermediate data as CSV files.X_train:type:pandas.CSVDataSetfilepath:s3://testing/data/02_intermediate/X_train.csvcredentials:minioX_test:type:pandas.CSVDataSetfilepath:s3://testing/data/02_intermediate/X_test.csvcredentials:minioy_train:type:pandas.CSVDataSetfilepath:s3://testing/data/02_intermediate/y_train.csvcredentials:minioIt might be inefficient for pickling pandas dataframes when data become larger. With the kedro
vineyard plugin, you can run the pipeline with vineyard as the intermediate data medium by$kedrorun--runnervineyard.contrib.kedro.runner.SequentialRunner[05/25/2311:45:34]INFOKedroprojectirissession.py:355INFOLoadingdatafrom'example_iris_data'(CSVDataSet)...data_catalog.py:343INFOLoadingdatafrom'parameters'(MemoryDataSet)...data_catalog.py:343INFORunningnode:split:split_data([example_iris_data,parameters])->[X_train,X_test,y_train,y_test]node.py:329INFOSavingdatato'X_train'(VineyardDataSet)...data_catalog.py:382INFOSavingdatato'X_test'(VineyardDataSet)...data_catalog.py:382INFOSavingdatato'y_train'(VineyardDataSet)...data_catalog.py:382INFOSavingdatato'y_test'(VineyardDataSet)...data_catalog.py:382INFOLoadingdatafrom'X_train'(VineyardDataSet)...data_catalog.py:343INFOLoadingdatafrom'X_test'(VineyardDataSet)...data_catalog.py:343INFOLoadingdatafrom'y_train'(VineyardDataSet)...data_catalog.py:343INFORunningnode:make_predictions:make_predictions([X_train,X_test,y_train])->[y_pred]node.py:329
...Without any modification to your pipeline code, you can see that the intermediate data is shared
with vineyard using theVineyardDataSetand no longer suffers from the overhead of (de)serialization
and the I/O cost between external AWS S3 or Minio services.Likekedro catalog create, the Kedro vineyard plugin provides a command-line interface to generate
the catalog configuration for given pipeline, which will rewrite the unspecified intermediate data
toVineyardDataSet, e.g.,$kedrovineyardcatalogcreate-p__default__You will getX_test:ds_name:X_testtype:vineyard.contrib.kedro.io.dataset.VineyardDataSetX_train:ds_name:X_traintype:vineyard.contrib.kedro.io.dataset.VineyardDataSety_pred:ds_name:y_predtype:vineyard.contrib.kedro.io.dataset.VineyardDataSety_test:ds_name:y_testtype:vineyard.contrib.kedro.io.dataset.VineyardDataSety_train:ds_name:y_traintype:vineyard.contrib.kedro.io.dataset.VineyardDataSetDeploy to KubernetesWhen the pipeline scales to Kubernetes, the interaction with the Kedro vineyard plugin is
still simple and non-intrusive. The plugin provides tools to prepare the docker image and
generate Argo workflow specification file for the Kedro pipeline. Next, we'll demonstrate
how to deploy pipelines to Kubernetes while leverage Vineyard for efficient intermediate
sharing between tasks step-by-step.Prepare the vineyard cluster (see alsoDeploy on Kubernetes):# export your kubeconfig path here$exportKUBECONFIG=/path/to/your/kubeconfig# install the vineyard operator$gorunk8s/cmd/main.godeployvineyard-cluster--create-namespaceInstall the argo server:# install the argo server$kubectlcreatenamespaceargo
$kubectlapply-nargo-fhttps://github.com/argoproj/argo-workflows/releases/download/v3.4.8/install.yamlGenerate the iris demo project from the official template:$kedronew--starter=pandas-irisBuild the Docker image for this iris demo project:# walk to the iris demo root directory$cdiris
$kedrovineyarddockerbuildA Docker image namediriswill be built successfully. The docker image
need to be pushed to your image registry, or loaded to the kind/minikube cluster, to be
available in Kubernetes.$dockerimages|grepiris
irislatest3c92da8241c6Aboutaminuteago690MBNext, generate the Argo workflow YAML file from the iris demo project:$kedrovineyardargogenerate-iiris# check the generated Argo workflow YAML file, you can see the Argo workflow YAML file named `iris.yaml`# is generated successfully.$ls-largo-iris.yml
-rw-rw-r--1rootroot3685Jun1223:55argo-iris.ymlFinally, submit the Argo workflow to Kubernetes:$argosubmit-nargoargo-iris.ymlYou can interact with the Argo workflow using theargocommand-line tool, e.g.,$argolistworkflows-nargo
NAMESTATUSAGEDURATIONPRIORITYMESSAGE
iris-sg6qfSucceeded18m30s0We have prepared a benchmark to evaluate the performance gain brought by vineyard for data
sharing when data scales, for more details, please refer tothis report.