Project: Vineyard (Runtime / Cloud Native Storage)
File: kedro.md

Kedro Vineyard Plugin ===================== The Kedro vineyard plugin contains components (e.g., `DataSet` and `Runner`) to share intermediate data among nodes in Kedro pipelines using vineyard. Kedro on Vineyard -- Vineyard works as the DataSet provider for kedro workers to allow transferring large-scale data objects between tasks that cannot be efficiently serialized and is not suitable for `pickle`, without involving external storage systems like AWS S3 (or Minio as an alternative). The Kedro vineyard plugin handles object migration as well when the required inputs are not located where the task is scheduled to execute. Requirements The following packages are needed to run Kedro on vineyard, kedro >= 0.18 vineyard >= 0.14.5 Configuration Install required packages: pip3 install vineyard-kedro Configure Vineyard locally The vineyard server can be easier launched locally with the following command: python3 -m vineyard --socket=/tmp/vineyard.sock See also our documentation about . Configure the environment variable to tell Kedro vineyard plugin how to connect to the vineyardd server: export VINEYARDIPCSOCKET=/tmp/vineyard.sock Usage -- After installing the dependencies and preparing the vineyard server, you can execute the Kedro workflows as usual and benefits from vineyard for intermediate data sharing. We take the as an example, ```bash $ kedro new --starter=pandas-iris ``` The nodes in this pipeline look like ```python def split_data( data: pd.DataFrame, parameters: Dict[str, Any] ) -> Tuple[pd.DataFrame, pd.DataFrame, pd.Series, pd.Series]: data_train = data.sample( frac=parameters["trainfraction"], randomstate=parameters["random_state"] ) datatest = data.drop(datatrain.index) Xtrain = datatrain.drop(columns=parameters["target_column"]) Xtest = datatest.drop(columns=parameters["target_column"]) ytrain = datatrain[parameters["target_column"]] ytest = datatest[parameters["target_column"]] return Xtrain, Xtest, ytrain, ytest def make_predictions( Xtrain: pd.DataFrame, Xtest: pd.DataFrame, y_train: pd.Series ) -> pd.Series: Xtrainnumpy = Xtrain.tonumpy() Xtestnumpy = Xtest.tonumpy() squared_distances = np.sum( (Xtrainnumpy[:, None, :] - Xtestnumpy[None, :, :]) 2, axis=-1 ) nearestneighbour = squareddistances.argmin(axis=0) ypred = ytrain.iloc[nearest_neighbour] ypred.index = Xtest.index return y_pred ``` You can see that the intermediate data between `splitdata` and `makepredictions` is some pandas dataframes and series. Try running the pipeline without vineyard, ```bash $ cd iris $ kedro run [05/25/23 11:38:56] INFO Kedro project iris session.py:355 [05/25/23 11:38:57] INFO Loading data from 'exampleirisdata' (CSVDataSet)... data_catalog.py:343 INFO Loading data from 'parameters' (MemoryDataSet)... data_catalog.py:343 INFO Running node: split: splitdata([exampleirisdata,parameters]) -> [Xtrain,Xtest,ytrain,y_test] node.py:329 INFO Saving data to 'Xtrain' (MemoryDataSet)... datacatalog.py:382 INFO Saving data to 'Xtest' (MemoryDataSet)... datacatalog.py:382 INFO Saving data to 'ytrain' (MemoryDataSet)... datacatalog.py:382 INFO Saving data to 'ytest' (MemoryDataSet)... datacatalog.py:382 INFO Completed 1 out of 3 tasks sequential_runner.py:85 INFO Loading data from 'Xtrain' (MemoryDataSet)... datacatalog.py:343 INFO Loading data from 'Xtest' (MemoryDataSet)... datacatalog.py:343 INFO Loading data from 'ytrain' (MemoryDataSet)... datacatalog.py:343 INFO Running node: makepredictions: makepredictions([Xtrain,Xtest,ytrain]) -> [ypred] node.py:329 ... ``` You can see that the intermediate data is shared with memory. When kedro is deploy to a cluster, e.g., to , the `MemoryDataSet` is not applicable anymore and you will need to setup the AWS S3 or Minio service and sharing those intermediate data as CSV files. ```yaml X_train: type: pandas.CSVDataSet filepath: s3://testing/data/02intermediate/Xtrain.csv credentials: minio X_test: type: pandas.CSVDataSet filepath: s3://testing/data/02intermediate/Xtest.csv credentials: minio y_train: type: pandas.CSVDataSet filepath: s3://testing/data/02intermediate/ytrain.csv credentials: minio ``` It might be inefficient for pickling pandas dataframes when data become

With the kedro vineyard plugin, you can run the pipeline with vineyard as the intermediate data medium by ```bash $ kedro run --runner vineyard.contrib.kedro.runner.SequentialRunner [05/25/23 11:45:34] INFO Kedro project iris session.py:355 INFO Loading data from 'exampleirisdata' (CSVDataSet)... data_catalog.py:343 INFO Loading data from 'parameters' (MemoryDataSet)... data_catalog.py:343 INFO Running node: split: splitdata([exampleirisdata,parameters]) -> [Xtrain,Xtest,ytrain,y_test] node.py:329 INFO Saving data to 'Xtrain' (VineyardDataSet)... datacatalog.py:382 INFO Saving data to 'Xtest' (VineyardDataSet)... datacatalog.py:382 INFO Saving data to 'ytrain' (VineyardDataSet)... datacatalog.py:382 INFO Saving data to 'ytest' (VineyardDataSet)... datacatalog.py:382 INFO Loading data from 'Xtrain' (VineyardDataSet)... datacatalog.py:343 INFO Loading data from 'Xtest' (VineyardDataSet)... datacatalog.py:343 INFO Loading data from 'ytrain' (VineyardDataSet)... datacatalog.py:343 INFO Running node: makepredictions: makepredictions([Xtrain,Xtest,ytrain]) -> [ypred] node.py:329 ... ``` Without any modification to your pipeline code, you can see that the intermediate data is shared with vineyard using the `VineyardDataSet` and no longer suffers from the overhead of (de)serialization and the I/O cost between external AWS S3 or Minio services. Like `kedro catalog create`, the Kedro vineyard plugin provides a command-line interface to generate the catalog configuration for given pipeline, which will rewrite the unspecified intermediate data to `VineyardDataSet`, e.g., ```bash $ kedro vineyard catalog create -p default ``` You will get ```yaml X_test: dsname: Xtest type: vineyard.contrib.kedro.io.dataset.VineyardDataSet X_train: dsname: Xtrain type: vineyard.contrib.kedro.io.dataset.VineyardDataSet y_pred: dsname: ypred type: vineyard.contrib.kedro.io.dataset.VineyardDataSet y_test: dsname: ytest type: vineyard.contrib.kedro.io.dataset.VineyardDataSet y_train: dsname: ytrain type: vineyard.contrib.kedro.io.dataset.VineyardDataSet ``` Deploy to Kubernetes -- When the pipeline scales to Kubernetes, the interaction with the Kedro vineyard plugin is still simple and non-intrusive. The plugin provides tools to prepare the docker image and generate Argo workflow specification file for the Kedro pipeline. Next, we'll demonstrate how to deploy pipelines to Kubernetes while leverage Vineyard for efficient intermediate sharing between tasks step-by-step. Prepare the vineyard cluster (see also ): ```bash $ export KUBECONFIG=/path/to/your/kubeconfig $ go run k8s/cmd/main.go deploy vineyard-cluster --create-namespace ``` Install the argo server: ```bash $ kubectl create namespace argo $ kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.4.8/install.yaml ``` Generate the iris demo project from the official template: ```bash $ kedro new --starter=pandas-iris ``` Build the Docker image for this iris demo project: ```bash $ cd iris $ kedro vineyard docker build ``` A Docker image named `iris` will be built successfully. The docker image need to be pushed to your image registry, or loaded to the kind/minikube cluster, to be available in Kubernetes. ```bash $ docker images | grep iris iris latest 3c92da8241c6 About a minute ago 690MB ``` Next, generate the Argo workflow YAML file from the iris demo project: ```bash $ kedro vineyard argo generate -i iris $ ls -l argo-iris.yml -rw-rw-r-- 1 root root 3685 Jun 12 23:55 argo-iris.yml ``` Finally, submit the Argo workflow to Kubernetes: ```bash $ argo submit -n argo argo-iris.yml ``` You can interact with the Argo workflow using the `argo` command-line tool, e.g., ```bash $ argo list workflows -n argo NAME STATUS AGE DURATION PRIORITY MESSAGE iris-sg6qf Succeeded 18m 30s 0 ``` We have prepared a benchmark to evaluate the performance gain brought by vineyard for data sharing when data scales, for more details, please refer to .