Package: gdmix-workflow
GDMix WorkflowGDMix-workflow is a workflow generation toolkit to orchestrate training jobs of theGDMix, which is a framework to train non-linear fixed effect and random effect models. By providing aGDMix config, GDMix-workflow can generate jobs and run them directly, or generate a YAML file that can be uploaded to Kubeflow Pipeline to run training job distributedly on Kubernetes cluster with Kubeflow Pipeline deployed.ConfigurationGDMix-workflow supports two modes, single_node and distributed. For single_node mode, user will need to install thegdmix-workflowpackage and spark, GDMix-workflow will prepare jobs and run them on the node. For distributed mode, GDMix-workflow generates a YAML file that can be uploaded to Kubeflow Pipeline. We'll explain more about distributed mode in the sectionRun on Kubernetes.
Once thegdmix-workflowpackage is installed (pip install gdmix-workflow),  user can callpython -m gdmixworkflow.mainplus following parameters:--config_path: path to gdmix config. Required.--mode: distributed or single_node. Required.--jar_path: path to the gdmix-data jar for GDMix processing intermediate data. Required by single_node mode only.--workflow_name: name for the generated zip file to upload to Kubeflow Pipeline. Required by distributed mode only.--namespace: Kubernetes namespace. Required by distributed mode only.--secret_name: secret name to access storage. Required by distributed mode only.--image: image used to launch gdmix jobs on Kubernetes. Required by distributed mode only.--service_account: service account for thespark-on-k8s-operatorto launch spark job. Required by distributed mode only.Run GDMix workflow on Kubernetes for distributed trainingGDMix's distributed training is based onKubernetes, and leverages Kubernetes job scheduling servicesKubeflowandspark-on-k8s-operatorto run TensorFlow and Spark job distributedly on Kubernetes, and usesKubeflow Pipelineto orchestrate jobs. Besides that, a centralized storage is needed for storing training data and models. User can useKubernetes-HDFSorNFSas the centralized storage.Create a Kubernetes cluster, deploy required servicesTo run GDMix in the distributed mode, user needs to create a Kubernetes cluster, and deploy following services:Kubeflow tf-operatorspark-on-k8s-operatorKubeflow PipelineNFS Server ProvisionerGenerate task YAML file and upload to Kubeflow Pipeline UIWhen the Kubernetes cluster and services are ready, with the provided GDMix config, GDMix-workflow can generate task YAML file that has job specification for the distributed TensorFlow and Spark jobs. User needs to upload it to Kubeflow Pipeline and start training.Run theMovieLensexampleIn this section we'll introduce how to train fixed effect and random effect models using GDMix for MovieLens data.
Please download and preprocess moveLens data to meet GDMix's need using the provided scriptdownload_process_movieLens_data.py:wget https://raw.githubusercontent.com/linkedin/gdmix/master/scripts/download_process_movieLens_data.py
pip install pandas
python download_process_movieLens_data.pyFor distributed training, the processed movieLens data need to be copied to the centralized storage.We'll also need a GDMix config, a reference of training logistic regression models for the fixed effectglobaland the random effectsper-userandper-moviewith distributed training can be found atlr-distributed-movieLens.config.Run on single nodePlease see the sectionTry out the movieLens examplein the rootREADME.mdfor details of how to run the movieLens example on single node.Run on Kubernetes for distributed trainingTo run on Kubernetes, as mentioned earlier, user will need to copy the processed movieLens data to the centralized storage, modify the input path fields such astraining_data_dir,validation_data_dir,feature_fileandmetadata_fileof the GDMix config for distributed traininglr-distributed-movieLens.config.If using the provided imagelinkedin/gdmix, user can mount the processed movieLens data from the centralized storage to path/workspace/notebook/movieLensfor each worker then no change is needed for the distributed training GDMix configlr-distributed-movieLens.config.Generate YAML fileUser will need to install theGDMix-worklfowpackage to generate the YAML file:pip install gdmix-workflowDownload the example GDMix config for distributed training and generate the YAML file with following command:wget https://raw.githubusercontent.com/linkedin/gdmix/master/gdmix-workflow/examples/movielens-100k/lr-distributed-movieLens.config

python -m gdmixworkflow.main --config_path lr-distributed-movieLens.config --mode=distributed --workflow_name=movieLens --namespace=default --secret_name default --image linkedin:gdmix --service_account defaultParameters ofnamespace,secret_nameandservice_accountrelate to the Kubernetes cluster setting and job scheduling operator deployments. A zip file namedmovieLens.zipis expected to be produced.Upload to Kubeflow PipelineIf the Kubeflow Pipeline is successfully deployed, use can forward the Pipeline UI to local browser, The command below forwards the Pipeline UI to the local port 9980:kubectl -n default port-forward svc/ml-pipeline-ui 9980:80Typelocalhost:9980in the local browser to view the Kubeflow Pipeline UI, upload the produced YAML filemovieLens.zip(click buttonUpload pipeline), and then click buttonCreate runto start the training. A snapshot of the movieLens workflow is shown below.GDMix Distributed Training on Kubeflow Pipeline