Package: ml-pipeline-gen
ML Pipeline GeneratorML Pipeline Generator is a tool for generating end-to-end pipelines composed of GCP components so that users can easily migrate their local ML models onto GCP and start realizing the benefits of the Cloud quickly.The following ML frameworks will be supported:TensorFlow (TF)Scikit-learn (SKL)XGBoost (XGB)The following backends are currently supported for model training:Google Cloud AI PlatformAI Platform Pipelines(managed Kubeflow Pipelines)Installationpipinstallml-pipeline-genSetupGCP credentialsgcloudauthlogin
gcloudauthapplication-defaultlogin
gcloudconfigsetproject[PROJECT_ID]Enabling required APIsThe tool requires following Google Cloud APIs to be enabled:Compute EngineAI Platform Training and PredictionCloud StorageEnable the above APIs by following the links, or run the below command to enable the APIs for your project.gcloudservicesenableml.googleapis.com\compute.googleapis.com\storage-component.googleapis.comPython environmentpython3-mvenvvenvsource./venv/bin/activate
pipinstallml-pipeline-genKubeflowCreate a Kubeflow deployment using Cloud Marketplace. Follow theseinstructionsto give the Kubeflow instance access to GCP services.A future release will automate provisioning of KFP clusters and incorporate
K8s Workload Identity for auth.Cloud AI Platform DemoThis demo uses the scikit-learn model inexamples/sklearn/model/sklearn_model.pyto create a training module to run on
CAIP. First, make a copy of the scikit-learn example.cp-rexamples/sklearnsklearn-democdsklearn-demoCreate aconfig.yamlby using theconfig.yaml.exampletemplate. See theInput argssection for details on the config parameters. Once the
config file is filled out, run the demo.pythondemo.pyRunning this demo uses the config file to generate atrainer/module that is
compatible with CAIP.KFP DemoThis demo uses the TensorFlow model inexamples/kfp/model/tf_model.pyto
create a KubeFlow Pipeline (hosted on AI Platform Pipelines). First, make a copy
of the kfp example.cp-rexamples/kfpkfp-democdkfp-demoCreate aconfig.yamlby using theconfig.yaml.exampletemplate. See theInput argssection for details on the config parameters. Once the
config file is filled out, run the demo.pythondemo.pyRunning this demo uses the config file to generate atrainer/module that is
compatible with CAIP. It also generatesorchestration/pipeline.py, which
compiles a Kubeflow Pipeline.TestsThe tests useunittest, Python's built-in unit testing framework. By runningpython -m unittest, the framework performs test discovery to find all tests
within this project. Tests can be run on a more granular level by feeding a
directory to test discover. Read more aboutunittesthere.python-munittestInput argsThe following input args are included by default. Overwrite them by adding them
as inputs in the config file.ArgDescriptiontrain_pathDir or bucket containing train data.eval_pathDir or bucket containing eval data.model_dirDir or bucket to save model files.batch_sizeNumber of rows of data to be fed into the model each iteration.max_stepsThe maximum number of iterations to train the model for.learning_rateMultiplier that controls how much the weights of our network are adjusted with respoect to the loss gradient.export_formatFile format expected by the exported model at inference time.save_checkpoints_stepsNumber of steps to run before saving a model checkpoint.keep_checkpoint_maxNumber of model checkpoints to keep.log_step_count_stepsNumber of steps to run before logging training performance.eval_stepsNumber of steps to use to evaluate the model.early_stopping_stepsNumber of steps with no loss decrease before stopping early.ContributeTo modify the behavior of the library, installml-pipeline-genusing:pipinstall-e.