Package: mlt
mltMachine Learning Container TemplatesMLT: it's like the keras of kubernetes- @mas-dse-greinamltaids in the creation of containers for machine learning jobs.
It does so by making it easy to use container and kubernetes object templates.We have found it useful to share project templates for various machine learning frameworks. Some have native support from Kubernetes operators, such as mxnet and TensorFlow. Others do not, but still have best practices for how to run on a Kubernetes cluster.On top of getting boiler plate code at the beginning of a project to work, the best practices may change over time.mltallows existing projects to adapt to these without having to reset and start over.mltaddresses another aspect of the application development:iterativecontainer creation. Storage and container creation is supposed to be fast - so why not rebuild containers automatically?mlthas a--watchoption, which lets you write code and have an IDE-like experience.
When changes are detected, a timer starts and triggers container rebuilds.
lint and unit tests can be run in this step, as an early indicator of whether the code will run in the cluster.
When the container is built, it is pushed to the cluster container registry.
From here, it is a quick step to redeploy the Kubernetes objects, throughmlt deployBuildPrerequisites:Python (2.7 or newer) development environment. This usually consists ofbuild-essential,libssl-dev,libffi-devandpython-devDockerkubectlgitpythonpipOptional prerequisites:kubetail(formlt logscommand)Ksync(formlt synccommands)ks(Needed0.11.0version for volume support to work inhorovodtemplate)TFJob operator(for thedistributed tensorflowtemplates)PyTorch operator(for thepytorch-distributedtemplate)jq(for thetensorboardtemplates)openssh-server(for thehorovodtemplate)Linux Debian-based distro users:Runmake debian_prereq_installto install every dependency required formltto work. We assume users already havekubectlanddockerset up though.InstallationInstall fromPyPI$pipinstallmltInstallation from Source$gitclonegit@github.com:IntelAI/mlt.git
Cloninginto'mlt'...
remote:Countingobjects:1981,done.
remote:Compressingobjects:100%(202/202),done.
remote:Total1981(delta202),reused280(delta121),pack-reused1599Receivingobjects:100%(1981/1981),438.10KiB|6.54MiB/s,done.
Resolvingdeltas:100%(1078/1078),done.

$cdmlt

$pipinstall.Create local Python distributions$makedist
$cddist
$lsmlt*
mlt-0.1.0a1+12.gf49c412.dirty-py2.py3-none-any.whlUsage summarySample mlt deployment$mlttemplatelist

TemplateDescription
---------------------------------------------------------------------------------------------------------------------
hello-worldATensorFlowpythonHelloWorldexamplerunthroughKubernetesJobs.
pytorchSampledistributedapplicationtakenfromhttp://pytorch.org/tutorials/intermediate/dist_tuto.html
pytorch-distributedAdistributedPyTorchMNISTexamplerunusingthepytorch-operator.
tf-dist-mnistAdistributedTensorFlowMNISTmodelwhichdesignatesworker0asthechief.
tf-distributedAdistributedTensorFlowmatrixmultiplicationrunthroughtheTensorFlowKubernetesOperator.

$mltinitmy-app--template=hello-world[master(root-commit)40239a2]Initialcommit.7fileschanged,191insertions(+)createmode100644mlt.jsoncreatemode100644Dockerfilecreatemode100644Makefilecreatemode100644k8s-templates/tfjob.yamlcreatemode100644k8s/README.mdcreatemode100644main.pycreatemode100644requirements.txt

$cdmy-app# NOTE: `mlt config` has been renamed to `mlt template_config` to avoid confusion regarding developer config settings.# List template-specific config parameters$mlttemplate_configlist
ParameterNameValue
--------------------------------------------------
registrymy-project-12345
namespacemy-app
namemy-app
template_parameters.greetingHello# Update the greeting parameter$mlttemplate_configsettemplate_parameters.greetingHi# Check the template_config list to see the updated parameter value$mlttemplate_configlist
ParameterNameValue
--------------------------------------------------
registryconstant-cubist-173123
namespacedmsuehir
namedmsuehir
template_parameters.greetingHi

$mltbuild
Startingbuildmy-app:71fb176d-28a9-46c2-ab51-fe3d4a88b02c
Building|######################################################| (ETA:  0:00:00)Pushing|######################################################| (ETA:  0:00:00)Builtandpushedtogcr.io/my-project-12345/my-app:71fb176d-28a9-46c2-ab51-fe3d4a88b02c

$mltdeploy
Deployinggcr.io/my-project-12345/my-app:71fb176d-28a9-46c2-ab51-fe3d4a88b02c

Inspectcreatedobjectsbyrunning:$kubectlget--namespace=my-appall### Provide -l flag to tail logs immediately after deploying.$mltdeploy--no-push-l
Skippingimagepush
Deployinggcr.io/my-project-12345/my-app:b9f124d2-ef34-4d66-b137-b8a6026bf782

Inspectcreatedobjectsbyrunning:
$kubectlget--namespace=my-appall

Checkingforpod(s)readiness
Willtail1logs...
my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg[my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg]2018-05-1722:28:34.578791:Itensorflow/core/platform/cpu_feature_guard.cc:140]YourCPUsupportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse:AVX2AVX512FFMA[my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg]b'Hello, TensorFlow!'$mltstatus
NAMEREADYSTATUSRESTARTSAGEIPNODE
my-app-897cb68f-e91f-42a0-968e-3e8073334450-vvpqj1/1Running014s10.23.45.67gke-my-cluster-highmem-8-skylake-1### To deploy in interactive mode (using no-push as an example)### NOTE: only basic functionality is supported at this time. Only one container and one pod in a deployment for now.#### If more than one container in a deployment, we'll pick the first one we find and deploy that.$mltdeploy-i--no-push
Skippingimagepush
Deployinglocalhost:5000/test:d6c9c06b-2b64-4038-a6a9-434bf90d6acc

$mltlogs
Checkingforpod(s)readiness
Willtail1logs...
my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg[my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg]2018-05-1722:28:34.578791:Itensorflow/core/platform/cpu_feature_guard.cc:140]YourCPUsupportsinstructionsthatthisTensorFlowbinarywasnotcompiledtouse:AVX2AVX512FFMA[my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg]b'Hello, TensorFlow!'Inspectcreatedobjectsbyrunning:
$kubectlget--namespace=my-appall

Connectingtopod...
root@test-9e035719-1d8b-4e0c-adcb-f706429ffeac-wl42v:/src/app#ls
DockerfileMakefileREADME.mdk8sk8s-templatesmain.pymlt.jsonrequirements.txt# Displays events for the current job$mltevents
LASTSEENFIRSTSEENCOUNTNAMEKINDSUBOBJECTTYPEREASONSOURCEMESSAGE

6m6m1my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg.152f8f13466696b4PodNormalScheduleddefault-schedulerSuccessfullyassignedmy-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqgtogke-dls-us-n1-highmem-8-skylake-82af83b4-8nvh
6m6m1my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg.152f8f134ff373d7PodNormalSuccessfulMountVolumekubelet,gke-dls-us-n1-highmem-8-skylake-82af83b4-8nvhMountVolume.SetUpsucceededforvolume"default-token-grq2c"6m6m1my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg.152f8f1399b33ba0Podspec.containers{my-app}NormalPulledkubelet,gke-dls-us-n1-highmem-8-skylake-82af83b4-8nvhContainerimage"gcr.io/my-project-12345/my-app:b9f124d2-ef34-4d66-b137-b8a6026bf782"alreadypresentonmachine
6m6m1my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg.152f8f139dec0dc3Podspec.containers{my-app}NormalCreatedkubelet,gke-dls-us-n1-highmem-8-skylake-82af83b4-8nvhCreatedcontainer
6m6m1my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqg.152f8f13a2ea0ff6Podspec.containers{my-app}NormalStartedkubelet,gke-dls-us-n1-highmem-8-skylake-82af83b4-8nvhStartedcontainer
6m6m1my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33.152f8f13461279e4JobNormalSuccessfulCreatejob-controllerCreatedpod:my-app-09aa35f4-bdf8-4da8-8400-8728bf7afa33-sqzqgExamplesDistributed U-Net model training using KVC and MLTDistributed U-Net model training using horovod and openmpi on KubernetesTemplate DevelopmentTo add new templates, see theTemplate Developers Manual.