Package: tfmesos
TFMesosis a lightweight framework to help running distributedTensorflowMachine Learning tasks onApache MesoswithinDockerandNvidia-Docker.TFMesosdynamically allocates resources from aMesoscluster, builds a distributed training cluster forTensorflow, and makes different training tasks mangeed and isolated in the sharedMesoscluster with the help ofDocker.PrerequisitesForMesos >= 1.0.0:MesosCluster (cf:Mesos Getting Started). All nodes in the cluster should be reachable using their hostnames, and all nodes have identical/etc/passwdand/etc/group.SetupMesos Agentto enableMesos ContainerizerandMesos Nvidia GPU Support(optional). eg:mesos-agent--containerizers=mesos--image_providers=docker--isolation=filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia(optional) A Distributed Filesystem (eg:MooseFS)Ensure latestTFMesosdocker image (tfmesos/tfmesos) is pulled across the whole clusterForMesos < 1.0.0:MesosCluster (cf:Mesos Getting Started). All nodes in the cluster should be reachable using their hostnames, and all nodes have identical/etc/passwdand/etc/group.Docker(cf:Docker Get Start Tutorial)Mesos Docker Containerizer Support(cf:Mesos Docker Containerizer)(optional)Nvidia-dockerinstallation (cf:Nvidia-docker installation) and make sure nvidia-plugin is accessible from remote host (with-l0.0.0.0:3476)(optional) A Distributed Filesystem (eg:MooseFS)Ensure latestTFMesosdocker image (tfmesos/tfmesos) is pulled across the whole clusterIf you are usingAWS G2instance, here is asamplescript to setup most of there prerequisites.Running simple TestAfter setting up the mesos and pulling the docker image on a single node (or a cluser), you should be able to use the following command to run a simple test.$dockerrun-eMESOS_MASTER=mesos-master:5050\-eDOCKER_IMAGE=tfmesos/tfmesos\--net=host\-v/path-to-your-tfmesos-code/tfmesos/examples/plus.py:/tmp/plus.py\--rm\-it\tfmesos/tfmesos\python/tmp/plus.pymesos-master:5050Successfully running the test should result in an output of 42 on the console.Running in replica modeThis mode is calledBetween-graph replicationin officialDistributed Tensorflow HowtoMost distributed training models that Google has open sourced (such asmnist_replicaandinception) are using this mode. In this mode, two kind of Jobs are defined with the names‘ps’and‘worker’.‘ps’tasks act as‘Parameter Server’and‘worker’tasks run the actual training process.Here we use our modified‘mnist_replica’as example:Checkout themnistexample codes into a directory in shared filesystem, eg:/nfs/mnistAssume Mesos master ismesos-master:5050Now we can launch this script using following commands:CPU:$dockerrun--rm-it-eMESOS_MASTER=mesos-master:5050\--net=host\-v/nfs/mnist:/nfs/mnist\-v/etc/passwd:/etc/passwd:ro\-v/etc/group:/etc/group:ro\-u`id-u`\-w/nfs/mnist\tfmesos/tfmesos\tfrun-w1-s1\-V/nfs/mnist:/nfs/mnist\--pythonmnist_replica.py\--ps_hosts{ps_hosts}--worker_hosts{worker_hosts}\--job_name{job_name}--worker_index{task_index}GPU (1 GPU per worker):$nvidia-dockerrun--rm-it-eMESOS_MASTER=mesos-master:5050\--net=host\-v/nfs/mnist:/nfs/mnist\-v/etc/passwd:/etc/passwd:ro\-v/etc/group:/etc/group:ro\-u`id-u`\-w/nfs/mnist\tfmesos/tfmesos\tfrun-w1-s1-Gw1--pythonmnist_replica.py\--ps_hosts{ps_hosts}--worker_hosts{worker_hosts}\--job_name{job_name}--worker_index{task_index}Note:In this mode,tfrunis used to prepare the cluster and launch the training script on each node, and worker #0 (the chief worker) will be launched in the local container.tfrunwill substitute{ps_hosts},{worker_hosts},{job_name},{task_index}with corresponding values of each task.Running in fine-grained modeThis mode is calledIn-graph replicationin officialDistributed Tensorflow HowtoIn this mode, we have more control over the cluster spec. All nodes in the cluster is remote and just running aGrpcserver. Each worker is driven by a local thread to run the training task.Here we use our modifiedmnistas example:Checkout themnistexample codes into a directory, eg:/tmp/mnistAssume Mesos master ismesos-master:5050Now we can launch this script using following commands:CPU:$dockerrun--rm-it-eMESOS_MASTER=mesos-master:5050\--net=host\-v/tmp/mnist:/tmp/mnist\-v/etc/passwd:/etc/passwd:ro\-v/etc/group:/etc/group:ro\-u`id-u`\-w/tmp/mnist\tfmesos/tfmesos\pythonmnist.pyGPU (1 GPU per worker):$nvidia-dockerrun--rm-it-eMESOS_MASTER=mesos-master:5050\--net=host\-v/tmp/mnist:/tmp/mnist\-v/etc/passwd:/etc/passwd:ro\-v/etc/group:/etc/group:ro\-u`id-u`\-w/tmp/mnist\tfmesos/tfmesos\pythonmnist.py--worker-gpus1