Package: training-noodles
Training NoodlesA simple and powerful tool to help trainingmultipleprograms onmultipleservers with only one human.FeaturesAutomatically deploys experiments to available serversNo need to change any existing codeConsiders CPU usage, GPU usage, memory usage, disk usage, and moreUses only SSH protocolRelies on minimal dependenciesAllows fast prototypingUse CaseIf we want to run 4 experiments on 3 servers, more specifically, we need toUpload the code to one of the servers which has low CPU usageRun the code on the serverDownload experimental results when they're readyIn the first deployment round (See image above), Noodles will use the user-defined commands to check CPU usage on the servers.The CPU usage is high onServer 1because there are some other programs running, so Noodles usesscpto upload the codeCode 1andCode 2, and run them onServer 2andServer 3respectively.As for how to upload the code, it's just a list of commands written by us, Noodles just follows the commands.In the second deployment round (See image above), we tell Noodles to check experimental results on all servers.Noodles finds thatServer 3has just finished runningCode 2, so it downloads the experimental results and process the data on local machine as we tell it to do so.In the third deployment round (See image above),Code 3andCode 4still need to be deployed. Noodles checks the CPU usage on all servers again. AsServer 1has just become free now, Noodles can deployCode 3andCode 4toServer 1andServer 3respectively.The deployment round would continue until all experiments are successfully deployed. As in this case, Noodles will try to download and process the experimental results ofCode 1,Code 3andCode 4in later rounds.How Noodles WorksThe general procedure is as follows:Initialize the list of experiments inEFor each deployment round:Initialize the list of servers inSFor each experiment inE:Noodles runs user-defined requirements on each server inSNoodles compares the metrics (results from the above step) to the user-defined expressionIf the expression is satisfied:Noodles runs the user-defined commands on the satisfied serverRemove the current experiment fromERemove the satisfied server fromSIfSis empty, breakIfEis empty, breakThe implementation of Noodles complies with the following rules:Simple (User can understand code and spec without looking documentation)Easy to debug (Noodles can take different actions when different error occurs)Stateless (The only state Noodles cares about is whether the deployment is successful or not, the states of the experiments must be handled by the user)DocumentationSee full documentationhere.PrerequisitesLinux-based terminals (For Windows, I recommend usinggit-sdk)Python 3.5 or higherInstallationRun the following command:pipinstalltraining-noodlesUsagenoodles<command_type><path_to_spec>It's just that simple.ExamplesHere are some examples showing how Noodles is used:noodlesrunmy_training.yml
noodlesstatusmy_training.yml
noodlesmonitormy_training.yml
noodlesstopmy_training.yml
noodlesdownloadmy_training.yml
noodlesuploadmy_training.yml
...You can also choose only some experiments:noodlesrun"my_training.yml:Experiment 1,Experiment 2"See the exampleTwo Localsto get started. SeeTrain TensorFlow Examplesfor a more complex example.Default SpecNoodles will use properties from default spec if the user spec doesn't specify them. Seetraining_noodles/specs/defaults.ymlfor the default spec.