Package: recompute
A sweet tool for Remote Execution.What is the chillest way one can train models in remote machines?Do not worry about environment setup (dependencies)Don't bother choosing an instance to run onNo more bash scripts to copy files back and forthPrinciplesreprovides a suite of features that empowers the user to focus on the experiments without having to worry about boring details listed above.Almost zero confAbstract away boring repetitive detailsEase of executionConventionsYou do need to follow a couple of conventions.Data goes intodata/Any non-python file that is necessary for remote execution should be added to.recompute/includeAny python file that shouldn't be pushed to remote machine should be added to.recompute/excludeSetup# install sshpassbrewinstallhttp://git.io/sshpass.rb
pipinstall--userrecomputeConfigurationThe configuration file is super-short.[general]instance=0remote_home=projects/[instance 0]username=grenouillehost=grasse.localpassword=hen0s3datru1hYou can add credentials for remote machines directly into the configuration file or add them sequentially via command-linere sshadd --instance='user@remotehost'.WorkflowMy machine learning workflow follows these steps:Copy code to remote machinersyncSetup dependenciespip installDownload dataset and place them indata/Execute code in remote machineGet execution logCopy binaries generatedbin/wihtre, the tasks listed above can be accomplished with 4 commands, as below:# re sshadd --instance='reinit# initalize [rsync, install]reasync"python3 x.py"# start execution in remote# (or) re sync "python3 x.py"  # blocking run (wait for completion)relog# after a whilerepull"bin/ ./bin/".# pull generated binariesinitcreates local configuration files, setting up the environment for remote executionMakes a list of local dependencies (python files)Populatesrequirements.txtwith required pypi packagesInstalls pypi packages in remote machineCopies local dependencies to remote machine usingrsyncA copy of local folder is created in the remote machine, under~/projects/We could start execution in remote machine and wait for it to complete by usingsyncmode or just start remote execution and move on, usingasyncmodeThe command to be executed in remote machine, should be given as a string next tosyncorasyncmodere logfetches log from remote machinere pullpulls any file from remote machineFiles are addressed by their relative pathsLoggingreredirects thestdoutandstderrof remote execution into<project-name>.log, which could be pulled to local machine by runningre log. More often than not, it takes a while for execution to complete. So we start the execution in remote machine and check the log once in a while usingre log. Or you could put this "once in a while" as a command-line argument andrepulls the log and shows you every "once in a while". It is recommended to useloggingmodule to print information onto stdout, instead ofprintstatements.# fetch log from remote machinerelog# . start execution in remote machine# .. fetch logreasync"python3 nn.py"relog# . start execution# .. pull log every 20 secondsreasync"python3 nn.py"relog--loop=20rsyncFiles (local dependencies) can be synchronized by usingrsynccommand.rsyncis run in the background which copies files listed in.recompute/rsync.dbto remote machine.--forceswitch forcesreto figure out the local dependencies and updatersync.db.rersync# --force updates .recompute/rsync.dbDependenciesrequirements.txtis populated with python packages necessary for execution (usespipreqsbehind the scenes).re installreadsrequirements.txtand installs the packages in remote system.# install dependenciesreinstall# --force updates requirements.txt# manual installreinstall"torch tqdm"Manages Processesrekeeps track of all the remote processes it has spawned. We could list them out usinglistcommand and selectively kill processes usingkillcommand.# list live processesrelist# +-------+--------------+-------+# | Index |     Name     |  PID  |# +-------+--------------+-------+# |   0   |     all      |   *   |# |   1   | zombie/spawn | 30601 |# |   2   |    runner    | 31036 |# +-------+--------------+-------+# kill process [1]rekill--idx=1# kill them allrepurge# or kill interactively with just `re kill`Upload/DownloadYou might wanna download or upload a file just once without having to include it in rsync database. We havepushandpullcommands. And there is a special command nameddatawhich downloads from space separated urls from command-line, into remote machine'sdata/directory.# . upload from local machine to remote# .. copy [current_dir/x/localfile] to [remote_home/projects/mynn1/x/]repush"x/localfile x/"# . download from remote machine to local# .. copy [remote_home/projects/mynn1/y/remotefile] to [current_dir/y/remotefile]repull"y/remotefile y/"# download IRIS dataset to remote machine's [data/]redatahttps://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data# more urls can be added, separated by spacesNotebookSometimes you wanna run code snippets in a notebook.re notebookstarts a remote jupyter notebook server and hooks it to a local port. The remote server is tracked (re list) and could be killed whenever necessary.# . start notebook server in remote machine# .. hook to local portrenotebook# Cntl-c to quitProbeprobecommand probes remote machines and provides us with a table of available machines with info on available resources.reprobe# +--------------------------------+--------+----------+-----------+# |        Machine                 | Status | GPU (MB) | Disk (MB) |# +--------------------------------+--------+----------+-----------+# | grenouille@grasse.local        | active |  10432   |     4238  |# | slartibartfast@magrathea.local | active |   8642   |    12012  |# +--------------------------------+--------+----------+-----------+Manualre mangives you a detailed manual.ModeDescriptionOptionsExampleinitSetup current directory for remote execution--instance-idxre initre init --instance-idx=1rsyncUse rsync to synchronize local files with remote--forcere rsyncsshaddAdd a new instance to config--instancere sshadd --instance="usr@host"installInstall pypi packages in requirements.txt in remotecmd, --forcere installre install "pytorch tqdm"syncSynchronous execution of "args.cmd" in remotecmd, --force, --rsyncre sync "python3 x.py"asyncAsynchronous execution of "args.cmd" in remotecmd, --force, --rsyncre async "python3 x.py"logFetch log from remote machine--loop, --filterre logre log --loop=2re log --filter="pattern"listList out processes alive in remote machine--forcere listkillKill a process by index--idxre killre kill --idx=1purgeKill all remote process that are aliveNonere purgesshCreate an ssh session in remote machineNonere sshnotebookCreate jupyter notebook in remote machine--run-asyncre notebookpushUpload file to remote machinecmdre push "x.py y/"pullDownload file from remote machinecmdre pull "y/z.py ."dataDownload data from web into data/ folder of remotecmdre data "url1 url2 url3"manShow this man pageNonere manContributionAll kinds of contribution are welcome.Somethin went wrong?What feature is missing?What could be done better?Raise anissue.
Add a pull request.LicenseCopyright (c) 2019 Suriyadeepan Ramamoorthy. All rights reserved.This work is licensed under the terms of the MIT license.For a copy, seehttps://opensource.org/licenses/MIT.