Package: labml-remote
🕹 LabML Remotelabml_remoteis a very simple tool that lets you setup python and
run python programs on remote computers.
It's mainly intended for deep learning model training.
It simply connects to remote computers via SSH to run commands,
and synchronises content using rsync.labml_remotecomes with a easy-to-useCommand-line Interface.
You can also use the API to launch
custom distributed training sessions.Here is an example.Install from PIPpipinstalllabml_remoteInitialize and add server detailsGo to your project folder.cd[PROJECT]Initialize for remote executionlabml_remoteinitlabml_remote initasks for your SSH credentials and creates two files[PROJECT].remote/configs.yamland[PROJECT].remote/exclude.txt.[PROJECT].remote/configs.yamlkeeps the remote configurations for the project.
Here's a sample.remote/configs.yaml:name:sampleservers:primary:hostname:3.19.32.53private_key:./.remote/private_keysecondary:hostname:3.19.32.54Each of the servers can have the following attributes:hostname:[IP address of hostname]private_key:[Location of the private key file; leave blank if not necessary]username:[Username to SSH with; defaults to 'ubuntu']password:[Password to connect with; leave blank if not necessary].remote/exclude.txtis like.gitignore- it specifies the files and folders that you dont need
to sync up with the remote server. The excludes generated bylabml_remote initexcludes
things like.git,.remote,logsand__pycache__.
You should edit this if you have things that you don't want to be synced
with your remote computer.See oursample projectfor a more complex example.💻 CLIGet the command line interface help with,labml_remote--helpUse the flag--helpwith any command to get the help for that command.Prepare the serverslabml_remoteprepareThis will install Conda on the servers, rsync your project content and install the
Python packages based on yourrequirements.txtorPipfile.Run a commandlabml_remoterun--cmd'python my_script.py'This will execute the command on the server and stream the output of it.Start a joblabml_remotejob-run--cmd'python my_script.py'--tagmy-jobList jobslabml_remotejob-list--rsync--rysncflag will sync up the job information from server to your local computer before
listing.Tail a job outputlabml_remotejob-tail--tagmy-jobThis will keep on tailing the output of the job.Kill jobslabml_remotejob-kill--tagmy-jobLaunch a PyTorch distributed training sessionlabml_remotehelper-torch-launch--cmd'train.py'--nproc-per-node2--envGLOO_SOCKET_IFNAMEenp1s0Heretrain.pyis the training script. We are using computers  with 2 GPUs, we want two processes per computer
so--nproc-per-nodeis 2.--env GLOO_SOCKET_IFNAME enp1s0sets environment variableGLOO_SOCKET_IFNAMEtoenp1s0. You can specify multiple environment variables with--env.How it worksIt sets upminicondaif it is not already installed and create a new environment for the project.
Then it creates a folder by the name of the project inside home folder and synchronises the contents
of your local folder with the remote computer.
It syncs usingrsyncso subsequent synchronisations only need to send the changes.
Then it installs packages fromrequirements.txtor withpipenvif aPipfileis found.
It will usepipenvto run your commands if aPipfileis present.
The outputs of commands are streamed backed to the local computer and the outputs of jobs redirected to
files on the server which are synchronized back to the local computer usingrsync.What it doesn't doThis won't install things like drivers or CUDA. So if you need them you should pick an
image that comes with those for your instance. For example, on AWS pick a deep learning
AMI if you want to use an instance with GPUs.