Package: clearml-session
clearml-sessionCLI for launching JupyterLab / VSCode on a remote machineclearml-sessionis a utility for launching detachable remote interactive sessions (MacOS, Windows, Linux)tl;drCLI to launch remote sessions for JupyterLab / VSCode-server / SSH, inside any docker image!What does it do?Starting a clearml (ob)session from your local machine triggers the following:ClearML allocates a remote instance (GPU) from your dedicated poolOn the allocated instance it will spinjupyter-lab+vscode server+SSHaccess for
interactive usage (i.e., development)ClearML will start monitoring machine performance, allowing DevOps to detect stale instances and spin them downℹ️Remote PyCharm:You can also work with PyCharm in a remote session over SSH. Use thePyCharm Pluginto automatically sync local configurations with a remote session.Use-cases for remote interactive sessions:Development requires resources not available on the current developer's machinesTeam resource sharing (e.g. how to dynamically assign GPUs to developers)Spin a copy of a previously executed experiment for remote debugging purposes (:open_mouth:!)Scale-out development to multiple clouds, assign development machines on AWS/GCP/Azure in a seamless wayPrerequisites:An SSH client installed on your machine- To verify open your terminal and executessh, if you did not receive an error, we are good to go.At least oneclearml-agentrunning on a remote host. See installationdetails.Supported OS: MacOS, Windows, LinuxSecure & Stableclearml-sessioncreates a single, secure, and encrypted connection to the remote machine over SSH.
SSH credentials are automatically generated by the CLI and contain fully random 32 bytes password.All http connections are tunneled over the SSH connection,
allowing users to add additional services on the remote machine (!)Furthermore, all tunneled connections have a special stable network layer allowing you to refresh the underlying SSH
connection without breaking any network sockets!This means that if the network connection is unstable, you can refresh
the base SSH network tunnel, without breaking JupyterLab/VSCode-server or your own SSH connection
(e.h. debugging over SSH with PyCharm)How to use: Interactive Sessionrunclearml-sessionselect the requested queue (resource)wait until a machine is up and readyclick on the link to the remote JupyterLab/VSCode OR connect with the provided SSH detailsNotice! You can also: Select adocker imageto execute in, install requiredpython packages, runbash script,
passgit credentials, etc.
See below for full CLI options.Frequently Asked Questions:How Does ClearML enable this?Theclearml-sessioncreates a new interactiveTaskin the system (default project: DevOps).ThisTaskis responsible for setting the SSH and JupyterLab/VSCode on the host machine.The localclearml-sessionawaits for the interactive Task to finish with the initial setup, then
it connects via SSH to the host machine (see "safe and stable" above), and tunnels
both SSH and JupyterLab over the SSH connection.The end results is a local link which you can use to access the JupyterLab/VSCode on the remote machine, over asecure and encryptedconnection!How can this be used to scale up/out development resources?Clearmlhas a cloud autoscaler, so you can easily and automatically spin machines for development!There is also a default docker image to use when initiating a task.This means that usingclearml-sessions
with the autoscaler enabled, allows for turn-key secure development environment inside a docker of your choosing.Learn more about ithereDoes this fit Work From Home situations?YES. Installclearml-agenton target machines inside the organization, connect over your company VPN
and useclearml-sessionto gain access to a dedicated on-prem machine with the docker of your choosing
(with out-of-the-box support for any internal docker artifactory).Learn more about how to utilize your office workstations and on-prem machineshereTutorialsGetting startedRequirementsclearmlpython package installed and configured (see detailedinstructions)pipinstallclearml-session
clearml-session--dockernvcr.io/nvidia/pytorch:20.11-py3--git-credentialsWait for the machine to spin up:
Expected CLI output would look something like:Creating new sessionNew session created [id=3d38e738c5ff458a9ec465e77e19da23]Waiting for remote machine allocation [id=3d38e738c5ff458a9ec465e77e19da23].Status [queued]....Status [in_progress]Remote machine allocatedSetting remote environment [Task id=3d38e738c5ff458a9ec465e77e19da23]Setup process details: https://app.community.clear.ml/projects/64ae77968db24b27abf86a501667c330/experiments/3d38e738c5ff458a9ec465e77e19da23/output/logWaiting for environment setup to complete [usually about 20-30 seconds]..............Remote machine is readySetting up connection to remote sessionStarting SSH tunnelWarning: Permanently added '[192.168.0.17]:10022' (ECDSA) to the list of known hosts.root@192.168.0.17's password: f7bae03235ff2a62b6bfbc6ab9479f9e28640a068b1208b63f60cb097b3a1784Interactive session is running:SSH: ssh root@localhost -p 8022 [password: f7bae03235ff2a62b6bfbc6ab9479f9e28640a068b1208b63f60cb097b3a1784]Jupyter Lab URL: http://localhost:8878/?token=df52806d36ad30738117937507b213ac14ed638b8c336a7eVSCode server available at http://localhost:8898/Connection is up and runningEnter "r" (or "reconnect") to reconnect the session (for example after suspend)`s` (or "shell") to connect to the SSH session`Ctrl-C` (or "quit") to abort (remote session remains active)or "Shutdown" to shut down remote interactive sessionClick on the JupyterLab link (http://localhost:8878/?token=xyz)
Open your terminal, clone your code & start working :)Leaving a session and reconnecting from the same machineOn theclearml-sessionCLI terminal, enter 'quit' or pressCtrl-CIt will close the CLI but preserve the remote session (i.e. remote session will remain running)When you want to reconnect to it, execute:clearml-sessionThen press "Y" (or enter) to reconnect to the already running sessionclearml-session - launch interactive sessionChecking previous sessionConnect to active session id=3d38e738c5ff458a9ec465e77e19da23 [Y]/n?Shutting down a remote sessionOn theclearml-sessionCLI terminal, enter 'shutdown' (case-insensitive)
It will shut down the remote session, free the resource and close the CLIEnter "r" (or "reconnect") to reconnect the session (for example after suspend)`s` (or "shell") to connect to the SSH session`Ctrl-C` (or "quit") to abort (remote session remains active)or "Shutdown" to shut down remote interactive sessionshutdownShutting down interactive sessionRemote session shutdownGoodbyeConnecting to a running interactive session from a different machineContinue working on an interactive session fromanymachine.
In theclearmlweb UI, go to DevOps project, and find your interactive session.
Click on the ID button next to the Task name, and copy the unique ID.clearml-session--attach<session_id_here>Click on the JupyterLab/VSCode link, or connect directly to the SSH sessionDebug a previously executed experimentIf you have a previously executed experiment in the system,
you can create an exact copy of the experiment and debug it on the remote interactive session.clearml-sessionwill replicate the exact remote environment, add JupyterLab/VSCode/SSH and allow you interactively
execute and debug the experiment, on the allocated remote machine.In theclearmlweb UI, find the experiment (Task) you wish to debug.
Click on the ID button next to the Task name, and copy the unique ID.clearml-session--debugging-session<experiment_id_here>Click on the JupyterLab/VSCode link, or connect directly to the SSH sessionCLI optionsclearml-session--helpclearml-session - CLI for launching JupyterLab / VSCode on a remote machineusage: clearml-session [-h] [--version] [--attach [ATTACH]][--shutdown [SHUTDOWN]] [--shell][--debugging-session DEBUGGING_SESSION] [--queue QUEUE][--docker DOCKER] [--docker-args DOCKER_ARGS][--public-ip [true/false]][--remote-ssh-port REMOTE_SSH_PORT][--vscode-server [true/false]][--vscode-version VSCODE_VERSION][--vscode-extensions VSCODE_EXTENSIONS][--jupyter-lab [true/false]][--upload-files UPLOAD_FILES][--git-credentials [true/false]][--user-folder USER_FOLDER][--packages [PACKAGES [PACKAGES ...]]][--requirements REQUIREMENTS][--init-script [INIT_SCRIPT]][--config-file CONFIG_FILE][--remote-gateway [REMOTE_GATEWAY]][--base-task-id BASE_TASK_ID] [--project PROJECT][--keepalive [true/false]][--queue-excluded-tag [QUEUE_EXCLUDED_TAG [QUEUE_EXCLUDED_TAG ...]]][--queue-include-tag [QUEUE_INCLUDE_TAG [QUEUE_INCLUDE_TAG ...]]][--skip-docker-network [true/false]][--password PASSWORD] [--username USERNAME][--force_dropbear [true/false]] [--verbose] [--yes]clearml-session - CLI for launching JupyterLab / VSCode on a remote machineoptional arguments:-h, --help            show this help message and exit--version             Display the clearml-session utility version--attach [ATTACH]     Attach to running interactive session (default:previous session)--shutdown [SHUTDOWN], -S [SHUTDOWN]Shut down an active session (default: previoussession)--shell               Open the SSH shell session directly, notice quitingthe SSH session will Not shutdown the remote session--debugging-session DEBUGGING_SESSIONPass existing Task id (experiment), create a copy ofthe experiment on a remote machine, and launchjupyter/ssh for interactive access. Example--debugging-session <task_id>--queue QUEUE         Select the queue to launch the interactive session on(default: previously used queue)--docker DOCKER       Select the docker image to use in the interactivesession on (default: previously used docker image or`nvidia/cuda:10.1-runtime-ubuntu18.04`)--docker-args DOCKER_ARGSAdd additional arguments for the docker image to usein the interactive session on (default: previouslyused docker-args)--public-ip [true/false]If True register the public IP of the remote machine.Set if running on the cloud. Default: false (use forlocal / on-premises)--remote-ssh-port REMOTE_SSH_PORTSet the remote ssh server port, running on the agent`smachine. (default: 10022)--vscode-server [true/false]Install vscode server (code-server) on interactivesession (default: true)--vscode-version VSCODE_VERSIONSet vscode server (code-server) version, as well asvscode python extension version <vscode:python-ext>(example: "3.7.4:2020.10.332292344")--vscode-extensions VSCODE_EXTENSIONSInstall additional vscode extensions, as well asvscode python extension (example: "ms-python.python,ms-python.black-formatter,ms-python.pylint,ms-python.flake8")--jupyter-lab [true/false]Install Jupyter-Lab on interactive session (default:true)--upload-files UPLOAD_FILESAdvanced: Upload local files/folders to the remotesession. Example: `/my/local/data/` will upload thelocal folder and extract it into the container in~/session-files/--git-credentials [true/false]If true, local .git-credentials file is sent to theinteractive session. (default: false)--user-folder USER_FOLDERAdvanced: Set the remote base folder (default: ~/)--packages [PACKAGES [PACKAGES ...]]Additional packages to add, supports version numbers(default: previously added packages). examples:--packages torch==1.7 tqdm--requirements REQUIREMENTSSpecify requirements.txt file to install when settingthe interactive session. Requirements file is read andstored in `packages` section as default for the nextsessions. Can be overridden by calling `--packages`--init-script [INIT_SCRIPT]Specify BASH init script file to be executed whensetting the interactive session. Script content isread and stored as default script for the nextsessions. To clear the init-script do not pass a file--config-file CONFIG_FILEAdvanced: Change the configuration file used to storethe previous state (default: ~/.clearml_session.json)--remote-gateway [REMOTE_GATEWAY]Advanced: Specify gateway ip/address:port to be passedto interactive session (for use with k8s ingestion /ELB)--base-task-id BASE_TASK_IDAdvanced: Set the base task ID for the interactivesession. (default: previously used Task). Use `none`for the default interactive session--project PROJECT     Advanced: Set the project name for the interactivesession Task--keepalive [true/false]Advanced: If set, enables the transparent proxy alwayskeeping the sockets alive. Default: False, do not usetransparent socket for mitigating connection drops.--queue-excluded-tag [QUEUE_EXCLUDED_TAG [QUEUE_EXCLUDED_TAG ...]]Advanced: Excluded queues with this specific tag fromthe selection--queue-include-tag [QUEUE_INCLUDE_TAG [QUEUE_INCLUDE_TAG ...]]Advanced: Only include queues with this specific tagfrom the selection--skip-docker-network [true/false]Advanced: If set, `--network host` is **not** passedto docker (assumes k8s network ingestion) (default:false)--password PASSWORD   Advanced: Select ssh password for the interactivesession (default: `randomly-generated` or previouslyused one)--username USERNAME   Advanced: Select ssh username for the interactivesession (default: `root` or previously used one)--force_dropbear [true/false]Force using `dropbear` instead of SSHd--verbose             Advanced: If set, print verbose progress information,e.g. the remote machine setup process log--yes, -y             Automatic yes to prompts; assume "yes" as answer toall prompts and run non-interactivelyNotice! all arguments are stored as new defaults for the next session