Package: remote-exec-api
RemoteTheremoteCLI lets you execute long or computation-heavy tasks (e.g., compilation, integration tests, etc.)
on a powerful remote host while you work on the source code locally.
This process is known as remote execution and can enable remote build capabilities, among other things.When you executeremote <cmd>, it will first sync your local workspace to the remote host you selected usingrsync.
It will then execute the command<cmd>on this host usingsshand finally, bring all the created/modified files back to your local workspace.remotesupports a host of configuration options to allow for complete customization of patterns for files and folders to include during the synchronization process in both directions.System RequirementsThe CLI supportsLinuxandMac OS Xoperating systems
withPython 3.6 or higherinstalled. You can also use it onWindowsif you haveWSLconfigured.The remote host must also be running onLinuxorMac OS X. The local and remote hosts can be running different operating systems. The only requirement is that the remote host must be accessible usingsshfrom the local host.Getting StartedInstalling on Mac OS XIf you use Mac OS X, you can installremoteusingHomebrewfrom ourcustom tap:brewinstallremote-cli/remote/remoteThen, you will always be able to update it to the latest version:brewupgraderemoteInstalling on other systemsremotedoesn't support any package managers other thanbrewyet. However, it can be manually downloaded
and installed. To do it, visithttps://github.com/remote-cli/remote/releasesand download the latest released-shivarchive, unpack it to some local directory (e.g.,~/.bin) and add it to PATH:mkdir-p~/.bin
tar-C~/.bin-xzf~/Downloads/remote-1.4.5-shiv.tgzecho'export PATH=$PATH:/home/username/.bin/remote/bin'>>~/.bash_profilesource~/.bash_profileDon't forget to replace the/home/usernameabove with the actual path to your home directory.Configuring the remote hostremoteCLI needs to be able to establish a passwordless SSH connection to the remote host.
Please runssh -o BatchMode=yes <your-host> echo OKto confirm that everything is ready for you.
If this command fails, please go throughSSH guideto set up
SSH keys locally and remotely.First runAfter you are done with the configuration, switch the working directory to the root of your workspace in
terminal and runremote-initto create a configuration file:cd~/path/to/workspace
remote-initremote-host.example.comThis will create a config file named.remote.tomlin the workspace root
(~/path/to/workspace/.remote.toml). This file controls the remote connection and synchronization options.
You can read more about this file in the Configuration section of this doc.After it, you can start using remote:# This will sync workspace and run './gradlew build' remotelyremote./gradlewbuild# This will forcefully push all local files to the remote machineremote-push# This will bring in ./build directory from the remote machine to local even if# the CLI is configured to ignore itremote-pullbuildDistributionremote's distribution comes with a set of executables:remote-init: set up a local directory to point to a remote directory on a target hostremote-ignore: set up directories/files to ignore while pushingremote-push: explicitly push local changes remoteremote-pull: pull a directory from remote to localremote: execute a command remotely, after first syncing the local tree with the remote treeremote-explain: explain your remote setup, explain what command actually will get runremote-quick: execute a command remotely without syncing the treesremote-add: add another remote host to the mirror listmremote: execute a remote command on all the hosts, after first syncing the local tree with the remote treesYou can run each of these commands with--helpflag to get a list of options and arguments they accept.ConfigurationThree configuration files control the behavior ofremote:~/.config/remote/defaults.tomlis a global config file. It sets options that affect all the workspaces
unless they are overwritten by.remote.tomlfile..remote.tomlis a workspace config that is expected to be placed in the root of every workspace.
TheremoteCLI cannot execute any commands remotely until this file is present, or the global config
overwrites this withallow_uninitiated_workspacesoption..remoteignore.tomlis a workspace config that controls only sync exclude and include patterns
and has the highest priority. While the same settings can be specified in the.remote.tomlfile,
you can use this file to check in project-specific ignore settings in the VCS because it doesn't contain
host-specific information in it.Both configs useTOMLformat.Workspace rootis a root directory of the project you're working on.
It is identified by the.remote.tomlfile. Each time you executeremotefrom workspace root or any of its
subdirectories,remotesyncs everything under workspace root with the destination host before running the command.Global Configuration FileGlobal configuration file should be placed in~/.config/remote/defaults.toml. This config file is optional
and theremoteCLI will work with the default values if it is absent. This is the example of how it looks like:[general]allow_uninitiated_workspaces=falseuse_relative_remote_paths=falseremote_root=".remotes"[[hosts]]host="linux-host.example.com"label="linux"[[hosts]]host="macos-host.example.com"port=2022supports_gssapi_auth=falsedefault=truelabel="mac"[push]exclude=[".git"][pull]exclude=["src/generated"]include=["build/reports"][both]include_vcs_ignore_patterns=true[general]block controls system-wide behavior for theremoteCLI.Reference:allow_uninitiated_workspaces(optional, defaults tofalse) - If this flag is set totrueand
the global config contains at least one remote host,remotewill treat its current working directory
as a workspace root even if it doesn't have.remote.tomlfile in it.Warning:If this option is on and you runremotein the subdirectory of an already configured workspace,remotewill ignore workspaces configuration and treat subdirectory as a separate workspace root.remote_root(optional, defaults to".remotes") - The default directory on the remote machine that
will be used to store synced workspaces. The path is expected to be relative to the remote user's home
directory, so.remoteswill resolve in/home/username/.remotes.
If the workspace-level configuration sets thedirectoryfor a host, this setting will be ignored.use_relative_remote_paths(optional, defaults tofalse)If set tofalse, all the workspaces will be stored in theremote_rootof the target host in a flat
structure. Each directory will have a name like<workspace_name>_<workspace_path_hash>.If set totrue, the remote path will be placed in theremote_roottree like it was placed in the user's
home directory tree locally. Some examples:If the local path is/home/username/projects/work/project_name, the remote path will be/home/username/.remotes/projects/work/project_nameIf the local path is/tmp/project_name, the remote path will be/home/username/.remotes/tmp/project_name[[hosts]]block lists all the remote hosts available for the workspaces. Used when the workspace
configuration doesn't overwrite it.You can provide multiple hosts in this block, but only one will be selected when you executeremote.
It would be either the host that is marked bydefault = trueor the first one in the list if no
default was set explicitly.You can run most of the commands with--label label|numberor-l label|numberoption to run a
command on the non-default host.labelhere is the text label you put in the config file,numberis
a number of required host in the hosts' list, starting from 1.Reference:host- a hostname, IP address, or ssh alias of a remote machine that you want to use for remote execution.port(optional, defaults to22) - a port used by the ssh daemon on the host.supports_gssapi_auth(optional, defaults totrue) -trueif the remote host supportsgssapi-*auth
methods. We recommend disabling it if the ssh connection to the host hangs for some time during establishing.default(optional, defaults tofalse) -trueif this host should be used by defaultlabel(optional) - a text label that later can be used to identify the host when running theremoteCLI.cmd_prefix(optional) - a string which is prefixed to all commands executed via theremoteCLI. The prefix isnotshell escaped.[push],[pull], and[both]blocks control what files are synced from local to a remote machine and back
before and after the execution. These blocks are used when the workspace configuration doesn't overwrite them.pushblock controls the files that are uploaded from local machine to the remote one.pullblock controls files that are downloaded from remote machine to local one.bothblock extends previous two.Each one of these blocks supports the following options:exclude(optional, defaults to empty list) - a list of rsync-style patterns. Every file in the workspace
that matches these patterns won't be synced unless it is explicitly specified ininclude.include(optional, defaults to empty list) - a list of rsync-style patterns. Every file in the workspace
that matches these patterns will be synced even if it matches theexclude.include_vcs_ignore_patterns(optional, defaults tofalse) - iftrueand.gitignoreis present,
all its patterns will be included in theexcludelist.Workspace Configuration FileThis is the example of how standalone workspace-level.remote.tomlconfiguration file looks like:[[hosts]]host="linux-host.example.com"directory=".remotes/workspace"label="linux"supports_gssapi_auth=truecmd_prefix="nice -n5"[[hosts]]host="macos-host.example.com"port=2022directory=".remotes/other-workspace"supports_gssapi_auth=falsedefault=truelabel="mac"[push]exclude=[".git"][pull]exclude=["src/generated"]include=["build/reports"][both]include_vcs_ignore_patterns=trueAll the used blocks here are similar to the ones in the global config file. However, you cannot put[general]block in this file. Also, you can provide one more option in[[hosts]]block:directory(optional) - a path relative to remote user's home. It will be used to store the workspace's
file on the remote machine.Also, if you set at least one value for any of the blocks in the workspace-level config,
all the values from this block in the global config will be ignored.
There is a way to change this behavior. You can use[extends.*]blocks to do it.Here is an example. Imagine, you have a following global config:[[hosts]]host="linux-host.example.com"label="linux"default=true[push]exclude=[".git"][both]include_vcs_ignore_patterns=trueIf you want to be able to use the same Linux host in the workspace, but you want to add one more and modify some exclude patterns, you can create the following workspace config:[[extends.hosts]]host="mac-host.example.com"directory=".remotes/mac-workspace"label="mac"default=true[extends.push]exclude=["workspace-specific-dir"]include=[".git/hooks"][both]include_vcs_ignore_patterns=falseAs you can see, some block names start withextends.. This name tells remote to merge the
workspace and global settings.There are a few things to note:If both workspace-level and global configs define a default host, the workspace-level config winsHosts ordering is preserved, and globally configured hosts always go first.If an option value is a list (e.g.exclude), it is extended. Otherwise, the value is overwritten.Workspace Files Sync Configuration File.remoteignore.tomlfiles is similar to.remote.toml, but only supportspush,pull,both,extends.push,extends.pullandextends.bothblocks. It also cannot be used to identify
the workspace root..remoteenv fileSometimes you will need to do some action each time before you execute some remote command.
A common example will be to executepytestin the virtual environment: you need to activate it
first, but the activation state won't be preserved between theremoteruns.There are two ways of solving this problem:Running both initiation logic and the command together:remote'source env/bin/activate && pytest'Creating a file called.remoteenvin the workspace root. If this file is present,remotewill
always runsource .remoteenvon the destination host before running the actual command. For example,
here is how you can runremote's tests on the other hosts:gitclonegit@github.com:remote-cli/remote.gitcdremote
remote-init<remote-host-name>
remotepython3-mvenvenvecho'. env/bin/activate'>>.remoteenv# starting from this point, all python commands will be executed in virtualenv remotely# This should print virtualenv's python pathremotewhichpython
remotepipinstall-e.
remotepipinstall-rtest_requirements.txt
remotepytestThe.remoteenvfile is guaranteed to sync to remote machine even if it is excluded by the workspace's.gitignorefile or other rules.Development & ContributionTo bootstrap the development run:gitclonegit@github.com:remote-cli/remote.gitcdremote
python3-mvenvenvsourceenv/bin/activate
pipinstall-e.
pipinstall-rtest_requirements.txtAfter it, you can open the code in any editor or IDE you like. If you prefer VSCode, the project contains the configuration file for it.Before submitting your pull request, please check it by running:flake8srctest&&mypy-premote&&black--check-l120srctest&&isort--check-onlysrctest&&pytestIfblackorisortfails, you can fix it using the following command:black-l120srctest&&isortsrctestDon't forget to add changed files to your commit after you do it.Changelog1.13.3Add cmd-prefix support in [[hosts]]Avoid gitignoring remoteignor.toml1.13.2Fix the shiv creation script1.13.1Move the pypi repo to remote-exec-api1.13.0Allow providing extra arguments for the SSH command1.12.0Allow providing paths for remote-pushSupport spaces in paths and improve shell escaping1.11.0Change order of application of API-provided env and user-provided env1.10.1Enable tunelling in remote-quick1.10.0Allow providing multiple ports for tunneling1.9.1fixed 1.9.0 release1.9.0improve sync include/exclude API1.8.0Introduce.remoteignore.tomlfileFix typo ininclude_vcs_ignore_patternsoption nameStart usingshlexto escape the shell command1.7.0improve the internal port forwarding API1.6.1add an API to set environment variables for the remote execution1.6.0support source changes streaming with watchdogadd remote-explain CLI entry pointadd --multi and --log flags to remote entry pointadd --push and --pull flags to remote-ignore entyr point1.5.0use TOML configs by defaultnew CLI option to select a host by label/indexnew config option to provide custom port for SSH.remoteenv usage fixes1.4.5Select host by their labelConfigure ssh connection options1.4.4Add the host labeling in toml config files1.4.3Fixed rsync bugs1.4.2Add the possibility to start an ssh tunnel during the remote execution1.4.1Fail ssh and rsync when password is requested by adding -o BatchMode=yesadd -K to rsync ssh command to make it homogenous with sshSupport rsync's --include-from to make exclude overrides possible1.4.0TOML config support.gitignore files can be used as a source for sync ignore1.3.8Unit testsReleases with good versions1.3.7Moving to pypi releases1.3.6Fix reverse-sync breakage for remote-dirs without trailing /Reverse-sync always, even when remote command fails1.3.5Support for relative and absolute paths (including leading ~ and /) in remote directorySupport for global variables $RSHELL and $RSHELL_OPTS to configure remote shell invocation1.3.4Support for choice of login shell and shell options for remote host in .remote fileBetter initialization for mremote screen1.3.3Fixing a bug in command execution for zsh based shells1.3.2Fixing signal handling, cleanup of temp filesFixing a bug in inferring verbose mode1.3.1More comprehensive ignore rule specifications (push, pull, both) for .remoteignore filesCommand-line argument improvements to remote, remote-push, remote-pull and remote-explain1.3.0Adding command line flags for remote, remote-pushDropping remote-push-dry since remote-push -n is an equivalent alternative.Improving remote-explain output to include diff status1.2.7remote-explain supports -c flag to test connection to the remote host1.2.6Support for remote-explain, a tool to explain your remote setup1.2.5Support for adding .remote related files to .gitignore1.2.4More robust parameter parsing1.2.3Support for initializing environment per remoted directory through .remoteenv file1.2.2Fail fast when local directory is not setup for remoteSupport kerberos auth (through ssh -K)1.2.1New executables: remote-set, remote-hostmremote now uses screen1.2.0Support for multiple-hostsNew executables: remote-add and mremote1.1.3remote-pull: now pulls without syncing the treesremote-quick: when you want to execute a command without syncing the trees1.1.2More fixes to md5 generation (verified to work on mac and linux)Silent ssh connections for reduced verbosity1.1.1Robust in finding appropriate md5 utility1.1Added remote-pullMoved remote-sync to remote-push1.0First drop