Package: gcloud-sync-ssh
gcloud_sync_sshAn improved version ofgcloud compute config-ssh, a convenient way to setup connectivity to your virtual machines hosted within GCP.This tools has some benefits:Can work on several projects at onceOnly sets up connectivity by setting Hostnames in SSH config.No side effects in project or instance metadata (madness!)Works quite faster because updating project metadata isslowCan generate user-specified SSH config for all hostsHost specific config is not handled by the tool. Edit your configuration instead.Can remove stopped ('terminated', in GCP parlance) instances from configGenerates smallest possible diff by preserving casing, separators and whitespaceSmallest diff = highest review impact.And some drawbacks:If the instance is not configured to let you in, this tool willnotfix that.I've used it for a couple of months for my modest needs and it did the job flawlessly. YMMV - see 'Limitations' below.InstallWith pipxpipx install gcloud_sync_sshRun from sourceClone itRecommended: create a virtualenv with [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenvi.e.pyenv virtualenv 3.6 gcloud_sync_sshpip install -r requirements.txtN.B. This will install development dependencies as wellpython -m gcloud_sync_sshUsagegcloud_sync_ssh --helpThe tool works in four phases. It optionally changes the gcloud authentication context, then it enumerates projects and instances, then it updates your SSH config in memory and finally saves it after making a backup and showing you the diff.Preliminary: Choose which config file to editBy default,gcloud_sync_sshedits the standard~/.ssh/config. This will be fine for most uses. It may be changed with-c/--config.Some people prefer the tool to operate on an isolated file, like~/.ssh/config.auto. A few things to keep in mind if you go down that path:You need tochmod 0600 ~/.ssh/config.autofor SSH to read itYou need to addInclude ~/.ssh/config.autoto your existing~/.ssh/configbeforeanyMatchorHostblock.Alternatively, usessh -F ~/.ssh/config.autoinstead of setting up theInclude.First phase: AuthenticationFirst it will optionally change yourgcloudauthentication context as desired, using the--loginoption for email accounts or--service-accountoption for service accounts.If none of these options are specified,gcloud_sync_sshwill not touch modify your gcloud
authentication settings - and will function with your current settings.Your current settings will be restored once the tools exits.Be aware that authentication is a user-level setting. When using special authentication, avoid runninggcloudwhilegcloud_sync_sshis running. This also applies to other tools that piggy back on gcloud auth, like Terraform gcloud backend in application-default mode.Second phase: EnumerationYou may select a project with the--projectoption. This option can be specified several times to select multiple projects. This option acceptsfnmatch-style globbing (i.e. using*,?...).Just one project :--project my-project-nameA couple of projects:--project project-web-1 --project-web-2A bunch of projects:--project 'project-web-*'(Beware shell quoting rules for globbing characters)All projects :--all-projectsThird phase: Configuration updatesThere are quite a few cases to consider.UpdatesThe simplest case is updates : running instances already in your configuration, but the external IP changes.Your configuration will be updated in place and nothing else will change. There are no options controlling this behavior.DeletionsThen we have deletions:Instances in your configuration that have entered theSTOPPEDstatus will be removed from your configuration. You can disable this using the-nrs|--no-remove-stoppedflag.Deleted instances (or instances that have otherwise vanished) will be removed from your SSH configif and only ifit's possible to attribute the instance to a project using its hostname i.e. its hostname terminates by.<project-name>. You can disable that using the-nrv|--no-remove-vanishedflag.AdditionsFinally, we have additions. A newHostblock will be added to your configuration for newly detected instances. The complexity comes from the many ways to specify how that block will be generated. There are two parts to this : the 'template' is the set of options that doesn't change from host to host, and the 'specifics' which are the couple of options that are allowed to vary.TheHosttemplateis computed as such:The tool has builtin defaults. You can see them with--no-inference --debug-template.The tool tries toinfera template from your existing configuration. Any keyword-argument pair that is shared byeveryHostblock in your config is added to this template. The inferred values override the defaults. You can turn this off by passing the--no-inferenceflag. You can see the results of this stage by passing-dt|--debug-template.You can specify additions to the template using the-kw/--kwargoption possibly several times.Here is an exemple of a Host template that overrides defaults :$ gcloud_sync_ssh --debug-template --no-inference -kw IdentityFile=/secret/id_rsa -kw UserKnownHostsFile=/data/known_hosts
2020-09-07 20:23:18.290 | INFO     | Displaying host template
    CheckHostIP no
    IdentitiesOnly yes
    IdentityFile /secret/id_rsa
    UserKnownHostsFile /data/known_hosts

2020-09-07 20:23:18.291 | INFO     | Done displaying host templateThe 'specifics' are:TheHostnamekwarg, that will be set to the instance external IP (or first external IP, if there are several). This is the whole point of this tool, and it cannot be controlled by an option.TheHostKeyAliaskwarg, that will be set tocompute.<instance_id>. This is whatgcloud compute config-sshdoes. This will prevent warnings because of external IP changes. You can disable generating those with-nk|--no-host-key-alias.Fourth phase: Configuration saveYou don't have to blindly trust the tool. By default it will show you the diff and ask for approval before saving - while still saving a backup.If you want to use it fromcronor CI, this behavior might be counterproductive, so it can be disabled:Don't show diff and don't ask for approval:--not-interactiveDon't save a backup:--no-backupExamplesLimitationsOnly works with one account at a time (TODO: Support iterating through all accounts exposed bygcloud auth list)Can only be setup through commandline options (TODO: Support configuration file on top of gazillion command line options)Doesn't support "jump box" setups or VPN setups - where you connect to the private IP address of your instances. (TODO: Support that!)Is single-threaded synchronous (TODO: Support parallelism with either threads or async)Formatting of new hosts is notexactlythe same as whatgcloud compute config-sshdoes. Notably, it has consistent space delimiting instead of having=on some lines andon others. (Probably won't fix)There are no ways to setup 'specific' options other than the two builtins for newHost. (TODO: Accept Python plugins to allow arbitrarily complex schemes to add/edit SSH config per host)Vanishing/deleted instances can only be removed from your config if their hostname is suffixed by.<project-name>. This is the GCP default. I found no other way to attribute a Host in your SSH config to a given SSH project. Workaround: remove everything withgcloud compute config-ssh --removethen usegcloud_sync_sshas usual. (TODO: Support--overwriteflag that removes everything in the config block before running)Instances in transitional states and suspended states are completely ignored by the tool.The above are roughly sorted given how concerning they are to me. None are preventing me to achieve my goals with the tool - but some may hinder you.AlternativesUsing this script to setup connectivity should work well enough for ~hundreds of instances whose external IPs change ~daily. I use it for ~dozens of instances whose IPs change ~daily.If your external IPs change very frequently, or if you have thousands of instances, this tool can still be a useful crutch, but you may want to look into alternatives.Here are my thoughts on the subject, ordered by effort required.Setup a shared private subnetwork between your instances, and you'll get DNS with.internalTLD for free. A jumpbox is a way to get access to your instances. [ XXX Other ways ? ]Use some form of service registration/discovery. Hashicorp Consul comes to mind.Use Google Logging to process instance startup/showdown logs. Tap those into Pub/Sub. Write a lambda function that produces ahostsfile or a SSH config file. Store that in Storage at a well known URI. Download that in whichever way you like, thenIncludeit in your SSH config.Same as above, but generate a zone file and feed that to a DNS server you control.Move all your things to K8s and welcome your new YAML overlords. You can't ssh into anything without usingkubectlbut that's okay.ContributingGrab a TODO from theLimitationslist, an issue or bring your own issue to solve.If it makes sense, please add tests and make sure they pass withpython -m pytest.LicenseThe code contained in this repository is licensed under the terms of theMIT licenseunless otherwise noted in the source code file.