Package: vault-redirector
Python/Twisted application to redirect Hashicorp Vault client requests to the active node in a HA cluster.API Documentation:http://vault-redirector-twisted.readthedocs.org/en/latest/StatusThis application is currently very young. Please ensure it meets your needs before using it in production.NOTE:My initial plan was to implement this in Go. My Go knowledge is severely lacking, and the performance of Python/Twisted at 1,000 requests per second is within 25% of the Go variant. Please consider this package to be temporary, until work on the Go version (https://github.com/manheim/vault-redirector) continues.PurposeThere’s a bit of a gap in usability ofVaultin aHigh Availabilitymode, at least in AWS:Vault’s HA architecture is based on an active/standby model; only one server can be active at a time, and any others are standby. Standby servers respond to all API requests with a 307 Temporary Redirect to the Active server, but can only do this if they’re unsealed (in the end of theHA docs: “It is important to note that only unsealed servers act as a standby. If a server is still in the sealed state, then it cannot act as a standby as it would be unable to serve any requests should the active server fail.”).HashiCorp recommends managing infrastructure individually, i.e. not in an auto-scaling group. In EC2, if you want to run Consul on the same nodes, this is an absolute requirement as Consul requires static IP addresses in order for disaster recovery to work without downtime and manual changes.As a result, we’re left with a conundrum:We can’t put Vault behind an Elastic Load Balancer, because that would cause all API requests to appear to have the ELB’s source IP address. Not only does this render any of the IP- or subnet-based authorization methods unusable, but it also means we lose client IPs in the audit logs (which is likely a deal-breaker for anyone concerned with security).The only remaining option for HA, at least in AWS, is to use Route53 round-robin DNS records that have the IPs of all of the cluster members. This poses a problem because if one node in an N-node cluster is either offline or sealed, approximately 1/N of all client requests will be directed to that node and fail.While it would be good for all clients to automatically retry these requests, it appears that most client libraries (and even thevaultcommand line client) do not currently support this. While retry logic would certainly be good to implement in any case, it adds latency to retrieving secrets (in the common case where the cluster is reachable, but some nodes are down) and also does not account for possible DNS caching issues. Furthermore, we’re providing Vault as a service to our organization; relying on retries would mean either adding retry logic to every Vault client library and getting those changes merged, or deviating from our plan of “here’s your credentials and endpoint, see the upstream docs for your language’s client library.”The best solution to this problem would be forVault issue #799, a request to addPROXY Protocolsupport to Vault, to be completed. BothAWS ELBsand HAProxy support this, and it would alleviate issue #1 above, allowing us to run Vault behind a load balancer but still have access to the original client IP address.This small service is intended to provide an interim workaround until that solution is implemented.FunctionalityWe take advantage of Vault’s 307 redirects (and the assumption that any protocol-compliant client library will honor them). Instead of connecting directly to the Vault service, clients connect to a load-balanced daemon running on the Vault nodes. This daemon asynchronously polls Consul for the health status of the Vault instances, and therefore knows the currently-active Vault instance at all times. All incoming HTTP(S) requests are simply 307 redirected to the active instance. As this service can safely be load balanced, it will tolerate failed nodes better than round-robin DNS. Since it redirects the client to the active node, the client’s IP address will be properly seen by Vault.RequirementsPython 2.7, 3.3, 3.4 or 3.5 andpip;virtualenvis recommended.gccor another suitable C compiler and the python headers/development package for your OS; thetwistedpackage has dependencies with native extensions which must be compiled at install time.Thetwistedpackage (will be installed automatically viapip).Therequestspackage (will be installed automatically viapip).If you wish to use TLS for incoming connections (highly recommended), thepyOpenSSLandpempackages. These can be automatically installed along with vault-redirector by usingpip installvault-redirector[tls].Consulrunning and configured with service checks for Vault (see below)Vault 0.6.0+Consul Service ChecksIn order to determine the active Vault instance,vault-redirectorrequires that Consul be running and monitoring the health of all Vault instances. Redirection can be to either the IP address or Consul node name running the active service.The current implementation in this package requires Vault 0.6.0+, which automatically
registers its own service and health checks with Consul.InstallationWe recommend installing inside an isolated virtualenv. If you don’t want to do that and would rather install system-wide, simply skip to the last two steps:Ensure thatgccor another suitable C compiler is installed.virtualenv vaultsource vault/bin/activatepip installvault-redirector; if you wish to use TLS for incoming connections (highly recommended) you’ll also need thepyOpenSSLandpempackages, which will be installed automatically if you instead runpip installvault-redirector[tls]UsageCommand Line UsageAll options and configuration are passed in via command-line options.jantman@exodus$vault-redirector-husage: vault-redirector [-h] [-v] [-l] [-V] [-S] [-I] [-p POLL_INTERVAL]
                        [-P BIND_PORT] [-C CHECKID] [-c CERT_PATH]
                        [-k KEY_PATH]
                        CONSUL_HOST_PORT

Python/Twisted application to redirect Hashicorp Vault client requests to the
active node in a HA cluster

positional arguments:
  CONSUL_HOST_PORT      Consul address in host:port form

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output. specify twice for debug-level output.
                        See also -l|--log-enable
  -l, --log-disable     If specified, disable ALL logging after initial setup.
                        This can be changed at runtime via signals
  -V, --version         show program's version number and exit
  -S, --https           Redirect to HTTPS scheme instead of plain HTTP.
  -I, --ip              redirect to active node IP instead of name
  -p POLL_INTERVAL, --poll-interval POLL_INTERVAL
                        Consul service health poll interval in seconds
                        (default 5.0)
  -P BIND_PORT, --port BIND_PORT
                        Port number to listen on (default 8080)
  -C CHECKID, --checkid CHECKID
                        Consul service CheckID for Vault (default:
                        "service:vault"
  -c CERT_PATH, --cert-path CERT_PATH
                        Path to PEM-encoded TLS certificate. If you need a
                        certificate chain to verify trust, this file should be
                        composed of the server certificate followed by one or
                        more chain certificates. If specified, you must also
                        specify -k|--key-path
  -k KEY_PATH, --key-path KEY_PATH
                        Path to PEM-encoded TLS private key. If specified, you
                        must also specify -c|--cert-pathBy default,vault-redirectorwill redirect clients to the hostname (Consul
health checknode name) of the active Vault node, over plain HTTP. This can
be changed via the-I|--ipand-S|--httpsoptions.Usage with TLSvault-redirector is capable of listening with TLS/HTTPS, which is strongly
recommended as clients will send their authentication tokens as HTTP headers.
To do so, specify the-k|--key-pathand-c|--cert-pathoptions with the
paths to the key and certificate files, respectively. Each should be a
PEM-encoded file; if your certificate requires a trust/CA chain to be presented
to clients, the file atcert-pathshould be a combined certificate and chain
file, composed of the PEM-encoded certificate concatenated with one or more PEM-encoded
chain certificates.Running as a Daemon / ServiceFor anything other than testing,vault-redirectorshould be run as a system
service. There is no built-in daemonizing support; this is left up to your
operating system.Here is an examplesystemdservice unit file forvault-redirector, assuming you wish to run it as avaultredirectoruser and group, and it is installed into a virtualenv at/usr/local/vault-redirector, and Consul is running on localhost (127.0.0.1)
on port 8500. This will start the service with logging disabled (-l) but set
to INFO level (-v); logging can be turned on with SIGUSR1 as described below.[Unit]Description=Vault RedirectorRequires=basic.target network.targetAfter=basic.target network.target[Service]Type=simpleUser=vaultredirectorGroup=vaultredirectorPrivateDevices=yesPrivateTmp=yesProtectSystem=fullProtectHome=read-onlyCapabilityBoundingSet=NoNewPrivileges=yesExecStart=/usr/local/vault-redirector/bin/vault-redirector -v -l 127.0.0.1:8500RestartSec=5sTimeoutStopSec=30sRestart=always# disable all rate limiting; let it restart foreverStartLimitInterval=0[Install]WantedBy=multi-user.targetIf you wish to both use TLS for incoming connections and redirect to a HTTPS URL for Vault,
theExecStartline would be:ExecStart=/usr/local/vault-redirector/bin/vault-redirector -v -l -S --cert-path=/path/to/server.crt --key-path=/path/to/server.key 127.0.0.1:8500Health CheckVault-redirector will respond to a request path of /vault-redirector-health with
a JSON body something like the following; this can be used for load balancer
health checks. If the active vault instance is known, the HTTP status code will
be 200. Otherwise (i.e. if there is no active vault node or if Consul is unreachable)
it will be a 503.{"healthy":true,"application":"vault-redirector","version":"0.1.0","consul_host_port":"127.0.0.1:8500","source":"https://github.com/manheim/vault-redirector-twisted","active_vault":"vault_hostname_or_ip:port","last_consul_poll":"YYYY-MM-DDTHH:MM:SS"}Logging and DebuggingPython’s logging framework can impose a slight performance penalty even for messages
which are below the level set to be displayed (simple testing reports 10x execution
time for logging to a level below what’s set, vs guarding the log statements with
a conditional). As a result, in addition to Python’s normal logging verbosity
levels, all logging statements after initial setup are guarded by a global
“logging enabled” boolean; if logging is not enabled, the calls to Python’s
logging framework will never be made. This behavior can be enabled by running
the process with the-lor--log-disableoptions (which is the
recommended production configuration).Note that this functionality is completely separate from the logging module’s
levels, which are controlled by the-v/-vvoptions (and are not currently
changeable at runtime).At any time, logging can be enabled by sending SIGUSR1 to the process, or disabled
by sending SIGUSR2 to the process.SupportPlease open any issues or feature requests in themanheim/vault-redirector-twisted GitHub issue trackerThey will be dealt with as time allows. Please include as much detail as possible, including your version ofvault-redirectorand the Python version and OS/distribution it’s running on, as well as the command line arguments used when running it. Debug-level logs will likely be very helpful.DevelopmentPull requests are welcome. Please cut them against themasterbranch of themanheim/vault-redirector-twistedrepository.It is expected that test coverage increase or stay the same, that all tests pass,
that any new code have complete test coverage, and that code conforms topep8and passespyflakes.After making any changes to the code, before submitting a pull request, runtox-edocsto regenerate the API documentation. Commit any changes to the auto-generated files underdocs/source.Installing for DevelopmentFork themanheim/vault-redirector-twistedrepository on GitHub.Clone your fork somewhere on your local machine andcdto the clone:$gitclonegit@github.com:YOUR-GITHUB-USER/vault-redirector-twisted.git$cdvault-redirector-twistedAdd the manheim upstream repository as a git upstream, so you can pull in
upstream changes, and fetch it:$gitremoteaddupstreamhttps://github.com/manheim/vault-redirector-twisted.git$gitfetchupstreamCreate a virtualenv for testing and running vault-redirector, install
your local source into it, and installtoxfor testing:$virtualenv.$sourcebin/activate$pipinstall-e.$pipinstalltoxpyOpenSSLpemCheck out a new git branch. If you’re working on a GitHub issue you opened, your
branch should be called “issues/N” where N is the issue number.TestingTesting is accomplished viapytestandtox. By default tests will be run
for Python 2.7, 3.3, 3.4. 3.5 and the documentation. Each supported Python interpreter has two test suites,unitandacceptance. Theacceptancesuite will actually run vault redirector bound to an available port (but with the Consul active node query code mocked out) and make example HTTP requests against it.To run the tests locally, with your virtualenv activated, runtox-epy<version>-(unit|acceptance)where<version>is one of the Python versions intox.ini(i.e. “27”, “33”, “34” or “35”). You will need to already have the appropriate Python interpreter version installed on your system. When the tests are run locally, coverage reports will be generated in thehtmlcov/directory.To generate documentation locally, runtox-edocs; the HTML output will be indocs/build/html. This must be done after making any code changes, and any changes to the auto-generated files underdocs/source/must be committed.Automated testing is accomplished via TravisCI (it’s free for any open source project). If you have a TravisCI account linked to your GitHub, you should be able to add your fork for automated testing without any changes to the repository.MaintenanceInstructions for repository maintainers follow:Fixing Issues / Making ChangesNote that all commit messages should be of the formissue #<ISSUE_NUM>: <descriptive message>. When you’ve verified that the issue is fixed and updateCHANGES.rst, your final commit message should be of the formfixes #<ISSUE_NUM>: <descriptive message>.Follow the instructions above for installing for development.Cut a new branch named after the GitHub issue (“issues/ISSUE_NUMBER”).Make your code changes as needed, and write or update tests. It’s preferred that you commit early and often, to make it easier to isolate work that needs improvements.Run tests locally at least for py27 and py35:tox-epy27-unit,py27-acceptance,py35-unit,py35-acceptanceExamine the test results and the coverage reports inhtmlcov/(the reports will be written for the last-run unit test suite). Iterate until you have full coverage and passing tests.Runtox-edocsto generate documentation locally. Examine it for correctness, and commit any changes to the auto-generated files underdocs/source/.UpdateCHANGES.rstwith a description of your change and a link to the GitHub issue. Commit that.Push your branch to origin. If you believe it’s ready, open a pull request for it.Handling PRsEnsure that all Travis tests are passing for the PR, and that code coverage is still 100% (for all Python versions).Check out the pull request locally. To do this simply, you can edit.git/configin your clone of the repository, and under the[remote "origin"]section add the following lines. Thengit fetch originand you can check out PRs locally likegit checkout refs/pull/origin/PR_NUM. Note that this will be read-only.fetch=+refs/pull/*/head:refs/pull/origin/*Runtox-edocsand ensure there are no changes to the auto-generated docs and that they look correct.Ensure there is an appropriateCHANGES.rstentry for the changes.Ensure thatREADME.rst, if it has been changed, renders correctly on GitHub.If there are any changes to the local repository, cut a new branch locally, commit them, and push it to your fork. You can either ask the original PR author to pull in your changes, or you can close their PR and open a new one for your branch (be sure to reference the closed PR in a comment).Merge the PR to master.Release ProcessOpen an issue for the release; cut a branch offmasterfor that issue.Build docs (tox-edocs) and ensure they look correct. Commit any changes to the auto-generated files.Ensure that Travis tests are passing in all environments.Ensure that test coverage is no less than the last release (ideally, 100%).Ensure there are entries inCHANGES.rstfor all changes since the last release, and that they link to the GitHub issues.Increment the version number invault_redirector/version.pyand add version and release date to CHANGES.rst. Mention the issue in the commit for this, and push to GitHub.Confirm that README.rst renders correctly on GitHub.Upload package to testpypi and confirm that README.rst renders correctly.Make sure your ~/.pypirc file is correct (a repo calledtestforhttps://testpypi.python.org/pypi) and that you havetwineinstalled in your virtualenv. Then:rm-Rfdistpython setup.py register-rhttps://testpypi.python.org/pypipython setup.py sdist bdist_wheeltwine upload-rtest dist/*Check that the README renders athttps://testpypi.python.org/pypi/vault-redirectorCreate a pull request for the release to be merge into master. Upon successful Travis build, merge it.Tag the release in Git, push tag to GitHub:tag the release. for now the message is quite simple:git tag-aX.Y.Z-m'X.Y.Z releasedYYYY-MM-DD'push the tag to GitHub:git push origin X.Y.ZUpload package to live pypi:twine upload dist/*make sure any GH issues fixed in the release were closed.Licensevault-redirector is licensed under the MIT license; seeLICENSEfor the text of the license.