Package: privateer
privateerThe ideaWe need a way of synchronising some docker volumes from a machine to some backup server, incrementally, usingrsync. We previously usedoffen/docker-volume-backupto backup volumes in their entirety to another machine as a tar file but the space and time requirements made this hard to use in practice.The setupWe assume some number ofservermachines -- these will receive data, and some number ofclientmachines -- these will send data to the server(s).  A client can back any number of volumes to any number of servers, and a server can receive and serve any number of volumes to any number of clients.A typical framework for us would be that we would have a "production" machine which is backing up to one or more servers, and then some additional set of "staging" machines that receive data from the servers, which in practice never send any data.Because we are going to use ssh for transport, we assume existence ofHashiCorp Vaultto store secrets.ConfigurationThe system is configured via a singlejsondocument,privateer.jsonwhich contains information about all the moving parts: servers, clients, volumes and the vault configuration. Seeexample/for some examples.We imagine that your configuration will exist in some repo, and that that repo will be checked out on all involved machines. Please add.privateer_identityto your.gitignorefor this repo.SetupAfter writing a configuration, on any machine runprivateer keygen --allwhich will generate ssh keypairs for all machines and put them in the vault. This only needs to be done once, but you might need to run it again ifyou add more machines to your systemyou want to rotate keysOnce keys are written to the vault, on each machine runprivateer configure <name>replacing<name>with the name of the machine within either theserversorclientssection of your configuration.  This sets up a special docker volume that will persist ssh keys and configurations so that communication between clients and servers is straightforward and secure.  It also leaves a file.privateer_identityat the same location as the configuration file, which is used as the default identity for subsequent commands. Typically this is what you want.Servers must be started before any backup is possible. To do this, runprivateer server startOnce started you can stop a server withprivateer server stop(or just kill the container) and find out how it's getting on withprivateer server statusManual backupTo back up a volume onto one of your configured servers, run:privateer backup <volume> [--server=NAME]Add--dry-runto see the commands to run it yourself.Scheduled backupsEach client can run a long-lived container to perform backups on some schedule usingyacron. If your client configuration contains aschedulesection then you can run the commandprivateer schedule startto start the scheduled tasks.RestoreRestoration is always manualprivateer restore <volume> [--server=NAME] [--source=NAME]where--servercontrols the server you are pulling from (useful if you have more than one configured) and--sourcecontrols the original machine that backed the data up (if more than one machine is pushing backups).For example, if you are on a "staging" machine, connecting to the "backup" server and want to pull the "user_data" volume that was backed up from "production" machine called  you would typeprivateer restore user_data --server=backup --source=productionPoint-in-time backup and recoveryPoint-in-time backup is always taken on the server side, and converts a copy of a volume held on the server to atarfile, on the host machine and outside of any docker volume. These can then be manually copied around and use to initialise the contents of new volumes, in a way similar to the normal restore path.The command to export the volume is:privateer export <volume> [--to-dir=PATH] [--source=NAME]which will bring up a new container and create the tar file within the directoryPATH. The name will be automatically generated and include the curent time, volume name and source.  Thesourceargument controls who backed the volume up in the first place, in the case where there are multiple clients.  It can be omitted in the case where there is only one client performing backups, andmustbe ommitted in the case where you are exporting a local volume.You can point this command at any volume on any system whereprivateeris installed to make atarfile; this might be useful for ad-hoc backup and recovery. If you have a volume calledredis_data, thenprivateer export redis_datawill create a new fileredis_data-<timestamp>.tarin your working directory.Given atarfile, recovery looks like:privateer [--dry-run] import <tarfile> <volume>This does not need to be run anywhere with aprivateer.jsonconfiguration, and indeed does not try and read one. It will fail if the volume exists already, making the command fairly safe.We could copy the file created in theredis_dataexample above to another machine and runprivateer import redis_data-<timestamp>.tar redis_datato export thetarfile into a new volumeredis_data.What's the problem anyway?Docker volumesare useful for abstracting away some persistent storage for an application. They're much nicer to use than bind mounts because they don't pollute the host system with immovable files (docker containers often running as root or with a uid different to the user running docker).  The dockerdocsdescribe some approaches to backup and restore but in practice this ignores many practical issues, especially when the volumes are large or off-site backup is important.We want to be able to synchronise a volume to another volume on a different machine; our setup looks like this:bob                            alice
+-------------------+          +-----------------------+
|                   |          |                       |
| application       |          |                       |
|  |                |          |                       |
| volume1           |          |     volume2           |
|  |                |   ssh/   |      |                |
| privateer-client--=----------=---> privateer-server  |
|  |                |  rsync   |      |                |
| keys              |          |     keys              |
|                   |          |                       |
+-------------------+          +-----------------------+so in this casebobruns aprivateerclient which sends data over ssh+rsync to a server running onalice, eventually meaning that the data involume1onbobis replicated tovolume2onalice.  This process uses a set of ssh keys that each client and server will hold in akeysvolume.  This means that they do not interact with any ssh systems on the host.  Note that ifaliceis also running sshd, this backup process will use asecondssh connection.In addition, we will support point-in-time backups onalice, creatingtarfiles of the volume onto disk that can be easily restored onto any host.Installationpip install privateerLicenseprivateeris distributed under the terms of theMITlicense.