Package: buttervolume
BTRFS Volume plugin for DockerWhat will Buttervolume allow you to do?Quickly recover recent data after an exploit or failure of your web sites or applicationsQuickly rollback your data to a previous version after a failed upgradeImplement automatic upgrade of your applications without fearKeep an history of your dataMake many backups without consuming more disk space than neededBuild a resilient hosting cluster with data replicationQuickly move your applications between nodesCreate preconfigured or templated applications to deploy in secondsWhat can Buttervolume do?Snapshot your Docker volumesRestore a snapshot to its original volume or under a new volumeList and remove existing snapshots of your volumesClone your Docker volumesReplicate or Sync your volumes to another hostSchedule periodic snapshots, sync or replication of your volumesSchedule periodic removal of your old snapshotsHow does it work?Buttervolume is a Docker Volume Plugin that sits on top of a BTRFS partition
and can manage and replicate BTRFS snapshots of your Docker volumes.ContentsBTRFS Volume plugin for DockerIntroductionPrerequisitesBuild and run as a contributorInstall and run as a userUpgradeConfigureUsageRunning the pluginRunning the plugin locally or in legacy modeCreating and deleting volumesManaging volumes and snapshotsCreate a snapshotList the snapshotsRestore a snapshotClone a volumeDelete a snapshotReplicate a snapshot to another hostSynchronize a volume from another host volumePurge old snapshotsSchedule a jobList scheduled jobsCopy-on-writeTestingWorking without a BTRFS partitionMigrate to version 3CreditsCHANGELOG3.10 (2023-04-09)3.9 (2022-10-11)3.8 (2022-10-07)3.7 (2018-12-13)3.6 (2018-12-11)3.5 (2018-06-07)3.4 (2018-04-27)3.3 (2018-04-27)3.2 (2018-04-27)3.1 (2018-04-27)3.0 (2018-04-24)2.0 (2018-03-24)1.4 (2018-02-01)1.3.1 (2017-10-22)1.3 (2017-07-30)1.2 (2017-07-16)1.1 (2017-07-13)1.0 (2017-05-24)IntroductionBTRFSis a next-generation copy-on-write
filesystem with subvolume and snapshot support. A BTRFSsubvolumecan be
seen as an independant file namespace that can live in a directory and can be
mounted as a filesystem and snapshotted individually.On the other hand,Docker volumesare commonly used
to store persistent data of stateful containers, such as a MySQL/PostgreSQL
database or an upload directory of a CMS. By default, Docker volumes are just
local directories in the host filesystem.  A number ofVolume pluginsalready exist for various storage backends, including distributed filesystems,
but small clusters often can’t afford to deploy a distributed filesystem.We believe BTRFS subvolumes are a powerful and lightweight storage solution for
Docker volumes, allowing fast and easy replication (and backup) across several
nodes of a small cluster.PrerequisitesMake sure the directory/var/lib/buttervolume/is living in a BTRFS
filesystem. It can be a BTRFS mountpoint or a BTRFS subvolume or both.You should also create the directories for the config and ssh on the host:sudo mkdir /var/lib/buttervolume
sudo mkdir /var/lib/buttervolume/config
sudo mkdir /var/lib/buttervolume/sshBuild and run as a contributorIf you want to be a contributor, read this chapter. Otherwise jump to the next section.You first need to create a root filesystem for the plugin, using the provided Dockerfile:git clone https://github.com/anybox/buttervolume
./build.shBy default the plugin is built for the latest commit (HEAD). You can build another version by specifying it like this:./build.sh 3.7At this point, you can set the SSH_PORT option for the plugin by running:docker plugin set anybox/buttervolume SSH_PORT=1122Note that this option is only relevant if you use the replication feature between two nodes.Now you can enable the plugin, which should start buttervolume in the plugin
container:docker plugin enable anybox/buttervolume:HEADYou can check it is responding by running a buttervolume command:export RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
alias drunc="sudo runc --root $RUNCROOT"
alias buttervolume="drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume"
sudo buttervolume scheduledIncrease the log level by writing a/var/lib/buttervolume/config/config.inifile with:[DEFAULT]
TIMER = 120Then check the logs with:sudo journalctl -f -u docker.serviceYou can also locally install and run the plugin in the foreground with:python3 -m venv venv
./venv/bin/python setup.py develop
sudo ./venv/bin/buttervolume runThen you can use the buttervolume CLI that was installed in developer mode in the venv:./venv/bin/buttervolume --versionInstall and run as a userIf the plugin is already pushed to the image repository, you can install it with:docker plugin install anybox/buttervolumeCheck it is running:docker plugin lsFind your runc root, then define useful aliases:export RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
alias drunc="sudo runc --root $RUNCROOT"
alias buttervolume="drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume"And try a buttervolume command:buttervolume scheduledOr create a volume with the driver. Note that the name of the driver is the
name of the plugin:docker volume create -d anybox/buttervolume:latest myvolumeNote that instead of using aliases, you can also define functions that you
can put in your .bash_profile or .bash_aliases:function drunc () {
  RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
  sudo runc --root $RUNCROOT $@
}
function buttervolume () {
  drunc exec -t $(docker plugin ls --no-trunc  | grep 'anybox/buttervolume:latest' |  awk '{print $1}') buttervolume $@
}UpgradeYou must force disable it before reinstalling it (as explained in the docker documentation):docker plugin disable -f anybox/buttervolume
docker plugin rm -f anybox/buttervolume
docker plugin install anybox/buttervolumeConfigureYou can configure the following variables:DRIVERNAME: the full name of the driver (with the tag)VOLUMES_PATH: the path where the BTRFS volumes are locatedSNAPSHOTS_PATH: the path where the BTRFS snapshots are locatedTEST_REMOTE_PATH: the path during unit tests where the remote BTRFS snapshots are locatedSCHEDULE: the path of the scheduler configurationRUNPATH: the path of the docker run directory (/run/docker)SOCKET: the path of the unix socket where buttervolume listensTIMER: the number of seconds between two runs of the schedulerDTFORMAT: the format of the datetime in the logsLOGLEVEL: the Python log level (INFO, DEBUG, etc.)The configuration can be done in this order of priority:from an environment variable prefixed withBUTTERVOLUME_(ex:BUTTERVOLUME_TIMER=120)from the [DEFAULT] section of the/etc/buttervolume/config.inifile
inside the container or/var/lib/buttervolume/config/config.inion the
hostExample ofconfig.inifile:[DEFAULT]
TIMER = 120If none of this is configured, the following default values are used:DRIVERNAME = anybox/buttervolume:latestVOLUMES_PATH = /var/lib/buttervolume/volumes/SNAPSHOTS_PATH = /var/lib/buttervolume/snapshots/TEST_REMOTE_PATH = /var/lib/buttervolume/received/SCHEDULE = /etc/buttervolume/schedule.csvRUNPATH = /run/dockerSOCKET = $RUNPATH/plugins/btrfs.sock# only if run manuallyTIMER = 60DTFORMAT =%Y-%m-%dT%H:%M:%S.%fLOGLEVEL = INFOUsageRunning the pluginThe normal way to run it is as a new-style Docker Plugin as described above in
the “Install and run” section, which will start it automatically.  This will
create a/run/docker/plugins/<uuid>/btrfs.sockfile to be used by the
Docker daemon. The<uuid>is the unique identifier of therunc/OCIcontainer running it.  This means you can probably run several versions of the
plugin simultaneously but this is currently not recommended unless you keep in
mind the volumes and snapshots are in the same place for the different
versions. Otherwise you can configure a different path for the volumes and
snapshots of each different versions using theconfig.inifile.Then the name of the volume driver is the name of the plugin:docker volume create -d anybox/buttervolume:latest myvolumeor:docker volume create --volume-driver=anybox/buttervolume:latestWhen creating a volume, you can choose to disable copy-on-write on a per-volume
basis. Just use the-oor–optoption as defined in theDocker documentationdocker volume create -d anybox/buttervolume -o copyonwrite=false myvolumeRunning the plugin locally or in legacy modeIf you installed it locally as a Python distribution, you can also
start it manually with:sudo buttervolume runIn this case it will create a unix socket in/run/docker/plugins/btrfs.sockfor use by Docker with the legacy plugin system. Then the name of the volume
driver is the name of the socket file:docker volume create -d btrfs myvolumeor:docker create --volume-driver=btrfsWhen started, the plugin will also start its own scheduler to run periodic jobs
(such as a snapshot, replication, purge or synchronization)Creating and deleting volumesOnce the plugin is running, whenever you create a container you can specify the
volume driver withdocker create--volume-driver=btrfs--name<name> <image>.  You can also manually create a BTRFS volume withdocker volume create-dbtrfs. It also works with docker-compose, by specifying thebtrfsdriver in thevolumessection of the compose file.When you delete the volume withdocker rm-v<container>ordocker volume rm <volume>, the BTRFS subvolume is deleted. If you snapshotted the volume
elsewhere in the meantime, the snapshots won’t be deleted.Managing volumes and snapshotsWhen buttervolume is installed, it provides a command line toolbuttervolume, with the following subcommands:run                 Run the plugin in foreground
snapshot            Snapshot a volume
snapshots           List snapshots
schedule            (un)Schedule a snapshot, replication or purge
scheduled           List scheduled actions
restore             Restore a snapshot (optionally to a different volume)
clone               Clone a volume as new volume
send                Send a snapshot to another host
sync                Synchronise a volume from a remote host volume
rm                  Delete a snapshot
purge               Purge old snapshot using a purge patternCreate a snapshotYou can create a readonly snapshot of the volume with:buttervolume snapshot <volume>The volumes are currently expected to live in/var/lib/buttervolume/volumesand
the snapshot will be created in/var/lib/buttervolume/snapshots, by appending the
datetime to the name of the volume, separated with@.List the snapshotsYou can list all the snapshots:buttervolume snapshotsor just the snapshots corresponding to a volume with:buttervolume snapshots <volume><volume>is the name of the volume, not the full path. It is expected
to live in/var/lib/buttervolume/volumes.Restore a snapshotYou can restore a snapshot as a volume. The current volume will first
be snapshotted, deleted, then replaced with the snapshot.  If you provide a
volume name instead of a snapshot, thelatest snapshotis restored. So no
data is lost if you do something wrong. Please take care of stopping the
container before restoring a snapshot:buttervolume restore <snapshot><snapshot>is the name of the snapshot, not the full path. It is expected
to live in/var/lib/buttervolume/snapshots.By default, the volume name corresponds to the volume the snapshot was created
from. But you can optionally restore the snapshot to a different volume name by
adding the target as the second argument:buttervolume restore <snapshot> <volume>Clone a volumeYou can clone a volume as a new volume. The current volume will be cloned
as a new volume name given as parameter. Please take care of stopping the
container before cloning a volume:buttervolume clone <volume> <new_volume><volume>is the name of the volume to be cloned, not the full path. It is expected
to live in/var/lib/buttervolume/volumes.<new_volume>is the name of the new volume to be created as clone of previous one,
not the full path. It is expected to be created in/var/lib/buttervolume/volumes.Delete a snapshotYou can delete a snapshot with:buttervolume rm <snapshot><snapshot>is the name of the snapshot, not the full path. It is expected
to live in/var/lib/buttervolume/snapshots.Replicate a snapshot to another hostYou can incrementally send snapshots to another host, so that data is
replicated to several machines, allowing to quickly move a stateful docker
container to another host. The first snapshot is first sent as a whole, then
the next snapshots are used to only send the difference between the current one
and the previous one. This allows to replicate snapshots very often without
consuming a lot of bandwith or disk space:buttervolume send <host> <snapshot><snapshot>is the name of the snapshot, not the full path. It is expected
to live in/var/lib/buttervolume/snapshotsand is replicated to the same path on
the remote host.<host>is the hostname or IP address of the remote host. The snapshot is
currently sent using BTRFS send/receive through ssh, with an ssh server direcly
included in the plugin. This requires that ssh keys be present and already
authorized on the target host (under/var/lib/buttervolume/ssh), and that
theStrictHostKeyChecking nooption be enabled in/var/lib/buttervolume/ssh/configon local host.Please note you have to restart you docker daemons each time you change ssh configuration.The default SSH_PORT of the ssh server included in the plugin is1122. You can
change it withdocker plugin set anybox/buttervolume SSH_PORT=<PORT>before
enabling the plugin.Synchronize a volume from another host volumeYou can receive data from a remote volume, so in case there is a volume on
the remote host with thesame name, it will get new and most recent data
from the distant volume and replace in the local volume. Before running thersynccommand a snapshot is made on the local machine to manage recovery:buttervolume sync <volume> <host1> [<host2>][...]The intent is to synchronize a volume between multi hosts on running
containers, so you should schedule that action on each nodes from all remote
hosts.NoteAs we are pulling data from multiple hosts we never remove data, consider
removing scheduled actions before removing data on each hosts.WarningMake sure your application is able to handle such synchronisationPurge old snapshotsYou can purge old snapshot corresponding to the specified volume, using a retention pattern:buttervolume purge <pattern> <volume>If you’re unsure whether you retention pattern is correct, you can run the
purge with the--dryrunoption, to inspect what snapshots would be deleted,
without deleting them:buttervolume purge --dryrun <pattern> <volume><volume>is the name of the volume, not the full path. It is expected
to live in/var/lib/buttervolume/volumes.<pattern>is the snapshot retention pattern. It is a semicolon-separated
list of time length specifiers with a unit. Units can bemfor minutes,hfor hours,dfor days,wfor weeks,yfor years. The pattern
should have at least 2 items.Here are a few examples of retention patterns:4h:1d:2w:2yKeep all snapshots in the last four hours, then keep only one snapshot
every four hours during the first day, then one snapshot per day during
the first two weeks, then one snapshot every two weeks during the first
two years, then delete everything after two years.4h:1wkeep all snapshots during the last four hours, then one snapshot every
four hours during the first week, then delete older snapshots.2h:2hkeep all snapshots during the last two hours, then delete older snapshots.Schedule a jobYou can schedule a periodic job, such as a snapshot, a replication, a
synchronization or a purge. The schedule it self is stored in/etc/buttervolume/schedule.csv.Schedule a snapshotof a volume every 60 minutes:buttervolume schedule snapshot 60 <volume>Remove the same schedule by specifying a timer of 0 min:buttervolume schedule snapshot 0 <volume>Schedule a replicationof volumefoovolumetoremote_host:buttervolume schedule replicate:remote_host 3600 foovolumeRemove the same schedule:buttervolume schedule replicate:remote_host 0 foovolumeSchedule a purgeevery hour of the snapshots of volumefoovolume, but
keep all the snapshots in the last 4 hours, then only one snapshot every 4
hours during the first week, then one snapshot every week during one year, then
delete all snapshots after one year:buttervolume schedule purge:4h:1w:1y 60 foovolumeRemove the same schedule:buttervolume schedule purge:4h:1w:1y 0 foovolumeUsing the right combination of snapshot schedule timer, purge schedule timer
and purge retention pattern, you can create you own backup strategy, from the
simplest ones to more elaborate ones. A common one is the following:buttervolume schedule snapshot 1440 <volume>
buttervolume schedule purge:1d:4w:1y 1440 <volume>It should create a snapshot every day, then purge snapshots everydays while
keeping all snapshots in the last 24h, then one snapshot per day during one
month, then one snapshot per month during only one year.Schedule a syncrhonizationof volumefoovolumefromremote_host1abdremote_host2:buttervolume schedule synchronize:remote_host1,remote_host2 60 foovolumeRemove the same schedule:buttervolume schedule synchronize:remote_host1,remote_host2 0 foovolumeList scheduled jobsYou can list all the scheduled job with:buttervolume scheduledIt will display the schedule in the same format used for adding the schedule,
which is convenient to remove an existing schedule or add a similar one.Copy-on-writeCopy-On-Write is enabled by default. You can disable it if you really want.Why disabling copy-on-write? If your docker volume stores databases such as
PostgreSQL or MariaDB, the copy-on-write feature may hurt performance, though
the latest kernels have improved a lot. The good news is that disabling
copy-on-write does not prevent from doing snaphots.TestingIf your volumes directory is a BTRFS partition or volume, tests can be run
with:./test.shWorking without a BTRFS partitionIf you have no BTRFS partitions or volumes you can setup a virtual partition
in a file as follows (tested on Debian 8):Setup BTRFS virtual partition:sudo qemu-img create /var/lib/docker/btrfs.img 10G
sudo mkfs.btrfs /var/lib/docker/btrfs.imgNoteyou can ignore the error, in fact the new FS is formattedMount the partition somewhere temporarily to create 3 new BTRFS subvolumes:sudo -s
mkdir /tmp/btrfs_mount_point
mount -o loop /var/lib/docker/btrfs.img /tmp/btrfs_mount_point/
btrfs subvolume create /tmp/btrfs_mount_point/snapshots
btrfs subvolume create /tmp/btrfs_mount_point/volumes
btrfs subvolume create /tmp/btrfs_mount_point/received
umount /tmp/btrfs_mount_point/
rm -r /tmp/btrfs_mount_point/Stop docker, create required mount point and restart docker:systemctl stop docker
mkdir -p /var/lib/buttervolume/volumes
mkdir -p /var/lib/buttervolume/snapshots
mkdir -p /var/lib/buttervolume/received
mount -o loop,subvol=volumes /var/lib/docker/btrfs.img /var/lib/buttervolume/volumes
mount -o loop,subvol=snapshots /var/lib/docker/btrfs.img /var/lib/buttervolume/snapshots
mount -o loop,subvol=received /var/lib/docker/btrfs.img /var/lib/buttervolume/received
systemctl start dockerOnce you are done with your test, you can unmount those volumes and you will
find back your previous docker volumes:systemctl stop docker
umount /var/lib/buttervolume/volumes
umount /var/lib/buttervolume/snapshots
umount /var/lib/buttervolume/received
systemctl start docker
rm /var/lib/docker/btrfs.imgMigrate to version 3If you’re currently using Buttervolume 1.x or 2.0 in production, you must
carefully follow the guidelines below to migrate to version 3.First copy the ssh and config files and disable the scheduler:sudo -s
docker cp buttervolume_plugin_1:/etc/buttervolume /var/lib/buttervolume/config
docker cp buttervolume_plugin_1:/root/.ssh /var/lib/buttervolume/ssh
mv /var/lib/buttervolume/config/schedule.csv /var/lib/buttervolume/config/schedule.csv.disabledThen stop all your containers, excepted buttervolumeNow snapshot and delete all your volumes:volumes=$(docker volume ls -f driver=btrfs --format "{{.Name}}")
# or: # volumes=$(docker volume ls -f driver=btrfs|tail -n+2|awk '{print $2}')
echo $volumes
for v in $volumes; do docker exec buttervolume_plugin_1 buttervolume snapshot $v; done
for v in $volumes; do docker volume rm $v; doneThen stop the buttervolume container,remove the old btrfs.sock file, and
restart docker:docker stop buttervolume_plugin_1
docker rm -v buttervolume_plugin_1
rm /run/docker/plugins/btrfs.sock
systemctl stop dockerIf you were using Buttervolume 1.x, you must move your snapshots to the new location:mkdir /var/lib/buttervolume/snapshots
cd /var/lib/docker/snapshots
for i in *; do btrfs subvolume snapshot -r $i /var/lib/buttervolume/snapshots/$i; doneRestore /var/lib/docker/volumes as the original folder:cd /var/lib/docker
mkdir volumes.new
mv volumes/* volumes.new/
umount volumes  # if this was a mounted btrfs subvolume
mv volumes.new/* volumes/
rmdir volumes.new
systemctl start dockerChange your volume configurations (in your compose files) to use the newanybox/buttervolume:latestdriver name instead ofbtrfsThen start the new buttervolume 3.x as a managed plugin and check it is started:docker plugin install anybox/buttervolume:latest
docker plugin lsThen recreate all your volumes with the new driver and restore them from the snapshots:for v in $volumes; do docker volume create -d anybox/buttervolume:latest $v; done
export RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
alias drunc="sudo runc --root $RUNCROOT"
alias buttervolume="drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume"
# WARNING : check the the volume you will restore are the correct ones
for v in $volumes; do buttervolume restore $v; doneThen restart your containers, check they are ok with the correct data.Reenable the schedule:mv /var/lib/buttervolume/config/schedule.csv.disabled /var/lib/buttervolume/config/schedule.csvCreditsThanks to:Christophe CombellesPierre VerkestMarcelo OchoaChristoph RistPhilip Nagler-FrankYoann MOUGNIBASCHANGELOG3.10 (2023-04-09)updated dependenciesfixed bug #453.9 (2022-10-11)Fixed a crash when option_copyonwrite is not setRestored the test suiteImproved and simplified the build script and test script3.8 (2022-10-07)Switched to copy-on-write by defaultAllow to choose to enable/disable copy-on-write for each volumeAllow to change the default SSH_PORT in the plugin configUpdated the base docker image and dependenciesAdded an option to show the version numberImproved documentation3.7 (2018-12-13)Unpinned urllib33.6 (2018-12-11)Fixed zombie sshd processes inside the pluginMinor documentation change3.5 (2018-06-07)Improved documentation3.4 (2018-04-27)Fix rights at startup so that ssh works3.3 (2018-04-27)Fixed a bug preventing a start in certain conditions3.2 (2018-04-27)Fixed the socket path for startup3.1 (2018-04-27)Fixed a declaration in Python 3.6Automatically detects the btrfs.sock pathMade the runpath and drivername configurable3.0 (2018-04-24)Now use the dockermanaged pluginsystemStop the scheduler before shutdown to avoid a 5s timeoutImproved loggingImproved the migration doc from version 1 or 22.0 (2018-03-24)BREAKING CHANGE: Please read the migration path from version 1 to version 2:BTRFS volumes and snapshots are now stored by default in different directories under/var/lib/buttervolumeConfiguration possible through environment variables or aconfig.inifileimplementedVolumeDriver.Capabilitiesand just return'local'other minor fixes and improvements1.4 (2018-02-01)Add clone commandreplace sync bybtrfs filesystem sync1.3.1 (2017-10-22)fixed packaging (missing README)1.3 (2017-07-30)fixed the cli for the restore command1.2 (2017-07-16)fixed the purge algorithm1.1 (2017-07-13)allow to restore a snapshot to a different volume name1.0 (2017-05-24)initial release, used in production