Package: planb
PlanB backs up your remote files to a local ZFS storage. Manage many
hosts and host groups. Automate hourly, daily, weekly, monthly and
yearly backups with snapshots.The following data transfer methods are supported:ssh+rsync (built-in);ssh+rsync of Kubernetes volume mounts (throughkubersync), like Rook managed Ceph;snapshots of ZFS (encrypted) datasets (throughplanb-zfssync);snapshots of ZFS volumes (throughplanb-zfssync);copies of (large)OpenStack Swiftcontainers (throughplanb-swiftsync);custom transfer (through your own customtransfer_execscript).What it looks likeAt the moment, the interface is just aDjangoadmin interface:The files are stored onZFSstorage. It usesZFSsnapshots to keep earlier
versions of files. See this example shell transscript:# zfs list | grep mongo2
tank/BACKUP/experience-mongo2   9,34G  1,60T   855M  /srv/backups/experience-mongo2

# ls -l /srv/backups/experience-mongo2/data/srv/mongodb
total 646610
-rw------- 1 planb nogroup   67108864 jun 17 17:03 experience.0
-rw------- 1 planb nogroup  134217728 jun  9 16:01 experience.1
...Those are the “current” files in the workspace. But you can go back in time:# zfs list -r -t all tank/BACKUP/experience-mongo2 | head -n4
NAME                                                 USED  AVAIL  REFER  MOUNTPOINT
tank/BACKUP/experience-mongo2                       9,34G  1,60T   855M  /srv/backups/experience-mongo2
tank/BACKUP/experience-mongo2@planb-20170603T1147Z      0      -   809M  -
tank/BACKUP/experience-mongo2@planb-20170603T1211Z      0      -   809M  -

# cd /srv/backups/experience-mongo2/.zfs/
# ls -1
planb-20170603T1147Z
planb-20170603T1211Z
planb-20170604T0001Z
planb-20170605T0002Z
...

# ls planb-20170603T1147Z/data/srv/mongodb -l
total 581434
-rw------- 1 planb nogroup   67108864 jun  2 18:21 experience.0
-rw------- 1 planb nogroup  134217728 mei 29 14:38 experience.1
...Requirements / setupPlanB can be installed as a standaloneDjangoapplication, or it can be
integrated in another Django project.Seerequirements.txtorsetup.pyfor up-to-date dependencies/requirements.Basically, you’ll need:ZFSstorage, ssh and rsync, a webserver
(nginx), python hosting (uwsgi), a database (mysql), a
communication/cache bus (redis) and a few python packages.For more detailed steps, seeSetting it all upbelow.TODOEncryption: right now, encryption keys are still a bit of a mess:stuff is stored in tank/_local; should use some kind of vault;when removing/renaming, those keys are not updated alongside;planb-zfssync.sh does not clean up snapshots created before
send/recv failure (e.g. because remote did not support –raw)add key rotation example scripts?Docs: add documentation for sync from previous unencrypted filesets?Docs: add a bit of documentation on how to work with encrypted filesetsConsider: move the hostgroup contents to separate filesets, so as to
create a more readable fileset listing. tank/HOSTGROUP/FILESET instead
of tank/HOSTGROUP-FILESET.RFE: Add post-backup.d directory somewhere where we can place
post-backup-done scripts to manually do X or Y.RFE: Add planb group for better permission management.RFE: Also store user/group permissions on/after rsync (using xattr
extended attributes?).BUG: Items added to /exclude list are not deleted from destination if
they have already been backed up once. The rsync job would need some
way to keep track of changes in include/exclude settings, and run a
cleanup in case they are changed. (See metadata storage like
planb-swiftsync.* files.)RFE: Standardize stdout/stderr output from Rsync/Exec success (and
prepend “> “ to output) to be more in line with failure.RFE: Add possibility to feed back snapshot size from the individual
Transport instead of using dutree. Parsing the swiftsync listings is
fast after all.FIX: Add uwsgi-uid==djangoq-uid check?Replace the exception mails for common errors (like failing rsync) to
use mail_admins style mail.After using mail_admins style mail, we can start introducing mail digests
instead: daily summary of backup successes and failures.Replace the “daily report” hack with a signal-receiver.Clarify why there’s a /contrib/ and a /planb/contrib/ directory.WARNINGThe Django-Q task scheduler is highly configurable from the/admin/-view. With a little effort it will run user-supplied python
code directly. Any user with access to the schedulers will have
tremendous powersRecommendation: don’t give your users powers to edit the schedulers.
Use the fine-grained permissions of the Django-admin systems to limit
them to Hosts and HostGroups only.Perhaps we should disable web-access to it altogether.Setting it all upIf you follow the HOWTO below, you’ll set up PlanB as a standalone
project. Those familiar withDjangowill know how to integrate it into
their own project.The setup below assumes you’ll be using theplanbuser. You’re free
to change that consistently of course.Setting up a ZFS poolYou should really do your own research on this. If you’re lucky, your
operating system has native support forZFS, and then this is
relatively easy.Please readREADME-zpool.rstfor a quick
introduction. When you’re done, things should look somewhat like this:#zpoolstatuspool: tank
 state: ONLINE
  scan: none requested
config:

  NAME                                  STATE
  tank                                  ONLINE
    raidz2-0                            ONLINE
      scsi-SSEAGATE_ST10000NM0226_6351  ONLINE
      scsi-SSEAGATE_ST10000NM0226_0226  ONLINE
      scsi-SSEAGATE_ST10000NM0226_8412  ONLINE
      scsi-SSEAGATE_ST10000NM0226_...   ONLINE
      ...
    raidz2-1                            ONLINE
      scsi-SSEAGATE_ST10000NM0226_0123  ONLINE
      scsi-SSEAGATE_ST10000NM0226_...   ONLINE
      scsi-SSEAGATE_ST10000NM0226_...   ONLINE
      scsi-SSEAGATE_ST10000NM0226_...   ONLINE
      ...
  spares
    scsi-SSEAGATE_ST10000NM0226_9866    AVAIL
    scsi-SSEAGATE_ST10000NM0226_5992    AVAILSetting up the projectThis section assumes you know a little about Python, pip and virtual
envs. Details may vary a slight bit across distro versions.Set up a virtualenv (optional):mkdir -p /srv/virtualenvs
echo 'WORKON_HOME=/srv/virtualenvs' >>~/.bashrc
apt-get install python3-virtualenv python3-pip virtualenvwrapper
# you may need to log in/out once after this

# you may need /usr/share/bash-completion/completions/virtualenvwrapper
# sources in your bashrc
mkvirtualenv planb --python=$(which python3) --system-site-packages
workon planb

mkdir /etc/planb
cd /etc/planb
pwd >$VIRTUAL_ENV/.project  # or the src dir, if you're going to edit a lotInstall PlanB prerequisites:apt-get install redis-server  # and: mysql-server or postgresqlInstall PlanB dependencies through apt (optional):apt-get install python3-redis python3-setproctitle
# .. and: python3-mysqldb or python3-psycopg2Install PlanB (including depedencies) from PyPI:pip3 install planbInstall PlanB (including dependencies) from git:pip3 install git+https://github.com/ossobv/planb.git@master#egg=planbSet up a localplanbuser:adduser planb --disabled-password --home=/var/spool/planb \
  --shell=/bin/bash --system

sudo -H -u planb ssh-keygen -t ed25519      # use elliptic curve
sudo -H -u planb ssh-keygen -t rsa -b 8192  # or use RSA if you're oldNoteYou may want to back that ssh key up somewhere.Set up the local environment:cat >/etc/planb/envvars <<EOF
USER=planb
PYTHONPATH=/etc/planb
DJANGO_SETTINGS_MODULE=settings
EOFNotePlanB looks for an environment file in the locations:- env PLANB_ENVFILE
- /etc/planb/envvars
- ./envvarsThe first file that can be loaded will be used.Set up the local configuration:cp ${VIRTUAL_ENV:-/usr/local}/share/planb/example_settings.py \
  /etc/planb/settings.py
${EDITOR:-vi} /etc/planb/settings.pyReplace all *FIXME* entries in the ``settings.py``NoteFor development you only need the settings module which can
be placed in the project root.cp-nexample_settings.py settings.pyYou can usepython setup.py developto install planb
in develop mode. This links the source directory to python
site-packages and is especially useful for production hacking.Make sure the SQL database exists. How to do that is beyond the scope of
this readme.At this point, you should be able to run theplanbscript.Set up the database and a web-user:planb migrate
planb createsuperuserSet up uwsgiplanb.ini:[uwsgi]
plugin = python3
workers = 4

chdir = /
virtualenv = /srv/virtualenvs/planb
wsgi-file = /srv/virtualenvs/planb/share/planb/wsgi.py

uid = planb
gid = www-data
chmod-socket = 660

for-readline = /etc/planb/envvars
   env = %(_)
endfor =Set up static path, static files and log path:# see the STATIC_ROOT entry in your settings.py
install -o planb -d /srv/http/YOURHOSTNAME/static

planb collectstatic

install -o planb -d /var/log/planbSet up nginx config:server {
    listen 80;
    server_name YOURHOSTNAME;

    root /srv/http/YOURHOSTNAME;

    location / {
        uwsgi_pass unix:/run/uwsgi/app/planb/socket;
        include uwsgi_params;
    }
    location = /favicon.ico {
        return 404;
    }
    location /static/ {
    }
}GivePlanBsudoaccess toZFStools and fix paths:cat >/etc/sudoers.d/planb <<EOF
planb ALL=NOPASSWD: /sbin/zfs, /bin/chown
EOF

zfs create tank/BACKUP -o mountpoint=/srv/backups
chown planb /srv/backups
chmod 700 /srv/backups(Note that setting up a different mount point is optional. See alsoREADME-zpool.rstfor additional tips.Set upqclusterfor scheduled tasks:# (in the source, this file is in rc.d)
cp ${VIRTUAL_ENV:-/usr/local}/share/planb/planb-queue.service \
  /etc/systemd/system/

${EDITOR:-vi} /etc/systemd/system/planb-queue.service

systemctl daemon-reload &&
  systemctl enable planb-queue &&
  systemctl start planb-queue &&
  systemctl status planb-queueSet up theqclusterfor dutree tasks. If you do not use dutree
or if you want to run dutree on the default qcluster you can setQ_DUTREE_QUEUE='PlanB'in/etc/planb/settings.py.:cp ${VIRTUAL_ENV:-/usr/local}/share/planb/planb-queue-dutree.service \
  /etc/systemd/system/

${EDITOR:-vi} /etc/systemd/system/planb-queue-dutree.service

systemctl daemon-reload &&
  systemctl enable planb-queue-dutree &&
  systemctl start planb-queue-dutree &&
  systemctl status planb-queue-dutreeInstall automatic jobs:planb loaddata planb_jobsDon’t forget a logrotate config:cat >/etc/logrotate.d/planb <<EOF
/var/log/planb/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 0644 planb www-data
        sharedscripts
}
EOFCreate aliases to quickly mount/unmount the current working directory
in your~/.bashrc:alias zfs-quick-mount="zfs load-key -L \
    "'"file:///tank/_local/zfskeys/${PWD#/}/_key.bin" "${PWD#/}" &&
    zfs mount "${PWD#/}" && cd .'
alias zfs-quick-umount='cd / && if zfs umount "${OLDPWD#/}"
    then zfs unload-key "${OLDPWD#/}"; cd "${OLDPWD}"
    else cd "${OLDPWD}"; false; fi'WarningWARNING: The example above uses local key files! This will be
fixed/replaced in upcoming commits.Configuring a remote hostCreate aremotebackupuser on the remote host (orencbackupfor
backups encrypted at the source[1][2], which is beyond the scope of
this document):useradd -m remotebackupConfiguresudoaccess usingvisudo-f/etc/sudoers.d/remotebackup:# Backup user needs to be able to get the files
remotebackup ALL=NOPASSWD: /usr/bin/rsync --server --sender *
remotebackup ALL=NOPASSWD: /usr/bin/ionice -c2 -n7 /usr/bin/rsync --server --sender *
remotebackup ALL=NOPASSWD: /usr/bin/ionice -c3 /usr/bin/rsync --server --sender *

# Optional, for planb-zfsync.sh (only destroy snapshots with @ in the name)
remotebackup ALL=NOPASSWD: /sbin/zfs destroy *@*
remotebackup ALL=NOPASSWD: /sbin/zfs list *
remotebackup ALL=NOPASSWD: /sbin/zfs send *
remotebackup ALL=NOPASSWD: /sbin/zfs set *
remotebackup ALL=NOPASSWD: /sbin/zfs snapshot *Observe how the--server--sendermakes the rsync read-only.Set up the ssh key like you’d normally do:mkdir -p ~remotebackup/.ssh
cat >>~remotebackup/.ssh/authorized_keys <<EOF
... ssh public key from /var/spool/planb/.ssh/id_rsa.pub goes here ...
EOF

chmod 640 ~remotebackup/.ssh/authorized_keys
chown remotebackup -R ~remotebackup/.sshWhen you use this pattern, you can tickuse_sudoand set the remote
user toremotebackup.Adding post-backup notificationDo you want a notification when a backup succeeds? Or when it fails?You can add something like this to your settings:from datetime import datetime
from subprocess import check_call
from django.dispatch import receiver
from planb.signals import backup_done

@receiver(backup_done)
def notify_zabbix(sender, fileset, success, **kwargs):
    if success:
        key = 'planb.get_latest[{}]'.format(fileset.unique_name)
        val = datetime.now().strftime('%s')
        cmd = (
            'zabbix_sender', '-c', '/etc/zabbix/zabbix_agentd.conf',
            '-k', key, '-o', val)
        check_call(cmd)That combines nicely with a backup host discovery rule usingblist:# Machine discovery (redirects stderr to mail).
UserParameter=planb.discovery, \
  ( planb blist --zabbix 3>&2 2>&1 1>&3 \
  | mail -E -s 'ERROR: planb.discovery (zabbix)' root ) 2>&1Doing daily jobsA quick hack to get daily reports up and running, is by placing something
like this in/etc/planb/planb_custom.py:from planb.contrib.billing import BossoBillingPoster, daily_hostgroup_report

def daily_billing_report():
    """
    This function is added into: Home >> Task Queue >> Scheduled task
    As: "Report to Billing" <planb_custom.daily_bosso_report>
    """
    daily_hostgroup_report(BossoBillingPoster('http://my.url.here/'))F.A.Q.Can I use the software and customize it to my own needs?It is licensed under the GNU GPL version 3.0 or higher. See the
LICENSE file for the full text. That means: probably yes, but you
may be required to share any changes you make. But you were going to
do that anyway, right?Mails for backup success are sent, but mails for failure are not.Check theDEBUGsetting. At the moment, error-mails are sent
through the logging subsystem and that is disabled when running in
debug-mode.Where are the ssh host fingerprints (known_hostsfiles) stored?They’re in~planb/.ssh/known_hosts.d/. If you want tosshmanually, you can add this to~planb/.profile:ssh() {
    for arg in "$@"; do
        case $arg in
        -*) ;;
        *) break ;;
        esac
    done
    if test -n "$arg"; then
        host=${arg##*@}
        echo "(adding: \
-o UserKnownHostsFile=$HOME/.ssh/known_hosts.d/$host)" >&2
        /usr/bin/ssh -o HashKnownHosts=no \
          -o UserKnownHostsFile=$HOME/.ssh/known_hosts.d/$host "$@"
    else
        /usr/bin/ssh "$@"
    fi
}Can I use ajump host?You can add-e'ssh-Jjumpuser@jumphost'to thersynctransport flags. Observe that the known hosts file oftargetwill
contain the fingerprint of thejump host.Are bandwidth limits in place?Yes, the default for thersynctransport is 10MB/s (megabyte). You
can lower or raise this by adding--bwlimit=10Mto the transport
flags.I’ve increased the bwlimit, but it’s still slow.If you notice that you’re limited by ssh encryption CPU speed, you
can consider setting the preferred ciphers in~planb/.ssh/config:Host *
    # The default is:
    #
    #   chacha20-poly1305@openssh.com,
    #   aes128-ctr,aes192-ctr,aes256-ctr,
    #   aes128-gcm@openssh.com,aes256-gcm@openssh.com
    #
    # The available ciphers may be obtained using "ssh -Q cipher".
    # (Adding a non-existent one will yield a "Bad SSH2 cipher spec".)
    #
    # The AES ciphers are commonly hardware/CPU accelerated.
    #
    Ciphers aes128-ctr,aes128-gcm@openssh.com,aes256-ctr,\
        aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,3des-cbcRemoving a fileset does not wipe the filesystem from disk, what should I do?This is done intentionally. You should periodically useplanb slist--staleto check forstalefilesystems.You can them remove them manually usingzfs destroy[-r]FILESYSTEM.Rsync complains aboutfailed to statormkdir failed.If rsync returns these messages:rsync: recv_generator: failed to stat "...": Permission denied (13)
rsync: recv_generator: mkdir "..." failed: Permission denied (13)Then you may be looking at parent directories with crooked
permissions, like 077. Fix the permissions on the remote end.However, many of these problems have likely been fixed by the
addition of the--chmod=Du+rwxrsync option.Rsync complains aboutInvalid or incomplete multibyte or wide character.If rsync returns with code 23 and says this:rsync: recv_generator: failed to stat "...\#351es-BCS 27-09-11.csv":
  Invalid or incomplete multibyte or wide character (84)Then you might be backing up old hosts with legacy Latin-1 encoding
on the filesystem. Adding--iconv=utf8,latin1to the rsync transport
flags should fix it.You may need rsync version 3 or higher for that.Right now we opt tonotimplement any of these workarounds:Patch rsync to cope withEILSEQ(84) “Illegal byte sequence”.Cope with error code 23 and pretend that everything went fine.Instead, you should install a recent rsync and/or fix the filenames
on your remote filesystem.Themkvirtualenvsaidlocale.Error: unsupported locale setting.You need to install the right locales untilperl-esetlocaleis
silent. How depends on your system and your config. Seelocaleand
e.g.locale-genen_US.UTF-8.Theuwsgilog complains about“No module named site”.If your uwsgi fails to start, and the log looks like this:Python version: 2.7.12 (default, Nov 19 2016, 06:48:10)
Set PythonHome to /srv/virtualenvs/planb
ImportError: No module named siteThen your uWSGI is missing the Python 3 module. Go installuwsgi-plugin-python3.AuthorsPlanB was started in 2013 as “OSSO backup” by Alex Boonstra at OSSO B.V. Since
then, it has been evolved intoPlanB. When it was Open Sourced by Walter
Doekes in 2017, the old commits were dropped to ensure that any private company
information was not disclosed. Since then, Harm Geerts has also been
busy on the project.Footnotes[1]If you want your data encrypted before it gets sent to the PlanB server,
check out the OSSO blog:on the fly encrypted backups using gocryptfs (2020)[2]An older OSSO blog about on the fly encryption at the source:on the fly encrypted backups using encfs (2015)Changesv1.7 -2022-09-20CoreRefactorStorageandDatasetclasses.Consolidate storage location inFilesetfielddataset_name.RefactorStorageconfiguration.Refactor backup retention.Add support for hourly backups.Add configurable blacklist hours.Refactor snapshots, drop retention prefixes.A backup now creates one snapshot.Improve snapshot interval stability.Replace custom planb script with normal python entry point.TasksImplement task to allow renaming of filesets in the Storage engine.Enforce globalFilesetlocks to prevent race conditions.Web interfaceShow a message when a rename task has spawned from a change.Don’t show manually queued Filesets in the backup failure warning.OtherFixblistto show any transport type.Fixbcloneto also clone transport.Fixbqueueflushto default to the main queue.Cleanup ZFS workdir switching.Fix many bugs.Add many tests.Add tox and travis-ci configuration.Drop python 3.5 support.Fix compatibility with latest django-q.Fix model validation with choices in django>=2.1.Exclude manual backups from triggering failure.Improvements and bug fixes to swiftsyncImprovements and bug fixes to zfssyncv1.6.post1 -2019-03-20Web interfaceShow last snapshot size distribution, and first backup success date,
in hostconfig edit-view.Show real-used-size instead of apparent-used-size in snapshot data
distribution. If you have ZFS compression enabled, you’ll see a drop
in the snapshot size summaries (not in the total disk usage).Hide last-error message in hostconfig edit-view for hosts that are
disabled.OtherSingle Sign-On (Discourse style) can be enabled (using the optional
kleides-dssoclient dependency). See KLEIDES_DSSO_ENDPOINT option.Tweak permissions so you don’t need is_superuser powers anymore.Update BossoBillingPoster for posting backup data counts to (internal)
Bosso system.v1.5 -2018-06-13Web interfaceShow “time since last backup” in listing, instead of just OK/FAIL.TasksAdd locking to dutree scanner, so the filesystem isn’t raped. Only do
one dutree scan at a time.Change scheduler: jobs are now scheduled ahead of time by deducting
the expected duration.Fix rsync issue when remote directory permissions are wrong
(unreadable by user). In that case, the download (as root user) would
succeed, but later changes would fail locally (planb user).OtherChange schema, removing unnecessary weekly/monthly booleans.Fixtotal_size_mbin report, which was too large.Improvebreportcommand to output to stdout by default.v1.4 -2018-04-06Web interfaceShow job failures in hostconfig detail view.Reduce clutter in hostconfig list view, using smaller items and less
clutter.Show average run time, instead of last run time.OtherFix bug with sending of breport emails.Use git version for pip-install if available; add makefile for quick
commands.Update qcluster argv so it’s still considered busy while doing the
dutree scan.v1.3 -2018-03-19Web interfaceDisallow deletion of non-empty host groups.CLIAddbreportcommand to send out backup reports. See the template
in templates/planb/report_email_body.txt. Note that the report is
still in alpha stage. NOTE: To get e-mail reports as well, you need
to haverst2htmlinstalled.Add--with-disabledtoconfexportcommand to get complete
exports.Fix that planb runserver can be used for development (through
PYTHONPATH propagation).OtherDependency updates to Django 2.0+.Add backup history record keeping, for better logging and averages.v1.2 -2017-09-18Fix release, this time without pyc files and with wheel package.
Run this for upload: python setup.py sdist bdist_wheel uploadv1.1 -2017-09-18SettingsAddPLANB_DEFAULT_INCLUDES.RenameZFS_BIN,SUDO_BINandRSYNC_BINtoPLANB_<setting>.Fix allowing use of alternateDJANGO_SETTINGS_MODULE.Web interfaceAdd hosts to hostgroup listing.Allow ordering hosts by enabled/queued/running.CLIAdd “stale mounts” listing (planb slist).Create “hostconfig” export in YAML or JSON format (planb confexport).QueueFix so long running jobs don’t suffer from lost DB connections.OtherMisc refactoring/cleanup.v1.0 -2017-07-11Initial release.