Package: contentmonster
ContentMonsterContentMonster is a Python package used to replicate the contents of directories on one server ("shore") to other servers ("vessels") using SFTP over unstable network connections. The files are split into smaller chunks which are transferred separately and reassembled on the server.It comes with a daemon application (worker.py) which monitors the configured local directories for changes and instantly pushes them to the vessels. Once a file has been replicated to all vessels, it is moved to a "processed" subdirectory of its source directory and removed from the queue.PrerequisitesContentMonster is written in Python3 and makes use of syntactical features introduced in Python 3.8. It depends on two packages installable by pip, paramiko (for SSH/SFTP connections) and watchdog (to monitor local directories for changes).It was tested on Ubuntu 21.04 and Debian 10, but I don't see a reason why it would not work on other Unixoids or even Windows (although it might need some changes to properly work on the latter) as all dependencies are platform-independent.Vessels (destination servers) need to have an SSH server with SFTP support. This has been tested with a default OpenSSH server as well as a Dropbear server with OpenSSH's sftp-server. They also have to provide thecatcommand which is used to reassemble the uploaded chunks.InstallationIt is recommended that you use a virtual environment in order to maintain a clean Python environment independent from system updates and other Python projects on the same host. Note that you may have to install thevenvpackage from your OS's package repositories first (on Debian-based distributions:apt install python3-venv).In a terminal, navigate to the ContentMonster directory, then (assuming you are running bash) execute the following commands:python3-mvenvvenv# Create a virtual environment in the "venv" subdirectory.venv/bin/activate# Activate the virtual environment (just in case)pipinstall-Urrequirements.txt# Install the package dependencies (paramiko/watchdog)ConfigurationThe application is configured using thesettings.inifile. Start off by copying the providedsettings.example.initosettings.iniand opening it in a text editor. Note that all keys and values are case-sensitive. Required keys are identified as such in the comments below, all other keys are optional. The file consists of (at least) three sections:MONSTERTheMONSTERsection contains a few global configuration options for the application:[MONSTER]ChunkSize=10485760# Size of individual chunks in bytes (default: 10 MiB)DirectoryYou can configure as many directories to be replicated as you want by adding multipleDirectorysections. The directories are replicated to the same location on the vessels that they are located at on the shore.[Directory sampledir]  # Each directory needs a unique name - here:"sampledir"Location=/home/user/replication# Required: File system location of the directoryNote: Currently, the same Location value is used on both the shore and the vessels, although this may be configurable in a future version. The directory has to be writable by the configured users on all of the configured vessels. In the above example, files are taken from /home/user/replication on the shore and put into /home/user/replication on each of the vessels.VesselYou can configure as many vessels to replicate your files to as you want by adding multipleVesselsections. All configured directories are replicated to all vessels by default, but you can use the IgnoreDirs directive to exclude a directory from a given vessel. If you want to use an SSH key to authenticate on the vessels, make sure that it is picked up by the local SSH agent (i.e. you can login using the key when connecting with thesshcommand).[Vessel samplevessel]  # Each vessel needs a unique name - here:"samplevessel"Address=example.com# Required: Hostname / IP address of the vesselTempDir=/tmp/.ContentMonster# Temporary directory for uploaded chunks (default: /tmp/.ContentMonster) - needs to be writableUsername=replication# Username to authenticate as on the vessel (default: same as user running ContentMonster)Password=verysecret# Password to use to authenticate on the vessel (default: none, use SSH key)Passphrase=moresecret# Passphrase of the SSH key you use to authenticate (default: none, key has no passphrase)Port=22# Port of the SSH server on the vessel (default: 22)IgnoreDirs=sampledir, anotherdir# Names of directories *not* to replicate to this vessel, separated by commasRunningTo run the application after creating thesettings.ini, navigate to ContentMonster's base directory in a terminal and make sure you are in the right virtual environment:.venv/bin/activateThen, you can run the worker like this:pythonworker.pyKeep an eye on the output for the first minute or so, to check for any issues during initialization.systemd ServiceYou may want to run ContentMonster as a systemd service to make sure it starts automatically after a system reboot. Assuming that it is installed into/opt/ContentMonster/following the instructions above and supposed to run as thereplicationuser, something like this should work:[Unit]Description=ContentMonsterAfter=syslog.target network.target[Service]Type=simpleUser=replicationWorkingDirectory=/opt/ContentMonster/ExecStart=/opt/ContentMonster/venv/bin/python -u /opt/ContentMonster/worker.pyRestart=on-abort[Install]WantedBy=multi-user.targetWrite this to/etc/systemd/system/contentmonster.service, then enable the service like this:systemctldaemon-reload
systemctlenable--nowcontentmonster
systemctlstatuscontentmonster# Check that the service started properlyThe service should now start automatically after every reboot. You can use commands likesystemctl status contentmonsterandjournalctl -xeu contentmonsterto keep an eye on the status of the service.