Flatten SolidFire Backups on S3
Channel: scaleoutSean
Date: 23/08/2021

If we take multiple backups of the same volume to an S3 bucket, we'll notice the first backup has the longest list of objects S3. The second should have fewer objects, and its manifest file should have most of pointers pointed at the "baseline" backup subdirectory, and relatively few to the backup directory containing backup objects for the current backup. 
The third will also reference "baseline" backup directory.

This means we could in theory move changed blocks from the subdirectory with baseline backup to the directory of second backup (without overwriting any existing objects, as those contain changed data or "diff" between baseline backup and the second backup) and edit its manifest to only refer to its own subdirectory.

This isn't something officially supported or practical, but it does work as we can see in this video.

To show that it works I downloaded S3 objects from 2 backups and 2 manifest files to local disk, made necessary changes, and uploaded the second backup data and manifest file to S3. But this could be written as a script that automates these steps and doesn't download unnecessary data. I don't recommend it, though - it's not supported and it's error-prone. Borg, Restic and other utilities already do that for you - take a look at SolidBackup on Github.