Making backups using rsync - Page 1
Forum section: electronics / general

author: Karel
author_rank: Super Contributor
timestamp: February 27, 2020, 08:19:33 pm
content: For creating backups on external harddrives I use a shell script, something like:rsync -vrth --progress --delete /data/src_dir /var/run/media/harddrive/dest_dirThe only disadvantage is that, if I accidently erase a file and make a backup, the file is gone forever.Because of that I'm thinking about to make incremental backups using tar --create --listed-incrementaland put it together with rsync in a shell script.Any thoughts, ideas?

author: Twoflower
author_rank: Frequent Contributor
timestamp: February 27, 2020, 08:26:35 pm
content: If you want to keep the deleted files, the --backup and --backup-dir options are your friend.

author: mrflibble
author_rank: Super Contributor
timestamp: February 27, 2020, 08:38:42 pm
content: Use rsnapshot. That uses rsync + LVM snapshots for its backups, and yes, it is incremental. So when you remove file and then run rsnapshot once, the file is still avilable in an older part of incremental backup.

author: sleemanj
author_rank: Super Contributor
timestamp: February 27, 2020, 08:39:11 pm
content: Use borg backup, way nicer, incremental, de-duped, compressed and mountable backups with optional encryption. The executable is statically built stand-alone so you can just drop that on your backup device to have it available when the shit hits the fan -https://www.borgbackup.org/Alternative you can use rsnapshot, which uses rsync and creates archives based around hardlinks -https://rsnapshot.org/

author: ataradov
author_rank: Super Contributor
timestamp: February 27, 2020, 09:23:01 pm
content: +1 for borg. Works perfectly. And if you want off-site backups, rsync.net has special pricing for borg hosting accounts. It is very cheap.

author: RoGeorge
author_rank: Super Contributor
timestamp: February 27, 2020, 09:23:59 pm
content: Quote from: sleemanj on February 27, 2020, 08:39:11 pmThere is a dd example but It's not clear from the demo videos, can Borg backup an entire OS disk (Linux) while keeping the enumerated advantages?

author: ataradov
author_rank: Super Contributor
timestamp: February 27, 2020, 09:29:04 pm
content: No, I don't think Borg is for backing up the whole image. It will save files, of course, so to that extent it will work. But it will not give you an image that you can simply write into a partition and have a bootable computer.It is really more for user data backup. At least that's the way I understand it. May be there is some way to make a full OS backups. I have not investigated it much.

author: bd139
author_rank: Super Contributor
timestamp: February 27, 2020, 09:30:58 pm
content: Borg has some problems I found on large datasets. At around the 1TiB mark on lots of small files it shit a gigantic brick one day irrecoverably.I switched to rdiff-backup which is an extrapolation of rsync that does extremely fast incremental backups so you can get point in time recovery. Also the latest increment is always a copy of the source filesystem so you can restore with "cp" if the tools ever fall over worst case.https://github.com/rdiff-backup/rdiff-backupAlso don't back the entire machine up. Total waste of space and time. Only things that are valuable are in /etc /var/ /home in theory. I tend to just dump the machine package list out and /home and that's it. If it's a server box it is deployed from ansible therefore you can recreate the "desired state" with that

author: sleemanj
author_rank: Super Contributor
timestamp: February 27, 2020, 11:46:13 pm
content: Quote from: bd139 on February 27, 2020, 09:30:58 pmI used to use rdiff-backup, it's also good.For my (somewhat peculiar with lots of duplicate files on my system for reasons) usage pattern though the de-dup of borg is a god-send...Code:Original size Compressed size Deduplicated sizeThis archive: 428.52 GB 240.07 GB 483.39 MBAll archives: 5.73 TB 3.19 TB 201.89 GBYeah that's 5.73 TB of backups in 201 GB of space total, and the most recent compressed and de-duped to a 10x reduction.I use a separate borg repo for my vmware image/disks backups and it's similarly impressive on the de-dup...Code:Original size Compressed size Deduplicated sizeThis archive: 572.59 GB 281.86 GB 638.64 MBAll archives: 11.36 TB 5.59 TB 233.19 GB

author: ve7xen
author_rank: Super Contributor
timestamp: February 27, 2020, 11:55:26 pm
content: I used to use rdiff-backup.I have now fully switched to using a BTRFS filesystem and btrbk. Incremental, copy-on-write, any version immediately accessible without any 'recovery' process, transfers are extremely fast & small since there is no comparison process, the journal just needs to be run forward. It has both online and offline dedupe if you wanted it, but using COW reflinks or old school hard links probably makes more sense than block-level copies in the first place. Dedupe is more for VM storage, for example, where you can't alter the semantics but many machines will have identical blocks.

author: brucehoult
author_rank: Super Contributor
timestamp: February 28, 2020, 01:27:24 am
content: Quote from: ataradov on February 27, 2020, 09:23:01 pmOK, borg looks interesting.It seems remote backups are supported using:1) ssh to a remote system with borg installed. (fastest)2) sshfs or similar3) I guess anything else that looks locally mounted e.g. DropBox, Google Drive, iCloud Drive.Question: can you simply copy whole repos from one place to another? For example, make a repo locally, mass copy it to Google Drive, and then do incremental backups to the Google Drive. Is the speed reasonable doing so?

author: ataradov
author_rank: Super Contributor
timestamp: February 28, 2020, 02:06:06 am
content: I have not tried it, but it should be possible. It just creates a bunch of files and there is nothing that ties them to the target system.

author: nctnico
author_rank: Super Contributor
timestamp: February 29, 2020, 08:26:50 pm
content: Quote from: Karel on February 27, 2020, 08:19:33 pmRemove the --delete from the line above and files won't be deleted. The downside is that you have to prune the backup every once in a while.

author: Karel
author_rank: Super Contributor
timestamp: March 01, 2020, 02:59:57 pm
content: Quote from: nctnico on February 29, 2020, 08:26:50 pmAgreed but I will still loose data in case the content of a file has changed, no?

author: nctnico
author_rank: Super Contributor
timestamp: March 01, 2020, 05:32:34 pm
content: True.

author: kripton2035
author_rank: Super Contributor
timestamp: March 01, 2020, 07:30:06 pm
content: this script mimics the mac time machine, which does incremential backupshttps://github.com/samdoran/rsync-time-machine

author: brucehoult
author_rank: Super Contributor
timestamp: March 01, 2020, 11:12:10 pm
content: Quote from: kripton2035 on March 01, 2020, 07:30:06 pmTime Machine has a big advantage, because Apple modified their file system to support hard linking directories.With other file systems you either have to recreate the entire directory tree in each incremental backup and hard link only the unchanged files, or else symlink the unchanged directories, which complicates selective pruning of backups later.Or store the directory tree in some other way such as using a normal file to store each directory as git does. Or a database.

author: bitman
author_rank: Unknown
timestamp: March 01, 2020, 11:16:46 pm
content: Quote from: nctnico on February 29, 2020, 08:26:50 pmIf you want to restore and have the files you had when you backed up, --delete is absolutely required. A file in the wrong location can easily change behavior of software, so if the intention with the backup is to ever do a full restore, --delete is a must. There are other options if you want to put those deleted files in a safe place.

author: bd139
author_rank: Super Contributor
timestamp: March 01, 2020, 11:17:12 pm
content: Yes like rdiff-backup

author: bitman
author_rank: Unknown
timestamp: March 01, 2020, 11:21:26 pm
content: Quote from: Karel on February 27, 2020, 08:19:33 pmIf you want full backup and features, look at deja-dup - it too can use rsync but a lot of other protocols too. What's great is that it keeps indexes of files, over multiple backups and if you're using Nautilus in Gnome it's connected directly with the backup metadata allowing you to restore files/directories with a single click in the gui. A nice feature is that it will use cloud services as well as a local server like you do with scp/rsync.rsync is great - very flexible, but it's solely focused on the latest backup. So when you run it, you overwrite what the backup before did. That may be exactly what you want, but do realize that a crash during backup will make your destination invalid (old/new files in a big mess). Then again, when the backup fails typically you can just re-run it once you're resolved the issue that made it crash.Another note - for stuff I work on - writing code, documentation etc. I use 'git' as my backup. It's much better at keeping track of things and even allows easy sharing/contributions to happen. But I do have deja-dup running daily to deal with everything else.

author: Karel
author_rank: Super Contributor
timestamp: March 02, 2020, 07:34:00 am
content: What about making incremental backups using Tar? Anybody tried that?https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.htmlhttps://www.unixmen.com/performing-incremental-backups-using-tar/https://www.theurbanpenguin.com/incremental-backups-with-tar/

author: sleemanj
author_rank: Super Contributor
timestamp: March 02, 2020, 08:15:11 am
content: I incorporate an incremental tar backup function into the websites I develop (which may be deployed on places I don't control and may or may not have good server backups). Now, if anybody actually uses it...One thing that did surprise me when making it is... tar is not nearly as standardised or universal as you think. I found that the incremental archives especially, some tar implementations were ok, others not so much, some were ok with one format and not another or whatever.Rather than trying to find a good middle-ground I just specified in the backup restoration notes intended for future me or developers, "use gnu tar".For that reason I would use incremental tar only and exactly for this... to allow easy downloads of incremental archives to "admins", it's old and crusty for anything else.

author: bd139
author_rank: Super Contributor
timestamp: March 02, 2020, 09:21:54 am
content: tar is the reason so many alternatives exist. Literally the only stable functions are the POSIX defined ones and most of them are poorly implemented.

author: Karel
author_rank: Super Contributor
timestamp: March 02, 2020, 09:39:19 am
content: Considering that I'm using Linux & GNU tar, I could give it a try.I'm thinking to put it into a shell script together with rsync.

author: Zucca
author_rank: Unknown
timestamp: March 02, 2020, 09:39:50 am
content: I use 7z, it has tons of options (incremental or not) and it save space.At every workday end I fire a .bat which locks the pc, makes the 7z backup and then turn off the PC.QuoteIt's surely not the best solution for everybody, but the best for me.PS: My trusty 7z backup file saved my ass a couple of times already.PSS: Oh shit I have to sign up ASAP to some cloud backup service, probably going to blackblaze for future FreeNAS integration.

author: ve7xen
author_rank: Super Contributor
timestamp: March 03, 2020, 12:05:41 am
content: Quote from: brucehoult on March 01, 2020, 11:12:10 pmMost modern filesystems support mounted snapshots. Use btrfs or ZFS and you can accomplish basically identical behaviour. The openSUSE 'snapper' tool does a good job of automatically managing the snapshots. There are some flashy GUI tools too, but I don't find any actual value in those.

author: brucehoult
author_rank: Super Contributor
timestamp: March 03, 2020, 12:45:11 am
content: Quote from: ve7xen on March 03, 2020, 12:05:41 amFUSE lets you write a driver to make virtually anything look like a mounted drive. So, yes, you can make a snapshot stored with dedup in a database look like it's in a filesystem.That's not the same thing as being able to hard link directories in your normal filesystem.

author: ve7xen
author_rank: Super Contributor
timestamp: March 03, 2020, 08:32:25 pm
content: Quote from: brucehoult on March 03, 2020, 12:45:11 amI have no idea what you're on about. Nothing in this post is relevant. I didn't mention FUSE or databases, and snapshots have nothing to do with dedup.You're right that snapshots aren't the same thing as hard links, but for periodically snapshotting (heh) the system, they are better. The semantics with respect to what Time Machine does are the same, but the implementation depends on a filesystem primitive instead of a userland process creating and maintaining a bunch of hardlinks. All you need the script for is to schedule the snapshots and optionally mount them so you can have a /backups/2020-03-03T01:00:00 kind of hierarchy.