Best Overall Amazing Back Up Setup? 1K-4K Budget! Software + Hardware? Help! - Page 1
Forum section: electronics / general

author: NeedsPractice
author_rank: Contributor
timestamp: June 23, 2023, 11:24:47 pm
content: I remember a lot of people before when my hard drive failed told me that I could do a couple of amazing things with backing up data. What stuck out is two things...1. Software that will backup your data live via a cloud solution or over the internet without having a corporate cloud product.2. Get a storage tower with hard drives and load them up.What I mean is there software out there that you can install on your computer. You tell it which data you want backed up. Lets say your main PC with ( 2 ) hard drives and ( 8 ) hard drives connected to it via a tower.Then you have and ( 10 ) hard drives connected to the internet or your LAN to which this software will automatically copy all of your data 1 for 1 match, live, not compressed from your current PC and ( 8 ) hard drives to your ( 10 ) hard drives in another location.Live and in the current moment. So if you make a change or add a file, it automatically copies the differences.Does this make sense? If not, what is simply the best hard drives I could buy, the best hard drive storage or bay system to hold them, and what is the best software that I can do this with that is non compression and not cloud base that cost a monthly fee.Thanks!

author: nctnico
author_rank: Super Contributor
timestamp: June 23, 2023, 11:51:17 pm
content: rsync at set intervals. I run it over night. So during the day I can fetch a file I accidentally deleted. You'll only lose a day's work at most this way. There are also NAS systems that let you do incremental backups with history but I have never used those myself. Using realtime file synchronisation is iffy. I have used systems like Dropbox and Google drive as a local hard drive but the performance is quite poor and there are chances of file synchronisation problems. For example: having a git repository on a Google drive is not a good idea.

author: ataradov
author_rank: Super Contributor
timestamp: June 24, 2023, 01:02:15 am
content: I use rsync.net + Borg backup. While it is technically corporate cloud, it is pretty easy to setup and they provide dedicated plans for Borg. Everything is encrypted before leaving your PC and rsync.net just provides storage.For local backup I use NAS (4 bay Synology with 4x4TB drives, which are all replicated). 4 TB overall is plenty for my needs and I value redundancy. I also rsync the local system to NAS overnight. It is all automated and just works until you need a fetch a file and it is there.There is no point in having more drives if you are just going to replicate them all. What are the realistic chances that multiple fail at the same time before you have a chance to order a replacement?I tried many dedicated customer-oriented backup providers, but they all don't last. They either fold or jack up the price. Or don't support Linux.

author: SiliconWizard
author_rank: Super Contributor
timestamp: June 24, 2023, 01:49:16 am
content: Quote from: ataradov on June 24, 2023, 01:02:15 amAgreed.i'm also wondering these days if it's worth considering, on top of your other backup schemes, backing up what you consider critical data on more durable media, such as M-DISC (as I talked about a while ago).Of course since these are limited to 100GB, this wouldn't be for large backups, but only a fraction of maybe really important stuff.Does any of you do that?

author: ataradov
author_rank: Super Contributor
timestamp: June 24, 2023, 02:00:42 am
content: Quote from: SiliconWizard on June 24, 2023, 01:49:16 amWhy though? What is the point of it lasting past the point were you are able to replace the drives?And one more thing I do. I have another 4TB USB drive that is offline most of the time and once a week I plug it into Synology and it does the backup itself.This is inspired by all the ransomware. If anything really bad happens, I always have at most a week old backup.

author: nctnico
author_rank: Super Contributor
timestamp: June 24, 2023, 10:58:23 am
content: Quote from: ataradov on June 24, 2023, 02:00:42 amI don't think that is good enough. I don't have the answer to the question 'what is good enough'. My guess is that ransomware doesn't encrypt files within the timespan of a single day but takes several days up to weeks in order to maximise damage and make it impossible to use a backup that is a few days old. Also your computer would slow down noticably if the files are encrypted as fast as possible. Probably a backup system that keeps a history of the files is a better idea. That would be the only way to go back to the last good version of a file. But this is my gut feeling.

author: hans
author_rank: Super Contributor
timestamp: June 24, 2023, 11:26:16 am
content: The question is to what faults you want to protect yourself from:The "oopsie" I shouldn't have deleted that fileThe "shit" I got ransomware on my PC.. (similar failure to HDD dying)And the "damn" my house is on fire! (all your devices lost)I personally use a piece of software called restic. It requires some command line terminal to set up, but in my opinion it's far more flexible than a rsync script. It uses rclone to talk to various file backends. That can be Amazon, Backblaze B2, FTP, or any provider that offers WebDAV.Restic creates a repository on your provider and protects all your data using a local password. Data can then only be unlocked with your own password. It then creates incremental snapshots of your backup folders you point to. Those snapshots capture the exact state on disk at that point in time. In the background it will manage data and index blocks seperately, so only incremental data changes are written. You can have it prune the repository to keep the latest x days, x months, x years of snapshots to prevent accumulation of old files.Personally for me 1 key requirement is to have my backups off-site. If my house burns down I still have all my data. I also test my backups regurarily to check if they are in tact. It could be as simple as setting up a new machine without using anything from your original infrastructure (like copying your keychain from another machine, or mounting your NAS straight away), and seeing if that works out. The last time I did, I discovered some circular dependencies on unlocking all my accounts and repositories to recover data.But after that I was able to mount my repository and browse all my files.A 2nd requirement for me is to have backups automated. I can't imagine the hassle people occupy themselves with by copying data to a cold drive every other day. At one point you'll forget, and you potentially lose weeks of data.I also wouldn't count on you being able to catch ransomware. It depends how clever the virus is but if they wait for computer inactivity before they start encryption then you wouldn't notice the slowdown. If you keep your PC on during lunch (say 1 hour) it could have encrypted your whole disk by then. Modern SSDs can sustain continuous read/writes of easily 100+MB/s, so after 1h they could encrypt 360GB of documents, and potentially more if you have a fast M.2 drive that can (initially) sustain several GB/s.This is why I like restic because it talks via rclone. I don't think many ransomwares would target such a backend, although I suppose it is possible (if they can find your config file). But I run rclone on my NAS, so then they cannot find the repository details unless my NAS gets infected, or the ransomware can also hack into my NAS at the same time..I also have local file sync using syncthing, and so at best those files would be corrupt, but I could restore those from another machine that I have kept offline, or by pulling the files from restic repository.

author: mendip_discovery
author_rank: Super Contributor
timestamp: June 24, 2023, 02:10:51 pm
content: I have used Backblaze for a few years and annoyingly I have lost data partly down to the way the software works. Thankfully it was only my stash of vehicle manuals and TV shows that I lost.I have Backblaze running, with a PC with multiple drives and rsync batch files that I run regularly. Then at my sisters' house, I have the ability to do a daily and weekly backup to a NAS box at hers, she does the same the other way. I use both internal and external drives and have the PC on a UPS to prevent the PC from shutting down during a power outage, spike or brown out.I have some DVDs of backups here and I dread to think how many of them still work. I remember the days spent trying to get the data to save and what best way to avoid part folders being on the disks.One drive here has 5613.4 days on it.

author: Marco
author_rank: Super Contributor
timestamp: June 24, 2023, 02:27:13 pm
content: I'm just cobbling together stuff, but if I wanted to spend money I'd look into Veeam. Technologically it looks most advanced, it supports CoW snapshot based backups (they call it fast clone) with XFS for Linux and ReFS for Windows. Acronis, Duplicacy, etc are all file based AFAICS.Of course technological wizardry doesn't mean it's reliable, caveat emptor.PS. that's just the software, also need a NAS for local backup and then some remote (cloud or hosting). But best to start with the software and then see what they recommend for the rest.

author: Marco
author_rank: Super Contributor
timestamp: June 24, 2023, 03:15:54 pm
content: Quote from: NeedsPractice on June 23, 2023, 11:24:47 pmLooking around Veeam with ReFS isn't nearly that fast.Synology can do 5 minute snapshots though, so if this is for a desktop you could use a Synology NAS with iSCSI and let it handle the snapshotting. The snapshots are your local backup at 5 minute intervals and it can replicate them to a cloud backup at a larger interval.

author: PlainName
author_rank: Super Contributor
timestamp: June 25, 2023, 08:23:08 pm
content: Quote from: NeedsPractice on June 23, 2023, 11:24:47 pmIt would help if you say what OS you are running. Everyone in this thread seems to assume Linux, but in your original post you go on about Windows apps, so on the basis that you're running Windows...I would recommendMacrium Reflectwhich Just Works and is proven to be reliable. It has anti-ransomware features which you may or may not consider useful, and uses VSS so you can run a full coherent image while using the PC (most other backup solutions require you to not be using the PC, which is pretty much a guarantee for backups to drop out of use). Images can be mounted as drives for getting hold of specific files, or you can just blast the entire thing back.As to the destinations, you can pick whatever you want. Local storage is simple easy to access, but a proper disaster can screw that up. Don't have your only backup media permanently connected to your PC - a PC-based drive for backups is fine if you also have off-PC media that either duplicates it has its own backup scheme. Cloud is OK as a backup to the backup if you trust it (always encrypt the data before letting any cloud program get hold of it), but bear in mind it's a destination of last resort and may not be there when you're needing to do a restore.Having more than one media type as destination is a good idea (hence spinning disk and cloud, tape if you don't find it too tedious, etc).If you just want to save data, as opposed to a bootable image of the PC, then something likeSyncBackis good - this one will even use email as a backup destination if you like! The downside to this type is that you need a booting PC to restore the data to, but in many situation that's not a problem.In either case, you would want to think about saving a sequence of backups, to recover from things like ransomware, or that file you deleted a month ago that you really, really need now, etc. Most backup apps will allow that and have some scheme, but I would suggest you determine your own scheme and get the apps to implement that if for no other reason than you're not locked into something that is specific to some app and you don't really understand.Finally, whatever you go for make sure you stress test it. When I build a new PC I use it to restore a backup to ensure itcanbe restored - the new build is just an opportunity to test on an unused machine. Failing that, take one of your backup disks, that's not used yet (you will have more than one, and also willneverbackup to the same media sequentially) and shove that in your PC in place of your normal drive. Restore to that, to prove it works, then put your real drive back in.

author: SiliconWizard
author_rank: Super Contributor
timestamp: June 25, 2023, 09:55:43 pm
content: Quote from: ataradov on June 24, 2023, 02:00:42 amWhile my data retention "mantra" has beenreplication onlyfor years, this is a question I'm now considering.Replicating data to keep the corresponding information looks like the most reasonable approach overall, after all this is kinda what "life" itself does, in the grand scheme of things.This is all well as long as you replicate on a regular basis.The "why" is, what if, for any reason (shit happens), your data was not replicated in due time, and you lose it. Storing it on media that can last decades or even centuries potentially gives it a different chance.Whether that matters to you is of course a completely personal question.

author: ataradov
author_rank: Super Contributor
timestamp: June 25, 2023, 10:14:39 pm
content: This is why I use NAS with 4 bays/drives in RAID1 configuration. The data is stored on 4 drives at the same time and replicated instantaneously.I switched to 4 drives after using 2 drives for a long time and each of the drives died (separately in time) and NAS ensured that no data was lost. I simply ordered a new drive and the remaining drive was mirrored to it automatically with no data loss. But the time it took Amazon to deliver the replacement drive I was with just one drive, and that made me uncomfortable, so I switched to a 4 bay drive when it was time to update.

author: PlainName
author_rank: Super Contributor
timestamp: June 26, 2023, 07:46:24 am
content: Doesn't help if you get a gas leak and your NAS ends up under what's left of your house, or your mains is zapped by lightning, or any number of other 'box take-out' situations. You need more than one destination, and if you have that it doesn't really matter if you use raid 0 or 20 because it's expected to fail at some point. All raid does is make replacement a bit quicker in most circumstances, but the risk is consequently increased.

author: ataradov
author_rank: Super Contributor
timestamp: June 26, 2023, 03:57:29 pm
content: Quote from: PlainName on June 26, 2023, 07:46:24 amYes, and this is why I also have off-site backup. The answer was to the proposal to store on a potentially more lasting media. This part is fully addressed by redundant drives.

author: asmi
author_rank: Super Contributor
timestamp: June 26, 2023, 05:44:24 pm
content: I use Synology NAS DS1821+ for backups (set up with 6 HDDs and can survive failures of any 2 drives) - I've set up Synology Drive client to do automatic backups every midnight of important stuff from my main desktop (initially I set it for real-time backup, but that interferes with some programs I use, so I had to settle for daily backups), set up the same thing for my wife's laptop, and Synology itself is set up to do a client-encrypted backup into Backblaze S2 every week, and keep 30 last backups there (which covers more than half a year). This comes to about 200 GB of data stored in the cloud, which costs me around $1 per month. So it's a very economic solution, yet it still meets the 3-2-1 rule - three copies of data, two of which are on a physically separate devices (in my case electrically too as my desktop connects to the switch via fiber optic cable), one of which is off-site.A bit of stats - I've been using Synology NAS for what seems like eternity, previous DS1512+ lived for over a decade before I upgraded it to DS1821+, in those 10 years my data survived 5 HDD failures.

author: PlainName
author_rank: Super Contributor
timestamp: June 26, 2023, 06:00:06 pm
content: QuoteThis probably doesn't apply to you or anyone setting up for themselves, but...I was called to a client to recover a server which had gone down and wouldn't boot. It was running RAID5 and could survive one drive going AWOL. The problem was that two drives had failed. However, only one hadoriginallyfailed but no-one had noticed, and some time later (quite a long time) another drive had gone, and that was it.I've seen this before: the same client had, in a previous life, backed up to tape and some employee had religiously changed the tape every night before they went home. Turned out that the backups weren't working (can't recall if they were failing or just not starting) so the tapes had nothing worthwhile on them. It was that experience that persuaded the client to go for a fail-safe unattended system using RAID.Moral of the story is that regardless of the system, the stuff isn't where you think it is unless you regularly verify that. Automation can be the death of regular verification.Edit: having said that, it can conflict with what I said earlier about manual requirements ensuring the job never gets done. There is a happy medium which will vary according to circumstances.

author: Kjelt
author_rank: Super Contributor
timestamp: June 26, 2023, 08:19:44 pm
content: Quote from: PlainName on June 26, 2023, 06:00:06 pmThat is pretty unusual that a drive failure is not reported.What is not that unusual with those RAID configurations is that if 1 drive fails and the new drive is inserted to rebuild the RAID yet another one of the original disks die. Since the disks are often from the same batch and rebuilding a RAID asks a lot of the disks.Two friends and a colleague experienced this. All three had disks that ran for more than four years without glitch.Still that is why RAID is NOT a backup.To prevent this scenario replace disks after a certain period or event. This period could be for instance the end of the manufacturers guarantee period, or something like 5 years in a 24/7 situation, you may decide.An event could be one of the critical SMART errors:https://www.pcworld.com/article/410623/these-5-smart-errors-help-you-predict-your-hard-drives-death.htmlGood NASses will notify you , even sent an email to you in such an event.That is why I can't understand how some drive coukd fail without notice.

author: PlainName
author_rank: Super Contributor
timestamp: June 26, 2023, 10:49:32 pm
content: QuoteI am sure it was reported, but no-one with the ability to interpret the report saw it. Which is why I say above that a self-build techy system is probably OK, but the same system built for a non-technical user is probably not. The OP here I would suggest is not a systems guru.

author: asmi
author_rank: Super Contributor
timestamp: June 27, 2023, 04:51:48 pm
content: Quote from: PlainName on June 26, 2023, 06:00:06 pmAutomation is THE ONLY reliable way to ensure you have somewhat reliable backups. Manual backups is the worst of all, because Murphy would always guarantee that disaster strikes just as you forget to make a backup for a while, and I suspect that pretty much nobody from this "manual backupers" check backup's integrity on a regular basis.That said, I set up NAS to send me an email whenever it completes a backup to the S2, and it's also setup to verify cloud backup's integrity every week as well (and inform me of this operation's status).Also whenever a HDD failure occurs, it starts beeping such that it's impossible to not to hear it, it also sends an email to me, and I always have a fresh HDD as a hot spare (hence I bought an 8 bays unit) precisely to avoid showballing of faults like you described. Restoring redundancy hit remanining HDDs rather hard, so a secondary failure is certainly a possibility, which is why I always maintain a hot spare so that this restoration would begin immediately without my intervention. To be fair I never had such secondary failure actually occur in my setup, but I feel it's enough of a threat to justify an expense of an extra hard drive. Once restoration has completed, I only need to purchase another fresh drive and replace the failed one as a new hot standby.

author: asmi
author_rank: Super Contributor
timestamp: June 27, 2023, 04:58:40 pm
content: Quote from: PlainName on June 26, 2023, 10:49:32 pmIn this case I would set up a system with a hot spare, as well as an email notification which would in no uncertain terms say what happened and what they need to do next. This measure likely pushes the price of solution up because one would need more bays that just for a data array, as well as an extra drive, but it eliminates any urgency from a fault response (you can buy a new HDD and set it as a hot spare at your earliest convenience instead of "I need a new drive RIGHT NOW, or I risk losing my data!!", because while you no longer have a hot spare, you still are at full redundancy), so in my mind this little extra expense is totally worth it.

author: PlainName
author_rank: Super Contributor
timestamp: June 27, 2023, 10:40:43 pm
content: QuoteYes, I already said manual is a killer. But the point I was trying to make there was that automation is not the be all and end all, and can be just as useless as a manual system. You cannot just kick it off and then ignore it until it's required to get you out of the shit, and expect that it will manage that.

author: sokoloff
author_rank: Super Contributor
timestamp: June 27, 2023, 10:54:55 pm
content: For “best overall”, I think you have to consider zfs, snapshots, and shipping zfs change sets (from the snapshots) to an offsite zfs target.That will give you rollback to prior versions of files in the event of user or application error or crytpto-locker/ransomware.

author: asmi
author_rank: Super Contributor
timestamp: June 29, 2023, 02:15:54 am
content: Quote from: PlainName on June 27, 2023, 10:40:43 pmYou can also setup automatic verification of backups (which is what I did and always recommend everyone to do). This way you will get notified if something goes wrong, but you don't have to do anything manually (because any manual step is a guarantee that every once in a while it won't happen for one reason or another, and so you're exposing yourself to a potential risk).

author: asmi
author_rank: Super Contributor
timestamp: June 29, 2023, 02:17:02 am
content: Quote from: sokoloff on June 27, 2023, 10:54:55 pmIt won't save you if your entire array goes "caboom".

author: sokoloff
author_rank: Super Contributor
timestamp: June 29, 2023, 11:07:19 am
content: Could you be a little more explicit about the “caboom” scenario you are contemplating?If you lose more drives than the redundancy limit, any non-replicated snapshots would indeed be lost, but that’s true of literally any system I think.

author: asmi
author_rank: Super Contributor
timestamp: June 29, 2023, 01:59:24 pm
content: Quote from: sokoloff on June 29, 2023, 11:07:19 amLoss of whole device.Quote from: sokoloff on June 29, 2023, 11:07:19 amNot if you have an offsite backup.

author: sokoloff
author_rank: Super Contributor
timestamp: June 29, 2023, 02:07:36 pm
content: That’s the reason I included “non-replicated” in the post.

author: Marco
author_rank: Super Contributor
timestamp: June 29, 2023, 05:36:12 pm
content: Quote from: NeedsPractice on June 23, 2023, 11:24:47 pmAs interesting as the rest of the discussion is, I do think it's interesting too to search the solution space for something which actually gets close to this requirement. There seems to be a need for speed. A fair amount of older solutions like Borg and Restic are directory scan based under windows, so slow.On Linux Synology already proves you can do near time real time incremental backups, with their 5 minute btrfs snapshots. Doing something similar with btrfs on your own computer would require a lot more tinkering, especially when you want file level access to an old snapshot ... no easy GUIs available AFAIK.For windows similar technology, block based backup using snapshots, is available from Veeam and Ferro Backup systems (also R1Soft and Dell, but those don't really market to individuals). For file based backup Duplicacy seems to be the software to beat as far as speed is concerned.