Are your backups up to date?  - Page 1
Forum section: electronics / general

author: wilfred
author_rank: Super Contributor
timestamp: November 20, 2017, 02:37:26 am
content: /.

author: EEVblog
author_rank: Unknown
timestamp: November 20, 2017, 02:41:45 am
content: I'm using BackBlaze now for HD backups

author: BradC
author_rank: Super Contributor
timestamp: November 20, 2017, 02:56:10 am
content: Quote from: wilfred on November 20, 2017, 02:37:26 amI-Ching calculator says "A suffusion of yellow".Maybe with a bit more detail we might have more of an idea.Yes, my backups are up to date. 3am every day.

author: andyturk
author_rank: Frequent Contributor
timestamp: November 20, 2017, 03:00:44 am
content: Amazon Glacier says, "Yes."

author: xrunner
author_rank: Super Contributor
timestamp: November 20, 2017, 03:11:16 am
content: Quote from: andyturk on November 20, 2017, 03:00:44 amYes, mine are.

author: rdl
author_rank: Super Contributor
timestamp: November 20, 2017, 03:49:18 am
content: I have everything on a FreeNAS machine with mirrored drives. Those are synched weekly with a second machine, and periodically with a third.As far as spinning drives go, I've had 4 times as many external fail vs. internal. I no longer trust them at all. I personally still have two in use but they will be phased out this year. I now use standard internal drives and put them in external enclosures myself.

author: AG6QR
author_rank: Frequent Contributor
timestamp: November 20, 2017, 04:20:42 am
content: I use Time Machine, which is a utility built into Mac OS to keep hourly backups. My home computer is set to back up to three external drives. I normally keep two of them connected to the computer at home, and a third is kept separately at my workplace. Every couple of weeks, I rotate one of my home drives to my workplace, and bring the old workplace drive to home.My primary work computer is a MacBook, which I also have set up with Time Machine. I've got one external drive at home, and one at work. I don't move those drives around, but I do move my laptop between my workplace and home daily, and keep an external drive connected in each place.The net result is that if my workplace or home burns down, I'll lose no more than a week or so's data from my home computer, and no more than a day or so's data from my work computer. If I have a single hard drive failure anywhere, I lose less than an hour's worth of data.I've had a couple of hard drive failures since I started using this system, and restoration was quick and painless. I've also had several incidents where I accidentally erased data. Again, restoration was quick and easy.Time Machine uses an intelligent system of unix filesystem links to do its backups. That way, only changed files need to be backed up, but each time a backup is made, the entire full directory tree is replicated. It keeps hourly backups for the previous 24 hours, daily backups for the past month, and weekly backups for as long as it has space, which works out to about the previous year on my home computer, and the past six months for my work computer.My backup drives are all encrypted, and they're all in my possession, not relying on Internet or cloud storage.

author: Halcyon
author_rank: Unknown
timestamp: November 20, 2017, 05:48:18 am
content: Quote from: rdl on November 20, 2017, 03:49:18 amExternal drives rarely have "enterprise" grade drives in them. They are usually just the cheap entry-level drives, or even worse "AV/CCTV drives" which have little to no ECC capabilities on-board. Great for your young kid's iTunes backups but probably not much more.My first line of backups synchronise with a secondary NAS box with an offline (less frequently updated) copy to LTO tape, which reminds me, I should get around to updating those tapes.Absolutely critical information (such as my password database and very important documents) get a further 3 encrypted copies... one I keep on me at all times on a USB flash drive, another copy on CD/DVD that gets refreshed when there is a major change and a final "everything has gone to shit" copy in hard copy form through the use ofPaperback(all I need is a scanner or camera and a copy of Paperback to recover the files).

author: edavid
author_rank: Super Contributor
timestamp: November 20, 2017, 06:05:21 am
content: Quote from: wilfred on November 20, 2017, 02:37:26 amLarge capacity portable external drives use single, 12.5mm or 15mm thick 2.5" drives.An example drive is the WD20NPVZ, 2TB 15mm.

author: kripton2035
author_rank: Super Contributor
timestamp: November 20, 2017, 07:13:06 am
content: Quote from: wilfred onTodayat 03:37:26if it is effectively two drives in a raid config, then it's mostly two 1TB drives, in raidit implies that if one drive fails, the whole raid fails, and most of the second drive datas are not recoverable ....so it's twice more likely to fail than one single drive.I would avoid such configuration for important datas ...raid 5 or no raid (eventually mirror raid but better 2 drives)

author: ChrisLX200
author_rank: Unknown
timestamp: November 20, 2017, 11:02:17 am
content: Can recommend Macrium Reflecthttps://www.macrium.com/because it's reliable, quick and simple to operate. Saved my bacon several times. For image restore it can either boot from a thumbdrive or from the boot partition on your system drive, the latter is useful if (for e.g.,) your Windows partition is corrupt and you just want to quickly restore it. I keep full backups on seperate systems: a NAS, a discrete RAID disk array box, and copies spread around the other 4 PCs on the network. Extra backup drives installed in every workstation for this purpose are cheap insurance against one or other machine vapourising, but it does load the network down when the data transfer happens (in the small hours when not being used mostly). For real time protection I use SuperFlexible File Synchroniser (new version re-named:https://www.syncovery.com/) which is set to protect specific directory/files daily or hourly - things like email folders and working directories for project programs.

author: VK5RC
author_rank: Unknown
timestamp: November 20, 2017, 11:29:29 am
content: NAS 2x2TB raid 0 configured, backed up weekly.

author: brucehoult
author_rank: Super Contributor
timestamp: November 20, 2017, 11:52:22 am
content: I haven't bothered with conventional whole-disk backups of my hard disk for a long time.I work on multiple computers in different locations, synchronised by git, mostly peer-to-peer but in some cases via private or public git servers. I do non-work stuff on multiple portable devices as well as those multiple computers. Everything is either automatically replicated between them, or stored on someone's servers, or both. My library of media I didn't create is either itself a backup of CDs and DVDs or re-downloadable from the original source. Installed software either comes with the OS, or is easily re-downloadable form the original source (whether AppStore, apt-get, MacPorts or whatever)The only thing I actually need to back up consciously is the media I did create -- photos and videos. I have a couple of external drives (including a 2nd gen Drobo I've been using since 2008), but most of the things I'd cry if I lost end up on twitter, facebook, youtube etc anyway. rsync does the job for them.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: November 20, 2017, 12:13:53 pm
content: Quote from: ChrisLX200 on November 20, 2017, 11:02:17 amI'd second that - I have it set to do an nightly image backup of the main drive to a second drive and to a NAS.Nice thing is you can mount an image file to pull individual files out.I also have another utility, Versionbackup ( no longer sold), which keeps zips of changed files such that you can recover any version of a changed file from the last month ( or however you set it up)

author: rdl
author_rank: Super Contributor
timestamp: November 20, 2017, 02:05:46 pm
content: There must be something I don't understand because two disks in Raid 0 on a NAS does not seem like an optimal choice to me.

author: PlainName
author_rank: Super Contributor
timestamp: November 20, 2017, 02:21:06 pm
content: QuoteSounds a lot like AJC Active Backup:http://www.ajcsoft.com/active-backup.htmI've run it for a long time and can't recall ever having a problem with it. Just Works.

author: vealmike
author_rank: Regular Contributor
timestamp: November 20, 2017, 02:59:34 pm
content: That would depend on whether you wanted to optimise for speed and capacity, or for redundancy.

author: Monkeh
author_rank: Super Contributor
timestamp: November 20, 2017, 03:01:29 pm
content: Quote from: VK5RC on November 20, 2017, 11:29:29 amWhhhhyyyyyy.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: November 20, 2017, 03:46:51 pm
content: All of my machines do nightly incremental (rsync --link-dest) backups to an external drive plugged into a central backup machine in my basement. That machine syncs the external the other machines are backing up to to a second external that's a mirror copy of it daily. It then syncs the latest backup from each machine to an off-site encrypted drive. So any changes I make to any machine get duplicated that night, and quadruplicated, including one off-site copy, the following day at noon.

author: CJay
author_rank: Super Contributor
timestamp: November 20, 2017, 07:54:38 pm
content: Quote from: VK5RC on November 20, 2017, 11:29:29 amYou sure you meant 0?That's not exactly 'secure'

author: bd139
author_rank: Super Contributor
timestamp: November 20, 2017, 08:14:44 pm
content: Yes that’s less secure than one disk!

author: ChrisLX200
author_rank: Unknown
timestamp: November 20, 2017, 08:23:52 pm
content: RAID 0=Pre-scrambled and doubles the chance of hardware failure losing your data (kind of... one disk fails you lose both). Zero advantage to RAID 0 in a NAS anyway as disk speed is not your bottleneck, more likely it's your LAN speed.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 20, 2017, 09:41:05 pm
content: Quote from: wilfred on November 20, 2017, 03:58:20 amGoogle does very extensive analysis of the regular hard drives they use in data centers and found temperature not to influence drive life, unless it's quite excessive.I'm glad you started the thread, though. Most people only take heed when it's too late. Very few people can tell you what a proper backup is, even when they're familiar with computers.

author: ChrisLX200
author_rank: Unknown
timestamp: November 20, 2017, 11:03:18 pm
content: Takes experience. You have to lose the contents of a hard disk to understand the importance of backups, you need to have a failed restore due to faulty backup to understand the importance of backing up properly!

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 20, 2017, 11:13:03 pm
content: Quote from: ChrisLX200 on November 20, 2017, 11:03:18 pmYes, testing your backups is as important as making them. That's not theoretical either, I've seen an example of a narrow escape just last month.However, making proper backups in the first place seems to be rare as hen's teeth, and even if people say they make them, it's generally done is an ineffective manner.

author: amspire
author_rank: Super Contributor
timestamp: November 20, 2017, 11:19:21 pm
content: Quote from: EEVblog on November 20, 2017, 02:41:45 amLooks like Backblaze have really good prices. I do worry about how long a company can last when it has data storage costs 1/4 of its competitors. At that price, they would probably have to have disks last at least two years before they even have a chance of making money. Don't know how they do it especially if they need to have redundancy. As long as they keep up the prices though, they are a fabulous deal.Just watch out for a sudden price jump in the future. It can be traumatic when you have Terabytes of cloud storage and suddenly the price skyrockets. It has happened before and often there is no option but to pay. Microsoft used to offer unlimited cloud storage with office and some people did have Terabytes stored. Then they suddenly introduce the 1T free cap for Office 365 and so the people had the choice of paying for the extra storage or dumping the cloud files.For my backup, I use a number of 2 disk RAID1 NAS boxes. With 10T NAS drives available, the 2 disk boxes will be all I need for a long time. I sync the PC's to the NAS boxes with rsync, and have a scheduled daily script file to rsync the different of-site NAS boxes. With the exception of video files and Altera/Xilinx installation programs, I seem to be able to comfortably sync remote NAS boxes via ADSL2+ connections overnight. In a couple of years, I hopefully will have enough bandwidth to sync everything. The NAS boxes have BTRFS daily snapshots, so I can access the backup from previous days at any site. The BTRFS snapshots provide excellent protection from crytpoviruses. I do a weekly scrub on the NAS boxes to eliminate bitrot in the mirrored drives.The basic principle is for a reliable backup, you want at least two copies locally (the mirrored disk) and at least one off-site backup. You also have to remember that a backup is only a backup as long as the original files are still on your PC. If you delete the files from your PC, then you should make an extra archival copy. Otherwise a "backup" ends up becoming the primary storage for the files, and it then it needs a backup.

author: bd139
author_rank: Super Contributor
timestamp: November 20, 2017, 11:50:04 pm
content: I've got about 200Gb of data and I'm using the following:1. time machine offline on 512 gig SSD for lazy local backups2. rdiff-backup hourly to an HP microserver with FreeBSD + ZFS mirror on two 1 TiB disks.3. rsync that to an EBS volume on an AWS instance nightly.4. Everything major (pictures/documents/irrecoverable stuff) is synced to iCloud as well and is on my phone handset.I've just set the whole stack above up this week. All automated.AWS + EBS is quite expensive but cheaper than the other options. You don't get screwed hard until you actually have to do a restore - costs money to pull 200Gb out of AWS.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 21, 2017, 12:05:14 am
content: Quote from: bd139 on November 20, 2017, 11:50:04 pmIs your system able to deal with creeping corruption? All backups seem fairly immediate. That can be an issue when something's been encrypting your files on the down low, or corrupt memory causes random and intermittent corruption of files.

author: bd139
author_rank: Super Contributor
timestamp: November 21, 2017, 12:19:57 am
content: rdiff-backup is “permanently incremental” I.e you can roll back to any point in time. Checksums everything. If anything is corrupted or you get cryptolocker then it will diff the corruption as a file increment so you can go backwards in time until the file is fixed. Same with time machine.Corruption is handled via multiple target media as well.We use it for production backups for 15 years now. ZFS for about 5 years. It’s amazingly solid. Trick is buy good disks and use good operating systems.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 21, 2017, 12:21:58 am
content: Quote from: bd139 on November 21, 2017, 12:19:57 amI assume you have a way of preventing malware from eating up all your incrementals and spreading throughout your network? Do you have something that's disconnected from the rest of the network somehow?

author: bd139
author_rank: Super Contributor
timestamp: November 21, 2017, 12:29:08 am
content: Yes time machine backups are offline and two drives are cycled.

author: ChrisLX200
author_rank: Unknown
timestamp: November 21, 2017, 12:37:34 am
content: I have about 70TB diskspace spread through my network, NAS and sundry external drives (built using decent drives and ICYBOX enclosures). With astro-imaging I can generate ~10GB data per night, about half of that is all-sky camera recordings which is not critical, the rest is CCD raw images and processed images which I want to keep safe. The QNAP NAS runs Linux and the external drive connected to that is fairly well isolated from common virus attack - but it is still scanned of course.

author: amspire
author_rank: Super Contributor
timestamp: November 21, 2017, 03:20:17 am
content: Quote from: Mr. Scram on November 21, 2017, 12:21:58 amThe good thing about using a ZFS or BTRFS NAS is that snapshots are instant to create and become read only. The snapshots are part of the filesystem - not files on a drive. I have had a cryptovirus start to infect a NAS box at a company I sometimes help, and in just an hour, it had managed to encrypt about 20G of files on the NAS. The snapshots from the previous day were all fine, but since then, I have disabled all SMB Windows network file sharing to the NAS boxes. Cryptoviruses do know how to find and attack network shares. They also target and attached USB drives, so a backup USB drive permanently plugged into a PC is pretty useless. It is much more valuable to attack and encrypt files on a server then a workstation.Also when I sync the local NAS box to a offsite one, I always initiate the connection from the offsite box - there is no information on the local NAS box that a virus could use to initiate a connection to the remote one. Any connection is always on terms dictated by the offsite NAS.

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 21, 2017, 03:32:42 am
content: Raid is not a backup, but I use Linux md raid on my file server for all mass data storage so don't have to worry about drive failures or at very least not a single one. Beauty is not having to lose any productivity, no need to copy data anywhere or restore anything. Just pop another drive in and I'm good to go. (or wait, does not matter, but I like to do it asap)For actual backups I have various rsync jobs that copy data to one of the raid arrays that is dedicated for backups, I also have a drive dock that I do a backup to when I remember... and now that I think of it, it's been a while. I have several drives I rotate around with different jobs assigned to them. Typically I bring the large one to work which covers my most critical stuff. That is like my total last resort backup if the house burns down type deal. I also have some offsite rsync backup jobs that run weekly to my web server. I need to also look at cloud, but I'd want something that supports rsync so I don't need to come up with a different way to automate it. Actually I need to look at rdiff backup too. Overall I do need a massage storage upgrade so I can have better backups. Most of my backups are simply a mirror, with some oddball stuff dated monthly/weekly/daily such as databases.Oh and to those curious this is my file server setup:The middle one is where all the data is, the top one is a VM server (only has an OS drive) and the bottom stuff I don't really use but it's some SAN enclosures that are hooked up to the file server via fibre channel. Was awesome at the time, but for the power it uses and fact that I can't put my own drives in it, I find it's practically useless, so I will take them out if I need the rack space.Right now the file server has 3 raid arrays:Code:Filesystem Size Used Avail Use% Mounted on/dev/md0 5.4T 3.5T 1.7T 69% /volumes/raid1/dev/md1 6.3T 5.4T 571G 91% /volumes/raid2/dev/md3 7.2T 1.8T 5.1T 26% /volumes/raid3Not that much disk space by today's standards but I can expand as I need. md1 is a bunch of 1TB drives, I am due to upgrade that array to much larger drives. In fact all my arrays use rather small drives as that's what was economical to get at the time. md0 and md3 are raid 10, md1 is raid 5. That is my oldest array, it's dated 2008. It's been transplanted between like 3 different machines lol. That's the beauty of software raid though, you're not depending on specific hardware.

author: BradC
author_rank: Super Contributor
timestamp: November 21, 2017, 05:47:27 am
content: Quote from: amspire on November 21, 2017, 03:20:17 amI lost a 16TB RAID6 to creeping corruption. In my instance I had a dicey SATA controller that gave read errors under heavy load on 2 ports, but never enough to even register. Where it did the damage was stripe read-modify-write cycles on parity where the chunk size was 1M (so 8M stripes). Over probably 7 months it slowly corrupted 2 drives on the array until it caused noticeable damage. By then I'd lost a considerable amount of (mostly replaceable) data and corrupted a whole archive of years of digital photos that were not backed up.ZFS would probably have mitigated that, but it was strictly Solaris only at the time. BTRFS hadn't even been thought of.Along with effective and tested backups, I now run periodic md5 checks over the array and keep an eye on the array scrub mismatch_cnt (which is how I noticed the issue in the first place).The day ZFS hits the mainline kernel I might switch.

author: Red Squirrel
author_rank: Super Contributor
timestamp: November 21, 2017, 06:01:33 am
content: ZFS is something I want to look into at some point myself. MD raid has been pretty solid though, but ZFS has lot of interesting features as well.

author: amspire
author_rank: Super Contributor
timestamp: November 21, 2017, 06:38:53 am
content: Quote from: BradC on November 21, 2017, 05:47:27 amThis is the problem with raid - particularly as the disks get bigger. Raid 1 (Mirroring) is the only raid I have ever chosen to use. I could be talked into using RAID 10. To me, if you want to use RAID5/6, you may as well be looking multiple RAID systems set up to run like a SAN array to give hardware redundancy. Otherwise RAID5/6 needs a good backup, and you have to be able to accept the down time while you are rebuilding the repaired or replaced RAID drive from the backup.BTRFS is probably not as RAM hungry as ZFS and is pretty solid. ZFS has been part of FreeBSD since 2007, so if you want a really solid ZFS, that is probably the way to go. Netgear have been using BTRFS in their NAS boxes for many years now. RAID has been retarded historically as imperfect in BTRFS. I have been using it on a server myself in a RAID 1 configuration for 3 years without any issue. Netgear has been using mdadm for its raid and is then running a single drive BTRFS on top of the raid. Netgear for ages was using BTRFS at about version 0.2 and it was very reliable then.Both these file systems offer Scrub to help protect against bitrot, but you do have to schedule the Scrub - it is not automatic. It also take a really long time for huge filesystems.

author: kripton2035
author_rank: Super Contributor
timestamp: November 21, 2017, 07:33:59 am
content: Quote from: BradC on November 21, 2017, 05:47:27 amit's a good practice to backup the NAS ... on a simple drive but backup.also good to make a backup that does not stay in the same building.

author: brucehoult
author_rank: Super Contributor
timestamp: November 21, 2017, 08:03:54 am
content: Quote from: amspire on November 20, 2017, 11:19:21 pmI would suspect they are counting on everyone backing up the same OS files, apps, porn, commercial music and movies etc and only having to store one copy of them.

author: BradC
author_rank: Super Contributor
timestamp: November 21, 2017, 08:05:58 am
content: Quote from: amspire on November 21, 2017, 06:38:53 amNo, that was the problem with dicey hardware, an inadequate backup / integrity checking mechanism and my complete ignorance of the risk and management factors. Nothing to do with RAID.Quote from: amspire on November 21, 2017, 06:38:53 amI'm just not comfortable with BTRFS. A couple of us (myself and some mates) tried it on some larger filesystems earlier in the year (>30TB) using what were theoretically stable parity features, but after it trashed a number of filesystems we walked away (as did Redhat apparently). Most of those crashes were not parity related and were apparently due to the filesystem filling up. Nice, an "oh, I've run out of space" error becomes "Oh, I've lost an entire filesystem, I wonder what happened". Restoring 30TB from backup takes a while. I'm sure if we'd worked with the BTRFS list and developers we might have salvaged some stuff, but man "fool me once...". Why would you trust it?Aside from bad hardware, I've never lost an ext filesystem. In fact even *with* bad hardware I didn't lose the filesystem, I just ended up with some stripes across my jpegs and other miscellaneous file damage. Metadata was completely recoverable with a simple fsck. A BTRFS crash was a total irrecoverable loss.Make no mistake, my failure would have been caught sooner with regular scrubs (even on my RAID6), so it was a combination of bad hardware and ignorance of the mechanics on my part. Heck, I wasn't doing regular scrubs or even SMART tests. I didn't even know what I didn't know.RAID1 / RAID10 have plenty of failure modes, but when you start getting the requirement for larger data pools, they're just not as cost effective. You need to learn to mitigate the risks associated with other storage topologies.

author: BradC
author_rank: Super Contributor
timestamp: November 21, 2017, 08:09:03 am
content: Quote from: brucehoult on November 21, 2017, 08:03:54 amThat won't work. Your data is encrypted at your end with your key, so there's no way they can de-dupe.Backblaze are actually really open about a lot of what they do and how they work, it's really worth reading up on how they do what they do. I don't see them going anywhere anytime soon. They've put together a storage methodology that works for them from a cost perspective, and they're constantly getting bigger (not smaller). They've been around long enough now that if it wasn't a sustainable business for them, they'd have gone bang.

author: bd139
author_rank: Super Contributor
timestamp: November 21, 2017, 08:14:40 am
content: Just as a heads up, tarsnap are pretty good as well. They front end S3 with something less crazy and offer source code etc.

author: kulla
author_rank: Unknown
timestamp: November 21, 2017, 08:39:02 am
content: I ran mine backups on 4 drive NAS backed up to BackBlaze.But backups as backups means nothing if they are not tested before disaster happens, both scheduled and unscheduled.

author: woody
author_rank: Frequent Contributor
timestamp: November 21, 2017, 09:25:45 am
content: Yes, they are.Every 30 minutes copy of new/changed, volatile data from pc -> nas1, to a $weekday directoryOnce per day all new/changed data to nas1, to the same $weekday directoryEvery night all new/changed data from nas1 to nas2.Nas1 in mirror RAID, nas2 in mirror RAID, 200 km away from nas1.I use basic file copy, so no smart compressing, incremental, journalling, state-of-the-art tricks that need even more fancy tricks to get my files out of the backup again. Also no cloud, no recurring costs and, knock wood, so far no problems.Apart from this once in a blue moon I make an image backup. Why I do that is not clear to me; for the few times in my life I ran into disk problems I always started with a freshly installed OS.And I think TS's portable 2TB drive might not be two drives in RAID, but one 'thick' one (12mm).

author: VK5RC
author_rank: Unknown
timestamp: November 21, 2017, 10:56:26 am
content: Quote from: Monkeh on November 20, 2017, 03:01:29 pmApologies , I always get the RAID 0 and 1 numbers the wrong way around, it is 1.

author: CJay
author_rank: Super Contributor
timestamp: November 21, 2017, 12:13:56 pm
content: Quote from: Mr. Scram on November 20, 2017, 11:13:03 pmCould not agree more, backing up and not testing thatA. they can be restoredB. What is restored isn't digital gibberishis a complete waste of time.I've seen it a few times where corporates haven't had good backups for, in one case, years.

author: Monkeh
author_rank: Super Contributor
timestamp: November 21, 2017, 04:28:05 pm
content: Quote from: VK5RC on November 21, 2017, 10:56:26 amIt's easy enough to remember - AID 0 is not RAID, and describes exactly how much help you'll get when (not if) it fails.

author: bd139
author_rank: Super Contributor
timestamp: November 21, 2017, 04:43:36 pm
content: Quote from: Monkeh on November 21, 2017, 04:28:05 pmPerhaps I'm just a sociopath but I can't help laughing slightly when something major goes wrong where there is a big misunderstanding in the tech like that. It's always like the punch line for a German joke.Quote from: CJay on November 21, 2017, 12:13:56 pmI worked for a company once. Well I say worked for, but accidentally landed chief technical monkey position because I needed to eat. Turned out they had been blindly cycling tapes for about 2 years. When I reviewed the steaming turd I was landed with I noticed that the external VS160 DLT drive wasn't even connected to the server. The SCSI cable was down the back of the rack. I think someone had moved stuff around and left it like that. Still the tapes ejected and got inserted so they had the illusion of a backup

author: CJay
author_rank: Super Contributor
timestamp: November 21, 2017, 06:54:26 pm
content: Quote from: bd139 on November 21, 2017, 04:43:36 pmIt'd have been fast too and none of those pesky backup errors

author: bd139
author_rank: Super Contributor
timestamp: November 21, 2017, 09:01:23 pm
content: They actually didn’t notice they weren’t running and didn’t check for errors. Someone just went into the cupboard in the office; took the tape out and put another one in once a day. What happened between two consecutive events wasn’t their problem.

author: macboy
author_rank: Super Contributor
timestamp: November 22, 2017, 02:38:03 pm
content: Quote from: bd139 on November 21, 2017, 09:01:23 pmRelated to this, I once suffered a major data loss while working at a large company (close to 100k employees at the time), where we used Unix machines at our desktops (HPUX based) and all file storage was, of course, over the network on servers. One day a filesystem got corrupted, affecting the home directory of myself and about 40 colleagues. The RAID didn't help since it was not a disk failure but filesystem issue, so they had to go to the tapes to restore. You can see where this is going. Well it turns out the daily and weekly incremental backups weren't working correctly (nothing was copied) so they needed to fall back to the most recent complete backup... from about 7 months earlier. Luckily the code repository is separate, but the affected users lost 7 months worth of data on their home directories including e-mails, documentation for active projects, personal research notes, unsubmitted code, etc. Though not quite as bad as simply losing everything, getting "rolled back" by several months is painful. The subsequent investigation revealed a great many other filesystems also running completely without any recent backup.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 22, 2017, 03:19:37 pm
content: Quote from: bd139 on November 21, 2017, 09:01:23 pmThat's what you get when you treat or pay people like drones. You get people that do their job and their job exactly.

author: bd139
author_rank: Super Contributor
timestamp: November 22, 2017, 03:37:09 pm
content: Sometimes drones is all you can get.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 22, 2017, 03:37:44 pm
content: Quote from: bd139 on November 21, 2017, 04:43:36 pmI've seen something similar. Someone must have gotten fed up with all the errors the backup software threw and checked or unchecked the right boxes to make them go away. The problem was that the backup jobs were reported to have been run completely, but no where it was mentioned that half the data was glossed over due to errors, which were no longer visible due to the checkboxes. So you get a "job successfully completed" message in the end, as all the required steps had been kicked off, and you could blissfully drink your morning coffee.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 22, 2017, 03:39:38 pm
content: Quote from: bd139 on November 22, 2017, 03:37:09 pmThen you'd better make sure you train them properly, or give them the right checklists. If they all dance their appointed monkey dance, it's no problem, but someone needs to oversee the bigger picture.

author: Tom45
author_rank: Frequent Contributor
timestamp: November 22, 2017, 03:42:58 pm
content: Quote from: Mr. Scram on November 22, 2017, 03:39:38 pmSometimes the drone's boss is just a higher level drone.

author: bd139
author_rank: Super Contributor
timestamp: November 22, 2017, 03:58:09 pm
content: This is why I replaced all my drones with scripts and automation

author: PlainName
author_rank: Super Contributor
timestamp: November 23, 2017, 12:14:31 am
content: QuoteNice soundbite, but I've had this when the 'drone' has been the owner of the business (and rich to boot). He wasn't dumb andwantedit to work, so I don't think this example (just one of many) fits yourbon mot.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 23, 2017, 12:33:45 am
content: Quote from: dunkemhigh on November 23, 2017, 12:14:31 amHe wasn't treated or paid like a drone, so that seems to work out. Though I never claimed it fits your situation, or every situation even. Only the laws of nature get to do that, and we're not even sure of it being so.

author: PlainName
author_rank: Super Contributor
timestamp: November 23, 2017, 01:01:02 am
content: QuoteHardly! Still have the religiously changed but still blank tapes, which I'd class as not working outNo, my point wasn't that this was an exception to the rule, but that perhaps the rule is arse about face. That is, the supposition is that treating someone like a drone gets you a bad service which, on the face of it, seems reasonable. But the ultimate drone has to be a computer, and that works out just fine (except when it doesn't). Typically, it fails to work because you've neglected to cover some situation in its programming, and maybe that's the problem with the drone thing. Make something explicitly part of the job and its covered.So maybe the fix is to treat your dronesasdrones, being explicit in what they should do, and realising that if you haven't 'programmed' a situation it won't be covered. A failure is then a programming issue (i.e. you've failed the think things through enough to let your drone know how to react to a bad situation). If you tell 'em to change a tape every day and they do that but haven't checked to see if the backup has been done, that isn't their fault but yours for not making that check part of the job.Which is not to say or imply that a 'drone' her is a brainless moron.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 23, 2017, 01:16:03 am
content: Quote from: dunkemhigh on November 23, 2017, 01:01:02 amI meant the statement itself was working out, as it didn't apply to the boss. It only applies to drones, which this boss isn't. Though I don't think finding an exception changes much. I'm not trying to write the laws of physics.I'm sure I addressed the rest of your comment a few posts back.Quote from: Mr. Scram on November 22, 2017, 03:39:38 pm

author: PlainName
author_rank: Super Contributor
timestamp: November 23, 2017, 02:56:23 am
content: QuoteOK.

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 02:08:43 pm
content: Quote from: wilfred on November 20, 2017, 02:37:26 amDo I have all my backups up to date ?Oh yes...........Absolutely up to dateSystems all backed up daily (incremental) and full every week - to Qnap TS453 Pro with 4 WD 3TB Red drives in Raid 5. Qnap backed up weekly to external drives - 2 x 6TB, 2 x 5TB. Main PC's also backed up weekly to 2.5 external SSDs. (Grandfather Father Son method). Main QNAP NAS backed monthly to 3 x 4TB WD external and stored at a friends.On top of thatalso still have the TS212 with 2 x 6TB in JBOD and a 'few other Nas units with Raid 1 sets' knocking about. All units and PC systems protected by a couple of APC2200 UPS units.Oh, and the PC's, all the windows user 'folders' are on seperate SSD's from the main OS drive. The OS is fully up to date and maintained that way -as are the programs. Many items are run sandboxed in VM's using VMware and also the QNAP Virtualisation station. Full antivirus / malware in real time and weekly run of 'Trend Housecall' every week or two .. also normal good common sense in usage and what you open or do not open. I have NEVER had a virus or Malware on a PC. I only use Android for watching movies on a tablet and do not have mobile data on my 'phone'. I avoid windows 10 like the plague it is ... nasty nasty OS...I was full time liveaboard 'yotty' for 10 years. Had ALL my life's backups of photos, videos, Movies, Music data etc etc etc stored on DVD's. Salt air and DVDs donota nice combination make. DVD's are subject to insidious failure in storage - on dry land, let alone on board a 'yott'. (GENUINE(Verbatim are the bloody WORSE !). Once you lose such valuable data you do your best to ensure it will never happen again. A past life as a Network Security Manager (for the UK NHS for the Northern UK) leaves one a tad 'on the safe side' - a CISSP qualification also introduces 'slight ocd'Backups and OCD often go hand in hand..... so my shrink says...............On top of all that, the house is on a hill (no flooding) and built of concrete (unlikely to burn)..I will ask my nurse for some more of my 'steady pills' nowI need to lie down............Now, where did that wire go....................

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 23, 2017, 02:16:27 pm
content: How often do you test your backups?

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 02:37:55 pm
content: Quote from: Mr. Scram on November 23, 2017, 02:16:27 pmI don' t test the individual PC backups as there is no need. Nothing of great importance could be lost due to other backups providing backupsThe NAS unit I am perfectly happy with as far as ANY (relatively) large raid 5 system can go. The raid is scrubbed every week. Recovery of a single drive failure large raid 5 array is always finger biting no matter what the machine used. The main raid is backed up at a folder / file level - NOT as an image. The PC's data is backed up to the NAS at a folder / file level (which is why the user directories are on a separate drive). The only 'sector image backups' are the individual PC's to the GFS SD units - and again - I can safely recover that from other areas. The GFS system always means there are 3 independent image copies with little date difference. (And again, this only applies to the PC's / Laptops) .All the data is therefore stored on a file by file basis with multiple backups. There is simply no need to test. It is utterly pointless. Image backups are a different animal which is why I limit that to PC's -(and VM's actually).All external drives are checked occasionally for any error using manufacturer's applications (non destructive).There is simply no need to 'test' anything. Relying on RAID - even with drive failure ' fault tolerance' is very very bad karma. Raid is not failsafe. It MUST be backed up. There is a very real chance of raid not recovering effectively and this is directly related to the amount of data our drives can store. I am 99.9999999% happy with the reliability and robustness of my systems. Delirious in fact.. If it was a commercial enterprise I would 'consider' use a real time off site link storage / backup, however, it isn't and I dont. Off site storage 'by hand' is usually quite sufficient even for a moderately sized commercial enterprise -- depending of course on the format of the data and number of copies stored - a GFS method should be used in that case (grandfather father son).

author: John Coloccia
author_rank: Super Contributor
timestamp: November 23, 2017, 02:55:56 pm
content: I have a Synology appliance set for Raid 1 that I backup to. I use Macrium Reflect for backups. I switched from Acronis (the business one, not the personal one) and dropped it because it was an absolute piece of junk that gave me headaches all the time. Macrium has been headache free.I also have a SATA dock and I occasionally take an image on a drive and keep it as an offsite backup.I use to have a backup plan that did a full backup every 2 weeks, differentials every 2 days, and incrementals every couple of hours. Now I just do a full backup every week or so. At some point, I'll kick off a differential every couple of days but haven't gotten around to it on my new system yet.

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 23, 2017, 03:20:47 pm
content: Quote from: Freelander on November 23, 2017, 02:37:55 pmHaving many copies of the original copy doesn't protect you from a lot of failure modes. You really do need to test whether you can recover relevant files from your backups on a regular basis. If you don't test, you don't know what you have. Your stacks of copies could be full of perfect files full of perfect garbage.

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 03:29:58 pm
content: Quote from: John Coloccia on November 23, 2017, 02:55:56 pmI believe Synology units are very well respected. ! . Do they not have some form of real-time 'sync' application ? - that may be useful. I only have experience of the QNAP Pro units for home use. The last commercial NAS units I used were Compaq Proliant external Raid units linked via Compaq failover- (serial port heartbeat link) - (and this wasn't really a true NAS as it used Windows NT4server to link to the network. NT was an excellent system with odd numbered service packs providing you wore adult pampers.... that dates it ...///Anyways, back to the Synology.There may be a method to use a sync program on the PC to keep your main designated data folders duplicated on the nas - (with the 'don't delete files on the nas when deleted on the host' checked). On the QNAP this (Qsync) syncs my D Drives (ssd with all user folders) to a 'homes' directory on the NAS. A schedule on the NAS syncs this to another area - a copy on the local nas and a copy to an external drive in my case . You could link that to an external drive if the sync software is similar to the qnap stuff. It does a good job I must confess. The nas spends 23 hours of most days in 'sleep' mode. It wakes at 10 in the morning for 1 hour to sync then sleeps again unless another main sync job is running. (System sleep schedule as opposed to drive sleep saves leccy and adds up over the year). The applications available for for the QNAP are excellent and most things are catered for, I am sure Synology is the same. A new 'hybrid backup' application is now provided with the QNAP O/S. This is excellent and now offers cloud backup which is great for my Google Drive and works with all major cloud based systems. Not looked at 'macrium' - I fully agree re Acronis - what a POS ! (and that is not 'point of sale!) .. Jeez, attempting to rid your PC of this /infection/ is a nightmare also- it is easier to get rid of herpes..... I use a nice little utility called EaseUS Todo free. VERY very good. ! Well chuffed with it. Lacks full scheduling but I always use it manually anyway. Only ever had to use the restore once for the main system drive after an SSD died and it was painless. The provided boot / recovery software does what it says on the tin.. totally free and no ads or strings attached. Never needed anything else. I must get a dock one day as I have a stack of odd drives knocking about. Have you found the dock reliable ?edit - just to clarify as I didn't specify - EaseUS Todo is a BACKUP application. "EaseUS Todo Backup (free)" - superb piece of code - thoroughly recommended.

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 04:13:13 pm
content: QuoteNot at all.It is not the gun, it's the gunner...Static files are subject to hashing when written, as they are static by nature they will be flagged by hash corruption. A total non issue.Dynamic files are also subject to hashing but of no use as a direct integrity check. The parent application should maintain this integrity check. - self corruption is therefore a non issue providing multiple date/state copies exist. A total non issue.Dynamic files and the issue of 'user' corruption / invalid or incorrect data entry or deletion or 'insert and other 'PBCAK' here' is managed by standard dynamic file methodology and also a far FAR longer TBO regime - (time before overwrite - if you are unfamiliar)Dynamic files that are not self managed for data integrity are also subject to extended TBO. Depending on needs and pockets and regulations TBO can be up of 5 years or more (7 in a lot of cases in the UK)Again, with my system and my regime I am 99.999999% happy and secure. It is a total NON issue. Simply unnecessary..

author: Mr. Scram
author_rank: Super Contributor
timestamp: November 23, 2017, 04:22:53 pm
content: Quote from: Freelander on November 23, 2017, 04:13:13 pmWhat happens when you RAM goes corrupt and slowly corrupts data over time, which shows itself after a while when the errors built up to critical mass? All the garbage is copied perfectly down the line, again and again, and your last clean backups will be months or even years ago. Not testing anything is setting yourself up for failure. You're not the first and won't be the last. In the end, there has to be a monkey checking to see if the recovery process output lines up with the input.Besides, anyone not nervous about his backups is complacent and will fall eventually

author: CJay
author_rank: Super Contributor
timestamp: November 23, 2017, 04:41:40 pm
content: Ahh, the NHS, they pay *really* well when they lose critical patient data because their backups failed to restore sensible data.I think I went to Cornwall for two weeks in 5 star on one of those or was that the private cosmetic surgery place, I forget...

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 04:58:02 pm
content: QuoteWhat happens ? - if you follow the perfectly standard and normal advice I have list above - nothing at all is what happens(ps - love thebit ..)The methodology serves well in mission critical systems. - 999 Emergency services for example(which I personally have MANY years of experience managing). Oh, and there is not a single computer in use in that system with non ECC memoryThe static stuff you have no worries with as I list before - if it is designated as static (ie - completed record) and has changed then it is flagged as problematic - if static and hash checked / verified it will be EXACTLY the same in 10 years as it was when written and verified. - A non issue.If dynamic, follow the dynamic methodology as I listed above. Your points are then invalid - A non issue. (again)The TBO is set for your tangible or perceived value and risk analysis - hence - a non issue.You should also save up for ECC memory if data is being manipulated on a regular basisMost issues are caused by lack of knowledge and lack of spending on appropriate infrastructure - nearly as many issues in fact as incompatible bloody updates on 'insert favourite data storage rip off software here'.Lack of computing ability often means inability to verify or validate in real time - monkey turns off 'feature' instead of monkey's manager telling monkey's manager's manager to get his hands in his pocket for suitable hardware...Lack of 'storage room' to store old outdated hardware in an air conditioned and safe environment, along with old outdated software and OS and updates/ patches to enable that very old TBO dusty data medium to actually RE-INSTALL / revert in the first place. Oracle and Citrix used to be the very worst culprits for hardware and OS specific (including SP / HF level) incompatibility and NO conversion application.There are far far more REAL problematic areas for holistic system integrity and validity. McDonnell Douglas have at least stopped putting the big unprotected OFF switch on their latest iteration of what was a Unix Mini - switch placed just at the right height for MD monkey to turn it off with his knee... APC system wide UPS units now don't completely die and kill the system when the cleaner plugs the hoover into the red socket after spending 20 minutes levering out the block plug with a teaspoon...Being 'nervous' about backups is perfectly normal. Being confident in ones systems is also perfectly normal and not mutually exclusive. If you drive to the IAM 'system (Institute of advanced motorists) standards - as in Police / Ambulance teachings, then you will never have an accident that is your fault. It doesnt mean you should not be nervous though......... 'tis the other buggers....

author: bd139
author_rank: Super Contributor
timestamp: November 23, 2017, 05:07:33 pm
content: My desktop PC (HP Z620 w/ CentOS 7) has ECC RAM. It's pretty essential for data integrity.I'd be happier if they did Macs with them though. I'd pay more for it.

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 05:10:37 pm
content: Quote from: CJay on November 23, 2017, 04:41:40 pm100% agree. Since they closed the specialist NHS Information Authority (NHSIA) of which I was a part of, and farmed it out to muppets to 'save a few quid' they have had HUGE issues, many of which are never reported as patients would run from the hospitals.....(I must confess though the pay as a senior manager was really excellent and the golden handshake at 47 when they farmed it to 'outside contractors - at more than 3 times the cost in the first year !!!!! - was worth more than my house ! and then my pension at 50 .. was rather good of them...) ...The NHS 'mail' system was commissioned by the government and given to EDS - yes, the EDS that installed the Passport system that self imploded and disappeared up it's own rectum after depositing its crap everywhere. It must have been a good bribe or two.. against all advice and warning, in writing to the highest levels of Government, against all risk assessments, they gave it to bloody EDS !!! - what a joke- Ed's mail we called it.. a total sack of donkey's doop. 195 Million squids written off and not a penny recouped from 'Ed' - EDS are a joke.... The government 'upper monkeys have a total inability to write appropriate contracts, FULL statements of needs and penalty clauses. And these are the same idiots that are handling brexit.... Mohahahahahaha... we are all doomed I tell ya..

author: bd139
author_rank: Super Contributor
timestamp: November 23, 2017, 05:17:44 pm
content: That's funny. I have some family members who work close to NHS IT. It's a shitfest and a half. Glad I work in private sector finance.They need to bring it all in house and run it like they ran Spine 2.

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 06:10:44 pm
content: Quote from: bd139 on November 23, 2017, 05:17:44 pmI would agree. ! - when we had NHSnet (1) with BT and Cable and Wireless it was very very good. !. for security there was a national team of security managers each with a portion of the country to administer - mine was Northwest UK. ANYTHING that health authorities / trusts wanted to do with reference to connection to other entities and visa versa had to go through one of us. There were effectively two national firewalls (BT and C&W) and even to apply a rule was down to our NHSIA team member. We were also 'inspectors' and enforcers for the total network security - all aspects - including backups and system integrity ! - and had the power to 'pull the plug' on a non conforming trust (it was threatened but never actually needed to be used). We pen-tested the systems at our leisure (which was fun), a great way to pass an evening and far better than 'telly'Our favourite pastime was locking the Finance Director of a trust or HA out of his systems completely by changing their password after cracking it. It was usually ridiculously simple and very worthwhile because the Finance Directors virtually always were IT directors as well - !!!! (Ridiculous yes, but that's the way it was - they knew bugger all about IT but controlled the purse strings).Many an IT department manager would contact us to .. errrrr.. how shall we put this .... 'stitch up' the Finance Director' when he refused to pay for a vital upgrade because he hadn't got a clue what it actually did or was for, and usually did not believe the IT Manager as these 'computer' things were expensive and it worked ok for 3 years and he ran windows 95 at home with no worries...... Oh what joy when he had to call me or one of the other guys on the national team as we could lock them out and keep them out for not following protocols. The money usually flowed in the right direction very rapidly from then on. A nice message on their screen saying something like - please contact your Chief Executive or the NHSIA always, for some reason, resulted in us being contacted and not the Chief exec.I am glad I am out of it - I left in 2004 at the time they disbanded the agency. They reap what they sow. It's gone to ratshit now as far as I can gather. Same old problems, lack of money and lack of wages. Prior to the NHSIA I managed IT at Lancs Ambulance service. We were the highest paid IT staff in the country - by far ( 1995 - 2000). That was only because we had a chief exec that had his head screwed on and also the fact that we (there were only two of us originally) put forward concise business cases to update and develop the systems - and save costs ! - (we were using MD Unix systems over 9600 baud serial) and paying nearly 200 grand a year maintenance. We factored in our wage costs and expenses and a complete change to NT / Compaq and won the case. We even got to be one of the main pilot schemes for NHS Direct as it is called now. A million quids worth of hardware and software all specced, built and set up in house and all the staff we needed to do the job - it was like being in a sweetie shop. My assistant manager was payed more than the assistant finance director. ! - we never, ever, had a staff retention problem. None of the staff were below MCSE qualification (we were microsoft only). All training paid for in house.Then the government came along and wanted to make a 'joined up' national system like something only a Jedi knight could envisage... which was, of course, the beginning of the end. - ah, but I digress... sad to hear it is still bad in many areas.

author: Freelander
author_rank: Regular Contributor
timestamp: November 23, 2017, 09:32:25 pm
content: Quote from: Mr. Scram on November 23, 2017, 04:22:53 pmHi Mr S.A bit more detail here. Overall I support your sentiment and ideals for 'testing' of backups and system maintainability (in an ideal world). I thoroughly stand by what I have said though but have been tongue in cheek on some points - forgive me.If it helps, what I found over my career in the 'business' was that what is 'ideal' in theory simply does not and could not work in practice. Because of this, one has to develop robust and sustainable strategies to maintain reliability, continuity and accuracy / availability of data - as well as the option of reversion to earlier states if needed with the minimum of time delay and offline status. It would be very nice in an ideal world to be able to test every backup in a physical sense. This is, in practically all cases simply not feasible.Take for instance a scenario I have a lot of experience with. A 999 (911 to our Yank friends) Emergency control room. If we look at the system as a whole - and there may well be three slightly different versions of the system running in a single control room in an attempt to remove single point failure and also versions of the non commonality even in each of the 'semi duplicate main systems' (as mentioned below) - along with this, there is also the real time logging to a fail over site - from EACH copy of the system...... so, for each 'failsafe minimised' copy of the main systems running ----- (and again - this is repeated for EACH instance of the C&C ! - ) let us say there is a main server, a failover server and appropriate storage attached however needed. The system runs various flavours of database, GIS (Graphical Information systems), Scripted advice systems (the scripts the operators can rely on to advise patients where needed - and depending on qualification of the operator), a vehicle radio and crew audio / video interface system, a system status management module ( an overwatch for all vehicle placements and call geographics and historical data based prediction units for vehicle repositioning) and of course, the main command and control interface. There are many more parts to the system as well that we can just refer to as /management and information systems and monitoring. The main command and control system is the main player here and let us say it is from company A. It is the overseer and controller / caller of the functionality of the various system parts. The database is from company B and although called and commanded by the C&C (command and control) is independently managed by its own main application. The GIS is from company C and tightly integrated to the C&C, it however, also has a completely independent supervisor / overseer and indeed, database storage system. This database storage and management package may not be from the same supplier as database B. ! - (Ideally yes, in the real world, often no). Again, this is integrated at a high level into the C&C (A). The vehicle communication system is from company D. It is again heavily integrated into C&C A. It may well have a bespoke data logging system of it's own. It will communicate with the radio comms stack (physical) and that will communicate with the the various GSM/G or Radio based systems and their command and control networks. The scripted patient advice system - lets call this E, is again tightly integrated into C&C A, but is in reality a separate module again with a full internal management package and database (which hopefully is on a platform used by another part of the system but often is not as it may be bespoke. ..... In the background is a stack of monitoring systems and overwatch. These systems tend to be so specialised that no one supplier provides a complete one stop shop, or indeed could do that. they rely on many many parts all working semi autonomously but under a master C&C.As you can imagine, the backups for these systems are HUGELY complex and disparate and can involve many pathways and timings.To add to all the above, the system also runs on hardware with an OS - or OSes! and the hardware and OS are updated in line with needs and fixes and security patches where needed all these aspects of the hardware and OS can impact the system. As you can imagine, there are also external links to the outside world via firewalls and gateways, each with their own OS. firmware etc.It is simply IMPOSSIBLE to 'test' a backup of the system. Even if one had a complete and identical setup and infrastructure. It would also be impossible to apply a full system load scenario onto any restoration. If a data block is verified and hashed at backup time it will restore if that verification is intact. If the data was corrupted prior to backup time and not flagged by the management software of that part of the system, it will be corrupted in the backup (but correctly hashed and verified) and will also restore correctly !. Unless the system is fully functional and operable you cannot evaluate it and it is simply not possible to have a complete identical highly complex clone of the system to test with. Many sub systems can run their own off site cloud backups that are specific to that system and fully integrity checked as the backup arm of the application is part and parcel of it. Other parts may well use a specific backup application whereas the database modules may use another backup application and system. Transaction logging may and does present complex multiple real time backups. Also add to this that many of the workstations will run differing workstation specifications and OS patch levels and multiple servers for failover will often run differing levels of system state / hardware / firmware as Identical OS, hardware, bios etc etc that fails at a level below the main application, may well suffer exactly the same failure on fail safe change over. !. a classic single point failure SPF. this is why workstations are often split 50:50 as to Hardware types and patch levels of OS and drivers. A failure occurring due to an interaction of ANY part on a workstation may well affect ALL unless one has the split hardware to remove another SPF. This is, of course, necessary to allow for and attempt to achieve the 'ideal world' removal of single point failures. (hah !) - one can only try... for that to be possible, the parts would all have to be fabricated from Unobtainium !!!.Again, to 'test' a backup (well multiple interdependent backupS) is simply not feasible. One cannot take a system 'offline' for a test of a backup restore, which would also be quite good fun (not) as the system would have to be wound down, backed up in all forms, prior to the test of the restore and then it would have to be restored again back to the state it was in prior to the restore in the first place.........Add to this that without load the system would be highly UNLIKELY to reveal any issues at all if data corruption occurred prior to backup (your /ram/ degradation hypotheses for example - although - there would be no non ecc memory anywhere near that system. - even in the late 90s, the section or module of ECC memory could be taken off line automatically - in real time - if issues were flagged. All transparently by the server.) Enterprise backup solutions are extremely (incredibly) good at flagging any discrepancy and run a huge amount of checks on each data pack. Errors that occur are almost certainly corruption or deletion PRIOR to backup and corruption AFTER backup and AFTER verify whilst in storage.The same 'problems' as above and inability to 'test' backups in real world occur in many systems - such as ATC (Air traffic control) etc.With this level of complexity and mission critical status - in fact - LIFE critical systems if you boil it down !. it is simply NOT possible to test integrity of backups after the fact. The multitude of protection systems must be in place to interrogate the data integrity and structure PRIOR to backup time. Testing DURING backup is an added bonus.Hope that helps. Real world is a beee-atch !!, but can be suitably tamed and controlled.One may ask then - HOW does one restore in the event of a system failure... well.. firstly, I have never had acompletesystem failure apart from a power issue in the mid 90's.... (it's all the fault of APC and a vacuum cleaner ho hum)... the system can still function with multiple sections not available. There are redundant parts to most sections, the GIS is backed up by a secondary mapping system, the patient advice is backed up by crib cards on the desks, the radio system is fully redundant and backed up by personal cell units .. etc etc etc... The whole system can be offloaded to another site if needed.How would we bring a system back up ? ..... it is like how a hedgehog screws... very carefully..... no, seriously, we bring it up one piece at a time and test each section and then bring another unit on line. The systems are never totally powered down either, due to the multiple systems (slightly different copies of systems) and failovers, sections can be taken down and updated or new updates tested in relative safety.. minimising SPF is the key really. There is so much more to a system of such complexity than a single backup - as a single 'backup' does not exist. One has to plan for remedial action after failure, but the key thing is to plan and invest in multiple redundancy and continuity in the first place. A bit like an airliner... and in the same way, when a system (well, subsystem) fails you investigate and add redundancy along with a solution where possible. Thats why every desk has a stack of note taking forms, pencils and pens, a cellphone or two and a set of maps and mapbooks, torches etc, that is why printers run continuously at the back of the room producing summary paper backups of ongoing jobs,,,,,,, thank god we never needed them.So, with the best will in the world, no, It is not essential to 'test' backups.. providing the appropriate steps are taken.Jeez, I am so glad I retired