From: rca@Ingres.COM (Bob Arnold)
Subject: Re: WANTED: Unix administration horror stories !
Reply-To: rca@Ingres.COM (Bob Arnold)
Organization: Ask Computer Systems Inc., Ingres Division, Alameda CA 94501
Date: 12 Oct 92 23:35:24 GMT

Many moons ago, in my first sysadmin job, learning via "on-the-job
training", I was in charge of a UNIX box who's user disk developed a
bad block.  (Maybe you can see it already ...)

The "format" man page seemed to indicate that it could repair bad
blocks.  (Can you see it now?)  I read the man page very carefully.
Nowhere did it indicate any kind of destructive behavior.

I was brave and bold, not to mention boneheaded, and formatted the user disk.
Heh.

The good news:
	1) The bad block was gone.
	2) I was about to learn a lot real fast :-)
The bad news:
	1) The user data was gone too.
	2) The users weren't happy, to say the least.

Having recently made a full backup of the disk, I knew I was in for a
miserable all day restore.  Why all day?  It took 8 hours to dump
that disk to 40 floppies.  And I had incrementals (levels 1, 2, 3, 4,
and 5, which were another sign of my novice state) to layer on top
of the full.

Only it got worse.  The floppy drive had intermittent problems reading
some of the floppies.  So I had to go back and retry to get the files
which were missed on the first attempt.

This was also a port of Version 7 UNIX (like I said, this was many
moons ago).  It had a program called "restor", primordial ancestor of
BSD's "restore".  If you used the "x" option to extract selected files
(the ones missed on earlier attempts), "restor" would use the *inode
number* as the name of the extracted files.  You had to move the
extracted files to their correct locations yourself (the man page said
to write a shellscript to do this :-().  I didn't know much about shell
scripts at the time, but I learned a lot more that week.

Yes, it took me a full week, including the weekend, maybe 120 hours or
more, to get what I could (probably 95% of the data) off the backups.
And there were a few ownership and permissions problems to be cleaned up
after that.

Once burned twice shy.  This is the only truly catastrophic mistake I've
ever made as a sysadmin, I'm glad to be able to say.

I kept a copy of my memo to the users after I had done what I could.
Reading it over now is sobering indeed!  I also kept my extensive notes
on the restore process - thank goodness I've never had to use them since.

Morals:
	1) The "man" pages don't tell you everything you need to know.
	2) Don't do backups to floppies.
	3) Test your backups to make sure they are readable.
	4) Handle the format program (and anything else that writes directly
	   to disk devices) like nitroglycerine.
	5) Strenuously avoid systems with inadequate backup and restore
	   programs wherever possible (thank goodness for "restore" with
	   an "e"!).
	6) If you've never done sysadmin work before, take a formal
	   training class.

Well, I haven't thought about that one in a while!  I can laugh about
it now ....

		Bob

--
  __   _    _   Bob Arnold		Ingres, An ASK Corporation
|/  \ / \  / \| 			1080 Marina Village Parkway
|    /    /   |				Alameda, CA, 94501
|    \__/ \__/| rca@ingres.com		510/748-2819
Newsgroups: comp.unix.admin