From: nagappa@menudo.uh.edu (Chaitanya Nagappa)
Newsgroups: comp.unix.admin
Subject: Re: WANTED: Unix administration horror stories !
Date: 16 Oct 92 13:03:59 GMT
Reply-To: ravi@usv.com
Organization: University of Houston

The following article is posted by a friend from my account:
Chai Nagappa
===================================================================
Hi,
This is Ravi. Needed to add just a couple of stories from all the wierd stuff
that have happenned. So, are these tales for around a campfire on Halloween?

At one time, there were three of us working on a unique SVR3.2 motorola based
machine, on a R&D project. I took care of all the SysAdmin tasks, I had a
back up administrator, and the third person had been stuck into my group
(company politics). The group project files were in /user and the individial
ones in /user2. We had managed to get backup from the operations department
for /user only (not even /; security paranoia?). Anyway, I had another scsi
hard disk that I used for making a disk copy of the primary scsi hard disk
every Friday. This disk was connected, but not mounted, so that I could
do the disk backup from my desk when I wanted to.
This machine used to sometimes get a scsi error such that you could not log
in, but the processes already running on the machine were not affected. If
were logged in the console, you just powered off the machine for a few minutes
and rebooted it. Around holidays time the other Admin was off in a long
vacation. I had taken Monday off, and headed off for a four day weekend.
The machine does the same blurp. The third person decides the power off the
machine & turn it back on immediately. It does not come up properly. She
decides to reinstall the machine using the installation tape that I had
unfortunately left in the open. Reformats the hard disk, installs the base
system, and is stuck at that point when I come back in on Tuesday. I almost
blow a blood vessel but try to keep calm 'cause I had made a disk copy about
10 days before (too anxious to get on my holiday the previous week). Try to
mount the disk... hit vaccuum. Try using dd to look at the disk... Seemed
to be a large /dev/null :-? When the lady decided to reinstall the system,
it asked her what scsi disks she wanted to reformat, and she said "y" for
both 0 & 1!! All my sample/trial&error work for a year had bitten the dust.
My only (small) consolation was that I was not the only one affected.

Story 2. Live 24 hour online system. Does backup over the ethernet to a
SCSI tape. Unfortunately, no SCSI on this system to recover if root/ethernet
dies. This was a Compaq Systempro running SCO Unix. Slated a downtime of
4-6am. I thought that it will take me only 30 minutes, as I had installed
a similar (Adaptec) SCSI board on a similiar hardware on SCO. Only difference
was that this machine was running MPX (multiprocess extension) and you had
to deinstall it, install the SCSI, and then reinstall MPX (proper procedure).
I had made all my slot/IRQ charts the previous day, and so got busy removing
MPX. Then said "mkdev tape", go through the IDs, and am almost at home
base. Then... "link kit not installed, use floppy X1" when I tried to remake
the kernel. For some reason, when I removed the multiprocessor extension,
the single processor files were not moved to their right location. And if
I reinstalled the single, all my changes would be lost. Finally, restored the
OS (from backup) on the remote machine, and then rcp-ed them over to bring back
the MPX version. Unfortunately, rcp does not maintain the date/ permissions,
etc. Got a limpimg version of the machine back on-line about 45 minutes
after its slated time, and spent the rest of the day fixing vagrant files.
The next week, I moved the online programs to another machine (a headache),
and reinstalled this machine from scratch.

Ok, that should be enough horror. Please send any replies to "ravi@usv.com"
instead of this account.

Thanks,
--Ravi Ramachandran