From: pinard@IRO.UMontreal.CA (Francois Pinard)
Newsgroups: comp.unix.admin
Subject: Re: WANTED: Unix administration horror stories !
Date: 15 Oct 92 14:22:37 GMT
Organization: Universite' de Montre'al

Many things happened in those many years I've been with computers.
The most horrorful story I've seen is not UNIX related, but it is
certainly worth a tale.  Here it goes.

This big (:-) CDC 6600 system was bootable from tape drive 0, using
these 12 inches wheels containing 1/2" tape.  The *whole* system was
reloaded anew from the tape each time we restarted the machine,
because there was no permanent file system yet, the disks were not
meant to retain files through computer restarts (unbelievable today, I
know :-).  The deadstart tapes (as they were called) were quite
valuable, and we were keeping at least a dozen backups of those, going
back maybe one or two years in development.

The problem was that the two vacuum capstans which were driving the
tape 0, near the magnetic heads, were not perfectly synchronized, due
to an hardware misadjustment.  So they were stretching the tape while
they were reading it, wearing it in a way invisible to the eye, but
nevertheless making the tape irrecoverable.  Besides that, everything
was looking normal in the tape physical and electrical operations.  Of
course, nobody knew about this problem when it suddenly appeared.

All this happened while all the system administration team went into
vacation at the same time.  Not being a traveler, I just stayed
available `on call'.  The knowledgeable operators were able solve many
situations, and being kind guys for me (I was for them :-), they would
not disturb me just for a non-working deadstart tape.  Further, they
had a full list of all deadstart backup tapes.  So, they first tried
(and destroyed) half a dozen backups before turning the machine to the
hardware guys, whom destroyed themselves a few more.

The technicians had their own systems for diagnostics, all bootable
from tape drive 0, of course.  They had far less backups to we did.
They destroyed almost them all before calling me in.  Once told what
happened, my only suggestion was to alter the deadstart sequence so to
become able to boot from another tape drive.  Strangely enough, nobody
thought about it yet.  In these old times, software guys were always
suspecting hardware, and vice versa :-).

Happily enough, the few tapes left started, both for production and
for the technicians.  Tape drive 0 being quite suspectable, the
technicians finally discovered the problem and repaired it.  My only
job left was to upgrade the system from almost one year back, before
turning it to operations.  This was at the time, now seemingly lost,
when system teams were heavily modifying their operating system
sources.  This was also the time when everything not on big tapes was
all on punched Hollerith cards, the only interactive device being the
system console.  It took me many days, alone, having the machine in
standalone mode.  The crowd of users stopped regularily in the windows
of the computer room, taking bets, as they were used to do, on how
fast I will get the machine back up (I got some of my supporters
loosing their money, this time :-).

This was quite hard work for me, done under high pressure.  When the
remainder of the staff returned from trip, and when I told them the
whole tale, we decided to never synchronize our holidays again.
--
Franc,ois Pinard        ``Vivement GNU!''        pinard@iro.umontreal.ca
(514) 588-4656                                    ...!uunet!iros1!pinard
Consider joining the League for Programming Freedom.  Email for details!
Newsgroups: comp.unix.admin