From: hirai@cc.swarthmore.edu (Eiji Hirai)
Subject: Re: WANTED: Unix administration horror stories !
Organization: Information Services, Swarthmore College, Swarthmore, PA, USA
Date: Sun, 11 Oct 1992 19:27:50 GMT

Some of these stories of pure stupidity rather than of interesting horror
but they did happen.

[ BTW, these happened at a different place at a different time than where I
  am now.  Don't bother my current employer about it. ]

(1) A consultant we had hired (and not a very good one) was installing Unix
on one our workstations.  He was mucking with creating and deleting
/dev/tty* files and made /dev/tty a regular file.  Weird things started to
happen.  Commands would only print their output if you pressed return twice,
etc.  Fortunately, we solved the problem by re-mknod-ing /dev/tty.  However,
it took a while to realize what was causing this problem.

(2) I wanted to create a second swap partition on another disk and made the
partition start at sector 0 of the disk! (which sounded ok at the time since
all other regular 'a' partitions started on sector 0) Every time I rebooted,
fsck would complain about missing partition tables - I initially suspected
that the disk was bad but I later realized that swapping was overwriting the
partition table.  I had lost an unknown percentage of the financial data for
the institution that I was working for at the time, right when they were
being audited!  Yikes!  Anyway, we were able to recover the data and life
returned to normal but I did wonder at the time whether I could still keep
my job there.

(3) At the same institution, we were running a system software that had a
serious bug where if anyone had logged out ungracefully, the system wouldn't
let any more users onto the system and users who were logged on couldn't
execute any new commands.  (The newest release of the software later on did
fix this bug.) I had to reboot the machine to restore the system to a sane
state.  I did a wall <<EOF We need to shutdown blah blah... EOF and then
shutdown.  Well, I should've waited since at the precise moment, one of our
users was doing a once-a-year massive conversion of our financial data (talk
about bad luck).  I had shutdown in the middle of a very long disk write and
thus, data was lost.  We did recover that data and life went on.  Moral:
make damn sure that *no one* is doing anything on your system before you
reboot, even if other users are vociferously clamoring for you to reboot.

(4) I heard this from a fellow sysadmin friend.  My friend was forced to
work with some sysadmins who didn't have their act together.  One day, one
of them was "cleaning" the filesytem and saw a file called "vmunix" in /.
"Hmm, this is taking up a lot of space - let's delete it".  "rm /vmunix".

My friend had to reinstall the entire OS on that machine after his coworker
did this "cleanup".  Ahh, the hazards of working with sysadmins who really
shouldn't be sysadmins in the first place.


Moral of all these stories:  if I had to hire a Unix sysadmin, the first
thing I'd look for is experience.  NOTHING can substitute for down-to-earth,
real-life grungy experience in this field.

--
hirai@cc.swarthmore.edu (Eiji Hirai)    :     :    :   :  : :: ::: :::: :::::
Unix Geek for Swarthmore College        :     :    :   :  : :: ::: :::: :::::
Information Services, Swarthmore, PA, US.       Copyright 1992 by Eiji Hirai.
I don't speak for Swarthmore College.                    All Rights Reserved.
Xref: sparky alt.folklore.computers:15073 comp.unix.admin:5563