From: eps@siivu.oulu.fi (Erkka Sutinen)
Subject: Re: WANTED: Unix administration horror stories !
Reply-To: eps@rieska.oulu.fi
Organization: University of Oulu
Date: Thu, 15 Oct 1992 18:37:47 GMT

In article <1992Oct12.080944.23519@jet.uk> djs@jet.uk (David J Stevenson)  
writes:
> When this happened to a colleague (when I worked somewhere else) he restored
> vmunix by copying from another machine.  Unfortunately, a 68000 kernel does
> not run very well on a Sparc...

Uh. This reminds me of an long and hard day couple of weeks ago.....

I was walking merrily towards my room while world was beautiful and
sun was shining. I got the first hint that something may be wrong when I saw
our sysadmin banging his head to an nearby door. I asked if something
might be wrong, and he told me so while continuing banging. He had made
backups of our old faithful, tolsun (sun3/160) and made an minor typo.
Instead of tar cf /dev/rst0 . he had written tar xf /dev/rst0 .
(Scripts?? we don't use any bleeding scripts!! They restrict creativity
and make improvising impossible! ) Oh well. And the tape happened to be
an old backup from an sparc.

So. No binaries worked, execpt that one could login as box. inetd had
crashed (at which point it did is irrelevant.) There was no active
sessions for that system and there was no way to get in.....

Lets take a break, have some coffee and think it over: There was a
lighter side: all of the disks were mounted to another system via nfs,
and that daemon was still working. List of the files overwritten was on
an log file, and there wasn'y very many of them. backups were on 8mm tapes
and our only 8mm drive was on our server, but with nfs, that wasn't problem.
On the other hand we had lost /usr/bin, /usr/etc,
/usr/kvm and /ucb . Ugh! I think I don't like this. 

Fine. Lets take everything back from tape.... wait a minute... We had 
just installed new operating system, which had taken several
days since additional upgrade hadn't worked due to lack of disk space
and we had an rather unorthodox system running, which worked fine
with our add-ons, such as appletalk box...  And we hadn't make backups
of this new system yet... Aww shit. We couldn't afford to lose this system,
it had too much sweat already in it. 

Fortunately we had all the binaries of the upgraded system on our servers
disks, and with nfs we copied everything to the faulty system. Blessed
is the network, who rules our lives! 
Nothing happened. Nothing worked. Ah! We just have to restart inetd.
Hmmmm. but how. Do not worry, this is Unix, full of possibilities! First
cron, but the clock of that system was totally out of this world showing
probably the time of Ouagadougou and we didn't know where was that! Ah! 
making some false login attempts will bring error messages to the console.
Nothing happened... I guess syslog had died too.... When daemons are in
agony, no program can stand without feeling some sympathy towards those who
suffer so.... 

Oh no... But of course. Removing roots password.... Why are you brain cells
sometimes so slow.... This made it.... But still..... Nothing worked...
Ah, we had the upgraded binaries... Hm. Ahh. the sun's unupgrade option
was used while upgrading the system, so we just reunupgraded the system.
No problem. Everything is fine again... ??  Why doesn't that system
work?? Why our server doesn't let anyone in... Awwww...Jesus Christ.
What had I done!!! Unupgraded our SparcServer with sun3's binaries....
Awww shit and <add your favourite curses here. Mine is sys V curses>.

At this point I was trying to find an good way to get rid of all of our
computers.... Would anyone notice if I just dumped our systems to river
and claimed that we have never had any computers.... No hope. Back to work.

And at the same moment, the cron miraculously worked, and we had two inetd's
running. No harm done. It is just a nuicance.

No problem. Don't panic. Both of the systems are up and running....
Well aware of the fact that if there are any more mistakes, the systems
will crash and they will not come up. We didn't even have our server's
operating system handy, (one disk in whole unversity... ) I took the
most recent backup of our server and carefully extracted the rpc.passwdauth
and couple of other files from the tape, and lo ... everything worked again.

The reunupgrade procedure was made at this time at the right computer,
and at our suprise it started to work again.... The world is a miraculous
place......While ps again operational, we checked the daemons still
running.... nfsd and getty were only systems that had stayed up in addition
to the usual bunch (init etc...)

At this point we thought everything was fine... Lets make the backup now
and take a good care not to touch anything..... It worked....

Couple of days later I noticed that the restoring of our servers daemons
had gone to an wrong directory and the mail hadn't been running for
two days... No problem... Execpt that delivering two days worth of mail
brought our server to its knees... Fortunately it didn't crash but it
was ssslllllooooowwwwww for couple of hours....

.....and once again our heroes had defeated evil nazis and were heading
home for meal.......

Moral of the story: DON'T PANIC . There are several ways to handle unix
				as long as it is running. Do NOT try to reboot
				it. It only brings sorrow with it.
Moral of the story 2: Douple-check every command you give. If one of your
			commands has an error, it is the one which causes
			biggest damage.
Moral of the story 3: Check the machine you are in, before you give any even
			possibly destructive commands... Networks can be
			a real nuicance you know.....

--
======================================#=======================================
Erkka Pietari Sutinen                 #Eke what availeth Maner and Gentlinesse
eps@rieska.oulu.fi / sutinen@csc.fi   # Without yow, benygne creature?
University of Oulu. Finland           # Shal Cruelte be your governesse?
Dep. of Information Prosessing Science#                      -Chaucher