From: rrsum@matt.ksu.ksu.edu (Rick Summerhill)
Newsgroups: comp.dcom.sys.cisco
Subject: Weird Ethernet Behavior
Date: 30 Sep 92 12:09:15 GMT
Article-I.D.: matt.1ad8ibINNdmo
Organization: Kansas State University
NNTP-Posting-Host: matt.ksu.ksu.edu

We recently had a problem with one of our main Unix hosts which was
Network related.  It was of a sufficiently bazaar nature that I am
posting this to several groups to see if this kind of thing has been
seen before, or if someone can explain the details of the network
problem.

Here is the basic scenario: We have a central Cisco AGS+ router with
24 Ethernet ports all of which have IP enabled.  Each AUI port on the
back is directly connected to a fiber optic transceiver that uses
FOIRL and connects different buildings on our campus through fiber.

In building A we have a bunch of Sun workstations and in building B we
have a main Unix server the exports filesystems using NFS.  The clients
in B mount these filesystems, and the main Unix guy is very busy - lots of
users and lots of NFS traffic.  It has been stable for over a year, and
really has performed better than we ever expected.

Three weeks ago the server became unstable.  It's load would become
extremely large and it would have to be rebooted.  The manufacturer
looked at the core files we dumped and determined that there was
suspicious nfs behavior.  They sent us a replacement kernel file that
recorded the suspicious events, and sure enough, the clients in
building B (and only in B) were asking the server for absurd amounts
of kernel resources, enough to push all processes out to disk and
force everything into disk wait!  Not good!

We then turned on udp checksums on both the clients in building B and
the server in building A, and then we banged real hard on the nfs
daemons on the server.  Sure enough, netstat on the server would
report checksum errors at the rate of about one every minute, so
something in between the server and the client was mangling packets.
But Ethernets do checksums also, so the mangling was being done in a
very careful manner!

We then decided to replace some of the dumb parts in between the
server and the clients, starting with the transceiver sitting directly
on the router and connected to the fiber running to building B.  We
did this replacement at the same time we were banging really hard on
the server, and lo and behold, the errors stopped and have not
recurred.  So, that sort of implies that it was the transceiver, or a
flaky "direct" connection between the transceiver and the router.
After about 6 hours, we decided to put the "bad" transceiver back in
place to try to regenerate the problem.  Well, much to our surprise,
we are not seeing any errors, and have not for the past 24 hours.

The implications are a bit scary.  This means that a worker brushing
against the router might cause a connection to become flaky and cause
a distant Unix machine to all of a sudden become unstable!

So, what is the problem?  Well, the Unix box, of course, should probably
not rely so much on network integrity, but more importantly, what
network integrity has been compromised?  How can a packet get munged
by a bad connection and still satisfy the CRC checks associated with
Ethernet?  Has anyone ever seen this type of behavior before?  Does
the problem lie with the Ethernet port on the router?  It could be that
the transceiver really is bad and simply will not show its true colors for
the next couple of days, but even if the transceiver is bad, how can this
type of activity occur?

--Rick

-- 
Rick Summerhill                          Internet:  rrsum@hermzel.ksu.ksu.edu
CNS, Cardwell Hall                       Bitnet:    rrsum@ksuvm
Kansas State University                  Phone:     (913)532-6311
Manhattan, KS 66506                      FAX:       (913)532-5914