From: spurgeon@ccwf.cc.utexas.edu (Charles Spurgeon)
Newsgroups: comp.dcom.sys.cisco
Subject: Re: Weird Ethernet Behavior
Date: 12 Oct 92 16:31:32 GMT
Organization: The University of Texas at Austin, Austin TX

In article <Oct.5.18.01.25.1992.7644@geneva.rutgers.edu> hedrick@geneva.rutgers.edu (Charles Hedrick) writes:
>So if for one reason or another you're getting corrupted packets,
>you'll see IP checksum failures and you won't see IPX checksum
>failures.  Now why you're getting corrupted packets is a question I
>can't answer.  Perhaps it indicates that Ethernet CRC is not being
>properly checked.  Perhaps there is a hardware problem that is
>corrupting the packet after the CRC has been checked (i.e. bad memory,
>in either the interface or the main box), or perhaps it means that
>something in one of your hosts is corrupting the packet before the
>Ethernet CRC is being added.  The only point I'm trying to make is
>that the fact that you're seeing checksum failures for IP and not for
>IPX does not indicate that Ethernet CRC's are being handled
>differently for the two protocols.  

There's a known bug in pre 1.9 versions of MCI microcode that causes
the Cisco to trash the last two bytes of a packet.  This bug seems to
be related to a series of edge-of-the-envelope bugs that MCI Ethernet
interfaces exhibit when presented with Ethernet frames that have
dribble bits.

Dribble bits are a condition on Ethernet where a frame picks up extra
"bits" on the end of the frame perhaps due to transmitting circuits
not squelching their output quite fast enough.  The Ethernet spec
notes that you deal with this by sensing end of frame, truncating back
to the last octet boundary, checking the CRC, and if it's all good,
accepting the frame.  No problem.  No problem that is except for buggy
Cisco MCIs.

We've been running into these bugs dating from the first Cisco
installations on campus in 1990.  As I recall, in MCI microcode V1.5
the response was to just drop frames with dribble bits.  For us this
meant that certain pairs of broadband Ethernet modems on campus could
communicate through routers, and certain others couldn't depending on
whether or not they emitted frames with dribble bits.  Before the
Ciscos were installed we had been using DEC LAN Bridge 100s and had
never seen this problem.

In V1.7 MCI microcode the frames are accepted (and counted as a frame
error in the interface stats), but the interface has a hang condition
which I recall being told is caused when back-to-back frames (like
NFS) show up and the interface has just dealt with a collision event.
The interface freezes and comes back after a minute or so.  These
events are marked as "line down restarts" in the MCI controller stats
("show cont mci").  Cisco's release notes state that this happens on
fiber or broadband Ethernet interfaces, but I found it happening on
plain old coax interfaces as well.

In 1.8 MCI microcode the restart bug is fixed, but the code still has
a bug where every 20-50 thousand packets or so in our testing the
interface will garble the last two bytes of data in the frame if the
frame is received in a sequence of back-to-back frames.  Apparently
the back-to-back frames have to happen in conjunction with dribble bit
frames.  This can happen on even relatively low traffic nets as long
as there is a rapid sequence of frames.  The back-to-back sort of
frames caused by NFS or other protocols trying to emulate disk IO are
apparently what trigger the bug.  This bug may be in earlier MCI code
as well - I'm not sure since we had upgraded every interface on campus
to 1.8 and didn't have any older code to check.

In my experience you can get frames with dribble bits from some
broadband Ethernet modems (which is where we first saw the
phenomenon), some thick-to-thin Ethernet repeaters, certain PC
interfaces, RS6000 interfaces etc.  I've also heard reports of dribble
bits and 10BASE-T hubs.  In one case on campus NFS traffic worked for
a long time, then an RS6000 was installed on the LAN and the failures
began.

When we had a problem with Cisco interfaces and dribble bits we were
able to install an old DEC LAN Bridge 100 between the network and the
Cisco interface and the problem would go away.  In other words, the
LB100 was not bothered by the dribble bit conditions that caused the
Cisco to fail or corrupt frames.  I used to keep an LB100 in my office
and would lug it around campus like an extra large amulet used to ward
off evil dribble bit Cisco failures.  :-)

Note that in the V1.8 code when the Cisco interface corrupts the last
two bytes of data of the received frame it then transmits the
corrupted frame with a new Ethernet CRC that is correct for the
corrupted data.  The host will receive the Ethernet frame with a good
CRC and then hand the corrupted data up to the high level apps.
Applications like NFS and some DECnet-based thingies die on this since
they do not use higher-level checksums for the data and assume that
the Ethernet is delivering correct data.  A bad assumption, as this
case makes clear.

Upgrading the MCI, as we have had to do repeatedly at this site, is a
tedious process.  It requires that you take the router out of service,
pull the board out, pull as many as 13 PROMs out of the MCI and
install new ones, and then reinstall the router.  Doing this
repeatedly for routers scattered all over a large campus is, as you
might expect, quite a task.  The worst case is the CGS router which
has no access to the boards other than taking the router completely
apart.



-- 

Charles Spurgeon		Computation Center Networking Services
c.spurgeon@utexas.edu		University of Texas at Austin
(512) 471-3241 ex 265		Room COM1, Austin TX 78712
-----------------------------------------------------------------------
There's more to networking than just hooking things up.
Newsgroups: comp.dcom.sys.cisco