From: rpw3@rigden.wpd.sgi.com (Rob Warnock)
Subject: Re: Advice needed on Bridges/Routers
Organization: Silicon Graphics, Inc.  Mountain View, CA
Date: Sat, 29 Aug 1992 09:56:22 GMT

Note: Everything I say below about Ethernet applies to 802.5 Token Ring
and FDDI as well -- the problem I discuss is *not* an "Ethernet problem".
All three do a good job of bandwidth-sharing *without* dropping packets
under heavy load -- the sharing is accomplished in all three cases by
*delaying* access to the medium. Religious arguments about one versus the
other are only over the amount of the delay as a function of load, which is
irrelevant to what I'm about to say. [I had this in a "p.s.", then realized
it needed saying up front to head off possible misunderstandings.]  --rpw3

seifert@netcom.com (Rich Seifert) writes:
+---------------
| randy@ninja.megatek.uucp (Randy Davis) writes:
| >   I would like to solicit advice concerning the best way to split the
| > network and why.
| 
| 1: For sure, when you segment your network, do it according to logical
| connectivity. Put the workstations on the same segment as the server
| with which most of the traffic is directed. This will ease the 
| internetworking burden of any device you choose to use. A good
| rule of thumb is to segment such that at least 80% of the network 
| traffic is local (if possible).
+---------------

I'll be even more emphatic: THE TRAFFIC BETWEEN PARTITIONED SEGMENTS HAD
BETTER BE *REALLY* LOW, OR YOU CAN MAKE YOUR PROBLEMS *WORSE* BY SPLITTING!

Once you split your net into separate "collision domains" (whether you do
it with bridges or routers matters not), you have lost the automatic nearly-
fair bandwidth sharing that comes with Ethernet's CSMA/CD. After the split,
the bridge or router port on each leg only gets *one* share of the net!
(After all, it's just another single station.) As a result, in many cases
it is quite possible to *increase* congestion -- and packet loss! -- by
splitting your net.

A first-order heuristic: If a fraction "X" of the traffic is non-local, you'd
better not have more than "1/X" stations on the local net, *including* the
bridge/router. To use Rich's example, if 80% of the network traffic is local,
then 20% (0.2) is non-local, and you'd better not have more than 1/0.2 = 5
stations on the net, which is a maximum of 4 hosts plus the bridge/router.
If 90% of the traffic is local, you can have up to 9 hosts (plus the brouter).

If you violate this rule, you will experience high delays through the
bridges/routers, and quite possibly sharply increased packet loss rates
as the output queues in the bridges/routers overflow (some bridges have
amazingly small output queues). What's worse, if you run any high-performance
TCP connections across such paths, the "slow-start" congestion avoidance
algorithm in TCP will *create* packet loss, by opening the TCP send window
just to the point that a packet is lost on *that* connection, then decreasing
the window, and then opening it again, etc.

This is not so bad if all your traffic across the bridges/routers is TCP,
since the "slow-start" code will effectively throttle all of them at once,
causing them to share the available bandwidth. [There is good experimental
evidence of this, namely, "The Internet"!] But if you have mixed protocols
on your LAN (and who doesn't?), the TCPs will cause losses on those *other*
protocols, too. And it's a sad fact of life that many "request/response"
protocols designed for a single local net (e.g., NFS) don't tolerate packet
loss *AT ALL*! They assume an essentially error-free (though not delay-free)
net. Throughputs for these protocols will *collapse* if regular losses occur.

The worst scenario is when your original unified net has a *lot* of hosts
that talked more or less equally to each other. In that case, it is almost
impossible to partition the net successfully, unless you put every host on
its own subnet or bridge segment.

The good news: There are some improved congestion-avoidance algorithms being
worked on for TCP (e.g., the work by Crowcroft, et al.) that attempt to stay
below the point of packet loss (e.g., by measuring increases in delay and
assuming they're due to queues filling up in the bridges/routers).

The bad news: Nearly nobody's production TCP has these improvements in them,
and *nobody's* production non-TCP protocols have even made a start in this
direction. Most don't even have "slow-start". [And the congestion/collapse can
occur even if you have no TCP on the net at all. Adding the TCP "slow-start"
just makes it more obvious.]

These problems are not new. The industry faced them (and solved only *some*
of them) back in the days when there weren't any fast "broadcast" local-area
networks, only hosts and switches connected by medium-to-slow point-to-point
links. Since then, most of the lessons about congestion have been forgotten
(except on the Internet!). LAN users managed to ignore congestion issues
as long as individual hosts could consume only a small fraction of the LAN's
bandwidth. But now nearly *any* high-performance workstation can consume 100%
of an Ethernet with useful traffic, and congestion is once again rearing it's
head. [FDDI is no solution per se: Single-connection TCP rates of 30-70% of
the FDDI rate are now common from various workstation manufacturers.]

SUMMARY
A high average load on an Ethernet [or TR or FDDI] is not necessarily a
sufficient reason to partition a net. Ethernet's CSMA/CD [or token passing]
is actually a very good load-sharing mechanism, and hardly ever causes packets
to be *dropped*, only delayed. Partition your net only if you have confidence
(from measurement or calculation) that such paritioning will not *create*
congestion in the bridges/routers, with attendant packet loss and throughput
collapse in your "LAN protocols".


-Rob

-----
Rob Warnock, MS-9U/510		rpw3@sgi.com
Silicon Graphics, Inc.		(415)390-1673
2011 N. Shoreline Blvd.
Mountain View, CA  94043