From: jbvb@vax.ftp.com (James B. VanBokkelen)
Newsgroups: comp.protocols.nfs
Subject: Re: NFS packet size
Keywords: TCP NFS
Date: 4 Jan 93 17:33:26 GMT
Reply-To: jbvb@ftp.com
Organization: FTP Software, Inc., Wakefield, MA
Nntp-Software: PC/TCP NNTP

In article <ubeuj2s@rhyolite.wpd.sgi.com> vjs@rhyolite.wpd.sgi.com (Vernon Schryver) writes:

    In article <1993Jan1.115015.17311@prl.dec.com>, boyd@prl.dec.com (Boyd Roberts) writes:
    > ...
    > > A dropped TCP fragment can result in a large amount of resending too.
    > 
    > True, but that all happens at the TCP/IP level which is far more
    > responsive than NFS's 1, 2, 4, 8 ... _second_ exponential backoff
    > when a datagram gets lost.
    
    In other implementations of NFS, the NFS timeout starts at 0.7 seconds
    not 1.0, and can be reduced to 0.1 seconds.  Please compare 0.1 seconds
    with the standard TCP timeouts.

In a good TCP (or a good UDP NFS), retransmit timeouts are adaptive.
Whether it's TCP or NFS over UDP, what you want for a floor on the
adaptive algorithm depends on your timer granularity, but it *really*
ought to be larger than the average Round Trip Time plus the standard
deviation of the RTT.

In most UDP NFS traffic I've watched, the standard deviation of the
server's response time is large, a significant fraction of a second.
Presumably this is due to disk contention and cache hit/miss/invalidation.
What this implies is that you'll benefit from having your network RTT
decoupled from the disk, which TCP provides.  As long as TCP is happy
with the connection, there's no reason to retry the RPC...

    > We had a flakey drop cable here a while back.  When it was bad it would
    > take in the order of a minute to write an 8K block.  You could see the
    > fragments being dropped and the NFS retries.  I _never_ want to see
    > that again -- it was _awful_.
    
    It could have been as bad or worse if you had been using NFS or TCP.
    Your TCP connections would have been timing-out and being recreated,
    throwing away any successfully moved data, and forcing the systems to
    attempt the extra packet exchanges that are required to start a TCP
    connection, incurring even more grief.

If you're actually losing enough packets to make a good TCP drop a
connection due to timeouts (25%+), you won't get any UDP NFS I've ever
seen to work at all.  If your TCP drops connections with 10% packet loss,
you've got one or more of the 4bsd "escalating retransmit timeout" bugs,
and should get it fixed.

    With large (e.g. 250KByte) or even just full sized (e.g. 60KB) TCP
    windows, TCP can involve the retransmission of far more than an 8KB UDP
    packet.  The advantage with TCP is that its "go back N" retransmission
    scheme only goes back to the first lost TCP segment, not the to start
    of the NFS request.  You can hope to continually make progress, to only
    be thrown back to the start of the effort when things get really bad.

A good TCP buffers most or all of the ahead-of-sequence segments, so at
best you only need to retransmit the one that was lost.  Even if you can
control the IP ID on an outgoing UDP NFS packet, you still need to re-send
the whole thing, because as you say you can't tell what's missing.  The
issue is discussed in detail in Kent & Mogul's "Fragmentation Considered
Harmful" (ACM Computer Communications Review Vol. 17 #5).

So, what is "a good TCP"?  One derived from 4.3 tahoe or later is fairly
good, but it may still mis-handle ICMP errors, or escalate its retransmit
timeout in the face of packet loss.  I don't know of any objective tests,
so reading RFC 793 and watching it perform in the face of adversity is
still the best way to find out.

James B. VanBokkelen		2 High St., North Andover, MA  01845
FTP Software Inc.		voice: (508) 685-4000  fax: (508) 794-4488