From: ken@cs.cornell.edu (Ken Birman)
Subject: A TCP bug in most SVR4 compatible UNIX systems; a work-around
Organization: Cornell Univ. CS Dept, Ithaca NY 14853
Date: Wed, 18 Nov 1992 16:09:45 GMT

We have recently run into a whole slew of performance problems with
vendor releases of TCP protocols that 
	1) Support a setsocketopt option TCP_NODELAY
	2) Default to have TCP_NODELAY 0 ("delay")
These TCP implementations have the annoying property that under
many conditions, TCP sends are delayed to attempt to buffer 
multiple sends into one packet.  Worse, on some machines (SWBTL
ran into this on a TANDEM under their version of UNIX), the sender
gets delayed too, which seems really strange.

Either way, performance is severely affected.  For example, the
TANDEM RPC time to protos was measured at 250ms whenever a packet
had to be transmitted in two or more fragments (depends on how Isis
represents the data), while it was in the 6ms range normally!  For
SUN OS you would see something similar in the same situation.

There is a very simple fix for this problem.  When opening a TCP
connection (after the connect or accept call) you do:

#ifdef IPPROTO_TCP
#    ifdef TCP_NODELAY
     {
	   int enable = 1;
	   setsocketopt(fdes, IPPROTO_TCP, TCP_NODELAY, &enable, sizeof(enable));
     }
#    endif
#endif

The ifdef's are to make your code more portable: on a UNIX that doesn't
define IPPROTO_TCP or support option TCP_NODELAY, nothing is compiled.
(If the options are defined and not working, all bets are off -- but
I doubt that this is a problem on the major vendor platforms).

In ISIS itself, we are adding this to the standard release as of the
patch-11 tape for V3.0.7, which is being made up today (you get it
if you ask for it and have support, or if you purchase the system after
today).  Same for V2.2.7.  We will set this option:

    - In TCP connections made using isis_connect
    - When using TCP to connect from protos to a remote client.

If you need to edit this into the obscured source yourself, you can
do so as follows:

1) in cl_isis.c, after a label "Ba004c8:" around line 497; e.g.:

Ba004c8:
#ifdef IPPROTO_TCP
#    ifdef TCP_NODELAY
     {
           int enable = 1;
           setsocketopt(He08aa6e, IPPROTO_TCP, TCP_NODELAY, &enable, sizeof(enable));
     }
#    endif
#endif

2) in pr_client.c, after the following line (218 in my code):
L60509d26 = D60b66334(R3c9965b2, (D315a3970 P3497e080 *)&B1481a1d2, (int*)&J934091e);
#ifdef IPPROTO_TCP
#    ifdef TCP_NODELAY
     {
           int enable = 1;
           setsocketopt(L60509d26, IPPROTO_TCP, TCP_NODELAY, &enable, sizeof(enable));
     }
#    endif
#endif

In the case of isis_connect, you can just do this in your own code
after getting the file descriptor back.

Again, I have no idea how common this problem is.  We have seen it on
TANDEM systems and on SUN OS systems -- but not always on the latter.
Indeed, we are unclear just what needs to be true to trigger it on SUN
platforms.  Perhaps the option isn't initialized in a consistent way
within one of the SUN releases of TCP.  Anyhow, explicitly enabling
NODELAY mode is the right thing to do when latency might matter on a 
TCP connection.

-- 
Kenneth P. Birman                              E-mail:  ken@cs.cornell.edu
4105 Upson Hall, Dept. of Computer Science     TEL:     607 255-9199 (office)
Cornell University Ithaca, NY 14853 (USA)      FAX:     607 255-4428
Newsgroups: comp.sys.isis