ckolivas/cgminer — pull request #88 by midnightmagic (2012-01-22 09:28:24+00:00)
Title: Bring back TCP_NODELAY

Nagle's may infact delay some packets longer than necessary..

Okie doke. Data:

/sbin/tc qdisc add dev eth0 root netem delay 200"msec"

Measured solution delivery tcpdump *with* TCP_NODELAY enabled:
tcpdump -w nodelay_200 -s 2049 -lni eth0 port 31337
(then)
tcpdump -s 2049 -vxX -r nodelay_200 -n | less

21:33:47.520616 (POST)
21:33:47.520628 (getwork) [0.012 ms]

21:35:22.647470 (POST)
21:35:22.647482 (getwork) [0.012 ms]

21:36:59.462853 (POST)
21:36:59.462866 (getwork) [0.013 ms]
[etc, low variance, since this is a local gbE network]

Measured solution delivery tcpdump TCP_NODELAY DISabled:
tcpdump -w no_nodelay_200 -s 2049 -i eth0 port 31337
(then)
tcpdump -s 2049 -vxX -r no_nodelay_200 -n | less

22:11:48.165626 (POST)
22:11:48.166076 (ack)
22:11:48.366110 (getwork) [200.484 ms]

22:14:12.579990 (POST)
22:14:12.580478 (ack)
22:14:12.780514 (getwork) [200.524 ms]

22:15:12.510339 (POST)
22:15:12.511131 (ack)
22:15:12.711168 (getwork) [200.829 ms]
[etc, low variance, since this is a local gbE network]

**Summary**

The request of CURL appears to be generating two writes; they are therefore broken into two packets. Nagle's appears to be forcing the second packet (with the getwork, read: share solution) to wait for the ACK of the first packet from the remote end. NOTE: Nagle's usually won't split up small single write syscalls into multiple packets. At least I've never seen it do that. Therefore, even after the overhead of a TCP handshake, we are additionally forced to wait for at least an RTT * 1.5 before the solution is actually delivered to the remote end.

With systems like P2Pool (and especially for remote miners) 1.5 * RTT can literally mean the difference on average, of a share delivered in time to count, and a stale. With the sharechain in P2Pool and a mere 50 ms one-way latency, that's 100 ms of unnecessary wait that could be eliminated.

Err.. assuming it's not too late for me to think clearly about it. In any event, TCP_NODELAY does appear to deliver the solution much faster than without it, and in my tcpdump tests I see no additional packets that are sent: only delayed ones.

P.S. I rebased the branch behind this pull and squashed the earlier commits.. in case that matters.