tcp friends

TCP/IP loopback socket pair stack bypass, based on an idea by, and
rough upstream patch from, David Miller <davem@davemloft.net> called
"friends", the data structure modifcations and connection scheme are
reused with extensive data-path changes.

A new sysctl, net.ipv4.tcp_friends, is added:
  0: disable friends and use the stock data path.
  1: enable friends and bypass the stack data path, the default.

Note, when friends is enabled any loopback interpose, e.g. tcpdump,
will only see the TCP/IP packets during connection establishment and
finish, all data bypasses the stack and instead is delivered to the
destination socket directly.

Testing done on a 4 socket 2.2GHz "Quad-Core AMD Opteron(tm) Processor
8354 CPU" based system, netperf results for a single connection show
increased TCP_STREAM throughput, increased TCP_RR and TCP_CRR transaction
rate for most message sizes vs baseline and comparable to AF_UNIX.

Significant increase (up to 5x) in aggregate throughput for multiple
netperf runs (STREAM 32KB I/O x N) is seen.

Some results:

Default netperf: netperf
                 netperf -t STREAM_STREAM
                 netperf -t STREAM_STREAM -- -s 51882 -m 16384 -M 87380
                 netperf

         Baseline  AF_UNIX      AF_UNIX           Friends
         Mbits/S   Mbits/S      Mbits/S           Mbits/S
           6860       714   8%    9444 138% 1323%  10576 154% 1481% 112%

Note, for the AF_UNIX (STREAM_STREAM) test 2 results are listed, 1st
with no options but as the defaults for AF_UNIX sockets are much lower
performaning a 2nd set of runs with a socket buffer size and send/recv
buffer sizes equivalent to AF_INET (TCP_STREAM) are done.

Note, all subsequent AF_UNIX (STREAM_STREAM, STREAM_RR) tests are done
with "-s 51882" such that the same total effective socket buffering is
used as for the AF_INET runs defaults (16384+NNNNN/2).

STREAM 32KB I/O x N: netperf -l 100 -t TCP_STREAM -- -m 32K -M 32K
                     netperf -l 100 -t STREAM_STREAM -- -s 51882 -m 32K -M 32K
                     netperf -l 100 -t TCP_STREAM -- -m 32K -M 32K

          Baseline  AF_UNIX      Friends
   N  COC Mbits/S   Mbits/S      Mbits/S
   1   -    8616      9416 109%   11116 129% 118%
   2   -   15419     17076 111%   20267 131% 119%
  16   2   59497    303029 509%  347349 584% 115%
  32   4   54223    273637 505%  272891 503% 100%
 256  32   58244     85476 147%  273696 470% 320%
 512  64   58745     87402 149%  260837 444% 298%
1600 200   83161    158915 191%  383947 462% 242%

COC = Cpu Over Commit ratio (16 core platform)

Stream: netperf -l 100 -t TCP_STREAM
        netperf -l 100 -t STREAM_STREAM -- -s 51882
        netperf -l 100 -t TCP_STREAM

netperf  Baseline  AF_UNIX      Friends
-m/-M N  Mbits/S   Mbits/S      Mbits/S
  64       1020       445  44%     515  50% 116%
  1K       4881      4340  89%    5070 104% 117%
  8K       5933      8387 141%    9770 165% 116%
 32K       8168      9538 117%   11067 135% 116%
 64K       9116      9774 107%   11515 126% 118%
128K       9053     10044 111%   13082 145% 130%
256K       9642     10351 107%   13470 140% 130%
512K      10050     10142 101%   13327 133% 131%
  1M       8640      9843 114%   12201 141% 124%
 16M       7179      9619 134%   11316 158% 118%

Rr: netperf -l 100 -t TCP_RR
    netperf -l 100 -t STREAM_RR -- -s 51882 -m 16384 -M 87380
    netperf -l 100 -t TCP_RR

netperf  Baseline  AF_UNIX      Friends
-r N,N   Trans./S  Trans./S     Trans./S
  64      47913     99681 208%   98225 205%  99%
  1K      44045     92327 210%   91608 208%  99%
  8K      26732     33201 124%   33025 124%  99%
 32K      10903     11972 110%   13574 124% 113%
 64K       7113      6718  94%    7176 101% 107%
128K       4191      3431  82%    3695  88% 108%
256K       2324      1937  83%    2147  92% 111%
512K        958      1056 110%    1202 125% 114%
  1M        404       508 126%     497 123%  98%
 16M       26.1      34.1 131%    32.9 126%  96%

Crr: netperf -l 100 -t TCP_CRR
     netperf -l 100 -t TCP_CRR

netperf  Baseline  AF_UNIX      Friends
  -r N   Trans./S  Trans./S     Trans./S
  64      14690         -        18191 124%   -
  1K      14258         -        17492 123%   -
  8K      11535         -        14012 121%   -
 32K       7035         -         8974 128%   -
 64K       4312         -         5654 131%   -
128K       2252         -         3179 141%   -
256K       1237         -         2008 162%   -
512K       17.5*        -         1079   ?    -
  1M       4.93*        -          458   ?    -
 16M       8.29*        -         32.5   ?    -

Note, "-" denotes test not supported for transport.
      "*" denotes test results reported without statistical confidence.
      "?" denotes results not comparable.

SPLICE 32KB I/O:

Source
 Sink   Baseline  Friends
 FSFS   Mbits/S   Mbits/S
 ----     8042     10810 134%
 Z---     7071      9773 138%
 --N-     8039     10820 135%
 Z-N-     7902      9796 124%
 -S--    17326     37496 216%
 ZS--     9008      9573 106%
 -SN-    16154     36269 225%
 ZSN-     9531      9640 101%
 ---S     8466      9933 117%
 Z--S     8000      9453 118%
 --NS    12783     11379  89%
 Z-NS    11055      9489  86%
 -S-S    12741     24380 191%
 ZS-S     8097     10242 126%
 -SNS    16657     30954 186%
 ZSNS    12108     12763 105%

Note, "Z" source File /dev/zero, "-" source user memory
      "N" sink File /dev/null, "-" sink user memory
      "S" Splice on, "-" Splice off

Note from Tao Ma:
1. rebase to 2.6.32-358.14.1
2. fix a bug when syncookies are used.

Signed-off-by: Bruce \"Brutus\" Curtis <brutus@google.com>
Signed-off-by: Li Yu <bingtian.ly@taobao.com>
Signed-off-by: Zhu Yanhai <gaoyang.zyh@taobao.com>
Signed-off-by: Tao Ma <boyu.mt@taobao.com>