Repository: tornadoweb/tornado

Issue:
Suboptimal performance sending large messages
I notice that when I send large messages over a TCP IOStream I get considerably less bandwidth than my network provides (about 10% of total).  A bit of profiling shows that a large fraction of time is spent in `str.join` and `str.find`.

Below follows a tiny example to replicate this on localhost.
### Server

``` python
from tornado.tcpserver import TCPServer
from tornado.ioloop import IOLoop
from tornado import gen

import numpy as np
data = bytes(np.random.randint(0, 255, dtype='u1', size=100000000).data) # 100M

sentinel = b'--my-sentinel--'

class MyServer(TCPServer):
    @gen.coroutine
    def handle_stream(self, stream, address):
        yield stream.write(data + sentinel)

s = MyServer()
s.listen(8000)

IOLoop.current().start()
```
### Client

``` python
from tornado.tcpclient import TCPClient
from tornado.ioloop import IOLoop
from tornado import gen

sentinel = b'--my-sentinel--'

@gen.coroutine
def f(): 
    client = TCPClient()
    stream = yield client.connect('localhost', 8000,
                                  max_buffer_size=int(1e9))
    msg = yield stream.read_until(sentinel)
    print(len(msg))

if __name__ == '__main__':
    IOLoop().run_sync(f)
```
### Add a bit of network delay

This adds 10ms on to localhost, just to simulate being in a real network

```
# tc qdisc add dev lo root netem delay 10ms
```
### Remove network delay when done

```
# tc qdisc del dev lo root netem
```
### Profiler output

```
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       92    0.430    0.005    0.430    0.005 {method 'find' of 'str' objects}
       46    0.249    0.005    0.249    0.005 {method 'join' of 'str' objects}
       25    0.216    0.009    0.216    0.009 {method 'poll' of 'select.epoll' objects}
     1553    0.019    0.000    0.019    0.000 {method 'recv' of '_socket.socket' objects}
       46    0.008    0.000    0.259    0.006 iostream.py:1501(_double_prefix)
     1553    0.002    0.000    0.022    0.000 iostream.py:721(_read_to_buffer)
       22    0.002    0.000    0.715    0.032 iostream.py:585(_read_to_buffer_loop)
       47    0.001    0.000    0.251    0.005 iostream.py:1510(_merge_prefix)
     1553    0.001    0.000    0.020    0.000 iostream.py:1010(read_from_fd)
        1    0.001    0.001    0.934    0.934 ioloop.py:746(start)
       48    0.001    0.000    0.690    0.014 iostream.py:772(_find_read_pos)
     6549    0.000    0.000    0.000    0.000 {len}

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.957    0.957 <string>:1(<module>)
        1    0.000    0.000    0.957    0.957 ioloop.py:400(run_sync)
        1    0.001    0.001    0.957    0.957 ioloop.py:746(start)
       29    0.000    0.000    0.757    0.026 stack_context.py:271(null_wrapper)
       23    0.000    0.000    0.756    0.033 iostream.py:497(_handle_events)
       22    0.000    0.000    0.755    0.034 iostream.py:645(_handle_read)
       23    0.002    0.000    0.755    0.033 iostream.py:585(_read_to_buffer_loop)
       49    0.001    0.000    0.727    0.015 iostream.py:772(_find_read_pos)
       94    0.454    0.005    0.454    0.005 {method 'find' of 'str' objects}
       47    0.008    0.000    0.272    0.006 iostream.py:1501(_double_prefix)
       48    0.002    0.000    0.264    0.005 iostream.py:1510(_merge_prefix)
       47    0.261    0.006    0.261    0.006 {method 'join' of 'str' objects}
       26    0.198    0.008    0.198    0.008 {method 'poll' of 'select.epoll' objects}
     1553    0.002    0.000    0.026    0.000 iostream.py:721(_read_to_buffer)
     1553    0.001    0.000    0.023    0.000 iostream.py:1010(read_from_fd)
     1553    0.022    0.000    0.022    0.000 {method 'recv' of '_socket.socket' objects}
        5    0.000    0.000    0.001    0.000 ioloop.py:594(_run_callback)
        3    0.000    0.000    0.001    0.000 client.py:7(f)
```



Discussion:
The cost is on the receiving side, not the sender. `read_until` is relatively inefficient for large messages. This could be improved (i.e. we don't need to put everything in one contiguous buffer; we just need to copy `len(delimiter)-1` bytes from the end of one packet to the beginning of the next), but hasn't been a priority because in general well-designed network protocols use length prefixes instead of static delimiters, and protocols that do rely on delimiters (i.e. http's `\r\n\r\n` for headers) do so only in a context in which you can set a reasonable `max_bytes` limit.  

I've run into other performance issues when using length prefixed frames
when processing very frequent messages.

Is the right way to do this something with two calls to IOStream.read_bytes
or is there something built in that I'm missing.

I'll cook up another example. Sometime tomorrow.
On Apr 7, 2016 7:45 PM, "Ben Darnell" notifications@github.com wrote:

> The cost is on the receiving side, not the sender. read_until is
> relatively inefficient for large messages. This could be improved (i.e. we
> don't need to put everything in one contiguous buffer; we just need to copy
> len(delimiter)-1 bytes from the end of one packet to the beginning of the
> next), but hasn't been a priority because in general well-designed network
> protocols use length prefixes instead of static delimiters, and protocols
> that do rely on delimiters (i.e. http's \r\n\r\n for headers) do so only
> in a context in which you can set a reasonable max_bytes limit.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> https://github.com/tornadoweb/tornado/issues/1685#issuecomment-207186134

Yes, in general the best approach is something like `stream.read_bytes(4)`, parse the integer, then `read_bytes(n)`. 

## Take two

### Server

``` python
from tornado.tcpserver import TCPServer
from tornado.ioloop import IOLoop
from tornado import gen
import struct

import numpy as np
data = bytes(np.random.randint(0, 255, dtype='u1', size=100000000).data) # 100M

class MyServer(TCPServer):
    @gen.coroutine
    def handle_stream(self, stream, address):
        yield stream.write(struct.pack('L', len(data)))
        yield stream.write(data)

s = MyServer()
s.listen(8000)

IOLoop.current().start()
```

### Client

``` python
from tornado.tcpclient import TCPClient
from tornado.ioloop import IOLoop
from tornado import gen
import struct

@gen.coroutine
def f():
    client = TCPClient()
    stream = yield client.connect('localhost', 8000, max_buffer_size=int(1e9))
    nbytes = yield stream.read_bytes(8)
    nbytes = struct.unpack('L', nbytes)[0]
    data = yield stream.read_bytes(nbytes)

if __name__ == '__main__':
    IOLoop().run_sync(f)
```

### Profile

This generally looks pretty good, we're moving data at 500MB/s, which seems actually faster than typical IPC limits.  Still, it's odd the amount to which `bytes.join` costs us.

```
         19170 function calls (19161 primitive calls) in 0.201 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.074    0.074    0.074    0.074 {method 'join' of 'bytes' objects}
     1529    0.064    0.000    0.064    0.000 {method 'recv' of '_socket.socket' objects}       4    0.044    0.011    0.044    0.011 {method 'poll' of 'select.epoll' objects}
     1529    0.004    0.000    0.071    0.000 iostream.py:721(_read_to_buffer)
        1    0.004    0.004    0.200    0.200 ioloop.py:746(start)
        4    0.002    0.001    0.074    0.019 iostream.py:585(_read_to_buffer_loop)
     1529    0.002    0.000    0.066    0.000 iostream.py:1010(read_from_fd)
        2    0.001    0.001    0.075    0.038 iostream.py:1510(_merge_prefix)
     6124    0.001    0.000    0.001    0.000 {built-in method len}
```

### Add a bit of latency into the network

```
tc qdisc add dev lo root netem delay 10ms
```

### Profile again

We're now down to 100 MB/s, which is well under IPC limits

```
         21410 function calls (21401 primitive calls) in 0.892 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      111    0.674    0.006    0.674    0.006 {method 'poll' of 'select.epoll' objects}
     1638    0.108    0.000    0.108    0.000 {method 'recv' of '_socket.socket' objects}       1    0.052    0.052    0.052    0.052 {method 'join' of 'bytes' objects}
     1638    0.014    0.000    0.130    0.000 iostream.py:721(_read_to_buffer)
        1    0.009    0.009    0.890    0.890 ioloop.py:746(start)
      109    0.008    0.000    0.140    0.001 iostream.py:585(_read_to_buffer_loop)
     1638    0.006    0.000    0.114    0.000 iostream.py:1010(read_from_fd)
        2    0.003    0.001    0.056    0.028 iostream.py:1510(_merge_prefix)
      108    0.002    0.000    0.200    0.002 iostream.py:497(_handle_events)
      113    0.002    0.000    0.002    0.000 {method 'update' of 'dict' objects}
      115    0.002    0.000    0.205    0.002 stack_context.py:271(null_wrapper)
     6140    0.002    0.000    0.002    0.000 {built-in method len}
     2182    0.001    0.000    0.001    0.000 iostream.py:478(closed)
     1533    0.001    0.000    0.001    0.000 iostream.py:754(_run_streaming_callback)
      108    0.001    0.000    0.001    0.000 iostream.py:474(writing)
     1531    0.001    0.000    0.001    0.000 {method 'append' of 'collections.deque' objects}220    0.001    0.000    0.001    0.000 iostream.py:772(_find_read_pos)
```

## Profiling `read_until` vs `read_bytes`

We find that using sentinels is _much_ faster for rapid back and forth of small messages.

### core_sentinel.py

``` python
from tornado import gen

sentinel = b'--my-sentinel--'

@gen.coroutine
def read(stream):
    msg = yield stream.read_until(sentinel)
    return msg[:-len(sentinel)]


@gen.coroutine
def write(stream, msg):
    yield stream.write(msg + sentinel)
```

### `core_len.py`

``` python
from tornado import gen
import struct

@gen.coroutine
def read(stream):
    nbytes = yield stream.read_bytes(8)
    nbytes = struct.unpack('L', nbytes)[0]
    data = yield stream.read_bytes(nbytes)
    return data


@gen.coroutine
def write(stream, msg):
    yield stream.write(struct.pack('L', len(msg)))
    yield stream.write(msg)
```

### Server

``` python
from tornado.tcpserver import TCPServer
from tornado.ioloop import IOLoop
from tornado import gen

# from core_sentinel import read, write
from core_len import read, write

class MyServer(TCPServer):
    @gen.coroutine
    def handle_stream(self, stream, address):
        while True:
            msg = yield read(stream)
            yield write(stream, msg)

s = MyServer()
s.listen(8000) 

IOLoop.current().start()
```

### Client

``` python
from tornado.tcpclient import TCPClient
from tornado.ioloop import IOLoop
from tornado import gen

# from core_sentinel import read, write
from core_len import read, write

@gen.coroutine
def f():
    client = TCPClient()
    stream = yield client.connect('localhost', 8000, max_buffer_size=int(1e9))
    for i in range(1000):
        yield write(stream, b'Hello!')
        response = yield read(stream)
        print(response)
        assert response == b'Hello!'

if __name__ == '__main__':
    IOLoop().run_sync(f)
```

### Results - `core_sentinel.py`

```
         225216 function calls (223157 primitive calls) in 0.346 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2978/1953    0.022    0.000    0.250    0.000 gen.py:990(run)
        1    0.019    0.019    0.344    0.344 ioloop.py:746(start)
     1000    0.017    0.000    0.017    0.000 {built-in method print}
     1000    0.015    0.000    0.015    0.000 {method 'send' of '_socket.socket' objects}    0.015    0.000    0.066    0.000 gen.py:1051(handle_yield)
    24985    0.013    0.000    0.021    0.000 {built-in method isinstance}
2002/2001    0.013    0.000    0.166    0.000 gen.py:257(wrapper)
     1975    0.011    0.000    0.011    0.000 {method 'recv' of '_socket.socket' objects}    0.010    0.000    0.010    0.000 {method 'poll' of 'select.epoll' objects}00    0.009    0.000    0.046    0.000 iostream.py:360(write)
     2002    0.008    0.000    0.061    0.000 gen.py:938(__init__)
     1975    0.008    0.000    0.035    0.000 iostream.py:585(_read_to_buffer_loop)  2931    0.007    0.000    0.306    0.000 stack_context.py:271(null_wrapper)
     1001    0.007    0.000    0.032    0.000 iostream.py:827(_handle_write)
     1954    0.007    0.000    0.010    0.000 ioloop.py:928(add_callback)
4005/4004    0.006    0.000    0.021    0.000 concurrent.py:318(_set_done)
     3910    0.006    0.000    0.011    0.000 stack_context.py:253(wrap)
     2002    0.006    0.000    0.190    0.000 client_fast.py:7(f)
```

### Results - `core_len.py`

```
         365627 function calls (362618 primitive calls) in 79.941 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     3002   78.231    0.026   78.231    0.026 {method 'poll' of 'select.epoll' objects}
        1    0.126    0.126   79.939   79.939 ioloop.py:746(start)
     3998    0.112    0.000    0.112    0.000 {method 'recv' of '_socket.socket' objects}002/3002    0.099    0.000    1.095    0.000 gen.py:990(run)
     1000    0.076    0.000    0.076    0.000 {built-in method print}
     6003    0.064    0.000    0.277    0.000 gen.py:1051(handle_yield)
    39034    0.058    0.000    0.094    0.000 {built-in method isinstance}
     2000    0.056    0.000    0.208    0.000 iostream.py:360(write)
     5004    0.054    0.000    1.517    0.000 stack_context.py:271(null_wrapper)
     2000    0.053    0.000    0.053    0.000 {method 'send' of '_socket.socket' objects}    2000    0.052    0.000    0.357    0.000 iostream.py:497(_handle_events)
     3998    0.046    0.000    0.252    0.000 iostream.py:585(_read_to_buffer_loop)
     3004    0.044    0.000    0.044    0.000 {method 'update' of 'dict' objects}
     3003    0.037    0.000    0.053    0.000 ioloop.py:928(add_callback)
2002/2001    0.037    0.000    0.615    0.000 gen.py:257(wrapper)
     3998    0.035    0.000    0.147    0.000 iostream.py:1010(read_from_fd)
     3998    0.033    0.000    0.184    0.000 iostream.py:721(_read_to_buffer)
     2001    0.029    0.000    0.128    0.000 iostream.py:827(_handle_write)
6005/6004    0.029    0.000    0.109    0.000 concurrent.py:318(_set_done)
     8008    0.028    0.000    0.046    0.000 stack_context.py:253(wrap)
     2002    0.027    0.000    0.301    0.000 gen.py:938(__init__)
     6004    0.025    0.000    0.040    0.000 functools.py:688(dispatch)
     6004    0.024    0.000    0.106    0.000 functools.py:726(wrapper)
     2000    0.023    0.000    0.204    0.000 iostream.py:294(read_bytes)
     2000    0.021    0.000    0.140    0.000 iostream.py:688(_try_inline_read)
     4004    0.020    0.000    0.035    0.000 abc.py:178(__instancecheck__)
     6005    0.019    0.000    0.019    0.000 concurrent.py:168(__init__)
     2000    0.019    0.000    0.120    0.000 iostream.py:669(_run_read_callback)
     6021    0.018    0.000    0.018    0.000 {built-in method hasattr}
     2002    0.018    0.000    0.710    0.000 client_fast.py:7(f)
```

It looks like I can avoid most of the slowdown in the latter example by only yielding to the write coroutine once, accepting a memcpy locally

### Before

``` python
yield stream.write(struct.pack('L', len(msg)))
yield stream.write(msg)
```

### After

``` python
yield stream.write(struct.pack('L', len(msg)) + msg)
```

I'm still not sure why there was such a significant cost for this though.

The first benchmark, of why we lose significant amounts of bandwidth when we add a bit of network latency still confuses me though.

OK, I can reproduce this without Tornado.  I suspect that I'm running into general TCP issues from having a high bandwidth delay product.

Your `read_bytes` benchmark is spending a lot of time in `join` because it needs to put everything in one giant string. It may be better, both in terms of CPU time and peak memory usage, to read the data in [chunks](https://github.com/tornadoweb/tornado/blob/50f9eedd89f26fc86974461b6ddd84343504be93/tornado/http1connection.py#L563-L566).

I suspect the difference with one write versus two has to do with the interaction between TCP_NODELAY and delayed acks, as seen in #110. Try using `stream.set_nodelay(True)`. 

OK, here is a side by side comparison using raw blocking sockets.  We implement an echo server (just sends back the message it gets.  We find that on two EC2 boxes (`m4.xlarge`) we can roundtrip 100MB of random data in 1 second with raw sockets and 1.5 seconds with Tornado IOStreams.  The time difference, 0.5s, happens to match the time spent waiting on epoll, though that might just be a coincidence.

At this stage this would be "nice to fix" but no longer my leading bottleneck.

## Raw Sockets

### Server

``` python
import socket

size = int(1e8)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

s.bind(('172.31.1.249', 5005))
s.listen(1)

while True:
    conn, addr = s.accept()
    print('Connection address:', addr)
    n = 0
    L = []
    while n < size:
        data = conn.recv(10240)
        L.append(data)
        n += len(data)
    conn.send(b''.join(L))  # echo
    conn.close()
```

### Client

``` python
import socket
from time import time

import numpy as np
data = bytes(np.random.randint(0, 255, dtype='u1', size=int(1e8)).data)
assert len(data) == int(1e8)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('172.31.1.249', 5005))

start = time()
s.send(data)

n = 0
L = []
while n < len(data):
    L.append(s.recv(10240))
    n += len(L[-1])

result = b''.join(L)
end = time()
print(end - start)

s.close()
```

### Profile

```
         39107 function calls in 0.996 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     9769    0.414    0.000    0.414    0.000 {method 'recv' of '_socket.socket' objects}
        1    0.279    0.279    0.279    0.279 {method 'send' of '_socket.socket' objects}
        1    0.135    0.135    0.135    0.135 {method 'join' of 'str' objects}
        1    0.074    0.074    0.996    0.996 <string>:1(<module>)
        1    0.060    0.060    0.922    0.922 client.py:8(f)
    19539    0.022    0.000    0.022    0.000 {len}
     9769    0.012    0.000    0.012    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.000    0.000 {method 'connect' of '_socket.socket' objects}
        1    0.000    0.000    0.000    0.000 socket.py:189(__init__)
       12    0.000    0.000    0.000    0.000 {setattr}
        1    0.000    0.000    0.000    0.000 socket.py:196(close)
        7    0.000    0.000    0.000    0.000 {getattr}
        1    0.000    0.000    0.000    0.000 socket.py:227(meth)
        2    0.000    0.000    0.000    0.000 {time.time}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
```

## Tornado IOStreams

### Server

``` python
from tornado.tcpserver import TCPServer
from tornado.ioloop import IOLoop
from tornado import gen

from core_len import read, write

class MyServer(TCPServer):
    @gen.coroutine
    def handle_stream(self, stream, address):
        data = yield read(stream)
        print(len(data))
        yield write(stream, data)

server = MyServer()
server.listen(8000)

IOLoop.current().start()
```

### Client

``` python
from tornado.tcpclient import TCPClient
from tornado.iostream import IOStream
from tornado.ioloop import IOLoop
from tornado import gen
from time import time

from core_len import read, write

import numpy as np
data = bytes(np.random.randint(0, 255, dtype='u1', size=100000000).data) # 100M

@gen.coroutine
def f():
    client = TCPClient()
    stream = yield client.connect('172.31.1.249', 8000, max_buffer_size=int(1e9))
    yield write(stream, data)
    msg = yield read(stream)
    print(len(msg))

if __name__ == '__main__':
    start = time()
    IOLoop().run_sync(f)
    end = time()
    print(end - start)
```

### Profile

``` python
         42699 function calls (42692 primitive calls) in 1.515 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       76    0.516    0.007    0.516    0.007 {method 'poll' of 'select.epoll' objects}
     1571    0.205    0.000    0.205    0.000 {method 'recv' of '_socket.socket' objects}      2    0.196    0.098    0.333    0.166 core_len.py:20(write)
     1694    0.144    0.000    0.144    0.000 {method 'join' of 'str' objects}
        1    0.136    0.136    0.137    0.137 iostream.py:360(write)
     1696    0.090    0.000    0.253    0.000 iostream.py:1510(_merge_prefix)
      892    0.062    0.000    0.062    0.000 {method 'send' of '_socket.socket' objects}      1    0.037    0.037    1.514    1.514 ioloop.py:746(start)
        2    0.036    0.018    0.179    0.090 iostream.py:876(_consume)
     1571    0.019    0.000    0.239    0.000 iostream.py:721(_read_to_buffer)
    13358    0.014    0.000    0.014    0.000 {len}
       28    0.012    0.000    0.257    0.009 iostream.py:585(_read_to_buffer_loop)
       47    0.010    0.000    0.187    0.004 iostream.py:827(_handle_write)
     1571    0.008    0.000    0.213    0.000 iostream.py:1010(read_from_fd)
     5130    0.004    0.000    0.004    0.000 {method 'popleft' of 'collections.deque' objects}7    0.004    0.000    0.004    0.000 {method 'append' of 'list' objects}
     1892    0.003    0.000    0.003    0.000 iostream.py:478(closed)
     2308    0.003    0.000    0.003    0.000 {method 'append' of 'collections.deque' objects}47    0.003    0.000    0.003    0.000 iostream.py:754(_run_streaming_callback)
      892    0.003    0.000    0.065    0.000 iostream.py:1023(write_to_fd)
     2822    0.003    0.000    0.003    0.000 {method 'appendleft' of 'collections.deque' objects}  0.001    0.000    0.625    0.009 iostream.py:497(_handle_events)
       82    0.000    0.000    0.960    0.012 stack_context.py:271(null_wrapper)
       78    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
```

There are 1571 calls to `recv` but only 76 to `epoll`, so the optimizations we make to avoid `epoll` calls are working most of the time. I think time spent in `epoll` is actually time while the network is idle, not the cost of `epoll` itself.

The sending side is significantly different between the two: the blocking version makes one call to the cpython `send` function`, which feeds the data to the kernel as fast as it can. The IOStream version makes 892 calls to`send`, returning back to python code each time (but not going all the way back through the IOLoop in most cases, as shown by the number of`epoll`calls). I think there's enough delay between the successive calls to`send` that it can't keep the network pipe full. 

Some rough numbers: the network is capable of sending 100MB in a second. Tornado sends at most 128KiB per call to `send` (this number was chosen to work around a windows bug but maybe it's time to reevaluate it). That data can be sent in 1.3 milliseconds. So if the loop in `_handle_write` sometimes takes more than 1.3ms to get to the next `write_to_fd` call, it would lead to a stall that could explain the difference in performance. The two calls to `merge_prefix` take 150µs each, getting us a quarter of the way there, but the rest of it would have to be something bursty (and therefore difficult to see in the averages) or invisible to the profiler. Garbage collection is one possibility. The profiler itself adds overhead to each function call and could be contributing to the difference between the two versions (since the latter has a lot more function calls).

Try increasing the limit in `IOStream._handle_write`. If that speeds things up then we can increase the limit for non-windows platforms. If not, we may just be hitting the limits of pure-python async IO.

Confirmed that using `IOStream.set_nodelay(True)` on the server does remove the multi-write slowdown issue.

I've observed that, at least on the EC2 machines that I'm using, memcpy is surprisingly slow, around 500MB/s.  It could be that the slowdown between the two versions is due to extra copies, both within Tornado and due to my single call to `write(len + message)`.

http://stackoverflow.com/questions/36523142/why-is-copying-memory-on-ec2-machines-slow?noredirect=1#comment60652232_36523142

This is a bit out of scope, but here is a profile from my actual application, which includes serializing down and compressing a numpy array on a 500MB/s connection (localhost, but similar to 10Gb-ish).  I found it interesting to see the performance issues here in the context of other costs.   

The results below are for sending a 1GB random numpy array to the same process over localhost. 

```
         381811 function calls (381774 primitive calls) in 1.718 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.384    0.384    0.384    0.384 {built-in method dumps}
        8    0.309    0.039    0.311    0.039 iostream.py:360(write)
    15358    0.229    0.000    0.229    0.000 {method 'join' of 'bytes' objects}
        1    0.196    0.196    0.196    0.196 {built-in method LZ4_compress}
    15362    0.174    0.000    0.174    0.000 {method 'recv' of '_socket.socket' objects}
    15368    0.163    0.000    0.405    0.000 iostream.py:1510(_merge_prefix)
     7769    0.105    0.000    0.105    0.000 {method 'send' of '_socket.socket' objects}
        1    0.040    0.040    1.718    1.718 ioloop.py:746(start)
        8    0.020    0.002    0.205    0.026 iostream.py:876(_consume)
    15362    0.020    0.000    0.204    0.000 iostream.py:721(_read_to_buffer)
       98    0.018    0.000    0.347    0.004 iostream.py:827(_handle_write)
       96    0.014    0.000    0.222    0.002 iostream.py:585(_read_to_buffer_loop)
   122593    0.009    0.000    0.009    0.000 {built-in method len}
    15362    0.007    0.000    0.181    0.000 iostream.py:1010(read_from_fd)
     7769    0.004    0.000    0.108    0.000 iostream.py:1023(write_to_fd)
```

This is fast enough that, on 1Gb network, we don't need to care much any more.  At 10Gb we still care, and probably the next easiest thing to optimize out is the bytestring handling within Tornado, which seems to be doable.  I may try a platform specific hack to just dump all bytes onto the socket on `IOStream.write` sometime in the near-to-moderate future.


Resolving patch:
diff --git a/tornado/iostream.py b/tornado/iostream.py
--- a/tornado/iostream.py
+++ b/tornado/iostream.py
@@ -82,6 +82,8 @@
 if hasattr(errno, "WSAEINPROGRESS"):
     _ERRNO_INPROGRESS += (errno.WSAEINPROGRESS,)  # type: ignore
 
+_WINDOWS = sys.platform.startswith('win')
+
 
 class StreamClosedError(IOError):
     """Exception raised by `IOStream` methods when the stream is closed.
@@ -158,11 +160,14 @@ def __init__(self, io_loop=None, max_buffer_size=None,
                                    self.max_buffer_size // 2)
         self.max_write_buffer_size = max_write_buffer_size
         self.error = None
-        self._read_buffer = collections.deque()
-        self._write_buffer = collections.deque()
+        self._read_buffer = bytearray()
+        self._read_buffer_pos = 0
         self._read_buffer_size = 0
+        self._write_buffer = bytearray()
+        self._write_buffer_pos = 0
         self._write_buffer_size = 0
         self._write_buffer_frozen = False
+        self._pending_writes_while_frozen = []
         self._read_delimiter = None
         self._read_regex = None
         self._read_max_bytes = None
@@ -373,21 +378,16 @@ def write(self, data, callback=None):
         .. versionchanged:: 4.0
             Now returns a `.Future` if no callback is given.
         """
-        assert isinstance(data, bytes)
         self._check_closed()
-        # We use bool(_write_buffer) as a proxy for write_buffer_size>0,
-        # so never put empty strings in the buffer.
         if data:
             if (self.max_write_buffer_size is not None and
                     self._write_buffer_size + len(data) > self.max_write_buffer_size):
                 raise StreamBufferFullError("Reached maximum write buffer size")
-            # Break up large contiguous strings before inserting them in the
-            # write buffer, so we don't have to recopy the entire thing
-            # as we slice off pieces to send to the socket.
-            WRITE_BUFFER_CHUNK_SIZE = 128 * 1024
-            for i in range(0, len(data), WRITE_BUFFER_CHUNK_SIZE):
-                self._write_buffer.append(data[i:i + WRITE_BUFFER_CHUNK_SIZE])
-            self._write_buffer_size += len(data)
+            if self._write_buffer_frozen:
+                self._pending_writes_while_frozen.append(data)
+            else:
+                self._write_buffer += data
+                self._write_buffer_size += len(data)
         if callback is not None:
             self._write_callback = stack_context.wrap(callback)
             future = None
@@ -396,7 +396,7 @@ def write(self, data, callback=None):
             future.add_done_callback(lambda f: f.exception())
         if not self._connecting:
             self._handle_write()
-            if self._write_buffer:
+            if self._write_buffer_size:
                 self._add_io_state(self.io_loop.WRITE)
             self._maybe_add_error_listener()
         return future
@@ -466,6 +466,7 @@ def _maybe_run_close_callback(self):
             # if the IOStream object is kept alive by a reference cycle.
             # TODO: Clear the read buffer too; it currently breaks some tests.
             self._write_buffer = None
+            self._write_buffer_size = 0
 
     def reading(self):
         """Returns true if we are currently reading from the stream."""
@@ -473,7 +474,7 @@ def reading(self):
 
     def writing(self):
         """Returns true if we are currently writing to the stream."""
-        return bool(self._write_buffer)
+        return self._write_buffer_size > 0
 
     def closed(self):
         """Returns true if the stream has been closed."""
@@ -743,7 +744,7 @@ def _read_to_buffer(self):
             break
         if chunk is None:
             return 0
-        self._read_buffer.append(chunk)
+        self._read_buffer += chunk
         self._read_buffer_size += len(chunk)
         if self._read_buffer_size > self.max_buffer_size:
             gen_log.error("Reached maximum read buffer size")
@@ -791,30 +792,25 @@ def _find_read_pos(self):
             # since large merges are relatively expensive and get undone in
             # _consume().
             if self._read_buffer:
-                while True:
-                    loc = self._read_buffer[0].find(self._read_delimiter)
-                    if loc != -1:
-                        delimiter_len = len(self._read_delimiter)
-                        self._check_max_bytes(self._read_delimiter,
-                                              loc + delimiter_len)
-                        return loc + delimiter_len
-                    if len(self._read_buffer) == 1:
-                        break
-                    _double_prefix(self._read_buffer)
+                loc = self._read_buffer.find(self._read_delimiter,
+                                             self._read_buffer_pos)
+                if loc != -1:
+                    loc -= self._read_buffer_pos
+                    delimiter_len = len(self._read_delimiter)
+                    self._check_max_bytes(self._read_delimiter,
+                                          loc + delimiter_len)
+                    return loc + delimiter_len
                 self._check_max_bytes(self._read_delimiter,
-                                      len(self._read_buffer[0]))
+                                      self._read_buffer_size)
         elif self._read_regex is not None:
             if self._read_buffer:
-                while True:
-                    m = self._read_regex.search(self._read_buffer[0])
-                    if m is not None:
-                        self._check_max_bytes(self._read_regex, m.end())
-                        return m.end()
-                    if len(self._read_buffer) == 1:
-                        break
-                    _double_prefix(self._read_buffer)
-                self._check_max_bytes(self._read_regex,
-                                      len(self._read_buffer[0]))
+                m = self._read_regex.search(self._read_buffer,
+                                            self._read_buffer_pos)
+                if m is not None:
+                    loc = m.end() - self._read_buffer_pos
+                    self._check_max_bytes(self._read_regex, loc)
+                    return loc
+                self._check_max_bytes(self._read_regex, self._read_buffer_size)
         return None
 
     def _check_max_bytes(self, delimiter, size):
@@ -824,35 +820,55 @@ def _check_max_bytes(self, delimiter, size):
                 "delimiter %r not found within %d bytes" % (
                     delimiter, self._read_max_bytes))
 
+    def _freeze_write_buffer(self, size):
+        self._write_buffer_frozen = size
+
+    def _unfreeze_write_buffer(self):
+        self._write_buffer_frozen = False
+        self._write_buffer += b''.join(self._pending_writes_while_frozen)
+        self._write_buffer_size += sum(map(len, self._pending_writes_while_frozen))
+        self._pending_writes_while_frozen[:] = []
+
+    def _got_empty_write(self, size):
+        """
+        Called when a non-blocking write() failed writing anything.
+        Can be overridden in subclasses.
+        """
+
     def _handle_write(self):
-        while self._write_buffer:
+        while self._write_buffer_size:
+            assert self._write_buffer_size >= 0
             try:
-                if not self._write_buffer_frozen:
+                start = self._write_buffer_pos
+                if self._write_buffer_frozen:
+                    size = self._write_buffer_frozen
+                elif _WINDOWS:
                     # On windows, socket.send blows up if given a
                     # write buffer that's too large, instead of just
                     # returning the number of bytes it was able to
                     # process.  Therefore we must not call socket.send
                     # with more than 128KB at a time.
-                    _merge_prefix(self._write_buffer, 128 * 1024)
-                num_bytes = self.write_to_fd(self._write_buffer[0])
+                    size = 128 * 1024
+                else:
+                    size = self._write_buffer_size
+                num_bytes = self.write_to_fd(
+                    memoryview(self._write_buffer)[start:start + size])
                 if num_bytes == 0:
-                    # With OpenSSL, if we couldn't write the entire buffer,
-                    # the very same string object must be used on the
-                    # next call to send.  Therefore we suppress
-                    # merging the write buffer after an incomplete send.
-                    # A cleaner solution would be to set
-                    # SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER, but this is
-                    # not yet accessible from python
-                    # (http://bugs.python.org/issue8240)
-                    self._write_buffer_frozen = True
+                    self._got_empty_write(size)
                     break
-                self._write_buffer_frozen = False
-                _merge_prefix(self._write_buffer, num_bytes)
-                self._write_buffer.popleft()
+                self._write_buffer_pos += num_bytes
                 self._write_buffer_size -= num_bytes
+                # Amortized O(1) shrink
+                # (this heuristic is implemented natively in Python 3.4+
+                #  but is replicated here for Python 2)
+                if self._write_buffer_pos > self._write_buffer_size:
+                    del self._write_buffer[:self._write_buffer_pos]
+                    self._write_buffer_pos = 0
+                if self._write_buffer_frozen:
+                    self._unfreeze_write_buffer()
             except (socket.error, IOError, OSError) as e:
                 if e.args[0] in _ERRNO_WOULDBLOCK:
-                    self._write_buffer_frozen = True
+                    self._got_empty_write(size)
                     break
                 else:
                     if not self._is_connreset(e):
@@ -863,7 +879,7 @@ def _handle_write(self):
                                         self.fileno(), e)
                     self.close(exc_info=True)
                     return
-        if not self._write_buffer:
+        if not self._write_buffer_size:
             if self._write_callback:
                 callback = self._write_callback
                 self._write_callback = None
@@ -874,11 +890,23 @@ def _handle_write(self):
                 future.set_result(None)
 
     def _consume(self, loc):
+        # Consume loc bytes from the read buffer and return them
         if loc == 0:
             return b""
-        _merge_prefix(self._read_buffer, loc)
+        assert loc <= self._read_buffer_size
+        # Slice the bytearray buffer into bytes, without intermediate copying
+        b = (memoryview(self._read_buffer)
+                       [self._read_buffer_pos:self._read_buffer_pos + loc]
+                       ).tobytes()
+        self._read_buffer_pos += loc
         self._read_buffer_size -= loc
-        return self._read_buffer.popleft()
+        # Amortized O(1) shrink
+        # (this heuristic is implemented natively in Python 3.4+
+        #  but is replicated here for Python 2)
+        if self._read_buffer_pos > self._read_buffer_size:
+            del self._read_buffer[:self._read_buffer_pos]
+            self._read_buffer_pos = 0
+        return b
 
     def _check_closed(self):
         if self.closed():
@@ -1251,6 +1279,17 @@ def reading(self):
     def writing(self):
         return self._handshake_writing or super(SSLIOStream, self).writing()
 
+    def _got_empty_write(self, size):
+        # With OpenSSL, if we couldn't write the entire buffer,
+        # the very same string object must be used on the
+        # next call to send.  Therefore we suppress
+        # merging the write buffer after an incomplete send.
+        # A cleaner solution would be to set
+        # SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER, but this is
+        # not yet accessible from python
+        # (http://bugs.python.org/issue8240)
+        self._freeze_write_buffer(size)
+
     def _do_ssl_handshake(self):
         # Based on code from test_ssl.py in the python stdlib
         try:
@@ -1498,53 +1537,6 @@ def read_from_fd(self):
         return chunk
 
 
-def _double_prefix(deque):
-    """Grow by doubling, but don't split the second chunk just because the
-    first one is small.
-    """
-    new_len = max(len(deque[0]) * 2,
-                  (len(deque[0]) + len(deque[1])))
-    _merge_prefix(deque, new_len)
-
-
-def _merge_prefix(deque, size):
-    """Replace the first entries in a deque of strings with a single
-    string of up to size bytes.
-
-    >>> d = collections.deque(['abc', 'de', 'fghi', 'j'])
-    >>> _merge_prefix(d, 5); print(d)
-    deque(['abcde', 'fghi', 'j'])
-
-    Strings will be split as necessary to reach the desired size.
-    >>> _merge_prefix(d, 7); print(d)
-    deque(['abcdefg', 'hi', 'j'])
-
-    >>> _merge_prefix(d, 3); print(d)
-    deque(['abc', 'defg', 'hi', 'j'])
-
-    >>> _merge_prefix(d, 100); print(d)
-    deque(['abcdefghij'])
-    """
-    if len(deque) == 1 and len(deque[0]) <= size:
-        return
-    prefix = []
-    remaining = size
-    while deque and remaining > 0:
-        chunk = deque.popleft()
-        if len(chunk) > remaining:
-            deque.appendleft(chunk[remaining:])
-            chunk = chunk[:remaining]
-        prefix.append(chunk)
-        remaining -= len(chunk)
-    # This data structure normally just contains byte strings, but
-    # the unittest gets messy if it doesn't use the default str() type,
-    # so do the merge based on the type of data that's actually present.
-    if prefix:
-        deque.appendleft(type(prefix[0])().join(prefix))
-    if not deque:
-        deque.appendleft(b"")
-
-
 def doctests():
     import doctest
     return doctest.DocTestSuite()


Test patch:
diff --git a/tornado/test/iostream_test.py b/tornado/test/iostream_test.py
--- a/tornado/test/iostream_test.py
+++ b/tornado/test/iostream_test.py
@@ -602,6 +602,17 @@ def close_callback():
             server.close()
             client.close()
 
+    def test_write_memoryview(self):
+        server, client = self.make_iostream_pair()
+        try:
+            client.read_bytes(4, self.stop)
+            server.write(memoryview(b"hello"))
+            data = self.wait()
+            self.assertEqual(data, b"hell")
+        finally:
+            server.close()
+            client.close()
+
     def test_read_bytes_partial(self):
         server, client = self.make_iostream_pair()
         try: