Chunked transfer encoding for SSE heartbeats failing

When subscribing to a Server Sent Events endpoint, I notice an error given the processing of a heartbeat. Heartbeats in SSE can be sent as an empty line. Consider the following SSE reply for chunked transfer encoding:

```
    00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d HTTP/1.1  200 OK.
    00000010  0a 53 65 72 76 65 72 3a  20 6f 70 65 6e 72 65 73 .Server:  openres
    00000020  74 79 2f 31 2e 39 2e 31  35 2e 31 0d 0a 44 61 74 ty/1.9.1 5.1..Dat
    00000030  65 3a 20 46 72 69 2c 20  32 38 20 4f 63 74 20 32 e: Fri,  28 Oct 2
    00000040  30 31 36 20 30 32 3a 33  39 3a 35 34 20 47 4d 54 016 02:3 9:54 GMT
    00000050  0d 0a 43 6f 6e 74 65 6e  74 2d 54 79 70 65 3a 20 ..Conten t-Type: 
    00000060  74 65 78 74 2f 65 76 65  6e 74 2d 73 74 72 65 61 text/eve nt-strea
    00000070  6d 0d 0a 54 72 61 6e 73  66 65 72 2d 45 6e 63 6f m..Trans fer-Enco
    00000080  64 69 6e 67 3a 20 63 68  75 6e 6b 65 64 0d 0a 43 ding: ch unked..C
    00000090  6f 6e 6e 65 63 74 69 6f  6e 3a 20 6b 65 65 70 2d onnectio n: keep-
    000000A0  61 6c 69 76 65 0d 0a 0d  0a                      alive... .
    000000A9  31 0d 0a 0a 0d 0a                                1.....
```

With the last line, the `31` indicates that there is just one byte to receive (ASCII `1`). The chunked size is then followed by a CRLF and then the empty line (`LF`), followed again by a CRLF.

This appears to cause the following stacktrace:

```
  ...
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/utils.py", line 377, in stream_decode_response_unicode
    for chunk in iterator:
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/models.py", line 676, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 353, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 502, in read_chunked
    self._update_chunk_length()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 448, in _update_chunk_length
    line = self._fp.fp.readline()
AttributeError: 'NoneType' object has no attribute 'readline'
```

What I'm left wondering is whether chunked responses that only contain a newline cause the requests library a problem.

I'm using requests  2.11.1.


I can't reproduce a problem on my system by duplicating that output. Is it possible for you to provide the code you're using?


I appreciate the situation, however a reproducer isn't immediately available. I'll try and work on something for you, but can't promise anything. Thanks for any further insights.


Thanks @huntc. In the meantime, the best I can give you is the knowledge that this isn't immediately obviously a requests problem. =( Sorry!


I've been looking at the request code most of today. The fact that it is throwing an error given a correct SSE reply should be cause for concern. My theory is that the body is consumed and eats the LF without considering that it is to be counted in the chunked reply. 


@huntc That theory has at least one counter-example:

``` python
import socket

s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 8080))
s.listen(5)

while True:
    new_sock, _ = s.accept()
    data = b''
    while not data.endswith(b'\r\n\r\n'):
        data += new_sock.recv(8192)

    new_sock.sendall(
        b'HTTP/1.1 200 OK\r\n'
        b'Server: socket server\r\n'
        b'Transfer-Encoding: chunked\r\n'
        b'\r\n'
    )
    new_sock.sendall(b'1\r\n\n\r\n')
    new_sock.sendall(b'1\r\n\n\r\n')
    new_sock.sendall(b'5\r\nhello\r\n')
    new_sock.sendall(b'5\r\nworld\r\n')
    new_sock.sendall(b'0\r\n\r\n')
    new_sock.close()
```

If you run that script and then do this:

``` python
r = requests.get('http://localhost:8080/', stream=True)
for chunk in r.iter_content(1024):
    print repr(chunk)
```

then I encounter no error.


Thanks. I'll look further. 


I've got a reproducer - same server code as your provided, but here's the client code. I shall investigate further, but I'm posting here in case you see anything obvious. The code was copied and modified from https://bitbucket.org/btubbs/sseclient, so I may have inherited a problem, or inadvertently introduced one.

``` python
import requests
import re


SSE_REQUEST_INPUT = {
    'headers': {
        'Cache-Control': 'no-cache',
        'Accept': 'text/event-stream'
    }
}


SSE_END_OF_FIELD = re.compile(r'\r\n\r\n|\r\r|\n\n')


def is_event_complete(buf):
    return re.search(SSE_END_OF_FIELD, buf) is not None


def parse_event(raw_sse_string):
    event, data = None, None
    lines = raw_sse_string.split('\n')
    for line in lines:
        if line.rstrip():
            key, value = line.split(':')
            if key == 'event':
                event = value
            elif key == 'data':
                data = value
    return Event(event, data)


class Event:
    def __init__(self, event, data):
        self.event = event
        self.data = data

    def __eq__(self, other):
        if type(other) is type(self):
            return self.__dict__ == other.__dict__
        else:
            return False

    def __ne__(self, other):
        return not self.__eq__(other)


class Client:
    """
    The following SSE Client has been backported, cut down version of https://bitbucket.org/btubbs/sseclient -
    look at sseclient.py.

    We were not be able to use the SSE client library as it does not have the support for Python 3.2 due
    to the presence of unicode string declaration (i.e. u''). The unicode string declaration is valid from Python 3.3 and
    above, while in Python 3.2 it will result in Syntax error.

    Changes introduced as part of the backport:
    - Support for Python 3.2 (i.e. do not use u'')
    - Parse only for event and data string within SSE
    - No support for retries and last event id
    """
    def __init__(self, url, headers=None):
        self.url = url
        self.headers = headers
        self.response = None

    def connect(self):
        sse_request_input = dict(SSE_REQUEST_INPUT)
        if self.headers:
            sse_request_input['headers'].update(self.headers)

        response = requests.get(self.url, stream=True, **sse_request_input)
        response.raise_for_status()
        self.response = response

    def __iter__(self):
        return self

    def __next__(self):
        buf = ''
        while not is_event_complete(buf):
            next_char = next(self.response.iter_content(decode_unicode=True))
            buf += next_char.decode('ascii')

        split = re.split(SSE_END_OF_FIELD, buf)
        raw_sse = split[0]
        return parse_event(raw_sse)


def get_events(url, headers=None):
    client = Client(url, headers)
    client.connect()
    return client

sse_events = get_events('http://localhost:8080/')
for event in sse_events:
    print(repr(event))
```


Further investigations show that the problem is occurring within:

``` python
            next_char = next(self.response.iter_content(decode_unicode=True))
```

Here's my stack trace:

```
ChristohersMBP2:test huntc$ python3 client.py
b'\n'
Traceback (most recent call last):
  File "client.py", line 97, in <module>
    for event in sse_events:
  File "client.py", line 82, in __next__
    next_char = next(self.response.iter_content(decode_unicode=True))
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/utils.py", line 372, in stream_decode_response_unicode
    for item in iterator:
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/models.py", line 676, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 353, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 502, in read_chunked
    self._update_chunk_length()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 448, in _update_chunk_length
    line = self._fp.fp.readline()
AttributeError: 'NoneType' object has no attribute 'readline'
```

I will continue looking further over the w/e.


That's ok, I've identified the problem. `iter_content` returns a _generator_, which necessarily contains some state. In the case of `iter_content` it contains two particularly important bits of state: one related to Unicode decoding (which you're using), and one related to response completion state. If you throw the generators away, or create more than one, you lose that state, which causes Requests to get confused about where the response body is or what data has already been read.

Change your code to call `iter_content` once per response and save its return value in a variable, and then call `next` on that variable instead. This should resolve your problem.


Got it! Thanks so much for your help!