Iterating over chunks that are not terminated by new lines

I would like to iterator over chunks coming from a service, where each chunk is a JSON blob (similar to the example in the docs).  However, in this service, there are no newlines after each chunk.  As a result, it seems that `iter_lines` only yields one line, which is the entire response (not chunked), and `iter_content` yields one character at a time.

Is it possible to use this library to stream data from this service?

Thanks!


`iter_content()` takes an argument, the [chunk size](http://docs.python-requests.org/en/latest/api/#requests.Response.iter_content). Set this larger and you'll get back larger chunks. =)


Right, but isn't the chunk size determined by the server?  Before each chunk, the server sends the size of the next chunk.  Chunks are not all going to be the same size, so how can I specify the chunk size before-hand?


For example, I have [forked httpbin](https://github.com/pkaeding/httpbin/commit/40aeb58992613f61c616bad85ae3aa867e050684) and modified the `/stream` resource to not add newlines to the end of each chunk (note that this it adding a newline to the chunk itself, not the CRLF that terminates the chunk, and is not counted as part of the chunk size).

I have this forked httpbin app running at http://shielded-thicket-4419.herokuapp.com/ if you want to take a look.  The following code snippet:

```
import json
import requests

r = requests.get('http://shielded-thicket-4419.herokuapp.com/stream/20', stream=True)

for line in r.iter_lines():

    # filter out keep-alive new lines
    if line:
        print json.loads(line)
```

Now produces this error, because it is lumping all chunks into one:

```
Traceback (most recent call last):
  File "chunked_request_client.py", line 10, in <module>
    print json.loads(line)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 369, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 381 - line 1 column 7630 (char 381 - 7630)
```


Oh, I apologise, it wasn't clear to me that when you said 'chunk' you meant as-in 'chunked encoding'.

The short answer is that requests provides no way to use `iter_content` in such a way that you obtain chunks. However, chunked encoding _mandates_ a newline between chunk length and the chunk body itself, and then between the body and the next chunk length, so `iter_lines` would normally be fine. Your problem seems to be a server that is _approximating_ chunked encoding without actually doing it. Requests unfortunately provides no way to easily handle that situation beyond iterating using `iter_content` and handling the decode yourself.


As I [understand it](http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1), chunked encoding mandates a CRLF in between chunks, but not that the chunk data itself end in a new line.  Thus, I believe the modification I made to httpbin is still valid HTTP, right?


Nope. From [RFC 2616](http://tools.ietf.org/html/rfc2616#section-3.6.1), here is the ABNF for chunked encoding:

```
       Chunked-Body   = *chunk
                        last-chunk
                        trailer
                        CRLF

       chunk          = chunk-size [ chunk-extension ] CRLF
                        chunk-data CRLF
       chunk-size     = 1*HEX
```

Note that each chunk is made up of the chunk data followed by a CRLF combination.


Right, but I mean that the CRLF is not part of the chunk data.  For example, try running this:

```
$ curl -v --raw http://shielded-thicket-4419.herokuapp.com/stream/20
```

You will see that there is line break after the chunk data, before the next chunk header.  Also, note that the chunk header does not include that CRLF in its length.

(You will also see that my chunks are not coming out on the right boundaries; it seems that heroku is doing something at the proxy level.)  When I run locally, it looks like this:

```
...
AB
{"url": "http://localhost:5000/stream/20", "headers": {"Host": "localhost:5000", "Accept": "*/*", "User-Agent": "curl/7.27.0"}, "args": {}, "id": 9, "origin": "127.0.0.1"}
AC
{"url": "http://localhost:5000/stream/20", "headers": {"Host": "localhost:5000", "Accept": "*/*", "User-Agent": "curl/7.27.0"}, "args": {}, "id": 10, "origin": "127.0.0.1"}
...
```

That first line in that snippet is 171 characters long, not including the line break.  171 in decimal is 0xAB in hex.  So, you can see that there is a CRLF terminating the chunk, but there is not one that is a part of the chunk.


Agreed, but the chunk still needs to be terminated by a CRLF: the fact that it's not included in the chunk length is irrelevant.

I'm really struggling to see the problem here. Any well-formed chunked encoding will have its chunks terminated by CRLF. `iter_lines` will split on those. Use `iter_lines` to get chunks. Why is that logic not working?


Oh, hang on, I see it. Presumably `httplib` is consuming the newline. Argh.


Yeah, in this situation requests doesn't give you the kind of low-level control you want. In requests' opinion, the Transfer Encoding is an implementation detail: the fact that chunked transfer encoding is being used should not be exposed to the user. Unfortunately, if you want that finer-grained control you'll need to use a lower-level library. Alternatively, consider a different method of obtaining the JSON (such as a streaming JSON decoder).


there are some hacks: http://mihai.ibanescu.net/chunked-encoding-and-python-requests
I hope this function could merged into requests core.


I'm open to having requests provide some useful way of handling chunked encoding as it arrives, though how it would fit into the API is a bit unclear.


Note also that chunked encoding is an artefact of HTTP/1.1 and is not present in HTTP/2. That may be an argument _against_ exposing it in requests: it's obviously a transport-level detail.


@Lukasa  http/2 may not call it chunked transfers, but it does have DATA frames as mentioned here: [rfc7540](http://httpwg.org/specs/rfc7540.html#HttpSequence). The 'pad length' concept is analogous to chunk size.

@electronicsguy pad length is not analagous to chunk size: it is how much padding is attached to the frame. 

More importantly, again, DATA frames are not semantic and should also not be exposed to users. 

@Lukasa   oh I see. Thanks for clearing that up about lad length. But I don't understand why DATA frames are not analogous to chunking. It is in-fact mentioned in this mailing list that it is the case, at least logically: [http/2-chunking](https://lists.w3.org/Archives/Public/ietf-http-wg/2014JulSep/1676.html). Even in the case of http/1.1, chunked transfer encoding is not visible to the users since it is at the transport layer. So when you said above that chunked encoding is an artefact of http/1.1 and not present in http/2, I'm not able to understand how that is.

@electronicsguy So the short answer is that we're in agreement. Note again what I said:

> Note also that chunked encoding is an artefact of HTTP/1.1 and is not present in HTTP/2. That may be an argument *against* exposing it in requests: it's obviously a transport-level detail.

That sentence can be rephrased as:

> Note that because chunked encoding was removed in HTTP/2, it is clearly not a semantic part of HTTP, as HTTP/2 was required to keep a one-to-one semantic mapping to HTTP/1.1. For this reason, we should probably consider not exposing the chunks: they aren't supposed to have semantic meaning to the user.

This is of course backed up by all the relevant RFCs. It's also the course we're taking in the urllib3 v2 work: the boundaries of data chunks are not being exposed to users.