reading multipart/x-mixed-replace stream

Hi, I'm trying to fetch multipart/x-mixed-replace data with the [Streaming Requests](http://docs.python-requests.org/en/latest/user/advanced/#streaming-requests) example. As I read [here](http://en.wikipedia.org/wiki/Push_technology#HTTP_server_push):

> The content type multipart/x-mixed-replace was developed as part of a technology to emulate server push and streaming over HTTP.

So how to fetch this correctly? 

``` python
import requests

settings = { 'interval': '1000', 'count':'20' }
url = 'http://agent.mtconnect.org/sample'

r = requests.get(url, params=settings, stream=True)

for line in r.iter_lines():
    if line:
        print line
```

the content:

``` xml

--0910dc57f12aa1452fb67c74b7b2ddce
Content-type: text/xml
Content-length: 6299

... long xml ...

--e8f953249be4753b6f93d61bde0c970a
Content-type: text/xml
Content-length: 531

<?xml version="1.0" encoding="UTF-8"?>
<MTConnectStreams xmlns:m="urn:mtconnect.org:MTConnectStreams:1.2" xmlns="urn:mtconnect.org:MTConnectStreams:1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mtconnect.org:MTConnectStreams:1.2 /schemas/MTConnectStreams_1.2.xsd">
  <Header creationTime="2013-08-26T19:29:34Z" sender="wiesson.local" instanceId="1377530136" version="1.2.0.21" bufferSize="131072" nextSequence="18234" firstSequence="1" lastSequence="18233"/>
  <Streams/>
</MTConnectStreams>
```

So, I'm not able to separate header and content easily. And suggestions? (Normally I don't use github to post my _problem_ but here I can't help myself ;))

A simple curl in the terminal:

``` bash
curl http://http://agent.mtconnect.org/sample\?interval\=0\&count\=50
```

grabs the data.


The response is broken into parts, delimited by strings of the form `--<stuff>`. So, read data until you find one of those, then join all the previous lines up with `\r\n` characters. You now have a single part, which contains the headers and then the body. Split on the string `\r\n\r\n`. The first part is the headers: all the rest is the body.

``` python
part = []

for line in r.iter_lines():
    if line.startswith('--'):  # start of new chunk
        if not part:
            continue  # Handle empty parts

        strpart = '\r\n'.join(part)
        headers, body = strpart.split('\r\n\r\n', 1)

        do_stuff(headers, body)

        # Clean up for the next chunk.
        part = []
    elif line:  # Portion of current chunk.
        part.append(line)
```

Obviously, clean up the code and break some of it into functions. =)


Wow, thanks @Lukasa ... I tried your code but it seems that I don't understand it ;)

``` python
strpart = '\r\n'.join(part)
```

will join all line-breaks? The desired XML paylout itself also contains "\r\n"

``` xml
['Content-type: text/xml\r\nContent-length: 4117\r\n
<?xml version="1.0" encoding="UTF-8"?>\r\n
<MTConnectStreams xmlns:m="urn:mtconnect.org:MTConnectStreams:1.2" xmlns="urn:mtconnect.org:MTConnectStreams:1.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mtconnect.org:MTConnectStreams:1.2 http://www.mtconnect.org/schemas/MTConnectStreams_1.2.xsd">\r\n 

... more xml with \r\n ...
```

so I helped me with: 

``` python
strpart = '\r\n'.join(part[2:])
headers = ''.join(part[:2])
body = ''.join(strpart.split('\r\n\r\n',1))
```

works fine for me :) 


Okay, sorry for reopening - the problem: It feels like that it is not reading to the end? I've added a screen to demonstrate what I mean ;) 

Left side is requests with iter_lines(), the boundaries are not visible due to if line.startswith('--'),
Right side is a simple curl in the termal.
![screenshot 2013-08-27 14 06 57](https://f.cloud.github.com/assets/933231/1034192/eed76a4c-0f11-11e3-8834-072193784915.png)

So maybe I have to use iter_content and read the body "manually" ?


Nope. :smile:

`Response.iter_lines()` takes an optional `chunk_size` parameter, as you can [see here](http://docs.python-requests.org/en/latest/api/#requests.Response.iter_lines). The `iter_lines()` call will block until that much data has been read from the wire, at which point it will return the next set of data. If a response length is not a multiple of that value, you will not get it all until more data arrives.

You can get around this, at the cost of some inefficiency, by setting `chunk_size=1`.

NB: The same limitation applies to `iter_content()`.


Thanks a lot :) I read the docs but I haven't noticed that `chunk_size` parameter :/ I will play around and test some settings. 


Sorry for coming back to this problem - is it possible to change the content-length / chunk_size while reading a stream? As stated in the following snipped, I thought about reading the first element with chunk_size 1, then figuring out the content-length for each interval (as stated in die figure https://f.cloud.github.com/assets/933231/1034192/eed76a4c-0f11-11e3-8834-072193784915.png) . Chunk-size 1 is indeed very slow as you mentioned afore.

``` python
url = 'agent.mtconnect.org/sample?interval=0'
if __name__ == '__main__':
    chunk_size = 1
    chunk_flag = True
    util = XMLBuilder()

    requests_stream = requests.get(url, stream=True, timeout=4)
    for line in requests_stream.iter_lines(chunk_size, decode_unicode=None):
        print line

        if line.startswith('Content-length') and chunk_flag:
            strpart = line.split(': ')
            chunk_size = int(strpart[1])  # + 83
            chunk_flag = False
            print "-- Content-length: %d --" % chunk_size
            #: something like:
            #: requests_stream.iter_lines(chunk_size)

        elif line.endswith('</MTConnectStreams>'):
            chunk_flag = True

```


Absolutely you can.

The `iter_content()` function returns a generator. If you stop iterating over it before you exhaust the generator, then call the `iter_content()` function again, you get a new generator with the new size that continues from the same point.


@Lukasa beat me to this, but I was going to include the caveat that questions should be posted to [StackOverflow](http://stackoverflow.com/questions/tagged/python-requests) in the future.


@sigmavirus24 sorry for that, I will post my problems on stackoverflow in the future. :) @Lukasa, thanks for the quick answer!


No worries @wiesson . I had the answer ready to go too. I don't mind helping, which is why I keep SO open in a tab 24/7. :)


@Lukasa, here is my approach: Any comments or suggestions for improvements? No clue how to catch a disconnect :)

``` python
import requests
from lxml import etree

if __name__ == '__main__':
    content_length = 512
    flag = True
    requests_stream = requests.get('http://agent.mtconnect.org/sample?interval=0', stream=True, timeout=2)
    while flag:
        for line in requests_stream.iter_lines(3, decode_unicode=None):
            if line.startswith('Content-length'):
                content_length = int(''.join(x for x in line if x.isdigit()))
                break

        for xml in requests_stream.iter_content(content_length):
            root = etree.fromstring(xml.lstrip('\r\n'))
            ns = dict(m = 'urn:mtconnect.org:MTConnectStreams:1.2')
            header = root.find('.//m:Header', namespaces=ns)
            nextSeq = header.attrib['nextSequence']
            elements = root.findall('.//m:Events/*', namespaces=ns)
            for e in elements:
                print "%s is %s %s" % (e.tag.lstrip('{urn:mtconnect.org:MTConnectStreams:1.2}'), e.text, e.attrib)
            break
```


@wiesson the first thing I see is that you never set `flag = False` so you're look will go forever. If that's what you intend you could simplify it to:

``` python
while True:
    # ...
```

As for waiting for a disconnect, you can probably simulate that by looking at the tests in [shazow/urllib3](/shazow/urllib3). I believe there are a few dummy servers which simulate dropped connections so you could see how your code responds.


Ok, thanks for the comment, a simple break will end the loop, too. But how do I catch an exception in the `requests_stream.iter_lines` or  `requests_stream.iter_content` loop? 


Are you explicitly worried about the generator throwing an exception?


Hmm, I'm just looking for an exception that catches an event that indicates, there is no new "message" or no new event since x seconds. 


You can set a timeout, which should throw an exception if you have to wait that long for data.


Hi @Lukasa !
I'm trying to download multipart/x-mixed-replace.
Will you help me? ;)


the content:
`
--myboundary
Content-Type: text/plain
Content-Length: 2497

****text****

--myboundary
Content-Type: image/jpeg
Content-Length: 591666

****image data******
`