Repository: docker/docker-py

Issue:
streaming from build() is broken in 0.3.0
Some client code that was working fine w/ docker-py 0.2.3 now breaks on 0.3.0

A minimal test case looks like:

```
ubuntu@ip-10-77-1-34:/tmp/foo$ cat Dockerfile
FROM stackbrew/ubuntu:13.04
RUN echo "hello"
CMD /bin/bash

ubuntu@ip-10-77-1-34:/tmp/foo$ cat test.py
import docker

doc = docker.Client()
gen = doc.build(path='.', stream=True)
for line in gen:
    print line
```

Executing `test.py` under 0.2.3 builds the image fine.  Under 0.3.0, I get 

```
ubuntu@ip-10-77-1-34:/tmp/foo$ python test.py
{"stream":" ---\u003e Using cache\n"}

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    for line in gen:
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 239, in _stream_helper
    size = int(socket.readline(), 16)
ValueError: invalid literal for int() with base 16: ''
```

This is Ubuntu 12.04 w/ Docker 0.9.0

`client.pull(stream=True)` doesn't split stream lines correctly?
This is on Ubuntu 13.10, with Docker 0.10.0:

```
Python 2.7.5+ (default, Sep 19 2013, 13:48:49)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker.client
>>> client = docker.Client()
>>> stream = client.pull('busybox', stream=True)
>>> chunks = [chunk for chunk in stream]
>>> len(chunks)
2
>>> chunks[0]
'{"status":"Pulling repository busybox"}'
>>> len(chunks[1])
22721
```

Basically `chunks[0]` is one line of the stream, and `chunks[1]` is every other line of the stream, concatenated. (I can take a look at making a PR for this tomorrow, if you'd like.)



Discussion:
Thanks for the report! Which Python version are you using?

ah, I figured I'd forgotten something relevant: 2.7.3.     Cheers.

Same problem in Python 2.7.5.

Workaround for me was to force version='1.6' in the Client constructor.

Sorry, I looked more closely, and my problem is close, but not the same as rosenhouse's problem.  I get this problem _without_ specifying stream=True in 0.3.0, but it does involve the same _stream_helper() code.

Sorry if I'm hijacking this for something that should be a different issue, but I'll post my findings.

Here is what I see:

```
In [3]: c = docker.Client(base_url='http://localhost:4243')

In [4]: r = c.build(path='/tmp/tmpdEEGN8', tag='eddy-test')

In [5]: r
Out[5]: <generator object _stream_helper at 0x2263960>

...

In [7]: r.next()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-8d25240d5a85> in <module>()
----> 1 r.next()

/home/ereyes/work/new-env/local/lib/python2.7/site-packages/docker/client.pyc in _stream_helper(self, response)
    233     def _stream_helper(self, response):
    234         """Generator for data coming from a chunked-encoded HTTP response."""
--> 235         socket_fp = self._get_raw_response_socket(response)
    236         socket_fp.setblocking(1)
    237         socket = socket_fp.makefile()

/home/ereyes/work/new-env/local/lib/python2.7/site-packages/docker/client.pyc in _get_raw_response_socket(self, response)
    229     def _get_raw_response_socket(self, response):
    230         self._raise_for_status(response)
--> 231         return response.raw._fp.fp._sock
    232 
    233     def _stream_helper(self, response):

AttributeError: 'NoneType' object has no attribute '_sock'
```

The image does get built, but my code was relying on the return value of build() to get at the image ID.

It looks like, by default as of "version 1.8", build will _always_ assume you want to stream the output back:

``` python
        if stream or utils.compare_version('1.8', self._version) >= 0:
            return self._stream_helper(response)
        else:
            output = self._result(response)
```

This is why the workaround for me was to force version='1.6' in the client.  This worked for me because I didn't want to stream at all.  I would imagine that if you really want to do stream=True, you may still be stuck if you force the version to something older like 1.6.

The reason for this is that from API version 1.8 onwards the `/build` endpoint IIRC always returns a stream.

Yes, I think these are 2 different issues. Absence of image id for version >= 1.8 is intended, errors when trying to read from the stream are not.

I'm getting this issue too but on pull(stream=True).  Ubuntu 13.10, Python 2.6.9 (yeah, i know that's a little odd), Docker 0.9.0, Docker-py 0.3.0.

```
    def _stream_helper(self, response):
        """Generator for data coming from a chunked-encoded HTTP response."""
        socket_fp = self._get_raw_response_socket(response)
        socket_fp.setblocking(1)
        socket = socket_fp.makefile()
        while True:
>           size = int(socket.readline(), 16)
E           ValueError: invalid literal for int() with base 16: ''
```

Is the root cause of this know?  I'm going to dig into this tonight, as it is breaking some stuff for me.  Let me know if the correct fix is already known.

So basically what I'm seeing is that between 0.8.x and 0.9.0, due to https://github.com/dotcloud/docker/pull/4276, docker now sends empty newline chunks in streamed response.

So before the response would look line

```
42\r\n
{......}\r\n
42\r\n
{......}\r\n
```

Now it looks like 

```
42\r\n
{......}\r\n
\r\n
42\r\n
{......}\r\n
\r\n
```

So I think just a simple check to ignore the newlines will probably fix it.  I'll try it out.

Ah, I see what's happening now.  So _stream_helper reads the chunk size, but then doesn't actually use the size to read the data

``` python
    def _stream_helper(self, response):
        """Generator for data coming from a chunked-encoded HTTP response."""
        socket_fp = self._get_raw_response_socket(response)
        socket_fp.setblocking(1)
        socket = socket_fp.makefile()
        while True:
            size = int(socket.readline(), 16)
            if size <= 0:
                break
            data = socket.readline()
            if not data:
                break
            yield data
```

So if the data has a newline in it (which in 0.8.x never happened), then socket.readline() will read a partial data chunk.  Then when the loop starts again, where you are expecting to read an int, you are really still reading the chunk.

I'm going to experiment with switching to using [Request.iter_lines()](http://docs.python-requests.org/en/latest/user/advanced/#streaming-requests) so that docker-py doesn't have to be so low level and read chunks

Thanks for reporting this. I don't think we've validated `docker-py` with Docker 0.10 yet. Correctly returning the streamed lines has been a constant challenge because Docker changed the outputted stream format, and it's wasn't consistent across all endpoints. Hopefully 0.10 brings some sanity to this.

Along these lines, pulling a non-existent repository doesn't result in an error.  It should throw an exception, but in my case I'm just getting

```
{"status":"Pulling repository apps/skydock-nginx"}
{"errorDetail":{"message":"HTTP code: 404"},"error":"HTTP code: 404"}
```

I don't think it warrants its own issue, but it is a problem I've run into.


Resolving patch:
diff --git a/docker/client.py b/docker/client.py
--- a/docker/client.py
+++ b/docker/client.py
@@ -209,9 +209,23 @@ def _get_raw_response_socket(self, response):
 
     def _stream_helper(self, response):
         """Generator for data coming from a chunked-encoded HTTP response."""
-        for line in response.iter_lines(chunk_size=32):
-            if line:
-                yield line
+        socket_fp = self._get_raw_response_socket(response)
+        socket_fp.setblocking(1)
+        socket = socket_fp.makefile()
+        while True:
+            # Because Docker introduced newlines at the end of chunks in v0.9,
+            # and only on some API endpoints, we have to cater for both cases.
+            size_line = socket.readline()
+            if size_line == '\r\n':
+                size_line = socket.readline()
+
+            size = int(size_line, 16)
+            if size <= 0:
+                break
+            data = socket.readline()
+            if not data:
+                break
+            yield data
 
     def _multiplexed_buffer_helper(self, response):
         """A generator of multiplexed data blocks read from a buffered


Test patch:
diff --git a/tests/integration_test.py b/tests/integration_test.py
--- a/tests/integration_test.py
+++ b/tests/integration_test.py
@@ -14,6 +14,7 @@
 
 import time
 import base64
+import json
 import io
 import os
 import signal
@@ -666,10 +667,8 @@ def runTest(self):
         self.assertIn('Images', info)
         img_count = info['Images']
         stream = self.client.pull('joffrey/test001', stream=True)
-        res = u''
         for chunk in stream:
-            res += chunk
-        self.assertEqual(type(res), six.text_type)
+            json.loads(chunk)  # ensure chunk is a single, valid JSON blob
         self.assertEqual(img_count + 3, self.client.info()['Images'])
         img_info = self.client.inspect_image('joffrey/test001')
         self.assertIn('id', img_info)
@@ -762,6 +761,7 @@ def runTest(self):
         stream = self.client.build(fileobj=script, stream=True)
         logs = ''
         for chunk in stream:
+            json.loads(chunk)  # ensure chunk is a single, valid JSON blob
             logs += chunk
         self.assertNotEqual(logs, '')