Repository: ansible/ansible-modules-core

Issue:
docker pull broken in devel for docker >= 1.8
## Issue type

Bug report
## Summary

The docker module fails when pulling new images when using docker >= 1.8.
## Ansible version:

```
ansible 2.0.0 (detached HEAD 90e005d234) last updated 2015/09/10 09:32:44 (GMT +100)
  lib/ansible/modules/core: (devel 1c7e8ef1a0) last updated 2015/09/09 17:21:32 (GMT +100)
  lib/ansible/modules/extras: (detached HEAD ff2386faf4) last updated 2015/07/21 18:18:28 (GMT +100)
  config file = /etc/ansible/ansible.cfg
  configured module search path = None
```
## Environment/config:

OSX source install.  Default config.
## Steps to reproduce

Add a docker image that does not exist on the target host.
Ansible will mark the pull as failed, even though it completes successfully.  Log below:

``` json
"{\"status\":\"Status: Downloaded newer image for docker.io/mesoscloud/mesos-master:0.23.0-centos-7\"}\r\n", "{}\r\n"], 
"failed": true, "msg": "Unrecognized status from pull.", "status": ""}
fatal: [172.21.22.225]: FAILED! => {"changed": false, "changes": ["{\"stream\":\"Trying to pull repository docker.io/mesoscloud/mesos-master ... \"}\r\n", 
"{\"status\":\"Pulling from mesoscloud/mesos-master\",\"id\":\"0.23.0-centos-7\"}\r\n", 

... Snipped lots of repeating progress messages  ...

"{\"status\":\"Extracting\",\"progressDetail\":{\"current\":32,\"total\":32},\"progress\":\"[==================================================\\u003e]     32 B/32 B\",\"id\":\"db34bb6f738d\"}", 
"{\"status\":\"Extracting\",\"progressDetail\":{\"current\":32,\"total\":32},\"progress\":\"[==================================================\\u003e]     32 B/32 B\",\"id\":\"db34bb6f738d\"}", "{\"status\":\"Pull complete\",\"progressDetail\":{},\"id\":\"db34bb6f738d\"}", "{\"status\":\"Digest: sha256:3cf66154ed913a72cb4d49254f45b4a0efa75aadc7aa98d04bc3344e9cfd1a19\"}\r\n",
 "{\"status\":\"Status: Downloaded newer image for docker.io/mesoscloud/mesos-master:0.23.0-centos-7\"}\r\n", "{}\r\n"], 
"failed": true, "msg": "Unrecognized status from pull.", "status": ""}
```

Note that the docker status is "Pull complete" and that the image exists in sudo docker show images after this.  It appears that ansible does not correctly parse the status message.

This only happens for me after upgrading from docker 1.7 to 1.8.

Re-running ansible works correctly, as now the image is downloaded and the pull is not required - however having to run ansible repeatedly to download multiple images is highly frustrating.



Discussion:
I'm seeing this too, with docker-1.8.2-5.gitcb216be.fc23.x86_64 , python-docker-py-1.3.1-1.fc23.noarch (on the docker host box) and ansible-1.9.3-3.fc23.noarch (on the machine running ansible).

Note: the problem here isn't that the status message has changed, I don't think. Note in the error message, it actually logs the status it got: `"status": ""` - the value is the empty string. So the problem is that it's not actually getting the status properly, somehow.

The code here is around https://github.com/AdamWill/ansible-modules-core/blob/devel/cloud/docker/docker.py#L1392 . It runs a `self.client.pull` ; `self.client` is a `docker.Client` instance, so there we're in https://github.com/docker/docker-py/blob/master/docker/api/image.py#L150 . It saves the result as a list and then pulls the last item out of the list, and treats it as JSON, and gets the `status` value out of that JSON.

Now I look at the output carefully it's pretty obvious what the problem is: the _very_ last item in the stream is an empty dict! Look at the end:

```
,"{\"status\":\"Status: Downloaded newer image for docker.io/mesoscloud/mesos-master:0.23.0-centos-7\"}\r\n", "{}\r\n"]
```

that second-to-last dict is the one we actually want - the one with the status message that the code is looking for - but there's a completely empty dict after it. So we're getting that empty dict, and obviously hence our status value will be the empty string.

So I guess the real problem here is in either docker-py or docker itself - whatever's causing that extra empty dict to show up in the stream. A dumb hack would be to take the last item, see if it's an empty dict, and take the second-to-last item if so, but that seems kinda awful. Having fun trying to track down what's causing this, though.


Resolving patch:
diff --git a/cloud/docker/docker.py b/cloud/docker/docker.py
--- a/cloud/docker/docker.py
+++ b/cloud/docker/docker.py
@@ -1392,6 +1392,11 @@ def pull_image(self):
             changes = list(self.client.pull(image, tag=tag, stream=True, **extra_params))
             try:
                 last = changes[-1]
+                # seems Docker 1.8 puts an empty dict at the end of the
+                # stream; catch that and get the previous instead
+                # https://github.com/ansible/ansible-modules-core/issues/2043
+                if last.strip() == '{}':
+                    last = changes[-2]
             except IndexError:
                 last = '{}'
             status = json.loads(last).get('status', '')


Test patch: