Repository: ansible/ansible

Issue:
docker_service may not provide useful errors when tasks fail
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
`docker_service`

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.2.0.0
  config file = /Users/shabble/work/mhn-automation/ansible.cfg
  configured module search path = Default w/o overrides
```

and

```
ansible 2.3.0 (devel 9d26d8b605) last updated 2017/01/19 20:34:38 (GMT +100)
  config file =
  configured module search path = Default w/o overrides
```



##### CONFIGURATION
Nothing interesting
```
[defaults]
hostfile=hosts
roles_path=roles/
command_warnings=True
nocows=1
```

<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Runner: OSX 10.11.6, python 2.7.12 (via macports)
Target: Debian 8

##### SUMMARY
<!--- Explain the problem briefly -->
When using the `docker_service` module, certain runtime or configuration errors are masked and not available in any of the ansible return values or logs.
The cause seems to be `docker-compose` logging (some) error messages directly to stdout or stderr, even when called as a library by `docker_service`.
 
This might be related to ansible/ansible-modules-core#5117 and #19371 

It is not fully resolved by PR ansible/ansible-modules-core#5165 since that only considers stdout.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
Minimal testcase below should cause a failure due to the 2 containers both trying to forward the same port, resulting in an error to stderr which isn't returned by ansible.

```yaml
---
- hosts: all
  tasks:
    - docker_service:
        project_name: foo
        debug: yes
        definition:
          version: '2'
          services:
            testcontainer:
              image: busybox
              ports:
                - "8001:8000"
              command: 'sleep 100'
            testcontainer2:
              image: busybox
              ports:
                - "8001:8000"
              command: 'sleep 100'
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
A useful error message similar to that produced when running `docker-compose up` explicitly.

For comparison, running the same testcase using just a docker-compose.yml produces the output:
```
Creating network "ansbug_default" with the default driver
Creating ansbug_testcontainer_1
Creating ansbug_testcontainer2_1

ERROR: for testcontainer  Cannot start service testcontainer: driver failed programming external connectivity on endpoint ansbug_testcontainer_1 (5ff302538b9881883b36afc21da5c9246dc304f1620c0bee98cca8093bbc0d01): Bind for 0.0.0.0:8001 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.
```
`docker-compose up 2>/dev/null` produces no output, indicating the messages are being sent to stderr.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
*Run with ansible 2.2.0*:

```
$ ansible-playbook -i hosts.dtest dc-errmsg-demo.yml -vvv                             
Using /Users/shabble/work/mhn-automation/ansible.cfg as config file

PLAYBOOK: dc-errmsg-demo.yml ***************************************************
1 plays in dc-errmsg-demo.yml

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
Using module file /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
<ansibledocker.mhn> ESTABLISH SSH CONNECTION FOR USER: shabble
<ansibledocker.mhn> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r ansibledocker.mhn '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1484858631.24-77548129045125 `" && echo ansible-tmp-1484858631.24-77548129045125="` echo $HOME/.ansible/tmp/ansible-tmp-1484858631.24-77548129045125 `" ) && sleep 0'"'"''
<ansibledocker.mhn> PUT /var/folders/lr/ph2sd0md7_5_51wtxztmcmm80000gn/T/tmp66feBd TO /home/shabble/.ansible/tmp/ansible-tmp-1484858631.24-77548129045125/setup.py
<ansibledocker.mhn> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r '[ansibledocker.mhn]'
<ansibledocker.mhn> ESTABLISH SSH CONNECTION FOR USER: shabble
<ansibledocker.mhn> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r ansibledocker.mhn '/bin/sh -c '"'"'chmod u+x /home/shabble/.ansible/tmp/ansible-tmp-1484858631.24-77548129045125/ /home/shabble/.ansible/tmp/ansible-tmp-1484858631.24-77548129045125/setup.py && sleep 0'"'"''
<ansibledocker.mhn> ESTABLISH SSH CONNECTION FOR USER: shabble
<ansibledocker.mhn> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r -tt ansibledocker.mhn '/bin/sh -c '"'"'/usr/bin/python /home/shabble/.ansible/tmp/ansible-tmp-1484858631.24-77548129045125/setup.py; rm -rf "/home/shabble/.ansible/tmp/ansible-tmp-1484858631.24-77548129045125/" > /dev/null 2>&1 && sleep 0'"'"''
ok: [ansibledocker.mhn]

TASK [docker_service] **********************************************************
task path: /Users/shabble/work/mhn-automation/dc-errmsg-demo.yml:4
Using module file /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/modules/core/cloud/docker/docker_service.py
<ansibledocker.mhn> ESTABLISH SSH CONNECTION FOR USER: shabble
<ansibledocker.mhn> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r ansibledocker.mhn '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1484858633.54-69159756081069 `" && echo ansible-tmp-1484858633.54-69159756081069="` echo $HOME/.ansible/tmp/ansible-tmp-1484858633.54-69159756081069 `" ) && sleep 0'"'"''
<ansibledocker.mhn> PUT /var/folders/lr/ph2sd0md7_5_51wtxztmcmm80000gn/T/tmprlIO_I TO /home/shabble/.ansible/tmp/ansible-tmp-1484858633.54-69159756081069/docker_service.py
<ansibledocker.mhn> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r '[ansibledocker.mhn]'
<ansibledocker.mhn> ESTABLISH SSH CONNECTION FOR USER: shabble
<ansibledocker.mhn> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r ansibledocker.mhn '/bin/sh -c '"'"'chmod u+x /home/shabble/.ansible/tmp/ansible-tmp-1484858633.54-69159756081069/ /home/shabble/.ansible/tmp/ansible-tmp-1484858633.54-69159756081069/docker_service.py && sleep 0'"'"''
<ansibledocker.mhn> ESTABLISH SSH CONNECTION FOR USER: shabble
<ansibledocker.mhn> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=shabble -o ConnectTimeout=10 -o ControlPath=/Users/shabble/.ansible/cp/ansible-ssh-%h-%p-%r -tt ansibledocker.mhn '/bin/sh -c '"'"'/usr/bin/python /home/shabble/.ansible/tmp/ansible-tmp-1484858633.54-69159756081069/docker_service.py; rm -rf "/home/shabble/.ansible/tmp/ansible-tmp-1484858633.54-69159756081069/" > /dev/null 2>&1 && sleep 0'"'"''
fatal: [ansibledocker.mhn]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "api_version": null,
            "build": false,
            "cacert_path": null,
            "cert_path": null,
            "debug": true,
            "definition": {
                "services": {
                    "testcontainer": {
                        "command": "sleep 100",
                        "image": "busybox",
                        "ports": [
                            "8001:8000"
                        ]
                    },
                    "testcontainer2": {
                        "command": "sleep 100",
                        "image": "busybox",
                        "ports": [
                            "8001:8000"
                        ]
                    }
                },
                "version": "2"
            },
            "dependencies": true,
            "docker_host": null,
            "files": null,
            "filter_logger": false,
            "hostname_check": false,
            "key_path": null,
            "nocache": false,
            "project_name": "foo",
            "project_src": null,
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": 10,
            "tls": null,
            "tls_hostname": null,
            "tls_verify": null
        },
        "module_name": "docker_service"
    },
    "msg": "Error starting project - "
}
	to retry, use: --limit @/Users/shabble/work/mhn-automation/dc-errmsg-demo.retry

PLAY RECAP *********************************************************************
ansibledocker.mhn          : ok=1    changed=0    unreachable=0    failed=1
```

Relevant part here is: ` "msg": "Error starting project - "`

*Run with current git HEAD*:

```
[... equivalent setup spam snipped...]
fatal: [ansibledocker.mhn]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "api_version": null,
            "build": false,
            "cacert_path": null,
            "cert_path": null,
            "debug": true,
            "definition": {
                "services": {
                    "testcontainer": {
                        "command": "sleep 100",
                        "image": "busybox",
                        "ports": [
                            "8001:8000"
                        ]
                    },
                    "testcontainer2": {
                        "command": "sleep 100",
                        "image": "busybox",
                        "ports": [
                            "8001:8000"
                        ]
                    }
                },
                "version": "2"
            },
            "dependencies": true,
            "docker_host": null,
            "files": null,
            "filter_logger": false,
            "hostname_check": false,
            "key_path": null,
            "nocache": false,
            "project_name": "foo",
            "project_src": null,
            "pull": false,
            "recreate": "smart",
            "remove_images": null,
            "remove_orphans": false,
            "remove_volumes": false,
            "restarted": false,
            "scale": null,
            "services": null,
            "ssl_version": null,
            "state": "present",
            "stopped": false,
            "timeout": 10,
            "tls": null,
            "tls_hostname": null,
            "tls_verify": null
        },
        "module_name": "docker_service"
    },
    "module_stderr": "",
    "module_stdout": "",
    "msg": "Error starting project "
}
```



Discussion:

Resolving patch:
diff --git a/lib/ansible/modules/cloud/docker/docker_service.py b/lib/ansible/modules/cloud/docker/docker_service.py
old mode 100644
new mode 100755
--- a/lib/ansible/modules/cloud/docker/docker_service.py
+++ b/lib/ansible/modules/cloud/docker/docker_service.py
@@ -503,20 +503,78 @@ def stdout_redirector(path_name):
     finally:
         sys.stdout = old_stdout
 
-def get_stdout(path_name):
-    full_stdout = ''
-    last_line = ''
+
+@contextmanager
+def stderr_redirector(path_name):
+    old_fh = sys.stderr
+    fd = open(path_name, 'w')
+    sys.stderr = fd
+    try:
+        yield
+    finally:
+        sys.stderr = old_fh
+
+
+def make_redirection_tempfiles():
+    _, out_redir_name = tempfile.mkstemp(prefix="ansible")
+    _, err_redir_name = tempfile.mkstemp(prefix="ansible")
+    return (out_redir_name, err_redir_name)
+
+
+def cleanup_redirection_tempfiles(out_name, err_name):
+    get_redirected_output(out_name)
+    get_redirected_output(err_name)
+
+
+def get_redirected_output(path_name):
+    output = []
     with open(path_name, 'r') as fd:
         for line in fd:
             # strip terminal format/color chars
             new_line = re.sub(r'\x1b\[.+m', '', line.encode('ascii'))
-            full_stdout += new_line
-            if new_line.strip():
-                # Assuming last line contains the error message
-                last_line = new_line.strip().encode('utf-8')
+            output.append(new_line)
     fd.close()
     os.remove(path_name)
-    return full_stdout, last_line
+    return output
+
+
+def attempt_extract_errors(exc_str, stdout, stderr):
+    errors = [l.strip() for l in stderr if l.strip().startswith('ERROR:')]
+    errors.extend([l.strip() for l in stdout if l.strip().startswith('ERROR:')])
+
+    warnings = [l.strip() for l in stderr if l.strip().startswith('WARNING:')]
+    warnings.extend([l.strip() for l in stdout if l.strip().startswith('WARNING:')])
+
+    # assume either the exception body (if present) or the last warning was the 'most'
+    # fatal.
+
+    if exc_str.strip():
+        msg = exc_str.strip()
+    elif errors:
+        msg = errors[-1].encode('utf-8')
+    else:
+        msg = 'unknown cause'
+
+    return {
+        'warnings': [w.encode('utf-8') for w in warnings],
+        'errors': [e.encode('utf-8') for e in errors],
+        'msg': msg,
+        'module_stderr': ''.join(stderr),
+        'module_stdout': ''.join(stdout)
+    }
+
+
+def get_failure_info(exc, out_name, err_name=None, msg_format='%s'):
+    if err_name is None:
+        stderr = []
+    else:
+        stderr = get_redirected_output(err_name)
+    stdout = get_redirected_output(out_name)
+
+    reason = attempt_extract_errors(str(exc), stdout, stderr)
+    reason['msg'] = msg_format % reason['msg']
+    return reason
+
 
 class ContainerManager(DockerBaseClass):
 
@@ -565,7 +623,8 @@ def __init__(self, client):
             self.options[u'--file'] = self.files
 
         if not HAS_COMPOSE:
-            self.client.fail("Unable to load docker-compose. Try `pip install docker-compose`. Error: %s" % HAS_COMPOSE_EXC)
+            self.client.fail("Unable to load docker-compose. Try `pip install docker-compose`. Error: %s" %
+                             HAS_COMPOSE_EXC)
 
         if LooseVersion(compose_version) < LooseVersion(MINIMUM_COMPOSE_VERSION):
             self.client.fail("Found docker-compose version %s. Minimum required version is %s. "
@@ -682,25 +741,26 @@ def cmd_up(self):
                     result['actions'].append(result_action)
 
         if not self.check_mode and result['changed']:
-            _, fd_name = tempfile.mkstemp(prefix="ansible")
+            out_redir_name, err_redir_name = make_redirection_tempfiles()
             try:
-                with stdout_redirector(fd_name):
-                    do_build = build_action_from_opts(up_options)
-                    self.log('Setting do_build to %s' % do_build)
-                    self.project.up(
-                        service_names=service_names,
-                        start_deps=start_deps,
-                        strategy=converge,
-                        do_build=do_build,
-                        detached=detached,
-                        remove_orphans=self.remove_orphans,
-                        timeout=self.timeout)
+                with stdout_redirector(out_redir_name):
+                    with stderr_redirector(err_redir_name):
+                        do_build = build_action_from_opts(up_options)
+                        self.log('Setting do_build to %s' % do_build)
+                        self.project.up(
+                            service_names=service_names,
+                            start_deps=start_deps,
+                            strategy=converge,
+                            do_build=do_build,
+                            detached=detached,
+                            remove_orphans=self.remove_orphans,
+                            timeout=self.timeout)
             except Exception as exc:
-                full_stdout, last_line= get_stdout(fd_name)
-                self.client.module.fail_json(msg="Error starting project %s" % str(exc), module_stderr=last_line,
-                                             module_stdout=full_stdout)
+                fail_reason = get_failure_info(exc, out_redir_name, err_redir_name,
+                                               msg_format="Error starting project %s")
+                self.client.module.fail_json(**fail_reason)
             else:
-                get_stdout(fd_name)
+                cleanup_redirection_tempfiles(out_redir_name, err_redir_name)
 
         if self.stopped:
             stop_output = self.cmd_stop(service_names)
@@ -903,16 +963,17 @@ def cmd_stop(self, service_names):
                     ))
                 result['actions'].append(service_res)
         if not self.check_mode and result['changed']:
-            _, fd_name = tempfile.mkstemp(prefix="ansible")
+            out_redir_name, err_redir_name = make_redirection_tempfiles()
             try:
-                with stdout_redirector(fd_name):
-                    self.project.stop(service_names=service_names, timeout=self.timeout)
+                with stdout_redirector(out_redir_name):
+                    with stderr_redirector(err_redir_name):
+                        self.project.stop(service_names=service_names, timeout=self.timeout)
             except Exception as exc:
-                full_stdout, last_line = get_stdout(fd_name)
-                self.client.module.fail_json(msg="Error stopping project %s" % str(exc), module_stderr=last_line,
-                                             module_stdout=full_stdout)
+                fail_reason = get_failure_info(exc, out_redir_name, err_redir_name,
+                                               msg_format="Error stopping project %s")
+                self.client.module.fail_json(**fail_reason)
             else:
-                get_stdout(fd_name)
+                cleanup_redirection_tempfiles(out_redir_name, err_redir_name)
         return result
 
     def cmd_restart(self, service_names):
@@ -937,16 +998,17 @@ def cmd_restart(self, service_names):
                 result['actions'].append(service_res)
 
         if not self.check_mode and result['changed']:
-            _, fd_name = tempfile.mkstemp(prefix="ansible")
+            out_redir_name, err_redir_name = make_redirection_tempfiles()
             try:
-                with stdout_redirector(fd_name):
-                    self.project.restart(service_names=service_names, timeout=self.timeout)
+                with stdout_redirector(out_redir_name):
+                    with stderr_redirector(err_redir_name):
+                        self.project.restart(service_names=service_names, timeout=self.timeout)
             except Exception as exc:
-                full_stdout, last_line = get_stdout(fd_name)
-                self.client.module.fail_json(msg="Error restarting project %s" % str(exc), module_stderr=last_line,
-                                             module_stdout=full_stdout)
+                fail_reason = get_failure_info(exc, out_redir_name, err_redir_name,
+                                               msg_format="Error restarting project %s")
+                self.client.module.fail_json(**fail_reason)
             else:
-                get_stdout(fd_name)
+                cleanup_redirection_tempfiles(out_redir_name, err_redir_name)
         return result
 
     def cmd_scale(self):
@@ -981,7 +1043,7 @@ def main():
         state=dict(type='str', choices=['absent', 'present'], default='present'),
         definition=dict(type='dict'),
         hostname_check=dict(type='bool', default=False),
-        recreate=dict(type='str', choices=['always','never','smart'], default='smart'),
+        recreate=dict(type='str', choices=['always', 'never', 'smart'], default='smart'),
         build=dict(type='bool', default=False),
         remove_images=dict(type='str', choices=['all', 'local']),
         remove_volumes=dict(type='bool', default=False),


Test patch:
diff --git a/test/sanity/pep8/legacy-files.txt b/test/sanity/pep8/legacy-files.txt
--- a/test/sanity/pep8/legacy-files.txt
+++ b/test/sanity/pep8/legacy-files.txt
@@ -108,7 +108,6 @@ lib/ansible/modules/cloud/docker/docker_image.py
 lib/ansible/modules/cloud/docker/docker_image_facts.py
 lib/ansible/modules/cloud/docker/docker_login.py
 lib/ansible/modules/cloud/docker/docker_network.py
-lib/ansible/modules/cloud/docker/docker_service.py
 lib/ansible/modules/cloud/google/gc_storage.py
 lib/ansible/modules/cloud/google/gcdns_record.py
 lib/ansible/modules/cloud/google/gcdns_zone.py