Repository: ansible/ansible

Issue:
Docker connection module: Set Popen to return str rather than bytes object
Update to docker connection module to return str object to handle communicate() properly. Using Python3 raises exeptions on bytes vs str

- Bugfix Pull Request




Discussion:
@djazo Greetings! Thanks for taking the time to open this pullrequest. In order for the community to handle your pullrequest effectively, we need a bit more information. 

Here are the items we could not find in your description:
- issue type

Please set the description of this pullrequest with this template:
https://raw.githubusercontent.com/ansible/ansible/devel/.github/PULL_REQUEST_TEMPLATE.md

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: issue_missing_data --->
Connection plugins deal in byte strings so this is not the way to fix your issue.  Can you show where things are breaking?  If it's within the connection plugin we probably need to convert text into bytes.  If it's outside of it, then we probably need to convert the bytes into text there.
Hi!
All comparisons fail due to str vs bytes.
For example:
LooseVersion(docker_version) < LooseVersion('1.3')
_sanitize_version regexp fails
cmd_output.split('\n') fails

That's because hyphens make str object, not bytes.
So the correct way is LooseVersion(b'1.3'), cmd_output.split(b'\n')  etc ?

@djazo yes, probably.
@djazo Your branch does not contain a shippable.yml file. Please [rebase your branch](http://docs.ansible.com/ansible/dev_guide/developing_rebasing.html) to trigger running of current tests.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: no_shippable_yaml --->
I'm encountering the following fatal error when running my playbook using `ansible-playbook-3` (version 2.3.1.0, python version = 3.5.2) instead of `ansible-playbook`. Can someone confirm it's the same problem being addressed by this PR?

```
...

TASK [Gathering Facts] ******************************************************************************************************************************
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/usr/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 472, in _execute
    self._connection = self._get_connection(variables=variables, templar=templar)
  File "/usr/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 719, in _get_connection
    connection = self._shared_loader_obj.connection_loader.get(conn_type, self._play_context, self._new_stdin)
  File "/usr/lib/python3.5/site-packages/ansible/plugins/__init__.py", line 373, in get
    obj = obj(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/ansible/plugins/connection/docker.py", line 78, in __init__
    docker_version = self._get_docker_version()
  File "/usr/lib/python3.5/site-packages/ansible/plugins/connection/docker.py", line 138, in _get_docker_version
    for line in cmd_output.split('\n'):
TypeError: a bytes-like object is required, not 'str'
fatal: [34e82054a6625e54a9bed112ca42a46191ab1a3e00f4de4f7b9c070bbf53a811]: FAILED! => {
    "failed": true,
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}
        to retry, use: ...
```
@mmathesius Seems to be the same issue. Might be good time to update PR. =)

Resolving patch:
diff --git a/lib/ansible/plugins/connection/docker.py b/lib/ansible/plugins/connection/docker.py
--- a/lib/ansible/plugins/connection/docker.py
+++ b/lib/ansible/plugins/connection/docker.py
@@ -51,7 +51,7 @@
 import ansible.constants as C
 from ansible.errors import AnsibleError, AnsibleFileNotFound
 from ansible.module_utils.six.moves import shlex_quote
-from ansible.module_utils._text import to_bytes, to_native
+from ansible.module_utils._text import to_bytes, to_native, to_text
 from ansible.plugins.connection import ConnectionBase, BUFSIZE
 
 
@@ -89,7 +89,7 @@ def __init__(self, play_context, new_stdin, *args, **kwargs):
                 raise AnsibleError("docker command not found in PATH")
 
         docker_version = self._get_docker_version()
-        if LooseVersion(docker_version) < LooseVersion('1.3'):
+        if LooseVersion(docker_version) < LooseVersion(u'1.3'):
             raise AnsibleError('docker connection type requires docker 1.3 or higher')
 
         # The remote user we will request from docker (if supported)
@@ -98,7 +98,7 @@ def __init__(self, play_context, new_stdin, *args, **kwargs):
         self.actual_user = None
 
         if self._play_context.remote_user is not None:
-            if LooseVersion(docker_version) >= LooseVersion('1.7'):
+            if LooseVersion(docker_version) >= LooseVersion(u'1.7'):
                 # Support for specifying the exec user was added in docker 1.7
                 self.remote_user = self._play_context.remote_user
                 self.actual_user = self.remote_user
@@ -106,8 +106,8 @@ def __init__(self, play_context, new_stdin, *args, **kwargs):
                 self.actual_user = self._get_docker_remote_user()
 
                 if self.actual_user != self._play_context.remote_user:
-                    display.warning('docker {0} does not support remote_user, using container default: {1}'
-                                    .format(docker_version, self.actual_user or '?'))
+                    display.warning(u'docker {0} does not support remote_user, using container default: {1}'
+                                    .format(docker_version, self.actual_user or u'?'))
         elif self._display.verbosity > 2:
             # Since we're not setting the actual_user, look it up so we have it for logging later
             # Only do this if display verbosity is high enough that we'll need the value
@@ -116,7 +116,7 @@ def __init__(self, play_context, new_stdin, *args, **kwargs):
 
     @staticmethod
     def _sanitize_version(version):
-        return re.sub('[^0-9a-zA-Z\.]', '', version)
+        return re.sub(u'[^0-9a-zA-Z\.]', u'', version)
 
     def _old_docker_version(self):
         cmd_args = []
@@ -148,15 +148,15 @@ def _get_docker_version(self):
 
         cmd, cmd_output, err, returncode = self._old_docker_version()
         if returncode == 0:
-            for line in cmd_output.split('\n'):
-                if line.startswith('Server version:'):  # old docker versions
+            for line in to_text(cmd_output, errors='surrogate_or_strict').split(u'\n'):
+                if line.startswith(u'Server version:'):  # old docker versions
                     return self._sanitize_version(line.split()[2])
 
         cmd, cmd_output, err, returncode = self._new_docker_version()
         if returncode:
-            raise AnsibleError('Docker version check (%s) failed: %s' % (cmd, err))
+            raise AnsibleError('Docker version check (%s) failed: %s' % (to_native(cmd), to_native(err)))
 
-        return self._sanitize_version(cmd_output)
+        return self._sanitize_version(to_text(cmd_output, errors='surrogate_or_strict'))
 
     def _get_docker_remote_user(self):
         """ Get the default user configured in the docker container """
@@ -164,13 +164,14 @@ def _get_docker_remote_user(self):
                              stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
         out, err = p.communicate()
+        out = to_text(out, errors='surrogate_or_strict')
 
         if p.returncode != 0:
-            display.warning('unable to retrieve default user from docker container: %s' % out + err)
+            display.warning(u'unable to retrieve default user from docker container: %s %s' % (out, to_text(err)))
             return None
 
         # The default exec user is root, unless it was changed in the Dockerfile with USER
-        return out.strip() or 'root'
+        return out.strip() or u'root'
 
     def _build_exec_cmd(self, cmd):
         """ Build the local docker exec command to run cmd on remote_host
@@ -184,13 +185,13 @@ def _build_exec_cmd(self, cmd):
         if self._play_context.docker_extra_args:
             local_cmd += self._play_context.docker_extra_args.split(' ')
 
-        local_cmd += ['exec']
+        local_cmd += [b'exec']
 
         if self.remote_user is not None:
-            local_cmd += ['-u', self.remote_user]
+            local_cmd += [b'-u', self.remote_user]
 
         # -i is needed to keep stdin open which allows pipelining to work
-        local_cmd += ['-i', self._play_context.remote_addr] + cmd
+        local_cmd += [b'-i', self._play_context.remote_addr] + cmd
 
         return local_cmd
 
@@ -199,7 +200,7 @@ def _connect(self, port=None):
         super(Connection, self)._connect()
         if not self._connected:
             display.vvv(u"ESTABLISH DOCKER CONNECTION FOR USER: {0}".format(
-                self.actual_user or '?'), host=self._play_context.remote_addr
+                self.actual_user or u'?'), host=self._play_context.remote_addr
             )
             self._connected = True
 
@@ -239,7 +240,7 @@ def put_file(self, in_path, out_path):
         out_path = self._prefix_login_path(out_path)
         if not os.path.exists(to_bytes(in_path, errors='surrogate_or_strict')):
             raise AnsibleFileNotFound(
-                "file or module does not exist: %s" % in_path)
+                "file or module does not exist: %s" % to_native(in_path))
 
         out_path = shlex_quote(out_path)
         # Older docker doesn't have native support for copying files into
@@ -257,7 +258,8 @@ def put_file(self, in_path, out_path):
             stdout, stderr = p.communicate()
 
             if p.returncode != 0:
-                raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" % (in_path, out_path, stdout, stderr))
+                raise AnsibleError("failed to transfer file %s to %s:\n%s\n%s" %
+                                   (to_native(in_path), to_native(out_path), to_native(stdout), to_native(stderr)))
 
     def fetch_file(self, in_path, out_path):
         """ Fetch a file from container to local. """


Test patch: