Repository: ansible/ansible-runner

Issue:
Unicode in environment variables break ansible-runner 
We run ansible-runner in Docker containers in Gitlab CI.
ansible-runner breaks when a user with Umlauts in his name starts the runner, as the user's name (eg. `Müller`) is saved as environment variable.

```console
[root@af4df578c2e7 /]# unset test
[root@af4df578c2e7 /]# ansible-runner -p test.yml run .
ERROR! the playbook: test.yml could not be found
[root@af4df578c2e7 /]# export test=Müller
[root@af4df578c2e7 /]# ansible-runner -p test.yml run .
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible_runner/__main__.py", line 329, in main
    res = run(**run_options)
  File "/usr/lib/python2.7/site-packages/ansible_runner/interface.py", line 162, in run
    r.run()
  File "/usr/lib/python2.7/site-packages/ansible_runner/runner.py", line 112, in run
    'env': self.config.env}, f, ensure_ascii=False
  File "/usr/lib64/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
  File "/usr/lib64/python2.7/codecs.py", line 691, in write
    return self.writer.write(data)
  File "/usr/lib64/python2.7/codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)
[root@af4df578c2e7 /]# ansible-runner --version
1.3.4
```


## related issues

similar to #234

## link to breaking source

https://github.com/ansible/ansible-runner/blob/1696ec2075f96b57bc9099cf018ed8d0d574aed6/ansible_runner/runner.py#L109-L113

Discussion:
ansible-zuul[bot]: Build failed.

- [ansible-tox-py36 ](https://logs.zuul.ansible.com/15/315/805ce3455baf9663d19bc0ec7b85e688b9369789/check/ansible-tox-py36/1e77548/) : SUCCESS in 7m 52s
- [ansible-tox-py37 ](https://logs.zuul.ansible.com/15/315/805ce3455baf9663d19bc0ec7b85e688b9369789/check/ansible-tox-py37/35a4067/) : SUCCESS in 7m 55s
- [ansible-build-python-tarball ](https://logs.zuul.ansible.com/15/315/805ce3455baf9663d19bc0ec7b85e688b9369789/check/ansible-build-python-tarball/132a56d/) : SUCCESS in 2m 37s
- [ansible-tox-linters ](https://logs.zuul.ansible.com/15/315/805ce3455baf9663d19bc0ec7b85e688b9369789/check/ansible-tox-linters/2788102/) : SUCCESS in 4m 50s
- [ansible-tox-py27 ](https://logs.zuul.ansible.com/15/315/805ce3455baf9663d19bc0ec7b85e688b9369789/check/ansible-tox-py27/dc98d32/) : FAILURE in 6m 56s

ansible-zuul[bot]: Build failed.

- [ansible-tox-py36 ](https://logs.zuul.ansible.com/15/315/af88addad082828a67932ea2f5e05c0f1043cf55/check/ansible-tox-py36/b45e3de/) : SUCCESS in 7m 35s
- [ansible-tox-py37 ](https://logs.zuul.ansible.com/15/315/af88addad082828a67932ea2f5e05c0f1043cf55/check/ansible-tox-py37/82fbe5f/) : SUCCESS in 7m 39s
- [ansible-build-python-tarball ](https://logs.zuul.ansible.com/15/315/af88addad082828a67932ea2f5e05c0f1043cf55/check/ansible-build-python-tarball/db937bb/) : SUCCESS in 2m 37s
- [ansible-tox-linters ](https://logs.zuul.ansible.com/15/315/af88addad082828a67932ea2f5e05c0f1043cf55/check/ansible-tox-linters/b365b7f/) : SUCCESS in 4m 57s
- [ansible-tox-py27 ](https://logs.zuul.ansible.com/15/315/af88addad082828a67932ea2f5e05c0f1043cf55/check/ansible-tox-py27/3cdf276/) : FAILURE in 6m 56s

matburt: recheck
ansible-zuul[bot]: Build failed.

- [ansible-tox-py36 ](https://logs.zuul.ansible.com/15/315/9f3d1f71ff321aa88d482c5330ed98a8ff49df5f/check/ansible-tox-py36/b61f96a/) : FAILURE in 8m 11s
- [ansible-tox-py37 ](https://logs.zuul.ansible.com/15/315/9f3d1f71ff321aa88d482c5330ed98a8ff49df5f/check/ansible-tox-py37/1e58bda/) : FAILURE in 7m 56s
- [ansible-build-python-tarball ](https://logs.zuul.ansible.com/15/315/9f3d1f71ff321aa88d482c5330ed98a8ff49df5f/check/ansible-build-python-tarball/cd2b118/) : SUCCESS in 3m 00s
- [ansible-tox-linters ](https://logs.zuul.ansible.com/15/315/9f3d1f71ff321aa88d482c5330ed98a8ff49df5f/check/ansible-tox-linters/0cedd54/) : SUCCESS in 5m 04s
- [ansible-tox-py27 ](https://logs.zuul.ansible.com/15/315/9f3d1f71ff321aa88d482c5330ed98a8ff49df5f/check/ansible-tox-py27/bdadcb6/) : FAILURE in 7m 24s

matburt: recheck
matburt: recheck
ansible-zuul[bot]: Build succeeded.

- [ansible-tox-py36 ](https://logs.zuul.ansible.com/15/315/7dd28b19aef3633e216222c7bdbcf92927ed65af/check/ansible-tox-py36/51a9d89/) : SUCCESS in 13m 37s
- [ansible-tox-py37 ](https://logs.zuul.ansible.com/15/315/7dd28b19aef3633e216222c7bdbcf92927ed65af/check/ansible-tox-py37/6c3aed3/) : SUCCESS in 15m 42s
- [ansible-build-python-tarball ](https://logs.zuul.ansible.com/15/315/7dd28b19aef3633e216222c7bdbcf92927ed65af/check/ansible-build-python-tarball/c92a50a/) : SUCCESS in 2m 31s
- [ansible-tox-linters ](https://logs.zuul.ansible.com/15/315/7dd28b19aef3633e216222c7bdbcf92927ed65af/check/ansible-tox-linters/3ac1847/) : SUCCESS in 4m 51s
- [ansible-tox-py27 ](https://logs.zuul.ansible.com/15/315/7dd28b19aef3633e216222c7bdbcf92927ed65af/check/ansible-tox-py27/11ae112/) : SUCCESS in 15m 09s


Patch:
diff --git a/ansible_runner/loader.py b/ansible_runner/loader.py
index 7e358e7..55f9f69 100644
--- a/ansible_runner/loader.py
+++ b/ansible_runner/loader.py
@@ -18,6 +18,7 @@
 #
 import os
 import json
+import codecs
 
 from yaml import safe_load, YAMLError
 from six import string_types
@@ -95,8 +96,7 @@ class ArtifactLoader(object):
         try:
             if not os.path.exists(path):
                 raise ConfigurationError('specified path does not exist %s' % path)
-
-            with open(path) as f:
+            with codecs.open(path, encoding='utf-8') as f:
                 data = f.read()
 
             return data
diff --git a/ansible_runner/runner.py b/ansible_runner/runner.py
index c9d4a89..3b31dd8 100644
--- a/ansible_runner/runner.py
+++ b/ansible_runner/runner.py
@@ -103,7 +103,10 @@ class Runner(object):
                 raise
         os.close(os.open(stdout_filename, os.O_CREAT, stat.S_IRUSR | stat.S_IWUSR))
 
-        command = [a.decode('utf-8') if six.PY2 else a for a in self.config.command]
+        if six.PY2:
+            command = [a.decode('utf-8') for a in self.config.command]
+        else:
+            command = self.config.command
         with codecs.open(command_filename, 'w', encoding='utf-8') as f:
             os.chmod(command_filename, stat.S_IRUSR | stat.S_IWUSR)
             json.dump(
diff --git a/ansible_runner/runner_config.py b/ansible_runner/runner_config.py
index 4c0d7bf..b333a2e 100644
--- a/ansible_runner/runner_config.py
+++ b/ansible_runner/runner_config.py
@@ -220,18 +220,20 @@ class RunnerConfig(object):
         self.expect_passwords[pexpect.TIMEOUT] = None
         self.expect_passwords[pexpect.EOF] = None
 
+        # seed env with existing shell env
+        self.env = os.environ.copy()
+        if self.envvars and isinstance(self.envvars, dict):
+            if six.PY2:
+                self.env.update({k.decode('utf-8'):v.decode('utf-8') for k, v in self.envvars.items()})
+            else:
+                self.env.update({k:v for k, v in self.envvars.items()})
         try:
-            # seed env with existing shell env
-            self.env = os.environ.copy()
             envvars = self.loader.load_file('env/envvars', Mapping)
             if envvars:
-                self.env.update({k:six.text_type(v) for k, v in envvars.items()})
-            if self.envvars and isinstance(self.envvars, dict):
-                self.env.update({k:six.text_type(v) for k, v in self.envvars.items()})
+                self.env.update({six.text_type(k):six.text_type(v) for k, v in envvars.items()})
         except ConfigurationError:
             output.debug("Not loading environment vars")
             # Still need to pass default environment to pexpect
-            self.env = os.environ.copy()
 
         try:
             self.settings = self.loader.load_file('env/settings', Mapping)


Test patch:
diff --git a/test/integration/test_main.py b/test/integration/test_main.py
index 8e3dff7..8416c85 100644
--- a/test/integration/test_main.py
+++ b/test/integration/test_main.py
@@ -1,7 +1,9 @@
+# -*- coding: utf-8 -*-
 from __future__ import print_function
 from ansible_runner.__main__ import main
 
 import os
+import codecs
 import multiprocessing
 import shutil
 import yaml
@@ -207,15 +209,15 @@ def test_role_run_artifacts_dir_abs():
 @pytest.mark.parametrize('envvars', [
     {'msg': 'hi'},
     {
-        'msg': b'\xf0\x98\x90\x9d\xe5\x83\xac\xe2\xb2\x82\xeb\x8d\xb6'.decode('utf-8'),
-        b'\xf0\x98\x90\x9d\xe5\x83\xac\xe2\xb2\x82\xeb\x8d\xb6'.decode('utf-8'): b'\xf0\x98\x90\x9d\xe5\x83\xac\xe2\xb2\x82\xeb\x8d\xb6'.decode('utf-8')
+        'msg': u'utf-8-䉪ቒ칸ⱷ?噂폄蔆㪗輥',
+        u'蔆㪗輥': u'䉪ቒ칸'
     }
 ])
 def test_role_run_env_vars(envvars):
 
     with temp_directory() as temp_dir:
         ensure_directory(os.path.join(temp_dir, 'env'))
-        with open(os.path.join(temp_dir, 'env/envvars'), 'w') as f:
+        with codecs.open(os.path.join(temp_dir, 'env/envvars'), 'w', encoding='utf-8') as f:
             f.write(yaml.dump(envvars))
 
         rc = main(['-r', 'benthomasson.hello_role',
diff --git a/test/integration/test_runner.py b/test/integration/test_runner.py
index 677b1ed..128cea1 100644
--- a/test/integration/test_runner.py
+++ b/test/integration/test_runner.py
@@ -42,6 +42,8 @@ def test_run_command_with_unicode(rc):
     if six.PY2:
         expected = expected.decode('utf-8')
     rc.command = ['echo', '"utf-8-䉪ቒ칸ⱷ?噂폄蔆㪗輥"']
+    rc.envvars = {"䉪ቒ칸": "蔆㪗輥"}
+    rc.prepare_env()
     status, exitcode = Runner(config=rc).run()
     assert status == 'successful'
     assert exitcode == 0
@@ -50,6 +52,7 @@ def test_run_command_with_unicode(rc):
         assert data.get('command') == ['echo', expected]
         assert 'cwd' in data
         assert isinstance(data.get('env'), dict)
+        assert u"䉪ቒ칸" in data.get('env')
 
 
 def test_run_command_finished_callback(rc):
diff --git a/test/unit/test_loader.py b/test/unit/test_loader.py
index ad23150..01aaaa2 100644
--- a/test/unit/test_loader.py
+++ b/test/unit/test_loader.py
@@ -112,7 +112,7 @@ def test_load_file_type_check(loader):
 
 
 def test_get_contents_ok(loader):
-    with patch('ansible_runner.loader.open') as mock_open:
+    with patch('codecs.open') as mock_open:
         handler = BytesIO()
         handler.write(b"test string")
         handler.seek(0)