Repository: google/clusterfuzz

Issue:
Issue with non-ascii environment variables
I have some non-ascii characters in my PS1, and this causes problems with some of the code in #1336 .  For example, running some integration tests, I see:

```
Traceback (most recent call last):
  File "~/clusterfuzz/src/python/tests/core/bot/fuzzers/libFuzzer/engine_test.py", line 1017, in test_qemu_logs_returned_on_error
    build_manager.setup_build()
  File "~/clusterfuzz/src/python/build_management/build_manager.py", line 1418, in setup_build
    return setup_trunk_build(bucket_paths, target_weights=target_weights)
  File "~/clusterfuzz/src/python/build_management/build_manager.py", line 1212, in setup_trunk_build
    target_weights=target_weights)
  File "~/clusterfuzz/src/python/build_management/build_manager.py", line 1256, in setup_regular_build
    if build.setup():
  File "~/clusterfuzz/src/python/build_management/build_manager.py", line 783, in setup
    fuchsia.device.initial_qemu_setup()
  File "~/clusterfuzz/src/python/platforms/fuchsia/device.py", line 99, in initial_qemu_setup
    extend_fvm(qemu_vars['fuchsia_resources_dir'], qemu_vars['drive_path'])
  File "~/clusterfuzz/src/python/platforms/fuchsia/device.py", line 228, in extend_fvm
    result = process.run_and_wait()
  File "~/clusterfuzz/src/python/system/new_process.py", line 344, in run_and_wait
    **popen_args)
  File "~/clusterfuzz/src/python/system/new_process.py", line 283, in run
    for k, v in env.items()
  File "~/clusterfuzz/src/python/system/new_process.py", line 283, in <dictcomp>
    for k, v in env.items()
  File "~/clusterfuzz/src/python/base/utils.py", line 949, in newstr_to_native_str
    return future_utils.native(str(s)).encode()
  File "~/clusterfuzz/ENV/local/lib/python2.7/site-packages/future/types/newstr.py", line 102, in __new__
    return super(newstr, cls).__new__(cls, value)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 6: ordinal not in range(128)
```

I was able to fix this by simply removing the conversion of environment variables in https://github.com/google/clusterfuzz/blob/25703fe2372817765e/src/python/system/new_process.py#L278-L284 , but I'm not sure this is a proper fix (I wasn't able to find the subprocess type checks mentioned in the TODO).  Happy to send a PR with this, but thought I'd ask for your input first.  Thanks!


Discussion:
I don't think removing the conversions is the correct fix here, as this was fixing another exception. @mbarbella-chromium maybe we can just add 'utf-8' to the str() call of: 

```python
# TODO(mbarbella): Delete this once fully migrated to Python 3.
def newstr_to_native_str(s):
  return future_utils.native(str(s, encoding='utf-8')).encode()  
```

While we're here, should .encode() be called under Python 3? As is, this funtions will not return the native "str" on Python 3 and return a bytes instead.
> I don't think removing the conversions is the correct fix here, as this was fixing another exception. @mbarbella-chromium maybe we can just add 'utf-8' to the str() call of:

That seems reasonable to me.

> While we're here, should .encode() be called under Python 3? As is, this funtions will not return the native "str" on Python 3 and return a bytes instead.

future_utils.native actually returns a unicode (since it's the native type backing newstr), so this should be safe.

Resolving patch:
diff --git a/src/python/base/utils.py b/src/python/base/utils.py
--- a/src/python/base/utils.py
+++ b/src/python/base/utils.py
@@ -946,4 +946,4 @@ def cpu_count():
 
 # TODO(mbarbella): Delete this once fully migrated to Python 3.
 def newstr_to_native_str(s):
-  return future_utils.native(str(s)).encode()
+  return future_utils.native(str(s, encoding='utf-8')).encode()
diff --git a/src/python/system/new_process.py b/src/python/system/new_process.py
--- a/src/python/system/new_process.py
+++ b/src/python/system/new_process.py
@@ -278,10 +278,17 @@ def run(self,
     # TODO(mbarbella): Remove this after the Python 3 conversion. Subprocess
     # contains some explicit type checks, causing errors when newstrs are used.
     if env:
-      env = {
-          utils.newstr_to_native_str(k): utils.newstr_to_native_str(v)
-          for k, v in env.items()
-      }
+      new_env = {}
+      for k, v in env.items():
+        try:
+          new_env[utils.decode_to_unicode(k)] = utils.newstr_to_native_str(v)
+        except UnicodeDecodeError:
+          # If any non-unicode keys or values are already set in the
+          # environment, we know those aren't of type newstr (since they could
+          # not have been decoded in the first place).
+          new_env[k] = v
+
+      env = new_env
 
     return ChildProcess(
         subprocess.Popen(


Test patch: