Repository: buildbot/buildbot

Issue:
unicode encoding problem in windows from version 0.9.13 to 0.9.15post1
If you try to excute a command containing a `unicode` to a **Windows** worker from a **Windows** master , you will get the error message like:
```python
 error in RunProcess._startCommand
Traceback (most recent call last):
  File "d:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 446, in start
    self._startCommand()
  File "d:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 572, in _startCommand
    usePTY=self.usePTY)
  File "d:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 603, in _spawnProcess
    path, usePTY=usePTY)
  File "d:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 619, in _spawnAsBatch
    tf.write(win32_batch_quote(self.command))
  File "d:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 70, in win32_batch_quote
    return ' '.join(map(escape_arg, cmd_list))
  File "d:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 63, in escape_arg
    arg = bytes2unicode(arg, unicode_encoding)
  File "d:\buildbot\sandbox\lib\site-packages\buildbot_worker\compat.py", line 87, in bytes2unicode
    return text_type(x, encoding, errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb3 in position 49: invalid start byte
```
I found that in _buildbot-worker/runprocess.py:323_:
```python
        def to_bytes(cmd):
            if isinstance(cmd, (tuple, list)):
                for i, a in enumerate(cmd):
                    if isinstance(a, text_type):
                        cmd[i] = a.encode(self.builder.unicode_encoding)
            elif isinstance(cmd, text_type):
                cmd = cmd.encode(self.builder.unicode_encoding)
            return cmd
```
where `self.builder.unicode_encoding` is set to 'mbcs' which results in a wrong encoded `str` for 
 `text_type` in _compat.py:87_ to encode.


Discussion:
This is probably related to #3797
@pmisik No you are wrong, it is not related to #3797
By the way, 0.9.11 doesn't have this problem because I can send command in gbk encoded str format.
0.9.15post1 still has this problem. 
Assume that we have a shell command list consisting of `unicode`: `[u'echo', u'\u6211']`. When they come to the worker, firstly, these commands are encoded to `bytes`.
```python
# runprocess.py:RunProcess.__init__:332
self.command = to_bytes(util.Obfuscated.get_real(command))
```
```python
def to_bytes(cmd):
    if isinstance(cmd, (tuple, list)):
        for i, a in enumerate(cmd):
            if isinstance(a, text_type):
                cmd[i] = a.encode(self.builder.unicode_encoding)
    elif isinstance(cmd, text_type):
        cmd = cmd.encode(self.builder.unicode_encoding)
    return cmd
```
As you can see, `to_bytes` encodes our `unicode` in encoding of `self.builder.unicode_encoding` whose default value is `mbcs` or `gbk` and is set by the below code.
```python
# base.py:238
class BotBase(service.MultiService):

    """I represent the worker-side bot."""
    name = "bot"
    WorkerForBuilder = WorkerForBuilderBase

    def __init__(self, basedir, unicode_encoding=None):
        service.MultiService.__init__(self)
        self.basedir = basedir
        self.numcpus = None
        self.unicode_encoding = unicode_encoding or sys.getfilesystemencoding(
        ) or 'ascii'
        self.builders = {}
```
Then, in `runprocess.py:619:win32_batch_quote`, our `mbcs` encoded `bytes` are decoded by `bytes2unicode` using `utf8` which results in a `UnicodeDecodeError`.
```python
class RunProcess(object):
    # ...
    def _spawnAsBatch(self, processProtocol, executable, args, env,
                      path, usePTY):
        # ...
            tf.write(win32_batch_quote(self.command))
```
```python
def win32_batch_quote(cmd_list, unicode_encoding='utf-8'):
    # Quote cmd_list to a string that is suitable for inclusion in a
    # Windows batch file. This is not quite the same as quoting it for the
    # shell, as cmd.exe doesn't support the %% escape in interactive mode.
    def escape_arg(arg):
        arg = bytes2unicode(arg, unicode_encoding)
        arg = quoteArguments([arg])
        # escape shell special characters
        arg = re.sub(r'[@()^"<>&|]', r'^\g<0>', arg)
        # prevent variable expansion
        return arg.replace('%', '%%')

    return ' '.join(map(escape_arg, cmd_list))
```
What if we change `tf.write(win32_batch_quote(self.command))` to `tf.write(win32_batch_quote(self.command, self.builder.unicode_encoding))`? This brings a new problem:
```python
echo 我
 in dir F:\buildbot\unicode-bug-master\workers\pass\runtests\build (timeout 1200 secs)
 watching logfiles {}
 argv: ['echo', '\xce\xd2']
 using PTY: False
error in RunProcess._startCommand
Traceback (most recent call last):
  File "f:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 446, in start
    self._startCommand()
  File "f:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 572, in _startCommand
    usePTY=self.usePTY)
  File "f:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 603, in _spawnProcess
    path, usePTY=usePTY)
  File "f:\buildbot\sandbox\lib\site-packages\buildbot_worker\runprocess.py", line 619, in _spawnAsBatch
    tf.write(win32_batch_quote(self.command, self.builder.unicode_encoding))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u6211' in position 5: ordinal not in range(128)
```
A simple illustration is as below:
![image](https://user-images.githubusercontent.com/19603852/35966141-f3fe6f50-0cf7-11e8-855c-58e8a849b2af.png)

This seems to be somthing wrong related to the `NamedTemporaryFile`.
@tardyp 

A reproduceable `master.cfg`:
```python
# -*- python -*-
# ex: set filetype=python:

from buildbot.plugins import *

# This is a sample buildmaster config file. It must be installed as
# 'master.cfg' in your buildmaster's base directory.

# This is the dictionary that the buildmaster pays attention to. We also use
# a shorter alias to save typing.
c = BuildmasterConfig = {}

####### WORKERS

# The 'workers' list defines the set of recognized workers. Each element is
# a Worker object, specifying a unique worker name and password.  The same
# worker name and password must be configured on the worker.
c['workers'] = [worker.LocalWorker("example-worker", "pass")]

# 'protocols' contains information about protocols which master will use for
# communicating with workers. You must define at least 'port' option that workers
# could connect to your master with this protocol.
# 'port' must match the value configured into the workers (with their
# --master option)
c['protocols'] = {'pb': {'port': 9989}}

####### CHANGESOURCES

# the 'change_source' setting tells the buildmaster how it should find out
# about source code changes.  Here we point to the buildbot version of a python hello-world project.

c['change_source'] = []

####### SCHEDULERS

# Configure the Schedulers, which decide how to react to incoming changes.  In this
# case, just kick off a 'runtests' build

c['schedulers'] = []
c['schedulers'].append(schedulers.ForceScheduler(
                            name="force",
                            builderNames=["runtests"]))

####### BUILDERS

# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
# what steps, and which workers can execute them.  Note that any particular build will
# only take place on one worker.

factory = util.BuildFactory()
# run the tests (note that this will require that 'trial' is installed)
factory.addStep(steps.ShellCommand(command=["echo",u"\u6211"],
                                    logEnviron=False))

c['builders'] = []
c['builders'].append(
    util.BuilderConfig(name="runtests",
      workernames=["example-worker"],
      factory=factory))

####### BUILDBOT SERVICES

# 'services' is a list of BuildbotService items like reporter targets. The
# status of each build will be pushed to these targets. buildbot/reporters/*.py
# has a variety to choose from, like IRC bots.

c['services'] = []

####### PROJECT IDENTITY

# the 'title' string will appear at the top of this buildbot installation's
# home pages (linked to the 'titleURL').

c['title'] = "Hello World CI"
c['titleURL'] = "https://buildbot.github.io/hello-world/"

# the 'buildbotURL' string should point to the location where the buildbot's
# internal web server is visible. This typically uses the port number set in
# the 'www' entry below, but with an externally-visible host name which the
# buildbot cannot figure out without some help.

c['buildbotURL'] = "http://localhost:8010/"

# minimalistic config to activate new web UI
c['www'] = dict(port=8010,
                plugins=dict(waterfall_view={}, console_view={}, grid_view={}))

####### DB URL

c['db'] = {
    # This specifies what database buildbot uses to store its state.  You can leave
    # this at its default for all but the largest installations.
    'db_url' : "sqlite:///state.sqlite",
}

```
  
This bug is caused by the following code in `runprocess.py:58`:
```python
def win32_batch_quote(cmd_list, unicode_encoding='utf-8'):
    # Quote cmd_list to a string that is suitable for inclusion in a
    # Windows batch file. This is not quite the same as quoting it for the
    # shell, as cmd.exe doesn't support the %% escape in interactive mode.
    def escape_arg(arg):
        arg = bytes2unicode(arg, unicode_encoding) # SHOULD BE unicode2bytes
        arg = quoteArguments([arg])
        # escape shell special characters
        arg = re.sub(r'[@()^"<>&|]', r'^\g<0>', arg)
        # prevent variable expansion
        return arg.replace('%', '%%')

    return ' '.join(map(escape_arg, cmd_list))
```
`bytes2unicode` should be replaced by `unicode2bytes` because `tf.write()`(runprocess.py:617) requires native string other than a unicode in Python2. You can try the following code to confirm that:
```python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from tempfile import NamedTemporaryFile
tf = NamedTemporaryFile(mode='w+', dir='.', suffix=".bat",
                                delete=False)
tf.write(u'\u6211')
tf.close()
```
The diff between 0.9.11 and 0.9.15post1 is as below:
![image](https://user-images.githubusercontent.com/19603852/35970731-6a8033da-0d06-11e8-9857-7e0c038298ba.png)

In Python2, `bytes2NativeString` returns `x` directly if `x` is a `str` object while `bytes2unicode` doesn't which causes a `UnicodeDecodeError` because `self.command` consists of `str` objects encoded by `mbcs` in `to_bytes` function.
```python
def bytes2NativeString(x, encoding='utf-8'):
    """
    Convert C{bytes} to a native C{str}.

    On Python 3 and higher, str and bytes
    are not equivalent.  In this case, decode
    the bytes, and return a native string.

    On Python 2 and lower, str and bytes
    are equivalent.  In this case, just
    just return the native string.

    @param x: a string of type C{bytes}
    @param encoding: an optional codec, default: 'utf-8'
    @return: a string of type C{str}
    """
    if isinstance(x, bytes) and str != bytes:
        return x.decode(encoding)
    return x
```

Resolving patch:
diff --git a/worker/buildbot_worker/runprocess.py b/worker/buildbot_worker/runprocess.py
--- a/worker/buildbot_worker/runprocess.py
+++ b/worker/buildbot_worker/runprocess.py
@@ -20,6 +20,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 from future.builtins import range
+from future.utils import PY3
 from future.utils import iteritems
 from future.utils import string_types
 from future.utils import text_type
@@ -47,6 +48,7 @@
 from twisted.python.win32 import quoteArguments
 
 from buildbot_worker import util
+from buildbot_worker.compat import bytes2NativeString
 from buildbot_worker.compat import bytes2unicode
 from buildbot_worker.compat import unicode2bytes
 from buildbot_worker.exceptions import AbandonChain
@@ -60,7 +62,7 @@ def win32_batch_quote(cmd_list, unicode_encoding='utf-8'):
     # Windows batch file. This is not quite the same as quoting it for the
     # shell, as cmd.exe doesn't support the %% escape in interactive mode.
     def escape_arg(arg):
-        arg = bytes2unicode(arg, unicode_encoding)
+        arg = bytes2NativeString(arg, unicode_encoding)
         arg = quoteArguments([arg])
         # escape shell special characters
         arg = re.sub(r'[@()^"<>&|]', r'^\g<0>', arg)
@@ -609,14 +611,22 @@ def _spawnAsBatch(self, processProtocol, executable, args, env,
         """A cheat that routes around the impedance mismatch between
         twisted and cmd.exe with respect to escaping quotes"""
 
-        tf = NamedTemporaryFile(mode='w+', dir='.', suffix=".bat",
-                                delete=False)
+        # NamedTemporaryFile differs in PY2 and PY3.
+        # In PY2, it needs encoded str and its encoding cannot be specified.
+        # In PY3, it needs str which is unicode and its encoding can be specified.
+        if PY3:
+            tf = NamedTemporaryFile(mode='w+', dir='.', suffix=".bat",
+                                    delete=False, encoding=self.builder.unicode_encoding)
+        else:
+            tf = NamedTemporaryFile(mode='w+', dir='.', suffix=".bat",
+                                    delete=False)
+
         # echo off hides this cheat from the log files.
         tf.write(u"@echo off\n")
         if isinstance(self.command, (string_types, bytes)):
-            tf.write(bytes2unicode(self.command, self.builder.unicode_encoding))
+            tf.write(bytes2NativeString(self.command, self.builder.unicode_encoding))
         else:
-            tf.write(win32_batch_quote(self.command))
+            tf.write(win32_batch_quote(self.command, self.builder.unicode_encoding))
         tf.close()
 
         argv = os.environ['COMSPEC'].split()  # allow %COMSPEC% to have args


Test patch:
diff --git a/worker/buildbot_worker/test/unit/test_runprocess.py b/worker/buildbot_worker/test/unit/test_runprocess.py
--- a/worker/buildbot_worker/test/unit/test_runprocess.py
+++ b/worker/buildbot_worker/test/unit/test_runprocess.py
@@ -508,6 +508,12 @@ def test_spawnAsBatchCommandString(self):
     def test_spawnAsBatchCommandList(self):
         return self._test_spawnAsBatch(stdoutCommand('hello'), "cmd.exe /c")
 
+    def test_spawnAsBatchCommandWithNonAscii(self):
+        return self._test_spawnAsBatch(u"echo \u6211", "cmd.exe")
+
+    def test_spawnAsBatchCommandListWithNonAscii(self):
+        return self._test_spawnAsBatch(['echo', u"\u6211"], "cmd.exe /c")
+
 
 class TestPOSIXKilling(BasedirMixin, unittest.TestCase):