Repository: buildbot/buildbot

Issue:
P4 with python 3.6 under windows fails on worker
Trying to migrate from python 2.7 to python 3.6 on worker and getting following error in Perforce sync command:

'b'p4'' is not recognized as an internal or external command,
operable program or batch file.

Which is interesting because the command is

b'p4' b'-p' b'192.168.1.10:4000' b'-u' b'ABC' b'-c' b'ABC-buildbot09-build-targets' b'client' b'-i'
 in dir D:\buildbot\build-targets\build (timeout 1200 secs)
 watching logfiles {}
 argv: [b"b'p4'", b"b'-p'", b"b'192.168.1.10:4000'", b"b'-u'", b"b'ABC'", b"b'-c'", b"b'ABC-buildbot09-build-targets'", b"b'client'", b"b'-i'"]
 writing 547 bytes to stdin
 using PTY: False

Which has doubly-quoted binary items here.

Not sure if the problem is the binary string or something else. Master is running 3.4.3


Discussion:
I found that if I modified line 245 in buildbot/p4.py  which reads:

 command = [encodeArg(c) for c in command]

and simply don't encode each argument the problem is fixed.

However, I suspect this would break python 2.7.

Any ideas?
I think your patch is on the right track.

Can you try your patch in python 2 and python 3 setups and verify if it works with perforce?
I don't have an actual p4 environment to test against.

Alternatively, if you can give me some quick steps I can do to set up p4 to test, I can try that.
I think `encodeArg` isn't doing it right then. I last touched that line, but it was there before so I don't know the original reason.
@seankelly I am no python expert but I wonder if the encodeArg is there to handle unicode strings in some of the client names, etc. In 2.7 it would need to be encoded as ascii, but in 3.x the encode is making it a byte-string which I think needs to be decoded as ascii. Other commands in p4 in the same file have no such encoding.

@rodrigc I don't have python 2 setup here anymore - but I think it would be moot test for me - we don't have utf-8/multibyte client or machine names here since we are in North America. The concern would be more with langues that use unicode to handle accents or non-latin text.

Maybe someone who knows more about this can decide what to do -- I suspect we need to decode the resulting byte string into ascii but I have not use 2.7 for a looonnng time.
I am wondering if `encodeArg` is needed. The rest of the source steps aren't doing the same thing. GitPoller, SVNPoller, and P4Poller all use `encodeArg`/`encode`, but HgPoller doesn't.
Are you using P4Poller to monitor changes too?
Yes. It works fine. The initial p4 -V is fine too. Its simply the final
command is doubly encoded.

On Fri, Aug 4, 2017 at 7:19 PM Sean Kelly <notifications@github.com> wrote:

> Are you using P4Poller to monitor changes too?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/buildbot/buildbot/issues/3493#issuecomment-320389157>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACnpqDf6LL-TiOrIpHTwqzTDzEeTFaGVks5sU9FEgaJpZM4OtCxb>
> .
>

@asomers can you provide me with a very small recipe for creating a Perforce repository which can be polled with buildbot with this: http://docs.buildbot.net/latest/manual/cfg-buildsteps.html#p4

I want to try to fix some of the issues preventing this from working with Python 3
Wow.. I went to look at perforce docs (which used to be well organized) and they've degraded significantly. I did p4 admin work a while back. 
O.k. look at this page:
http://answers.perforce.com/articles/KB/2451?startURL=%2Farticles%2FKB_Article%2FExample-Unix-Installation

And:
http://answers.perforce.com/articles/KB/2439

I think the latter may be what you want.
I've been working on this off-and-on and have something that can create a repository that Buildbot can poll. This is what I have in a Dockerfile:

    /opt/perforce/sbin/configure-helix-p4d.sh master -n -p 1666 -r /srv/perforce/master -u super -P SuperSuper
    echo 'SuperSuper' | p4 login
    p4 configure set security=0
    p4 depot -i < /root/perforce/test-depot
    p4 client -i < /root/perforce/client
    mkdir -p /root/work/test
    cd /root/work/test
    echo 00 > 00
    P4CLIENT=buildbot_test p4 add 00
    P4CLIENT=buildbot_test p4 submit -i < /root/perforce/submit
    p4dctl stop -a

The redirected files aren't important. So long as an editor is set, Perforce will open something usable.
@rodrigc I'm travelling right now, but I can give it a shot next Thursday if you haven't solved it yet.
Thanks for following up on this.
I created a perforce repository by doing:
```
export P4HOST=localhost
export P4PORT=1666
mkdir /tmp/sample-p4-repo
p4d -r /tmp/sample-p4-repo
```

On client, I was able to do:

```
export P4HOST=localhost
export P4PORT=1666
p4 info
```

In master.cfg, I was able to add:

```
p4source = changes.P4Source(p4base='//depot/project/', 
                                                     p4port="localhost:1666",
                                                     pollInterval=15, pollAtLaunch=True,
                                                     split_file=lambda branchfile: branchfile.split('/',1))
c['change_source'].append(p4source)

factory.addStep(steps.P4(p4port='localhost:1666',
                         p4client=util.WithProperties('localhost'),
                         p4user='crodrigues',
                         p4base='//depot',
                         mode='incremental'))
```

What else do I need in master.cfg to get this to work?
The problems I had were not from the change_source, but actually from the
sync command issues by steps.P4:

 steps.P4( ) - it was the sync operation that caused it.

OK, I added a steps.P4 to my master.cfg, and I have a working setp where I can reproduce your problem.
I see the problem.  One place where the double encoding is here:
https://github.com/buildbot/buildbot/blob/v0.9.9/worker/buildbot_worker/util/__init__.py#L75

Resolving patch:
diff --git a/master/buildbot/process/remotecommand.py b/master/buildbot/process/remotecommand.py
--- a/master/buildbot/process/remotecommand.py
+++ b/master/buildbot/process/remotecommand.py
@@ -16,7 +16,7 @@
 from __future__ import absolute_import
 from __future__ import print_function
 from future.utils import iteritems
-from future.utils import string_types
+from future.utils import text_type
 
 from twisted.internet import defer
 from twisted.internet import error
@@ -360,7 +360,7 @@ def __init__(self, workdir, command, env=None,
         if decodeRC is None:
             decodeRC = {0: SUCCESS}
         self.command = command  # stash .command, set it later
-        if isinstance(self.command, string_types):
+        if isinstance(self.command, (text_type, bytes)):
             # Single string command doesn't support obfuscation.
             self.fake_command = command
         else:
diff --git a/master/buildbot/steps/master.py b/master/buildbot/steps/master.py
--- a/master/buildbot/steps/master.py
+++ b/master/buildbot/steps/master.py
@@ -16,13 +16,11 @@
 from __future__ import absolute_import
 from __future__ import print_function
 from future.utils import iteritems
-from future.utils import string_types
+from future.utils import text_type
 
 import os
 import pprint
 import re
-from builtins import bytes
-from builtins import str
 
 from twisted.internet import defer
 from twisted.internet import error
@@ -85,7 +83,7 @@ def start(self):
         # render properties
         command = self.command
         # set up argv
-        if isinstance(command, (str, bytes)):
+        if isinstance(command, (text_type, bytes)):
             if runtime.platformType == 'win32':
                 # allow %COMSPEC% to have args
                 argv = os.environ['COMSPEC'].split()
@@ -108,7 +106,7 @@ def start(self):
 
         self.stdio_log = stdio_log = self.addLog("stdio")
 
-        if isinstance(command, str):
+        if isinstance(command, (text_type, bytes)):
             stdio_log.addHeader(command.strip() + "\n\n")
         else:
             stdio_log.addHeader(" ".join(command) + "\n\n")
@@ -139,7 +137,7 @@ def subst(match):
             newenv = {}
             for key, v in iteritems(env):
                 if v is not None:
-                    if not isinstance(v, string_types):
+                    if not isinstance(v, (text_type, bytes)):
                         raise RuntimeError("'env' values must be strings or "
                                            "lists; key '%s' is incorrect" % (key,))
                     newenv[key] = p.sub(subst, env[key])
diff --git a/worker/buildbot_worker/util/__init__.py b/worker/buildbot_worker/util/__init__.py
--- a/worker/buildbot_worker/util/__init__.py
+++ b/worker/buildbot_worker/util/__init__.py
@@ -13,7 +13,7 @@
 #
 # Copyright Buildbot Team Members
 
-from future.utils import string_types
+from future.utils import text_type
 
 import itertools
 import textwrap
@@ -70,7 +70,7 @@ def __eq__(self, other):
 
     @staticmethod
     def to_text(s):
-        if isinstance(s, string_types):
+        if isinstance(s, (text_type, bytes)):
             return s
         return str(s)
 


Test patch:
diff --git a/worker/buildbot_worker/test/unit/test_util.py b/worker/buildbot_worker/test/unit/test_util.py
--- a/worker/buildbot_worker/test/unit/test_util.py
+++ b/worker/buildbot_worker/test/unit/test_util.py
@@ -1,3 +1,4 @@
+# coding: utf-8
 # This file is part of Buildbot.  Buildbot is free software: you can
 # redistribute it and/or modify it under the terms of the GNU General Public
 # License as published by the Free Software Foundation, version 2.
@@ -68,16 +69,30 @@ def testSimple(self):
 
     def testObfuscatedCommand(self):
         cmd = ['echo', util.Obfuscated('password', '*******')]
+        cmd_bytes = [b'echo', util.Obfuscated(b'password', b'*******')]
+        cmd_unicode = [u'echo', util.Obfuscated(u'password', u'привет')]
 
         self.assertEqual(
             ['echo', 'password'], util.Obfuscated.get_real(cmd))
         self.assertEqual(
             ['echo', '*******'], util.Obfuscated.get_fake(cmd))
+        self.assertEqual(
+            [b'echo', b'password'], util.Obfuscated.get_real(cmd_bytes))
+        self.assertEqual(
+            [b'echo', b'*******'], util.Obfuscated.get_fake(cmd_bytes))
+        self.assertEqual(
+            [u'echo', u'password'], util.Obfuscated.get_real(cmd_unicode))
+        self.assertEqual(
+            [u'echo', u'привет'], util.Obfuscated.get_fake(cmd_unicode))
 
     def testObfuscatedNonString(self):
         cmd = ['echo', 1]
+        cmd_bytes = [b'echo', 2]
+        cmd_unicode = [u'привет', 3]
         self.assertEqual(['echo', '1'], util.Obfuscated.get_real(cmd))
-        self.assertEqual(['echo', '1'], util.Obfuscated.get_fake(cmd))
+        self.assertEqual([b'echo', '2'], util.Obfuscated.get_fake(cmd_bytes))
+        self.assertEqual([u'привет', u'3'],
+                         util.Obfuscated.get_fake(cmd_unicode))
 
     def testObfuscatedNonList(self):
         cmd = 1