Repository: buildbot/buildbot

Issue:
random-like UTF-8 problem when starting workers
Hi!

I run a server with multiple workers on identical hard/software. When connecting these workers some refuse to connect with the following log:
```txt
2023-07-10 15:04:33+0200 [-] Loading buildbot.tac...
2023-07-10 15:04:33+0200 [-] Loaded.
2023-07-10 15:04:33+0200 [-] twistd 22.10.0 (/home/m/m21AAAA/.conda/envs/bb-3.11/bin/python3.11 3.11.4) starting up.
2023-07-10 15:04:33+0200 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2023-07-10 15:04:33+0200 [-] Starting Worker -- version: 3.8.0
2023-07-10 15:04:33+0200 [-] recording hostname in twistd.hostname
2023-07-10 15:04:33+0200 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x7f8b07c0bc10>
2023-07-10 15:04:33+0200 [Broker,client] message from master: attached
2023-07-10 15:04:33+0200 [Broker,client] Peer will receive following PB traceback:
2023-07-10 15:04:33+0200 [Broker,client] Unhandled Error
        Traceback (most recent call last):
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/banana.py", line 176, in gotItem
            self.callExpressionReceived(item)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/banana.py", line 137, in callExpressionReceived
            self.expressionReceived(obj)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/pb.py", line 607, in expressionReceived
            method(*sexp[1:])
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/pb.py", line 999, in proto_message
            self._recvMessage(
        --- <exception caught here> ---
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/pb.py", line 1053, in _recvMessage
            netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/flavors.py", line 131, in remoteMessageReceived
            state = method(*args, **kw)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/buildbot_worker/base.py", line 222, in remote_getWorkerInfo
            files[f] = fin.read()
          File "<frozen codecs>", line 322, in decode

        builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte

2023-07-10 15:04:33+0200 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2023-07-10 15:04:33+0200 [Broker,client] While trying to connect:
        Traceback from remote host -- twisted.spread.pb.RemoteError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte

2023-07-10 15:04:33+0200 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x7f8b07c0bc10>
2023-07-10 15:04:33+0200 [-] Main loop terminated.
2023-07-10 15:04:33+0200 [-] Server Shut Down.
```

my starting command is `TERM=xterm buildbot-worker start $(pwd)`. I also tested with `export LANG=C` but error is persistent.

This log is from a python-3.11 test setup. 

The production version throws this error with python-3.10:

```txt
2023-07-10 15:24:05+0200 [-] Loaded.
2023-07-10 15:24:05+0200 [-] twistd 22.2.0 (/home/m/m21AAAA/.conda/envs/buildbot-worker/bin/python3.10 3.10.9) starting up.
2023-07-10 15:24:05+0200 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2023-07-10 15:24:05+0200 [-] Starting Worker -- version: 3.7.0
2023-07-10 15:24:05+0200 [-] recording hostname in twistd.hostname
2023-07-10 15:24:05+0200 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x7f725e708100>
2023-07-10 15:24:05+0200 [Broker,client] message from master: attached
2023-07-10 15:24:05+0200 [Broker,client] Peer will receive following PB traceback:
2023-07-10 15:24:05+0200 [Broker,client] Unhandled Error
	Traceback (most recent call last):
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/banana.py", line 176, in gotItem
	    self.callExpressionReceived(item)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/banana.py", line 137, in callExpressionReceived
	    self.expressionReceived(obj)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/pb.py", line 607, in expressionReceived
	    method(*sexp[1:])
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/pb.py", line 999, in proto_message
	    self._recvMessage(
	--- <exception caught here> ---
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/pb.py", line 1053, in _recvMessage
	    netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/flavors.py", line 131, in remoteMessageReceived
	    state = method(*args, **kw)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/buildbot_worker/base.py", line 222, in remote_getWorkerInfo
	    files[f] = fin.read()
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/codecs.py", line 322, in decode
	    (result, consumed) = self._buffer_decode(data, self.errors, final)
	builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte
	
2023-07-10 15:24:05+0200 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2023-07-10 15:24:05+0200 [Broker,client] While trying to connect:
	Traceback from remote host -- twisted.spread.pb.RemoteError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte
	
2023-07-10 15:24:05+0200 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x7f725e708100>
2023-07-10 15:24:05+0200 [-] Main loop terminated.
2023-07-10 15:24:05+0200 [-] Server Shut Down.
```
Hence: same error with different versions of the worker software and different python versions.

Server info:
- Python version: 3.10.9
- Buildbot version: 3.6.0
- Twisted version: 22.10.0

I searched around but could not find any further input on this. Any ideas and tips are welcome!

thx


Discussion:
By accident I noticed, that I can get avoid the error, when I remove the `info` directory on the client side. The error also does not show up, when I create the info directory with `buildbot-worker create-worker` and leave it unchanged. But as soon as I edit the two textfiles (keeping them pure ASCII) the error is there again.

Resolving patch:
diff --git a/worker/buildbot_worker/base.py b/worker/buildbot_worker/base.py
--- a/worker/buildbot_worker/base.py
+++ b/worker/buildbot_worker/base.py
@@ -25,12 +25,14 @@
 from twisted.application import service
 from twisted.internet import defer
 from twisted.internet import reactor
+from twisted.python import failure
 from twisted.python import log
 from twisted.spread import pb
 
 import buildbot_worker
 from buildbot_worker.commands import base
 from buildbot_worker.commands import registry
+from buildbot_worker.compat import bytes2unicode
 from buildbot_worker.util import buffer_manager
 from buildbot_worker.util import lineboundaries
 
@@ -218,7 +220,11 @@ def remote_getWorkerInfo(self):
                 filename = os.path.join(basedir, f)
                 if os.path.isfile(filename):
                     with open(filename, "r") as fin:
-                        files[f] = fin.read()
+                        try:
+                            files[f] = bytes2unicode(fin.read())
+                        except UnicodeDecodeError:
+                            log.err(failure.Failure(),
+                                    'error while reading file: %s' % (filename))
 
         self._read_os_release(self.os_release_file, files)
 


Test patch:
diff --git a/worker/buildbot_worker/test/unit/test_bot.py b/worker/buildbot_worker/test/unit/test_bot.py
--- a/worker/buildbot_worker/test/unit/test_bot.py
+++ b/worker/buildbot_worker/test/unit/test_bot.py
@@ -127,6 +127,41 @@ def test_getWorkerInfo_nodir(self):
             ['environ', 'system', 'numcpus', 'basedir', 'worker_commands', 'version',
              'delete_leftover_dirs']))
 
+    @defer.inlineCallbacks
+    def test_getWorkerInfo_decode_error(self):
+        infodir = os.path.join(self.basedir, "info")
+        os.makedirs(infodir)
+        with open(os.path.join(infodir, "admin"), "w") as f:
+            f.write("testy!")
+        with open(os.path.join(infodir, "foo"), "w") as f:
+            f.write("bar")
+        with open(os.path.join(infodir, "environ"), "w") as f:
+            f.write("something else")
+        # This will not be part of worker info
+        with open(os.path.join(infodir, "binary"), "wb") as f:
+            f.write(b"\x90")
+
+        # patch the log.err, otherwise trial will think something *actually*
+        # failed
+        self.patch(log, "err", lambda f, x: None)
+
+        info = yield self.bot.callRemote("getWorkerInfo")
+
+        # remove any os_ fields as they are dependent on the test environment
+        info = {k: v for k, v in info.items() if not k.startswith("os_")}
+
+        self.assertEqual(info, {
+            "admin": 'testy!',
+            "foo": 'bar',
+            "environ": os.environ,
+            "system": os.name,
+            "basedir": self.basedir,
+            "worker_commands": self.real_bot.remote_getCommands(),
+            "version": self.real_bot.remote_getVersion(),
+            "numcpus": multiprocessing.cpu_count(),
+            "delete_leftover_dirs": False
+            })
+
     def test_shutdown(self):
         d1 = defer.Deferred()
         self.patch(reactor, "stop", lambda: d1.callback(None))