Repository: buildbot/buildbot

Issue:
random-like UTF-8 problem when starting workers
Hi!

I run a server with multiple workers on identical hard/software. When connecting these workers some refuse to connect with the following log:
```txt
2023-07-10 15:04:33+0200 [-] Loading buildbot.tac...
2023-07-10 15:04:33+0200 [-] Loaded.
2023-07-10 15:04:33+0200 [-] twistd 22.10.0 (/home/m/m21AAAA/.conda/envs/bb-3.11/bin/python3.11 3.11.4) starting up.
2023-07-10 15:04:33+0200 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2023-07-10 15:04:33+0200 [-] Starting Worker -- version: 3.8.0
2023-07-10 15:04:33+0200 [-] recording hostname in twistd.hostname
2023-07-10 15:04:33+0200 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x7f8b07c0bc10>
2023-07-10 15:04:33+0200 [Broker,client] message from master: attached
2023-07-10 15:04:33+0200 [Broker,client] Peer will receive following PB traceback:
2023-07-10 15:04:33+0200 [Broker,client] Unhandled Error
        Traceback (most recent call last):
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/banana.py", line 176, in gotItem
            self.callExpressionReceived(item)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/banana.py", line 137, in callExpressionReceived
            self.expressionReceived(obj)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/pb.py", line 607, in expressionReceived
            method(*sexp[1:])
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/pb.py", line 999, in proto_message
            self._recvMessage(
        --- <exception caught here> ---
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/pb.py", line 1053, in _recvMessage
            netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/twisted/spread/flavors.py", line 131, in remoteMessageReceived
            state = method(*args, **kw)
          File "/home/m/m21AAAA/.conda/envs/bb-3.11/lib/python3.11/site-packages/buildbot_worker/base.py", line 222, in remote_getWorkerInfo
            files[f] = fin.read()
          File "<frozen codecs>", line 322, in decode

        builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte

2023-07-10 15:04:33+0200 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2023-07-10 15:04:33+0200 [Broker,client] While trying to connect:
        Traceback from remote host -- twisted.spread.pb.RemoteError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte

2023-07-10 15:04:33+0200 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x7f8b07c0bc10>
2023-07-10 15:04:33+0200 [-] Main loop terminated.
2023-07-10 15:04:33+0200 [-] Server Shut Down.
```

my starting command is `TERM=xterm buildbot-worker start $(pwd)`. I also tested with `export LANG=C` but error is persistent.

This log is from a python-3.11 test setup. 

The production version throws this error with python-3.10:

```txt
2023-07-10 15:24:05+0200 [-] Loaded.
2023-07-10 15:24:05+0200 [-] twistd 22.2.0 (/home/m/m21AAAA/.conda/envs/buildbot-worker/bin/python3.10 3.10.9) starting up.
2023-07-10 15:24:05+0200 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2023-07-10 15:24:05+0200 [-] Starting Worker -- version: 3.7.0
2023-07-10 15:24:05+0200 [-] recording hostname in twistd.hostname
2023-07-10 15:24:05+0200 [-] Starting factory <buildbot_worker.pb.BotFactory object at 0x7f725e708100>
2023-07-10 15:24:05+0200 [Broker,client] message from master: attached
2023-07-10 15:24:05+0200 [Broker,client] Peer will receive following PB traceback:
2023-07-10 15:24:05+0200 [Broker,client] Unhandled Error
	Traceback (most recent call last):
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/banana.py", line 176, in gotItem
	    self.callExpressionReceived(item)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/banana.py", line 137, in callExpressionReceived
	    self.expressionReceived(obj)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/pb.py", line 607, in expressionReceived
	    method(*sexp[1:])
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/pb.py", line 999, in proto_message
	    self._recvMessage(
	--- <exception caught here> ---
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/pb.py", line 1053, in _recvMessage
	    netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/twisted/spread/flavors.py", line 131, in remoteMessageReceived
	    state = method(*args, **kw)
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/site-packages/buildbot_worker/base.py", line 222, in remote_getWorkerInfo
	    files[f] = fin.read()
	  File "/home/m/m21AAAA/.conda/envs/buildbot-worker/lib/python3.10/codecs.py", line 322, in decode
	    (result, consumed) = self._buffer_decode(data, self.errors, final)
	builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte
	
2023-07-10 15:24:05+0200 [Broker,client] ReconnectingPBClientFactory.failedToGetPerspective
2023-07-10 15:24:05+0200 [Broker,client] While trying to connect:
	Traceback from remote host -- twisted.spread.pb.RemoteError: 'utf-8' codec can't decode byte 0x9f in position 3: invalid start byte
	
2023-07-10 15:24:05+0200 [-] Stopping factory <buildbot_worker.pb.BotFactory object at 0x7f725e708100>
2023-07-10 15:24:05+0200 [-] Main loop terminated.
2023-07-10 15:24:05+0200 [-] Server Shut Down.
```
Hence: same error with different versions of the worker software and different python versions.

Server info:
- Python version: 3.10.9
- Buildbot version: 3.6.0
- Twisted version: 22.10.0

I searched around but could not find any further input on this. Any ideas and tips are welcome!

thx


Discussion:
By accident I noticed, that I can get avoid the error, when I remove the `info` directory on the client side. The error also does not show up, when I create the info directory with `buildbot-worker create-worker` and leave it unchanged. But as soon as I edit the two textfiles (keeping them pure ASCII) the error is there again.

Resolving patch:
diff --git a/master/buildbot/scripts/copydb.py b/master/buildbot/scripts/copydb.py
--- a/master/buildbot/scripts/copydb.py
+++ b/master/buildbot/scripts/copydb.py
@@ -91,11 +91,29 @@ def _copy_single_table(src_db, dst_db, table, table_name, buildset_to_parent_bui
     written_count = [0]
     total_count = [0]
 
+    autoincrement_foreign_key_column = None
+    for column_name, column in table.columns.items():
+        if not column.foreign_keys and column.primary_key and isinstance(column.type, sa.Integer):
+            autoincrement_foreign_key_column = column_name
+
     def thd_write(conn):
+        max_column_id = 0
         while True:
             try:
                 rows = rows_queue.get(timeout=1)
                 if rows is None:
+
+                    if autoincrement_foreign_key_column is not None and max_column_id != 0:
+                        if dst_db.pool.engine.dialect.name == 'postgresql':
+                            # Explicitly inserting primary row IDs does not bump the primary key
+                            # sequence on Postgres
+                            seq_name = f"{table_name}_{autoincrement_foreign_key_column}_seq"
+                            transaction = conn.begin()
+                            conn.execute(
+                                f"ALTER SEQUENCE {seq_name} RESTART WITH {max_column_id + 1}"
+                            )
+                            transaction.commit()
+
                     rows_queue.task_done()
                     return
 
@@ -103,6 +121,10 @@ def thd_write(conn):
                     {k: getattr(row, k) for k in column_keys} for row in rows
                 ]
 
+                if autoincrement_foreign_key_column is not None:
+                    for row in row_dicts:
+                        max_column_id = max(max_column_id, row[autoincrement_foreign_key_column])
+
                 if table_name == "buildsets":
                     for row_dict in row_dicts:
                         if row_dict["parent_buildid"] is not None:
@@ -201,7 +223,7 @@ def _copy_database_with_db(src_db, dst_db, print_log):
 
     for table_name in table_names:
         table = metadata.tables[table_name]
-        _copy_single_table(
+        yield _copy_single_table(
             src_db,
             dst_db,
             table,
diff --git a/worker/buildbot_worker/base.py b/worker/buildbot_worker/base.py
--- a/worker/buildbot_worker/base.py
+++ b/worker/buildbot_worker/base.py
@@ -25,12 +25,14 @@
 from twisted.application import service
 from twisted.internet import defer
 from twisted.internet import reactor
+from twisted.python import failure
 from twisted.python import log
 from twisted.spread import pb
 
 import buildbot_worker
 from buildbot_worker.commands import base
 from buildbot_worker.commands import registry
+from buildbot_worker.compat import bytes2unicode
 from buildbot_worker.util import buffer_manager
 from buildbot_worker.util import lineboundaries
 
@@ -218,7 +220,11 @@ def remote_getWorkerInfo(self):
                 filename = os.path.join(basedir, f)
                 if os.path.isfile(filename):
                     with open(filename, "r") as fin:
-                        files[f] = fin.read()
+                        try:
+                            files[f] = bytes2unicode(fin.read())
+                        except UnicodeDecodeError:
+                            log.err(failure.Failure(),
+                                    'error while reading file: %s' % (filename))
 
         self._read_os_release(self.os_release_file, files)
 
diff --git a/worker/buildbot_worker/scripts/runner.py b/worker/buildbot_worker/scripts/runner.py
--- a/worker/buildbot_worker/scripts/runner.py
+++ b/worker/buildbot_worker/scripts/runner.py
@@ -217,7 +217,7 @@ def postOptions(self):
                                        argument))
 
         for argument in ["log-count", "maxretries", "umask", "numcpus"]:
-            if not re.match(r'^(0o)?\d+$', self[argument]) and \
+            if not re.match(r'^((0o)\d+|0|[1-9]\d*)$', self[argument]) and \
                     self[argument] != 'None':
                 raise usage.UsageError("{} parameter needs to be a number"
                                     " or None".format(argument))


Test patch:
diff --git a/worker/buildbot_worker/test/unit/test_bot.py b/worker/buildbot_worker/test/unit/test_bot.py
--- a/worker/buildbot_worker/test/unit/test_bot.py
+++ b/worker/buildbot_worker/test/unit/test_bot.py
@@ -127,6 +127,41 @@ def test_getWorkerInfo_nodir(self):
             ['environ', 'system', 'numcpus', 'basedir', 'worker_commands', 'version',
              'delete_leftover_dirs']))
 
+    @defer.inlineCallbacks
+    def test_getWorkerInfo_decode_error(self):
+        infodir = os.path.join(self.basedir, "info")
+        os.makedirs(infodir)
+        with open(os.path.join(infodir, "admin"), "w") as f:
+            f.write("testy!")
+        with open(os.path.join(infodir, "foo"), "w") as f:
+            f.write("bar")
+        with open(os.path.join(infodir, "environ"), "w") as f:
+            f.write("something else")
+        # This will not be part of worker info
+        with open(os.path.join(infodir, "binary"), "wb") as f:
+            f.write(b"\x90")
+
+        # patch the log.err, otherwise trial will think something *actually*
+        # failed
+        self.patch(log, "err", lambda f, x: None)
+
+        info = yield self.bot.callRemote("getWorkerInfo")
+
+        # remove any os_ fields as they are dependent on the test environment
+        info = {k: v for k, v in info.items() if not k.startswith("os_")}
+
+        self.assertEqual(info, {
+            "admin": 'testy!',
+            "foo": 'bar',
+            "environ": os.environ,
+            "system": os.name,
+            "basedir": self.basedir,
+            "worker_commands": self.real_bot.remote_getCommands(),
+            "version": self.real_bot.remote_getVersion(),
+            "numcpus": multiprocessing.cpu_count(),
+            "delete_leftover_dirs": False
+            })
+
     def test_shutdown(self):
         d1 = defer.Deferred()
         self.patch(reactor, "stop", lambda: d1.callback(None))
diff --git a/worker/buildbot_worker/test/unit/test_scripts_runner.py b/worker/buildbot_worker/test/unit/test_scripts_runner.py
--- a/worker/buildbot_worker/test/unit/test_scripts_runner.py
+++ b/worker/buildbot_worker/test/unit/test_scripts_runner.py
@@ -244,6 +244,11 @@ def test_inv_umask(self):
                                "umask parameter needs to be a number or None"):
             self.parse("--umask=X", *self.req_args)
 
+    def test_inv_umask2(self):
+        with self.assertRaisesRegex(usage.UsageError,
+                               "umask parameter needs to be a number or None"):
+            self.parse("--umask=022", *self.req_args)
+
     def test_inv_allow_shutdown(self):
         with self.assertRaisesRegex(usage.UsageError,
                        "allow-shutdown needs to be one of 'signal' or 'file'"):