internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: buildbot/buildbot

Issue:
Paused bots resume automatically on master restart
Paused bots resume automatically on master restart. They should remain paused until someone unpause them. 

Usually I pause a bot when a bot is having some issues (e.g.: disk full, unexpected failure etc.). However, it automatically resumes on master restart, which is unexpected. Master restart should not override a manual action (of pausing the bot).


Discussion:

Resolving patch:
diff --git a/master/buildbot/worker/base.py b/master/buildbot/worker/base.py
--- a/master/buildbot/worker/base.py
+++ b/master/buildbot/worker/base.py
@@ -235,6 +235,7 @@ def _applyWorkerInfo(self, info):
     def _getWorkerInfo(self):
         worker = yield self.master.data.get(
             ('workers', self.workerid))
+        self._paused = worker["paused"]
         self._applyWorkerInfo(worker['workerinfo'])
 
     def setServiceParent(self, parent):


Test patch:
diff --git a/master/buildbot/test/fakedb/workers.py b/master/buildbot/test/fakedb/workers.py
--- a/master/buildbot/test/fakedb/workers.py
+++ b/master/buildbot/test/fakedb/workers.py
@@ -77,8 +77,8 @@ def insertTestData(self, rows):
                 self.workers[row.id] = dict(
                     id=row.id,
                     name=row.name,
-                    paused=0,
-                    graceful=0,
+                    paused=row.paused,
+                    graceful=row.graceful,
                     info=row.info)
             elif isinstance(row, ConfiguredWorker):
                 row.id = row.buildermasterid * 10000 + row.workerid
diff --git a/master/buildbot/test/unit/worker/test_base.py b/master/buildbot/test/unit/worker/test_base.py
--- a/master/buildbot/test/unit/worker/test_base.py
+++ b/master/buildbot/test/unit/worker/test_base.py
@@ -470,6 +470,34 @@ def test_setServiceParent_workerLocks(self):
         bs = ConcreteWorker('bot', 'pass', locks=[lock.access("counting")])
         yield bs.setServiceParent(bsmanager)
 
+    @defer.inlineCallbacks
+    def test_startService_paused_true(self):
+        """Test that paused state is restored on a buildbot restart"""
+        self.master.db.insertTestData([
+            fakedb.Worker(id=9292, name='bot', paused=1)
+        ])
+
+        worker = yield self.createWorker()
+
+        yield worker.startService()
+
+        self.assertTrue(worker.isPaused())
+        self.assertFalse(worker._graceful)
+
+    @defer.inlineCallbacks
+    def test_startService_graceful_true(self):
+        """Test that graceful state is NOT restored on a buildbot restart"""
+        self.master.db.insertTestData([
+            fakedb.Worker(id=9292, name='bot', graceful=1)
+        ])
+
+        worker = yield self.createWorker()
+
+        yield worker.startService()
+
+        self.assertFalse(worker.isPaused())
+        self.assertFalse(worker._graceful)
+
     @defer.inlineCallbacks
     def test_startService_getWorkerInfo_empty(self):
         worker = yield self.createWorker()
Repository: buildbot/buildbot

Issue:
builtins.AttributeError: instead of intended log message
Hello,

Because I had set up an incorrect url, or an url that was not possible to access due to limited api token access I got this error message:
```
          File "/usr/local/lib/python3.10/site-packages/buildbot/changes/github.py", line 142, in _getPulls
            f"while loading {result.url}")
        builtins.AttributeError: '_Response' object has no attribute 'url'
```

However, looking at the code the error should probably be something more like:
```
"GitHubPullrequestPoller error {error_code} '{error message}' while loading {some_url}")
```

Is the attribute error the intended behaviour? If not then I guess I would do some kind of patch for it (i.e. checking for existing url attribute).


Discussion:
Hmm. I am going to assume this is not the intended behaviour because the returned log message and error code made a whole lot of sense in my case if I made some local modifications to not print the url.

I will create a pull request when I have time.

Resolving patch:
diff --git a/master/buildbot/util/httpclientservice.py b/master/buildbot/util/httpclientservice.py
--- a/master/buildbot/util/httpclientservice.py
+++ b/master/buildbot/util/httpclientservice.py
@@ -35,8 +35,6 @@
 
 try:
     import treq
-    implementer(IHttpResponse)(treq.response._Response)
-
 except ImportError:
     treq = None
 
@@ -64,6 +62,27 @@ def url(self):
         return self._res.url
 
 
+@implementer(IHttpResponse)
+class TreqResponseWrapper:
+
+    def __init__(self, res):
+        self._res = res
+
+    def content(self):
+        return self._res.content()
+
+    def json(self):
+        return self._res.json()
+
+    @property
+    def code(self):
+        return self._res.code
+
+    @property
+    def url(self):
+        return self._res.request.absoluteURI.decode()
+
+
 class HTTPClientService(service.SharedService):
     """A SharedService class that can make http requests to remote services.
 
@@ -202,7 +221,7 @@ def _doTReq(self, method, ep, **kwargs):
         kwargs['agent'] = self._agent
 
         res = yield getattr(treq, method)(url, **kwargs)
-        return IHttpResponse(res)
+        return IHttpResponse(TreqResponseWrapper(res))
 
     # lets be nice to the auto completers, and don't generate that code
     def get(self, ep, **kwargs):


Test patch:
diff --git a/master/buildbot/test/integration/test_setup_entrypoints.py b/master/buildbot/test/integration/test_setup_entrypoints.py
--- a/master/buildbot/test/integration/test_setup_entrypoints.py
+++ b/master/buildbot/test/integration/test_setup_entrypoints.py
@@ -116,6 +116,7 @@ def test_util(self):
             'buildbot.util.git.GitStepMixin',
             'buildbot.util.giturlparse.GitUrl',
             'buildbot.util.httpclientservice.HTTPClientService',
+            'buildbot.util.httpclientservice.TreqResponseWrapper',
             'buildbot.util.httpclientservice.TxRequestsResponseWrapper',
             'buildbot.util.kubeclientservice.KubeClientService',
             'buildbot.util.kubeclientservice.KubeConfigLoaderBase',
Repository: buildbot/buildbot

Issue:
elapsed time for current build is negative
Frequently, while loading webpages, buildbot shows negative time for time elapsed for current build/step. It gets corrected automatically after few seconds. However, if buildbot is not very fast, this is very noticeable. Seems like some script taking long time to load and displaying negative time meanwhile.

Few screenshots below.

<img width="290" alt="screen shot 2017-07-20 at 6 47 57 pm" src="https://user-images.githubusercontent.com/3098702/28449022-5b899822-6d91-11e7-9c97-8d5bd959e631.png">

<img width="1262" alt="screen shot 2017-07-20 at 9 20 18 pm" src="https://user-images.githubusercontent.com/3098702/28449026-5e92a176-6d91-11e7-82d5-360c7fb7d2fa.png">



Discussion:
That is probably an easy fix. Any help appreciated!
This seems to happen when the time on server is different than the time on client. In my case time on server was somehow ~30s ahead of actual time (probably some misconfiguration).

Nevertheless, buildbot should not display negative time.
@aakashja do you mean server == bb-master host, client == user, who open web-page ?
@Frodox yes
Taking a look at where this might be happening, does it sound right that it's happening because of this line? https://github.com/buildbot/buildbot/blob/d5616a3b9fa17108b767c32806ca916923094835/master/buildbot/statistics/capture.py#L286 
Either this could be solved by setting the min duration to 0, or by getting different start/end time values somewhere else? 
(I'm new to open source) 

Resolving patch:
diff --git a/master/buildbot/statistics/capture.py b/master/buildbot/statistics/capture.py
--- a/master/buildbot/statistics/capture.py
+++ b/master/buildbot/statistics/capture.py
@@ -14,6 +14,7 @@
 # Copyright Buildbot Team Members
 
 import abc
+import datetime
 import re
 
 from twisted.internet import defer
@@ -282,7 +283,10 @@ def default_callback(start_time, end_time):
                 divisor = 60
             elif report_in == 'hours':
                 divisor = 60 * 60
-            duration = end_time - start_time
+            if end_time < start_time:
+                duration = datetime.timedelta(0)
+            else:
+                duration = end_time - start_time
             # cannot use duration.total_seconds() on Python 2.6
             duration = ((duration.microseconds + (duration.seconds +
                                                   duration.days * 24 * 3600) * 1e6) / 1e6)


Test patch:
Repository: buildbot/buildbot

Issue:
p4port is not renderable
Hello,

I'm trying to override p4port per-client. I need this because we want some workers to connect to a local perforce proxy but still use the actual perforce server by default.

Basically something like this:

```
P4PORT = 'perforce.company.com:1666'
P4PORT_PROXY = 'proxy.lan:1666'

c = BuildmasterConfig = {}

c['workers'] = [
                worker.Worker("worker1", "password1"),
                worker.Worker("worker2", "password2", properties={'P4PORT': P4PORT_PROXY}),
]


factory = util.BuildFactory()
factory.addStep(steps.P4(p4port=util.Property('P4PORT', default=P4PORT), p4user=P4USER, p4base='//depot', p4branch=getBranch, stream=True, p4client=getClientName, p4client_spec_options='rmdir clobber', mode='incremental'))

...
```

This doesn't work because p4port is not a renderable:

> Perforce client error:
> 	Connect to server failed; check $P4PORT.
> 	TCP connect to Unpersistable("Unpersistable data: b'instance of class buildbot.process.properties.Property deemed insecure'") failed.
> 	The specified class was not found. 

This is similar to #3954 (`p4passwd` was not renderable).


Discussion:

Resolving patch:
diff --git a/master/buildbot/steps/source/p4.py b/master/buildbot/steps/source/p4.py
--- a/master/buildbot/steps/source/p4.py
+++ b/master/buildbot/steps/source/p4.py
@@ -45,7 +45,7 @@ class P4(Source):
 
     name = 'p4'
 
-    renderables = ['mode', 'p4base', 'p4client', 'p4viewspec', 'p4branch', 'p4passwd']
+    renderables = ['mode', 'p4base', 'p4client', 'p4viewspec', 'p4branch', 'p4passwd', 'p4port']
     possible_modes = ('incremental', 'full')
 
     def __init__(self, mode='incremental',


Test patch:
Repository: buildbot/buildbot

Issue:
There's a small window where events will be missed by reporters during reconfig
The code in `ReporterBase.reconfigService()` stops consuming events for all event keys and then starts consuming events for all event keys again. This is not necessary for any wanted event keys which are wanted in both old and new configuration. The code should stop consuming events for no longer wanted event keys, start consuming events for newly wanted event keys and leave the rest untouched.


Discussion:

Resolving patch:
diff --git a/master/buildbot/reporters/base.py b/master/buildbot/reporters/base.py
--- a/master/buildbot/reporters/base.py
+++ b/master/buildbot/reporters/base.py
@@ -35,7 +35,7 @@ class ReporterBase(service.BuildbotService):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.generators = None
-        self._event_consumers = []
+        self._event_consumers = {}
         self._pending_got_event_calls = {}
 
     def checkConfig(self, generators):
@@ -52,26 +52,29 @@ def checkConfig(self, generators):
 
     @defer.inlineCallbacks
     def reconfigService(self, generators):
-
-        for consumer in self._event_consumers:
-            yield consumer.stopConsuming()
-        self._event_consumers = []
-
         self.generators = generators
 
         wanted_event_keys = set()
         for g in self.generators:
             wanted_event_keys.update(g.wanted_event_keys)
 
+        # Remove consumers for keys that are no longer wanted
+        for key in list(self._event_consumers.keys()):
+            if key not in wanted_event_keys:
+                yield self._event_consumers[key].stopConsuming()
+                del self._event_consumers[key]
+
+        # Add consumers for new keys
         for key in sorted(list(wanted_event_keys)):
-            consumer = yield self.master.mq.startConsuming(self._got_event, key)
-            self._event_consumers.append(consumer)
+            if key not in self._event_consumers:
+                self._event_consumers[key] = \
+                    yield self.master.mq.startConsuming(self._got_event, key)
 
     @defer.inlineCallbacks
     def stopService(self):
-        for consumer in self._event_consumers:
+        for consumer in self._event_consumers.values():
             yield consumer.stopConsuming()
-        self._event_consumers = []
+        self._event_consumers = {}
 
         for pending_call in list(self._pending_got_event_calls.values()):
             yield pending_call


Test patch:
Repository: buildbot/buildbot

Issue:
Rebuilt buildsets lack parent_buildid on the new messages in the Data API
Pretty much what the title says.


Discussion:

Resolving patch:
diff --git a/master/buildbot/data/buildsets.py b/master/buildbot/data/buildsets.py
--- a/master/buildbot/data/buildsets.py
+++ b/master/buildbot/data/buildsets.py
@@ -173,6 +173,7 @@ def addBuildset(self, waited_for, scheduler=None, sourcestamps=None, reason='',
             "bsid": bsid,
             "external_idstring": external_idstring,
             "reason": reason,
+            "parent_buildid": parent_buildid,
             "submitted_at": submitted_at,
             "complete": False,
             "complete_at": None,


Test patch:
diff --git a/master/buildbot/test/unit/data/test_buildsets.py b/master/buildbot/test/unit/data/test_buildsets.py
--- a/master/buildbot/test/unit/data/test_buildsets.py
+++ b/master/buildbot/test/unit/data/test_buildsets.py
@@ -221,6 +221,7 @@ def _buildsetMessage(self, bsid, external_idstring='extid',
                 "complete": False,
                 "complete_at": None,
                 "external_idstring": external_idstring,
+                "parent_buildid": None,
                 "reason": reason,
                 "results": None,
                 "scheduler": scheduler,
Repository: buildbot/buildbot

Issue:
Address PEP 706 - Filter for tarfile.extractall
Given proposal improves security of tarfile extraction to help avoid CVE-2007-4559.

- In Python 3.12-3.13, a DeprecationWarning is emitted and extraction uses `fully_trusted` filter.
- In Python 3.14+, it will use the `data` filter.

It seems given proposal was backported also to older version of Python.

Reference: https://peps.python.org/pep-0706/



Discussion:

Resolving patch:
diff --git a/master/buildbot/process/remotetransfer.py b/master/buildbot/process/remotetransfer.py
--- a/master/buildbot/process/remotetransfer.py
+++ b/master/buildbot/process/remotetransfer.py
@@ -126,7 +126,10 @@ def remote_unpack(self):
 
         # Unpack archive and clean up after self
         with tarfile.open(name=self.tarname, mode=mode) as archive:
-            archive.extractall(path=self.destroot)
+            if hasattr(tarfile, 'data_filter'):
+                archive.extractall(path=self.destroot, filter='data')
+            else:
+                archive.extractall(path=self.destroot)
         os.remove(self.tarname)
 
 


Test patch:
diff --git a/master/buildbot/test/integration/test_upgrade.py b/master/buildbot/test/integration/test_upgrade.py
--- a/master/buildbot/test/integration/test_upgrade.py
+++ b/master/buildbot/test/integration/test_upgrade.py
@@ -77,7 +77,10 @@ def setUpUpgradeTest(self):
             with tarfile.open(tarball) as tf:
                 prefixes = set()
                 for inf in tf:
-                    tf.extract(inf)
+                    if hasattr(tarfile, 'data_filter'):
+                        tf.extract(inf, filter='data')
+                    else:
+                        tf.extract(inf)
                     prefixes.add(inf.name.split('/', 1)[0])
 
             # get the top-level dir from the tarball
Repository: buildbot/buildbot

Issue:
Wrong error message when buildbot uses React web frontend and it is not installed
Buildbot shows incorrect error message when using React frontend and it's not installed.

```
buildtins.RuntimeError: could not find buildbot-www; is it installed;
```

The correct package is buildbot-www-react. This is especially confusing now during the process of migration to React web frontend.


Discussion:

Resolving patch:
diff --git a/master/buildbot/www/service.py b/master/buildbot/www/service.py
--- a/master/buildbot/www/service.py
+++ b/master/buildbot/www/service.py
@@ -306,8 +306,10 @@ def setupSite(self, new_config):
 
         self.reconfigurableResources = []
 
-        # we're going to need at least the base plugin (buildbot-www)
+        # we're going to need at least the base plugin (buildbot-www or buildbot-www-react)
         if self.base_plugin_name not in self.apps:
+            if self.base_plugin_name == 'base_react':
+                raise RuntimeError("could not find buildbot-www-react; is it installed?")
             raise RuntimeError("could not find buildbot-www; is it installed?")
 
         root = self.apps.get(self.base_plugin_name).resource


Test patch:
Repository: buildbot/buildbot

Issue:
Address PEP 706 - Filter for tarfile.extractall
Given proposal improves security of tarfile extraction to help avoid CVE-2007-4559.

- In Python 3.12-3.13, a DeprecationWarning is emitted and extraction uses `fully_trusted` filter.
- In Python 3.14+, it will use the `data` filter.

It seems given proposal was backported also to older version of Python.

Reference: https://peps.python.org/pep-0706/



Discussion:

Resolving patch:
diff --git a/master/buildbot/process/remotetransfer.py b/master/buildbot/process/remotetransfer.py
--- a/master/buildbot/process/remotetransfer.py
+++ b/master/buildbot/process/remotetransfer.py
@@ -126,7 +126,10 @@ def remote_unpack(self):
 
         # Unpack archive and clean up after self
         with tarfile.open(name=self.tarname, mode=mode) as archive:
-            archive.extractall(path=self.destroot)
+            if hasattr(tarfile, 'data_filter'):
+                archive.extractall(path=self.destroot, filter='data')
+            else:
+                archive.extractall(path=self.destroot)
         os.remove(self.tarname)
 
 


Test patch:
diff --git a/master/buildbot/test/integration/test_upgrade.py b/master/buildbot/test/integration/test_upgrade.py
--- a/master/buildbot/test/integration/test_upgrade.py
+++ b/master/buildbot/test/integration/test_upgrade.py
@@ -77,7 +77,10 @@ def setUpUpgradeTest(self):
             with tarfile.open(tarball) as tf:
                 prefixes = set()
                 for inf in tf:
-                    tf.extract(inf)
+                    if hasattr(tarfile, 'data_filter'):
+                        tf.extract(inf, filter='data')
+                    else:
+                        tf.extract(inf)
                     prefixes.add(inf.name.split('/', 1)[0])
 
             # (note that tf.extractall isn't available in py2.4)
Repository: buildbot/buildbot

Issue:
Typo in IRC Bot help
https://github.com/fabianjuelich/buildbot/blob/ee5472a6671a2f2931b861a2448cfd944cc68d15/master/buildbot/reporters/irc.py#L137C1-L137C29

`join` -> `leave`


Discussion:

Resolving patch:
diff --git a/master/buildbot/reporters/irc.py b/master/buildbot/reporters/irc.py
--- a/master/buildbot/reporters/irc.py
+++ b/master/buildbot/reporters/irc.py
@@ -134,7 +134,7 @@ def command_JOIN(self, args, **kwargs):
 
     @dangerousCommand
     def command_LEAVE(self, args, **kwargs):
-        """join a channel"""
+        """leave a channel"""
         args = self.splitArgs(args)
         for channel in args:
             self.bot.leave(channel)


Test patch:
Repository: buildbot/buildbot

Issue:
Gitpoller does not assess "branches" type
Gitpoller won't complain if "branches" is only a string, but does not handle it correctly. I think either a conversion to a list or a type check during initialization and reconfiguration would be beneficial.

Cheers,
Jan
Gitpoller does not assess "branches" type
Gitpoller won't complain if "branches" is only a string, but does not handle it correctly. I think either a conversion to a list or a type check during initialization and reconfiguration would be beneficial.

Cheers,
Jan


Discussion:

Resolving patch:
diff --git a/master/buildbot/changes/gitpoller.py b/master/buildbot/changes/gitpoller.py
--- a/master/buildbot/changes/gitpoller.py
+++ b/master/buildbot/changes/gitpoller.py
@@ -17,6 +17,7 @@
 
 import os
 import re
+from typing import TYPE_CHECKING
 from urllib.parse import quote as urlquote
 
 from twisted.internet import defer
@@ -32,6 +33,10 @@
 from buildbot.util.git import check_ssh_config
 from buildbot.util.state import StateMixin
 
+if TYPE_CHECKING:
+    from typing import Callable
+    from typing import Literal
+
 
 class GitError(Exception):
     """Raised when git exits with code 128."""
@@ -65,11 +70,11 @@ def __init__(self, repourl, **kwargs) -> None:
             kwargs["name"] = repourl
         super().__init__(repourl, **kwargs)
 
-    def checkConfig(
+    def checkConfig(  # type: ignore[override]
         self,
         repourl,
-        branches=None,
-        branch=None,
+        branches: list[str] | Literal[True] | Callable[[str], bool] | None = None,
+        branch: str | None = None,
         workdir=None,
         pollInterval=10 * 60,
         gitbin="git",
@@ -97,6 +102,17 @@ def checkConfig(
             config.error("GitPoller: can't specify only_tags and branch/branches")
         if branch and branches:
             config.error("GitPoller: can't specify both branch and branches")
+        if branch and not isinstance(branch, str):
+            config.error("GitPoller: 'branch' argument must be a str")
+        if branches is not None and not (
+            (isinstance(branches, list) and all(isinstance(e, str) for e in branches))
+            or branches is True
+            or callable(branches)
+        ):
+            config.error(
+                "GitPoller: 'branches' argument must be one of "
+                "list of str, True, or Callable[[str], bool]"
+            )
 
         check_ssh_config('GitPoller', sshPrivateKey, sshHostKey, sshKnownHosts)
 


Test patch:
Repository: buildbot/buildbot

Issue:
Flaky test test_userpass_wait
Hello,
I'm a NixOS maintainer, a distribution featuring buildbot as a package. Currently, buildbot is broken on NixOS because of some tests not passing, including ``test_userpass_wait`` and ``test_userpass_wait_bytes``. I've tried to investigate the root cause of this, but I'm not sure what is happening. It looks some timing are involved because running several times the tests, sometimes they are passing. 
Here is a complete trace of a build : https://hydra.nixos.org/build/204211282/log

 


Discussion:

Resolving patch:
diff --git a/master/buildbot/schedulers/trysched.py b/master/buildbot/schedulers/trysched.py
--- a/master/buildbot/schedulers/trysched.py
+++ b/master/buildbot/schedulers/trysched.py
@@ -370,11 +370,13 @@ def buildEvent(key, msg):
             if key[-1] == 'finished':
                 d.callback(None)
 
-        consumer = yield self.master.mq.startConsuming(
-            buildEvent, ('builds', str(self.builddict['buildid']), None)
-        )
+        buildid = self.builddict['buildid']
+        consumer = yield self.master.mq.startConsuming(buildEvent, ('builds', str(buildid), None))
+        builddict = yield self.master.data.get(('builds', buildid))
+        # build might have finished before we called startConsuming
+        if not builddict.get('complete', False):
+            yield d  # wait for event
 
-        yield d  # wait for event
         consumer.stopConsuming()
         return self  # callers expect result=self
 


Test patch: