internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: buildbot/buildbot

Issue:
Workers configured_on gets corrupted for multi-master
https://github.com/buildbot/buildbot/issues/3981 fixed the issue of Workers configured_on getting corrupted. It works for single master scenario. However, for multi-master scenario, this bug is still present.

I have two masters (running 1.1.1):
1: handling the web-server
2: handling workers communication, builds and everything else

If I restart 1 and then 2, then it's fine. However, if I restart 2 and then 1, then the workers configured_on is corrupted. The issue reproduce immediately on restarting 1.

If I restart only 2, then the issue doesn't happen. However, restarting 1 consistently reproduces the issue.

I suspect that the HouseKeeping which is done while starting the masters might be causing the issue.

https://github.com/buildbot/buildbot/blob/master/master/buildbot/master.py#L300



Discussion:

Resolving patch:
diff --git a/master/buildbot/data/masters.py b/master/buildbot/data/masters.py
--- a/master/buildbot/data/masters.py
+++ b/master/buildbot/data/masters.py
@@ -115,7 +115,7 @@ def expireMasters(self, forceHouseKeeping=False, _reactor=reactor):
         too_old = epoch2datetime(_reactor.seconds() - 60 * EXPIRE_MINUTES)
         masters = yield self.master.db.masters.getMasters()
         for m in masters:
-            if not forceHouseKeeping and m['last_active'] is not None and m['last_active'] >= too_old:
+            if m['last_active'] is not None and m['last_active'] >= too_old:
                 continue
 
             # mark the master inactive, and send a message on its behalf


Test patch:
Repository: buildbot/buildbot

Issue:
Broken links to and unsuccessful docs building at Read the Docs
PDFs unavailable.

http://media.readthedocs.org/pdf/buildbot/v1.3.0/buildbot.pdf
https://docs.buildbot.net/
https://readthedocs.org/projects/buildbot/
https://readthedocs.org/projects/buildbot/downloads/
https://readthedocs.org/projects/buildbot/builds/


Discussion:

Resolving patch:
diff --git a/master/setup.py b/master/setup.py
--- a/master/setup.py
+++ b/master/setup.py
@@ -540,7 +540,8 @@ def define_plugin_entries(groups):
         'pyenchant',
         'docutils>=0.8',
         'sphinx-jinja',
-        'towncrier'
+        'towncrier',
+        'yaml'
     ],
 }
 


Test patch:
Repository: buildbot/buildbot

Issue:
Custom templates still not working
Hi, this is the original issue with broken custom templates #4980

But it doesn't work even after the fix (2.4.1).
The web part of Buildbot is far to complicated for me. But I was able to find lines like this in scripts.js?_1568233606304
```
    , function(e, t) {
        e.exports = window.T["undefined/properties.html"] || '<table class="table table-hover...
    }
```
And I presume there is something wrong if there is "**undefined**/properties.html".


Discussion:

Resolving patch:
diff --git a/master/buildbot/www/config.py b/master/buildbot/www/config.py
--- a/master/buildbot/www/config.py
+++ b/master/buildbot/www/config.py
@@ -91,7 +91,7 @@ def parseCustomTemplateDir(self, template_dir):
                         compiler = pyjade.ext.html.Compiler(
                             block, pretty=False)
                         html = compiler.compile()
-                res[template_name % (basename,)] = json.dumps(html)
+                res[template_name % (basename,)] = html
 
         return res
 


Test patch:
diff --git a/master/buildbot/test/unit/test_www_config.py b/master/buildbot/test/unit/test_www_config.py
--- a/master/buildbot/test/unit/test_www_config.py
+++ b/master/buildbot/test/unit/test_www_config.py
@@ -78,14 +78,14 @@ def test_render(self):
         self.assertEqual(res, exp)
 
     def test_parseCustomTemplateDir(self):
-        exp = {'views/builds.html': json.dumps('<div>\n</div>')}
+        exp = {'views/builds.html': '<div>\n</div>'}
         try:
             # we make the test work if pyjade is present or note
             # It is better than just skip if pyjade is not there
             import pyjade
             [pyjade]
             exp.update({'plugin/views/plugin.html':
-                        json.dumps('<div class="myclass"><pre>this is customized</pre></div>')})
+                        '<div class="myclass"><pre>this is customized</pre></div>'})
         except ImportError:
             log.msg("Only testing html based template override")
         template_dir = util.sibpath(__file__, "test_templates_dir")
Repository: buildbot/buildbot

Issue:
buildbot-2.6.0.gitarchive.tar.gz is not generated correctly
```
<rjarry> tardyp: the .gitarchive.tar.gz file of release 2.6.0 is corrupted
16:57:31 <rjarry> buildbot-v2.5.1.gitarchive.tar.gz
16:57:31 <rjarry> 4.69 MB
16:57:31 <rjarry> buildbot-v2.5.1.gitarchive.tar.gz.sig
16:57:31 <rjarry> 310 Bytes
16:57:47 <rjarry> last one does not
16:57:49 <rjarry> buildbot-2.6.0.gitarchive.tar.gz
16:57:49 <rjarry> 15 Bytes
16:57:49 <rjarry> buildbot-2.6.0.gitarchive.tar.gz.sig
16:57:49 <rjarry> 310 Bytes
```
We shall regenerate the archive and signature manually.
preferably @p12tic , which originaly signed the rest of the release

This archive is used by debian packaging as they require to build everything from source


Discussion:

Resolving patch:
diff --git a/common/download_release.py b/common/download_release.py
--- a/common/download_release.py
+++ b/common/download_release.py
@@ -6,18 +6,19 @@
 import yaml
 
 
-def download(url, fn):
-    print(url, fn)
+def download(session, url, fn):
     if os.path.exists(fn):
-        return
+        print('Removing old file {}'.format(fn))
+        os.unlink(fn)
+    print('Downloading {} from {}'.format(fn, url))
     with open(fn, 'wb') as f:
-        r = s.get(url, stream=True)
+        r = session.get(url, stream=True)
+        r.raise_for_status()
         for c in r.iter_content(1024):
             f.write(c)
 
 
 def main():
-    global s
     with open(os.path.expanduser("~/.config/hub")) as f:
         conf = yaml.safe_load(f)
         token = conf['github.com'][0]['oauth_token']
@@ -32,15 +33,15 @@ def main():
     assets = s.get("https://api.github.com/repos/buildbot/buildbot/releases/{id}/assets".format(id=r['id']))
     assets.raise_for_status()
     assets = assets.json()
-    os.system("mkdir -p dist")
+    os.makedirs('dist', exist_ok=True)
     for url in (a['browser_download_url'] for a in assets):
         if url.endswith(".whl") or url.endswith(".tar.gz"):
             fn = os.path.join('dist', url.split('/')[-1])
-            download(url, fn)
+            download(s, url, fn)
     # download tag archive
     url = "https://github.com/buildbot/buildbot/archive/{tag}.tar.gz".format(tag=tag)
     fn = os.path.join('dist', "buildbot-{tag}.gitarchive.tar.gz".format(tag=tag))
-    download(url, fn)
+    download(s, url, fn)
     sigfn = fn + ".sig"
     if os.path.exists(sigfn):
         os.unlink(sigfn)


Test patch:
Repository: buildbot/buildbot

Issue:
Cache-control header is filled incorrectly
When a cache-control header is formed a ';' character is used as a delimiter:

https://github.com/buildbot/buildbot/blob/144eb7e82dc261e6506f1f68493446bcb24d77a0/master/buildbot/www/config.py#L120

This is not allowed by [RFC 7234](https://tools.ietf.org/html/rfc7234). The RFC states the following format of the header:
```
Cache-Control = *( "," OWS ) cache-directive *( OWS "," [ OWS
    cache-directive ] )
 cache-directive = token [ "=" ( token / quoted-string ) ]
```

Thus a replace `;` -> `, ` is required.


Discussion:
Hi @ikoveshnikov 
To me it seems you are right. 

I just wonder what issue you have observed you figured this problem?
Any http proxy/browser issue you have seen?
It might be worth to mention in release notes if someone else has the same environment.

Well, it happens on our web proxy server which is still under development, so I don't think anyone has the same environment. 

After I put it in front of buildbot I observed that buildbot responses are filtered out because of wrong `Cache-control` format. A little investigation showed me that our server follows RFC definitions and the buildbot response was malformed. In the same same time browsers and other proxy servers doesn't have such strict rules for `Cache-control` headers and buildbot website works well with them (tried: firefox, chrome, nginx). Probably they just ignore parsing errors in `Cache-control` header and treat response as non-cacheable, not really sure here.

So I came to conclusion that our server should be updated to follow not only the RFC but also behaviour of other servers/clients. And there is an issue in open-source project I cant report.

I didn't dig deep into buildbot sources and the only string `public;max-age=0` I could grep is on that line. I have no idea, why the header is set for `request` while I get this header only on first response from buildbot.

Resolving patch:
diff --git a/master/buildbot/www/config.py b/master/buildbot/www/config.py
--- a/master/buildbot/www/config.py
+++ b/master/buildbot/www/config.py
@@ -117,7 +117,7 @@ def getEnvironmentVersions():
     def renderIndex(self, request):
         config = {}
         request.setHeader(b"content-type", b'text/html')
-        request.setHeader(b"Cache-Control", b"public;max-age=0")
+        request.setHeader(b"Cache-Control", b"public,max-age=0")
 
         try:
             yield self.config['auth'].maybeAutoLogin(request)


Test patch:
Repository: buildbot/buildbot

Issue:
Windows service fails after upgrade from 2.7.0 to 2.8.1
Hello,
Windows 7
Buildbot 2.8.1
Twisted 20.3.0
pywin32 227

I upgraded my Buildbot master from version 2.7.0 to version 2.8.1.
Buildbot is configured to run as a Windows service. After the upgrade, the service starts then quickly stops. 
The twistd.log does not get written to.
The Windows Event log(Application) shows:
```
The instance's SvcRun() method failed 
<Error getting traceback - traceback.print_exception() failed 
%2: %3
```
The Windows Event log(System) shows:
```
The BuildBot service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.
```

buildbot/master/buildbot/scripts/windows_service.py (line 233)
`cmd = '{} --spawn {} start --nodaemon {}'.format(self.runner_prefix, hstop, bbdir)`
needs to be changed to:
`cmd = '{} --spawn {} start --nodaemon {}'.format(self.runner_prefix, int(hstop), bbdir)`

Notice the call to int(hstop)
The first version produces something like:
`python.exe --spawn <PyHANDLE:456> start --nodaemon c:/master`
instead of:
`python.exe" --spawn 456 start --nodaemon c:/master`
Thanks,
Joe













Discussion:

Resolving patch:
diff --git a/master/buildbot/scripts/windows_service.py b/master/buildbot/scripts/windows_service.py
--- a/master/buildbot/scripts/windows_service.py
+++ b/master/buildbot/scripts/windows_service.py
@@ -230,7 +230,7 @@ def SvcDoRun(self):
             self.info("Starting BuildBot in directory '{}'".format(bbdir))
             hstop = self.hWaitStop
 
-            cmd = '{} --spawn {} start --nodaemon {}'.format(self.runner_prefix, hstop, bbdir)
+            cmd = '{} --spawn {} start --nodaemon {}'.format(self.runner_prefix, int(hstop), bbdir)
             # print "cmd is", cmd
             h, t, output = self.createProcess(cmd)
             child_infos.append((bbdir, h, t, output))


Test patch:
Repository: buildbot/buildbot

Issue:
threading.Thread.isAlive has been deprecated and removed in Python 3.9 in favour of is_alive
https://github.com/buildbot/buildbot/blob/5e302c011332c69712e62166785a5f7459250d78/master/buildbot/scripts/windows_service.py#L299
https://github.com/buildbot/buildbot/blob/5e302c011332c69712e62166785a5f7459250d78/worker/buildbot_worker/scripts/windows_service.py#L304


Discussion:

Resolving patch:
diff --git a/master/buildbot/scripts/windows_service.py b/master/buildbot/scripts/windows_service.py
--- a/master/buildbot/scripts/windows_service.py
+++ b/master/buildbot/scripts/windows_service.py
@@ -296,7 +296,7 @@ def SvcDoRun(self):
             for i in range(5):
                 t.join(1)
                 self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
-                if not t.isAlive():
+                if not t.is_alive():
                     break
             else:
                 self.warning("Redirect thread did not stop!")
diff --git a/worker/buildbot_worker/scripts/windows_service.py b/worker/buildbot_worker/scripts/windows_service.py
--- a/worker/buildbot_worker/scripts/windows_service.py
+++ b/worker/buildbot_worker/scripts/windows_service.py
@@ -301,7 +301,7 @@ def SvcDoRun(self):
             for i in range(5):
                 t.join(1)
                 self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
-                if not t.isAlive():
+                if not t.is_alive():
                     break
             else:
                 self.warning("Redirect thread did not stop!")


Test patch:
Repository: buildbot/buildbot

Issue:
base64.decodestring has been deprecated and removed in Python 3.9 in favor of decodebytes
https://github.com/buildbot/buildbot/blob/144eb7e82dc261e6506f1f68493446bcb24d77a0/master/buildbot/manhole.py#L115

https://github.com/python/cpython/pull/18022


Discussion:

Resolving patch:
diff --git a/master/buildbot/manhole.py b/master/buildbot/manhole.py
--- a/master/buildbot/manhole.py
+++ b/master/buildbot/manhole.py
@@ -112,7 +112,7 @@ def checkKey(self, credentials):
                     if len(l2) < 2:
                         continue
                     try:
-                        if base64.decodestring(l2[1]) == credentials.blob:
+                        if base64.decodebytes(l2[1]) == credentials.blob:
                             return 1
                     except binascii.Error:
                         continue


Test patch:
Repository: buildbot/buildbot

Issue:
Can't use secret interpolation in p4poller.P4Source and steps.P4
I tried to use secrets for perforce passwords (using buildbot 1.0.0).

First I tried to use it in the P4 change source:
p4ChangeSource = p4poller.P4Source(p4port=p4Port,
                                                           p4user=p4User,
                                                           p4passwd=util.Secret('p4passwd'),
                                                           ...)
which fails with:
... [-] P4 poll failed ..
...
exceptions.TypeError: Arguments contain a non-string value

Then I tried to use the secret in the P4 source checkout step:
factory.addStep(steps.P4(p4port=p4Port,
                         ...,
                         p4passwd=util.Secret('p4passwd'),
                         ....)
This fails with:
Upon execvpe p4 ['p4', ..., '-P', Unpersistable('Unpersistable data: instance of class buildbot.process.properties.Secret deemed insecure'), ...] in environment id 56699632
:Traceback (most recent call last):
  File "/.../internet/process.py", line 445, in _fork environment)
  File "/.../internet/process.py", line 523, in _execChild os.execvpe(executable, args, environment)
  File "/.../lib/python2.7/os.py", line 353, in execvpe _execvpe(file, args, env)
  File "/.../lib/python2.7/os.py", line 380, in _execvpe func(fullname, *argrest)
TypeError: coercing to Unicode: need string or buffer, instance found

I suppose, the secrets are still not implemented for perforce functions.
Or am I doing something wrong?


Discussion:
`p4passwd` is not renderable in these two class.
This issue is still present in Buildbot version: 1.7.0

Resolving patch:
diff --git a/master/buildbot/steps/source/p4.py b/master/buildbot/steps/source/p4.py
--- a/master/buildbot/steps/source/p4.py
+++ b/master/buildbot/steps/source/p4.py
@@ -44,7 +44,7 @@ class P4(Source):
 
     name = 'p4'
 
-    renderables = ['mode', 'p4base', 'p4client', 'p4viewspec', 'p4branch']
+    renderables = ['mode', 'p4base', 'p4client', 'p4viewspec', 'p4branch', 'p4passwd']
     possible_modes = ('incremental', 'full')
 
     def __init__(self, mode='incremental',


Test patch:
Repository: buildbot/buildbot

Issue:
Cannot filter on started_at in REST API
I wanted to list all builds in the last week using the REST API.  `.../api/v2/builds?order=-started` gets the list of all builds ever sorted in date order, but adding for example `started_at__ge=1598375285` to filter to just builds started after a specific timestamp results in an `invalid filter value for started_at__ge`.



Discussion:
Can you give some guidelines to reproduce?

Resolving patch:
diff --git a/master/buildbot/data/types.py b/master/buildbot/data/types.py
--- a/master/buildbot/data/types.py
+++ b/master/buildbot/data/types.py
@@ -116,8 +116,25 @@ def valueFromString(self, arg):
 
 
 class DateTime(Instance):
+
     name = "datetime"
-    types = (datetime.datetime)
+    types = (datetime.datetime,)
+
+    def valueFromString(self, arg):
+        return int(arg)
+
+    def validate(self, name, object):
+        if isinstance(object, datetime.datetime):
+            return
+        if isinstance(object, int):
+            try:
+                datetime.datetime.fromtimestamp(object)
+            except (OverflowError, OSError):
+                pass
+            else:
+                return
+        yield "{} ({}) is not a valid timestamp".format(name, object)
+
     ramlType = "date"
 
 


Test patch:
diff --git a/master/buildbot/test/unit/data/test_types.py b/master/buildbot/test/unit/data/test_types.py
--- a/master/buildbot/test/unit/data/test_types.py
+++ b/master/buildbot/test/unit/data/test_types.py
@@ -13,6 +13,8 @@
 #
 # Copyright Buildbot Team Members
 
+from datetime import datetime
+
 from twisted.trial import unittest
 
 from buildbot.data import types
@@ -78,6 +80,17 @@ class Integer(TypeMixin, unittest.TestCase):
     cmpResults = [(10, '9', 1), (-2, '-1', -1)]
 
 
+class DateTime(TypeMixin, unittest.TestCase):
+
+    klass = types.DateTime
+    good = [0, 1604843464, datetime(2020, 11, 15, 18, 40, 1, 630219)]
+    bad = [int(1e60), 'bad', 1604843464.388657]
+    stringValues = [
+        ('1604843464', 1604843464),
+    ]
+    badStringValues = ['one', '', '0x10']
+
+
 class String(TypeMixin, unittest.TestCase):
 
     klass = types.String
Repository: buildbot/buildbot

Issue:
Fix reference to tuplematch in base reporters
The tuplematch object was not referenced in the ReporterBase class when referenced from the GitHubStatusPush class.

It seems this is not an issue when it is executed from the test but only occures when called from GitHubStatusPush. I don't know how I can test this.

Fixes #5765 


## Contributor Checklist:

* [ ] I have updated the unit tests
* [x] I have created a file in the `master/buildbot/newsfragments` directory (and read the `README.txt` in that directory)
* [x] I have updated the appropriate documentation



Discussion:

Resolving patch:
diff --git a/master/buildbot/reporters/base.py b/master/buildbot/reporters/base.py
--- a/master/buildbot/reporters/base.py
+++ b/master/buildbot/reporters/base.py
@@ -19,9 +19,9 @@
 from twisted.python import log
 
 from buildbot import config
-from buildbot import util
 from buildbot.reporters import utils
 from buildbot.util import service
+from buildbot.util import tuplematch
 
 ENCODING = 'utf-8'
 
@@ -75,7 +75,7 @@ def stopService(self):
 
     def _does_generator_want_key(self, generator, key):
         for filter in generator.wanted_event_keys:
-            if util.tuplematch.matchTuple(key, filter):
+            if tuplematch.matchTuple(key, filter):
                 return True
         return False
 


Test patch:
Repository: buildbot/buildbot

Issue:
Send keepalive messages on websocket to prevent cloudflare disconnect
buildbot behind cloudflare is currently reloading for me every 100 seconds. This is due to a cloudflare limitation that is resetting websockets with 100 seconds of inactivity. It seems the only ways around that is either an upgrade to their enterprise plan or adding keepalive messages to the application.

https://community.cloudflare.com/t/cloudflare-websocket-timeout/5865

I'm on buildbot 1.0.0 due to openbsd -stable, I've grepped through the source on master and it seems there's already a ping command implemented on the server, but it's not automatically used by the client yet. The client should invoke it every 60 seconds to avoid a disconnect due to timeouts.

Thanks!


Discussion:
This is also true for AWS ALB. The default active connection timeout is 60 seconds (instructions on how to change it: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout).
Sending pings every X seconds sounds good to me. I suggest ~45 seconds as the default value, but it would be nice if it's easily configurable.
This is same when proxying though nginx, as it sets [default timeout to 60 seconds](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout)

Resolving patch:
diff --git a/master/buildbot/www/ws.py b/master/buildbot/www/ws.py
--- a/master/buildbot/www/ws.py
+++ b/master/buildbot/www/ws.py
@@ -135,7 +135,8 @@ class WsProtocolFactory(WebSocketServerFactory):
     def __init__(self, master):
         super().__init__()
         self.master = master
-        self.setProtocolOptions(webStatus=False)
+        pingInterval = self.master.config.www.get('ws_ping_interval', 0)
+        self.setProtocolOptions(webStatus=False, autoPingInterval=pingInterval)
 
     def buildProtocol(self, addr):
         p = WsProtocol(self.master)


Test patch:
Repository: buildbot/buildbot

Issue:
AttributeError: module 'sqlalchemy.engine.strategies' has no attribute 'PlainEngineStrategy'
On buildbot version 3.0.1, since upgrading SQLAlchemy to version 1.4, `buildbot create-master` now throws an error:

```
Traceback (most recent call last):
  File "/opt/buildbot/master/venv/bin/buildbot", line 8, in <module>
    sys.exit(run())
  File "/opt/buildbot/master/venv/lib/python3.8/site-packages/buildbot/scripts/runner.py", line 772, in run
    subcommandFunction = reflect.namedObject(subconfig.subcommandFunction)
  File "/opt/buildbot/master/venv/lib/python3.8/site-packages/twisted/python/reflect.py", line 170, in namedObject
    module = namedModule(".".join(classSplit[:-1]))
  File "/opt/buildbot/master/venv/lib/python3.8/site-packages/twisted/python/reflect.py", line 157, in namedModule
    topLevel = __import__(name)
  File "/opt/buildbot/master/venv/lib/python3.8/site-packages/buildbot/scripts/create_master.py", line 25, in <module>
    from buildbot.master import BuildMaster
  File "/opt/buildbot/master/venv/lib/python3.8/site-packages/buildbot/master.py", line 35, in <module>
    from buildbot.db import connector as dbconnector
  File "/opt/buildbot/master/venv/lib/python3.8/site-packages/buildbot/db/connector.py", line 30, in <module>
    from buildbot.db import enginestrategy
  File "/opt/buildbot/master/venv/lib/python3.8/site-packages/buildbot/db/enginestrategy.py", line 154, in <module>
    class BuildbotEngineStrategy(strategies.PlainEngineStrategy):
AttributeError: module 'sqlalchemy.engine.strategies' has no attribute 'PlainEngineStrategy'
````

Restricting the sqlalchemy version to 1.3.23 seems to resolve this issue.

Seems related to the removal of this interface in SQLAlchemy 1.4: https://github.com/sqlalchemy/sqlalchemy/commit/dfb20f07d8796ec27732df84c40b4ce4857fd83b


Discussion:

Resolving patch:
diff --git a/master/setup.py b/master/setup.py
--- a/master/setup.py
+++ b/master/setup.py
@@ -491,7 +491,7 @@ def define_plugin_entries(groups):
     'Jinja2 >= 2.1',
     # required for tests, but Twisted requires this anyway
     'zope.interface >= 4.1.1',
-    'sqlalchemy>=1.2.0',
+    'sqlalchemy >= 1.2.0, < 1.4',
     'sqlalchemy-migrate>=0.13',
     'python-dateutil>=1.5',
     'txaio ' + txaio_ver,


Test patch: