internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: buildbot/buildbot

Issue:
UnboundLocalError in mq/base.py on master shutdown
Hello,

We're using buildbot in multi-master mode and got this stacktrace on one of the master when shutting it down:
```
2017-07-17 12:33:29+0000 [-] Waiting for 1 build(s) to finish
2017-07-17 12:33:29+0000 [-] Builder <Builder 'u'sql-monitor-bitbucket_scality_ring-monitor_ring_frequent-prod-frontend-0'' at 140555339856784> has 1 builds running
2017-07-17 12:33:29+0000 [-] Not shutting down, there are 1 builds running
2017-07-17 12:33:29+0000 [-] Trying shutdown sequence again
2017-07-17 12:33:30+0000 [-] <Build sql-monitor-bitbucket_scality_ring-monitor_ring_frequent-prod-frontend-0 number:32108L results:exception>: stopping build: Master Shutdown 5
2017-07-17 12:33:30+0000 [-] Unhandled error in Deferred:
2017-07-17 12:33:30+0000 [-] Unhandled Error
Traceback (most recent call last):
File "/root/bitbucket/scality/ring/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = g.send(result)
File "/root/bitbucket/scality/ring/venv/local/lib/python2.7/site-packages/buildbot/process/botmaster.py", line 105, in cleanShutdown
l.append(build.waitUntilFinished())
File "/root/bitbucket/scality/ring/venv/local/lib/python2.7/site-packages/buildbot/process/build.py", line 687, in waitUntilFinished
lambda: self.finished)
File "/root/bitbucket/scality/ring/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1445, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
— <exception caught here> —
File "/root/bitbucket/scality/ring/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
result = g.send(result)
File "/root/bitbucket/scality/ring/venv/local/lib/python2.7/site-packages/buildbot/mq/base.py", line 40, in waitUntilEvent
defer.returnValue(res)
exceptions.UnboundLocalError: local variable 'res' referenced before assignment
```
Looking at the code at the end of `waitUntilEvent()`:
```
        if not check:
            res = yield d
        yield buildCompleteConsumer.stopConsuming
        defer.returnValue(res)
```

If the check returned false, we try to return a value (`res`) that was never defined.


Discussion:

Resolving patch:
diff --git a/master/buildbot/mq/base.py b/master/buildbot/mq/base.py
--- a/master/buildbot/mq/base.py
+++ b/master/buildbot/mq/base.py
@@ -36,7 +36,9 @@ def waitUntilEvent(self, filter, check_callback):
         # we only wait if the check callback return true
         if not check:
             res = yield d
-        yield buildCompleteConsumer.stopConsuming
+        else:
+            res = None
+        yield buildCompleteConsumer.stopConsuming()
         defer.returnValue(res)
 
 


Test patch:
Repository: buildbot/buildbot

Issue:
Add a `region` arg to `OpenStackLatentWorker`
Would be nice to allow passing a region_name to novaclient in `OpenStackLatentWorker`.


Discussion:
@seankelly isn't that possible already?
There is `nova_args` for arbitrary options, but I think that's only for creating an instance. I think it's not possible right now.
I recall cmouse raising this in irc too so I think it requires code changes.
What we currently do is inherit from `OpenStackLatentWorker` and override the `__init__()` to add `self.novaclient.client.region_name = region` at the end.

Resolving patch:
diff --git a/master/buildbot/worker/openstack.py b/master/buildbot/worker/openstack.py
--- a/master/buildbot/worker/openstack.py
+++ b/master/buildbot/worker/openstack.py
@@ -60,6 +60,7 @@ def __init__(self, name, password,
                  os_tenant_name,
                  os_auth_url,
                  block_devices=None,
+                 region=None,
                  image=None,
                  meta=None,
                  # Have a nova_args parameter to allow passing things directly
@@ -88,6 +89,8 @@ def __init__(self, name, password,
             self.novaclient = self._constructClient(
                 client_version, os_username, os_password, os_tenant_name,
                 os_auth_url)
+            if region is not None:
+                self.novaclient.client.region_name = region
 
         if block_devices is not None:
             self.block_devices = [


Test patch:
diff --git a/master/buildbot/test/fake/openstack.py b/master/buildbot/test/fake/openstack.py
--- a/master/buildbot/test/fake/openstack.py
+++ b/master/buildbot/test/fake/openstack.py
@@ -32,6 +32,10 @@
 }
 
 
+class FakeNovaClient():
+    region_name = ""
+
+
 # Parts used from novaclient
 class Client():
 
@@ -43,6 +47,7 @@ def __init__(self, version, session):
         self.volume_snapshots = ItemManager()
         self.volume_snapshots._add_items([Snapshot(TEST_UUIDS['snapshot'], 'CirrOS 0.3.4', 2)])
         self.servers = Servers()
+        self.client = FakeNovaClient()
 
 
 class ItemManager():
diff --git a/master/buildbot/test/unit/test_worker_openstack.py b/master/buildbot/test/unit/test_worker_openstack.py
--- a/master/buildbot/test/unit/test_worker_openstack.py
+++ b/master/buildbot/test/unit/test_worker_openstack.py
@@ -71,6 +71,11 @@ def test_constructor_minimal(self):
         self.assertEqual(bs.block_devices, None)
         self.assertIsInstance(bs.novaclient, novaclient.Client)
 
+    def test_constructor_region(self):
+        bs = openstack.OpenStackLatentWorker(
+            'bot', 'pass', region="test-region", **self.bs_image_args)
+        self.assertEqual(bs.novaclient.client.region_name, "test-region")
+
     def test_constructor_block_devices_default(self):
         block_devices = [{'uuid': 'uuid', 'volume_size': 10}]
         bs = openstack.OpenStackLatentWorker('bot', 'pass', flavor=1,
Repository: buildbot/buildbot

Issue:
Buildbot should check for duplicates in addURL 
In a step if we call self.addURL() multiple times with same URL, buildbot doesn't check if URL is already added, and adds the duplicate URL. Buildbot should check if the URL is already added before adding the URL.

This happens when a script parses a lot of output and generate links for various things. We can manually maintain a list ourself and check for duplicates before calling addURL(), but it feels so basic that buildbot should be doing this check instead of callers of addURL() checking for duplicates.



Discussion:
feel free to add this improvement. 

Resolving patch:
diff --git a/master/buildbot/db/steps.py b/master/buildbot/db/steps.py
--- a/master/buildbot/db/steps.py
+++ b/master/buildbot/db/steps.py
@@ -151,10 +151,13 @@ def thd(conn):
             if _racehook is not None:
                 _racehook()
             urls = json.loads(row.urls_json)
-            urls.append(dict(name=name, url=url))
 
-            q = tbl.update(whereclause=wc)
-            conn.execute(q, urls_json=json.dumps(urls))
+            url_item = dict(name=name, url=url)
+
+            if url_item not in urls:
+                urls.append(url_item)
+                q = tbl.update(whereclause=wc)
+                conn.execute(q, urls_json=json.dumps(urls))
 
         return self.url_lock.run(lambda: self.db.pool.do(thd))
 


Test patch:
diff --git a/master/buildbot/test/fake/fakedb.py b/master/buildbot/test/fake/fakedb.py
--- a/master/buildbot/test/fake/fakedb.py
+++ b/master/buildbot/test/fake/fakedb.py
@@ -2063,7 +2063,9 @@ def addURL(self, stepid, name, url, _racehook=None):
         b = self.steps.get(stepid)
         if b:
             urls = json.loads(b['urls_json'])
-            urls.append(dict(name=name, url=url))
+            url_item = dict(name=name, url=url)
+            if url_item not in urls:
+                urls.append(url_item)
             b['urls_json'] = json.dumps(urls)
         return defer.succeed(None)
 
diff --git a/master/buildbot/test/unit/test_db_steps.py b/master/buildbot/test/unit/test_db_steps.py
--- a/master/buildbot/test/unit/test_db_steps.py
+++ b/master/buildbot/test/unit/test_db_steps.py
@@ -272,6 +272,18 @@ def urlKey(url):
                                  {'name': u'foo2', 'url': u'bar2'}],
                                 key=urlKey))
 
+    @defer.inlineCallbacks
+    def test_addURL_no_duplicate(self):
+        yield self.insertTestData(self.backgroundData + [self.stepRows[2]])
+        yield defer.gatherResults([
+            self.db.steps.addURL(stepid=72, name=u'foo', url=u'bar'),
+            self.db.steps.addURL(stepid=72, name=u'foo', url=u'bar')])
+
+        stepdict = yield self.db.steps.getStep(stepid=72)
+
+        self.assertEqual(stepdict['urls'],
+                         [{'name': u'foo', 'url': u'bar'}])
+
     @defer.inlineCallbacks
     def test_finishStep(self):
         clock = task.Clock()
Repository: buildbot/buildbot

Issue:
Running with --nodaemon and PID files
This ticket is a migrated Trac ticket [3608](http://trac.buildbot.net/ticket/3608)

People contributed to the original ticket: @unknown_contributor
Ticket created on: `Sep 12 2016`
Ticket last modified on: `Sep 12 2016`

---

Both the master and the worker still use PID files while running with --nodaemon. I've had issues with workers running with --nodaemon not starting because of an old, stale PID file, corresponding to an actual process belonging to another user.

This can be thought to be inconsistent, and harms the writing ofreally simple non forking configuration files to manage the process through an external service manager (systemd, supervisord etc.).

It seems that the cousin ``twistd`` executable has a --pidfile argument, that can be used to mean "no pid file". This is explained notably in https://twistedmatrix.com/documents/current/core/howto/systemd.html.

We could expose that, or we could decide that --nodaemon implies no pid file.

I believe this is true in Eight and Nine, and that a fix could be easily backported
(marking this as minor, because it's easy to circumvent by putting appropriate rms in unit files or other confs)


---





Discussion:

Resolving patch:
diff --git a/master/buildbot/scripts/start.py b/master/buildbot/scripts/start.py
--- a/master/buildbot/scripts/start.py
+++ b/master/buildbot/scripts/start.py
@@ -84,9 +84,14 @@ def launchNoDaemon(config):
 
     argv = ["twistd",
             "--no_save",
-            '--nodaemon',
+            "--nodaemon",
             "--logfile=twistd.log",  # windows doesn't use the same default
             "--python=buildbot.tac"]
+
+    if platformType != 'win32':
+        # windows doesn't use pidfile option.
+        argv.extend(["--pidfile="])
+
     sys.argv = argv
 
     # this is copied from bin/twistd. twisted-2.0.0 through 2.4.0 use
diff --git a/worker/buildbot_worker/scripts/start.py b/worker/buildbot_worker/scripts/start.py
--- a/worker/buildbot_worker/scripts/start.py
+++ b/worker/buildbot_worker/scripts/start.py
@@ -131,7 +131,11 @@ def launch(nodaemon):
             "--logfile=twistd.log",  # windows doesn't use the same default
             "--python=buildbot.tac"]
     if nodaemon:
-        argv.extend(['--nodaemon'])
+        argv.extend(["--nodaemon"])
+        if platformType != 'win32':
+            # windows doesn't use pidfile option.
+            argv.extend(["--pidfile="])
+
     sys.argv = argv
 
     # this is copied from bin/twistd. twisted-2.0.0 through 2.4.0 use


Test patch:
Repository: buildbot/buildbot

Issue:
EC2LatentWorker ignores tags parameter when spot_instance is True
Currently EC2LatentWorker only sets EC2 Tags on an instance when spot_instance is False.  When spot_instance is True, the tags parameter is ignored.

It looks like it is any easy fix to move the call to self.instance.create_tags() into _wait_for_instance(), and then it will apply to both normal and spot instances.  I'll submit a patch.



Discussion:

Resolving patch:
diff --git a/master/buildbot/worker/ec2.py b/master/buildbot/worker/ec2.py
--- a/master/buildbot/worker/ec2.py
+++ b/master/buildbot/worker/ec2.py
@@ -413,9 +413,6 @@ def _start_instance(self):
         self.instance = reservations[0]
         instance_id, start_time = self._wait_for_instance()
         if None not in [instance_id, image.id, start_time]:
-            if self.tags:
-                self.instance.create_tags(Tags=[{"Key": k, "Value": v}
-                                                for k, v in self.tags.items()])
             return [instance_id, image.id, start_time]
         else:
             self.failed_to_start(self.instance.id, self.instance.state['Name'])
@@ -558,6 +555,9 @@ def _wait_for_instance(self):
                 minutes // 60, minutes % 60, seconds)
             if self.volumes:
                 self._attach_volumes()
+            if self.tags:
+                self.instance.create_tags(Tags=[{"Key": k, "Value": v}
+                                                for k, v in self.tags.items()])
             return self.instance.id, start_time
         else:
             self.failed_to_start(self.instance.id, self.instance.state['Name'])


Test patch:
Repository: buildbot/buildbot

Issue:
Two mail.MailNotifier instances with the same auto-generated name
Thank you guys, BuildBot is great!

I have found something like a bug.
I have **two** mail.MailNotifiers in my master.cfg with only **two differences**:
- a list of **recipients**
- a boolean parameter **buildSetSummary**

but BuildBot generates the same name for both:

`buildbot/util/service.py:64: self.name = MailNotifier_builders_Check REST [debug]+Check REST [debug]/1 Check clients/test-otc-1658failing_passing_warnings
`

I have fixed it setting theirs names explicitly.
But the behavior is **misleading**.

Many thanks for the people from freenode/#buildbot


Discussion:

Resolving patch:
diff --git a/master/buildbot/config.py b/master/buildbot/config.py
--- a/master/buildbot/config.py
+++ b/master/buildbot/config.py
@@ -848,6 +848,10 @@ def load_services(self, filename, config_dict):
 
                 continue
 
+            if _service.name in self.services:
+                error('Duplicate service name %r' % _service.name)
+                continue
+
             self.services[_service.name] = _service
 
     def check_single_master(self):


Test patch:
diff --git a/master/buildbot/test/unit/test_config.py b/master/buildbot/test/unit/test_config.py
--- a/master/buildbot/test/unit/test_config.py
+++ b/master/buildbot/test/unit/test_config.py
@@ -1052,6 +1052,20 @@ class MyService(object):
         errMsg += "object should be an instance of buildbot.util.service.BuildbotService"
         self.assertConfigError(self.errors, errMsg)
 
+    def test_load_services_duplicate(self):
+
+        class MyService(service.BuildbotService):
+            name = 'myservice'
+
+            def reconfigService(self, x=None):
+                self.x = x
+
+        self.cfg.load_services(self.filename, dict(
+            services=[MyService(x='a'), MyService(x='b')]))
+
+        self.assertConfigError(
+            self.errors, 'Duplicate service name %r' % MyService.name)
+
     def test_load_configurators_norminal(self):
 
         class MyConfigurator(configurators.ConfiguratorBase):
Repository: buildbot/buildbot

Issue:
Authentication problem Buildbot 0.9.14
I created the class below
```
class MyAuth(CustomAuth):
    def check_credentials(user, password):
        if user == 'snow' and password == 'white':
            return True
        else:
            return False
```
and set it as my auth class.
```
c['www']['auth']=MyAuth()
```
But it throws following exception.
```
web.Server Traceback (most recent call last):
exceptions.AttributeError: 'str' object has no attribute 'providedBy'
/home/buildbot/virtualenv.buildbot/local/lib/python2.7/site-packages/twisted/web/server.py:195 in process
194                    self._encoder = encoder
195            self.render(resrc)
196        except:
/home/buildbot/virtualenv.buildbot/local/lib/python2.7/site-packages/twisted/web/server.py:255 in render
254        try:
255            body = resrc.render(self)
256        except UnsupportedMethod as e:
/home/buildbot/virtualenv.buildbot/local/lib/python2.7/site-packages/twisted/web/_auth/wrapper.py:138 in render
137        """
138        return self._authorizedResource(request).render(request)
139
/home/buildbot/virtualenv.buildbot/local/lib/python2.7/site-packages/twisted/web/_auth/wrapper.py:116 in _authorizedResource
115        if not authheader:
116            return util.DeferredResource(self._login(Anonymous()))
117
/home/buildbot/virtualenv.buildbot/local/lib/python2.7/site-packages/twisted/web/_auth/wrapper.py:162 in _login
161        """
162        d = self._portal.login(credentials, None, IResource)
163        d.addCallbacks(self._loginSucceeded, self._loginFailed)
/home/buildbot/virtualenv.buildbot/local/lib/python2.7/site-packages/twisted/cred/portal.py:118 in login
117        for i in self.checkers:
118            if i.providedBy(credentials):
119                return maybeDeferred(self.checkers[i].requestAvatarId, credentials
exceptions.AttributeError: 'str' object has no attribute 'providedBy'
```


Discussion:

Resolving patch:
diff --git a/master/buildbot/www/auth.py b/master/buildbot/www/auth.py
--- a/master/buildbot/www/auth.py
+++ b/master/buildbot/www/auth.py
@@ -189,7 +189,7 @@ def __init__(self, users, **kwargs):
 @implementer(ICredentialsChecker)
 class CustomAuth(TwistedICredAuthBase):
     __metaclass__ = ABCMeta
-    credentialInterfaces = IUsernamePassword
+    credentialInterfaces = [IUsernamePassword]
 
     def __init__(self, **kwargs):
         TwistedICredAuthBase.__init__(


Test patch:
Repository: buildbot/buildbot

Issue:
Badge plugin prevents UI load when doing buildbot reconfig
if the badge plugin is configured, buildbot start works and UI starts correctly.

after reconfiguration, the UI then does not load, because it tries to load badges/scripts.js, which does not exist.
 


Discussion:

Resolving patch:
diff --git a/master/buildbot/www/service.py b/master/buildbot/www/service.py
--- a/master/buildbot/www/service.py
+++ b/master/buildbot/www/service.py
@@ -259,14 +259,7 @@ def getPortnum(self):
         # was not made.
         return self._getPort().getHost().port
 
-    def setupSite(self, new_config):
-        self.reconfigurableResources = []
-
-        # we're going to need at least the base plugin (buildbot-www)
-        if 'base' not in self.apps:
-            raise RuntimeError("could not find buildbot-www; is it installed?")
-
-        root = self.apps.get('base').resource
+    def configPlugins(self, root, new_config):
         known_plugins = set(new_config.www.get('plugins', {})) | set(['base'])
         for key, plugin in list(iteritems(new_config.www.get('plugins', {}))):
             log.msg("initializing www plugin %r" % (key,))
@@ -283,6 +276,15 @@ def setupSite(self, new_config):
             log.msg("NOTE: www plugin %r is installed but not "
                     "configured" % (plugin_name,))
 
+    def setupSite(self, new_config):
+        self.reconfigurableResources = []
+
+        # we're going to need at least the base plugin (buildbot-www)
+        if 'base' not in self.apps:
+            raise RuntimeError("could not find buildbot-www; is it installed?")
+
+        root = self.apps.get('base').resource
+        self.configPlugins(root, new_config)
         # /
         root.putChild(b'', wwwconfig.IndexResource(
             self.master, self.apps.get('base').static_dir))
@@ -339,6 +341,8 @@ def resourceNeedsReconfigs(self, resource):
         self.reconfigurableResources.append(resource)
 
     def reconfigSite(self, new_config):
+        root = self.apps.get('base').resource
+        self.configPlugins(root, new_config)
         new_config.www['auth'].reconfigAuth(self.master, new_config)
         cookie_expiration_time = new_config.www.get('cookie_expiration_time')
         if cookie_expiration_time is not None:


Test patch:
Repository: buildbot/buildbot

Issue:
Web UI User settings should have defaults configurable in master.cfg
c['www']['default_ui_settings'] = {
'Builders.buildFetchLimit': 1000
}

This can be done easily by whitelisting in https://github.com/buildbot/buildbot/blob/master/master/buildbot/config.py#L807
And the the settings module in the UI can fetch its default value there (overriding the default hardcoded in the coffee)



Discussion:
This would be awesome.

Resolving patch:
diff --git a/master/buildbot/config.py b/master/buildbot/config.py
--- a/master/buildbot/config.py
+++ b/master/buildbot/config.py
@@ -804,12 +804,13 @@ def load_www(self, filename, config_dict):
         if 'www' not in config_dict:
             return
         www_cfg = config_dict['www']
-        allowed = set(['port', 'debug', 'json_cache_seconds',
-                       'rest_minimum_version', 'allowed_origins', 'jsonp',
-                       'plugins', 'auth', 'authz', 'avatar_methods', 'logfileName',
-                       'logRotateLength', 'maxRotatedFiles', 'versions',
-                       'change_hook_dialects', 'change_hook_auth',
-                       'custom_templates_dir', 'cookie_expiration_time'])
+        allowed = {'port', 'debug', 'json_cache_seconds',
+                   'rest_minimum_version', 'allowed_origins', 'jsonp',
+                   'plugins', 'auth', 'authz', 'avatar_methods', 'logfileName',
+                   'logRotateLength', 'maxRotatedFiles', 'versions',
+                   'change_hook_dialects', 'change_hook_auth',
+                   'custom_templates_dir', 'cookie_expiration_time',
+                   'ui_default_config'}
         unknown = set(list(www_cfg)) - allowed
 
         if unknown:


Test patch:
Repository: buildbot/buildbot

Issue:
MultipleFileUpload makes directory component part of the filename
When transferring files from a Windows Buildbot worker to a Linux Buildbot server via something like:

```python
MultipleFileUpload(
  glob=True,
  workersrcs='package/*.exe',
  masterdest="/mnt/buildbot/Installers")
```

Buildbot unexpectedly creates files named like e.g. `package\foo.exe` (where the backslash is part of the filename) at the destination.


Discussion:

Resolving patch:
diff --git a/master/buildbot/steps/transfer.py b/master/buildbot/steps/transfer.py
--- a/master/buildbot/steps/transfer.py
+++ b/master/buildbot/steps/transfer.py
@@ -405,16 +405,11 @@ def start(self):
 
         @defer.inlineCallbacks
         def globSources(sources):
-            dl = defer.DeferredList([
+            results = yield defer.gatherResults([
                 self.runGlob(
                     os.path.join(self.workdir, source), abandonOnFailure=False) for source in sources
             ])
-            results = yield dl
-            results = [
-                result[1]
-                for result in filter(lambda result: result[0], results)
-            ]
-            results = flatten(results)
+            results = [self.workerPathToMasterPath(p) for p in flatten(results)]
             defer.returnValue(results)
 
         @defer.inlineCallbacks
diff --git a/master/buildbot/steps/worker.py b/master/buildbot/steps/worker.py
--- a/master/buildbot/steps/worker.py
+++ b/master/buildbot/steps/worker.py
@@ -233,6 +233,9 @@ def commandComplete(self, cmd):
 
 class CompositeStepMixin():
 
+    def workerPathToMasterPath(self, path):
+        return os.path.join(*self.worker.path_module.split(path))
+
     def addLogForRemoteCommands(self, logname):
         """This method must be called by user classes
         composite steps could create several logs, this mixin functions will write


Test patch:
diff --git a/master/buildbot/test/util/steps.py b/master/buildbot/test/util/steps.py
--- a/master/buildbot/test/util/steps.py
+++ b/master/buildbot/test/util/steps.py
@@ -211,6 +211,7 @@ def getWorkerVersion(cmd, oldversion):
         # step.worker
 
         self.worker = step.worker = worker.FakeWorker(self.master)
+        self.worker.attached(None)
 
         # step overrides