Repository: buildbot/buildbot

Issue:
mailNotifier can't use Secrets
The following settings in my master.cfg prevent buildbot from starting.

```
###### SECRETS

c['secretsProviders'] = [secrets.SecretInAFile("/srv/buildbot/secrets")]

###### NOTIFIERS

mn = reporters.MailNotifier(
    fromaddr="{REDACTED}",
    sendToInterestedUsers=False,
    extraRecipients=[{REDACTED}],
    relayhost="mail3.gridhost.co.uk",
    useSmtps=True,
    smtpPort=465,
    smtpUser="{REDACTED}",
    smtpPassword=util.Interpolate("%(secret:fwm-ftp-password)s"),
    mode=('all')
)
c['services'].append(mn)
```

In the example above, I have changed the secret file being used from one which specifically represents my SMTP password to a different file which is successfully used later on in the same master.cfg so I know the file is accessible and readable by the SecretInAFile system.

I've attached the complete startup log from a typical failed startup.

[twistd.log - Error with Interpolate](https://github.com/buildbot/buildbot/files/1797122/twistd.log)

I have also attempted to use util.Secret() rather than util.Interpolate(). This also results in a failed startup but a slightly different failures. Interpolate reports NoneType errors, Secrets report KeyError. I think these different error types have the same root cause.

[twistd (1).log - Error with Secret](https://github.com/buildbot/buildbot/files/1797175/twistd.1.log)

Directly typing the password into master.cfg for smtpPassword rather than using Interpolate or Secret gives me a working mailNotifier so the rest of the mailNotifier setup is correct and the issue is in the use of Secrets.

Looking at the unittests for mailNotifier, I see there is a test for mailNotifier to be able to use Interpolate but not for it to be provided a full secret, I believe the issue here is in providing the secret rather than mailNotifier correctly swapping in strings hence the unittests passing.

Q. Looking at the mail.py file which implements mailNotifier, I see smtpUser is in the secret list as well, does that mean both fields have to be defined as secrets or is that list used to exclude fields from log messages? (don't know enough of the code to be able to tell without following it down the rabbit hole).

Trevor


Discussion:

Resolving patch:
diff --git a/master/buildbot/master.py b/master/buildbot/master.py
--- a/master/buildbot/master.py
+++ b/master/buildbot/master.py
@@ -193,6 +193,7 @@ def create_child_services(self):
 
         self.secrets_manager = SecretManager()
         self.secrets_manager.setServiceParent(self)
+        self.secrets_manager.reconfig_priority = 2000
 
         self.service_manager = service.BuildbotServiceManager()
         self.service_manager.setServiceParent(self)
diff --git a/master/buildbot/process/properties.py b/master/buildbot/process/properties.py
--- a/master/buildbot/process/properties.py
+++ b/master/buildbot/process/properties.py
@@ -487,13 +487,13 @@ def __init__(self, secret_name):
 
     @defer.inlineCallbacks
     def getRenderingFor(self, properties):
-        secretsSrv = properties.getBuild().master.namedServices.get("secrets")
+        secretsSrv = properties.master.namedServices.get("secrets")
         if not secretsSrv:
             error_message = "secrets service not started, need to configure" \
                             " SecretManager in c['services'] to use 'secrets'" \
                             "in Interpolate"
             raise KeyError(error_message)
-        credsservice = properties.getBuild().master.namedServices['secrets']
+        credsservice = properties.master.namedServices['secrets']
         secret_detail = yield credsservice.get(self.secret_name)
         if secret_detail is None:
             raise KeyError("secret key %s is not found in any provider" % self.secret_name)


Test patch:
diff --git a/master/buildbot/test/integration/interop/test_integration_secrets.py b/master/buildbot/test/integration/interop/test_integration_secrets.py
--- a/master/buildbot/test/integration/interop/test_integration_secrets.py
+++ b/master/buildbot/test/integration/interop/test_integration_secrets.py
@@ -20,15 +20,23 @@
 from twisted.internet import defer
 
 from buildbot.process.properties import Interpolate
+from buildbot.reporters.http import HttpStatusPush
 from buildbot.test.fake.secrets import FakeSecretStorage
 from buildbot.test.util.integration import RunMasterBase
 
 
+class FakeSecretReporter(HttpStatusPush):
+    def send(self, build):
+        assert self.auth == ('user', 'myhttppasswd')
+        self.reported = True
+
+
 class SecretsConfig(RunMasterBase):
 
     @defer.inlineCallbacks
     def test_secret(self):
-        yield self.setupConfig(masterConfig())
+        c = masterConfig()
+        yield self.setupConfig(c)
         build = yield self.doForceBuild(wantSteps=True, wantLogs=True)
         self.assertEqual(build['buildid'], 1)
         res = yield self.checkBuildStepLogExist(build, "echo <foo>")
@@ -38,6 +46,7 @@ def test_secret(self):
         # at this point, build contains all the log and steps info that is in the db
         # we check that our secret is not in there!
         self.assertNotIn("bar", repr(build))
+        self.assertTrue(c['services'][0].reported)
 
     @defer.inlineCallbacks
     def test_secretReconfig(self):
@@ -66,13 +75,15 @@ def masterConfig():
     from buildbot.process.factory import BuildFactory
     from buildbot.plugins import schedulers, steps
 
+    c['services'] = [FakeSecretReporter('http://example.com/hook',
+                                        auth=('user', Interpolate('%(secret:httppasswd)s')))]
     c['schedulers'] = [
         schedulers.ForceScheduler(
             name="force",
             builderNames=["testy"])]
 
     c['secretsProviders'] = [FakeSecretStorage(
-        secretdict={"foo": "bar", "something": "more"})]
+        secretdict={"foo": "bar", "something": "more", 'httppasswd': 'myhttppasswd'})]
     f = BuildFactory()
     if os.name == "posix":
         f.addStep(steps.ShellCommand(command=Interpolate(