Repository: buildbot/buildbot

Issue:
flawed HashiCorpVaultSecretProvider implementation
After unsuccessfuly trying to get vault working with buildbot I realized, that current naive implementation has several major flaws preventing it to be usable in real world scenarios. To give more context, we are using hashicorp vault in production at my work for about 5 years now, having several thousand secrets stored in generic kv store. We use typically 5 level deep paths allowing us for fine grained access control to secrets.

### Current issues:
1. separating optional key with `/` is ambiguos, as vault path components are separated by `/` as well,
2. only flat (single component) path can be used now, which is not viable real world use (it would severely hinder management of access control in vault),
3. only token authentication is supported, but even when I write some python code in config to obtain it, tokens are short lived with forced TTL. So in real world scenarios buildbot uptime will typically exceed token TTL, breaking pipelines after some time after config/restart,
4. vault paths in general may contain any characters so it is good practice to URL escape path components.

### To resolve this I suggest following changes:
1. use [HVAC](https://github.com/hvac/hvac) module for vault communication (it will help with points 3. and 4.),
2. use colon `:` for separating path from key (this convention is used in [Ansible hashi_vault lookup plugin](https://docs.ansible.com/ansible/latest/collections/community/hashi_vault/hashi_vault_lookup.html)), ideally with escaping extra colons in path or key name using `\`,
3. support additional login methods, at least *approle* which is probably most typical for CI/CD pipelines,
4. drop support for default `value` key, it's only confusing with no real world benefit; mentioned Ansible plugin returns dictionary with all keys when no key is specified, but I'm not sure if buildbot could handle this (if not, we can throw exception to enforce key specification).

I would welcome any comments or objections regarding this topic. I'll make PR with new HashiCorpVaultSecretProvider once we agree on this.


Discussion:
@mixicz Thanks for filing this bug report. I agree on all of your points.

Regarding (2), escaping sounds good, `Interpolate` should handle the subsequent colons and just pass it as part of string for the secret engine to interpret.

Regarding (4), let's just remove implicit `value` key support altogether.

Since the code of the current `HashiCorpVaultSecretProvider` is rather simple, would it be possible to implement your changes within that class? The only backwards incompatible changes are the path/key separator and getting rid of the implicit key support. I think we can handle both by adding e.g. `path_key_sep` argument that is equal to `/` by default and throws a deprecation warning. Users would need to pass `path_key_sep=':'` to get rid of the warning. No other values are supported. `path_key_sep=':'` would also turn off the default `value` key logic.

What do you think about the above?

Looking forward to your PR.
> Regarding (4), let's just remove implicit `value` key support altogether.
> 
> Since the code of the current `HashiCorpVaultSecretProvider` is rather simple, would it be possible to implement your changes within that class? The only backwards incompatible changes are the path/key separator and getting rid of the implicit key support. I think we can handle both by adding e.g. `path_key_sep` argument that is equal to `/` by default and throws a deprecation warning. Users would need to pass `path_key_sep=':'` to get rid of the warning. No other values are supported. `path_key_sep=':'` would also turn off the default `value` key logic.

While I think it is generally good practice to do such changes in backwards compatible way as you suggest, I have moral problem forcing users to specify extra parameter to switch from broken to correct behavior.

Use `path_key_sep` parameter is good idea, but I would rather set default to new `path_key_sep=':'`. For backward compatibility I'd evaluate if path contains unescaped separator, and if not, use *last* slash as path-key separator with deprecation warning (and in case no slash is present in the path, use default `value` key). I would also allow to specify any separator character, not just `:` or `/`, it might be handy feature in case someone often uses colon in vault paths.

This way it would remain backwards compatible and would not require any extra parameters for new users.
@mixicz The problem I want to avoid is to break users' configurations in a way that's difficult to understand. At least when they are explicitly fixing a deprecation warning they'll are more likely to notice issues with their secret paths. When one is upgrading from Buildbot 2.2.0 to 2.10.1 backwards incompatible changes just lead to non-working configuration and it's hard to know what exactly is wrong.

If we immediately default to `path_key_sep=':'` then we would break all configuration that use `':'` in key paths.

The requirement to have `path_key_sep` for new users will be temporary - until we remove the deprecated previous default and switch to the colon. This will happen with a new major version of Buildbot.
@p12tic I do understand your concern and while I think it's unlikely anyone has such configuration, there is no way to be sure.

I'm still not inclined to force users specifying extra parameter though. What if I create new class instead (`HashiCorpVaultHvacSecretProvider` or `HvacSecretProvider`) and old class would show deprecation message to migrate to new class? Old configuration would remain bug-to-bug compatible, new secret provider would be way cleaner not having to support old buggy implementation and easier to use without need to specify extra parameters.
@mixicz I agree that having a different class would address my concerns. It would be easier for you to implement it this way too.
@p12tic OK, I'll create new class named `HashiCorpVaultHvacSecretProvider`. I can't promise any specific timeline as this is part of my free time hobby project, but I expect 2-3 weeks to have PR tested and ready.

Resolving patch:
diff --git a/master/buildbot/secrets/providers/vault.py b/master/buildbot/secrets/providers/vault.py
--- a/master/buildbot/secrets/providers/vault.py
+++ b/master/buildbot/secrets/providers/vault.py
@@ -16,12 +16,12 @@
 vault based providers
 """
 
-
 from twisted.internet import defer
 
 from buildbot import config
 from buildbot.secrets.providers.base import SecretProviderBase
 from buildbot.util import httpclientservice
+from buildbot.warnings import warn_deprecated
 
 
 class HashiCorpVaultSecretProvider(SecretProviderBase):
@@ -33,6 +33,8 @@ class HashiCorpVaultSecretProvider(SecretProviderBase):
 
     def checkConfig(self, vaultServer=None, vaultToken=None, secretsmount=None,
                     apiVersion=1):
+        warn_deprecated("3.4.0", "Use of HashiCorpVaultSecretProvider is deprecated and will be "
+                        "removed in future releases. Use HashiCorpVaultKvSecretProvider instead")
         if not isinstance(vaultServer, str):
             config.error("vaultServer must be a string while it is {}".format(type(vaultServer)))
         if not isinstance(vaultToken, str):
diff --git a/master/buildbot/secrets/providers/vault_hvac.py b/master/buildbot/secrets/providers/vault_hvac.py
new file mode 100644
--- /dev/null
+++ b/master/buildbot/secrets/providers/vault_hvac.py
@@ -0,0 +1,189 @@
+# This file is part of Buildbot.  Buildbot is free software: you can
+# redistribute it and/or modify it under the terms of the GNU General Public
+# License as published by the Free Software Foundation, version 2.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc., 51
+# Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Copyright Buildbot Team Members
+"""
+HVAC based providers
+"""
+
+from twisted.internet import defer
+from twisted.internet import threads
+
+from buildbot import config
+from buildbot.secrets.providers.base import SecretProviderBase
+
+
+class VaultAuthenticator:
+    """
+    base HVAC authenticator class
+    """
+
+    def authenticate(self, client):
+        pass
+
+
+class VaultAuthenticatorToken(VaultAuthenticator):
+    """
+    HVAC authenticator for static token
+    """
+
+    def __init__(self, token):
+        self.token = token
+
+    def authenticate(self, client):
+        client.token = self.token
+
+
+class VaultAuthenticatorApprole(VaultAuthenticator):
+    """
+    HVAC authenticator for Approle login method
+    """
+
+    def __init__(self, roleId, secretId):
+        self.roleId = roleId
+        self.secretId = secretId
+
+    def authenticate(self, client):
+        client.auth.approle.login(role_id=self.roleId, secret_id=self.secretId)
+
+
+class HashiCorpVaultKvSecretProvider(SecretProviderBase):
+    """
+    Basic provider where each secret is stored in Vault KV secret engine.
+    In case more secret engines are going to be supported, each engine should have it's own class.
+    """
+
+    name = 'SecretInVaultKv'
+
+    def checkConfig(self, vault_server=None, authenticator=None, secrets_mount=None,
+                    api_version=2, path_delimiter='|', path_escape='\\'):
+        try:
+            import hvac
+            [hvac]
+        except ImportError:  # pragma: no cover
+            config.error(f"{self.__class__.__name__} needs the hvac package installed " +
+                         "(pip install hvac)")
+
+        if not isinstance(vault_server, str):
+            config.error("vault_server must be a string while it is {}".format(type(vault_server)))
+        if not isinstance(path_delimiter, str) or len(path_delimiter) > 1:
+            config.error("path_delimiter must be a single character")
+        if not isinstance(path_escape, str) or len(path_escape) > 1:
+            config.error("path_escape must be a single character")
+        if not isinstance(authenticator, VaultAuthenticator):
+            config.error("authenticator must be instance of VaultAuthenticator while it is {}"
+                         .format(type(authenticator)))
+        if api_version not in [1, 2]:
+            config.error("api_version {} is not supported".format(api_version))
+
+    def reconfigService(self, vault_server=None, authenticator=None, secrets_mount=None,
+                        api_version=2, path_delimiter='|', path_escape='\\'):
+        try:
+            import hvac
+        except ImportError:  # pragma: no cover
+            config.error(f"{self.__class__.__name__} needs the hvac package installed " +
+                         "(pip install hvac)")
+
+        if secrets_mount is None:
+            secrets_mount = "secret"
+        self.secrets_mount = secrets_mount
+        self.path_delimiter = path_delimiter
+        self.path_escape = path_escape
+        self.authenticator = authenticator
+        self.api_version = api_version
+        if vault_server.endswith('/'):  # pragma: no cover
+            vault_server = vault_server[:-1]
+        self.client = hvac.Client(vault_server)
+        self.client.secrets.kv.default_kv_version = api_version
+        return self
+
+    def escaped_split(self, s):
+        """
+        parse and split string, respecting escape characters
+        """
+        ret = []
+        current = []
+        itr = iter(s)
+        for ch in itr:
+            if ch == self.path_escape:
+                try:
+                    # skip the next character; it has been escaped and remove
+                    # escape character
+                    current.append(next(itr))
+                except StopIteration:
+                    # escape character on end of the string is safest to ignore, as buildbot for
+                    # each secret identifier tries all secret providers until value is found,
+                    # meaning we may end up parsing identifiers for other secret providers, where
+                    # our escape character may be valid on end of string
+                    pass
+            elif ch == self.path_delimiter:
+                # split! (add current to the list and reset it)
+                ret.append(''.join(current))
+                current = []
+            else:
+                current.append(ch)
+        ret.append(''.join(current))
+        return ret
+
+    def thd_hvac_wrap_read(self, path):
+        if self.api_version == 1:
+            return self.client.secrets.kv.v1.read_secret(path=path, mount_point=self.secrets_mount)
+        else:
+            return self.client.secrets.kv.v2.read_secret_version(path=path,
+                                                                 mount_point=self.secrets_mount)
+
+    def thd_hvac_get(self, path):
+        """
+        query secret from Vault and try to re-authenticate in case Unauthorized
+        exception when active token reaches its TTL
+        """
+
+        # no need to "try" import, it was already handled by reconfigService()
+        import hvac
+
+        try:
+            response = self.thd_hvac_wrap_read(path=path)
+        except (hvac.exceptions.Unauthorized, hvac.exceptions.InvalidRequest):
+            self.authenticator.authenticate(self.client)
+            response = self.thd_hvac_wrap_read(path=path)
+
+        return response
+
+    @defer.inlineCallbacks
+    def get(self, entry):
+        """
+        get the value from vault secret backend
+        """
+
+        parts = self.escaped_split(entry)
+        if len(parts) == 1:
+            raise KeyError("Vault secret specification must contain attribute name separated from "
+                           "path by '{}'".format(self.path_delimiter))
+        if len(parts) > 2:
+            raise KeyError("Multiple separators ('{0}') found in vault path '{1}'. All occurences "
+                           "of '{0}' in path or attribute name must be escaped using '{2}'"
+                           .format(self.path_delimiter, entry, self.path_escape))
+        name = parts[0]
+        key = parts[1]
+
+        response = yield threads.deferToThread(self.thd_hvac_get, path=name)
+
+        # in KVv2 we have extra "data" dictionary, as vault provides metadata as well
+        if self.api_version == 2:
+            response = response['data']
+
+        try:
+            return response['data'][key]
+        except KeyError as e:
+            raise KeyError(
+                "The secret {} does not exist in Vault provider: {}".format(entry, e)) from e
diff --git a/master/setup.py b/master/setup.py
--- a/master/setup.py
+++ b/master/setup.py
@@ -250,7 +250,10 @@ def define_plugin_entries(groups):
         ('buildbot.secrets', [
             ('buildbot.secrets.providers.file', ['SecretInAFile']),
             ('buildbot.secrets.providers.passwordstore', ['SecretInPass']),
-            ('buildbot.secrets.providers.vault', ['HashiCorpVaultSecretProvider'])
+            ('buildbot.secrets.providers.vault', ['HashiCorpVaultSecretProvider']),
+            ('buildbot.secrets.providers.vault_hvac', [
+                'HashiCorpVaultKvSecretProvider', 'VaultAuthenticatorToken',
+                'VaultAuthenticatorApprole'])
         ]),
         ('buildbot.worker', [
             ('buildbot.worker.base', ['Worker']),


Test patch:
diff --git a/master/buildbot/test/__init__.py b/master/buildbot/test/__init__.py
--- a/master/buildbot/test/__init__.py
+++ b/master/buildbot/test/__init__.py
@@ -108,6 +108,9 @@
 if sys.version_info[0] >= 3 and "pg8000" in os.getenv("BUILDBOT_TEST_DB_URL", ""):
     warnings.filterwarnings('ignore', ".*unclosed .*socket", ResourceWarning)
 
+# ignore ResourceWarnings when connecting to a HashiCorp vault via hvac in integration tests
+warnings.filterwarnings('ignore', r".*unclosed .*socket.*raddr=.*, 8200[^\d]", ResourceWarning)
+
 # Python 3.5-3.8 shows this warning
 warnings.filterwarnings('ignore', ".*the imp module is deprecated in favour of importlib*")
 
diff --git a/master/buildbot/test/integration/test_integration_secrets_with_vault.py b/master/buildbot/test/integration/test_integration_secrets_with_vault.py
--- a/master/buildbot/test/integration/test_integration_secrets_with_vault.py
+++ b/master/buildbot/test/integration/test_integration_secrets_with_vault.py
@@ -23,6 +23,8 @@
 from buildbot.secrets.providers.vault import HashiCorpVaultSecretProvider
 from buildbot.steps.shell import ShellCommand
 from buildbot.test.util.integration import RunMasterBase
+from buildbot.test.util.warnings import assertProducesWarning
+from buildbot.warnings import DeprecatedApiWarning
 
 # This integration test creates a master and worker environment,
 # with one builders and a shellcommand step
@@ -63,7 +65,8 @@ def remove_container(self):
 
     @defer.inlineCallbacks
     def do_secret_test(self, secret_specifier, expected_obfuscation, expected_value):
-        yield self.setupConfig(masterConfig(secret_specifier=secret_specifier))
+        with assertProducesWarning(DeprecatedApiWarning):
+            yield self.setupConfig(masterConfig(secret_specifier=secret_specifier))
         build = yield self.doForceBuild(wantSteps=True, wantLogs=True)
         self.assertEqual(build['buildid'], 1)
 
diff --git a/master/buildbot/test/integration/test_integration_secrets_with_vault_hvac.py b/master/buildbot/test/integration/test_integration_secrets_with_vault_hvac.py
new file mode 100644
--- /dev/null
+++ b/master/buildbot/test/integration/test_integration_secrets_with_vault_hvac.py
@@ -0,0 +1,121 @@
+# This file is part of Buildbot.  Buildbot is free software: you can
+# redistribute it and/or modify it under the terms of the GNU General Public
+# License as published by the Free Software Foundation, version 2.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc., 51
+# Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Copyright Buildbot Team Members
+
+import base64
+import subprocess
+import time
+from unittest.case import SkipTest
+
+from twisted.internet import defer
+
+from buildbot.process.properties import Interpolate
+from buildbot.secrets.providers.vault_hvac import HashiCorpVaultKvSecretProvider
+from buildbot.secrets.providers.vault_hvac import VaultAuthenticatorToken
+from buildbot.steps.shell import ShellCommand
+from buildbot.test.util.integration import RunMasterBase
+
+# This integration test creates a master and worker environment,
+# with one builders and a shellcommand step
+
+
+class TestVaultHvac(RunMasterBase):
+    def setUp(self):
+        try:
+            subprocess.check_call(['docker', 'pull', 'vault'])
+
+            subprocess.check_call(['docker', 'run', '-d',
+                                   '-e', 'SKIP_SETCAP=yes',
+                                   '-e', 'VAULT_DEV_ROOT_TOKEN_ID=my_vaulttoken',
+                                   '-e', 'VAULT_TOKEN=my_vaulttoken',
+                                   '--name=vault_for_buildbot',
+                                   '-p', '8200:8200', 'vault'])
+            time.sleep(1)  # the container needs a little time to setup itself
+            self.addCleanup(self.remove_container)
+
+            subprocess.check_call(['docker', 'exec',
+                                   '-e', 'VAULT_ADDR=http://127.0.0.1:8200/',
+                                   'vault_for_buildbot',
+                                   'vault', 'kv', 'put', 'secret/key', 'value=word'])
+
+            subprocess.check_call(['docker', 'exec',
+                                   '-e', 'VAULT_ADDR=http://127.0.0.1:8200/',
+                                   'vault_for_buildbot',
+                                   'vault', 'kv', 'put', 'secret/anykey', 'anyvalue=anyword'])
+
+            subprocess.check_call(['docker', 'exec',
+                                   '-e', 'VAULT_ADDR=http://127.0.0.1:8200/',
+                                   'vault_for_buildbot',
+                                   'vault', 'kv', 'put', 'secret/key1/key2', 'id=val'])
+        except (FileNotFoundError, subprocess.CalledProcessError):
+            raise SkipTest("Vault integration needs docker environment to be setup")
+
+    def remove_container(self):
+        subprocess.call(['docker', 'rm', '-f', 'vault_for_buildbot'])
+
+    @defer.inlineCallbacks
+    def do_secret_test(self, secret_specifier, expected_obfuscation, expected_value):
+        yield self.setupConfig(master_config(secret_specifier=secret_specifier))
+        build = yield self.doForceBuild(wantSteps=True, wantLogs=True)
+        self.assertEqual(build['buildid'], 1)
+
+        patterns = [
+            "echo {}".format(expected_obfuscation),
+            base64.b64encode((expected_value + "\n").encode('utf-8')).decode('utf-8'),
+        ]
+
+        res = yield self.checkBuildStepLogExist(build, patterns)
+        self.assertTrue(res)
+
+    @defer.inlineCallbacks
+    def test_key(self):
+        yield self.do_secret_test('%(secret:key|value)s', '<key|value>', 'word')
+
+    @defer.inlineCallbacks
+    def test_key_any_value(self):
+        yield self.do_secret_test('%(secret:anykey|anyvalue)s', '<anykey|anyvalue>', 'anyword')
+
+    @defer.inlineCallbacks
+    def test_nested_key(self):
+        yield self.do_secret_test('%(secret:key1/key2|id)s', '<key1/key2|id>', 'val')
+
+
+def master_config(secret_specifier):
+    c = {}
+    from buildbot.config import BuilderConfig
+    from buildbot.process.factory import BuildFactory
+    from buildbot.plugins import schedulers
+
+    c['schedulers'] = [
+        schedulers.ForceScheduler(name="force", builderNames=["testy"])
+    ]
+
+    # note that as of August 2021, the vault docker image default to kv
+    # version 2 to be enabled by default
+    c['secretsProviders'] = [
+        HashiCorpVaultKvSecretProvider(authenticator=VaultAuthenticatorToken('my_vaulttoken'),
+                                       vault_server="http://localhost:8200",
+                                       secrets_mount="secret")
+    ]
+
+    f = BuildFactory()
+    f.addStep(ShellCommand(command=Interpolate('echo {} | base64'.format(secret_specifier))))
+
+    c['builders'] = [
+        BuilderConfig(name="testy",
+                      workernames=["local1"],
+                      factory=f)
+    ]
+
+    return c
diff --git a/master/buildbot/test/integration/test_setup_entrypoints.py b/master/buildbot/test/integration/test_setup_entrypoints.py
--- a/master/buildbot/test/integration/test_setup_entrypoints.py
+++ b/master/buildbot/test/integration/test_setup_entrypoints.py
@@ -191,6 +191,7 @@ def test_secrets(self):
             'buildbot.secrets.manager.SecretManager',
             'buildbot.secrets.providers.base.SecretProviderBase',
             'buildbot.secrets.secret.SecretDetails',
+            'buildbot.secrets.providers.vault_hvac.VaultAuthenticator',
         }
         self.verify_plugins_registered('secrets', 'buildbot.secrets', None, known_not_exported)
 
diff --git a/master/buildbot/test/unit/test_secret_in_hvac.py b/master/buildbot/test/unit/test_secret_in_hvac.py
new file mode 100644
--- /dev/null
+++ b/master/buildbot/test/unit/test_secret_in_hvac.py
@@ -0,0 +1,230 @@
+# This file is part of Buildbot.  Buildbot is free software: you can
+# redistribute it and/or modify it under the terms of the GNU General Public
+# License as published by the Free Software Foundation, version 2.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc., 51
+# Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# Copyright Buildbot Team Members
+
+from unittest.mock import patch
+
+from parameterized import parameterized
+
+from twisted.internet import defer
+from twisted.trial import unittest
+
+from buildbot.secrets.providers.vault_hvac import HashiCorpVaultKvSecretProvider
+from buildbot.secrets.providers.vault_hvac import VaultAuthenticatorApprole
+from buildbot.secrets.providers.vault_hvac import VaultAuthenticatorToken
+from buildbot.test.util import interfaces
+from buildbot.test.util.config import ConfigErrorsMixin
+
+try:
+    import hvac
+    assert hvac
+except ImportError:
+    hvac = None
+
+
+class FakeHvacApprole:
+    def login(self, role_id, secret_id):
+        self.role_id = role_id
+        self.secret_id = secret_id
+
+
+class FakeHvacAuth:
+    approle = FakeHvacApprole()
+
+
+class FakeHvacKvV1:
+    token = None
+
+    def read_secret(self, path, mount_point):
+        if self.token is None:
+            raise hvac.exceptions.Unauthorized
+        if path == "wrong/path":
+            raise hvac.exceptions.InvalidPath(message="Fake InvalidPath exception")
+        return {'data': {'key': "value"}}
+
+
+class FakeHvacKvV2:
+    token = None
+
+    def read_secret_version(self, path, mount_point):
+        if self.token is None:
+            raise hvac.exceptions.Unauthorized(message="Fake Unauthorized exception")
+        if path == "wrong/path":
+            raise hvac.exceptions.InvalidPath(message="Fake InvalidPath exception")
+        return {'data': {'data': {'key': "value"}}}
+
+
+class FakeHvacKv:
+    default_kv_version = 2
+    v1 = FakeHvacKvV1()
+    v2 = FakeHvacKvV2()
+
+
+class FakeHvacSecrets:
+    kv = FakeHvacKv()
+
+
+class FakeHvacClient:
+    auth = FakeHvacAuth()
+    secrets = FakeHvacSecrets()
+
+    _token = None
+
+    @property
+    def token(self):
+        return self._token
+
+    @token.setter
+    def token(self, new_token):
+        self._token = new_token
+        self.secrets.kv.v1.token = new_token
+        self.secrets.kv.v2.token = new_token
+
+
+def mock_vault(*args, **kwargs):
+    client = FakeHvacClient()
+    client.token = "mockToken"
+    return client
+
+
+class TestSecretInVaultAuthenticator(interfaces.InterfaceTests):
+
+    def test_authenticate(self):
+        raise NotImplementedError
+
+
+class TestSecretInVaultAuthenticatorToken(unittest.TestCase, TestSecretInVaultAuthenticator):
+
+    def setUp(self):
+        if hvac is None:
+            raise unittest.SkipTest(
+                "Need to install hvac to test VaultAuthenticatorToken")
+
+    def test_authenticate(self):
+        token = "mockToken"
+        authenticator = VaultAuthenticatorToken(token)
+        client = hvac.Client()
+        authenticator.authenticate(client)
+        self.assertEqual(client.token, token)
+
+
+class TestSecretInVaultAuthenticatorApprole(unittest.TestCase, TestSecretInVaultAuthenticator):
+
+    def test_authenticate(self):
+        authenticator = VaultAuthenticatorApprole("testRole", "testSecret")
+        client = FakeHvacClient()
+        authenticator.authenticate(client)
+        self.assertEqual(client.auth.approle.secret_id, "testSecret")
+
+
+class TestSecretInHashiCorpVaultKvSecretProvider(ConfigErrorsMixin, unittest.TestCase):
+
+    def setUp(self):
+        if hvac is None:
+            raise unittest.SkipTest(
+                "Need to install hvac to test HashiCorpVaultKvSecretProvider")
+        param = dict(vault_server="", authenticator=VaultAuthenticatorToken("mockToken"),
+                     path_delimiter='|', path_escape='\\', api_version=2)
+        self.provider = HashiCorpVaultKvSecretProvider(**param)
+        self.provider.reconfigService(**param)
+        self.provider.client = FakeHvacClient()
+        self.provider.client.secrets.kv.default_kv_version = param['api_version']
+        self.provider.client.token = "mockToken"
+
+    @parameterized.expand([
+        ('vault_server_not_string', {'vault_server': {}}, 'vault_server must be a string'),
+        ('path_delimiter_not_char', {'vault_server': 'abc', 'path_delimiter': {}},
+         'path_delimiter must be a single character'),
+        ('path_delimiter_too_long', {'vault_server': 'abc', 'path_delimiter': 'ab'},
+         'path_delimiter must be a single character'),
+        ('path_escape_not_char', {'vault_server': 'abc', 'path_escape': {}},
+         'path_escape must be a single character'),
+        ('path_escape_too_long', {'vault_server': 'abc', 'path_escape': 'ab'},
+         'path_escape must be a single character'),
+        ('api_version_unsupported', {'vault_server': 'abc', 'api_version': 3},
+         'api_version 3 is not supported'),
+    ])
+    def test_check_config(self, name, params, error):
+        with self.assertRaisesConfigError(error):
+            HashiCorpVaultKvSecretProvider(authenticator=VaultAuthenticatorToken("mockToken"),
+                                           **params)
+
+    def test_check_config_authenticator(self):
+        with self.assertRaisesConfigError('authenticator must be instance of VaultAuthenticator'):
+            HashiCorpVaultKvSecretProvider(vault_server='abc')
+
+    def test_escaped_split(self):
+        parts = self.provider.escaped_split("a/b\\|c/d|e/f\\|g/h")
+        self.assertEqual(parts, ["a/b|c/d", "e/f|g/h"])
+
+    def test_escaped_split_ends_with_escape(self):
+        parts = self.provider.escaped_split("a|b\\")
+        self.assertEqual(parts, ["a", "b"])
+
+    def test_thd_hvac_wrap_read_v1(self):
+        self.provider.api_version = 1
+        self.provider.client.token = "mockToken"
+        value = self.provider.thd_hvac_wrap_read("some/path")
+        self.assertEqual(value['data']['key'], "value")
+
+    def test_thd_hvac_wrap_read_v2(self):
+        self.provider.client.token = "mockToken"
+        value = self.provider.thd_hvac_wrap_read("some/path")
+        self.assertEqual(value['data']['data']['key'], "value")
+
+    # for some reason, errors regarding generator function were thrown
+    @patch("hvac.Client", side_effect=mock_vault)
+    def test_thd_hvac_wrap_read_unauthorized(self, mock_vault):
+        self.provider.client.token = None
+        yield self.assertFailure(self.provider.thd_hvac_wrap_read("some/path"),
+                                 hvac.exceptions.Unauthorized)
+
+    def test_thd_hvac_get_reauthorize(self):
+        """
+        When token is None, provider gets unauthorized exception and is forced to re-authenticate
+        """
+        self.provider.client.token = None
+        value = self.provider.thd_hvac_get("some/path")
+        self.assertEqual(value['data']['data']['key'], "value")
+
+    @defer.inlineCallbacks
+    def test_get_v1(self):
+        self.provider.api_version = 1
+        self.provider.client.token = "mockToken"
+        value = yield self.provider.get("some/path|key")
+        self.assertEqual(value, "value")
+
+    @defer.inlineCallbacks
+    def test_get_v2(self):
+        self.provider.client.token = "mockToken"
+        value = yield self.provider.get("some/path|key")
+        self.assertEqual(value, "value")
+
+    @defer.inlineCallbacks
+    def test_get_fail_no_key(self):
+        self.provider.client.token = "mockToken"
+        with self.assertRaises(KeyError):
+            yield self.provider.get("some/path")
+
+    @defer.inlineCallbacks
+    def test_get_fail_wrong_key(self):
+        self.provider.client.token = "mockToken"
+        with self.assertRaises(KeyError):
+            yield self.provider.get("some/path|wrong_key")
+
+    @defer.inlineCallbacks
+    def test_get_fail_multiple_separators(self):
+        self.provider.client.token = "mockToken"
+        with self.assertRaises(KeyError):
+            yield self.provider.get("some/path|unescaped|key")
diff --git a/master/buildbot/test/unit/test_secret_in_vault.py b/master/buildbot/test/unit/test_secret_in_vault.py
--- a/master/buildbot/test/unit/test_secret_in_vault.py
+++ b/master/buildbot/test/unit/test_secret_in_vault.py
@@ -13,6 +13,8 @@
 #
 # Copyright Buildbot Team Members
 
+import warnings
+
 from twisted.internet import defer
 from twisted.trial import unittest
 
@@ -28,6 +30,7 @@ class TestSecretInVaultHttpFakeBase(ConfigErrorsMixin, TestReactorMixin,
 
     @defer.inlineCallbacks
     def setUp(self, version):
+        warnings.simplefilter('ignore')
         self.setUpTestReactor()
         self.srvcVault = HashiCorpVaultSecretProvider(vaultServer="http://vaultServer",
                                                       vaultToken="someToken",