Repository: kroxylicious/kroxylicious
Pull request #935 (2024-01-31 03:16:41)

Enable user to configure alternative source of keys for vault KMS client
### Type of change

- Enhancement / new feature

### Description

Enable user to configure alternative source of keys for vault like:

```yaml
tls:
    key:
        storeFile: /opt/cert/server.p12
        storePassword:
            passwordFile: /opt/cert/store.password
        keyPassword:
            passwordFile: /opt/cert/key.password
        storeType: PKCS12
```

Why:
Vault can require TLS client authentication where clients must present certificates proving who they are. These could be private certs so we should support configurable sources of keys.

Note that we do not support PEM keystores or PKCS8 KeyPairs yet. In kroxylicious core we depend on netty to handle parsing PEMS. This PR at least lets us support a source of keys. We can add support for these alternatives later, for now Kroxylicious throws with a message about how they are not supported yet.

### Checklist

_Please go through this checklist and make sure all applicable tasks have been done_

- [ ] Write tests
- [ ] Make sure all tests pass
- [ ] Review performance test results. Ensure that any degradations to performance numbers are understood and justified.
- [ ] Make sure all Sonar-Lint warnings are addressed or are justifiably ignored.
- [ ] Update documentation
- [ ] Reference relevant issue(s) and close them after merging
- [ ] For user facing changes, update CHANGELOG.md (remember to include changes affecting the API of the test artefacts too).


sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[92.1% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935)


sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[92.1% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935)


sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[92.1% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935)


strimzi-ci: :arrow_forward: **Build started** - check Jenkins for more info. :arrow_forward:
strimzi-ci: ### :heavy_check_mark: Test Summary :heavy_check_mark:
**BUILD NUMBER:** [1](https://master-jenkins-csb-amq-streams.apps.ocp-c1.prod.psi.redhat.com/job/kroxylicious/job/kroxylicious-performance-tests-pr/1/)
**KAFKA VERSION:** 3.6.0
**NUMBER OF BROKERS:** 1
**NUMBER OF MESSAGES:** 30000000
**MESSAGE SIZE:** 1024 bytes
**PRODUCER PROPERTIES:** acks=all

:small_blue_diamond: Producer Results
| Latency | Current Value | Previous Value |
| ------------- | ------------- | ------------- |
| **AVG Latency** | 204.68 ms | 208.99 ms |
| **95th Latency** | 279 ms | 264 ms |
| **99th Latency** | 359 ms | 401 ms |

:small_blue_diamond: Consumer Results
| Latency | Current Value | Previous Value |
| ------------- | ------------- | ------------- |
| **data.consumed.in.MB** | 29296.973  | 29296.973  |
| **MB.sec** | 276.7024  | 276.82  |
| **data.consumed.in.nMsg** | 30000100  | 30000100  |
| **nMsg.sec** | 283343.25  | 283463.72  |
| **rebalance.time.ms** | 1019  | 964  |
| **fetch.time.ms** | 104860  | 104870  |
| **fetch.MB.sec** | 279.3913  | 279.3647  |
| **fetch.nMsg.sec** | 286096.7  | 286069.4  |

sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=935&resolved=false&inNewCodePeriod=true)  
[92.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=935&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=935)



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ba1c1054..32f772e3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@ Please enumerate **all user-facing** changes using format `<githib issue/pr numb
 
 ## 0.5.0
 
+* [#935](https://github.com/kroxylicious/kroxylicious/pull/935): Enable user to configure alternative source of keys for vault KMS client
 * [#787](https://github.com/kroxylicious/kroxylicious/issues/787): Initial documentation for the envelope-encryption feature.
 * [#940](https://github.com/kroxylicious/kroxylicious/issues/940): Support vault namespaces and support secrets transit engine at locations other than /transit
 * [#951](https://github.com/kroxylicious/kroxylicious/pull/951): Include the kroxylicious maintained filters in the dist by default
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTls.java b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTls.java
index 86bd7d5a..2a419ec5 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTls.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTls.java
@@ -10,7 +10,10 @@ import java.io.FileInputStream;
 import java.security.KeyStore;
 import java.security.SecureRandom;
 import java.security.cert.X509Certificate;
+import java.util.Optional;
 
+import javax.net.ssl.KeyManager;
+import javax.net.ssl.KeyManagerFactory;
 import javax.net.ssl.SSLContext;
 import javax.net.ssl.TrustManager;
 import javax.net.ssl.TrustManagerFactory;
@@ -20,12 +23,17 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import io.kroxylicious.proxy.config.tls.InsecureTls;
+import io.kroxylicious.proxy.config.tls.KeyPair;
+import io.kroxylicious.proxy.config.tls.KeyProvider;
+import io.kroxylicious.proxy.config.tls.KeyProviderVisitor;
+import io.kroxylicious.proxy.config.tls.PasswordProvider;
 import io.kroxylicious.proxy.config.tls.Tls;
 import io.kroxylicious.proxy.config.tls.TrustProvider;
 import io.kroxylicious.proxy.config.tls.TrustProviderVisitor;
 import io.kroxylicious.proxy.config.tls.TrustStore;
 
 import edu.umd.cs.findbugs.annotations.NonNull;
+import edu.umd.cs.findbugs.annotations.Nullable;
 
 /**
  * Builds a JDK SSLContext used for vault communicateion.
@@ -69,8 +77,16 @@ public record JdkTls(Tls tls) {
 
     public SSLContext sslContext() {
         try {
-            if (tls != null && tls.trust() != null) {
-                return customSSLContext(tls.trust());
+            if (tls != null) {
+                TrustManager[] trustManagers = null;
+                KeyManager[] keyManagers = null;
+                if (tls.trust() != null) {
+                    trustManagers = getTrustManagers(tls.trust());
+                }
+                if (tls.key() != null) {
+                    keyManagers = getKeyManagers(tls.key());
+                }
+                return getSslContext(trustManagers, keyManagers);
             }
             else {
                 return SSLContext.getDefault();
@@ -81,17 +97,51 @@ public record JdkTls(Tls tls) {
         }
     }
 
-    @NonNull
-    private SSLContext customSSLContext(TrustProvider trust) {
-        TrustManager[] trustManagers = getTrustManagers(trust);
-        return getSslContext(trustManagers);
+    /* exposed for testing */
+    static KeyManager[] getKeyManagers(KeyProvider key) {
+        return key.accept(new KeyProviderVisitor<>() {
+            @Override
+            public KeyManager[] visit(KeyPair keyPair) {
+                throw new SslConfigurationException("KeyPair is not supported by vault KMS yet");
+            }
+
+            @Override
+            public KeyManager[] visit(io.kroxylicious.proxy.config.tls.KeyStore keyStore) {
+                try {
+                    if (keyStore.isPemType()) {
+                        throw new SslConfigurationException("PEM is not supported by vault KMS yet");
+                    }
+                    KeyStore store = KeyStore.getInstance(keyStore.getType());
+                    char[] storePassword = passwordOrNull(keyStore.storePasswordProvider());
+                    try (FileInputStream fileInputStream = new FileInputStream(keyStore.storeFile())) {
+                        store.load(fileInputStream, storePassword);
+                    }
+                    KeyManagerFactory instance = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
+                    char[] keyPassword = passwordOrNull(keyStore.keyPasswordProvider());
+                    keyPassword = keyPassword == null ? storePassword : keyPassword;
+                    instance.init(store, keyPassword);
+                    return instance.getKeyManagers();
+                }
+                catch (Exception e) {
+                    throw new SslConfigurationException(e);
+                }
+            }
+
+            @Nullable
+            private static char[] passwordOrNull(PasswordProvider value) {
+                return Optional.ofNullable(value).map(PasswordProvider::getProvidedPassword).map(String::toCharArray).orElse(null);
+            }
+        });
     }
 
     @NonNull
-    private static SSLContext getSslContext(TrustManager[] trustManagers) {
+    private static SSLContext getSslContext(TrustManager[] trustManagers, KeyManager[] keyManagers) {
         try {
+            if (trustManagers == null && keyManagers == null) {
+                return SSLContext.getDefault();
+            }
             SSLContext context = SSLContext.getInstance("TLS");
-            context.init(null, trustManagers, new SecureRandom());
+            context.init(keyManagers, trustManagers, new SecureRandom());
             return context;
         }
         catch (Exception e) {


Test patch:
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/CertificateGenerator.java b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/CertificateGenerator.java
index c0d9da90..cc229d01 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/CertificateGenerator.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/CertificateGenerator.java
@@ -16,9 +16,9 @@ import java.nio.charset.StandardCharsets;
 import java.nio.file.Path;
 import java.security.KeyPair;
 import java.security.KeyPairGenerator;
-import java.security.KeyStore;
 import java.security.NoSuchAlgorithmException;
 import java.security.SecureRandom;
+import java.security.cert.Certificate;
 import java.security.cert.X509Certificate;
 import java.time.Duration;
 import java.time.Instant;
@@ -100,7 +100,12 @@ public class CertificateGenerator {
             var validFrom = Date.from(now);
             var validTo = Date.from(now.plus(Duration.ofDays(9999)));
             var certBuilder = new X509v3CertificateBuilder(
-                    new X500Name("CN=issuer"),
+                    // Currently it is important that the issuer name equals the subject name. We use these certs on vault to authenticate
+                    // the client. We tell vault that the client certificate is a trusted CA. With client authentication enabled,
+                    // vault sends a list of trusted issuers to the client as part of the handshake (the subject name from the cert).
+                    // When the client is told about the trusted issuers it tries to locate a certificate with that issuer to present
+                    // to the server. It's at that point that we need the issuer name to match what vault sent.
+                    new X500Name("CN=localhost"),
                     BigInteger.ONE,
                     validFrom,
                     validTo,
@@ -130,7 +135,7 @@ public class CertificateGenerator {
     private static Path buildTrustStore(X509Certificate cert, String password, String suffix, String type) {
         try {
             File certFile = createTempFile("trust", suffix);
-            KeyStore pkcs12Store = KeyStore.getInstance(type);
+            java.security.KeyStore pkcs12Store = java.security.KeyStore.getInstance(type);
             pkcs12Store.load(null, null);
             pkcs12Store.setCertificateEntry(ALIAS, cert);
             char[] pass = password != null ? password.toCharArray() : null;
@@ -146,19 +151,26 @@ public class CertificateGenerator {
 
     }
 
+    public record KeyStore(@NonNull Path path, @NonNull String type, @Nullable String storePassword, @Nullable Path storePasswordFile, @Nullable String keyPassword,
+                           @Nullable Path keyPasswordFile) {
+
+    }
+
     public record Keys(KeyPair serverKey,
-                       Path serverPrivateKeyPem,
-                       Path serverCertPem,
+                       Path privateKeyPem,
+                       Path selfSignedCertificatePem,
                        TrustStore pkcs12ClientTruststore,
                        TrustStore jksClientTruststore,
-                       TrustStore pkcs12NoPasswordClientTruststore) {}
+                       TrustStore pkcs12NoPasswordClientTruststore,
+                       KeyStore jksServerKeystore) {}
 
     public static Keys generate() {
+        String password = "changeit";
         KeyPair pair = generateRsaKeyPair();
         Path privateKeyPem = writeRsaPrivateKeyPem(pair);
         X509Certificate x509Certificate = generateSelfSignedX509Certificate(pair);
+        KeyStore keyStore = createJksKeystore(pair, x509Certificate, password, "keypass");
         Path serverCert = generateCertPem(x509Certificate);
-        String password = "changeit";
         Path pkcs12Trust = buildPkcs12TrustStore(x509Certificate, password);
         Path noPasswordPkcs12Trust = buildPkcs12TrustStore(x509Certificate, null);
         Path jksTrust = buildJksTrustStore(x509Certificate, password);
@@ -166,7 +178,26 @@ public class CertificateGenerator {
         TrustStore pkcs12ClientTruststore = new TrustStore(pkcs12Trust, PKCS_12, password, passwordFile);
         TrustStore pkcs12NoPasswordTruststore = new TrustStore(noPasswordPkcs12Trust, PKCS_12, null, null);
         TrustStore jksClientTruststore = new TrustStore(jksTrust, JKS, password, passwordFile);
-        return new Keys(pair, privateKeyPem, serverCert, pkcs12ClientTruststore, jksClientTruststore, pkcs12NoPasswordTruststore);
+        return new Keys(pair, privateKeyPem, serverCert, pkcs12ClientTruststore, jksClientTruststore, pkcs12NoPasswordTruststore, keyStore);
+    }
+
+    public static KeyStore createJksKeystore(KeyPair privateKeyPem, X509Certificate x509Certificate, String storePassword, String keyPassword) {
+        try {
+            File tempFile = createTempFile("keystore", "jks");
+            java.security.KeyStore store = java.security.KeyStore.getInstance("JKS");
+            store.load(null);
+            store.setKeyEntry("alias", privateKeyPem.getPrivate(), keyPassword.toCharArray(), new Certificate[]{ x509Certificate });
+            try (FileOutputStream stream = new FileOutputStream(tempFile)) {
+                store.store(stream, storePassword.toCharArray());
+            }
+            Path path = tempFile.toPath();
+            Path storePasswordFile = writeToTempFile(storePassword);
+            Path keyPasswordFile = writeToTempFile(keyPassword);
+            return new KeyStore(path, "JKS", storePassword, storePasswordFile, keyPassword, keyPasswordFile);
+        }
+        catch (Exception e) {
+            throw new RuntimeException(e);
+        }
     }
 
     private static Path writeToTempFile(String password) {
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java
index 855c22fd..3cc666d8 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java
@@ -12,6 +12,7 @@ import java.io.UncheckedIOException;
 import java.net.URI;
 import java.util.Arrays;
 
+import org.testcontainers.containers.wait.strategy.HttpWaitStrategy;
 import org.testcontainers.containers.wait.strategy.Wait;
 import org.testcontainers.utility.MountableFile;
 import org.testcontainers.vault.VaultContainer;
@@ -22,6 +23,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
 import static org.assertj.core.api.Assertions.assertThat;
 
 public class TestVault implements Closeable {
+    public static final int PLAIN_PORT = 8200;
     private static final String VAULT_TOKEN = "token";
 
     private static final String HASHICORP_VAULT = "hashicorp/vault:1.15";
@@ -31,33 +33,52 @@ public class TestVault implements Closeable {
 
     private static final int TLS_PORT = 8202;
 
-    private TestVault(CertificateGenerator.Keys keys) {
+    private TestVault(CertificateGenerator.Keys serverKeys, CertificateGenerator.Keys clientKeys) {
+        if (clientKeys != null && serverKeys == null) {
+            throw new IllegalArgumentException("server TLS key/cert must be supplied if we want to configure client TLS auth");
+        }
         VaultContainer<?> vault = new VaultContainer<>(HASHICORP_VAULT)
                 .withVaultToken(VAULT_TOKEN)
                 .withEnv("VAULT_FORMAT", "json")
-                .withInitCommand("secrets enable transit ");
-        if (keys != null) {
-            // in vault server -dev mode the 8200 port cannot be reconfigured to TLS. So we configure a second listener on a
-            // different port. We expose only the tls port to avoid any chance of a misconfigured client pointing at the plain listener.
-            vault = vault.withExposedPorts(TLS_PORT)
-                    .withCopyFileToContainer(MountableFile.forClasspathResource("Vault.hcl"), "/vault/config/Vault.hcl")
-                    .withCopyFileToContainer(MountableFile.forHostPath(keys.serverCertPem()), "/vault/config/cert.pem")
-                    .withCopyFileToContainer(MountableFile.forHostPath(keys.serverPrivateKeyPem()), "/vault/config/key.pem");
-            // we need testcontainers to allow insecure connections to the health as the only exposed port is now using TLS
-            vault.setWaitStrategy(Wait.forHttps("/v1/sys/health").allowInsecure().forStatusCode(200));
+                .withInitCommand("secrets enable transit");
+        if (serverKeys != null) {
+            withTls(vault, serverKeys, clientKeys);
         }
         vault.start();
         this.vault = vault;
-        endpoint = URI.create(keys == null ? vault.getHttpHostAddress() : String.format("https://%s:%s", vault.getHost(), vault.getMappedPort(TLS_PORT)))
+        endpoint = URI.create(serverKeys == null ? vault.getHttpHostAddress() : String.format("https://%s:%s", vault.getHost(), vault.getMappedPort(TLS_PORT)))
                 .resolve("v1/transit");
     }
 
+    private static void withTls(VaultContainer<?> vault, CertificateGenerator.Keys serverKeys, CertificateGenerator.Keys clientKeys) {
+        boolean isMutualTls = clientKeys != null;
+        String configFile = isMutualTls ? "VaultMutualTls.hcl" : "Vault.hcl";
+        int healthPort = isMutualTls ? PLAIN_PORT : TLS_PORT;
+        vault.withExposedPorts(TLS_PORT, healthPort);
+        vault.withCopyFileToContainer(MountableFile.forClasspathResource(configFile), "/vault/config/Vault.hcl")
+                .withCopyFileToContainer(MountableFile.forHostPath(serverKeys.selfSignedCertificatePem()), "/vault/config/cert.pem")
+                .withCopyFileToContainer(MountableFile.forHostPath(serverKeys.privateKeyPem()), "/vault/config/key.pem");
+        if (isMutualTls) {
+            vault.withCopyFileToContainer(MountableFile.forHostPath(clientKeys.selfSignedCertificatePem()), "/vault/config/client-cert.pem");
+        }
+        HttpWaitStrategy wait = Wait.forHttp("/v1/sys/health").forPort(healthPort).forStatusCode(200);
+        if (!isMutualTls) {
+            // when using server TLS only, we only expose the tls port.
+            wait.usingTls().allowInsecure().forStatusCode(200);
+        }
+        vault.setWaitStrategy(wait);
+    }
+
     public static TestVault startWithTls(CertificateGenerator.Keys keys) {
-        return new TestVault(keys);
+        return new TestVault(keys, null);
+    }
+
+    public static TestVault startWithClientAuthTls(CertificateGenerator.Keys serverKeys, CertificateGenerator.Keys clientKeys) {
+        return new TestVault(serverKeys, clientKeys);
     }
 
     public static TestVault start() {
-        return new TestVault(null);
+        return new TestVault(null, null);
     }
 
     public URI getEndpoint() {
@@ -86,6 +107,11 @@ public class TestVault implements Closeable {
         }
     }
 
+    VaultResponse.ReadKeyData createKek(String keyId) {
+        return runVaultCommand(new TypeReference<>() {
+        }, "vault", "write", "-f", "transit/keys/%s".formatted(keyId));
+    }
+
     public String rootToken() {
         return VAULT_TOKEN;
     }
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTlsIT.java b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTlsIT.java
index c27ed070..16dd5656 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTlsIT.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTlsIT.java
@@ -20,13 +20,12 @@ import org.junit.jupiter.params.provider.Arguments;
 import org.junit.jupiter.params.provider.MethodSource;
 import org.testcontainers.DockerClientFactory;
 
-import com.fasterxml.jackson.core.type.TypeReference;
-
 import io.kroxylicious.kms.provider.hashicorp.vault.VaultResponse.ReadKeyData;
 import io.kroxylicious.kms.provider.hashicorp.vault.config.Config;
 import io.kroxylicious.proxy.config.tls.FilePassword;
 import io.kroxylicious.proxy.config.tls.InlinePassword;
 import io.kroxylicious.proxy.config.tls.InsecureTls;
+import io.kroxylicious.proxy.config.tls.KeyStore;
 import io.kroxylicious.proxy.config.tls.Tls;
 import io.kroxylicious.proxy.config.tls.TrustStore;
 
@@ -89,6 +88,41 @@ class VaultKmsTlsIT {
                 .withMessageContaining("unable to find valid certification path to requested target");
     }
 
+    @Test
+    void testMutualTlsConnectionRejectedIfClientOffersNoCert() {
+        CertificateGenerator.Keys clientKeys = CertificateGenerator.generate();
+        TestVault testVault = TestVault.startWithClientAuthTls(keys, clientKeys);
+        var keyName = "mykey";
+        testVault.createKek(keyName);
+        VaultKms service = getTlsVaultKms(tlsForTrustStoreFilePassword(keys.pkcs12ClientTruststore()), testVault.getEndpoint());
+        var resolved = service.resolveAlias(keyName);
+        assertThat(resolved)
+                .failsWithin(Duration.ofSeconds(5))
+                .withThrowableThat().havingCause()
+                .isInstanceOf(SSLHandshakeException.class)
+                .withMessageContaining("Received fatal alert: certificate_required");
+        testVault.close();
+    }
+
+    @Test
+    void testMutualTlsConnectionAcceptedIfClientOffersTrustedCert() {
+        CertificateGenerator.Keys clientKeys = CertificateGenerator.generate();
+        TestVault testVault = TestVault.startWithClientAuthTls(keys, clientKeys);
+        var keyName = "mykey";
+        testVault.createKek(keyName);
+        CertificateGenerator.TrustStore trustStore = keys.pkcs12ClientTruststore();
+        CertificateGenerator.KeyStore keyStore = clientKeys.jksServerKeystore();
+        VaultKms service = getTlsVaultKms(
+                new Tls(new KeyStore(keyStore.path().toString(), new InlinePassword(keyStore.storePassword()), new InlinePassword(keyStore.keyPassword()), "JKS"),
+                        new TrustStore(trustStore.path().toString(), new FilePassword(trustStore.passwordFile().toString()), trustStore.type())),
+                testVault.getEndpoint());
+        var resolved = service.resolveAlias(keyName);
+        assertThat(resolved)
+                .succeedsWithin(Duration.ofSeconds(5))
+                .isEqualTo(keyName);
+        testVault.close();
+    }
+
     @ParameterizedTest(name = "{0}")
     @MethodSource("tlsConfigurations")
     void testArbitraryKmsOperationSucceedsWithTls(String kms, KmsCreator creator) {
@@ -102,8 +136,7 @@ class VaultKmsTlsIT {
     }
 
     private ReadKeyData createKek(String keyId) {
-        return vaultContainer.runVaultCommand(new TypeReference<>() {
-        }, "vault", "write", "-f", "transit/keys/%s".formatted(keyId));
+        return vaultContainer.createKek(keyId);
     }
 
     @NonNull
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTlsTest.java b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTlsTest.java
index 0898737a..92fa4bcc 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTlsTest.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/JdkTlsTest.java
@@ -13,6 +13,7 @@ import java.security.cert.CertificateException;
 import java.security.cert.X509Certificate;
 import java.util.UUID;
 
+import javax.net.ssl.KeyManager;
 import javax.net.ssl.SSLContext;
 import javax.net.ssl.TrustManager;
 import javax.net.ssl.X509TrustManager;
@@ -22,15 +23,20 @@ import org.junit.jupiter.api.Test;
 import io.kroxylicious.kms.provider.hashicorp.vault.CertificateGenerator;
 import io.kroxylicious.proxy.config.tls.InlinePassword;
 import io.kroxylicious.proxy.config.tls.InsecureTls;
+import io.kroxylicious.proxy.config.tls.KeyPair;
+import io.kroxylicious.proxy.config.tls.KeyStore;
 import io.kroxylicious.proxy.config.tls.Tls;
 import io.kroxylicious.proxy.config.tls.TrustStore;
 
+import static io.kroxylicious.kms.provider.hashicorp.vault.CertificateGenerator.createJksKeystore;
+import static io.kroxylicious.kms.provider.hashicorp.vault.CertificateGenerator.generateRsaKeyPair;
+import static io.kroxylicious.kms.provider.hashicorp.vault.CertificateGenerator.generateSelfSignedX509Certificate;
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.assertj.core.api.Assertions.assertThatCode;
 import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 class JdkTlsTest {
-    public static final X509Certificate SELF_SIGNED_X_509_CERTIFICATE = CertificateGenerator.generateSelfSignedX509Certificate(CertificateGenerator.generateRsaKeyPair());
+    public static final X509Certificate SELF_SIGNED_X_509_CERTIFICATE = generateSelfSignedX509Certificate(generateRsaKeyPair());
 
     @Test
     void testInsecureTlsEnabled() {
@@ -118,11 +124,49 @@ class JdkTlsTest {
     void testPkcs12() {
         CertificateGenerator.Keys keys = CertificateGenerator.generate();
         CertificateGenerator.TrustStore trustStore = keys.pkcs12ClientTruststore();
-        TrustStore store = new TrustStore(trustStore.path().toString(), new InlinePassword(trustStore.password()), "PKCS12");
+        TrustStore store = new TrustStore(trustStore.path().toString(), new InlinePassword(trustStore.password()), trustStore.type());
         TrustManager[] trustManagers = JdkTls.getTrustManagers(store);
         assertThat(trustManagers).isNotEmpty();
     }
 
+    @Test
+    void testKeyStore() {
+        CertificateGenerator.Keys keys = CertificateGenerator.generate();
+        CertificateGenerator.KeyStore keyStore = keys.jksServerKeystore();
+        KeyStore store = new KeyStore(keyStore.path().toString(), new InlinePassword(keyStore.storePassword()), new InlinePassword(keyStore.keyPassword()),
+                keyStore.type());
+        KeyManager[] trustManagers = JdkTls.getKeyManagers(store);
+        assertThat(trustManagers).isNotEmpty();
+    }
+
+    @Test
+    void testKeyStoreKeyPasswordDefaultsToStorePassword() {
+        java.security.KeyPair pair = generateRsaKeyPair();
+        X509Certificate x509Certificate = generateSelfSignedX509Certificate(pair);
+        String password = "password";
+        CertificateGenerator.KeyStore keyStore = createJksKeystore(pair, x509Certificate, password, password);
+        KeyStore store = new KeyStore(keyStore.path().toString(), new InlinePassword(keyStore.storePassword()), null,
+                keyStore.type());
+        KeyManager[] trustManagers = JdkTls.getKeyManagers(store);
+        assertThat(trustManagers).isNotEmpty();
+    }
+
+    @Test
+    void testKeyPairNotSupported() {
+        KeyPair store = new KeyPair("/tmp/keypair", "/tmp/cert", null);
+        assertThatThrownBy(() -> {
+            JdkTls.getKeyManagers(store);
+        }).isInstanceOf(SslConfigurationException.class).hasMessageContaining("KeyPair is not supported by vault KMS yet");
+    }
+
+    @Test
+    void testKeyStorePemNotSupported() {
+        KeyStore store = new KeyStore("/tmp/pem", null, null, "PEM");
+        assertThatThrownBy(() -> {
+            JdkTls.getKeyManagers(store);
+        }).isInstanceOf(SslConfigurationException.class).hasMessageContaining("PEM is not supported by vault KMS yet");
+    }
+
     @Test
     void testPkcs12WrongPassword() {
         CertificateGenerator.Keys keys = CertificateGenerator.generate();
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/resources/VaultMutualTls.hcl b/kroxylicious-kms-provider-hashicorp-vault/src/test/resources/VaultMutualTls.hcl
new file mode 100644
index 00000000..016cc591
--- /dev/null
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/resources/VaultMutualTls.hcl
@@ -0,0 +1,7 @@
+listener "tcp" {
+  address       = "0.0.0.0:8202"
+  tls_cert_file      = "/vault/config/cert.pem"
+  tls_key_file       = "/vault/config/key.pem"
+  tls_require_and_verify_client_cert = "true"
+  tls_client_ca_file = "/vault/config/client-cert.pem"
+}