Repository: kroxylicious/kroxylicious
Pull request #952 (2024-02-06 17:05:36)

Support HashiCorp Vault namespaces (Enterprise feature) and allow non-standard transit engine names
### Type of change

_Select the type of your PR_

- Bugfix

### Description

Currently the EnvelopeEncryption filter fails to operate against a HCE.  HCEs make use of the Enterprise namespace feature,
which means the URLs generated by the Kms implementation are wrong.

I took the approach of letting the vaultUrl provide full path to the transit engine.  That gives support for both namespaces, and gives scope to work with non-standard engine path names.

An alternative would have been to let the implementation accept a namespaces configuration parameter, with would be passed to the [X-Vault-Namespace](https://developer.hashicorp.com/vault/docs/enterprise/namespaces#vault-api-and-namespaces) on the request.  A separate parameter would be required to allow for the customisation the name of the transit engine.



### Additional Context

_Why are you making this pull request?_

### Checklist

_Please go through this checklist and make sure all applicable tasks have been done_

- [x] Write tests
- [x] Make sure all tests pass
- [ ] Review performance test results. Ensure that any degradations to performance numbers are understood and justified.
- [ ] Make sure all Sonar-Lint warnings are addressed or are justifiably ignored.
- [ ] Update documentation
- [x] Reference relevant issue(s) and close them after merging
- [x] For user facing changes, update CHANGELOG.md (remember to include changes affecting the API of the test artefacts too).


k-wall: With this fix, the filter works with HCE too.  Here's I'm using a kroxylicious namespace created beneath the parent (admin) in my HCE instance.

```
    - type: EnvelopeEncryption
      config:
        kms: VaultKmsService
        kmsConfig:
          vaultUrl: https://sample-cluster-public-vault-a157adce.14dc13da.z1.hashicorp.cloud:8200/v1/admin/kroxylicious/transit/
          vaultToken: hvs.CAESIA9yak-E_ixxxxxxxxxxxxxxxxxxxxxxxxxxxx
        selector: TemplateKekSelector
        selectorConfig:
          template: "topic.${topicName}"
```
sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952) **Quality Gate passed**  
The SonarCloud Quality Gate passed, but some issues were introduced.

[1 New issue](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[86.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952)


robobario: Is it a bit confusing that `vaultUrl` can be `https://vault:8080` or `https://vault:8080/v1/your/path/to/transit`? It means `vaultUrl` in config can be either a base url, or a base transit url. Should we split it into `vaultBaseUrl` and `transitEnginePath` (default "/v1/transit")?
k-wall: @robobario you make a good point.  The vaultUrl field name no longer describes the purpose.  I think I prefer a single field.

How about I rename the field vaultTransitEngineUrl and make the specification of the path mandatory?
robobario: > How about I rename the field vaultTransitEngineUrl and make the specification of the path mandatory?

Yep that works for me, thanks
SamBarker: > How about I rename the field vaultTransitEngineUrl and make the specification of the path mandatory?

Yes, assuming we don't need to support any other engines in the future. 
k-wall: > > How about I rename the field vaultTransitEngineUrl and make the specification of the path mandatory?
> 
> Yes, assuming we don't need to support any other engines in the future.

I can't foreesee myself.  Maybe the Kroxylicious core might have a HashiCorp Secrets integration, but I don't think that would be a concern for EE or KMS services.  

k-wall: @strimzi-ci run system tests
strimzi-ci: :arrow_forward: **Build started** - check Jenkins for more info. :arrow_forward:
sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[80.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952)


strimzi-ci: ### :x: Test Summary :x:
**BUILD NUMBER:** [11](https://master-jenkins-csb-amq-streams.apps.ocp-c1.prod.psi.redhat.com/job/kroxylicious/job/kroxylicious-system-tests-pr/11/)
**KAFKA VERSION:** 3.6.1
**TOTAL:** 6
**PASS:** 3
**FAIL:** 2
**SKIP:** 1
### :heavy_exclamation_mark: Test Failures :heavy_exclamation_mark:
- ensureClusterHasEncryptedMessage(String) in io.kroxylicious.systemtests.EnvelopeEncryptionST

**Re-run command**:
@strimzi-ci run system tests
sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[80.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952)


k-wall: @franvila it looks to me like the systests for this PR ran with the wrong production code.  The tests pass locally.
I'll try running them again.
k-wall: @strimzi-ci run system tests
strimzi-ci: :arrow_forward: **Build started** - check Jenkins for more info. :arrow_forward:
strimzi-ci: ### :x: Test Summary :x:
**BUILD NUMBER:** [13](https://master-jenkins-csb-amq-streams.apps.ocp-c1.prod.psi.redhat.com/job/kroxylicious/job/kroxylicious-system-tests-pr/13/)
**KAFKA VERSION:** 3.6.1
**TOTAL:** 6
**PASS:** 3
**FAIL:** 2
**SKIP:** 1
### :heavy_exclamation_mark: Test Failures :heavy_exclamation_mark:
- ensureClusterHasEncryptedMessage(String) in io.kroxylicious.systemtests.EnvelopeEncryptionST

**Re-run command**:
@strimzi-ci run system tests
k-wall: I'm sorry, the mistake is mine.  

This PR refactors the `vaultUrl` config key to have a different name.  I thought I'd got all the occurrences, but I missed one in the system-test tree.
k-wall: > I'm sorry, the mistake is mine.
> 
> This PR refactors the `vaultUrl` config key to have a different name. I thought I'd got all the occurrences, but I missed one in the system-test tree.  I have no idea why my EESTs worked locally.


k-wall: @strimzi-ci run system tests
sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[80.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952)


strimzi-ci: ### :heavy_check_mark: Test Summary :heavy_check_mark:
**BUILD NUMBER:** [14](https://master-jenkins-csb-amq-streams.apps.ocp-c1.prod.psi.redhat.com/job/kroxylicious/job/kroxylicious-system-tests-pr/14/)
**KAFKA VERSION:** 3.6.1
**TOTAL:** 6
**PASS:** 5
**FAIL:** 0
**SKIP:** 1

sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[80.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952)


sonarcloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952) **Quality Gate passed**  
Kudos, no new issues were introduced!

[0 New issues](https://sonarcloud.io/project/issues?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=kroxylicious_kroxylicious&pullRequest=952&resolved=false&inNewCodePeriod=true)  
[80.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_coverage&view=list)  
[0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=kroxylicious_kroxylicious&pullRequest=952&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=kroxylicious_kroxylicious&pullRequest=952)



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ae5676f2..ef8c13d8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@ Please enumerate **all user-facing** changes using format `<githib issue/pr numb
 
 ## 0.5.0
 
+* [#940](https://github.com/kroxylicious/kroxylicious/issues/940): Support vault namespaces and support secrets transit engine at locations other than /transit
 * [#951](https://github.com/kroxylicious/kroxylicious/pull/951): Include the kroxylicious maintained filters in the dist by default
 * [#910](https://github.com/kroxylicious/kroxylicious/pull/910): Envelope encryption preserve batches within MemoryRecords
 * [#883](https://github.com/kroxylicious/kroxylicious/pull/883): Ensure we only initialise a filter factory once.
@@ -22,6 +23,9 @@ Please enumerate **all user-facing** changes using format `<githib issue/pr numb
   adjust your dependencies as your adopt this release.
 * `io.kroxylicious:kroxylicious-filter-test-support` now contains RecordTestUtils for creating example `Record`, `RecordBatch` and `MemoryRecords`. It also contains 
   assertj assertions for those same classes to enable us to write fluent assertions, accessible via `io.kroxylicious.test.assertj.KafkaAssertions`.
+* The configuration for VaultKMS service has changed.  Instead of the `vaultUrl` config key, the provider now
+  requires `vaultTransitEngineUrl`.  This must provide the complete path to the Transit Engine on the
+  HashiCorp Vault instance (e.g. https://myvault:8200/v1/transit or https://myvault:8200/v1/mynamespace/transit).  
 
 ## 0.4.1
 
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKms.java b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKms.java
index a54610d3..5d513894 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKms.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKms.java
@@ -17,6 +17,7 @@ import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.CompletionStage;
 import java.util.function.Function;
 import java.util.function.Supplier;
+import java.util.regex.Pattern;
 
 import javax.crypto.SecretKey;
 import javax.crypto.spec.SecretKeySpec;
@@ -46,9 +47,6 @@ import static java.nio.charset.StandardCharsets.UTF_8;
  * <br/>
  * <h2>TODO</h2>
  * <ul>
- *    <li>don't assume /transit endpoint</li>
- *    <li>HTTPs</li>
- *    <li>Client trust store</li>
  *    <li>Securely pass vault token</li>
  * </ul>
  */
@@ -56,19 +54,37 @@ public class VaultKms implements Kms<String, VaultEdek> {
 
     private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
     private static final String AES_KEY_ALGO = "AES";
+    private static final Pattern LEGAL_API_VERSION_REGEX = Pattern.compile("^/?v1/.+");
     private final Duration timeout;
     private final HttpClient vaultClient;
 
-    private final URI vaultUrl;
+    /**
+     * The vault url which will include the path to the transit engine.
+     */
+    private final URI vaultTransitEngineUrl;
     private final String vaultToken;
 
-    VaultKms(URI vaultUrl, String vaultToken, Duration timeout, SSLContext sslContext) {
-        this.vaultUrl = vaultUrl;
+    VaultKms(URI vaultTransitEngineUrl, String vaultToken, Duration timeout, SSLContext sslContext) {
+        this.vaultTransitEngineUrl = ensureEndsInSlash(validateTransitPath(vaultTransitEngineUrl));
         this.vaultToken = vaultToken;
         this.timeout = timeout;
         vaultClient = createClient(sslContext);
     }
 
+    private URI validateTransitPath(URI vaultTransitEngineUrl) {
+        String path = vaultTransitEngineUrl.getPath();
+        if (path == null || !LEGAL_API_VERSION_REGEX.matcher(path).matches()) {
+            throw new IllegalArgumentException(("vaultTransitEngineUrl path (%s) must start with v1/ and must specify the complete path to"
+                    + " the transit engine e.g 'v1/transit' or 'v1/mynamespace/transit' if namespaces are in use.").formatted(path));
+        }
+        return vaultTransitEngineUrl;
+    }
+
+    private URI ensureEndsInSlash(URI uri) {
+        // We use #resolve to build the urls for the endpoints, so the path must end in a slash.
+        return uri.resolve(uri.getPath() + "/").normalize();
+    }
+
     private HttpClient createClient(SSLContext sslContext) {
         HttpClient.Builder builder = HttpClient.newBuilder();
         if (sslContext != null) {
@@ -90,7 +106,7 @@ public class VaultKms implements Kms<String, VaultEdek> {
     public CompletionStage<DekPair<VaultEdek>> generateDekPair(@NonNull String kekRef) {
 
         var request = createVaultRequest()
-                .uri(vaultUrl.resolve("v1/transit/datakey/plaintext/%s".formatted(encode(kekRef, UTF_8))))
+                .uri(vaultTransitEngineUrl.resolve("datakey/plaintext/%s".formatted(encode(kekRef, UTF_8))))
                 .POST(HttpRequest.BodyPublishers.noBody())
                 .build();
 
@@ -118,7 +134,7 @@ public class VaultKms implements Kms<String, VaultEdek> {
         var body = createDecryptPostBody(edek);
 
         var request = createVaultRequest()
-                .uri(vaultUrl.resolve("v1/transit/decrypt/%s".formatted(encode(edek.kekRef(), UTF_8))))
+                .uri(vaultTransitEngineUrl.resolve("decrypt/%s".formatted(encode(edek.kekRef(), UTF_8))))
                 .POST(HttpRequest.BodyPublishers.ofString(body))
                 .build();
 
@@ -150,7 +166,7 @@ public class VaultKms implements Kms<String, VaultEdek> {
     public CompletableFuture<String> resolveAlias(@NonNull String alias) {
 
         var request = createVaultRequest()
-                .uri(vaultUrl.resolve("v1/transit/keys/%s".formatted(encode(alias, UTF_8))))
+                .uri(vaultTransitEngineUrl.resolve("keys/%s".formatted(encode(alias, UTF_8))))
                 .build();
 
         return vaultClient.sendAsync(request, statusHandler(alias, new JsonBodyHandler<VaultResponse<ReadKeyData>>(new TypeReference<>() {
@@ -186,4 +202,8 @@ public class VaultKms implements Kms<String, VaultEdek> {
             return handler.apply(r);
         };
     }
+
+    /* testing */ URI getVaultTransitEngineUri() {
+        return vaultTransitEngineUrl;
+    }
 }
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsService.java b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsService.java
index 626761cc..eca4db81 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsService.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsService.java
@@ -23,7 +23,7 @@ public class VaultKmsService implements KmsService<Config, String, VaultEdek> {
     @NonNull
     @Override
     public VaultKms buildKms(Config options) {
-        return new VaultKms(options.vaultUrl(), options.vaultToken(), Duration.ofSeconds(20), options.sslContext());
+        return new VaultKms(options.vaultTransitEngineUrl(), options.vaultToken(), Duration.ofSeconds(20), options.sslContext());
     }
 
 }
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/Config.java b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/Config.java
index c399d3f3..c37d244f 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/Config.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/config/Config.java
@@ -20,15 +20,15 @@ import edu.umd.cs.findbugs.annotations.NonNull;
 
 /**
  * Configuration for the Vault KMS service.
- * @param vaultUrl vault url
- * @param vaultToken vault token.
+ * @param vaultTransitEngineUrl URL of the Vault Transit Engine e.g. {@code https://myhashicorpvault:8200/v1/transit}
+ * @param vaultToken Vault token.
  */
 public record Config(
-                     @JsonProperty(required = true) URI vaultUrl,
+                     @JsonProperty(value = "vaultTransitEngineUrl", required = true) URI vaultTransitEngineUrl,
                      @JsonProperty(required = true) String vaultToken,
                      Tls tls) {
     public Config {
-        Objects.requireNonNull(vaultUrl);
+        Objects.requireNonNull(vaultTransitEngineUrl);
         Objects.requireNonNull(vaultToken);
     }
 
diff --git a/kubernetes-examples/envelope-encryption/base/kroxylicious/kroxylicious-config.yaml b/kubernetes-examples/envelope-encryption/base/kroxylicious/kroxylicious-config.yaml
index d78fef41..a2eec9ad 100644
--- a/kubernetes-examples/envelope-encryption/base/kroxylicious/kroxylicious-config.yaml
+++ b/kubernetes-examples/envelope-encryption/base/kroxylicious/kroxylicious-config.yaml
@@ -34,7 +34,7 @@ data:
       config:
         kms: VaultKmsService
         kmsConfig:
-          vaultUrl: http://vault.vault.svc.cluster.local:8200
+          vaultTransitEngineUrl: http://vault.vault.svc.cluster.local:8200/v1/transit
           vaultToken: myroottoken
         selector: TemplateKekSelector
         selectorConfig:


Test patch:
diff --git a/kroxylicious-kms-provider-hashicorp-vault-test-support/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultTestKmsFacade.java b/kroxylicious-kms-provider-hashicorp-vault-test-support/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultTestKmsFacade.java
index ba645ac4..7390f513 100644
--- a/kroxylicious-kms-provider-hashicorp-vault-test-support/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultTestKmsFacade.java
+++ b/kroxylicious-kms-provider-hashicorp-vault-test-support/src/main/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultTestKmsFacade.java
@@ -69,7 +69,7 @@ public class VaultTestKmsFacade implements TestKmsFacade<Config, String, VaultEd
 
     @Override
     public Config getKmsServiceConfig() {
-        return new Config(URI.create(vaultContainer.getHttpHostAddress()), VAULT_TOKEN, null);
+        return new Config(URI.create(vaultContainer.getHttpHostAddress()).resolve("v1/transit"), VAULT_TOKEN, null);
     }
 
     private class VaultTestKekManager implements TestKekManager {
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java
index 9403275d..855c22fd 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/TestVault.java
@@ -35,7 +35,7 @@ public class TestVault implements Closeable {
         VaultContainer<?> vault = new VaultContainer<>(HASHICORP_VAULT)
                 .withVaultToken(VAULT_TOKEN)
                 .withEnv("VAULT_FORMAT", "json")
-                .withInitCommand("secrets enable transit");
+                .withInitCommand("secrets enable transit ");
         if (keys != null) {
             // in vault server -dev mode the 8200 port cannot be reconfigured to TLS. So we configure a second listener on a
             // different port. We expose only the tls port to avoid any chance of a misconfigured client pointing at the plain listener.
@@ -48,7 +48,8 @@ public class TestVault implements Closeable {
         }
         vault.start();
         this.vault = vault;
-        endpoint = URI.create(keys == null ? vault.getHttpHostAddress() : String.format("https://%s:%s", vault.getHost(), vault.getMappedPort(TLS_PORT)));
+        endpoint = URI.create(keys == null ? vault.getHttpHostAddress() : String.format("https://%s:%s", vault.getHost(), vault.getMappedPort(TLS_PORT)))
+                .resolve("v1/transit");
     }
 
     public static TestVault startWithTls(CertificateGenerator.Keys keys) {
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTest.java b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTest.java
index 53c2cdea..b4b3c519 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTest.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/VaultKmsTest.java
@@ -15,18 +15,23 @@ import java.net.http.HttpTimeoutException;
 import java.time.Duration;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ExecutionException;
+import java.util.stream.Stream;
 
 import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
 
 import com.sun.net.httpserver.HttpServer;
 
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 class VaultKmsTest {
 
     // address in TEST-NET-1, reserved for use in example specifications and other documents
-    private static final URI NON_ROUTABLE = URI.create("http://192.0.2.1/");
+    private static final URI NON_ROUTABLE = URI.create("http://192.0.2.1/v1/transit");
     private static final byte ARBITRARY_BODY_BYTE = 5;
 
     @Test
@@ -41,8 +46,9 @@ class VaultKmsTest {
     @Test
     void testRequestTimeoutWaitingForHeaders() {
         HttpServer httpServer = delayResponse(Duration.ofSeconds(1), Duration.ZERO);
-        URI address = URI.create("http://" + httpServer.getAddress().getHostName() + ":" + httpServer.getAddress().getPort() + "/");
-        VaultKms kms = new VaultKms(address, "token", Duration.ofMillis(500), null);
+        var addr = httpServer.getAddress();
+        var uri = URI.create("http://" + addr.getHostName() + ":" + addr.getPort() + "/v1/transit");
+        VaultKms kms = new VaultKms(uri, "token", Duration.ofMillis(500), null);
         CompletableFuture<String> alias = kms.resolveAlias("alias");
         assertThat(alias).failsWithin(Duration.ofSeconds(2))
                 .withThrowableOfType(ExecutionException.class)
@@ -50,12 +56,51 @@ class VaultKmsTest {
         httpServer.stop(0);
     }
 
+    static Stream<Arguments> acceptableVaultTransitEnginePaths() {
+        var uri = URI.create("https://localhost:1234");
+        return Stream.of(
+                Arguments.of("basic", uri.resolve("v1/transit"), "/v1/transit/"),
+                Arguments.of("trailing slash", uri.resolve("v1/transit/"), "/v1/transit/"),
+                Arguments.of("single namespace", uri.resolve("v1/ns1/transit"), "/v1/ns1/transit/"),
+                Arguments.of("many namespaces", uri.resolve("v1/ns1/ns2/transit"), "/v1/ns1/ns2/transit/"),
+                Arguments.of("non standard engine name", uri.resolve("v1/mytransit"), "/v1/mytransit/"));
+    }
+
+    @ParameterizedTest(name = "{0}")
+    @MethodSource("acceptableVaultTransitEnginePaths")
+    void acceptsVaultTransitEnginePaths(String name, URI uri, String expected) {
+        var kms = new VaultKms(uri, "token", Duration.ofMillis(500), null);
+        assertThat(kms.getVaultTransitEngineUri())
+                .extracting(URI::getPath)
+                .isEqualTo(expected);
+
+    }
+
+    static Stream<Arguments> unacceptableVaultTransitEnginePaths() {
+        var uri = URI.create("https://localhost:1234");
+        return Stream.of(
+                Arguments.of("no path", uri),
+                Arguments.of("missing path", uri.resolve("/")),
+                Arguments.of("unrecognized API version", uri.resolve("v999/transit")),
+                Arguments.of("missing engine", uri.resolve("v1")),
+                Arguments.of("empty engine", uri.resolve("v1/")));
+    }
+
+    @ParameterizedTest(name = "{0}")
+    @MethodSource("unacceptableVaultTransitEnginePaths")
+    void detectsUnacceptableVaultTransitEnginePaths(String name, URI uri) {
+        assertThatThrownBy(() -> new VaultKms(uri, "token", Duration.ZERO, null))
+                .isInstanceOf(IllegalArgumentException.class);
+
+    }
+
     @Disabled("JDK http client request timeout cancelled after headers received currently, todo add our own timeout future")
     @Test
     void testRequestTimeoutWaitingForBody() {
         HttpServer httpServer = delayResponse(Duration.ZERO, Duration.ofSeconds(1));
-        URI address = URI.create("http://" + httpServer.getAddress().getHostName() + ":" + httpServer.getAddress().getPort() + "/");
-        VaultKms kms = new VaultKms(address, "token", Duration.ofMillis(500), null);
+        var addr = httpServer.getAddress();
+        URI uri = URI.create("http://" + addr.getHostName() + ":" + httpServer.getAddress().getPort() + "/v1/transit");
+        VaultKms kms = new VaultKms(uri, "token", Duration.ofMillis(500), null);
         CompletableFuture<String> alias = kms.resolveAlias("alias");
         assertThat(alias).failsWithin(Duration.ofSeconds(2))
                 .withThrowableOfType(ExecutionException.class)
@@ -82,6 +127,7 @@ class VaultKmsTest {
         }
     }
 
+    @SuppressWarnings("java:S2925")
     private static void sleep(Duration delayHeaders) {
         try {
             Thread.sleep(delayHeaders.toMillis());
diff --git a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/ConfigParseTest.java b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/ConfigParseTest.java
index f04c52dd..8322ef7f 100644
--- a/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/ConfigParseTest.java
+++ b/kroxylicious-kms-provider-hashicorp-vault/src/test/java/io/kroxylicious/kms/provider/hashicorp/vault/config/ConfigParseTest.java
@@ -30,13 +30,13 @@ class ConfigParseTest {
     void testVaultUrlAndToken() throws IOException {
         String json = """
                 {
-                    "vaultUrl": "http://vault",
+                    "vaultTransitEngineUrl": "http://vault",
                     "vaultToken": "token"
                 }
                 """;
         Config config = readConfig(json);
         assertThat(config.vaultToken()).isEqualTo("token");
-        assertThat(config.vaultUrl()).isEqualTo(URI.create("http://vault"));
+        assertThat(config.vaultTransitEngineUrl()).isEqualTo(URI.create("http://vault"));
     }
 
     @Test
@@ -48,7 +48,7 @@ class ConfigParseTest {
                     }
                     """;
             readConfig(json);
-        }).isInstanceOf(MismatchedInputException.class).hasMessageContaining("vaultUrl");
+        }).isInstanceOf(MismatchedInputException.class).hasMessageContaining("vaultTransitEngineUrl");
     }
 
     @Test
@@ -56,7 +56,7 @@ class ConfigParseTest {
         Assertions.assertThatThrownBy(() -> {
             String json = """
                     {
-                        "vaultUrl": null,
+                        "vaultTransitEngineUrl": null,
                         "vaultToken": "token"
                     }
                     """;
@@ -69,7 +69,7 @@ class ConfigParseTest {
         Assertions.assertThatThrownBy(() -> {
             String json = """
                     {
-                        "vaultUrl": "https://vault"
+                        "vaultTransitEngineUrl": "https://vault"
                     }
                     """;
             readConfig(json);
@@ -81,7 +81,7 @@ class ConfigParseTest {
         Assertions.assertThatThrownBy(() -> {
             String json = """
                     {
-                        "vaultUrl": "https://vault",
+                        "vaultTransitEngineUrl": "https://vault",
                         "vaultToken": null
                     }
                     """;
@@ -93,7 +93,7 @@ class ConfigParseTest {
     void testEmptyTls() throws Exception {
         String json = """
                 {
-                    "vaultUrl": "https://vault",
+                    "vaultTransitEngineUrl": "https://vault",
                     "vaultToken": "token",
                     "tls": {}
                 }
@@ -108,7 +108,7 @@ class ConfigParseTest {
     void testMissingTls() throws Exception {
         String json = """
                 {
-                    "vaultUrl": "https://vault",
+                    "vaultTransitEngineUrl": "https://vault",
                     "vaultToken": "token"
                 }
                 """;
@@ -122,7 +122,7 @@ class ConfigParseTest {
     void testTlsTrust() throws Exception {
         String json = """
                 {
-                    "vaultUrl": "https://vault",
+                    "vaultTransitEngineUrl": "https://vault",
                     "vaultToken": "token",
                     "tls": {
                         "trust": {
diff --git a/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/resources/vault/KubeVaultTestKmsFacade.java b/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/resources/vault/KubeVaultTestKmsFacade.java
index 5d46ad39..ec27e2d2 100644
--- a/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/resources/vault/KubeVaultTestKmsFacade.java
+++ b/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/resources/vault/KubeVaultTestKmsFacade.java
@@ -71,7 +71,7 @@ public class KubeVaultTestKmsFacade implements TestKmsFacade<Config, String, Vau
 
     @Override
     public Config getKmsServiceConfig() {
-        return new Config(URI.create(vault.getBootstrap()), Vault.VAULT_ROOT_TOKEN, null);
+        return new Config(URI.create(vault.getBootstrap()).resolve("v1/transit"), Vault.VAULT_ROOT_TOKEN, null);
     }
 
     private class VaultTestKekManager implements TestKekManager {
diff --git a/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/templates/kroxylicious/KroxyliciousConfigMapTemplates.java b/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/templates/kroxylicious/KroxyliciousConfigMapTemplates.java
index 07444e1d..1e21e08e 100644
--- a/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/templates/kroxylicious/KroxyliciousConfigMapTemplates.java
+++ b/kroxylicious-systemtests/src/main/java/io/kroxylicious/systemtests/templates/kroxylicious/KroxyliciousConfigMapTemplates.java
@@ -71,7 +71,7 @@ public class KroxyliciousConfigMapTemplates {
                   config:
                     kms: VaultKmsService
                     kmsConfig:
-                      vaultUrl: http://%VAULT_SERVICE%.%VAULT_NAMESPACE%.svc.cluster.local:8200
+                      vaultTransitEngineUrl: http://%VAULT_SERVICE%.%VAULT_NAMESPACE%.svc.cluster.local:8200/v1/transit
                       vaultToken: %ROOT_TOKEN%
                     selector: TemplateKekSelector
                     selectorConfig: