Repository: quarkiverse/quarkus-vault
Pull request #360: Support refresh of limited use tokens (num_use > 0)
Author: login: kdubb
Created: 2025-01-23T20:30:05Z | Merged: 2025-04-24T06:16:35Z

Fixes #338

While enhancing the test for caching tokens I realized supporting limited use tokens should be fairly straightforward.

@vsevel What do you think? The only tricky thing is that we copy token values (e.g., for caching) so I used an `AtomicReference` as a shared reference count. Other than that, I added `getClientTokenForUsage` to decrement the count when it's being used for authorization and overloaded the expiration tests to take into account it's max # of uses.

I added a test to the caching token provider tests suite that tests limited use tokens and it seems to perform as expected.

It does not attempt to `extend` limited use tokens that are over their use limit because from my investigation it doesn't seem that's allowed.

@alexjaravete Can you build this this locally and see if this "just works" for you?

Linked issue #338: No support for token_num_uses in auth token
Hi,

we stumbled across an issue where `token_num_uses` is not supported in the vault extension.
When you have a token that has a `token_num_uses` value of anything else other than `0`, and it reaches the limit, it does not automatically renew the token.

So any further requests fail with `403 Not Authorized` and a restart of the application is necessary or you need to wait until the TTL expires, and a new token is created.

Thank you!
Linked issue comments: user: login: vsevel
created_at: 2025-04-25T12:57:22Z
body: this has been fixed in `4.3.0` and will be part of the quarkus platform `3.22`
author_association: CONTRIBUTOR

Diff:
diff --git a/client/src/main/java/io/quarkus/vault/client/VaultClient.java b/client/src/main/java/io/quarkus/vault/client/VaultClient.java
index d37aa9ce..18a4532a 100644
--- a/client/src/main/java/io/quarkus/vault/client/VaultClient.java
+++ b/client/src/main/java/io/quarkus/vault/client/VaultClient.java
@@ -6,14 +6,14 @@
 import java.nio.file.Path;
 import java.time.Duration;
 import java.time.InstantSource;
-import java.util.Objects;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.CompletionException;
 import java.util.concurrent.CompletionStage;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.atomic.AtomicReference;
-import java.util.function.Function;
 import java.util.function.Supplier;
+import java.util.logging.Level;
+import java.util.logging.Logger;
 
 import io.quarkus.vault.client.api.VaultAuthAccessor;
 import io.quarkus.vault.client.api.VaultSecretsAccessor;
@@ -37,6 +37,7 @@ public static class Builder {
         private Duration requestTimeout;
         private LogConfidentialityLevel logConfidentialityLevel;
         private InstantSource instantSource = InstantSource.system();
+        private int maxRetries = 2;
 
         public Builder baseUrl(URL baseUrl) {
             this.baseUrl = requireNonNull(baseUrl, "baseUrl is required");
@@ -118,6 +119,23 @@ public Builder requestTimeout(Duration requestTimeout) {
             return this;
         }
 
+        /**
+         * Sets the maximum number of retries for requests that fail due to token expiration.
+         * <p>
+         * The default is 2 retries, for a total of 3 attempts.
+         * <p>
+         *
+         * @param maxRetries the maximum number of retries, excluding the initial request, must be greater than or equal to 0
+         * @return this builder
+         */
+        public Builder maxRetries(int maxRetries) {
+            if (maxRetries < 0) {
+                throw new IllegalArgumentException("maxRetries must be greater than or equal to 0");
+            }
+            this.maxRetries = maxRetries;
+            return this;
+        }
+
         public Builder logConfidentialityLevel(LogConfidentialityLevel logConfidentialityLevel) {
             this.logConfidentialityLevel = logConfidentialityLevel != null ? logConfidentialityLevel
                     : LogConfidentialityLevel.HIGH;
@@ -155,6 +173,8 @@ public static Builder builder() {
         return new Builder();
     }
 
+    private static final Logger log = Logger.getLogger(VaultClient.class.getName());
+
     private final AtomicReference<VaultSecretsAccessor> secrets = new AtomicReference<>();
     private final AtomicReference<VaultAuthAccessor> auth = new AtomicReference<>();
     private final URL baseUrl;
@@ -165,6 +185,7 @@ public static Builder builder() {
     private final VaultTokenProvider tokenProvider;
     private final String namespace;
     private final InstantSource instantSource;
+    private final int maxAttempts;
 
     private VaultClient(Builder builder) {
         this.baseUrl = requireNonNull(builder.baseUrl, "baseUrl is required");
@@ -175,6 +196,7 @@ private VaultClient(Builder builder) {
         this.tokenProvider = builder.tokenProvider;
         this.namespace = builder.namespace;
         this.instantSource = builder.instantSource;
+        this.maxAttempts = builder.maxRetries + 1;
     }
 
     public VaultSecretsAccessor secrets() {
@@ -244,27 +266,76 @@ public <T> CompletionStage<VaultResponse<T>> execute(VaultRequest<T> request) {
         }
 
         if (request.hasToken() || tokenProvider == null) {
+            log.finer(() -> "Executing authorized request " + request.getOperation());
             return executor.execute(requestBuilder.rebuild());
         }
 
-        var appliedToken = new AtomicReference<VaultToken>(null);
+        log.finer(() -> "Executing unauthorized request " + request.getOperation());
+
+        Supplier<CompletionStage<VaultToken>> providedToken = () -> {
+            log.finer(() -> "Requesting token for request " + request.getOperation());
+            return tokenProvider.apply(VaultAuthRequest.of(this, request, instantSource));
+        };
 
-        var providedToken = tokenProvider.apply(VaultAuthRequest.of(this, request, instantSource));
+        var appliedToken = new AtomicReference<VaultToken>(null);
 
-        Supplier<CompletionStage<VaultResponse<T>>> responseSupplier = () -> providedToken.thenCompose(token -> {
+        Supplier<CompletionStage<VaultResponse<T>>> responseSupplier = () -> providedToken.get().thenCompose(token -> {
 
             appliedToken.set(token);
 
             if (token == null) {
                 requestBuilder.noToken();
             } else {
-                requestBuilder.token(token.getClientToken());
+                requestBuilder.token(token.getClientTokenForUsage());
             }
 
             return executor.execute(requestBuilder.rebuild());
         });
 
-        return CompletableFuture.completedStage(0).thenCompose(new Retry<>(responseSupplier, 1, appliedToken));
+        var execution = attempt(0, responseSupplier, request, appliedToken);
+
+        if (log.isLoggable(Level.FINER)) {
+            return execution.whenComplete((response, error) -> {
+                if (error != null) {
+                    log.finer(() -> "Failed request " + request.getOperation() + " with error " + error);
+                } else {
+                    log.finer(() -> "Successful request " + request.getOperation());
+                }
+            });
+        } else {
+            return execution;
+        }
+    }
+
+    public <T> CompletionStage<VaultResponse<T>> attempt(Integer attempt,
+            Supplier<CompletionStage<VaultResponse<T>>> responseSupplier, VaultRequest<T> request,
+            AtomicReference<VaultToken> appliedToken) {
+        log.finer(() -> "Attempt %d of %d for request %s".formatted(attempt + 1, maxAttempts, request.getOperation()));
+
+        return responseSupplier.get().exceptionallyCompose(failure -> {
+
+            if (attempt < maxAttempts && shouldRetry(failure, appliedToken.get())) {
+                log.finer(() -> "Retrying request %s due to %s".formatted(request.getOperation(), failure));
+
+                return attempt(attempt + 1, responseSupplier, request, appliedToken);
+            }
+            return CompletableFuture.failedFuture(failure);
+        });
+    }
+
+    private boolean shouldRetry(Throwable failure, VaultToken appliedToken) {
+        failure = unwrapException(failure);
+        if (failure instanceof VaultTokenException) {
+            return true;
+        }
+        if (failure instanceof VaultClientException ve) {
+            var cachedPermissionFailure = ve.isPermissionDenied() && appliedToken != null && appliedToken.isFromCache();
+            if (cachedPermissionFailure) {
+                tokenProvider.invalidateCache();
+            }
+            return cachedPermissionFailure;
+        }
+        return false;
     }
 
     public VaultClient.Builder configure() {
@@ -278,38 +349,11 @@ public VaultClient.Builder configure() {
         builder.logConfidentialityLevel = logConfidentialityLevel;
         return builder;
     }
-}
 
-class Retry<T> implements Function<Integer, CompletionStage<T>> {
-    private final Supplier<CompletionStage<T>> upstreamSupplier;
-    private final long maxRetries;
-    private final AtomicReference<VaultToken> appliedToken;
-
-    public Retry(Supplier<CompletionStage<T>> upstreamSupplier, long maxRetries, AtomicReference<VaultToken> appliedToken) {
-        assert maxRetries > 0;
-        this.upstreamSupplier = Objects.requireNonNull(upstreamSupplier, "upstreamSupplier");
-        this.maxRetries = maxRetries;
-        this.appliedToken = appliedToken;
-    }
-
-    @Override
-    public CompletionStage<T> apply(Integer attempt) {
-        return upstreamSupplier.get().exceptionallyCompose(failure -> {
-            if (attempt < maxRetries && shouldRetry(failure)) {
-                return apply(attempt + 1);
-            }
-            return CompletableFuture.failedFuture(failure);
-        });
-    }
-
-    private boolean shouldRetry(Throwable failure) {
-        if (failure instanceof CompletionException || failure instanceof ExecutionException) {
-            failure = failure.getCause();
+    private static Throwable unwrapException(Throwable e) {
+        if (e instanceof CompletionException || e instanceof ExecutionException) {
+            return e.getCause();
         }
-        if (failure instanceof VaultClientException ve) {
-            var token = appliedToken.get();
-            return ve.isPermissionDenied() && token != null && token.isFromCache();
-        }
-        return false;
+        return e;
     }
 }
diff --git a/client/src/main/java/io/quarkus/vault/client/auth/VaultAppRoleTokenProvider.java b/client/src/main/java/io/quarkus/vault/client/auth/VaultAppRoleTokenProvider.java
index 444c0795..aeac5562 100644
--- a/client/src/main/java/io/quarkus/vault/client/auth/VaultAppRoleTokenProvider.java
+++ b/client/src/main/java/io/quarkus/vault/client/auth/VaultAppRoleTokenProvider.java
@@ -31,7 +31,7 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                 .thenApply(res -> {
                     var auth = res.getAuth();
                     return VaultToken.from(auth.getClientToken(), auth.isRenewable(), auth.getLeaseDuration(),
-                            authRequest.getInstantSource());
+                            auth.getNumUses(), authRequest.getInstantSource());
                 });
     }
 }
diff --git a/client/src/main/java/io/quarkus/vault/client/auth/VaultCachingTokenProvider.java b/client/src/main/java/io/quarkus/vault/client/auth/VaultCachingTokenProvider.java
index dda18439..41851196 100644
--- a/client/src/main/java/io/quarkus/vault/client/auth/VaultCachingTokenProvider.java
+++ b/client/src/main/java/io/quarkus/vault/client/auth/VaultCachingTokenProvider.java
@@ -30,12 +30,20 @@ public VaultCachingTokenProvider(VaultTokenProvider delegate, Duration renewGrac
         this.renewGracePeriod = renewGracePeriod;
     }
 
+    public Optional<VaultToken> getCachedToken() {
+        return Optional.ofNullable(cachedToken.get());
+    }
+
     @Override
     public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
 
-        var cachedToken = Optional.ofNullable(this.cachedToken.get())
+        var cachedToken = getCachedToken()
                 .map(token -> {
                     var logLevel = authRequest.getRequest().getLogConfidentialityLevel();
+                    if (!token.hasAllowedUsesRemaining()) {
+                        log.fine("cached token " + token.getConfidentialInfo(logLevel) + " has exhausted its allowed usages");
+                        return null;
+                    }
                     if (token.isExpired()) {
                         log.fine("cached token " + token.getConfidentialInfo(logLevel) + " has expired");
                         return null;
@@ -94,7 +102,7 @@ public CompletionStage<VaultToken> extend(VaultAuthRequest authRequest, String c
                 .thenApply(res -> {
                     var auth = res.getAuth();
                     var vaultToken = VaultToken.from(auth.getClientToken(), auth.isRenewable(), auth.getLeaseDuration(),
-                            authRequest.getInstantSource());
+                            auth.getNumUses(), authRequest.getInstantSource());
                     sanityCheck(vaultToken);
                     log.fine("extended login token: " + vaultToken.getConfidentialInfo(logLevel));
                     return vaultToken;
diff --git a/client/src/main/java/io/quarkus/vault/client/auth/VaultKubernetesTokenProvider.java b/client/src/main/java/io/quarkus/vault/client/auth/VaultKubernetesTokenProvider.java
index 013061c4..c0114fa4 100644
--- a/client/src/main/java/io/quarkus/vault/client/auth/VaultKubernetesTokenProvider.java
+++ b/client/src/main/java/io/quarkus/vault/client/auth/VaultKubernetesTokenProvider.java
@@ -44,7 +44,7 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                     .thenApply(res -> {
                         var auth = res.getAuth();
                         return VaultToken.from(auth.getClientToken(), auth.isRenewable(), auth.getLeaseDuration(),
-                                authRequest.getInstantSource());
+                                auth.getNumUses(), authRequest.getInstantSource());
                     });
         });
     }
diff --git a/client/src/main/java/io/quarkus/vault/client/auth/VaultTimeLimited.java b/client/src/main/java/io/quarkus/vault/client/auth/VaultTimeLimited.java
index 98bc409e..327d6fec 100644
--- a/client/src/main/java/io/quarkus/vault/client/auth/VaultTimeLimited.java
+++ b/client/src/main/java/io/quarkus/vault/client/auth/VaultTimeLimited.java
@@ -41,6 +41,19 @@ public Duration getLeaseDuration() {
         return leaseDuration;
     }
 
+    /**
+     * Check if the token is valid, in the sense that using it for authentication
+     * *should* be successful.
+     * <p>
+     * This base class implementation checks if the token is expired, but subclasses
+     * may override this method to provide additional checks.
+     *
+     * @return true if the token is valid, false otherwise
+     */
+    public boolean isValid() {
+        return !isExpired();
+    }
+
     public boolean isExpired() {
         return instantSource.instant().isAfter(getExpiresAt());
     }
diff --git a/client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java b/client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java
index bf3dfdfe..a7f46ad9 100644
--- a/client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java
+++ b/client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java
@@ -1,10 +1,12 @@
 package io.quarkus.vault.client.auth;
 
 import static io.quarkus.vault.client.logging.LogConfidentialityLevel.LOW;
+import static java.lang.Math.max;
 
 import java.time.Duration;
 import java.time.Instant;
 import java.time.InstantSource;
+import java.util.concurrent.atomic.AtomicInteger;
 
 import io.quarkus.vault.client.logging.LogConfidentialityLevel;
 
@@ -12,45 +14,88 @@ public class VaultToken extends VaultTimeLimited {
 
     private final String clientToken;
     private final boolean fromCache;
+    private final AtomicInteger allowedUsesRemaining;
 
-    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, InstantSource instantSource) {
-        return new VaultToken(clientToken, renewable, leaseDuration, false, instantSource);
+    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return new VaultToken(clientToken, renewable, leaseDuration, allowedUsesRemaining, false, instantSource);
     }
 
-    public static VaultToken renewable(String clientToken, Duration leaseDuration, InstantSource instantSource) {
-        return from(clientToken, true, leaseDuration, instantSource);
+    public static VaultToken renewable(String clientToken, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return from(clientToken, true, leaseDuration, allowedUsesRemaining, instantSource);
     }
 
     public static VaultToken neverExpires(String clientToken, InstantSource instantSource) {
-        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), instantSource);
+        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), null, instantSource);
     }
 
-    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, boolean fromCache,
+    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            boolean fromCache,
             InstantSource instantSource) {
         super(renewable, leaseDuration, instantSource);
         this.clientToken = clientToken;
         this.fromCache = fromCache;
+        this.allowedUsesRemaining = new AtomicInteger(
+                allowedUsesRemaining != null && allowedUsesRemaining > 0 ? allowedUsesRemaining : Integer.MAX_VALUE);
     }
 
     public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, Instant created, boolean fromCache,
-            InstantSource instantSource) {
+            AtomicInteger allowedUsesRemaining, InstantSource instantSource) {
         super(renewable, leaseDuration, created, instantSource);
         this.clientToken = clientToken;
         this.fromCache = fromCache;
+        this.allowedUsesRemaining = allowedUsesRemaining;
     }
 
     public VaultToken cached() {
-        return new VaultToken(clientToken, isRenewable(), getLeaseDuration(), getCreated(), true, getInstantSource());
+        return new VaultToken(clientToken, isRenewable(), getLeaseDuration(), getCreated(), true, allowedUsesRemaining,
+                getInstantSource());
     }
 
     public String getClientToken() {
         return clientToken;
     }
 
+    public int getAllowedUsesRemaining() {
+        return allowedUsesRemaining.get();
+    }
+
+    public boolean hasAllowedUsesRemaining() {
+        return allowedUsesRemaining.get() > 0;
+    }
+
     public boolean isFromCache() {
         return fromCache;
     }
 
+    /**
+     * Returns the client token for usage, decrementing the allowed uses remaining.
+     * <p>
+     * During normal operation, this method should always return the client token, as the remaining
+     * use count should be checked before calling this method. However, if the token is requested for use
+     * multiple times in parallel, it is possible that the use count will be exhausted between the check and the
+     * call to this method. In that case, the method will throw a {@link VaultTokenException}. The
+     * {@link io.quarkus.vault.client.VaultClient} implementation should handle this exception and retry the
+     * request, after obtaining a new token (if possible).
+     *
+     * @apiNote This ignores the token's expiration; it is the caller's responsibility to ensure the token has not expired.
+     * @return the client token
+     * @throws VaultTokenException if the token has no allowed uses remaining
+     */
+    public String getClientTokenForUsage() {
+        var remainingUses = allowedUsesRemaining.updateAndGet(remaining -> max(remaining - 1, -1));
+        if (remainingUses == -1) {
+            throw new VaultTokenException(VaultTokenException.Reason.TOKEN_USES_EXHAUSTED);
+        }
+        return clientToken;
+    }
+
+    @Override
+    public boolean isValid() {
+        return super.isValid() && hasAllowedUsesRemaining();
+    }
+
     public String getConfidentialInfo(LogConfidentialityLevel level) {
         return "{clientToken: " + level.maskWithTolerance(clientToken, LOW) + ", " + super.info() + "}";
     }
diff --git a/client/src/main/java/io/quarkus/vault/client/auth/VaultTokenException.java b/client/src/main/java/io/quarkus/vault/client/auth/VaultTokenException.java
new file mode 100644
index 00000000..0bfedb5d
--- /dev/null
+++ b/client/src/main/java/io/quarkus/vault/client/auth/VaultTokenException.java
@@ -0,0 +1,27 @@
+package io.quarkus.vault.client.auth;
+
+import io.quarkus.vault.client.VaultException;
+
+public class VaultTokenException extends VaultException {
+
+    public enum Reason {
+        TOKEN_EXPIRED("Token has expired"),
+        TOKEN_USES_EXHAUSTED("Token has exhausted its allowed usages"),
+
+        ;
+
+        public final String message;
+
+        Reason(String message) {
+            this.message = message;
+        }
+    }
+
+    public final Reason reason;
+
+    public VaultTokenException(Reason reason) {
+        super(reason.message);
+        this.reason = reason;
+    }
+
+}
diff --git a/client/src/main/java/io/quarkus/vault/client/auth/VaultUserPassTokenProvider.java b/client/src/main/java/io/quarkus/vault/client/auth/VaultUserPassTokenProvider.java
index 0dc58b19..3c84b35b 100644
--- a/client/src/main/java/io/quarkus/vault/client/auth/VaultUserPassTokenProvider.java
+++ b/client/src/main/java/io/quarkus/vault/client/auth/VaultUserPassTokenProvider.java
@@ -31,7 +31,7 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                     .thenApply(res -> {
                         var auth = res.getAuth();
                         return VaultToken.from(auth.getClientToken(), auth.isRenewable(), auth.getLeaseDuration(),
-                                authRequest.getInstantSource());
+                                auth.getNumUses(), authRequest.getInstantSource());
                     });
         });
     }
diff --git a/client/src/main/java/io/quarkus/vault/client/common/VaultTracingExecutor.java b/client/src/main/java/io/quarkus/vault/client/common/VaultTracingExecutor.java
index bb7fa8bc..513864b3 100644
--- a/client/src/main/java/io/quarkus/vault/client/common/VaultTracingExecutor.java
+++ b/client/src/main/java/io/quarkus/vault/client/common/VaultTracingExecutor.java
@@ -19,16 +19,16 @@ public VaultTracingExecutor(VaultRequestExecutor delegate) {
 
     @Override
     public <T> CompletionStage<VaultResponse<T>> execute(VaultRequest<T> request) {
-        log.info("Executing: " + request.getOperation() + lineSeparator() + getCurlFormattedRequest(request));
+        log.info("REQUEST: " + request.getOperation() + lineSeparator() + getCurlFormattedRequest(request));
         return delegate.execute(request)
                 .thenApply((response) -> {
-                    var message = "Response: " + request.getOperation() + lineSeparator() + getHTTPFormattedResponse(response)
+                    var message = "RESPONSE: " + request.getOperation() + lineSeparator() + getHTTPFormattedResponse(response)
                             + lineSeparator();
                     log.log(response.isSuccessful() ? Level.INFO : Level.WARNING, message);
                     return response;
                 })
                 .exceptionallyCompose(error -> {
-                    log.log(Level.SEVERE, "Request failed: " + error);
+                    log.log(Level.SEVERE, "EXCEPTION: " + error);
                     return CompletableFuture.failedStage(error);
                 });
     }
diff --git a/client/src/test/java/io/quarkus/vault/client/VaultCachingTokenProviderTest.java b/client/src/test/java/io/quarkus/vault/client/VaultCachingTokenProviderTest.java
index 06ccd9a7..3b0f7d02 100644
--- a/client/src/test/java/io/quarkus/vault/client/VaultCachingTokenProviderTest.java
+++ b/client/src/test/java/io/quarkus/vault/client/VaultCachingTokenProviderTest.java
@@ -1,15 +1,23 @@
 package io.quarkus.vault.client;
 
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.BDDAssertions.as;
+import static org.assertj.core.api.InstanceOfAssertFactories.BOOLEAN;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.Mockito.*;
 
 import java.time.Duration;
 import java.time.Instant;
+import java.util.List;
+import java.util.Map;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.CompletionStage;
+import java.util.logging.Logger;
 
+import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
 
 import io.quarkus.vault.client.api.auth.token.VaultAuthTokenCreateTokenParams;
 import io.quarkus.vault.client.api.common.VaultAuthResult;
@@ -18,13 +26,21 @@
 import io.quarkus.vault.client.auth.VaultToken;
 import io.quarkus.vault.client.auth.VaultTokenProvider;
 import io.quarkus.vault.client.common.VaultRequest;
-import io.quarkus.vault.client.test.TickableInstanceSource;
+import io.quarkus.vault.client.test.TickableInstantSource;
 import io.quarkus.vault.client.test.VaultClientTest;
 
-@VaultClientTest
+@VaultClientTest(secrets = {
+        @VaultClientTest.Mount(type = "kv", path = "empty", options = { "-version=2" }),
+})
 public class VaultCachingTokenProviderTest {
 
-    private static final TickableInstanceSource tickableInstanceSource = new TickableInstanceSource(Instant.now());
+    private static final TickableInstantSource tickableInstantSource = new TickableInstantSource(Instant.now());
+
+    @BeforeAll
+    public static void beforeAll(VaultClient vaultClient) {
+        vaultClient.secrets().kv2().updateSecret("test", null, Map.of("a", "1", "b", "2"))
+                .toCompletableFuture().join();
+    }
 
     @Test
     public void testCachedTokensAreNotChanged(VaultClient client) throws Exception {
@@ -32,16 +48,17 @@ public void testCachedTokensAreNotChanged(VaultClient client) throws Exception {
         var clientToken = client.auth().token().create(null)
                 .thenApply(VaultAuthResult::getClientToken)
                 .toCompletableFuture().get();
-        var tokenProvider = spy(new VaultTokenProvider() {
+        //noinspection Convert2Lambda
+        var tokenProvider = spy((VaultTokenProvider) new VaultTokenProvider() {
             @Override
             public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
-                var token = VaultToken.from(clientToken, true, Duration.ofMinutes(1), tickableInstanceSource);
+                var token = VaultToken.from(clientToken, true, Duration.ofMinutes(1), null, tickableInstantSource);
                 return CompletableFuture.completedStage(token);
             }
         });
         var cachingTokenProvider = new VaultCachingTokenProvider(tokenProvider, Duration.ofSeconds(30));
 
-        var authRequest = new VaultAuthRequest(client.getExecutor(), VaultRequest.get("test").build(), tickableInstanceSource);
+        var authRequest = new VaultAuthRequest(client.getExecutor(), VaultRequest.get("test").build(), tickableInstantSource);
 
         var originalToken = cachingTokenProvider.apply(authRequest)
                 .toCompletableFuture().get();
@@ -52,54 +69,38 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         var cachedToken3 = cachingTokenProvider.apply(authRequest)
                 .toCompletableFuture().get();
 
-        assertThat(originalToken.isFromCache())
-                .isFalse();
-        assertThat(cachedToken1.isFromCache())
-                .isTrue();
-        assertThat(cachedToken2.isFromCache())
-                .isTrue();
-        assertThat(cachedToken3.isFromCache())
-                .isTrue();
-
-        assertThat(cachedToken1.getClientToken())
-                .isEqualTo(originalToken.getClientToken());
-        assertThat(cachedToken2.getClientToken())
-                .isEqualTo(originalToken.getClientToken());
-        assertThat(cachedToken3.getClientToken())
-                .isEqualTo(originalToken.getClientToken());
-
-        assertThat(cachedToken1.getCreated())
-                .isEqualTo(originalToken.getCreated());
-        assertThat(cachedToken2.getCreated())
-                .isEqualTo(originalToken.getCreated());
-        assertThat(cachedToken3.getCreated())
-                .isEqualTo(originalToken.getCreated());
-
-        assertThat(cachedToken1.getLeaseDuration())
-                .isEqualTo(originalToken.getLeaseDuration());
-        assertThat(cachedToken2.getLeaseDuration())
-                .isEqualTo(originalToken.getLeaseDuration());
-        assertThat(cachedToken3.getLeaseDuration())
-                .isEqualTo(originalToken.getLeaseDuration());
-
-        assertThat(cachedToken1.isRenewable())
-                .isEqualTo(originalToken.isRenewable());
-        assertThat(cachedToken2.isRenewable())
-                .isEqualTo(originalToken.isRenewable());
-        assertThat(cachedToken3.isRenewable())
-                .isEqualTo(originalToken.isRenewable());
+        assertThat(List.of(cachedToken1, cachedToken2, cachedToken3))
+                .allSatisfy(s -> {
+                    assertThat(s)
+                            .extracting(VaultToken::isFromCache, as(BOOLEAN))
+                            .isTrue();
+                    assertThat(s)
+                            .extracting(VaultToken::getClientToken)
+                            .isEqualTo(originalToken.getClientToken());
+                    assertThat(s)
+                            .extracting(VaultToken::getCreated)
+                            .isEqualTo(originalToken.getCreated());
+                    assertThat(s)
+                            .extracting(VaultToken::getLeaseDuration)
+                            .isEqualTo(originalToken.getLeaseDuration());
+                    assertThat(s)
+                            .extracting(VaultToken::isRenewable)
+                            .isEqualTo(originalToken.isRenewable());
+                });
     }
 
-    @Test
-    public void testExpiredTokensAreRequestedAgain(VaultClient client) throws Exception {
+    @ParameterizedTest
+    @ValueSource(strings = { "secret", "empty" })
+    public void testExpiredTokensAreRequestedAgain(String mount, VaultClient client) throws Exception {
 
         var clientToken = client.auth().token().create(null)
                 .thenApply(VaultAuthResult::getClientToken)
                 .toCompletableFuture().get();
-        var tokenProvider = spy(new VaultTokenProvider() {
+        //noinspection Convert2Lambda
+        var tokenProvider = spy((VaultTokenProvider) new VaultTokenProvider() {
             @Override
             public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
-                var token = VaultToken.from(clientToken, true, Duration.ofMinutes(1), tickableInstanceSource);
+                var token = VaultToken.from(clientToken, true, Duration.ofMinutes(1), null, tickableInstantSource);
                 return CompletableFuture.completedStage(token);
             }
         });
@@ -107,13 +108,13 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         var executor = spy(client.getExecutor());
 
         var testClient = client.configure()
-                .instantSource(tickableInstanceSource)
+                .instantSource(tickableInstantSource)
                 .executor(executor)
                 .tokenProvider(cachingTokenProvider)
                 .build();
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -126,10 +127,10 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         verify(cachingTokenProvider, times(0))
                 .extend(any(), any());
 
-        tickableInstanceSource.tick(Duration.ofSeconds(90));
+        tickableInstantSource.tick(Duration.ofSeconds(90));
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -143,17 +144,19 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                 .extend(any(), any());
     }
 
-    @Test
-    public void testExpiringTokensRenew(VaultClient client) throws Exception {
+    @ParameterizedTest
+    @ValueSource(strings = { "secret", "empty" })
+    public void testExpiringTokensRenew(String mount, VaultClient client) throws Exception {
 
         var token = client.auth().token().create(new VaultAuthTokenCreateTokenParams()
                 .setTtl(Duration.ofMinutes(1))
                 .setRenewable(true))
                 .thenApply(
-                        a -> VaultToken.from(a.getClientToken(), a.isRenewable(), a.getLeaseDuration(), tickableInstanceSource))
+                        a -> VaultToken.from(a.getClientToken(), a.isRenewable(), a.getLeaseDuration(), a.getNumUses(),
+                                tickableInstantSource))
                 .toCompletableFuture().get();
         //noinspection Convert2Lambda
-        var tokenProvider = spy(new VaultTokenProvider() {
+        var tokenProvider = spy((VaultTokenProvider) new VaultTokenProvider() {
             @Override
             public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                 return CompletableFuture.completedStage(token);
@@ -163,13 +166,13 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         var executor = spy(client.getExecutor());
 
         var testClient = client.configure()
-                .instantSource(tickableInstanceSource)
+                .instantSource(tickableInstantSource)
                 .executor(executor)
                 .tokenProvider(cachingTokenProvider)
                 .build();
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -182,10 +185,10 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         verify(cachingTokenProvider, times(0))
                 .extend(any(), any());
 
-        tickableInstanceSource.tick(Duration.ofSeconds(45));
+        tickableInstantSource.tick(Duration.ofSeconds(45));
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -199,18 +202,20 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                 .extend(any(), any());
     }
 
-    @Test
-    public void testExpiredNonRenewableTokensAreRequestedAgain(VaultClient client) throws Exception {
+    @ParameterizedTest
+    @ValueSource(strings = { "secret", "empty" })
+    public void testExpiredNonRenewableTokensAreRequestedAgain(String mount, VaultClient client) throws Exception {
 
         var clientToken = client.auth().token().create(new VaultAuthTokenCreateTokenParams()
                 .setTtl(Duration.ofMinutes(5))
                 .setRenewable(false))
                 .thenApply(VaultAuthResult::getClientToken)
                 .toCompletableFuture().get();
+        //noinspection Convert2Lambda
         var tokenProvider = spy(new VaultTokenProvider() {
             @Override
             public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
-                var token = VaultToken.from(clientToken, false, Duration.ofMinutes(1), tickableInstanceSource);
+                var token = VaultToken.from(clientToken, false, Duration.ofMinutes(1), null, tickableInstantSource);
                 return CompletableFuture.completedStage(token);
             }
         });
@@ -218,13 +223,13 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         var executor = spy(client.getExecutor());
 
         var testClient = client.configure()
-                .instantSource(tickableInstanceSource)
+                .instantSource(tickableInstantSource)
                 .executor(executor)
                 .tokenProvider(cachingTokenProvider)
                 .build();
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -237,10 +242,10 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         verify(cachingTokenProvider, times(0))
                 .extend(any(), any());
 
-        tickableInstanceSource.tick(Duration.ofSeconds(90));
+        tickableInstantSource.tick(Duration.ofSeconds(90));
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -254,8 +259,9 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                 .extend(any(), any());
     }
 
-    @Test
-    public void testExpiringTokensFailingToRenewAreRequestedAgain(VaultClient client) throws Exception {
+    @ParameterizedTest
+    @ValueSource(strings = { "secret", "empty" })
+    public void testExpiringTokensFailingToRenewAreRequestedAgain(String mount, VaultClient client) throws Exception {
 
         var tokenProvider = spy(new VaultTokenProvider() {
             @Override
@@ -264,20 +270,21 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                         .setTtl(Duration.ofSeconds(30))
                         .setRenewable(false))
                         .thenApply(
-                                a -> VaultToken.from(a.getClientToken(), true, Duration.ofMinutes(1), tickableInstanceSource));
+                                a -> VaultToken.from(a.getClientToken(), true, Duration.ofMinutes(1), a.getNumUses(),
+                                        tickableInstantSource));
             }
         });
         var cachingTokenProvider = spy(new VaultCachingTokenProvider(tokenProvider, Duration.ofSeconds(30)));
         var executor = spy(client.getExecutor());
 
         var testClient = client.configure()
-                .instantSource(tickableInstanceSource)
+                .instantSource(tickableInstantSource)
                 .executor(executor)
                 .tokenProvider(cachingTokenProvider)
                 .build();
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -290,10 +297,10 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         verify(cachingTokenProvider, times(0))
                 .extend(any(), any());
 
-        tickableInstanceSource.tick(Duration.ofSeconds(45));
+        tickableInstantSource.tick(Duration.ofSeconds(45));
 
         for (int i = 0; i < 10; i++) {
-            testClient.secrets().kv2().listSecrets("/")
+            testClient.secrets().kv2(mount).listSecrets("/")
                     .toCompletableFuture().get();
         }
 
@@ -306,4 +313,115 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
         verify(cachingTokenProvider, times(1))
                 .extend(any(), any());
     }
+
+    @ParameterizedTest
+    @ValueSource(strings = { "secret", "empty" })
+    public void testLimitedUseTokensAreRequestedAgain(String mount, VaultClient client) throws Exception {
+
+        var tokenProvider = spy(new VaultTokenProvider() {
+            @Override
+            public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
+                return client.auth().token().create(new VaultAuthTokenCreateTokenParams()
+                        .setTtl(Duration.ofMinutes(5))
+                        .setRenewable(false)
+                        .setNumUses(2))
+                        .thenApply(
+                                a -> VaultToken.from(a.getClientToken(), a.isRenewable(), a.getLeaseDuration(), a.getNumUses(),
+                                        tickableInstantSource));
+            }
+        });
+        var cachingTokenProvider = spy(new VaultCachingTokenProvider(tokenProvider, Duration.ofSeconds(30)));
+        var executor = spy(client.getExecutor());
+
+        var testClient = client.configure()
+                .instantSource(tickableInstantSource)
+                .executor(executor)
+                .tokenProvider(cachingTokenProvider)
+                .build();
+
+        for (int i = 0; i < 10; i++) {
+            testClient.secrets().kv2(mount).listSecrets("/")
+                    .toCompletableFuture().get();
+        }
+
+        verify(executor, times(10))
+                .execute(any());
+        verify(tokenProvider, times(5))
+                .apply(any());
+        verify(cachingTokenProvider, times(5))
+                .request(any());
+        verify(cachingTokenProvider, times(0))
+                .extend(any(), any());
+
+        tickableInstantSource.tick(Duration.ofSeconds(90));
+
+        for (int i = 0; i < 10; i++) {
+            testClient.secrets().kv2(mount).listSecrets("/")
+                    .toCompletableFuture().get();
+        }
+
+        verify(executor, times(20))
+                .execute(any());
+        verify(tokenProvider, times(10))
+                .apply(any());
+        verify(cachingTokenProvider, times(10))
+                .request(any());
+        verify(cachingTokenProvider, times(0))
+                .extend(any(), any());
+    }
+
+    @ParameterizedTest
+    @ValueSource(strings = { "secret", "empty" })
+    public void testLimitedUseTokenExhaustion(String mount, VaultClient client) throws Exception {
+
+        var tokenProvider = spy(new VaultTokenProvider() {
+            @Override
+            public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
+                return client.auth().token().create(new VaultAuthTokenCreateTokenParams()
+                        .setTtl(Duration.ofMinutes(5))
+                        .setRenewable(false)
+                        .setNumUses(2))
+                        .thenApply(
+                                a -> VaultToken.from(a.getClientToken(), a.isRenewable(), a.getLeaseDuration(), a.getNumUses(),
+                                        tickableInstantSource));
+            }
+        });
+        var cachingTokenProvider = spy(new VaultCachingTokenProvider(tokenProvider, Duration.ofSeconds(30)));
+        var executor = spy(client.getExecutor());
+
+        var useStealingTokenProvider = new VaultTokenProvider() {
+            @Override
+            public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
+                return cachingTokenProvider.apply(authRequest)
+                        .thenApply(t -> {
+                            if (t.getAllowedUsesRemaining() == 1) {
+                                Logger.getLogger(VaultCachingTokenProviderTest.class.getName())
+                                        .fine("### Stealing last token usage ###");
+                                t.getClientTokenForUsage();
+                            }
+                            return t;
+                        });
+            }
+        };
+
+        var testClient = client.configure()
+                .instantSource(tickableInstantSource)
+                .executor(executor)
+                .tokenProvider(useStealingTokenProvider)
+                .build();
+
+        for (int i = 0; i < 10; i++) {
+            testClient.secrets().kv2(mount).listSecrets("/")
+                    .toCompletableFuture().get();
+        }
+
+        verify(executor, times(10))
+                .execute(any());
+        verify(tokenProvider, times(10))
+                .apply(any());
+        verify(cachingTokenProvider, times(10))
+                .request(any());
+        verify(cachingTokenProvider, times(0))
+                .extend(any(), any());
+    }
 }
diff --git a/client/src/test/java/io/quarkus/vault/client/VaultClientTest.java b/client/src/test/java/io/quarkus/vault/client/VaultClientRetryTest.java
similarity index 86%
rename from client/src/test/java/io/quarkus/vault/client/VaultClientTest.java
rename to client/src/test/java/io/quarkus/vault/client/VaultClientRetryTest.java
index 6f80092d..cbf6b3c5 100644
--- a/client/src/test/java/io/quarkus/vault/client/VaultClientTest.java
+++ b/client/src/test/java/io/quarkus/vault/client/VaultClientRetryTest.java
@@ -6,6 +6,7 @@
 
 import java.net.URL;
 import java.time.Duration;
+import java.time.InstantSource;
 import java.util.List;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.CompletionStage;
@@ -21,10 +22,10 @@
 import io.quarkus.vault.client.common.VaultResponse;
 import io.quarkus.vault.client.logging.LogConfidentialityLevel;
 
-public class VaultClientTest {
+public class VaultClientRetryTest {
 
     @Test
-    public void testPermissionDeniedInvalidatesCacheAndRetriesOnce() {
+    public void testPermissionDeniedInvalidatesCacheAndRetriesOnce() throws Exception {
 
         var executor = spy(new VaultRequestExecutor() {
             @Override
@@ -34,12 +35,19 @@ public <T> CompletionStage<VaultResponse<T>> execute(VaultRequest<T> request) {
             }
         });
         var tokenProvider = spy(new VaultTokenProvider() {
+            VaultToken token = VaultToken.from("baaaaad", true, Duration.ofMinutes(1), null, InstantSource.system()).cached();
+
             @Override
             public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
-                var token = VaultToken.from("test", true, Duration.ofMinutes(1), authRequest.getInstantSource())
-                        .cached();
                 return CompletableFuture.completedStage(token);
             }
+
+            @Override
+            public void invalidateCache() {
+                // simulate token refresh
+                token = VaultToken.from(token.getClientToken(), token.isRenewable(), token.getLeaseDuration(),
+                        token.getAllowedUsesRemaining(), token.getInstantSource());
+            }
         });
 
         var client = VaultClient.builder()
@@ -53,7 +61,7 @@ public CompletionStage<VaultToken> apply(VaultAuthRequest authRequest) {
                 .isInstanceOf(VaultClientException.class)
                 .hasMessageContaining("permission denied");
 
-        verify(tokenProvider, times(1))
+        verify(tokenProvider, times(2))
                 .apply(any());
         verify(executor, times(2))
                 .execute(any());
diff --git a/client/src/test/java/io/quarkus/vault/client/test/TickableInstanceSource.java b/client/src/test/java/io/quarkus/vault/client/test/TickableInstantSource.java
similarity index 75%
rename from client/src/test/java/io/quarkus/vault/client/test/TickableInstanceSource.java
rename to client/src/test/java/io/quarkus/vault/client/test/TickableInstantSource.java
index 2d8d3dd4..59e96011 100644
--- a/client/src/test/java/io/quarkus/vault/client/test/TickableInstanceSource.java
+++ b/client/src/test/java/io/quarkus/vault/client/test/TickableInstantSource.java
@@ -4,11 +4,11 @@
 import java.time.Instant;
 import java.time.InstantSource;
 
-public class TickableInstanceSource implements InstantSource {
+public class TickableInstantSource implements InstantSource {
 
     private Instant instant;
 
-    public TickableInstanceSource(Instant instant) {
+    public TickableInstantSource(Instant instant) {
         this.instant = instant;
     }
 
diff --git a/client/src/test/java/io/quarkus/vault/client/test/VaultClientTestExtension.java b/client/src/test/java/io/quarkus/vault/client/test/VaultClientTestExtension.java
index 13ee3f5f..584ddfd2 100644
--- a/client/src/test/java/io/quarkus/vault/client/test/VaultClientTestExtension.java
+++ b/client/src/test/java/io/quarkus/vault/client/test/VaultClientTestExtension.java
@@ -3,10 +3,13 @@
 import static org.testcontainers.containers.BindMode.READ_ONLY;
 
 import java.io.IOException;
+import java.io.InputStream;
 import java.net.URL;
 import java.net.http.HttpClient;
 import java.security.MessageDigest;
 import java.util.HexFormat;
+import java.util.Optional;
+import java.util.logging.LogManager;
 
 import org.junit.jupiter.api.extension.*;
 import org.testcontainers.containers.Network;
@@ -63,6 +66,18 @@ public static String getRandomString(int length) {
     @Override
     public void beforeAll(ExtensionContext extensionContext) throws Exception {
 
+        Optional.ofNullable(System.getenv("TRACE_ENABLED"))
+                .ifPresent(logLevel -> {
+                    try {
+                        try (InputStream traceLogPropStream = Thread.currentThread().getContextClassLoader()
+                                .getResourceAsStream("trace-log.properties")) {
+                            LogManager.getLogManager().readConfiguration(traceLogPropStream);
+                        }
+                    } catch (IOException e) {
+                        throw new RuntimeException("Failed to load trace-log.properties", e);
+                    }
+                });
+
         var annotation = extensionContext.getRequiredTestClass().getAnnotation(VaultClientTest.class);
 
         vaultContainer
diff --git a/client/src/test/resources/trace-log.properties b/client/src/test/resources/trace-log.properties
new file mode 100644
index 00000000..16977acb
--- /dev/null
+++ b/client/src/test/resources/trace-log.properties
@@ -0,0 +1,5 @@
+handlers = java.util.logging.ConsoleHandler
+.level= INFO
+io.quarkus.vault.level = FINEST
+java.util.logging.ConsoleHandler.level = FINEST
+java.util.logging.SimpleFormatter.format=%4$s[%1$tM:%1$tS.%1$tN] %2$s:%n    %5$s%n
diff --git a/docs/modules/ROOT/pages/includes/attributes.adoc b/docs/modules/ROOT/pages/includes/attributes.adoc
index e39b74f8..e9853832 100644
--- a/docs/modules/ROOT/pages/includes/attributes.adoc
+++ b/docs/modules/ROOT/pages/includes/attributes.adoc
@@ -1,4 +1,4 @@
-:quarkus-version: 3.19.0
+:quarkus-version: 3.19.3
 :quarkus-vault-version: 4.2.1
 :maven-version: 3.8.1+
 


Reviews:
user: login: vsevel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-18T14:50:44Z
user: login: kdubb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T13:26:03Z
user: login: vsevel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T15:48:19Z
user: login: vsevel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T15:55:03Z
user: login: vsevel
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T15:56:09Z

Review comments:
diff_hunk: @@ -5,42 +5,52 @@
 import java.time.Duration;
 import java.time.Instant;
 import java.time.InstantSource;
+import java.util.concurrent.atomic.AtomicInteger;
 
+import io.quarkus.vault.client.VaultException;
 import io.quarkus.vault.client.logging.LogConfidentialityLevel;
 
 public class VaultToken extends VaultTimeLimited {
 
     private final String clientToken;
     private final boolean fromCache;
+    private final AtomicInteger allowedUsesRemaining;
 
-    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, InstantSource instantSource) {
-        return new VaultToken(clientToken, renewable, leaseDuration, false, instantSource);
+    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return new VaultToken(clientToken, renewable, leaseDuration, allowedUsesRemaining, false, instantSource);
     }
 
-    public static VaultToken renewable(String clientToken, Duration leaseDuration, InstantSource instantSource) {
-        return from(clientToken, true, leaseDuration, instantSource);
+    public static VaultToken renewable(String clientToken, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return from(clientToken, true, leaseDuration, allowedUsesRemaining, instantSource);
     }
 
     public static VaultToken neverExpires(String clientToken, InstantSource instantSource) {
-        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), instantSource);
+        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), null, instantSource);
     }
 
-    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, boolean fromCache,
+    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            boolean fromCache,
             InstantSource instantSource) {
         super(renewable, leaseDuration, instantSource);
         this.clientToken = clientToken;
         this.fromCache = fromCache;
+        this.allowedUsesRemaining = new AtomicInteger(
+                allowedUsesRemaining != null && allowedUsesRemaining > 0 ? allowedUsesRemaining : Integer.MAX_VALUE);
path: client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java
user: login: vsevel
body: why this condition `allowedUsesRemaining > 0` ?
if `allowedUsesRemaining <= 0`, then the remaining allowed uses should really be `MAX_VALUE`?
created_at: 2025-02-18T14:50:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,42 +5,52 @@
 import java.time.Duration;
 import java.time.Instant;
 import java.time.InstantSource;
+import java.util.concurrent.atomic.AtomicInteger;
 
+import io.quarkus.vault.client.VaultException;
 import io.quarkus.vault.client.logging.LogConfidentialityLevel;
 
 public class VaultToken extends VaultTimeLimited {
 
     private final String clientToken;
     private final boolean fromCache;
+    private final AtomicInteger allowedUsesRemaining;
 
-    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, InstantSource instantSource) {
-        return new VaultToken(clientToken, renewable, leaseDuration, false, instantSource);
+    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return new VaultToken(clientToken, renewable, leaseDuration, allowedUsesRemaining, false, instantSource);
     }
 
-    public static VaultToken renewable(String clientToken, Duration leaseDuration, InstantSource instantSource) {
-        return from(clientToken, true, leaseDuration, instantSource);
+    public static VaultToken renewable(String clientToken, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return from(clientToken, true, leaseDuration, allowedUsesRemaining, instantSource);
     }
 
     public static VaultToken neverExpires(String clientToken, InstantSource instantSource) {
-        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), instantSource);
+        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), null, instantSource);
     }
 
-    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, boolean fromCache,
+    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            boolean fromCache,
             InstantSource instantSource) {
         super(renewable, leaseDuration, instantSource);
         this.clientToken = clientToken;
         this.fromCache = fromCache;
+        this.allowedUsesRemaining = new AtomicInteger(
+                allowedUsesRemaining != null && allowedUsesRemaining > 0 ? allowedUsesRemaining : Integer.MAX_VALUE);
path: client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java
user: login: kdubb
body: The value is either `0` (which is unlimited) or the number of allowed uses, I believe the null check was just me being pedantic.

In either case we have to map `0`/`null` to something or we'd have to check `allowedUsesRemaining != null` before every access. This seemed like the cleaner method. 
created_at: 2025-04-07T13:26:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,42 +5,52 @@
 import java.time.Duration;
 import java.time.Instant;
 import java.time.InstantSource;
+import java.util.concurrent.atomic.AtomicInteger;
 
+import io.quarkus.vault.client.VaultException;
 import io.quarkus.vault.client.logging.LogConfidentialityLevel;
 
 public class VaultToken extends VaultTimeLimited {
 
     private final String clientToken;
     private final boolean fromCache;
+    private final AtomicInteger allowedUsesRemaining;
 
-    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, InstantSource instantSource) {
-        return new VaultToken(clientToken, renewable, leaseDuration, false, instantSource);
+    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return new VaultToken(clientToken, renewable, leaseDuration, allowedUsesRemaining, false, instantSource);
     }
 
-    public static VaultToken renewable(String clientToken, Duration leaseDuration, InstantSource instantSource) {
-        return from(clientToken, true, leaseDuration, instantSource);
+    public static VaultToken renewable(String clientToken, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return from(clientToken, true, leaseDuration, allowedUsesRemaining, instantSource);
     }
 
     public static VaultToken neverExpires(String clientToken, InstantSource instantSource) {
-        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), instantSource);
+        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), null, instantSource);
     }
 
-    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, boolean fromCache,
+    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            boolean fromCache,
             InstantSource instantSource) {
         super(renewable, leaseDuration, instantSource);
         this.clientToken = clientToken;
         this.fromCache = fromCache;
+        this.allowedUsesRemaining = new AtomicInteger(
+                allowedUsesRemaining != null && allowedUsesRemaining > 0 ? allowedUsesRemaining : Integer.MAX_VALUE);
path: client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java
user: login: vsevel
body: I can see it now in https://developer.hashicorp.com/vault/api-docs/auth/token#num_uses
seemed strange that 0 uses means unlimited. 

created_at: 2025-04-08T15:48:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -5,42 +5,52 @@
 import java.time.Duration;
 import java.time.Instant;
 import java.time.InstantSource;
+import java.util.concurrent.atomic.AtomicInteger;
 
+import io.quarkus.vault.client.VaultException;
 import io.quarkus.vault.client.logging.LogConfidentialityLevel;
 
 public class VaultToken extends VaultTimeLimited {
 
     private final String clientToken;
     private final boolean fromCache;
+    private final AtomicInteger allowedUsesRemaining;
 
-    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, InstantSource instantSource) {
-        return new VaultToken(clientToken, renewable, leaseDuration, false, instantSource);
+    public static VaultToken from(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return new VaultToken(clientToken, renewable, leaseDuration, allowedUsesRemaining, false, instantSource);
     }
 
-    public static VaultToken renewable(String clientToken, Duration leaseDuration, InstantSource instantSource) {
-        return from(clientToken, true, leaseDuration, instantSource);
+    public static VaultToken renewable(String clientToken, Duration leaseDuration, Integer allowedUsesRemaining,
+            InstantSource instantSource) {
+        return from(clientToken, true, leaseDuration, allowedUsesRemaining, instantSource);
     }
 
     public static VaultToken neverExpires(String clientToken, InstantSource instantSource) {
-        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), instantSource);
+        return from(clientToken, false, Duration.ofSeconds(Long.MAX_VALUE), null, instantSource);
     }
 
-    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, boolean fromCache,
+    public VaultToken(String clientToken, boolean renewable, Duration leaseDuration, Integer allowedUsesRemaining,
+            boolean fromCache,
             InstantSource instantSource) {
         super(renewable, leaseDuration, instantSource);
         this.clientToken = clientToken;
         this.fromCache = fromCache;
+        this.allowedUsesRemaining = new AtomicInteger(
+                allowedUsesRemaining != null && allowedUsesRemaining > 0 ? allowedUsesRemaining : Integer.MAX_VALUE);
path: client/src/main/java/io/quarkus/vault/client/auth/VaultToken.java
user: login: vsevel
body: I looked at the structure . it is an `int64`
https://github.com/hashicorp/vault/blob/main/sdk/plugin/pb/backend.proto#L204
it may have been more appropriate to treat it as a long (like many other fields).
probably not worth the change.

created_at: 2025-04-08T15:55:03Z
author_association: CONTRIBUTOR

PR comments:
user: login: vsevel
created_at: 2025-01-27T08:17:27Z
body: ideally we would want decrementing the token close to where it is actually being used, so in the real `VaultRequestExecutor`, to make sure it is actually called exactly once. `VaultClient` is an executor, but so are `JDKVaultHttpClient`, `VertxVaultHttpClient`. are we sure we always go through `VaultClient` first, then to one of the concrete classes? it seems so. 
for my own comprehension what are you supposed to do when you exhausted the number of uses? you can still renew it?
so when we call `getClientTokenForUsage` and it fails with a `VaultException` do we auto renew it? 
I see the overload on `isExpired()`, so the sequence what should happen is that when we reaches 0, we should renew the token. correct?
I suppose the only pitfall would be if there was 1 use left, with 2 concurrent calls. they would both decrement the counter, after they have called `isExpired()`, then fail when making the real call. is that correct?
if true, then this approach would not be reliable in highly concurrent applications.
a solution then would be that once `isExpired()` has returned `false`, we can't be rejected because of the num uses.
but that might not be easily implementable?


author_association: CONTRIBUTOR
user: login: kdubb
created_at: 2025-01-28T02:47:40Z
body: > VaultClient is an executor, but so are JDKVaultHttpClient, VertxVaultHttpClient. are we sure we always go through VaultClient first, then to one of the concrete classes? it seems so.
 
Yes all roads go through `VaultClient`, it basically wraps another executor that is the client implementation. The only calls to `getClientToken` were the one in `VaultClient` and one in `VaultCachingTokenProvider`. I added `getClientTokenForUsage` only to be very clear about what it's doing.

> I suppose the only pitfall would be if there was 1 use left, with 2 concurrent calls. they would both decrement the counter, after they have called isExpired(), then fail when making the real call. is that correct?

This is a great point. I'll look into solving this problem and testing it.
author_association: CONTRIBUTOR
user: login: alexjaravete
created_at: 2025-01-28T09:57:55Z
body: @kdubb Thanks for taking a look at it and creating a fix. I will try to run it locally in the next few days and let you know if it works.
author_association: NONE
user: login: alexjaravete
created_at: 2025-01-30T20:31:14Z
body: I did a small test in our system with the new fix, and it seems to work as expected. The potential pitfall mentioned by @vsevel could still be a problem, but I was not able to test this in my limited test run.
And thank you @kdubb for tackling this issue.
author_association: NONE
user: login: vsevel
created_at: 2025-02-05T14:54:35Z
body: @kdubb I would like to do a release for the next quarkus version.
do you think this could go in?
author_association: CONTRIBUTOR
user: login: vsevel
created_at: 2025-02-10T11:49:23Z
body: @kdubb have you been able to make any progress? worst case scenario, can the current non perfect fix be considered as an improvement, and we improve on the concurrency later?
author_association: CONTRIBUTOR
user: login: vsevel
created_at: 2025-02-14T17:31:40Z
body: @kdubb I would like to make a release for `3.19`. according to you, we know this PR is not perfect (sensitive to race condition), but would you say that is an improvement anyhow, with no regression?
author_association: CONTRIBUTOR
user: login: vsevel
created_at: 2025-02-18T15:30:10Z
body: I am planning a vault release end of this week with `3.19`. it would be nice to finalize this PR
author_association: CONTRIBUTOR
user: login: kdubb
created_at: 2025-03-14T15:41:00Z
body: @vsevel Apologies for the quietness been slammed with other things. I do think this an improvement either way because limited use tokens are more likely to used in a managed way. By that I mean that you *probably* are not using a limited use token as the "general" authorization token. Maybe @alexjaravete can shed some light here.

In either case I'm looking into this a bit today to see what can be done to mitigate this issue.
author_association: CONTRIBUTOR
user: login: kdubb
created_at: 2025-03-14T23:08:55Z
body: @vsevel @alexjaravete I did a deep dive and added some extensive tests and I think the current implementation mitigates the race condition well.

### Token Acquisition/Use
The two step process token acquisition and then application naturally creates a race condition regardless of caching. I don't really see how we could change this it's just worth pointing out that the client allows _a lot_ of flexibility in how a token is acquired (even possibly creating it on the fly). Given this, we'll always have a race condition to deal with.

### Execution is a "use"
This implementation treats any attempted execution of a request as a "use" of the token. Generally, that means a _sent_ HTTP request but considering the allowed flexibility in `VaultClient` (it just passes the request to an executor) there could potentially be something else fail before ever sending a request. Even in the common case of passing the request directly to an HTTP executor, a network failure would still incorrectly consume a token usage.

I don't see this as a problem. It would only mean that a new token is being requested while the current token still has a couple uses left. Maybe somebody else has a different scenario that would mean this is not an allowable state.

### Over "use" is properly handled
Under normal operation tokens are cached and prior to usage their expiry and usage count are  checked before being applied. If either case fails, an "extension" or "refresh" is attempted. In the race condition case, a `VaultTokenException` (newly added) is thrown and `VaultClient` handles this by retrying the request, with a call to the token provider first. Currently, the default retry count is `2` making the max number of attempts `3` for any request; although this is now configurable in `VaultClient`.

In some bizarre case of multiple clients, that are not sharing a token provider, using a limited token, the server returns the error and the client initiates a retry as expected.

### Other Considerations

Token over "use" could potentially be enhanced. For example, we could "lease" a token `use` to requests and only permanently decrement the token's remaining use count once a request has been returned from the Vault instance. Part of request execution would have to be waiting for the lease of a `use`. This would essentially mean some kind of serialization *might* happen as tokens near usage exhaustion; which sounds correct if we believe sending a possibly overused token should always be avoided.

Alternatively, we could yank the checks for over use altogether and let the server tell us when to refresh (via a 403 response, which should now be appropriately retried).

### Wrap Up

All the above suggests to me that the current implementation is "good". If a token over use is detected by the client, or server, it is retried up to 2 times. A high volume client can increase the `maxRetries` in `VaultClient` if they expect a lot of requests when using a limited use token.

We could up the default # of retries as well because they only apply to token & authentication failures. Separately (in another PR) we should probably add a general "retry" count for things like network failures.

So.. unless I'm missing something... 👍👍
author_association: CONTRIBUTOR
user: login: vsevel
created_at: 2025-03-31T13:47:01Z
body: > I don't really see how we could change this it's just worth pointing out that the client allows a lot of flexibility in how a token is acquired (even possibly creating it on the fly). Given this, we'll always have a race condition to deal with.

I agree. the only way would be to `synchronize` code between checking if there are use left, and making the call, which can't be done obviously. it is not specific to the vault extension, any multithreaded client checking the token then making the call without atomicity would have the issue.

when I though about this issue, the two only possible partial solutions I thought of were:
- do not wait for num tokens = 0 before refreshing. make it a config parameter, and applications could adjust how often they refresh depending on the parallelism.
- or catch the exception, and offer to refresh and retry on certain operations, which I believe is exactly what you are describing here: _In the race condition case, a VaultTokenException (newly added) is thrown and VaultClient handles this by retrying the request, with a call to the token provider first_

> This implementation treats any attempted execution of a request as a "use" of the token.

I like that. was this in the original implementation? this approach in itself would be almost sufficient.
the only drawback would be to try calling vault, fail (for a reason other than num uses), refresh token, and retry, and fail, and ... so you might generate a lot of token refresh, because of non related failures. 

>  only permanently decrement the token's remaining use count once a request has been returned from the Vault instance

not sure this is better. if you have 1000 calls to vault in parallel and num uses = 1, they will all think they can get through, and only one will succeed.

I like better assuming the call will go through, so decrement first, then call vault, plus catch the exception and refresh+retry.

> we could yank the checks for over use altogether and let the server tell us when to refresh (via a 403 response, which should now be appropriately retried).

if we were sure this would not cause a perf issue, I would say yes.
but if you have 1000 calls to vault, and all except one get rejected, we are going to have may be some refreshs, and 9999 retries.

as a conclusion, I like keeping a counter in the application, decrementing before use, catching the failure, then refreshing and retrying.

in terms of implementation, I wish we could make more atomic `token.hasAllowedUsesRemaining()` called from the token provider, then later `token.hasAllowedUsesRemaining()` and `allowedUsesRemaining().updateAndGet(...)` in `getClientTokenForUsage`.

ideally, we should:
- decrement
- if < 0 refresh
- make the call
author_association: CONTRIBUTOR
user: login: kdubb
created_at: 2025-04-01T19:17:47Z
body: @vsevel I switched around the `getClientTokenForUsage` to check the result of the `updateAndGet` instead of using a pre-check using `hasAllowedUsesRemaining`.

It's more correct and I believe that's what you were suggesting.

Should we merge now?
author_association: CONTRIBUTOR
user: login: vsevel
created_at: 2025-04-07T12:58:34Z
body: looks good.
there is this point left: https://github.com/quarkiverse/quarkus-vault/pull/360#discussion_r1959908645
should this be `this.allowedUsesRemaining = new AtomicInteger(allowedUsesRemaining != null ? allowedUsesRemaining : Integer.MAX_VALUE);`
do we really need the extra condition `allowedUsesRemaining > 0`?
is the value is non null, should not we trust it?
specifically if the value is 0, then with the current impl, we would get `MAX_VALUE` reuses. seems strange.
author_association: CONTRIBUTOR
user: login: kdubb
created_at: 2025-04-07T13:46:20Z
body: @vsevel I responded to the review inline.
author_association: CONTRIBUTOR
user: login: vsevel
created_at: 2025-04-08T15:55:46Z
body: @alexjaravete would you care to do one more test to make sure it works for you?
author_association: CONTRIBUTOR
user: login: alexjaravete
created_at: 2025-04-24T08:39:34Z
body: @vsevel Sorry I was not able to test it anymore, but I will test it with the new release. Thank You!
author_association: NONE
user: login: vsevel
created_at: 2025-04-24T11:59:51Z
body: I am going to try to release before the end of the week
author_association: CONTRIBUTOR