Repository: GoogleCloudPlatform/cloud-sql-jdbc-socket-factory (java)

Issue:
Unable to connect to CloudSQL databases after certificates expire, application can't recover
### Bug Description

Some of our applications are unable to connect to their CloudsQL database after running for a few days. They never recover until we restart them. We've enabled debug logging and suspect a problem with refreshing certificates.

We see that all applications continuously verify the certificate expiry:

```
[bolcom-pro-sre-health-538:europe-west4:sre-health] Now = 2024-07-30T09:04:39.959408416Z, Current client certificate expiration = 2024-07-30T10:04:14.941Z
```

At some point in time we see a refresh operation scheduled:

```
Jul 30, 2024 @ 11:04:39.553 DEBUG com.google.cloud.sql.core.RefreshAheadStrategy [bolcom-pro-sre-health-538:europe-west4:sre-health] Refresh Operation: Next operation scheduled at 2024-07-30T10:00:14Z.
```

Most applications then refresh the certificate at the scheduled time, but a few applications fail to do this and break as seen above.

In the logs below it looks like the refresh never happened at 10:00:14Z. Or at least that an old certificate is still lingering. Because at 12:05:17 the applications tries to use the certificate that expired 2 hours ago.


```
Jul 30, 2024 @ 12:05:17.156 DEBUG com.google.cloud.sql.core.RefreshAheadStrategy [bolcom-pro-sre-health-538:europe-west4:sre-health] Client certificate has expired. Starting next refresh operation immediately.
Jul 30, 2024 @ 12:05:17.156 DEBUG com.google.cloud.sql.core.RefreshAheadStrategy [bolcom-pro-sre-health-538:europe-west4:sre-health] Now = 2024-07-30T10:05:17.156356729Z, Current client certificate expiration = 2024-07-30T10:04:14.941Z
```
It looks like refreshing the certificate fails, or at least the application is still using the wrong one. As after connections to the database fail.

```
Jul 30, 2024 @ 12:05:17.157 DEBUG com.google.cloud.sql.core.Connector [10.25.8.40] Connecting to instance.
Jul 30, 2024 @ 12:05:17.172 DEBUG com.google.cloud.sql.core.Connector [10.25.8.40] Connected to instance successfully.
Jul 30, 2024 @ 12:05:17.174 DEBUG com.zaxxer.hikari.pool.HikariPool HikariPool-1 - Cannot acquire connection from data source org.postgresql.util.PSQLException: The connection attempt failed. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:358) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:273) at org.postgresql.Driver.makeConnection(Driver.java:446) at org.postgresql.Driver.connect(Driver.java:298) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:137) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:360) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:202) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:461) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:724) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:703) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365) at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1509) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1480) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:162) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:129) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:74) at org.postgresql.core.PGStream.receiveChar(PGStream.java:467) at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:683) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:207) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:262) ... 14 common frames elided
Jul 30, 2024 @ 12:05:17.174 DEBUG com.zaxxer.hikari.pool.PoolBase HikariPool-1 - Failed to create/setup connection: The connection attempt failed.
```





We haven't been able to identify what is different between applications that work and fail. In some cases even copies of applications (like one for testing and one for production) behave differently. With one working flawlessly for months, while the other one fails every few days. One thing I have noticed is that these applications all have a relatively low load and can idle for hours before a new database interaction happens. But I'm not sure if that is a clue or just an unrelated factor. 

We are currently using the following library versions:
- spring-cloud-gcp 5.4.1
- spring-boot 3.3.1
- HikariCP 5.1.0
- com.google.cloud.sql:jdbc-socket-factory-core:jar:1.18.0

We did not observe the issue when using:
- spring-cloud-gcp 4.9.4
- spring-boot 3.1.9
- HikariCP 5.0.1
- com.google.cloud.sql:jdbc-socket-factory-core:jar:1.14.1

We started seeing the issue when using: 
- spring-cloud-gcp 5.4.0
- spring-boot 3.2.6
- HikariCP 5.0.1
- com.google.cloud.sql:jdbc-socket-factory-core:jar:1.18.0

I haven't been able to narrow it down further yet. 

We found the following issue: https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory/issues/1314 which looks similar, but was fixed in `jdbc-socket-factory-core:jar:1.13.1`.  We are currently using 1.18.0.

### Example code (or command)

_No response_

### Stacktrace

```bash
om.zaxxer.hikari.pool.HikariPool HikariPool-1 - Cannot acquire connection from data source org.postgresql.util.PSQLException: The connection attempt failed. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:358) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:273) at org.postgresql.Driver.makeConnection(Driver.java:446) at org.postgresql.Driver.connect(Driver.java:298) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:137) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:360) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:202) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:461) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:724) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:703) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365) at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1509) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1480) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:162) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:129) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:114) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:74) at org.postgresql.core.PGStream.receiveChar(PGStream.java:467) at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:683) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:207) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:262) ... 14 common frames elided
Jul 30, 2024 @ 12:05:17.174 DEBUG com.zaxxer.hikari.pool.PoolBase HikariPool-1 - Failed to create/setup connection: The connection attempt failed.
```


### Steps to reproduce?

Unfortunately I don't have a good reproduction path. The issue seems to be consistently happening with only a few of our applications. Only happens after working well for a couple of days. 

### Environment

1. OS: Debian 12, running on Google Kubernetes Engine
2. JRE: Azul Zulu 21
3. Cloud SQL Java Socket Factory version: 1.18.0


### Additional Details

_No response_

Pull request description:
fix: Lazy refresh should refresh tokens 4 minutes before expiration.
Added a 4 minute buffer to refreshing tokens and certificates to avoid creating race condition that would allow the connector to create an ephemeral certificate with an expired auth token. 

Now, IAM auth tokens are now refreshed 4 minutes before they token expire. Also, the Lazy Refresh Strategy will refresh the client certificate 4 minutes before the expiration of the certificate and the IAM auth token. 

This should mitigate some of the strange certificate expiration errors commonly found in Cloud Run, see: #2059 

Patch:
diff --git a/core/src/main/java/com/google/cloud/sql/core/DefaultAccessTokenSupplier.java b/core/src/main/java/com/google/cloud/sql/core/DefaultAccessTokenSupplier.java
index d7c7260a..5a59e287 100644
--- a/core/src/main/java/com/google/cloud/sql/core/DefaultAccessTokenSupplier.java
+++ b/core/src/main/java/com/google/cloud/sql/core/DefaultAccessTokenSupplier.java
@@ -16,6 +16,8 @@
 
 package com.google.cloud.sql.core;
 
+import static com.google.cloud.sql.core.RefreshCalculator.DEFAULT_REFRESH_BUFFER;
+
 import com.google.auth.oauth2.AccessToken;
 import com.google.auth.oauth2.GoogleCredentials;
 import com.google.cloud.sql.AuthType;
@@ -76,12 +78,12 @@ class DefaultAccessTokenSupplier implements AccessTokenSupplier {
             () -> {
               final GoogleCredentials credentials = credentialFactory.getCredentials();
               try {
-                credentials.refreshIfExpired();
+                refreshIfRequired(credentials);
               } catch (IllegalStateException e) {
                 throw new IllegalStateException("Error refreshing credentials " + credentials, e);
               }
-              if (credentials.getAccessToken() == null
-                  || "".equals(credentials.getAccessToken().getTokenValue())) {
+
+              if (isAccessTokenEmpty(credentials)) {
 
                 String errorMessage = "Access Token has length of zero";
                 logger.debug(errorMessage);
@@ -97,10 +99,9 @@ class DefaultAccessTokenSupplier implements AccessTokenSupplier {
               // For some implementations of GoogleCredentials, particularly
               // ImpersonatedCredentials, down-scoped credentials are not
               // initialized with a token and need to be explicitly refreshed.
-              if (downscoped.getAccessToken() == null
-                  || "".equals(downscoped.getAccessToken().getTokenValue())) {
+              if (isAccessTokenEmpty(downscoped)) {
                 try {
-                  downscoped.refreshIfExpired();
+                  downscoped.refresh();
                 } catch (Exception e) {
                   throw new IllegalStateException(
                       "Error refreshing downscoped credentials " + credentials, e);
@@ -108,8 +109,7 @@ class DefaultAccessTokenSupplier implements AccessTokenSupplier {
 
                 // After attempting to refresh once, if the downscoped credentials do not have
                 // an access token after attempting to refresh, then throw an IllegalStateException
-                if (downscoped.getAccessToken() == null
-                    || "".equals(downscoped.getAccessToken().getTokenValue())) {
+                if (isAccessTokenEmpty(downscoped)) {
                   String errorMessage = "Downscoped access token has length of zero";
                   logger.debug(errorMessage);
 
@@ -135,6 +135,28 @@ class DefaultAccessTokenSupplier implements AccessTokenSupplier {
     }
   }
 
+  private static boolean isAccessTokenEmpty(GoogleCredentials credentials) {
+    return credentials.getAccessToken() == null
+        || "".equals(credentials.getAccessToken().getTokenValue());
+  }
+
+  private void refreshIfRequired(GoogleCredentials credentials) throws IOException {
+    // if the token does not exist, or if the token expires in less than 4 minutes, refresh it.
+    if (credentials.getAccessToken() == null) {
+      logger.debug("Current IAM AuthN Token is not set. Refreshing the token.");
+      credentials.refresh();
+    } else if (credentials.getAccessToken().getExpirationTime() != null
+        && credentials
+            .getAccessToken()
+            .getExpirationTime()
+            .toInstant()
+            .minus(DEFAULT_REFRESH_BUFFER)
+            .isBefore(Instant.now())) {
+      logger.debug("Current IAM AuthN Token expires in less than 4 minutes. Refreshing the token.");
+      credentials.refresh();
+    }
+  }
+
   private void validateAccessTokenExpiration(AccessToken accessToken) {
     Date expirationTimeDate = accessToken.getExpirationTime();
 
diff --git a/core/src/main/java/com/google/cloud/sql/core/DefaultConnectionInfoRepository.java b/core/src/main/java/com/google/cloud/sql/core/DefaultConnectionInfoRepository.java
index cb141848..2f3e92a9 100644
--- a/core/src/main/java/com/google/cloud/sql/core/DefaultConnectionInfoRepository.java
+++ b/core/src/main/java/com/google/cloud/sql/core/DefaultConnectionInfoRepository.java
@@ -46,6 +46,7 @@ import java.security.cert.CertificateFactory;
 import java.security.cert.X509Certificate;
 import java.time.Instant;
 import java.util.Arrays;
+import java.util.Base64;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -196,7 +197,15 @@ class DefaultConnectionInfoRepository implements ConnectionInfoRepository {
               .orElse(x509Certificate.getNotAfter().toInstant());
     }
 
-    logger.debug(String.format("[%s] INSTANCE DATA DONE", instanceName));
+    logger.debug(
+        "[{}] INSTANCE DATA DONE - Ephemeral cert id: {} cert expiration: {} token expiration: {}",
+        instanceName,
+        Base64.getEncoder().encodeToString(((X509Certificate) ephemeralCertificate).getSignature()),
+        token
+            .map(tok -> tok.getExpirationTime())
+            .filter(time -> time != null)
+            .map(time -> time.toInstant().toString())
+            .orElse("(none)"));
 
     return new ConnectionInfo(metadata, sslContext, expiration);
   }
diff --git a/core/src/main/java/com/google/cloud/sql/core/LazyRefreshConnectionInfoCache.java b/core/src/main/java/com/google/cloud/sql/core/LazyRefreshConnectionInfoCache.java
index 7b49f473..424c5024 100644
--- a/core/src/main/java/com/google/cloud/sql/core/LazyRefreshConnectionInfoCache.java
+++ b/core/src/main/java/com/google/cloud/sql/core/LazyRefreshConnectionInfoCache.java
@@ -16,6 +16,8 @@
 
 package com.google.cloud.sql.core;
 
+import static com.google.cloud.sql.core.RefreshCalculator.DEFAULT_REFRESH_BUFFER;
+
 import com.google.cloud.sql.CredentialFactory;
 import java.security.KeyPair;
 
@@ -54,7 +56,8 @@ class LazyRefreshConnectionInfoCache implements ConnectionInfoCache {
             config.getCloudSqlInstance(),
             () ->
                 connectionInfoRepository.getConnectionInfoSync(
-                    instanceName, accessTokenSupplier, config.getAuthType(), keyPair));
+                    instanceName, accessTokenSupplier, config.getAuthType(), keyPair),
+            DEFAULT_REFRESH_BUFFER);
   }
 
   @Override
diff --git a/core/src/main/java/com/google/cloud/sql/core/LazyRefreshStrategy.java b/core/src/main/java/com/google/cloud/sql/core/LazyRefreshStrategy.java
index bd8edc70..20852855 100644
--- a/core/src/main/java/com/google/cloud/sql/core/LazyRefreshStrategy.java
+++ b/core/src/main/java/com/google/cloud/sql/core/LazyRefreshStrategy.java
@@ -17,6 +17,7 @@
 package com.google.cloud.sql.core;
 
 import com.google.errorprone.annotations.concurrent.GuardedBy;
+import java.time.Duration;
 import java.time.Instant;
 import java.util.function.Supplier;
 import org.slf4j.Logger;
@@ -28,6 +29,7 @@ public class LazyRefreshStrategy implements RefreshStrategy {
 
   private final String name;
   private final Supplier<ConnectionInfo> refreshOperation;
+  private final Duration refreshBuffer;
 
   private final Object connectionInfoGuard = new Object();
 
@@ -38,9 +40,11 @@ public class LazyRefreshStrategy implements RefreshStrategy {
   private boolean closed;
 
   /** Creates a new LazyRefreshStrategy instance. */
-  public LazyRefreshStrategy(String name, Supplier<ConnectionInfo> refreshOperation) {
+  public LazyRefreshStrategy(
+      String name, Supplier<ConnectionInfo> refreshOperation, Duration refreshDuration) {
     this.name = name;
     this.refreshOperation = refreshOperation;
+    this.refreshBuffer = refreshDuration;
   }
 
   @Override
@@ -59,7 +63,7 @@ public class LazyRefreshStrategy implements RefreshStrategy {
                 name));
         fetchConnectionInfo();
       }
-      if (Instant.now().isAfter(connectionInfo.getExpiration())) {
+      if (Instant.now().isAfter(connectionInfo.getExpiration().minus(refreshBuffer))) {
         logger.debug(
             String.format(
                 "[%s] Lazy Refresh Operation: Client certificate has expired. Starting next "
diff --git a/core/src/main/java/com/google/cloud/sql/core/RefreshCalculator.java b/core/src/main/java/com/google/cloud/sql/core/RefreshCalculator.java
index de89c18a..6245f20a 100644
--- a/core/src/main/java/com/google/cloud/sql/core/RefreshCalculator.java
+++ b/core/src/main/java/com/google/cloud/sql/core/RefreshCalculator.java
@@ -28,7 +28,7 @@ class RefreshCalculator {
   // defaultRefreshBuffer is the minimum amount of time for which a
   // certificate must be valid to ensure the next refresh attempt has adequate
   // time to complete.
-  private static final Duration DEFAULT_REFRESH_BUFFER = Duration.ofMinutes(4);
+  static final Duration DEFAULT_REFRESH_BUFFER = Duration.ofMinutes(4);
 
   long calculateSecondsUntilNextRefresh(Instant now, Instant expiration) {
     Duration timeUntilExp = Duration.between(now, expiration);


Test patch:
diff --git a/core/src/test/java/com/google/cloud/sql/core/LazyRefreshStrategyTest.java b/core/src/test/java/com/google/cloud/sql/core/LazyRefreshStrategyTest.java
index c6310948..e8876f19 100644
--- a/core/src/test/java/com/google/cloud/sql/core/LazyRefreshStrategyTest.java
+++ b/core/src/test/java/com/google/cloud/sql/core/LazyRefreshStrategyTest.java
@@ -19,6 +19,7 @@ package com.google.cloud.sql.core;
 import static com.google.common.truth.Truth.assertThat;
 import static org.junit.Assert.assertThrows;
 
+import java.time.Duration;
 import java.time.Instant;
 import java.time.temporal.ChronoUnit;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -32,7 +33,9 @@ public class LazyRefreshStrategyTest {
     final ExampleData data = new ExampleData(Instant.now().plus(1, ChronoUnit.HOURS));
     LazyRefreshStrategy r =
         new LazyRefreshStrategy(
-            "LazyRefresherTest.testCloudSqlInstanceDataRetrievedSuccessfully", () -> data);
+            "LazyRefresherTest.testCloudSqlInstanceDataRetrievedSuccessfully",
+            () -> data,
+            Duration.ZERO);
     ConnectionInfo gotInfo = r.getConnectionInfo(TEST_TIMEOUT_MS);
     assertThat(gotInfo).isSameInstanceAs(data);
   }
@@ -44,7 +47,8 @@ public class LazyRefreshStrategyTest {
             "LazyRefresherTest.testInstanceFailsOnConnectionError",
             () -> {
               throw new RuntimeException("always fails");
-            });
+            },
+            Duration.ZERO);
 
     RuntimeException ex =
         assertThrows(RuntimeException.class, () -> r.getConnectionInfo(TEST_TIMEOUT_MS));
@@ -62,7 +66,8 @@ public class LazyRefreshStrategyTest {
             () -> {
               refreshCount.incrementAndGet();
               return data;
-            });
+            },
+            Duration.ZERO);
 
     r.getConnectionInfo(TEST_TIMEOUT_MS);
     assertThat(refreshCount.get()).isEqualTo(1);
@@ -96,24 +101,59 @@ public class LazyRefreshStrategyTest {
                 return initialData;
               }
               return data;
-            });
+            },
+            Duration.ZERO);
 
     // Get the first data that is about to expire
     ConnectionInfo d = r.getConnectionInfo(TEST_TIMEOUT_MS);
     assertThat(refreshCount.get()).isEqualTo(1);
     assertThat(d).isSameInstanceAs(initialData);
 
-    waitForExpiration(initialData);
+    waitForExpiration(initialData.getExpiration());
 
     assertThat(r.getConnectionInfo(TEST_TIMEOUT_MS)).isSameInstanceAs(data);
     assertThat(refreshCount.get()).isEqualTo(2);
   }
 
-  private static void waitForExpiration(ExampleData initialData) throws InterruptedException {
+  @Test
+  public void testCloudSqlRefreshesExpiredDataWithRefreshBuffer() throws Exception {
+    Duration buffer = Duration.ofSeconds(5);
+    ExampleData initialData = new ExampleData(Instant.now().plus(8, ChronoUnit.SECONDS));
+    ExampleData data = new ExampleData(Instant.now().plus(1, ChronoUnit.HOURS));
+
+    AtomicInteger refreshCount = new AtomicInteger();
+
+    LazyRefreshStrategy r =
+        new LazyRefreshStrategy(
+            "LazyRefresherTest.testCloudSqlRefreshesExpiredData",
+            () -> {
+              int c = refreshCount.getAndIncrement();
+              if (c == 0) {
+                return initialData;
+              }
+              return data;
+            },
+            buffer);
+
+    // Get the first data that is about to expire, but not yet expired
+    ConnectionInfo d = r.getConnectionInfo(TEST_TIMEOUT_MS);
+    assertThat(refreshCount.get()).isEqualTo(1);
+    assertThat(d).isSameInstanceAs(initialData);
+
+    // Wait 5 seconds. Now the initialData expires in 3 seconds, less than the buffer.
+    waitForExpiration(Instant.now().plus(buffer));
+
+    // Assert that it gets data instead of initialData.
+    assertThat(r.getConnectionInfo(TEST_TIMEOUT_MS)).isSameInstanceAs(data);
+    assertThat(refreshCount.get()).isEqualTo(2);
+  }
+
+  private static void waitForExpiration(Instant expiration) throws InterruptedException {
     // Wait for the instance to expire
-    while (!Instant.now().isAfter(initialData.getExpiration())) {
+    while (!Instant.now().isAfter(expiration)) {
       Thread.sleep(10);
     }
+
     // Sleep a few more ms to make sure that Instant.now() really is after expiration.
     // Fixes a date math race condition only present in Java 8.
     Thread.sleep(10);
@@ -135,7 +175,8 @@ public class LazyRefreshStrategyTest {
                 return initialData;
               }
               return data;
-            });
+            },
+            Duration.ZERO);
 
     // Get the first data that is about to expire
     ConnectionInfo d = r.getConnectionInfo(TEST_TIMEOUT_MS);
@@ -143,7 +184,7 @@ public class LazyRefreshStrategyTest {
     assertThat(d).isSameInstanceAs(initialData);
 
     // Wait for the instance to expire
-    waitForExpiration(initialData);
+    waitForExpiration(initialData.getExpiration());
 
     // Start multiple threads and request connection info
     Thread t1 = new Thread(() -> r.getConnectionInfo(TEST_TIMEOUT_MS));
@@ -166,7 +207,9 @@ public class LazyRefreshStrategyTest {
     ExampleData data = new ExampleData(Instant.now().plus(1, ChronoUnit.HOURS));
     LazyRefreshStrategy r =
         new LazyRefreshStrategy(
-            "RefresherTest.testClosedCloudSqlInstanceDataThrowsException", () -> data);
+            "RefresherTest.testClosedCloudSqlInstanceDataThrowsException",
+            () -> data,
+            Duration.ZERO);
     r.close();
 
     assertThrows(IllegalStateException.class, () -> r.getConnectionInfo(TEST_TIMEOUT_MS));