Repository: govuk-one-login/ipv-core-back
Pull request #3010: PYIC-8127 Rethrow fatal IOExceptions on all HTTP requests
Author: login: blakeyp
Created: 2025-03-03T12:01:57Z | Merged: 2025-03-04T09:25:31Z

## Proposed changes

### What changed

Throw unchecked exception for 'fatal' IOExceptions on all HTTP requests to force lambda instance to crash. If timeout exception, just rethrow. If "Connection reset" exception or HttpRetryException, retry once and then throw unchecked exception.

### Why did it change

We've seen a further instance in prod of a lambda spawned with apparently no network connectivity, raising IOExceptions on HTTP requests, this time in process-cri-callback, and again eventually crashing with a seg fault. As a further mitigation, we should rethrow fatal IOExceptions on all HTTP requests, not just EVCS requests as previously.

### Issue tracking
- [PYIC-8127](https://govukverify.atlassian.net/browse/PYIC-8127)



[PYIC-8127]: https://govukverify.atlassian.net/browse/PYIC-8127?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Diff:
diff --git a/libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java b/libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
index 1a8c922b19..39d06d6715 100644
--- a/libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
+++ b/libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
@@ -11,12 +11,15 @@
 import javax.net.ssl.SSLParameters;
 
 import java.io.IOException;
+import java.io.UncheckedIOException;
 import java.net.Authenticator;
 import java.net.CookieHandler;
+import java.net.HttpRetryException;
 import java.net.ProxySelector;
 import java.net.http.HttpClient;
 import java.net.http.HttpRequest;
 import java.net.http.HttpResponse;
+import java.net.http.HttpTimeoutException;
 import java.time.Duration;
 import java.time.Instant;
 import java.util.Optional;
@@ -51,15 +54,25 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
+            if (e instanceof HttpTimeoutException) {
+                throw e;
+            }
+            // In the build environment we see connection resets for idle connections in the
+            // pool. Retrying uses a different connection.
+            if (e.getMessage().contains("Connection reset") || e instanceof HttpRetryException) {
                 LOGGER.warn(
-                        LogHelper.buildErrorMessage("Retrying after HTTP IOException", e)
+                        LogHelper.buildErrorMessage("Retrying after non-fatal HTTP IOException", e)
                                 .with("host", request.uri().getHost()));
-                return baseClient.send(request, responseBodyHandler);
+                try {
+                    return baseClient.send(request, responseBodyHandler);
+                } catch (IOException ex) {
+                    throw new UncheckedIOException(ex);
+                }
             }
-            throw e;
+            // Rethrow any other IOException as unchecked exception to force a crash (see PYIC-8058
+            // and linked incident INC0014124)
+            throw new UncheckedIOException(e);
         }
     }
 
diff --git a/libs/common-services/src/test/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClientTest.java b/libs/common-services/src/test/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClientTest.java
index 0efe74f2fe..faf1bddcfe 100644
--- a/libs/common-services/src/test/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClientTest.java
+++ b/libs/common-services/src/test/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClientTest.java
@@ -7,11 +7,14 @@
 import org.mockito.junit.jupiter.MockitoExtension;
 
 import java.io.IOException;
+import java.io.UncheckedIOException;
 import java.net.URI;
 import java.net.http.HttpClient;
 import java.net.http.HttpRequest;
 import java.net.http.HttpResponse;
+import java.net.http.HttpTimeoutException;
 
+import static org.junit.jupiter.api.Assertions.assertThrowsExactly;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
@@ -34,4 +37,26 @@ void sendShouldRetryConnectionReset() throws Exception {
 
         verify(mockHttpClient, times(2)).send(any(), any());
     }
+
+    @Test
+    void sendShouldRethrowIfTimeoutException() throws Exception {
+        when(mockHttpClient.send(any(), any()))
+                .thenThrow(new HttpTimeoutException("Timed out"))
+                .thenReturn(null);
+        var req = HttpRequest.newBuilder().uri(URI.create("https://example.com")).build();
+        var handler = HttpResponse.BodyHandlers.ofString();
+
+        assertThrowsExactly(HttpTimeoutException.class, () -> tracingHttpClient.send(req, handler));
+    }
+
+    @Test
+    void sendShouldThrowUncheckedExceptionIfFatalIOException() throws Exception {
+        when(mockHttpClient.send(any(), any()))
+                .thenThrow(new IOException("Something fatal"))
+                .thenReturn(null);
+        var req = HttpRequest.newBuilder().uri(URI.create("https://example.com")).build();
+        var handler = HttpResponse.BodyHandlers.ofString();
+
+        assertThrowsExactly(UncheckedIOException.class, () -> tracingHttpClient.send(req, handler));
+    }
 }
diff --git a/libs/evcs-service/src/main/java/uk/gov/di/ipv/core/library/client/EvcsClient.java b/libs/evcs-service/src/main/java/uk/gov/di/ipv/core/library/client/EvcsClient.java
index 4f60cb1344..cc036a1465 100644
--- a/libs/evcs-service/src/main/java/uk/gov/di/ipv/core/library/client/EvcsClient.java
+++ b/libs/evcs-service/src/main/java/uk/gov/di/ipv/core/library/client/EvcsClient.java
@@ -26,7 +26,6 @@
 import uk.gov.di.ipv.core.library.tracing.TracingHttpClient;
 
 import java.io.IOException;
-import java.io.UncheckedIOException;
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.net.URLEncoder;
@@ -233,15 +232,7 @@ private HttpResponse<String> sendHttpRequest(HttpRequest evcsHttpRequest)
                                             evcsHttpRequest, HttpResponse.BodyHandlers.ofString());
                             checkResponseStatusCode(res);
                             return res;
-                        } catch (IOException e) {
-                            LOGGER.error(
-                                    LogHelper.buildErrorMessage(
-                                            "HTTP request failed with IOException", e));
-                            // Rethrow IOException as unchecked exception for now to force lambda to
-                            // crash (see
-                            // PYIC-8058 and linked incident INC0014124)
-                            throw new UncheckedIOException(e);
-                        } catch (EvcsServiceException e) {
+                        } catch (EvcsServiceException | IOException e) {
                             throw new NonRetryableException(e);
                         } catch (InterruptedException e) {
                             // This should never happen running in Lambda as it's single
diff --git a/libs/evcs-service/src/test/java/uk/gov/di/ipv/core/library/client/EvcsClientTest.java b/libs/evcs-service/src/test/java/uk/gov/di/ipv/core/library/client/EvcsClientTest.java
index 51d94d75ac..75687007ea 100644
--- a/libs/evcs-service/src/test/java/uk/gov/di/ipv/core/library/client/EvcsClientTest.java
+++ b/libs/evcs-service/src/test/java/uk/gov/di/ipv/core/library/client/EvcsClientTest.java
@@ -27,8 +27,6 @@
 import uk.gov.di.ipv.core.library.retry.Sleeper;
 import uk.gov.di.ipv.core.library.service.ConfigService;
 
-import java.io.IOException;
-import java.io.UncheckedIOException;
 import java.net.URLEncoder;
 import java.net.http.HttpClient;
 import java.net.http.HttpRequest;
@@ -265,21 +263,6 @@ void testGetUserVCs_shouldThrowException_ifHttpClientException() throws Exceptio
                                 TEST_USER_ID, TEST_EVCS_ACCESS_TOKEN, VC_STATES_FOR_QUERY));
     }
 
-    @Test
-    void testGetUserVCs_shouldRethrowUncheckedException_ifIOException() throws Exception {
-        // Arrange
-        when(mockHttpClient.send(any(HttpRequest.class), any(HttpResponse.BodyHandler.class)))
-                .thenThrow(new IOException("I/O error"));
-
-        // Act
-        // Assert
-        assertThrows(
-                UncheckedIOException.class,
-                () ->
-                        evcsClient.getUserVcs(
-                                TEST_USER_ID, TEST_EVCS_ACCESS_TOKEN, VC_STATES_FOR_QUERY));
-    }
-
     @Test
     void testGetUserVCs_shouldThrowException_ifResponseBodyParsingFail() throws Exception {
         // Arrange


Reviews:
user: login: Joe-Edwards-GDS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T12:18:22Z
user: login: Joe-Edwards-GDS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T12:23:12Z
user: login: blakeyp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:51:24Z
user: login: blakeyp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:52:59Z
user: login: Joe-Edwards-GDS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:55:37Z
user: login: blakeyp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:57:42Z
user: login: Joe-Edwards-GDS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T14:01:59Z
user: login: blakeyp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T15:00:42Z
user: login: Joe-Edwards-GDS
body: This looks ok to me.. Might want to monitor the logs a little bit once it hits to make sure we're not crashing on anything unexpected.

As mentioned, I don't think our error logs include stack traces at all, but that can be done as a separate change.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T15:59:02Z
user: login: blakeyp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T16:20:04Z
user: login: Joe-Edwards-GDS
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T16:31:29Z

Review comments:
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
+            if (isNonFatalIoException(e)) {
                 LOGGER.warn(
-                        LogHelper.buildErrorMessage("Retrying after HTTP IOException", e)
+                        LogHelper.buildErrorMessage("Retrying after non-fatal HTTP IOException", e)
                                 .with("host", request.uri().getHost()));
                 return baseClient.send(request, responseBodyHandler);
             }
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: Joe-Edwards-GDS
body: This logic doesn't seem quite right - I think this means timeouts will be retried, which probably isn't what you want?

I suspect there are 3(?) paths:
- Timeouts -> throw IOException as-is
- Connection reset -> retry once, and throw unchecked IO if it fails?
- Any other IO Exception -> throw unchecked IO immediately?
created_at: 2025-03-03T12:16:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: Joe-Edwards-GDS
body: Do we need this? If it's rethrown then we should log it where its caught, and if it's retried then we get the warning.

(Doesn't matter too much in the fatal case, but e.g. if we recover gracefully then maybe it's only a warning?)
created_at: 2025-03-03T12:18:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
+            if (isNonFatalIoException(e)) {
                 LOGGER.warn(
-                        LogHelper.buildErrorMessage("Retrying after HTTP IOException", e)
+                        LogHelper.buildErrorMessage("Retrying after non-fatal HTTP IOException", e)
                                 .with("host", request.uri().getHost()));
                 return baseClient.send(request, responseBodyHandler);
             }
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: Joe-Edwards-GDS
body: If you can solve the connection reset some other way then that would be ideal, because we can reduce down to two cases... - I know Chris looked into it a bit and didn't really get anywhere :(
created_at: 2025-03-03T12:23:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: blakeyp
body: I wanted to put this in for now so we can easily see exactly what IOExceptions we're getting and AWS Support have asked for stack traces (not that they'll be much use..)
created_at: 2025-03-03T13:51:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
+            if (isNonFatalIoException(e)) {
                 LOGGER.warn(
-                        LogHelper.buildErrorMessage("Retrying after HTTP IOException", e)
+                        LogHelper.buildErrorMessage("Retrying after non-fatal HTTP IOException", e)
                                 .with("host", request.uri().getHost()));
                 return baseClient.send(request, responseBodyHandler);
             }
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: blakeyp
body: I had thought be no harm in retrying on timeouts, now I've remembered TICF 🤦‍♂️ 
created_at: 2025-03-03T13:52:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: Joe-Edwards-GDS
body: Ok fair enough - I don't think our error logging includes stack traces at the moment though.

I can't remember all the history tbh. I came very close to adding them in a while back, but it all got a bit muddied and I ended up not including it. Something about not including (possibly sensitive) stack traces in logs, but IMO the value they add is a lot greater than the risk.
created_at: 2025-03-03T13:55:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
+            if (isNonFatalIoException(e)) {
                 LOGGER.warn(
-                        LogHelper.buildErrorMessage("Retrying after HTTP IOException", e)
+                        LogHelper.buildErrorMessage("Retrying after non-fatal HTTP IOException", e)
                                 .with("host", request.uri().getHost()));
                 return baseClient.send(request, responseBodyHandler);
             }
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: blakeyp
body: I don't have much context on the "Connection reset" errors other than the comment 😕
created_at: 2025-03-03T13:57:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
+            if (isNonFatalIoException(e)) {
                 LOGGER.warn(
-                        LogHelper.buildErrorMessage("Retrying after HTTP IOException", e)
+                        LogHelper.buildErrorMessage("Retrying after non-fatal HTTP IOException", e)
                                 .with("host", request.uri().getHost()));
                 return baseClient.send(request, responseBodyHandler);
             }
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: Joe-Edwards-GDS
body: Some context on https://github.com/govuk-one-login/ipv-core-back/pull/2704 (including a stack overflow link blaming the NAT gateway...)
created_at: 2025-03-03T14:01:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
+            if (isNonFatalIoException(e)) {
                 LOGGER.warn(
-                        LogHelper.buildErrorMessage("Retrying after HTTP IOException", e)
+                        LogHelper.buildErrorMessage("Retrying after non-fatal HTTP IOException", e)
                                 .with("host", request.uri().getHost()));
                 return baseClient.send(request, responseBodyHandler);
             }
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: blakeyp
body: I've updated, it's not particularly pretty but I think does what we want for now
created_at: 2025-03-03T15:00:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: blakeyp
body: `LOGGER.error(message, ex)` should print the stack trace, at least it does in unit tests but since it's not a structured message it might come out a bit funky in cloudwatch
> Logs a message at the ERROR level including the stack trace of the Throwable throwable passed as parameter.
created_at: 2025-03-03T16:20:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -51,15 +54,16 @@ public <T> HttpResponse<T> send(
         try {
             return baseClient.send(request, responseBodyHandler);
         } catch (IOException e) {
-            // In the build environment we see connection resets for idle connections in
-            // the pool. Retrying uses a different connection.
-            if (e.getMessage().contains("Connection reset")) {
+            LOGGER.error("HTTP request failed with IOException", e);
path: libs/common-services/src/main/java/uk/gov/di/ipv/core/library/tracing/TracingHttpClient.java
user: login: Joe-Edwards-GDS
body: Ah sorry, I'm not looking closely enough - this is distinct from `buildErrorMessage`!

Yes, the underlying logger should be able to log stack traces (though as you say, I don't know exactly what it'll look like in CW/splunk).

I do still wonder if we should make our default error logging include the stack trace too, so we can get it in our standard log format.
created_at: 2025-03-03T16:31:29Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-03-03T15:06:39Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=ipv-core-back&pullRequest=3010) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=ipv-core-back&pullRequest=3010&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=ipv-core-back&pullRequest=3010&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=ipv-core-back&pullRequest=3010&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=ipv-core-back&pullRequest=3010&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=ipv-core-back&pullRequest=3010&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=ipv-core-back&pullRequest=3010)


author_association: NONE