Repository: c-ares/c-ares (2026-07-04T22:19:05Z)

Issue:
v1.34.6 regression: ares_fetch_connection rejects connections when consec_failures > 0, causing UDP socket exhaustion under load
A regression introduced in v1.34.6 (via #1032) causes c-ares to open a new UDP
socket for every DNS query when the DNS server has any consecutive failures,
instead of reusing existing connections. Under real-world load this produces a
235x increase in UDP socket usage.

## Versions affected

- **v1.34.5**: working correctly (baseline)
- **v1.34.6**: broken (regression introduced via #1032)


node: 22.22.2-bookworm result:
<img width="643" height="201" alt="Image" src="https://github.com/user-attachments/assets/5ab1ed68-749e-4fe7-aba4-5884f2dacca9" />


node:22.21.1-bookworm result:
<img width="636" height="226" alt="Image" src="https://github.com/user-attachments/assets/3f250c4f-6820-453b-8776-6e22283c5149" />

Metric | Node 22.21.1 | Node 22.22.2 | Delta
-- | -- | -- | --
Peak sockets | 13 (+7) | 1,651 (+1,645) | 235x more sockets
Amplification | 4.30x | 4.76x | +11%
Packets OUT | 26,866 | 29,744 | +11%
Packets IN | 14,388 | 13,490 | -6% (more drops)


## Root cause

Commit db67e35 in #1032 added the following guard to `ares_fetch_connection()`:

```c
if (conn->server->consec_failures > 0) {
  return NULL;
}







Discussion:
can you elaborate a bit more on your scenario here?  what is causing the socket to have failures so frequently in your case that this causes any sort of issue?
> can you elaborate a bit more on your scenario here? what is causing the socket to have failures so frequently in your case that this causes any sort of issue?

The scenario: A Node.js application server making a high volume of outbound DNS lookups under production load — hostname resolution for API calls, external service connections, etc. At this scale, tens of thousands of lookups per minute is typical.

Why consec_failures gets set: Under high query volume, transient failures are normal even with a perfectly healthy DNS server:

Occasional UDP packet loss on the network path triggers a timeout → failure
The DNS server may briefly drop or rate-limit queries when the source port floods it
A momentary SERVFAIL for an upstream delegation issue
These are noise, not signs of a broken server. v1.34.5 handled this correctly — it kept reusing the existing UDP connection, retried over it, and only escalated after sustained failure.

The 1.34.6 behavior: The consec_failures > 0 guard means any single transient failure permanently stops connection reuse for that server. Every subsequent query opens a new UDP socket. At high query rates this compounds fast:

In our controlled test: peak sockets 1,651 vs 13 on v1.34.5 — a 235x increase
In production, with real reverse DNS lookup volume, UDP socket count climbs past 25,000 and causes the process to hang — services become unresponsive until restarted
Each new socket holds an open inflight query from a unique ephemeral port. The DNS server sees this as a flood from many sources → drops more → consec_failures stays > 0 → the cycle never recovers. Eventually the OS runs out of ephemeral ports or file descriptors and the process hangs entirely.

The core issue: consec_failures is a server health metric, but the guard uses it as a per-connection eviction trigger. A server can have consec_failures == 1 from a single dropped packet and still be perfectly healthy. The change conflates "this server has experienced a problem" with "never reuse any existing connection to it," which breaks connection reuse under any realistic high-load pattern where transient failures are inevitable.
I do suspect I can have similar issues/regression from 1.34.5 to 1.34.6. I can't say for sure its this one. Do you have a proposed fix?



Patch:
diff --git a/src/lib/ares_close_sockets.c b/src/lib/ares_close_sockets.c
index 347f43e6fc..7bb72c31ec 100644
--- a/src/lib/ares_close_sockets.c
+++ b/src/lib/ares_close_sockets.c
@@ -112,11 +112,11 @@ void ares_check_cleanup_conns(const ares_channel_t *channel)
         do_cleanup = ARES_TRUE;
       }
 
-      /* If the associated server has failures, close it out. Resetting the
-       * connection (and specifically the source port number) can help resolve
-       * situations where packets are being dropped.
+      /* If the connection has been retired for new queries, close it out once
+       * idle.  Resetting the connection (and specifically the source port
+       * number) can help resolve situations where packets are being dropped.
        */
-      if (conn->server->consec_failures > 0) {
+      if (conn->flags & ARES_CONN_FLAG_NONEW) {
         do_cleanup = ARES_TRUE;
       }
 
diff --git a/src/lib/ares_conn.h b/src/lib/ares_conn.h
index 16ecefdd19..c332de14b1 100644
--- a/src/lib/ares_conn.h
+++ b/src/lib/ares_conn.h
@@ -43,7 +43,13 @@ typedef enum {
   ARES_CONN_FLAG_TFO = 1 << 1,
   /*! TCP Fast Open has not yet sent its first packet. Gets unset on first
    *  write to a connection */
-  ARES_CONN_FLAG_TFO_INITIAL = 1 << 2
+  ARES_CONN_FLAG_TFO_INITIAL = 1 << 2,
+  /*! Connection has been retired: it must not be handed any NEW queries (e.g.
+   *  it experienced a query timeout, suggesting packets are being dropped on
+   *  it).  Its in-flight queries continue to drain and it is cleaned up once
+   *  idle.  This is a per-connection signal so that a transient failure does
+   *  not evict otherwise-healthy connections to the same server. */
+  ARES_CONN_FLAG_NONEW = 1 << 3
 } ares_conn_flags_t;
 
 typedef enum {
diff --git a/src/lib/ares_process.c b/src/lib/ares_process.c
index aebea1af27..c93f4d41d9 100644
--- a/src/lib/ares_process.c
+++ b/src/lib/ares_process.c
@@ -735,6 +735,12 @@ static ares_status_t process_timeouts(ares_channel_t       *channel,
     query->timeouts++;
 
     conn = query->conn;
+    /* Retire this connection for NEW queries.  A timeout suggests packets are
+     * being dropped on it, so route new queries to a fresh source port while
+     * the in-flight queries here drain (it is cleaned up once idle).  This is
+     * per-connection so a transient failure doesn't stop reuse of healthy
+     * connections to the same server. */
+    conn->flags |= ARES_CONN_FLAG_NONEW;
     server_increment_failures(conn->server, query->using_tcp);
     status = ares_requeue_query(query, now, ARES_ETIMEOUT, ARES_TRUE, NULL,
       NULL);
@@ -1236,9 +1242,13 @@ static ares_conn_t *ares_fetch_connection(const ares_channel_t *channel,
     return NULL;
   }
 
-  /* If the associated server has failures, don't use it.  It should be cleaned
-   * up later. */
-  if (conn->server->consec_failures > 0) {
+  /* Don't hand new queries to a connection that has been retired (e.g. it saw
+   * a timeout).  It keeps servicing its in-flight queries and is cleaned up
+   * once idle.  Note this is a per-connection check: a server-wide failure
+   * counter must not be used here or a single transient failure would evict
+   * every (including healthy) connection to the server and spawn a new socket
+   * per query. */
+  if (conn->flags & ARES_CONN_FLAG_NONEW) {
     return NULL;
   }
 


Test patch:
diff --git a/test/ares-test-mock.cc b/test/ares-test-mock.cc
index 5b9c15eb43..3b63d1b65a 100644
--- a/test/ares-test-mock.cc
+++ b/test/ares-test-mock.cc
@@ -526,6 +526,69 @@ TEST_P(MockUDPMaxQueriesTest, GetHostByNameParallelLookups) {
   }
 }
 
+// Regression test for #1152.  A transient failure (a single query timeout)
+// must not force a brand new UDP socket to be opened for every subsequent
+// query to the same server.  The connection that saw the timeout is retired
+// for new queries (ARES_CONN_FLAG_NONEW) and drains, while new queries share
+// one fresh connection.  Before the fix, ares_fetch_connection() rejected
+// reuse whenever the server's consec_failures was > 0, so a single dropped
+// packet caused a new socket per query and exhausted sockets/ports under load.
+class MockUDPTransientFailureTest
+    : public MockChannelOptsTest,
+      public ::testing::WithParamInterface<int> {
+ public:
+  MockUDPTransientFailureTest()
+    : MockChannelOptsTest(1, GetParam(), false, false,
+                          FillOptions(&opts_),
+                          ARES_OPT_TIMEOUTMS | ARES_OPT_TRIES) {}
+  static struct ares_options* FillOptions(struct ares_options * opts) {
+    memset(opts, 0, sizeof(struct ares_options));
+    opts->timeout = 150; /* ms (ARES_OPT_TIMEOUTMS) - keep the test quick */
+    opts->tries   = 3;
+    return opts;
+  }
+ private:
+  struct ares_options opts_;
+};
+
+TEST_P(MockUDPTransientFailureTest, NoSocketExplosionOnTimeout) {
+  // Configure the server to never reply, so every query times out (a
+  // transient failure).  No SetReply() means ProcessRequest() drops the
+  // packet; the AnyNumber() expectation just silences uninteresting-call
+  // warnings for the many dropped requests.
+  EXPECT_CALL(server_, OnRequest("www.example.com", T_A))
+    .Times(testing::AnyNumber());
+
+  int rc = ARES_SUCCESS;
+  ares_set_socket_callback(channel_, SocketConnectCallback, &rc);
+  sock_cb_count = 0;
+
+  const size_t nqueries = 16;
+  HostResult result[nqueries];
+  for (size_t i = 0; i < nqueries; i++) {
+    ares_gethostbyname(channel_, "www.example.com.", AF_INET, HostCallback,
+                       &result[i]);
+  }
+  Process();
+
+  // All queries complete (they fail with a timeout since the server never
+  // replied).
+  for (size_t i = 0; i < nqueries; i++) {
+    EXPECT_TRUE(result[i].done_);
+  }
+
+  // The key assertion: with tries=3 the fix opens roughly one connection per
+  // retry round (the retired connection is skipped and the single replacement
+  // is reused by all queries), not one per query.  Before the fix this was on
+  // the order of nqueries*tries sockets.
+  EXPECT_LE(sock_cb_count, 6)
+    << "opened " << sock_cb_count << " sockets for " << nqueries
+    << " queries; a transient timeout must not force a new socket per query";
+}
+
+INSTANTIATE_TEST_SUITE_P(AddressFamilies, MockUDPTransientFailureTest,
+                         ::testing::ValuesIn(ares::test::families), PrintFamily);
+
 class CacheQueriesTest
     : public MockChannelOptsTest,
       public ::testing::WithParamInterface<int> {