Repository: valkey-io/valkey (2025-06-26T09:31:15Z)

Issue:
[NEW] Performance bottleneck in clusterNodeCleanupFailureReports under heavy failure load
The current implementation of `clusterNodeCleanupFailureReports` walks the entire fail reports in the list on every invocation and performs a listDelNode() for each expired or non‑voting entry. When a cluster node has accumulated hundreds or thousands of failure reports, this function becomes a CPU hotspot.

https://github.com/valkey-io/valkey/blob/3ceae81fc4cf065dae00888ab92b7aa069fde111/src/cluster_legacy.c#L1702-L1721

![Image](https://github.com/user-attachments/assets/4d4c8117-8613-4f99-860f-08d60d9f78e1)
![Image](https://github.com/user-attachments/assets/f544c279-ca8f-4134-946c-bcc6aec7f812)
(The above result was observed during a 2,000‑node cluster failover scenario)


**Current behavior:**
- Where N is the number of failure reports, O(N) scan on every call, regardless of how many reports are actually expired.

**Improvement ideas:**
- ~~Use a priority queue with expiration time, so expired nodes can be removed in O(RlogN) rather than scanned in full. (R is # of actually expired reports)~~
- Use lazy deletion instead of removing them on every call
- Use early exit when meeting quorum






Discussion:
Thanks @sungming2 for filing the issue. We had seen this as one of the major bottleneck during the failure of large number of nodes.

> Use a priority queue with expiration time, so expired nodes can be removed in O(RlogN) rather than scanned in full. (R is # of actually expired reports)
There will be cost associated while we are adding a bunch of nodes to the priority queue. We should account for that time as well.

Some of the other ideas from the top of my head ~ 

1. If there is a way to automatically evict the old failed reports, maybe using clusterCron(), that would be great. It should be O(1) when there are no failovers since the queue is empty.

2. Have we considered other data structures? Is there a way to store the failed reports in the dictionary, and maybe use the time of expiry as the key, and nodes as the values, so we can even evict the nodes in the O(K)?

It would be good to come up with a proposal so we can discuss pros / cons in the community.
We should also consider if we need to throttle the cleanup of failure reports itself. The failed nodes reports isn't the source of truth but an indicator, so maybe we don't need to perform cleanup so frequently? The con is that we may lose up some cluster convergence time as the number of failure report is used to form consensus. 
@sarthakaggarwal97  Thanks for the suggestion

> If there is a way to automatically evict the old failed reports, maybe using clusterCron(), that would be great

Yes, I thought about that too. It should be part of the `Use lazy deletion instead of removing them on every call` that I mentioned above. I think we also should be careful of not overloading cluster cron. if it tries to clean up too many entries at once, it could introduce another CPU bottleneck and delay other critical tasks. 

>There will be cost associated while we are adding a bunch of nodes to the priority queue. We should account for that time as well.

Yes, It is true
what about this early exit method
```
while (ln = listNext(iterator)) {
    fr = ln.value

    if (now - fr.time > TTL) {
        listDelNode(node.fail_reports, ln)
        continue
    }

    count += 1
    if (count >= quorum) {
            // quorum reached
    }
}
```

In this way, we can avoid cleanup on every gossip call.
The best case: O(K) (valid, recent reports only) and early exit when quorum is met.
The worst case: O(N) (e.g., half the primaries down)
@sungming2 IIUC this method doesn't deal with figuring out if we have reach quorum or not. So, I don't see how the early exit would help. 

Also, what % of CPU gets spent on `clusterNodeCleanupFailureReports` ?
> Also, what % of CPU gets spent on clusterNodeCleanupFailureReports

We end up spending somewhere between 30-60% of the compute when a bunch of primary nodes are killed. During that time, a lot of nodes are evaluated if they are deemed to be in the fail state or not. The CPU recovers automatically once this phase is over, and all the killed nodes are marked as failed.
@hpatro 

This optimization allows to determine if quorum is reached without having to iterate through the entire failure report list. For example, if only 3 out of 1000 primaries are down, and the list contains 900 reports, we can send fail broadcast message as soon as the quorum condition is met. It can replace current `clusterNodeFailureReportsCount` and remove `clusterNodeCleanupFailureReports`
```
    count += 1
    if (count >= quorum) {
            // quorum reached
    }
```

To further optimize, we could introduce a cleanup task in clusterCron() to periodically prune stale reports. Keeping the list healthy would reduce unnecessary iterations and improve overall efficiency.
> For example, if only 3 out of 1000 primaries are down, and the list contains 900 reports, we can send fail broadcast message as soon as the quorum condition is met. It can replace current clusterNodeFailureReportsCount and remove clusterNodeCleanupFailureReports

The failure report is per node. You would be performing the cleanup at each node level. So, you're reducing the iteration count by half at the max. For your example stated above, you would need to go through atleast 501 reports for each of the node.
> you're reducing the iteration count by half at the max. For your example stated above, you would need to go through atleast 501 reports for each of the node.

Yes, I will think about other ways too, but doesn't it improve at least in the best case? I know its probably minor thing though
> > you're reducing the iteration count by half at the max. For your example stated above, you would need to go through atleast 501 reports for each of the node.
> 
> Yes, I will think about other ways too, but doesn't it improve at least in the best case?

Give it a try. We would need to perform the cleanup somewhere else though so it's not completely eliminated. I would also try to think on the lines of representing this information in some other form to avoid this O(N) operation.
> avoid this O(N) operation.

Yes that is the ultimate goal



Patch:
diff --git a/src/cluster_legacy.c b/src/cluster_legacy.c
index bb5376283f..c541f8475d 100644
--- a/src/cluster_legacy.c
+++ b/src/cluster_legacy.c
@@ -1671,48 +1671,91 @@ clusterNode *createClusterNode(char *nodename, int flags) {
     node->tcp_port = 0;
     node->cport = 0;
     node->tls_port = 0;
-    node->fail_reports = listCreate();
+    node->fail_reports = raxNew();
     node->orphaned_time = 0;
     node->repl_offset = 0;
-    listSetFreeMethod(node->fail_reports, zfree);
     node->is_node_healthy = 0;
     return node;
 }
 
+/* 8 bytes mstime + 8 bytes cluster node pointer (padded with zeros on 32-bit systems). */
+#define FAILURE_REPORT_KEYLEN 16
+#define SEC_IN_MS 1000ULL
+#define CEIL_OFFSET_MS (SEC_IN_MS - 1)
+
+/* The report time is rounded up (ceiled) to the nearest second in order to bucket
+ * timestamps and keep the radix‐tree keys compact.  The resulting timestamp is stored
+ * in big‑endian format, followed by the pointer to the clusterNode. */
+static void encodeFailureReportKey(clusterNode *node, mstime_t report_time, unsigned char *buf_out) {
+    const size_t node_ptr_pad_bytes = (sizeof(clusterNode *) == 4) ? 4 : 0; // pad on 32-bit
+
+    /* Round up to the next second for fewer key splits and quorum grace */
+    mstime_t bucketed_time = (report_time / SEC_IN_MS) * SEC_IN_MS + SEC_IN_MS;
+
+    /* Big endian timestamp for numeric time order in rax */
+    uint64_t big_endian_time = htonu64((uint64_t)bucketed_time);
+    memcpy(buf_out, &big_endian_time, sizeof(uint64_t));
+
+    /* Append the node pointer, plus padding if necessary */
+    memcpy(buf_out + sizeof(uint64_t), &node, sizeof(node));
+    if (node_ptr_pad_bytes) memset(buf_out + sizeof(uint64_t) + sizeof(node), 0, node_ptr_pad_bytes);
+}
+
+/* Reverses the operation of encodeFailureReportKey, reading back a timestamp
+ * and the pointer to the associated clusterNode. */
+static void decodeFailureReportKey(unsigned char *buf, mstime_t *report_time, clusterNode **node_ptr) {
+    uint64_t big_endian_time;
+    memcpy(&big_endian_time, buf, sizeof(uint64_t));
+    *report_time = (mstime_t)ntohu64(big_endian_time);
+    memcpy(node_ptr, buf + sizeof(uint64_t), sizeof(*node_ptr));
+}
+
 /* This function is called every time we get a failure report from a node.
- * The side effect is to populate the fail_reports list (or to update
- * the timestamp of an existing report).
+ * We maintain fail_reports as a radix tree keyed by bucketed timestamp
+ * (ceiled to the next second) and the sender pointer, which naturally orders
+ * entries by time.
  *
  * 'failing' is the node that is in failure state according to the
  * 'sender' node.
  *
- * The function returns 0 if it just updates a timestamp of an existing
- * failure report from the same sender. 1 is returned if a new failure
- * report is created. */
+ * The function returns 0 if it early‐exits (same sender & time bucket)
+ * or updates a timestamp of an existing failure report from the same sender.
+ * 1 is returned if a new failure report is created. */
 int clusterNodeAddFailureReport(clusterNode *failing, clusterNode *sender) {
-    list *l = failing->fail_reports;
-    listNode *ln;
-    listIter li;
-    clusterNodeFailReport *fr;
-
-    /* If a failure report from the same sender already exists, just update
-     * the timestamp. */
-    listRewind(l, &li);
+    unsigned char buf[FAILURE_REPORT_KEYLEN];
     mstime_t now = mstime();
-    while ((ln = listNext(&li)) != NULL) {
-        fr = ln->value;
-        if (fr->node == sender) {
-            fr->time = now;
-            return 0;
+    const mstime_t bucketed_time = (now / SEC_IN_MS) * SEC_IN_MS + SEC_IN_MS;
+    int is_new = 1;
+
+    /* Look for any existing entry from this sender and remove it */
+    raxIterator ri;
+    raxStart(&ri, failing->fail_reports);
+    raxSeek(&ri, "^", NULL, 0);
+    while (raxNext(&ri)) {
+        mstime_t reported_time;
+        clusterNode *reported_node;
+        decodeFailureReportKey(ri.key, &reported_time, &reported_node);
+        if (reported_node == sender) {
+            if (reported_time == bucketed_time) {
+                /* Early exit if it is the exact same sender and bucket */
+                raxStop(&ri);
+                return 0;
+            }
+
+            raxRemove(failing->fail_reports,
+                      ri.key, ri.key_len, NULL);
+            is_new = 0;
+            break;
         }
     }
+    raxStop(&ri);
 
-    /* Otherwise create a new report. */
-    fr = zmalloc(sizeof(*fr));
-    fr->node = sender;
-    fr->time = now;
-    listAddNodeTail(l, fr);
-    return 1;
+    /* Encode new key (now + sender) and store the fresh timestamp */
+    encodeFailureReportKey(sender, now, buf);
+    raxInsert(failing->fail_reports,
+              buf, sizeof(buf), NULL, NULL);
+
+    return is_new;
 }
 
 /* Remove failure reports that are too old, where too old means reasonably
@@ -1721,27 +1764,34 @@ int clusterNodeAddFailureReport(clusterNode *failing, clusterNode *sender) {
  * older than the global node timeout, so we don't just trust the number
  * of failure reports from other nodes.
  *
- * If the reporting node loses its voting right during this time, we will
- * also clear its report. */
+ * This function scans the radix tree of failure reports (keyed by bucketed
+ * timestamps plus sender pointer) for the given node and removes any entries that
+ * have expired beyond the cutoff window
+ *
+ * By expiring entries older than the global node timeout window,
+ * we guarantee only fresh votes remain. */
 void clusterNodeCleanupFailureReports(clusterNode *node) {
-    list *l = node->fail_reports;
-    if (!listLength(l)) return;
-
-    listNode *ln;
-    listIter li;
-    clusterNodeFailReport *fr;
-    mstime_t maxtime = server.cluster_node_timeout * CLUSTER_FAIL_REPORT_VALIDITY_MULT;
     mstime_t now = mstime();
+    mstime_t timeout = server.cluster_node_timeout * CLUSTER_FAIL_REPORT_VALIDITY_MULT;
+    mstime_t cutoff = now - timeout;
 
-    listRewind(l, &li);
-    while ((ln = listNext(&li)) != NULL) {
-        fr = ln->value;
-        if (now - fr->time > maxtime) {
-            listDelNode(l, ln);
-        } else if (!clusterNodeIsVotingPrimary(fr->node)) {
-            listDelNode(l, ln);
-        }
+    raxIterator ri;
+    raxStart(&ri, node->fail_reports);
+    raxSeek(&ri, "^", NULL, 0);
+
+    while (raxNext(&ri)) {
+        mstime_t reported_time;
+        clusterNode *reported_node;
+        decodeFailureReportKey(ri.key, &reported_time, &reported_node);
+        if (reported_time > cutoff) break;
+
+        int retval = raxRemove(node->fail_reports, ri.key, ri.key_len, NULL);
+        serverAssert(retval == 1);
+        /* Move directly to the next element after the tree mutation due to removal */
+        raxSeek(&ri, ">=", ri.key, ri.key_len);
     }
+
+    raxStop(&ri);
 }
 
 /* Remove the failing report for 'node' if it was previously considered
@@ -1756,25 +1806,23 @@ void clusterNodeCleanupFailureReports(clusterNode *node) {
  * The function returns 1 if the failure report was found and removed.
  * Otherwise 0 is returned. */
 int clusterNodeDelFailureReport(clusterNode *node, clusterNode *sender) {
-    list *l = node->fail_reports;
-    if (!listLength(l)) return 0;
-
-    listNode *ln;
-    listIter li;
-    clusterNodeFailReport *fr;
-
-    /* Search for a failure report from this sender. */
-    listRewind(l, &li);
-    while ((ln = listNext(&li)) != NULL) {
-        fr = ln->value;
-        if (fr->node == sender) break;
+    raxIterator ri;
+    raxStart(&ri, node->fail_reports);
+    raxSeek(&ri, "^", NULL, 0);
+
+    while (raxNext(&ri)) {
+        mstime_t reported_time;
+        clusterNode *reported_node;
+        decodeFailureReportKey(ri.key, &reported_time, &reported_node);
+        if (reported_node == sender) {
+            raxRemove(node->fail_reports,
+                      ri.key, ri.key_len, NULL);
+            raxStop(&ri);
+            return 1;
+        }
     }
-    if (!ln) return 0; /* No failure report from this sender. */
-
-    /* Remove the failure report. */
-    listDelNode(l, ln);
-    clusterNodeCleanupFailureReports(node);
-    return 1;
+    raxStop(&ri);
+    return 0;
 }
 
 /* Return the number of external nodes that believe 'node' is failing,
@@ -1782,7 +1830,7 @@ int clusterNodeDelFailureReport(clusterNode *node, clusterNode *sender) {
  * node as well. */
 int clusterNodeFailureReportsCount(clusterNode *node) {
     clusterNodeCleanupFailureReports(node);
-    return listLength(node->fail_reports);
+    return raxSize(node->fail_reports);
 }
 
 static int clusterNodeNameComparator(const void *node1, const void *node2) {
@@ -1855,7 +1903,7 @@ void freeClusterNode(clusterNode *n) {
     sdsfree(n->human_nodename);
     sdsfree(n->announce_client_ipv4);
     sdsfree(n->announce_client_ipv6);
-    listRelease(n->fail_reports);
+    raxFree(n->fail_reports);
     zfree(n->replicas);
     zfree(n);
 }
diff --git a/src/cluster_legacy.h b/src/cluster_legacy.h
index a49cb3e87f..3914550902 100644
--- a/src/cluster_legacy.h
+++ b/src/cluster_legacy.h
@@ -71,12 +71,6 @@ typedef struct clusterLink {
 #define nodeSupportsLightMsgHdrForModule(n) ((n)->flags & CLUSTER_NODE_LIGHT_HDR_MODULE_SUPPORTED)
 #define nodeInNormalState(n) (!((n)->flags & (CLUSTER_NODE_HANDSHAKE | CLUSTER_NODE_MEET | CLUSTER_NODE_PFAIL | CLUSTER_NODE_FAIL)))
 
-/* This structure represent elements of node->fail_reports. */
-typedef struct clusterNodeFailReport {
-    clusterNode *node; /* Node reporting the failure condition. */
-    mstime_t time;     /* Time of the last report from this node. */
-} clusterNodeFailReport;
-
 /* Cluster messages header */
 
 /* Message types.
@@ -364,7 +358,7 @@ struct _clusterNode {
     int cport;                              /* Latest known cluster port of this node. */
     clusterLink *link;                      /* TCP/IP link established toward this node */
     clusterLink *inbound_link;              /* TCP/IP link accepted from this node */
-    list *fail_reports;                     /* List of nodes signaling this as failing */
+    rax *fail_reports;                      /* Radix tree for failure reports with sorted order by timestamp */
     int is_node_healthy;                    /* Boolean indicating the cached node health.
                                                Update with updateAndCountChangedNodeHealth(). */
 };


Test patch:
diff --git a/tests/unit/cluster/failure-marking.tcl b/tests/unit/cluster/failure-marking.tcl
index 787713b9fa..0ec85e9549 100644
--- a/tests/unit/cluster/failure-marking.tcl
+++ b/tests/unit/cluster/failure-marking.tcl
@@ -111,6 +111,16 @@ tags {external:skip tls:skip cluster singledb} {
             }
 
             resume_process $replica_pid
+
+            # Check there are no failure reports left.
+            wait_for_condition 1000 50 {
+                [R 0 CLUSTER COUNT-FAILURE-REPORTS $replica_id] == 0 &&
+                [R 2 CLUSTER COUNT-FAILURE-REPORTS $replica_id] == 0 &&
+                [R 3 CLUSTER COUNT-FAILURE-REPORTS $replica_id] == 0 &&
+                [R 4 CLUSTER COUNT-FAILURE-REPORTS $replica_id] == 0
+            } else {
+                fail "Cluster COUNT-FAILURE-REPORTS is not right."
+            }
         }
     }
 }