Repository: redis/redis

Problem statement:
Gossiped node deletion
Typically you need to send a cluster forget to each node in a cluster to delete a node. If you don't do this fast enough, the node will be re-added through gossip. Ideally you just need to forget a node and it will eventually be forgotten throughout the cluster.

This is part of #8948.

Possible ways to implement this:

1. Append the contents of the clusterBlacklistNodes (node id and timestamp) to all cluster ping and pong messages. A ping extension (mechanism introduced for hostnames) can be used for this. (The blacklist expire timestamp needs to be gossiped too, so that we don't end up with the node being re-forgotten and re-added to the blacklist by gossip after the blacklist TTL is passed.) This increases the size of the cluster bus messages while any node is blacklisted (1 minute), but if the blacklist is short and the ping extension is small, this might be acceptable.
2. Append the contents of the clusterBlacklistNodes (node id and timestamp) to every pong message which is a response to a ping, but only when we detect that the ping's gossip section contains a forgotten node, i.e. a node in our blacklist. This minimizes the extra data sent over the cluster bus, but it is enough to make sure a node is forgotten throughout the cluster?
3. When a CLUSTER FORGET command is received, broadcast this to the cluster, for example in a pong with the forgotten node in a ping extension. This is done only once and by the node which receives the command. If a client sends CLUSTER FORGET to all nodes in a cluster, there will be NxN pongs sent in a short period of time.

@madolson et.al. any preference among the above or other ideas?


Hints:
First, some context. Some folks from RL and the core group have been iterating on a design for completing replacing the clusterbus with a new consistent storage. With that in place, it becomes trivial to "remove" a node from a cluster, since all operations are driven through consensus. @ushachar ping since he is supposed to be preparing that for general discussion.

Since the idea of Cluster V2 is it will replace the cluster in the fullness of time, we can still make incremental progress until then on other parts of the cluster system, since it'll be at least 1+ years. We decided to continue iterating on slot migration, for instance.

So, to your points. I generally like the option 1, it can also be "enhanced" like option 3 by immediately broadcasting out the ping to all nodes. I don't have much issue with "extra" metadata on the cluster protocol since it's already so verbose (it's ~2kb today, already larger than a single TCP packet!). Option 2 seems to try to be more of an optimization, but I'm not convinced it buys us all that much. One thing worth highlighting is that the current system does not broadcasting any time information, to allow nodes to have different clocks, so we should consider broadcasting out relative TTL which can only decrement.
>Some folks from RL and the core group have been iterating on a design for completing replacing the clusterbus with a new consistent storage. 

@madolson is this the doc that you were referring to [here](https://github.com/redis/redis/issues/2807#issuecomment-1152794422)?

>We decided to continue iterating on slot migration, for instance.

Do you mind circling back to thread #2807? I don't intend to hijack this thread but now with this new information, I think it would be great to align with the community on how to best evolve the existing slot migration protocol. 

>Typically you need to send a cluster forget to each node in a cluster to delete a node. If you don't do this fast enough, the node will be re-added through gossip. Ideally you just need to forget a node and it will eventually be forgotten throughout the cluster.

I agree we need a one-shot `CLUSTER FORGET`. The timeout can  indeed become an issue in a large cluster. However, I am not sure if gossip is the right solution. It is not entirely clear to me how we could ensure consistency in the face of failures. What'd happen if a few nodes in the cluster didn't receive the gossip for various reasons, either temporarily network issues or simply some races between `CLUSTER MEET` and `CLUSTER FORGET` such that the `FORGET` initiator wasn't aware of the presence of these nodes. A consensus-based solution would be ideal IMO though I don't have the details, yet.

I'll go for option 1 with relative TTL then!

@PingXie I don't think we'll get any guarantees here, just better chances. redis-cli can still try to send cluster forget to all nodes. Guarantees will come in the new future cluster bus, hopefully.

@madolson Any chance that I (and @PingXie I suppose) can be involved in the Cluster V2 project? I have some time to spend and it seems wiser to spend it on the long-term solution. Or do you prefer that we continue to improve Cluster V1 meanwhile?
Option 1 seems  good, and we need to avoid broadcast operations as much as possible, broadcasting in a large cluster (1000 nodes) may cause fluctuations in latency

Patch:
diff --git a/src/cluster.c b/src/cluster.c
index 09c3880138a..813e3119490 100644
--- a/src/cluster.c
+++ b/src/cluster.c
@@ -2012,6 +2012,21 @@ int writeHostnamePingExt(clusterMsgPingExt **cursor) {
     return extension_size;
 }
 
+/* Write the forgotten node ping extension at the start of the cursor, update
+ * the cursor to point to the end of the written extension and return the number
+ * of bytes written. */
+int writeForgottenNodePingExt(clusterMsgPingExt **cursor, sds name, uint64_t ttl) {
+    serverAssert(sdslen(name) == CLUSTER_NAMELEN);
+    clusterMsgPingExtForgottenNode *ext = &(*cursor)->ext[0].forgotten_node;
+    memcpy(ext->name, name, CLUSTER_NAMELEN);
+    ext->ttl = htonu64(ttl);
+    uint32_t extension_size = sizeof(clusterMsgPingExt) + sizeof(clusterMsgPingExtForgottenNode);
+    (*cursor)->type = htons(CLUSTERMSG_EXT_TYPE_FORGOTTEN_NODE);
+    (*cursor)->length = htonl(extension_size);
+    *cursor = (clusterMsgPingExt *) (ext->name + sizeof(clusterMsgPingExtForgottenNode));
+    return extension_size;
+}
+
 /* We previously validated the extensions, so this function just needs to
  * handle the extensions. */
 void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
@@ -2025,6 +2040,19 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
         if (type == CLUSTERMSG_EXT_TYPE_HOSTNAME) {
             clusterMsgPingExtHostname *hostname_ext = (clusterMsgPingExtHostname *) &(ext->ext[0].hostname);
             ext_hostname = hostname_ext->hostname;
+        } else if (type == CLUSTERMSG_EXT_TYPE_FORGOTTEN_NODE) {
+            clusterMsgPingExtForgottenNode *forgotten_node_ext = &(ext->ext[0].forgotten_node);
+            clusterNode *n = clusterLookupNode(forgotten_node_ext->name, CLUSTER_NAMELEN);
+            if (n && n != myself && !(nodeIsSlave(myself) && myself->slaveof == n)) {
+                sds id = sdsnewlen(forgotten_node_ext->name, CLUSTER_NAMELEN);
+                dictEntry *de = dictAddRaw(server.cluster->nodes_black_list, id, NULL);
+                serverAssert(de != NULL);
+                uint64_t expire = server.unixtime + ntohu64(forgotten_node_ext->ttl);
+                dictSetUnsignedIntegerVal(de, expire);
+                clusterDelNode(n);
+                clusterDoBeforeSleep(CLUSTER_TODO_UPDATE_STATE|
+                                     CLUSTER_TODO_SAVE_CONFIG);
+            }
         } else {
             /* Unknown type, we will ignore it but log what happened. */
             serverLog(LL_WARNING, "Received unknown extension type %d", type);
@@ -2924,6 +2952,8 @@ void clusterSendPing(clusterLink *link, int type) {
     estlen = sizeof(clusterMsg) - sizeof(union clusterMsgData);
     estlen += (sizeof(clusterMsgDataGossip)*(wanted + pfail_wanted));
     estlen += getHostnamePingExtSize();
+    estlen += dictSize(server.cluster->nodes_black_list) *
+        (sizeof(clusterMsgPingExt) + sizeof(clusterMsgPingExtForgottenNode));
 
     /* Note: clusterBuildMessageHdr() expects the buffer to be always at least
      * sizeof(clusterMsg) or more. */
@@ -3004,6 +3034,22 @@ void clusterSendPing(clusterLink *link, int type) {
         extensions++;
     }
 
+    /* Gossip forgotten nodes */
+    if (dictSize(server.cluster->nodes_black_list) > 0) {
+        dictIterator *di = dictGetIterator(server.cluster->nodes_black_list);
+        dictEntry *de;
+        while ((de = dictNext(di)) != NULL) {
+            sds name = dictGetKey(de);
+            uint64_t expire = dictGetUnsignedIntegerVal(de);
+            if ((time_t)expire < server.unixtime) continue; /* already expired */
+            uint64_t ttl = expire - server.unixtime;
+            hdr->mflags[0] |= CLUSTERMSG_FLAG0_EXT_DATA;
+            totlen += writeForgottenNodePingExt(&cursor, name, ttl);
+            extensions++;
+        }
+        dictReleaseIterator(di);
+    }
+
     /* Compute the actual total length and send! */
     totlen += sizeof(clusterMsg)-sizeof(union clusterMsgData);
     totlen += (sizeof(clusterMsgDataGossip)*gossipcount);
@@ -5612,7 +5658,12 @@ NULL
         /* CLUSTER FORGET <NODE ID> */
         clusterNode *n = clusterLookupNode(c->argv[2]->ptr, sdslen(c->argv[2]->ptr));
         if (!n) {
-            addReplyErrorFormat(c,"Unknown node %s", (char*)c->argv[2]->ptr);
+            if (clusterBlacklistExists((char*)c->argv[2]->ptr))
+                /* Already forgotten. The deletion may have been gossipped by
+                 * another node, so we pretend it succeeded. */
+                addReply(c,shared.ok);
+            else
+                addReplyErrorFormat(c,"Unknown node %s", (char*)c->argv[2]->ptr);
             return;
         } else if (n == myself) {
             addReplyError(c,"I tried hard but I can't forget myself...");
diff --git a/src/cluster.h b/src/cluster.h
index c50cfff8010..ba1b4edb472 100644
--- a/src/cluster.h
+++ b/src/cluster.h
@@ -253,6 +253,7 @@ typedef struct {
  * consistent manner. */
 typedef enum {
     CLUSTERMSG_EXT_TYPE_HOSTNAME,
+    CLUSTERMSG_EXT_TYPE_FORGOTTEN_NODE,
 } clusterMsgPingtypes; 
 
 /* Helper function for making sure extensions are eight byte aligned. */
@@ -262,12 +263,20 @@ typedef struct {
     char hostname[1]; /* The announced hostname, ends with \0. */
 } clusterMsgPingExtHostname;
 
+typedef struct {
+    char name[CLUSTER_NAMELEN]; /* Node name. */
+    uint64_t ttl; /* Remaining time to blacklist the node, in seconds. */
+} clusterMsgPingExtForgottenNode;
+
+static_assert(sizeof(clusterMsgPingExtForgottenNode) % 8 == 0, "");
+
 typedef struct {
     uint32_t length; /* Total length of this extension message (including this header) */
     uint16_t type; /* Type of this extension message (see clusterMsgPingExtTypes) */
     uint16_t unused; /* 16 bits of padding to make this structure 8 byte aligned. */
     union {
         clusterMsgPingExtHostname hostname;
+        clusterMsgPingExtForgottenNode forgotten_node;
     } ext[]; /* Actual extension information, formatted so that the data is 8 
               * byte aligned, regardless of its content. */
 } clusterMsgPingExt;


Test patch:
diff --git a/tests/unit/cluster/hostnames.tcl b/tests/unit/cluster/hostnames.tcl
index 124ceb643fb..02fb8361581 100644
--- a/tests/unit/cluster/hostnames.tcl
+++ b/tests/unit/cluster/hostnames.tcl
@@ -1,12 +1,25 @@
+# Returns 1 if no node knows node_id, 0 if any node knows it.
+proc node_is_forgotten {node_id} {
+    for {set j 0} {$j < [llength $::servers]} {incr j} {
+        set cluster_nodes [R $j CLUSTER NODES]
+        if { [string match "*$node_id*" $cluster_nodes] } {
+            return 0
+        }
+    }
+    return 1
+}
+
 # Isolate a node from the cluster and give it a new nodeid
 proc isolate_node {id} {
     set node_id [R $id CLUSTER MYID]
     R $id CLUSTER RESET HARD
-    for {set j 0} {$j < [llength $::servers]} {incr j} {
-        if { $j eq $id } {
-            continue
-        }
-        R $j CLUSTER FORGET $node_id
+    # Here we additionally test that CLUSTER FORGET propagates to all nodes.
+    set other_id [expr $id == 0 ? 1 : 0]
+    R $other_id CLUSTER FORGET $node_id
+    wait_for_condition 50 100 {
+        [node_is_forgotten $node_id]
+    } else {
+        fail "CLUSTER FORGET was not propagated to all nodes"
     }
 }