Repository: redis/redis

Issue:
[BUG]"corrupted cluster config file" on redis 7.2.3 error when running redis cluster with mixed 7.0 and 7.2 nodes
**Describe the bug**

"corrupted cluster config file" on redis 7.2 error when running redis cluster with mixed 7.0 and 7.2 nodes.

**To reproduce**

1. Create a 3 node cluster with redis 7.0 (I've used 7.0.14)
2. Create 3 slave nodes with redis 7.2 (I've used 7.2.3)
3. Stop any of the 7.2 nodes and try to start it again. It fails with the following error:

> 70878:M 14 Nov 2023 09:47:27.115 # Unrecoverable error: corrupted cluster config file "c5cb6e214d955fe19cb2eb2d5d3e8a35a165f7d2 127.0.0.1:6381@16381,,tls-port=0,shard-id=f5de5cc8bc87f66210d34f1017149ebd89e03ad7 master - 0 1699951301788 3 connected 10923-16383
".

**Expected behavior**

7.2 nodes can be restarted and can rejoin the cluster.

**Additional information**

I've came across this problem when trying to update one of my 7.0 cluster to 7.2. I've updated several slaves, as usual. Everything seemed to work fine, until I had to restart one of them and it failed. So it seems that the rolling update from 7.0 to 7.2 is right now impossible, but maybe I'm doing something wrong?

Corrupted config file generated by redis 7.2 (6379-6381 nodes are 7.0, 36379-36381 are 7.2):

> 8f542d6e23d29a9106e4d5db7c028dafd67d4649 127.0.0.1:6380@16380,,tls-port=0,shard-id=502af3db7f3f11593bb9754e9fd4f58e309cd719 master - 0 1699951299000 2 connected 5461-10922
f83290999eab469341f81dea9a783eaaec18b96a 127.0.0.1:6379@16379,,tls-port=0,shard-id=653bfbe1b8b4f698f81e93c173a04fb886641e51 master - 0 1699951300780 1 connected 0-5460
9662b7ddd38a17cd7a294e1c2bac692c5fc1cbcc 127.0.0.1:36381@46381,,tls-port=0,shard-id=a6af4619eec99aabd7c4b9e4ccc5070f43f642b8 slave,fail c5cb6e214d955fe19cb2eb2d5d3e8a35a165f7d2 1699951288673 1699951282616 3 disconnected
c5cb6e214d955fe19cb2eb2d5d3e8a35a165f7d2 127.0.0.1:6381@16381,,tls-port=0,shard-id=f5de5cc8bc87f66210d34f1017149ebd89e03ad7 master - 0 1699951301788 3 connected 10923-16383
0eabd0c4889de11a25ac373c66251e5021d157a0 127.0.0.1:36380@46380,,tls-port=0,shard-id=98461c28bba1a38fdb7100cba6e98eacbb95e97b slave 8f542d6e23d29a9106e4d5db7c028dafd67d4649 0 1699951300000 2 connected
bd1667d21001886891be7def40c298d6873967ce 127.0.0.1:36379@46379,,tls-port=0,shard-id=653bfbe1b8b4f698f81e93c173a04fb886641e51 myself,slave f83290999eab469341f81dea9a783eaaec18b96a 0 1699951301000 1 connected
vars currentEpoch 3 lastVoteEpoch 0

I've also used gdb to check why it is failing and it seems to fail in this place when trying to parse shard-id field:
https://github.com/redis/redis/blob/7f4bae817614988c43c3024402d16edcbf3b3277/src/cluster.c#L502


Discussion:
@madolson this is a issue i mentioned in https://github.com/redis/redis/pull/12604#issuecomment-1759100649
do you have any ideas how should we fix it?

Patch:
diff --git a/src/cluster.h b/src/cluster.h
index 0cf71ad4dd7..1856cd41741 100644
--- a/src/cluster.h
+++ b/src/cluster.h
@@ -89,6 +89,7 @@ int clusterNodeIsMaster(clusterNode *n);
 char *clusterNodeIp(clusterNode *node);
 int clusterNodeIsSlave(clusterNode *node);
 clusterNode *clusterNodeGetSlaveof(clusterNode *node);
+clusterNode *clusterNodeGetMaster(clusterNode *node);
 char *clusterNodeGetName(clusterNode *node);
 int clusterNodeTimedOut(clusterNode *node);
 int clusterNodeIsFailing(clusterNode *node);
diff --git a/src/cluster_legacy.c b/src/cluster_legacy.c
index 0f20cbdbb12..e36378d84a6 100644
--- a/src/cluster_legacy.c
+++ b/src/cluster_legacy.c
@@ -2596,11 +2596,23 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
         /* We know this will be valid since we validated it ahead of time */
         ext = getNextPingExt(ext);
     }
+
     /* If the node did not send us a hostname extension, assume
      * they don't have an announced hostname. Otherwise, we'll
      * set it now. */
     updateAnnouncedHostname(sender, ext_hostname);
     updateAnnouncedHumanNodename(sender, ext_humannodename);
+    /* If the node did not send us a shard-id extension, it means the sender
+     * does not support it (old version), node->shard_id is randomly generated.
+     * A cluster-wide consensus for the node's shard_id is not necessary.
+     * The key is maintaining consistency of the shard_id on each individual 7.2 node.
+     * As the cluster progressively upgrades to version 7.2, we can expect the shard_ids
+     * across all nodes to naturally converge and align.
+     *
+     * If sender is a replica, set the shard_id to the shard_id of its master.
+     * Otherwise, we'll set it now. */
+    if (ext_shardid == NULL) ext_shardid = clusterNodeGetMaster(sender)->shard_id;
+
     updateShardId(sender, ext_shardid);
 }
 
@@ -5543,7 +5555,7 @@ void addShardReplyForClusterShards(client *c, list *nodes) {
     addReplyBulkCString(c, "slots");
 
     /* Use slot_info_pairs from the primary only */
-    while (n->slaveof != NULL) n = n->slaveof;
+    n = clusterNodeGetMaster(n);
 
     if (n->slot_info_pairs != NULL) {
         serverAssert((n->slot_info_pairs_count % 2) == 0);
@@ -5856,6 +5868,11 @@ clusterNode *clusterNodeGetSlaveof(clusterNode *node) {
     return node->slaveof;
 }
 
+clusterNode *clusterNodeGetMaster(clusterNode *node) {
+    while (node->slaveof != NULL) node = node->slaveof;
+    return node;
+}
+
 char *clusterNodeGetName(clusterNode *node) {
     return node->name;
 }