Bug: Cassandra non-seed node doing auto_bootstrap from other DC
Status: Triage Needed
Priority: Normal
Created: 01/Apr/20 21:47

I was always under the impression when using "NetworkTopologyStrategy" for a keyspace that bootstrapping a non-seed node would stream the data from the local DC.

In my test environment I'm busy adding a third node to dc2, with auto_bootstrap: true, and it is always streaming its data from dc1 nodes for some unexplained reason.

The user keyspace (~1GB of sstables for complete data set) has been created with:
{noformat}replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3', 'dc2': '3'}{noformat}

Nodetool status is showing:
{noformat}Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.30.29.60   622.13 MiB  256          63.4%             e26aa9c5-c4df-4015-9609-0b1337c90f06  rack1
UN  172.30.29.252  568.22 MiB  256          58.2%             16e0db7b-25b6-4316-8e8d-37ddb851e0af  rack1
UN  172.30.29.61   610.9 MiB  256          61.4%             51b7d47d-ef19-4ea2-b895-e01a26c2652d  rack1
UN  172.30.29.251  568.34 MiB  256          57.6%             22978c0d-989c-4cf7-a7e8-0c93d126fe44  rack1
UN  172.30.29.64   586.41 MiB  256          59.4%             fd39d945-205c-47d6-8ced-dc597028a7cd  rack1
Datacenter: dc2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.30.25.250  984.97 MiB  256          100.0%            8ee74a25-e7db-4044-bdbc-6fac20daa3b5  rack1
UN  172.30.29.248  984.95 MiB  256          100.0%            25d05594-95ce-4981-8169-6201ad37fc1e  rack1
{noformat}

My seed configuration:
{noformat}- seeds: "172.30.25.250,172.30.29.60,172.30.29.61,172.30.29.248"{noformat}

From the system.log:
{noformat}INFO  [STREAM-IN-/172.30.29.252:7000] 2020-04-01 23:27:35,124 StreamResultFuture.java:173 - [Stream #974ccf50-745f-11ea-b7ee-f9cc893c8823 ID#0] Prepare completed. Receiving 22 files(376.803MiB), sending 0 files(0.000KiB)
INFO  [STREAM-IN-/172.30.29.60:7000] 2020-04-01 23:27:36,150 StreamResultFuture.java:173 - [Stream #974ccf50-745f-11ea-b7ee-f9cc893c8823 ID#0] Prepare completed. Receiving 24 files(485.766MiB), sending 0 files(0.000KiB)
INFO  [STREAM-IN-/172.30.29.61:7000] 2020-04-01 23:27:36,657 StreamResultFuture.java:173 - [Stream #974ccf50-745f-11ea-b7ee-f9cc893c8823 ID#0] Prepare completed. Receiving 14 files(91.114MiB), sending 0 files(0.000KiB)
INFO  [STREAM-IN-/172.30.25.250:7000] 2020-04-01 23:27:36,749 StreamResultFuture.java:173 - [Stream #974ccf50-745f-11ea-b7ee-f9cc893c8823 ID#0] Prepare completed. Receiving 3 files(3.411KiB), sending 0 files(0.000KiB){noformat}

From the above log lines, it streamed my user keyspace of ~1GB from the dc1 nodes and some other small data from dc2.

I can repeat the condition by rm -rf /var/lib/cassandra3/data, performing nodetool removenode <nodeid> and then starting the node again.

Environment:
Cassandra 3.11.5 (also experienced it on 3.11.0) with GossipingPropertyFileSnitch, no initial token, num_tokens: 256
OpenJDK 1.8.0_232
CentOS 7