Repository: scylladb/scylla-cdc-java
Pull request #88: Make load balancing policy token aware when local-dc is provided
Author: login: dkropachev
Labels: name: bug
description: Something isn't working
name: enhancement
description: New feature or request
Created: 2025-03-05T05:29:28Z | Merged: 2025-03-05T12:43:44Z

Following changes:
1. Make load balancing policy token aware when local-dc is provided, it used to be just dc-aware, probably by mistake
2. Make default load balancing policy cluster-wide token aware, before it was token and dc-aware, with random dc.
3. Implement `localRackName` option that allows to target particular rack
4. Implement `replicaOrdering` to change how token-aware load balancing policy iterates over replicas.


Diff:
diff --git a/scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/CQLConfiguration.java b/scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/CQLConfiguration.java
index cf71bfef..8a27a8b4 100644
--- a/scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/CQLConfiguration.java
+++ b/scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/CQLConfiguration.java
@@ -63,12 +63,14 @@ public enum ConsistencyLevel {
     public final String password;
     private final ConsistencyLevel consistencyLevel;
     private final String localDCName;
+    private final String localRackName;
+    private final ReplicaOrdering replicaOrdering;
     public final SslConfig sslConfig;
     public final int queryOptionsFetchSize;
 
     private CQLConfiguration(List<InetSocketAddress> contactPoints,
                             String user, String password, ConsistencyLevel consistencyLevel,
-                            String localDCName, SslConfig sslConfig, int queryOptionsFetchSize) {
+                            String localDCName, String localRackName, ReplicaOrdering replicaOrdering, SslConfig sslConfig, int queryOptionsFetchSize) {
         this.contactPoints = Preconditions.checkNotNull(contactPoints);
         Preconditions.checkArgument(!contactPoints.isEmpty());
 
@@ -81,6 +83,8 @@ private CQLConfiguration(List<InetSocketAddress> contactPoints,
 
         this.consistencyLevel = Preconditions.checkNotNull(consistencyLevel);
         this.localDCName = localDCName;
+        this.localRackName = localRackName;
+        this.replicaOrdering = Preconditions.checkNotNull(replicaOrdering);
         this.sslConfig = sslConfig;
         this.queryOptionsFetchSize = queryOptionsFetchSize;
     }
@@ -114,6 +118,33 @@ public String getLocalDCName() {
         return localDCName;
     }
 
+    /**
+     * Returns the name of the configured local rack.
+     * <p>
+     * This local rack name will be used to setup
+     * the connection to Scylla to prioritize sending requests to
+     * the nodes in the local rack (in the local datacenter). If this parameter
+     * was not configured, this method returns <code>null</code>.
+     *
+     * @return the name of configured local rack or
+     * <code>null</code> if it was not configured.
+     */
+    public String getLocalRackName() {
+        return localRackName;
+    }
+
+    /**
+     * Returns replica ordering.
+     * <p>
+     * Replica ordering defines how CQL driver iterates over data replicas
+     * when reads from CDC tables.
+     *
+     * @return replica ordering
+     */
+    public ReplicaOrdering getReplicaOrdering() {
+        return replicaOrdering;
+    }
+
     public static Builder builder() {
         return new Builder();
     }
@@ -124,6 +155,8 @@ public static class Builder {
         private String password = null;
         private ConsistencyLevel consistencyLevel = DEFAULT_CONSISTENCY_LEVEL;
         private String localDCName = null;
+        private String localRackName = null;
+        private ReplicaOrdering replicaOrdering = ReplicaOrdering.RANDOM;
         private SslConfig sslConfig = null;
         private int queryOptionsFetchSize = 0;
 
@@ -187,6 +220,35 @@ public Builder withLocalDCName(String localDCName) {
             return this;
         }
 
+        /**
+         * Sets the name of local rack.
+         * <p>
+         * This local rack name will be used to setup
+         * the connection to Scylla to prioritize sending requests to
+         * the nodes in the local rack (in local datacenter).
+         *
+         * @param localRackName the name of local rack to set.
+         * @return a reference to this builder.
+         */
+        public Builder withLocalRackName(String localRackName) {
+            this.localRackName = Preconditions.checkNotNull(localRackName);
+            return this;
+        }
+
+        /**
+         * Sets the replica ordering for load balancing policy.
+         * <p>
+         * It allows to change the way reader iterates over data replicas
+         * when read data from
+         *
+         * @param replicaOrdering replica ordering to set.
+         * @return a reference to this builder.
+         */
+        public Builder withReplicaOrdering(ReplicaOrdering replicaOrdering) {
+            this.replicaOrdering = Preconditions.checkNotNull(replicaOrdering);
+            return this;
+        }
+
         public Builder withSslConfig(SslConfig sslConfig) {
             this.sslConfig = sslConfig;
             return this;
@@ -204,7 +266,7 @@ public Builder withQueryOptionsFetchSize(int fetchSize) {
         }
 
         public CQLConfiguration build() {
-            return new CQLConfiguration(contactPoints, user, password, consistencyLevel, localDCName, sslConfig, queryOptionsFetchSize);
+            return new CQLConfiguration(contactPoints, user, password, consistencyLevel, localDCName, localRackName, replicaOrdering, sslConfig, queryOptionsFetchSize);
         }
     }
 }
\ No newline at end of file
diff --git a/scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/ReplicaOrdering.java b/scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/ReplicaOrdering.java
new file mode 100644
index 00000000..a6a8a96a
--- /dev/null
+++ b/scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/ReplicaOrdering.java
@@ -0,0 +1,12 @@
+package com.scylladb.cdc.cql;
+
+/**
+ A copy-cat TokenAwarePolicy.ReplicaOrdering of the driver
+ Needed here not to leak driver types
+ */
+public enum ReplicaOrdering {
+	TOPOLOGICAL,
+	RANDOM,
+	NEUTRAL;
+}
+
diff --git a/scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java b/scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
index 5f6851df..1b942377 100644
--- a/scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
+++ b/scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
@@ -2,6 +2,9 @@
 
 import com.datastax.driver.core.*;
 import com.datastax.driver.core.policies.DCAwareRoundRobinPolicy;
+import com.datastax.driver.core.policies.RackAwareRoundRobinPolicy;
+import com.datastax.driver.core.policies.RoundRobinPolicy;
+import com.datastax.driver.core.policies.TokenAwarePolicy;
 import com.scylladb.cdc.cql.CQLConfiguration;
 import com.scylladb.cdc.cql.SslConfig;
 import io.netty.handler.ssl.SslContext;
@@ -111,9 +114,35 @@ public Driver3Session(CQLConfiguration cqlConfiguration) {
             clusterBuilder = clusterBuilder.withCredentials(user, password);
         }
 
-        if (cqlConfiguration.getLocalDCName() != null) {
+        TokenAwarePolicy.ReplicaOrdering replicaOrdering;
+        switch (cqlConfiguration.getReplicaOrdering()) {
+            case NEUTRAL:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.NEUTRAL;
+                break;
+            case TOPOLOGICAL:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.TOPOLOGICAL;
+                break;
+            default:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.RANDOM;
+                break;
+        }
+
+        if (cqlConfiguration.getLocalRackName() != null) {
+            RackAwareRoundRobinPolicy.Builder builder = RackAwareRoundRobinPolicy.builder().withLocalRack(cqlConfiguration.getLocalRackName());
+            if (cqlConfiguration.getLocalDCName() != null) {
+                builder = builder.withLocalDc(cqlConfiguration.getLocalDCName());
+            }
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(builder.build(), replicaOrdering)
+            );
+        } else if (cqlConfiguration.getLocalDCName() != null) {
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(cqlConfiguration.getLocalDCName()).build(), replicaOrdering)
+            );
+        } else {
             clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
-                    DCAwareRoundRobinPolicy.builder().withLocalDc(cqlConfiguration.getLocalDCName()).build());
+                new TokenAwarePolicy(new RoundRobinPolicy(), replicaOrdering)
+            );
         }
 
         if (cqlConfiguration.queryOptionsFetchSize > 0) {


Reviews:
user: login: Bouncheck
body: There's at least 1 unused import introduced
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T09:20:07Z
user: login: dkropachev
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T09:49:33Z
user: login: dkropachev
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T10:07:19Z
user: login: Bouncheck
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T10:33:59Z
user: login: Bouncheck
body: Code LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T10:44:52Z
user: login: dkropachev
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T12:43:06Z
user: login: fee-mendes
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T13:40:36Z
user: login: fee-mendes
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T13:45:34Z

Review comments:
diff_hunk: @@ -1,6 +1,7 @@
 package com.scylladb.cdc.cql;
 
 import com.google.common.base.Preconditions;
+import org.checkerframework.checker.nullness.qual.NonNull;
path: scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/CQLConfiguration.java
user: login: Bouncheck
body: unused import
created_at: 2025-03-05T08:36:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -116,6 +117,10 @@ public Driver3Session(CQLConfiguration cqlConfiguration) {
             clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
                 new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(cqlConfiguration.getLocalDCName()).build())
             );
+        } else {
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(new RoundRobinPolicy())
path: scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
user: login: Bouncheck
body: Why not DCAware in this case like it was previously? Or the same as above without explicitly set local dc?
created_at: 2025-03-05T09:07:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -116,6 +117,10 @@ public Driver3Session(CQLConfiguration cqlConfiguration) {
             clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
                 new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(cqlConfiguration.getLocalDCName()).build())
             );
+        } else {
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(new RoundRobinPolicy())
path: scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
user: login: dkropachev
body: If you do that then how do you make it target whole cluster ?
created_at: 2025-03-05T09:49:33Z
author_association: COLLABORATOR
diff_hunk: @@ -1,6 +1,7 @@
 package com.scylladb.cdc.cql;
 
 import com.google.common.base.Preconditions;
+import org.checkerframework.checker.nullness.qual.NonNull;
path: scylla-cdc-base/src/main/java/com/scylladb/cdc/cql/CQLConfiguration.java
user: login: dkropachev
body: removed
created_at: 2025-03-05T10:07:18Z
author_association: COLLABORATOR
diff_hunk: @@ -116,6 +117,10 @@ public Driver3Session(CQLConfiguration cqlConfiguration) {
             clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
                 new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(cqlConfiguration.getLocalDCName()).build())
             );
+        } else {
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(new RoundRobinPolicy())
path: scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
user: login: Bouncheck
body: You cannot, but I'm not aware of cases where someone would want to target remote DCs over local (maybe there are).
created_at: 2025-03-05T10:33:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -116,6 +117,10 @@ public Driver3Session(CQLConfiguration cqlConfiguration) {
             clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
                 new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(cqlConfiguration.getLocalDCName()).build())
             );
+        } else {
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(new RoundRobinPolicy())
path: scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
user: login: dkropachev
body: it is possible if they want to get more throughput
created_at: 2025-03-05T12:43:06Z
author_association: COLLABORATOR
diff_hunk: @@ -111,9 +114,35 @@ public Driver3Session(CQLConfiguration cqlConfiguration) {
             clusterBuilder = clusterBuilder.withCredentials(user, password);
         }
 
-        if (cqlConfiguration.getLocalDCName() != null) {
+        TokenAwarePolicy.ReplicaOrdering replicaOrdering;
+        switch (cqlConfiguration.getReplicaOrdering()) {
+            case NEUTRAL:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.NEUTRAL;
+                break;
+            case TOPOLOGICAL:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.TOPOLOGICAL;
+                break;
+            default:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.RANDOM;
+                break;
+        }
+
+        if (cqlConfiguration.getLocalRackName() != null) {
+            RackAwareRoundRobinPolicy.Builder builder = RackAwareRoundRobinPolicy.builder().withLocalRack(cqlConfiguration.getLocalRackName());
+            if (cqlConfiguration.getLocalDCName() != null) {
path: scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
user: login: fee-mendes
body: Orthogonal to this PR, I would like to point out that `getLocalDCName()` should be enforced, instead of optional.

Our core driver `RackAwareRoundRobinPolicy` is too _kind_ on figuring out the `localDC` name given the provided contact point found as part of the drivers' initial control connection. If `localDC` is unset, it prints an info message back to the caller stating the fact and uses whichever `localDC` it found as part of the initial connection process. 

However, I feel a bit concerned of any specific situation where one would call `RackAwareRoundRobinPolicy` **without** specifying a DC name and actually make it work as intended. To do that, an application would need to dynamically specify contact point addresses on a per DC basis, which is very unlikely. Most applications just hardcode the same contact points through their lifetime and only change it when something goes wrong.

Instead, the correct way for an app to rely on this load balancing strategy is to always specify the localDc **and** rack to connect to and reject everything else, in turn making the contact points provided irrelevant. The placement should always be inferred by the app: it queries the instance/pod metadata and set the `rack` + `localDc` combination from there. 

You're probably correct to assume this is a _minor_ issue. It is minor because most deployments out there simply span 3 AZs and have a single DC. However, remember that a DC by definition is only a logical abstraction. You may see (and we have it) deployments out there with a setup where a DC represents replicas in a single AZ, and each rack under that AZ is logically set to hardcoded name(s). Under these topologies, you end up with too much room for mistakes, when we should simply throw instead.

A user mistake implies that we aren't sure whether the `RackAwareness` is working as intended. If we can't guarantee it, we would rather save the user from continuing to burn money on cross-AZ costs, which is the primary (and perhaps only) reason why one would use this policy in the first place. 

In fact, if I'm reading the policy code correctly, we don't even enforce the `localRack` to be specified, which also seems wrong to me for the very same reasons. :)
created_at: 2025-03-05T13:40:36Z
author_association: MEMBER
diff_hunk: @@ -111,9 +114,35 @@ public Driver3Session(CQLConfiguration cqlConfiguration) {
             clusterBuilder = clusterBuilder.withCredentials(user, password);
         }
 
-        if (cqlConfiguration.getLocalDCName() != null) {
+        TokenAwarePolicy.ReplicaOrdering replicaOrdering;
+        switch (cqlConfiguration.getReplicaOrdering()) {
+            case NEUTRAL:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.NEUTRAL;
+                break;
+            case TOPOLOGICAL:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.TOPOLOGICAL;
+                break;
+            default:
+                replicaOrdering = TokenAwarePolicy.ReplicaOrdering.RANDOM;
+                break;
+        }
+
+        if (cqlConfiguration.getLocalRackName() != null) {
+            RackAwareRoundRobinPolicy.Builder builder = RackAwareRoundRobinPolicy.builder().withLocalRack(cqlConfiguration.getLocalRackName());
+            if (cqlConfiguration.getLocalDCName() != null) {
+                builder = builder.withLocalDc(cqlConfiguration.getLocalDCName());
+            }
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(builder.build(), replicaOrdering)
+            );
+        } else if (cqlConfiguration.getLocalDCName() != null) {
+            clusterBuilder = clusterBuilder.withLoadBalancingPolicy(
+                new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(cqlConfiguration.getLocalDCName()).build(), replicaOrdering)
+            );
+        } else {
path: scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3Session.java
user: login: fee-mendes
body: Yeah, this is arguable. There is no problem with is per see, but we could simply enforce a localDC to be specified because... why not?

The notion of whether querying remoteDCs or not is purely a matter of using a non local consistency level. That said, I don't see anything wrong with it - but a warning would be nice so users and the field could troubleshoot when they use the default. 
created_at: 2025-03-05T13:45:33Z
author_association: MEMBER

PR comments:
user: login: Bouncheck
created_at: 2025-03-05T09:21:55Z
body: It's probably hard to say without benchmarks but do we want to expose LBP configuration to the user? Querying CDC streams is already rather specific workload so maybe it would be better to just choose one that performs the best and hide this detail from the user.
This also avoids the hack with reimplementing ReplicaOrdering enums in cdc library.
author_association: CONTRIBUTOR
user: login: dkropachev
created_at: 2025-03-05T09:55:43Z
body: > It's probably hard to say without benchmarks but do we want to expose LBP configuration to the user? Querying CDC streams is already rather specific workload so maybe it would be better to just choose one that performs the best and hide this detail from the user. This also avoids the hack with reimplementing ReplicaOrdering enums in cdc library.

Benchmark results could comeup different for different setups, how many dcs, how many racks, are you using DC-aware / Rack-aware policy.
What is the replication factor.
All that contributes to the results.

Probably when we introduce proper benchmarks we can run all scenarios and default to something that is most productive in most of the cases.

As you know certain replica ordering can provide improper behavior in some scenarios when used with tablets.
Let's make it possible to confgure it.
Default `RANDOM` - still the same.

author_association: COLLABORATOR
user: login: Bouncheck
created_at: 2025-03-05T10:47:12Z
body: > As you know certain replica ordering can provide improper behavior in some scenarios when used with tablets. Let's make it possible to confgure it. Default `RANDOM` - still the same.

I believe that is because of the bugged combinations on the driver side. On server side there is also issue that low number of tablets can lead to imbalances IIRC (but that may have been resolved somehow).


author_association: CONTRIBUTOR
user: login: fee-mendes
created_at: 2025-03-05T13:46:39Z
body: I see this is merged so I just left some words for thought and bother you people with annoying email notifications.
author_association: MEMBER