Bug: TableSkew cost scales beyond 1
Status: Open | Priority: Major | Resolution: 
Created: 06/Sep/24 11:20
Affects Version/s: 2.5.7

This may already be covered by later releases, but we noticed that the table skew cost function can produce cost values beyond 1. In our case with over 1000 tables caused the table skew cost to suppress the region count skew (and other) cost functions.

I think this is because the cost per table are 'simply' summed in TableSkewCostFunction#cost. So if the number of tables with skew is large, this cost function may cause the balancer to favour actions that decrease this cost at to big of an expense of other costs such as region count skew.

Logging from the HBase master that shows this:
{code:java}
[...] balancer.StochasticLoadBalancer: dBalancer.balancer, initial weighted average imbalance=0.25500371101846336, functionCost=RegionCountSkewCostFunction : (multiplier=100000.0, imbalance=0.24272066309658274, need balance); PrimaryRegionCountSkewCostFunction : (not needed); MoveCostFunction : (multiplier=7.0, imbalance=0.0); ServerLocalityCostFunction : (multiplier=25.0, imbalance=0.6022498608833904, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=35.24784226006047, need balance); RegionReplicaHostCostFunction : (not needed); RegionReplicaRackCostFunction : (not needed); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.24057323733439073, need balance); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.3233739875438904, need balance); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.3195880383071082, need balance); StoreFileCostFunction : (multiplier=5.0, imbalance=0.23335375436276784, need balance);  computedMaxSteps=1000000 {code}
Note the {{TableSkewCostFunction : (multiplier=35.0, imbalance=35.24784226006047)}} part.

In order to work-around this we temporarily reduced the multiplier of the table skew cost function to 0.

The test case below fails on HBase the 2.5 and 2.6 branches. It simply assigns two tables with two regions each to a single server.
{code:java}
@Test
public void testTableSkewCost() {
  TableName t1 = TableName.valueOf("t1");
  TableName t2 = TableName.valueOf("t2");

  TreeMap<ServerName, List<RegionInfo>> clusterState = new TreeMap<>();
  clusterState.put(ServerName.valueOf("n1", 16020,0), Arrays.asList(
    RegionInfoBuilder.newBuilder(t1).setRegionId(11).build(),
    RegionInfoBuilder.newBuilder(t1).setRegionId(12).build()
  ));
  clusterState.put(ServerName.valueOf("n2", 16020,0), Arrays.asList(
    RegionInfoBuilder.newBuilder(t2).setRegionId(21).build(),
    RegionInfoBuilder.newBuilder(t2).setRegionId(22).build()
  ));

  BalancerClusterState cluster = new BalancerClusterState(clusterState, null, null, null);

  Configuration conf = HBaseConfiguration.create();
  CostFunction costFunction = new TableSkewCostFunction(conf);
  costFunction.prepare(cluster);
  double cost = costFunction.cost();
  assertTrue(cost >= 0);
  assertTrue(cost <= 1.01);
} {code}
 

It's the second assertion that fails since the computed cost for this cluster state is 2.

I guess none of the existing cluster state test/mock configurations have a real table skew.