Repository: apache/hudi
Pull request #13265: [HUDI-9379] Spark RemotePartitioner For BucketIndex
Author: login: zhangyue19921010
Labels: name: size:M
description: PR with lines of changes in (100, 300]
Created: 2025-05-06T04:04:32Z | Merged: 2025-05-19T03:44:12Z

### Change Logs

**Background**
Spark Bulk insert Bucket index Table，will use
```
public static Functions.Function3<Integer, String, Integer, Integer> getPartitionIndexFunc(int parallelism) {
    return (bucketNum, partition, curBucket) -> {
      long partitionIndex = (partition.hashCode() & Integer.MAX_VALUE) % parallelism * (long) bucketNum;
      long globalIndex = partitionIndex + curBucket;
      int partitionId = (int) (globalIndex % parallelism);
      ValidationUtils.checkArgument(partitionId >= 0 && partitionId < parallelism,
          () -> "Partition id should be in range [0, " + parallelism + "), but got " + partitionId);
      return partitionId;
    };
  }

```
For a given partition and BucketNumber, calculating the Spark Partition number based on Hash and Mode during repartition operations offers the advantage of high performance and simple logic. However, its drawbacks include inevitable data skew and an inability to fully utilize all task resources. This results in poor performance for large-scale data initialization scenarios. For example, in our full product table scenario with 1.2 trillion existing records and 20,000 buckets, initialization takes 8 hours. Due to data skew limitations, scaling resources cannot further accelerate the process.

**Proposed Solution**
![image](https://github.com/user-attachments/assets/a5ea64be-1119-4b83-b7ab-53eae66d57db)

We propose a RemotePartitioner algorithm leveraging Hudi's built-in Javalin API service. This RemotePartitioner requests the current data's partition-specific BucketStartIndex from the Driver (cached with two-level caching at both Driver and Executor levels) and combines it with the computed bucketID to determine the target SparkPartitionNumber. The total number of requests equals SparkTaskNumber × DataPartitionNumber, ensuring overall controllability.

**Results**
This approach completely resolves data skew issues. The aforementioned task was optimized from 8 hours to 2 hours, with performance remaining continuously scalable.
![image](https://github.com/user-attachments/assets/ca4813ab-ebf8-4f44-a9d8-c72b97ad226b)

### Impact

Spark write bucket index table using bulkinsert operation

### Risk level (write none, low medium or high below)

low

### Documentation Update

_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._

- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
  changes to the website._

### Contributor's checklist

- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed



Diff:
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java
index e9c7f5ca8b906..d92334756dfbd 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java
@@ -170,6 +170,10 @@ private void startServer(TimelineServiceCreator timelineServiceCreator) throws I
                   * writeConfig.getHoodieClientHeartbeatTolerableMisses());
     }
 
+    if (writeConfig.isUsingRemotePartitioner()) {
+      timelineServiceConfBuilder.enableRemotePartitioner(true);
+    }
+
     this.serviceConfig = timelineServiceConfBuilder.build();
 
     server = timelineServiceCreator.create(context, storageConf.newInstance(), serviceConfig, viewManager);
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java
index 9aac9cf353499..1932e63cd3103 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java
@@ -301,6 +301,12 @@ public class HoodieIndexConfig extends HoodieConfig {
       .withDocumentation("Only applies if index type is BUCKET. Determine the number of buckets in the hudi table, "
           + "and each partition is divided to N buckets.");
 
+  public static final ConfigProperty<Boolean> BUCKET_PARTITIONER = ConfigProperty
+      .key("hoodie.bucket.index.remote.partitioner.enable")
+      .defaultValue(false)
+      .withDocumentation("Use Remote Partitioner using centralized allocation of partition "
+          + "IDs to do repartition based on bucket aiming to resolve data skew. Default local hash partitioner");
+
   public static final ConfigProperty<String> BUCKET_INDEX_PARTITION_RULE_TYPE = ConfigProperty
       .key("hoodie.bucket.index.partition.rule.type")
       .defaultValue(PartitionBucketIndexRule.REGEX.name)
@@ -728,6 +734,11 @@ public Builder withBucketMaxNum(int bucketMaxNum) {
       return this;
     }
 
+    public Builder enableBucketRemotePartitioner(boolean enableRemotePartitioner) {
+      hoodieIndexConfig.setValue(BUCKET_PARTITIONER, String.valueOf(enableRemotePartitioner));
+      return this;
+    }
+
     public Builder withBucketMinNum(int bucketMinNum) {
       hoodieIndexConfig.setValue(BUCKET_INDEX_MIN_NUM_BUCKETS, String.valueOf(bucketMinNum));
       return this;
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
index f69a8fd25bad4..922ba35028487 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
@@ -2265,6 +2265,10 @@ public boolean getRecordIndexUpdatePartitionPath() {
   public String getRecordIndexInputStorageLevel() {
     return getStringOrDefault(HoodieIndexConfig.RECORD_INDEX_INPUT_STORAGE_LEVEL_VALUE);
   }
+  
+  public boolean isUsingRemotePartitioner() {
+    return getBoolean(HoodieIndexConfig.BUCKET_PARTITIONER);
+  }
 
   /**
    * storage properties.
diff --git a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BucketIndexBulkInsertPartitionerWithRows.java b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BucketIndexBulkInsertPartitionerWithRows.java
index db7a64de69fe0..9816febb04145 100644
--- a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BucketIndexBulkInsertPartitionerWithRows.java
+++ b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BucketIndexBulkInsertPartitionerWithRows.java
@@ -19,10 +19,12 @@
 package org.apache.hudi.execution.bulkinsert;
 
 import org.apache.hudi.common.model.PartitionBucketIndexHashingConfig;
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.index.bucket.partition.NumBucketsFunction;
 import org.apache.hudi.table.BulkInsertPartitioner;
 
+import org.apache.spark.Partitioner;
 import org.apache.spark.sql.BucketPartitionUtils$;
 import org.apache.spark.sql.Dataset;
 import org.apache.spark.sql.Row;
@@ -34,21 +36,34 @@ public class BucketIndexBulkInsertPartitionerWithRows implements BulkInsertParti
 
   private final String indexKeyFields;
   private final NumBucketsFunction numBucketsFunction;
+  private final HoodieWriteConfig writeConfig;
+  private FileSystemViewStorageConfig viewConfig;
 
   public BucketIndexBulkInsertPartitionerWithRows(String indexKeyFields, HoodieWriteConfig writeConfig) {
     this.indexKeyFields = indexKeyFields;
     this.numBucketsFunction = NumBucketsFunction.fromWriteConfig(writeConfig);
+    this.writeConfig = writeConfig;
+    if (writeConfig.isUsingRemotePartitioner()) {
+      this.viewConfig = writeConfig.getViewStorageConfig();
+    }
   }
 
-  public BucketIndexBulkInsertPartitionerWithRows(String indexKeyFields, PartitionBucketIndexHashingConfig hashingConfig) {
-    this.indexKeyFields = indexKeyFields;
+  public BucketIndexBulkInsertPartitionerWithRows(HoodieWriteConfig writeConfig, PartitionBucketIndexHashingConfig hashingConfig) {
+    this.indexKeyFields = writeConfig.getBucketIndexHashFieldWithDefault();
     this.numBucketsFunction = new NumBucketsFunction(hashingConfig.getExpressions(),
         hashingConfig.getRule(), hashingConfig.getDefaultBucketNumber());
+    this.writeConfig = writeConfig;
+    if (writeConfig.isUsingRemotePartitioner()) {
+      this.viewConfig = writeConfig.getViewStorageConfig();
+    }
   }
 
   @Override
   public Dataset<Row> repartitionRecords(Dataset<Row> rows, int outputPartitions) {
-    return BucketPartitionUtils$.MODULE$.createDataFrame(rows, indexKeyFields, numBucketsFunction, outputPartitions);
+    Partitioner partitioner = writeConfig.isUsingRemotePartitioner() && writeConfig.isEmbeddedTimelineServerEnabled()
+        ? BucketPartitionUtils$.MODULE$.getRemotePartitioner(viewConfig, numBucketsFunction, outputPartitions) 
+        : BucketPartitionUtils$.MODULE$.getLocalePartitioner(numBucketsFunction, outputPartitions);
+    return BucketPartitionUtils$.MODULE$.createDataFrame(rows, indexKeyFields, numBucketsFunction, partitioner);
   }
 
   @Override
diff --git a/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/BucketPartitionUtils.scala b/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/BucketPartitionUtils.scala
index ecdf406871f36..9e3784cf0ca00 100644
--- a/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/BucketPartitionUtils.scala
+++ b/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/BucketPartitionUtils.scala
@@ -19,7 +19,8 @@
 package org.apache.spark.sql
 
 import org.apache.hudi.common.model.HoodieRecord
-import org.apache.hudi.common.util.Functions
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig
+import org.apache.hudi.common.util.{Functions, RemotePartitionHelper}
 import org.apache.hudi.common.util.hash.BucketIndexUtil
 import org.apache.hudi.index.bucket.BucketIdentifier
 import org.apache.hudi.index.bucket.partition.NumBucketsFunction
@@ -28,7 +29,7 @@ import org.apache.spark.Partitioner
 import org.apache.spark.sql.catalyst.InternalRow
 
 object BucketPartitionUtils {
-  def createDataFrame(df: DataFrame, indexKeyFields: String, numBucketsFunction: NumBucketsFunction, partitionNum: Int): DataFrame = {
+  def createDataFrame(df: DataFrame, indexKeyFields: String, numBucketsFunction: NumBucketsFunction, partitioner: Partitioner): DataFrame = {
     def getPartitionKeyExtractor(): InternalRow => (String, Int) = row => {
       val partition = row.getString(HoodieRecord.PARTITION_PATH_META_FIELD_ORD)
       val kb = BucketIdentifier
@@ -42,8 +43,29 @@ object BucketPartitionUtils {
     }
 
     val getPartitionKey = getPartitionKeyExtractor()
-    val partitioner = new Partitioner {
+    // use internalRow to avoid extra convert.
+    val reRdd = df.queryExecution.toRdd
+      .keyBy(row => getPartitionKey(row))
+      .repartitionAndSortWithinPartitions(partitioner)
+      .values
+    df.sparkSession.internalCreateDataFrame(reRdd, df.schema)
+  }
+
+  def getRemotePartitioner(viewConf: FileSystemViewStorageConfig, numBucketsFunction: NumBucketsFunction, partitionNum: Int): Partitioner = {
+    new Partitioner {
+      private val helper = new RemotePartitionHelper(viewConf)
+
+      override def numPartitions: Int = partitionNum
+
+      override def getPartition(value: Any): Int = {
+        val partitionKeyPair = value.asInstanceOf[(String, Int)]
+        helper.getPartition(numBucketsFunction.getNumBuckets(partitionKeyPair._1), partitionKeyPair._1, partitionKeyPair._2, partitionNum)
+      }
+    }
+  }
 
+  def getLocalePartitioner(numBucketsFunction: NumBucketsFunction, partitionNum: Int): Partitioner = {
+    new Partitioner {
       private val partitionIndexFunc: Functions.Function3[Integer, String, Integer, Integer] =
         BucketIndexUtil.getPartitionIndexFunc(partitionNum)
 
@@ -54,11 +76,5 @@ object BucketPartitionUtils {
         partitionIndexFunc.apply(numBucketsFunction.getNumBuckets(partitionKeyPair._1), partitionKeyPair._1, partitionKeyPair._2)
       }
     }
-    // use internalRow to avoid extra convert.
-    val reRdd = df.queryExecution.toRdd
-      .keyBy(row => getPartitionKey(row))
-      .repartitionAndSortWithinPartitions(partitioner)
-      .values
-    df.sparkSession.internalCreateDataFrame(reRdd, df.schema)
   }
 }
diff --git a/hudi-common/src/main/java/org/apache/hudi/common/util/RemotePartitionHelper.java b/hudi-common/src/main/java/org/apache/hudi/common/util/RemotePartitionHelper.java
new file mode 100644
index 0000000000000..626f443fd2208
--- /dev/null
+++ b/hudi-common/src/main/java/org/apache/hudi/common/util/RemotePartitionHelper.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.http.Consts;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.HashMap;
+
+public class RemotePartitionHelper implements Serializable {
+  private static final Logger LOG = LoggerFactory.getLogger(RemotePartitionHelper.class);
+
+  public static final String URL = "/v1/hoodie/partitioner/getpartitionindex";
+  public static final String NUM_BUCKETS_PARAM = "numbuckets";
+  public static final String PARTITION_PATH_PARAM = "partitionpath";
+  public static final String PARTITION_NUM_PARAM = "partitionnum";
+  private final RetryHelper retryHelper;
+  private final String serverHost;
+  private final Integer serverPort;
+  private final ObjectMapper mapper;
+  private final int timeoutMs;
+  private final HashMap<String, Integer> cache; // dataPartition -> sparkPartitionIndex
+  public RemotePartitionHelper(FileSystemViewStorageConfig viewConf) {
+    this.retryHelper = new RetryHelper(
+        viewConf.getRemoteTimelineClientMaxRetryIntervalMs(),
+        viewConf.getRemoteTimelineClientMaxRetryNumbers(),
+        viewConf.getRemoteTimelineInitialRetryIntervalMs(),
+        viewConf.getRemoteTimelineClientRetryExceptions(),
+        "Sending request");
+    this.serverHost = viewConf.getRemoteViewServerHost();
+    this.serverPort = viewConf.getRemoteViewServerPort();
+    this.timeoutMs = viewConf.getRemoteTimelineClientTimeoutSecs() * 1000;
+    this.mapper = new ObjectMapper();
+    this.cache = new HashMap<>();
+  }
+
+  public int getPartition(int numBuckets, String partitionPath, int curBucket, int partitionNum) throws Exception {
+    if (cache.containsKey(partitionPath)) {
+      return computeActualPartition(cache.get(partitionPath), curBucket, partitionNum);
+    }
+    URIBuilder builder =
+        new URIBuilder().setHost(serverHost).setPort(serverPort).setPath(URL).setScheme("http");
+
+    builder.addParameter(NUM_BUCKETS_PARAM, String.valueOf(numBuckets));
+    builder.addParameter(PARTITION_PATH_PARAM, partitionPath);
+    builder.addParameter(PARTITION_NUM_PARAM, String.valueOf(partitionNum));
+
+    String url = builder.toString();
+    LOG.debug("Sending request : (" + url + ").");
+    Response response = (Response)(retryHelper != null ? retryHelper.start(() -> Request.Get(url).connectTimeout(timeoutMs).socketTimeout(timeoutMs).execute())
+        : Request.Get(url).connectTimeout(timeoutMs).socketTimeout(timeoutMs).execute());
+    String content = response.returnContent().asString(Consts.UTF_8);
+    int partitionIndex = Integer.parseInt(mapper.readValue(content, new TypeReference<String>() {}));
+    cache.put(partitionPath, partitionIndex);
+    return computeActualPartition(partitionIndex, curBucket, partitionNum);
+  }
+
+  private int computeActualPartition(int startOffset, int curBucket, int partitionNum) {
+    int res = startOffset + curBucket;
+    return res >= partitionNum ? res % partitionNum : res;
+  }
+
+}
diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBucketRescaleCommitActionExecutor.java b/hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBucketRescaleCommitActionExecutor.java
index ab623643a6280..449283d872c00 100644
--- a/hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBucketRescaleCommitActionExecutor.java
+++ b/hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/commit/DatasetBucketRescaleCommitActionExecutor.java
@@ -59,7 +59,7 @@ public DatasetBucketRescaleCommitActionExecutor(HoodieWriteConfig config,
    */
   @Override
   protected BulkInsertPartitioner<Dataset<Row>> getPartitioner(boolean populateMetaFields, boolean isTablePartitioned) {
-    return new BucketIndexBulkInsertPartitionerWithRows(writeConfig.getBucketIndexHashFieldWithDefault(), hashingConfig);
+    return new BucketIndexBulkInsertPartitionerWithRows(writeClient.getConfig(), hashingConfig);
   }
 
   /**
diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestBucketIndexSupport.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestBucketIndexSupport.scala
index 67915f9a58847..adf34641989b5 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestBucketIndexSupport.scala
+++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestBucketIndexSupport.scala
@@ -22,21 +22,27 @@ import org.apache.hudi.common.config.{HoodieMetadataConfig, TypedProperties}
 import org.apache.hudi.common.config.HoodieMetadataConfig.ENABLE
 import org.apache.hudi.common.fs.FSUtils
 import org.apache.hudi.common.table.HoodieTableConfig
-import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.common.table.view.{FileSystemViewManager, FileSystemViewStorageConfig}
+import org.apache.hudi.config.{HoodieIndexConfig, HoodieWriteConfig}
+import org.apache.hudi.hadoop.fs.HadoopFSUtils
 import org.apache.hudi.index.HoodieIndex
 import org.apache.hudi.index.bucket.BucketIdentifier
+import org.apache.hudi.index.bucket.partition.NumBucketsFunction
 import org.apache.hudi.keygen.{ComplexKeyGenerator, NonpartitionedKeyGenerator}
 import org.apache.hudi.keygen.constant.KeyGeneratorOptions
 import org.apache.hudi.testutils.HoodieSparkClientTestBase
+import org.apache.hudi.timeline.service.TimelineService
 
 import org.apache.avro.Schema
 import org.apache.avro.generic.GenericData
-import org.apache.spark.sql.{HoodieCatalystExpressionUtils, SparkSession}
+import org.apache.spark.sql.{BucketPartitionUtils, HoodieCatalystExpressionUtils, SparkSession}
 import org.apache.spark.sql.catalyst.encoders.DummyExpressionHolder
 import org.apache.spark.sql.catalyst.expressions.PredicateHelper
 import org.apache.spark.sql.types._
 import org.junit.jupiter.api.{AfterEach, BeforeEach, Tag, Test}
 
+import java.util
+
 @Tag("functional")
 class TestBucketIndexSupport extends HoodieSparkClientTestBase with PredicateHelper {
 
@@ -78,6 +84,52 @@ class TestBucketIndexSupport extends HoodieSparkClientTestBase with PredicateHel
     cleanupSparkContexts()
   }
 
+  @Test
+  def testBucketIndexRemotePartitioner(): Unit = {
+    val numBuckets = 511
+    val config = HoodieWriteConfig.newBuilder
+      .withPath(basePath).
+      withIndexConfig(HoodieIndexConfig.newBuilder()
+        .withBucketNum(numBuckets.toString)
+        .enableBucketRemotePartitioner(true).build())
+      .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder
+        .withRemoteServerPort(incrementTimelineServicePortToUse).build)
+      .build
+
+    val timelineService = new TimelineService(context, HadoopFSUtils.getStorageConf,
+      TimelineService.Config.builder.enableMarkerRequests(true).enableRemotePartitioner(true)
+      .serverPort(config.getViewStorageConfig.getRemoteViewServerPort).build,
+      FileSystemViewManager.createViewManager(context, config.getMetadataConfig, config.getViewStorageConfig, config.getCommonConfig))
+
+    timelineService.startService
+    this.timelineService = timelineService
+
+    timelineService.startService()
+    val numBucketsFunction = NumBucketsFunction.fromWriteConfig(config)
+    val partitionNum = 1533
+    val partitioner = BucketPartitionUtils.getRemotePartitioner(config.getViewStorageConfig, numBucketsFunction, partitionNum)
+    val dataPartitions = List("dt=20250501", "dt=20250502", "dt=20250503", "dt=20250504", "dt=20250505", "dt=20250506")
+    val res = new util.HashMap[Int, Int]
+    dataPartitions.foreach(dataPartition => {
+      for (i <- 1 to numBuckets) {
+        // for bucket id from 1 to numBuckets
+        // mock 1000 spark partitions
+        val sparkPartition = partitioner.getPartition((dataPartition, i))
+        if (res.containsKey(sparkPartition)) {
+          val value = res.get(sparkPartition) + 1
+          res.put(sparkPartition, value)
+        } else {
+          res.put(sparkPartition, 1)
+        }
+      }
+    })
+    timelineService.close()
+
+    res.values().stream().forEach(value => {
+      assert(value == (numBuckets*dataPartitions.size/partitionNum))
+    })
+  }
+
   @Test
   def testSingleHashFieldsExpression: Unit = {
     val bucketNumber = 19
diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestInsertTable.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestInsertTable.scala
index 474c7f3c0181c..fd0e7b328c1bb 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestInsertTable.scala
+++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestInsertTable.scala
@@ -1890,6 +1890,51 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
     }
   }
 
+  test("Test Bulk Insert Into Bucket Index Table With Remote Partitioner") {
+    withSQLConf(
+      "hoodie.datasource.write.operation" -> "bulk_insert",
+      "hoodie.bulkinsert.shuffle.parallelism" -> "1") {
+      withTempDir { tmp =>
+        val tableName = generateTableName
+        // Create a partitioned table
+        spark.sql(
+          s"""
+             |create table $tableName (
+             |  id int,
+             |  dt string,
+             |  name string,
+             |  price double,
+             |  ts long
+             |) using hudi
+             | tblproperties (
+             | primaryKey = 'id,name',
+             | type = 'cow',
+             | preCombineField = 'ts',
+             | hoodie.index.type = 'BUCKET',
+             | hoodie.bucket.index.hash.field = 'id,name',
+             | hoodie.bucket.index.remote.partitioner.enable = 'true')
+             | partitioned by (dt)
+             | location '${tmp.getCanonicalPath}'
+       """.stripMargin)
+
+        // Note: Do not write the field alias, the partition field must be placed last.
+        spark.sql(
+          s"""
+             | insert into $tableName values
+             | (1, 'a1,1', 10, 1000, "2021-01-05"),
+             | (2, 'a2', 20, 2000, "2021-01-06"),
+             | (3, 'a3,3', 30, 3000, "2021-01-07")
+       """.stripMargin)
+
+        checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+          Seq(1, "a1,1", 10.0, 1000, "2021-01-05"),
+          Seq(2, "a2", 20.0, 2000, "2021-01-06"),
+          Seq(3, "a3,3", 30.0, 3000, "2021-01-07")
+        )
+      }
+    }
+  }
+
   test("Test Insert Overwrite Bucket Index Table") {
     withSQLConf(
       "hoodie.datasource.write.operation" -> "bulk_insert",
diff --git a/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java b/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java
index 701e92e3902d6..8050c62057f2a 100644
--- a/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java
+++ b/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java
@@ -35,12 +35,14 @@
 import org.apache.hudi.common.table.view.SyncableFileSystemView;
 import org.apache.hudi.common.util.HoodieTimer;
 import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.RemotePartitionHelper;
 import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieIOException;
 import org.apache.hudi.storage.StorageConfiguration;
 import org.apache.hudi.timeline.service.handlers.BaseFileHandler;
 import org.apache.hudi.timeline.service.handlers.FileSliceHandler;
 import org.apache.hudi.timeline.service.handlers.MarkerHandler;
+import org.apache.hudi.timeline.service.handlers.RemotePartitionerHandler;
 import org.apache.hudi.timeline.service.handlers.TimelineHandler;
 
 import com.fasterxml.jackson.core.JsonProcessingException;
@@ -83,6 +85,7 @@ public class RequestHandler {
   private final FileSliceHandler sliceHandler;
   private final BaseFileHandler dataFileHandler;
   private final MarkerHandler markerHandler;
+  private RemotePartitionerHandler partitionerHandler;
   private final Registry metricsRegistry = Registry.getRegistry("TimelineService");
   private final ScheduledExecutorService asyncResultService;
 
@@ -101,6 +104,9 @@ public RequestHandler(Javalin app, StorageConfiguration<?> conf, TimelineService
     } else {
       this.markerHandler = null;
     }
+    if (timelineServiceConfig.enableRemotePartitioner) {
+      this.partitionerHandler = new RemotePartitionerHandler(conf, timelineServiceConfig, viewManager);
+    }
     if (timelineServiceConfig.async) {
       this.asyncResultService = Executors.newSingleThreadScheduledExecutor();
     } else {
@@ -182,6 +188,9 @@ public void register() {
     if (markerHandler != null) {
       registerMarkerAPI();
     }
+    if (partitionerHandler != null) {
+      registerRemotePartitionerAPI();
+    }
   }
 
   public void stop() {
@@ -530,6 +539,16 @@ private void registerMarkerAPI() {
     }, false));
   }
 
+  private void registerRemotePartitionerAPI() {
+    app.get(RemotePartitionHelper.URL, new ViewHandler(ctx -> {
+      int partition = partitionerHandler.gePartitionIndex(
+          ctx.queryParamAsClass(RemotePartitionHelper.NUM_BUCKETS_PARAM, String.class).getOrDefault(""),
+          ctx.queryParamAsClass(RemotePartitionHelper.PARTITION_PATH_PARAM, String.class).getOrDefault(""),
+          ctx.queryParamAsClass(RemotePartitionHelper.PARTITION_NUM_PARAM, String.class).getOrDefault(""));
+      writeValueAsString(ctx, partition);
+    }, false));
+  }
+
   /**
    * Used for logging and performing refresh check.
    */
diff --git a/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/TimelineService.java b/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/TimelineService.java
index 7dfe55b3e93e0..a5f3780d25315 100644
--- a/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/TimelineService.java
+++ b/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/TimelineService.java
@@ -111,6 +111,9 @@ public static class Config implements Serializable {
     @Parameter(names = {"--enable-marker-requests", "-em"}, description = "Enable handling of marker-related requests")
     public boolean enableMarkerRequests = false;
 
+    @Parameter(names = {"--enable-remote-partitioner"}, description = "Enable remote partitioner")
+    public boolean enableRemotePartitioner = false;
+
     @Parameter(names = {"--marker-batch-threads", "-mbt"}, description = "Number of threads to use for batch processing marker creation requests")
     public int markerBatchNumThreads = 20;
 
@@ -185,6 +188,7 @@ public static class Builder {
       private Long asyncConflictDetectorInitialDelayMs = 0L;
       private Long asyncConflictDetectorPeriodMs = 30000L;
       private Long maxAllowableHeartbeatIntervalInMs = 120000L;
+      private boolean enableRemotePartitioner = false;
 
       public Builder() {
       }
@@ -239,6 +243,11 @@ public Builder enableMarkerRequests(boolean enableMarkerRequests) {
         return this;
       }
 
+      public Builder enableRemotePartitioner(boolean enableRemotePartitioner) {
+        this.enableRemotePartitioner = enableRemotePartitioner;
+        return this;
+      }
+
       public Builder markerBatchNumThreads(int markerBatchNumThreads) {
         this.markerBatchNumThreads = markerBatchNumThreads;
         return this;
@@ -296,6 +305,7 @@ public Config build() {
         config.async = this.async;
         config.compress = this.compress;
         config.enableMarkerRequests = this.enableMarkerRequests;
+        config.enableRemotePartitioner = this.enableRemotePartitioner;
         config.markerBatchNumThreads = this.markerBatchNumThreads;
         config.markerBatchIntervalMs = this.markerBatchIntervalMs;
         config.markerParallelism = this.markerParallelism;
diff --git a/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/RemotePartitionerHandler.java b/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/RemotePartitionerHandler.java
new file mode 100644
index 0000000000000..74d930b8d800e
--- /dev/null
+++ b/hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/RemotePartitionerHandler.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.storage.StorageConfiguration;
+import org.apache.hudi.timeline.service.TimelineService;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RemotePartitionerHandler extends Handler {
+
+  // cache Map<PartitionPath, BucketStartIndex>
+  private final ConcurrentHashMap<String, Integer> cache;
+  private final AtomicInteger nextIndex = new AtomicInteger(0);
+
+  public RemotePartitionerHandler(StorageConfiguration<?> conf, TimelineService.Config timelineServiceConfig,
+                                  FileSystemViewManager viewManager) {
+    super(conf, timelineServiceConfig, viewManager);
+    this.cache = new ConcurrentHashMap<>();
+  }
+
+  public int gePartitionIndex(String numBuckets, String partitionPath, String partitionNum) {
+    int num = Integer.parseInt(numBuckets);
+    int partNum = Integer.parseInt(partitionNum);
+
+    return cache.computeIfAbsent(partitionPath, key -> {
+      int current;
+      int newNext;
+      int res;
+      do {
+        current = nextIndex.get();
+        res = current;
+        newNext = current + num;
+        if (newNext >= partNum) {
+          newNext = newNext % partNum;
+        }
+      } while (!nextIndex.compareAndSet(current, newNext));
+      return res;
+    });
+  }
+}


Reviews:
user: login: zhangyue19921010
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T04:07:04Z
user: login: KnightChess
body: If there is no centralized allocation of partition IDs, it is indeed impossible to fully handle the issue of partition key conflicts. However, this algorithm has a drawback, which may lead to an excessive number of reduce tasks. When the data record is large and there is no remote shuffle service, it is likely to cause the shuffle read to fail. If it is enabled by default, it may be necessary to supplement in the documentation what problems may be caused.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-10T03:23:01Z
user: login: danny0405
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T00:25:21Z
user: login: danny0405
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T00:26:28Z
user: login: KnightChess
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T02:37:30Z
user: login: KnightChess
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T02:37:50Z
user: login: danny0405
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T03:43:44Z

Review comments:
diff_hunk: @@ -287,6 +287,11 @@ public class HoodieIndexConfig extends HoodieConfig {
       .withDocumentation("Only applies if index type is BUCKET. Determine the number of buckets in the hudi table, "
           + "and each partition is divided to N buckets.");
 
+  public static final ConfigProperty<Boolean> BUCKET_PARTITIONER = ConfigProperty
+      .key("hoodie.bucket.remote.partitioner.enable")
+      .defaultValue(true)
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java
user: login: zhangyue19921010
body: Default enable this new remote partitioner
created_at: 2025-05-06T04:07:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -287,6 +287,11 @@ public class HoodieIndexConfig extends HoodieConfig {
       .withDocumentation("Only applies if index type is BUCKET. Determine the number of buckets in the hudi table, "
           + "and each partition is divided to N buckets.");
 
+  public static final ConfigProperty<Boolean> BUCKET_PARTITIONER = ConfigProperty
+      .key("hoodie.bucket.remote.partitioner.enable")
+      .defaultValue(true)
path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieIndexConfig.java
user: login: danny0405
body: `hoodie.bucket.remote.partitioner.enable` -> `hoodie.bucket.index.remote.partitioner.enable`?
created_at: 2025-05-13T00:25:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.table.view.FileSystemViewStorageConfig;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.http.Consts;
+import org.apache.http.client.fluent.Request;
+import org.apache.http.client.fluent.Response;
+import org.apache.http.client.utils.URIBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.HashMap;
+
+public class RemotePartitionHelper implements Serializable {
+  private static final Logger LOG = LoggerFactory.getLogger(RemotePartitionHelper.class);
+
+  public static final String URL = "/v1/hoodie/partitioner/getpartitionindex";
+  public static final String NUM_BUCKETS_PARAM = "numbuckets";
+  public static final String PARTITION_PATH_PARAM = "partitionpath";
+  public static final String CUR_BUCKET_PARAM = "curbucket";
+  public static final String PARTITION_NUM_PARAM = "partitionnum";
+  private final RetryHelper retryHelper;
+  private final String serverHost;
+  private final Integer serverPort;
+  private final ObjectMapper mapper;
+  private final int timeoutMs;
+  private final HashMap<String, Integer> cache; // dataPartition -> sparkPartitionIndex
+  public RemotePartitionHelper(FileSystemViewStorageConfig viewConf) {
+    this.retryHelper = new RetryHelper(
+        viewConf.getRemoteTimelineClientMaxRetryIntervalMs(),
+        viewConf.getRemoteTimelineClientMaxRetryNumbers(),
+        viewConf.getRemoteTimelineInitialRetryIntervalMs(),
+        viewConf.getRemoteTimelineClientRetryExceptions(),
+        "Sending request");
+    this.serverHost = viewConf.getRemoteViewServerHost();
+    this.serverPort = viewConf.getRemoteViewServerPort();
+    this.timeoutMs = viewConf.getRemoteTimelineClientTimeoutSecs() * 1000;
+    this.mapper = new ObjectMapper();
+    this.cache = new HashMap<>();
+  }
+
+  public int getPartition(int numBuckets, String partitionPath, int curBucket, int partitionNum) throws Exception {
+    if (cache.containsKey(partitionPath)) {
+      return computeActualPartition(cache.get(partitionPath), curBucket, partitionNum);
+    }
+    URIBuilder builder =
+        new URIBuilder().setHost(serverHost).setPort(serverPort).setPath(URL).setScheme("http");
+
+    builder.addParameter(NUM_BUCKETS_PARAM, String.valueOf(numBuckets));
+    builder.addParameter(PARTITION_PATH_PARAM, partitionPath);
+    builder.addParameter(CUR_BUCKET_PARAM, String.valueOf(curBucket));
+    builder.addParameter(PARTITION_NUM_PARAM, String.valueOf(partitionNum));
+
+    String url = builder.toString();
+    LOG.info("Sending request : (" + url + ").");
path: hudi-common/src/main/java/org/apache/hudi/common/util/RemotePartitionHelper.java
user: login: danny0405
body: is the log msg too many there? should we switch to `DEBUG` level.
created_at: 2025-05-13T00:26:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.timeline.service.handlers;
+
+import org.apache.hudi.common.table.view.FileSystemViewManager;
+import org.apache.hudi.storage.StorageConfiguration;
+import org.apache.hudi.timeline.service.TimelineService;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RemotePartitionerHandler extends Handler {
+
+  // cache Map<PartitionPath, BucketStartIndex>
+  private final ConcurrentHashMap<String, Integer> cache;
+  private final AtomicInteger nextIndex = new AtomicInteger(0);
+
+  public RemotePartitionerHandler(StorageConfiguration<?> conf, TimelineService.Config timelineServiceConfig,
+                                  FileSystemViewManager viewManager) {
+    super(conf, timelineServiceConfig, viewManager);
+    this.cache = new ConcurrentHashMap<>();
+  }
+
+  public int gePartitionIndex(String numBuckets, String partitionPath, String curBucket, String partitionNum) {
path: hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/RemotePartitionerHandler.java
user: login: KnightChess
body: nit: the `curBucket` not use
created_at: 2025-05-13T02:37:30Z
author_association: CONTRIBUTOR

PR comments:
user: login: zhangyue19921010
created_at: 2025-05-07T02:25:47Z
body: @hudi-bot run azure
author_association: CONTRIBUTOR
user: login: zhangyue19921010
created_at: 2025-05-08T06:43:16Z
body: @KnightChess would u mind to take a look?
author_association: CONTRIBUTOR
user: login: danny0405
created_at: 2025-05-08T07:38:04Z
body: what is the part the local partitioner missed so that data skew was induced compared to the global partitioner proposed here? 
author_association: CONTRIBUTOR
user: login: zhangyue19921010
created_at: 2025-05-08T12:07:01Z
body: > what is the part the local partitioner missed so that data skew was induced compared to the global partitioner proposed here?

# Analysis of Local Hash Algorithm Limitations

The Local Hash algorithm has two critical flaws:

1. **Uneven Distribution** causing long-tail effects, degrading resource utilization and execution efficiency.
2. **Underutilized Parallelism** leading to idle Tasks (empty processing).

---

## Case 1: Data Skew 

### Problem
When `bucketNum` and `parallelism` share a common factor `d > 1`, the `curBucket` values for each partition are confined to a fixed subset of `d` partition IDs. This results in **Uneven Distribution** and **Data Skew**.

### Example
- **Parameters**:  
  `parallelism = 6`, `bucketNum = 4` (common factor `d = 2`).  
- **Partition A** (hash mod 6 = 2):  
  - `partitionIndex = 2 * 4 = 8`  
  - `globalIndex` for `curBucket = 0,1,2,3`: `8, 9, 10, 11`  
  - **Result**: Partition IDs `2, 3, 4, 5` (skewed to upper half).  
- **Partition B** (hash mod 6 = 0):  
  - `curBucket` maps to Partition IDs `0, 1, 2, 3`.  
- **Outcome**:  
  Partitions `0-3` handle most data, while `4-5` are underutilized.

---

## Case 2: Incomplete Coverage of Partition IDs

### Problem
A shared factor `d > 1` between `bucketNum` and `parallelism` restricts the coverage of partition IDs to **`parallelism/d` unique values**. The modulo operation cycles within a fixed subset of residues, leaving some partitions unused.

### Example
- **Parameters**:  
  `parallelism = 6`, `bucketNum = 4` (common factor `d = 2`).  
- **Partition** (hash mod 6 = 2):  
  - `partitionIndex = 2 * 4 = 8`  
  - `globalIndex` for `curBucket = 0,1,2,3`: `8, 9, 10, 11`  
  - **Modulo 6 Results**: `2, 3, 4, 5`  
- **Consequence**:  
  Partition IDs `0` and `1` are **never assigned**, resulting in idle Tasks.

---

## Root Cause
The algorithm’s reliance on multiplicative offsets (partitionIndex = hash % parallelism * bucketNum) and modulo arithmetic amplifies skew **when bucketNum and parallelism share common factors**. This design flaw inherently limits the effective parallelism and exacerbates resource imbalance.


author_association: CONTRIBUTOR
user: login: zhangyue19921010
created_at: 2025-05-08T12:14:13Z
body: <img width="1675" alt="image" src="https://github.com/user-attachments/assets/f1c9ba6c-bd3e-47fd-a6b1-22261eeedbeb" />
Here is another example for this case which about 50% of the tasks are running idly.
author_association: CONTRIBUTOR
user: login: danny0405
created_at: 2025-05-09T01:33:55Z
body: > The algorithm’s reliance on multiplicative offsets (partitionIndex = hash % parallelism * bucketNum) and modulo arithmetic amplifies skew when bucketNum and parallelism share common factors. This design flaw inherently limits the effective parallelism

So there is no way to improve the local algorithm anyway? And based your practice, how is the stability based on the fs view request?
author_association: CONTRIBUTOR
user: login: KnightChess
created_at: 2025-05-10T03:28:10Z
body: @danny0405 I think one improved approach is to modify the following line of code for local algorithm. However, this would still result in an excessively large number of reduce tasks and fail to completely resolve the issue of partition key conflicts. If there is no centralized allocation of partition IDs, it is indeed impossible to fully handle the issue of partition key conflicts.
`int partitionId = (int) (globalIndex % parallelism); `
-> `int partitionId = globalIndex;`
use globalindex
```
public static Functions.Function3<Integer, String, Integer, Integer> getPartitionIndexFunc(int parallelism) {
    return (bucketNum, partition, curBucket) -> {
      long partitionIndex = (partition.hashCode() & Integer.MAX_VALUE) % parallelism * (long) bucketNum;
      long globalIndex = partitionIndex + curBucket;
      int partitionId = (int) (globalIndex % parallelism);
      ValidationUtils.checkArgument(partitionId >= 0 && partitionId < parallelism,
          () -> "Partition id should be in range [0, " + parallelism + "), but got " + partitionId);
      return partitionId;
    };
  }
```
author_association: CONTRIBUTOR
user: login: zhangyue19921010
created_at: 2025-05-11T04:47:54Z
body: > If there is no centralized allocation of partition IDs, it is indeed impossible to fully handle the issue of partition key conflicts. However, this algorithm has a drawback, which may lead to an excessive number of reduce tasks. When the data record is large and there is no remote shuffle service, it is likely to cause the shuffle read to fail. If it is enabled by default, it may be necessary to supplement in the documentation what problems may be caused.

Hi, thanks for your response. Indeed, without a centralized planner, addressing skewness completely—especially in large-scale data scenarios—is challenging. There’s one point I’m not entirely clear about: when comparing RemotePartitioner and LocalPartitioner, how do they differ in terms of shuffle data volume? My understanding is that in a bucket index scenario, the amount of shuffled data should be the same regardless of which partitioner is used, as both rely on a global shuffle based on the bucket ID.

author_association: CONTRIBUTOR
user: login: zhangyue19921010
created_at: 2025-05-11T04:48:15Z
body: Also It’s okay for me to provided remote partitioner as default false
author_association: CONTRIBUTOR
user: login: KnightChess
created_at: 2025-05-12T12:14:33Z
body: @zhangyue19921010 Sorry, I didn't notice before that the new algorithm also calculates the remainder of the parallelism.
author_association: CONTRIBUTOR
user: login: zhangyue19921010
created_at: 2025-05-14T01:54:52Z
body: @hudi-bot run azure
author_association: CONTRIBUTOR
user: login: zhangyue19921010
created_at: 2025-05-14T11:49:56Z
body: @hudi-bot run azure
author_association: CONTRIBUTOR
user: login: hudi-bot
created_at: 2025-05-19T02:35:53Z
body: <!--
Meta data
{
  "version" : 1,
  "metaDataEntries" : [ {
    "hash" : "e599fc4d3376465abdbbb96576be152901c9195d",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5317",
    "triggerID" : "e599fc4d3376465abdbbb96576be152901c9195d",
    "triggerType" : "PUSH"
  }, {
    "hash" : "0e3d5091a6ba3c393ec19fa56213bfa03081a8ac",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5330",
    "triggerID" : "0e3d5091a6ba3c393ec19fa56213bfa03081a8ac",
    "triggerType" : "PUSH"
  }, {
    "hash" : "0e3d5091a6ba3c393ec19fa56213bfa03081a8ac",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5336",
    "triggerID" : "2856834623",
    "triggerType" : "MANUAL"
  }, {
    "hash" : "c4bc3568a3f37d14cd8d12ea90b8f5884f1d06b7",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5441",
    "triggerID" : "c4bc3568a3f37d14cd8d12ea90b8f5884f1d06b7",
    "triggerType" : "PUSH"
  }, {
    "hash" : "c4bc3568a3f37d14cd8d12ea90b8f5884f1d06b7",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5454",
    "triggerID" : "2878188120",
    "triggerType" : "MANUAL"
  }, {
    "hash" : "c4bc3568a3f37d14cd8d12ea90b8f5884f1d06b7",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5457",
    "triggerID" : "2878396537",
    "triggerType" : "MANUAL"
  }, {
    "hash" : "c4bc3568a3f37d14cd8d12ea90b8f5884f1d06b7",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5465",
    "triggerID" : "2879905797",
    "triggerType" : "MANUAL"
  }, {
    "hash" : "4625a14cf6c32729d9f740a422546ba99c548385",
    "status" : "DELETED",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5479",
    "triggerID" : "4625a14cf6c32729d9f740a422546ba99c548385",
    "triggerType" : "PUSH"
  }, {
    "hash" : "ff91f6bec2406d4c4019b170312b919ab3bcd613",
    "status" : "FAILURE",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5530",
    "triggerID" : "ff91f6bec2406d4c4019b170312b919ab3bcd613",
    "triggerType" : "PUSH"
  }, {
    "hash" : "59454b94ab007c3a3a12f65e341b155c0939bf36",
    "status" : "PENDING",
    "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5536",
    "triggerID" : "59454b94ab007c3a3a12f65e341b155c0939bf36",
    "triggerType" : "PUSH"
  } ]
}-->
## CI report:

* ff91f6bec2406d4c4019b170312b919ab3bcd613 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5530) 
* 59454b94ab007c3a3a12f65e341b155c0939bf36 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5536) 

<details>
<summary>Bot commands</summary>
  @hudi-bot supports the following commands:

 - `@hudi-bot run azure` re-run the last Azure build
</details>
author_association: COLLABORATOR
user: login: danny0405
created_at: 2025-05-19T03:43:08Z
body: CI passed: https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=5536&view=results
author_association: CONTRIBUTOR