Repository: apache/pinot
Pull request #15778: make dedup table use strict replica group assignment too
Author: login: klsince
Labels: name: enhancement
description: Improvement to existing functionality
name: upsert
description: Related to upsert functionality
name: dedup
description: Changes related to realtime ingestion dedup handling
Created: 2025-05-13T18:27:42Z | Merged: 2025-05-15T19:58:43Z

This PR ensures dedup table to use strict replica group assignment too. Like upsert table, for dedup table, the segments from same partition must be hosted on same server as well.


Diff:
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java
index e32a7246b2d4..7e37026438d7 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java
@@ -21,6 +21,7 @@
 import javax.annotation.Nullable;
 import org.apache.helix.HelixManager;
 import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.spi.config.table.DedupConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.TableType;
 import org.apache.pinot.spi.config.table.UpsertConfig;
@@ -40,7 +41,9 @@ public static SegmentAssignment getSegmentAssignment(HelixManager helixManager,
       segmentAssignment = new OfflineSegmentAssignment();
     } else {
       UpsertConfig upsertConfig = tableConfig.getUpsertConfig();
-      if (upsertConfig != null && upsertConfig.getMode() != UpsertConfig.Mode.NONE) {
+      DedupConfig dedupConfig = tableConfig.getDedupConfig();
+      if ((upsertConfig != null && upsertConfig.getMode() != UpsertConfig.Mode.NONE) || (dedupConfig != null
+          && dedupConfig.isDedupEnabled())) {
         segmentAssignment = new StrictRealtimeSegmentAssignment();
       } else {
         segmentAssignment = new RealtimeSegmentAssignment();
diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java
index 2bc0ada5bd8d..6c76332a24ec 100644
--- a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java
+++ b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java
@@ -37,7 +37,7 @@
 
 
 /**
- * Segment assignment for LLC real-time table using upsert. The assignSegment() of RealtimeSegmentAssignment is
+ * Segment assignment for LLC real-time table using upsert or dedup. The assignSegment() of RealtimeSegmentAssignment is
  * overridden to add new segment for a table partition in a way that's consistent with the assignment in idealState to
  * make sure that at any time the segments from the same table partition is hosted by the same server.
  * <ul>
@@ -47,12 +47,13 @@
  *     InstancePartition and the one in idealState are different, the one in idealState must be used so that segments
  *     from the same table partition are always hosted on the same server as set in current idealState. If the
  *     idealState is not honored, segments from the same table partition may be assigned to different servers,
- *     breaking the key assumption for queries to be correct for the table using upsert.
+ *     breaking the key assumption for queries to be correct for the table using upsert or dedup.
  *   </li>
  *   <li>
- *     There is no need to handle COMPLETED segments for tables using upsert, because their completed segments should
- *     not be relocated to servers tagged to host COMPLETED segments. Basically, upsert-enabled tables can only use
- *     servers tagged for CONSUMING segments to host both consuming and completed segments from a table partition.
+ *     There is no need to handle COMPLETED segments for tables using upsert or dedup, because their completed
+ *     segments should not be relocated to servers tagged to host COMPLETED segments. Basically, tables using upsert
+ *     or dedup can only use servers tagged for CONSUMING segments to host both consuming and completed segments from
+ *     a table partition.
  *   </li>
  * </ul>
  */
@@ -63,9 +64,9 @@ public class StrictRealtimeSegmentAssignment extends RealtimeSegmentAssignment {
   // 1. This cache is used for table rebalance only, but not segment assignment. During rebalance, rebalanceTable() can
   //    be invoked multiple times when the ideal state changes during the rebalance process.
   // 2. The cache won't be refreshed when an existing segment is replaced with a segment from a different partition.
-  //    Replacing a segment with a segment from a different partition should not be allowed for upsert table because it
-  //    will cause the segment being served by the wrong servers. If this happens during the table rebalance, another
-  //    rebalance might be needed to fix the assignment.
+  //    Replacing a segment with a segment from a different partition should not be allowed for upsert/dedup table
+  //    because it will cause the segment being served by the wrong servers. If this happens during the table
+  //    rebalance, another rebalance might be needed to fix the assignment.
   private final Object2IntOpenHashMap<String> _segmentPartitionIdMap = new Object2IntOpenHashMap<>();
 
   @Override
@@ -163,8 +164,10 @@ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, S
     Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
         _tableNameWithType);
     Preconditions.checkArgument(config.isIncludeConsuming(),
-        "Consuming segment must be included when rebalancing upsert table: %s", _tableNameWithType);
-    Preconditions.checkState(sortedTiers == null, "Tiers must not be specified for upsert table: %s",
+        "Consuming segment must be included when rebalancing upsert/dedup table: %s", _tableNameWithType);
+    // TODO: consider to support tiers for segments out of metadata TTL for upsert/dedup table, as those segments
+    //       won't be included in the upsert/dedup metadata as kept on CONSUMING tier.
+    Preconditions.checkState(sortedTiers == null, "Tiers must not be specified for upsert/dedup table: %s",
         _tableNameWithType);
     _logger.info("Rebalancing table: {} with instance partitions: {}", _tableNameWithType, instancePartitions);
 
diff --git a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java
index 6775329d378a..43cfb9998d98 100644
--- a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java
+++ b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java
@@ -30,6 +30,7 @@
 import org.apache.pinot.common.assignment.InstancePartitions;
 import org.apache.pinot.common.utils.LLCSegmentName;
 import org.apache.pinot.core.realtime.impl.fakestream.FakeStreamConfigUtils;
+import org.apache.pinot.spi.config.table.DedupConfig;
 import org.apache.pinot.spi.config.table.ReplicaGroupStrategyConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.TableType;
@@ -39,6 +40,7 @@
 import org.apache.pinot.spi.utils.CommonConstants.Segment.AssignmentStrategy;
 import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
 import org.testng.annotations.BeforeClass;
+import org.testng.annotations.DataProvider;
 import org.testng.annotations.Test;
 
 import static org.mockito.ArgumentMatchers.anyInt;
@@ -67,7 +69,6 @@ public class StrictRealtimeSegmentAssignmentTest {
       InstancePartitionsType.CONSUMING.getInstancePartitionsName(RAW_TABLE_NAME);
 
   private List<String> _segments;
-  private SegmentAssignment _segmentAssignment;
   private Map<InstancePartitionsType, InstancePartitions> _instancePartitionsMap;
   private InstancePartitions _newConsumingInstancePartitions;
 
@@ -78,16 +79,6 @@ public void setUp() {
       _segments.add(new LLCSegmentName(RAW_TABLE_NAME, segmentId % NUM_PARTITIONS, segmentId / NUM_PARTITIONS,
           System.currentTimeMillis()).getSegmentName());
     }
-
-    Map<String, String> streamConfigs = FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap();
-    UpsertConfig upsertConfig = new UpsertConfig(UpsertConfig.Mode.FULL);
-    TableConfig tableConfig =
-        new TableConfigBuilder(TableType.REALTIME).setTableName(RAW_TABLE_NAME).setNumReplicas(NUM_REPLICAS)
-            .setStreamConfigs(streamConfigs).setUpsertConfig(upsertConfig)
-            .setSegmentAssignmentStrategy(AssignmentStrategy.REPLICA_GROUP_SEGMENT_ASSIGNMENT_STRATEGY)
-            .setReplicaGroupStrategyConfig(new ReplicaGroupStrategyConfig(PARTITION_COLUMN, 1)).build();
-    _segmentAssignment = SegmentAssignmentFactory.getSegmentAssignment(createHelixManager(), tableConfig, null);
-
     _instancePartitionsMap = new TreeMap<>();
     // CONSUMING instances:
     // {
@@ -126,25 +117,40 @@ public void setUp() {
     }
   }
 
-  @Test
-  public void testFactory() {
-    assertTrue(_segmentAssignment instanceof StrictRealtimeSegmentAssignment);
+  @DataProvider(name = "tableTypes")
+  public Object[] getTableTypes() {
+    return new Object[]{"upsert", "dedup"};
+  }
+
+  private static SegmentAssignment createSegmentAssignment(String tableType) {
+    TableConfigBuilder builder = new TableConfigBuilder(TableType.REALTIME).setTableName(RAW_TABLE_NAME)
+        .setNumReplicas(NUM_REPLICAS)
+        .setStreamConfigs(FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs().getStreamConfigsMap())
+        .setSegmentAssignmentStrategy(AssignmentStrategy.REPLICA_GROUP_SEGMENT_ASSIGNMENT_STRATEGY)
+        .setReplicaGroupStrategyConfig(new ReplicaGroupStrategyConfig(PARTITION_COLUMN, 1));
+    TableConfig tableConfig;
+    if ("upsert".equalsIgnoreCase(tableType)) {
+      tableConfig = builder.setUpsertConfig(new UpsertConfig(UpsertConfig.Mode.FULL)).build();
+    } else {
+      tableConfig = builder.setDedupConfig(new DedupConfig()).build();
+    }
+    return SegmentAssignmentFactory.getSegmentAssignment(createHelixManager(), tableConfig, null);
   }
 
-  @Test
-  public void testAssignSegment() {
-    assertTrue(_segmentAssignment instanceof StrictRealtimeSegmentAssignment);
+  @Test(dataProvider = "tableTypes")
+  public void testAssignSegment(String tableType) {
+    SegmentAssignment segmentAssignment = createSegmentAssignment(tableType);
+    assertTrue(segmentAssignment instanceof StrictRealtimeSegmentAssignment);
     Map<InstancePartitionsType, InstancePartitions> onlyConsumingInstancePartitionMap =
         ImmutableMap.of(InstancePartitionsType.CONSUMING, _instancePartitionsMap.get(InstancePartitionsType.CONSUMING));
     int numInstancesPerReplicaGroup = NUM_CONSUMING_INSTANCES / NUM_REPLICAS;
     Map<String, Map<String, String>> currentAssignment = new TreeMap<>();
     // Add segments for partition 0/1/2, but add no segment for partition 3.
     List<String> instancesAssigned;
-    boolean consistent;
     for (int segmentId = 0; segmentId < 3; segmentId++) {
       String segmentName = _segments.get(segmentId);
       instancesAssigned =
-          _segmentAssignment.assignSegment(segmentName, currentAssignment, onlyConsumingInstancePartitionMap);
+          segmentAssignment.assignSegment(segmentName, currentAssignment, onlyConsumingInstancePartitionMap);
       assertEquals(instancesAssigned.size(), NUM_REPLICAS);
       // Segment 0 (partition 0) should be assigned to instance 0, 3, 6
       // Segment 1 (partition 1) should be assigned to instance 1, 4, 7
@@ -171,7 +177,7 @@ public void testAssignSegment() {
     int segmentId = 3;
     String segmentName = _segments.get(segmentId);
     instancesAssigned =
-        _segmentAssignment.assignSegment(segmentName, currentAssignment, newConsumingInstancePartitionMap);
+        segmentAssignment.assignSegment(segmentName, currentAssignment, newConsumingInstancePartitionMap);
     assertEquals(instancesAssigned,
         Arrays.asList("new_consumingInstance_0", "new_consumingInstance_3", "new_consumingInstance_6"));
     addToAssignment(currentAssignment, segmentId, instancesAssigned);
@@ -180,7 +186,7 @@ public void testAssignSegment() {
     for (segmentId = 4; segmentId < 7; segmentId++) {
       segmentName = _segments.get(segmentId);
       instancesAssigned =
-          _segmentAssignment.assignSegment(segmentName, currentAssignment, newConsumingInstancePartitionMap);
+          segmentAssignment.assignSegment(segmentName, currentAssignment, newConsumingInstancePartitionMap);
       assertEquals(instancesAssigned.size(), NUM_REPLICAS);
 
       // Those segments are assigned according to the assignment from idealState, instead of using new_xxx instances
@@ -197,20 +203,20 @@ public void testAssignSegment() {
     }
   }
 
-  @Test
-  public void testAssignSegmentWithOfflineSegment() {
-    assertTrue(_segmentAssignment instanceof StrictRealtimeSegmentAssignment);
+  @Test(dataProvider = "tableTypes")
+  public void testAssignSegmentWithOfflineSegment(String tableType) {
+    SegmentAssignment segmentAssignment = createSegmentAssignment(tableType);
+    assertTrue(segmentAssignment instanceof StrictRealtimeSegmentAssignment);
     Map<InstancePartitionsType, InstancePartitions> onlyConsumingInstancePartitionMap =
         ImmutableMap.of(InstancePartitionsType.CONSUMING, _instancePartitionsMap.get(InstancePartitionsType.CONSUMING));
     int numInstancesPerReplicaGroup = NUM_CONSUMING_INSTANCES / NUM_REPLICAS;
     Map<String, Map<String, String>> currentAssignment = new TreeMap<>();
     // Add segments for partition 0/1/2, but add no segment for partition 3.
     List<String> instancesAssigned;
-    boolean consistent;
     for (int segmentId = 0; segmentId < 3; segmentId++) {
       String segmentName = _segments.get(segmentId);
       instancesAssigned =
-          _segmentAssignment.assignSegment(segmentName, currentAssignment, onlyConsumingInstancePartitionMap);
+          segmentAssignment.assignSegment(segmentName, currentAssignment, onlyConsumingInstancePartitionMap);
       assertEquals(instancesAssigned.size(), NUM_REPLICAS);
       // Segment 0 (partition 0) should be assigned to instance 0, 3, 6
       // Segment 1 (partition 1) should be assigned to instance 1, 4, 7
@@ -238,7 +244,7 @@ public void testAssignSegmentWithOfflineSegment() {
     for (int segmentId = 3; segmentId < 7; segmentId++) {
       String segmentName = _segments.get(segmentId);
       instancesAssigned =
-          _segmentAssignment.assignSegment(segmentName, currentAssignment, newConsumingInstancePartitionMap);
+          segmentAssignment.assignSegment(segmentName, currentAssignment, newConsumingInstancePartitionMap);
       assertEquals(instancesAssigned.size(), NUM_REPLICAS);
 
       // Those segments are assigned according to the assignment from idealState, instead of using new_xxx instances
@@ -255,9 +261,10 @@ public void testAssignSegmentWithOfflineSegment() {
     }
   }
 
-  @Test(expectedExceptions = IllegalStateException.class)
-  public void testAssignSegmentToCompletedServers() {
-    _segmentAssignment.assignSegment("seg01", new TreeMap<>(), new TreeMap<>());
+  @Test(expectedExceptions = IllegalStateException.class, dataProvider = "tableTypes")
+  public void testAssignSegmentToCompletedServers(String tableType) {
+    SegmentAssignment segmentAssignment = createSegmentAssignment(tableType);
+    segmentAssignment.assignSegment("seg01", new TreeMap<>(), new TreeMap<>());
   }
 
   private void addToAssignment(Map<String, Map<String, String>> currentAssignment, int segmentId,
@@ -276,7 +283,7 @@ private void addToAssignment(Map<String, Map<String, String>> currentAssignment,
         SegmentAssignmentUtils.getInstanceStateMap(instancesAssigned, SegmentStateModel.CONSUMING));
   }
 
-  private HelixManager createHelixManager() {
+  private static HelixManager createHelixManager() {
     HelixManager helixManager = mock(HelixManager.class);
     ZkHelixPropertyStore<ZNRecord> propertyStore = mock(ZkHelixPropertyStore.class);
     when(helixManager.getHelixPropertyStore()).thenReturn(propertyStore);


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR extends the strict assignment mechanism to dedup tables, in addition to upsert tables, ensuring that segments from the same partition are hosted on the same server.  
- Refactors tests in StrictRealtimeSegmentAssignmentTest.java to use a DataProvider and a helper method for both "upsert" and "dedup" configurations.  
- Removes the redundant _segmentAssignment field from the tests to improve isolation.  
- Updates SegmentAssignmentFactory.java to check dedup configuration via dedupConfig.isDedupEnabled() and route dedup tables to StrictRealtimeSegmentAssignment.

### Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

| File | Description |
| ---- | ----------- |
| pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java | Refactored test methods to leverage a DataProvider and local SegmentAssignment creation for both upsert and dedup table types. |
| pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java | Adjusted the assignment factory to consider dedup configuration in addition to upsert configuration. |


<details>
<summary>Comments suppressed due to low confidence (2)</summary>

**pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java:122**
* [nitpick] Consider defining constants or an enum for the table types instead of using raw string literals to enhance clarity and avoid potential typos.
```
return new Object[]{"upsert", "dedup"};
```
**pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java:45**
* Verify that dedupConfig.isDedupEnabled() accurately reflects when a dedup table should use strict assignment, ensuring that default or edge cases are correctly handled.
```
if ((upsertConfig != null && upsertConfig.getMode() != UpsertConfig.Mode.NONE) || (dedupConfig != null && dedupConfig.isDedupEnabled())) {
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T18:38:19Z
user: login: Jackie-Jiang
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T00:07:47Z
user: login: somandal
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T22:03:27Z
user: login: klsince
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T16:50:09Z
user: login: somandal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T16:51:58Z
user: login: klsince
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T17:07:28Z
user: login: klsince
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T19:58:23Z

Review comments:
diff_hunk: @@ -163,8 +164,10 @@ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, S
     Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
         _tableNameWithType);
     Preconditions.checkArgument(config.isIncludeConsuming(),
-        "Consuming segment must be included when rebalancing upsert table: %s", _tableNameWithType);
-    Preconditions.checkState(sortedTiers == null, "Tiers must not be specified for upsert table: %s",
+        "Consuming segment must be included when rebalancing upsert/dedup table: %s", _tableNameWithType);
+    // TODO: consider to support tiers for segments out of metadata TTL for upsert/dedup table, as those segments
path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java
user: login: somandal
body: nit: should we also allow support for COMPLETED tagged server move for the same reasons? or will that still be disallowed
created_at: 2025-05-14T21:55:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -40,7 +41,9 @@ public static SegmentAssignment getSegmentAssignment(HelixManager helixManager,
       segmentAssignment = new OfflineSegmentAssignment();
     } else {
       UpsertConfig upsertConfig = tableConfig.getUpsertConfig();
-      if (upsertConfig != null && upsertConfig.getMode() != UpsertConfig.Mode.NONE) {
+      DedupConfig dedupConfig = tableConfig.getDedupConfig();
path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java
user: login: somandal
body: @klsince @Jackie-Jiang is the end goal to use the `strictReplicaGroup` routing config to decide which assignment strategy to use? Or is this one only to be used for upsert / dedup?

We should discuss the invariants for `strictReplicaGroup` routing config for non-upsert/dedup REALTIME and OFFLINE tables
created_at: 2025-05-14T22:03:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,8 +164,10 @@ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, S
     Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
         _tableNameWithType);
     Preconditions.checkArgument(config.isIncludeConsuming(),
-        "Consuming segment must be included when rebalancing upsert table: %s", _tableNameWithType);
-    Preconditions.checkState(sortedTiers == null, "Tiers must not be specified for upsert table: %s",
+        "Consuming segment must be included when rebalancing upsert/dedup table: %s", _tableNameWithType);
+    // TODO: consider to support tiers for segments out of metadata TTL for upsert/dedup table, as those segments
path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java
user: login: klsince
body: I think dedup/upsert still can't use COMPLETED tag, which moves committed segments to different servers immediately, w/o checking metadata TTL. 
created_at: 2025-05-15T16:50:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,8 +164,10 @@ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, S
     Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
         _tableNameWithType);
     Preconditions.checkArgument(config.isIncludeConsuming(),
-        "Consuming segment must be included when rebalancing upsert table: %s", _tableNameWithType);
-    Preconditions.checkState(sortedTiers == null, "Tiers must not be specified for upsert table: %s",
+        "Consuming segment must be included when rebalancing upsert/dedup table: %s", _tableNameWithType);
+    // TODO: consider to support tiers for segments out of metadata TTL for upsert/dedup table, as those segments
path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java
user: login: somandal
body: yeah makes sense, but if checking metadata TTL, would it make sense to add support? I guess this is probably less important than tiering though
created_at: 2025-05-15T16:51:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -40,7 +41,9 @@ public static SegmentAssignment getSegmentAssignment(HelixManager helixManager,
       segmentAssignment = new OfflineSegmentAssignment();
     } else {
       UpsertConfig upsertConfig = tableConfig.getUpsertConfig();
-      if (upsertConfig != null && upsertConfig.getMode() != UpsertConfig.Mode.NONE) {
+      DedupConfig dedupConfig = tableConfig.getDedupConfig();
path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/SegmentAssignmentFactory.java
user: login: klsince
body: > ... is the end goal to use the strictReplicaGroup routing config to decide which assignment strategy to use? Or is this one only to be used for upsert / dedup?

I don't think so. The 'strictReplicaGroup' routing policy and StrictRealtimeSegmentAssignment are kinda orthogonal. The former applies to broker side routing decisions (i.e. if a replica group has unavailable segment, this whole group is skipped); while the latter applies to controller when it decides where to place a segment (i.e. to put the segment according to current IdealState, instead of following the current InstancePartition). 

For dedup/upsert tables, they happen to have to use StrictRealtimeSegmentAssignment to track partition level states properly and strictReplicaGroup routing policy so query always access one partition as a whole.

> We should discuss the invariants for strictReplicaGroup routing config for non-upsert/dedup REALTIME and OFFLINE tables

No very sure what invariants mean here. But I think any table can choose to use 'strictReplicaGroup' routing, just that it doesn't benefit for non-upsert/dedup table as their committed segments are consistent across replicas anyway (and in fact, using 'strctReplicaGroup' would reduce data availability for them)

created_at: 2025-05-15T17:07:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,8 +164,10 @@ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, S
     Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
         _tableNameWithType);
     Preconditions.checkArgument(config.isIncludeConsuming(),
-        "Consuming segment must be included when rebalancing upsert table: %s", _tableNameWithType);
-    Preconditions.checkState(sortedTiers == null, "Tiers must not be specified for upsert table: %s",
+        "Consuming segment must be included when rebalancing upsert/dedup table: %s", _tableNameWithType);
+    // TODO: consider to support tiers for segments out of metadata TTL for upsert/dedup table, as those segments
path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignment.java
user: login: klsince
body: yeah, using tiered storage would be able to handle such ask (more intuitively IMO), when wanting to move segments out of TTL to different servers.
created_at: 2025-05-15T19:58:23Z
author_association: CONTRIBUTOR

PR comments:
user: login: somandal
created_at: 2025-05-14T13:51:43Z
body: @klsince just curious, will this change work with dedup + tiered storage (where only those beyond TTL are moved)?
author_association: CONTRIBUTOR
user: login: klsince
created_at: 2025-05-14T21:36:11Z
body: > ... will this change work with dedup + tiered storage (where only those beyond TTL are moved)?

looks like this check in StrictRealtimeSegmentAssignment.rebalanceTable() doesn't allow to configure tiers for upsert/dedup, so we can't move segments to other tiers by rebalancing tables. We may need to remove this limitation and I think can be addressed in a separate PR.
```
Preconditions.checkState(sortedTiers == null, "Tiers must not be specified for upsert table: %s",
        _tableNameWithType);
```
author_association: CONTRIBUTOR
user: login: codecov-commenter
created_at: 2025-05-14T22:36:37Z
body: ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15778?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `25.00000%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 63.36%. Comparing base ([`1a476de`](https://app.codecov.io/gh/apache/pinot/commit/1a476dea72ebf1c15e14ded28ae60c385f71b2c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ee52ea9`](https://app.codecov.io/gh/apache/pinot/commit/ee52ea9bde6a04b1f6bbaaaf5fc7c3953f603a58?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1739 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15778?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...e/assignment/segment/SegmentAssignmentFactory.java](https://app.codecov.io/gh/apache/pinot/pull/15778?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fhelix%2Fcore%2Fassignment%2Fsegment%2FSegmentAssignmentFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL2Fzc2lnbm1lbnQvc2VnbWVudC9TZWdtZW50QXNzaWdubWVudEZhY3RvcnkuamF2YQ==) | 33.33% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15778?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...nment/segment/StrictRealtimeSegmentAssignment.java](https://app.codecov.io/gh/apache/pinot/pull/15778?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fhelix%2Fcore%2Fassignment%2Fsegment%2FStrictRealtimeSegmentAssignment.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL2Fzc2lnbm1lbnQvc2VnbWVudC9TdHJpY3RSZWFsdGltZVNlZ21lbnRBc3NpZ25tZW50LmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15778?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |

<details><summary>Additional details and impacted files</summary>



```diff
@@             Coverage Diff              @@
##             master   #15778      +/-   ##
============================================
+ Coverage     62.90%   63.36%   +0.45%     
- Complexity     1386     1394       +8     
============================================
  Files          2867     2893      +26     
  Lines        163354   165516    +2162     
  Branches      24952    25313     +361     
============================================
+ Hits         102755   104873    +2118     
+ Misses        52847    52721     -126     
- Partials       7752     7922     +170     
```

| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.33% <25.00%> (+0.46%)` | :arrow_up: |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.31% <25.00%> (+0.49%)` | :arrow_up: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.36% <25.00%> (+0.45%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.35% <25.00%> (+0.45%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.34% <ø> (+0.52%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15778/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.54% <25.00%> (-0.03%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15778?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>

- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE