Repository: apache/pinot
Pull request #15261: Consumes segments in strict order of sequence number
Author: login: noob-se7en
Labels: name: bug
description: Something is not working as expected
name: ingestion
description: Related to data ingestion pipeline
name: real-time
description: Related to realtime table ingestion and serving
Created: 2025-03-13T11:27:12Z | Merged: 2025-03-27T01:42:08Z

**Problem Statement**
Dedup and Partial Upserts require consumption of segments to be done in strict order. This is a constraint in-order for the implementation of Dedup and Partial Upserts to work.

**Solution**

TLDR:
- Adds a Coordinator which coordinates all of the offline -> consuming helix transitions.
- Segment order is determined using segment's sequence number present in the segment name.
- Coordinator checks for each offline -> consuming helix transition whether the previous segment is present on server or not.

**Approach**:
1. For the first helix transition (offline -> consuming) of any segment arriving on Server, Always rely on Ideal State to get the previous segment. 

   Example: If the first helix transition (offline -> consuming) of a segment ‘K’ arrives on server, The previous segment will be the segment having the highest sequence number lower than the sequence number of the segment ‘K’.

2. By the time, Above is done, All existing segments will be loaded and the first (offline -> consuming) helix transition will be processed. Whenever a segment is loaded to a server (loaded here means that the segment is registered to segmentDataManagerMap in RealtimeTableDataManager), The Coordinator tracks the highest sequence number across all loaded segments for a particular partition GroupId. The next segment which will be allowed for (offline -> consuming) transition by the Coordinator will be the segment having its previous segment loaded. 

    While the server is alive, Once the first (offline -> consuming) helix transition has been processed, All of the upcoming 
  (offline -> consuming) helix transition will be processed in strict order of sequence number.

     Example: If the first (offline -> consuming) helix transition on server was of segment ‘K’ with sequence number: ‘n’, The second (offline -> consuming) helix transition to go ahead will be of segment with sequence number: n+1, Third will be of sequence number: n+2, fourth: n+3, .., Tenth: n+9, .. and so on.

3. The above sequence order will be maintained until the server restarts. 

     Above will be true because, If (offline -> consuming) helix transition arrives for segment with sequence num: m, It's always guaranteed that controller emitted an (offline -> consuming) helix transition of segment with sequence num: m-1. Event if segment m-1 was deleted, There must be a helix transition like consuming/online -> offline for the segment, which will only arrive on server if offline -> consuming will succeed.

     Since Coordinator tracks the highest sequence number irrespective of whether a segment was deleted or not, the 
     consumption will go on the server in strict order of segment sequence number.



Diff:
diff --git a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
index 1618f73047d6..917a18794681 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
@@ -63,6 +63,12 @@ public enum ServerTimer implements AbstractMetrics.Timer {
   SECONDARY_Q_WAIT_TIME_MS("milliseconds", false,
       "Time spent waiting in the secondary queue when BinaryWorkloadScheduler is used."),
 
+  PREV_SEGMENT_FETCH_IDEAL_STATE_TIME_MS("milliseconds", false,
+      "Time spent while fetching previous segment from ideal state for any segment."),
+
+  PREV_SEGMENT_WAIT_TIME_MS("milliseconds", false,
+      "Time spent while waiting on previous segment to be registered."),
+
   // Multi-stage
   /**
    * Time spent building the hash table for the join.
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
new file mode 100644
index 000000000000..75ba5f73d442
--- /dev/null
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
@@ -0,0 +1,291 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _firstTransitionProcessed;
+
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _firstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    if (_enforceConsumptionInOrder) {
+      long startTimeMs = System.currentTimeMillis();
+      waitForPrevSegment(llcSegmentName);
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(), ServerTimer.PREV_SEGMENT_WAIT_TIME_MS,
+          System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire consumer semaphore for segment: {} in: {}ms. Retrying.", llcSegmentName,
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  public void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  public void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_firstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      awaitForPreviousSegmentFromIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _firstTransitionProcessed.set(true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    awaitForPreviousSegmentFromIdealState(currSegment);
+  }
+
+  private void awaitForPreviousSegmentFromIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    String previousSegment = getPreviousSegmentFromIdealState(currSegment);
+    if (previousSegment == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      long startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          if (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waited on previous segment: {} for: {}ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+            // waited until timeout, fetch previous segment again from ideal state as previous segment might be
+            // changed in ideal state.
+            previousSegment = getPreviousSegmentFromIdealState(currSegment);
+            if (previousSegment == null) {
+              return;
+            }
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+      }
+    } finally {
+      if (segmentDataManager != null) {
+        _realtimeTableDataManager.releaseSegment(segmentDataManager);
+      }
+    }
+  }
+
+  /***
+   * @param currSegment is the segment of current helix transition.
+   * @param timeoutMs is max time to wait in millis
+   * @return true if previous Segment was registered to the server, else false.
+   * @throws InterruptedException
+   */
+  @VisibleForTesting
+  boolean awaitForPreviousSegmentSequenceNumber(LLCSegmentName currSegment, long timeoutMs)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumRegistered < prevSeqNum) {
+        // it means the previous segment is not loaded in the server. Wait until it's loaded.
+        if (!_condition.await(timeoutMs, TimeUnit.MILLISECONDS)) {
+          LOGGER.warn(
+              "Semaphore access denied to segment: {}. Waited on previous segment with sequence number: {} for: {}ms.",
+              currSegment.getSegmentName(), prevSeqNum, System.currentTimeMillis() - startTimeMs);
+          // waited until the timeout. Rely on ideal state now.
+          return _maxSegmentSeqNumRegistered >= prevSeqNum;
+        }
+      }
+      return true;
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  String getPreviousSegmentFromIdealState(LLCSegmentName currSegment) {
+    long startTimeMs = System.currentTimeMillis();
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    String previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+    Map<String, Map<String, String>> segmentAssignment = getSegmentAssignment();
+    String currentServerInstanceId = _realtimeTableDataManager.getServerInstance();
+
+    for (Map.Entry<String, Map<String, String>> entry : segmentAssignment.entrySet()) {
+      String segmentName = entry.getKey();
+      Map<String, String> instanceStateMap = entry.getValue();
+      String state = instanceStateMap.get(currentServerInstanceId);
+
+      if (!CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE.equals(state)) {
+        // if server is looking for previous segment to current transition's segment, it means the previous segment
+        // has to be online in the instance. If all previous segments are not online, we just allow the current helix
+        // transition to go ahead.
+        continue;
+      }
+
+      LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+      if (llcSegmentName == null) {
+        // ignore uploaded segments
+        continue;
+      }
+
+      if (llcSegmentName.getPartitionGroupId() != currPartitionGroupId) {
+        // ignore segments of different partitions.
+        continue;
+      }
+
+      if (llcSegmentName.getSequenceNumber() >= currSequenceNum) {
+        // ignore segments with higher sequence number than existing helix transition segment.
+        continue;
+      }
+
+      if (llcSegmentName.getSequenceNumber() > maxSequenceNumBelowCurrentSegment) {
+        maxSequenceNumBelowCurrentSegment = llcSegmentName.getSequenceNumber();
+        // also track the name of segment
+        previousSegment = segmentName;
+      }
+    }
+
+    long timeSpentMs = System.currentTimeMillis() - startTimeMs;
+    LOGGER.info("Fetched previous segment: {} to current segment: {} in: {}ms.", previousSegment, currSegment,
+        timeSpentMs);
+    _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+        ServerTimer.PREV_SEGMENT_FETCH_IDEAL_STATE_TIME_MS, timeSpentMs, TimeUnit.MILLISECONDS);
+
+    return previousSegment;
+  }
+
+  @VisibleForTesting
+  Map<String, Map<String, String>> getSegmentAssignment() {
+    IdealState idealState = HelixHelper.getTableIdealState(_realtimeTableDataManager.getHelixManager(),
+        _realtimeTableDataManager.getTableName());
+    Preconditions.checkState(idealState != null, "Failed to find ideal state for table: %s",
+        _realtimeTableDataManager.getTableName());
+    return idealState.getRecord().getMapFields();
+  }
+
+  @VisibleForTesting
+  Lock getLock() {
+    return _lock;
+  }
+
+  @VisibleForTesting
+  AtomicBoolean getFirstTransitionProcessed() {
+    return _firstTransitionProcessed;
+  }
+
+  // this should not be used outside of tests.
+  @VisibleForTesting
+  int getMaxSegmentSeqNumLoaded() {
+    return _maxSegmentSeqNumRegistered;
+  }
+}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
index 229799e8d2e7..90113188912f 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
@@ -247,7 +247,7 @@ public void deleteSegmentFile() {
   // Semaphore for each partitionGroupId only, which is to prevent two different stream consumers
   // from consuming with the same partitionGroupId in parallel in the same host.
   // See the comments in {@link RealtimeTableDataManager}.
-  private final Semaphore _partitionGroupConsumerSemaphore;
+  private final ConsumerCoordinator _consumerCoordinator;
   // A boolean flag to check whether the current thread has acquired the semaphore.
   // This boolean is needed because the semaphore is shared by threads; every thread holding this semaphore can
   // modify the permit. This boolean make sure the semaphore gets released only once when the partition group stops
@@ -1066,7 +1066,7 @@ SegmentBuildDescriptor getSegmentBuildDescriptor() {
 
   @VisibleForTesting
   Semaphore getPartitionGroupConsumerSemaphore() {
-    return _partitionGroupConsumerSemaphore;
+    return _consumerCoordinator.getSemaphore();
   }
 
   @VisibleForTesting
@@ -1280,8 +1280,8 @@ private void closeStreamConsumers() {
     closePartitionGroupConsumer();
     closePartitionMetadataProvider();
     if (_acquiredConsumerSemaphore.compareAndSet(true, false)) {
-      _segmentLogger.info("Releasing the _partitionGroupConsumerSemaphore");
-      _partitionGroupConsumerSemaphore.release();
+      _segmentLogger.info("Releasing the consumer semaphore");
+      _consumerCoordinator.release();
     }
   }
 
@@ -1540,7 +1540,7 @@ public void stop()
   // If the transition is OFFLINE to ONLINE, the caller should have downloaded the segment and we don't reach here.
   public RealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig,
       RealtimeTableDataManager realtimeTableDataManager, String resourceDataDir, IndexLoadingConfig indexLoadingConfig,
-      Schema schema, LLCSegmentName llcSegmentName, Semaphore partitionGroupConsumerSemaphore,
+      Schema schema, LLCSegmentName llcSegmentName, ConsumerCoordinator consumerCoordinator,
       ServerMetrics serverMetrics, @Nullable PartitionUpsertMetadataManager partitionUpsertMetadataManager,
       @Nullable PartitionDedupMetadataManager partitionDedupMetadataManager, BooleanSupplier isReadyToConsumeData)
       throws AttemptsExceededException, RetriableOperationException {
@@ -1585,7 +1585,7 @@ public RealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata, TableConf
             _segmentZKMetadata.getEndOffset() == null ? null
                 : _streamPartitionMsgOffsetFactory.create(_segmentZKMetadata.getEndOffset()),
             _segmentZKMetadata.getStatus().toString());
-    _partitionGroupConsumerSemaphore = partitionGroupConsumerSemaphore;
+    _consumerCoordinator = consumerCoordinator;
     _acquiredConsumerSemaphore = new AtomicBoolean(false);
     InstanceDataManagerConfig instanceDataManagerConfig = indexLoadingConfig.getInstanceDataManagerConfig();
     String clientIdSuffix =
@@ -1679,11 +1679,7 @@ public RealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata, TableConf
 
     // Acquire semaphore to create stream consumers
     try {
-      long startTimeMs = System.currentTimeMillis();
-      while (!_partitionGroupConsumerSemaphore.tryAcquire(5, TimeUnit.MINUTES)) {
-        _segmentLogger.warn("Failed to acquire partitionGroupConsumerSemaphore in: {} ms. Retrying.",
-            System.currentTimeMillis() - startTimeMs);
-      }
+      _consumerCoordinator.acquire(llcSegmentName);
       _acquiredConsumerSemaphore.set(true);
     } catch (InterruptedException e) {
       String errorMsg = "InterruptedException when acquiring the partitionConsumerSemaphore";
@@ -1714,8 +1710,8 @@ public RealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata, TableConf
       // In case of exception thrown here, segment goes to ERROR state. Then any attempt to reset the segment from
       // ERROR -> OFFLINE -> CONSUMING via Helix Admin fails because the semaphore is acquired, but not released.
       // Hence releasing the semaphore here to unblock reset operation via Helix Admin.
-      _segmentLogger.info("Releasing the _partitionGroupConsumerSemaphore");
-      _partitionGroupConsumerSemaphore.release();
+      _segmentLogger.info("Releasing the consumer semaphore");
+      _consumerCoordinator.release();
       _acquiredConsumerSemaphore.set(false);
       _realtimeTableDataManager.addSegmentError(_segmentNameStr, new SegmentErrorInfo(now(),
           "Failed to initialize segment data manager", t));
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
index b070b743e08c..7bffd4fdd80f 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
@@ -75,6 +75,8 @@
 import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.UpsertConfig;
+import org.apache.pinot.spi.config.table.ingestion.IngestionConfig;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
 import org.apache.pinot.spi.data.DateTimeFieldSpec;
 import org.apache.pinot.spi.data.DateTimeFormatSpec;
 import org.apache.pinot.spi.data.FieldSpec;
@@ -106,8 +108,8 @@ public class RealtimeTableDataManager extends BaseTableDataManager {
   // consuming from the same stream partition can lead to bugs.
   // The semaphores will stay in the hash map even if the consuming partitions move to a different host.
   // We expect that there will be a small number of semaphores, but that may be ok.
-  private final Map<Integer, Semaphore> _partitionGroupIdToSemaphoreMap = new ConcurrentHashMap<>();
-
+  private final Map<Integer, ConsumerCoordinator> _partitionGroupIdToConsumerCoordinatorMap =
+      new ConcurrentHashMap<>();
   // The old name of the stats file used to be stats.ser which we changed when we moved all packages
   // from com.linkedin to org.apache because of not being able to deserialize the old files using the newer classes
   private static final String STATS_FILE_NAME = "segment-stats.ser";
@@ -142,6 +144,7 @@ public class RealtimeTableDataManager extends BaseTableDataManager {
   private TableDedupMetadataManager _tableDedupMetadataManager;
   private TableUpsertMetadataManager _tableUpsertMetadataManager;
   private BooleanSupplier _isTableReadyToConsumeData;
+  private boolean _enforceConsumptionInOrder = false;
 
   public RealtimeTableDataManager(Semaphore segmentBuildSemaphore) {
     this(segmentBuildSemaphore, () -> true);
@@ -223,6 +226,8 @@ protected void doInit() {
       _tableUpsertMetadataManager.init(_tableConfig, schema, this);
     }
 
+    _enforceConsumptionInOrder = isEnforceConsumptionInOrder();
+
     // For dedup and partial-upsert, need to wait for all segments loaded before starting consuming data
     if (isDedupEnabled() || isPartialUpsertEnabled()) {
       _isTableReadyToConsumeData = new BooleanSupplier() {
@@ -511,7 +516,7 @@ public void addConsumingSegment(String segmentName)
   private void doAddConsumingSegment(String segmentName)
       throws AttemptsExceededException, RetriableOperationException {
     SegmentZKMetadata zkMetadata = fetchZKMetadata(segmentName);
-    if (zkMetadata.getStatus() != Status.IN_PROGRESS) {
+    if ((zkMetadata.getStatus() != Status.IN_PROGRESS) && (!_enforceConsumptionInOrder)) {
       // NOTE: We do not throw exception here because the segment might have just been committed before the state
       //       transition is processed. We can skip adding this segment, and the segment will enter CONSUMING state in
       //       Helix, then we can rely on the following CONSUMING -> ONLINE state transition to add it.
@@ -543,7 +548,7 @@ private void doAddConsumingSegment(String segmentName)
     // Generates only one semaphore for every partition
     LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
     int partitionGroupId = llcSegmentName.getPartitionGroupId();
-    Semaphore semaphore = _partitionGroupIdToSemaphoreMap.computeIfAbsent(partitionGroupId, k -> new Semaphore(1));
+    ConsumerCoordinator consumerCoordinator = getConsumerCoordinator(partitionGroupId);
 
     // Create the segment data manager and register it
     PartitionUpsertMetadataManager partitionUpsertMetadataManager =
@@ -553,8 +558,9 @@ private void doAddConsumingSegment(String segmentName)
         _tableDedupMetadataManager != null ? _tableDedupMetadataManager.getOrCreatePartitionManager(partitionGroupId)
             : null;
     RealtimeSegmentDataManager realtimeSegmentDataManager =
-        createRealtimeSegmentDataManager(zkMetadata, tableConfig, indexLoadingConfig, schema, llcSegmentName, semaphore,
-            partitionUpsertMetadataManager, partitionDedupMetadataManager, _isTableReadyToConsumeData);
+        createRealtimeSegmentDataManager(zkMetadata, tableConfig, indexLoadingConfig, schema, llcSegmentName,
+            consumerCoordinator, partitionUpsertMetadataManager, partitionDedupMetadataManager,
+            _isTableReadyToConsumeData);
     registerSegment(segmentName, realtimeSegmentDataManager, partitionUpsertMetadataManager);
     if (partitionUpsertMetadataManager != null) {
       partitionUpsertMetadataManager.trackNewlyAddedSegment(segmentName);
@@ -641,12 +647,12 @@ private long getDownloadTimeoutMs(TableConfig tableConfig) {
   @VisibleForTesting
   protected RealtimeSegmentDataManager createRealtimeSegmentDataManager(SegmentZKMetadata zkMetadata,
       TableConfig tableConfig, IndexLoadingConfig indexLoadingConfig, Schema schema, LLCSegmentName llcSegmentName,
-      Semaphore semaphore, PartitionUpsertMetadataManager partitionUpsertMetadataManager,
+      ConsumerCoordinator consumerCoordinator, PartitionUpsertMetadataManager partitionUpsertMetadataManager,
       PartitionDedupMetadataManager partitionDedupMetadataManager, BooleanSupplier isTableReadyToConsumeData)
       throws AttemptsExceededException, RetriableOperationException {
     return new RealtimeSegmentDataManager(zkMetadata, tableConfig, this, _indexDir.getAbsolutePath(),
-        indexLoadingConfig, schema, llcSegmentName, semaphore, _serverMetrics, partitionUpsertMetadataManager,
-        partitionDedupMetadataManager, isTableReadyToConsumeData);
+        indexLoadingConfig, schema, llcSegmentName, consumerCoordinator, _serverMetrics,
+        partitionUpsertMetadataManager, partitionDedupMetadataManager, isTableReadyToConsumeData);
   }
 
   /**
@@ -803,6 +809,21 @@ private void registerSegment(String segmentName, SegmentDataManager segmentDataM
     registerSegment(segmentName, segmentDataManager);
   }
 
+  @Override
+  protected SegmentDataManager registerSegment(String segmentName, SegmentDataManager segmentDataManager) {
+    SegmentDataManager oldSegmentDataManager = super.registerSegment(segmentName, segmentDataManager);
+    if (_enforceConsumptionInOrder) {
+      // helix threads might be waiting for their respective previous segments to be loaded.
+      // they need to be notified here.
+      LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+      if (llcSegmentName != null) {
+        ConsumerCoordinator consumerCoordinator = getConsumerCoordinator(llcSegmentName.getPartitionGroupId());
+        consumerCoordinator.trackSegment(llcSegmentName);
+      }
+    }
+    return oldSegmentDataManager;
+  }
+
   /**
    * Replaces the CONSUMING segment with a downloaded committed one.
    */
@@ -852,6 +873,23 @@ public Map<Integer, Long> getUpsertPartitionToPrimaryKeyCount() {
     return Collections.emptyMap();
   }
 
+  @Nullable
+  public StreamIngestionConfig getStreamIngestionConfig() {
+    IngestionConfig ingestionConfig = _tableConfig.getIngestionConfig();
+    return ingestionConfig != null ? ingestionConfig.getStreamIngestionConfig() : null;
+  }
+
+  @VisibleForTesting
+  ConsumerCoordinator getConsumerCoordinator(int partitionGroupId) {
+    return _partitionGroupIdToConsumerCoordinatorMap.computeIfAbsent(partitionGroupId,
+        k -> new ConsumerCoordinator(_enforceConsumptionInOrder, this));
+  }
+
+  @VisibleForTesting
+  void setEnforceConsumptionInOrder(boolean enforceConsumptionInOrder) {
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+  }
+
   /**
    * Validate a schema against the table config for real-time record consumption.
    * Ideally, we should validate these things when schema is added or table is created, but either of these
@@ -886,4 +924,9 @@ private void validate(TableConfig tableConfig, Schema schema) {
     // 2. Validate the schema itself
     SchemaUtils.validate(schema);
   }
+
+  private boolean isEnforceConsumptionInOrder() {
+    StreamIngestionConfig streamIngestionConfig = getStreamIngestionConfig();
+    return streamIngestionConfig != null && streamIngestionConfig.isEnforceConsumptionInOrder();
+  }
 }
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinatorTest.java b/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinatorTest.java
new file mode 100644
index 000000000000..a2b01a44ea61
--- /dev/null
+++ b/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinatorTest.java
@@ -0,0 +1,485 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.cache.CacheBuilder;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.ReentrantLock;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.util.TestUtils;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+
+public class ConsumerCoordinatorTest {
+
+  private static class FakeRealtimeTableDataManager extends RealtimeTableDataManager {
+    private final StreamIngestionConfig _streamIngestionConfig;
+    private ConsumerCoordinator _consumerCoordinator;
+
+    public FakeRealtimeTableDataManager(Semaphore segmentBuildSemaphore,
+        boolean useIdealStateToCalculatePreviousSegment) {
+      super(segmentBuildSemaphore);
+      super._recentlyDeletedSegments = CacheBuilder.newBuilder().build();
+      StreamIngestionConfig streamIngestionConfig = new StreamIngestionConfig(List.of(new HashMap<>()));
+      streamIngestionConfig.setEnforceConsumptionInOrder(true);
+      if (useIdealStateToCalculatePreviousSegment) {
+        streamIngestionConfig.setUseIdealStateToCalculatePreviousSegment(true);
+      }
+      _streamIngestionConfig = streamIngestionConfig;
+    }
+
+    @Override
+    ConsumerCoordinator getConsumerCoordinator(int partitionId) {
+      return _consumerCoordinator;
+    }
+
+    public void setConsumerCoordinator(ConsumerCoordinator consumerCoordinator) {
+      _consumerCoordinator = consumerCoordinator;
+    }
+
+    @Override
+    public StreamIngestionConfig getStreamIngestionConfig() {
+      return _streamIngestionConfig;
+    }
+
+    @Override
+    public String getServerInstance() {
+      return "server_1";
+    }
+  }
+
+  private static class FakeConsumerCoordinator extends ConsumerCoordinator {
+    private final Map<String, Map<String, String>> _segmentAssignmentMap;
+
+    public FakeConsumerCoordinator(boolean enforceConsumptionInOrder,
+        RealtimeTableDataManager realtimeTableDataManager) {
+      super(enforceConsumptionInOrder, realtimeTableDataManager);
+      Map<String, String> serverSegmentStatusMap = new HashMap<>() {{
+        put("server_1", "ONLINE");
+        put("server_3", "ONLINE");
+      }};
+      _segmentAssignmentMap = new HashMap<>() {{
+        put("tableTest_REALTIME__1__101__20250304T0035Z", serverSegmentStatusMap);
+        put("tableTest_REALTIME__2__101__20250304T0035Z", serverSegmentStatusMap);
+        put("tableTest_REALTIME__2__100__20250304T0035Z", serverSegmentStatusMap);
+        put("tableTest_REALTIME__1__1__20250304T0035Z", serverSegmentStatusMap);
+        put("tableTest_REALTIME__1__14__20250304T0035Z", serverSegmentStatusMap);
+        put("tableTest_REALTIME__1__91__20250304T0035Z", serverSegmentStatusMap);
+        put("tableTest_REALTIME__1__90__20250304T0035Z", serverSegmentStatusMap);
+      }};
+    }
+
+    @Override
+    public Map<String, Map<String, String>> getSegmentAssignment() {
+      return _segmentAssignmentMap;
+    }
+  }
+
+  @Test
+  public void testAwaitForPreviousSegmentSequenceNumber()
+      throws InterruptedException {
+    // 1. enable tracking segment seq num.
+    FakeRealtimeTableDataManager realtimeTableDataManager = new FakeRealtimeTableDataManager(null, false);
+    realtimeTableDataManager.setEnforceConsumptionInOrder(true);
+    FakeConsumerCoordinator consumerCoordinator = new FakeConsumerCoordinator(true, realtimeTableDataManager);
+    realtimeTableDataManager.setConsumerCoordinator(consumerCoordinator);
+
+    // 2. check if thread waits on prev segment seq
+    AtomicBoolean atomicBoolean = new AtomicBoolean(false);
+    Thread thread1 = new Thread(() -> {
+      LLCSegmentName llcSegmentName = getLLCSegment(101);
+      try {
+        boolean b = consumerCoordinator.awaitForPreviousSegmentSequenceNumber(llcSegmentName, 5000);
+        atomicBoolean.set(b);
+      } catch (InterruptedException e) {
+        throw new RuntimeException(e);
+      }
+    });
+    thread1.start();
+
+    // 3. add prev segment and check if thread is unblocked.
+    consumerCoordinator.trackSegment(getLLCSegment(100));
+
+    TestUtils.waitForCondition(aVoid -> atomicBoolean.get(), 4000,
+        "Thread waiting on previous segment should have been unblocked.");
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 100);
+
+    // 4. check if second thread waits on prev segment seq until timeout and returns false
+    AtomicBoolean atomicBoolean2 = new AtomicBoolean(false);
+    Thread thread2 = new Thread(() -> {
+      LLCSegmentName llcSegmentName = getLLCSegment(102);
+      try {
+        boolean b = consumerCoordinator.awaitForPreviousSegmentSequenceNumber(llcSegmentName, 500);
+        atomicBoolean2.set(b);
+      } catch (InterruptedException e) {
+        throw new RuntimeException(e);
+      }
+    });
+    thread2.start();
+
+    Thread.sleep(1500);
+
+    Assert.assertFalse(atomicBoolean2.get());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 100);
+  }
+
+  @Test
+  public void testFirstConsumer()
+      throws InterruptedException {
+    // 1. Enable tracking segment seq num.
+    FakeRealtimeTableDataManager realtimeTableDataManager = new FakeRealtimeTableDataManager(null, false);
+    realtimeTableDataManager.setEnforceConsumptionInOrder(true);
+    FakeConsumerCoordinator consumerCoordinator = new FakeConsumerCoordinator(true, realtimeTableDataManager);
+    realtimeTableDataManager.setConsumerCoordinator(consumerCoordinator);
+    ReentrantLock lock = (ReentrantLock) consumerCoordinator.getLock();
+    RealtimeSegmentDataManager mockedRealtimeSegmentDataManager = getMockedRealtimeSegmentDataManager();
+    Map<String, String> serverSegmentStatusMap = new HashMap<>() {{
+      put("server_1", "ONLINE");
+      put("server_3", "ONLINE");
+    }};
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(100), serverSegmentStatusMap);
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(102), serverSegmentStatusMap);
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(104), serverSegmentStatusMap);
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(106), serverSegmentStatusMap);
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(107), serverSegmentStatusMap);
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(109), serverSegmentStatusMap);
+
+    // 2. create multiple helix transitions in this order: 106, 109, 104, 107
+    Thread thread1 = getNewThread(consumerCoordinator, getLLCSegment(106));
+    Thread thread2 = getNewThread(consumerCoordinator, getLLCSegment(109));
+    Thread thread3 = getNewThread(consumerCoordinator, getLLCSegment(104));
+    Thread thread4 = getNewThread(consumerCoordinator, getLLCSegment(107));
+
+    thread1.start();
+
+    Thread.sleep(1000);
+
+    // 3. load segment 100, 101, 102
+    realtimeTableDataManager.registerSegment(getSegmentName(100), mockedRealtimeSegmentDataManager);
+    realtimeTableDataManager.registerSegment(getSegmentName(101), mockedRealtimeSegmentDataManager);
+    realtimeTableDataManager.registerSegment(getSegmentName(102), mockedRealtimeSegmentDataManager);
+    Thread.sleep(1000);
+
+    // 4. check all of the above threads wait
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 102);
+    Assert.assertFalse(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    thread2.start();
+    thread3.start();
+    thread4.start();
+
+    Thread.sleep(1000);
+
+    // 5. check that first thread acquiring semaphore is of segment 104
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 102);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    realtimeTableDataManager.registerSegment(getSegmentName(104), mockedRealtimeSegmentDataManager);
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 104);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 1);
+
+    // 6. check the next threads acquiring semaphore is 106
+    consumerCoordinator.getSemaphore().release();
+    realtimeTableDataManager.registerSegment(getSegmentName(106), mockedRealtimeSegmentDataManager);
+
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 106);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 1);
+  }
+
+  @Test
+  public void testSequentialOrderNotRelyingOnIdealState()
+      throws InterruptedException {
+    // 1. Enable tracking segment seq num.
+    FakeRealtimeTableDataManager realtimeTableDataManager = new FakeRealtimeTableDataManager(null, false);
+    realtimeTableDataManager.setEnforceConsumptionInOrder(true);
+
+    FakeConsumerCoordinator consumerCoordinator = new FakeConsumerCoordinator(true, realtimeTableDataManager);
+    realtimeTableDataManager.setConsumerCoordinator(consumerCoordinator);
+    ReentrantLock lock = (ReentrantLock) consumerCoordinator.getLock();
+
+    // 2. check first transition blocked on ideal state
+    Thread thread1 = getNewThread(consumerCoordinator, getLLCSegment(101));
+    thread1.start();
+
+    Thread.sleep(2000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), -1);
+    Assert.assertFalse(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    RealtimeSegmentDataManager mockedRealtimeSegmentDataManager = getMockedRealtimeSegmentDataManager();
+
+    // 3. register older segment and check seq num watermark and semaphore.
+    realtimeTableDataManager.registerSegment(getSegmentName(90), mockedRealtimeSegmentDataManager);
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 90);
+    Assert.assertFalse(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    // 4. register prev segment and check watermark and if thread was unblocked
+    realtimeTableDataManager.registerSegment(getSegmentName(91), mockedRealtimeSegmentDataManager);
+    Thread.sleep(1000);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 91);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    // 5. check that all the following transitions rely on seq num watermark and gets blocked.
+    Thread thread2 = getNewThread(consumerCoordinator, getLLCSegment(102));
+    Thread thread3 = getNewThread(consumerCoordinator, getLLCSegment(103));
+    Thread thread4 = getNewThread(consumerCoordinator, getLLCSegment(104));
+    thread3.start();
+    thread2.start();
+    thread4.start();
+
+    Thread.sleep(2000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 91);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    // 6. check that all above threads are still blocked even if semaphore is released.
+    consumerCoordinator.getSemaphore().release();
+
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 91);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    // 6. mark 101 seg as complete. Check 102 acquired the semaphore.
+    realtimeTableDataManager.registerSegment(getSegmentName(101), mockedRealtimeSegmentDataManager);
+
+    Thread.sleep(1000);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 101);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 0);
+
+    // 7. register 102 seg, check if seg 103 is waiting on semaphore.
+    realtimeTableDataManager.registerSegment(getSegmentName(102), mockedRealtimeSegmentDataManager);
+
+    Thread.sleep(1000);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 102);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 1);
+
+    // 8. release the semaphore and check if semaphore is acquired by seg 103.
+    consumerCoordinator.getSemaphore().release();
+    Thread.sleep(1000);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 102);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 0);
+
+    // 8. register 103 seg and check if seg 104 is now queued on semaphore
+    realtimeTableDataManager.registerSegment(getSegmentName(103), mockedRealtimeSegmentDataManager);
+    Thread.sleep(1000);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), 103);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 1);
+  }
+
+  @Test
+  public void testSequentialOrderRelyingOnIdealState()
+      throws InterruptedException {
+    FakeRealtimeTableDataManager realtimeTableDataManager = new FakeRealtimeTableDataManager(null, true);
+    realtimeTableDataManager.setEnforceConsumptionInOrder(true);
+
+    FakeConsumerCoordinator consumerCoordinator = new FakeConsumerCoordinator(true, realtimeTableDataManager);
+    realtimeTableDataManager.setConsumerCoordinator(consumerCoordinator);
+
+    // 1. test that acquire blocks when prev segment is not loaded.
+    Thread thread = getNewThread(consumerCoordinator, getLLCSegment(101));
+    thread.start();
+
+    ReentrantLock lock = (ReentrantLock) consumerCoordinator.getLock();
+
+    RealtimeSegmentDataManager mockedRealtimeSegmentDataManager = Mockito.mock(RealtimeSegmentDataManager.class);
+    Mockito.when(mockedRealtimeSegmentDataManager.increaseReferenceCount()).thenReturn(true);
+    Assert.assertNotNull(realtimeTableDataManager);
+
+    // prev segment has seq 91, so registering seq 90 won't do anything.
+    realtimeTableDataManager.registerSegment(getSegmentName(90), mockedRealtimeSegmentDataManager);
+
+    Thread.sleep(2000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+
+    // 2. test that registering prev segment will unblock thread.
+    realtimeTableDataManager.registerSegment(getSegmentName(91), mockedRealtimeSegmentDataManager);
+
+    TestUtils.waitForCondition(aVoid -> (consumerCoordinator.getSemaphore().availablePermits() == 0), 5000,
+        "Semaphore must be acquired after registering previous segment");
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertTrue(consumerCoordinator.getFirstTransitionProcessed().get());
+
+    consumerCoordinator.release();
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+    Assert.assertFalse(consumerCoordinator.getSemaphore().hasQueuedThreads());
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), -1);
+    realtimeTableDataManager.registerSegment(getSegmentName(101), mockedRealtimeSegmentDataManager);
+
+    // 3. test that segment 103 will be blocked.
+    Map<String, String> serverSegmentStatusMap = new HashMap<>() {{
+      put("server_1", "ONLINE");
+      put("server_3", "ONLINE");
+    }};
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(102), serverSegmentStatusMap);
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(103), serverSegmentStatusMap);
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(104), serverSegmentStatusMap);
+
+    Thread thread1 = getNewThread(consumerCoordinator, getLLCSegment(103));
+    thread1.start();
+
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+
+    // 3. test that segment 102 will acquire semaphore.
+    Thread thread2 = getNewThread(consumerCoordinator, getLLCSegment(102));
+    thread2.start();
+
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(consumerCoordinator.getSemaphore().hasQueuedThreads());
+
+    // 4. registering seg 102 should unblock seg 103
+    realtimeTableDataManager.registerSegment(getSegmentName(102), mockedRealtimeSegmentDataManager);
+
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 1);
+
+    // 5. releasing semaphore should let seg 103 acquire it
+    consumerCoordinator.getSemaphore().release();
+
+    Thread.sleep(1000);
+
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().getQueueLength(), 0);
+    Assert.assertFalse(lock.hasQueuedThreads() && lock.isLocked());
+    Assert.assertEquals(consumerCoordinator.getMaxSegmentSeqNumLoaded(), -1);
+  }
+
+  @Test
+  public void testRandomOrder()
+      throws InterruptedException {
+    RealtimeTableDataManager realtimeTableDataManager = Mockito.mock(RealtimeTableDataManager.class);
+    Mockito.when(realtimeTableDataManager.getTableName()).thenReturn("tableTest_REALTIME");
+
+    FakeConsumerCoordinator consumerCoordinator = new FakeConsumerCoordinator(false, realtimeTableDataManager);
+
+    String segmentName = "tableTest_REALTIME__1__101__20250304T0035Z";
+    LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+    Assert.assertNotNull(llcSegmentName);
+    consumerCoordinator.acquire(llcSegmentName);
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 0);
+    Assert.assertFalse(consumerCoordinator.getSemaphore().hasQueuedThreads());
+
+    consumerCoordinator.release();
+    Assert.assertEquals(consumerCoordinator.getSemaphore().availablePermits(), 1);
+    Assert.assertFalse(consumerCoordinator.getSemaphore().hasQueuedThreads());
+  }
+
+  @Test
+  public void testPreviousSegment() {
+    RealtimeTableDataManager realtimeTableDataManager = Mockito.mock(RealtimeTableDataManager.class);
+    Mockito.when(realtimeTableDataManager.getTableName()).thenReturn("tableTest_REALTIME");
+    Mockito.when(realtimeTableDataManager.getServerInstance()).thenReturn("server_1");
+
+    FakeConsumerCoordinator consumerCoordinator = new FakeConsumerCoordinator(true, realtimeTableDataManager);
+
+    String segmentName = "tableTest_REALTIME__1__101__20250304T0035Z";
+    LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+    Assert.assertNotNull(llcSegmentName);
+    String previousSegment = consumerCoordinator.getPreviousSegmentFromIdealState(llcSegmentName);
+    Assert.assertEquals(previousSegment, "tableTest_REALTIME__1__91__20250304T0035Z");
+
+    consumerCoordinator.getSegmentAssignment().clear();
+    Map<String, String> serverSegmentStatusMap = new HashMap<>() {{
+      put("server_3", "ONLINE");
+    }};
+    consumerCoordinator.getSegmentAssignment().put(getSegmentName(100), serverSegmentStatusMap);
+    previousSegment = consumerCoordinator.getPreviousSegmentFromIdealState(llcSegmentName);
+    Assert.assertNull(previousSegment);
+  }
+
+  private Thread getNewThread(FakeConsumerCoordinator consumerCoordinator, LLCSegmentName llcSegmentName) {
+    return new Thread(() -> {
+      try {
+        consumerCoordinator.acquire(llcSegmentName);
+      } catch (InterruptedException e) {
+        throw new RuntimeException(e);
+      }
+    }, String.valueOf(llcSegmentName.getSequenceNumber()));
+  }
+
+  private RealtimeSegmentDataManager getMockedRealtimeSegmentDataManager() {
+    RealtimeSegmentDataManager mockedRealtimeSegmentDataManager = Mockito.mock(RealtimeSegmentDataManager.class);
+    Mockito.when(mockedRealtimeSegmentDataManager.increaseReferenceCount()).thenReturn(true);
+    Assert.assertNotNull(mockedRealtimeSegmentDataManager);
+    return mockedRealtimeSegmentDataManager;
+  }
+
+  private LLCSegmentName getLLCSegment(int seqNum) {
+    String segmentName = getSegmentName(seqNum);
+    LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+    Assert.assertNotNull(llcSegmentName);
+    return llcSegmentName;
+  }
+
+  private String getSegmentName(int seqNum) {
+    return "tableTest_REALTIME__1__" + seqNum + "__20250304T0035Z";
+  }
+}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManagerTest.java b/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManagerTest.java
index 623d0de37dfb..72dbf26acb5a 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManagerTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManagerTest.java
@@ -97,7 +97,8 @@ public class RealtimeSegmentDataManagerTest {
   private static final long START_OFFSET_VALUE = 198L;
   private static final LongMsgOffset START_OFFSET = new LongMsgOffset(START_OFFSET_VALUE);
 
-  private final Map<Integer, Semaphore> _partitionGroupIdToSemaphoreMap = new ConcurrentHashMap<>();
+  private final Map<Integer, ConsumerCoordinator> _partitionGroupIdToConsumerCoordinatorMap =
+      new ConcurrentHashMap<>();
 
   private static TableConfig createTableConfig()
       throws Exception {
@@ -166,12 +167,13 @@ private FakeRealtimeSegmentDataManager createFakeSegmentManager(boolean noUpsert
     tableConfig.getIngestionConfig().setRetryOnSegmentBuildPrecheckFailure(true);
     RealtimeTableDataManager tableDataManager = createTableDataManager(tableConfig);
     LLCSegmentName llcSegmentName = new LLCSegmentName(SEGMENT_NAME_STR);
-    _partitionGroupIdToSemaphoreMap.putIfAbsent(PARTITION_GROUP_ID, new Semaphore(1));
+    _partitionGroupIdToConsumerCoordinatorMap.putIfAbsent(PARTITION_GROUP_ID,
+        new ConsumerCoordinator(false, tableDataManager));
     Schema schema = Fixtures.createSchema();
     ServerMetrics serverMetrics = new ServerMetrics(PinotMetricUtils.getPinotMetricsRegistry());
     return new FakeRealtimeSegmentDataManager(segmentZKMetadata, tableConfig, tableDataManager,
         new File(TEMP_DIR, REALTIME_TABLE_NAME).getAbsolutePath(), schema, llcSegmentName,
-        _partitionGroupIdToSemaphoreMap, serverMetrics, timeSupplier);
+        _partitionGroupIdToConsumerCoordinatorMap, serverMetrics, timeSupplier);
   }
 
   @BeforeClass
@@ -992,7 +994,7 @@ public static class FakeRealtimeSegmentDataManager extends RealtimeSegmentDataMa
     private boolean _notifySegmentBuildFailedWithDeterministicErrorCalled = false;
     public boolean _throwExceptionFromConsume = false;
     public boolean _postConsumeStoppedCalled = false;
-    public Map<Integer, Semaphore> _semaphoreMap;
+    public Map<Integer, ConsumerCoordinator> _consumerCoordinatorMap;
     public boolean _stubConsumeLoop = true;
     private TimeSupplier _timeSupplier;
     private boolean _indexCapacityThresholdBreached;
@@ -1009,12 +1011,13 @@ private static InstanceDataManagerConfig makeInstanceDataManagerConfig() {
 
     public FakeRealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig,
         RealtimeTableDataManager realtimeTableDataManager, String resourceDataDir, Schema schema,
-        LLCSegmentName llcSegmentName, Map<Integer, Semaphore> semaphoreMap, ServerMetrics serverMetrics,
-        TimeSupplier timeSupplier)
+        LLCSegmentName llcSegmentName, Map<Integer, ConsumerCoordinator> consumerCoordinatorMap,
+        ServerMetrics serverMetrics, TimeSupplier timeSupplier)
         throws Exception {
       super(segmentZKMetadata, tableConfig, realtimeTableDataManager, resourceDataDir,
           new IndexLoadingConfig(makeInstanceDataManagerConfig(), tableConfig), schema, llcSegmentName,
-          semaphoreMap.get(llcSegmentName.getPartitionGroupId()), serverMetrics, null, null, () -> true);
+          consumerCoordinatorMap.get(llcSegmentName.getPartitionGroupId()), serverMetrics, null, null,
+          () -> true);
       _state = RealtimeSegmentDataManager.class.getDeclaredField("_state");
       _state.setAccessible(true);
       _shouldStop = RealtimeSegmentDataManager.class.getDeclaredField("_shouldStop");
@@ -1024,7 +1027,7 @@ public FakeRealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata, Table
       _segmentBuildFailedWithDeterministicError =
           RealtimeSegmentDataManager.class.getDeclaredField("_segmentBuildFailedWithDeterministicError");
       _segmentBuildFailedWithDeterministicError.setAccessible(true);
-      _semaphoreMap = semaphoreMap;
+      _consumerCoordinatorMap = consumerCoordinatorMap;
       _streamMsgOffsetFactory = RealtimeSegmentDataManager.class.getDeclaredField("_streamPartitionMsgOffsetFactory");
       _streamMsgOffsetFactory.setAccessible(true);
       _streamMsgOffsetFactory.set(this, new LongMsgOffsetFactory());
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeSegmentDataManager.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeSegmentDataManager.java
index 960138e8a088..8899fbd723dd 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeSegmentDataManager.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeSegmentDataManager.java
@@ -18,10 +18,10 @@
  */
 package org.apache.pinot.integration.tests.realtime.utils;
 
-import java.util.concurrent.Semaphore;
 import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
 import org.apache.pinot.common.metrics.ServerMetrics;
 import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.core.data.manager.realtime.ConsumerCoordinator;
 import org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager;
 import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
 import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
@@ -43,16 +43,15 @@ public class FailureInjectingRealtimeSegmentDataManager extends RealtimeSegmentD
   /**
    * Creates a manager that will forcibly fail the commit segment step.
    */
-  public FailureInjectingRealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata,
-      TableConfig tableConfig, RealtimeTableDataManager realtimeTableDataManager, String resourceDataDir,
-      IndexLoadingConfig indexLoadingConfig, Schema schema, LLCSegmentName llcSegmentName,
-      Semaphore partitionGroupConsumerSemaphore, ServerMetrics serverMetrics,
-      boolean failCommit) throws AttemptsExceededException, RetriableOperationException {
+  public FailureInjectingRealtimeSegmentDataManager(SegmentZKMetadata segmentZKMetadata, TableConfig tableConfig,
+      RealtimeTableDataManager realtimeTableDataManager, String resourceDataDir, IndexLoadingConfig indexLoadingConfig,
+      Schema schema, LLCSegmentName llcSegmentName, ConsumerCoordinator consumerCoordinator,
+      ServerMetrics serverMetrics, boolean failCommit)
+      throws AttemptsExceededException, RetriableOperationException {
     // Pass through to the real parent constructor
-    super(segmentZKMetadata, tableConfig, realtimeTableDataManager, resourceDataDir,
-        indexLoadingConfig, schema, llcSegmentName, partitionGroupConsumerSemaphore, serverMetrics,
-        null /* no PartitionUpsertMetadataManager */, null /* no PartitionDedupMetadataManager */,
-        () -> true /* isReadyToConsumeData always true for tests */);
+    super(segmentZKMetadata, tableConfig, realtimeTableDataManager, resourceDataDir, indexLoadingConfig, schema,
+        llcSegmentName, consumerCoordinator, serverMetrics, null /* no PartitionUpsertMetadataManager */,
+        null /* no PartitionDedupMetadataManager */, () -> true /* isReadyToConsumeData always true for tests */);
 
     _failCommit = failCommit;
   }
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeTableDataManager.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeTableDataManager.java
index 278e2bcb589e..af65cd1a7929 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeTableDataManager.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/realtime/utils/FailureInjectingRealtimeTableDataManager.java
@@ -25,6 +25,7 @@
 import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
 import org.apache.pinot.common.utils.LLCSegmentName;
 import org.apache.pinot.common.utils.PauselessConsumptionUtils;
+import org.apache.pinot.core.data.manager.realtime.ConsumerCoordinator;
 import org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager;
 import org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager;
 import org.apache.pinot.segment.local.dedup.PartitionDedupMetadataManager;
@@ -52,7 +53,7 @@ public FailureInjectingRealtimeTableDataManager(Semaphore segmentBuildSemaphore,
   @Override
   protected RealtimeSegmentDataManager createRealtimeSegmentDataManager(SegmentZKMetadata zkMetadata,
       TableConfig tableConfig, IndexLoadingConfig indexLoadingConfig, Schema schema, LLCSegmentName llcSegmentName,
-      Semaphore semaphore, PartitionUpsertMetadataManager partitionUpsertMetadataManager,
+      ConsumerCoordinator consumerCoordinator, PartitionUpsertMetadataManager partitionUpsertMetadataManager,
       PartitionDedupMetadataManager partitionDedupMetadataManager, BooleanSupplier isTableReadyToConsumeData)
       throws AttemptsExceededException, RetriableOperationException {
 
@@ -61,6 +62,6 @@ protected RealtimeSegmentDataManager createRealtimeSegmentDataManager(SegmentZKM
       addFailureToCommits = false;
     }
     return new FailureInjectingRealtimeSegmentDataManager(zkMetadata, tableConfig, this, _indexDir.getAbsolutePath(),
-        indexLoadingConfig, schema, llcSegmentName, semaphore, _serverMetrics, addFailureToCommits);
+        indexLoadingConfig, schema, llcSegmentName, consumerCoordinator, _serverMetrics, addFailureToCommits);
   }
 }
diff --git a/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java b/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
index e1400620fbc2..da17a2c0207b 100644
--- a/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
+++ b/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
@@ -39,10 +39,17 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   private boolean _columnMajorSegmentBuilderEnabled = true;
 
   @JsonPropertyDescription("Whether to track offsets of the filtered stream messages during consumption.")
-  private boolean _trackFilteredMessageOffsets = false;
+  private boolean _trackFilteredMessageOffsets;
 
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
-  private boolean _pauselessConsumptionEnabled = false;
+  private boolean _pauselessConsumptionEnabled;
+
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder;
+
+  @JsonPropertyDescription("If enabled, Server always relies on ideal state to get previous segment. If disabled, "
+      + "server uses sequence id - 1 for previous segment")
+  private boolean _useIdealStateToCalculatePreviousSegment;
 
   @JsonPropertyDescription("Policy to determine the behaviour of parallel consumption.")
   private ParallelSegmentConsumptionPolicy _parallelSegmentConsumptionPolicy;
@@ -80,6 +87,22 @@ public void setPauselessConsumptionEnabled(boolean pauselessConsumptionEnabled)
     _pauselessConsumptionEnabled = pauselessConsumptionEnabled;
   }
 
+  public boolean isEnforceConsumptionInOrder() {
+    return _enforceConsumptionInOrder;
+  }
+
+  public void setEnforceConsumptionInOrder(boolean enforceConsumptionInOrder) {
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+  }
+
+  public boolean isUseIdealStateToCalculatePreviousSegment() {
+    return _useIdealStateToCalculatePreviousSegment;
+  }
+
+  public void setUseIdealStateToCalculatePreviousSegment(boolean useIdealStateToCalculatePreviousSegment) {
+    _useIdealStateToCalculatePreviousSegment = useIdealStateToCalculatePreviousSegment;
+  }
+
   @Nullable
   public ParallelSegmentConsumptionPolicy getParallelSegmentConsumptionPolicy() {
     return _parallelSegmentConsumptionPolicy;


Reviews:
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T12:32:30Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T07:23:46Z
user: login: Jackie-Jiang
body: If new consuming segment is assigned to a different server (e.g. new server added to the tenant), the state transition will be blocked forever.
Does it work if we do a try-lock, and keep checking the segment status (CONSUMING/DONE) if lock cannot be acquired?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T21:24:28Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T12:05:35Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T12:06:28Z
user: login: Jackie-Jiang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T18:09:54Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T19:23:30Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T19:26:20Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T19:29:21Z
user: login: KKcorps
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:00:34Z
user: login: KKcorps
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:00:51Z
user: login: KKcorps
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:02:28Z
user: login: KKcorps
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:02:53Z
user: login: KKcorps
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:03:33Z
user: login: KKcorps
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:06:51Z
user: login: KKcorps
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:13:29Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:57:34Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T07:00:33Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T07:13:03Z
user: login: Jackie-Jiang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T21:19:24Z
user: login: Jackie-Jiang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T21:47:26Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:44:57Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:52:20Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:58:49Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:08:10Z
user: login: Jackie-Jiang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T22:55:52Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T12:29:34Z
user: login: Jackie-Jiang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T17:48:31Z
user: login: Jackie-Jiang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T17:50:31Z
user: login: Jackie-Jiang
body: Mostly good
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T19:15:25Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T20:36:44Z
user: login: noob-se7en
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T20:40:42Z
user: login: Jackie-Jiang
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T01:41:58Z

Review comments:
diff_hunk: @@ -1063,12 +1063,7 @@ AtomicBoolean getAcquiredConsumerSemaphore() {
 
   @VisibleForTesting
   protected SegmentBuildDescriptor buildSegmentInternal(boolean forCommit) {
-    // for partial upsert tables, do not release _partitionGroupConsumerSemaphore proactively and rely on offload()
-    // to release the semaphore. This ensures new consuming segment is not consuming until the segment replacement is
-    // complete.
-    if (_allowConsumptionDuringCommit) {
-      closeStreamConsumers();
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
user: login: noob-se7en
body: Need to confirm side effect of removing this. Since  A new consumer can start consuming and updating the dedup metadata and immutable segment of currently building segment also accesses the metadata.
created_at: 2025-03-13T12:32:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1063,12 +1063,7 @@ AtomicBoolean getAcquiredConsumerSemaphore() {
 
   @VisibleForTesting
   protected SegmentBuildDescriptor buildSegmentInternal(boolean forCommit) {
-    // for partial upsert tables, do not release _partitionGroupConsumerSemaphore proactively and rely on offload()
-    // to release the semaphore. This ensures new consuming segment is not consuming until the segment replacement is
-    // complete.
-    if (_allowConsumptionDuringCommit) {
-      closeStreamConsumers();
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java
user: login: noob-se7en
body: will handle this in dedicate separate PR.
created_at: 2025-03-17T07:23:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -927,6 +927,8 @@ private void createNewSegmentZKMetadata(TableConfig tableConfig, StreamConfig st
       newSegmentZKMetadata.setPartitionMetadata(partitionMetadata);
     }
 
+    newSegmentZKMetadata.setPreviousSegment(committingSegmentDescriptor.getSegmentName());
path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
user: login: Jackie-Jiang
body: Could this be `null`? We should probably check if `committingSegmentZKMetadata == null`
created_at: 2025-03-18T21:21:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  @Nullable
+  private Set<Integer> _segmentSequenceNumSet = null;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _segmentSequenceNumSet = new HashSet<>();
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: noob-se7en
body: This is unrelated to the problem this PR is fixing. It's just an optimisation.
created_at: 2025-03-21T12:05:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  @Nullable
+  private Set<Integer> _segmentSequenceNumSet = null;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _segmentSequenceNumSet = new HashSet<>();
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: noob-se7en
body: Can be separated out from this PR
created_at: 2025-03-21T12:06:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  @Nullable
+  private Set<Integer> _segmentSequenceNumSet = null;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _segmentSequenceNumSet = new HashSet<>();
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
+      }
+    }
+  }
+
+  public void release() {
+    _semaphore.release();
+  }
+
+  public Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  public void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (_segmentSequenceNumSet != null) {
+        _segmentSequenceNumSet.add(llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitUntilTableIsReady()
+      throws InterruptedException {
+    while (!_realtimeTableDataManager.getIsTableReadyToConsumeData().getAsBoolean()) {
+      Thread.sleep(RealtimeTableDataManager.READY_TO_CONSUME_DATA_CHECK_INTERVAL_MS);
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_segmentSequenceNumSet == null) || (!_isFirstTransitionProcessed.get())) {
+      // if _segmentSequenceNumSet is null or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (!_segmentSequenceNumSet.contains(prevSeqNum)) {
+        // if _segmentSequenceNumSet does not contain prev segment's seq num, it means segment is not loaded in the
+        // server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @Nullable
+  private String getPreviousSegment(LLCSegmentName currSegment) {
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    String previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+
+    Map<String, Map<String, String>> segmentAssignment =
+        HelixHelper.getSegmentAssignment(_realtimeTableDataManager.getTableName(),
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: Jackie-Jiang
body: I prefer the previous version of storing previous segment in segment ZK metadata. Reading ideal state per consuming segment could cause overhead for large table.
You may also store the sequence_id of the previous segment
created_at: 2025-03-21T18:01:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: Jackie-Jiang
body: Since you are extracting this out, consider changing the `Semaphore` to a `Lock`. It is not really used as a `Semaphore`
created_at: 2025-03-21T18:02:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
+
+  @JsonPropertyDescription("If enabled, Server in-memory tracks the sequence number of all loaded segments")
+  private boolean _trackSegmentSeqNumber = true;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: Jackie-Jiang
body: We probably don't need to have a config for this. Does it work if we only keep a watermark of highest sequence id of the loaded segment?
created_at: 2025-03-21T18:06:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: Jackie-Jiang
body: We should always enforce order for dedup and partial upsert to have consistent result. We can consider adding a config to turn it off for dedup and partial upsert, and another config to force it even if it is not dedup or partial upsert. It can also be configured as an enum so that we can use the same key to control the behavior
created_at: 2025-03-21T18:09:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  @Nullable
+  private Set<Integer> _segmentSequenceNumSet = null;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _segmentSequenceNumSet = new HashSet<>();
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
+      }
+    }
+  }
+
+  public void release() {
+    _semaphore.release();
+  }
+
+  public Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  public void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (_segmentSequenceNumSet != null) {
+        _segmentSequenceNumSet.add(llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitUntilTableIsReady()
+      throws InterruptedException {
+    while (!_realtimeTableDataManager.getIsTableReadyToConsumeData().getAsBoolean()) {
+      Thread.sleep(RealtimeTableDataManager.READY_TO_CONSUME_DATA_CHECK_INTERVAL_MS);
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_segmentSequenceNumSet == null) || (!_isFirstTransitionProcessed.get())) {
+      // if _segmentSequenceNumSet is null or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (!_segmentSequenceNumSet.contains(prevSeqNum)) {
+        // if _segmentSequenceNumSet does not contain prev segment's seq num, it means segment is not loaded in the
+        // server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @Nullable
+  private String getPreviousSegment(LLCSegmentName currSegment) {
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    String previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+
+    Map<String, Map<String, String>> segmentAssignment =
+        HelixHelper.getSegmentAssignment(_realtimeTableDataManager.getTableName(),
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: noob-se7en
body: Is reading ideal state just for first few helix transitions also a problem? As even when we store sequence number we use ideal state to fetch prev segment during server restarts.

storing prevSegment in segment ZK metadata was not straightforward considering deletion of segment edge-cases.
created_at: 2025-03-21T19:23:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
+
+  @JsonPropertyDescription("If enabled, Server in-memory tracks the sequence number of all loaded segments")
+  private boolean _trackSegmentSeqNumber = true;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: noob-se7en
body: HashSet was kept as for some few segment deletion edge cases. Considering they are not open as of now, I have removed the set.
But we still feel that this feature flag is required to just use ideal state incase of any unforeseen bug related to segment sequence number during runtime.
created_at: 2025-03-21T19:26:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: noob-se7en
body: > and another config to force it even if it is not dedup or partial upsert. 

why? _enforceConsumptionInOrder makes ingestion slow. What benefit it will have for cases where dedup or partial upsert is disabled?
created_at: 2025-03-21T19:29:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -885,4 +921,21 @@ private void validate(TableConfig tableConfig, Schema schema) {
     // 2. Validate the schema itself
     SchemaUtils.validate(schema);
   }
+
+  private SemaphoreAccessCoordinator getSemaphoreAccessCoordinator(int partitionGroupId) {
+    return _partitionGroupIdToSemaphoreCoordinatorMap.computeIfAbsent(partitionGroupId,
+        k -> new SemaphoreAccessCoordinator(new Semaphore(1), _enforceConsumptionInOrder, this));
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
user: login: KKcorps
body: keep the sempahore count as constant. 
created_at: 2025-03-24T06:00:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -885,4 +921,21 @@ private void validate(TableConfig tableConfig, Schema schema) {
     // 2. Validate the schema itself
     SchemaUtils.validate(schema);
   }
+
+  private SemaphoreAccessCoordinator getSemaphoreAccessCoordinator(int partitionGroupId) {
+    return _partitionGroupIdToSemaphoreCoordinatorMap.computeIfAbsent(partitionGroupId,
+        k -> new SemaphoreAccessCoordinator(new Semaphore(1), _enforceConsumptionInOrder, this));
+  }
+
+  public BooleanSupplier getIsTableReadyToConsumeData() {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
user: login: KKcorps
body: why public?
created_at: 2025-03-24T06:00:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: KKcorps
body: +1. However I feel we should not touch semaphore at all and use futures to achieve this. The approach would be much cleaner and more reactive
created_at: 2025-03-24T06:02:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -885,4 +921,21 @@ private void validate(TableConfig tableConfig, Schema schema) {
     // 2. Validate the schema itself
     SchemaUtils.validate(schema);
   }
+
+  private SemaphoreAccessCoordinator getSemaphoreAccessCoordinator(int partitionGroupId) {
+    return _partitionGroupIdToSemaphoreCoordinatorMap.computeIfAbsent(partitionGroupId,
+        k -> new SemaphoreAccessCoordinator(new Semaphore(1), _enforceConsumptionInOrder, this));
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
user: login: KKcorps
body: or just change it to a lock
created_at: 2025-03-24T06:02:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,241 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = false;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (!trackSegmentSeqNumber) {
+      _relyOnIdealState = true;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: KKcorps
body: use constant here instead of 5
created_at: 2025-03-24T06:03:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,241 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = false;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (!trackSegmentSeqNumber) {
+      _relyOnIdealState = true;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
+      }
+    }
+  }
+
+  public void release() {
+    _semaphore.release();
+  }
+
+  public Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  public void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: KKcorps
body: Let's not make it a flag imo. We should keep both approaches. 

A good solution looks something like 

1. Never keep track of sequence number
2. Use the combination of `segmentDataManagerMap`, `recentlyDeletedSegments` and ideal state to determine previous segment. 
created_at: 2025-03-24T06:06:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  @Nullable
+  private Set<Integer> _segmentSequenceNumSet = null;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _segmentSequenceNumSet = new HashSet<>();
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
+      }
+    }
+  }
+
+  public void release() {
+    _semaphore.release();
+  }
+
+  public Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  public void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (_segmentSequenceNumSet != null) {
+        _segmentSequenceNumSet.add(llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitUntilTableIsReady()
+      throws InterruptedException {
+    while (!_realtimeTableDataManager.getIsTableReadyToConsumeData().getAsBoolean()) {
+      Thread.sleep(RealtimeTableDataManager.READY_TO_CONSUME_DATA_CHECK_INTERVAL_MS);
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_segmentSequenceNumSet == null) || (!_isFirstTransitionProcessed.get())) {
+      // if _segmentSequenceNumSet is null or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (!_segmentSequenceNumSet.contains(prevSeqNum)) {
+        // if _segmentSequenceNumSet does not contain prev segment's seq num, it means segment is not loaded in the
+        // server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @Nullable
+  private String getPreviousSegment(LLCSegmentName currSegment) {
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    String previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+
+    Map<String, Map<String, String>> segmentAssignment =
+        HelixHelper.getSegmentAssignment(_realtimeTableDataManager.getTableName(),
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: KKcorps
body: Instead of waiting on previous segment, I feel we can simply let the previous segment signal which transition to start next
this is because the coordinator has complete view of what all transitions are waiting and can take correct decision
it doesn't need to care about missing segments and all
created_at: 2025-03-24T06:13:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  @Nullable
+  private Set<Integer> _segmentSequenceNumSet = null;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _segmentSequenceNumSet = new HashSet<>();
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
+      }
+    }
+  }
+
+  public void release() {
+    _semaphore.release();
+  }
+
+  public Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  public void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (_segmentSequenceNumSet != null) {
+        _segmentSequenceNumSet.add(llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitUntilTableIsReady()
+      throws InterruptedException {
+    while (!_realtimeTableDataManager.getIsTableReadyToConsumeData().getAsBoolean()) {
+      Thread.sleep(RealtimeTableDataManager.READY_TO_CONSUME_DATA_CHECK_INTERVAL_MS);
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_segmentSequenceNumSet == null) || (!_isFirstTransitionProcessed.get())) {
+      // if _segmentSequenceNumSet is null or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (!_segmentSequenceNumSet.contains(prevSeqNum)) {
+        // if _segmentSequenceNumSet does not contain prev segment's seq num, it means segment is not loaded in the
+        // server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @Nullable
+  private String getPreviousSegment(LLCSegmentName currSegment) {
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    String previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+
+    Map<String, Map<String, String>> segmentAssignment =
+        HelixHelper.getSegmentAssignment(_realtimeTableDataManager.getTableName(),
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: noob-se7en
body: We need to rely on previous segment atleast for first transition. Signalling next segment also has similar edge-cases and approach.
created_at: 2025-03-24T06:57:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -885,4 +921,21 @@ private void validate(TableConfig tableConfig, Schema schema) {
     // 2. Validate the schema itself
     SchemaUtils.validate(schema);
   }
+
+  private SemaphoreAccessCoordinator getSemaphoreAccessCoordinator(int partitionGroupId) {
+    return _partitionGroupIdToSemaphoreCoordinatorMap.computeIfAbsent(partitionGroupId,
+        k -> new SemaphoreAccessCoordinator(new Semaphore(1), _enforceConsumptionInOrder, this));
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
user: login: noob-se7en
body: With lock, Other thread might not be able to unlock the lock acquired by consumer thread.
created_at: 2025-03-24T07:00:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,241 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(SemaphoreAccessCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = false;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public SemaphoreAccessCoordinator(Semaphore semaphore, boolean enforceConsumptionInOrder,
+      RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = semaphore;
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (!trackSegmentSeqNumber) {
+      _relyOnIdealState = true;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  public void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    String segmentName = llcSegmentName.getSegmentName();
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+
+      // if segment is marked online/offline by any instance, just return the current offline -> consuming
+      // transition.
+      if (!_enforceConsumptionInOrder && !isSegmentInProgress(segmentName)) {
+        throw new SegmentAlreadyConsumedException("segment: " + segmentName + " status must be in progress");
+      }
+    }
+  }
+
+  public void release() {
+    _semaphore.release();
+  }
+
+  public Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  public void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: noob-se7en
body: > 1. Never keep track of sequence number

Is solution possible for this? If we are not relying on ideal state, Coordinator needs to have some state regarding which sequence number is in line to proceed ahead. Even with consumer signalling next segment, we need to maintain state as helix transitions can come up anytime in future.

> 2. Use the combination of segmentDataManagerMap, recentlyDeletedSegments and ideal state to determine previous segment.

Agree with `segmentDataManagerMap` and ideal state. But don't think we need `recentlyDeletedSegments`.
created_at: 2025-03-24T07:13:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: Jackie-Jiang
body: I don't have a concrete example in mind, but just for flexibility. We can choose to not have it as well
created_at: 2025-03-24T21:19:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -250,6 +250,13 @@ public static IdealState getTableIdealState(HelixManager manager, String resourc
     return accessor.getProperty(builder.idealStates(resourceName));
   }
 
+  public static Map<String, Map<String, String>> getSegmentAssignment(String tableNameWithType,
path: pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
user: login: Jackie-Jiang
body: (minor) Suggest not adding this as a general util because:
1. We don't usually perform existence check in this util class
2. It is not very clear where the assignment from by the method name

It would be more readable to perform the IS null check on the caller side
created_at: 2025-03-24T21:30:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: Should we remove this flag? Do we always want to rely on IS for the first CONSUMING segment, then rely on sequence id for the following segments?
created_at: 2025-03-24T21:35:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -802,6 +808,21 @@ private void registerSegment(String segmentName, SegmentDataManager segmentDataM
     registerSegment(segmentName, segmentDataManager);
   }
 
+  @Override
+  protected SegmentDataManager registerSegment(String segmentName, SegmentDataManager segmentDataManager) {
+    SegmentDataManager oldSegmentDataManager = super.registerSegment(segmentName, segmentDataManager);
+    if (_enforceConsumptionInOrder) {
+      // helix threads might be waiting for their respective previous segments to be loaded.
+      // they need to be notified here.
+      LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+      Preconditions.checkNotNull(llcSegmentName);
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
user: login: Jackie-Jiang
body: This will break uploaded segments which might not follow the LLC segment name format. We should skip these segments
created_at: 2025-03-24T21:38:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _relyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
+        _maxSegmentSeqNumLoaded = Math.max(_maxSegmentSeqNumLoaded, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_relyOnIdealState) || (!_isFirstTransitionProcessed.get())) {
+      // if _relyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumLoaded < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumLoaded is not loaded in the server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  String getPreviousSegment(LLCSegmentName currSegment) {
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    String previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+
+    Map<String, Map<String, String>> segmentAssignment = getSegmentAssignment();
+
+    if (segmentAssignment == null) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: (minor) This will never be `null`
created_at: 2025-03-24T21:40:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _relyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
+        _maxSegmentSeqNumLoaded = Math.max(_maxSegmentSeqNumLoaded, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_relyOnIdealState) || (!_isFirstTransitionProcessed.get())) {
+      // if _relyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumLoaded < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumLoaded is not loaded in the server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  String getPreviousSegment(LLCSegmentName currSegment) {
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    String previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+
+    Map<String, Map<String, String>> segmentAssignment = getSegmentAssignment();
+
+    if (segmentAssignment == null) {
+      return null;
+    }
+
+    for (Map.Entry<String, Map<String, String>> entry : segmentAssignment.entrySet()) {
+      String segmentName = entry.getKey();
+      Map<String, String> instanceStateMap = entry.getValue();
+
+      if (!instanceStateMap.containsValue(CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE)) {
+        // if server is looking for previous segment to current transition's segment, it means the previous segment
+        // has to be online in any instance. If all previous segments are not online, we just allow the current helix
+        // transition to go ahead.
+        continue;
+      }
+
+      LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+      Preconditions.checkNotNull(llcSegmentName);
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: Skip the segments not following the LLC naming convention
created_at: 2025-03-24T21:40:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _relyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
+        _maxSegmentSeqNumLoaded = Math.max(_maxSegmentSeqNumLoaded, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_relyOnIdealState) || (!_isFirstTransitionProcessed.get())) {
+      // if _relyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumLoaded < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumLoaded is not loaded in the server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  String getPreviousSegment(LLCSegmentName currSegment) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: For the first state transition, checking only the previous segment is not really enough. Currently we rely on the start-up check to ensure all segments loaded, so we can skip the first state transition check. We can revisit this when trying to solve the rebalance problem
created_at: 2025-03-24T21:45:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _relyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
+        _maxSegmentSeqNumLoaded = Math.max(_maxSegmentSeqNumLoaded, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_relyOnIdealState) || (!_isFirstTransitionProcessed.get())) {
+      // if _relyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumLoaded < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumLoaded is not loaded in the server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: This PR doesn't integrate the change from #15246. Do you want to open a separate PR for that?
created_at: 2025-03-24T21:47:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.base.Preconditions;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class SemaphoreAccessCoordinator {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SemaphoreAccessCoordinator.java
user: login: noob-se7en
body: I believe this is adressed: https://github.com/apache/pinot/pull/15261#discussion_r2009563467
created_at: 2025-03-25T11:44:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _relyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
+        _maxSegmentSeqNumLoaded = Math.max(_maxSegmentSeqNumLoaded, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_relyOnIdealState) || (!_isFirstTransitionProcessed.get())) {
+      // if _relyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumLoaded < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumLoaded is not loaded in the server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: noob-se7en
body: Yes. As https://github.com/apache/pinot/pull/15246 has nothing to do with consumption in order problem for dedup and upserts. Will handle that separately.
created_at: 2025-03-25T11:52:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: noob-se7en
body: This flag is meant to be removed in future once we gain confidence that just relying on previous segment seq id enough for all (offline-> consuming) helix transitions except the first one. 

Because incase of any unforeseen edge-case, ingestion will stop completely on server and we can just set this flag as true and restart the server.
created_at: 2025-03-25T11:58:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _relyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(5, TimeUnit.MINUTES)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
+        _maxSegmentSeqNumLoaded = Math.max(_maxSegmentSeqNumLoaded, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_relyOnIdealState) || (!_isFirstTransitionProcessed.get())) {
+      // if _relyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      String previousSegment = getPreviousSegment(currSegment);
+      if (previousSegment == null) {
+        // previous segment can only be null if either all the previous segments are deleted or this is the starting
+        // sequence segment of the partition Group.
+        return;
+      }
+
+      SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+      try {
+        _lock.lock();
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(5, TimeUnit.MINUTES)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+        if (segmentDataManager != null) {
+          _realtimeTableDataManager.releaseSegment(segmentDataManager);
+        }
+      }
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumLoaded < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumLoaded is not loaded in the server. Wait until it's loaded.
+        while (!_condition.await(5, TimeUnit.MINUTES)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+        }
+      }
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  String getPreviousSegment(LLCSegmentName currSegment) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: noob-se7en
body: > Currently we rely on the start-up check to ensure all segments loaded, so we can skip the first state transition check.

Didn't completely get this. I think we need this. I am thinking for case where a slow server comes back online after being dead. While it was dead segment 101 was deleted. When it comes back online it's slow and multiple (offline->consuming) transitions are pending starting from segment 102. There is no guarantee that the first helix transition racing to acquire semaphore will be segment 102. 
created_at: 2025-03-25T12:08:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: I see. I mis-read the logic and thought it is used only for the first transition.
(minor) We can change it to `final`
created_at: 2025-03-25T22:37:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: Jackie-Jiang
body: This comment is not addressed
created_at: 2025-03-25T22:39:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
+
+  @JsonPropertyDescription("If enabled, Server in-memory tracks the sequence number of all loaded segments")
+  private boolean _trackSegmentSeqNumber = true;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: Jackie-Jiang
body: Since we are using this to determine whether to use IS to calculate the previous segment, suggest making it more explicit (such as _useIdealStateToCalculatePreviousSegment`). It is good practice to have boolean config as `false` by default, which is less error-prone
created_at: 2025-03-25T22:43:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,240 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
+  private boolean _relyOnIdealState = true;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(5);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    boolean trackSegmentSeqNumber = false;
+    if (streamIngestionConfig != null) {
+      trackSegmentSeqNumber = streamIngestionConfig.isTrackSegmentSeqNumber();
+    }
+    // if trackSegmentSeqNumber is false, server relies on ideal state to fetch previous segment to a segment.
+    if (trackSegmentSeqNumber) {
+      _relyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+
+    if (_enforceConsumptionInOrder) {
+      waitForPrevSegment(llcSegmentName);
+    }
+
+    long startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_relyOnIdealState) {
+        _maxSegmentSeqNumLoaded = Math.max(_maxSegmentSeqNumLoaded, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    long startTimeMs = System.currentTimeMillis();
+
+    if ((_relyOnIdealState) || (!_isFirstTransitionProcessed.get())) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: (nit) parenthesis seems redundant
created_at: 2025-03-25T22:44:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,240 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private int _maxSegmentSeqNumLoaded = -1;
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: This needs to be `volatile`
created_at: 2025-03-25T22:55:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,12 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: noob-se7en
body: We initially decided to keep it default false for backward compatibility. 
@KKcorps 

I think we should keep it false by default atleast until rebalancing edge-case is fully solved.
created_at: 2025-03-26T12:29:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,291 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    acquireSegmentRelyingOnIdealState(currSegment);
+  }
+
+  private void acquireSegmentRelyingOnIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+
+    // this can be slow. And should not happen within lock.
+    LLCSegmentName previousLLCSegmentName = getPreviousSegment(currSegment);
+    if (previousLLCSegmentName == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+    String previousSegment = previousLLCSegmentName.getSegmentName();
+
+    long duration = System.currentTimeMillis() - startTimeMs;
+    LOGGER.info("Fetched previous segment: {} to current segment: {} in: {} ms.", previousSegment,
+        currSegment.getSegmentName(), duration);
+    _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+        ServerTimer.PREV_SEGMENT_FETCH_IDEAL_STATE_DURATION_MS, duration, TimeUnit.MILLISECONDS);
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+      }
+    } finally {
+      if (segmentDataManager != null) {
+        _realtimeTableDataManager.releaseSegment(segmentDataManager);
+      }
+    }
+  }
+
+  /***
+   * @param currSegment is the segment of current helix transition.
+   * @param timeoutMs is max time to wait in millis
+   * @return true if previous Segment was registered to the server, else false.
+   * @throws InterruptedException
+   */
+  @VisibleForTesting
+  boolean awaitForPreviousSegmentSequenceNumber(LLCSegmentName currSegment, long timeoutMs)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumRegistered < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumRegistered is not loaded in the server. Wait until it's loaded.
+        if (!_condition.await(timeoutMs, TimeUnit.MILLISECONDS)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+          // waited until the timeout. Rely on ideal state now.
+          return false;
+        }
+      }
+
+      return (_maxSegmentSeqNumRegistered >= prevSeqNum);
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  LLCSegmentName getPreviousSegment(LLCSegmentName currSegment) {
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    LLCSegmentName previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+    Map<String, Map<String, String>> segmentAssignment = getSegmentAssignment();
+
+    for (Map.Entry<String, Map<String, String>> entry : segmentAssignment.entrySet()) {
+      String segmentName = entry.getKey();
+      Map<String, String> instanceStateMap = entry.getValue();
+
+      if (!instanceStateMap.containsValue(CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE)) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: We should check the state for the current server only
created_at: 2025-03-26T17:48:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,291 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    acquireSegmentRelyingOnIdealState(currSegment);
+  }
+
+  private void acquireSegmentRelyingOnIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+
+    // this can be slow. And should not happen within lock.
+    LLCSegmentName previousLLCSegmentName = getPreviousSegment(currSegment);
+    if (previousLLCSegmentName == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+    String previousSegment = previousLLCSegmentName.getSegmentName();
+
+    long duration = System.currentTimeMillis() - startTimeMs;
+    LOGGER.info("Fetched previous segment: {} to current segment: {} in: {} ms.", previousSegment,
+        currSegment.getSegmentName(), duration);
+    _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+        ServerTimer.PREV_SEGMENT_FETCH_IDEAL_STATE_DURATION_MS, duration, TimeUnit.MILLISECONDS);
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: We should refresh the previous segment here
created_at: 2025-03-26T17:50:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -63,6 +63,12 @@ public enum ServerTimer implements AbstractMetrics.Timer {
   SECONDARY_Q_WAIT_TIME_MS("milliseconds", false,
       "Time spent waiting in the secondary queue when BinaryWorkloadScheduler is used."),
 
+  PREV_SEGMENT_FETCH_IDEAL_STATE_DURATION_MS("milliseconds", false,
+      "Time spent while fetching previous segment from ideal state for any segment."),
+
+  PREV_SEGMENT_WAIT_DURATION_MS("milliseconds", false,
+      "Time spent while waiting on previous segment to be registered."),
path: pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
user: login: Jackie-Jiang
body: For consistency:
```suggestion
  PREV_SEGMENT_FETCH_IDEAL_STATE_TIME_MS("milliseconds", false,
      "Time spent while fetching previous segment from ideal state for any segment."),

  PREV_SEGMENT_WAIT_TIME_MS("milliseconds", false,
      "Time spent while waiting on previous segment to be registered."),
```
created_at: 2025-03-26T18:57:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: (minor) Re-order variables. Put constant first, followed by `final`, then this one
created_at: 2025-03-26T18:59:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: Change the public facing method to `public`
created_at: 2025-03-26T19:00:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: (minor) `set()` should be enough
created_at: 2025-03-26T19:05:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    acquireSegmentRelyingOnIdealState(currSegment);
+  }
+
+  private void acquireSegmentRelyingOnIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    LLCSegmentName previousLLCSegmentName = getPreviousSegment(currSegment);
+    if (previousLLCSegmentName == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+    String previousSegment = previousLLCSegmentName.getSegmentName();
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      long startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+            // waited until timeout, fetch previous segment again from ideal state as previous segment might be
+            // changed in ideal state.
+            previousLLCSegmentName = getPreviousSegment(currSegment);
+            if (previousLLCSegmentName == null) {
+              return;
+            }
+            previousSegment = previousLLCSegmentName.getSegmentName();
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+      }
+    } finally {
+      if (segmentDataManager != null) {
+        _realtimeTableDataManager.releaseSegment(segmentDataManager);
+      }
+    }
+  }
+
+  /***
+   * @param currSegment is the segment of current helix transition.
+   * @param timeoutMs is max time to wait in millis
+   * @return true if previous Segment was registered to the server, else false.
+   * @throws InterruptedException
+   */
+  @VisibleForTesting
+  boolean awaitForPreviousSegmentSequenceNumber(LLCSegmentName currSegment, long timeoutMs)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumRegistered < prevSeqNum) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: We should perform check twice, once before the wait and once after the wait. Currently it only checks once
created_at: 2025-03-26T19:08:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    acquireSegmentRelyingOnIdealState(currSegment);
+  }
+
+  private void acquireSegmentRelyingOnIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    LLCSegmentName previousLLCSegmentName = getPreviousSegment(currSegment);
+    if (previousLLCSegmentName == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+    String previousSegment = previousLLCSegmentName.getSegmentName();
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      long startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+            // waited until timeout, fetch previous segment again from ideal state as previous segment might be
+            // changed in ideal state.
+            previousLLCSegmentName = getPreviousSegment(currSegment);
+            if (previousLLCSegmentName == null) {
+              return;
+            }
+            previousSegment = previousLLCSegmentName.getSegmentName();
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+      }
+    } finally {
+      if (segmentDataManager != null) {
+        _realtimeTableDataManager.releaseSegment(segmentDataManager);
+      }
+    }
+  }
+
+  /***
+   * @param currSegment is the segment of current helix transition.
+   * @param timeoutMs is max time to wait in millis
+   * @return true if previous Segment was registered to the server, else false.
+   * @throws InterruptedException
+   */
+  @VisibleForTesting
+  boolean awaitForPreviousSegmentSequenceNumber(LLCSegmentName currSegment, long timeoutMs)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumRegistered < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumRegistered is not loaded in the server. Wait until it's loaded.
+        if (!_condition.await(timeoutMs, TimeUnit.MILLISECONDS)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+          // waited until the timeout. Rely on ideal state now.
+          return false;
+        }
+      }
+
+      return (_maxSegmentSeqNumRegistered >= prevSeqNum);
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  LLCSegmentName getPreviousSegment(LLCSegmentName currSegment) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: (minor) Make it more specific
```suggestion
  LLCSegmentName getPreviousSegmentFromIdealState(LLCSegmentName currSegment) {
```
created_at: 2025-03-26T19:10:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    acquireSegmentRelyingOnIdealState(currSegment);
+  }
+
+  private void acquireSegmentRelyingOnIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    LLCSegmentName previousLLCSegmentName = getPreviousSegment(currSegment);
+    if (previousLLCSegmentName == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+    String previousSegment = previousLLCSegmentName.getSegmentName();
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      long startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+            // waited until timeout, fetch previous segment again from ideal state as previous segment might be
+            // changed in ideal state.
+            previousLLCSegmentName = getPreviousSegment(currSegment);
+            if (previousLLCSegmentName == null) {
+              return;
+            }
+            previousSegment = previousLLCSegmentName.getSegmentName();
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+      }
+    } finally {
+      if (segmentDataManager != null) {
+        _realtimeTableDataManager.releaseSegment(segmentDataManager);
+      }
+    }
+  }
+
+  /***
+   * @param currSegment is the segment of current helix transition.
+   * @param timeoutMs is max time to wait in millis
+   * @return true if previous Segment was registered to the server, else false.
+   * @throws InterruptedException
+   */
+  @VisibleForTesting
+  boolean awaitForPreviousSegmentSequenceNumber(LLCSegmentName currSegment, long timeoutMs)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumRegistered < prevSeqNum) {
+        // it means all segments until _maxSegmentSeqNumRegistered is not loaded in the server. Wait until it's loaded.
+        if (!_condition.await(timeoutMs, TimeUnit.MILLISECONDS)) {
+          LOGGER.warn("Semaphore access denied to segment: {}."
+                  + " Waiting on previous segment with sequence number: {} since: {} ms.", currSegment.getSegmentName(),
+              prevSeqNum, System.currentTimeMillis() - startTimeMs);
+          // waited until the timeout. Rely on ideal state now.
+          return false;
+        }
+      }
+
+      return (_maxSegmentSeqNumRegistered >= prevSeqNum);
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  @VisibleForTesting
+  @Nullable
+  LLCSegmentName getPreviousSegment(LLCSegmentName currSegment) {
+    long startTimeMs = System.currentTimeMillis();
+    // if seq num of current segment is 102, maxSequenceNumBelowCurrentSegment must be highest seq num of any segment
+    // created before current segment
+    int maxSequenceNumBelowCurrentSegment = -1;
+    LLCSegmentName previousSegment = null;
+    int currPartitionGroupId = currSegment.getPartitionGroupId();
+    int currSequenceNum = currSegment.getSequenceNumber();
+    Map<String, Map<String, String>> segmentAssignment = getSegmentAssignment();
+    String currentServerInstanceId = _realtimeTableDataManager.getServerInstance();
+
+    for (Map.Entry<String, Map<String, String>> entry : segmentAssignment.entrySet()) {
+      String segmentName = entry.getKey();
+      Map<String, String> instanceStateMap = entry.getValue();
+      String state = instanceStateMap.get(currentServerInstanceId);
+
+      if (!CommonConstants.Helix.StateModel.SegmentStateModel.ONLINE.equals(state)) {
+        // if server is looking for previous segment to current transition's segment, it means the previous segment
+        // has to be online in the instance. If all previous segments are not online, we just allow the current helix
+        // transition to go ahead.
+        continue;
+      }
+
+      LLCSegmentName llcSegmentName = LLCSegmentName.of(segmentName);
+      if (llcSegmentName == null) {
+        // can't compare with this segment, hence skip.
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: (minor)
```suggestion
        // ignore uploaded segments
```
created_at: 2025-03-26T19:11:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: Jackie-Jiang
body: (nit)
```suggestion
  private final AtomicBoolean _firstTransitionProcessed;
```
created_at: 2025-03-26T19:13:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,6 +43,13 @@ public class StreamIngestionConfig extends BaseJsonConfig {
   @JsonPropertyDescription("Whether pauseless consumption is enabled for the table")
   private boolean _pauselessConsumptionEnabled = false;
 
+  @JsonPropertyDescription("Enforce consumption of segments in order of segment creation by the controller")
+  private boolean _enforceConsumptionInOrder = false;
+
+  @JsonPropertyDescription("If enabled, Server always relies on ideal state to get previous segment. Else server uses"
path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/StreamIngestionConfig.java
user: login: Jackie-Jiang
body: If disabled, server uses sequence id - 1
created_at: 2025-03-26T19:15:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    acquireSegmentRelyingOnIdealState(currSegment);
+  }
+
+  private void acquireSegmentRelyingOnIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    LLCSegmentName previousLLCSegmentName = getPreviousSegment(currSegment);
+    if (previousLLCSegmentName == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+    String previousSegment = previousLLCSegmentName.getSegmentName();
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      long startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+            // waited until timeout, fetch previous segment again from ideal state as previous segment might be
+            // changed in ideal state.
+            previousLLCSegmentName = getPreviousSegment(currSegment);
+            if (previousLLCSegmentName == null) {
+              return;
+            }
+            previousSegment = previousLLCSegmentName.getSegmentName();
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+      }
+    } finally {
+      if (segmentDataManager != null) {
+        _realtimeTableDataManager.releaseSegment(segmentDataManager);
+      }
+    }
+  }
+
+  /***
+   * @param currSegment is the segment of current helix transition.
+   * @param timeoutMs is max time to wait in millis
+   * @return true if previous Segment was registered to the server, else false.
+   * @throws InterruptedException
+   */
+  @VisibleForTesting
+  boolean awaitForPreviousSegmentSequenceNumber(LLCSegmentName currSegment, long timeoutMs)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumRegistered < prevSeqNum) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: noob-se7en
body: why?
created_at: 2025-03-26T20:36:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,302 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.Map;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+import javax.annotation.Nullable;
+import org.apache.helix.model.IdealState;
+import org.apache.pinot.common.metrics.ServerMetrics;
+import org.apache.pinot.common.metrics.ServerTimer;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+import org.apache.pinot.segment.local.data.manager.SegmentDataManager;
+import org.apache.pinot.spi.config.table.ingestion.StreamIngestionConfig;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The ConsumerCoordinator coordinates the offline->consuming helix transitions.
+ */
+public class ConsumerCoordinator {
+
+  private static final Logger LOGGER = LoggerFactory.getLogger(ConsumerCoordinator.class);
+
+  private final Semaphore _semaphore;
+  private final boolean _enforceConsumptionInOrder;
+  private final Condition _condition;
+  private final Lock _lock;
+  private final ServerMetrics _serverMetrics;
+  private volatile int _maxSegmentSeqNumRegistered = -1;
+  private final boolean _alwaysRelyOnIdealState;
+  private final RealtimeTableDataManager _realtimeTableDataManager;
+  private final AtomicBoolean _isFirstTransitionProcessed;
+  private static final long WAIT_INTERVAL_MS = TimeUnit.MINUTES.toMillis(3);
+
+  public ConsumerCoordinator(boolean enforceConsumptionInOrder, RealtimeTableDataManager realtimeTableDataManager) {
+    _semaphore = new Semaphore(1);
+    _lock = new ReentrantLock();
+    _condition = _lock.newCondition();
+    _enforceConsumptionInOrder = enforceConsumptionInOrder;
+    _realtimeTableDataManager = realtimeTableDataManager;
+    StreamIngestionConfig streamIngestionConfig = realtimeTableDataManager.getStreamIngestionConfig();
+    if (streamIngestionConfig != null) {
+      // if isUseIdealStateToCalculatePreviousSegment is true, server relies on ideal state to fetch previous segment
+      // to a segment for all helix transitions.
+      _alwaysRelyOnIdealState = streamIngestionConfig.isUseIdealStateToCalculatePreviousSegment();
+    } else {
+      _alwaysRelyOnIdealState = false;
+    }
+    _isFirstTransitionProcessed = new AtomicBoolean(false);
+    _serverMetrics = ServerMetrics.get();
+  }
+
+  void acquire(LLCSegmentName llcSegmentName)
+      throws InterruptedException {
+    long startTimeMs;
+
+    if (_enforceConsumptionInOrder) {
+      startTimeMs = System.currentTimeMillis();
+
+      waitForPrevSegment(llcSegmentName);
+
+      _serverMetrics.addTimedTableValue(_realtimeTableDataManager.getTableName(),
+          ServerTimer.PREV_SEGMENT_WAIT_DURATION_MS, System.currentTimeMillis() - startTimeMs, TimeUnit.MILLISECONDS);
+    }
+
+    startTimeMs = System.currentTimeMillis();
+    while (!_semaphore.tryAcquire(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+      LOGGER.warn("Failed to acquire partitionGroup consumer semaphore in: {} ms. Retrying.",
+          System.currentTimeMillis() - startTimeMs);
+    }
+  }
+
+  void release() {
+    _semaphore.release();
+  }
+
+  @VisibleForTesting
+  Semaphore getSemaphore() {
+    return _semaphore;
+  }
+
+  void trackSegment(LLCSegmentName llcSegmentName) {
+    _lock.lock();
+    try {
+      if (!_alwaysRelyOnIdealState) {
+        _maxSegmentSeqNumRegistered = Math.max(_maxSegmentSeqNumRegistered, llcSegmentName.getSequenceNumber());
+      }
+      // notify all helix threads waiting for their offline -> consuming segment's prev segment to be loaded
+      _condition.signalAll();
+    } finally {
+      _lock.unlock();
+    }
+  }
+
+  private void waitForPrevSegment(LLCSegmentName currSegment)
+      throws InterruptedException {
+
+    if (_alwaysRelyOnIdealState || !_isFirstTransitionProcessed.get()) {
+      // if _alwaysRelyOnIdealState or no offline -> consuming transition has been processed, it means rely on
+      // ideal state to fetch previous segment.
+      acquireSegmentRelyingOnIdealState(currSegment);
+
+      // the first transition will always be prone to error, consider edge case where segment previous to current
+      // helix transition's segment was deleted and this server came alive after successful deletion. the prev
+      // segment will not exist, hence first transition is handled using isFirstTransitionSuccessful.
+      _isFirstTransitionProcessed.compareAndSet(false, true);
+      return;
+    }
+
+    // rely on _maxSegmentSeqNumRegistered watermark for previous segment.
+    if (awaitForPreviousSegmentSequenceNumber(currSegment, WAIT_INTERVAL_MS)) {
+      return;
+    }
+
+    // tried using prevSegSeqNumber watermark, but could not acquire the previous segment.
+    // fallback to acquire prev segment from ideal state.
+    acquireSegmentRelyingOnIdealState(currSegment);
+  }
+
+  private void acquireSegmentRelyingOnIdealState(LLCSegmentName currSegment)
+      throws InterruptedException {
+    LLCSegmentName previousLLCSegmentName = getPreviousSegment(currSegment);
+    if (previousLLCSegmentName == null) {
+      // previous segment can only be null if either all the previous segments are deleted or this is the starting
+      // sequence segment of the partition Group.
+      return;
+    }
+    String previousSegment = previousLLCSegmentName.getSegmentName();
+
+    SegmentDataManager segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+    try {
+      long startTimeMs = System.currentTimeMillis();
+      _lock.lock();
+      try {
+        while (segmentDataManager == null) {
+          // if segmentDataManager == null, it means segment is not loaded in the server.
+          // wait until it's loaded.
+          while (!_condition.await(WAIT_INTERVAL_MS, TimeUnit.MILLISECONDS)) {
+            LOGGER.warn("Semaphore access denied to segment: {}. Waiting on previous segment: {} since: {} ms.",
+                currSegment.getSegmentName(), previousSegment, System.currentTimeMillis() - startTimeMs);
+            // waited until timeout, fetch previous segment again from ideal state as previous segment might be
+            // changed in ideal state.
+            previousLLCSegmentName = getPreviousSegment(currSegment);
+            if (previousLLCSegmentName == null) {
+              return;
+            }
+            previousSegment = previousLLCSegmentName.getSegmentName();
+          }
+          segmentDataManager = _realtimeTableDataManager.acquireSegment(previousSegment);
+        }
+      } finally {
+        _lock.unlock();
+      }
+    } finally {
+      if (segmentDataManager != null) {
+        _realtimeTableDataManager.releaseSegment(segmentDataManager);
+      }
+    }
+  }
+
+  /***
+   * @param currSegment is the segment of current helix transition.
+   * @param timeoutMs is max time to wait in millis
+   * @return true if previous Segment was registered to the server, else false.
+   * @throws InterruptedException
+   */
+  @VisibleForTesting
+  boolean awaitForPreviousSegmentSequenceNumber(LLCSegmentName currSegment, long timeoutMs)
+      throws InterruptedException {
+    long startTimeMs = System.currentTimeMillis();
+    int prevSeqNum = currSegment.getSequenceNumber() - 1;
+    _lock.lock();
+    try {
+      while (_maxSegmentSeqNumRegistered < prevSeqNum) {
path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/ConsumerCoordinator.java
user: login: noob-se7en
body: I see, I am adding I will return `return (_maxSegmentSeqNumRegistered >= prevSeqNum);` instead false.
created_at: 2025-03-26T20:40:41Z
author_association: CONTRIBUTOR

PR comments:
user: login: noob-se7en
created_at: 2025-03-18T22:01:35Z
body: @Jackie-Jiang this PR is not in ready state as of now. We want to rely on sequence number instead of segmentZkMetadata for previous segment.
author_association: CONTRIBUTOR
user: login: noob-se7en
created_at: 2025-03-18T22:09:05Z
body: The problem you mentioned is fixed in this PR: https://github.com/apache/pinot/pull/15246
This PR needs to merge above.
author_association: CONTRIBUTOR
user: login: codecov-commenter
created_at: 2025-03-19T20:50:11Z
body: ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15261?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `76.38889%` with `34 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 63.19%. Comparing base ([`59551e4`](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`58cd003`](https://app.codecov.io/gh/apache/pinot/commit/58cd003feaf6657972d409dfdc363d0b6db90ed2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3829 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15261?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ore/data/manager/realtime/ConsumerCoordinator.java](https://app.codecov.io/gh/apache/pinot/pull/15261?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2Frealtime%2FConsumerCoordinator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvQ29uc3VtZXJDb29yZGluYXRvci5qYXZh) | 78.70% | [16 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/15261?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2Frealtime%2FRealtimeTableDataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | 57.14% | [5 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/manager/realtime/RealtimeSegmentDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/15261?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2Frealtime%2FRealtimeSegmentDataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | 71.42% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |

<details><summary>Additional details and impacted files</summary>



```diff
@@             Coverage Diff              @@
##             master   #15261      +/-   ##
============================================
+ Coverage     61.75%   63.19%   +1.44%     
- Complexity      207     1373    +1166     
============================================
  Files          2436     2812     +376     
  Lines        133233   158798   +25565     
  Branches      20636    24316    +3680     
============================================
+ Hits          82274   100352   +18078     
- Misses        44911    50866    +5955     
- Partials       6048     7580    +1532     
```

| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+99.99%)` | :arrow_up: |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+99.99%)` | :arrow_up: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+99.99%)` | :arrow_up: |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.16% <76.38%> (+1.45%)` | :arrow_up: |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.18% <76.38%> (+1.55%)` | :arrow_up: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.18% <76.38%> (+1.43%)` | :arrow_up: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.15% <76.38%> (+35.43%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.19% <76.38%> (+1.44%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.19% <76.38%> (+1.44%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.21% <76.38%> (+9.32%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15261/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `33.81% <1.38%> (+6.08%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15261?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>

- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE