Repository: hiero-ledger/hiero-consensus-node
Pull request #18915: feat: block stream buffer
Author: login: timfn-hg
Created: 2025-04-21T18:09:32Z | Merged: 2025-05-07T18:32:45Z

**Description**:
This PR creates a buffer for storing the most recent blocks. The number of blocks to retain is based on a new configuration property where blocks will be pruned from the buffer if they exceed the configured time threshold. Besides time, blocks are also only pruned if they are acknowledged. This pruning process is performed asynchronously by a background thread at a configurable interval.

If a block is not acknowledged but expired (that is they are older than the TTL threshold), it will not be pruned. If there are blocks in this state, the buffer is considered to be "saturated". This state is not normal and usually indicates a problem with the block node receiving or processing blocks. In this state we want to apply some form of backpressure to prevent more blocks (and transactions) from being processed.

**Related issue(s)**:

Fixes #18430 

**Checklist**

- [X] Documented (Code comments, README, etc.)
- [X] Tested (unit, integration, etc.)


Linked issue #18430: Design Buffering of Blocks for streaming
### Background
The consensus node will need to keep blocks in an in-memory/on-disk buffer for X number of minutes (default to 5 minutes) for the purpose of keeping unacknowledged blocks and streaming blocks which are not the current block the consensus node is producing.

-  Design buffering X minutes of Blocks in-memory.
   - A Ring Buffer, configurable time.
   - Write the buffer to disk only when shutting down the consensus node.
   - Upon restart, reload any information into the in-memory ring buffer.
   - Block Acknowledgement handling.
       - When a Block Acknowledgement is received the CN should maintain knowledge that it is ack'd but still keep it in the buffer until expiration.

### Acceptance Criteria

- Design diagram showing key interactions and classes
- Reviewed design with the Execution team

### Dependencies

_No response_

### Definition of Ready (DoR) Checklist

- [ ] Clear acceptance criteria
- [ ] Clear and detailed description
- [ ] Dependencies identified
- [ ] Links to documentation
- [ ] Should be completable in 2-3 Days
- [ ] Initial draft of Low-level design document
- [ ] At least high level test plan
- [ ] Groomed/Estimated

### Definition of Done (DoD) Checklist

- [ ] Acceptance Criteria complete
- [ ] No Codacy issues greater than minor (in new code)
- [ ] JavaDocs updated/created
- [ ] Code commented
- [ ] Unit tests created/updated
- [ ] 80% test code coverage (in new code)
- [ ] Happy Path and major negative cases in HAPI tests as applicable

Diff:
diff --git a/hedera-node/configuration/dev/log4j2.xml b/hedera-node/configuration/dev/log4j2.xml
index 3c353842ecd1..d19f307a7ca7 100644
--- a/hedera-node/configuration/dev/log4j2.xml
+++ b/hedera-node/configuration/dev/log4j2.xml
@@ -222,6 +222,10 @@
       <AppenderRef ref="StateLogs"/>
     </Logger>
 
+    <Logger name="com.hedera.node.app.blocks.impl.streaming.BlockStreamStateManager" level="debug" additivity="false">
+      <AppenderRef ref="Console"/>
+      <AppenderRef ref="RollingFile"/>
+    </Logger>
     <Logger name="com.hedera" level="info" additivity="false">
       <AppenderRef ref="Console"/>
       <AppenderRef ref="RollingFile"/>
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/BlockStreamModule.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/BlockStreamModule.java
index c4cf6f00a003..fcbbeee7a1c5 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/BlockStreamModule.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/BlockStreamModule.java
@@ -23,7 +23,6 @@
 import dagger.Provides;
 import edu.umd.cs.findbugs.annotations.NonNull;
 import java.nio.file.FileSystem;
-import java.time.Duration;
 import java.util.function.Supplier;
 import javax.inject.Singleton;
 
@@ -47,8 +46,7 @@ static BlockNodeConfigExtractor provideBlockNodeConfigExtractor(@NonNull final C
     @Singleton
     static BlockStreamStateManager provideBlockStreamStateManager(
             @NonNull final ConfigProvider configProvider, @NonNull final BlockStreamMetrics blockStreamMetrics) {
-        final Duration expiryPeriod = Duration.ofMinutes(5L);
-        return new BlockStreamStateManager(configProvider, blockStreamMetrics, expiryPeriod);
+        return new BlockStreamStateManager(configProvider, blockStreamMetrics);
     }
 
     @Provides
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java
index 8f3748f56b85..f8b8f3bbb45e 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java
@@ -693,6 +693,15 @@ class SequentialTask extends AbstractTask {
 
         @Override
         protected boolean onExecute() {
+            try {
+                return doExecute();
+            } catch (final RuntimeException e) {
+                log.error("Error occurred while executing task", e);
+                throw e;
+            }
+        }
+
+        private boolean doExecute() {
             final var kind = item.item().kind();
             switch (kind) {
                 case EVENT_HEADER, EVENT_TRANSACTION, ROUND_HEADER -> inputTreeHasher.addLeaf(hash);
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnection.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnection.java
index 7c816db752ed..823df58a01c2 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnection.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnection.java
@@ -124,7 +124,7 @@ public void createRequestObserver() {
     /**
      * @param newState the new state to transition to
      */
-    public void updateConnectionState(@NonNull ConnectionState newState) {
+    public void updateConnectionState(@NonNull final ConnectionState newState) {
         connectionState = newState;
     }
 
@@ -223,14 +223,14 @@ && getCurrentRequestIndex() == blockState.requests().size()) {
                         }
                     }
                 }
-            } catch (InterruptedException e) {
+            } catch (final InterruptedException e) {
                 logger.error(
                         "[{}] Request worker thread interrupted for node {}",
                         Thread.currentThread().getName(),
                         connectionDescriptor);
                 Thread.currentThread().interrupt();
                 return;
-            } catch (Exception e) {
+            } catch (final Exception e) {
                 logger.error(
                         "[{}] Error in request worker thread for node {}",
                         Thread.currentThread().getName(),
@@ -271,7 +271,7 @@ private void waitForNewRequests() throws InterruptedException {
         }
     }
 
-    private void logBlockProcessingInfo(BlockState blockState) {
+    private void logBlockProcessingInfo(final BlockState blockState) {
         logger.trace(
                 "[{}] Processing block {} for node {}, isComplete: {}, requests: {}",
                 Thread.currentThread().getName(),
@@ -281,12 +281,12 @@ private void logBlockProcessingInfo(BlockState blockState) {
                 blockState.requests().size());
     }
 
-    private boolean needToWaitForMoreRequests(@NonNull BlockState blockState) {
+    private boolean needToWaitForMoreRequests(@NonNull final BlockState blockState) {
         return getCurrentRequestIndex() >= blockState.requests().size() && !blockState.isComplete();
     }
 
-    private void processAvailableRequests(@NonNull BlockState blockState) {
-        List<PublishStreamRequest> requests = blockState.requests();
+    private void processAvailableRequests(@NonNull final BlockState blockState) {
+        final List<PublishStreamRequest> requests = blockState.requests();
         while (getCurrentRequestIndex() < requests.size()) {
             if (!isActive()) {
                 return;
@@ -333,7 +333,7 @@ private void handleEndOfStreamError() {
                 TimeUnit.SECONDS);
     }
 
-    private void handleAcknowledgement(@NonNull Acknowledgement acknowledgement) {
+    private void handleAcknowledgement(@NonNull final Acknowledgement acknowledgement) {
         if (acknowledgement.hasBlockAck()) {
             final var blockAck = acknowledgement.blockAck();
             final var acknowledgedBlockNumber = blockAck.blockNumber();
@@ -376,8 +376,6 @@ private void handleAcknowledgement(@NonNull Acknowledgement acknowledgement) {
                             connectionDescriptor,
                             acknowledgedBlockNumber,
                             acknowledgedBlockNumber + 1L);
-                    // Remove all block states up to and including this block number
-                    blockStreamStateManager.removeBlockStatesUpTo(acknowledgedBlockNumber);
                     jumpToBlock(acknowledgedBlockNumber + 1);
                 } else if (currentBlockStreaming == acknowledgedBlockNumber
                         && currentBlockProducing == acknowledgedBlockNumber) {
@@ -390,9 +388,6 @@ private void handleAcknowledgement(@NonNull Acknowledgement acknowledgement) {
                             acknowledgedBlockNumber,
                             acknowledgedBlockNumber + 1L);
                     jumpToBlock(acknowledgedBlockNumber + 1);
-                } else if (currentBlockStreaming > acknowledgedBlockNumber) {
-                    // We are already streaming a block after the acknowledged block number, so remove from buffer
-                    blockStreamStateManager.removeBlockStatesUpTo(acknowledgedBlockNumber);
                 } else if (currentBlockStreaming < acknowledgedBlockNumber) {
                     logger.debug(
                             "[{}] Currently producing Block {} to Block Node {} and acknowledged Block {} - moving streaming ahead to Block {}",
@@ -409,7 +404,7 @@ private void handleAcknowledgement(@NonNull Acknowledgement acknowledgement) {
         }
     }
 
-    private void handleEndOfStream(@NonNull EndOfStream endOfStream) {
+    private void handleEndOfStream(@NonNull final EndOfStream endOfStream) {
         final var blockNumber = endOfStream.blockNumber();
         final var responseCode = endOfStream.status();
 
@@ -444,7 +439,7 @@ private void handleEndOfStream(@NonNull EndOfStream endOfStream) {
                     STREAM_ITEMS_BAD_STATE_PROOF -> {
                 // We should restart the stream at the block immediately
                 // following the block where the node fell behind.
-                long restartBlockNumber = blockNumber == Long.MAX_VALUE ? 0 : blockNumber + 1;
+                final long restartBlockNumber = blockNumber == Long.MAX_VALUE ? 0 : blockNumber + 1;
                 logger.warn(
                         "[{}] Block node {} reported status indicating immediate restart should be attempted. Will restart stream at block {}.",
                         Thread.currentThread().getName(),
@@ -460,7 +455,7 @@ private void handleEndOfStream(@NonNull EndOfStream endOfStream) {
             case STREAM_ITEMS_BEHIND -> {
                 // The block node is behind us, check if we have the last verified block still available in order to
                 // restart the stream from there
-                long restartBlockNumber = blockNumber == Long.MAX_VALUE ? 0 : blockNumber + 1;
+                final long restartBlockNumber = blockNumber == Long.MAX_VALUE ? 0 : blockNumber + 1;
                 if (blockStreamStateManager.getBlockState(restartBlockNumber) != null) {
                     logger.warn(
                             "[{}] Block node {} reported it is behind. Will restart stream at block {}.",
@@ -496,7 +491,7 @@ private void handleEndOfStream(@NonNull EndOfStream endOfStream) {
         }
     }
 
-    private void handleSkipBlock(@NonNull SkipBlock skipBlock) {
+    private void handleSkipBlock(@NonNull final SkipBlock skipBlock) {
         final var skipBlockNumber = skipBlock.blockNumber();
 
         // Only jump if the skip is for the block we are currently processing
@@ -513,7 +508,7 @@ private void handleSkipBlock(@NonNull SkipBlock skipBlock) {
         }
     }
 
-    private void handleResendBlock(@NonNull ResendBlock resendBlock) {
+    private void handleResendBlock(@NonNull final ResendBlock resendBlock) {
         final var resendBlockNumber = resendBlock.blockNumber();
 
         logger.debug(
@@ -532,7 +527,7 @@ private void handleResendBlock(@NonNull ResendBlock resendBlock) {
         restartStreamAtBlock(resendBlockNumber);
     }
 
-    private String generateConnectionDescriptor(BlockNodeConfig nodeConfig) {
+    private String generateConnectionDescriptor(final BlockNodeConfig nodeConfig) {
         return nodeConfig.address() + ":" + nodeConfig.port();
     }
 
@@ -626,7 +621,7 @@ public void notifyNewRequestAvailable() {
             return;
         }
         synchronized (newRequestAvailable) {
-            BlockState blockState = blockStreamStateManager.getBlockState(currentBlock);
+            final BlockState blockState = blockStreamStateManager.getBlockState(currentBlock);
             if (blockState != null) {
                 logger.trace(
                         "Notifying of new request available for node {} - block: {}, requests: {}, isComplete: {}",
@@ -650,7 +645,7 @@ public void notifyNewBlockAvailable() {
         }
     }
 
-    public void setCurrentBlockNumber(long blockNumber) {
+    public void setCurrentBlockNumber(final long blockNumber) {
         currentBlockNumber.set(blockNumber);
         currentRequestIndex.set(0); // Reset the request index when setting a new block
         logger.debug(
@@ -666,7 +661,7 @@ public void setCurrentBlockNumber(long blockNumber) {
      *
      * @param blockNumber the block number to restart at
      */
-    public void restartStreamAtBlock(long blockNumber) {
+    public void restartStreamAtBlock(final long blockNumber) {
         logger.debug("Restarting stream at block {} for node {}", blockNumber, connectionDescriptor);
 
         setCurrentBlockNumber(blockNumber);
@@ -683,7 +678,7 @@ public void restartStreamAtBlock(long blockNumber) {
      *
      * @param blockNumber the block number to jump to
      */
-    public void jumpToBlock(long blockNumber) {
+    public void jumpToBlock(final long blockNumber) {
         logger.debug(
                 "[{}] Jumping to block {} for node {}",
                 Thread.currentThread().getName(),
@@ -721,7 +716,7 @@ private void stopWorkerThread() {
     }
 
     @Override
-    public void onNext(PublishStreamResponse response) {
+    public void onNext(final PublishStreamResponse response) {
         if (response.hasAcknowledgement()) {
             handleAcknowledgement(response.acknowledgement());
         } else if (response.hasEndStream()) {
@@ -741,7 +736,7 @@ public void onNext(PublishStreamResponse response) {
     }
 
     @Override
-    public void onError(Throwable error) {
+    public void onError(final Throwable error) {
         logger.error(
                 "[{}] Error on stream from block node {}",
                 Thread.currentThread().getName(),
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManager.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManager.java
index 41a7a16290ec..a9ef78eecca9 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManager.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManager.java
@@ -526,9 +526,11 @@ public void updateLastVerifiedBlock(
         requireNonNull(blockNodeConfig);
 
         final Long latestBlock = getLastVerifiedBlock(blockNodeConfig);
-        if (blockNumber != null && blockNumber > latestBlock) {
-            lastVerifiedBlockPerConnection.put(blockNodeConfig, blockNumber);
-            blockStreamMetrics.setLatestAcknowledgedBlockNumber(blockNumber);
+        if (blockNumber != null) {
+            if (blockNumber > latestBlock) {
+                lastVerifiedBlockPerConnection.put(blockNodeConfig, blockNumber);
+            }
+            blockStreamStateManager.setLatestAcknowledgedBlock(blockNumber);
         }
     }
 
@@ -541,16 +543,6 @@ public Long getLastVerifiedBlock(@NonNull final BlockNodeConfig blockNodeConfig)
         return lastVerifiedBlockPerConnection.computeIfAbsent(blockNodeConfig, key -> -1L);
     }
 
-    /**
-     * @param blockNumber the block number to check for acknowledgements
-     * @return whether the block has been acknowledged by any connection.
-     */
-    public boolean isBlockAlreadyAcknowledged(@NonNull final Long blockNumber) {
-        requireNonNull(blockNumber);
-        return lastVerifiedBlockPerConnection.values().stream()
-                .anyMatch(verifiedBlock -> verifiedBlock.equals(blockNumber));
-    }
-
     /**
      * Runnable task to handle the connection attempt logic.
      * Schedules itself for subsequent retries upon failure using the connectionExecutor.
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockState.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockState.java
index ef5260092ddd..8b0fe7d7a8b2 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockState.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockState.java
@@ -17,14 +17,14 @@ public class BlockState {
     private final List<BlockItem> items;
     private final List<PublishStreamRequest> requests;
     private final AtomicBoolean isComplete = new AtomicBoolean(false);
-    private Instant completionTime = null;
+    private Instant completionTimestamp = null;
 
     /**
      * Create a new block state for a block number
      *
      * @param blockNumber the block number
      */
-    public BlockState(long blockNumber, @NonNull List<BlockItem> items) {
+    public BlockState(final long blockNumber, @NonNull final List<BlockItem> items) {
         this.blockNumber = blockNumber;
         this.items = items;
         this.requests = new ArrayList<>();
@@ -70,7 +70,7 @@ public boolean isComplete() {
      * Set the block as complete
      */
     public void setComplete() {
-        this.completionTime = Instant.now();
+        this.completionTimestamp = Instant.now();
         this.isComplete.set(true);
     }
 
@@ -79,7 +79,17 @@ public void setComplete() {
      *
      * @return the completion time, or null if the block is not complete
      */
-    public Instant getCompletionTime() {
-        return completionTime;
+    public Instant completionTimestamp() {
+        return completionTimestamp;
+    }
+
+    @Override
+    public String toString() {
+        return "BlockState{" + "blockNumber="
+                + blockNumber + ", items="
+                + items + ", requests="
+                + requests + ", isComplete="
+                + isComplete + ", completionTimestamp="
+                + completionTimestamp + '}';
     }
 }
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManager.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManager.java
index 20ea5d74b6d2..8bd928fb8361 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManager.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManager.java
@@ -10,11 +10,25 @@
 import com.hedera.node.config.ConfigProvider;
 import com.hedera.node.config.data.BlockStreamConfig;
 import edu.umd.cs.findbugs.annotations.NonNull;
+import java.math.BigDecimal;
+import java.math.RoundingMode;
 import java.time.Duration;
 import java.time.Instant;
 import java.util.ArrayList;
-import java.util.Map;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import javax.annotation.Nullable;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 
@@ -26,32 +40,110 @@
 public class BlockStreamStateManager {
     private static final Logger logger = LogManager.getLogger(BlockStreamStateManager.class);
 
-    private final Map<Long, BlockState> blockStates = new ConcurrentHashMap<>();
-    private final int blockItemBatchSize;
-    private long blockNumber = 0;
+    /**
+     * Buffer that stores recent blocks. This buffer is unbounded, however when opening a new block the buffer will be
+     * pruned. Generally speaking, the buffer should contain only blocks that are recent (that is within the configured
+     * {@link BlockStreamConfig#blockBufferTtl() TTL}) and have yet to be acknowledged. There may be cases where older
+     * blocks still exist in the buffer if they are unacknowledged, but once they are acknowledged they will be pruned
+     * the next time {@link #openBlock(long)} is invoked. {@link #isBufferSaturated()} can be used to check if the
+     * buffer contains unacknowledged old blocks.
+     */
+    private final BlockingQueue<BlockState> blockBuffer = new LinkedBlockingQueue<>();
+    /**
+     * Map for quickly looking up blocks by their ID/number. This will get pruned along with the buffer periodically.
+     */
+    private final ConcurrentMap<Long, BlockState> blockStatesById = new ConcurrentHashMap<>();
+    /**
+     * Flag to indicate if the buffer contains blocks that have expired but are still unacknowledged.
+     */
+    private final AtomicBoolean isBufferSaturated = new AtomicBoolean(false);
+    /**
+     * This tracks the highest block number that has been acknowledged by the connected block node. This is kept
+     * separately instead of individual acknowledgement tracking on a per-block basis because it is possible that after
+     * a block node reconnects, it (being the block node) may have processed blocks from another consensus node that are
+     * newer than the blocks processed by this consensus node.
+     */
+    private final AtomicLong highestAckedBlockNumber = new AtomicLong(Long.MIN_VALUE);
+    /**
+     * Executor that is used to schedule buffer pruning and triggering backpressure if needed.
+     */
+    private final ScheduledExecutorService execSvc = Executors.newSingleThreadScheduledExecutor();
+    /**
+     * Global CompletableFuture reference that is used to apply backpressure via {@link #ensureNewBlocksPermitted()}. If
+     * the completed future has a value of {@code true}, then it means that the buffer is no longer saturated and no
+     * blocking/backpressure is needed. If the value is {@code false} then it means this future was completed but
+     * another one took its place and backpressure is still enabled.
+     */
+    private static final AtomicReference<CompletableFuture<Boolean>> backpressureCompletableFutureRef =
+            new AtomicReference<>();
+
+    private long blockNumber = -1;
+    private final ConfigProvider configProvider;
 
     // Reference to the connection manager for notifications
     private BlockNodeConnectionManager blockNodeConnectionManager;
 
-    private BlockStreamMetrics blockStreamMetrics = null;
-
-    private final Duration expiryPeriod;
+    private final BlockStreamMetrics blockStreamMetrics;
 
     /**
      * Creates a new BlockStreamStateManager with the given configuration.
      *
      * @param configProvider the configuration provider
+     * @param blockStreamMetrics metrics factory for block stream
      */
     public BlockStreamStateManager(
-            @NonNull final ConfigProvider configProvider,
-            @NonNull BlockStreamMetrics blockStreamMetrics,
-            @NonNull Duration expiryPeriod) {
-        this.blockItemBatchSize = configProvider
+            @NonNull final ConfigProvider configProvider, @NonNull final BlockStreamMetrics blockStreamMetrics) {
+        this.configProvider = configProvider;
+        this.blockStreamMetrics = blockStreamMetrics;
+
+        scheduleNextPruning();
+    }
+
+    /**
+     * @return the interval in which the block buffer will be pruned (a duration of 0 means pruning is disabled)
+     */
+    private Duration blockBufferPruneInterval() {
+        return configProvider
+                .getConfiguration()
+                .getConfigData(BlockStreamConfig.class)
+                .blockBufferPruneInterval();
+    }
+
+    /**
+     * @return the current batch size for block items
+     */
+    private int blockItemBatchSize() {
+        return configProvider
                 .getConfiguration()
                 .getConfigData(BlockStreamConfig.class)
                 .blockItemBatchSize();
-        this.blockStreamMetrics = blockStreamMetrics;
-        this.expiryPeriod = expiryPeriod;
+    }
+
+    /**
+     * @return the current TTL for items in the block buffer
+     */
+    private Duration blockBufferTtl() {
+        return configProvider
+                .getConfiguration()
+                .getConfigData(BlockStreamConfig.class)
+                .blockBufferTtl();
+    }
+
+    /**
+     * @return the block period duration (i.e. the amount of time a single block represents)
+     */
+    private Duration blockPeriod() {
+        return configProvider
+                .getConfiguration()
+                .getConfigData(BlockStreamConfig.class)
+                .blockPeriod();
+    }
+
+    /**
+     * @return true if the block buffer has blocks that are expired but unacknowledged, else false
+     */
+    public boolean isBufferSaturated() {
+        return isBufferSaturated.get();
     }
 
     /**
@@ -59,24 +151,37 @@ public BlockStreamStateManager(
      *
      * @param blockNodeConnectionManager the block node connection manager
      */
-    public void setBlockNodeConnectionManager(@NonNull BlockNodeConnectionManager blockNodeConnectionManager) {
+    public void setBlockNodeConnectionManager(@NonNull final BlockNodeConnectionManager blockNodeConnectionManager) {
         this.blockNodeConnectionManager =
                 requireNonNull(blockNodeConnectionManager, "blockNodeConnectionManager must not be null");
     }
 
     /**
-     * Opens a new block with the given block number.
+     * Opens a new block with the given block number. This will also attempt to prune older blocks from the buffer.
      *
      * @param blockNumber the block number
      * @throws IllegalArgumentException if the block number is negative
      */
-    public void openBlock(long blockNumber) {
+    public void openBlock(final long blockNumber) {
         if (blockNumber < 0) throw new IllegalArgumentException("Block number must be non-negative");
+
+        if (this.blockNumber >= blockNumber) {
+            logger.error(
+                    "Attempted to open a new block with number {}, but a block with the same or later number "
+                            + "(latest: {}) has already been opened",
+                    blockNumber,
+                    this.blockNumber);
+            throw new IllegalStateException("Attempted to open a new block with number " + blockNumber
+                    + ", but a block with the same or later number (latest: " + this.blockNumber
+                    + ") has already been opened");
+        }
+
         // Create a new block state
-        blockStates.put(blockNumber, new BlockState(blockNumber, new ArrayList<>()));
+        final BlockState blockState = new BlockState(blockNumber, new ArrayList<>());
+        blockBuffer.add(blockState);
+        blockStatesById.put(blockNumber, blockState);
         this.blockNumber = blockNumber;
         blockStreamMetrics.setProducingBlockNumber(blockNumber);
-
         blockNodeConnectionManager.openBlock(blockNumber);
     }
 
@@ -87,9 +192,9 @@ public void openBlock(long blockNumber) {
      * @param blockItem the block item to add
      * @throws IllegalStateException if no block is currently open
      */
-    public void addItem(final long blockNumber, @NonNull BlockItem blockItem) {
+    public void addItem(final long blockNumber, @NonNull final BlockItem blockItem) {
         requireNonNull(blockItem, "blockItem must not be null");
-        BlockState blockState = getBlockState(blockNumber);
+        final BlockState blockState = getBlockState(blockNumber);
         if (blockState == null) {
             throw new IllegalStateException("Block state not found for block " + blockNumber);
         }
@@ -97,19 +202,43 @@ public void addItem(final long blockNumber, @NonNull BlockItem blockItem) {
         blockState.items().add(blockItem);
 
         // If we have enough items, create a new request
-        if (blockState.items().size() >= blockItemBatchSize) {
-            createRequestFromCurrentItems(blockState);
-        }
+        createRequestFromCurrentItems(blockState, false);
     }
 
     /**
-     * Creates a new PublishStreamRequest from the current items in the block.
+     * Creates zero, one, or many {@link PublishStreamRequest} for the current pending items in the block. This is a
+     * batched operation and thus a new {@link PublishStreamRequest} will be created only if there are enough pending
+     * items to fill the batch. If {@code force} is true, then a new {@link PublishStreamRequest} will be created even
+     * if the number of pending items is fewer than the configured batch size.
+     *
+     * @param blockState the block that may contain pending items
+     * @param force true if a new {@link PublishStreamRequest} should be created regardless of if there are enough items
+     *              to create a batch, otherwise if false, then a new {@link PublishStreamRequest} will only be created
+     *              if there are enough items to complete a full batch
      */
-    public void createRequestFromCurrentItems(@NonNull BlockState blockState) {
+    public void createRequestFromCurrentItems(@NonNull final BlockState blockState, final boolean force) {
+        requireNonNull(blockState, "blockState is required");
+
+        if (blockState.items().isEmpty()) {
+            return;
+        }
+
+        final int cfgBatchSize = blockItemBatchSize();
+        final int batchSize = Math.max(1, cfgBatchSize); // if cfgBatchSize is less than 1, set the size to 1
+        final List<BlockItem> items = new ArrayList<>(batchSize);
+
+        if (force || blockState.items().size() >= batchSize) {
+            final Iterator<BlockItem> it = blockState.items().iterator();
+            while (it.hasNext() && items.size() != batchSize) {
+                items.add(it.next());
+                it.remove();
+            }
+        } else {
+            return;
+        }
+
         // Create BlockItemSet by adding all items at once
-        final BlockItemSet itemSet = BlockItemSet.newBuilder()
-                .blockItems(new ArrayList<>(blockState.items()))
-                .build();
+        final BlockItemSet itemSet = BlockItemSet.newBuilder().blockItems(items).build();
 
         // Create the request and add it to the list
         final PublishStreamRequest request =
@@ -121,11 +250,13 @@ public void createRequestFromCurrentItems(@NonNull BlockState blockState) {
                 blockState.blockNumber(),
                 blockState.requests().size());
 
-        // Clear the items list
-        blockState.items().clear();
-
         // Notify the connection manager
         blockNodeConnectionManager.notifyConnectionsOfNewRequest();
+
+        if ((!blockState.items().isEmpty() && force) || blockState.items().size() >= batchSize) {
+            // another request can be created
+            createRequestFromCurrentItems(blockState, force);
+        }
     }
 
     /**
@@ -134,30 +265,14 @@ public void createRequestFromCurrentItems(@NonNull BlockState blockState) {
      * @throws IllegalStateException if no block is currently open
      */
     public void closeBlock(final long blockNumber) {
-        BlockState blockState = getBlockState(blockNumber);
+        final BlockState blockState = getBlockState(blockNumber);
         if (blockState == null) {
             throw new IllegalStateException("Block state not found for block " + blockNumber);
         }
 
-        // Check if there are remaining items
-        if (!blockState.items().isEmpty()) {
-            logger.debug(
-                    "Creating request from remaining items in block {} size {}",
-                    blockNumber,
-                    blockState.items().size());
-            createRequestFromCurrentItems(blockState);
-            // Mark the block as complete
-            blockState.setComplete();
-
-            // TODO: temporarily set here until we have the buffering implemented
-            final Long oldestUnackBlockNum =
-                    blockStates.keySet().stream().min(Long::compareTo).orElse(blockNumber);
-            blockStreamMetrics.setOldestUnacknowledgedBlockTime(
-                    getBlockState(oldestUnackBlockNum).getCompletionTime().toEpochMilli() / 1000L);
-
-            // Notify the connection manager
-            blockNodeConnectionManager.notifyConnectionsOfNewRequest();
-        }
+        // Mark the block as complete
+        blockState.setComplete();
+        createRequestFromCurrentItems(blockState, true);
 
         logger.debug(
                 "Closed block in BlockStreamStateManager {} - request count: {}",
@@ -171,16 +286,27 @@ public void closeBlock(final long blockNumber) {
      * @param blockNumber the block number
      * @return the block state, or null if no block state exists for the given block number
      */
-    public BlockState getBlockState(long blockNumber) {
-        return blockStates.get(blockNumber);
+    public @Nullable BlockState getBlockState(final long blockNumber) {
+        return blockStatesById.get(blockNumber);
+    }
+
+    /**
+     * Retrieves if the specified block has been marked as acknowledged.
+     *
+     * @param blockNumber the block to check
+     * @return true if the block has been acknowledged, else false
+     * @throws IllegalArgumentException if the specified block is not found
+     */
+    public boolean isAcked(final long blockNumber) {
+        return highestAckedBlockNumber.get() >= blockNumber;
     }
 
     /**
      * Creates a new request from the current items in the block prior to BlockProof if there are any.
      * @param blockNumber the block number
      */
-    public void streamPreBlockProofItems(long blockNumber) {
-        BlockState blockState = getBlockState(blockNumber);
+    public void streamPreBlockProofItems(final long blockNumber) {
+        final BlockState blockState = getBlockState(blockNumber);
         if (blockState == null) {
             throw new IllegalStateException("Block state not found for block " + blockNumber);
         }
@@ -191,53 +317,251 @@ public void streamPreBlockProofItems(long blockNumber) {
                     "Prior to BlockProof, creating request from items in block {} size {}",
                     blockNumber,
                     blockState.items().size());
-            createRequestFromCurrentItems(blockState);
+            createRequestFromCurrentItems(blockState, true);
         }
     }
 
     /**
-     * Removes all completed block states older than expiry period with block numbers less than or equal to the given block number. This operation is
-     * thread-safe.
+     * Marks all blocks up to and including the specified block as being acknowledged.
      *
-     * @param blockNumber the block number
+     * @param blockNumber the block number to mark acknowledged up to and including
+     */
+    public void setLatestAcknowledgedBlock(final long blockNumber) {
+        final long highestBlock = highestAckedBlockNumber.updateAndGet(current -> Math.max(current, blockNumber));
+        blockStreamMetrics.setLatestAcknowledgedBlockNumber(highestBlock);
+    }
+
+    /**
+     * Gets the current block number.
+     *
+     * @return the current block number or -1 if no blocks have been opened yet
+     */
+    public long getBlockNumber() {
+        return blockNumber;
+    }
+
+    /**
+     * Ensures that there is enough capacity in the block buffer to permit a new block being created. If there is not
+     * enough capacity - i.e. the buffer is saturated - then this method will block until there is enough capacity.
+     */
+    public static void ensureNewBlocksPermitted() {
+        final CompletableFuture<Boolean> cf = backpressureCompletableFutureRef.get();
+        if (cf != null && !cf.isDone()) {
+            try {
+                logger.error("!!! Block buffer is saturated; blocking thread until buffer is no longer saturated");
+                final long startMs = System.currentTimeMillis();
+                final boolean bufferAvailable = cf.get();
+                final long durationMs = System.currentTimeMillis() - startMs;
+                logger.warn("Thread was blocked for {}ms waiting for block buffer to free space", durationMs);
+
+                if (!bufferAvailable) {
+                    logger.warn("Block buffer still not available to accept new blocks; reentering wait...");
+                    ensureNewBlocksPermitted();
+                }
+            } catch (final InterruptedException e) {
+                Thread.currentThread().interrupt();
+            } catch (final Exception e) {
+                logger.warn("Failed to wait for block buffer to be available", e);
+            }
+        }
+    }
+
+    /**
+     * Prunes the block buffer by removing blocks that have been acknowledged and exceeded the configured TTL. By doing
+     * this, we also inadvertently can know if buffer is "saturated" due to blocks not being acknowledged in a timely
+     * manner.
      */
-    public void removeBlockStatesUpTo(long blockNumber) {
-        // Use keySet().removeIf for atomic removal of multiple entries
-        final Instant fiveMinutesAgo = Instant.now().minus(expiryPeriod);
-        final var removed = blockStates.entrySet().removeIf(entry -> {
-            final long key = entry.getKey();
-            final BlockState state = entry.getValue();
-
-            // Check if the block number is less than or equal to the acknowledged block number
-            if (key <= blockNumber) {
-                // Check if the block state is complete and older than 5 minutes
-                if (state.isComplete()) {
-                    final Instant completionTime = state.getCompletionTime();
-                    // If completionTime is null (shouldn't happen if isComplete is true, but defensive check),
-                    // or if it's older than 5 minutes, mark for removal.
-                    return completionTime == null || completionTime.isBefore(fiveMinutesAgo);
+    private @NonNull PruneResult pruneBuffer() {
+        final Duration ttl = blockBufferTtl();
+        final Instant cutoffInstant = Instant.now().minus(ttl);
+        final Iterator<BlockState> it = blockBuffer.iterator();
+        final long highestBlockAcked = highestAckedBlockNumber.get();
+        /*
+        Calculate the ideal max buffer size. This is calculated as the block buffer TTL (e.g. 5 minutes) divided by the
+        block period (e.g. 2 seconds). This gives us an ideal number of blocks in the buffer.
+         */
+        final long idealMaxBufferSize = ttl.dividedBy(blockPeriod());
+        int numPruned = 0;
+        int numChecked = 0;
+        int numPendingAck = 0;
+        final AtomicReference<Instant> oldestUnackedTimestamp = new AtomicReference<>(Instant.MAX);
+
+        while (it.hasNext()) {
+            final BlockState block = it.next();
+            ++numChecked;
+
+            if (block.isComplete()) {
+                if (block.blockNumber() <= highestBlockAcked) {
+                    // this block is eligible for pruning if it is old enough
+                    if (block.completionTimestamp().isBefore(cutoffInstant)) {
+                        blockStatesById.remove(block.blockNumber());
+                        it.remove();
+                        ++numPruned;
+                    }
                 } else {
-                    // If the block state is not complete, it's not eligible for removal based on the new rule.
-                    // Keep blocks that are <= blockNumber but not yet complete.
-                    return false;
+                    ++numPendingAck;
+                    oldestUnackedTimestamp.updateAndGet(current ->
+                            current.compareTo(block.completionTimestamp()) < 0 ? current : block.completionTimestamp());
                 }
-            } else {
-                // If the block number is greater than the acknowledged block number, don't remove.
-                return false;
             }
-        });
+        }
+
+        final long oldestUnackedMillis = Instant.MAX.equals(oldestUnackedTimestamp.get())
+                ? -1 // sentinel value indicating no blocks are unacked
+                : oldestUnackedTimestamp.get().toEpochMilli();
+        blockStreamMetrics.setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+
+        return new PruneResult(idealMaxBufferSize, numChecked, numPendingAck, numPruned);
+    }
+
+    /*
+    Simple record that contains information related to the outcome of a block buffer prune operation.
+     */
+    private record PruneResult(
+            long idealMaxBufferSize, int numBlocksChecked, int numBlocksPendingAck, int numBlocksPruned) {
+
+        /**
+         * Calculate the saturation percent based on the size of the buffer and the number of unacked blocks found.
+         * @return the saturation percent
+         */
+        double calculateSaturationPercent() {
+            if (idealMaxBufferSize == 0) {
+                return 0D;
+            }
+
+            final BigDecimal size = BigDecimal.valueOf(idealMaxBufferSize);
+            final BigDecimal pending = BigDecimal.valueOf(numBlocksPendingAck);
+            return pending.divide(size, 6, RoundingMode.HALF_EVEN)
+                    .multiply(BigDecimal.valueOf(100))
+                    .doubleValue();
+        }
 
-        if (removed) {
-            logger.debug("Removed completed block states up to block {} that were older than 5 minutes.", blockNumber);
+        /**
+         * Check if the buffer is considered saturated.
+         *
+         * @return true if the block buffer is considered saturated, else false
+         */
+        boolean isSaturated() {
+            return idealMaxBufferSize != 0 && numBlocksPendingAck >= idealMaxBufferSize;
         }
     }
 
     /**
-     * Gets the current block number.
-     *
-     * @return the current block number
+     * Prunes the block buffer and checks if the buffer is saturated. If the buffer is saturated, then a backpressure
+     * mechanism is activated. The backpressure will be enabled until the next time this method is invoked, after which
+     * the backpressure mechanism will be disabled if the buffer is no longer saturated, or maintained if the buffer
+     * continues to be saturated.
      */
-    public long getBlockNumber() {
-        return blockNumber;
+    private void checkBuffer() {
+        final boolean isSaturatedBeforePrune = isBufferSaturated.get();
+        final BlockStreamStateManager.PruneResult result = pruneBuffer();
+        final boolean isSaturatedAfterPrune = result.isSaturated();
+        isBufferSaturated.set(isSaturatedAfterPrune);
+        final double saturationPercent = result.calculateSaturationPercent();
+
+        logger.debug(
+                "Block buffer status: idealMaxBufferSize={}, blocksChecked={}, blocksPruned={}, blocksPendingAck={}, saturation={}%",
+                result.idealMaxBufferSize,
+                result.numBlocksChecked,
+                result.numBlocksPruned,
+                result.numBlocksPendingAck,
+                saturationPercent);
+
+        blockStreamMetrics.updateBlockBufferSaturation(saturationPercent);
+
+        if (isSaturatedBeforePrune == isSaturatedAfterPrune) {
+            // no state change detected, escape early
+            return;
+        }
+
+        if (isSaturatedAfterPrune) {
+            // we've transitioned to a saturated state, apply backpressure
+            logger.warn(
+                    "Block buffer is saturated; backpressure is being enabled "
+                            + "(idealMaxBufferSize={}, blocksChecked={}, blocksPruned={}, blocksPendingAck={}, saturation={}%)",
+                    result.idealMaxBufferSize,
+                    result.numBlocksChecked,
+                    result.numBlocksPruned,
+                    result.numBlocksPendingAck,
+                    saturationPercent);
+
+            CompletableFuture<Boolean> oldCf;
+            CompletableFuture<Boolean> newCf;
+            do {
+                oldCf = backpressureCompletableFutureRef.get();
+                if (oldCf != null) {
+                    /**
+                     * If everything is behaving as expected, then this condition should never be encountered. At any
+                     * given time there should only be one state manager and thus one scheduled prune task. However, if
+                     * there are multiple instances of the manager or something gets messed up threading-wise, then we
+                     * need to handle the possibility that there are multiple blocking futures concurrently. With this
+                     * in mind, we will set the CompletableFuture we use to block in {@link #ensureNewBlocksPermitted()}
+                     * to complete with a value of {@code false}. This false indicates that the CompletableFuture was
+                     * completed but another CompletableFuture took its place and that blocking should continue to
+                     * be enabled.
+                     */
+                    logger.warn("Multiple backpressure blocking futures encountered; this may indicate multiple state "
+                            + "managers or buffer pruning tasks were concurrently active");
+                    oldCf.complete(false);
+                }
+                newCf = new CompletableFuture<>();
+            } while (!backpressureCompletableFutureRef.compareAndSet(oldCf, newCf));
+        } else {
+            // we've transitioned to a non-saturated state, disable backpressure
+            CompletableFuture<Boolean> oldCf;
+            CompletableFuture<Boolean> newCf;
+
+            do {
+                oldCf = backpressureCompletableFutureRef.get();
+                if (oldCf != null) {
+                    /**
+                     * If everything is behaving as expected, then this condition should never be encountered. At any
+                     * given time there should only be one state manager and thus one scheduled prune task. However, if
+                     * there are multiple instances of the manager or something gets messed up threading-wise, then we
+                     * need to handle the possibility that there are multiple blocking futures concurrently. With this
+                     * in mind, we will set the CompletableFuture we use to block in {@link #ensureNewBlocksPermitted()}
+                     * to complete with a value of {@code true}. This true indicates that the CompletableFuture was
+                     * completed and that we are no longer applying backpressure and thus no longer blocking.
+                     */
+                    logger.warn("Multiple backpressure blocking futures encountered; this may indicate multiple state "
+                            + "managers or buffer pruning tasks were concurrently active");
+                    oldCf.complete(true);
+                }
+                newCf = CompletableFuture.completedFuture(true);
+            } while (!backpressureCompletableFutureRef.compareAndSet(oldCf, newCf));
+        }
+    }
+
+    private void scheduleNextPruning() {
+        /*
+        The prune interval may be set to 0, which will effectively disable the pruning. However, we still want to
+        maintain some sensible interval to re-check if the interval has changed, in particular if it is no longer set to
+        0 and thus pruning should be enabled.
+         */
+        final Duration pruneInterval = blockBufferPruneInterval();
+        final long millis = pruneInterval.toMillis() != 0 ? pruneInterval.toMillis() : TimeUnit.SECONDS.toMillis(1);
+        execSvc.schedule(new BufferPruneTask(), millis, TimeUnit.MILLISECONDS);
+    }
+
+    /**
+     * Task that prunes the block buffer.
+     * @see #checkBuffer()
+     */
+    private class BufferPruneTask implements Runnable {
+
+        @Override
+        public void run() {
+            final Duration pruneInterval = blockBufferPruneInterval();
+            try {
+                // If the interval is 0, pruning is disabled, so only do the prune if the interval is NOT 0.
+                if (!pruneInterval.isZero()) {
+                    checkBuffer();
+                }
+            } catch (final RuntimeException e) {
+                logger.warn("Periodic buffer pruning failed", e);
+            } finally {
+                scheduleNextPruning();
+            }
+        }
     }
 }
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/metrics/BlockStreamMetrics.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/metrics/BlockStreamMetrics.java
index 35f83c4f20e7..37bdf7f6d8a8 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/metrics/BlockStreamMetrics.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/metrics/BlockStreamMetrics.java
@@ -5,6 +5,7 @@
 
 import com.hedera.hapi.block.PublishStreamResponseCode;
 import com.swirlds.metrics.api.Counter;
+import com.swirlds.metrics.api.DoubleGauge;
 import com.swirlds.metrics.api.LongGauge;
 import com.swirlds.metrics.api.Metrics;
 import com.swirlds.state.lifecycle.info.NodeInfo;
@@ -43,6 +44,7 @@ public class BlockStreamMetrics {
     private LongGauge producingBlockNumberGauge;
     private LongGauge oldestUnacknowledgedBlockTimeGauge;
     private LongGauge latestAcknowledgedBlockNumberGauge;
+    private DoubleGauge blockBufferSaturationGauge;
 
     @Inject
     public BlockStreamMetrics(@NonNull final Metrics metrics, @NonNull final NodeInfo selfNodeInfo) {
@@ -110,6 +112,18 @@ public void registerMetrics() {
                 metrics.getOrCreate(new LongGauge.Config(APP_CATEGORY, latestAckBlockNumMetricName)
                         .withDescription("Latest block number acknowledged for node " + localNodeId));
 
+        /*
+        Buffer saturation gauge - higher values mean the buffer is nearing saturation. Values over 100 mean the buffer
+        is saturated and "overflowing" though this should be minimal since backpressure should be applied to prevent
+        more blocks from being created
+         */
+        final String bufferSaturationMetricName = "blockBufferSaturation" + nodeLabel;
+        blockBufferSaturationGauge =
+                metrics.getOrCreate(new DoubleGauge.Config(APP_CATEGORY, bufferSaturationMetricName)
+                        .withDescription("Block buffer saturation; Values closer to 100 mean the buffer is nearing"
+                                + "saturation and backpressure may be applied, and values at or above 100 mean the "
+                                + "buffer is fully saturated and potentially overflowing"));
+
         logger.info("Finished registering BlockStreamMetrics for node {}", localNodeId);
     }
 
@@ -171,7 +185,7 @@ public void incrementBlockAckReceivedCount() {
      *
      * @param blockNumber The current block number.
      */
-    public void setProducingBlockNumber(long blockNumber) {
+    public void setProducingBlockNumber(final long blockNumber) {
         if (producingBlockNumberGauge != null) {
             producingBlockNumberGauge.set(blockNumber);
         } else {
@@ -185,7 +199,7 @@ public void setProducingBlockNumber(long blockNumber) {
      *
      * @param timestamp The timestamp in milliseconds since epoch.
      */
-    public void setOldestUnacknowledgedBlockTime(long timestamp) {
+    public void setOldestUnacknowledgedBlockTime(final long timestamp) {
         if (oldestUnacknowledgedBlockTimeGauge != null) {
             oldestUnacknowledgedBlockTimeGauge.set(timestamp);
         } else {
@@ -199,7 +213,7 @@ public void setOldestUnacknowledgedBlockTime(long timestamp) {
      *
      * @param blockNumber The block number of the latest acknowledgement.
      */
-    public void setLatestAcknowledgedBlockNumber(long blockNumber) {
+    public void setLatestAcknowledgedBlockNumber(final long blockNumber) {
         if (latestAcknowledgedBlockNumberGauge != null) {
             latestAcknowledgedBlockNumberGauge.set(blockNumber);
         } else {
@@ -219,4 +233,14 @@ public void incrementOnErrorCount() {
             logger.warn("onErrorCounter not found.");
         }
     }
+
+    /**
+     * Updates the current block buffer saturation percent.
+     * @param saturation the current block buffer saturation percent
+     */
+    public void updateBlockBufferSaturation(final double saturation) {
+        if (blockBufferSaturationGauge != null) {
+            blockBufferSaturationGauge.set(saturation);
+        }
+    }
 }
diff --git a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/HandleWorkflow.java b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/HandleWorkflow.java
index d2b9ad25f926..df4d670f6e37 100644
--- a/hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/HandleWorkflow.java
+++ b/hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/HandleWorkflow.java
@@ -39,6 +39,7 @@
 import com.hedera.node.app.blocks.BlockStreamManager;
 import com.hedera.node.app.blocks.impl.BoundaryStateChangeListener;
 import com.hedera.node.app.blocks.impl.KVStateChangeListener;
+import com.hedera.node.app.blocks.impl.streaming.BlockStreamStateManager;
 import com.hedera.node.app.fees.ExchangeRateManager;
 import com.hedera.node.app.hints.HintsService;
 import com.hedera.node.app.hints.impl.ReadableHintsStoreImpl;
@@ -235,6 +236,7 @@ public void handleRound(
             @NonNull final Round round,
             @NonNull final Consumer<ScopedSystemTransaction<StateSignatureTransaction>> stateSignatureTxnCallback) {
         logStartRound(round);
+        BlockStreamStateManager.ensureNewBlocksPermitted();
         cacheWarmer.warm(state, round);
         if (streamMode != RECORDS) {
             blockStreamManager.startRound(round, state);
diff --git a/hedera-node/hedera-app/src/main/java/module-info.java b/hedera-node/hedera-app/src/main/java/module-info.java
index e5a09dee0a64..7e3e93f1c099 100644
--- a/hedera-node/hedera-app/src/main/java/module-info.java
+++ b/hedera-node/hedera-app/src/main/java/module-info.java
@@ -31,6 +31,7 @@
     requires transitive io.grpc;
     requires transitive io.helidon.grpc.core;
     requires transitive io.helidon.webclient.grpc;
+    requires transitive java.annotation;
     requires transitive javax.inject;
     requires transitive org.apache.logging.log4j;
     requires transitive org.hyperledger.besu.datatypes;
diff --git a/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManagerTest.java b/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManagerTest.java
index bf5f48472bd1..bd604089e34e 100644
--- a/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManagerTest.java
+++ b/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockNodeConnectionManagerTest.java
@@ -5,7 +5,6 @@
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.mockito.BDDMockito.given;
 import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.never;
 import static org.mockito.Mockito.times;
 import static org.mockito.Mockito.verify;
 import static org.mockito.Mockito.when;
@@ -329,7 +328,7 @@ void testUpdateLastVerifiedBlock_WhenNewBlockNumberIsGreater() {
         // Then
         assertThat(blockNodeConnectionManager.getLastVerifiedBlock(blockNodeConfig))
                 .isEqualTo(newBlockNumber);
-        verify(blockStreamMetrics).setLatestAcknowledgedBlockNumber(newBlockNumber);
+        verify(mockStateManager).setLatestAcknowledgedBlock(newBlockNumber);
     }
 
     @Test
@@ -348,7 +347,7 @@ void testUpdateLastVerifiedBlock_WhenNewBlockNumberIsLess() {
         // Then
         assertThat(blockNodeConnectionManager.getLastVerifiedBlock(blockNodeConfig))
                 .isEqualTo(initialBlockNumber);
-        verify(blockStreamMetrics, never()).setLatestAcknowledgedBlockNumber(newBlockNumber);
+        verify(mockStateManager).setLatestAcknowledgedBlock(newBlockNumber);
     }
 
     @Test
@@ -360,9 +359,6 @@ void testUpdateLastVerifiedBlock_WhenNewBlockNumberIsEqual() {
         // Set initial block number
         blockNodeConnectionManager.updateLastVerifiedBlock(blockNodeConfig, blockNumber);
 
-        // Reset mock to verify it's not called again
-        verify(blockStreamMetrics).setLatestAcknowledgedBlockNumber(blockNumber);
-
         // When
         blockNodeConnectionManager.updateLastVerifiedBlock(blockNodeConfig, blockNumber);
 
@@ -370,7 +366,7 @@ void testUpdateLastVerifiedBlock_WhenNewBlockNumberIsEqual() {
         assertThat(blockNodeConnectionManager.getLastVerifiedBlock(blockNodeConfig))
                 .isEqualTo(blockNumber);
         // Verify it was called only once (from the initial setup)
-        verify(blockStreamMetrics, times(1)).setLatestAcknowledgedBlockNumber(blockNumber);
+        verify(mockStateManager, times(2)).setLatestAcknowledgedBlock(blockNumber);
     }
 
     @Test
@@ -388,6 +384,6 @@ void testUpdateLastVerifiedBlock_WhenNewBlockNumberIsNull() {
         // Then
         assertThat(blockNodeConnectionManager.getLastVerifiedBlock(blockNodeConfig))
                 .isEqualTo(initialBlockNumber);
-        verify(blockStreamMetrics, times(1)).setLatestAcknowledgedBlockNumber(initialBlockNumber);
+        verify(mockStateManager, times(1)).setLatestAcknowledgedBlock(initialBlockNumber);
     }
 }
diff --git a/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManagerTest.java b/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManagerTest.java
index ad479d9bacaf..f5f774a8cd9b 100644
--- a/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManagerTest.java
+++ b/hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManagerTest.java
@@ -7,6 +7,10 @@
 import static org.junit.jupiter.api.Assertions.assertAll;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.mockito.BDDMockito.given;
+import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
 
 import com.hedera.hapi.block.stream.BlockItem;
 import com.hedera.hapi.block.stream.BlockProof;
@@ -17,7 +21,21 @@
 import com.hedera.node.config.VersionedConfigImpl;
 import com.hedera.node.config.data.BlockStreamConfig;
 import com.hedera.node.config.testfixtures.HederaTestConfigBuilder;
+import com.swirlds.config.api.Configuration;
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodHandles.Lookup;
+import java.lang.invoke.VarHandle;
+import java.lang.reflect.Method;
 import java.time.Duration;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
@@ -26,6 +44,26 @@
 
 @ExtendWith(MockitoExtension.class)
 class BlockStreamStateManagerTest {
+
+    private static final VarHandle execSvcHandle;
+    private static final VarHandle blockBufferHandle;
+    private static final MethodHandle checkBufferHandle;
+
+    static {
+        try {
+            final Lookup lookup = MethodHandles.lookup();
+            blockBufferHandle = MethodHandles.privateLookupIn(BlockStreamStateManager.class, lookup)
+                    .findVarHandle(BlockStreamStateManager.class, "blockBuffer", BlockingQueue.class);
+            execSvcHandle = MethodHandles.privateLookupIn(BlockStreamStateManager.class, lookup)
+                    .findVarHandle(BlockStreamStateManager.class, "execSvc", ScheduledExecutorService.class);
+            final Method checkBufferMethod = BlockStreamStateManager.class.getDeclaredMethod("checkBuffer");
+            checkBufferMethod.setAccessible(true);
+            checkBufferHandle = lookup.unreflect(checkBufferMethod);
+        } catch (final Exception e) {
+            throw new RuntimeException(e);
+        }
+    }
+
     private static final long TEST_BLOCK_NUMBER = 1L;
     private static final long TEST_BLOCK_NUMBER2 = 2L;
     private static final long TEST_BLOCK_NUMBER3 = 3L;
@@ -41,16 +79,22 @@ class BlockStreamStateManagerTest {
 
     private BlockStreamStateManager blockStreamStateManager;
 
-    private static final Duration EXPIRY_PERIOD = Duration.ofMinutes(5L);
-
     @BeforeEach
     void setUp() {
-        given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(DEFAULT_CONFIG, 1));
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        lenient().when(configProvider.getConfiguration()).thenReturn(new VersionedConfigImpl(DEFAULT_CONFIG, 1));
+    }
+
+    @AfterEach
+    void afterEach() throws InterruptedException {
+        // stop the async pruning thread(s)
+        final ScheduledExecutorService execSvc = (ScheduledExecutorService) execSvcHandle.get(blockStreamStateManager);
+        execSvc.shutdownNow();
+        assertThat(execSvc.awaitTermination(3, TimeUnit.SECONDS)).isTrue();
     }
 
     @Test
     void testOpenNewBlock() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         // given
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         // when
@@ -70,15 +114,17 @@ void testOpenNewBlock() {
 
     @Test
     void testCleanUp_NotCompletedBlockState_ShouldNotBeRemoved() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         // given
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
 
         // when
-        blockStreamStateManager.removeBlockStatesUpTo(TEST_BLOCK_NUMBER);
+        blockStreamStateManager.setLatestAcknowledgedBlock(TEST_BLOCK_NUMBER);
 
         // then
         // not completed states should not be removed
+        assertThat(blockStreamStateManager.isAcked(TEST_BLOCK_NUMBER)).isTrue();
         final BlockState actualBlockState = blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER);
         assertThat(actualBlockState).isNotNull();
         assertFalse(actualBlockState.isComplete());
@@ -86,6 +132,7 @@ void testCleanUp_NotCompletedBlockState_ShouldNotBeRemoved() {
 
     @Test
     void testCleanUp_CompletedNotExpiredBlockState_ShouldNotBeRemoved() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         // given
         // expiry period set to zero in order for completed state to be cleared
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
@@ -93,32 +140,16 @@ void testCleanUp_CompletedNotExpiredBlockState_ShouldNotBeRemoved() {
         blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).setComplete();
 
         // when
-        blockStreamStateManager.removeBlockStatesUpTo(TEST_BLOCK_NUMBER);
+        blockStreamStateManager.setLatestAcknowledgedBlock(TEST_BLOCK_NUMBER);
 
         // then
         // completed states should be removed
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER)).isNotNull();
     }
 
-    @Test
-    void testCleanUp_CompletedExpiredBlockState_ShouldBeRemoved() {
-        // given
-        // expiry period set to zero in order for completed state to be cleared
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, Duration.ofNanos(0L));
-        blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
-        blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).setComplete();
-
-        // when
-        blockStreamStateManager.removeBlockStatesUpTo(TEST_BLOCK_NUMBER);
-
-        // then
-        // completed states should be removed
-        assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER)).isNull();
-    }
-
     @Test
     void testMaintainMultipleBlockStates() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         // given
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         // when
@@ -144,8 +175,9 @@ void testMaintainMultipleBlockStates() {
 
     @Test
     void testHandleNonExistentBlockState() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         // when
-        BlockState blockState = blockStreamStateManager.getBlockState(999L);
+        final BlockState blockState = blockStreamStateManager.getBlockState(999L);
 
         // then
         assertThat(blockState).isNull();
@@ -155,23 +187,20 @@ void testHandleNonExistentBlockState() {
     void testPublishStreamRequestIsNotCreatedWhenBatchSizeIsNotMet() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 4)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
@@ -180,7 +209,7 @@ void testPublishStreamRequestIsNotCreatedWhenBatchSizeIsNotMet() {
         // then
         // verify that request is still not created and block items are added to state
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).requests())
-                .hasSize(0);
+                .isEmpty();
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).items())
                 .hasSize(2);
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).items())
@@ -191,23 +220,20 @@ void testPublishStreamRequestIsNotCreatedWhenBatchSizeIsNotMet() {
     void testPublishStreamRequestIsCreatedWhenBatchSizeIsMet() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 2)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
@@ -218,33 +244,28 @@ void testPublishStreamRequestIsCreatedWhenBatchSizeIsMet() {
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).requests())
                 .hasSize(1);
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).items())
-                .hasSize(0);
+                .isEmpty();
     }
 
     @Test
     void testWithMoreBlockItemsThanBlockItemBatchSize() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 2)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
-        var blockItem3 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
+        final var blockItem3 = newBlockTxItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
@@ -265,33 +286,28 @@ void testWithMoreBlockItemsThanBlockItemBatchSize() {
     void testPublishStreamRequestIsCreatedWithRemainingItemsAndBlockProof() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 5)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
-        var blockProof = BlockItem.newBuilder()
-                .blockProof(BlockProof.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
+        final var blockProof = newBlockHeaderItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem2);
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockProof);
-        var blockState = blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER);
-        blockStreamStateManager.createRequestFromCurrentItems(blockState);
+        final var blockState = blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER);
+        blockStreamStateManager.createRequestFromCurrentItems(blockState, true);
 
         // then
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).requests())
@@ -302,25 +318,24 @@ void testPublishStreamRequestIsCreatedWithRemainingItemsAndBlockProof() {
     void testPublishStreamRequestIsCreatedWithBlockProofOnly() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 5)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockProof = BlockItem.newBuilder()
-                .blockProof(BlockProof.newBuilder().build())
-                .build();
+        final var blockProof = newBlockProofItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockProof);
-        var blockState = blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER);
-        blockStreamStateManager.createRequestFromCurrentItems(blockState);
+        final var blockState = blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER);
+        blockStreamStateManager.createRequestFromCurrentItems(blockState, true);
 
         // then
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).requests())
@@ -331,23 +346,20 @@ void testPublishStreamRequestIsCreatedWithBlockProofOnly() {
     void testPublishStreamRequestCreatedWithRemainingBlockItemsOnBlockCLose() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 5)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
@@ -369,14 +381,15 @@ void testPublishStreamRequestCreatedWithRemainingBlockItemsOnBlockCLose() {
     void testCompletedExpiredBlockStateIsRemovedUpToSpecificBlockNumber() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 5)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, Duration.ofNanos(0L));
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
@@ -385,45 +398,38 @@ void testCompletedExpiredBlockStateIsRemovedUpToSpecificBlockNumber() {
         blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER2).setComplete();
 
         // when
-        blockStreamStateManager.removeBlockStatesUpTo(TEST_BLOCK_NUMBER);
+        blockStreamStateManager.setLatestAcknowledgedBlock(TEST_BLOCK_NUMBER);
 
         // then
-        assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER)).isNull();
+        assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER)).isNotNull();
+        assertThat(blockStreamStateManager.isAcked(TEST_BLOCK_NUMBER)).isTrue();
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER2)).isNotNull();
+        assertThat(blockStreamStateManager.isAcked(TEST_BLOCK_NUMBER2)).isFalse();
     }
 
     @Test
     void testPublishStreamRequestsCreatedForMultipleBLocks() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 2)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER2);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER3);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
-        var blockItem3 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
-        var blockItem4 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
-        var blockItem5 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
+        final var blockItem3 = newBlockTxItem();
+        final var blockItem4 = newBlockTxItem();
+        final var blockItem5 = newBlockTxItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
@@ -438,21 +444,24 @@ void testPublishStreamRequestsCreatedForMultipleBLocks() {
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER2).requests())
                 .hasSize(1);
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER3).requests())
-                .hasSize(0);
+                .isEmpty();
     }
 
     @Test
     void testGetCurrentBlockNumberWhenNoNewBlockIsOpened() {
         // given
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
 
         // when and then
-        assertThat(blockStreamStateManager.getBlockNumber()).isEqualTo(0L);
+        // -1 is a sentinel value indicating no block has been opened
+        assertThat(blockStreamStateManager.getBlockNumber()).isEqualTo(-1);
     }
 
     @Test
     void testGetCurrentBlockNumberWhenNewBlockIsOpened() {
         // given
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER2);
 
@@ -464,17 +473,21 @@ void testGetCurrentBlockNumberWhenNewBlockIsOpened() {
     @Test
     void testOpenBlockWithNegativeBlockNumber() {
         // given
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
 
         // when and then
         assertThatThrownBy(() -> blockStreamStateManager.openBlock(-1L))
                 .isInstanceOf(IllegalArgumentException.class)
                 .hasMessageContaining("Block number must be non-negative");
-        assertThat(blockStreamStateManager.getBlockNumber()).isEqualTo(0L);
+
+        // -1 is a sentinel value indicating no block has been opened
+        assertThat(blockStreamStateManager.getBlockNumber()).isEqualTo(-1L);
     }
 
     @Test
     void testAddNullBlockItem() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         // given
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
@@ -488,6 +501,7 @@ void testAddNullBlockItem() {
     @Test
     void testAddBlockItemToNonExistentBlockState() {
         // given
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
 
         // when and then
@@ -500,6 +514,7 @@ void testAddBlockItemToNonExistentBlockState() {
     @Test
     void testCloseBlockForNonExistentBlockState() {
         // given
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
 
         // when and then
@@ -511,6 +526,7 @@ void testCloseBlockForNonExistentBlockState() {
     @Test
     void testGetNonExistentBlockState() {
         // given
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
 
         // when and then
@@ -520,6 +536,7 @@ void testGetNonExistentBlockState() {
     @Test
     void testStreamPreBlockProofItemsForNonExistentBlockState() {
         // given
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
 
         // when and then
@@ -532,23 +549,20 @@ void testStreamPreBlockProofItemsForNonExistentBlockState() {
     void testSetBlockItemBatchSizeToZero() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 0)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
@@ -563,23 +577,20 @@ void testSetBlockItemBatchSizeToZero() {
     void testSetBlockItemBatchSizeToOne() {
         // given
         // mock the number of batch items by modifying the default config
-        var mockConfig = HederaTestConfigBuilder.create()
+        final var mockConfig = HederaTestConfigBuilder.create()
                 .withConfigDataType(BlockStreamConfig.class)
                 .withValue("blockStream.blockItemBatchSize", 1)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
                 .getOrCreateConfig();
         given(configProvider.getConfiguration()).willReturn(new VersionedConfigImpl(mockConfig, 1));
 
         // make blockStreamStateManager use the mocked config
-        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics, EXPIRY_PERIOD);
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
 
         blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
         blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
-        var blockItem1 = BlockItem.newBuilder()
-                .blockHeader(BlockHeader.newBuilder().build())
-                .build();
-        var blockItem2 = BlockItem.newBuilder()
-                .transactionOutput(TransactionOutput.newBuilder().build())
-                .build();
+        final var blockItem1 = newBlockHeaderItem();
+        final var blockItem2 = newBlockTxItem();
 
         // when
         blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, blockItem1);
@@ -589,4 +600,339 @@ void testSetBlockItemBatchSizeToOne() {
         assertThat(blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER).requests())
                 .hasSize(2);
     }
+
+    @Test
+    void testMultiBatch() {
+        final Configuration config = HederaTestConfigBuilder.create()
+                .withConfigDataType(BlockStreamConfig.class)
+                .withValue("blockStream.blockItemBatchSize", 3)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
+                .getOrCreateConfig();
+        when(configProvider.getConfiguration()).thenReturn(new VersionedConfigImpl(config, 1));
+
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
+
+        blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
+        blockStreamStateManager.openBlock(TEST_BLOCK_NUMBER);
+        assertThat(blockStreamStateManager.isBufferSaturated()).isFalse();
+
+        // batch size is 3, add 8 items
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockHeaderItem());
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockTxItem());
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockTxItem());
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockTxItem());
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockTxItem());
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockTxItem());
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockTxItem());
+        blockStreamStateManager.addItem(TEST_BLOCK_NUMBER, newBlockProofItem());
+
+        final BlockState blockState = blockStreamStateManager.getBlockState(TEST_BLOCK_NUMBER);
+
+        // #addItem(...) will attempt to create a new request if there are enough items. With a batch size of 3, we
+        // should expect 2 requests created with 2 more pending items
+        assertThat(blockState.requests()).hasSize(2);
+        assertThat(blockState.items()).hasSize(2);
+
+        // now force the creation of the final request
+        blockStreamStateManager.createRequestFromCurrentItems(blockState, true);
+
+        // there should be 3 requests now and no outstanding items
+        assertThat(blockState.requests()).hasSize(3);
+        assertThat(blockState.items()).isEmpty();
+    }
+
+    @Test
+    void testOpenExistingBlock() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
+        blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
+        blockStreamStateManager.openBlock(2L);
+
+        // try to open the same block number
+        assertThatThrownBy(() -> blockStreamStateManager.openBlock(2L))
+                .isInstanceOf(IllegalStateException.class)
+                .hasMessageContaining("Attempted to open a new block with number 2, but a block with the same or "
+                        + "later number (latest: 2) has already been opened");
+
+        // try to open an older block
+        assertThatThrownBy(() -> blockStreamStateManager.openBlock(1L))
+                .isInstanceOf(IllegalStateException.class)
+                .hasMessageContaining("Attempted to open a new block with number 1, but a block with the same or "
+                        + "later number (latest: 2) has already been opened");
+    }
+
+    @Test
+    void testBuffer() throws Throwable {
+        final Duration blockTtl = Duration.ofSeconds(5);
+        final Configuration config = HederaTestConfigBuilder.create()
+                .withConfigDataType(BlockStreamConfig.class)
+                .withValue("blockStream.blockPeriod", Duration.ofSeconds(1))
+                .withValue("blockStream.blockItemBatchSize", 3)
+                .withValue("blockStream.blockBufferTtl", blockTtl)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
+                .getOrCreateConfig();
+        when(configProvider.getConfiguration()).thenReturn(new VersionedConfigImpl(config, 1));
+
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
+        blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
+        final BlockingQueue<?> buffer = (BlockingQueue<?>) blockBufferHandle.get(blockStreamStateManager);
+
+        // IdealMaxBufferSize = BlockTtl (5s) / BlockPeriod (1s) = 5
+
+        // add some blocks, but don't ack them
+        blockStreamStateManager.openBlock(1L);
+        blockStreamStateManager.closeBlock(1L);
+        blockStreamStateManager.openBlock(2L);
+        blockStreamStateManager.closeBlock(2L);
+        blockStreamStateManager.openBlock(3L);
+        blockStreamStateManager.closeBlock(3L);
+        blockStreamStateManager.openBlock(4L);
+        blockStreamStateManager.closeBlock(4L);
+
+        // wait for the TTL period, with a little padding
+        Thread.sleep(blockTtl.plusMillis(250));
+        // prune the buffer, nothing should be removed since nothing is acked and we are not yet saturated
+        checkBufferHandle.invoke(blockStreamStateManager);
+        assertThat(blockStreamStateManager.isBufferSaturated()).isFalse();
+        verify(blockStreamMetrics).updateBlockBufferSaturation(80.0); // the buffer is 80% saturated
+        long oldestUnackedMillis =
+                blockStreamStateManager.getBlockState(1L).completionTimestamp().toEpochMilli();
+        verify(blockStreamMetrics).setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+        assertThat(buffer).hasSize(4);
+
+        // reset the block stream metrics mock to capture the next interaction that has the same value as before
+        reset(blockStreamMetrics);
+
+        // add another block and prune again, this will cause the buffer to be fully saturated
+        blockStreamStateManager.openBlock(5L);
+        blockStreamStateManager.closeBlock(5L);
+        checkBufferHandle.invoke(blockStreamStateManager);
+        // the buffer is now marked as saturated because multiple blocks have not been acked yet and they are expired
+        assertThat(blockStreamStateManager.isBufferSaturated()).isTrue();
+        verify(blockStreamMetrics).updateBlockBufferSaturation(100.0); // the buffer is 100% saturated
+        oldestUnackedMillis =
+                blockStreamStateManager.getBlockState(1L).completionTimestamp().toEpochMilli();
+        verify(blockStreamMetrics).setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+
+        // reset the block stream metrics mock to capture the next interaction that has the same value as before
+        reset(blockStreamMetrics);
+
+        assertThat(buffer).hasSize(5);
+
+        // "overflow" the buffer
+        blockStreamStateManager.openBlock(6L);
+        blockStreamStateManager.closeBlock(6L);
+        checkBufferHandle.invoke(blockStreamStateManager);
+        assertThat(blockStreamStateManager.isBufferSaturated()).isTrue();
+        verify(blockStreamMetrics).updateBlockBufferSaturation(120.0); // the buffer is 120% saturated
+        oldestUnackedMillis =
+                blockStreamStateManager.getBlockState(1L).completionTimestamp().toEpochMilli();
+        verify(blockStreamMetrics).setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+        reset(blockStreamMetrics);
+        assertThat(buffer).hasSize(6);
+
+        // ack up to block 3
+        blockStreamStateManager.setLatestAcknowledgedBlock(3L);
+        verify(blockStreamMetrics).setLatestAcknowledgedBlockNumber(3L);
+
+        // now blocks 1-3 are acked
+        assertThat(blockStreamStateManager.isAcked(1L)).isTrue();
+        assertThat(blockStreamStateManager.isAcked(2L)).isTrue();
+        assertThat(blockStreamStateManager.isAcked(3L)).isTrue();
+
+        // now that multiple blocks are acked, run pruning again and verify we are no longer saturated
+        checkBufferHandle.invoke(blockStreamStateManager);
+        assertThat(blockStreamStateManager.isBufferSaturated()).isFalse();
+        verify(blockStreamMetrics).updateBlockBufferSaturation(60.0); // the buffer is 60% saturated
+        oldestUnackedMillis =
+                blockStreamStateManager.getBlockState(4L).completionTimestamp().toEpochMilli();
+        verify(blockStreamMetrics).setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+        reset(blockStreamMetrics);
+        assertThat(buffer).hasSize(3);
+
+        // ack up to block 6, run pruning, and verify the buffer is not saturated
+        blockStreamStateManager.setLatestAcknowledgedBlock(6L);
+        Thread.sleep(blockTtl.plusMillis(250));
+        checkBufferHandle.invoke(blockStreamStateManager);
+        assertThat(blockStreamStateManager.isBufferSaturated()).isFalse();
+        verify(blockStreamMetrics).updateBlockBufferSaturation(0.0); // the buffer is 0% saturated
+        verify(blockStreamMetrics).setOldestUnacknowledgedBlockTime(-1); // there is no unacked block
+        reset(blockStreamMetrics);
+        assertThat(buffer).isEmpty();
+
+        // now add another block without acking and ensure the buffer is partially saturated
+        blockStreamStateManager.openBlock(7L);
+        blockStreamStateManager.closeBlock(7L);
+        checkBufferHandle.invoke(blockStreamStateManager);
+        assertThat(blockStreamStateManager.isBufferSaturated()).isFalse();
+        verify(blockStreamMetrics).updateBlockBufferSaturation(20.0); // the buffer is 20% saturated
+        oldestUnackedMillis =
+                blockStreamStateManager.getBlockState(7L).completionTimestamp().toEpochMilli();
+        verify(blockStreamMetrics).setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+        reset(blockStreamMetrics);
+        assertThat(buffer).hasSize(1);
+    }
+
+    @Test
+    void testFutureBlockAcked() throws Throwable {
+        /*
+         * There is a scenario where a block node (BN) may have a later block than what the active consensus node (CN)
+         * has. For example, if a CN goes down then then another CN node may send blocks to the BN. When the original
+         * CN reconnects to the BN, the BN may indicate that is has later blocks from another CN.
+         */
+
+        final Duration blockTtl = Duration.ofSeconds(1);
+        final Configuration config = HederaTestConfigBuilder.create()
+                .withConfigDataType(BlockStreamConfig.class)
+                .withValue("blockStream.blockItemBatchSize", 3)
+                .withValue("blockStream.blockBufferTtl", blockTtl)
+                .withValue("blockStream.blockBufferPruneInterval", Duration.ZERO) // disable auto pruning
+                .getOrCreateConfig();
+        when(configProvider.getConfiguration()).thenReturn(new VersionedConfigImpl(config, 1));
+
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
+
+        blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
+        blockStreamStateManager.openBlock(1L);
+
+        // Block 1 has been added. Now lets ack up to block 5
+        blockStreamStateManager.setLatestAcknowledgedBlock(5L);
+
+        // Since we've acked up to block 5, the block we opened _and_ any blocks we've yet to process up to 5 should
+        // be considered acked
+        assertThat(blockStreamStateManager.isAcked(1L)).isTrue();
+        assertThat(blockStreamStateManager.isAcked(2L)).isTrue();
+        assertThat(blockStreamStateManager.isAcked(3L)).isTrue();
+        assertThat(blockStreamStateManager.isAcked(4L)).isTrue();
+        assertThat(blockStreamStateManager.isAcked(5L)).isTrue();
+        assertThat(blockStreamStateManager.isAcked(6L)).isFalse(); // only blocks up to 5 have been acked
+
+        // Since we've acked up to block 5, that also means any blocks up to 5 will also be pruned as soon as they
+        // expire
+        // Add some more blocks, then check after pruning
+        blockStreamStateManager.openBlock(2L);
+        blockStreamStateManager.openBlock(3L);
+        blockStreamStateManager.openBlock(4L);
+        blockStreamStateManager.openBlock(5L);
+        blockStreamStateManager.openBlock(6L);
+
+        // close the blocks
+        blockStreamStateManager.closeBlock(1L);
+        blockStreamStateManager.closeBlock(2L);
+        blockStreamStateManager.closeBlock(3L);
+        blockStreamStateManager.closeBlock(4L);
+        blockStreamStateManager.closeBlock(5L);
+        blockStreamStateManager.closeBlock(6L);
+
+        // wait for the TTL period, with a little padding
+        Thread.sleep(blockTtl.plusMillis(250));
+        checkBufferHandle.invoke(blockStreamStateManager);
+
+        // Add another block to trigger the prune, then verify the state... there should only be blocks 6 and 7 buffered
+        blockStreamStateManager.openBlock(7L);
+
+        final BlockingQueue<?> buffer = (BlockingQueue<?>) blockBufferHandle.get(blockStreamStateManager);
+        assertThat(buffer).hasSize(2);
+        assertThat(blockStreamStateManager.getBlockState(6L)).isNotNull();
+        assertThat(blockStreamStateManager.getBlockState(7L)).isNotNull();
+    }
+
+    @Test
+    void testBufferBackpressure() throws Throwable {
+        // ensure block TTL is greater than prune interval for this test to work as expected
+        final Duration blockTtl = Duration.ofSeconds(2);
+        final Duration pruneInterval = Duration.ofSeconds(1);
+        final Configuration config = HederaTestConfigBuilder.create()
+                .withConfigDataType(BlockStreamConfig.class)
+                .withValue("blockStream.blockItemBatchSize", 3)
+                .withValue("blockStream.blockBufferTtl", blockTtl)
+                .withValue("blockStream.blockBufferPruneInterval", pruneInterval)
+                .getOrCreateConfig();
+        when(configProvider.getConfiguration()).thenReturn(new VersionedConfigImpl(config, 1));
+
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
+        blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
+
+        final CountDownLatch startLatch = new CountDownLatch(1);
+        final CountDownLatch doneLatch = new CountDownLatch(1);
+        final AtomicLong waitDurationMs = new AtomicLong(0L);
+        final AtomicReference<Throwable> exceptionRef = new AtomicReference<>(null);
+
+        ForkJoinPool.commonPool().execute(() -> {
+            try {
+                startLatch.await();
+
+                final long start = System.currentTimeMillis();
+                BlockStreamStateManager.ensureNewBlocksPermitted();
+                final long durationMs = System.currentTimeMillis() - start;
+                waitDurationMs.set(durationMs);
+            } catch (final Exception e) {
+                exceptionRef.set(e);
+            } finally {
+                doneLatch.countDown();
+            }
+        });
+
+        // create some blocks such that the buffer will be saturated
+        blockStreamStateManager.openBlock(1L);
+        blockStreamStateManager.closeBlock(1L);
+        blockStreamStateManager.openBlock(2L);
+        blockStreamStateManager.closeBlock(2L);
+        blockStreamStateManager.openBlock(3L);
+        blockStreamStateManager.closeBlock(3L);
+
+        // Auto-pruning is enabled and since the prune internal is less than the block TTL, by waiting for the block TTL
+        // period, plus some extra time, the pruning should detect that the buffer is saturated and enable backpressure
+        Thread.sleep(blockTtl.plusMillis(250));
+        // Now start the thread we spawned earlier and have this current thread sleep for a couple seconds to prove the
+        // other thread is blocked
+        startLatch.countDown();
+        Thread.sleep(2_000);
+        // ack the blocks and wait for some more time... this should allow the
+        blockStreamStateManager.setLatestAcknowledgedBlock(3L);
+        Thread.sleep(1_000);
+        // wait for the spawned thread to complete
+        assertThat(doneLatch.await(3, TimeUnit.SECONDS)).isTrue();
+
+        // the spawned thread has completed, now verify state
+        assertThat(exceptionRef).hasNullValue(); // no exception should have occurred
+        // between the time the spawned thread was started and the time the buffer was marked as not being saturated
+        // should be at least 2 seconds - since we slept for that long before doing the ack
+        assertThat(waitDurationMs).hasValueGreaterThan(2_000L);
+    }
+
+    @Test
+    void testSetLatestAcknowledgedBlock() {
+        blockStreamStateManager = new BlockStreamStateManager(configProvider, blockStreamMetrics);
+        blockStreamStateManager.setBlockNodeConnectionManager(blockNodeConnectionManager);
+
+        blockStreamStateManager.setLatestAcknowledgedBlock(1L);
+        verify(blockStreamMetrics).setLatestAcknowledgedBlockNumber(1L);
+        reset(blockStreamMetrics);
+
+        blockStreamStateManager.setLatestAcknowledgedBlock(0L);
+        verify(blockStreamMetrics).setLatestAcknowledgedBlockNumber(1L);
+        reset(blockStreamMetrics);
+
+        blockStreamStateManager.setLatestAcknowledgedBlock(100L);
+        verify(blockStreamMetrics).setLatestAcknowledgedBlockNumber(100L);
+    }
+
+    @Test
+    private static BlockItem newBlockHeaderItem() {
+        return BlockItem.newBuilder()
+                .blockHeader(BlockHeader.newBuilder().build())
+                .build();
+    }
+
+    private static BlockItem newBlockTxItem() {
+        return BlockItem.newBuilder()
+                .transactionOutput(TransactionOutput.newBuilder().build())
+                .build();
+    }
+
+    private static BlockItem newBlockProofItem() {
+        return BlockItem.newBuilder()
+                .blockProof(BlockProof.newBuilder().build())
+                .build();
+    }
 }
diff --git a/hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java b/hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
index 0e3fa6e19735..734b4a392761 100644
--- a/hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
+++ b/hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
@@ -19,6 +19,9 @@
  * @param roundsPerBlock the number of rounds per block
  * @param blockPeriod the block period
  * @param blockItemBatchSize the number of items to send in a batch to block nodes
+ * @param blockBufferTtl the TTL for entries in the block buffer
+ * @param blockBufferPruneInterval interval to prune block buffer and check for whether backpressure is needed; if set
+ *                                 to 0 then pruning is effectively disabled
  */
 @ConfigData("blockStream")
 public record BlockStreamConfig(
@@ -28,7 +31,9 @@ public record BlockStreamConfig(
         @ConfigProperty(defaultValue = "32") @NetworkProperty int hashCombineBatchSize,
         @ConfigProperty(defaultValue = "1") @NetworkProperty int roundsPerBlock,
         @ConfigProperty(defaultValue = "2s") @Min(0) @NetworkProperty Duration blockPeriod,
-        @ConfigProperty(defaultValue = "256") @Min(0) @NetworkProperty int blockItemBatchSize) {
+        @ConfigProperty(defaultValue = "256") @Min(0) @NetworkProperty int blockItemBatchSize,
+        @ConfigProperty(defaultValue = "5m") @Min(0) @NetworkProperty Duration blockBufferTtl,
+        @ConfigProperty(defaultValue = "1s") @Min(0) @NetworkProperty Duration blockBufferPruneInterval) {
 
     /**
      * Whether to stream to block nodes.


Reviews:
user: login: timfn-hg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T19:37:42Z
user: login: timfn-hg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T16:48:32Z
user: login: tinker-michaelj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T12:15:43Z
user: login: tinker-michaelj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T12:22:45Z
user: login: timfn-hg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T13:20:55Z
user: login: timfn-hg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T14:58:23Z
user: login: Nana-EC
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T21:21:30Z
user: login: timfn-hg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:56:20Z
user: login: petreze
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T18:26:11Z

Review comments:
diff_hunk: @@ -235,6 +236,7 @@ public void handleRound(
             @NonNull final Round round,
             @NonNull final Consumer<ScopedSystemTransaction<StateSignatureTransaction>> stateSignatureTxnCallback) {
         logStartRound(round);
+        BlockStreamStateManager.ensureNewBlocksPermitted();
path: hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/HandleWorkflow.java
user: login: timfn-hg
body: I'm not sure if this is remotely the correct place to apply the backpressure; still need to do some additional testing. Thoughts/suggestions are welcome regardless.

This is a very naive place to put the check, but it does prevent a limbo state where the blocking is performed later in the process and we have a partially open block. 
created_at: 2025-05-01T19:37:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,6 +236,7 @@ public void handleRound(
             @NonNull final Round round,
             @NonNull final Consumer<ScopedSystemTransaction<StateSignatureTransaction>> stateSignatureTxnCallback) {
         logStartRound(round);
+        BlockStreamStateManager.ensureNewBlocksPermitted();
path: hedera-node/hedera-app/src/main/java/com/hedera/node/app/workflows/handle/HandleWorkflow.java
user: login: timfn-hg
body: I did confirm with the check in the current position, when the buffer becomes full the thread is blocked and the platform state goes from ACTIVE to CHECKING and no new blocks are created.
created_at: 2025-05-05T16:48:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -693,6 +693,15 @@ class SequentialTask extends AbstractTask {
 
         @Override
         protected boolean onExecute() {
+            try {
+                return doExecute();
+            } catch (final RuntimeException e) {
+                log.error("Error occurred while executing task", e);
path: hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java
user: login: tinker-michaelj
body: Were there exceptions silently killing task threads before this?
created_at: 2025-05-06T12:15:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,53 +317,247 @@ public void streamPreBlockProofItems(long blockNumber) {
                     "Prior to BlockProof, creating request from items in block {} size {}",
                     blockNumber,
                     blockState.items().size());
-            createRequestFromCurrentItems(blockState);
+            createRequestFromCurrentItems(blockState, true);
         }
     }
 
     /**
-     * Removes all completed block states older than expiry period with block numbers less than or equal to the given block number. This operation is
-     * thread-safe.
+     * Marks all blocks up to and including the specified block as being acknowledged.
      *
-     * @param blockNumber the block number
+     * @param blockNumber the block number to mark acknowledged up to and including
+     */
+    public void setLatestAcknowledgedBlock(final long blockNumber) {
+        final long highestBlock = highestAckedBlockNumber.updateAndGet(current -> Math.max(current, blockNumber));
+        blockStreamMetrics.setLatestAcknowledgedBlockNumber(highestBlock);
+    }
+
+    /**
+     * Gets the current block number.
+     *
+     * @return the current block number or -1 if no blocks have been opened yet
+     */
+    public long getBlockNumber() {
+        return blockNumber;
+    }
+
+    /**
+     * Ensures that there is enough capacity in the block buffer to permit a new block being created. If there is not
+     * enough capacity - i.e. the buffer is saturated - then this method will block until there is enough capacity.
+     */
+    public static void ensureNewBlocksPermitted() {
+        final CompletableFuture<Boolean> cf = backpressureCompletableFutureRef.get();
+        if (cf != null && !cf.isDone()) {
+            try {
+                logger.error("!!! Block buffer is saturated; blocking thread until buffer is no longer saturated");
+                final long startMs = System.currentTimeMillis();
+                final boolean bufferAvailable = cf.get();
+                final long durationMs = System.currentTimeMillis() - startMs;
+                logger.warn("Thread was blocked for {}ms waiting for block buffer to free space", durationMs);
+
+                if (!bufferAvailable) {
+                    logger.warn("Block buffer still not available to accept new blocks; reentering wait...");
+                    ensureNewBlocksPermitted();
+                }
+            } catch (final InterruptedException e) {
+                Thread.currentThread().interrupt();
+            } catch (final Exception e) {
+                logger.warn("Failed to wait for block buffer to be available", e);
+            }
+        }
+    }
+
+    /**
+     * Prunes the block buffer by removing blocks that have been acknowledged and exceeded the configured TTL. By doing
+     * this, we also inadvertently can know if buffer is "saturated" due to blocks not being acknowledged in a timely
+     * manner.
      */
-    public void removeBlockStatesUpTo(long blockNumber) {
-        // Use keySet().removeIf for atomic removal of multiple entries
-        final Instant fiveMinutesAgo = Instant.now().minus(expiryPeriod);
-        final var removed = blockStates.entrySet().removeIf(entry -> {
-            final long key = entry.getKey();
-            final BlockState state = entry.getValue();
-
-            // Check if the block number is less than or equal to the acknowledged block number
-            if (key <= blockNumber) {
-                // Check if the block state is complete and older than 5 minutes
-                if (state.isComplete()) {
-                    final Instant completionTime = state.getCompletionTime();
-                    // If completionTime is null (shouldn't happen if isComplete is true, but defensive check),
-                    // or if it's older than 5 minutes, mark for removal.
-                    return completionTime == null || completionTime.isBefore(fiveMinutesAgo);
+    private @NonNull PruneResult pruneBuffer() {
+        final Duration ttl = blockBufferTtl();
+        final Instant cutoffInstant = Instant.now().minus(ttl);
+        final Iterator<BlockState> it = blockBuffer.iterator();
+        final long highestBlockAcked = highestAckedBlockNumber.get();
+        /*
+        Calculate the ideal max buffer size. This is calculated as the block buffer TTL (e.g. 5 minutes) divided by the
+        block period (e.g. 2 seconds). This gives us an ideal number of blocks in the buffer.
+         */
+        final long idealMaxBufferSize = ttl.dividedBy(blockPeriod());
+        int numPruned = 0;
+        int numChecked = 0;
+        int numPendingAck = 0;
+        final AtomicReference<Instant> oldestUnackedTimestamp = new AtomicReference<>(Instant.MAX);
+
+        while (it.hasNext()) {
+            final BlockState block = it.next();
+            ++numChecked;
+
+            if (block.isComplete()) {
+                if (block.blockNumber() <= highestBlockAcked) {
+                    // this block is eligible for pruning if it is old enough
+                    if (block.completionTimestamp().isBefore(cutoffInstant)) {
+                        blockStatesById.remove(block.blockNumber());
+                        it.remove();
+                        ++numPruned;
+                    }
                 } else {
-                    // If the block state is not complete, it's not eligible for removal based on the new rule.
-                    // Keep blocks that are <= blockNumber but not yet complete.
-                    return false;
+                    ++numPendingAck;
+                    oldestUnackedTimestamp.updateAndGet(current ->
+                            current.compareTo(block.completionTimestamp()) < 0 ? current : block.completionTimestamp());
                 }
-            } else {
-                // If the block number is greater than the acknowledged block number, don't remove.
-                return false;
             }
-        });
+        }
+
+        final long oldestUnackedMillis = Instant.MAX.equals(oldestUnackedTimestamp.get())
+                ? -1 // sentinel value indicating no blocks are unacked
+                : oldestUnackedTimestamp.get().toEpochMilli();
+        blockStreamMetrics.setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+
+        return new PruneResult(idealMaxBufferSize, numChecked, numPendingAck, numPruned);
+    }
+
+    /*
+    Simple record that contains information related to the outcome of a block buffer prune operation.
+     */
+    private record PruneResult(
+            long idealMaxBufferSize, int numBlocksChecked, int numBlocksPendingAck, int numBlocksPruned) {
+
+        /**
+         * Calculate the saturation percent based on the size of the buffer and the number of unacked blocks found.
+         * @return the saturation percent
+         */
+        double calculateSaturationPercent() {
+            if (idealMaxBufferSize == 0) {
+                return 0D;
+            }
+
+            final BigDecimal size = BigDecimal.valueOf(idealMaxBufferSize);
+            final BigDecimal pending = BigDecimal.valueOf(numBlocksPendingAck);
+            return pending.divide(size, 6, RoundingMode.HALF_EVEN)
+                    .multiply(BigDecimal.valueOf(100))
+                    .doubleValue();
+        }
 
-        if (removed) {
-            logger.debug("Removed completed block states up to block {} that were older than 5 minutes.", blockNumber);
+        /**
+         * Check if the buffer is considered saturated.
+         *
+         * @return true if the block buffer is considered saturated, else false
+         */
+        boolean isSaturated() {
+            return idealMaxBufferSize != 0 && numBlocksPendingAck >= idealMaxBufferSize;
         }
     }
 
     /**
-     * Gets the current block number.
-     *
-     * @return the current block number
+     * Prunes the block buffer and checks if the buffer is saturated. If the buffer is saturated, then a backpressure
+     * mechanism is activated. The backpressure will be enabled until the next time this method is invoked, after which
+     * the backpressure mechanism will be disabled if the buffer is no longer saturated, or maintained if the buffer
+     * continues to be saturated.
      */
-    public long getBlockNumber() {
-        return blockNumber;
+    private void checkBuffer() {
+        final boolean isSaturatedBeforePrune = isBufferSaturated.get();
+        final BlockStreamStateManager.PruneResult result = pruneBuffer();
+        final boolean isSaturatedAfterPrune = result.isSaturated();
+        isBufferSaturated.set(isSaturatedAfterPrune);
+        final double saturationPercent = result.calculateSaturationPercent();
+
+        logger.debug(
+                "Block buffer status: idealMaxBufferSize={}, blocksChecked={}, blocksPruned={}, blocksPendingAck={}, saturation={}%",
+                result.idealMaxBufferSize,
+                result.numBlocksChecked,
+                result.numBlocksPruned,
+                result.numBlocksPendingAck,
+                saturationPercent);
+
+        blockStreamMetrics.updateBlockBufferSaturation(saturationPercent);
+
+        if (isSaturatedBeforePrune == isSaturatedAfterPrune) {
+            // no state change detected, escape early
+            return;
+        }
+
+        if (isSaturatedAfterPrune) {
+            // we've transitioned to a saturated state, apply backpressure
+            logger.warn(
+                    "Block buffer is saturated; backpressure is being enabled "
+                            + "(idealMaxBufferSize={}, blocksChecked={}, blocksPruned={}, blocksPendingAck={}, saturation={}%)",
+                    result.idealMaxBufferSize,
+                    result.numBlocksChecked,
+                    result.numBlocksPruned,
+                    result.numBlocksPendingAck,
+                    saturationPercent);
+
+            CompletableFuture<Boolean> oldCf;
+            CompletableFuture<Boolean> newCf;
+            do {
+                oldCf = backpressureCompletableFutureRef.get();
+                if (oldCf != null) {
+                    /**
+                     * If everything is behaving as expected, then this condition should never be encountered. At any
path: hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManager.java
user: login: tinker-michaelj
body: Imo we should have at least a `WARN` log event if the system recovers from an unexpected state.
created_at: 2025-05-06T12:22:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -693,6 +693,15 @@ class SequentialTask extends AbstractTask {
 
         @Override
         protected boolean onExecute() {
+            try {
+                return doExecute();
+            } catch (final RuntimeException e) {
+                log.error("Error occurred while executing task", e);
path: hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/BlockStreamManagerImpl.java
user: login: timfn-hg
body: Yeah. Originally in `BlockStreamStateManager` the `blockNumber` was initialized to `0`. When the first block would be opened (block 0) an exception would be thrown in `BlockStreamStateManager#openBlock(long)` because it was equal to the initialized value and thus an error condition. This was silently being absorbed in the task and causing the state to go into CHECKING without any indication of why. By repeating this same scenario with the log statement, it is/was immediately clear what was wrong. (It took me over 2 days trying to figure out why it was silently failing before.)
created_at: 2025-05-06T13:20:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,53 +317,247 @@ public void streamPreBlockProofItems(long blockNumber) {
                     "Prior to BlockProof, creating request from items in block {} size {}",
                     blockNumber,
                     blockState.items().size());
-            createRequestFromCurrentItems(blockState);
+            createRequestFromCurrentItems(blockState, true);
         }
     }
 
     /**
-     * Removes all completed block states older than expiry period with block numbers less than or equal to the given block number. This operation is
-     * thread-safe.
+     * Marks all blocks up to and including the specified block as being acknowledged.
      *
-     * @param blockNumber the block number
+     * @param blockNumber the block number to mark acknowledged up to and including
+     */
+    public void setLatestAcknowledgedBlock(final long blockNumber) {
+        final long highestBlock = highestAckedBlockNumber.updateAndGet(current -> Math.max(current, blockNumber));
+        blockStreamMetrics.setLatestAcknowledgedBlockNumber(highestBlock);
+    }
+
+    /**
+     * Gets the current block number.
+     *
+     * @return the current block number or -1 if no blocks have been opened yet
+     */
+    public long getBlockNumber() {
+        return blockNumber;
+    }
+
+    /**
+     * Ensures that there is enough capacity in the block buffer to permit a new block being created. If there is not
+     * enough capacity - i.e. the buffer is saturated - then this method will block until there is enough capacity.
+     */
+    public static void ensureNewBlocksPermitted() {
+        final CompletableFuture<Boolean> cf = backpressureCompletableFutureRef.get();
+        if (cf != null && !cf.isDone()) {
+            try {
+                logger.error("!!! Block buffer is saturated; blocking thread until buffer is no longer saturated");
+                final long startMs = System.currentTimeMillis();
+                final boolean bufferAvailable = cf.get();
+                final long durationMs = System.currentTimeMillis() - startMs;
+                logger.warn("Thread was blocked for {}ms waiting for block buffer to free space", durationMs);
+
+                if (!bufferAvailable) {
+                    logger.warn("Block buffer still not available to accept new blocks; reentering wait...");
+                    ensureNewBlocksPermitted();
+                }
+            } catch (final InterruptedException e) {
+                Thread.currentThread().interrupt();
+            } catch (final Exception e) {
+                logger.warn("Failed to wait for block buffer to be available", e);
+            }
+        }
+    }
+
+    /**
+     * Prunes the block buffer by removing blocks that have been acknowledged and exceeded the configured TTL. By doing
+     * this, we also inadvertently can know if buffer is "saturated" due to blocks not being acknowledged in a timely
+     * manner.
      */
-    public void removeBlockStatesUpTo(long blockNumber) {
-        // Use keySet().removeIf for atomic removal of multiple entries
-        final Instant fiveMinutesAgo = Instant.now().minus(expiryPeriod);
-        final var removed = blockStates.entrySet().removeIf(entry -> {
-            final long key = entry.getKey();
-            final BlockState state = entry.getValue();
-
-            // Check if the block number is less than or equal to the acknowledged block number
-            if (key <= blockNumber) {
-                // Check if the block state is complete and older than 5 minutes
-                if (state.isComplete()) {
-                    final Instant completionTime = state.getCompletionTime();
-                    // If completionTime is null (shouldn't happen if isComplete is true, but defensive check),
-                    // or if it's older than 5 minutes, mark for removal.
-                    return completionTime == null || completionTime.isBefore(fiveMinutesAgo);
+    private @NonNull PruneResult pruneBuffer() {
+        final Duration ttl = blockBufferTtl();
+        final Instant cutoffInstant = Instant.now().minus(ttl);
+        final Iterator<BlockState> it = blockBuffer.iterator();
+        final long highestBlockAcked = highestAckedBlockNumber.get();
+        /*
+        Calculate the ideal max buffer size. This is calculated as the block buffer TTL (e.g. 5 minutes) divided by the
+        block period (e.g. 2 seconds). This gives us an ideal number of blocks in the buffer.
+         */
+        final long idealMaxBufferSize = ttl.dividedBy(blockPeriod());
+        int numPruned = 0;
+        int numChecked = 0;
+        int numPendingAck = 0;
+        final AtomicReference<Instant> oldestUnackedTimestamp = new AtomicReference<>(Instant.MAX);
+
+        while (it.hasNext()) {
+            final BlockState block = it.next();
+            ++numChecked;
+
+            if (block.isComplete()) {
+                if (block.blockNumber() <= highestBlockAcked) {
+                    // this block is eligible for pruning if it is old enough
+                    if (block.completionTimestamp().isBefore(cutoffInstant)) {
+                        blockStatesById.remove(block.blockNumber());
+                        it.remove();
+                        ++numPruned;
+                    }
                 } else {
-                    // If the block state is not complete, it's not eligible for removal based on the new rule.
-                    // Keep blocks that are <= blockNumber but not yet complete.
-                    return false;
+                    ++numPendingAck;
+                    oldestUnackedTimestamp.updateAndGet(current ->
+                            current.compareTo(block.completionTimestamp()) < 0 ? current : block.completionTimestamp());
                 }
-            } else {
-                // If the block number is greater than the acknowledged block number, don't remove.
-                return false;
             }
-        });
+        }
+
+        final long oldestUnackedMillis = Instant.MAX.equals(oldestUnackedTimestamp.get())
+                ? -1 // sentinel value indicating no blocks are unacked
+                : oldestUnackedTimestamp.get().toEpochMilli();
+        blockStreamMetrics.setOldestUnacknowledgedBlockTime(oldestUnackedMillis);
+
+        return new PruneResult(idealMaxBufferSize, numChecked, numPendingAck, numPruned);
+    }
+
+    /*
+    Simple record that contains information related to the outcome of a block buffer prune operation.
+     */
+    private record PruneResult(
+            long idealMaxBufferSize, int numBlocksChecked, int numBlocksPendingAck, int numBlocksPruned) {
+
+        /**
+         * Calculate the saturation percent based on the size of the buffer and the number of unacked blocks found.
+         * @return the saturation percent
+         */
+        double calculateSaturationPercent() {
+            if (idealMaxBufferSize == 0) {
+                return 0D;
+            }
+
+            final BigDecimal size = BigDecimal.valueOf(idealMaxBufferSize);
+            final BigDecimal pending = BigDecimal.valueOf(numBlocksPendingAck);
+            return pending.divide(size, 6, RoundingMode.HALF_EVEN)
+                    .multiply(BigDecimal.valueOf(100))
+                    .doubleValue();
+        }
 
-        if (removed) {
-            logger.debug("Removed completed block states up to block {} that were older than 5 minutes.", blockNumber);
+        /**
+         * Check if the buffer is considered saturated.
+         *
+         * @return true if the block buffer is considered saturated, else false
+         */
+        boolean isSaturated() {
+            return idealMaxBufferSize != 0 && numBlocksPendingAck >= idealMaxBufferSize;
         }
     }
 
     /**
-     * Gets the current block number.
-     *
-     * @return the current block number
+     * Prunes the block buffer and checks if the buffer is saturated. If the buffer is saturated, then a backpressure
+     * mechanism is activated. The backpressure will be enabled until the next time this method is invoked, after which
+     * the backpressure mechanism will be disabled if the buffer is no longer saturated, or maintained if the buffer
+     * continues to be saturated.
      */
-    public long getBlockNumber() {
-        return blockNumber;
+    private void checkBuffer() {
+        final boolean isSaturatedBeforePrune = isBufferSaturated.get();
+        final BlockStreamStateManager.PruneResult result = pruneBuffer();
+        final boolean isSaturatedAfterPrune = result.isSaturated();
+        isBufferSaturated.set(isSaturatedAfterPrune);
+        final double saturationPercent = result.calculateSaturationPercent();
+
+        logger.debug(
+                "Block buffer status: idealMaxBufferSize={}, blocksChecked={}, blocksPruned={}, blocksPendingAck={}, saturation={}%",
+                result.idealMaxBufferSize,
+                result.numBlocksChecked,
+                result.numBlocksPruned,
+                result.numBlocksPendingAck,
+                saturationPercent);
+
+        blockStreamMetrics.updateBlockBufferSaturation(saturationPercent);
+
+        if (isSaturatedBeforePrune == isSaturatedAfterPrune) {
+            // no state change detected, escape early
+            return;
+        }
+
+        if (isSaturatedAfterPrune) {
+            // we've transitioned to a saturated state, apply backpressure
+            logger.warn(
+                    "Block buffer is saturated; backpressure is being enabled "
+                            + "(idealMaxBufferSize={}, blocksChecked={}, blocksPruned={}, blocksPendingAck={}, saturation={}%)",
+                    result.idealMaxBufferSize,
+                    result.numBlocksChecked,
+                    result.numBlocksPruned,
+                    result.numBlocksPendingAck,
+                    saturationPercent);
+
+            CompletableFuture<Boolean> oldCf;
+            CompletableFuture<Boolean> newCf;
+            do {
+                oldCf = backpressureCompletableFutureRef.get();
+                if (oldCf != null) {
+                    /**
+                     * If everything is behaving as expected, then this condition should never be encountered. At any
path: hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockStreamStateManager.java
user: login: timfn-hg
body: I've added a `WARN` message regarding this.
created_at: 2025-05-06T14:58:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -28,7 +31,9 @@ public record BlockStreamConfig(
         @ConfigProperty(defaultValue = "32") @NetworkProperty int hashCombineBatchSize,
         @ConfigProperty(defaultValue = "1") @NetworkProperty int roundsPerBlock,
         @ConfigProperty(defaultValue = "2s") @Min(0) @NetworkProperty Duration blockPeriod,
-        @ConfigProperty(defaultValue = "256") @Min(0) @NetworkProperty int blockItemBatchSize) {
+        @ConfigProperty(defaultValue = "256") @Min(0) @NetworkProperty int blockItemBatchSize,
+        @ConfigProperty(defaultValue = "5m") @Min(0) @NetworkProperty Duration blockBufferTtl,
+        @ConfigProperty(defaultValue = "1s") @Min(0) @NetworkProperty Duration blockBufferPruneInterval) {
path: hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
user: login: Nana-EC
body: Q: Might this be too aggressive to start with?
If the buffer itself is populated ever 2 seconds because on block production then acknowledgment and pruning on success is likely around that cadence also.
If prune checking is expensive you could consider setting ti to match the block production time initially.
Of course faster is helpful in clearing out a backed up buffer.
Something to note
created_at: 2025-05-06T20:05:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -28,7 +31,9 @@ public record BlockStreamConfig(
         @ConfigProperty(defaultValue = "32") @NetworkProperty int hashCombineBatchSize,
         @ConfigProperty(defaultValue = "1") @NetworkProperty int roundsPerBlock,
         @ConfigProperty(defaultValue = "2s") @Min(0) @NetworkProperty Duration blockPeriod,
-        @ConfigProperty(defaultValue = "256") @Min(0) @NetworkProperty int blockItemBatchSize) {
+        @ConfigProperty(defaultValue = "256") @Min(0) @NetworkProperty int blockItemBatchSize,
+        @ConfigProperty(defaultValue = "5m") @Min(0) @NetworkProperty Duration blockBufferTtl,
+        @ConfigProperty(defaultValue = "1s") @Min(0) @NetworkProperty Duration blockBufferPruneInterval) {
path: hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
user: login: timfn-hg
body: Pruning/checking the buffer state is fairly lightweight and the aggressive nature of it is on purpose, because like you say, we want to be quick in both applying backpressure when BNs are unhealthy and we want to be quick in removing backpressure because the longer we have backpressure the longer the CN is rejecting transactions.
created_at: 2025-05-07T15:56:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: Nana-EC
created_at: 2025-05-06T19:56:01Z
body: > If a block is not acknowledged but expired (that is they are older than the TTL threshold), it will not be pruned. If there are blocks in this state, the buffer is considered to be "saturated". This state is not normal and usually indicates a problem with the block node receiving or processing blocks. In this state we want to apply some form of backpressure to prevent more blocks (and transactions) from being processed.

Does this assume a CN has exhausted all available BN's in its list?
If not prior to back pressure a CN can explore other blocks nodes to feed the Block to.
@jsync-swirlds thoughts?
author_association: CONTRIBUTOR
user: login: jsync-swirlds
created_at: 2025-05-06T20:05:51Z
body: > Does this assume a CN has exhausted all available BN's in its list? If not prior to back pressure a CN can explore other blocks nodes to feed the Block to. @jsync-swirlds thoughts?

That might be a subsequent improvement (or might already be implemented elsewhere, not sure), but the assumption has been that _well_ before the buffer fills, unacknowledged blocks will cause the `BlockManager` to send blocks to a different block node (if available) on the assumption that the current block node is likely unhealthy and the CN is better off sending blocks to a different recipient.
Generally, if a block node is more than around 20-30 blocks behind in acknowledgement, something is probably off-nominal with that block node.

That also means that (in most cases) multiple block nodes would need to be unhealthy before any CN might have to enter a CHECKING state due to back-pressure.
author_association: CONTRIBUTOR
user: login: timfn-hg
created_at: 2025-05-07T15:53:56Z
body: @Nana-EC , @jsync-swirlds: Correct in that we plan on moving to a different block node when the buffer becomes saturated to some threshold. This work is not included in this PR but will be in a follow up PR once we think a little more on it.
author_association: CONTRIBUTOR