Repository: apache/flink-connector-aws
Pull request #190: [FLINK-36203][Connectors/DynamoDB] Added polling delay between getrecords calls
Author: login: gguptp
Created: 2025-02-19T04:37:48Z | Merged: 2025-03-07T11:15:45Z


<!--
*Thank you for contributing to Apache Flink AWS Connectors - we are happy that you want to help us improve our Flink connectors. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*

## Contribution Checklist

- The name of the pull request should correspond to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Commits should be in the form of "[FLINK-XXXX][component] Title of the pull request", where [FLINK-XXXX] should be replaced by the actual issue number. 
    Generally, [component] should be the connector you are working on.
    For example: "[FLINK-XXXX][Connectors/Kinesis] XXXX" if you are working on the Kinesis connector or "[FLINK-XXXX][Connectors/AWS] XXXX" if you are working on components shared among all the connectors.
- Each pull request should only have one JIRA issue.
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
-->

## Purpose of the change

DDB Streams customers are charged on number of getrecords calls and given the connector maintains a queue of shards and if a task manager has only a few shards in the queue, it can cause a shard being called (1000/(x*y)) times in a second where x = number of shards in the queue, y = average getrecords latency. 

We should ideally be having some delay in case the last poll was an empty poll so that the customer gets charged less 

## Verifying this change

Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing

*(Please pick either of the following options)*


This change added tests and can be verified as follows:

- *Added unit tests*
- *Manually verified by running the Kinesis connector on a local Flink cluster.*

## Significant changes
*(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)*
- [ ] Dependencies have been added or upgraded
- [ ] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`)
- [ ] Serializers have been changed
- [ x ] New feature has been introduced
  - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)



Diff:
diff --git a/flink-connector-aws/flink-connector-dynamodb/pom.xml b/flink-connector-aws/flink-connector-dynamodb/pom.xml
index b67e2831..3cbb7e08 100644
--- a/flink-connector-aws/flink-connector-dynamodb/pom.xml
+++ b/flink-connector-aws/flink-connector-dynamodb/pom.xml
@@ -151,6 +151,12 @@ under the License.
             <version>3.14.1</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.awaitility</groupId>
+            <artifactId>awaitility</artifactId>
+            <version>4.3.0</version>
+            <scope>test</scope>
+        </dependency>
 
     </dependencies>
 </project>
diff --git a/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/DynamoDbStreamsSource.java b/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/DynamoDbStreamsSource.java
index 14f94e7b..5b20bc13 100644
--- a/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/DynamoDbStreamsSource.java
+++ b/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/DynamoDbStreamsSource.java
@@ -66,6 +66,8 @@
 
 import static org.apache.flink.connector.dynamodb.source.config.DynamodbStreamsSourceConfigConstants.DYNAMODB_STREAMS_EXPONENTIAL_BACKOFF_MAX_DELAY;
 import static org.apache.flink.connector.dynamodb.source.config.DynamodbStreamsSourceConfigConstants.DYNAMODB_STREAMS_EXPONENTIAL_BACKOFF_MIN_DELAY;
+import static org.apache.flink.connector.dynamodb.source.config.DynamodbStreamsSourceConfigConstants.DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS;
+import static org.apache.flink.connector.dynamodb.source.config.DynamodbStreamsSourceConfigConstants.DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS;
 import static org.apache.flink.connector.dynamodb.source.config.DynamodbStreamsSourceConfigConstants.DYNAMODB_STREAMS_RETRY_COUNT;
 
 /**
@@ -133,13 +135,25 @@ public SourceReader<T, DynamoDbStreamsShardSplit> createReader(
         setUpDeserializationSchema(readerContext);
 
         Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap = new ConcurrentHashMap<>();
+        // Delay between calling GetRecords API when last poll returned some records. Default - 250
+        // millis
+        final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS);
+
+        // Delay between calling GetRecords API when last poll returned no records. Default - 1
+        // second
+        final Duration getRecordsIdlePollingTimeBetweenEmptyPolls =
+                sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS);
 
         // We create a new stream proxy for each split reader since they have their own independent
         // lifecycle.
         Supplier<PollingDynamoDbStreamsShardSplitReader> splitReaderSupplier =
                 () ->
                         new PollingDynamoDbStreamsShardSplitReader(
-                                createDynamoDbStreamsProxy(sourceConfig), shardMetricGroupMap);
+                                createDynamoDbStreamsProxy(sourceConfig),
+                                getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+                                getRecordsIdlePollingTimeBetweenEmptyPolls,
+                                shardMetricGroupMap);
         DynamoDbStreamsRecordEmitter<T> recordEmitter =
                 new DynamoDbStreamsRecordEmitter<>(deserializationSchema);
 
diff --git a/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java b/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
index 290a9758..b16833e5 100644
--- a/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
+++ b/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
@@ -76,6 +76,21 @@ public enum InitialPosition {
     public static final String BASE_DDB_STREAMS_USER_AGENT_PREFIX_FORMAT =
             "Apache Flink %s (%s) DynamoDb Streams Connector";
 
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.empty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(1000))
+                            .withDescription(
+                                    "The idle time between empty polls for DynamoDB Streams GetRecords API");
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.nonempty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(250))
+                            .withDescription(
+                                    "The default idle time between non-empty polls for DynamoDB Streams GetRecords API");
+
     /** DynamoDb Streams identifier for user agent prefix. */
     public static final String DDB_STREAMS_CLIENT_USER_AGENT_PREFIX =
             "aws.dynamodbstreams.client.user-agent-prefix";
diff --git a/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java b/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
index aa5181a3..8a516bed 100644
--- a/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
+++ b/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
@@ -28,12 +28,15 @@
 import org.apache.flink.connector.dynamodb.source.split.DynamoDbStreamsShardSplitState;
 import org.apache.flink.connector.dynamodb.source.split.StartingPosition;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 import software.amazon.awssdk.services.dynamodb.model.GetRecordsResponse;
 import software.amazon.awssdk.services.dynamodb.model.Record;
 
 import javax.annotation.Nullable;
 
 import java.io.IOException;
+import java.time.Duration;
 import java.util.ArrayDeque;
 import java.util.Collection;
 import java.util.Collections;
@@ -57,48 +60,90 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final Deque<DynamoDbStreamsShardSplitWithContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
-    private final Set<String> pausedSplitIds = new HashSet<>();
+    private final Set<String> pausedSplitIds;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(PollingDynamoDbStreamsShardSplitReader.class);
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits = new ArrayDeque<>();
+        this.pausedSplitIds = new HashSet<>();
+    }
+
+    private long getNextEligibleTime(DynamoDbStreamsShardSplitWithContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+        DynamoDbStreamsShardSplitWithContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        long currentTime = System.currentTimeMillis();
+        long nextEligibleTime = getNextEligibleTime(splitContext);
+
+        LOG.debug(
+                "Polling split: {}, currentTime: {}, eligibleTime: {}, wasEmptyPoll: {}",
+                splitContext.splitState.getSplitId(),
+                currentTime,
+                nextEligibleTime,
+                splitContext.wasLastPollEmpty);
+
+        // Check if split is not ready due to empty poll and non-empty poll delay
+        if (nextEligibleTime > currentTime) {
+            assignedSplits.add(splitContext);
+            sleep(1);
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
 
         GetRecordsResponse getRecordsResponse =
                 dynamodbStreams.getRecords(
-                        splitState.getStreamArn(),
-                        splitState.getShardId(),
-                        splitState.getNextStartingPosition());
+                        splitContext.splitState.getStreamArn(),
+                        splitContext.splitState.getShardId(),
+                        splitContext.splitState.getNextStartingPosition());
         boolean isComplete = getRecordsResponse.nextShardIterator() == null;
+        boolean isEmptyPoll = hasNoRecords(getRecordsResponse);
 
-        if (hasNoRecords(getRecordsResponse)) {
+        splitContext.lastPollTimeMillis = currentTime;
+        splitContext.wasLastPollEmpty = isEmptyPoll;
+
+        if (isEmptyPoll) {
             if (isComplete) {
                 return new DynamoDbStreamRecordsWithSplitIds(
-                        Collections.emptyIterator(), splitState.getSplitId(), true);
+                        Collections.emptyIterator(), splitContext.splitState.getSplitId(), true);
             } else {
-                assignedSplits.add(splitState);
+                assignedSplits.add(splitContext);
                 return INCOMPLETE_SHARD_EMPTY_RECORDS;
             }
         } else {
             DynamoDbStreamsShardMetrics shardMetrics =
-                    shardMetricGroupMap.get(splitState.getShardId());
+                    shardMetricGroupMap.get(splitContext.splitState.getShardId());
             Record lastRecord =
                     getRecordsResponse.records().get(getRecordsResponse.records().size() - 1);
             shardMetrics.setMillisBehindLatest(
@@ -111,7 +156,7 @@ public RecordsWithSplitIds<Record> fetch() throws IOException {
                             0));
         }
 
-        splitState.setNextStartingPosition(
+        splitContext.splitState.setNextStartingPosition(
                 StartingPosition.continueFromSequenceNumber(
                         getRecordsResponse
                                 .records()
@@ -120,10 +165,20 @@ public RecordsWithSplitIds<Record> fetch() throws IOException {
                                 .sequenceNumber()));
 
         if (!isComplete) {
-            assignedSplits.add(splitState);
+            assignedSplits.add(splitContext);
         }
         return new DynamoDbStreamRecordsWithSplitIds(
-                getRecordsResponse.records().iterator(), splitState.getSplitId(), isComplete);
+                getRecordsResponse.records().iterator(),
+                splitContext.splitState.getSplitId(),
+                isComplete);
+    }
+
+    private void sleep(long milliseconds) throws IOException {
+        try {
+            Thread.sleep(milliseconds);
+        } catch (InterruptedException e) {
+            throw new IOException("Split reader was interrupted while sleeping", e);
+        }
     }
 
     private boolean hasNoRecords(GetRecordsResponse getRecordsResponse) {
@@ -133,7 +188,9 @@ private boolean hasNoRecords(GetRecordsResponse getRecordsResponse) {
     @Override
     public void handleSplitsChanges(SplitsChange<DynamoDbStreamsShardSplit> splitsChanges) {
         for (DynamoDbStreamsShardSplit split : splitsChanges.splits()) {
-            assignedSplits.add(new DynamoDbStreamsShardSplitState(split));
+            assignedSplits.add(
+                    new DynamoDbStreamsShardSplitWithContext(
+                            new DynamoDbStreamsShardSplitState(split)));
         }
     }
 
@@ -191,4 +248,17 @@ public Set<String> finishedSplits() {
             return isComplete ? singleton(splitId) : Collections.emptySet();
         }
     }
+
+    @Internal
+    private static class DynamoDbStreamsShardSplitWithContext {
+        final DynamoDbStreamsShardSplitState splitState;
+        long lastPollTimeMillis;
+        boolean wasLastPollEmpty;
+
+        DynamoDbStreamsShardSplitWithContext(DynamoDbStreamsShardSplitState splitState) {
+            this.splitState = splitState;
+            this.lastPollTimeMillis = System.currentTimeMillis();
+            this.wasLastPollEmpty = false;
+        }
+    }
 }
diff --git a/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/DynamoDbStreamsSourceReaderTest.java b/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/DynamoDbStreamsSourceReaderTest.java
index d9a33cd5..e8cf786f 100644
--- a/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/DynamoDbStreamsSourceReaderTest.java
+++ b/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/DynamoDbStreamsSourceReaderTest.java
@@ -35,6 +35,7 @@
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 
+import java.time.Duration;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
@@ -53,6 +54,8 @@ class DynamoDbStreamsSourceReaderTest {
     private DynamoDbStreamsSourceReader<TestData> sourceReader;
     private MetricListener metricListener;
     private Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
+    private static final Duration NON_EMPTY_POLLING_DELAY_MILLIS = Duration.ofMillis(250);
+    private static final Duration EMPTY_POLLING_DELAY_MILLIS = Duration.ofMillis(1000);
 
     @BeforeEach
     public void init() {
@@ -62,7 +65,10 @@ public void init() {
         Supplier<PollingDynamoDbStreamsShardSplitReader> splitReaderSupplier =
                 () ->
                         new PollingDynamoDbStreamsShardSplitReader(
-                                testStreamProxy, shardMetricGroupMap);
+                                testStreamProxy,
+                                NON_EMPTY_POLLING_DELAY_MILLIS,
+                                EMPTY_POLLING_DELAY_MILLIS,
+                                shardMetricGroupMap);
 
         testingReaderContext =
                 DynamoDbStreamsContextProvider.DynamoDbStreamsTestingContext
diff --git a/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java b/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
index 0c8c3099..f3cc1692 100644
--- a/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
+++ b/flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
@@ -30,6 +30,7 @@
 import org.junit.jupiter.api.Test;
 import software.amazon.awssdk.services.dynamodb.model.Record;
 
+import java.time.Duration;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
@@ -45,6 +46,7 @@
 import static org.apache.flink.connector.dynamodb.source.util.TestUtil.getTestSplit;
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.assertj.core.api.AssertionsForClassTypes.assertThatNoException;
+import static org.awaitility.Awaitility.await;
 
 class PollingDynamoDbStreamsShardSplitReaderTest {
     private PollingDynamoDbStreamsShardSplitReader splitReader;
@@ -53,6 +55,9 @@ class PollingDynamoDbStreamsShardSplitReaderTest {
     private Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
     private static final String TEST_SHARD_ID = TestUtil.generateShardId(1);
 
+    private static final Duration NON_EMPTY_POLLING_DELAY_MILLIS = Duration.ofMillis(250);
+    private static final Duration EMPTY_POLLING_DELAY_MILLIS = Duration.ofMillis(1000);
+
     @BeforeEach
     public void init() {
         testStreamProxy = getTestStreamProxy();
@@ -64,7 +69,11 @@ public void init() {
                 new DynamoDbStreamsShardMetrics(
                         TestUtil.getTestSplit(TEST_SHARD_ID), metricListener.getMetricGroup()));
         splitReader =
-                new PollingDynamoDbStreamsShardSplitReader(testStreamProxy, shardMetricGroupMap);
+                new PollingDynamoDbStreamsShardSplitReader(
+                        testStreamProxy,
+                        NON_EMPTY_POLLING_DELAY_MILLIS,
+                        EMPTY_POLLING_DELAY_MILLIS,
+                        shardMetricGroupMap);
     }
 
     @Test
@@ -117,8 +126,15 @@ void testSingleAssignedSplitAllConsumed() throws Exception {
         // When fetching records
         List<Record> records = new ArrayList<>();
         for (int i = 0; i < expectedRecords.size(); i++) {
-            RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
-            records.addAll(readAllRecords(retrievedRecords));
+            // Wait for non-empty poll delay after getting records
+            await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                    .untilAsserted(
+                            () -> {
+                                RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
+                                List<Record> polledRecords = readAllRecords(retrievedRecords);
+                                assertThat(polledRecords).isNotEmpty();
+                                records.addAll(polledRecords);
+                            });
         }
 
         assertThat(records).containsExactlyInAnyOrderElementsOf(expectedRecords);
@@ -155,8 +171,14 @@ void testMultipleAssignedSplitsAllConsumed() throws Exception {
         // When records are fetched
         List<Record> fetchedRecords = new ArrayList<>();
         for (int i = 0; i < expectedRecords.size(); i++) {
-            RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
-            fetchedRecords.addAll(readAllRecords(retrievedRecords));
+            await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS) // Wait first
+                    .untilAsserted(
+                            () -> {
+                                RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
+                                List<Record> polledRecords = readAllRecords(retrievedRecords);
+                                assertThat(polledRecords).isNotEmpty();
+                                fetchedRecords.addAll(polledRecords);
+                            });
         }
 
         // Then all records are fetched
@@ -173,13 +195,17 @@ void testHandleEmptyCompletedShard() throws Exception {
         splitReader.handleSplitsChanges(new SplitsAddition<>(Collections.singletonList(split)));
         testStreamProxy.setShouldCompleteNextShard(true);
 
-        // When fetching records
-        RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
-
-        // Returns completed split with no records
-        assertThat(retrievedRecords.nextRecordFromSplit()).isNull();
-        assertThat(retrievedRecords.nextSplit()).isNull();
-        assertThat(retrievedRecords.finishedSplits()).contains(split.splitId());
+        await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                .untilAsserted(
+                        () -> {
+                            // When fetching records
+                            RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
+
+                            // Returns completed split with no records
+                            assertThat(retrievedRecords.nextRecordFromSplit()).isNull();
+                            assertThat(retrievedRecords.nextSplit()).isNull();
+                            assertThat(retrievedRecords.finishedSplits()).contains(split.splitId());
+                        });
     }
 
     @Test
@@ -197,18 +223,24 @@ void testFinishedSplitsReturned() throws Exception {
         // When fetching records
         List<Record> fetchedRecords = new ArrayList<>();
         testStreamProxy.setShouldCompleteNextShard(true);
-        RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
-
-        // Then records can be read successfully, with finishedSplit returned once all records are
-        // completed
-        for (int i = 0; i < expectedRecords.size(); i++) {
-            assertThat(retrievedRecords.nextSplit()).isEqualTo(split.splitId());
-            assertThat(retrievedRecords.finishedSplits()).isEmpty();
-            fetchedRecords.add(retrievedRecords.nextRecordFromSplit());
-        }
-        assertThat(retrievedRecords.nextSplit()).isNull();
-        assertThat(retrievedRecords.finishedSplits()).contains(split.splitId());
-        assertThat(fetchedRecords).containsExactlyInAnyOrderElementsOf(expectedRecords);
+        await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                .untilAsserted(
+                        () -> {
+                            RecordsWithSplitIds<Record> retrievedRecords = splitReader.fetch();
+
+                            // Then records can be read successfully, with finishedSplit returned
+                            // once all records are completed
+                            for (int i = 0; i < expectedRecords.size(); i++) {
+                                assertThat(retrievedRecords.nextSplit()).isEqualTo(split.splitId());
+                                assertThat(retrievedRecords.finishedSplits()).isEmpty();
+                                fetchedRecords.add(retrievedRecords.nextRecordFromSplit());
+                            }
+
+                            assertThat(retrievedRecords.nextSplit()).isNull();
+                            assertThat(retrievedRecords.finishedSplits()).contains(split.splitId());
+                            assertThat(fetchedRecords)
+                                    .containsExactlyInAnyOrderElementsOf(expectedRecords);
+                        });
     }
 
     @Test
@@ -233,13 +265,18 @@ void testPauseOrResumeSplits() throws Exception {
         splitReader.handleSplitsChanges(new SplitsAddition<>(Collections.singletonList(testSplit)));
 
         // read data from split
-        RecordsWithSplitIds<Record> records = splitReader.fetch();
-        assertThat(readAllRecords(records)).containsExactlyInAnyOrder(expectedRecords.get(0));
+        await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                .untilAsserted(
+                        () -> {
+                            RecordsWithSplitIds<Record> records = splitReader.fetch();
+                            assertThat(readAllRecords(records))
+                                    .containsExactlyInAnyOrder(expectedRecords.get(0));
+                        });
 
         // pause split
         splitReader.pauseOrResumeSplits(
                 Collections.singletonList(testSplit), Collections.emptyList());
-        records = splitReader.fetch();
+        RecordsWithSplitIds<Record> records = splitReader.fetch();
         // returns incomplete split with no records
         assertThat(records.finishedSplits()).isEmpty();
         assertThat(records.nextSplit()).isNull();
@@ -248,8 +285,13 @@ void testPauseOrResumeSplits() throws Exception {
         // resume split
         splitReader.pauseOrResumeSplits(
                 Collections.emptyList(), Collections.singletonList(testSplit));
-        records = splitReader.fetch();
-        assertThat(readAllRecords(records)).containsExactlyInAnyOrder(expectedRecords.get(1));
+        await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                .untilAsserted(
+                        () -> {
+                            RecordsWithSplitIds<Record> resumedRecords = splitReader.fetch();
+                            assertThat(readAllRecords(resumedRecords))
+                                    .containsExactlyInAnyOrder(expectedRecords.get(1));
+                        });
     }
 
     @Test
@@ -306,8 +348,15 @@ record ->
         // read data from splits and verify that only records from split 2 were fetched by reader
         List<Record> fetchedRecords = new ArrayList<>();
         for (int i = 0; i < 10; i++) {
-            RecordsWithSplitIds<Record> records = splitReader.fetch();
-            fetchedRecords.addAll(readAllRecords(records));
+            await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                    .untilAsserted(
+                            () -> {
+                                RecordsWithSplitIds<Record> records = splitReader.fetch();
+                                List<Record> polledRecords = readAllRecords(records);
+                                if (!polledRecords.isEmpty()) {
+                                    fetchedRecords.addAll(polledRecords);
+                                }
+                            });
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit2.toArray(new Record[0]));
 
@@ -318,12 +367,138 @@ record ->
         // read data from splits and verify that only records from split 3 had been read
         fetchedRecords.clear();
         for (int i = 0; i < 10; i++) {
-            RecordsWithSplitIds<Record> records = splitReader.fetch();
-            fetchedRecords.addAll(readAllRecords(records));
+            await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                    .untilAsserted(
+                            () -> {
+                                RecordsWithSplitIds<Record> records = splitReader.fetch();
+                                List<Record> polledRecords = readAllRecords(records);
+                                if (!polledRecords.isEmpty()) {
+                                    fetchedRecords.addAll(polledRecords);
+                                }
+                            });
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Use a very long delay for this test to ensure it's not flaky
+        Duration testEmptyPollDelay = Duration.ofMinutes(1);
+        splitReader =
+                new PollingDynamoDbStreamsShardSplitReader(
+                        testStreamProxy,
+                        NON_EMPTY_POLLING_DELAY_MILLIS,
+                        testEmptyPollDelay,
+                        shardMetricGroupMap);
+
+        // Immediate second poll - should return empty due to polling delay
+        RecordsWithSplitIds<Record> secondPoll = splitReader.fetch();
+        assertThat(secondPoll.nextRecordFromSplit()).isNull();
+        assertThat(secondPoll.nextSplit()).isNull();
+        assertThat(secondPoll.finishedSplits()).isEmpty();
+    }
+
+    @Test
+    void testLessPollingDelayForNonEmptyRecords() {
+        // Given assigned split with records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        Record record1 = getTestRecord("data-1");
+        Record record2 = getTestRecord("data-2");
+
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record1));
+
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return record1
+        List<Record> fetchedRecords = new ArrayList<>();
+        await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                .untilAsserted(
+                        () -> {
+                            RecordsWithSplitIds<Record> records = splitReader.fetch();
+                            List<Record> polledRecords = readAllRecords(records);
+                            assertThat(polledRecords).isNotEmpty();
+                            fetchedRecords.addAll(polledRecords);
+                        });
+
+        // Add second record
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record2));
+
+        // Immediate second poll - should return record2 despite no delay
+        fetchedRecords.clear();
+        await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                .atMost(Duration.ofSeconds(1))
+                .untilAsserted(
+                        () -> {
+                            RecordsWithSplitIds<Record> records = splitReader.fetch();
+                            List<Record> polledRecords = readAllRecords(records);
+                            assertThat(polledRecords).isNotEmpty();
+                            fetchedRecords.addAll(polledRecords);
+                        });
+        assertThat(fetchedRecords).containsExactly(record2);
+    }
+
+    @Test
+    void testPollingDelayRespectedAfterEmptyPoll() throws Exception {
+        // Given assigned split with records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - empty
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+
+        // Add a record but try to poll before empty poll delay expires
+        Record record = getTestRecord("data-1");
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record));
+
+        // Second poll - should still be empty due to empty poll delay
+        RecordsWithSplitIds<Record> secondPoll = splitReader.fetch();
+        assertThat(secondPoll.nextRecordFromSplit()).isNull();
+
+        // Wait for empty poll delay
+        // Third poll - should get the record
+        await().pollDelay(EMPTY_POLLING_DELAY_MILLIS)
+                .untilAsserted(
+                        () -> {
+                            RecordsWithSplitIds<Record> thirdPoll = splitReader.fetch();
+                            assertThat(readAllRecords(thirdPoll)).containsExactly(record);
+                        });
+
+        // Add new record but try to poll before non-empty poll delay expires
+        Record newRecord = getTestRecord("data-2");
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(newRecord));
+
+        // Fourth poll - should be empty due to non-empty poll delay
+        RecordsWithSplitIds<Record> fourthPoll = splitReader.fetch();
+        assertThat(fourthPoll.nextRecordFromSplit()).isNull();
+
+        // Wait for non-empty poll delay
+        // Fifth poll - should get the new record
+        await().pollDelay(NON_EMPTY_POLLING_DELAY_MILLIS)
+                .untilAsserted(
+                        () -> {
+                            RecordsWithSplitIds<Record> fifthPoll = splitReader.fetch();
+                            assertThat(readAllRecords(fifthPoll)).containsExactly(newRecord);
+                        });
+    }
+
     @Test
     void testCloseClosesStreamProxy() {
         assertThatNoException().isThrownBy(splitReader::close);


Reviews:
user: login: hlteoh37
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:05:56Z
user: login: hlteoh37
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:07:33Z
user: login: hlteoh37
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:07:56Z
user: login: hlteoh37
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:08:48Z
user: login: hlteoh37
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:09:07Z
user: login: hlteoh37
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:10:53Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:11:04Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:11:13Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T13:11:25Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T16:41:39Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T16:42:07Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T16:42:40Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T16:43:47Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:21:18Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:21:49Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:23:58Z
user: login: hlteoh37
body: Added some questions. Thanks @gguptp 

state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T14:19:59Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T14:22:36Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T14:23:07Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T14:24:33Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T14:25:17Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T14:25:59Z
user: login: hlteoh37
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T14:36:25Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T18:35:39Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T18:40:26Z
user: login: gguptp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T18:41:16Z
user: login: hlteoh37
body: Approved. Thanks @gguptp 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T15:08:26Z

Review comments:
diff_hunk: @@ -57,48 +59,101 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final PriorityQueue<SplitContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
     private final Set<String> pausedSplitIds = new HashSet<>();
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits =
+                new PriorityQueue<>(
+                        (a, b) -> {
+                            // First, handle paused splits
+                            boolean aIsPaused = pausedSplitIds.contains(a.splitState.getSplitId());
+                            boolean bIsPaused = pausedSplitIds.contains(b.splitState.getSplitId());
+                            if (aIsPaused && !bIsPaused) {
+                                return 1;
+                            }
+                            if (!aIsPaused && bIsPaused) {
+                                return -1;
+                            }
+                            if (aIsPaused && bIsPaused) {
+                                return 0;
+                            }
+
+                            // Get next eligible time for both splits
+                            long aNextEligibleTime = getNextEligibleTime(a);
+                            long bNextEligibleTime = getNextEligibleTime(b);
+
+                            return Long.compare(aNextEligibleTime, bNextEligibleTime);
+                        });
+    }
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: hlteoh37
body: This seems like it is introducing unnecessary complexity. Can we keep it simple first, and just keep the `Deque`? Alternatively, if we want to do this, let's put it behind a feature flag and add debugging logs.

I prefer just keeping it simple first!
created_at: 2025-02-27T13:05:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +59,101 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final PriorityQueue<SplitContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
     private final Set<String> pausedSplitIds = new HashSet<>();
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits =
+                new PriorityQueue<>(
+                        (a, b) -> {
+                            // First, handle paused splits
+                            boolean aIsPaused = pausedSplitIds.contains(a.splitState.getSplitId());
+                            boolean bIsPaused = pausedSplitIds.contains(b.splitState.getSplitId());
+                            if (aIsPaused && !bIsPaused) {
+                                return 1;
+                            }
+                            if (!aIsPaused && bIsPaused) {
+                                return -1;
+                            }
+                            if (aIsPaused && bIsPaused) {
+                                return 0;
+                            }
+
+                            // Get next eligible time for both splits
+                            long aNextEligibleTime = getNextEligibleTime(a);
+                            long bNextEligibleTime = getNextEligibleTime(b);
+
+                            return Long.compare(aNextEligibleTime, bNextEligibleTime);
+                        });
+    }
+
+    private long getNextEligibleTime(SplitContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
+        SplitContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+
+        // Check if split is paused or not ready due to empty poll delay
+        long currentTime = System.currentTimeMillis();
+        long nextEligibleTime = getNextEligibleTime(splitContext);
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        if (nextEligibleTime > currentTime) {
+            assignedSplits.add(splitContext);
+            sleep(20);
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: hlteoh37
body: We should not call `Thread.sleep()` in this method. This will cause pauses in the Flink runtime processing.

Can we instead store the "next read timestamp" in the split record, and the split back to the `Deque`? The next time we call this, we can check if the next read timestamp has arrived
created_at: 2025-02-27T13:07:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,10 +175,20 @@ public RecordsWithSplitIds<Record> fetch() throws IOException {
                                 .sequenceNumber()));
 
         if (!isComplete) {
-            assignedSplits.add(splitState);
+            assignedSplits.add(splitContext);
         }
         return new DynamoDbStreamRecordsWithSplitIds(
-                getRecordsResponse.records().iterator(), splitState.getSplitId(), isComplete);
+                getRecordsResponse.records().iterator(),
+                splitContext.splitState.getSplitId(),
+                isComplete);
+    }
+
+    private void sleep(long milliseconds) throws IOException {
+        try {
+            Thread.sleep(milliseconds);
+        } catch (InterruptedException e) {
+            throw new IOException("Split reader was interrupted while sleeping", e);
+        }
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: hlteoh37
body: As mentioned above, we should not be calling `Thread.sleep`. Let's remove this
created_at: 2025-02-27T13:07:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,4 +256,17 @@ public Set<String> finishedSplits() {
             return isComplete ? singleton(splitId) : Collections.emptySet();
         }
     }
+
+    @Internal
+    private static class SplitContext {
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: hlteoh37
body: Let's name this something more meaningful like `DynamoDbStreamsShardSplitWithContext`!
created_at: 2025-02-27T13:08:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -133,13 +135,20 @@ public SourceReader<T, DynamoDbStreamsShardSplit> createReader(
         setUpDeserializationSchema(readerContext);
 
         Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap = new ConcurrentHashMap<>();
+        final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS);
+        final Duration getRecordsIdlePollingTimeBetweenEmptyPolls =
+                sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS);
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/DynamoDbStreamsSource.java
user: login: hlteoh37
body: Let's add documentation for this. What is the default?
created_at: 2025-02-27T13:09:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,6 +76,23 @@ public enum InitialPosition {
     public static final String BASE_DDB_STREAMS_USER_AGENT_PREFIX_FORMAT =
             "Apache Flink %s (%s) DynamoDb Streams Connector";
 
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS =
+                    ConfigOptions.key(
+                                    "flink.dynamodbstreams.getrecords.idle.time.between.empty.polls")
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
user: login: hlteoh37
body: this name is abit long, can we shorten
`flink.dynamodbstreams.getrecords.empty.mindelay`

`flink.dynamodbstreams.getrecords.nonempty.mindelay`
created_at: 2025-02-27T13:10:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +59,101 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final PriorityQueue<SplitContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
     private final Set<String> pausedSplitIds = new HashSet<>();
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits =
+                new PriorityQueue<>(
+                        (a, b) -> {
+                            // First, handle paused splits
+                            boolean aIsPaused = pausedSplitIds.contains(a.splitState.getSplitId());
+                            boolean bIsPaused = pausedSplitIds.contains(b.splitState.getSplitId());
+                            if (aIsPaused && !bIsPaused) {
+                                return 1;
+                            }
+                            if (!aIsPaused && bIsPaused) {
+                                return -1;
+                            }
+                            if (aIsPaused && bIsPaused) {
+                                return 0;
+                            }
+
+                            // Get next eligible time for both splits
+                            long aNextEligibleTime = getNextEligibleTime(a);
+                            long bNextEligibleTime = getNextEligibleTime(b);
+
+                            return Long.compare(aNextEligibleTime, bNextEligibleTime);
+                        });
+    }
+
+    private long getNextEligibleTime(SplitContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
+        SplitContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+
+        // Check if split is paused or not ready due to empty poll delay
+        long currentTime = System.currentTimeMillis();
+        long nextEligibleTime = getNextEligibleTime(splitContext);
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        if (nextEligibleTime > currentTime) {
+            assignedSplits.add(splitContext);
+            sleep(20);
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: yeah, i was initially doing that but without thread sleep, this leads to the job manager CPU runnign at 100%. is there any good way we can provide to control the fetch calls
created_at: 2025-02-27T13:11:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,4 +256,17 @@ public Set<String> finishedSplits() {
             return isComplete ? singleton(splitId) : Collections.emptySet();
         }
     }
+
+    @Internal
+    private static class SplitContext {
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: makes sense
created_at: 2025-02-27T13:11:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +59,101 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final PriorityQueue<SplitContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
     private final Set<String> pausedSplitIds = new HashSet<>();
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits =
+                new PriorityQueue<>(
+                        (a, b) -> {
+                            // First, handle paused splits
+                            boolean aIsPaused = pausedSplitIds.contains(a.splitState.getSplitId());
+                            boolean bIsPaused = pausedSplitIds.contains(b.splitState.getSplitId());
+                            if (aIsPaused && !bIsPaused) {
+                                return 1;
+                            }
+                            if (!aIsPaused && bIsPaused) {
+                                return -1;
+                            }
+                            if (aIsPaused && bIsPaused) {
+                                return 0;
+                            }
+
+                            // Get next eligible time for both splits
+                            long aNextEligibleTime = getNextEligibleTime(a);
+                            long bNextEligibleTime = getNextEligibleTime(b);
+
+                            return Long.compare(aNextEligibleTime, bNextEligibleTime);
+                        });
+    }
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: yeah, makes sense we can do that

created_at: 2025-02-27T13:11:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +59,101 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final PriorityQueue<SplitContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
     private final Set<String> pausedSplitIds = new HashSet<>();
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits =
+                new PriorityQueue<>(
+                        (a, b) -> {
+                            // First, handle paused splits
+                            boolean aIsPaused = pausedSplitIds.contains(a.splitState.getSplitId());
+                            boolean bIsPaused = pausedSplitIds.contains(b.splitState.getSplitId());
+                            if (aIsPaused && !bIsPaused) {
+                                return 1;
+                            }
+                            if (!aIsPaused && bIsPaused) {
+                                return -1;
+                            }
+                            if (aIsPaused && bIsPaused) {
+                                return 0;
+                            }
+
+                            // Get next eligible time for both splits
+                            long aNextEligibleTime = getNextEligibleTime(a);
+                            long bNextEligibleTime = getNextEligibleTime(b);
+
+                            return Long.compare(aNextEligibleTime, bNextEligibleTime);
+                        });
+    }
+
+    private long getNextEligibleTime(SplitContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
+        SplitContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+
+        // Check if split is paused or not ready due to empty poll delay
+        long currentTime = System.currentTimeMillis();
+        long nextEligibleTime = getNextEligibleTime(splitContext);
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        if (nextEligibleTime > currentTime) {
+            assignedSplits.add(splitContext);
+            sleep(20);
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: as discussed offline, since we didnt find any way to do this, we will stick with a Thread.sleep, but with a lower sleep time (1ms)
created_at: 2025-03-03T16:41:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,10 +175,20 @@ public RecordsWithSplitIds<Record> fetch() throws IOException {
                                 .sequenceNumber()));
 
         if (!isComplete) {
-            assignedSplits.add(splitState);
+            assignedSplits.add(splitContext);
         }
         return new DynamoDbStreamRecordsWithSplitIds(
-                getRecordsResponse.records().iterator(), splitState.getSplitId(), isComplete);
+                getRecordsResponse.records().iterator(),
+                splitContext.splitState.getSplitId(),
+                isComplete);
+    }
+
+    private void sleep(long milliseconds) throws IOException {
+        try {
+            Thread.sleep(milliseconds);
+        } catch (InterruptedException e) {
+            throw new IOException("Split reader was interrupted while sleeping", e);
+        }
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: as discussed offline, since we dont have a easy way to do this from flink, we will continue with a thread.sleep of 1 ms
created_at: 2025-03-03T16:42:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -191,4 +256,17 @@ public Set<String> finishedSplits() {
             return isComplete ? singleton(splitId) : Collections.emptySet();
         }
     }
+
+    @Internal
+    private static class SplitContext {
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: done
created_at: 2025-03-03T16:42:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,6 +76,23 @@ public enum InitialPosition {
     public static final String BASE_DDB_STREAMS_USER_AGENT_PREFIX_FORMAT =
             "Apache Flink %s (%s) DynamoDb Streams Connector";
 
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS =
+                    ConfigOptions.key(
+                                    "flink.dynamodbstreams.getrecords.idle.time.between.empty.polls")
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
user: login: gguptp
body: done
created_at: 2025-03-03T16:43:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,6 +76,23 @@ public enum InitialPosition {
     public static final String BASE_DDB_STREAMS_USER_AGENT_PREFIX_FORMAT =
             "Apache Flink %s (%s) DynamoDb Streams Connector";
 
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS =
+                    ConfigOptions.key(
+                                    "flink.dynamodbstreams.getrecords.idle.time.between.empty.polls")
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
user: login: gguptp
body: done
created_at: 2025-03-03T17:21:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +59,101 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final PriorityQueue<SplitContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
     private final Set<String> pausedSplitIds = new HashSet<>();
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits =
+                new PriorityQueue<>(
+                        (a, b) -> {
+                            // First, handle paused splits
+                            boolean aIsPaused = pausedSplitIds.contains(a.splitState.getSplitId());
+                            boolean bIsPaused = pausedSplitIds.contains(b.splitState.getSplitId());
+                            if (aIsPaused && !bIsPaused) {
+                                return 1;
+                            }
+                            if (!aIsPaused && bIsPaused) {
+                                return -1;
+                            }
+                            if (aIsPaused && bIsPaused) {
+                                return 0;
+                            }
+
+                            // Get next eligible time for both splits
+                            long aNextEligibleTime = getNextEligibleTime(a);
+                            long bNextEligibleTime = getNextEligibleTime(b);
+
+                            return Long.compare(aNextEligibleTime, bNextEligibleTime);
+                        });
+    }
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: as discussed offline, for heavy polling cases, having pq and array dequeu should give similar polling rates hence moving with array dequeue
created_at: 2025-03-03T17:21:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -133,13 +135,20 @@ public SourceReader<T, DynamoDbStreamsShardSplit> createReader(
         setUpDeserializationSchema(readerContext);
 
         Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap = new ConcurrentHashMap<>();
+        final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS);
+        final Duration getRecordsIdlePollingTimeBetweenEmptyPolls =
+                sourceConfig.get(DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS);
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/DynamoDbStreamsSource.java
user: login: gguptp
body: done.
created_at: 2025-03-03T17:23:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +60,89 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final Deque<DynamoDbStreamsShardSplitWithContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
-    private final Set<String> pausedSplitIds = new HashSet<>();
+    private final Set<String> pausedSplitIds;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(PollingDynamoDbStreamsShardSplitReader.class);
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits = new ArrayDeque<>();
+        this.pausedSplitIds = new HashSet<>();
+    }
+
+    private long getNextEligibleTime(DynamoDbStreamsShardSplitWithContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+        DynamoDbStreamsShardSplitWithContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        // Check if split is paused or not ready due to empty poll delay
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: hlteoh37
body: This comment doesn't seem appropriate (since pause is checked above)
created_at: 2025-03-04T14:01:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +60,89 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final Deque<DynamoDbStreamsShardSplitWithContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
-    private final Set<String> pausedSplitIds = new HashSet<>();
+    private final Set<String> pausedSplitIds;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(PollingDynamoDbStreamsShardSplitReader.class);
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits = new ArrayDeque<>();
+        this.pausedSplitIds = new HashSet<>();
+    }
+
+    private long getNextEligibleTime(DynamoDbStreamsShardSplitWithContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+        DynamoDbStreamsShardSplitWithContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        // Check if split is paused or not ready due to empty poll delay
+        long currentTime = System.currentTimeMillis();
+        long nextEligibleTime = getNextEligibleTime(splitContext);
+
+        LOG.debug(
+                "Polling split: {}, currentTime: {}, eligibleTime: {}, wasEmptyPoll: {}",
+                splitContext.splitState.getSplitId(),
+                currentTime,
+                nextEligibleTime,
+                splitContext.wasLastPollEmpty);
+        if (nextEligibleTime > currentTime) {
+            assignedSplits.add(splitContext);
+            sleep(1);
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: hlteoh37
body: Does this actually help with CPU 100%?
created_at: 2025-03-04T14:03:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,10 +336,103 @@ record ->
         for (int i = 0; i < 10; i++) {
             RecordsWithSplitIds<Record> records = splitReader.fetch();
             fetchedRecords.addAll(readAllRecords(records));
+            Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Immediate second poll - should return empty due to polling delay
path: flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
user: login: hlteoh37
body: Can we make sure this test is not flaky by making split delay something like 1 min?
created_at: 2025-03-04T14:15:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,10 +336,103 @@ record ->
         for (int i = 0; i < 10; i++) {
             RecordsWithSplitIds<Record> records = splitReader.fetch();
             fetchedRecords.addAll(readAllRecords(records));
+            Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Immediate second poll - should return empty due to polling delay
+        RecordsWithSplitIds<Record> secondPoll = splitReader.fetch();
+        assertThat(secondPoll.nextRecordFromSplit()).isNull();
+        assertThat(secondPoll.nextSplit()).isNull();
+        assertThat(secondPoll.finishedSplits()).isEmpty();
+    }
+
+    @Test
+    void testLessPollingDelayForNonEmptyRecords() throws Exception {
+        // Given assigned split with records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        Record record1 = getTestRecord("data-1");
+        Record record2 = getTestRecord("data-2");
+
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record1));
+
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
+        // First poll - should return record1
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(readAllRecords(firstPoll)).containsExactly(record1);
+
+        // Add second record
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record2));
+        Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
path: flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
user: login: hlteoh37
body: Can we use Awaitility instead of Thread.sleep?
https://github.com/awaitility/awaitility
created_at: 2025-03-04T14:19:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,6 +76,21 @@ public enum InitialPosition {
     public static final String BASE_DDB_STREAMS_USER_AGENT_PREFIX_FORMAT =
             "Apache Flink %s (%s) DynamoDb Streams Connector";
 
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.empty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(1000))
+                            .withDescription(
+                                    "The idle time between empty polls for DynamoDB Streams GetRecords API");
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.nonempty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(250))
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
user: login: hlteoh37
body: Why not make this default to 0?
created_at: 2025-03-04T14:19:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +60,89 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final Deque<DynamoDbStreamsShardSplitWithContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
-    private final Set<String> pausedSplitIds = new HashSet<>();
+    private final Set<String> pausedSplitIds;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(PollingDynamoDbStreamsShardSplitReader.class);
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits = new ArrayDeque<>();
+        this.pausedSplitIds = new HashSet<>();
+    }
+
+    private long getNextEligibleTime(DynamoDbStreamsShardSplitWithContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+        DynamoDbStreamsShardSplitWithContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        // Check if split is paused or not ready due to empty poll delay
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: yes, will update this
created_at: 2025-03-04T14:22:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,48 +60,89 @@ public class PollingDynamoDbStreamsShardSplitReader
             new DynamoDbStreamRecordsWithSplitIds(Collections.emptyIterator(), null, false);
 
     private final StreamProxy dynamodbStreams;
+    private final Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+    private final Duration getRecordsIdlePollingTimeBetweenEmptyPolls;
 
-    private final Deque<DynamoDbStreamsShardSplitState> assignedSplits = new ArrayDeque<>();
+    private final Deque<DynamoDbStreamsShardSplitWithContext> assignedSplits;
     private final Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap;
-    private final Set<String> pausedSplitIds = new HashSet<>();
+    private final Set<String> pausedSplitIds;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(PollingDynamoDbStreamsShardSplitReader.class);
 
     public PollingDynamoDbStreamsShardSplitReader(
             StreamProxy dynamodbStreamsProxy,
+            Duration getRecordsIdlePollingTimeBetweenNonEmptyPolls,
+            Duration getRecordsIdlePollingTimeBetweenEmptyPolls,
             Map<String, DynamoDbStreamsShardMetrics> shardMetricGroupMap) {
         this.dynamodbStreams = dynamodbStreamsProxy;
+        this.getRecordsIdlePollingTimeBetweenNonEmptyPolls =
+                getRecordsIdlePollingTimeBetweenNonEmptyPolls;
+        this.getRecordsIdlePollingTimeBetweenEmptyPolls =
+                getRecordsIdlePollingTimeBetweenEmptyPolls;
         this.shardMetricGroupMap = shardMetricGroupMap;
+        this.assignedSplits = new ArrayDeque<>();
+        this.pausedSplitIds = new HashSet<>();
+    }
+
+    private long getNextEligibleTime(DynamoDbStreamsShardSplitWithContext splitContext) {
+        long requiredDelay =
+                splitContext.wasLastPollEmpty
+                        ? getRecordsIdlePollingTimeBetweenEmptyPolls.toMillis()
+                        : getRecordsIdlePollingTimeBetweenNonEmptyPolls.toMillis();
+
+        return splitContext.lastPollTimeMillis + requiredDelay;
     }
 
     @Override
     public RecordsWithSplitIds<Record> fetch() throws IOException {
-        DynamoDbStreamsShardSplitState splitState = assignedSplits.poll();
-        if (splitState == null) {
+        if (assignedSplits.isEmpty()) {
+            return INCOMPLETE_SHARD_EMPTY_RECORDS;
+        }
+        DynamoDbStreamsShardSplitWithContext splitContext = assignedSplits.poll();
+
+        if (pausedSplitIds.contains(splitContext.splitState.getSplitId())) {
+            assignedSplits.add(splitContext);
             return INCOMPLETE_SHARD_EMPTY_RECORDS;
         }
 
-        if (pausedSplitIds.contains(splitState.getSplitId())) {
-            assignedSplits.add(splitState);
+        // Check if split is paused or not ready due to empty poll delay
+        long currentTime = System.currentTimeMillis();
+        long nextEligibleTime = getNextEligibleTime(splitContext);
+
+        LOG.debug(
+                "Polling split: {}, currentTime: {}, eligibleTime: {}, wasEmptyPoll: {}",
+                splitContext.splitState.getSplitId(),
+                currentTime,
+                nextEligibleTime,
+                splitContext.wasLastPollEmpty);
+        if (nextEligibleTime > currentTime) {
+            assignedSplits.add(splitContext);
+            sleep(1);
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReader.java
user: login: gguptp
body: yeah, if i remove this sleep, the CPU goes full 100% all the time, presumably flink calls the fetch function every microsecond
created_at: 2025-03-04T14:23:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,6 +76,21 @@ public enum InitialPosition {
     public static final String BASE_DDB_STREAMS_USER_AGENT_PREFIX_FORMAT =
             "Apache Flink %s (%s) DynamoDb Streams Connector";
 
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.empty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(1000))
+                            .withDescription(
+                                    "The idle time between empty polls for DynamoDB Streams GetRecords API");
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.nonempty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(250))
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
user: login: gguptp
body: I thought 0 would lead to an undeterministic number of calls, 250 would ensure that the calls are capped at 4 rps
created_at: 2025-03-04T14:24:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,10 +336,103 @@ record ->
         for (int i = 0; i < 10; i++) {
             RecordsWithSplitIds<Record> records = splitReader.fetch();
             fetchedRecords.addAll(readAllRecords(records));
+            Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Immediate second poll - should return empty due to polling delay
+        RecordsWithSplitIds<Record> secondPoll = splitReader.fetch();
+        assertThat(secondPoll.nextRecordFromSplit()).isNull();
+        assertThat(secondPoll.nextSplit()).isNull();
+        assertThat(secondPoll.finishedSplits()).isEmpty();
+    }
+
+    @Test
+    void testLessPollingDelayForNonEmptyRecords() throws Exception {
+        // Given assigned split with records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        Record record1 = getTestRecord("data-1");
+        Record record2 = getTestRecord("data-2");
+
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record1));
+
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
+        // First poll - should return record1
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(readAllRecords(firstPoll)).containsExactly(record1);
+
+        // Add second record
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record2));
+        Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
path: flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
user: login: gguptp
body: wasnt aware about this, we can use
created_at: 2025-03-04T14:25:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,10 +336,103 @@ record ->
         for (int i = 0; i < 10; i++) {
             RecordsWithSplitIds<Record> records = splitReader.fetch();
             fetchedRecords.addAll(readAllRecords(records));
+            Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Immediate second poll - should return empty due to polling delay
path: flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
user: login: gguptp
body: sorry i didnt understand this comment?
created_at: 2025-03-04T14:25:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,10 +336,103 @@ record ->
         for (int i = 0; i < 10; i++) {
             RecordsWithSplitIds<Record> records = splitReader.fetch();
             fetchedRecords.addAll(readAllRecords(records));
+            Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Immediate second poll - should return empty due to polling delay
path: flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
user: login: hlteoh37
body: I just meant let's make sure we ensure that this test is not flaky if we run on a slow host
created_at: 2025-03-04T14:36:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,10 +336,103 @@ record ->
         for (int i = 0; i < 10; i++) {
             RecordsWithSplitIds<Record> records = splitReader.fetch();
             fetchedRecords.addAll(readAllRecords(records));
+            Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Immediate second poll - should return empty due to polling delay
+        RecordsWithSplitIds<Record> secondPoll = splitReader.fetch();
+        assertThat(secondPoll.nextRecordFromSplit()).isNull();
+        assertThat(secondPoll.nextSplit()).isNull();
+        assertThat(secondPoll.finishedSplits()).isEmpty();
+    }
+
+    @Test
+    void testLessPollingDelayForNonEmptyRecords() throws Exception {
+        // Given assigned split with records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        Record record1 = getTestRecord("data-1");
+        Record record2 = getTestRecord("data-2");
+
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record1));
+
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
+        // First poll - should return record1
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(readAllRecords(firstPoll)).containsExactly(record1);
+
+        // Add second record
+        testStreamProxy.addRecords(
+                TestUtil.STREAM_ARN, TEST_SHARD_ID, Collections.singletonList(record2));
+        Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
path: flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
user: login: gguptp
body: done
created_at: 2025-03-04T18:35:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -320,10 +336,103 @@ record ->
         for (int i = 0; i < 10; i++) {
             RecordsWithSplitIds<Record> records = splitReader.fetch();
             fetchedRecords.addAll(readAllRecords(records));
+            Thread.sleep(NON_EMPTY_POLLING_DELAY_MILLIS.toMillis());
         }
         assertThat(fetchedRecords).containsExactly(recordsFromSplit3.toArray(new Record[0]));
     }
 
+    @Test
+    void testPollingDelayForEmptyRecords() throws Exception {
+        // Given assigned split with no records
+        testStreamProxy.addShards(TEST_SHARD_ID);
+        splitReader.handleSplitsChanges(
+                new SplitsAddition<>(Collections.singletonList(getTestSplit(TEST_SHARD_ID))));
+
+        // First poll - should return empty records
+        RecordsWithSplitIds<Record> firstPoll = splitReader.fetch();
+        assertThat(firstPoll.nextRecordFromSplit()).isNull();
+        assertThat(firstPoll.nextSplit()).isNull();
+        assertThat(firstPoll.finishedSplits()).isEmpty();
+
+        // Immediate second poll - should return empty due to polling delay
path: flink-connector-aws/flink-connector-dynamodb/src/test/java/org/apache/flink/connector/dynamodb/source/reader/PollingDynamoDbStreamsShardSplitReaderTest.java
user: login: gguptp
body: added the empty poll delay of 1 min for this, that should ensure the test is not flaky
created_at: 2025-03-04T18:40:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,6 +76,21 @@ public enum InitialPosition {
     public static final String BASE_DDB_STREAMS_USER_AGENT_PREFIX_FORMAT =
             "Apache Flink %s (%s) DynamoDb Streams Connector";
 
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.empty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(1000))
+                            .withDescription(
+                                    "The idle time between empty polls for DynamoDB Streams GetRecords API");
+    public static final ConfigOption<Duration>
+            DYNAMODB_STREAMS_GET_RECORDS_IDLE_TIME_BETWEEN_NON_EMPTY_POLLS =
+                    ConfigOptions.key("flink.dynamodbstreams.getrecords.nonempty.mindelay")
+                            .durationType()
+                            .defaultValue(Duration.ofMillis(250))
path: flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/source/config/DynamodbStreamsSourceConfigConstants.java
user: login: gguptp
body: Keeping it 250 millis, we can make it 0 if need requires
created_at: 2025-03-04T18:41:15Z
author_association: CONTRIBUTOR

PR comments:
user: login: gguptp
created_at: 2025-03-04T03:32:48Z
body: @hlteoh37 i have addressed the comments and updated the PR. Thanks!
author_association: CONTRIBUTOR
user: login: gguptp
created_at: 2025-03-04T18:44:05Z
body: > Added some questions. Thanks @gguptp

Thanks @hlteoh37 . I have updated the PR with the requested changes
author_association: CONTRIBUTOR