Repository: apache/ozone
Pull request #8157: HDDS-12596. OM fs snapshot max limit is not enforced
Author: login: peterxcli
Labels: name: snapshot
description: https://issues.apache.org/jira/browse/HDDS-6517
Created: 2025-03-25T15:03:53Z | Merged: 2025-05-02T06:39:01Z

## What changes were proposed in this pull request?

```xml
//ozone-default.xml
<property>
  <name>ozone.om.fs.snapshot.max.limit</name>
  <value>1000</value>
  <tag>OZONE, OM, MANAGEMENT</tag>
  <description>
    The maximum number of filesystem snapshot allowed in an Ozone Manager.
  </description>
</property>
```
The value of the above property is passed to RDBStore but it is never used.

We should verify & limit the number of snapshots created in `OMSnapshotCreateRequest`.

- Get non-deleted rows count directly from `NumSnapshotActive` metrics.
  - It would be prepared on the OM node restart, and be inc/dec on  `TestOMSnapshotCreateRequest`/`TestOMSnapshotDeleteRequest`. So it would reflect the real snapshotTable row count.
https://github.com/apache/ozone/blob/a4d878ef7ad0bd8697fddffa689b8159ca22df1f/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L1949-L1980

## What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12596

## How was this patch tested?

CI:
~~https://github.com/peterxcli/ozone/actions/runs/14679627514~~
https://github.com/peterxcli/ozone/actions/runs/14683601913


Diff:
diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
index 660b21c45bcf..596eb1276560 100644
--- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
+++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java
@@ -272,6 +272,8 @@ public enum ResultCodes {
     TOO_MANY_BUCKETS,
     KEY_UNDER_LEASE_RECOVERY,
     KEY_ALREADY_CLOSED,
-    KEY_UNDER_LEASE_SOFT_LIMIT_PERIOD
+    KEY_UNDER_LEASE_SOFT_LIMIT_PERIOD,
+
+    TOO_MANY_SNAPSHOTS,
   }
 }
diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerHASnapshot.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerHASnapshot.java
index 3d4bc628e2e1..728ef1c88051 100644
--- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerHASnapshot.java
+++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerHASnapshot.java
@@ -344,6 +344,29 @@ public void testKeyAndSnapshotDeletionService() throws IOException, InterruptedE
     checkSnapshotIsPurgedFromDB(omFollower, tableKey);
   }
 
+  @Test
+  public void testSnapshotInFlightCount() throws Exception {
+    // snapshot inflight count should be reset to 0 when leader changes
+
+    // first do some snapshot creations
+    String snapshotName1 = UUID.randomUUID().toString();
+    store.createSnapshot(volumeName, bucketName, snapshotName1);
+
+    // then shutdown the leader
+    OzoneManager omLeader = cluster.getOMLeader();
+    cluster.shutdownOzoneManager(omLeader);
+
+    // wait for the new leader to be elected
+    cluster.waitForLeaderOM();
+
+    // check the inflight count on the new leader is 0
+    OzoneManager newLeader = cluster.getOMLeader();
+    assertEquals(0, newLeader.getOmSnapshotManager().getInFlightSnapshotCount());
+
+    // restart the previous shutdowned node
+    cluster.restartOzoneManager(omLeader, true);
+  }
+
   private void createSnapshot(String volName, String buckName, String snapName) throws IOException {
     store.createSnapshot(volName, buckName, snapName);
 
diff --git a/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto b/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
index 4108af198bff..92edafa9426d 100644
--- a/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
+++ b/hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
@@ -564,6 +564,7 @@ enum Status {
     KEY_ALREADY_CLOSED = 96;
     KEY_UNDER_LEASE_SOFT_LIMIT_PERIOD = 97;
 
+    TOO_MANY_SNAPSHOTS = 98;
 }
 
 /**
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
index 6629e911af98..788271042212 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
@@ -24,6 +24,8 @@
 import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_CHECKPOINT_DIR;
 import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_DIFF_DB_NAME;
 import static org.apache.hadoop.ozone.OzoneConsts.OM_SNAPSHOT_INDICATOR;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT_DEFAULT;
 import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_CLEANUP_SERVICE_RUN_INTERVAL;
 import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_CLEANUP_SERVICE_RUN_INTERVAL_DEFAULT;
 import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE;
@@ -63,6 +65,8 @@
 import java.util.Set;
 import java.util.UUID;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
 import java.util.stream.Collectors;
 import org.apache.commons.lang3.tuple.Pair;
 import org.apache.hadoop.hdds.StringUtils;
@@ -173,6 +177,9 @@ public final class OmSnapshotManager implements AutoCloseable {
   // Soft limit of the snapshot cache size.
   private final int softCacheSize;
 
+  private int fsSnapshotMaxLimit;
+  private final AtomicInteger inFlightSnapshotCount = new AtomicInteger(0);
+
   public OmSnapshotManager(OzoneManager ozoneManager) {
 
     boolean isFilesystemSnapshotEnabled =
@@ -247,6 +254,9 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
     this.softCacheSize = ozoneManager.getConfiguration().getInt(
         OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE,
         OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE_DEFAULT);
+      
+    fsSnapshotMaxLimit = ozoneManager.getConfiguration().getInt(OZONE_OM_FS_SNAPSHOT_MAX_LIMIT,
+        OZONE_OM_FS_SNAPSHOT_MAX_LIMIT_DEFAULT);
 
     CacheLoader<UUID, OmSnapshot> loader = createCacheLoader();
 
@@ -859,6 +869,42 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
+
+    AtomicReference<OMException> exceptionRef = new AtomicReference<>(null);
+    inFlightSnapshotCount.updateAndGet(count -> {
+      if (currentSnapshotNum + count >= fsSnapshotMaxLimit) {
+        exceptionRef.set(new OMException(
+            String.format("Snapshot limit of %d reached. Cannot create more snapshots. " +
+                "Current snapshots: %d, In-flight creations: %d",
+                fsSnapshotMaxLimit, currentSnapshotNum, count) +
+                " If you already deleted some snapshots, " +
+                "please wait for the background service to complete the cleanup.",
+            OMException.ResultCodes.TOO_MANY_SNAPSHOTS));
+        return count;
+      }
+      return count + 1;
+    });
+    if (exceptionRef.get() != null) {
+      throw exceptionRef.get();
+    }
+  }
+
+  public void decrementInFlightSnapshotCount() {
+    inFlightSnapshotCount.decrementAndGet();
+  }
+
+  public void resetInFlightSnapshotCount() {
+    inFlightSnapshotCount.set(0);
+  }
+
+  public int getInFlightSnapshotCount() {
+    return inFlightSnapshotCount.get();
+  }
+
   private int getIndexFromToken(final String token) throws IOException {
     if (isBlank(token)) {
       return 0;
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
index 71e810d78eb8..c26140424c25 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
@@ -156,6 +156,11 @@ public SnapshotInfo getLatestSnapshot() {
     return snapshotInfo;
   }
 
+  @Override
+  public void notifyLeaderReady() {
+    ozoneManager.getOmSnapshotManager().resetInFlightSnapshotCount();
+  }
+
   @Override
   public void notifyLeaderChanged(RaftGroupMemberId groupMemberId,
                                   RaftPeerId newLeaderId) {
diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
index 900419ad4223..8efa517a750b 100644
--- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
@@ -120,6 +120,8 @@ public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
           "Only bucket owners and Ozone admins can create snapshots",
           OMException.ResultCodes.PERMISSION_DENIED);
     }
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
     CreateSnapshotRequest.Builder createSnapshotRequest = omRequest.getCreateSnapshotRequest().toBuilder()
         .setSnapshotId(toProtobuf(UUID.randomUUID()))
         .setVolumeName(volumeName)
@@ -188,7 +190,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
       //  pre-replicated key size counter in OmBucketInfo.
       snapshotInfo.setReferencedSize(estimateBucketDataSize(omBucketInfo));
 
-      addSnapshotInfoToSnapshotChainAndCache(omMetadataManager, context.getIndex());
+      addSnapshotInfoToSnapshotChainAndCache(ozoneManager, omMetadataManager, context.getIndex());
 
       omResponse.setCreateSnapshotResponse(
           CreateSnapshotResponse.newBuilder()
@@ -250,6 +252,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
    * it was removed at T-5.
    */
   private void addSnapshotInfoToSnapshotChainAndCache(
+      OzoneManager ozoneManager,
       OmMetadataManagerImpl omMetadataManager,
       long transactionLogIndex
   ) throws IOException {
@@ -288,6 +291,8 @@ private void addSnapshotInfoToSnapshotChainAndCache(
         removeSnapshotInfoFromSnapshotChainManager(snapshotChainManager,
             snapshotInfo);
         throw new IOException(exception.getMessage(), exception);
+      } finally {
+        ozoneManager.getOmSnapshotManager().decrementInFlightSnapshotCount();
       }
     }
   }
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
index 7cb1e2e38ab8..8e2a53e7c0d8 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
@@ -32,6 +32,7 @@
 import static org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.getINode;
 import static org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.truncateFileName;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertNotNull;
@@ -54,6 +55,7 @@
 import java.util.Arrays;
 import java.util.HashMap;
 import java.util.HashSet;
+import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
@@ -76,6 +78,7 @@
 import org.apache.ozone.test.GenericTestUtils;
 import org.apache.ozone.test.GenericTestUtils.LogCapturer;
 import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.AfterEach;
 import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
@@ -90,6 +93,9 @@
 class TestOmSnapshotManager {
 
   private OzoneManager om;
+  private SnapshotChainManager snapshotChainManager;
+  private OmMetadataManagerImpl omMetadataManager;
+  private OmSnapshotManager omSnapshotManager;
   private static final String CANDIDATE_DIR_NAME = OM_DB_NAME +
       SNAPSHOT_CANDIDATE_DIR;
   private File leaderDir;
@@ -114,9 +120,15 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
@@ -124,6 +136,21 @@ void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
+
+    Iterator<UUID> iter = snapshotChainManager.iterator(true);
+    while (iter.hasNext()) {
+      UUID snapshotId = iter.next();
+      String snapshotInfoKey = snapshotChainManager.getTableKey(snapshotId);
+      SnapshotInfo snapshotInfo = snapshotInfoTable.get(snapshotInfoKey);
+      snapshotChainManager.deleteSnapshot(snapshotInfo);
+      snapshotInfoTable.delete(snapshotInfoKey);
+    }
+    omSnapshotManager.invalidateCache();
+  }
+
   @Test
   public void testSnapshotFeatureFlagSafetyCheck() throws IOException {
     // Verify that the snapshot feature config safety check method
@@ -151,11 +178,11 @@ public void testCloseOnEviction() throws IOException,
     Table<String, OmBucketInfo> bucketTable = mock(Table.class);
     Table<String, SnapshotInfo> snapshotInfoTable = mock(Table.class);
     HddsWhiteboxTestUtils.setInternalState(
-        om.getMetadataManager(), VOLUME_TABLE, volumeTable);
+        omMetadataManager, VOLUME_TABLE, volumeTable);
     HddsWhiteboxTestUtils.setInternalState(
-        om.getMetadataManager(), BUCKET_TABLE, bucketTable);
+        omMetadataManager, BUCKET_TABLE, bucketTable);
     HddsWhiteboxTestUtils.setInternalState(
-        om.getMetadataManager(), SNAPSHOT_INFO_TABLE, snapshotInfoTable);
+        omMetadataManager, SNAPSHOT_INFO_TABLE, snapshotInfoTable);
 
     final String volumeName = UUID.randomUUID().toString();
     final String dbVolumeKey = om.getMetadataManager().getVolumeKey(volumeName);
@@ -185,8 +212,8 @@ public void testCloseOnEviction() throws IOException,
     when(snapshotInfoTable.get(first.getTableKey())).thenReturn(first);
     when(snapshotInfoTable.get(second.getTableKey())).thenReturn(second);
 
-    ((OmMetadataManagerImpl) om.getMetadataManager()).getSnapshotChainManager().addSnapshot(first);
-    ((OmMetadataManagerImpl) om.getMetadataManager()).getSnapshotChainManager().addSnapshot(second);
+    snapshotChainManager.addSnapshot(first);
+    snapshotChainManager.addSnapshot(second);
     RDBBatchOperation rdbBatchOperation = new RDBBatchOperation();
     // create the first snapshot checkpoint
     OmSnapshotManager.createOmSnapshotCheckpoint(om.getMetadataManager(),
@@ -194,7 +221,6 @@ public void testCloseOnEviction() throws IOException,
     om.getMetadataManager().getStore().commitBatchOperation(rdbBatchOperation);
 
     // retrieve it and setup store mock
-    OmSnapshotManager omSnapshotManager = om.getOmSnapshotManager();
     OmSnapshot firstSnapshot = omSnapshotManager
         .getActiveSnapshot(first.getVolumeName(), first.getBucketName(), first.getName())
         .get();
@@ -230,6 +256,37 @@ public void testCloseOnEviction() throws IOException,
     }, 100, 30_000);
   }
 
+  @Test
+  public void testValidateSnapshotLimit() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = mock(Table.class);
+    HddsWhiteboxTestUtils.setInternalState(
+        omMetadataManager, SNAPSHOT_INFO_TABLE, snapshotInfoTable);
+
+    SnapshotInfo first = createSnapshotInfo("vol1", "buck1");
+    SnapshotInfo second = createSnapshotInfo("vol1", "buck1");
+
+    first.setGlobalPreviousSnapshotId(null);
+    first.setPathPreviousSnapshotId(null);
+    second.setGlobalPreviousSnapshotId(first.getSnapshotId());
+    second.setPathPreviousSnapshotId(first.getSnapshotId());
+
+    when(snapshotInfoTable.get(first.getTableKey())).thenReturn(first);
+    when(snapshotInfoTable.get(second.getTableKey())).thenReturn(second);
+
+    snapshotChainManager.addSnapshot(first);
+    assertDoesNotThrow(() -> omSnapshotManager.snapshotLimitCheck());
+    omSnapshotManager.decrementInFlightSnapshotCount();
+
+    snapshotChainManager.addSnapshot(second);
+
+    OMException exception = assertThrows(OMException.class, () -> omSnapshotManager.snapshotLimitCheck());
+    assertEquals(OMException.ResultCodes.TOO_MANY_SNAPSHOTS, exception.getResult());
+
+    snapshotChainManager.deleteSnapshot(second);
+
+    assertDoesNotThrow(() -> omSnapshotManager.snapshotLimitCheck());
+  }
+
   @BeforeEach
   void setupData(@TempDir File testDir) throws IOException {
     // Set up the leader with the following files:
@@ -333,16 +390,12 @@ public void testHardLinkCreation() throws IOException {
   @Test
   public void testGetSnapshotInfo() throws IOException {
     SnapshotInfo s1 = createSnapshotInfo("vol", "buck");
-    UUID latestGlobalSnapId =
-        ((OmMetadataManagerImpl) om.getMetadataManager()).getSnapshotChainManager()
-            .getLatestGlobalSnapshotId();
+    UUID latestGlobalSnapId = snapshotChainManager.getLatestGlobalSnapshotId();
     UUID latestPathSnapId =
-        ((OmMetadataManagerImpl) om.getMetadataManager()).getSnapshotChainManager()
-            .getLatestPathSnapshotId(String.join("/", "vol", "buck"));
+        snapshotChainManager.getLatestPathSnapshotId(String.join("/", "vol", "buck"));
     s1.setPathPreviousSnapshotId(latestPathSnapId);
     s1.setGlobalPreviousSnapshotId(latestGlobalSnapId);
-    ((OmMetadataManagerImpl) om.getMetadataManager()).getSnapshotChainManager()
-        .addSnapshot(s1);
+    snapshotChainManager.addSnapshot(s1);
     OMException ome = assertThrows(OMException.class,
         () -> om.getOmSnapshotManager().getSnapshot(s1.getSnapshotId()));
     assertEquals(OMException.ResultCodes.FILE_NOT_FOUND, ome.getResult());
@@ -351,6 +404,12 @@ public void testGetSnapshotInfo() throws IOException {
     ome = assertThrows(OMException.class,
         () -> om.getOmSnapshotManager().getSnapshot(s2.getSnapshotId()));
     assertEquals(OMException.ResultCodes.FILE_NOT_FOUND, ome.getResult());
+
+    // add to make cleanup work
+    Table<String, SnapshotInfo> snapshotInfoTable = mock(Table.class);
+    HddsWhiteboxTestUtils.setInternalState(
+        omMetadataManager, SNAPSHOT_INFO_TABLE, snapshotInfoTable);
+    when(snapshotInfoTable.get(s1.getTableKey())).thenReturn(s1);
   }
 
   /*
diff --git a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotCreateRequest.java b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotCreateRequest.java
index 7842bc1cab25..a45991816b1c 100644
--- a/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotCreateRequest.java
+++ b/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotCreateRequest.java
@@ -18,6 +18,7 @@
 package org.apache.hadoop.ozone.om.request.snapshot;
 
 import static org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor.THREE;
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT;
 import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.getFromProtobuf;
 import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.getTableKey;
 import static org.apache.hadoop.ozone.om.request.OMRequestTestUtils.createSnapshotRequest;
@@ -37,6 +38,7 @@
 import org.apache.hadoop.hdds.client.RatisReplicationConfig;
 import org.apache.hadoop.hdds.utils.TransactionInfo;
 import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
 import org.apache.hadoop.ozone.om.OzoneManager;
 import org.apache.hadoop.ozone.om.ResolvedBucket;
 import org.apache.hadoop.ozone.om.exceptions.OMException;
@@ -64,11 +66,17 @@
 public class TestOMSnapshotCreateRequest extends TestSnapshotRequestAndResponse {
   private String snapshotName1;
   private String snapshotName2;
+  private String snapshotName3;
+  private String snapshotName4;
+  private String snapshotName5;
 
   @BeforeEach
   public void setup() throws Exception {
     snapshotName1 = UUID.randomUUID().toString();
     snapshotName2 = UUID.randomUUID().toString();
+    snapshotName3 = UUID.randomUUID().toString();
+    snapshotName4 = UUID.randomUUID().toString();
+    snapshotName5 = UUID.randomUUID().toString();
   }
 
   @ValueSource(strings = {
@@ -264,6 +272,52 @@ public void testEntryExists() throws Exception {
     assertEquals(2, getOmMetrics().getNumSnapshotCreates());
   }
 
+  @Test
+  public void testSnapshotLimit() throws Exception {
+    when(getOzoneManager().isAdmin(any())).thenReturn(true);
+    getOzoneManager().getOmSnapshotManager().close();
+    getOzoneManager().getConfiguration().setInt(OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 3);
+    OmSnapshotManager omSnapshotManager = new OmSnapshotManager(getOzoneManager());
+    when(getOzoneManager().getOmSnapshotManager()).thenReturn(omSnapshotManager);
+
+    // Test Case 1: No snapshots in chain, no in-flight
+    String key1 = getTableKey(getVolumeName(), getBucketName(), snapshotName1);
+    OMRequest omRequest = createSnapshotRequest(getVolumeName(), getBucketName(), snapshotName1);
+    OMSnapshotCreateRequest omSnapshotCreateRequest = doPreExecute(omRequest);
+    assertNull(getOmMetadataManager().getSnapshotInfoTable().get(key1));
+    omSnapshotCreateRequest.validateAndUpdateCache(getOzoneManager(), 1);
+    assertNotNull(getOmMetadataManager().getSnapshotInfoTable().get(key1));
+
+    // Test Case 2: One snapshot in chain, no in-flight
+    String key2 = getTableKey(getVolumeName(), getBucketName(), snapshotName2);
+    OMRequest snapshotRequest2 = createSnapshotRequest(getVolumeName(), getBucketName(), snapshotName2);
+    OMSnapshotCreateRequest omSnapshotCreateRequest2 = doPreExecute(snapshotRequest2);
+    omSnapshotCreateRequest2.validateAndUpdateCache(getOzoneManager(), 2);
+    assertNotNull(getOmMetadataManager().getSnapshotInfoTable().get(key2));
+
+    // Test Case 3: Two snapshots in chain, one in-flight
+    // First create an in-flight snapshot
+    String key3 = getTableKey(getVolumeName(), getBucketName(), snapshotName3);
+    OMRequest snapshotRequest3 = createSnapshotRequest(getVolumeName(), getBucketName(), snapshotName3);
+    OMSnapshotCreateRequest omSnapshotCreateRequest3 = doPreExecute(snapshotRequest3);
+    // Don't call validateAndUpdateCache to keep it in-flight
+
+    // Try to create another snapshot - should fail as total would be 4 (2 in chain + 1 in-flight + 1 new)
+    OMRequest snapshotRequest4 = createSnapshotRequest(getVolumeName(), getBucketName(), snapshotName4);
+    OMException omException = assertThrows(OMException.class, () -> doPreExecute(snapshotRequest4));
+    assertEquals(OMException.ResultCodes.TOO_MANY_SNAPSHOTS, omException.getResult());
+
+    // Complete the in-flight snapshot
+    omSnapshotCreateRequest3.validateAndUpdateCache(getOzoneManager(), 3);
+    assertNotNull(getOmMetadataManager().getSnapshotInfoTable().get(key3));
+
+    // Test Case 4: Three snapshots in chain, no in-flight
+    // Try to create another snapshot - should fail as we've reached the limit  
+    OMRequest snapshotRequest5 = createSnapshotRequest(getVolumeName(), getBucketName(), snapshotName5);
+    omException = assertThrows(OMException.class, () -> doPreExecute(snapshotRequest5));
+    assertEquals(OMException.ResultCodes.TOO_MANY_SNAPSHOTS, omException.getResult());
+  }
+
   private void renameKey(String fromKey, String toKey, long offset)
       throws IOException {
     OmKeyInfo toKeyInfo = addKey(toKey, offset + 1L);


Reviews:
user: login: hemantk-12
body: Thanks @peterxcli for working on this.

state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T18:01:23Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T18:49:17Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T19:05:11Z
user: login: SaketaChalamchala
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:31:06Z
user: login: SaketaChalamchala
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:33:05Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T11:17:18Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T01:12:59Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T18:01:53Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-29T02:24:04Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-29T23:29:50Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T03:32:55Z
user: login: hemantk-12
body: Source code looks good. Left some comments on tests. Please take a look.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T18:16:56Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T22:02:37Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T23:06:30Z
user: login: swamirishi
body: @peterxcli thanks for working on the patch. I have a few suggestions for making this implementation correct in case of concurrent ratis requests.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T23:20:04Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T23:21:52Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T23:23:03Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T23:58:55Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T03:16:22Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T04:00:34Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T16:41:54Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T17:10:56Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T17:30:20Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T08:32:06Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T20:59:36Z
user: login: SaketaChalamchala
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:37:54Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T03:29:33Z
user: login: hemantk-12
body: Overall changes look good to me.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T21:22:05Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T01:34:41Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T07:11:11Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T18:53:17Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T20:08:49Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T20:11:24Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T20:42:20Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T12:48:15Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T12:49:03Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T16:42:20Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T16:48:18Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-15T11:30:45Z
user: login: hemantk-12
body: Thanks, @peterxcli for addressing all the review comments.
Overall changes look good to me.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T20:11:50Z
user: login: jojochuang
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T23:46:47Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T06:25:06Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T06:34:03Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T06:35:03Z
user: login: hemantk-12
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T19:54:50Z
user: login: hemantk-12
body: Thanks @peterxcli for the improvement.

LGTM+1.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T18:17:56Z
user: login: jojochuang
body: It looks like snapshotLimitCheck() can still be prone to race conditions. But it should be fine in this case. The race is adding currentSnapshotNum to inFlightSnapshotCount. But the result is not saved so it doesn't cause data loss or anything. It's a guardrail and it's okay even if it's off by a few.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T06:38:43Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T17:14:38Z
user: login: swamirishi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T17:15:49Z
user: login: peterxcli
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-02T17:22:20Z

Review comments:
diff_hunk: @@ -165,6 +165,14 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
         throw new OMException("Snapshot already exists", FILE_ALREADY_EXISTS);
       }
 
+      // Check snapshot limit
+      int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+      if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: 1. We can't rely on this metric. If a bootstrapped node becomes the leader, it may not have the correct active snapshot count. We should use SnapshotInfoTable size.
2. Should we compare it against total snapshots? I'm thinking that if there is an issue in the SnapshotDeletionService, we may accumulate a lot of snapshots without knowing that there is a problem in the snapshot purge unless someone has good dashboard and alarms in place.
created_at: 2025-03-25T17:48:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,6 +165,14 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
         throw new OMException("Snapshot already exists", FILE_ALREADY_EXISTS);
       }
 
+      // Check snapshot limit
+      int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+      if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: peterxcli
body: > 1\. We can't rely on this metric. If a bootstrapped node becomes the leader, it may not have the correct active snapshot count. We should use SnapshotInfoTable size.

I think it still have consistent value, which is guarantee by ratis. 



> 2\. Should we compare it against total snapshots? I'm thinking that if there is an issue in the SnapshotDeletionService, we may accumulate a lot of snapshots without knowing that there is a problem in the snapshot purge unless someone has good dashboard and alarms in place.

Maybe use the size of globalSnapshotChain map in SnapshotChainManager?
created_at: 2025-03-25T18:49:17Z
author_association: MEMBER
diff_hunk: @@ -165,6 +165,14 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
         throw new OMException("Snapshot already exists", FILE_ALREADY_EXISTS);
       }
 
+      // Check snapshot limit
+      int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+      if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: > Maybe use the size of globalSnapshotChain map in SnapshotChainManager? 

Yes, this works too.
created_at: 2025-03-25T19:05:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,6 +165,14 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
         throw new OMException("Snapshot already exists", FILE_ALREADY_EXISTS);
       }
 
+      // Check snapshot limit
+      int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+      if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: SaketaChalamchala
body: Thanks for the patch @peterxcli. Could we move this check up before we take any locks?
created_at: 2025-03-26T16:31:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -264,6 +264,39 @@ public void testEntryExists() throws Exception {
     assertEquals(2, getOmMetrics().getNumSnapshotCreates());
   }
 
+  @Test
+  public void testSnapshotLimit() throws Exception {
+    when(getOzoneManager().isAdmin(any())).thenReturn(true);
+
+    when(getOzoneManager().getFsSnapshotMaxLimit()).thenReturn(1);
+
+    String key1 = getTableKey(getVolumeName(), getBucketName(), snapshotName1);
+
+    OMRequest omRequest =
+        createSnapshotRequest(getVolumeName(), getBucketName(), snapshotName1);
+    OMSnapshotCreateRequest omSnapshotCreateRequest = doPreExecute(omRequest);
+
+    assertNull(getOmMetadataManager().getSnapshotInfoTable().get(key1));
+    omSnapshotCreateRequest.validateAndUpdateCache(getOzoneManager(), 1);
+
+    assertNotNull(getOmMetadataManager().getSnapshotInfoTable().get(key1));
+
+    // Should fail as snapshot limit is 1
+    omRequest = createSnapshotRequest(getVolumeName(), getBucketName(), snapshotName2);
+    omSnapshotCreateRequest = doPreExecute(omRequest);
+    OMClientResponse omClientResponse =
+        omSnapshotCreateRequest.validateAndUpdateCache(getOzoneManager(), 2);
+
+    OMResponse omResponse = omClientResponse.getOMResponse();
+    assertNotNull(omResponse.getCreateSnapshotResponse());
+    assertEquals(OzoneManagerProtocolProtos.Status.INVALID_SNAPSHOT_ERROR,
+        omResponse.getStatus());
+
+    assertEquals(1, getOmMetrics().getNumSnapshotCreateFails());
+    assertEquals(1, getOmMetrics().getNumSnapshotActive());
+    assertEquals(2, getOmMetrics().getNumSnapshotCreates());
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotCreateRequest.java
user: login: SaketaChalamchala
body: Could you also expand the test to delete a snapshot and create one again?
created_at: 2025-03-26T16:33:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,6 +165,14 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
         throw new OMException("Snapshot already exists", FILE_ALREADY_EXISTS);
       }
 
+      // Check snapshot limit
+      int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+      if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: peterxcli
body: > > 2. Should we compare it against total snapshots? I'm thinking that if there is an issue in the SnapshotDeletionService, we may accumulate a lot of snapshots without knowing that there is a problem in the snapshot purge unless someone has good dashboard and alarms in place.
> 
> Maybe use the size of globalSnapshotChain map in SnapshotChainManager?

@SaketaChalamchala Do you have any thoughts on this? Thanks
created_at: 2025-03-27T11:17:17Z
author_association: MEMBER
diff_hunk: @@ -147,6 +147,14 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, Execut
     OzoneManagerProtocolProtos.UserInfo userInfo = getOmRequest().getUserInfo();
     String key = snapshotInfo.getTableKey();
     try {
+      // Check snapshot limit
+      int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+      if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
+        throw new OMException(
+            String.format("Snapshot limit of %d reached. Cannot create more snapshots.", maxSnapshots),
+            OMException.ResultCodes.INVALID_SNAPSHOT_ERROR);
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: 1. Can you please create a new ResultCode `TOO_MANY_SNAPSHOTS`?
2. We should move this check to preExecute() similar to [buckets](https://github.com/apache/ozone/blob/770d3e62144267d2970f19ad29916186bec2b085/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java#L97). 
created_at: 2025-03-28T01:06:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -988,6 +989,9 @@ private void instantiateServices(boolean withNewSnapshot) throws IOException {
     fsSnapshotEnabled = configuration.getBoolean(
         OMConfigKeys.OZONE_FILESYSTEM_SNAPSHOT_ENABLED_KEY,
         OMConfigKeys.OZONE_FILESYSTEM_SNAPSHOT_ENABLED_DEFAULT);
+    fsSnapshotMaxLimit = configuration.getInt(
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
user: login: hemantk-12
body: 1.  Should this config be moved to [OmSnapshotManager](https://github.com/apache/ozone/blob/dc4789772168efe312f7832b6bb3bc3e7bc40d1b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java#L109)? If agree, we can just move `validateSnapshotLimit` to `OmSnapshotManager` as well.
1. Please remove this config from [OmMetadataManagerImpl](https://github.com/apache/ozone/blob/86d2027e328cd66c3319b355f3438842440ffa70/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java#L592) and [DBStoreBuilder](https://github.com/apache/ozone/blob/eb96ff489209ad9dab10a3cded277ae5ccd7cb4a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/DBStoreBuilder.java#L101).
created_at: 2025-03-28T17:58:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -99,11 +99,23 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
     snapshotPath = snapshotInfo.getSnapshotPath();
   }
 
+  private void validateSnapshotLimit(OzoneManager ozoneManager) throws IOException {
+    int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+    OMMetrics omMetrics = ozoneManager.getMetrics();
+    if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: I would prefer SnapshotChain over metric. Please use SnapshotChain.
created_at: 2025-03-28T18:01:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -988,6 +989,9 @@ private void instantiateServices(boolean withNewSnapshot) throws IOException {
     fsSnapshotEnabled = configuration.getBoolean(
         OMConfigKeys.OZONE_FILESYSTEM_SNAPSHOT_ENABLED_KEY,
         OMConfigKeys.OZONE_FILESYSTEM_SNAPSHOT_ENABLED_DEFAULT);
+    fsSnapshotMaxLimit = configuration.getInt(
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
user: login: peterxcli
body: > 2\. Please remove this config from [OmMetadataManagerImpl](https://github.com/apache/ozone/blob/86d2027e328cd66c3319b355f3438842440ffa70/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java#L592) and [DBStoreBuilder](https://github.com/apache/ozone/blob/eb96ff489209ad9dab10a3cded277ae5ccd7cb4a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/DBStoreBuilder.java#L101).

I was planning to clean them in https://issues.apache.org/jira/browse/HDDS-12690.
But if you feel ok, I can make the changes together in this patch.
created_at: 2025-03-29T02:24:04Z
author_association: MEMBER
diff_hunk: @@ -988,6 +989,9 @@ private void instantiateServices(boolean withNewSnapshot) throws IOException {
     fsSnapshotEnabled = configuration.getBoolean(
         OMConfigKeys.OZONE_FILESYSTEM_SNAPSHOT_ENABLED_KEY,
         OMConfigKeys.OZONE_FILESYSTEM_SNAPSHOT_ENABLED_DEFAULT);
+    fsSnapshotMaxLimit = configuration.getInt(
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
user: login: hemantk-12
body: Yes, you can do it as part of [HDDS-12690](https://issues.apache.org/jira/browse/HDDS-12690).
created_at: 2025-03-29T23:29:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -99,11 +99,23 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
     snapshotPath = snapshotInfo.getSnapshotPath();
   }
 
+  private void validateSnapshotLimit(OzoneManager ozoneManager) throws IOException {
+    int maxSnapshots = ozoneManager.getFsSnapshotMaxLimit();
+    OMMetrics omMetrics = ozoneManager.getMetrics();
+    if (omMetrics.getNumSnapshotActive() >= maxSnapshots) {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: peterxcli
body: Done. PTAL.
created_at: 2025-03-31T03:32:55Z
author_association: MEMBER
diff_hunk: @@ -272,11 +273,16 @@ public void testEntryExist() throws Exception {
 
   private OMSnapshotDeleteRequest doPreExecute(
       OMRequest originalRequest) throws Exception {
+    return doPreExecute(originalRequest, getOzoneManager());
+  }
+
+  public static OMSnapshotDeleteRequest doPreExecute(
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotDeleteRequest.java
user: login: hemantk-12
body: Why do we need this change?
created_at: 2025-03-31T16:46:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -228,6 +231,52 @@ public void testCloseOnEviction() throws IOException,
     GenericTestUtils.waitFor(() -> {
       return logCapture.getOutput().contains(msg);
     }, 100, 30_000);
+
+    // clean up all snapshot
+    snapshotChainManager.deleteSnapshot(second);
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: hemantk-12
body: Why do we need to remove these snapshots from the chain? Shouldn't `setupData` take care of it? If not, I'll suggest to move this code to `@BeforeEach` and `@AfterEach`. 

created_at: 2025-03-31T18:14:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -352,6 +402,9 @@ public void testGetSnapshotInfo() throws IOException {
     ome = assertThrows(OMException.class,
         () -> om.getOmSnapshotManager().getSnapshot(s2.getSnapshotId()));
     assertEquals(OMException.ResultCodes.FILE_NOT_FOUND, ome.getResult());
+
+    // clean up
+    snapshotChainManager.deleteSnapshot(s1);
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: hemantk-12
body: Same as previous.
created_at: 2025-03-31T18:15:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -228,6 +231,52 @@ public void testCloseOnEviction() throws IOException,
     GenericTestUtils.waitFor(() -> {
       return logCapture.getOutput().contains(msg);
     }, 100, 30_000);
+
+    // clean up all snapshot
+    snapshotChainManager.deleteSnapshot(second);
+    snapshotChainManager.deleteSnapshot(first);
+    assertEquals(snapshotChainManager.getGlobalSnapshotChain().size(), 0);
+  }
+
+  @Test
+  public void testValidateSnapshotLimit() throws IOException {
+    Table<String, OmVolumeArgs> volumeTable = mock(Table.class);
+    Table<String, OmBucketInfo> bucketTable = mock(Table.class);
+    Table<String, SnapshotInfo> snapshotInfoTable = mock(Table.class);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), VOLUME_TABLE, volumeTable);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), BUCKET_TABLE, bucketTable);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), SNAPSHOT_INFO_TABLE, snapshotInfoTable);
+
+    om.getOmSnapshotManager().setFsSnapshotMaxLimit(2);
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: hemantk-12
body: Is it possible to set it in `OzoneConfig` rather than this way?
created_at: 2025-03-31T18:15:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().validateSnapshotLimit();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: I discussed with @swamirish, it makes more sense to do this check in `validateAndUpdateCache` rather than `preExecute`. Limiting BucketCreation reason is different from SnapshotCreation. Since `validateAndUpdateCache` gives a consistence view, we should do it there. Sorry for the confusion.
cc: @swamirishi 
created_at: 2025-03-31T22:02:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().validateSnapshotLimit();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: swamirishi
body: yeah this validation should move to the validateAndUpdateCacheMethod
created_at: 2025-03-31T23:06:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +856,25 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void validateSnapshotLimit() throws IOException {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: Make this method synchronized and have a parameter in memory which keeps the number of snapshots creation in flight in memory. On hindsight I believe we should not put this check inside validateAndUpdateCache since this can impact ratis replays. @hemantk-12 
The in memory count should be increased once the pre execute passes inside a synchronized block and should be decremented once the snapshot gets added on to the snapshot chain in validateAndUpdate cache. The total number of snapshots at any time would be:
totalSnapshots = snapshotChainManager.size() + inmemoryCounter
created_at: 2025-03-31T23:19:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +856,25 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void validateSnapshotLimit() throws IOException {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: To add more details:
Ratis replay would not be deterministic across various om nodes if the config is not matching across all nodes. We should only look at leader. Since validateAndUpdateCache runs on all nodes, this could lead to an inconsistent state.
created_at: 2025-03-31T23:21:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +856,25 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void validateSnapshotLimit() throws IOException {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: Rename this function addSnapshotLimitCheck() or something
created_at: 2025-03-31T23:23:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().validateSnapshotLimit();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: Based on https://github.com/apache/ozone/pull/8157#discussion_r2021899830 let's leave it here and make other changes.
created_at: 2025-03-31T23:58:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -228,6 +231,52 @@ public void testCloseOnEviction() throws IOException,
     GenericTestUtils.waitFor(() -> {
       return logCapture.getOutput().contains(msg);
     }, 100, 30_000);
+
+    // clean up all snapshot
+    snapshotChainManager.deleteSnapshot(second);
+    snapshotChainManager.deleteSnapshot(first);
+    assertEquals(snapshotChainManager.getGlobalSnapshotChain().size(), 0);
+  }
+
+  @Test
+  public void testValidateSnapshotLimit() throws IOException {
+    Table<String, OmVolumeArgs> volumeTable = mock(Table.class);
+    Table<String, OmBucketInfo> bucketTable = mock(Table.class);
+    Table<String, SnapshotInfo> snapshotInfoTable = mock(Table.class);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), VOLUME_TABLE, volumeTable);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), BUCKET_TABLE, bucketTable);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), SNAPSHOT_INFO_TABLE, snapshotInfoTable);
+
+    om.getOmSnapshotManager().setFsSnapshotMaxLimit(2);
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: peterxcli
body: Yes, but the validation function should read the value from config every time it's called. Do we want it to be that?
created_at: 2025-04-01T03:16:21Z
author_association: MEMBER
diff_hunk: @@ -228,6 +231,52 @@ public void testCloseOnEviction() throws IOException,
     GenericTestUtils.waitFor(() -> {
       return logCapture.getOutput().contains(msg);
     }, 100, 30_000);
+
+    // clean up all snapshot
+    snapshotChainManager.deleteSnapshot(second);
+    snapshotChainManager.deleteSnapshot(first);
+    assertEquals(snapshotChainManager.getGlobalSnapshotChain().size(), 0);
+  }
+
+  @Test
+  public void testValidateSnapshotLimit() throws IOException {
+    Table<String, OmVolumeArgs> volumeTable = mock(Table.class);
+    Table<String, OmBucketInfo> bucketTable = mock(Table.class);
+    Table<String, SnapshotInfo> snapshotInfoTable = mock(Table.class);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), VOLUME_TABLE, volumeTable);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), BUCKET_TABLE, bucketTable);
+    HddsWhiteboxTestUtils.setInternalState(
+        om.getMetadataManager(), SNAPSHOT_INFO_TABLE, snapshotInfoTable);
+
+    om.getOmSnapshotManager().setFsSnapshotMaxLimit(2);
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: hemantk-12
body: Nah, I don't want that. I was thinking if we could load a new config every time a test runs.
created_at: 2025-04-02T04:00:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: With `inFlightSnapshotCount` approach, it should be moved just before the request is submitted to Ratis at line 125. Because if the name or permission check fails, it will never reach `validateAndUpdateCache` to decrement the `inFlightSnapshotCount`.
created_at: 2025-04-03T16:32:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +856,25 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void validateSnapshotLimit() throws IOException {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: hemantk-12
body: This approach has an issue. `preExecute` runs only on a single node (the leader node, if I recall correctly), whereas `validateAndUpdateCache` runs on all nodes. If we decrement in `validateAndUpdateCache`, other nodes will keep decrementing without a corresponding increment, causing the value to go negative. Additionally, if leadership changes, it could result in exceeding the allowed limit.

IMO, we can simplify this by relying on the `SnapshotChain` count, which will act as a soft limit initially and eventually become a hard limit. I doubt that `applyingTransaction` is slower than the write request rate to the extent that it would create a backlog of snapshot creation requests.
created_at: 2025-04-03T16:41:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: peterxcli
body: Should we revert the "inflight count" change? Maybe we can make it to become a real hard limit after migrating to pre-ratis execution?
created_at: 2025-04-03T17:10:56Z
author_association: MEMBER
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: I agree that `inFlightCount` gives a real hard limit, but it comes with race condition problems as mentioned in [this](https://github.com/apache/ozone/pull/8157#discussion_r2027392804) comment.
created_at: 2025-04-03T17:30:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: peterxcli
body: @hemantk-12 @swamirishi Would you like me to revert the "inflight count" changes?
If you have any other idea on this, please also let me know, Thanks!
created_at: 2025-04-04T08:32:06Z
author_association: MEMBER
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: I'm OK with keeping the SnapshotChain-based approach and using it as an eventual limit rather than increasing the complexity with `inFlightCount` approach.

Even if `preExecute` runs only on a leader node and the count is decremented on the leader node, the issue persists in case of Ratis leadership change and log replay.
created_at: 2025-04-07T20:59:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: SaketaChalamchala
body: I agree with @hemantk-12. We need not reject a snapshot create because of inFlight Snapshot create requests. inFlight snapshot creates may not always result in a snapshot being created and added to the chain. 
created_at: 2025-04-07T23:37:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: peterxcli
body: Reverted the change.
created_at: 2025-04-08T03:29:32Z
author_association: MEMBER
diff_hunk: @@ -113,16 +119,37 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
   void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: hemantk-12
body: nit: we should delete snapshots and wait for purge.
created_at: 2025-04-08T21:21:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -113,16 +119,37 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
   void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: peterxcli
body: Do you mean calling `SnapshotDiffCleanupService#run`?
created_at: 2025-04-09T01:34:41Z
author_association: MEMBER
diff_hunk: @@ -113,16 +119,37 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
   void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: peterxcli
body: Or `invalidateAllCache` ?
created_at: 2025-04-09T07:11:11Z
author_association: MEMBER
diff_hunk: @@ -113,16 +119,37 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
   void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: hemantk-12
body: No, currently you are just deleting entries directly from the SnapshotInfoTable and chain. Ideally, we should issue a delete snapshot (like [this](https://github.com/apache/ozone/blob/master/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOmSnapshot.java#L1820)) for all the snapshots, and wait for `SnapshotInfoTable` to be empty.
created_at: 2025-04-09T18:53:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +856,25 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void validateSnapshotLimit() throws IOException {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: @hemantk-12 We can ignore the inflight counter on the followers. We would be just using this counter on the leader. We don't have to be worried about the value in the followers. On leader switch this number can be reset to 0 again.
created_at: 2025-04-09T20:08:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -856,23 +858,31 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
-  public void validateSnapshotLimit() throws IOException {
+  /*
+   * Check snapshot limit
+   * Note: This method increments the snapshot in-flight counter
+   */
+  public synchronized void snapshotLimitCheck() throws IOException {
     OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl)
         ozoneManager.getMetadataManager();
     SnapshotChainManager snapshotChainManager =
         omMetadataManager.getSnapshotChainManager();
     int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
-    if (currentSnapshotNum >= fsSnapshotMaxLimit) {
+    int inFlightCount = inFlightSnapshotCount.get();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: @peterxcli We should do this entire validation inside inFlightCount.update() block
created_at: 2025-04-09T20:11:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +856,25 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void validateSnapshotLimit() throws IOException {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: hemantk-12
body: Yes, we can ignore the inflight counter on the followers and start from `0`, but if there is a leadership change while `preExecute` has completed and `validateAndUpdateCache` has not. When a new leader (former follower) tries to apply the transactions (basically executing `validateAndUpdateCache`), `inflight` counter will go negative. 
created_at: 2025-04-09T20:42:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -113,16 +119,37 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
   void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: peterxcli
body: If we want to purge the snapshot through `ObjectStore#deleteSnapsot`, we have to set the mock `volumeTable` and `bucketTable` for each `OmSnapshot` to make the deleted keys scanning of `SnapshotDeletionService` work. 
https://github.com/apache/ozone/blob/3cb97fecdd167a99eb86193c480220bc31f15a35/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java#L192-L199

And I had looked through the behaviours in `TestOmSnapshotManager`, it seems they all do direct operation on snapshotChain or the table. So I think it's ok to do cleanup via direct public function calling on them, and also make the cleanup logic explicit.
created_at: 2025-04-11T12:48:15Z
author_association: MEMBER
diff_hunk: @@ -113,16 +119,37 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
   void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: peterxcli
body: @hemantk-12 What's your thoughts on this. Thanks.
created_at: 2025-04-11T12:49:03Z
author_association: MEMBER
diff_hunk: @@ -113,16 +119,37 @@ void init(@TempDir File tempDir) throws Exception {
         OMConfigKeys.OZONE_OM_SNAPSHOT_CACHE_MAX_SIZE, 1);
     configuration.setBoolean(
         OMConfigKeys.OZONE_OM_SNAPSHOT_ROCKSDB_METRICS_ENABLED, false);
+    // Allow 2 fs snapshots
+    configuration.setInt(
+        OMConfigKeys.OZONE_OM_FS_SNAPSHOT_MAX_LIMIT, 2);
 
     OmTestManagers omTestManagers = new OmTestManagers(configuration);
     om = omTestManagers.getOzoneManager();
+    omMetadataManager = (OmMetadataManagerImpl) om.getMetadataManager();
+    omSnapshotManager = om.getOmSnapshotManager();
+    snapshotChainManager = omMetadataManager.getSnapshotChainManager();
   }
 
   @AfterAll
   void stop() {
     om.stop();
   }
 
+  @AfterEach
+  void cleanup() throws IOException {
+    Table<String, SnapshotInfo> snapshotInfoTable = omMetadataManager.getSnapshotInfoTable();
path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
user: login: hemantk-12
body: I guess it is OK to directly delete it from the test.
created_at: 2025-04-11T16:42:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +856,25 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void validateSnapshotLimit() throws IOException {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: hemantk-12
body: @swamirishi and I were discussing this, and one way to fix it is to implement `notifyLeaderReady()` and `notifyNotLeader()` similar to [SCMStateMachine](https://github.com/apache/ozone/blob/7dfd8c1a7889b5947c87f81d1c4d24a30bf629f6/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java#L372) for OzoneManager and reset the `inflight` counter on leadership change.

cc: @swamirishi @SaketaChalamchala 
created_at: 2025-04-11T16:48:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -345,6 +345,29 @@ public void testKeyAndSnapshotDeletionService() throws IOException, InterruptedE
     checkSnapshotIsPurgedFromDB(omFollower, tableKey);
   }
 
+  @Test
+  public void testSnapshotInFlightCount() throws Exception {
+    // snapshot inflight count should be reset to 0 when leader changes
+
+    // first do some snapshot creations
+    String snapshotName1 = UUID.randomUUID().toString();
+    store.createSnapshot(volumeName, bucketName, snapshotName1);
+
+    // then shutdown the leader
+    OzoneManager omLeader = cluster.getOMLeader();
+    cluster.shutdownOzoneManager(omLeader);
+
+    // wait for the new leader to be elected
+    cluster.waitForLeaderOM();
+
+    // check the inflight count on the new leader is 0
+    OzoneManager newLeader = cluster.getOMLeader();
+    assertEquals(0, newLeader.getOmSnapshotManager().getInFlightSnapshotCount());
path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneManagerHASnapshot.java
user: login: peterxcli
body: I have check that if I comment out the reset logic, this test would failed with:
```java
ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 112.5 s <<< FAILURE! -- in org.apache.hadoop.ozone.om.snapshot.TestOzoneManagerHASnapshot
[ERROR] org.apache.hadoop.ozone.om.snapshot.TestOzoneManagerHASnapshot.testSnapshotInFlightCount -- Time elapsed: 1.501 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <0> but was: <-24>
        at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
        at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
        at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
        at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
        at org.apache.hadoop.ozone.om.snapshot.TestOzoneManagerHASnapshot.testSnapshotInFlightCount(TestOzoneManagerHASnapshot.java:365)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
```
created_at: 2025-04-15T11:30:33Z
author_association: MEMBER
diff_hunk: @@ -849,6 +858,40 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
+    int oldCount = inFlightSnapshotCount.get();
+    int newCount = inFlightSnapshotCount.updateAndGet(count -> {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: hemantk-12
body: Since we are using two steps, get and update, we may throw an exception for a few snapshot requests due to the case condition between get and update. It is not a big deal, but we can avoid it by putting it inside a synchronized block on `inFlightSnapshotCount` object.
created_at: 2025-04-22T20:10:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,6 +175,9 @@ public final class OmSnapshotManager implements AutoCloseable {
   // Soft limit of the snapshot cache size.
   private final int softCacheSize;
 
+  private int fsSnapshotMaxLimit;
+  private final AtomicInteger inFlightSnapshotCount = new AtomicInteger(0);
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: jojochuang
body: I'm new to this. What does inFlightSnapshotCount count?
created_at: 2025-04-23T23:34:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +858,40 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
+    int oldCount = inFlightSnapshotCount.get();
+    int newCount = inFlightSnapshotCount.updateAndGet(count -> {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: jojochuang
body: agreed this section is prone to race conditions.

Semaphore would be a more appropriate solution, but it will be require a larger rewrite.
created_at: 2025-04-23T23:44:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,6 +175,9 @@ public final class OmSnapshotManager implements AutoCloseable {
   // Soft limit of the snapshot cache size.
   private final int softCacheSize;
 
+  private int fsSnapshotMaxLimit;
+  private final AtomicInteger inFlightSnapshotCount = new AtomicInteger(0);
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: hemantk-12
body: `inFlightSnapshotCount` is the count of snapshot transactions in Ratis log that have not been applied yet.
created_at: 2025-04-24T06:25:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +858,40 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
+    int oldCount = inFlightSnapshotCount.get();
+    int newCount = inFlightSnapshotCount.updateAndGet(count -> {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: @peterxcli Why can't we directly have an AtomicReference<OMException> outside the block? This can be just set to the exception that needs to be thrown if at all it needs to be thrown
with a null check it can be checked outside of the getAndUpdate
created_at: 2025-04-25T06:34:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -849,6 +858,40 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
+    int oldCount = inFlightSnapshotCount.get();
+    int newCount = inFlightSnapshotCount.updateAndGet(count -> {
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: We wouldn't need the oldCount and newCount with the above proposed change
created_at: 2025-04-25T06:35:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,8 @@ public OMSnapshotCreateRequest(OMRequest omRequest) {
   @RequireSnapshotFeatureState(true)
   public OMRequest preExecute(OzoneManager ozoneManager) throws IOException {
     final OMRequest omRequest = super.preExecute(ozoneManager);
+    // verify snapshot limit
+    ozoneManager.getOmSnapshotManager().snapshotLimitCheck();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
user: login: hemantk-12
body: We should move this after all other validations, before line 125. Otherwise, inflightCount will never be decreased if snapshot name validation or permission check fails.
created_at: 2025-04-25T19:54:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -859,6 +869,42 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: @peterxcli Can't we call this inside inflightSnapshotCount.updateAndGet() function that would make this threadsafe and consistent.
created_at: 2025-05-02T17:14:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -859,6 +869,42 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: swamirishi
body: I was off the previous week so couldn't review this PR in time. Can you please make this change in a follow up jira.
created_at: 2025-05-02T17:15:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -859,6 +869,42 @@ private void validateSnapshotsExistAndActive(final String volumeName,
     checkSnapshotActive(toSnapInfo, false);
   }
 
+  public void snapshotLimitCheck() throws IOException, OMException {
+    OmMetadataManagerImpl omMetadataManager = (OmMetadataManagerImpl) ozoneManager.getMetadataManager();
+    SnapshotChainManager snapshotChainManager = omMetadataManager.getSnapshotChainManager();
+    int currentSnapshotNum = snapshotChainManager.getGlobalSnapshotChain().size();
path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
user: login: peterxcli
body: Sure and thanks for the review! Created [HDDS-12952](https://issues.apache.org/jira/browse/HDDS-12952) for this.
created_at: 2025-05-02T17:22:20Z
author_association: MEMBER

PR comments:
user: login: peterxcli
created_at: 2025-03-25T15:05:21Z
body: > The value of the above property is passed to RDBStore but it is never used.

https://issues.apache.org/jira/browse/HDDS-12690
author_association: MEMBER
user: login: peterxcli
created_at: 2025-03-25T15:06:14Z
body: @szetszwo @SaketaChalamchala please take a look if you have time. Thanks!
author_association: MEMBER
user: login: hemantk-12
created_at: 2025-03-25T19:05:44Z
body: @SaketaChalamchala can you please take a look?
author_association: CONTRIBUTOR
user: login: peterxcli
created_at: 2025-03-28T05:49:43Z
body: @hemantk-12 Do you know how to sync the changes in `interface-client` proto file to `proto.lock`?
author_association: MEMBER
user: login: adoroszlai
created_at: 2025-03-28T05:53:58Z
body: > Do you know how to sync the changes in `interface-client` proto file to `proto.lock`?

You shouldn't.  `proto.lock` captures the previous release's state, so it's updated only before releasing Ozone.  It is used as a reference to check for incompatible changes.
author_association: CONTRIBUTOR
user: login: peterxcli
created_at: 2025-03-28T05:55:05Z
body: > > Do you know how to sync the changes in `interface-client` proto file to `proto.lock`?
> 
> You shouldn't. `proto.lock` captures the previous release's state, so it's updated only before releasing Ozone. It is used as a reference to check for incompatible changes.

Got it. Thanks @adoroszlai!
author_association: MEMBER
user: login: swamirishi
created_at: 2025-04-11T23:44:21Z
body: @peterxcli Had a discussion with @hemantk-12 we should go ahead with enforcing hard limits on the number of snapshots. For that the number of snapshot creates inflight would have to be considered. This counter should be only relevant on leader. So on every leader switch this counter needs to be reset to zero, which can be detected by implementing 
NotifyLeaderChanged(I am not really sure if we need this, maybe it could be helpful resetting the counters in the followers @hemantk-12 ) & notifyLeaderReady(The counter might go into a negative value if raft logs are being replayed and are just running validateAndUpdateCache)
https://github.com/apache/ratis/blob/c81902a094344d953f2fc5f5d7ed08dc7c171cdf/ratis-server-api/src/main/java/org/apache/ratis/statemachine/StateMachine.java#L215-L218 
https://github.com/apache/ratis/blob/c81902a094344d953f2fc5f5d7ed08dc7c171cdf/ratis-server-api/src/main/java/org/apache/ratis/statemachine/StateMachine.java#L310

for ozoneManagerStateMachine

https://github.com/apache/ozone/blob/dc9952e446940b82b277a7e8c9d02239a8417a06/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java#L92
author_association: CONTRIBUTOR
user: login: swamirishi
created_at: 2025-04-11T23:48:47Z
body: > @peterxcli Had a discussion with @hemantk-12 we should go ahead with enforcing hard limits on the number of snapshots. For that the number of snapshot creates inflight would have to be considered. This counter should be only relevant on leader. So on every leader switch this counter needs to be reset, which can be detected by implementing

As a side note we can actually implement a notifier similar to what SCM has to notify status of OM has changed, maybe resuing the same interface is also a good idea. What do you think @hemantk-12 ? 
https://github.com/apache/ozone/blob/0e109d19eaaf82f8c5220d506c4b3d3a2ab03899/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMService.java#L27-L31
author_association: CONTRIBUTOR
user: login: peterxcli
created_at: 2025-04-15T06:43:49Z
body: @hemantk-12 @swamirishi 
Just came up one question: Does `notifyStatusChanged` happen before any request applied on new leader node and also executed in serial with other ratis logs?
author_association: MEMBER
user: login: peterxcli
created_at: 2025-04-15T08:36:02Z
body: Ok I think `StartupLogEntry` provide this guarantee.
author_association: MEMBER
user: login: peterxcli
created_at: 2025-04-15T11:56:32Z
body: > which can be detected by implementing
> NotifyLeaderChanged(I am not really sure if we need this, maybe it could be helpful resetting the counters in the followers

@swamirishi I think `notifyLeaderReady` is enough at this time. If we need more logic about follower/leader in future we can then migrate the reset logic to `notifyLeaderChanged`.

Using `notifyLeaderReady` also make the code cleaner.
author_association: MEMBER
user: login: peterxcli
created_at: 2025-04-25T06:31:48Z
body: > agreed this section is prone to race conditions.
Semaphore would be a more appropriate solution, but it will be require a larger rewrite.

> Agreed, this section is prone to race conditions.  
A semaphore would be a more appropriate solution, but it would require a larger rewrite.

~~@jojochuang I have refactored that section by putting it into a synchronized block. I think it now has the same effect as a `Semaphore`:~~

Changed to use atomicReference to handle the exception and place the increment logic within the `AtomicReference#updateAndGet` block
author_association: MEMBER
user: login: hemantk-12
created_at: 2025-04-28T18:19:26Z
body: @swamirishi can you please take another look?
author_association: CONTRIBUTOR
user: login: jojochuang
created_at: 2025-05-01T20:30:54Z
body: BTW ozone.om.fs.snapshot.max.limit of 1000 is probably too low. We have users at a few thousand snapshots already and will scale up even further. It looks like 1000 is not a problem. We should raise it a lot higher. That's a separate task.

EDIT: opened https://issues.apache.org/jira/browse/HDDS-12948 to track this task
author_association: CONTRIBUTOR
user: login: peterxcli
created_at: 2025-05-02T06:45:55Z
body: Thanks @hemantk-12, @swamirishi, @jojochuang and @SaketaChalamchala for the review, and @adoroszlai for the clarification of my  `proto.lock` question!
author_association: MEMBER
user: login: adoroszlai
created_at: 2025-05-02T07:27:18Z
body: @jojochuang @peterxcli This caused [build failure](https://github.com/apache/ozone/actions/runs/14790150556/job/41525587482#step:16:24) on `master` due to other recent changes around OM metadata classes.  The compile error can be fixed by simply adding the required import, but the test is failing after that.  Thus I had to revert this change, sorry.

As a sanity check, I recommend running local build and checkstyle/pmd before merging PRs.

```
gh pr checkout 8157
git merge --no-commit origin/master
mvn -DskipRecon -DskipShade -DskipTests clean verify
hadoop-ozone/dev-support/checks/checkstyle.sh
hadoop-ozone/dev-support/checks/pmd.sh
```

(Note: check exit code of pmd, Maven always shows success.)

When there are related changes on `master`, I also recommend merging it into the PR branch to trigger complete CI.
author_association: CONTRIBUTOR