Repository: floci-io/floci (2026-07-23T13:35:52Z)

Issue:
[BUG] Duplicate storage backends clobber persisted state on graceful shutdown (DynamoDB table definitions lost)
**Version:** reproduced on 1.5.31 and 1.5.33 (deterministic on 1.5.33). All durable storage modes affected (`persistent`, `hybrid`, `wal`).

## Symptom

After a clean `docker stop`, `dynamodb-items.json` is intact but `dynamodb-tables.json` retains only tables written *after* some point in the process lifetime (in our stack: 5–7 of 63 table definitions survive; `hybrid` flushed 0). On restart every DynamoDB Streams event source mapping references a stale stream ARN and the poller logs `Stream not found` every second.

`docker kill` (skipping graceful shutdown) preserves all 63 table definitions under `persistent` mode — the graceful shutdown path is what loses the data.

## Root cause (from reading main)

`StorageFactory.create()` builds a new backend per call and never dedupes by file path, so two components resolving the same file get independent instances with independent in-memory stores (`allBackends` collects them all; each backend's `load()` also runs twice — once in `create()`, again in `loadAll()`). At shutdown, `EmulatorLifecycle.onStop()` → `storageFactory.flushAll()` iterates `allBackends`; a stale duplicate flushes after the live one and rewrites the file from its outdated store.

`persistent` mode appeared safe before 1.5.30 only because the flush ran after container teardown and was usually SIGKILLed at the grace deadline; moving the flush first (1.5.30, "flush persisted state before container teardown on shutdown") made the clobber deterministic.

## Repro

1. Fresh data dir, `FLOCI_STORAGE_MODE=persistent` (or `hybrid`/`wal`)
2. Create ~60 DynamoDB tables via the API
3. Let a second component touch the table map later (e.g. deploy a Lambda with a DynamoDB Streams ESM, or create a few more tables minutes later)
4. `docker stop`
5. Inspect `dynamodb-tables.json`: only the late-touched tables remain

Second stop/start cycles are stable (all instances load the same complete file at boot), which is why the loss only strikes the first stop after table creation.

## Suggested fix

Cache backends by resolved file path in `StorageFactory.create()` (or refuse duplicate registration), and drop the redundant `load()` double-call.


Discussion:

Patch:
diff --git a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
index c2a04d7c29..0f9e4fb742 100644
--- a/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
+++ b/src/main/java/io/github/hectorvent/floci/core/storage/StorageFactory.java
@@ -11,6 +11,7 @@
 
 import java.nio.file.Path;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 
@@ -28,6 +29,9 @@ public class StorageFactory {
     private final EmulatorConfig config;
     private final ServiceConfigAccess serviceConfigAccess;
     private final List<StorageBackend<?, ?>> allBackends = new ArrayList<>();
+    // A file path identifies one logical store: callers sharing a path are expected to agree on
+    // its value type and storage mode. The first create() wins; repeat calls reuse that backend.
+    private final Map<Path, StorageBackend<?, ?>> backendsByPath = new HashMap<>();
     private final List<HybridStorage<?, ?>> hybridBackends = new ArrayList<>();
     private final List<WalStorage<?, ?>> walBackends = new ArrayList<>();
 
@@ -50,13 +54,24 @@ public StorageFactory(EmulatorConfig config, ServiceConfigAccess serviceConfigAc
      * @param fileName      the JSON file name for persistent storage
      * @param typeReference Jackson type reference for deserialization
      */
-    public <V> StorageBackend<String, V> create(String serviceName, String fileName,
+    public synchronized <V> StorageBackend<String, V> create(String serviceName, String fileName,
                                                  TypeReference<Map<String, V>> typeReference) {
         String mode = resolveMode(serviceName);
         long flushInterval = resolveFlushInterval(serviceName);
         Path basePath = Path.of(config.storage().persistentPath());
         Path filePath = basePath.resolve(fileName);
 
+        // Reuse an existing backend for the same file. Handing out a second backend bound to the
+        // same path creates a duplicate in-memory store; on shutdown the stale duplicate flushes
+        // after the active instance and clobbers persisted state (issue #1921).
+        StorageBackend<?, ?> existing = backendsByPath.get(filePath);
+        if (existing != null) {
+            LOG.debugv("Reusing existing {0} storage for service {1} (file: {2})", mode, serviceName, filePath);
+            @SuppressWarnings("unchecked")
+            StorageBackend<String, V> typed = (StorageBackend<String, V>) existing;
+            return typed;
+        }
+
         LOG.debugv("Creating {0} storage for service {1} (file: {2})", mode, serviceName, filePath);
 
         StorageBackend<String, V> inner = switch (mode) {
@@ -83,25 +98,26 @@ public <V> StorageBackend<String, V> create(String serviceName, String fileName,
         AccountAwareStorageBackend<V> backend = new AccountAwareStorageBackend<>(
                 inner, requestContextInstance, config.defaultAccountId());
         allBackends.add(backend);
+        backendsByPath.put(filePath, backend);
         return backend;
     }
 
     /** Load all storage backends from disk. */
-    public void loadAll() {
+    public synchronized void loadAll() {
         for (StorageBackend<?, ?> backend : allBackends) {
             backend.load();
         }
     }
 
     /** Flush all storage backends to disk. */
-    public void flushAll() {
+    public synchronized void flushAll() {
         for (StorageBackend<?, ?> backend : allBackends) {
             backend.flush();
         }
     }
 
     /** Clear all storage backends. */
-    public void clearAll() {
+    public synchronized void clearAll() {
         for (StorageBackend<?, ?> backend : allBackends) {
             backend.clear();
         }
@@ -109,7 +125,7 @@ public void clearAll() {
     }
 
     /** Shutdown all managed backends (stop schedulers, close connections). */
-    public void shutdownAll() {
+    public synchronized void shutdownAll() {
         for (HybridStorage<?, ?> hybrid : hybridBackends) {
             hybrid.shutdown();
         }


Test patch:
diff --git a/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryDuplicateBackendTest.java b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryDuplicateBackendTest.java
new file mode 100644
index 0000000000..ea55e886e0
--- /dev/null
+++ b/src/test/java/io/github/hectorvent/floci/core/storage/StorageFactoryDuplicateBackendTest.java
@@ -0,0 +1,79 @@
+package io.github.hectorvent.floci.core.storage;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import io.quarkus.test.junit.QuarkusTest;
+import io.quarkus.test.junit.QuarkusTestProfile;
+import io.quarkus.test.junit.TestProfile;
+import jakarta.inject.Inject;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Reproduces issue #1921: two components request the same storage file, the factory hands out
+ * two independent backends, and on graceful shutdown the stale duplicate flushes last and
+ * clobbers the persisted state written by the active instance.
+ */
+@QuarkusTest
+@TestProfile(StorageFactoryDuplicateBackendTest.PersistentProfile.class)
+class StorageFactoryDuplicateBackendTest {
+
+    private static final String STORAGE_PATH = "/tmp/floci-dedupe-backend-test";
+    private static final String FILE_NAME = "dynamodb-tables.json";
+
+    @Inject
+    StorageFactory storageFactory;
+
+    @BeforeEach
+    void reset() throws IOException {
+        // StorageFactory is an application-scoped singleton, so a backend for this file may already
+        // exist and hold in-memory state from app startup. Clear all backends (wipes memory + disk),
+        // then remove the file so both create() calls below start from an empty store.
+        storageFactory.clearAll();
+        Files.deleteIfExists(Path.of(STORAGE_PATH, FILE_NAME));
+    }
+
+    @Test
+    void duplicateBackendsForSameFileDoNotClobberOnFlush() {
+        TypeReference<Map<String, String>> typeRef = new TypeReference<>() {};
+
+        // Active component (e.g. DynamoDbService) opens the tables file.
+        StorageBackend<String, String> active = storageFactory.create("dynamodb", FILE_NAME, typeRef);
+        // A second component (e.g. DynamoDbStreamService) opens the same file later.
+        storageFactory.create("dynamodb", FILE_NAME, typeRef);
+
+        // Active component writes tables; persistent mode write-throughs to disk.
+        for (int i = 0; i < 60; i++) {
+            active.put("table-" + i, "definition-" + i);
+        }
+
+        // Graceful shutdown flushes every registered backend. The stale duplicate must not
+        // overwrite the active instance's data.
+        storageFactory.flushAll();
+
+        // Read the persisted file back with an independent reader.
+        PersistentStorage<String, String> reader =
+                new PersistentStorage<>(Path.of(STORAGE_PATH, FILE_NAME), typeRef);
+        reader.load();
+
+        assertEquals(60, reader.keys().size(),
+                "persisted table definitions should survive flushAll despite the duplicate backend");
+    }
+
+    public static final class PersistentProfile implements QuarkusTestProfile {
+        @Override
+        public Map<String, String> getConfigOverrides() {
+            return Map.of(
+                    "floci.storage.mode", "memory",
+                    "floci.storage.services.dynamodb.mode", "persistent",
+                    "floci.storage.persistent-path", STORAGE_PATH
+            );
+        }
+    }
+}