Repository: TraceMachina/nativelink (language: rust)

Issue:
[Discussion] How to Improve the Readability and Composability of the NativeLink Config JSON part 1
Currently, one area of confusion stems from the fact that we accept user-defined keys (in other words, dynamic keys) for the names of stores and other entities. It's fine to have user-defined names, but they should be anchored to a non-dynamic key, and added as fields. Here is an example config in our current state:

"AC_MAIN_STORE" and "FS_CONTENT_STORE" are both user-supplied values.


```json5
{
  "stores": {
    "AC_MAIN_STORE": {
        "filesystem": {}
    },
    "FS_CONTENT_STORE": {
        "filesystem": {}
    }
  }
}
```
Below is an improved version of the same config where:

- the name is dynamic but the key is always name
- the stores objects are nested in an array because multiple objects should be in an array

  - and must be when they share a key (e.g. "name")
- the details about each store are embedded in a self-describing key called `options`. 
 
The sum of all these changes results in many improvements. 

```json5
{
  "stores": [
    {
      "name": "AC_MAIN_STORE",
      "options": {
        "filesystem": {
            "content_path": "/tmp/nativelink/data-worker-test/content_path-ac",
            "temp_path": "/tmp/nativelink/data-worker-test/tmp_path-ac"
            ...
        }
      }
    },
    {
      "name": "FS_CONTENT_STORE",
      "options": {
        "filesystem": {}
      }
    }
  ]
}
```


Discussion / hints:
I like this. It reminds me of how such configs would look like in YAML.

Some ideas (which effectively can be compiled down to json):

*GHA style* (seems nice and concise initially, but the more I think about it the less convinced I get as it seems very easy to mess up when configs become more complex):

```yaml
---
stores:
  - name: ac-main-store-fs
    uses: filesystem-store
    with:
      contentPath: /tmp/nativelink/data-worker-test/content_path-ac
      tempPath: /tmp/nativelink/data-worker-test/tmp_path-ac
  - name: ac-main-store-memory
    uses: memory-store
    with:
      maxSize: 20G
  - name: ac-main-store
    uses: fast-slow-store
    with:
      fast: ac-main-store-memory
      slow: ac-main-store-fs
```

*K8s style* (seems overly verbose initially, but the more I think about it the more I like it as it provides a clear, familiar structure and a standardized way to iterate on the API):

```yaml
---
apiVersion: nativelink/v1alpha1
kind: FilesystemStore
metadata:
  name: ac-main-store-fs
spec:
  contentPath: /tmp/nativelink/data-worker-test/content_path-ac
  tempPath: /tmp/nativelink/data-worker-test/tmp_path-ac
---
apiVersion: nativelink/v1alpha1
kind: MemoryStore
metadata:
  name: ac-main-store-memory
spec:
  maxSize: 20G
---
apiVersion: nativelink/v1alpha1
kind: FastSlowStore
metadata:
  name: ac-main-store
spec:
  fast: ac-main-store-memory
  slow: ac-main-store-fs
```
I thought about this for a long time and the only possible downside of the new format is that lookup in the vector format (example 2) changes from direct access to O(n) time. However, given that we will never have more than 5 or 10 stores in NativeLink, I don't see this as a material concern. In the worst case scenario, 100 stores would still not cause any negligible performance problems. 
In this case the Big-O complexity doesn't change. We have to translate these configs into the implementation which requires iterating through them all at startup time. Internally it'll still hold them as a `HashMap`.

I'm fine with this, the only bikeshed I have is `"options"`. I would like to find a more intuitive name... maybe `"config"` or something?
I'd like to work on this issue.
I've questions while I was working on this issue.
I just had felt that `schedulers` had the same issue as `stores` and I think it needs to updated as well.

Before:

```
"schedulers": {
    "MAIN_SCHEDULER": {
      "simple": {
        "supported_platform_properties": {
          "cpu_count": "minimum",
          "memory_kb": "minimum",
          ...
        }
      }
    }
  }
```

After:

```
"schedulers": [
    "name": "MAIN_SCHEDULER",
    "config": {
      "simple": {
        "supported_platform_properties": {
          "cpu_count": "minimum",
          "memory_kb": "minimum",
          ...
        }
      }
    }
  ]
```

Do we need this change in this PR?

cc: @MarcusSorealheis , @aaronmondal , @allada , @adam-singer 

Okay, I already made PR for both `stores` and `schedulers`.

https://github.com/TraceMachina/nativelink/pull/889

Fix patch:
diff --git a/Cargo.lock b/Cargo.lock
index 48b59ab7c..956d2180b 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2011,6 +2011,7 @@ dependencies = [
  "humantime",
  "pretty_assertions",
  "serde",
+ "serde_json",
  "serde_json5",
  "shellexpand",
 ]
diff --git a/deployment-examples/docker-compose/local-storage-cas.json5 b/deployment-examples/docker-compose/local-storage-cas.json5
index 273b7fafc..5dd31d87c 100644
--- a/deployment-examples/docker-compose/local-storage-cas.json5
+++ b/deployment-examples/docker-compose/local-storage-cas.json5
@@ -4,8 +4,9 @@
 // so objects are compressed, deduplicated and uses some in-memory
 // optimizations for certain hot paths.
 {
-  "stores": {
-    "CAS_MAIN_STORE": {
+  "stores": [
+    {
+      "name": "CAS_MAIN_STORE",
       "compression": {
         "compression_algorithm": {
           "lz4": {}
@@ -21,8 +22,8 @@
           }
         }
       }
-    },
-    "AC_MAIN_STORE": {
+    }, {
+      "name": "AC_MAIN_STORE",
       "filesystem": {
         "content_path": "~/.cache/nativelink/content_path-ac",
         "temp_path": "~/.cache/nativelink/tmp_path-ac",
@@ -32,7 +33,7 @@
         }
       }
     }
-  },
+  ],
   "servers": [{
     "listener": {
       "http": {
diff --git a/deployment-examples/docker-compose/scheduler.json5 b/deployment-examples/docker-compose/scheduler.json5
index c04f20d3f..14eaa15f8 100644
--- a/deployment-examples/docker-compose/scheduler.json5
+++ b/deployment-examples/docker-compose/scheduler.json5
@@ -1,6 +1,7 @@
 {
-  "stores": {
-    "GRPC_LOCAL_STORE": {
+  "stores": [
+    {
+      "name": "GRPC_LOCAL_STORE",
       // Note: This file is used to test GRPC store.
       "grpc": {
         "instance_name": "",
@@ -9,8 +10,8 @@
         ],
         "store_type": "cas"
       }
-    },
-    "GRPC_LOCAL_AC_STORE": {
+    }, {
+      "name": "GRPC_LOCAL_AC_STORE",
       // Note: This file is used to test GRPC store.
       "grpc": {
         "instance_name": "",
@@ -20,9 +21,10 @@
         "store_type": "ac"
       }
     }
-  },
-  "schedulers": {
-    "MAIN_SCHEDULER": {
+  ],
+  "schedulers": [
+    {
+      "name": "MAIN_SCHEDULER",
       "simple": {
         "supported_platform_properties": {
           "cpu_count": "minimum",
@@ -32,7 +34,7 @@
         }
       }
     }
-  },
+  ],
   "servers": [{
     "listener": {
       "http": {
diff --git a/deployment-examples/docker-compose/worker.json5 b/deployment-examples/docker-compose/worker.json5
index 09df99e60..57cc5ad1d 100644
--- a/deployment-examples/docker-compose/worker.json5
+++ b/deployment-examples/docker-compose/worker.json5
@@ -1,6 +1,7 @@
 {
-  "stores": {
-    "GRPC_LOCAL_STORE": {
+  "stores": [
+    {
+      "name": "GRPC_LOCAL_STORE",
       // Note: This file is used to test GRPC store.
       "grpc": {
         "instance_name": "",
@@ -9,8 +10,8 @@
         ],
         "store_type": "cas"
       }
-    },
-    "GRPC_LOCAL_AC_STORE": {
+    }, {
+      "name": "GRPC_LOCAL_AC_STORE",
       // Note: This file is used to test GRPC store.
       "grpc": {
         "instance_name": "",
@@ -19,8 +20,8 @@
         ],
         "store_type": "ac"
       }
-    },
-    "WORKER_FAST_SLOW_STORE": {
+    }, {
+      "name": "WORKER_FAST_SLOW_STORE",
       "fast_slow": {
         "fast": {
           "filesystem": {
@@ -39,7 +40,7 @@
         }
       }
     }
-  },
+  ],
   "workers": [{
     "local": {
       "worker_api_endpoint": {
diff --git a/kubernetes/components/worker/worker.json5 b/kubernetes/components/worker/worker.json5
index 284a6beef..3a452b1aa 100644
--- a/kubernetes/components/worker/worker.json5
+++ b/kubernetes/components/worker/worker.json5
@@ -1,6 +1,7 @@
 {
-  "stores": {
-    "GRPC_LOCAL_STORE": {
+  "stores": [
+    {
+      "name": "GRPC_LOCAL_STORE",
       // Note: This file is used to test GRPC store.
       "grpc": {
         "instance_name": "",
@@ -9,8 +10,8 @@
         ],
         "store_type": "cas"
       }
-    },
-    "GRPC_LOCAL_AC_STORE": {
+    }, {
+      "name": "GRPC_LOCAL_AC_STORE",
       // Note: This file is used to test GRPC store.
       "grpc": {
         "instance_name": "",
@@ -19,8 +20,8 @@
         ],
         "store_type": "ac"
       }
-    },
-    "WORKER_FAST_SLOW_STORE": {
+    }, {
+      "name": "WORKER_FAST_SLOW_STORE",
       "fast_slow": {
         "fast": {
           "filesystem": {
@@ -38,7 +39,7 @@
         }
       }
     }
-  },
+  ],
   "workers": [{
     "local": {
       "worker_api_endpoint": {
diff --git a/kubernetes/nativelink/nativelink-config.json5 b/kubernetes/nativelink/nativelink-config.json5
index e06d78269..156ab3425 100644
--- a/kubernetes/nativelink/nativelink-config.json5
+++ b/kubernetes/nativelink/nativelink-config.json5
@@ -2,8 +2,9 @@
 // `~/.cache/nativelink`. When this location is mounted as a PersistentVolume
 // it persists the cache across restarts.
 {
-  "stores": {
-    "CAS_MAIN_STORE": {
+  "stores": [
+    {
+      "name": "CAS_MAIN_STORE",
       "existence_cache": {
         "backend": {
           "compression": {
@@ -22,8 +23,8 @@
           }
         }
       }
-    },
-    "AC_MAIN_STORE": {
+    }, {
+      "name": "AC_MAIN_STORE",
       "completeness_checking": {
         "backend": {
           "filesystem": {
@@ -41,9 +42,10 @@
         }
       }
     }
-  },
-  "schedulers": {
-    "MAIN_SCHEDULER": {
+  ],
+  "schedulers": [
+    {
+      "name": "MAIN_SCHEDULER",
       // TODO(adams): use the right scheduler because reclient doesn't use the cached results?
       // TODO(adams): max_bytes_per_stream
       "simple": {
@@ -68,7 +70,7 @@
         }
       }
     }
-  },
+  ],
   "servers": [{
     // Only publish metrics on a private port.
     "listener": {
diff --git a/nativelink-config/BUILD.bazel b/nativelink-config/BUILD.bazel
index 9af292cd0..5d78c0cd8 100644
--- a/nativelink-config/BUILD.bazel
+++ b/nativelink-config/BUILD.bazel
@@ -31,6 +31,7 @@ rust_test_suite(
     name = "integration",
     timeout = "short",
     srcs = [
+        "tests/backwards_compat_test.rs",
         "tests/deserialization_test.rs",
     ],
     deps = [
@@ -40,6 +41,7 @@ rust_test_suite(
         "@crates//:humantime",
         "@crates//:pretty_assertions",
         "@crates//:serde",
+        "@crates//:serde_json",
         "@crates//:serde_json5",
     ],
 )
diff --git a/nativelink-config/Cargo.toml b/nativelink-config/Cargo.toml
index fdd28dff8..b39aeecb1 100644
--- a/nativelink-config/Cargo.toml
+++ b/nativelink-config/Cargo.toml
@@ -12,3 +12,4 @@ shellexpand = { version = "3.1.0", default-features = false, features = ["base-0
 
 [dev-dependencies]
 pretty_assertions = { version = "1.4.1", features = ["std"] }
+serde_json = { version = "1.0.138", default-features = false }
diff --git a/nativelink-config/examples/basic_cas.json5 b/nativelink-config/examples/basic_cas.json5
index 8f04e0825..0f53600a5 100644
--- a/nativelink-config/examples/basic_cas.json5
+++ b/nativelink-config/examples/basic_cas.json5
@@ -1,6 +1,7 @@
 {
-  "stores": {
-    "AC_MAIN_STORE": {
+  "stores": [
+    {
+      "name": "AC_MAIN_STORE",
       "filesystem": {
         "content_path": "/tmp/nativelink/data-worker-test/content_path-ac",
         "temp_path": "/tmp/nativelink/data-worker-test/tmp_path-ac",
@@ -9,8 +10,8 @@
           "max_bytes": 1000000000
         }
       }
-    },
-    "WORKER_FAST_SLOW_STORE": {
+    }, {
+      "name": "WORKER_FAST_SLOW_STORE",
       "fast_slow": {
         // "fast" must be a "filesystem" store because the worker uses it to make
         // hardlinks on disk to a directory where the jobs are running.
@@ -34,9 +35,10 @@
         }
       }
     }
-  },
-  "schedulers": {
-    "MAIN_SCHEDULER": {
+  ],
+  "schedulers": [
+    {
+      "name": "MAIN_SCHEDULER",
       "simple": {
         "supported_platform_properties": {
           "cpu_count": "minimum",
@@ -59,7 +61,7 @@
         }
       }
     }
-  },
+  ],
   "workers": [{
     "local": {
       "worker_api_endpoint": {
diff --git a/nativelink-config/examples/filesystem_cas.json5 b/nativelink-config/examples/filesystem_cas.json5
index 4baa85c7c..154aceb7d 100644
--- a/nativelink-config/examples/filesystem_cas.json5
+++ b/nativelink-config/examples/filesystem_cas.json5
@@ -4,8 +4,9 @@
 // so objects are compressed, deduplicated and uses some in-memory
 // optimizations for certain hot paths.
 {
-  "stores": {
-    "FS_CONTENT_STORE": {
+  "stores": [
+    {
+      "name": "FS_CONTENT_STORE",
       "compression": {
         "compression_algorithm": {
           "lz4": {}
@@ -21,8 +22,8 @@
           }
         }
       }
-    },
-    "CAS_MAIN_STORE": {
+    }, {
+      "name": "CAS_MAIN_STORE",
       "verify": {
         "backend": {
           // Because we are using a dedup store, we can bypass small objects
@@ -78,8 +79,8 @@
         "verify_size": true,
         "verify_hash": true
       }
-    },
-    "AC_MAIN_STORE": {
+    }, {
+      "name": "AC_MAIN_STORE",
       "filesystem": {
         "content_path": "/tmp/nativelink/data/content_path-ac",
         "temp_path": "/tmp/nativelink/data/tmp_path-ac",
@@ -89,9 +90,10 @@
         }
       }
     }
-  },
-  "schedulers": {
-    "MAIN_SCHEDULER": {
+  ],
+  "schedulers": [
+    {
+      "name": "MAIN_SCHEDULER",
       "simple": {
         "supported_platform_properties": {
           "cpu_count": "minimum",
@@ -113,7 +115,7 @@
         }
       }
     }
-  },
+  ],
   "servers": [{
     "listener": {
       "http": {
diff --git a/nativelink-config/examples/redis.json b/nativelink-config/examples/redis.json
index 1f06fe560..a990c28e9 100644
--- a/nativelink-config/examples/redis.json
+++ b/nativelink-config/examples/redis.json
@@ -1,18 +1,19 @@
 {
-  "stores": {
-    "CAS_FAST_SLOW_STORE": {
+  "stores": [
+    {
+      "name": "CAS_FAST_SLOW_STORE",
       "redis_store": {
         "addresses": ["redis://127.0.0.1:6379/"],
         "mode": "cluster"
       }
-    },
-    "AC_FAST_SLOW_STORE": {
+    }, {
+      "name": "AC_FAST_SLOW_STORE",
       "redis_store": {
         "addresses": ["redis://127.0.0.1:6379/"],
         "mode": "cluster"
       }
-    },
-    "AC_MAIN_STORE": {
+    }, {
+      "name": "AC_MAIN_STORE",
       "completeness_checking": {
         "backend": {
           "ref_store": {
@@ -25,8 +26,8 @@
           }
         }
       }
-    },
-    "CAS_MAIN_STORE": {
+    }, {
+      "name": "CAS_MAIN_STORE",
       "existence_cache": {
         "backend": {
           "compression": {
@@ -42,7 +43,7 @@
         }
       }
     }
-  },
+  ],
   "servers": [
     {
       "listener": {
diff --git a/nativelink-config/examples/s3_backend_with_local_fast_cas.json5 b/nativelink-config/examples/s3_backend_with_local_fast_cas.json5
index e1221fd14..accccae1b 100644
--- a/nativelink-config/examples/s3_backend_with_local_fast_cas.json5
+++ b/nativelink-config/examples/s3_backend_with_local_fast_cas.json5
@@ -1,6 +1,7 @@
 {
-  "stores": {
-    "CAS_MAIN_STORE": {
+  "stores": [
+    {
+      "name": "CAS_MAIN_STORE",
       "verify": {
         "backend": {
           "dedup": {
@@ -70,8 +71,8 @@
         "verify_size": true,
         "hash_verification_function": "sha256"
       }
-    },
-    "AC_MAIN_STORE": {
+    }, {
+      "name": "AC_MAIN_STORE",
       "fast_slow": {
         "fast": {
           "memory": {
@@ -104,9 +105,10 @@
         }
       }
     }
-  },
-  "schedulers": {
-    "MAIN_SCHEDULER": {
+  ],
+  "schedulers": [
+    {
+      "name": "MAIN_SCHEDULER",
       "simple": {
         "supported_platform_properties": {
           "cpu_count": "minimum",
@@ -128,7 +130,7 @@
         }
       }
     }
-  },
+  ],
   "servers": [{
     "listener": {
       "http": {
diff --git a/nativelink-config/src/cas_server.rs b/nativelink-config/src/cas_server.rs
index aefb1c1d5..e1f325985 100644
--- a/nativelink-config/src/cas_server.rs
+++ b/nativelink-config/src/cas_server.rs
@@ -16,14 +16,14 @@ use std::collections::HashMap;
 
 use serde::Deserialize;
 
-use crate::schedulers::SchedulerSpec;
 use crate::serde_utils::{
     convert_data_size_with_shellexpand, convert_duration_with_shellexpand,
     convert_numeric_with_shellexpand, convert_optional_numeric_with_shellexpand,
     convert_optional_string_with_shellexpand, convert_string_with_shellexpand,
     convert_vec_string_with_shellexpand,
 };
-use crate::stores::{ClientTlsConfig, ConfigDigestHashFunction, StoreRefName, StoreSpec};
+use crate::stores::{ClientTlsConfig, ConfigDigestHashFunction, StoreRefName};
+use crate::{SchedulerConfigs, StoreConfigs};
 
 /// Name of the scheduler. This type will be used when referencing a
 /// scheduler in the `CasConfig::schedulers`'s map key.
@@ -764,7 +764,7 @@ pub struct GlobalConfig {
 pub struct CasConfig {
     /// List of stores available to use in this config.
     /// The keys can be used in other configs when needing to reference a store.
-    pub stores: HashMap<StoreRefName, StoreSpec>,
+    pub stores: StoreConfigs,
 
     /// Worker configurations used to execute jobs.
     pub workers: Option<Vec<WorkerConfig>>,
@@ -772,7 +772,7 @@ pub struct CasConfig {
     /// List of schedulers available to use in this config.
     /// The keys can be used in other configs when needing to reference a
     /// scheduler.
-    pub schedulers: Option<HashMap<SchedulerRefName, SchedulerSpec>>,
+    pub schedulers: Option<SchedulerConfigs>,
 
     /// Servers to setup for this process.
     pub servers: Vec<ServerConfig>,
diff --git a/nativelink-config/src/lib.rs b/nativelink-config/src/lib.rs
index 0607e28c5..2bcd45cf5 100644
--- a/nativelink-config/src/lib.rs
+++ b/nativelink-config/src/lib.rs
@@ -16,3 +16,147 @@ pub mod cas_server;
 pub mod schedulers;
 pub mod serde_utils;
 pub mod stores;
+
+use std::any::type_name;
+use std::collections::HashMap;
+use std::fmt;
+use std::marker::PhantomData;
+
+use serde::de::{MapAccess, SeqAccess, Visitor};
+use serde::{Deserialize, Deserializer};
+
+#[derive(Debug, Clone, Deserialize)]
+pub struct NamedConfig<Spec> {
+    pub name: String,
+    #[serde(flatten)]
+    pub spec: Spec,
+}
+
+pub type StoreConfig = NamedConfig<crate::stores::StoreSpec>;
+pub type SchedulerConfig = NamedConfig<crate::schedulers::SchedulerSpec>;
+
+// TODO(aaronmondal): Remove all the iterator impls and the Deserializer once we
+//                    fully migrate to the new config schema.
+pub type StoreConfigs = NamedConfigs<crate::stores::StoreSpec>;
+pub type SchedulerConfigs = NamedConfigs<crate::schedulers::SchedulerSpec>;
+
+#[derive(Debug)]
+pub struct NamedConfigs<T>(pub Vec<NamedConfig<T>>);
+
+impl<T> NamedConfigs<T> {
+    pub fn iter(&self) -> std::slice::Iter<'_, NamedConfig<T>> {
+        self.0.iter()
+    }
+}
+
+impl<T> IntoIterator for NamedConfigs<T> {
+    type Item = NamedConfig<T>;
+    type IntoIter = std::vec::IntoIter<Self::Item>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.0.into_iter()
+    }
+}
+
+impl<'a, T> IntoIterator for &'a NamedConfigs<T> {
+    type Item = &'a NamedConfig<T>;
+    type IntoIter = std::slice::Iter<'a, NamedConfig<T>>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.0.iter()
+    }
+}
+
+struct NamedConfigsVisitor<T> {
+    phantom: PhantomData<T>,
+}
+
+impl<T> NamedConfigsVisitor<T> {
+    fn new() -> Self {
+        NamedConfigsVisitor {
+            phantom: PhantomData,
+        }
+    }
+}
+
+impl<'de, T: Deserialize<'de>> Visitor<'de> for NamedConfigsVisitor<T> {
+    type Value = NamedConfigs<T>;
+
+    fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
+        formatter.write_str("a sequence or map of named configs")
+    }
+
+    fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error>
+    where
+        A: SeqAccess<'de>,
+    {
+        let mut vec = Vec::new();
+        while let Some(config) = seq.next_element()? {
+            vec.push(config);
+        }
+        Ok(NamedConfigs(vec))
+    }
+
+    fn visit_map<M>(self, mut access: M) -> Result<Self::Value, M::Error>
+    where
+        M: MapAccess<'de>,
+    {
+        let config_type = if type_name::<T>().contains("StoreSpec") {
+            "stores"
+        } else if type_name::<T>().contains("SchedulerSpec") {
+            "schedulers"
+        } else {
+            "stores and schedulers"
+        };
+        eprintln!(
+            r#"
+WARNING: Using deprecated map format for {config_type}. Please migrate to the new array format:
+
+  // Old:
+  "stores": {{
+    "SOMESTORE": {{
+      "memory": {{}}
+    }}
+  }},
+  "schedulers": {{
+    "SOMESCHEDULER": {{
+      "simple": {{}}
+    }}
+  }}
+
+  // New:
+  "stores": [
+    {{
+      "name": "SOMESTORE",
+      "memory": {{}}
+    }}
+  ],
+  "schedulers": [
+    {{
+      "name": "SOMESCHEDULER",
+      "simple": {{}}
+    }}
+  ]
+"#
+        );
+
+        let mut map = HashMap::new();
+        while let Some((key, value)) = access.next_entry()? {
+            map.insert(key, value);
+        }
+        Ok(NamedConfigs(
+            map.into_iter()
+                .map(|(name, spec)| NamedConfig { name, spec })
+                .collect(),
+        ))
+    }
+}
+
+impl<'de, T: Deserialize<'de>> Deserialize<'de> for NamedConfigs<T> {
+    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+    where
+        D: Deserializer<'de>,
+    {
+        deserializer.deserialize_any(NamedConfigsVisitor::new())
+    }
+}
diff --git a/src/bin/nativelink.rs b/src/bin/nativelink.rs
index 1e97a0089..27184ccd6 100644
--- a/src/bin/nativelink.rs
+++ b/src/bin/nativelink.rs
@@ -31,6 +31,7 @@ use nativelink_config::cas_server::{
     CasConfig, GlobalConfig, HttpCompressionAlgorithm, ListenerConfig, ServerConfig, WorkerConfig,
 };
 use nativelink_config::stores::ConfigDigestHashFunction;
+use nativelink_config::{SchedulerConfig, StoreConfig};
 use nativelink_error::{make_err, make_input_err, Code, Error, ResultExt};
 use nativelink_metric::{
     MetricFieldData, MetricKind, MetricPublishKnownKindData, MetricsComponent, RootMetricsComponent,
@@ -190,11 +191,11 @@ async fn inner_main(
     {
         let mut health_registry_lock = health_registry_builder.lock().await;
 
-        for (name, store_cfg) in cfg.stores {
+        for StoreConfig { name, spec } in cfg.stores {
             let health_component_name = format!("stores/{name}");
             let mut health_register_store =
                 health_registry_lock.sub_builder(&health_component_name);
-            let store = store_factory(&store_cfg, &store_manager, Some(&mut health_register_store))
+            let store = store_factory(&spec, &store_manager, Some(&mut health_register_store))
                 .await
                 .err_tip(|| format!("Failed to create store '{name}'"))?;
             store_manager.add_store(&name, store);
@@ -203,17 +204,15 @@ async fn inner_main(
 
     let mut action_schedulers = HashMap::new();
     let mut worker_schedulers = HashMap::new();
-    if let Some(schedulers_cfg) = cfg.schedulers {
-        for (name, scheduler_cfg) in schedulers_cfg {
-            let (maybe_action_scheduler, maybe_worker_scheduler) =
-                scheduler_factory(&scheduler_cfg, &store_manager)
-                    .err_tip(|| format!("Failed to create scheduler '{name}'"))?;
-            if let Some(action_scheduler) = maybe_action_scheduler {
-                action_schedulers.insert(name.clone(), action_scheduler.clone());
-            }
-            if let Some(worker_scheduler) = maybe_worker_scheduler {
-                worker_schedulers.insert(name.clone(), worker_scheduler.clone());
-            }
+    for SchedulerConfig { name, spec } in cfg.schedulers.iter().flatten() {
+        let (maybe_action_scheduler, maybe_worker_scheduler) =
+            scheduler_factory(spec, &store_manager)
+                .err_tip(|| format!("Failed to create scheduler '{name}'"))?;
+        if let Some(action_scheduler) = maybe_action_scheduler {
+            action_schedulers.insert(name.clone(), action_scheduler.clone());
+        }
+        if let Some(worker_scheduler) = maybe_worker_scheduler {
+            worker_schedulers.insert(name.clone(), worker_scheduler.clone());
         }
     }
 


Test patch:
diff --git a/nativelink-config/tests/backwards_compat_test.rs b/nativelink-config/tests/backwards_compat_test.rs
new file mode 100644
index 000000000..26261e843
--- /dev/null
+++ b/nativelink-config/tests/backwards_compat_test.rs
@@ -0,0 +1,55 @@
+// Copyright 2024 The NativeLink Authors. All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use std::collections::HashMap;
+
+use nativelink_config::NamedConfigs;
+use serde::Deserialize;
+use serde_json::json;
+
+#[derive(Debug, Deserialize, PartialEq)]
+struct DummySpec {
+    memory: HashMap<String, String>,
+}
+
+#[test]
+fn test_configs_deserialization() {
+    let expected = json!([
+        {
+            "name": "store1",
+            "memory": {}
+        },
+        {
+            "name": "store2",
+            "memory": {}
+        }
+    ]);
+
+    let config: NamedConfigs<DummySpec> = serde_json::from_value(expected.clone()).unwrap();
+    let mut new_stores: Vec<_> = config.into_iter().collect();
+    new_stores.sort_by(|a, b| a.name.cmp(&b.name));
+
+    let old_format = json!({
+        "store1": { "memory": {} },
+        "store2": { "memory": {} }
+    });
+
+    let config: NamedConfigs<DummySpec> = serde_json::from_value(old_format).unwrap();
+    let mut transformed_stores: Vec<_> = config.into_iter().collect();
+    transformed_stores.sort_by(|a, b| a.name.cmp(&b.name));
+
+    assert_eq!(new_stores.len(), transformed_stores.len());
+    assert_eq!(new_stores[0].name, transformed_stores[0].name);
+    assert_eq!(new_stores[1].name, transformed_stores[1].name);
+}