Repository: 0xPolygonMiden/miden-base
Pull request #1250: Hash keys in `StorageMap` before insertion into SMT
Author: login: PhilippGackstatter
Created: 2025-03-26T13:03:38Z | Merged: 2025-03-28T07:21:29Z

Hash keys in `StorageMap` before insertion into SMT.

Adds some missing docs to `StorageMap`, cleans up some names and removes unnecessary (imo) APIs. Also adds a dedicated `StorageMapError` instead of re-exporting `MerkleError`, which has a practical reason. It is inconvenient to call `Smt::with_entries` before building the `BTreeMap`, and calling it afterwards wouldn't catch the duplicates because BTreeMap would have deduplicated them.

closes #1129 

Linked issue #1129: Salt `StorageMap` keys
### What should be done?

Useres can currently use `StorageMap`s in accounts to authenticate many key-value pairs in a single storage slot. The keys of the `Smt` underlying the `StorageMap` are taken directly from user input. This means users or an attacker could insert many key-value pairs that end up in the same subtree. However, a uniform distribution of keys across an `Smt` is important for storage optimizations, so it would be desirable for us to ensure keys are uniformly distributed and not entirely user-controlled.

### How should it be done?

To that end, we can add a salt to the user keys, like the account ID. So in essence, the actual key in an `Smt` should be the hash of the tuple `(KEY || [account_id_prefix, account_id_suffix, 0, 0])` or something similar.



### When is this task done?

When user input can no longer affect the distribution of keys in a `StorageMap`.

### Additional context

_No response_
Linked issue comments: user: login: bobbinth
created_at: 2025-02-17T07:22:21Z
body: One additional consideration: by looking at the salted keys it would not be possible to reconstruct the original key values. This may be fine in some cases, but may be problematic in others. For example, if we have some kind of a "storage explorer" which lets people visualize the contents of the storage, then such storage explorer would not be able to show the original keys and this may be confusing to users.
author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-24T09:12:10Z
body: I wonder if a way to get both uniform distributions across storage as well as support explorers is by salting the keys just before insertion into storage. For example in the node we have this [table](https://github.com/0xPolygonMiden/miden-node/blob/94d9269e1811cbd1441a122dd4547fc0630598dd/crates/store/src/db/migrations/001-init.sql#L81-L90):

```sql
CREATE TABLE account_storage_map_updates (
    account_id  BLOB NOT NULL,
    block_num   INTEGER NOT NULL,
    slot        INTEGER NOT NULL,
    key         BLOB    NOT NULL,
    value       BLOB    NOT NULL,

    PRIMARY KEY (account_id, block_num, slot, key),
    FOREIGN KEY (account_id, block_num) REFERENCES account_deltas (account_id, block_num)
) STRICT, WITHOUT ROWID;
```

(I don't think salting is really necessary here, since this reflects just an update of a map, but I'm taking this as an example to make a larger point.)

We could set `key = hash(storage_map_key, account_id)` and set `value = (storage_map_key, storage_map_value)`.
When retrieving from the DB, we rebuild the original key-value pair from the value.

That way, what gets into the DB is salted and uniformly distributed but any layer above the DB works with the original keys, which should be fine. If we're concerned about in-memory uniformity, we can use a `HashMap` which applies its own hash function to each key, conceptually in the same way as the DB approach I described.

The downside is the extra work needed before and after every DB operation, but if we do salting in general, then we'll have to do that work _somewhere_. And this way, the salting is closer to where we actually need it, i.e. when storing maps on disk.
author_association: CONTRIBUTOR
user: login: bobbinth
created_at: 2025-03-25T08:33:15Z
body: I think this makes sense, but one concrete question: how would we define the `StorageMap` struct now? I think the simplest approach could be:
```rust
pub struct StorageMap {
    map: Smt,
    salt: AccountId,
    entries: BTreeMap<RpoDigest, Word>, // key-value map using the original keys
}
```
But having `entries` there seems a bit "redundant" and also not sure if having the account ID be a part of it is a good idea (mostly because of extra serialization overhead).
author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-25T14:38:54Z
body: > how would we define the `StorageMap` struct now?

What I meant was that the struct would stay exactly the same, because at that level we wouldn't salt the keys. The keys would only be salted when inserted into the DB, and upon retrievel, the original key-value pairs are restored from the `value`, and so the original `StorageMap` can be reconstructed as well.
I imagine we always store a `StorageMap` next to the account ID it belongs to, so the salt can just be the account ID that is stored in the same table, without having to duplicate it. 
author_association: CONTRIBUTOR
user: login: bobbinth
created_at: 2025-03-25T15:00:56Z
body: > What I meant was that the struct would stay exactly the same, because at that level we wouldn't salt the keys. The keys would only be salted when inserted into the DB, and upon retrievel, the original key-value pairs are restored from the `value`, and so the original `StorageMap` can be reconstructed as well.
> I imagine we always store a `StorageMap` next to the account ID it belongs to, so the salt can just be the account ID that is stored in the same table, without having to duplicate it.

Ah, I see - but then the value needs to be 2 words, right? (vs. the single word that we currently assume for it). And this would affect the `StorageMap`.
author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-25T16:23:59Z
body: The value in the DB is defined as `value       BLOB    NOT NULL` so whether we store just the original value of the storage map or the key _and_ the value shouldn't matter that much. And if we look up a key in the DB we can also hash it, and then reconstruct the original key-value pair which then shouldn't affect the storage map, or am I missing something? I thought the main point was to make the `key` in the DB, which is a primary key, uniformly distributed which the hash achieves.

author_association: CONTRIBUTOR
user: login: bobbinth
created_at: 2025-03-26T06:31:34Z
body: I think that maybe I didn't explain the intent of the original issue clearly: what we really care about is which `SmtLeaf` they key-value pairs end up in. It would be a bad outcome if keys `0` and `1` end up in the same leaf of the `Smt` because this would result in a multi-key leaf and these are expensive (this would happen because `0` and `1` share the same 64-bit prefix). So, before inserting a key-value pair into `StorageMap` we need to compute `hash(0)` and `hash(1)` and use these ask keys. This way `hash(0)` and `hash(1)`, with very high probability, will end up in different leaves of `Smt` (we could also use salted hashes to make key distribution unique for each account).

Given the above, we need to modify the transaction kernel to first hash the user-provided key, and then to use this hashed key as the key with which to make the request to the SMT.

This works fine if we know user-provided keys (e.g., in the transaction kernel), but it won't work when we don't have these keys. So, somewhere we need to keep track of them, and most likely this somewhere needs to be the `StorageMap` struct. The reason for this is that when we serialize accounts, we also serialize storage maps. And if the user-specified keys are not serialized together with the account somehow, we won't be able to get them when we de-serialize the struct.
author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-26T13:02:08Z
body: I see, thanks for explaining.

In that case, I agree we have to deal with it in `StorageMap` itself. If we do want to retain the original keys, there doesn't seem to be a way around duplicating them in the map.

I pretty much went with your approach, except I thought that hashing the keys should be good enough and that salting them is not necessary. Storage maps of different accounts do not affect each other's trees so if multiple storage maps have the same layout then having the same keys shouldn't be a problem for efficiency, afaict.
author_association: CONTRIBUTOR
user: login: Mirko-von-Leipzig
created_at: 2025-03-26T18:34:30Z
body: > I see, thanks for explaining.
> 
> In that case, I agree we have to deal with it in `StorageMap` itself. If we do want to retain the original keys, there doesn't seem to be a way around duplicating them in the map.
> 
> I pretty much went with your approach, except I thought that hashing the keys should be good enough and that salting them is not necessary. Storage maps of different accounts do not affect each other's trees so if multiple storage maps have the same layout then having the same keys shouldn't be a problem for efficiency, afaict.

I thought maybe the concern was that without salting you can still find a bad ordering/hash set. And salting restricts this "attack" to one per account/salt. But maybe I was reading too much into it.
author_association: CONTRIBUTOR
user: login: bobbinth
created_at: 2025-03-26T19:08:20Z
body: > I thought maybe the concern was that without salting you can still find a bad ordering/hash set. And salting restricts this "attack" to one per account/salt. But maybe I was reading too much into it.

Yes, definitely was thinking about this - but more out of abundance of caution then any real reason. One the one hand, I think it is good to use salting "just in case" - but also, if this complicates the code too much, maybe it is not worth it.
author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-27T08:40:27Z
body: Good point, thanks!

I looked into it, but adding the `AccountId` to `StorageMap` results in a circular dependency during account creation. We have to instantiate `AccountStorage` to get its commitment which is used to grind an account ID. But if the account storage should contain a map with the ID as the salt, we can't initialize the map with the account ID yet. So we'd have to do awkward work arounds like making the salt an `Option<AccountId>` and inserting it after the account has been instantiated, or disallowing storage maps at account creation and requiring them to be inserted in a subsequent account update. Not sure this is worth it.
author_association: CONTRIBUTOR
user: login: Mirko-von-Leipzig
created_at: 2025-03-27T08:55:08Z
body: > Good point, thanks!
> 
> I looked into it, but adding the `AccountId` to `StorageMap` results in a circular dependency during account creation. We have to instantiate `AccountStorage` to get its commitment which is used to grind an account ID. But if the account storage should contain a map with the ID as the salt, we can't initialize the map with the account ID yet. So we'd have to do awkward work arounds like making the salt an `Option<AccountId>` and inserting it after the account has been instantiated, or disallowing storage maps at account creation and requiring them to be inserted in a subsequent account update. Not sure this is worth it.

Are there alternatives to account ID for the salt?

- block hash -- though this cannot be the latest hash and would instead be the reference block hash iiuc? So that this is doable locally.
- ... 🤷 
author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-27T13:50:23Z
body: > Are there alternatives to account ID for the salt?

Yeah, good point. The block hash is one potential idea. We'd have to set this once at account creation time and use it from then on. Instead of making this a separate parameter on the `AccountBuilder` and using the account-creating tx reference block, we could re-use the block commitment from the `AccountIdAnchor`. Feels a bit unclean to use it for two purposes, but other than that I don't see an issue with that. It does commit us a bit more to the anchor which ideally we want to get rid of (so we can deterministically recreate account IDs from seeds, iirc).
author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-28T07:21:45Z
body: As discussed in the call attacking account storage maps to make it a problem for the network:
- is expensive due to the hashing involved.
- only applies to network accounts.
- only applies to account code where users can choose keys freely, which authors may or may not allow.
- is noticeable because it involves network accounts and many transactions, so there likely is time to react (e.g. to upgrade the account code).
- the same bad set of keys applies to accounts with the same code, but likely not to account's with other code.
- and its benefits are somewhat unclear. At worst, an attacker can force the network to no longer update an account if it becomes too expensive due to too many key-value pairs residing in the same leaf.

Adding an additional salt would only ensure that different accounts with the same code cannot be attacked, but would not otherwise prevent a targetted attack.

Given these mitigating circumstances, for now it should be good enough to use only hashing. 

Closed by #1250.
author_association: CONTRIBUTOR

Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 26c734ade6..20a80dd60c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,8 @@
 
 ## 0.9.0 (TBD)
 
+- [BREAKING] Hash keys in storage maps before insertion into the SMT (#1250).
+
 ## 0.8.1 (2025-03-26)
 
 ### Changes
diff --git a/crates/miden-lib/asm/kernels/transaction/lib/account.masm b/crates/miden-lib/asm/kernels/transaction/lib/account.masm
index 17d85c9a53..9e12d4407c 100644
--- a/crates/miden-lib/asm/kernels/transaction/lib/account.masm
+++ b/crates/miden-lib/asm/kernels/transaction/lib/account.masm
@@ -584,7 +584,11 @@ export.get_map_item
     exec.get_item swapw
     # => [KEY, ROOT]
 
-    # fetch the VALUE located under KEY in the tree
+    # see hash_map_key's docs for why this is done
+    exec.hash_map_key
+    # => [HASHED_KEY, ROOT]
+
+    # fetch the VALUE located under HASHED_KEY in the tree
     exec.smt::get
     # => [VALUE, ROOT]
 
@@ -629,13 +633,17 @@ export.set_map_item.12
     emit.ACCOUNT_STORAGE_BEFORE_SET_MAP_ITEM_EVENT
     # => [index, KEY, NEW_VALUE, OLD_ROOT]
 
-    # duplicate the KEY and the NEW_VALUE to be able to emit an event after an account storage item
-    # is being updated
+    # duplicate the original KEY and the NEW_VALUE to be able to emit an event after the
+    # account storage item was updated
     movdn.12 movupw.2 dupw.2 dupw.2
     # => [KEY, NEW_VALUE, OLD_ROOT, KEY, NEW_VALUE, index, ...]
 
-    # set the NEW_VALUE under KEY in the tree
-    # note smt::set expects the stack to be [NEW_VALUE, KEY, OLD_ROOT, ...]
+    # see hash_map_key's docs for why this is done
+    exec.hash_map_key
+    # => [HASHED_KEY, NEW_VALUE, OLD_ROOT, KEY, NEW_VALUE, index, ...]
+
+    # set the NEW_VALUE under HASHED_KEY in the tree
+    # note smt::set expects the stack to be [NEW_VALUE, HASHED_KEY, OLD_ROOT, ...]
     swapw exec.smt::set
     # => [OLD_MAP_VALUE, NEW_ROOT, KEY, NEW_VALUE, index, ...]
 
@@ -1132,3 +1140,16 @@ export.validate_current_foreign_account
     # clean the stack
     dropw drop drop dropw
 end
+
+#! Hashes the provided map key before using it as the key in an SMT.
+#!
+#! Because the keys of the map are user-chosen and thus not necessarily uniformly distributed, the
+#! tree could be imbalanced and made less efficient. To mitigate that, the keys used in the
+#! storage map are hashed before they are inserted into the SMT, which creates a uniform distribution.
+#!
+#! Inputs:  [KEY]
+#! Outputs: [HASHED_KEY]
+proc.hash_map_key
+  hash
+  # => [HASHED_KEY]
+end
diff --git a/crates/miden-lib/src/transaction/procedures/kernel_v0.rs b/crates/miden-lib/src/transaction/procedures/kernel_v0.rs
index 15d932df5b..8b9fa19ed3 100644
--- a/crates/miden-lib/src/transaction/procedures/kernel_v0.rs
+++ b/crates/miden-lib/src/transaction/procedures/kernel_v0.rs
@@ -26,9 +26,9 @@ pub const KERNEL0_PROCEDURES: [Digest; 36] = [
     // account_set_item
     digest!("0x61104ec016c3ed9b49aee53650ddde9e984a72e4c4e13001cbf98b9cef426758"),
     // account_get_map_item
-    digest!("0x21237825d10004f77d3e7b32aee9052b519752fd03f839c19440e2010f73132e"),
+    digest!("0xf751b0762e049923d31ec6fd18512254f307f22140a7daa40e71ae7577b03f67"),
     // account_set_map_item
-    digest!("0x876168292d11aea0e2886ee6d7f9e723a95ec3aa4e467e6f46c898567bfc3604"),
+    digest!("0x47be5d18f62a46c8c75076d1c63df9a0bbf1bf42b1b865a64cddb52489b7c7d6"),
     // account_get_vault_root
     digest!("0x279b4a9e5adca07f01cadf8ecc1303fa3c670003a7a4e69f09506b070c4023df"),
     // account_add_asset
@@ -40,13 +40,13 @@ pub const KERNEL0_PROCEDURES: [Digest; 36] = [
     // account_has_non_fungible_asset
     digest!("0x4fea67ed25474d5494a23c5e1e06a93f8aa140d0a673c6e140e0d4f1dd8bd835"),
     // faucet_mint_asset
-    digest!("0x836ecba4b7caa770b24ffa51115e2c08724ecb7f0854ee3a94848bec5b0ee2ce"),
+    digest!("0xc8f5ec2b280ba12b685b8730b4a8e32c1b9199081fd6070db89dc87af598a317"),
     // faucet_burn_asset
-    digest!("0x37e6b7c80f478c11ea7f12c065013c4bc3a09100fd6ecc4893e17d0abd675254"),
+    digest!("0xabf28cea013cbef5838c49f0e4a9118f32437e2086e31d1dea14959c3a7e862d"),
     // faucet_get_total_fungible_asset_issuance
     digest!("0xd2ee4bd330f989165ee2be0f121a4db916f95e58f6fd2d040d57672f2f0cef63"),
     // faucet_is_non_fungible_asset_issued
-    digest!("0x04c49c1f9cc628c1447e19d46c307084e995044faa86154353eec6af6b5b5041"),
+    digest!("0x1587b082609234cbb46300f8ce342ecf2b7757ae98d13c679df7c439b0b33cbc"),
     // note_get_assets_info
     digest!("0x34e4f1ea83eb4342ab8f5acec89962b2ab4b56d9c631e807d8e4dc8efd270bf2"),
     // note_add_asset
diff --git a/crates/miden-objects/src/account/delta/storage.rs b/crates/miden-objects/src/account/delta/storage.rs
index ff92495b07..809126c3e3 100644
--- a/crates/miden-objects/src/account/delta/storage.rs
+++ b/crates/miden-objects/src/account/delta/storage.rs
@@ -11,7 +11,6 @@ use super::{
 use crate::{
     Digest, EMPTY_WORD,
     account::{AccountStorage, StorageMap, StorageSlot},
-    crypto::merkle::SmtLeaf,
 };
 // ACCOUNT STORAGE DELTA
 // ================================================================================================
@@ -276,21 +275,7 @@ impl StorageMapDelta {
 /// Converts a [StorageMap] into a [StorageMapDelta] for initial delta construction.
 impl From<StorageMap> for StorageMapDelta {
     fn from(map: StorageMap) -> Self {
-        // TODO: implement `IntoIterator` for `Smt` and get rid of copying keys/values:
-        let mut delta = StorageMapDelta::new(BTreeMap::default());
-        for (_, leaf) in map.leaves() {
-            match leaf {
-                SmtLeaf::Empty(_) => continue,
-                SmtLeaf::Single((key, value)) => {
-                    delta.insert(*key, *value);
-                },
-                SmtLeaf::Multiple(vec) => {
-                    vec.iter().for_each(|(key, value)| delta.insert(*key, *value));
-                },
-            }
-        }
-
-        delta
+        StorageMapDelta::new(map.into_entries())
     }
 }
 
diff --git a/crates/miden-objects/src/account/storage/header.rs b/crates/miden-objects/src/account/storage/header.rs
index 69c364f418..a0423ae1e3 100644
--- a/crates/miden-objects/src/account/storage/header.rs
+++ b/crates/miden-objects/src/account/storage/header.rs
@@ -146,19 +146,13 @@ mod tests {
 
     #[test]
     fn test_from_account_storage() {
+        let storage_map = AccountStorage::mock_map();
+
         // create new storage header from AccountStorage
         let slots = vec![
             (StorageSlotType::Value, [Felt::new(1), Felt::new(2), Felt::new(3), Felt::new(4)]),
             (StorageSlotType::Value, [Felt::new(5), Felt::new(6), Felt::new(7), Felt::new(8)]),
-            (
-                StorageSlotType::Map,
-                [
-                    Felt::new(12405212884040084310),
-                    Felt::new(17614307840949763446),
-                    Felt::new(6101527485586301500),
-                    Felt::new(14442045877206841081),
-                ],
-            ),
+            (StorageSlotType::Map, storage_map.root().into()),
         ];
 
         let expected_header = AccountStorageHeader { slots };
diff --git a/crates/miden-objects/src/account/storage/map.rs b/crates/miden-objects/src/account/storage/map.rs
index ccff666d83..ba4a65ee0b 100644
--- a/crates/miden-objects/src/account/storage/map.rs
+++ b/crates/miden-objects/src/account/storage/map.rs
@@ -1,29 +1,53 @@
-use miden_crypto::merkle::{EmptySubtreeRoots, MerkleError};
+use alloc::collections::BTreeMap;
+
+use miden_crypto::merkle::EmptySubtreeRoots;
+use vm_core::EMPTY_WORD;
 
 use super::{
     ByteReader, ByteWriter, Deserializable, DeserializationError, Digest, Serializable, Word,
 };
 use crate::{
+    Hasher,
     account::StorageMapDelta,
     crypto::{
         hash::rpo::RpoDigest,
         merkle::{InnerNodeInfo, LeafIndex, SMT_DEPTH, Smt, SmtLeaf, SmtProof},
     },
+    errors::StorageMapError,
 };
 
 // ACCOUNT STORAGE MAP
 // ================================================================================================
 
 /// Empty storage map root.
-pub const EMPTY_STORAGE_MAP_ROOT: Digest =
-    *EmptySubtreeRoots::entry(StorageMap::STORAGE_MAP_TREE_DEPTH, 0);
-
-/// Account storage map is a Sparse Merkle Tree of depth 64. It can be used to store more data as
-/// there is in plain usage of the storage slots. The root of the SMT consumes one account storage
-/// slot.
+pub const EMPTY_STORAGE_MAP_ROOT: Digest = *EmptySubtreeRoots::entry(StorageMap::TREE_DEPTH, 0);
+
+/// An account storage map is a sparse merkle tree of depth [`Self::TREE_DEPTH`] (64).
+///
+/// It can be used to store a large amount of data in an account than would be otherwise possible
+/// using just the account's storage slots. This works by storing the root of the map's underlying
+/// SMT in one account storage slot. Each map entry is a leaf in the tree and its inclusion is
+/// proven while retrieving it (e.g. via `account::get_map_item`).
+///
+/// As a side-effect, this also means that _not all_ entries of the map have to be present at
+/// transaction execution time in order to access or modify the map. It is sufficient if _just_ the
+/// accessed/modified items are present in the advice provider.
+///
+/// Because the keys of the map are user-chosen and thus not necessarily uniformly distributed, the
+/// tree could be imbalanced and made less efficient. To mitigate that, the keys used in the
+/// storage map are hashed before they are inserted into the SMT, which creates a uniform
+/// distribution. The original keys are retained in a separate map. This causes redundancy but
+/// allows for introspection of the map, e.g. by querying the set of stored (original) keys which is
+/// useful in debugging and explorer scenarios.
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct StorageMap {
-    map: Smt,
+    /// The SMT where each key is the hashed original key.
+    smt: Smt,
+    /// The entries of the map where the key is the original user-chosen one.
+    ///
+    /// It is an invariant of this type that the map's entries are always consistent with the SMT's
+    /// entries and vice-versa.
+    map: BTreeMap<RpoDigest, Word>,
 }
 
 impl StorageMap {
@@ -31,7 +55,7 @@ impl StorageMap {
     // --------------------------------------------------------------------------------------------
 
     /// Depth of the storage tree.
-    pub const STORAGE_MAP_TREE_DEPTH: u8 = SMT_DEPTH;
+    pub const TREE_DEPTH: u8 = SMT_DEPTH;
 
     /// The default value of empty leaves.
     pub const EMPTY_VALUE: Word = Smt::EMPTY_VALUE;
@@ -43,61 +67,98 @@ impl StorageMap {
     ///
     /// All leaves in the returned tree are set to [Self::EMPTY_VALUE].
     pub fn new() -> Self {
-        StorageMap { map: Smt::new() }
+        StorageMap { smt: Smt::new(), map: BTreeMap::new() }
     }
 
     /// Creates a new [`StorageMap`] from the provided key-value entries.
     ///
     /// # Errors
     ///
-    /// Returns an error if the provided entries contain multiple values for the same key.
-    pub fn with_entries(
-        entries: impl IntoIterator<Item = (RpoDigest, Word)>,
-    ) -> Result<Self, MerkleError> {
-        Smt::with_entries(entries).map(|map| StorageMap { map })
+    /// Returns an error if:
+    /// - the provided entries contain multiple values for the same key.
+    pub fn with_entries<I: ExactSizeIterator<Item = (RpoDigest, Word)>>(
+        entries: impl IntoIterator<Item = (RpoDigest, Word), IntoIter = I>,
+    ) -> Result<Self, StorageMapError> {
+        let mut map = BTreeMap::new();
+
+        for (key, value) in entries {
+            if let Some(prev_value) = map.insert(key, value) {
+                return Err(StorageMapError::DuplicateKey {
+                    key,
+                    value0: prev_value,
+                    value1: value,
+                });
+            }
+        }
+
+        Ok(Self::from_btree_map(map))
     }
 
-    // PUBLIC ACCESSORS
-    // --------------------------------------------------------------------------------------------
+    /// Creates a new [`StorageMap`] from the given map. For internal use.
+    fn from_btree_map(map: BTreeMap<RpoDigest, Word>) -> Self {
+        let hashed_keys_iter = map.iter().map(|(key, value)| (Self::hash_key(*key), *value));
+        let smt = Smt::with_entries(hashed_keys_iter)
+            .expect("btree maps should not contain duplicate keys");
 
-    pub const fn depth(&self) -> u8 {
-        SMT_DEPTH
+        StorageMap { smt, map }
     }
 
-    pub fn root(&self) -> RpoDigest {
-        self.map.root() // Delegate to Smt's root method
-    }
+    // PUBLIC ACCESSORS
+    // --------------------------------------------------------------------------------------------
 
-    pub fn get_leaf(&self, key: &RpoDigest) -> SmtLeaf {
-        self.map.get_leaf(key) // Delegate to Smt's get_leaf method
+    /// Returns the root of the underlying sparse merkle tree.
+    pub fn root(&self) -> RpoDigest {
+        self.smt.root()
     }
 
-    pub fn get_value(&self, key: &RpoDigest) -> Word {
-        self.map.get_value(key) // Delegate to Smt's get_value method
+    /// Returns the value corresponding to the key or [`Self::EMPTY_VALUE`] if the key is not
+    /// associated with a value.
+    pub fn get(&self, key: &RpoDigest) -> Word {
+        self.map.get(key).copied().unwrap_or_default()
     }
 
+    /// Returns an opening of the leaf associated with `key`.
+    ///
+    /// Conceptually, an opening is a Merkle path to the leaf, as well as the leaf itself.
     pub fn open(&self, key: &RpoDigest) -> SmtProof {
-        self.map.open(key) // Delegate to Smt's open method
+        let key = Self::hash_key(*key);
+        self.smt.open(&key)
     }
 
     // ITERATORS
     // --------------------------------------------------------------------------------------------
+
+    /// Returns an iterator over the leaves of the underlying [`Smt`].
     pub fn leaves(&self) -> impl Iterator<Item = (LeafIndex<SMT_DEPTH>, &SmtLeaf)> {
-        self.map.leaves() // Delegate to Smt's leaves method
+        self.smt.leaves() // Delegate to Smt's leaves method
     }
 
-    pub fn entries(&self) -> impl Iterator<Item = &(RpoDigest, Word)> {
-        self.map.entries() // Delegate to Smt's entries method
+    /// Returns an iterator over the key value pairs of the map.
+    pub fn entries(&self) -> impl Iterator<Item = (&RpoDigest, &Word)> {
+        self.map.iter()
     }
 
+    /// Returns an iterator over the inner nodes of the underlying [`Smt`].
     pub fn inner_nodes(&self) -> impl Iterator<Item = InnerNodeInfo> + '_ {
-        self.map.inner_nodes() // Delegate to Smt's inner_nodes method
+        self.smt.inner_nodes() // Delegate to Smt's inner_nodes method
     }
 
     // DATA MUTATORS
     // --------------------------------------------------------------------------------------------
+
+    /// Inserts or updates the given key value pair and returns the previous value, or
+    /// [`Self::EMPTY_VALUE`] if no entry was previously present.
+    ///
+    /// If the provided `value` is [`Self::EMPTY_VALUE`] the entry will be removed.
     pub fn insert(&mut self, key: RpoDigest, value: Word) -> Word {
-        self.map.insert(key, value) // Delegate to Smt's insert method
+        if value == EMPTY_WORD {
+            self.map.remove(&key);
+        } else {
+            self.map.insert(key, value);
+        }
+
+        let key = Self::hash_key(key);
+        self.smt.insert(key, value) // Delegate to Smt's insert method
     }
 
     /// Applies the provided delta to this account storage.
@@ -109,6 +170,16 @@ impl StorageMap {
 
         self.root()
     }
+
+    /// Consumes the map and returns the underlying map of entries.
+    pub fn into_entries(self) -> BTreeMap<RpoDigest, Word> {
+        self.map
+    }
+
+    /// Hashes the given key to get the key of the SMT.
+    pub fn hash_key(key: RpoDigest) -> RpoDigest {
+        Hasher::hash_elements(key.as_elements())
+    }
 }
 
 impl Default for StorageMap {
@@ -122,27 +193,28 @@ impl Default for StorageMap {
 
 impl Serializable for StorageMap {
     fn write_into<W: ByteWriter>(&self, target: &mut W) {
-        self.map.write_into(target)
+        self.map.write_into(target);
     }
 
     fn get_size_hint(&self) -> usize {
-        self.map.get_size_hint()
+        self.smt.get_size_hint()
     }
 }
 
 impl Deserializable for StorageMap {
     fn read_from<R: ByteReader>(source: &mut R) -> Result<Self, DeserializationError> {
-        let smt = Smt::read_from(source)?;
-        Ok(StorageMap { map: smt })
+        let map = BTreeMap::read_from(source)?;
+        Ok(Self::from_btree_map(map))
     }
 }
 
 #[cfg(test)]
 mod tests {
     use assert_matches::assert_matches;
-    use miden_crypto::{Felt, hash::rpo::RpoDigest, merkle::MerkleError};
+    use miden_crypto::{Felt, hash::rpo::RpoDigest};
 
     use super::{Deserializable, EMPTY_STORAGE_MAP_ROOT, Serializable, StorageMap, Word};
+    use crate::errors::StorageMapError;
 
     #[test]
     fn account_storage_serialization() {
@@ -165,7 +237,11 @@ mod tests {
         let storage_map = StorageMap::with_entries(storage_map_leaves_2).unwrap();
 
         let bytes = storage_map.to_bytes();
-        assert_eq!(storage_map, StorageMap::read_from_bytes(&bytes).unwrap());
+        let deserialized_map = StorageMap::read_from_bytes(&bytes).unwrap();
+
+        assert_eq!(storage_map.root(), deserialized_map.root());
+
+        assert_eq!(storage_map, deserialized_map);
     }
 
     #[test]
@@ -189,6 +265,6 @@ mod tests {
         ];
 
         let error = StorageMap::with_entries(storage_map_leaves_2).unwrap_err();
-        assert_matches!(error, MerkleError::DuplicateValuesForIndex(_));
+        assert_matches!(error, StorageMapError::DuplicateKey { .. });
     }
 }
diff --git a/crates/miden-objects/src/account/storage/mod.rs b/crates/miden-objects/src/account/storage/mod.rs
index 0a388060f6..aa72c237de 100644
--- a/crates/miden-objects/src/account/storage/mod.rs
+++ b/crates/miden-objects/src/account/storage/mod.rs
@@ -132,7 +132,7 @@ impl AccountStorage {
             slots_len: self.slots.len() as u8,
             index,
         })? {
-            StorageSlot::Map(map) => Ok(map.get_value(&Digest::from(key))),
+            StorageSlot::Map(map) => Ok(map.get(&Digest::from(key))),
             _ => Err(AccountError::StorageSlotNotMap(index)),
         }
     }
diff --git a/crates/miden-objects/src/errors.rs b/crates/miden-objects/src/errors.rs
index bf1dc58275..39811d3d8a 100644
--- a/crates/miden-objects/src/errors.rs
+++ b/crates/miden-objects/src/errors.rs
@@ -227,6 +227,18 @@ pub enum AccountDeltaError {
     NotAFungibleFaucetId(AccountId),
 }
 
+// STORAGE MAP ERROR
+// ================================================================================================
+
+#[derive(Debug, Error)]
+pub enum StorageMapError {
+    #[error("map entries contain key {key} twice with values {value0} and {value1}",
+      value0 = vm_core::utils::to_hex(Felt::elements_as_bytes(value0)),
+      value1 = vm_core::utils::to_hex(Felt::elements_as_bytes(value1))
+    )]
+    DuplicateKey { key: Digest, value0: Word, value1: Word },
+}
+
 // BATCH ACCOUNT UPDATE ERROR
 // ================================================================================================
 
diff --git a/crates/miden-tx/src/tests/kernel_tests/test_faucet.rs b/crates/miden-tx/src/tests/kernel_tests/test_faucet.rs
index c3b57ab97d..1617000f16 100644
--- a/crates/miden-tx/src/tests/kernel_tests/test_faucet.rs
+++ b/crates/miden-tx/src/tests/kernel_tests/test_faucet.rs
@@ -11,7 +11,7 @@ use miden_lib::{
 };
 use miden_objects::{
     FieldElement,
-    account::AccountId,
+    account::{AccountId, StorageMap},
     asset::{FungibleAsset, NonFungibleAsset},
     testing::{
         account_id::{
@@ -222,7 +222,7 @@ fn test_mint_non_fungible_asset_succeeds() {
         end
         ",
         non_fungible_asset = word_to_masm_push_string(&non_fungible_asset.into()),
-        asset_vault_key = word_to_masm_push_string(&asset_vault_key),
+        asset_vault_key = word_to_masm_push_string(&StorageMap::hash_key(asset_vault_key.into())),
     );
 
     tx_context.execute_code(&code).unwrap();


Reviews:
user: login: bobbinth
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T17:34:11Z
user: login: Mirko-von-Leipzig
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T18:26:35Z
user: login: Mirko-von-Leipzig
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T18:28:19Z
user: login: bobbinth
body: Looks good! Thank you! I left a few small comments inline. One potentially bigger thing is related to https://github.com/0xPolygonMiden/miden-base/pull/1250#discussion_r2014676929 - but I'm actually not sure what's the best way to go there.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T00:34:48Z
user: login: PhilippGackstatter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T07:34:40Z
user: login: PhilippGackstatter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T07:38:20Z
user: login: PhilippGackstatter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T08:04:56Z
user: login: bobbinth
body: Looks good! Thank you!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T01:48:09Z
user: login: bobbinth
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T01:48:51Z
user: login: PhilippGackstatter
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T07:10:51Z

Review comments:
diff_hunk: @@ -1,37 +1,60 @@
-use miden_crypto::merkle::{EmptySubtreeRoots, MerkleError};
+use alloc::collections::BTreeMap;
+
+use miden_crypto::merkle::EmptySubtreeRoots;
+use vm_core::{EMPTY_WORD, Felt, FieldElement};
 
 use super::{
     ByteReader, ByteWriter, Deserializable, DeserializationError, Digest, Serializable, Word,
 };
 use crate::{
+    Hasher,
     account::StorageMapDelta,
     crypto::{
         hash::rpo::RpoDigest,
         merkle::{InnerNodeInfo, LeafIndex, SMT_DEPTH, Smt, SmtLeaf, SmtProof},
     },
+    errors::StorageMapError,
 };
 
 // ACCOUNT STORAGE MAP
 // ================================================================================================
 
 /// Empty storage map root.
-pub const EMPTY_STORAGE_MAP_ROOT: Digest =
-    *EmptySubtreeRoots::entry(StorageMap::STORAGE_MAP_TREE_DEPTH, 0);
-
-/// Account storage map is a Sparse Merkle Tree of depth 64. It can be used to store more data as
-/// there is in plain usage of the storage slots. The root of the SMT consumes one account storage
-/// slot.
+pub const EMPTY_STORAGE_MAP_ROOT: Digest = *EmptySubtreeRoots::entry(StorageMap::TREE_DEPTH, 0);
+
+/// An account storage map which is a sparse merkle tree of depth [`Self::TREE_DEPTH`] (64).
+///
+/// It can be used to store a large amount of data in an account than would be otherwise possible
+/// using just the account's storage slots. This works by storing the root of the map's underlying
+/// SMT in one account storage slot. Each map entry is a leaf in the tree and its inclusion is
+/// proven while retrieving it (e.g. via `account::get_map_item`).
+///
+/// As a side-effect, this also means that _not all_ entries of the map have to be present at
+/// transaction execution time in order to access or modify the map. It is sufficient if _just_ the
+/// accessed/modified items are present in the advice provider.
+///
+/// Because the keys of the map are user-chosen and thus not necessarily uniformly distributed, the
+/// trees could be imbalanced and made less efficient. To mitigate that, the keys used in the
+/// storage map are hashed before they are inserted into the SMT. The original keys are retained in
+/// a separate map. This causes redundancy but allows for introspection of the map, e.g. by querying
+/// the set of stored (original) keys which is useful in debugging and explorer scenarios.
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct StorageMap {
-    map: Smt,
+    /// The SMT where each key is the hashed original key.
+    smt: Smt,
+    /// The entries of the map where the key is the original user-chosen one.
+    ///
+    /// It is an invariant of this type that the map's entries are always consistent with the SMT's
+    /// entries and vice-versa.
+    map: BTreeMap<RpoDigest, Word>,
path: crates/miden-objects/src/account/storage/map.rs
user: login: bobbinth
body: One thing that I was thinking: could this be just a `BTreeSet` of keys? Technically, if we have the original keys, with one extra hash we can always compute the actual key and so, we should be able to retrieve the value from the `Smt`.

On the other hand, storing entries in a separate map is convenient. So, I guess we'd be giving up some convenience and performance for smaller memory footprint. Not sure if it is worth it though.
created_at: 2025-03-26T17:34:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +169,19 @@ impl StorageMap {
 
         self.root()
     }
+
+    /// Consumes the map and returns the underlying map of entries.
+    pub fn into_entries(self) -> BTreeMap<RpoDigest, Word> {
+        self.map
+    }
+
+    /// Hashes the given key to get the key of the SMT.
+    pub fn hash_key(key: RpoDigest) -> RpoDigest {
+        let padded_input =
+            [Felt::ZERO, Felt::ZERO, Felt::ZERO, Felt::ZERO, key[0], key[1], key[2], key[3]];
path: crates/miden-objects/src/account/storage/map.rs
user: login: Mirko-von-Leipzig
body: Why does this need padding?
created_at: 2025-03-26T18:24:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,37 +1,60 @@
-use miden_crypto::merkle::{EmptySubtreeRoots, MerkleError};
+use alloc::collections::BTreeMap;
+
+use miden_crypto::merkle::EmptySubtreeRoots;
+use vm_core::{EMPTY_WORD, Felt, FieldElement};
 
 use super::{
     ByteReader, ByteWriter, Deserializable, DeserializationError, Digest, Serializable, Word,
 };
 use crate::{
+    Hasher,
     account::StorageMapDelta,
     crypto::{
         hash::rpo::RpoDigest,
         merkle::{InnerNodeInfo, LeafIndex, SMT_DEPTH, Smt, SmtLeaf, SmtProof},
     },
+    errors::StorageMapError,
 };
 
 // ACCOUNT STORAGE MAP
 // ================================================================================================
 
 /// Empty storage map root.
-pub const EMPTY_STORAGE_MAP_ROOT: Digest =
-    *EmptySubtreeRoots::entry(StorageMap::STORAGE_MAP_TREE_DEPTH, 0);
-
-/// Account storage map is a Sparse Merkle Tree of depth 64. It can be used to store more data as
-/// there is in plain usage of the storage slots. The root of the SMT consumes one account storage
-/// slot.
+pub const EMPTY_STORAGE_MAP_ROOT: Digest = *EmptySubtreeRoots::entry(StorageMap::TREE_DEPTH, 0);
+
+/// An account storage map which is a sparse merkle tree of depth [`Self::TREE_DEPTH`] (64).
+///
+/// It can be used to store a large amount of data in an account than would be otherwise possible
+/// using just the account's storage slots. This works by storing the root of the map's underlying
+/// SMT in one account storage slot. Each map entry is a leaf in the tree and its inclusion is
+/// proven while retrieving it (e.g. via `account::get_map_item`).
+///
+/// As a side-effect, this also means that _not all_ entries of the map have to be present at
+/// transaction execution time in order to access or modify the map. It is sufficient if _just_ the
+/// accessed/modified items are present in the advice provider.
+///
+/// Because the keys of the map are user-chosen and thus not necessarily uniformly distributed, the
+/// trees could be imbalanced and made less efficient. To mitigate that, the keys used in the
+/// storage map are hashed before they are inserted into the SMT. The original keys are retained in
+/// a separate map. This causes redundancy but allows for introspection of the map, e.g. by querying
+/// the set of stored (original) keys which is useful in debugging and explorer scenarios.
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct StorageMap {
-    map: Smt,
+    /// The SMT where each key is the hashed original key.
+    smt: Smt,
+    /// The entries of the map where the key is the original user-chosen one.
+    ///
+    /// It is an invariant of this type that the map's entries are always consistent with the SMT's
+    /// entries and vice-versa.
+    map: BTreeMap<RpoDigest, Word>,
path: crates/miden-objects/src/account/storage/map.rs
user: login: Mirko-von-Leipzig
body: I think I'd always take the performance - memory is generally cheap.

For me the benefit would rather be one less invariant to worry about maintaining.

But no strong opinion tbh.
created_at: 2025-03-26T18:28:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1132,3 +1138,17 @@ export.validate_current_foreign_account
     # clean the stack
     dropw drop drop dropw
 end
+
+#! Hashes the map key.
+#!
+#! Inputs:  [KEY]
+#! Outputs: [HASHED_KEY]
+proc.hash_map_key
+  # pad four elements for the capacity of the hasher
+  # pad another four elements so the hash input is a multiple of 8
+  padw swapw padw swapw
+  # => [KEY, pad(8)]
+
+  hperm exec.rpo::squeeze_digest
+  # => [HASHED_KEY]
+end
path: crates/miden-lib/asm/kernels/transaction/lib/account.masm
user: login: bobbinth
body: We could probably just use the `hash` [instruction](https://0xpolygonmiden.github.io/miden-vm/user_docs/assembly/cryptographic_operations.html#hashing-and-merkle-trees).
created_at: 2025-03-27T00:17:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -629,13 +632,16 @@ export.set_map_item.12
     emit.ACCOUNT_STORAGE_BEFORE_SET_MAP_ITEM_EVENT
     # => [index, KEY, NEW_VALUE, OLD_ROOT]
 
-    # duplicate the KEY and the NEW_VALUE to be able to emit an event after an account storage item
-    # is being updated
+    # duplicate the original KEY and the NEW_VALUE to be able to emit an event after the
+    # account storage item was updated
     movdn.12 movupw.2 dupw.2 dupw.2
     # => [KEY, NEW_VALUE, OLD_ROOT, KEY, NEW_VALUE, index, ...]
 
-    # set the NEW_VALUE under KEY in the tree
-    # note smt::set expects the stack to be [NEW_VALUE, KEY, OLD_ROOT, ...]
+    exec.hash_map_key
+    # => [HASHED_KEY, NEW_VALUE, OLD_ROOT, KEY, NEW_VALUE, index, ...]
path: crates/miden-lib/asm/kernels/transaction/lib/account.masm
user: login: bobbinth
body: Should we add comments (either here or as doc comments) explaining why this is being done?
created_at: 2025-03-27T00:20:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -584,7 +584,10 @@ export.get_map_item
     exec.get_item swapw
     # => [KEY, ROOT]
 
-    # fetch the VALUE located under KEY in the tree
+    exec.hash_map_key
+    # => [HASHED_KEY, ROOT]
path: crates/miden-lib/asm/kernels/transaction/lib/account.masm
user: login: bobbinth
body: Similar to the above comment: I'd add comments explaining this somewhere.
created_at: 2025-03-27T00:20:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +169,19 @@ impl StorageMap {
 
         self.root()
     }
+
+    /// Consumes the map and returns the underlying map of entries.
+    pub fn into_entries(self) -> BTreeMap<RpoDigest, Word> {
+        self.map
+    }
+
+    /// Hashes the given key to get the key of the SMT.
+    pub fn hash_key(key: RpoDigest) -> RpoDigest {
+        let padded_input =
+            [Felt::ZERO, Felt::ZERO, Felt::ZERO, Felt::ZERO, key[0], key[1], key[2], key[3]];
path: crates/miden-objects/src/account/storage/map.rs
user: login: bobbinth
body: If we use the `hash` instruction in MASM - we shouldn't need this - i.e., we should be able to do something like:
```rust
Hasher::hash_elements(key.as_elements())
```
created_at: 2025-03-27T00:29:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,37 +1,60 @@
-use miden_crypto::merkle::{EmptySubtreeRoots, MerkleError};
+use alloc::collections::BTreeMap;
+
+use miden_crypto::merkle::EmptySubtreeRoots;
+use vm_core::{EMPTY_WORD, Felt, FieldElement};
 
 use super::{
     ByteReader, ByteWriter, Deserializable, DeserializationError, Digest, Serializable, Word,
 };
 use crate::{
+    Hasher,
     account::StorageMapDelta,
     crypto::{
         hash::rpo::RpoDigest,
         merkle::{InnerNodeInfo, LeafIndex, SMT_DEPTH, Smt, SmtLeaf, SmtProof},
     },
+    errors::StorageMapError,
 };
 
 // ACCOUNT STORAGE MAP
 // ================================================================================================
 
 /// Empty storage map root.
-pub const EMPTY_STORAGE_MAP_ROOT: Digest =
-    *EmptySubtreeRoots::entry(StorageMap::STORAGE_MAP_TREE_DEPTH, 0);
-
-/// Account storage map is a Sparse Merkle Tree of depth 64. It can be used to store more data as
-/// there is in plain usage of the storage slots. The root of the SMT consumes one account storage
-/// slot.
+pub const EMPTY_STORAGE_MAP_ROOT: Digest = *EmptySubtreeRoots::entry(StorageMap::TREE_DEPTH, 0);
+
+/// An account storage map which is a sparse merkle tree of depth [`Self::TREE_DEPTH`] (64).
+///
+/// It can be used to store a large amount of data in an account than would be otherwise possible
+/// using just the account's storage slots. This works by storing the root of the map's underlying
+/// SMT in one account storage slot. Each map entry is a leaf in the tree and its inclusion is
+/// proven while retrieving it (e.g. via `account::get_map_item`).
+///
+/// As a side-effect, this also means that _not all_ entries of the map have to be present at
+/// transaction execution time in order to access or modify the map. It is sufficient if _just_ the
+/// accessed/modified items are present in the advice provider.
+///
+/// Because the keys of the map are user-chosen and thus not necessarily uniformly distributed, the
+/// trees could be imbalanced and made less efficient. To mitigate that, the keys used in the
+/// storage map are hashed before they are inserted into the SMT. The original keys are retained in
+/// a separate map. This causes redundancy but allows for introspection of the map, e.g. by querying
+/// the set of stored (original) keys which is useful in debugging and explorer scenarios.
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct StorageMap {
path: crates/miden-objects/src/account/storage/map.rs
user: login: bobbinth
body: Some of this info should probably make it into the mdBook - but maybe this is something we leave for https://github.com/0xPolygonMiden/miden-base/issues/1213.
created_at: 2025-03-27T00:34:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +169,19 @@ impl StorageMap {
 
         self.root()
     }
+
+    /// Consumes the map and returns the underlying map of entries.
+    pub fn into_entries(self) -> BTreeMap<RpoDigest, Word> {
+        self.map
+    }
+
+    /// Hashes the given key to get the key of the SMT.
+    pub fn hash_key(key: RpoDigest) -> RpoDigest {
+        let padded_input =
+            [Felt::ZERO, Felt::ZERO, Felt::ZERO, Felt::ZERO, key[0], key[1], key[2], key[3]];
path: crates/miden-objects/src/account/storage/map.rs
user: login: PhilippGackstatter
body: Thanks, will change it.

One thing I'm generally missing is documentation on hashing in the VM and the equivalent operations in Rust. The existing docs on [hashing](https://0xpolygonmiden.github.io/miden-docs/imported/miden-vm/src/user_docs/assembly/cryptographic_operations.html#hashing-and-merkle-trees) do not really explain when to use `hash`, `hperm` or `hmerge`, what the "rate" and "capacity" words are, how to initialize them, how to get the final digest after an `hperm` (e.g. using `rpo::squeeze_digest`) or if `hmerge` should only be used on two digests or if it can also be used on non-digests. I think this would be good to document for external users as well.
created_at: 2025-03-27T07:34:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,37 +1,60 @@
-use miden_crypto::merkle::{EmptySubtreeRoots, MerkleError};
+use alloc::collections::BTreeMap;
+
+use miden_crypto::merkle::EmptySubtreeRoots;
+use vm_core::{EMPTY_WORD, Felt, FieldElement};
 
 use super::{
     ByteReader, ByteWriter, Deserializable, DeserializationError, Digest, Serializable, Word,
 };
 use crate::{
+    Hasher,
     account::StorageMapDelta,
     crypto::{
         hash::rpo::RpoDigest,
         merkle::{InnerNodeInfo, LeafIndex, SMT_DEPTH, Smt, SmtLeaf, SmtProof},
     },
+    errors::StorageMapError,
 };
 
 // ACCOUNT STORAGE MAP
 // ================================================================================================
 
 /// Empty storage map root.
-pub const EMPTY_STORAGE_MAP_ROOT: Digest =
-    *EmptySubtreeRoots::entry(StorageMap::STORAGE_MAP_TREE_DEPTH, 0);
-
-/// Account storage map is a Sparse Merkle Tree of depth 64. It can be used to store more data as
-/// there is in plain usage of the storage slots. The root of the SMT consumes one account storage
-/// slot.
+pub const EMPTY_STORAGE_MAP_ROOT: Digest = *EmptySubtreeRoots::entry(StorageMap::TREE_DEPTH, 0);
+
+/// An account storage map which is a sparse merkle tree of depth [`Self::TREE_DEPTH`] (64).
+///
+/// It can be used to store a large amount of data in an account than would be otherwise possible
+/// using just the account's storage slots. This works by storing the root of the map's underlying
+/// SMT in one account storage slot. Each map entry is a leaf in the tree and its inclusion is
+/// proven while retrieving it (e.g. via `account::get_map_item`).
+///
+/// As a side-effect, this also means that _not all_ entries of the map have to be present at
+/// transaction execution time in order to access or modify the map. It is sufficient if _just_ the
+/// accessed/modified items are present in the advice provider.
+///
+/// Because the keys of the map are user-chosen and thus not necessarily uniformly distributed, the
+/// trees could be imbalanced and made less efficient. To mitigate that, the keys used in the
+/// storage map are hashed before they are inserted into the SMT. The original keys are retained in
+/// a separate map. This causes redundancy but allows for introspection of the map, e.g. by querying
+/// the set of stored (original) keys which is useful in debugging and explorer scenarios.
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct StorageMap {
-    map: Smt,
+    /// The SMT where each key is the hashed original key.
+    smt: Smt,
+    /// The entries of the map where the key is the original user-chosen one.
+    ///
+    /// It is an invariant of this type that the map's entries are always consistent with the SMT's
+    /// entries and vice-versa.
+    map: BTreeMap<RpoDigest, Word>,
path: crates/miden-objects/src/account/storage/map.rs
user: login: PhilippGackstatter
body: Yes, initially I had started the implementation with a `BTreeSet` but then I thought having the extra hash operation in the common operations (like `StorageMap::get`) would be somewhat inefficient. Iirc, RPO hashing is not _very_ efficient in Rust, so trading memory for performance makes sense to me. Plus, the difference between having a `BTreeSet<Digest>` or a `BTreeMap<Digest, Word>` where the value takes up just four more field elements should be almost the same.

And the nice thing is that serialization can just use the map and reconstruct the SMT instead, so the byte representation is more or less the same as before.
created_at: 2025-03-27T07:38:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,37 +1,60 @@
-use miden_crypto::merkle::{EmptySubtreeRoots, MerkleError};
+use alloc::collections::BTreeMap;
+
+use miden_crypto::merkle::EmptySubtreeRoots;
+use vm_core::{EMPTY_WORD, Felt, FieldElement};
 
 use super::{
     ByteReader, ByteWriter, Deserializable, DeserializationError, Digest, Serializable, Word,
 };
 use crate::{
+    Hasher,
     account::StorageMapDelta,
     crypto::{
         hash::rpo::RpoDigest,
         merkle::{InnerNodeInfo, LeafIndex, SMT_DEPTH, Smt, SmtLeaf, SmtProof},
     },
+    errors::StorageMapError,
 };
 
 // ACCOUNT STORAGE MAP
 // ================================================================================================
 
 /// Empty storage map root.
-pub const EMPTY_STORAGE_MAP_ROOT: Digest =
-    *EmptySubtreeRoots::entry(StorageMap::STORAGE_MAP_TREE_DEPTH, 0);
-
-/// Account storage map is a Sparse Merkle Tree of depth 64. It can be used to store more data as
-/// there is in plain usage of the storage slots. The root of the SMT consumes one account storage
-/// slot.
+pub const EMPTY_STORAGE_MAP_ROOT: Digest = *EmptySubtreeRoots::entry(StorageMap::TREE_DEPTH, 0);
+
+/// An account storage map which is a sparse merkle tree of depth [`Self::TREE_DEPTH`] (64).
+///
+/// It can be used to store a large amount of data in an account than would be otherwise possible
+/// using just the account's storage slots. This works by storing the root of the map's underlying
+/// SMT in one account storage slot. Each map entry is a leaf in the tree and its inclusion is
+/// proven while retrieving it (e.g. via `account::get_map_item`).
+///
+/// As a side-effect, this also means that _not all_ entries of the map have to be present at
+/// transaction execution time in order to access or modify the map. It is sufficient if _just_ the
+/// accessed/modified items are present in the advice provider.
+///
+/// Because the keys of the map are user-chosen and thus not necessarily uniformly distributed, the
+/// trees could be imbalanced and made less efficient. To mitigate that, the keys used in the
+/// storage map are hashed before they are inserted into the SMT. The original keys are retained in
+/// a separate map. This causes redundancy but allows for introspection of the map, e.g. by querying
+/// the set of stored (original) keys which is useful in debugging and explorer scenarios.
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct StorageMap {
path: crates/miden-objects/src/account/storage/map.rs
user: login: PhilippGackstatter
body: Agreed, I would leave that for that issue.
created_at: 2025-03-27T08:04:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +169,19 @@ impl StorageMap {
 
         self.root()
     }
+
+    /// Consumes the map and returns the underlying map of entries.
+    pub fn into_entries(self) -> BTreeMap<RpoDigest, Word> {
+        self.map
+    }
+
+    /// Hashes the given key to get the key of the SMT.
+    pub fn hash_key(key: RpoDigest) -> RpoDigest {
+        let padded_input =
+            [Felt::ZERO, Felt::ZERO, Felt::ZERO, Felt::ZERO, key[0], key[1], key[2], key[3]];
path: crates/miden-objects/src/account/storage/map.rs
user: login: bobbinth
body: Agreed! Let's create an issue for this in the VM.
created_at: 2025-03-28T01:48:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,6 +169,19 @@ impl StorageMap {
 
         self.root()
     }
+
+    /// Consumes the map and returns the underlying map of entries.
+    pub fn into_entries(self) -> BTreeMap<RpoDigest, Word> {
+        self.map
+    }
+
+    /// Hashes the given key to get the key of the SMT.
+    pub fn hash_key(key: RpoDigest) -> RpoDigest {
+        let padded_input =
+            [Felt::ZERO, Felt::ZERO, Felt::ZERO, Felt::ZERO, key[0], key[1], key[2], key[3]];
path: crates/miden-objects/src/account/storage/map.rs
user: login: PhilippGackstatter
body: Created https://github.com/0xPolygonMiden/miden-vm/issues/1730 :+1:.
created_at: 2025-03-28T07:10:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: PhilippGackstatter
created_at: 2025-03-26T13:52:11Z
body: One other thing I noticed while implementing this is that we seem to use `Word` and `RpoDigest` inconsistently/incorrectly. The keys of a storage map are not actual digests, they are just words, but we use `RpoDigest` in the API. And we have to use that currently, because `Word` doesn't implement `PartialOrd/Ord` because `Felt` doesn't implement that.

Further up in the code we do use `key: Word` and then we just convert it, which suggests that it really should be `Word` in the map as well.

https://github.com/0xPolygonMiden/miden-base/blob/67c5ff8d2f53356a5a1f75a9f71ddee1f4e94346/crates/miden-objects/src/account/storage/mod.rs#L130-L136

It seems to me the correct thing to do would be to implement those traits for `Felt` and then use `Word` as the key of an SMT and StorageMap.
Relatedly, the `value` of `PartialMerkleTree` is currently `RpoDigest` (e.g. https://docs.rs/miden-crypto/latest/miden_crypto/merkle/struct.PartialMerkleTree.html#method.add_path), but it should be `Word` afaict - the value is generally not a digest. We could check other data structures in miden-crypto as well for this. I'm happy to open an issue in miden-crypto unless I'm missing the reason why this is correct as-is. The fixes would be mostly cosmetic and therefore simple afaict.
author_association: CONTRIBUTOR
user: login: Mirko-von-Leipzig
created_at: 2025-03-26T18:32:43Z
body: > One other thing I noticed while implementing this is that we seem to use `Word` and `RpoDigest` inconsistently/incorrectly. The keys of a storage map are not actual digests, they are just words, but we use `RpoDigest` in the API. And we have to use that currently, because `Word` doesn't implement `PartialOrd/Ord` because `Felt` doesn't implement that.

iirc I brought this up before and `Digest` shouldn't have an order because its a hash was the reasoning i.e. one hash shouldn't literally be smaller/larger than another. 

tbh I think `Ord` doesn't really imply that it has the LT/GT ordering and more that their _does_ exist some way of consistently ordering thins i.e. I'm in favor of adding `Ord`. It would also be nice to use `HashMap` and friends instead of `BTreeMap` but that's maybe a separate discussion :)

author_association: CONTRIBUTOR
user: login: PhilippGackstatter
created_at: 2025-03-27T07:47:07Z
body: > tbh I think `Ord` doesn't really imply that it has the LT/GT ordering

I don't have a strong opinion on whether `Digest` should or should not have `PartialOrd`/`Ord`. But I think your reasoning makes sense, so I'm happy to keep it.

I think having `Digest` as the key in a `StorageMap` or `Smt` while those keys are often not actual digests may not be good typing. It would be reasonable for someone to assume that digests are uniformly distributed and make implementation decisions based on that, and those decisions not being reasonable when the keys are actually just `Word`s.

> It would also be nice to use HashMap and friends instead of BTreeMap but that's maybe a separate discussion :)

Yes, I think we have some places where using a `HashMap` would be better than a `BTreeMap`. Probably even whenever we don't need ordering.
author_association: CONTRIBUTOR