Repository: recallnet/ipc
Pull request #574: fix: resolve and get blobs
Author: login: sanderpick
Created: 2025-03-18T00:57:49Z | Merged: 2025-03-19T16:09:41Z

Four issues fixed here:
- The new size check during blob resolve was checking the top level hash seq hash size, not the internal blob hash size cc @avichalp 
- We also need to explicitly download using the hash sequence format during resolve in order to recursively download all hashes in the sequence.
- The buckets actor was trying to read a Cid for a HAMT that belongs to the blobs actor cc @joewagner 
- I added an `expiry` field to the bucket actors's `ObjectState` and some logic that will hide expired blobs when listing. 

Goes with https://github.com/recallnet/rust-recall/pull/238.


Diff:
diff --git a/Cargo.lock b/Cargo.lock
index df7e4ff36..15f475c5a 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -3817,6 +3817,7 @@ dependencies = [
  "ipc_ipld_resolver",
  "iroh",
  "iroh-base",
+ "iroh_manager",
  "k256 0.11.6",
  "lazy_static",
  "libipld",
@@ -6701,6 +6702,7 @@ dependencies = [
  "log",
  "lru_time_cache",
  "multihash 0.18.1",
+ "num-traits",
  "prometheus",
  "quickcheck",
  "quickcheck_macros",
@@ -7121,6 +7123,7 @@ version = "0.1.0"
 dependencies = [
  "anyhow",
  "iroh",
+ "num-traits",
 ]
 
 [[package]]
diff --git a/fendermint/actors/blobs/shared/src/lib.rs b/fendermint/actors/blobs/shared/src/lib.rs
index fac2ba33a..8601917a4 100644
--- a/fendermint/actors/blobs/shared/src/lib.rs
+++ b/fendermint/actors/blobs/shared/src/lib.rs
@@ -172,7 +172,10 @@ pub fn add_blob(
     ))?)
 }
 
-pub fn get_blob(rt: &impl Runtime, hash: state::Hash) -> Result<Option<state::Blob>, ActorError> {
+pub fn get_blob(
+    rt: &impl Runtime,
+    hash: state::Hash,
+) -> Result<Option<state::BlobInfo>, ActorError> {
     deserialize_block(extract_send_result(rt.send(
         &BLOBS_ACTOR_ADDR,
         Method::GetBlob as MethodNum,
diff --git a/fendermint/actors/blobs/shared/src/state.rs b/fendermint/actors/blobs/shared/src/state.rs
index fe3b4f49f..b629a6649 100644
--- a/fendermint/actors/blobs/shared/src/state.rs
+++ b/fendermint/actors/blobs/shared/src/state.rs
@@ -120,6 +120,66 @@ impl Account {
     }
 }
 
+/// The return type used for Account.
+#[derive(Debug, Serialize_tuple, Deserialize_tuple)]
+pub struct AccountInfo {
+    /// Total size of all blobs managed by the account.
+    pub capacity_used: u64,
+    /// Current free credit in byte-blocks that can be used for new commitments.
+    pub credit_free: Credit,
+    /// Current committed credit in byte-blocks that will be used for debits.
+    pub credit_committed: Credit,
+    /// Optional default sponsor account address.
+    pub credit_sponsor: Option<Address>,
+    /// The chain epoch of the last debit.
+    pub last_debit_epoch: ChainEpoch,
+    /// Credit approvals to other accounts from this account, keyed by receiver.
+    pub approvals_to: HashMap<Address, CreditApproval>,
+    /// Credit approvals to this account from other accounts, keyed by sender.
+    pub approvals_from: HashMap<Address, CreditApproval>,
+    /// The maximum allowed TTL for actor's blobs.
+    pub max_ttl: ChainEpoch,
+    /// The total token value an account has used to buy credits.
+    pub gas_allowance: TokenAmount,
+}
+
+impl AccountInfo {
+    pub fn from(rt: &impl Runtime, account: Account) -> Result<Self, ActorError> {
+        let store = rt.store();
+        let mut approvals_to = HashMap::new();
+        account
+            .approvals_to
+            .hamt(store)?
+            .for_each(|address, approval| {
+                let external_account_address = to_delegated_address(rt, address)?;
+                approvals_to.insert(external_account_address, approval.clone());
+                Ok(())
+            })?;
+
+        let mut approvals_from = HashMap::new();
+        account
+            .approvals_from
+            .hamt(store)?
+            .for_each(|address, approval| {
+                let external_account_address = to_delegated_address(rt, address)?;
+                approvals_from.insert(external_account_address, approval.clone());
+                Ok(())
+            })?;
+
+        Ok(AccountInfo {
+            capacity_used: account.capacity_used,
+            credit_free: account.credit_free,
+            credit_committed: account.credit_committed,
+            credit_sponsor: account.credit_sponsor,
+            last_debit_epoch: account.last_debit_epoch,
+            approvals_to,
+            approvals_from,
+            max_ttl: account.max_ttl,
+            gas_allowance: account.gas_allowance,
+        })
+    }
+}
+
 /// A credit approval from one account to another.
 #[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple)]
 pub struct CreditApproval {
@@ -238,6 +298,41 @@ pub struct Blob {
     pub status: BlobStatus,
 }
 
+/// The return type used for Blob.
+#[derive(Debug, Serialize_tuple, Deserialize_tuple)]
+pub struct BlobInfo {
+    /// The size of the content.
+    pub size: u64,
+    /// Blob metadata that contains information for blob recovery.
+    pub metadata_hash: Hash,
+    /// Active subscribers (accounts) that are paying for the blob to expiry.
+    pub subscribers: HashMap<SubscriptionId, ChainEpoch>,
+    /// Blob status.
+    pub status: BlobStatus,
+}
+
+impl BlobInfo {
+    /// Returns [`BlobInfo`] from [`Blob`].
+    /// TODO: HAMTs should carry max expiry such that we don't full scan here.
+    pub fn from(rt: &impl Runtime, blob: Blob) -> Result<Self, ActorError> {
+        let store = rt.store();
+        let mut subscribers = HashMap::new();
+        blob.subscribers.hamt(store)?.for_each(|_, group| {
+            group.hamt(store)?.for_each(|id, sub| {
+                subscribers.insert(id, sub.expiry);
+                Ok(())
+            })?;
+            Ok(())
+        })?;
+        Ok(Self {
+            size: blob.size,
+            metadata_hash: blob.metadata_hash,
+            subscribers,
+            status: blob.status,
+        })
+    }
+}
+
 #[derive(Debug, Clone, PartialEq, Serialize_tuple, Deserialize_tuple)]
 pub struct BlobSubscribers {
     pub root: hamt::Root<Address, SubscriptionGroup>,
@@ -561,65 +656,6 @@ impl CreditApprovals {
     }
 }
 
-/// The return type used for Account.
-#[derive(Debug, Serialize_tuple, Deserialize_tuple)]
-pub struct AccountInfo {
-    /// Total size of all blobs managed by the account.
-    pub capacity_used: u64,
-    /// Current free credit in byte-blocks that can be used for new commitments.
-    pub credit_free: Credit,
-    /// Current committed credit in byte-blocks that will be used for debits.
-    pub credit_committed: Credit,
-    /// Optional default sponsor account address.
-    pub credit_sponsor: Option<Address>,
-    /// The chain epoch of the last debit.
-    pub last_debit_epoch: ChainEpoch,
-    /// Credit approvals to other accounts from this account, keyed by receiver.
-    pub approvals_to: HashMap<String, CreditApproval>,
-    /// Credit approvals to this account from other accounts, keyed by sender.
-    pub approvals_from: HashMap<String, CreditApproval>,
-    /// The maximum allowed TTL for actor's blobs.
-    pub max_ttl: ChainEpoch,
-    /// The total token value an account has used to buy credits.
-    pub gas_allowance: TokenAmount,
-}
-
-impl AccountInfo {
-    pub fn from(account: Account, rt: &impl Runtime) -> Result<Self, ActorError> {
-        let mut approvals_to = HashMap::new();
-        account
-            .approvals_to
-            .hamt(rt.store())?
-            .for_each(|address, approval| {
-                let external_account_address = to_delegated_address(rt, address)?;
-                approvals_to.insert(external_account_address.to_string(), approval.clone());
-                Ok(())
-            })?;
-
-        let mut approvals_from = HashMap::new();
-        account
-            .approvals_from
-            .hamt(rt.store())?
-            .for_each(|address, approval| {
-                let external_account_address = to_delegated_address(rt, address)?;
-                approvals_from.insert(external_account_address.to_string(), approval.clone());
-                Ok(())
-            })?;
-
-        Ok(AccountInfo {
-            capacity_used: account.capacity_used,
-            credit_free: account.credit_free,
-            credit_committed: account.credit_committed,
-            credit_sponsor: account.credit_sponsor,
-            last_debit_epoch: account.last_debit_epoch,
-            approvals_to,
-            approvals_from,
-            max_ttl: account.max_ttl,
-            gas_allowance: account.gas_allowance,
-        })
-    }
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;
diff --git a/fendermint/actors/blobs/src/actor.rs b/fendermint/actors/blobs/src/actor.rs
index 4d9e805af..39f663e4c 100644
--- a/fendermint/actors/blobs/src/actor.rs
+++ b/fendermint/actors/blobs/src/actor.rs
@@ -12,7 +12,7 @@ use fendermint_actor_blobs_shared::params::{
     SetSponsorParams, TrimBlobExpiriesParams, UpdateGasAllowanceParams,
 };
 use fendermint_actor_blobs_shared::state::{
-    AccountInfo, Blob, BlobStatus, Credit, CreditApproval, GasAllowance, Hash, PublicKey,
+    AccountInfo, BlobInfo, BlobStatus, Credit, CreditApproval, GasAllowance, Hash, PublicKey,
     Subscription, SubscriptionId,
 };
 use fendermint_actor_blobs_shared::Method;
@@ -107,7 +107,7 @@ impl BlobsActor {
             credit_purchased(delegated_addr, token_to_biguint(Some(credit_amount))),
         )?;
 
-        AccountInfo::from(account, rt)
+        AccountInfo::from(rt, account)
     }
 
     /// Updates gas allowance for the `from` address.
@@ -319,8 +319,7 @@ impl BlobsActor {
                     .credit_sponsor
                     .map(|sponsor| to_delegated_address(rt, sponsor))
                     .transpose()?;
-
-                AccountInfo::from(account, rt)
+                AccountInfo::from(rt, account)
             });
 
         account.transpose()
@@ -476,10 +475,12 @@ impl BlobsActor {
     }
 
     /// Returns a blob by [`Hash`] if it exists.
-    fn get_blob(rt: &impl Runtime, params: GetBlobParams) -> Result<Option<Blob>, ActorError> {
+    fn get_blob(rt: &impl Runtime, params: GetBlobParams) -> Result<Option<BlobInfo>, ActorError> {
         rt.validate_immediate_caller_accept_any()?;
-        let blob = rt.state::<State>()?.get_blob(rt.store(), params.0)?;
-        Ok(blob)
+        match rt.state::<State>()?.get_blob(rt.store(), params.0)? {
+            Some(blob) => Ok(Some(BlobInfo::from(rt, blob)?)),
+            None => Ok(None),
+        }
     }
 
     /// Returns the current [`BlobStatus`] for a blob by [`Hash`].
@@ -1365,6 +1366,25 @@ mod tests {
         assert_eq!(subscription.expiry, 3605);
         assert_eq!(subscription.delegate, None);
         rt.verify();
+
+        // Get it back
+        rt.expect_validate_caller_any();
+        let get_params = GetBlobParams(hash.0);
+        let blob = rt
+            .call::<BlobsActor>(
+                Method::GetBlob as u64,
+                IpldBlock::serialize_cbor(&get_params).unwrap(),
+            )
+            .unwrap()
+            .unwrap()
+            .deserialize::<Option<BlobInfo>>()
+            .unwrap();
+        assert!(blob.is_some());
+        let blob = blob.unwrap();
+        assert_eq!(blob.size, add_params.size);
+        assert_eq!(blob.metadata_hash, add_params.metadata_hash);
+        assert_eq!(blob.subscribers.len(), 1);
+        assert_eq!(blob.status, BlobStatus::Added);
     }
 
     #[test]
diff --git a/fendermint/actors/bucket/src/actor.rs b/fendermint/actors/bucket/src/actor.rs
index 335170d9e..e314fbbc5 100644
--- a/fendermint/actors/bucket/src/actor.rs
+++ b/fendermint/actors/bucket/src/actor.rs
@@ -6,7 +6,7 @@ use std::collections::HashMap;
 
 use fendermint_actor_blobs_shared::{
     add_blob, delete_blob, get_blob, has_credit_approval, overwrite_blob,
-    state::{Blob, BlobStatus, SubscriptionId},
+    state::{BlobInfo, BlobStatus, SubscriptionId},
 };
 use fendermint_actor_machine::{
     events::emit_evm_event,
@@ -18,7 +18,6 @@ use fil_actors_runtime::{
     runtime::{ActorCode, Runtime},
     ActorError,
 };
-use fvm_ipld_blockstore::Blockstore;
 use fvm_ipld_hamt::BytesKey;
 use fvm_shared::address::Address;
 use recall_sol_facade::bucket::{object_added, object_deleted, object_metadata_updated};
@@ -59,8 +58,9 @@ impl Actor {
         validate_metadata(&params.metadata)?;
 
         let sub = if let Some(object) = state.get(rt.store(), &key)? {
-            // If we have existing blob
-            if params.overwrite {
+            // If we have existing blob and it's not expired
+            let expired = object.expiry <= rt.curr_epoch();
+            if params.overwrite || expired {
                 // Overwrite if the flag is passed
                 overwrite_blob(
                     rt,
@@ -101,6 +101,7 @@ impl Actor {
                 key,
                 params.hash,
                 params.size,
+                sub.expiry,
                 params.metadata.clone(),
                 params.overwrite,
             )
@@ -160,7 +161,7 @@ impl Actor {
         let key = BytesKey(params.0);
         if let Some(object_state) = state.get(rt.store(), &key)? {
             if let Some(blob) = get_blob(rt, object_state.hash)? {
-                let object = build_object(rt.store(), &blob, &object_state, sub_id, owner)?;
+                let object = build_object(&blob, &object_state, sub_id, owner)?;
                 Ok(object)
             } else {
                 Ok(None)
@@ -177,6 +178,7 @@ impl Actor {
     ) -> Result<ListObjectsReturn, ActorError> {
         rt.validate_immediate_caller_accept_any()?;
 
+        let current_epoch = rt.curr_epoch();
         let mut objects = Vec::new();
         let start_key = params.start_key.map(BytesKey::from);
         let state = rt.state::<State>()?;
@@ -187,7 +189,9 @@ impl Actor {
             start_key.as_ref(),
             params.limit,
             |key: Vec<u8>, object_state: ObjectState| -> anyhow::Result<(), ActorError> {
-                objects.push((key, object_state));
+                if object_state.expiry > current_epoch {
+                    objects.push((key, object_state));
+                }
                 Ok(())
             },
         )?;
@@ -258,6 +262,7 @@ impl Actor {
                 key,
                 object.hash,
                 object.size,
+                object.expiry,
                 object.metadata.clone(),
                 true,
             )?;
@@ -280,37 +285,27 @@ fn get_blob_id(state: &State, key: &[u8]) -> anyhow::Result<SubscriptionId, Acto
 }
 
 /// Build an object from its state and blob.
-fn build_object<BS: Blockstore>(
-    store: &BS,
-    blob: &Blob,
+fn build_object(
+    blob: &BlobInfo,
     object_state: &ObjectState,
     sub_id: SubscriptionId,
     subscriber: Address,
 ) -> anyhow::Result<Option<Object>, ActorError> {
     match blob.status {
         BlobStatus::Resolved => {
-            let subscribers = blob.subscribers.hamt(store)?;
-            let group = subscribers.get(&subscriber)?.ok_or_else(|| {
+            blob.subscribers.get(&sub_id).cloned().ok_or_else(|| {
                 ActorError::illegal_state(format!(
                     "owner {} is not subscribed to blob {}; this should not happen",
                     subscriber, object_state.hash
                 ))
             })?;
-            let (expiry, _) = group.max_expiries(store, &sub_id, None)?;
-            if let Some(expiry) = expiry {
-                Ok(Some(Object {
-                    hash: object_state.hash,
-                    recovery_hash: blob.metadata_hash,
-                    size: blob.size,
-                    expiry,
-                    metadata: object_state.metadata.clone(),
-                }))
-            } else {
-                Err(ActorError::illegal_state(format!(
-                    "subscription group is empty for blob {}; this should not happen",
-                    object_state.hash,
-                )))
-            }
+            Ok(Some(Object {
+                hash: object_state.hash,
+                recovery_hash: blob.metadata_hash,
+                size: blob.size,
+                expiry: object_state.expiry,
+                metadata: object_state.metadata.clone(),
+            }))
         }
         BlobStatus::Added | BlobStatus::Pending | BlobStatus::Failed => Ok(None),
     }
@@ -401,7 +396,7 @@ mod tests {
             AddBlobParams, DeleteBlobParams, GetBlobParams, GetCreditApprovalParams,
             OverwriteBlobParams,
         },
-        state::{BlobSubscribers, CreditApproval, Hash, Subscription, SubscriptionGroup},
+        state::{CreditApproval, Hash, Subscription},
         Method as BlobMethod, BLOBS_ACTOR_ADDR,
     };
     use fendermint_actor_blobs_testing::{new_hash, new_pk, setup_logs};
@@ -691,7 +686,14 @@ mod tests {
             })
             .unwrap(),
             TokenAmount::from_whole(0),
-            IpldBlock::serialize_cbor(&Subscription::default()).unwrap(),
+            IpldBlock::serialize_cbor(&Subscription {
+                added: 0,
+                expiry: ChainEpoch::from(3600),
+                source: add_params.source,
+                delegate: None,
+                failed: false,
+            })
+            .unwrap(),
             ExitCode::OK,
         );
         expect_emitted_add_event(&rt, &add_params);
@@ -872,7 +874,14 @@ mod tests {
             })
             .unwrap(),
             TokenAmount::from_whole(0),
-            IpldBlock::serialize_cbor(&Subscription::default()).unwrap(),
+            IpldBlock::serialize_cbor(&Subscription {
+                added: 0,
+                expiry: ttl,
+                source: add_params.source,
+                delegate: None,
+                failed: false,
+            })
+            .unwrap(),
             ExitCode::OK,
         );
         expect_emitted_add_event(&rt, &add_params);
@@ -886,40 +895,14 @@ mod tests {
         .unwrap();
         rt.verify();
 
-        let store = rt.store();
-        let blob_subscribers = BlobSubscribers::new(store).unwrap();
-
-        let mut subscribers = blob_subscribers.hamt(store).unwrap();
-
         // Get the object
-        let mut blob = Blob {
+        let blob = BlobInfo {
             size: add_params.size,
-            subscribers: blob_subscribers,
+            subscribers: HashMap::from([(sub_id, ttl)]),
             status: BlobStatus::Resolved,
             metadata_hash: add_params.recovery_hash,
         };
 
-        let mut group = SubscriptionGroup::new(store).unwrap();
-        let mut group_hamt = group.hamt(store).unwrap();
-
-        group.save_tracked(
-            group_hamt
-                .set_and_flush_tracked(
-                    &sub_id,
-                    Subscription {
-                        added: 0,
-                        expiry: ChainEpoch::from(3600),
-                        source: add_params.source,
-                        delegate: Some(origin),
-                        failed: false,
-                    },
-                )
-                .unwrap(),
-        );
-
-        blob.subscribers
-            .save_tracked(subscribers.set_and_flush_tracked(&origin, group).unwrap());
-
         rt.expect_validate_caller_any();
         rt.expect_send(
             BLOBS_ACTOR_ADDR,
@@ -962,13 +945,14 @@ mod tests {
         // Add an object
         let hash = new_hash(256);
         let key = vec![0, 1, 2];
+        let ttl = ChainEpoch::from(3600);
         let add_params: AddParams = AddParams {
             source: new_pk(),
             key: key.clone(),
             hash: hash.0,
             size: hash.1,
             recovery_hash: new_hash(256).0,
-            ttl: None,
+            ttl: Some(ttl),
             metadata: HashMap::from([("foo".into(), "bar".into()), ("foo2".into(), "bar".into())]),
             from: origin,
             overwrite: false,
@@ -984,14 +968,21 @@ mod tests {
                 source: add_params.source,
                 hash: add_params.hash,
                 metadata_hash: add_params.recovery_hash,
-                id: sub_id,
+                id: sub_id.clone(),
                 size: add_params.size,
                 ttl: add_params.ttl,
                 from: origin,
             })
             .unwrap(),
             TokenAmount::from_whole(0),
-            IpldBlock::serialize_cbor(&Subscription::default()).unwrap(),
+            IpldBlock::serialize_cbor(&Subscription {
+                added: 0,
+                expiry: ttl,
+                source: add_params.source,
+                delegate: None,
+                failed: false,
+            })
+            .unwrap(),
             ExitCode::OK,
         );
         expect_emitted_add_event(&rt, &add_params);
@@ -1004,7 +995,6 @@ mod tests {
             .unwrap()
             .deserialize::<Object>()
             .unwrap();
-        let state = rt.state::<State>().unwrap();
         assert_eq!(add_params.hash, result.hash);
         assert_eq!(add_params.metadata, result.metadata);
         assert_eq!(add_params.recovery_hash, result.recovery_hash);
@@ -1125,39 +1115,12 @@ mod tests {
         rt.verify();
 
         // Get the object and check metadata
-        let store = rt.store();
-        let blob_subscribers = BlobSubscribers::new(store).unwrap();
-        let mut subscribers = blob_subscribers.hamt(store).unwrap();
-
-        let sub_id = get_blob_id(&state, &key).unwrap();
-        let mut blob = Blob {
+        let blob = BlobInfo {
             size: add_params.size,
-            subscribers: blob_subscribers,
+            subscribers: HashMap::from([(sub_id, ttl)]),
             status: BlobStatus::Resolved,
             metadata_hash: add_params.recovery_hash,
         };
-
-        let mut group = SubscriptionGroup::new(store).unwrap();
-        let mut group_hamt = group.hamt(store).unwrap();
-
-        group.save_tracked(
-            group_hamt
-                .set_and_flush_tracked(
-                    &sub_id,
-                    Subscription {
-                        added: 0,
-                        expiry: ChainEpoch::from(3600),
-                        source: add_params.source,
-                        delegate: Some(origin),
-                        failed: false,
-                    },
-                )
-                .unwrap(),
-        );
-
-        blob.subscribers
-            .save_tracked(subscribers.set_and_flush_tracked(&origin, group).unwrap());
-
         rt.expect_validate_caller_any();
         rt.expect_send(
             BLOBS_ACTOR_ADDR,
diff --git a/fendermint/actors/bucket/src/state.rs b/fendermint/actors/bucket/src/state.rs
index 1d681c9b1..89e55f76d 100644
--- a/fendermint/actors/bucket/src/state.rs
+++ b/fendermint/actors/bucket/src/state.rs
@@ -12,7 +12,7 @@ use fil_actors_runtime::ActorError;
 use fvm_ipld_blockstore::Blockstore;
 use fvm_ipld_encoding::tuple::*;
 use fvm_ipld_hamt::{BytesKey, Config, Hamt};
-use fvm_shared::address::Address;
+use fvm_shared::{address::Address, clock::ChainEpoch};
 use serde::{Deserialize, Serialize};
 
 const MAX_LIST_LIMIT: usize = 1000;
@@ -95,6 +95,8 @@ pub struct ObjectState {
     pub hash: Hash,
     /// The object size.
     pub size: u64,
+    /// Expiry block.
+    pub expiry: ChainEpoch,
     /// User-defined object metadata (e.g., last modified timestamp, etc.).
     pub metadata: HashMap<String, String>,
 }
@@ -116,6 +118,7 @@ impl State {
         key: BytesKey,
         hash: Hash,
         size: u64,
+        expiry: ChainEpoch,
         metadata: HashMap<String, String>,
         overwrite: bool,
     ) -> anyhow::Result<Cid, ActorError> {
@@ -124,6 +127,7 @@ impl State {
         let object = ObjectState {
             hash,
             size,
+            expiry,
             metadata,
         };
         if overwrite {
@@ -226,6 +230,7 @@ mod tests {
             let hash = new_hash(u16::arbitrary(g) as usize);
             ObjectState {
                 hash: hash.0,
+                expiry: i64::arbitrary(g),
                 size: u64::arbitrary(g),
                 metadata: HashMap::arbitrary(g),
             }
@@ -240,11 +245,12 @@ mod tests {
         ObjectState {
             hash,
             size,
+            expiry: 123456789,
             metadata,
         }
     }
 
-    const OBJECT_ONE_CID: &str = "bafy2bzacealtpdigmoweehfr3573mdks5r3eaj4djoh7dzcdl7zdbkxnx2kds";
+    const OBJECT_ONE_CID: &str = "bafy2bzacea5tbd4x6okckdkb2yl7wbyjqpxkow6whr46dswwv5xj7va4uro2g";
 
     fn object_two() -> ObjectState {
         let (hash, size) = new_hash_from_vec([6, 7, 8, 9, 10, 11].to_vec());
@@ -254,6 +260,7 @@ mod tests {
         ObjectState {
             hash,
             size,
+            expiry: 123456789,
             metadata,
         }
     }
@@ -266,6 +273,7 @@ mod tests {
         ObjectState {
             hash,
             size,
+            expiry: 123456789,
             metadata,
         }
     }
@@ -335,6 +343,7 @@ mod tests {
                 BytesKey(vec![1, 2, 3]),
                 object.hash,
                 object.size,
+                object.expiry,
                 object.metadata,
                 true,
             )
@@ -354,6 +363,7 @@ mod tests {
                 key.clone(),
                 object.hash,
                 object.size,
+                object.expiry,
                 object.metadata,
                 true,
             )
@@ -372,7 +382,15 @@ mod tests {
         let key = BytesKey(vec![1, 2, 3]);
         let md = object.metadata.clone();
         state
-            .add(&store, key.clone(), object.hash, object.size, md, true)
+            .add(
+                &store,
+                key.clone(),
+                object.hash,
+                object.size,
+                object.expiry,
+                md,
+                true,
+            )
             .unwrap();
         let result = state.get(&store, &key);
 
@@ -391,6 +409,7 @@ mod tests {
             baz_key.clone(),
             object.hash,
             object.size,
+            object.expiry,
             object.metadata,
             false,
         )?;
@@ -401,6 +420,7 @@ mod tests {
             bar_key.clone(),
             object.hash,
             object.size,
+            object.expiry,
             object.metadata,
             false,
         )?;
@@ -412,6 +432,7 @@ mod tests {
             other_key.clone(),
             hash.0,
             8,
+            123456789,
             HashMap::<String, String>::new(),
             false,
         )?;
@@ -422,6 +443,7 @@ mod tests {
             jpeg_key.clone(),
             object.hash,
             object.size,
+            object.expiry,
             object.metadata,
             false,
         )?;
@@ -459,6 +481,7 @@ mod tests {
                     key.clone(),
                     object.hash,
                     object.size,
+                    object.expiry,
                     object.metadata,
                     false,
                 )
@@ -498,6 +521,7 @@ mod tests {
                     key.clone(),
                     object.hash,
                     object.size,
+                    object.expiry,
                     object.metadata,
                     false,
                 )
@@ -566,6 +590,7 @@ mod tests {
                 jpeg_key.clone(),
                 hash.0,
                 8,
+                123456789,
                 HashMap::<String, String>::new(),
                 false,
             )
@@ -578,6 +603,7 @@ mod tests {
                 bar_key.clone(),
                 hash.0,
                 8,
+                123456789,
                 HashMap::<String, String>::new(),
                 false,
             )
@@ -590,6 +616,7 @@ mod tests {
                 baz_key.clone(),
                 hash.0,
                 8,
+                123456789,
                 HashMap::<String, String>::new(),
                 false,
             )
@@ -642,6 +669,7 @@ mod tests {
                 one.clone(),
                 hash.0,
                 8,
+                123456789,
                 HashMap::<String, String>::new(),
                 false,
             )
@@ -654,6 +682,7 @@ mod tests {
                 two.clone(),
                 hash.0,
                 8,
+                123456789,
                 HashMap::<String, String>::new(),
                 false,
             )
diff --git a/fendermint/app/Cargo.toml b/fendermint/app/Cargo.toml
index 6e69c0c9d..b1f0c1e1f 100644
--- a/fendermint/app/Cargo.toml
+++ b/fendermint/app/Cargo.toml
@@ -94,6 +94,8 @@ ipc-observability = { workspace = true }
 entangler = { workspace = true }
 entangler_storage = { workspace = true }
 
+iroh_manager = { path = "../../recall/iroh_manager" }
+
 [dev-dependencies]
 tempfile = { workspace = true }
 iroh-base = { workspace = true }
diff --git a/fendermint/app/src/cmd/objects.rs b/fendermint/app/src/cmd/objects.rs
index a3b67e255..980560fea 100644
--- a/fendermint/app/src/cmd/objects.rs
+++ b/fendermint/app/src/cmd/objects.rs
@@ -11,7 +11,6 @@ use anyhow::Context;
 use bytes::Buf;
 use entangler::{ChunkRange, Config, EntanglementResult, Entangler};
 use entangler_storage::iroh::IrohStorage as EntanglerIrohStorage;
-use fendermint_actor_blobs_shared::state::Hash as BlobHash;
 use fendermint_actor_bucket::{GetParams, Object};
 use fendermint_app_settings::objects::ObjectsSettings;
 use fendermint_rpc::{client::FendermintClient, message::GasParams, QueryClient};
@@ -27,12 +26,12 @@ use iroh::{
     client::blobs::BlobStatus,
     net::NodeAddr,
 };
+use iroh_manager::{get_blob_hash_and_size, IrohManager};
 use lazy_static::lazy_static;
-use num_traits::Zero;
 use prometheus::{register_histogram, register_int_counter, Histogram, IntCounter};
 use serde::{Deserialize, Serialize};
 use thiserror::Error;
-use tracing::info;
+use tracing::{debug, info};
 use uuid::Uuid;
 use warp::{
     filters::multipart::Part,
@@ -71,28 +70,23 @@ cmd! {
                 }
 
                 let client = FendermintClient::new_http(tendermint_url, None)?;
-
-                let iroh_addr = iroh_addr
-                    .to_socket_addrs()?
-                    .next()
-                    .ok_or(anyhow!("failed to convert iroh_addr to a socket address"))?;
-                let iroh_client = iroh::client::Iroh::connect_addr(iroh_addr).await?;
+                let iroh_manager = IrohManager::from_addr(Some(iroh_addr));
 
                 // Admin routes
                 let health = warp::path!("health")
                     .and(warp::get()).and_then(handle_health);
                 let node_addr = warp::path!("v1" / "node" )
                 .and(warp::get())
-                .and(with_iroh(iroh_client.clone()))
-                .and_then(handle_node_addr);
+                .and(with_iroh(iroh_manager.clone()))
+                .and_then(handle_node_addr_with_manager);
 
                 // Objects routes
                 let objects_upload = warp::path!("v1" / "objects" )
                 .and(warp::post())
-                .and(with_iroh(iroh_client.clone()))
+                .and(with_iroh(iroh_manager.clone()))
                 .and(warp::multipart::form().max_length(settings.max_object_size + 1024 * 1024)) // max_object_size + 1MB for form overhead
                 .and(with_max_size(settings.max_object_size))
-                .and_then(handle_object_upload);
+                .and_then(handle_object_upload_with_manager);
 
                 let objects_download = warp::path!("v1" / "objects" / String / ..)
                 .and(warp::path::tail())
@@ -102,8 +96,8 @@ cmd! {
                 .and(warp::header::optional::<String>("Range"))
                 .and(warp::query::<HeightQuery>())
                 .and(with_client(client.clone()))
-                .and(with_iroh(iroh_client.clone()))
-                .and_then(handle_object_download);
+                .and(with_iroh(iroh_manager.clone()))
+                .and_then(handle_object_download_with_manager);
 
                 let router = health
                     .or(node_addr)
@@ -132,8 +126,8 @@ fn with_client(
 }
 
 fn with_iroh(
-    client: iroh::client::Iroh,
-) -> impl Filter<Extract = (iroh::client::Iroh,), Error = Infallible> + Clone {
+    client: IrohManager,
+) -> impl Filter<Extract = (IrohManager,), Error = Infallible> + Clone {
     warp::any().map(move || client.clone())
 }
 
@@ -229,7 +223,7 @@ impl ObjectParser {
                 }
                 // Ignore but accept signature-related fields for backward compatibility
                 "chain_id" | "msg" => {
-                    // Just read and discard the data
+                    // Read and discard the data
                     let _ = object_parser.read_part(part).await?;
                 }
                 _ => {
@@ -278,6 +272,15 @@ async fn handle_health() -> Result<impl Reply, Rejection> {
     Ok(warp::reply::reply())
 }
 
+async fn handle_node_addr_with_manager(mut iroh: IrohManager) -> Result<impl Reply, Rejection> {
+    let iroh_client = iroh.client().await.map_err(|e| {
+        Rejection::from(BadRequest {
+            message: format!("failed to load iroh client: {}", e),
+        })
+    })?;
+    handle_node_addr(iroh_client).await
+}
+
 async fn handle_node_addr(iroh: iroh::client::Iroh) -> Result<impl Reply, Rejection> {
     let node_addr = iroh.net().node_addr().await.map_err(|e| {
         Rejection::from(BadRequest {
@@ -293,6 +296,19 @@ struct UploadResponse {
     metadata_hash: String,
 }
 
+async fn handle_object_upload_with_manager(
+    mut iroh: IrohManager,
+    form_data: warp::multipart::FormData,
+    max_size: u64,
+) -> Result<impl Reply, Rejection> {
+    let iroh_client = iroh.client().await.map_err(|e| {
+        Rejection::from(BadRequest {
+            message: format!("failed to load iroh client: {}", e),
+        })
+    })?;
+    handle_object_upload(iroh_client, form_data, max_size).await
+}
+
 async fn handle_object_upload(
     iroh: iroh::client::Iroh,
     form_data: warp::multipart::FormData,
@@ -483,16 +499,28 @@ async fn tag_entangled_data(
         .filter(|h| h != &metadata_hash)
         .collect::<Vec<_>>();
 
-    let mut hashes = vec![orig_hash.clone(), metadata_hash];
+    let mut hashes = vec![orig_hash, metadata_hash];
     hashes.extend(upload_hashes);
 
+    let hashes_str = hashes
+        .iter()
+        .map(|h| h.to_string())
+        .collect::<Vec<_>>()
+        .join(", ");
+
     let batch = iroh.blobs().batch().await?;
 
     // make a hash sequence object from the hashes and upload it to iroh
     let hash_seq = hashes.into_iter().collect::<HashSeq>();
     let temp_tag = batch.add_bytes(hash_seq).await?;
-
     let hash_seq_hash = *temp_tag.hash();
+
+    debug!(
+        "storing hash sequence: {} ({})",
+        hash_seq_hash.to_string(),
+        hashes_str
+    );
+
     // this tag will be replaced later by the validator to "stored-seq-{hash_seq_hash}"
     let hash_seq_tag = iroh::blobs::Tag(format!("temp-seq-{hash_seq_hash}").into());
     batch.persist_to(temp_tag, hash_seq_tag).await?;
@@ -564,6 +592,32 @@ pub(crate) struct ObjectRange {
     body: Body,
 }
 
+async fn handle_object_download_with_manager<F: QueryClient + Send + Sync>(
+    address: String,
+    tail: Tail,
+    method: String,
+    range: Option<String>,
+    height_query: HeightQuery,
+    client: F,
+    mut iroh: IrohManager,
+) -> Result<impl Reply, Rejection> {
+    let iroh_client = iroh.client().await.map_err(|e| {
+        Rejection::from(BadRequest {
+            message: format!("failed to load iroh client: {}", e),
+        })
+    })?;
+    handle_object_download(
+        address,
+        tail,
+        method,
+        range,
+        height_query,
+        client,
+        iroh_client,
+    )
+    .await
+}
+
 async fn handle_object_download<F: QueryClient + Send + Sync>(
     address: String,
     tail: Tail,
@@ -594,13 +648,12 @@ async fn handle_object_download<F: QueryClient + Send + Sync>(
 
     match maybe_object {
         Some(object) => {
-            let (hash, size) = extract_blob_hash_and_size(&iroh, &object.hash)
-                .await
-                .map_err(|e| {
-                    Rejection::from(BadRequest {
-                        message: e.to_string(),
-                    })
-                })?;
+            let hash = Hash::from_bytes(object.hash.0);
+            let (hash, size) = get_blob_hash_and_size(&iroh, hash).await.map_err(|e| {
+                Rejection::from(BadRequest {
+                    message: e.to_string(),
+                })
+            })?;
 
             let ent = new_entangler(iroh).map_err(|e| {
                 Rejection::from(BadRequest {
@@ -735,72 +788,6 @@ async fn handle_object_download<F: QueryClient + Send + Sync>(
     }
 }
 
-async fn extract_blob_hash_and_size(
-    iroh: &iroh::client::Iroh,
-    object_hash: &BlobHash,
-) -> Result<(Hash, u64), anyhow::Error> {
-    let hash_seq_hash = Hash::from_bytes(object_hash.0);
-    let status = iroh.blobs().status(hash_seq_hash).await.map_err(|e| {
-        anyhow!(
-            "failed to get status for hash sequence object: {} {}",
-            hash_seq_hash,
-            e
-        )
-    })?;
-
-    let BlobStatus::Complete { size } = status else {
-        return Err(anyhow!(
-            "hash sequence object {} is not available",
-            hash_seq_hash
-        ));
-    };
-
-    if size.is_zero() {
-        return Err(anyhow!(
-            "hash sequence object {} has zero size",
-            hash_seq_hash
-        ));
-    }
-
-    let res = iroh
-        .blobs()
-        .read_to_bytes(hash_seq_hash)
-        .await
-        .map_err(|e| {
-            anyhow!(
-                "failed to read hash sequence object: {} {}",
-                hash_seq_hash,
-                e
-            )
-        })?;
-
-    let hash_seq = HashSeq::try_from(res).map_err(|e| {
-        anyhow!(
-            "failed to parse hash sequence object: {} {}",
-            hash_seq_hash,
-            e
-        )
-    })?;
-
-    let blob_hash = hash_seq.get(0).ok_or_else(|| {
-        anyhow!(
-            "failed to get hash with index 0 from hash sequence object: {}",
-            hash_seq_hash
-        )
-    })?;
-
-    let status = iroh
-        .blobs()
-        .status(blob_hash)
-        .await
-        .map_err(|e| anyhow!("failed to read object: {} {}", blob_hash, e))?;
-
-    let BlobStatus::Complete { size } = status else {
-        return Err(anyhow!("object {} is not available", blob_hash));
-    };
-    Ok((blob_hash, size))
-}
-
 /// Parse an f/eth-address from string.
 pub fn parse_address(s: &str) -> anyhow::Result<Address> {
     let addr = Network::Mainnet
@@ -886,6 +873,7 @@ mod tests {
     use super::*;
     use async_trait::async_trait;
     use bytes::Bytes;
+    use fendermint_actor_blobs_shared::state::Hash as BlobHash;
     use fendermint_vm_message::query::FvmQuery;
     use rand_chacha::rand_core::{RngCore, SeedableRng};
     use rand_chacha::ChaCha8Rng;
@@ -944,8 +932,8 @@ mod tests {
         metadata_iroh_hash: Hash,
     ) -> MockQueryClient {
         let object = Object {
-            hash: BlobHash(hash_seq_hash.as_bytes().clone()),
-            recovery_hash: BlobHash(metadata_iroh_hash.as_bytes().clone()),
+            hash: BlobHash(*hash_seq_hash.as_bytes()),
+            recovery_hash: BlobHash(*metadata_iroh_hash.as_bytes()),
             metadata: HashMap::from([
                 ("foo".to_string(), "bar".to_string()),
                 (
@@ -1019,7 +1007,7 @@ mod tests {
         let client_node_addr = client_iroh.net().node_addr().await.unwrap();
         let size = 11;
 
-        // Create multipart form for source-based upload
+        // Create the multipart form for source-based upload
         let boundary = "--abcdef1234--";
         let mut body = Vec::new();
         let form_data = format!(
diff --git a/fendermint/app/src/cmd/run.rs b/fendermint/app/src/cmd/run.rs
index fb38f59d3..755d2994f 100644
--- a/fendermint/app/src/cmd/run.rs
+++ b/fendermint/app/src/cmd/run.rs
@@ -641,7 +641,7 @@ async fn dispatch_vote(
             }
         }
         AppVote::BlobFinality(f) => {
-            debug!(hash = ?f.hash, success = ?f.success, "received vote for blob finality");
+            debug!(hash = %f.hash, success = ?f.success, "received vote for blob finality");
             match atomically_or_err(|| {
                 parent_finality_votes.add_blob_vote(
                     vote.public_key.clone(),
@@ -676,7 +676,7 @@ async fn dispatch_vote(
             }
         }
         AppVote::ReadRequestClosed(r) => {
-            debug!(hash = ?r.hash, "received vote for read request completion");
+            debug!(hash = %r.hash, "received vote for read request completion");
             match atomically_or_err(|| {
                 parent_finality_votes.add_blob_vote(
                     vote.public_key.clone(),
diff --git a/fendermint/vm/interpreter/src/chain.rs b/fendermint/vm/interpreter/src/chain.rs
index b7093dd4a..ac3b186bf 100644
--- a/fendermint/vm/interpreter/src/chain.rs
+++ b/fendermint/vm/interpreter/src/chain.rs
@@ -338,7 +338,7 @@ where
             state.state_tree_mut().begin_transaction();
             for item in local_finalized_blobs.iter() {
                 if is_blob_finalized(&mut state, item.subscriber, item.hash, item.id.clone())? {
-                    tracing::debug!(hash = ?item.hash, "blob already finalized on chain; removing from pool");
+                    tracing::debug!(hash = %item.hash, "blob already finalized on chain; removing from pool");
                     atomically(|| chain_env.blob_pool.remove_task(item)).await;
                     // Remove the result from the pool
                     atomically(|| chain_env.blob_pool.remove_result(item)).await;
@@ -351,7 +351,7 @@ where
                 })
                 .await;
                 if is_globally_finalized {
-                    tracing::debug!(hash = ?item.hash, size = item.size, "blob has quorum; adding tx to chain");
+                    tracing::debug!(hash = %item.hash, size = item.size, "blob has quorum; adding tx to chain");
                     blobs.push(ChainMessage::Ipc(IpcMessage::BlobFinalized(
                         FinalizedBlob {
                             subscriber: item.subscriber,
@@ -533,7 +533,7 @@ where
                     // Check that blobs that are being enqueued are still in "added" state in the actor
                     // Once we enqueue a blob, the actor will transition it to "pending" state.
                     if !is_blob_added(&mut state, blob.subscriber, blob.hash, blob.id)? {
-                        tracing::debug!(hash = ?blob.hash, "blob is not added onchain; rejecting proposal");
+                        tracing::debug!(hash = %blob.hash, "blob is not added onchain; rejecting proposal");
                         return Ok(false);
                     }
                 }
@@ -545,7 +545,7 @@ where
                         is_blob_finalized(state, blob.subscriber, blob.hash, blob.id.clone())
                     })?;
                     if is_blob_finalized {
-                        tracing::debug!(hash = ?blob.hash, "blob is already finalized on chain; rejecting proposal");
+                        tracing::debug!(hash = %blob.hash, "blob is already finalized on chain; rejecting proposal");
                         return Ok(false);
                     }
 
@@ -556,12 +556,12 @@ where
                     })
                     .await;
                     if !is_globally_finalized {
-                        tracing::debug!(hash = ?blob.hash, "blob is not globally finalized; rejecting proposal");
+                        tracing::debug!(hash = %blob.hash, "blob is not globally finalized; rejecting proposal");
                         return Ok(false);
                     }
                     if blob.succeeded != succeeded {
                         tracing::debug!(
-                            hash = ?blob.hash,
+                            hash = %blob.hash,
                             quorum = ?succeeded,
                             message = ?blob.succeeded,
                             "blob finalization mismatch; rejecting proposal"
@@ -584,12 +584,12 @@ where
                         })
                         .await;
                     if is_locally_finalized {
-                        tracing::debug!(hash = ?blob.hash, "blob is locally finalized; removing from pool");
+                        tracing::debug!(hash = %blob.hash, "blob is locally finalized; removing from pool");
                         atomically(|| chain_env.blob_pool.remove_task(&item)).await;
                         // Remove the result from the pool
                         atomically(|| chain_env.blob_pool.remove_result(&item)).await;
                     } else {
-                        tracing::debug!(hash = ?blob.hash, "blob is not locally finalized");
+                        tracing::debug!(hash = %blob.hash, "blob is not locally finalized");
                     }
                 }
                 ChainMessage::Ipc(IpcMessage::ReadRequestPending(read_request)) => {
@@ -608,7 +608,7 @@ where
                         get_read_request_status(state, read_request.id)
                     })?;
                     if !matches!(status, Some(ReadRequestStatus::Pending)) {
-                        tracing::info!(hash = ?read_request.id, "only pending read requests can be closed; rejecting proposal");
+                        tracing::info!(hash = %read_request.id, "only pending read requests can be closed; rejecting proposal");
                         return Ok(false);
                     }
 
@@ -626,7 +626,7 @@ where
                     })
                     .await;
                     if !is_globally_finalized {
-                        tracing::info!(hash = ?read_request.id, "read request is not globally finalized; rejecting proposal");
+                        tracing::info!(hash = %read_request.id, "read request is not globally finalized; rejecting proposal");
                         return Ok(false);
                     }
 
@@ -884,7 +884,7 @@ where
                     let (apply_ret, emitters) = state.execute_implicit(msg)?;
 
                     tracing::debug!(
-                        hash = ?blob.hash,
+                        hash = %blob.hash,
                         "chain interpreter has set blob to pending"
                     );
 
@@ -933,7 +933,7 @@ where
                     let (apply_ret, emitters) = state.execute_implicit(msg)?;
 
                     tracing::debug!(
-                        hash = ?blob.hash,
+                        hash = %blob.hash,
                         "chain interpreter has finalized blob"
                     );
 
@@ -985,7 +985,7 @@ where
                     let ret = close_read_request(&mut state, read_request.id)?;
 
                     tracing::debug!(
-                        hash = ?read_request.id,
+                        hash = %read_request.id,
                         "read request is closed"
                     );
 
diff --git a/fendermint/vm/iroh_resolver/src/iroh.rs b/fendermint/vm/iroh_resolver/src/iroh.rs
index 1808048ea..b545802e0 100644
--- a/fendermint/vm/iroh_resolver/src/iroh.rs
+++ b/fendermint/vm/iroh_resolver/src/iroh.rs
@@ -101,7 +101,7 @@ fn start_resolve<V>(
     V: Clone + Send + Sync + Serialize + DeserializeOwned + 'static,
 {
     tokio::spawn(async move {
-        tracing::debug!(hash = ?task.hash(), "starting iroh blob resolve");
+        tracing::debug!(hash = %task.hash(), "starting iroh blob resolve");
         match task.task_type() {
             TaskType::ResolveBlob { source, size } => {
                 match client
@@ -117,7 +117,7 @@ fn start_resolve<V>(
                         // By not quitting, we should see this error every time there is a new task, which is at least a constant reminder.
                     }
                     Ok(Ok(())) => {
-                        tracing::debug!(hash = ?task.hash(), "iroh blob resolved");
+                        tracing::debug!(hash = %task.hash(), "iroh blob resolved");
                         atomically(|| task.set_resolved()).await;
                         if add_own_vote(
                             task.hash(),
@@ -276,7 +276,7 @@ where
 async fn reenqueue(task: ResolveTask, queue: ResolveQueue, retry_delay: Duration) -> bool {
     if atomically(|| task.add_attempt()).await {
         tracing::error!(
-            hash = ?task.hash(),
+            hash = %task.hash(),
             "iroh blob resolution failed; retrying later"
         );
         schedule_retry(task, queue, retry_delay).await;
diff --git a/fendermint/vm/topdown/src/voting.rs b/fendermint/vm/topdown/src/voting.rs
index 8c168964f..711e0cd12 100644
--- a/fendermint/vm/topdown/src/voting.rs
+++ b/fendermint/vm/topdown/src/voting.rs
@@ -444,6 +444,9 @@ where
                 } else {
                     power_table.get(vk).cloned().unwrap_or_default()
                 };
+
+                tracing::debug!("voter; key={}, power={}", vk.to_string(), power);
+
                 if *resolved {
                     resolved_weight += power;
                 } else {
@@ -452,6 +455,14 @@ where
             }
         }
 
+        tracing::debug!(
+            resolved_weight,
+            failed_weight,
+            quorum_threshold,
+            "blob quorum; votes={}",
+            votes_for_blob.len()
+        );
+
         if resolved_weight >= quorum_threshold {
             Ok((true, true))
         } else if failed_weight >= quorum_threshold {
diff --git a/ipld/resolver/Cargo.toml b/ipld/resolver/Cargo.toml
index 1fc80ee60..7d2eccff9 100644
--- a/ipld/resolver/Cargo.toml
+++ b/ipld/resolver/Cargo.toml
@@ -31,6 +31,7 @@ serde_json = { workspace = true, features = ["raw_value"] }
 thiserror = { workspace = true }
 tokio = { workspace = true }
 hex = { workspace = true }
+num-traits = { workspace = true }
 
 fvm_ipld_encoding = { workspace = true }
 fvm_shared = { workspace = true }
diff --git a/ipld/resolver/src/service.rs b/ipld/resolver/src/service.rs
index 221ebec1a..d8d69eb15 100644
--- a/ipld/resolver/src/service.rs
+++ b/ipld/resolver/src/service.rs
@@ -19,7 +19,7 @@ use iroh::blobs::Hash;
 use iroh::client::blobs::ReadAtLen;
 use iroh::client::Iroh;
 use iroh::net::NodeAddr;
-use iroh_manager::IrohManager;
+use iroh_manager::{get_blob_hash_and_size, IrohManager};
 use libipld::store::StoreParams;
 use libipld::Cid;
 use libp2p::connection_limits::ConnectionLimits;
@@ -672,20 +672,19 @@ pub fn build_transport(local_key: Keypair) -> Boxed<(PeerId, StreamMuxerBox)> {
 
 async fn download_blob(
     iroh: Iroh,
-    hash: Hash,
+    seq_hash: Hash,
     size: u64,
     node_addr: NodeAddr,
 ) -> anyhow::Result<()> {
+    // Download top-level blob
     // Use an explicit tag so we can keep track of it
-    // TODO: this needs to be tagged with a "user id"
-    let tag = iroh::blobs::Tag(format!("stored-seq-{hash}").into());
-    let res = iroh
-        .blobs()
+    let tag = iroh::blobs::Tag(format!("stored-seq-{seq_hash}").into());
+    iroh.blobs()
         .download_with_opts(
-            hash,
+            seq_hash,
             iroh::client::blobs::DownloadOptions {
-                format: iroh::blobs::BlobFormat::Raw,
-                nodes: vec![node_addr],
+                format: iroh::blobs::BlobFormat::HashSeq,
+                nodes: vec![node_addr.clone()],
                 tag: iroh::blobs::util::SetTagOption::Named(tag),
                 mode: iroh::client::blobs::DownloadMode::Queued,
             },
@@ -693,25 +692,27 @@ async fn download_blob(
         .await?
         .await?;
 
-    if res.local_size + res.downloaded_size != size {
+    // Verify downloaded size of user blob matches the expected size
+    let (_, size_actual) = get_blob_hash_and_size(&iroh, seq_hash).await?;
+    if size != size_actual {
         return Err(anyhow!(
             "downloaded blob size {} does not match expected size {}",
-            res.local_size + res.downloaded_size,
+            size_actual,
             size
         ));
     }
 
-    debug!("downloaded blob {}: {:?}", hash, res);
-
     // Delete the temporary tag (this might fail as not all nodes will have one).
-    // TODO: this needs to be tagged with a "user id"
-    let tag = iroh::blobs::Tag(format!("temp-seq-{hash}").into());
+    let tag = iroh::blobs::Tag(format!("temp-seq-{seq_hash}").into());
     iroh.tags().delete(tag).await.ok();
 
+    debug!("downloaded blob {}", seq_hash);
+
     Ok(())
 }
 
 async fn read_blob(iroh: Iroh, hash: Hash, offset: u32, len: u32) -> anyhow::Result<bytes::Bytes> {
+    let (hash, _) = get_blob_hash_and_size(&iroh, hash).await?;
     let len = ReadAtLen::AtMost(len as u64);
     let res = iroh
         .blobs()
diff --git a/recall/iroh_manager/Cargo.toml b/recall/iroh_manager/Cargo.toml
index e982f9e1b..dffe2e9ba 100644
--- a/recall/iroh_manager/Cargo.toml
+++ b/recall/iroh_manager/Cargo.toml
@@ -7,4 +7,5 @@ license.workspace = true
 
 [dependencies]
 anyhow = { workspace = true }
+num-traits = { workspace = true }
 iroh = { workspace = true }
diff --git a/recall/iroh_manager/src/lib.rs b/recall/iroh_manager/src/lib.rs
index 01c7104c8..abde8b42d 100644
--- a/recall/iroh_manager/src/lib.rs
+++ b/recall/iroh_manager/src/lib.rs
@@ -5,8 +5,13 @@
 use std::net::ToSocketAddrs;
 
 use anyhow::anyhow;
+use iroh::blobs::hashseq::HashSeq;
+use iroh::blobs::Hash;
+use iroh::client::blobs::BlobStatus;
 use iroh::client::Iroh;
+use num_traits::Zero;
 
+/// Helper for managing Iroh connections.
 #[derive(Clone, Debug)]
 pub struct IrohManager {
     addr: Option<String>,
@@ -14,10 +19,13 @@ pub struct IrohManager {
 }
 
 impl IrohManager {
+    /// Returns a manager for the address.
     pub fn from_addr(addr: Option<String>) -> IrohManager {
         Self { addr, client: None }
     }
 
+    /// Returns the Iroh client.
+    /// The underlying client will be created if it does not exist.  
     pub async fn client(&mut self) -> anyhow::Result<Iroh> {
         if let Some(c) = self.client.clone() {
             return Ok(c);
@@ -38,3 +46,57 @@ impl IrohManager {
         }
     }
 }
+
+/// Returns the user blob hash and size from the hash sequence.
+/// The user blob hash is the first hash in the sequence.
+pub async fn get_blob_hash_and_size(
+    iroh: &Iroh,
+    seq_hash: Hash,
+) -> Result<(Hash, u64), anyhow::Error> {
+    // Get the hash sequence status (it needs to be available)
+    let status = iroh.blobs().status(seq_hash).await.map_err(|e| {
+        anyhow!(
+            "failed to get status for hash sequence object: {} {}",
+            seq_hash,
+            e
+        )
+    })?;
+    let BlobStatus::Complete { size } = status else {
+        return Err(anyhow!(
+            "hash sequence object {} is not available",
+            seq_hash
+        ));
+    };
+    if size.is_zero() {
+        return Err(anyhow!("hash sequence object {} has zero size", seq_hash));
+    }
+
+    // Read the bytes and create a hash sequence
+    let res = iroh
+        .blobs()
+        .read_to_bytes(seq_hash)
+        .await
+        .map_err(|e| anyhow!("failed to read hash sequence object: {} {}", seq_hash, e))?;
+    let hash_seq = HashSeq::try_from(res)
+        .map_err(|e| anyhow!("failed to parse hash sequence object: {} {}", seq_hash, e))?;
+
+    // Get the user blob status at index 0 (it needs to be available)
+    let blob_hash = hash_seq.get(0).ok_or_else(|| {
+        anyhow!(
+            "failed to get hash with index 0 from hash sequence object: {}",
+            seq_hash
+        )
+    })?;
+    let status = iroh
+        .blobs()
+        .status(blob_hash)
+        .await
+        .map_err(|e| anyhow!("failed to read object: {} {}", blob_hash, e))?;
+
+    // Finally, get the size from the status
+    let BlobStatus::Complete { size } = status else {
+        return Err(anyhow!("object {} is not available", blob_hash));
+    };
+
+    Ok((blob_hash, size))
+}
diff --git a/scripts/deploy_subnet/deploy.sh b/scripts/deploy_subnet/deploy.sh
index 55b6d8b9a..d468de36e 100755
--- a/scripts/deploy_subnet/deploy.sh
+++ b/scripts/deploy_subnet/deploy.sh
@@ -520,8 +520,8 @@ cd "$IPC_FOLDER"
 # Approve each validator to stake
 for i in {0..2}
 do
-  # Approve power min 1 RECALL max 10 RECALL
-  cast send --private-key "$pk" --rpc-url "$rpc_url" --timeout 120 "$VALIDATOR_GATER_ADDRESS" "approve(address,uint256,uint256)" "${wallet_addresses[i]}" 1000000000000000000 100000000000000000000
+  # Approve power min 1 RECALL max 100 RECALL
+  cast send --private-key "$pk" --rpc-url "$rpc_url" --timeout 120 "$VALIDATOR_GATER_ADDRESS" "approve(address,uint256,uint256)" "${wallet_addresses[i]}" 1000000000000000000 1000000000000000000000
 done
 echo "Approved validators to stake"
 
@@ -589,7 +589,7 @@ do
   echo "Joining subnet ${subnet_id} for validator ${wallet_addresses[i]}"
   # Approve subnet contract to lock up to 10 RECALL from collateral contract (which is also the supply source contract)
   vpk=$(cat "${IPC_CONFIG_FOLDER}"/validator_"$i".sk)
-  cast send --private-key "$vpk" --rpc-url "$rpc_url" --timeout 120 "$SUPPLY_SOURCE_ADDRESS" "approve(address,uint256)" "$subnet_eth_addr" 10000000000000000000
+  cast send --private-key "$vpk" --rpc-url "$rpc_url" --timeout 120 "$SUPPLY_SOURCE_ADDRESS" "approve(address,uint256)" "$subnet_eth_addr" 100000000000000000000
   # Join and stake 10 RECALL
   ipc-cli subnet join --from "${wallet_addresses[i]}" --subnet "$subnet_id" --collateral 10
 done
@@ -827,7 +827,7 @@ if [[ $local_deploy = true ]]; then
     # transaction until it succeeds—else, we'll continue and simply skip the account since it's not
     # recommended to use the validator accounts during development, anyways
     while [ $attempt -le $max_retries ]; do
-      if cast send --rpc-url http://localhost:"${ETHAPI_HOST_PORTS[0]}" $CREDIT_MANAGER_ADDRESS "buyCredit()" --value "${credit_amount}ether" --private-key $private_key 2>/dev/null; then
+      if cast send --rpc-url http://localhost:"${ETHAPI_HOST_PORTS[0]}" "$CREDIT_MANAGER_ADDRESS" "buyCredit()" --value "${credit_amount}ether" --private-key "$private_key" 2>/dev/null; then
         echo "Successfully bought credits for account $i"
         break
       else
@@ -914,7 +914,7 @@ if [[ $local_deploy = true ]]; then
   echo "Parent RECALL:   ${parent_recall%.*} RECALL"
   echo "Subnet native:  ${subnet_native%.*} RECALL"
   # get credit balance (i.e., to help verify the credit purchases above worked)
-  account_info_output=$(cast call --rpc-url http://localhost:"${ETHAPI_HOST_PORTS[0]}" $CREDIT_MANAGER_ADDRESS "getAccount(address)" "${addr}")
+  account_info_output=$(cast call --rpc-url http://localhost:"${ETHAPI_HOST_PORTS[0]}" "$CREDIT_MANAGER_ADDRESS" "getAccount(address)" "${addr}")
   account_info=$(cast abi-decode "getAccount(address)((uint64,uint256,uint256,address,uint64,(address,(uint256,uint256,uint64,uint256,uint256))[],(address,(uint256,uint256,uint64,uint256,uint256))[],uint64,uint256))" "$account_info_output")
   credit_balance=$(echo "$account_info" | awk -F',' '{gsub(/[^0-9]/, "", $2); print substr($2, 1, length($2)-18)}')
   echo "Subnet credits: ${credit_balance}"


Reviews:
user: login: avichalp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T01:39:55Z
user: login: avichalp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T01:50:28Z
user: login: avichalp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T01:57:58Z
user: login: avichalp
body: Looks good!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T02:02:14Z
user: login: sanderpick
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T03:19:44Z
user: login: joewagner
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T14:58:12Z
user: login: joewagner
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T15:10:33Z
user: login: sanderpick
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T18:39:35Z
user: login: sanderpick
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T07:12:36Z
user: login: dignifiedquire
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T09:23:10Z
user: login: dignifiedquire
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T09:26:51Z
user: login: dtbuchholz
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T14:08:53Z
user: login: joewagner
body: This is great!  
I had to nuke all of my docker artifacts, but I just got the integration tests to pass with this change :partying_face: 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T14:33:02Z
user: login: sanderpick
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T16:07:17Z

Review comments:
diff_hunk: @@ -280,37 +279,27 @@ fn get_blob_id(state: &State, key: &[u8]) -> anyhow::Result<SubscriptionId, Acto
 }
 
 /// Build an object from its state and blob.
-fn build_object<BS: Blockstore>(
-    store: &BS,
-    blob: &Blob,
+fn build_object(
+    blob: &BlobInfo,
     object_state: &ObjectState,
     sub_id: SubscriptionId,
     subscriber: Address,
 ) -> anyhow::Result<Option<Object>, ActorError> {
     match blob.status {
         BlobStatus::Resolved => {
-            let subscribers = blob.subscribers.hamt(store)?;
-            let group = subscribers.get(&subscriber)?.ok_or_else(|| {
path: fendermint/actors/bucket/src/actor.rs
user: login: avichalp
body: 👍 
created_at: 2025-03-18T01:39:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -238,6 +298,41 @@ pub struct Blob {
     pub status: BlobStatus,
 }
 
+/// The return type used for Blob.
+#[derive(Debug, Serialize_tuple, Deserialize_tuple)]
+pub struct BlobInfo {
+    /// The size of the content.
+    pub size: u64,
+    /// Blob metadata that contains information for blob recovery.
+    pub metadata_hash: Hash,
+    /// Active subscribers (accounts) that are paying for the blob to expiry.
+    pub subscribers: HashMap<SubscriptionId, ChainEpoch>,
+    /// Blob status.
+    pub status: BlobStatus,
+}
+
+impl BlobInfo {
+    /// Returns [`BlobInfo`] from [`Blob`].
+    /// TODO: HAMTs should carry max expiry such that we don't full scan here.
path: fendermint/actors/blobs/shared/src/state.rs
user: login: avichalp
body: we do paginate the expiry now so this TODO could be removed, right? or is this a TODO for something else?
created_at: 2025-03-18T01:50:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -720,3 +724,54 @@ async fn read_blob(iroh: Iroh, hash: Hash, offset: u32, len: u32) -> anyhow::Res
     debug!("read blob {}: {:?}", hash, res);
     Ok(res)
 }
+
+async fn extract_blob_hash_and_size(
path: ipld/resolver/src/service.rs
user: login: avichalp
body: ah thanks for catching this! so, now every blob has a corresponding blob that stores its "hash sequence" and the actual blob is retrieved by fetching the first hash from the sequence.
created_at: 2025-03-18T01:57:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -238,6 +298,41 @@ pub struct Blob {
     pub status: BlobStatus,
 }
 
+/// The return type used for Blob.
+#[derive(Debug, Serialize_tuple, Deserialize_tuple)]
+pub struct BlobInfo {
+    /// The size of the content.
+    pub size: u64,
+    /// Blob metadata that contains information for blob recovery.
+    pub metadata_hash: Hash,
+    /// Active subscribers (accounts) that are paying for the blob to expiry.
+    pub subscribers: HashMap<SubscriptionId, ChainEpoch>,
+    /// Blob status.
+    pub status: BlobStatus,
+}
+
+impl BlobInfo {
+    /// Returns [`BlobInfo`] from [`Blob`].
+    /// TODO: HAMTs should carry max expiry such that we don't full scan here.
path: fendermint/actors/blobs/shared/src/state.rs
user: login: sanderpick
body: this is new, for the building of `BlobInfo`
created_at: 2025-03-18T03:19:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -886,40 +875,14 @@ mod tests {
         .unwrap();
         rt.verify();
 
-        let store = rt.store();
-        let blob_subscribers = BlobSubscribers::new(store).unwrap();
-
-        let mut subscribers = blob_subscribers.hamt(store).unwrap();
-
         // Get the object
-        let mut blob = Blob {
+        let blob = BlobInfo {
             size: add_params.size,
-            subscribers: blob_subscribers,
+            subscribers: HashMap::from([(sub_id, ttl)]),
path: fendermint/actors/bucket/src/actor.rs
user: login: joewagner
body: Nice!
created_at: 2025-03-18T14:58:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -962,13 +925,14 @@ mod tests {
         // Add an object
         let hash = new_hash(256);
         let key = vec![0, 1, 2];
+        let ttl = ChainEpoch::from(3600);
         let add_params: AddParams = AddParams {
             source: new_pk(),
             key: key.clone(),
             hash: hash.0,
             size: hash.1,
             recovery_hash: new_hash(256).0,
-            ttl: None,
+            ttl: Some(ttl),
path: fendermint/actors/bucket/src/actor.rs
user: login: joewagner
body: Does this change testing logic?
created_at: 2025-03-18T15:10:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -962,13 +925,14 @@ mod tests {
         // Add an object
         let hash = new_hash(256);
         let key = vec![0, 1, 2];
+        let ttl = ChainEpoch::from(3600);
         let add_params: AddParams = AddParams {
             source: new_pk(),
             key: key.clone(),
             hash: hash.0,
             size: hash.1,
             recovery_hash: new_hash(256).0,
-            ttl: None,
+            ttl: Some(ttl),
path: fendermint/actors/bucket/src/actor.rs
user: login: sanderpick
body: not for this test
created_at: 2025-03-18T18:39:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -59,8 +58,9 @@ impl Actor {
         validate_metadata(&params.metadata)?;
 
         let sub = if let Some(object) = state.get(rt.store(), &key)? {
-            // If we have existing blob
-            if params.overwrite {
+            // If we have existing blob and it's not expired
+            let expired = object.expiry <= rt.curr_epoch();
+            if params.overwrite || expired {
path: fendermint/actors/bucket/src/actor.rs
user: login: sanderpick
body: don't require overwrite flag if the object is expired. otherwise it would be confusing when looking at the output from listing objects... the key won't show there anymore.
created_at: 2025-03-19T07:06:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -187,7 +189,9 @@ impl Actor {
             start_key.as_ref(),
             params.limit,
             |key: Vec<u8>, object_state: ObjectState| -> anyhow::Result<(), ActorError> {
-                objects.push((key, object_state));
+                if object_state.expiry > current_epoch {
+                    objects.push((key, object_state));
+                }
path: fendermint/actors/bucket/src/actor.rs
user: login: sanderpick
body: only add to list if not expired
created_at: 2025-03-19T07:06:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -71,28 +70,23 @@ cmd! {
                 }
 
                 let client = FendermintClient::new_http(tendermint_url, None)?;
-
-                let iroh_addr = iroh_addr
-                    .to_socket_addrs()?
-                    .next()
-                    .ok_or(anyhow!("failed to convert iroh_addr to a socket address"))?;
-                let iroh_client = iroh::client::Iroh::connect_addr(iroh_addr).await?;
+                let iroh_manager = IrohManager::from_addr(Some(iroh_addr));
path: fendermint/app/src/cmd/objects.rs
user: login: sanderpick
body: use the iroh manager here so we don't have to worry about iroh being available when starting the objects API
created_at: 2025-03-19T07:08:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -483,16 +499,28 @@ async fn tag_entangled_data(
         .filter(|h| h != &metadata_hash)
         .collect::<Vec<_>>();
 
-    let mut hashes = vec![orig_hash.clone(), metadata_hash];
+    let mut hashes = vec![orig_hash, metadata_hash];
     hashes.extend(upload_hashes);
 
+    let hashes_str = hashes
+        .iter()
+        .map(|h| h.to_string())
+        .collect::<Vec<_>>()
+        .join(", ");
+
     let batch = iroh.blobs().batch().await?;
 
     // make a hash sequence object from the hashes and upload it to iroh
     let hash_seq = hashes.into_iter().collect::<HashSeq>();
     let temp_tag = batch.add_bytes(hash_seq).await?;
-
     let hash_seq_hash = *temp_tag.hash();
+
+    debug!(
+        "storing hash sequence: {} ({})",
+        hash_seq_hash.to_string(),
+        hashes_str
+    );
path: fendermint/app/src/cmd/objects.rs
user: login: sanderpick
body: the changes here are just for logging
created_at: 2025-03-19T07:08:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -735,72 +788,6 @@ async fn handle_object_download<F: QueryClient + Send + Sync>(
     }
 }
 
-async fn extract_blob_hash_and_size(
path: fendermint/app/src/cmd/objects.rs
user: login: sanderpick
body: moved to iroh manager crate since we also need it in the resolver. 
created_at: 2025-03-19T07:09:07Z
author_association: CONTRIBUTOR
path: fendermint/vm/topdown/src/voting.rs
user: login: sanderpick
body: just some more logging
created_at: 2025-03-19T07:09:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -672,46 +674,77 @@ pub fn build_transport(local_key: Keypair) -> Boxed<(PeerId, StreamMuxerBox)> {
 
 async fn download_blob(
     iroh: Iroh,
-    hash: Hash,
+    seq_hash: Hash,
     size: u64,
     node_addr: NodeAddr,
 ) -> anyhow::Result<()> {
+    // Download top-level blob
     // Use an explicit tag so we can keep track of it
-    // TODO: this needs to be tagged with a "user id"
-    let tag = iroh::blobs::Tag(format!("stored-seq-{hash}").into());
-    let res = iroh
-        .blobs()
+    let tag = iroh::blobs::Tag(format!("stored-seq-{seq_hash}").into());
+    iroh.blobs()
         .download_with_opts(
-            hash,
+            seq_hash,
             iroh::client::blobs::DownloadOptions {
                 format: iroh::blobs::BlobFormat::Raw,
-                nodes: vec![node_addr],
+                nodes: vec![node_addr.clone()],
                 tag: iroh::blobs::util::SetTagOption::Named(tag),
                 mode: iroh::client::blobs::DownloadMode::Queued,
             },
         )
         .await?
         .await?;
 
-    if res.local_size + res.downloaded_size != size {
-        return Err(anyhow!(
-            "downloaded blob size {} does not match expected size {}",
-            res.local_size + res.downloaded_size,
-            size
-        ));
+    // Read top-level blob as a hash sequence
+    let res = iroh
+        .blobs()
+        .read_to_bytes(seq_hash)
+        .await
+        .map_err(|e| anyhow!("failed to read hash sequence object: {} {}", seq_hash, e))?;
+    let hash_seq = HashSeq::try_from(res)
+        .map_err(|e| anyhow!("failed to parse hash sequence object: {} {}", seq_hash, e))?;
+
+    // Download blobs in the hash sequence
path: ipld/resolver/src/service.rs
user: login: sanderpick
body: @dignifiedquire seems this is required, ie, downloading just the top level blob does not make the hashes in the sequence available. 
created_at: 2025-03-19T07:10:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -672,46 +674,77 @@ pub fn build_transport(local_key: Keypair) -> Boxed<(PeerId, StreamMuxerBox)> {
 
 async fn download_blob(
     iroh: Iroh,
-    hash: Hash,
+    seq_hash: Hash,
     size: u64,
     node_addr: NodeAddr,
 ) -> anyhow::Result<()> {
+    // Download top-level blob
     // Use an explicit tag so we can keep track of it
-    // TODO: this needs to be tagged with a "user id"
-    let tag = iroh::blobs::Tag(format!("stored-seq-{hash}").into());
-    let res = iroh
-        .blobs()
+    let tag = iroh::blobs::Tag(format!("stored-seq-{seq_hash}").into());
+    iroh.blobs()
         .download_with_opts(
-            hash,
+            seq_hash,
             iroh::client::blobs::DownloadOptions {
                 format: iroh::blobs::BlobFormat::Raw,
-                nodes: vec![node_addr],
+                nodes: vec![node_addr.clone()],
                 tag: iroh::blobs::util::SetTagOption::Named(tag),
                 mode: iroh::client::blobs::DownloadMode::Queued,
             },
         )
         .await?
         .await?;
 
-    if res.local_size + res.downloaded_size != size {
-        return Err(anyhow!(
-            "downloaded blob size {} does not match expected size {}",
-            res.local_size + res.downloaded_size,
-            size
-        ));
+    // Read top-level blob as a hash sequence
+    let res = iroh
+        .blobs()
+        .read_to_bytes(seq_hash)
+        .await
+        .map_err(|e| anyhow!("failed to read hash sequence object: {} {}", seq_hash, e))?;
+    let hash_seq = HashSeq::try_from(res)
+        .map_err(|e| anyhow!("failed to parse hash sequence object: {} {}", seq_hash, e))?;
+
+    // Download blobs in the hash sequence
+    for (i, hash) in hash_seq.iter().enumerate() {
+        let tmp_id = Uuid::new_v4();
+        let tmp_tag = iroh::blobs::Tag(format!("temp-{hash}-{tmp_id}").into());
+        let res = iroh
+            .blobs()
+            .download_with_opts(
+                hash,
+                iroh::client::blobs::DownloadOptions {
+                    format: iroh::blobs::BlobFormat::Raw,
+                    nodes: vec![node_addr.clone()],
+                    tag: iroh::blobs::util::SetTagOption::Named(tmp_tag.clone()),
+                    mode: iroh::client::blobs::DownloadMode::Queued,
+                },
+            )
+            .await?
+            .await?;
+        iroh.tags().delete(tmp_tag).await.ok();
path: ipld/resolver/src/service.rs
user: login: sanderpick
body: delete tmp tag. this works assuming the top-level tag (already present at this point) protects these blobs from GC.
created_at: 2025-03-19T07:11:20Z
author_association: CONTRIBUTOR
path: scripts/deploy_subnet/deploy.sh
user: login: sanderpick
body: some changes that allow for staking more collateral, which is useful to test
created_at: 2025-03-19T07:12:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -672,46 +674,77 @@ pub fn build_transport(local_key: Keypair) -> Boxed<(PeerId, StreamMuxerBox)> {
 
 async fn download_blob(
     iroh: Iroh,
-    hash: Hash,
+    seq_hash: Hash,
     size: u64,
     node_addr: NodeAddr,
 ) -> anyhow::Result<()> {
+    // Download top-level blob
     // Use an explicit tag so we can keep track of it
-    // TODO: this needs to be tagged with a "user id"
-    let tag = iroh::blobs::Tag(format!("stored-seq-{hash}").into());
-    let res = iroh
-        .blobs()
+    let tag = iroh::blobs::Tag(format!("stored-seq-{seq_hash}").into());
+    iroh.blobs()
         .download_with_opts(
-            hash,
+            seq_hash,
             iroh::client::blobs::DownloadOptions {
                 format: iroh::blobs::BlobFormat::Raw,
path: ipld/resolver/src/service.rs
user: login: dignifiedquire
body: @sanderpick 
this line is the issue, you need to use `iroh::blobs::BlobFormat::HashSeq` to trigger downloading the whole sequence, passing in `Raw` treats this as a single blob
created_at: 2025-03-19T09:23:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -672,46 +674,77 @@ pub fn build_transport(local_key: Keypair) -> Boxed<(PeerId, StreamMuxerBox)> {
 
 async fn download_blob(
     iroh: Iroh,
-    hash: Hash,
+    seq_hash: Hash,
     size: u64,
     node_addr: NodeAddr,
 ) -> anyhow::Result<()> {
+    // Download top-level blob
     // Use an explicit tag so we can keep track of it
-    // TODO: this needs to be tagged with a "user id"
-    let tag = iroh::blobs::Tag(format!("stored-seq-{hash}").into());
-    let res = iroh
-        .blobs()
+    let tag = iroh::blobs::Tag(format!("stored-seq-{seq_hash}").into());
+    iroh.blobs()
         .download_with_opts(
-            hash,
+            seq_hash,
             iroh::client::blobs::DownloadOptions {
                 format: iroh::blobs::BlobFormat::Raw,
-                nodes: vec![node_addr],
+                nodes: vec![node_addr.clone()],
                 tag: iroh::blobs::util::SetTagOption::Named(tag),
                 mode: iroh::client::blobs::DownloadMode::Queued,
             },
         )
         .await?
         .await?;
 
-    if res.local_size + res.downloaded_size != size {
-        return Err(anyhow!(
-            "downloaded blob size {} does not match expected size {}",
-            res.local_size + res.downloaded_size,
-            size
-        ));
+    // Read top-level blob as a hash sequence
+    let res = iroh
+        .blobs()
+        .read_to_bytes(seq_hash)
+        .await
+        .map_err(|e| anyhow!("failed to read hash sequence object: {} {}", seq_hash, e))?;
+    let hash_seq = HashSeq::try_from(res)
+        .map_err(|e| anyhow!("failed to parse hash sequence object: {} {}", seq_hash, e))?;
+
+    // Download blobs in the hash sequence
path: ipld/resolver/src/service.rs
user: login: dignifiedquire
body: see my comment above, this loop is not needed when adjusting the download of the hashseq itself
created_at: 2025-03-19T09:26:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -672,46 +674,77 @@ pub fn build_transport(local_key: Keypair) -> Boxed<(PeerId, StreamMuxerBox)> {
 
 async fn download_blob(
     iroh: Iroh,
-    hash: Hash,
+    seq_hash: Hash,
     size: u64,
     node_addr: NodeAddr,
 ) -> anyhow::Result<()> {
+    // Download top-level blob
     // Use an explicit tag so we can keep track of it
-    // TODO: this needs to be tagged with a "user id"
-    let tag = iroh::blobs::Tag(format!("stored-seq-{hash}").into());
-    let res = iroh
-        .blobs()
+    let tag = iroh::blobs::Tag(format!("stored-seq-{seq_hash}").into());
+    iroh.blobs()
         .download_with_opts(
-            hash,
+            seq_hash,
             iroh::client::blobs::DownloadOptions {
                 format: iroh::blobs::BlobFormat::Raw,
-                nodes: vec![node_addr],
+                nodes: vec![node_addr.clone()],
                 tag: iroh::blobs::util::SetTagOption::Named(tag),
                 mode: iroh::client::blobs::DownloadMode::Queued,
             },
         )
         .await?
         .await?;
 
-    if res.local_size + res.downloaded_size != size {
-        return Err(anyhow!(
-            "downloaded blob size {} does not match expected size {}",
-            res.local_size + res.downloaded_size,
-            size
-        ));
+    // Read top-level blob as a hash sequence
+    let res = iroh
+        .blobs()
+        .read_to_bytes(seq_hash)
+        .await
+        .map_err(|e| anyhow!("failed to read hash sequence object: {} {}", seq_hash, e))?;
+    let hash_seq = HashSeq::try_from(res)
+        .map_err(|e| anyhow!("failed to parse hash sequence object: {} {}", seq_hash, e))?;
+
+    // Download blobs in the hash sequence
path: ipld/resolver/src/service.rs
user: login: sanderpick
body: great ty. fixed
created_at: 2025-03-19T16:07:17Z
author_association: CONTRIBUTOR

PR comments:
user: login: dtbuchholz
created_at: 2025-03-18T02:16:33Z
body: @sanderpick unfortunately, i'm still seeing the same error on localnet for `Error: object not found for key '...'`

steps to reproduce:
1. checkout `sander/fix-getblob` in this repo locally, and run the localnet deploy script
2. check out `rust-recall`'s branch [`dtb/fix-getblob`](https://github.com/recallnet/rust-recall/tree/dtb/fix-getblob), and run `make install`
3. with the CLI set up for localnet, create a bucket and add/get an object...you'll see the error about the object not being found
```
export PRIVATE_KEY=0x7c852118294e51e653712a81e05800f419141751be58f605c371e15141b007a6
export RECALL_NETWORK=localnet
export BUCKET_ADDR=""

BUCKET_ADDR=$(recall bu create | jq '.address' | tr -d '"')
echo 'test' > ~/test.txt
recall bu add --address $BUCKET_ADDR --key test ~/test.txt
recall bu get --address $BUCKET_ADDR test
```
author_association: CONTRIBUTOR
user: login: avichalp
created_at: 2025-03-18T02:34:20Z
body: I am actually seeing this in the current `testnet` too which should not have these commits 🤔 

![Screenshot 2025-03-17 at 7 33 18 PM](https://github.com/user-attachments/assets/e7e5271f-bbf5-4fb7-acd9-2b1df6823fad)



```sh
cargo run -- --network testnet bu get -a 0xFf0000000000000000000000000000000000033B foo

Error: object not found for key 'foo'
```
author_association: CONTRIBUTOR
user: login: dtbuchholz
created_at: 2025-03-18T02:44:54Z
body: > I am actually seeing this in the current `testnet` too which should not have these commits 🤔

@avichalp yeah, there's some convo about this on discord...i think it's purely coincidental because the testnet is using https://github.com/recallnet/ipc/commit/963ec53b1f45e6ddda135d66b3358f3713c8f6cd, which is before where this issue seems to start showing up https://github.com/recallnet/ipc/commit/ec02304bd74e5f1f37c7e8becc0f4eb3c07042dc.

although....we just had new validators join, right? were any of them using the _latest_ from `ipc`, which means they _are_ running `ipc` with this bug? if so, would mismatched software versions break the network like this?

edit: is there an API that validators run to show which `ipc` commit/sha they're using? this could be helpful for debugging. the closest thing i can find is this, but i think we'd need to version `fendermint` to make it meaningful?:
```
>  curl -X GET -H "accept: application/json" https://api.testnet.recall.chain.love/abci_info | jq

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "response": {
      "data": "fendermint",
      "version": "0.1.0",
      "last_block_height": "847485",
      "last_block_app_hash": "AXGg5AIgtSiAsBSH+cBtprxEYp8pDkxpE1lWSjOf1HthjH8b0nQ="
    }
  }
}
```
author_association: CONTRIBUTOR
user: login: joewagner
created_at: 2025-03-18T16:01:02Z
body: @dtbuchholz @sanderpick I'm using this branch locally, then running the portal and integration tests.  I had to make a change to the rust sdk to get the build working, but I'm seeing a different error than Dan.

The portal and intergration tests both result in:
`Serialization error for Cbor protocol: Mismatch { expect_major: 4, byte: 161 }`

Maybe my local setup is not linked together correctly?  
author_association: CONTRIBUTOR
user: login: joewagner
created_at: 2025-03-18T16:05:04Z
body: Here is the rust sdk branch I am testing on. https://github.com/recallnet/rust-recall/pull/236
author_association: CONTRIBUTOR
user: login: dtbuchholz
created_at: 2025-03-18T17:17:15Z
body: @joewagner idk the testing suite well enough to comment on that aspect. but, i have sander's `ipc` branch set up locally with localnet, and then i installed the CLI via https://github.com/recallnet/rust-recall/pull/236. i dont see the error you do. i _can_ create a bucket and add an objects, and when i try to get the object, i see the `not found` error. i dont see anything about cbor.
author_association: CONTRIBUTOR
user: login: sanderpick
created_at: 2025-03-19T07:02:53Z
body: okay @dtbuchholz, this should be gtg. can you try localnet again. it works for me. i can help with the contract changes in the morning if helpful. requires https://github.com/recallnet/rust-recall/pull/238.
author_association: CONTRIBUTOR
user: login: sanderpick
created_at: 2025-03-19T07:04:56Z
body: also see updated description above. 
author_association: CONTRIBUTOR
user: login: dtbuchholz
created_at: 2025-03-19T14:08:44Z
body: > okay @dtbuchholz, this should be gtg. can you try localnet again. it works for me. i can help with the contract changes in the morning if helpful. requires [recallnet/rust-recall#238](https://github.com/recallnet/rust-recall/pull/238).

@sanderpick confirmed, it works for me on localnet!
author_association: CONTRIBUTOR
user: login: asutula
created_at: 2025-03-19T15:39:37Z
body: Nice work everyone!
author_association: MEMBER