Repository: near/nearcore
Pull request #13527: network: include Chunk Validators in Tier1
Author: login: saketh-are
Created: 2025-05-12T19:40:49Z | Merged: 2025-05-14T22:26:46Z

Adds chunk validators to the Tier1 participant set. In mainnet, T1 will grow in size from 100 to 250 participants.

Changing the function which calculates the participant set should be sufficient to implement this change. The code in the network crate which initiates outbound connections and accepts incoming ones operates according to the calculated participant set.

-----

OptimisticBlock was being broadcast over Tier1 to all peers. The CVs are not interested in receiving OB, so the behavior is changed to send only to BPs.**

The following message types sent over T1 are not affected in any way:
- `Tier1Handshake`, `HandshakeFailure`, and `LastEdge`, are used only to manage the connection.
- `BlockApproval` is routed specifically to the next block producer.
- `VersionedChunkEndorsement`  is routed specifically to the next few block producers.
- `VersionedPartialEncodedChunk` is routed to the block producers of current and next epochs.
- `ContractCodeRequest` is routed to one randomly chosen chunk producer.
- `ContractCodeResponse` is routed back to the author of the incoming request.

The following routed message types were already allowed on T1 and will now be sent over T1 more consistently:
- `PartialEncodedStateWitness` is routed in parts to all chunk validators.
- `PartialEncodedStateWitnessForward` is routed to all chunk validators.
- `ChunkContractAccesses` is routed to the chunk validators which aren't producers.


**I tested out sending the OB only to the next 5 block producers instead of sending it to all of them, but it caused `tests::optimistic_block::test_optimistic_block` to fail.


Diff:
diff --git a/chain/client/src/client.rs b/chain/client/src/client.rs
index c5e833a6e57..5c516f5ccec 100644
--- a/chain/client/src/client.rs
+++ b/chain/client/src/client.rs
@@ -69,7 +69,7 @@ use near_primitives::validator_signer::ValidatorSigner;
 use near_primitives::version::PROTOCOL_VERSION;
 use near_primitives::views::{CatchupStatusView, DroppedReason};
 use std::cmp::max;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
 use std::num::NonZeroUsize;
 use std::sync::Arc;
 use tracing::{debug, debug_span, error, info, warn};
@@ -174,6 +174,8 @@ pub struct Client {
     upgrade_schedule: ProtocolUpgradeVotingSchedule,
     /// Produced optimistic block.
     last_optimistic_block_produced: Option<OptimisticBlock>,
+    /// Cached precomputed set of the chunk producers for current and next epochs.
+    chunk_producer_accounts_cache: Option<(EpochId, Arc<Vec<AccountId>>)>,
 }
 
 impl AsRef<Client> for Client {
@@ -366,6 +368,7 @@ impl Client {
             chunk_distribution_network,
             upgrade_schedule,
             last_optimistic_block_produced: None,
+            chunk_producer_accounts_cache: None,
         })
     }
 
@@ -2211,7 +2214,7 @@ impl Client {
 
 impl Client {
     /// Each epoch defines a set of important accounts: block producers, chunk producers,
-    /// approvers. Low-latency reliable communication between those accounts is critical,
+    /// chunk validators. Low-latency reliable communication between those accounts is critical,
     /// so that the blocks can be produced on time. This function computes the set of
     /// important accounts (aka TIER1 accounts) so that it can be fed to PeerManager, which
     /// will take care of the traffic prioritization.
@@ -2233,8 +2236,8 @@ impl Client {
         let _guard =
             tracing::debug_span!(target: "client", "get_tier1_accounts(): recomputing").entered();
 
-        // What we really need are: chunk producers, block producers and block approvers for
-        // this epoch and the beginning of the next epoch (so that all required connections are
+        // What we really need are: chunk producers/validators and block producers/approvers
+        // for this epoch and the beginning of the next epoch (so that all required connections are
         // established in advance). Note that block producers and block approvers are not
         // exactly the same - last blocks of this epoch will also need to be signed by the
         // block producers of the next epoch. On the other hand, block approvers
@@ -2242,10 +2245,10 @@ impl Client {
         // definitely don't need to connect to right now). Still, as long as there is no big churn
         // in the set of block producers, it doesn't make much difference.
         //
-        // With the current implementation we just fetch chunk producers and block producers
+        // With the current implementation we just fetch chunk producers/validators and block producers
         // of this and the next epoch (which covers what we need, as described above), but may
         // require some tuning in the future. In particular, if we decide that connecting to
-        // block & chunk producers of the next epoch is too expensive, we can postpone it
+        // the newly important nodes of the next epoch is too expensive, we can postpone it
         // till almost the end of this epoch.
         let mut account_keys = AccountKeys::new();
         for epoch_id in [&tip.epoch_id, &tip.next_epoch_id] {
@@ -2266,12 +2269,43 @@ impl Client {
                     .or_default()
                     .insert(bp.public_key().clone());
             }
+            for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+                account_keys
+                    .entry(v.account_id().clone())
+                    .or_default()
+                    .insert(v.public_key().clone());
+            }
         }
         let account_keys = Arc::new(account_keys);
-        self.tier1_accounts_cache = Some((tip.epoch_id, account_keys.clone()));
+        self.tier1_accounts_cache = Some((tip.epoch_id, Arc::clone(&account_keys)));
         Ok(account_keys)
     }
 
+    /// We send the optimistic block to the chunk producers of the current and next epochs.
+    pub(crate) fn get_optimistic_block_targets(
+        &mut self,
+        tip: &Tip,
+    ) -> Result<Arc<Vec<AccountId>>, Error> {
+        match &self.chunk_producer_accounts_cache {
+            Some(it) if it.0 == tip.epoch_id => return Ok(it.1.clone()),
+            _ => {}
+        }
+
+        let _span =
+            tracing::debug_span!(target: "client", "get_optimistic_block_targets(): recomputing")
+                .entered();
+
+        let mut account_ids = HashSet::new();
+        for epoch_id in [&tip.epoch_id, &tip.next_epoch_id] {
+            for cp in self.epoch_manager.get_epoch_chunk_producers(epoch_id)? {
+                account_ids.insert(cp.account_id().clone());
+            }
+        }
+        let account_ids = Arc::new(account_ids.into_iter().collect_vec());
+        self.chunk_producer_accounts_cache = Some((tip.epoch_id, Arc::clone(&account_ids)));
+        Ok(account_ids)
+    }
+
     /// send_network_chain_info sends ChainInfo to PeerManagerActor.
     /// ChainInfo contains chain information relevant to p2p networking.
     /// It is expected to be called every time the head of the chain changes (or more often).
diff --git a/chain/client/src/client_actor.rs b/chain/client/src/client_actor.rs
index 62c698f8a99..c2151258993 100644
--- a/chain/client/src/client_actor.rs
+++ b/chain/client/src/client_actor.rs
@@ -1388,8 +1388,13 @@ impl ClientActorInner {
         };
 
         // If we produced the optimistic block, send it out before we save it.
+        let tip = self.client.chain.head()?;
+        let targets = self.client.get_optimistic_block_targets(&tip)?;
         self.network_adapter.send(PeerManagerMessageRequest::NetworkRequests(
-            NetworkRequests::OptimisticBlock { optimistic_block: optimistic_block.clone() },
+            NetworkRequests::OptimisticBlock {
+                chunk_producers: targets,
+                optimistic_block: optimistic_block.clone(),
+            },
         ));
 
         // We’ve produced the optimistic block, mark it as done so we don't produce it again.
diff --git a/chain/network/src/peer_manager/network_state/tier1.rs b/chain/network/src/peer_manager/network_state/tier1.rs
index 2aa677bc922..a057a243304 100644
--- a/chain/network/src/peer_manager/network_state/tier1.rs
+++ b/chain/network/src/peer_manager/network_state/tier1.rs
@@ -12,6 +12,7 @@ use near_async::time;
 use near_crypto::PublicKey;
 use near_o11y::log_assert;
 use near_primitives::network::PeerId;
+use near_primitives::types::AccountId;
 use rand::seq::IteratorRandom as _;
 use rand::seq::SliceRandom as _;
 use std::collections::{HashMap, HashSet};
@@ -374,4 +375,24 @@ impl super::NetworkState {
         }
         None
     }
+
+    /// Finds a TIER1 connection for the given AccountId. Currently used only for OptimisticBlock,
+    /// which is implemented as a PeerMessage but has targets identified by AccountId.
+    /// TODO(saketh): consider simplifying things by changing the message type of OptimisticBlock.
+    pub fn get_tier1_proxy_for_account_id(
+        &self,
+        account_id: &AccountId,
+    ) -> Option<Arc<connection::Connection>> {
+        let accounts_data = self.accounts_data.load();
+        for key in accounts_data.keys_by_id.get(account_id).iter().flat_map(|keys| keys.iter()) {
+            let Some(data) = accounts_data.data.get(key) else {
+                continue;
+            };
+            let Some(conn) = self.get_tier1_proxy(data) else {
+                continue;
+            };
+            return Some(conn);
+        }
+        None
+    }
 }
diff --git a/chain/network/src/peer_manager/peer_manager_actor.rs b/chain/network/src/peer_manager/peer_manager_actor.rs
index f8b2c292543..e7fc64a718c 100644
--- a/chain/network/src/peer_manager/peer_manager_actor.rs
+++ b/chain/network/src/peer_manager/peer_manager_actor.rs
@@ -783,12 +783,16 @@ impl PeerManagerActor {
                 self.state.tier2.broadcast_message(Arc::new(PeerMessage::Block(block)));
                 NetworkResponses::NoResponse
             }
-            NetworkRequests::OptimisticBlock { optimistic_block } => {
-                // TODO(#10584): send this message to all the producers.
-                // Maybe we just need to send this to the next producers.
-                self.state
-                    .tier1
-                    .broadcast_message(Arc::new(PeerMessage::OptimisticBlock(optimistic_block)));
+            NetworkRequests::OptimisticBlock { chunk_producers, optimistic_block } => {
+                // TODO(saketh): the chunk_producers are identified by their validator AccountId,
+                // but OptimisticBlock is sent over a direct PeerMessage. Hence we have to perform
+                // a conversion here from AccountId to peer connection. Consider reworking this.
+                let msg = Arc::new(PeerMessage::OptimisticBlock(optimistic_block));
+                for target_account in &*chunk_producers {
+                    if let Some(conn) = self.state.get_tier1_proxy_for_account_id(&target_account) {
+                        conn.send_message(msg.clone());
+                    }
+                }
                 NetworkResponses::NoResponse
             }
             NetworkRequests::Approval { approval_message } => {
diff --git a/chain/network/src/types.rs b/chain/network/src/types.rs
index 11e513afab6..515503c2236 100644
--- a/chain/network/src/types.rs
+++ b/chain/network/src/types.rs
@@ -242,7 +242,7 @@ pub enum NetworkRequests {
     /// Sends block, either when block was just produced or when requested.
     Block { block: Block },
     /// Sends optimistic block as soon as the production window for the height starts.
-    OptimisticBlock { optimistic_block: OptimisticBlock },
+    OptimisticBlock { chunk_producers: Arc<Vec<AccountId>>, optimistic_block: OptimisticBlock },
     /// Sends approval.
     Approval { approval_message: ApprovalMessage },
     /// Request block with given hash from given peer.
diff --git a/integration-tests/src/env/setup.rs b/integration-tests/src/env/setup.rs
index e90286788f4..6a2ad28eb4b 100644
--- a/integration-tests/src/env/setup.rs
+++ b/integration-tests/src/env/setup.rs
@@ -618,16 +618,17 @@ fn process_peer_manager_message_default(
 
             hash_to_height.write().insert(*block.header().hash(), block.header().height());
         }
-        NetworkRequests::OptimisticBlock { optimistic_block } => {
+        NetworkRequests::OptimisticBlock { chunk_producers, optimistic_block } => {
             // TODO(#10584): maybe go through an adapter to facilitate testing.
-            for actor_handles in connectors {
-                actor_handles.client_actor.do_send(
-                    OptimisticBlockMessage {
-                        optimistic_block: optimistic_block.clone(),
-                        from_peer: PeerInfo::random().id,
-                    }
-                    .with_span_context(),
-                );
+            for (i, name) in validators.iter().enumerate() {
+                if !chunk_producers.contains(name) {
+                    continue;
+                }
+                let msg = OptimisticBlockMessage {
+                    optimistic_block: optimistic_block.clone(),
+                    from_peer: PeerInfo::random().id,
+                };
+                connectors[i].client_actor.do_send(msg.with_span_context());
             }
         }
         NetworkRequests::PartialEncodedChunkRequest { target, request, .. } => {
diff --git a/test-loop-tests/src/tests/optimistic_block.rs b/test-loop-tests/src/tests/optimistic_block.rs
index f5721b94a47..0724261948a 100644
--- a/test-loop-tests/src/tests/optimistic_block.rs
+++ b/test-loop-tests/src/tests/optimistic_block.rs
@@ -253,7 +253,9 @@ fn alter_optimistic_block_at_height(
         peer_actor.register_override_handler(Box::new({
             let validator_signer = signer.clone();
             move |request: NetworkRequests| {
-                if let NetworkRequests::OptimisticBlock { optimistic_block } = &request {
+                if let NetworkRequests::OptimisticBlock { chunk_producers, optimistic_block } =
+                    &request
+                {
                     if optimistic_block.height() == height {
                         let altered_ob = optimistic_block::OptimisticBlock::alter(
                             optimistic_block,
@@ -263,6 +265,7 @@ fn alter_optimistic_block_at_height(
                             ),
                         );
                         return Some(NetworkRequests::OptimisticBlock {
+                            chunk_producers: chunk_producers.clone(),
                             optimistic_block: altered_ob,
                         });
                     }
diff --git a/test-loop-tests/src/utils/peer_manager_actor.rs b/test-loop-tests/src/utils/peer_manager_actor.rs
index 170728fcce7..6eb24fdb6d2 100644
--- a/test-loop-tests/src/utils/peer_manager_actor.rs
+++ b/test-loop-tests/src/utils/peer_manager_actor.rs
@@ -421,18 +421,20 @@ fn network_message_to_client_handler(
             }
             None
         }
-        NetworkRequests::OptimisticBlock { optimistic_block } => {
+        NetworkRequests::OptimisticBlock { chunk_producers, optimistic_block } => {
             let my_peer_id = shared_state.account_to_peer_id(&my_account_id);
             for account_id in shared_state.accounts() {
-                if account_id != my_account_id {
-                    let _ = shared_state
-                        .senders_for_account(&my_account_id, &account_id)
-                        .client_sender
-                        .send(OptimisticBlockMessage {
-                            optimistic_block: optimistic_block.clone(),
-                            from_peer: my_peer_id.clone(),
-                        });
+                if !chunk_producers.contains(&account_id) {
+                    continue;
                 }
+                let msg = OptimisticBlockMessage {
+                    optimistic_block: optimistic_block.clone(),
+                    from_peer: my_peer_id.clone(),
+                };
+                let _ = shared_state
+                    .senders_for_account(&my_account_id, &account_id)
+                    .client_sender
+                    .send(msg);
             }
             None
         }


Reviews:
user: login: shreyan-gupta
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T06:44:12Z
user: login: wacban
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T09:16:08Z
user: login: VanBarbascu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T13:16:26Z
user: login: saketh-are
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T14:42:17Z
user: login: saketh-are
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T15:04:05Z
user: login: saketh-are
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T15:22:42Z
user: login: saketh-are
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T16:02:10Z
user: login: shreyan-gupta
body: Approving for unblocking, but otherwise it would be a good idea to get a review from someone for OB.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T20:17:21Z
user: login: wacban
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T07:45:59Z
user: login: VanBarbascu
body: A bit concerned about duplicating the logic of `tier1_accounts_cache` with `optimistic_block_targets_cache`.

We could combine them all under `tier1_accounts_cache` and have an object `T1AccountsCache{epoch_id: EpochId, bp: Arc<...>,cp: Arc<...>, cv: Arc<...>}` and functions for `getAll` and `getProducers`.

state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T11:50:18Z
user: login: saketh-are
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T21:14:30Z
user: login: saketh-are
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T21:24:44Z
user: login: wacban
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T09:17:45Z
user: login: wacban
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T09:20:55Z

Review comments:
diff_hunk: @@ -783,12 +783,16 @@ impl PeerManagerActor {
                 self.state.tier2.broadcast_message(Arc::new(PeerMessage::Block(block)));
                 NetworkResponses::NoResponse
             }
-            NetworkRequests::OptimisticBlock { optimistic_block } => {
-                // TODO(#10584): send this message to all the producers.
-                // Maybe we just need to send this to the next producers.
-                self.state
-                    .tier1
-                    .broadcast_message(Arc::new(PeerMessage::OptimisticBlock(optimistic_block)));
+            NetworkRequests::OptimisticBlock { block_producers, optimistic_block } => {
+                // TODO(saketh): the block_producers are identified by their validator AccountId,
+                // but OptimisticBlock is sent over a direct PeerMessage. Hence we have to perform
+                // a conversion here from AccountId to peer connection. Consider reworking this.
+                let msg = Arc::new(PeerMessage::OptimisticBlock(optimistic_block));
+                for target_account in block_producers {
+                    if let Some(conn) = self.state.get_tier1_proxy_for_account_id(&target_account) {
path: chain/network/src/peer_manager/peer_manager_actor.rs
user: login: shreyan-gupta
body: Is it possible that we silently drop the message in case we are not able to find the tier1 proxy?
created_at: 2025-05-13T06:40:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1381,8 +1383,25 @@ impl ClientActorInner {
         };
 
         // If we produced the optimistic block, send it out before we save it.
+        let epoch_id = self
+            .client
+            .chain
+            .epoch_manager
path: chain/client/src/client_actor.rs
user: login: shreyan-gupta
body: nit: I guess we can add and additional line to get the epoch manager? `let epoch_manager = &self.client.chain.epoch_manager`?
created_at: 2025-05-13T06:41:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -374,4 +375,26 @@ impl super::NetworkState {
         }
         None
     }
+
+    /// Finds a TIER1 connection for the given AccountId. Currently used only for OptimisticBlock,
+    /// which is implemented as a PeerMessage but has targets identified by AccountId.
+    /// TODO(saketh): consider simplifying things by changing the message type of OptimisticBlock.
+    pub fn get_tier1_proxy_for_account_id(
+        &self,
+        account_id: &AccountId,
+    ) -> Option<Arc<connection::Connection>> {
path: chain/network/src/peer_manager/network_state/tier1.rs
user: login: shreyan-gupta
body: We have almost exactly the same block in `send_message_to_account` function in chain/network/src/peer_manager/network_state/mod.rs

Might we want to merge/reuse this function there?
created_at: 2025-05-13T06:43:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2237,6 +2237,10 @@ impl Client {
         // require some tuning in the future. In particular, if we decide that connecting to
         // block & chunk producers of the next epoch is too expensive, we can postpone it
         // till almost the end of this epoch.
+        //
+        // We include also all chunk validators of the current epoch, due to the importance of:
path: chain/client/src/client.rs
user: login: wacban
body: Can you also adjust this method's comment? 
created_at: 2025-05-13T09:06:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2257,6 +2261,9 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
path: chain/client/src/client.rs
user: login: wacban
body: Why not also do it for the next epoch? 
created_at: 2025-05-13T09:06:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1381,8 +1383,25 @@ impl ClientActorInner {
         };
 
         // If we produced the optimistic block, send it out before we save it.
+        let epoch_id = self
+            .client
+            .chain
+            .epoch_manager
+            .get_epoch_id_from_prev_block(optimistic_block.prev_block_hash())?;
+        //  TODO(#10584): Maybe we just need to send this to the next producers.
+        let block_producers: HashSet<AccountId> = self
path: chain/client/src/client_actor.rs
user: login: wacban
body: yes, but chunk producers, not block ptoducers
created_at: 2025-05-13T09:10:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -618,16 +618,18 @@ fn process_peer_manager_message_default(
 
             hash_to_height.write().insert(*block.header().hash(), block.header().height());
         }
-        NetworkRequests::OptimisticBlock { optimistic_block } => {
+        NetworkRequests::OptimisticBlock { block_producers, optimistic_block } => {
             // TODO(#10584): maybe go through an adapter to facilitate testing.
-            for actor_handles in connectors {
-                actor_handles.client_actor.do_send(
-                    OptimisticBlockMessage {
-                        optimistic_block: optimistic_block.clone(),
-                        from_peer: PeerInfo::random().id,
-                    }
-                    .with_span_context(),
-                );
+            for (i, name) in validators.iter().enumerate() {
+                if block_producers.contains(name) {
+                    connectors[i].client_actor.do_send(
+                        OptimisticBlockMessage {
+                            optimistic_block: optimistic_block.clone(),
+                            from_peer: PeerInfo::random().id,
+                        }
+                        .with_span_context(),
+                    );
+                }
path: integration-tests/src/env/setup.rs
user: login: wacban
body: mini nit1: invert the condition to continue early and reduce nesting
mini nit2: move the message to a temporary variable to not clump too many things into a single line

both optional and based on personal preference ;) 
created_at: 2025-05-13T09:13:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -253,7 +253,9 @@ fn alter_optimistic_block_at_height(
         peer_actor.register_override_handler(Box::new({
             let validator_signer = signer.clone();
             move |request: NetworkRequests| {
-                if let NetworkRequests::OptimisticBlock { optimistic_block } = &request {
+                if let NetworkRequests::OptimisticBlock { block_producers, optimistic_block } =
path: test-loop-tests/src/tests/optimistic_block.rs
user: login: wacban
body: mini nit again about early returns:
```
let OB { .. } = &request else { return Some(request); };
if optimistic_block.height() != height { return Some(request); }
...
return ...
```
created_at: 2025-05-13T09:15:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1381,8 +1383,25 @@ impl ClientActorInner {
         };
 
         // If we produced the optimistic block, send it out before we save it.
+        let epoch_id = self
+            .client
+            .chain
+            .epoch_manager
+            .get_epoch_id_from_prev_block(optimistic_block.prev_block_hash())?;
+        //  TODO(#10584): Maybe we just need to send this to the next producers.
+        let block_producers: HashSet<AccountId> = self
path: chain/client/src/client_actor.rs
user: login: VanBarbascu
body: To address the limit case of epoch boundary, we want to send to the intersection of this and next epoch producers. 

This means that the node will always send to CV waiting to become producers, but that is fine as they will also benefit from OB because they are keeping up with the chain.
created_at: 2025-05-13T13:16:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -783,12 +783,16 @@ impl PeerManagerActor {
                 self.state.tier2.broadcast_message(Arc::new(PeerMessage::Block(block)));
                 NetworkResponses::NoResponse
             }
-            NetworkRequests::OptimisticBlock { optimistic_block } => {
-                // TODO(#10584): send this message to all the producers.
-                // Maybe we just need to send this to the next producers.
-                self.state
-                    .tier1
-                    .broadcast_message(Arc::new(PeerMessage::OptimisticBlock(optimistic_block)));
+            NetworkRequests::OptimisticBlock { block_producers, optimistic_block } => {
+                // TODO(saketh): the block_producers are identified by their validator AccountId,
+                // but OptimisticBlock is sent over a direct PeerMessage. Hence we have to perform
+                // a conversion here from AccountId to peer connection. Consider reworking this.
+                let msg = Arc::new(PeerMessage::OptimisticBlock(optimistic_block));
+                for target_account in block_producers {
+                    if let Some(conn) = self.state.get_tier1_proxy_for_account_id(&target_account) {
path: chain/network/src/peer_manager/peer_manager_actor.rs
user: login: saketh-are
body: Yes. Note that this is not a behavior change: OB was previously being broadcast to T1 peers only. 

If OB were a RoutedMessage instead of a PeerMessage we could use `send_message_to_account` which falls back to routing over T2 if the T1 connection is absent. But that change involves a more difficult migration and we will revisit this in the future.
created_at: 2025-05-13T14:42:17Z
author_association: COLLABORATOR
diff_hunk: @@ -2257,6 +2261,9 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
path: chain/client/src/client.rs
user: login: saketh-are
body: I guess I am wary of increasing the size of the T1 set more aggressively than we need to. A few missed endorsements at the start of the new epoch while new T1 connections are established should be a relatively minor issue.

OTOH, if we want to say that the CV set is generally quite stable and this doesn't really increase the connection count much, I am fine with including next epoch CVs.
created_at: 2025-05-13T15:04:05Z
author_association: COLLABORATOR
diff_hunk: @@ -1381,8 +1383,25 @@ impl ClientActorInner {
         };
 
         // If we produced the optimistic block, send it out before we save it.
+        let epoch_id = self
+            .client
+            .chain
+            .epoch_manager
+            .get_epoch_id_from_prev_block(optimistic_block.prev_block_hash())?;
+        //  TODO(#10584): Maybe we just need to send this to the next producers.
+        let block_producers: HashSet<AccountId> = self
path: chain/client/src/client_actor.rs
user: login: saketh-are
body: I changed the logic to use `epoch_manager::get_epoch_chunk_producers`.
created_at: 2025-05-13T15:22:41Z
author_association: COLLABORATOR
diff_hunk: @@ -374,4 +375,26 @@ impl super::NetworkState {
         }
         None
     }
+
+    /// Finds a TIER1 connection for the given AccountId. Currently used only for OptimisticBlock,
+    /// which is implemented as a PeerMessage but has targets identified by AccountId.
+    /// TODO(saketh): consider simplifying things by changing the message type of OptimisticBlock.
+    pub fn get_tier1_proxy_for_account_id(
+        &self,
+        account_id: &AccountId,
+    ) -> Option<Arc<connection::Connection>> {
path: chain/network/src/peer_manager/network_state/tier1.rs
user: login: saketh-are
body: They are very subtly different and I would prefer to limit blast radius of this PR. As mentioned in other comments we can revisit the exact handling of OptimisticBlock in the future and possibly convert it to a routed message. In that case we can delete this function. 
created_at: 2025-05-13T16:02:10Z
author_association: COLLABORATOR
diff_hunk: @@ -2257,6 +2261,9 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
path: chain/client/src/client.rs
user: login: wacban
body: TBH I would just make it consistent with the BPs and CPs to avoid any confusion around this in the future. Yes, I think we can safely assume this set is quite stable over time. 
created_at: 2025-05-14T07:27:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2257,11 +2264,39 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
         let account_keys = Arc::new(account_keys);
         self.tier1_accounts_cache = Some((tip.epoch_id, account_keys.clone()));
         Ok(account_keys)
     }
 
+    /// We send the optimistic block to the chunk producers of the current and next epochs.
+    pub(crate) fn get_optimistic_block_targets(
+        &mut self,
+        tip: &Tip,
+    ) -> Result<Arc<Vec<AccountId>>, Error> {
+        match &self.optimistic_block_targets_cache {
+            Some(it) if it.0 == tip.epoch_id => return Ok(it.1.clone()),
path: chain/client/src/client.rs
user: login: wacban
body: Can you get it by reference instead of cloning? 
nit: Can you unpack it further? `Some(epoch_id, other_thing)`

created_at: 2025-05-14T07:29:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2257,11 +2264,39 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
         let account_keys = Arc::new(account_keys);
         self.tier1_accounts_cache = Some((tip.epoch_id, account_keys.clone()));
         Ok(account_keys)
     }
 
+    /// We send the optimistic block to the chunk producers of the current and next epochs.
+    pub(crate) fn get_optimistic_block_targets(
+        &mut self,
+        tip: &Tip,
+    ) -> Result<Arc<Vec<AccountId>>, Error> {
+        match &self.optimistic_block_targets_cache {
+            Some(it) if it.0 == tip.epoch_id => return Ok(it.1.clone()),
path: chain/client/src/client.rs
user: login: wacban
body: Ah I see it's an arc so clone should be cheap. Perhaps make it explicit with Arc::clone to make it less scary (guessing this syntax but there should be something like that). 
created_at: 2025-05-14T07:31:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -174,6 +174,8 @@ pub struct Client {
     upgrade_schedule: ProtocolUpgradeVotingSchedule,
     /// Produced optimistic block.
     last_optimistic_block_produced: Option<OptimisticBlock>,
+    /// Cached precomputed set of TIER1 accounts to which Optimistic Block should be sent.
+    optimistic_block_targets_cache: Option<(EpochId, Arc<Vec<AccountId>>)>,
path: chain/client/src/client.rs
user: login: wacban
body: Is it the targets just within this epoch or this and next? Either way can you mention that in the comment? 
created_at: 2025-05-14T07:35:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -174,6 +174,8 @@ pub struct Client {
     upgrade_schedule: ProtocolUpgradeVotingSchedule,
     /// Produced optimistic block.
     last_optimistic_block_produced: Option<OptimisticBlock>,
+    /// Cached precomputed set of TIER1 accounts to which Optimistic Block should be sent.
+    optimistic_block_targets_cache: Option<(EpochId, Arc<Vec<AccountId>>)>,
path: chain/client/src/client.rs
user: login: wacban
body: I'd be tempted to try to remove the Arc and make it use references - though I'm not sure if that would be any better. 
created_at: 2025-05-14T07:36:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1381,8 +1381,13 @@ impl ClientActorInner {
         };
 
         // If we produced the optimistic block, send it out before we save it.
+        let tip = self.client.chain.head()?;
+        let targets = self.client.get_optimistic_block_targets(&tip)?;
         self.network_adapter.send(PeerManagerMessageRequest::NetworkRequests(
-            NetworkRequests::OptimisticBlock { optimistic_block: optimistic_block.clone() },
+            NetworkRequests::OptimisticBlock {
+                chunk_producers: (*targets).clone(),
+                optimistic_block: optimistic_block.clone(),
path: chain/client/src/client_actor.rs
user: login: wacban
body: What do you think about wrapping those in `Arc`s to avoid the cloning? 
created_at: 2025-05-14T07:38:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -374,4 +375,26 @@ impl super::NetworkState {
         }
         None
     }
+
+    /// Finds a TIER1 connection for the given AccountId. Currently used only for OptimisticBlock,
+    /// which is implemented as a PeerMessage but has targets identified by AccountId.
+    /// TODO(saketh): consider simplifying things by changing the message type of OptimisticBlock.
+    pub fn get_tier1_proxy_for_account_id(
+        &self,
+        account_id: &AccountId,
+    ) -> Option<Arc<connection::Connection>> {
+        let accounts_data = self.accounts_data.load();
+        for key in accounts_data.keys_by_id.get(account_id).iter().flat_map(|keys| keys.iter()) {
+            let data = match accounts_data.data.get(key) {
+                Some(data) => data,
+                None => continue,
+            };
path: chain/network/src/peer_manager/network_state/tier1.rs
user: login: wacban
body: nit: 
```suggestion
            let Some(data) = accounts_data.data.get(key) else { 
                continue;
            };
```
created_at: 2025-05-14T07:43:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -374,4 +375,26 @@ impl super::NetworkState {
         }
         None
     }
+
+    /// Finds a TIER1 connection for the given AccountId. Currently used only for OptimisticBlock,
+    /// which is implemented as a PeerMessage but has targets identified by AccountId.
+    /// TODO(saketh): consider simplifying things by changing the message type of OptimisticBlock.
+    pub fn get_tier1_proxy_for_account_id(
+        &self,
+        account_id: &AccountId,
+    ) -> Option<Arc<connection::Connection>> {
+        let accounts_data = self.accounts_data.load();
+        for key in accounts_data.keys_by_id.get(account_id).iter().flat_map(|keys| keys.iter()) {
+            let data = match accounts_data.data.get(key) {
+                Some(data) => data,
+                None => continue,
+            };
+            let conn = match self.get_tier1_proxy(data) {
+                Some(conn) => conn,
+                None => continue,
+            };
path: chain/network/src/peer_manager/network_state/tier1.rs
user: login: wacban
body: ditto
created_at: 2025-05-14T07:44:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -783,12 +783,16 @@ impl PeerManagerActor {
                 self.state.tier2.broadcast_message(Arc::new(PeerMessage::Block(block)));
                 NetworkResponses::NoResponse
             }
-            NetworkRequests::OptimisticBlock { optimistic_block } => {
-                // TODO(#10584): send this message to all the producers.
-                // Maybe we just need to send this to the next producers.
-                self.state
-                    .tier1
-                    .broadcast_message(Arc::new(PeerMessage::OptimisticBlock(optimistic_block)));
+            NetworkRequests::OptimisticBlock { chunk_producers, optimistic_block } => {
+                // TODO(saketh): the block_producers are identified by their validator AccountId,
+                // but OptimisticBlock is sent over a direct PeerMessage. Hence we have to perform
+                // a conversion here from AccountId to peer connection. Consider reworking this.
path: chain/network/src/peer_manager/peer_manager_actor.rs
user: login: wacban
body: nit: block_producers -> chunk_producers
created_at: 2025-05-14T07:45:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2257,11 +2264,39 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
         let account_keys = Arc::new(account_keys);
         self.tier1_accounts_cache = Some((tip.epoch_id, account_keys.clone()));
         Ok(account_keys)
     }
 
+    /// We send the optimistic block to the chunk producers of the current and next epochs.
+    pub(crate) fn get_optimistic_block_targets(
+        &mut self,
+        tip: &Tip,
+    ) -> Result<Arc<Vec<AccountId>>, Error> {
+        match &self.optimistic_block_targets_cache {
+            Some(it) if it.0 == tip.epoch_id => return Ok(it.1.clone()),
+            _ => {}
+        }
+
+        let _guard =
path: chain/client/src/client.rs
user: login: VanBarbascu
body: ```suggestion
        let _span =
```
created_at: 2025-05-14T11:36:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -174,6 +174,8 @@ pub struct Client {
     upgrade_schedule: ProtocolUpgradeVotingSchedule,
     /// Produced optimistic block.
     last_optimistic_block_produced: Option<OptimisticBlock>,
+    /// Cached precomputed set of TIER1 accounts to which Optimistic Block should be sent.
+    optimistic_block_targets_cache: Option<(EpochId, Arc<Vec<AccountId>>)>,
path: chain/client/src/client.rs
user: login: VanBarbascu
body: Instead of calling it optimistic block target, I would use something generic like: producers to make it clear where we are sending the OBs.
This is a subset of `tier1_accounts_cache`. Maybe we can combine them?
created_at: 2025-05-14T11:41:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2257,11 +2264,39 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
path: chain/client/src/client.rs
user: login: VanBarbascu
body: Any chance we can add this logic only for Chunk Producers and Block Producers? 
This way CV will only connect to CP and BP and not to other CV.
created_at: 2025-05-14T11:43:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2257,11 +2264,39 @@ impl Client {
                     .insert(bp.public_key().clone());
             }
         }
+        for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
+            account_keys.entry(v.account_id().clone()).or_default().insert(v.public_key().clone());
+        }
path: chain/client/src/client.rs
user: login: saketh-are
body: I think I'd prefer to keep it simple, considering that this optimization does not reduce the worst case number of connections.
created_at: 2025-05-14T21:14:30Z
author_association: COLLABORATOR
diff_hunk: @@ -1381,8 +1381,13 @@ impl ClientActorInner {
         };
 
         // If we produced the optimistic block, send it out before we save it.
+        let tip = self.client.chain.head()?;
+        let targets = self.client.get_optimistic_block_targets(&tip)?;
         self.network_adapter.send(PeerManagerMessageRequest::NetworkRequests(
-            NetworkRequests::OptimisticBlock { optimistic_block: optimistic_block.clone() },
+            NetworkRequests::OptimisticBlock {
+                chunk_producers: (*targets).clone(),
+                optimistic_block: optimistic_block.clone(),
path: chain/client/src/client_actor.rs
user: login: saketh-are
body: I made the change for `chunk_producers`. For `optimistic_block` further changes are needed on the network side to avoid the clone so I'm leaving it for now.
created_at: 2025-05-14T21:24:44Z
author_association: COLLABORATOR
diff_hunk: @@ -2266,12 +2269,43 @@ impl Client {
                     .or_default()
                     .insert(bp.public_key().clone());
             }
+            for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
path: chain/client/src/client.rs
user: login: wacban
body: This should be `epoch_id`, not `tip.epoch_id`
created_at: 2025-05-15T09:17:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2266,12 +2269,43 @@ impl Client {
                     .or_default()
                     .insert(bp.public_key().clone());
             }
+            for v in self.epoch_manager.get_epoch_all_validators(&tip.epoch_id)? {
path: chain/client/src/client.rs
user: login: wacban
body: #13546 to fix it
created_at: 2025-05-15T09:20:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-05-12T21:14:00Z
body: ## [Codecov](https://app.codecov.io/gh/near/nearcore/pull/13527?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) Report
Attention: Patch coverage is `84.72222%` with `11 lines` in your changes missing coverage. Please review.
> Project coverage is 69.58%. Comparing base [(`14f7f22`)](https://app.codecov.io/gh/near/nearcore/commit/14f7f22a3fe2ae28b8c44192132cc7cc62b2ee73?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) to head [(`a2743aa`)](https://app.codecov.io/gh/near/nearcore/commit/a2743aaf65725549364d94e06bac76672ab08033?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near).
> Report is 1 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/near/nearcore/pull/13527?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Patch % | Lines |
|---|---|---|
| [integration-tests/src/env/setup.rs](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&filepath=integration-tests%2Fsrc%2Fenv%2Fsetup.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#diff-aW50ZWdyYXRpb24tdGVzdHMvc3JjL2Vudi9zZXR1cC5ycw==) | 40.00% | [6 Missing :warning: ](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) |
| [chain/client/src/client.rs](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&filepath=chain%2Fclient%2Fsrc%2Fclient.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#diff-Y2hhaW4vY2xpZW50L3NyYy9jbGllbnQucnM=) | 93.75% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) |
| [chain/client/src/client\_actor.rs](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&filepath=chain%2Fclient%2Fsrc%2Fclient_actor.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#diff-Y2hhaW4vY2xpZW50L3NyYy9jbGllbnRfYWN0b3IucnM=) | 66.66% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) |
| [...in/network/src/peer\_manager/network\_state/tier1.rs](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&filepath=chain%2Fnetwork%2Fsrc%2Fpeer_manager%2Fnetwork_state%2Ftier1.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#diff-Y2hhaW4vbmV0d29yay9zcmMvcGVlcl9tYW5hZ2VyL25ldHdvcmtfc3RhdGUvdGllcjEucnM=) | 92.30% | [1 Missing :warning: ](https://app.codecov.io/gh/near/nearcore/pull/13527?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master   #13527      +/-   ##
==========================================
+ Coverage   69.56%   69.58%   +0.02%     
==========================================
  Files         859      859              
  Lines      171152   171199      +47     
  Branches   171152   171199      +47     
==========================================
+ Hits       119059   119127      +68     
+ Misses      47273    47252      -21     
  Partials     4820     4820              
```

| [Flag](https://app.codecov.io/gh/near/nearcore/pull/13527/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | Coverage Δ | |
|---|---|---|
| [pytests](https://app.codecov.io/gh/near/nearcore/pull/13527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.53% <0.00%> (-0.01%)` | :arrow_down: |
| [pytests-nightly](https://app.codecov.io/gh/near/nearcore/pull/13527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `1.60% <0.00%> (-0.01%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/near/nearcore/pull/13527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `69.20% <84.72%> (-0.01%)` | :arrow_down: |
| [unittests-nightly](https://app.codecov.io/gh/near/nearcore/pull/13527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near) | `69.12% <84.72%> (+0.02%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/near/nearcore/pull/13527?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=near).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: wacban
created_at: 2025-05-13T09:03:46Z
body: > OB, ..., send only to BPs.

@VanBarbascu please correct me if I'm wrong, I think the OB should be sent to CPs, not BPs? Or both to be safe? I do realize today they are the same but in the future we may want decouple those roles. 
author_association: CONTRIBUTOR
user: login: saketh-are
created_at: 2025-05-13T16:04:22Z
body: @wacban @VanBarbascu I changed the Optimistic Block targets calculation to include chunk producers of the current and next epochs. I also added some caching behavior over it. PTAL.
author_association: COLLABORATOR
user: login: saketh-are
created_at: 2025-05-14T23:05:40Z
body: Seems like upgradable pytest is still flaky...
author_association: COLLABORATOR