Repository: anza-xyz/agave
Pull request #5104: Bound gossip channels
Author: login: cpubot
Created: 2025-02-28T17:36:31Z | Merged: 2025-03-03T22:25:55Z

#### Problem

The `unbounded` channel uses what is effectively a linked list of "blocks" of 31 slots. As volume surges, it has to dynamically allocate additional blocks to accommodate the extra slots required. It does this on demand, one-by-one, which adds extra synchronization overhead on the sending and receiving side in addition to the time waiting for allocation / deallocation -- this slows down the entire pipeline of packet processing. Contrast this to a `bounded` channel, which uses a single contiguous allocation without any of the block synchronization overhead. 

#### Summary of Changes

Gossip packet buffers now used `bounded` channels, with a fixed, preallocated, capacity. 

It was observed that under extreme load, the channel caps out at around 11k packet batches. The channel capacity introduced here is `16,384` (`2^14`) packet batches, which is the 11k figure rounded up to the next power of 2. This ensures that load shedding is highly unlikely to occur on the sender and continues to be done on the receiver side.

Senders are also updated to use `try_send` and introduce error handling in the event that channels are full. It is highly unlikely that these errors are actually hit in practice given the large capacity of the channels and the draining that is occurring in the receivers. This adjustment is made here for correctness and completeness, and also dovetail with forthcoming updates to load shed at the send site. 

To reiterate, there is no practical change to current load shedding behavior in this branch, save for edge cases that were never observed in load testing and are highly unlikely to be encountered.

Broken out of #5065 


Diff:
diff --git a/gossip/src/cluster_info.rs b/gossip/src/cluster_info.rs
index 433fae763d0b11..c053eb4af1645a 100644
--- a/gossip/src/cluster_info.rs
+++ b/gossip/src/cluster_info.rs
@@ -44,7 +44,7 @@ use {
         },
         weighted_shuffle::WeightedShuffle,
     },
-    crossbeam_channel::{Receiver, RecvTimeoutError, Sender},
+    crossbeam_channel::{Receiver, RecvTimeoutError, Sender, TrySendError},
     itertools::{Either, Itertools},
     rand::{seq::SliceRandom, CryptoRng, Rng},
     rayon::{prelude::*, ThreadPool, ThreadPoolBuilder},
@@ -118,6 +118,13 @@ const MAX_GOSSIP_TRAFFIC: usize = 128_000_000 / PACKET_DATA_SIZE;
 /// so any extra capacity that may be reserved due to traffic variations will be preserved,
 /// avoiding excessive resizing and re-allocation.
 const CHANNEL_RECV_BUFFER_INITIAL_CAPACITY: usize = MAX_GOSSIP_TRAFFIC.div_ceil(28);
+/// Channel capacity for gossip channels.
+///
+/// It was observed that under extreme load, the channel caps out
+/// around 11k capacity. This rounds that up to the next power of 2
+/// such that load shedding is highly unlikely to occur on the sender
+/// and continues to be done on the receiver side.
+pub(crate) const GOSSIP_CHANNEL_CAPACITY: usize = 16_384; // 2^14
 const GOSSIP_PING_CACHE_CAPACITY: usize = 126976;
 const GOSSIP_PING_CACHE_TTL: Duration = Duration::from_secs(1280);
 const GOSSIP_PING_CACHE_RATE_LIMIT_DELAY: Duration = Duration::from_secs(1280 / 64);
@@ -1357,7 +1364,11 @@ impl ClusterInfo {
         .filter_map(|(addr, data)| make_gossip_packet(addr, &data, &self.stats))
         .for_each(|pkt| packet_batch.push(pkt));
         if !packet_batch.is_empty() {
-            sender.send(packet_batch)?;
+            if let Err(TrySendError::Full(packet_batch)) = sender.try_send(packet_batch) {
+                self.stats
+                    .gossip_transmit_packets_dropped_count
+                    .add_relaxed(packet_batch.len() as u64);
+            }
         }
         self.stats
             .gossip_transmit_loop_iterations_since_last_report
@@ -1599,7 +1610,11 @@ impl ClusterInfo {
         if !requests.is_empty() {
             let response = self.handle_pull_requests(thread_pool, recycler, requests, stakes);
             if !response.is_empty() {
-                let _ = response_sender.send(response);
+                if let Err(TrySendError::Full(response)) = response_sender.try_send(response) {
+                    self.stats
+                        .gossip_packets_dropped_count
+                        .add_relaxed(response.len() as u64);
+                }
             }
         }
     }
@@ -1874,7 +1889,11 @@ impl ClusterInfo {
             .filter_map(|(addr, data)| make_gossip_packet(addr, &data, &self.stats))
             .for_each(|pkt| packet_batch.push(pkt));
         if !packet_batch.is_empty() {
-            let _ = response_sender.send(packet_batch);
+            if let Err(TrySendError::Full(packet_batch)) = response_sender.try_send(packet_batch) {
+                self.stats
+                    .gossip_packets_dropped_count
+                    .add_relaxed(packet_batch.len() as u64);
+            }
         }
     }
 
@@ -2158,7 +2177,7 @@ impl ClusterInfo {
             .map(EpochSpecs::current_epoch_staked_nodes)
             .cloned()
             .unwrap_or_default();
-        let packets: Vec<_> = {
+        let packets_verified: Vec<_> = {
             let _st = ScopedTimer::from(&self.stats.verify_gossip_packets_time);
             thread_pool.install(|| {
                 if packet_buf.len() == 1 {
@@ -2175,8 +2194,15 @@ impl ClusterInfo {
                 }
             })
         };
+        if let Err(TrySendError::Full(_)) = sender.try_send(packets_verified) {
+            self.stats.gossip_packets_dropped_count.add_relaxed(
+                packet_buf
+                    .iter()
+                    .fold(0, |acc, packet_batch| acc + packet_batch.len()) as u64,
+            );
+        }
         packet_buf.clear();
-        Ok(sender.send(packets)?)
+        Ok(())
     }
 
     /// Process messages from the network
@@ -3006,7 +3032,11 @@ fn send_gossip_packets<S: Borrow<SocketAddr>>(
     let pkts = pkts.into_iter();
     if pkts.len() != 0 {
         let pkts = make_gossip_packet_batch(pkts, recycler, stats);
-        let _ = sender.send(pkts);
+        if let Err(TrySendError::Full(pkts)) = sender.try_send(pkts) {
+            stats
+                .gossip_packets_dropped_count
+                .add_relaxed(pkts.len() as u64);
+        }
     }
 }
 
diff --git a/gossip/src/cluster_info_metrics.rs b/gossip/src/cluster_info_metrics.rs
index a9bafd4ddf7432..77e954a03c64b6 100644
--- a/gossip/src/cluster_info_metrics.rs
+++ b/gossip/src/cluster_info_metrics.rs
@@ -111,6 +111,7 @@ pub struct GossipStats {
     pub(crate) gossip_pull_request_sent_bytes: Counter,
     pub(crate) gossip_transmit_loop_iterations_since_last_report: Counter,
     pub(crate) gossip_transmit_loop_time: Counter,
+    pub(crate) gossip_transmit_packets_dropped_count: Counter,
     pub(crate) handle_batch_ping_messages_time: Counter,
     pub(crate) handle_batch_pong_messages_time: Counter,
     pub(crate) handle_batch_prune_messages_time: Counter,
@@ -431,6 +432,11 @@ pub(crate) fn submit_gossip_stats(
             stats.gossip_transmit_loop_time.clear(),
             i64
         ),
+        (
+            "gossip_transmit_packets_dropped_count",
+            stats.gossip_transmit_packets_dropped_count.clear(),
+            i64
+        ),
         (
             "gossip_transmit_loop_iterations_since_last_report",
             stats
diff --git a/gossip/src/gossip_service.rs b/gossip/src/gossip_service.rs
index 1c3c8481212c99..8b886469f556eb 100644
--- a/gossip/src/gossip_service.rs
+++ b/gossip/src/gossip_service.rs
@@ -1,8 +1,11 @@
 //! The `gossip_service` module implements the network control plane.
 
 use {
-    crate::{cluster_info::ClusterInfo, contact_info::ContactInfo},
-    crossbeam_channel::{unbounded, Sender},
+    crate::{
+        cluster_info::{ClusterInfo, GOSSIP_CHANNEL_CAPACITY},
+        contact_info::ContactInfo,
+    },
+    crossbeam_channel::{bounded, Sender},
     rand::{thread_rng, Rng},
     solana_client::{connection_cache::ConnectionCache, tpu_client::TpuClientWrapper},
     solana_keypair::Keypair,
@@ -43,7 +46,7 @@ impl GossipService {
         stats_reporter_sender: Option<Sender<Box<dyn FnOnce() + Send>>>,
         exit: Arc<AtomicBool>,
     ) -> Self {
-        let (request_sender, request_receiver) = unbounded();
+        let (request_sender, request_receiver) = bounded(GOSSIP_CHANNEL_CAPACITY);
         let gossip_socket = Arc::new(gossip_socket);
         trace!(
             "GossipService: id: {}, listening on: {:?}",
@@ -63,14 +66,14 @@ impl GossipService {
             None,
             false,
         );
-        let (consume_sender, listen_receiver) = unbounded();
+        let (consume_sender, listen_receiver) = bounded(GOSSIP_CHANNEL_CAPACITY);
         let t_socket_consume = cluster_info.clone().start_socket_consume_thread(
             bank_forks.clone(),
             request_receiver,
             consume_sender,
             exit.clone(),
         );
-        let (response_sender, response_receiver) = unbounded();
+        let (response_sender, response_receiver) = bounded(GOSSIP_CHANNEL_CAPACITY);
         let t_listen = cluster_info.clone().listen(
             bank_forks.clone(),
             listen_receiver,
diff --git a/streamer/src/streamer.rs b/streamer/src/streamer.rs
index b8dc0eb1699b30..4767322cfeb6bf 100644
--- a/streamer/src/streamer.rs
+++ b/streamer/src/streamer.rs
@@ -7,7 +7,7 @@ use {
         sendmmsg::{batch_send, SendPktsError},
         socket::SocketAddrSpace,
     },
-    crossbeam_channel::{Receiver, RecvTimeoutError, SendError, Sender},
+    crossbeam_channel::{Receiver, RecvTimeoutError, SendError, Sender, TrySendError},
     histogram::Histogram,
     itertools::Itertools,
     solana_packet::Packet,
@@ -61,6 +61,7 @@ pub struct StreamerReceiveStats {
     pub packet_batches_count: AtomicUsize,
     pub full_packet_batches_count: AtomicUsize,
     pub max_channel_len: AtomicUsize,
+    pub num_packets_dropped: AtomicUsize,
 }
 
 impl StreamerReceiveStats {
@@ -71,6 +72,7 @@ impl StreamerReceiveStats {
             packet_batches_count: AtomicUsize::default(),
             full_packet_batches_count: AtomicUsize::default(),
             max_channel_len: AtomicUsize::default(),
+            num_packets_dropped: AtomicUsize::default(),
         }
     }
 
@@ -97,6 +99,11 @@ impl StreamerReceiveStats {
                 self.max_channel_len.swap(0, Ordering::Relaxed) as i64,
                 i64
             ),
+            (
+                "num_packets_dropped",
+                self.num_packets_dropped.swap(0, Ordering::Relaxed) as i64,
+                i64
+            ),
         );
     }
 }
@@ -153,7 +160,9 @@ fn recv_loop(
                     packet_batch
                         .iter_mut()
                         .for_each(|p| p.meta_mut().set_from_staked_node(is_staked_service));
-                    packet_batch_sender.send(packet_batch)?;
+                    if let Err(TrySendError::Full(_)) = packet_batch_sender.try_send(packet_batch) {
+                        stats.num_packets_dropped.fetch_add(len, Ordering::Relaxed);
+                    }
                 }
                 break;
             }


Reviews:
user: login: t-nelson
body: +r network team approval
state: COMMENTED
author_association: NONE
submitted_at: 2025-02-28T23:18:26Z
user: login: gregcusack
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-01T01:53:22Z
user: login: gregcusack
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-01T06:06:05Z
user: login: cpubot
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-01T19:03:42Z
user: login: cpubot
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-01T19:06:14Z
user: login: gregcusack
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-03T17:52:30Z
user: login: gregcusack
body: lgtm! would be good to start reporting streaming metrics for gossip in another PR so we know if any packets start getting dropped from the sender side. Wouldn't expect there to be many but will help us debug if issues. 
state: DISMISSED
author_association: NONE
submitted_at: 2025-03-03T19:01:01Z
user: login: gregcusack
body: lgtm!
state: APPROVED
author_association: NONE
submitted_at: 2025-03-03T20:56:31Z

Review comments:
diff_hunk: @@ -71,6 +72,7 @@ impl StreamerReceiveStats {
             packet_batches_count: AtomicUsize::default(),
             full_packet_batches_count: AtomicUsize::default(),
             max_channel_len: AtomicUsize::default(),
+            num_packets_dropped: AtomicUsize::default(),
path: streamer/src/streamer.rs
user: login: gregcusack
body: side note: is it time to start reporting `StreamerReceiveStats`? or is there a reason we don't actually report these stats.
created_at: 2025-03-01T00:57:20Z
author_association: NONE
diff_hunk: @@ -2167,7 +2186,14 @@ impl ClusterInfo {
                 }
             })
         };
-        Ok(sender.send(packets)?)
+        if let Err(TrySendError::Full(_)) = sender.try_send(packets_verified) {
+            let mut dropped_packets_counts = [0u64; 7];
+            for packet_batch in packets {
+                count_dropped_packets(&packet_batch, &mut dropped_packets_counts);
+            }
path: gossip/src/cluster_info.rs
user: login: gregcusack
body: looks like if we ever hit this we'll be double counting dropped packets. packets received are already incremented in `verify_packet()`: 
```
let mut protocol: Protocol =
                stats.record_received_packet(packet.deserialize_slice::<Protocol, _>(..))?;
```
^ so this will increment packets received. and then if we fail to send any of them, we increment packets received again. 
created_at: 2025-03-01T06:05:57Z
author_association: NONE
diff_hunk: @@ -2167,7 +2186,14 @@ impl ClusterInfo {
                 }
             })
         };
-        Ok(sender.send(packets)?)
+        if let Err(TrySendError::Full(_)) = sender.try_send(packets_verified) {
+            let mut dropped_packets_counts = [0u64; 7];
+            for packet_batch in packets {
+                count_dropped_packets(&packet_batch, &mut dropped_packets_counts);
+            }
path: gossip/src/cluster_info.rs
user: login: cpubot
body: Good catch - thanks!
created_at: 2025-03-01T19:03:41Z
author_association: NONE
diff_hunk: @@ -71,6 +72,7 @@ impl StreamerReceiveStats {
             packet_batches_count: AtomicUsize::default(),
             full_packet_batches_count: AtomicUsize::default(),
             max_channel_len: AtomicUsize::default(),
+            num_packets_dropped: AtomicUsize::default(),
path: streamer/src/streamer.rs
user: login: cpubot
body: I can certainly include stat reporting in this PR. 

As far as I can tell, when the stats component of streamer was added, _reporting_ of those stats was not added to all usage sites, presumably to satisfy type checking of that change without making assumptions about reporting. I.e., I don't think we're deliberately not reporting stats
created_at: 2025-03-01T19:06:14Z
author_association: NONE
diff_hunk: @@ -71,6 +72,7 @@ impl StreamerReceiveStats {
             packet_batches_count: AtomicUsize::default(),
             full_packet_batches_count: AtomicUsize::default(),
             max_channel_len: AtomicUsize::default(),
+            num_packets_dropped: AtomicUsize::default(),
path: streamer/src/streamer.rs
user: login: gregcusack
body: no don't need to report stats in this PR. but maybe it makes sense to start reporting them in a follow up PR. Will need to think about it. 
created_at: 2025-03-03T17:52:30Z
author_association: NONE

PR comments:
user: login: gregcusack
created_at: 2025-03-01T01:53:02Z
body: pr looks good but i am running some tests to get a better understanding of the behavior
author_association: NONE
user: login: sakridge
created_at: 2025-03-03T17:26:50Z
body: +1 from me
author_association: NONE