Repository: zeek/zeek
Pull request #4387: Add telemetry for Broker's per-peer send buffers
Author: login: ckreibich
Created: 2025-04-24T06:55:45Z | Merged: 2025-04-25T17:22:29Z

This adds three metrics for each Broker peering's send buffer: the current buffer fill level, the largest fill level seen recently, and a count of the number of buffer overflows. See the commits  for details, but the essence is that I'm tracking low-level counts in the Broker manager by extending the `event_observer`, and use a script-level `Telemetry:sync()` hook to tie this to the cluster framework's node-naming.

I've tried to look out for performance, particularly in the per-message observer callbacks. The sync hook could be optimized further, for example we could store the resulting Cluster node name in a redef'd version of the `BrokerPeeringStats` record. It felt a bit like premature optimization so I steered clear for now.

Also includes a fix for an unrelated race condition in the `cluster.broker.publish-any` test that triggered on my machine.

Once this makes it, I'm going to follow up with a tiny PR that switches the default buffer overflow policy to `drop_oldest`. I've tested this quite a bit and its behavior is much tamer — since no peerings ever get disconnected there's nothing to reconnect, and the buffer overflow count becomes a simple indicator of the number of messages lost. It also means that if there's marginal overflow, it's smaller loss than the entire send buffer plus data in flight, of a disconnected peering. 

Resolves #4319.

Linked issue #4319: Establish visibility into Broker peer buffer fill levels
Another follow-up to #3981: the size of the send buffers (`Broker::peer_buffer_size` and `Broker::web_socket_buffer_size`) is a tunable. We can provide reasonable defaults, but sites should be able to gauge the fit. Currently you either see no backpressure issues, or you do — you have to tune sizing from backpressure overflow frequency.

We'd like an API into the buffering system to retrieve the current fill level of a given peering's send buffer, for example to expose them as a telemetry gauges. This should happen at the cluster level, not Broker's, so endpoint naming is fixed (think "broker_peerbuffer_entries_worker23_proxy02", not "broker_peerbuffer_entries_864f81a0-0c93-574c-954c-024e19e739bb_6dca6236-e666-52bc-a154-d5e400469359").

#3998 is a building-block for this.

Diff:
diff --git a/scripts/base/frameworks/broker/main.zeek b/scripts/base/frameworks/broker/main.zeek
index a3704b7157b..7402d39ecb4 100644
--- a/scripts/base/frameworks/broker/main.zeek
+++ b/scripts/base/frameworks/broker/main.zeek
@@ -104,6 +104,10 @@ export {
 	## Same as :zeek:see:`Broker::peer_overflow_policy` but for WebSocket clients.
 	const web_socket_overflow_policy = "disconnect" &redef;
 
+	## How frequently Zeek resets some peering/client buffer statistics,
+	## such as ``max_queued_recently`` in :zeek:see:`BrokerPeeringStats`.
+	const buffer_stats_reset_interval = 1min &redef;
+
 	## The CAF scheduling policy to use.  Available options are "sharing" and
 	## "stealing".  The "sharing" policy uses a single, global work queue along
 	## with mutex and condition variable used for accessing it, which may be
@@ -392,6 +396,12 @@ export {
 	## Returns: a unique identifier for the local broker endpoint.
 	global node_id: function(): string;
 
+	## Obtain each peering's send-buffer statistics. The keys are Broker
+	## endpoint IDs.
+	##
+	## Returns: per-peering statistics.
+	global peering_stats: function(): table[string] of BrokerPeeringStats;
+
 	## Sends all pending log messages to remote peers.  This normally
 	## doesn't need to be used except for test cases that are time-sensitive.
 	global flush_logs: function(): count;
@@ -554,6 +564,11 @@ function node_id(): string
 	return __node_id();
 	}
 
+function peering_stats(): table[string] of BrokerPeeringStats
+	{
+	return __peering_stats();
+	}
+
 function flush_logs(): count
 	{
 	return __flush_logs();
diff --git a/scripts/base/frameworks/cluster/__load__.zeek b/scripts/base/frameworks/cluster/__load__.zeek
index 0d6372e3d46..85fef40c5fc 100644
--- a/scripts/base/frameworks/cluster/__load__.zeek
+++ b/scripts/base/frameworks/cluster/__load__.zeek
@@ -14,8 +14,11 @@ redef Broker::log_topic = Cluster::rr_log_topic;
 # Add a cluster prefix.
 @prefixes += cluster
 
-# This should soon condition on loading only when Broker is in use.
+# Broker-specific additions:
+@if ( Cluster::backend == Cluster::CLUSTER_BACKEND_BROKER )
 @load ./broker-backpressure
+@load ./broker-telemetry
+@endif
 
 @if ( Supervisor::is_supervised() )
 # When running a supervised cluster, populate Cluster::nodes from the node table
diff --git a/scripts/base/frameworks/cluster/broker-telemetry.zeek b/scripts/base/frameworks/cluster/broker-telemetry.zeek
new file mode 100644
index 00000000000..7aa1e8fb3f1
--- /dev/null
+++ b/scripts/base/frameworks/cluster/broker-telemetry.zeek
@@ -0,0 +1,69 @@
+# Additional Broker-specific metrics that use Zeek cluster-level node names.
+
+@load base/frameworks/telemetry
+
+module Cluster;
+
+## This gauge tracks the current number of locally queued messages in each
+## Broker peering's send buffer. The "peer" label identifies the remote side of
+## the peering, containing a Zeek cluster node name.
+global broker_peer_buffer_messages_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-messages",
+    $unit="",
+    $label_names=vector("peer"),
+    $help_text="Number of messages queued in Broker's send buffers",
+]);
+
+## This gauge tracks recent maximum queue lengths for each Broker peering's send
+## buffer. Most of the time the send buffers are nearly empty, so this gauge
+## helps understand recent bursts of messages.  "Recent" here means
+## :zeek:see:`Broker::buffer_stats_reset_interval`. The time window advances in
+## increments of at least the stats interval, not incrementally with every new
+## observed message. That is, Zeek keeps a timestamp of when the window started,
+## and once it notices that the interval has passed, it moves the start of the
+## window to current time.
+global broker_peer_buffer_recent_max_messages_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-recent-max-messages",
+    $unit="",
+    $label_names=vector("peer"),
+    $help_text="Maximum number of messages recently queued in Broker's send buffers",
+]);
+
+## This counter tracks for each Broker peering the number of times its send
+## buffer has overflowed. For the "disconnect" policy this can at most be 1,
+## since Broker stops the peering at this time. For the "drop_oldest" and
+## "drop_newest" policies (see :zeek:see:`Broker:peer_overflow_policy`) the count
+## instead reflects the number of messages lost.
+global broker_peer_buffer_overflows_cf = Telemetry::register_counter_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-overflows",
+    $unit="",
+    $label_names=vector("peer"),
+    $help_text="Number of overflows in Broker's send buffers",
+]);
+
+hook Telemetry::sync()
+	{
+	local peers = Broker::peering_stats();
+	local nn: NamedNode;
+
+	for ( peer, stats in peers )
+		{
+		# Translate the Broker IDs to Zeek-level node names. We skip
+		# telemetry for peers where this mapping fails, i.e. ones for
+		# connections to external systems.
+		nn = nodeid_to_node(peer);
+
+		if ( |nn$name| > 0 )
+			{
+			Telemetry::gauge_family_set(broker_peer_buffer_messages_gf,
+			    vector(nn$name), stats$num_queued);
+			Telemetry::gauge_family_set(broker_peer_buffer_recent_max_messages_gf,
+			    vector(nn$name), stats$max_queued_recently);
+			Telemetry::counter_family_set(broker_peer_buffer_overflows_cf,
+			    vector(nn$name), stats$num_overflows);
+			}
+		}
+	}
diff --git a/scripts/base/init-bare.zeek b/scripts/base/init-bare.zeek
index 98b16e103ef..c8e6c7440ab 100644
--- a/scripts/base/init-bare.zeek
+++ b/scripts/base/init-bare.zeek
@@ -1135,6 +1135,20 @@ type BrokerStats: record {
 	num_ids_outgoing: count;
 };
 
+## Broker statistics for an individual peering.
+##
+type BrokerPeeringStats: record {
+	## The number of messages currently queued locally for transmission.
+	num_queued: count;
+	## The maximum number of messages queued in the recent
+	## :zeek:see:`Broker::buffer_stats_reset_interval` time interval.
+	max_queued_recently: count;
+	## The number of times the send buffer has overflowed.
+	num_overflows: count;
+};
+
+type BrokerPeeringStatsTable: table[string] of BrokerPeeringStats;
+
 ## Statistics about reporter messages and weirds.
 ##
 ## .. zeek:see:: get_reporter_stats
diff --git a/src/broker/Manager.cc b/src/broker/Manager.cc
index dee1c162cfa..853255523bc 100644
--- a/src/broker/Manager.cc
+++ b/src/broker/Manager.cc
@@ -92,6 +92,174 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        // The rendered peer ID. Storing this here helps reuse.
+        // Note that we only ever touch this from Zeek's main thread, not
+        // any of Broker's.
+        zeek::StringValPtr peer_id;
+
+        // Whether Broker has removed the peer, and this instance still
+        // needs to be removed.
+        bool is_zombie = false;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_reset_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_reset_interval)
+        : buffer_size(a_buffer_size), stats_reset_interval(a_stats_reset_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    void SetEndpoint(const broker::endpoint* a_endpoint) { endpoint = a_endpoint; }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, is_push == true) and departures (a pull, is_push ==
+    // false) as they happen. Note that this must not touch Zeek-side Vals.
+    void Observe(const broker::endpoint_id& peer, bool is_push) {
+        std::lock_guard<std::mutex> lock(mutex);
+        auto it = stats_map.find(peer);
+
+        if ( it == stats_map.end() ) {
+            stats_map.emplace(peer, Stats());
+            it = stats_map.find(peer);
+        }
+
+        auto& stats = it->second;
+
+        // Stick to Broker's notion of time here.
+        double now{0};
+        if ( endpoint != nullptr )
+            broker::convert(endpoint->now(), now);
+
+        if ( now - stats.last_interval > stats_reset_interval ) {
+            stats.last_interval = now;
+            stats.max_queued_recently = stats.queued;
+        }
+
+        if ( stats.queued == 0 ) {
+            // Watch for underflows. We could report somehow. Note that this
+            // runs in the context of Broker's threads.
+            assert(is_push);
+        }
+
+        if ( is_push && stats.queued == buffer_size )
+            stats.overflows += 1;
+        else {
+            stats.queued += is_push ? 1 : -1;
+            if ( stats.queued > stats.max_queued_recently )
+                stats.max_queued_recently = stats.queued;
+        }
+    }
+
+    // Updates the internal table[string] of BrokerPeeringStats and returns it.
+    const zeek::TableValPtr& GetPeeringStatsTable() {
+        std::lock_guard<std::mutex> lock(mutex);
+
+        for ( auto it = stats_map.begin(); it != stats_map.end(); ) {
+            auto& peer = it->first;
+            auto& stats = it->second;
+
+            if ( stats.peer_id == nullptr )
+                stats.peer_id = PeerIdToStringVal(peer);
+
+            // Broker told us the peer is gone, in RemovePeer() below. Remove it
+            // now from both tables. We add/remove from stats_table only here,
+            // not in Observer() and/or RemovePeer(), to ensure we only touch
+            // the Zeek-side Table from Zeek's main thread.
+            if ( stats.is_zombie ) {
+                stats_table->Remove(*stats.peer_id);
+                it = stats_map.erase(it);
+                continue;
+            }
+
+            auto stats_v = stats_table->Find(stats.peer_id);
+
+            if ( stats_v == nullptr ) {
+                stats_v = zeek::make_intrusive<zeek::RecordVal>(stats_record_type);
+                stats_table->Assign(stats.peer_id, stats_v);
+            }
+
+            // We may get here more than stats_reset_interval after the last
+            // Observe(), in which case the max_queued_recently value is now
+            // stale. Update if so.
+            double now{0};
+            if ( endpoint != nullptr )
+                broker::convert(endpoint->now(), now);
+
+            if ( now - stats.last_interval > stats_reset_interval ) {
+                stats.last_interval = now;
+                stats.max_queued_recently = stats.queued;
+            }
+
+            int n = 0;
+            stats_v->AsRecordVal()->Assign(n++, zeek::val_mgr->Count(stats.queued));
+            stats_v->AsRecordVal()->Assign(n++, zeek::val_mgr->Count(stats.max_queued_recently));
+            stats_v->AsRecordVal()->Assign(n++, zeek::val_mgr->Count(stats.overflows));
+
+            ++it;
+        }
+
+        return stats_table;
+    }
+
+    void RemovePeer(const broker::endpoint_id& peer) {
+        std::lock_guard<std::mutex> lock(mutex);
+        if ( auto it = stats_map.find(peer); it != stats_map.end() )
+            it->second.is_zombie = true;
+    }
+
+private:
+    zeek::StringValPtr PeerIdToStringVal(const broker::endpoint_id& peer) const {
+        std::string peer_s;
+        broker::convert(peer, peer_s);
+        return zeek::make_intrusive<zeek::StringVal>(peer_s);
+    }
+
+    // The maximum number of messages queueable for transmission to a peer,
+    // see Broker::peer_buffer_size and Broker::web_socket_buffer_size.
+    size_t buffer_size;
+
+    // Seconds after which we reset stats tracked per time window.
+    double stats_reset_interval;
+
+    EndpointMetricMap stats_map;
+    zeek::TableValPtr stats_table;
+    zeek::RecordTypePtr stats_record_type;
+
+    mutable std::mutex mutex;
+    const broker::endpoint* endpoint = nullptr;
+};
+
+using PeerBufferStatePtr = std::shared_ptr<PeerBufferState>;
+
 class LoggerQueue {
 public:
     void Push(broker::event_ptr event) {
@@ -128,26 +296,40 @@ using LoggerQueuePtr = std::shared_ptr<LoggerQueue>;
 
 using BrokerSeverityLevel = broker::event::severity_level;
 
-class LoggerAdapter : public broker::event_observer {
+class Observer : public broker::event_observer {
 public:
-    using SeverityLevel = broker::event::severity_level;
+    using LogSeverityLevel = broker::event::severity_level;
+
+    explicit Observer(LogSeverityLevel severity, LoggerQueuePtr queue, PeerBufferStatePtr pbstate)
+        : severity_(severity), queue_(std::move(queue)), pbstate_(std::move(pbstate)) {}
+
+    void on_peer_buffer_push(const broker::endpoint_id& peer, const broker::node_message&) override {
+        pbstate_->Observe(peer, true);
+    }
 
-    explicit LoggerAdapter(SeverityLevel severity, LoggerQueuePtr queue)
-        : severity_(severity), queue_(std::move(queue)) {}
+    void on_peer_buffer_pull(const broker::endpoint_id& peer, const broker::node_message&) override {
+        pbstate_->Observe(peer, false);
+    }
+
+    void on_peer_disconnect(const broker::endpoint_id& peer, const broker::error&) override {
+        pbstate_->RemovePeer(peer);
+    }
 
     void observe(broker::event_ptr what) override { queue_->Push(std::move(what)); }
 
-    bool accepts(SeverityLevel severity, broker::event::component_type) const override { return severity <= severity_; }
+    bool accepts(LogSeverityLevel severity, broker::event::component_type) const override {
+        return severity <= severity_;
+    }
 
 private:
-    SeverityLevel severity_;
+    LogSeverityLevel severity_;
     LoggerQueuePtr queue_;
+    PeerBufferStatePtr pbstate_;
 };
 
 } // namespace
 
 namespace zeek::Broker {
-
 static inline Val* get_option(const char* option) {
     const auto& id = zeek::detail::global_scope()->Find(option);
 
@@ -220,19 +402,24 @@ struct opt_mapping {
 
 class BrokerState {
 public:
-    using SeverityLevel = LoggerAdapter::SeverityLevel;
+    using LogSeverityLevel = Observer::LogSeverityLevel;
 
-    BrokerState(broker::configuration config, size_t congestion_queue_size, LoggerQueuePtr queue)
+    BrokerState(broker::configuration config, size_t congestion_queue_size, LoggerQueuePtr queue,
+                PeerBufferStatePtr pbstate)
         : endpoint(std::move(config), telemetry_mgr->GetRegistry()),
           subscriber(
               endpoint.make_subscriber({broker::topic::statuses(), broker::topic::errors()}, congestion_queue_size)),
-          loggerQueue(std::move(queue)) {}
+          loggerQueue(std::move(queue)),
+          peerBufferState(std::move(pbstate)) {
+        peerBufferState->SetEndpoint(&endpoint);
+    }
 
     broker::endpoint endpoint;
     broker::subscriber subscriber;
     LoggerQueuePtr loggerQueue;
-    SeverityLevel logSeverity = SeverityLevel::critical;
-    SeverityLevel stderrSeverity = SeverityLevel::critical;
+    PeerBufferStatePtr peerBufferState;
+    LogSeverityLevel logSeverity = LogSeverityLevel::critical;
+    LogSeverityLevel stderrSeverity = LogSeverityLevel::critical;
     std::unordered_set<broker::network_info> outbound_peerings;
 };
 
@@ -402,11 +589,13 @@ void Manager::DoInitPostScript() {
     checkLogSeverity(stderrSeverityVal);
     auto adapterVerbosity = static_cast<BrokerSeverityLevel>(std::max(logSeverityVal, stderrSeverityVal));
     auto queue = std::make_shared<LoggerQueue>();
-    auto adapter = std::make_shared<LoggerAdapter>(adapterVerbosity, queue);
-    broker::logger(adapter); // *must* be called before creating the BrokerState
+    auto pbstate = std::make_shared<PeerBufferState>(options.peer_buffer_size,
+                                                     get_option("Broker::buffer_stats_reset_interval")->AsDouble());
+    auto observer = std::make_shared<Observer>(adapterVerbosity, queue, pbstate);
+    broker::logger(observer); // *must* be called before creating the BrokerState
 
     auto cqs = get_option("Broker::congestion_queue_size")->AsCount();
-    bstate = std::make_shared<BrokerState>(std::move(config), cqs, queue);
+    bstate = std::make_shared<BrokerState>(std::move(config), cqs, queue, pbstate);
     bstate->logSeverity = static_cast<BrokerSeverityLevel>(logSeverityVal);
     bstate->stderrSeverity = static_cast<BrokerSeverityLevel>(stderrSeverityVal);
 
@@ -1968,6 +2157,8 @@ const Stats& Manager::GetStatistics() {
     return statistics;
 }
 
+TableValPtr Manager::GetPeeringStatsTable() { return bstate->peerBufferState->GetPeeringStatsTable(); }
+
 bool Manager::AddForwardedStore(const std::string& name, TableValPtr table) {
     if ( forwarded_stores.find(name) != forwarded_stores.end() ) {
         reporter->Error("same &broker_store %s specified for two different variables", name.c_str());
diff --git a/src/broker/Manager.h b/src/broker/Manager.h
index 16738183493..d9c4a4fa81d 100644
--- a/src/broker/Manager.h
+++ b/src/broker/Manager.h
@@ -384,6 +384,14 @@ class Manager : public zeek::cluster::Backend, public iosource::IOSource {
      */
     const Stats& GetStatistics();
 
+    /**
+     * Returns a table[string] of BrokerPeeringStats, with each peering's
+     * send-buffer stats filled in. The keys are Broker node IDs identifying the
+     * current peers.
+     * @return Each peering's send-buffer statistics.
+     */
+    TableValPtr GetPeeringStatsTable();
+
     /**
      * Creating an instance of this struct simply helps the manager
      * keep track of whether calls into its API are coming from script
diff --git a/src/broker/comm.bif b/src/broker/comm.bif
index d348b5ebffb..29e9d84680a 100644
--- a/src/broker/comm.bif
+++ b/src/broker/comm.bif
@@ -264,3 +264,8 @@ function Broker::__node_id%(%): string
 	zeek::Broker::Manager::ScriptScopeGuard ssg;
 	return zeek::make_intrusive<zeek::StringVal>(broker_mgr->NodeId());
 	%}
+
+function Broker::__peering_stats%(%): BrokerPeeringStatsTable
+	%{
+	return broker_mgr->GetPeeringStatsTable();
+	%}
diff --git a/testing/btest/Baseline/broker.telemetry/manager.out b/testing/btest/Baseline/broker.telemetry/manager.out
new file mode 100644
index 00000000000..ce348b2b028
--- /dev/null
+++ b/testing/btest/Baseline/broker.telemetry/manager.out
@@ -0,0 +1,4 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+Telemetry::COUNTER, zeek, zeek_broker_peer_buffer_overflows_total, [endpoint, peer], [manager, worker]
+Telemetry::GAUGE, zeek, zeek_broker_peer_buffer_messages, [endpoint, peer], [manager, worker]
+Telemetry::GAUGE, zeek, zeek_broker_peer_buffer_recent_max_messages, [endpoint, peer], [manager, worker]
diff --git a/testing/btest/Baseline/broker.telemetry/worker.out b/testing/btest/Baseline/broker.telemetry/worker.out
new file mode 100644
index 00000000000..b2486b0acee
--- /dev/null
+++ b/testing/btest/Baseline/broker.telemetry/worker.out
@@ -0,0 +1,4 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+Telemetry::COUNTER, zeek, zeek_broker_peer_buffer_overflows_total, [endpoint, peer], [worker, manager]
+Telemetry::GAUGE, zeek, zeek_broker_peer_buffer_messages, [endpoint, peer], [worker, manager]
+Telemetry::GAUGE, zeek, zeek_broker_peer_buffer_recent_max_messages, [endpoint, peer], [worker, manager]
diff --git a/testing/btest/Baseline/coverage.init-default/missing_loads b/testing/btest/Baseline/coverage.init-default/missing_loads
index 9997ec4fd8d..93af34614e4 100644
--- a/testing/btest/Baseline/coverage.init-default/missing_loads
+++ b/testing/btest/Baseline/coverage.init-default/missing_loads
@@ -1,6 +1,7 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
 -./frameworks/cluster/broker-backpressure.zeek
 -./frameworks/cluster/broker-stores.zeek
+-./frameworks/cluster/broker-telemetry.zeek
 -./frameworks/cluster/nodes/logger.zeek
 -./frameworks/cluster/nodes/manager.zeek
 -./frameworks/cluster/nodes/proxy.zeek
diff --git a/testing/btest/Baseline/opt.ZAM-bif-tracking/output b/testing/btest/Baseline/opt.ZAM-bif-tracking/output
index c20efb9801f..67cc8f48385 100644
--- a/testing/btest/Baseline/opt.ZAM-bif-tracking/output
+++ b/testing/btest/Baseline/opt.ZAM-bif-tracking/output
@@ -1,2 +1,2 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
-558 seen BiFs, 0 unseen BiFs (), 0 new BiFs ()
+559 seen BiFs, 0 unseen BiFs (), 0 new BiFs ()
diff --git a/testing/btest/broker/telemetry.zeek b/testing/btest/broker/telemetry.zeek
new file mode 100644
index 00000000000..3008dd72f0a
--- /dev/null
+++ b/testing/btest/broker/telemetry.zeek
@@ -0,0 +1,55 @@
+# @TEST-DOC: run a mini two-node cluster and check that Broker's peering telemetry is available.
+#
+# @TEST-PORT: BROKER_PORT
+#
+# @TEST-EXEC: btest-bg-run manager ZEEKPATH=$ZEEKPATH:.. CLUSTER_NODE=manager zeek -b manager.zeek
+# @TEST-EXEC: btest-bg-run worker ZEEKPATH=$ZEEKPATH:.. CLUSTER_NODE=worker zeek -b worker.zeek
+# @TEST-EXEC: btest-bg-wait 15
+#
+# @TEST-EXEC: btest-diff manager/out
+# @TEST-EXEC: btest-diff worker/out
+
+# @TEST-START-FILE cluster-layout.zeek
+redef Cluster::nodes = {
+	["manager"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT"))],
+	["worker"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $manager="manager"],
+};
+# @TEST-END-FILE
+
+# @TEST-START-FILE common.zeek
+@load base/frameworks/cluster
+@load policy/frameworks/cluster/experimental
+
+redef exit_only_after_terminate = T;
+redef Log::enable_local_logging = T;
+redef Log::default_rotation_interval = 0secs;
+redef Cluster::retry_interval = 1sec;
+
+function print_metrics(metrics: vector of Telemetry::Metric)
+	{
+	local f = open("out");
+
+	for (i in metrics)
+		{
+		local m = metrics[i];
+		print f, m$opts$metric_type, m$opts$prefix, m$opts$name, m$label_names, m?$label_values ? m$label_values : vector();
+		}
+
+	close(f);
+	}
+
+event Cluster::Experimental::cluster_started()
+	{
+	local broker_metrics = Telemetry::collect_metrics("zeek_broker_peer_buffer*", "*");
+	print_metrics(broker_metrics);
+	terminate();
+	}
+# @TEST-END-FILE
+
+# @TEST-START-FILE manager.zeek
+@load ./common.zeek
+# @TEST-END-FILE
+
+# @TEST-START-FILE worker.zeek
+@load ./common.zeek
+# @TEST-END-FILE
diff --git a/testing/btest/cluster/broker/publish-any.zeek b/testing/btest/cluster/broker/publish-any.zeek
index 62715481160..818d153ce44 100644
--- a/testing/btest/cluster/broker/publish-any.zeek
+++ b/testing/btest/cluster/broker/publish-any.zeek
@@ -56,8 +56,6 @@ event send_any()
 	local e = Cluster::make_event(ping, i, type_name(val), val);
 	Cluster::publish_hrw(Cluster::worker_pool, cat(i), e);
 	++i;
-
-	schedule 0.05sec { send_any() };
 	}
 
 event pong(c: count, what: string, val: any)
@@ -65,10 +63,17 @@ event pong(c: count, what: string, val: any)
 	++pongs;
 	print "got pong", pongs, "for ping", c, what, type_name(val), val;
 
-	# We send 5 pings in 3 different variations and
-	# get 4 one pong for each.
+	# The manager send 5 types of pings, in 3 different ways. The worker
+	# answers each ping in 4 ways, for a total of 60 expected pongs at the
+	# manager. Every batch of pings for one type involves 12 pongs.
+
 	if ( pongs == 60 )
 		Cluster::publish(Cluster::worker_topic, finish);
+	else if ( pongs > 0 && pongs % 12 == 0 )
+		{
+		# Wait for a batch to complete before sending the next.
+		event send_any();
+		}
 	}
 
 event Cluster::node_up(name: string, id: string)
diff --git a/testing/btest/opt/ZAM-bif-tracking.zeek b/testing/btest/opt/ZAM-bif-tracking.zeek
index 4dbce8e247b..5184cd68a6a 100644
--- a/testing/btest/opt/ZAM-bif-tracking.zeek
+++ b/testing/btest/opt/ZAM-bif-tracking.zeek
@@ -7,7 +7,7 @@
 # @TEST-EXEC: btest-diff output
 
 # This set tracks the BiFs that have been characterized for ZAM analysis.
-# As new ones are added or old ones removed, attend to updating FuncInfo.cc
+# As new ones are added or old ones removed, update src/script_opt/FuncInfo.cc
 # for ZAM, and then update the list here.
 global known_BiFs = set(
 	"Analyzer::__disable_all_analyzers",
@@ -45,6 +45,7 @@ global known_BiFs = set(
 	"Broker::__opaque_clone_through_serialization",
 	"Broker::__peer",
 	"Broker::__peer_no_retry",
+	"Broker::__peering_stats",
 	"Broker::__peers",
 	"Broker::__pop",
 	"Broker::__publish_id",


Reviews:
user: login: awelzel
body: I think I'd be happy to approve, but leaving just a "comment" for now. Thanks for fixing the flaky cluster test!
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T13:29:51Z
user: login: ckreibich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T19:26:55Z
user: login: ckreibich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T19:27:10Z
user: login: ckreibich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T19:28:04Z
user: login: ckreibich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T19:41:37Z
user: login: ckreibich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T19:44:42Z
user: login: awelzel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T08:05:29Z
user: login: awelzel
body: I've tested this branch (actually the one from #4396) with [zeek-cluster-perf](https://github.com/awelzel/zeek-cluster-perf) with 56 workers. It's maybe 1-2% slower. These tests set `peer_buffer_size` to `512 * 1024` in order to avoid any drops/disconnects and are likely triggering this overflow code a lot. I'm not worried about environments that do actual packet processing, too.

Here's some entries from the `telemetry.log` file collected during the `broadcast-highrate` test:

```
worker-51       counter zeek_broker_peer_buffer_overflows_total endpoint,peer   worker-51,proxy-3       0.0
worker-51       gauge   zeek_broker_peer_buffer_levels  endpoint,peer   worker-51,proxy-4       11765.0
worker-51       gauge   zeek_broker_peer_buffer_levels  endpoint,peer   worker-51,proxy-1       11021.0
worker-54       gauge   zeek_broker_peer_buffer_recent_max_levels       endpoint,peer   worker-54,proxy-3       64573.0
worker-54       gauge   zeek_broker_peer_buffer_recent_max_levels       endpoint,peer   worker-54,proxy-4       64661.0
worker-56       counter zeek_broker_peer_buffer_overflows_total endpoint,peer   worker-56,proxy-1       0.0
worker-56       gauge   zeek_broker_peer_buffer_levels  endpoint,peer   worker-56,proxy-2       10612.0
worker-55       counter zeek_broker_peer_buffer_overflows_total endpoint,peer   worker-55,proxy-1       0.0
worker-55       gauge   zeek_broker_peer_buffer_levels  endpoint,peer   worker-55,proxy-2       10668.0
```

I've left one comment about naming and the prefix usage that'd be good to adapt, but :+1: from me otherwise.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T08:46:20Z
user: login: ckreibich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T15:53:19Z
user: login: awelzel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T16:00:39Z
user: login: ckreibich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T17:25:54Z

Review comments:
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
+        std::lock_guard<std::mutex> lock(mutex);
+        auto it = stats_map.find(peer);
+
+        if ( it == stats_map.end() ) {
+            Stats stats{peer};
+            stats_map.emplace(peer, stats);
+            stats_table->Assign(stats.peer_id, zeek::make_intrusive<zeek::RecordVal>(stats_record_type));
+            it = stats_map.find(peer);
+        }
+
+        auto& stats = it->second;
+        auto now = zeek::util::current_time();
+
+        if ( now - stats.last_interval > stats_interval ) {
+            stats.last_interval = now;
+            stats.max_queued_recently = 0;
path: src/broker/Manager.cc
user: login: awelzel
body: ```suggestion
            stats.max_queued_recently = stats.queued;
```

Not sure that matters all that much, but given we may not update `max_queued_recently` below if we're in a overflow situation, might make sense.
created_at: 2025-04-24T13:07:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
+        std::lock_guard<std::mutex> lock(mutex);
+        auto it = stats_map.find(peer);
+
+        if ( it == stats_map.end() ) {
+            Stats stats{peer};
+            stats_map.emplace(peer, stats);
+            stats_table->Assign(stats.peer_id, zeek::make_intrusive<zeek::RecordVal>(stats_record_type));
+            it = stats_map.find(peer);
+        }
+
+        auto& stats = it->second;
+        auto now = zeek::util::current_time();
+
+        if ( now - stats.last_interval > stats_interval ) {
+            stats.last_interval = now;
+            stats.max_queued_recently = 0;
+        }
+
+        if ( stats.queued == 0 ) {
+            // Watch for underflows. We could report somehow. Note that this
+            // runs in the context of Broker's threads.
+            assert(push_or_pull);
+        }
+
+        if ( push_or_pull && stats.queued == buffer_size )
+            stats.overflows += 1;
+        else {
+            stats.queued += push_or_pull ? 1 : -1;
+            if ( stats.queued > stats.max_queued_recently )
+                stats.max_queued_recently = stats.queued;
+        }
+    }
+
+    // Updates the internal table[string] of BrokerPeeringStats and returns it.
+    const zeek::TableValPtr& GetPeeringStatsTable() {
+        std::lock_guard<std::mutex> lock(mutex);
+
+        for ( auto& [peer, stats] : stats_map ) {
+            auto stats_v = stats_table->Find(stats.peer_id);
+
+            assert(stats_v != nullptr);
+
+            // We may get here more than stats_interval after the last
+            // Observe(), in which case the max_queued_recently value is now
+            // stale. Reset if so.
+            auto now = zeek::util::current_time();
+
+            if ( now - stats.last_interval > stats_interval ) {
+                stats.last_interval = now;
+                stats.max_queued_recently = 0;
path: src/broker/Manager.cc
user: login: awelzel
body: I guess it matters more here. It'd be weird to observe a record with `queued > 0` but `max_queued_recently == 0,`, no?
created_at: 2025-04-24T13:08:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
path: src/broker/Manager.cc
user: login: awelzel
body: I find this `push_or_pull` bool super cryptic to read, but I guess you got used to me. To me it first read as "either it is a push or pull", but the usage is obviously much different. Maybe `is_push` like `is_orig` and comment that it's otherwise a pull?
created_at: 2025-04-24T13:14:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
+        std::lock_guard<std::mutex> lock(mutex);
path: src/broker/Manager.cc
user: login: awelzel
body: Are we concerned that this is potentially hot and/or contested? E.g. if one broadcasts a lot of messages to a lot of nodes, is this invoked from various threads in broker or just from a single thread?
created_at: 2025-04-24T13:27:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,6 +104,11 @@ export {
 	## Same as :zeek:see:`Broker::peer_overflow_policy` but for WebSocket clients.
 	const web_socket_overflow_policy = "disconnect" &redef;
 
+	## The time interval over which Zeek tracks windowed buffer statistics
+	## for each peering/client. See e.g. the ``max_queued_recently`` member
+	## of :zeek:see:`BrokerPeeringStats`.
+	const buffer_stats_interval = 1min &redef;
path: scripts/base/frameworks/broker/main.zeek
user: login: awelzel
body: I wonder if the word `reset` to add to the name. `max_queued_stats_reset_interval` for a mouth full.
created_at: 2025-04-24T13:28:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
+        std::lock_guard<std::mutex> lock(mutex);
+        auto it = stats_map.find(peer);
+
+        if ( it == stats_map.end() ) {
+            Stats stats{peer};
+            stats_map.emplace(peer, stats);
+            stats_table->Assign(stats.peer_id, zeek::make_intrusive<zeek::RecordVal>(stats_record_type));
+            it = stats_map.find(peer);
+        }
+
+        auto& stats = it->second;
+        auto now = zeek::util::current_time();
+
+        if ( now - stats.last_interval > stats_interval ) {
+            stats.last_interval = now;
+            stats.max_queued_recently = 0;
path: src/broker/Manager.cc
user: login: ckreibich
body: No I think you're right, it's more precise, thanks! (I did notice this in my testing on live machines: it's entirely realistic that no further observations come in for a while.)
created_at: 2025-04-24T19:26:54Z
author_association: MEMBER
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
+        std::lock_guard<std::mutex> lock(mutex);
+        auto it = stats_map.find(peer);
+
+        if ( it == stats_map.end() ) {
+            Stats stats{peer};
+            stats_map.emplace(peer, stats);
+            stats_table->Assign(stats.peer_id, zeek::make_intrusive<zeek::RecordVal>(stats_record_type));
+            it = stats_map.find(peer);
+        }
+
+        auto& stats = it->second;
+        auto now = zeek::util::current_time();
+
+        if ( now - stats.last_interval > stats_interval ) {
+            stats.last_interval = now;
+            stats.max_queued_recently = 0;
+        }
+
+        if ( stats.queued == 0 ) {
+            // Watch for underflows. We could report somehow. Note that this
+            // runs in the context of Broker's threads.
+            assert(push_or_pull);
+        }
+
+        if ( push_or_pull && stats.queued == buffer_size )
+            stats.overflows += 1;
+        else {
+            stats.queued += push_or_pull ? 1 : -1;
+            if ( stats.queued > stats.max_queued_recently )
+                stats.max_queued_recently = stats.queued;
+        }
+    }
+
+    // Updates the internal table[string] of BrokerPeeringStats and returns it.
+    const zeek::TableValPtr& GetPeeringStatsTable() {
+        std::lock_guard<std::mutex> lock(mutex);
+
+        for ( auto& [peer, stats] : stats_map ) {
+            auto stats_v = stats_table->Find(stats.peer_id);
+
+            assert(stats_v != nullptr);
+
+            // We may get here more than stats_interval after the last
+            // Observe(), in which case the max_queued_recently value is now
+            // stale. Reset if so.
+            auto now = zeek::util::current_time();
+
+            if ( now - stats.last_interval > stats_interval ) {
+                stats.last_interval = now;
+                stats.max_queued_recently = 0;
path: src/broker/Manager.cc
user: login: ckreibich
body: Agree. Will tweak.
created_at: 2025-04-24T19:27:10Z
author_association: MEMBER
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
path: src/broker/Manager.cc
user: login: ckreibich
body: > I guess you got used to me

LOL — I did hesitate about that naming too, but I've seen similar patterns before. I'll make a pass, easy enough!
created_at: 2025-04-24T19:28:04Z
author_association: MEMBER
diff_hunk: @@ -92,6 +92,142 @@ void print_escaped(std::string& buf, std::string_view str) {
     buf.push_back('"');
 }
 
+// Track metrics for a given peering's send buffer.
+class PeerBufferState {
+public:
+    struct Stats {
+        Stats(const broker::endpoint_id& peer) {
+            std::string peer_s;
+            broker::convert(peer, peer_s);
+            peer_id = zeek::make_intrusive<zeek::StringVal>(peer_s);
+        }
+
+        // The rendered peer ID. Storing this here helps reuse.
+        zeek::StringValPtr peer_id;
+
+        // Number of messages queued locally in the send buffer.
+        uint32_t queued = 0;
+
+        // Maximum number queued in the last Broker::buffer_stats_interval.
+        // This improces visibility into message bursts since instantaneous
+        // queueing (captured above) can be short-lived.
+        uint32_t max_queued_recently = 0;
+
+        // Number of times the buffer overflowed at send time.  For the
+        // "disconnect" overflow policy (via Broker::peer_overflow_policy), this
+        // count will at most be 1 since Broker will remove the peering upon
+        // overflow. The existing Zeek-level metric for tracking disconnects
+        // (see frameworks/broker/broker-backpressure.zeek) covers this one more
+        // permanently. For the "drop_newest" and "drop_oldest" policies it
+        // equals a count of the number of messages lost, since the peering
+        // continues.
+        uint64_t overflows = 0;
+
+        // When we last started a stats-tracking interval for this peering.
+        double last_interval = 0;
+    };
+
+    // For per-peering tracking, map endpoint IDs to the above state.
+    using EndpointMetricMap = std::unordered_map<broker::endpoint_id, Stats>;
+
+    PeerBufferState(size_t a_buffer_size, double a_stats_interval)
+        : buffer_size(a_buffer_size), stats_interval(a_stats_interval) {
+        stats_table =
+            zeek::make_intrusive<zeek::TableVal>(zeek::id::find_type<zeek::TableType>("BrokerPeeringStatsTable"));
+        stats_record_type = zeek::id::find_type<zeek::RecordType>("BrokerPeeringStats");
+    }
+
+    // Update the peering's stats. This runs in Broker's execution context.
+    // Broker does not expose send-buffer/queue state explicitly, so track
+    // arrivals (a push, push_or_pull == true) and departures (pull,
+    // push_or_pull == false) as they happen.
+    void Observe(const broker::endpoint_id& peer, bool push_or_pull) {
+        std::lock_guard<std::mutex> lock(mutex);
path: src/broker/Manager.cc
user: login: ckreibich
body: I wondered too and looked at that a bit. I _think_ we're okay — there's no linearity in CAF threads to the number of peers. My understanding is that interaction is confined to a single one, the CAF scheduler. This changes when you increase `Broker::num_threads` from its default of one, but I don't think that's common, and even if one does, it's not clear it's immediately problematic.

I also wondered whether to make the whole thing optional and decided to be optimistic for now. It would be quite easy to iterate on this in a bugfix release.
created_at: 2025-04-24T19:41:37Z
author_association: MEMBER
diff_hunk: @@ -104,6 +104,11 @@ export {
 	## Same as :zeek:see:`Broker::peer_overflow_policy` but for WebSocket clients.
 	const web_socket_overflow_policy = "disconnect" &redef;
 
+	## The time interval over which Zeek tracks windowed buffer statistics
+	## for each peering/client. See e.g. the ``max_queued_recently`` member
+	## of :zeek:see:`BrokerPeeringStats`.
+	const buffer_stats_interval = 1min &redef;
path: scripts/base/frameworks/broker/main.zeek
user: login: ckreibich
body: Sure thing. The naming in all of this a bit tricky. For starters, there are in fact no buffers anywhere, it's all just queue management. So ...... :-)
created_at: 2025-04-24T19:44:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,69 @@
+# Additional Broker-specific metrics that use Zeek cluster-level node names.
+
+@load base/frameworks/telemetry
+
+module Cluster;
+
+## This gauge tracks the current number of locally queued messages in each
+## Broker peering's send buffer. The "peer" label identifies the remote side of
+## the peering, containing a Zeek cluster node name.
+global broker_peer_buffer_queued_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-levels",
+    $unit="",
+    $label_names=vector("peer"),
+    $help_text="Number of messages queued in Broker's send buffers",
+]);
+
+## This gauge tracks recent fill levels for each Broker peering's send
+## buffer. Most of the time the send buffers are nearly empty, so this gauge
+## helps understand recent bursts of messages.  "Recent" here means
+## :zeek:see:`Broker::buffer_stats_reset_interval`. The time window advances in
+## increments of at least the stats interval, not incrementally with every new
+## observed message. That is, Zeek keeps a timestamp of when the window started,
+## and once it notices that the interval has passed, it moves the start of the
+## window to current time.
+global broker_peer_buffer_max_queued_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-recent-max-levels",
+    $unit="",
+    $label_names=vector("peer"),
+    $help_text="Maximum number of messages recently queued in Broker's send buffers",
+]);
+
+## This counter tracks for each Broker peering the number of times its send
+## buffer has overflowed. For the "disconnect" policy this can at most be 1,
+## since Broker stops the peering at this time. For the "drop_oldest" and
+## "drop_newest" policies (see :zeek:see:`Broker:peer_overflow_policy`) the count
+## instead reflects the number of messages lost.
+global broker_peer_buffer_overflows_cf = Telemetry::register_counter_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-overflows",
+    $unit="",
+    $label_names=vector("peer"),
+    $help_text="Number of overflows in Broker's send buffers",
+]);
+
+hook Telemetry::sync()
+	{
+	local peers = Broker::peering_stats();
+	local nn: NamedNode;
+
+	for ( peer, stats in peers )
+		{
+		# Translate the Broker IDs to Zeek-level node names. We skip
+		# telemetry for peers where this mapping fails, i.e. ones for
+		# connections to external systems.
+		nn = nodeid_to_node(peer);
path: scripts/base/frameworks/cluster/broker-telemetry.zeek
user: login: awelzel
body: I was taking a step back, wondering again why broker couldn't just do metrics itself internally and we need to loop this all through Zeek. Nothing to do here, just thinking out loud :+1: 

> hook to tie this to the cluster framework's node-naming.

This is sort of the crux, right? Broker doesn't know anything about Zeek's node names for labeling of metrics. It  only knows about randomly generated identifiers.  An alternative might have been to inform the broker layer about the node name association at `Cluster::hello` time. E.g. something like `Broker::__set_peer_name(peer, nodeid_to_node(peer)$name)`.

The thinking is that broker could then update internal prometheus-cpp metrics directly where all the logic is already, rather than the decoupling via the observer, exposing some of the logic/intricacies to the Zeek layer.
created_at: 2025-04-25T08:05:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,69 @@
+# Additional Broker-specific metrics that use Zeek cluster-level node names.
+
+@load base/frameworks/telemetry
+
+module Cluster;
+
+## This gauge tracks the current number of locally queued messages in each
+## Broker peering's send buffer. The "peer" label identifies the remote side of
+## the peering, containing a Zeek cluster node name.
+global broker_peer_buffer_queued_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-levels",
path: scripts/base/frameworks/cluster/broker-telemetry.zeek
user: login: awelzel
body: I'm cringing on the name. The "levels" here sound like new terminology that could be avoided and it doesn't feel like good fit for prom metrics. "messages" isn't a unit, but "levels" even less :-) Maybe "broker-peer-buffer-queued-messages" - that's essentially what the help text says.

Or even `broker-peer-buffered-messages` or `broker-peer-queued-messages`.

I'm also unsure about the `zeek` prefix. Maybe this should use just a `broker` prefix?
created_at: 2025-04-25T08:45:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,69 @@
+# Additional Broker-specific metrics that use Zeek cluster-level node names.
+
+@load base/frameworks/telemetry
+
+module Cluster;
+
+## This gauge tracks the current number of locally queued messages in each
+## Broker peering's send buffer. The "peer" label identifies the remote side of
+## the peering, containing a Zeek cluster node name.
+global broker_peer_buffer_queued_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-levels",
path: scripts/base/frameworks/cluster/broker-telemetry.zeek
user: login: ckreibich
body: Well. :-) These buffers are measured in number of messages, so I'd argue a message is a unit here. If they were in bytes you'd agree that bytes are a unit? Levels came from "fill level", and I wasn't super happy with them either. Happy to drop that.

I thought about the prefix too, but I like the distinction of `broker_*` coming directly out of Broker (the library) vs ones generated by Zeek. There's a btest that verifies whether Broker-internal metrics make it into Zeek's prometheus registry, and it works by prefix, so we'd be adding these there too.
created_at: 2025-04-25T15:53:19Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,69 @@
+# Additional Broker-specific metrics that use Zeek cluster-level node names.
+
+@load base/frameworks/telemetry
+
+module Cluster;
+
+## This gauge tracks the current number of locally queued messages in each
+## Broker peering's send buffer. The "peer" label identifies the remote side of
+## the peering, containing a Zeek cluster node name.
+global broker_peer_buffer_queued_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-levels",
path: scripts/base/frameworks/cluster/broker-telemetry.zeek
user: login: awelzel
body: > so I'd argue a message is a unit here. 

Sorry - I was distracted by this sentence in the [naming docs](https://prometheus.io/docs/practices/naming/) - as I wasn't explicit, I like messages much more than levels.

>  For example, connections or notifications are not considered units for this rule and do not have to be at the end of the metric name.

created_at: 2025-04-25T16:00:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,69 @@
+# Additional Broker-specific metrics that use Zeek cluster-level node names.
+
+@load base/frameworks/telemetry
+
+module Cluster;
+
+## This gauge tracks the current number of locally queued messages in each
+## Broker peering's send buffer. The "peer" label identifies the remote side of
+## the peering, containing a Zeek cluster node name.
+global broker_peer_buffer_queued_gf = Telemetry::register_gauge_family([
+    $prefix="zeek",
+    $name="broker-peer-buffer-levels",
path: scripts/base/frameworks/cluster/broker-telemetry.zeek
user: login: ckreibich
body: It's all good! And thanks a bunch for doing performance tests, much appreciated. It's been running on `zeek-test` since yesterday without problems, btw.
created_at: 2025-04-25T17:25:54Z
author_association: MEMBER

PR comments:
user: login: timwoj
created_at: 2025-04-25T02:06:49Z
body: Looks like something about shutdown is causing a TSan failure: https://api.cirrus-ci.com/v1/task/4854649158631424/logs/test.log

I thought this was part of the follow-on PR, but it reproduces reliably on my machine with this branch. There's actually three data races there. I'm not sure what's happening in a couple of them, but the third one is because you can't use `TableVal` objects in a non-main thread. This is because `TableEntryVal` tries to access `run_state::network_time` in the constructor, and that variable is [not thread-safe](https://github.com/zeek/zeek/issues/4302).
author_association: MEMBER
user: login: ckreibich
created_at: 2025-04-25T02:11:25Z
body: Interesting. I'll take a look.
author_association: MEMBER
user: login: timwoj
created_at: 2025-04-25T02:18:30Z
body: It's curious that `TableEntryVal` even uses that value. We don't care about it unless the table has the `&expire_create` attribute on it, so it doesn't always need to be set. I could see adding an argument to the constructor, and possibly a field to the object itself, that avoids that for tables that don't need it. That would at least give a way out to avoid the lack of thread-safety in this case.
author_association: MEMBER
user: login: ckreibich
created_at: 2025-04-25T05:37:09Z
body: Okay, I was able to essentially keep the same approach, but shift all of the Zeek-side work into `PeerBufferState::GetPeeringStatsTable()`. Using `util::current_time()` was the other problem, just as you predicted Tim. I switched this to using Broker's notion of time (since it runs in a Broker event observer callback), which has its own problems since I think there's a chicken-and-egg problem here between registering the observer and availability of the local endpoint instance to the observer. I simply provide it once available, and take care when accessing in case the observer callbacks can trigger before then.

I've also rebased `topic/christian/broker-tuning` for #4396.
author_association: MEMBER