Repository: dask/distributed

Issue:
expose transfer bytes total (count) rather than just gauge of current value
Currently the worker Prometheus endpoint exposes `transfer_incoming_bytes` and `transfer_outgoing_bytes` as gauges—i.e., the current value at a single point in time.

A better way to expose this sort of data is as a monotonically increasing count metric type (this should be exposed as `transfer_incoming_bytes_total` and `transfer_outgoing_bytes_total`).

It's easy to get rate from an accumulated count, but you can't get accurate count from a sampled rate.


Discussion:
I'm in favor of this generally

On Thu, Oct 6, 2022, 9:36 AM Nat Tabris ***@***.***> wrote:

> Currently the worker Prometheus endpoint exposes transfer_incoming_bytes
> and transfer_outgoing_bytes as gauges—i.e., the current value at a single
> point in time.
>
> A better way to expose this sort of data is as a monotonically increasing
> count metric type (this should be exposed as transfer_outgoing_bytes_total
> and transfer_outgoing_bytes_total).
>
> It's easy to get rate from an accumulated count, but you can't get
> accurate count from a sampled rate.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/7123>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTGM2ZFQI42QDXJL2F3WB35XXANCNFSM6AAAAAAQ6ZYKTI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

We expose `transfer_incoming_bytes` and `transfer_outgoing_bytes` as gauges as these point-in-time metrics inform data transfer throttling and should therefore be visible. If you find exposing `transfer_{incoming|outgoing}_bytes_total` useful, we can add those. We should just make sure we have a good definition for those metrics (see https://github.com/dask/distributed/pull/6936#issuecomment-1230524443) for a more in-depth comment).
@ntabris is there anything left to do or are your concerns addressed?
I think we don't expose actually bytes transferred over time, i.e., what @crusaderky calls "cumulative" values in https://github.com/dask/distributed/pull/6936#issuecomment-1230524443

That's what I was asking for. If it's not high-value, feel free to ignore for now though.

For context, host metrics can tell us how much data moves in/out of each worker. What it can't exactly tell us (at least not easily) is how much of that is transfer vs data moving into/out of cluster (e.g., S3). I think it would be nice if Dask could tell us how many bytes of host network traffic is for transfer.
See also #7357
I would also find this useful for benchmarking. Total amount of data transferred is a useful metric to compare when working on changes to scheduling.

Resolving patch:
diff --git a/distributed/http/worker/prometheus/core.py b/distributed/http/worker/prometheus/core.py
--- a/distributed/http/worker/prometheus/core.py
+++ b/distributed/http/worker/prometheus/core.py
@@ -112,6 +112,15 @@ def collect(self):
             value=self.server.transfer_outgoing_count,
         )
 
+        yield CounterMetricFamily(
+            self.build_name("transfer_outgoing_bytes_total"),
+            (
+                "Total size of data transfers to other workers "
+                "since the worker was started (including in-progress and failed transfers)"
+            ),
+            value=self.server.transfer_outgoing_bytes_total,
+        )
+
         yield CounterMetricFamily(
             self.build_name("transfer_outgoing_count_total"),
             (
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -394,6 +394,8 @@ class Worker(BaseWorker, ServerNode):
     transfer_outgoing_log: deque[dict[str, Any]]
     #: Total number of data transfers to other workers since the worker was started
     transfer_outgoing_count_total: int
+    #: Total size of data transfers to other workers (including in-progress and failed transfers)
+    transfer_outgoing_bytes_total: int
     #: Current total size of open data transfers to other workers
     transfer_outgoing_bytes: int
     #: Current number of open data transfers to other workers
@@ -556,6 +558,7 @@ def __init__(
         self.transfer_incoming_log = deque(maxlen=100000)
         self.transfer_outgoing_log = deque(maxlen=100000)
         self.transfer_outgoing_count_total = 0
+        self.transfer_outgoing_bytes_total = 0
         self.transfer_outgoing_bytes = 0
         self.transfer_outgoing_count = 0
         self.bandwidth = parse_bytes(dask.config.get("distributed.scheduler.bandwidth"))
@@ -1746,6 +1749,7 @@ async def get_data(
         bytes_per_task = {k: self.state.tasks[k].nbytes or 0 for k in data}
         total_bytes = sum(bytes_per_task.values())
         self.transfer_outgoing_bytes += total_bytes
+        self.transfer_outgoing_bytes_total += total_bytes
         stop = time()
         if self.digests is not None:
             self.digests["get-data-load-duration"].add(stop - start)


Test patch:
diff --git a/distributed/http/worker/tests/test_worker_http.py b/distributed/http/worker/tests/test_worker_http.py
--- a/distributed/http/worker/tests/test_worker_http.py
+++ b/distributed/http/worker/tests/test_worker_http.py
@@ -36,6 +36,7 @@ async def test_prometheus(c, s, a):
         "dask_worker_transfer_outgoing_bytes",
         "dask_worker_transfer_outgoing_count",
         "dask_worker_transfer_outgoing_count_total",
+        "dask_worker_transfer_outgoing_bytes_total",
         "dask_worker_tick_count_total",
         "dask_worker_tick_duration_maximum_seconds",
     }