Repository: dask/distributed

Issue:
Event loop health metric
It would be helpful to have a prometheus metric for event loop health, probably measured in terms of time.  

Today we have a digest, but this depends on `crick` which has proven less reliable.  Crick might also be overkill for this.  Maybe it makes sense to keep an exponentially weighted moving average over, say, the last ten seconds or the last minute.  

cc @fjetter , thoughts?

cc @ntabris @dchudz 


Discussion:
I already added a max in https://github.com/dask/distributed/pull/7301 I think this is much more informative than an average. Max values impact connection instabilities, etc. and we can still average later. We can also add both, of course. It's pretty cheap after all
Max seems not granular enough.  Ideally we would see, for example, that
during disk writes the event loop health degrades.  OK great, we should
probably avoid disk.

With max we lose out on all of the history.  My sense is that an EWMA is a
more informative metric.

On Tue, Nov 15, 2022 at 3:26 AM Florian Jetter ***@***.***>
wrote:

> I already added a max in #7301
> <https://github.com/dask/distributed/pull/7301> I think this is much more
> informative than an average. Max values impact connection instabilities,
> etc. and we can still average later. We can also add both, of course. It's
> pretty cheap after all
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/7307#issuecomment-1315028737>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTEPRN7NHBGLD5IANTTWINJLHANCNFSM6AAAAAASAH4DLA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Max for some period, like the last 10s or minute?
We could also do a jumpy EWMA

```python
if new > avg:
    avg = new
else:
    avg = avg * 0.99 + new * 0.01
```
This way we're sensitive to spikes, but also make sure that polling systems like Prometheus will have a chance to pick them up
> Max seems not granular enough.  Ideally we would see, for example, that
during disk writes the event loop health degrades.  OK great, we should
probably avoid disk.

It's a max between prometheus polls. not global max
> It's a max between prometheus polls. 

Hm, you and Matt know best, but that does sound good! It's never going get more granular than prometheus polls I imagine.
Oh yeah, if it's between polls then I'm good.  Sorry for the confusion.  I should have read the code more.

Resolving patch:
diff --git a/distributed/core.py b/distributed/core.py
--- a/distributed/core.py
+++ b/distributed/core.py
@@ -394,9 +394,10 @@ def stop() -> bool:
         self.periodic_callbacks["monitor"] = pc
 
         self._last_tick = time()
+        self._max_tick_duration = 0
         self._tick_counter = 0
-        self._tick_count = 0
-        self._tick_count_last = time()
+        self._last_tick_counter = 0
+        self._last_tick_cycle = time()
         self._tick_interval = parse_timedelta(
             dask.config.get("distributed.admin.tick.interval"), default="ms"
         )
@@ -581,27 +582,32 @@ def listener(self):
 
     def _measure_tick(self):
         now = time()
-        diff = now - self._last_tick
+        tick_duration = now - self._last_tick
         self._last_tick = now
         self._tick_counter += 1
-        if diff > tick_maximum_delay:
+        # This metric is exposed in Prometheus and is reset there during
+        # collection
+        self._max_tick_duration = max(self._max_tick_duration, tick_duration)
+        if tick_duration > tick_maximum_delay:
             logger.info(
                 "Event loop was unresponsive in %s for %.2fs.  "
                 "This is often caused by long-running GIL-holding "
                 "functions or moving large chunks of data. "
                 "This can cause timeouts and instability.",
                 type(self).__name__,
-                diff,
+                tick_duration,
             )
         if self.digests is not None:
-            self.digests["tick-duration"].add(diff)
+            self.digests["tick-duration"].add(tick_duration)
 
     def _cycle_ticks(self):
         if not self._tick_counter:
             return
-        last, self._tick_count_last = self._tick_count_last, time()
-        count, self._tick_counter = self._tick_counter, 0
-        self._tick_interval_observed = (time() - last) / (count or 1)
+        now = time()
+        last_tick_cycle, self._last_tick_cycle = self._last_tick_cycle, now
+        count = self._tick_counter - self._last_tick_counter
+        self._last_tick_counter = self._tick_counter
+        self._tick_interval_observed = (now - last_tick_cycle) / (count or 1)
 
     @property
     def address(self) -> str:
diff --git a/distributed/http/scheduler/prometheus/core.py b/distributed/http/scheduler/prometheus/core.py
--- a/distributed/http/scheduler/prometheus/core.py
+++ b/distributed/http/scheduler/prometheus/core.py
@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from time import time
+
 import prometheus_client
 import toolz
 from prometheus_client.core import CounterMetricFamily, GaugeMetricFamily
@@ -87,6 +89,23 @@ def collect(self):
                 prefix_state_counts.add_metric([tp.name, state], count)
         yield prefix_state_counts
 
+        now = time()
+        max_tick_duration = max(
+            self.server._max_tick_duration, now - self.server._last_tick
+        )
+        self.server._max_tick_duration = 0
+        yield GaugeMetricFamily(
+            self.build_name("tick_duration_maximum_seconds"),
+            "Maximum tick duration observed since Prometheus last scraped metrics",
+            value=max_tick_duration,
+        )
+
+        yield CounterMetricFamily(
+            self.build_name("tick_count_total"),
+            "Total number of ticks observed since the server started",
+            value=self.server._tick_counter,
+        )
+
 
 COLLECTORS = [
     SchedulerMetricCollector,
diff --git a/distributed/http/worker/prometheus/core.py b/distributed/http/worker/prometheus/core.py
--- a/distributed/http/worker/prometheus/core.py
+++ b/distributed/http/worker/prometheus/core.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import logging
+from time import time
 from typing import ClassVar
 
 import prometheus_client
@@ -143,6 +144,23 @@ def collect(self):
                 .quantile(50),
             )
 
+        now = time()
+        max_tick_duration = max(
+            self.server._max_tick_duration, now - self.server._last_tick
+        )
+        self.server._max_tick_duration = 0
+        yield GaugeMetricFamily(
+            self.build_name("tick_duration_maximum_seconds"),
+            "Maximum tick duration observed since Prometheus last scraped metrics",
+            value=max_tick_duration,
+        )
+
+        yield CounterMetricFamily(
+            self.build_name("tick_count_total"),
+            "Total number of ticks observed since the server started",
+            value=self.server._tick_counter,
+        )
+
 
 class PrometheusHandler(RequestHandler):
     _collector: ClassVar[WorkerMetricCollector | None] = None


Test patch:
diff --git a/distributed/http/scheduler/tests/test_scheduler_http.py b/distributed/http/scheduler/tests/test_scheduler_http.py
--- a/distributed/http/scheduler/tests/test_scheduler_http.py
+++ b/distributed/http/scheduler/tests/test_scheduler_http.py
@@ -111,9 +111,11 @@ async def test_prometheus(c, s, a, b):
         "dask_scheduler_tasks_suspicious",
         "dask_scheduler_tasks_forgotten",
         "dask_scheduler_prefix_state_totals",
+        "dask_scheduler_tick_count",
+        "dask_scheduler_tick_duration_maximum_seconds",
     }
 
-    assert active_metrics.keys() == expected_metrics
+    assert set(active_metrics.keys()) == expected_metrics
     assert active_metrics["dask_scheduler_clients"].samples[0].value == 1.0
 
     # request data twice since there once was a case where metrics got registered multiple times resulting in
diff --git a/distributed/http/worker/tests/test_worker_http.py b/distributed/http/worker/tests/test_worker_http.py
--- a/distributed/http/worker/tests/test_worker_http.py
+++ b/distributed/http/worker/tests/test_worker_http.py
@@ -36,6 +36,8 @@ async def test_prometheus(c, s, a):
         "dask_worker_transfer_outgoing_bytes",
         "dask_worker_transfer_outgoing_count",
         "dask_worker_transfer_outgoing_count_total",
+        "dask_worker_tick_count_total",
+        "dask_worker_tick_duration_maximum_seconds",
     }
 
     try: