Repository: jupyterhub/jupyterhub

Issue:
Can we with a metric measure if jupyterhub is CPU constrained?
I've watched a few grafana dashboards, most recently [examples from here](https://github.com/jupyterhub/grafana-dashboards/pull/59) where hub response times can peak to seconds. Seeing that I got thinking that the hub pod from time to time get starved on CPU.

In practice though, it is perhaps rather that some responses are stuck waiting on something, like stopping a server, before it completes. So, it would be incorrect to draw the conclusion that the hub pod is overburdened by requests when we see long response times.

The question I raise is then instead, can we understand if jupyterhub is overburdened somehow using exposed prometheus metrics? If not, can we develop a new metric for this somehow?


Discussion:
I think the most useful metric here is 'tick duration' - how long does a single 'tick' of the event loop take? I know that when we were on the tornado ioloop @minrk has this be printed during debug, and it was *super* helpful. However when we moved to asyncio ioloop we lost this. A tick duration that's too long usually means there's too much CPU bound stuff going on that's not async enough, and causes responsiveness problems.

I don't know if we can measure that though.
> when we moved to asyncio ioloop we lost this

To be clear, it's _tornado_ that switched to asyncio.

asyncio has its own [debug mode](https://docs.python.org/3/library/asyncio-dev.html) that logs similar info. I don't think it quite includes tick duration, but it does note each _individual coroutine_ that blocks for more than 100ms (configurable threshold). Lacking the 'tick duration' metric means the asyncio debug log wouldn't capture the 'everything individually is async and cooperative and fast, but there's just too damn many coroutines right now' case.

 I'm not sure how we could hook it up to metrics. Maybe we could hook up the asyncio debug logger and count 'long task block' events? It would be hard to get labels on them, I think.
Looking into asyncio, we _could_ patch asyncio's `_run_once` as is done [in the asyncio test](https://github.com/python/cpython/blob/8a2b7ee64d1bde762438b458ea7fe88f054a3a88/Lib/test/test_asyncio/test_events.py#L1708-L1714) to add a timer for the tick, equivalent to what we had in tornado. This is a private API, but private APIs in the standard library are very stable, so can generally be safely written against for debugging purposes, as long as you gracefully handle their eventual removal, which should be easy enough for a metric like this.

Something like:

```python
loop = asyncio.get_running_loop()
original_run_once = loop._run_once
def _run_once(*a, **kw):
    tic = time.perf_counter()
    original_run_once(*a, **kw)
    toc = time.perf_counter()
    gauge_metric.set(toc-tic)
loop._run_once = _run_once
```

gated by `if hasattr(loop, '_run_once')`.

Profiling inner loops like this (or asyncio.set_debug) does have a nontrivial cost on the performance you are measuring, since they are called _a lot_, so should generally be opt-in.
I opened a feature request after digging some more into tornado history: https://github.com/python/cpython/issues/101946

We definitely _can_ implement this ourselves, with a small amount of private-api usage.

What would be the right metric to use? Histogram? It is an event that happens _a lot_.
![image](https://github.com/jupyterhub/jupyterhub/assets/3837114/841a1bc7-dd75-4f66-a697-76fd60e779b1)

I saw that the dask scheduler provides a dashboard, in it it lists the workers it controls, and they report `event_loop_interval`. I found code to emit that data [in dask/distributed 's worker.py](https://github.com/dask/distributed/blob/7d0ba9c402f435ce0cdeef5fae33173c59d17153/distributed/worker.py#L1074)
Nice, we could definitely do a similar metric.

It's a little tricky to wrap my head around [the metrics](https://github.com/dask/distributed/blob/7d0ba9c402f435ce0cdeef5fae33173c59d17153/distributed/core.py#L734-L759), but it _doesn't_ actually measure event loop ticks when things are responsive below a certain threshold, though it _can_ catch when ticks take _longer_ than that threshold.

In their case, the metric is an instant scalar metric for the _average_ event loop interval by counting the number of times a PeriodicCallback with a short interval is called over a longer interval. The default behavior is 20ms for the short interval and 1s for the collection.

Another way to represent the metric is a _histogram_ of tick intervals, which might better represent how often ticks are within different buckets, since it would capture more granularity. But would be a bit more costly to represent.

I've run this metric with an artificial blocking sleep of ~1 second every second, and it still reports an _average_ event loop interval of ~0.1 seconds, because it's still running 10-20 fast ticks for each slow one. It doesn't really capture that 50% of the time, it's blocking for 1 full second.

Maybe that's right for dask, but I'm not sure if the _average_ ("in general, how many ticks to we get in a second") is as useful for us than the Histogram ("how often is a tick more than X milliseconds; Y seconds; etc.").
and I realize you can use count/sum of a histogram to compute the average (over the prometheus collection interval, at least), so maybe just the histogram is ideal?

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -1665,6 +1665,8 @@ def default_url_fn(handler):
         config=True,
     )
 
+    metrics_collector = Any()
+
     def init_handlers(self):
         h = []
         # load handlers from the authenticator
@@ -3180,8 +3182,10 @@ async def finish_init_spawners():
                 await self.proxy.check_routes(self.users, self._service_map)
 
             asyncio.ensure_future(finish_init_spawners())
-        metrics_updater = PeriodicMetricsCollector(parent=self, db=self.db)
-        metrics_updater.start()
+        metrics_collector = self.metrics_collector = PeriodicMetricsCollector(
+            parent=self, db=self.db
+        )
+        metrics_collector.start()
 
     async def cleanup(self):
         """Shutdown managed services and various subprocesses. Cleanup runtime files."""
@@ -3617,6 +3621,8 @@ def stop(self):
             return
         if self.http_server:
             self.http_server.stop()
+        if self.metrics_collector:
+            self.metrics_collector.stop()
         self.io_loop.add_callback(self.shutdown_cancel_tasks)
 
     async def start_show_config(self):
diff --git a/jupyterhub/metrics.py b/jupyterhub/metrics.py
--- a/jupyterhub/metrics.py
+++ b/jupyterhub/metrics.py
@@ -23,12 +23,13 @@
 """
 
 import os
+import time
 from datetime import timedelta
 from enum import Enum
 
 from prometheus_client import Gauge, Histogram
 from tornado.ioloop import PeriodicCallback
-from traitlets import Any, Bool, Integer
+from traitlets import Any, Bool, Dict, Float, Integer
 from traitlets.config import LoggingConfigurable
 
 from . import orm
@@ -231,6 +232,37 @@ class ActiveUserPeriods(Enum):
     ACTIVE_USERS.labels(period=s.value)
 
 
+EVENT_LOOP_INTERVAL_SECONDS = Histogram(
+    'event_loop_interval_seconds',
+    'Distribution of measured event loop intervals',
+    namespace=metrics_prefix,
+    # Increase resolution to 5ms below 50ms
+    # because this is where we are most sensitive.
+    # No need to have buckets below 25, since we only measure every 20ms.
+    buckets=[
+        # 5ms from 25-50ms
+        25e-3,
+        30e-3,
+        35e-3,
+        40e-3,
+        45e-3,
+        50e-3,
+        # from here, default prometheus buckets
+        75e-3,
+        0.1,
+        0.25,
+        0.5,
+        0.75,
+        1,
+        2.5,
+        5,
+        7.5,
+        10,
+        float("inf"),
+    ],
+)
+
+
 def prometheus_log_method(handler):
     """
     Tornado log handler for recording RED metrics.
@@ -281,6 +313,42 @@ class PeriodicMetricsCollector(LoggingConfigurable):
         config=True,
     )
 
+    event_loop_interval_enabled = Bool(
+        True,
+        config=True,
+        help="""
+        Enable event_loop_interval_seconds metric.
+        
+        Measures event-loop responsiveness.
+        """,
+    )
+    event_loop_interval_resolution = Float(
+        0.02,
+        config=True,
+        help="""
+        Interval (in seconds) on which to measure the event loop interval.
+        
+        This is the _sensitivity_ of the event_loop_interval metric.
+        Setting it too low (e.g. below 20ms) can end up slowing down the whole event loop
+        by measuring too often,
+        while setting it too high (e.g. above a few seconds) may limit its resolution and usefulness.
+        The Prometheus Histogram populated by this metric
+        doesn't resolve differences below 25ms,
+        so setting this below ~20ms won't result in increased resolution of the histogram metric,
+        except for the average value, computed by:
+            event_loop_interval_seconds_sum / event_loop_interval_seconds_count
+        """,
+    )
+    event_loop_interval_log_threshold = Float(
+        1,
+        config=True,
+        help="""Log when the event loop blocks for at least this many seconds.""",
+    )
+
+    # internal state
+    _last_tick = Float()
+    _periodic_callbacks = Dict()
+
     db = Any(help="SQLAlchemy db session to use for performing queries")
 
     def update_active_users(self):
@@ -303,18 +371,50 @@ def update_active_users(self):
             self.log.info(f'Found {value} active users in the last {period}')
             ACTIVE_USERS.labels(period=period.value).set(value)
 
+    def _event_loop_tick(self):
+        """Measure a single tick of the event loop
+
+        This measures the time since the last tick
+        """
+        now = time.perf_counter()
+        tick_duration = now - self._last_tick
+        self._last_tick = now
+        EVENT_LOOP_INTERVAL_SECONDS.observe(tick_duration)
+        if tick_duration >= self.event_loop_interval_log_threshold:
+            # warn about slow ticks
+            self.log.warning("Event loop was unresponsive for %.2fs!", tick_duration)
+
     def start(self):
         """
         Start the periodic update process
         """
         if self.active_users_enabled:
             # Setup periodic refresh of the metric
-            pc = PeriodicCallback(
+            self._periodic_callbacks["active_users"] = PeriodicCallback(
                 self.update_active_users,
                 self.active_users_update_interval * 1000,
                 jitter=0.01,
             )
-            pc.start()
 
             # Update the metrics once on startup too
             self.update_active_users()
+
+        if self.event_loop_interval_enabled:
+            now = time.perf_counter()
+            self._last_tick = self._last_tick_collect = now
+            self._tick_durations = []
+            self._periodic_callbacks["event_loop_tick"] = PeriodicCallback(
+                self._event_loop_tick,
+                self.event_loop_interval_resolution * 1000,
+            )
+
+        # start callbacks
+        for pc in self._periodic_callbacks.values():
+            pc.start()
+
+    def stop(self):
+        """
+        Stop collecting metrics
+        """
+        for pc in self._periodic_callbacks.values():
+            pc.stop()


Test patch: