Repository: jupyterhub/jupyterhub

Issue:
v5: Constant CPU usage of at least 1.9%
### Bug description

JupyterHub 5.0 consistently uses at least 1.9% CPU – even when *idle*.

#### Expected behaviour

JupyterHub 4.1 has a CPU usage of 0.05% when *idle*.

### Your personal set up

JupyterHub: [`glcr.b-data.ch/jupyterhub/jupyterhub:5.0-oauth-docker`](https://gitlab.b-data.ch/jupyterhub/jupyterhub/container_registry)



Discussion:
@minrk Do you have an explanation for this?
I do not, we'll need to do some profiling to figure out what's going on. Is JupyterHub the only upgraded package?
> I do not, we'll need to do some profiling to figure out what's going on. Is JupyterHub the only upgraded package?

This projects' [`jupyterhub/jupyterhub`](https://hub.docker.com/r/jupyterhub/jupyterhub) image plus `dockerspawner`, `jupyterhub-idle-culler`, `oauthenticator` and `psycopg2` installed on top.

👉 It is `/usr/bin/python3 /usr/local/bin/jupyterhub` that is consistently using at least 1.9% CPU.
To reproduce with this projects' docker image:

```bash
docker run --name jupyterhub-idle --rm -ti jupyterhub/jupyterhub
```

```bash
docker stats jupyterhub-idle
```

```
CONTAINER ID   NAME              CPU %     ...
c0398c58f7e1   jupyterhub-idle   1.92%     ...
```
Running JupyterHub under py-spy, but not connecting to it in a browser, uses around 3% CPU continuously on my system
`py-spy record -o profile.svg -- jupyterhub --JupyterHub.authenticator_class=dummy --JupyterHub.spawner_class=simple --Authenticator.admin_users=test`:
![profile.svg](https://github.com/jupyterhub/jupyterhub/assets/1644105/97cad099-908d-4c83-999c-f2b00ae7288a)

If I disable `PeriodicMetricsCollector`:
```diff
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
index 8bc5b937..8295bd48 100644
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -3414,7 +3414,7 @@ class JupyterHub(Application):
         metrics_collector = self.metrics_collector = PeriodicMetricsCollector(
             parent=self, db=self.db
         )
-        metrics_collector.start()
+        # metrics_collector.start()
 
     async def cleanup(self):
         """Shutdown managed services and various subprocesses. Cleanup runtime files."""
```
CPU usage is <1%
https://github.com/jupyterhub/jupyterhub/pull/4615
Increasing `PeriodicMetricsCollector.event_loop_interval_resolution` from the default 0.02 to a larger number brings down the CPU usage, but obviously makes the metric less useful.
IMHO there should be an option to disable metrics (default: enabled).
>IMHO there should be an option to disable metrics (default: enabled).

Definitely! There is exactly that:

```python
c.PeriodicMetricsCollector.event_loop_interval_enabled = False
```

but the default behavior shouldn't be this expensive, we should explore that. Possibly making the metric more efficient somehow, or if there isn't something particularly inefficient there, just make the default sensitivity longer, e.g. 100ms, 200ms or even 500ms.

FWIW, with this metric disabled, an idle JupyterHub uses < 0.1% CPU. On my machine with an idle JupyterHub doing nothing else:

| interval | % CPU |
| --- | --- |
| disabled | < 0.1% |
| .02 | ~2% |
| .05 | ~1% |
| 0.1 | ~0.7% |
| 0.2 | ~0.5% |
| 0.5 | ~0.1% |

So I think the default interval probably shouldn't be lower than 100ms, unless there's some inefficiency we can find.
I think a simpler measurer that is just

```python
while True:
    tick
    await asyncio.sleep(interval)
    tock
```

will be a bit more efficient than PeriodicCallback, which involves a number of function calls, which is a relevant fraction on this timescale.
> > IMHO there should be an option to disable metrics (default: enabled).
> 
> Definitely! There is exactly that:
> 
> ```python
> c.PeriodicMetricsCollector.event_loop_interval_enabled = False
> ```

That is what I was looking for. I could not find it in the documentation...
Now, the CPU usage of v5.0.0 is the same as v4.1.5:

```
CONTAINER ID   NAME              CPU %     ...
abf075e95af2   jupyterhub-idle   0.05%     ...
```

Resolving patch:
diff --git a/jupyterhub/metrics.py b/jupyterhub/metrics.py
--- a/jupyterhub/metrics.py
+++ b/jupyterhub/metrics.py
@@ -22,6 +22,7 @@
     added ``jupyterhub_`` prefix to metric names.
 """
 
+import asyncio
 import os
 import time
 from datetime import timedelta
@@ -236,17 +237,17 @@ class ActiveUserPeriods(Enum):
     'event_loop_interval_seconds',
     'Distribution of measured event loop intervals',
     namespace=metrics_prefix,
-    # Increase resolution to 5ms below 50ms
+    # don't measure below 50ms, our default
+    # Increase resolution to 5ms below 75ms
     # because this is where we are most sensitive.
-    # No need to have buckets below 25, since we only measure every 20ms.
+    # No need to have buckets below 50, since we only measure every 50ms.
     buckets=[
-        # 5ms from 25-50ms
-        25e-3,
-        30e-3,
-        35e-3,
-        40e-3,
-        45e-3,
+        # 5ms from 50-75ms
         50e-3,
+        55e-3,
+        60e-3,
+        65e-3,
+        70e-3,
         # from here, default prometheus buckets
         75e-3,
         0.1,
@@ -323,19 +324,20 @@ class PeriodicMetricsCollector(LoggingConfigurable):
         """,
     )
     event_loop_interval_resolution = Float(
-        0.02,
+        0.05,
         config=True,
         help="""
         Interval (in seconds) on which to measure the event loop interval.
         
-        This is the _sensitivity_ of the event_loop_interval metric.
+        This is the _sensitivity_ of the `event_loop_interval` metric.
         Setting it too low (e.g. below 20ms) can end up slowing down the whole event loop
         by measuring too often,
         while setting it too high (e.g. above a few seconds) may limit its resolution and usefulness.
         The Prometheus Histogram populated by this metric
         doesn't resolve differences below 25ms,
         so setting this below ~20ms won't result in increased resolution of the histogram metric,
-        except for the average value, computed by:
+        except for the average value, computed by::
+
             event_loop_interval_seconds_sum / event_loop_interval_seconds_count
         """,
     )
@@ -346,7 +348,7 @@ class PeriodicMetricsCollector(LoggingConfigurable):
     )
 
     # internal state
-    _last_tick = Float()
+    _tasks = Dict()
     _periodic_callbacks = Dict()
 
     db = Any(help="SQLAlchemy db session to use for performing queries")
@@ -371,18 +373,39 @@ def update_active_users(self):
             self.log.info(f'Found {value} active users in the last {period}')
             ACTIVE_USERS.labels(period=period.value).set(value)
 
-    def _event_loop_tick(self):
-        """Measure a single tick of the event loop
+    async def _measure_event_loop_interval(self):
+        """Measure the event loop responsiveness
 
-        This measures the time since the last tick
+        A single long-running coroutine because PeriodicCallback is too expensive
+        to measure small intervals.
         """
-        now = time.perf_counter()
-        tick_duration = now - self._last_tick
-        self._last_tick = now
-        EVENT_LOOP_INTERVAL_SECONDS.observe(tick_duration)
-        if tick_duration >= self.event_loop_interval_log_threshold:
-            # warn about slow ticks
-            self.log.warning("Event loop was unresponsive for %.2fs!", tick_duration)
+        tick = time.perf_counter
+
+        last_tick = tick()
+        resolution = self.event_loop_interval_resolution
+        lower_bound = 2 * resolution
+        # This loop runs _very_ often, so try to keep it efficient.
+        # Even excess comparisons and assignments have a measurable effect on overall cpu usage.
+        while True:
+            await asyncio.sleep(resolution)
+            now = tick()
+            # measure the _difference_ between the sleep time and the measured time
+            # the event loop blocked for somewhere in the range [delay, delay + resolution]
+            tick_duration = now - last_tick
+            last_tick = now
+            if tick_duration < lower_bound:
+                # don't report numbers less than measurement resolution,
+                # we don't really have that information
+                delay = resolution
+            else:
+                delay = tick_duration - resolution
+                if delay >= self.event_loop_interval_log_threshold:
+                    # warn about slow ticks
+                    self.log.warning(
+                        "Event loop was unresponsive for at least %.2fs!", delay
+                    )
+
+            EVENT_LOOP_INTERVAL_SECONDS.observe(delay)
 
     def start(self):
         """
@@ -400,12 +423,8 @@ def start(self):
             self.update_active_users()
 
         if self.event_loop_interval_enabled:
-            now = time.perf_counter()
-            self._last_tick = self._last_tick_collect = now
-            self._tick_durations = []
-            self._periodic_callbacks["event_loop_tick"] = PeriodicCallback(
-                self._event_loop_tick,
-                self.event_loop_interval_resolution * 1000,
+            self._tasks["event_loop_tick"] = asyncio.create_task(
+                self._measure_event_loop_interval()
             )
 
         # start callbacks
@@ -418,3 +437,5 @@ def stop(self):
         """
         for pc in self._periodic_callbacks.values():
             pc.stop()
+        for task in self._tasks.values():
+            task.cancel()


Test patch: