Repository: jupyterhub/jupyterhub

Issue:
Add jitter to poll checks
Some deployments use spawners that may consume resources to execute Spawner.poll() calls.  For instance, SlurmSpawner in [batchspawner](https://github.com/jupyterhub/batchspawner) makes a call to Slurm via [`squeue`](https://github.com/jupyterhub/batchspawner/blob/e89bf6a69ccfb9119b7066ec49df35694afb2c9e/batchspawner/batchspawner.py#L724) to get the status of notebook jobs.

If there are a large number of these jobs, and the Hub is redeployed, poll checks for all the notebook jobs synchronize, so every `poll_interval` seconds the Slurm controller gets a large number of status requests in a short amount of time, which Slurm does not like.  Over time the poll checks smear out, but this happens very slowly.

### Proposed change
<!-- Use this section to describe the feature you'd like to be added. -->

The smear-out could be accelerated if poll checks could be jittered.  I think the very first poll check when the Hub starts up maybe cannot/should not be jittered, but subsequent ones could be, by setting the `jitter` argument to the [`PeriodicCallback`](https://www.tornadoweb.org/en/stable/ioloop.html#tornado.ioloop.PeriodicCallback) setup in [`Spawner.start_polling`](https://github.com/jupyterhub/jupyterhub/blob/73b2307b367f8582e015fb6f8da5dd213523a943/jupyterhub/spawner.py#L1407).  By default this could be 0, but could be adjusted by adding a Spawner config parameter like `poll_jitter`.

### Alternative options
<!-- Use this section to describe alternative options and why you've decided on the proposed feature above. -->

* Another option would be to proxy all notebook server state through a reflector service like I think is used in KubeSpawner.  This would entail implementation changes to spawners that wanted to use that service.
* Don't allow "too many" servers to run in the first place (set an active_server_limit), or, more aggressively cull idle servers to cut down on the number running.

Adding jitter to the poll interval isn't the best solution maybe, but it is very simple to implement and we could maintain the current behavior by keeping the default jitter to 0 seconds. 

### Who would use this feature?
<!-- Describe who would benefit from using this feature. -->

Anybody with poll operations that consume any kind of resource in the course of execution.

### (Optional): Suggest a solution
<!-- Describe what you think needs to be done. Doing that is an excellent first step to get the feature implemented. -->

I'd add a `poll_jitter` parameter to Spawner and then set up `start_polling` to use it.  I thought for sure this conversation had happened in the past though but I can't find it in the issues (maybe issues where there are large numbers of servers running) but I might have missed it.  There may be reasons I haven't thought of that make this not a good idea. 


Discussion:
Jitter's a sensible and simple idea, if it would help.

For a more general solution, is it possible to use a shared background check for BatchSpawner.poll, like KubeSpawner, so you actually keep a local cache of _all_ running jobs up-to-date and `.poll()` only looks up in the local data structure? I wish our design made it easier to do this, but we don't have a configurable object that represents _all_ the spawners (e.g. SpawnManager if we want to go the Jupyter way). So KubeSpawner has to deal with [shared reflectors](https://github.com/jupyterhub/kubespawner/blob/812d309f2bf5cec7e107f2ea85883fc677289d16/kubespawner/spawner.py#L2352-L2373) created at the instance level but stored at the class level. A similar pattern could be followed for any Spawner where polling has a noticeable cost.
Right, a reflector service like KubeSpawner's is one of the alternatives I consider, but it requires implementation changes for each spawner class that has the same general issue.  It's worth pursuing, but in the meantime, enabling jitter mitigates poll synchronization for deployments where it may be an issue.  I'm just wondering if there were consequences I haven't considered with enabling jitter but it sounds like there aren't any that spring to your mind, so I'll try it out.

I think the shared reflector comment you made about KubeSpawner maybe addresses an issue I'd been thinking through for a reflector service for other spawners: A general issue there is also that it may introduce a kind of higher-level credential needed to check all users' servers that doesn't necessarily exist now.  Right now, if there's a credential per user used to execute a poll (say an ssh key, token, etc) we should really only use that to get the users' own state and not everybody else's.  This comment maybe doesn't make sense in the KubeSpawner context, since there's a token for talking to the k8s API, but for other spawners making a switch from direct polling to proxied polling it may came up.  Or maybe you meant something else.
> A general issue there is also that it may introduce a kind of higher-level credential needed to check all users' servers that doesn't necessarily exist now

Yeah, I was actually wondering if batchspawner had the necessary permissions to even do it.

I think adding jitter generally makes sense, and maybe also suggesting increasing the default poll interval on more costly implementations like BatchSpawner.
> I think adding jitter generally makes sense, and maybe also suggesting increasing the default poll interval on more costly implementations like BatchSpawner.

Right, we use 10 minutes, I've heard for even longer pollings.  Thanks!


Resolving patch:
diff --git a/jupyterhub/spawner.py b/jupyterhub/spawner.py
--- a/jupyterhub/spawner.py
+++ b/jupyterhub/spawner.py
@@ -490,6 +490,20 @@ async def _get_oauth_client_allowed_scopes(self):
         """,
     ).tag(config=True)
 
+    poll_jitter = Float(
+        0.1,
+        min=0,
+        max=1,
+        help="""
+        Jitter fraction for poll_interval.
+        
+        Avoids alignment of poll calls for many Spawners,
+        e.g. when restarting JupyterHub, which restarts all polls for running Spawners.
+
+        `poll_jitter=0` means no jitter, 0.1 means 10%, etc.
+        """,
+    ).tag(config=True)
+
     _callbacks = List()
     _poll_callback = Any()
 
@@ -1405,7 +1419,9 @@ def start_polling(self):
         self.stop_polling()
 
         self._poll_callback = PeriodicCallback(
-            self.poll_and_notify, 1e3 * self.poll_interval
+            self.poll_and_notify,
+            1e3 * self.poll_interval,
+            jitter=self.poll_jitter,
         )
         self._poll_callback.start()
 


Test patch: