Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Fix jupyterhub-idle-culler's permissions - our attempted implementation has failed
I think we have failed to provide the permissions properly to jupyterhub-idle-culler somehow.

I observe the following in the hub logs after #2395 has merged. I figure this error can show if:
- jupyterhub/jupyterhub-idle-culler permission failure? No.
  The idle culler could be making a request to /hub/api/info without passing its token. That's [not it](https://github.com/jupyterhub/jupyterhub-idle-culler/blob/ae59078ccf3f929702d9833b6b9e644dae0e191d/jupyterhub_idle_culler/__init__.py#L160-L164).
- Helm chart config failure.
  - We may have failed to pass permissions to the managed JupyterHub service.
    - Is the managed service coupled with a role granting it more permissions? Hmmm?
      I think it should say something like the following if this is done, but I don't think it is.
      ```
      [I 2021-09-26 01:03:16.379 JupyterHub roles:451] Added role jupyterhub-idle-culler to token <APIToken('72e4...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
      ```
    - Is the role granting the service enough permissions? Yes.
- jupyterhub/jupyterhub permission failure?
  JupyterHub itself may fail somehow, for example with some race condition?

```
[D 2021-09-26 01:03:16.299 JupyterHub app:1987] Loading default roles to database
[I 2021-09-26 01:03:16.337 JupyterHub roles:304] Role jupyterhub-idle-culler added to database
[I 2021-09-26 01:03:16.338 JupyterHub roles:304] Role test-role-1 added to database
[I 2021-09-26 01:03:16.352 JupyterHub roles:304] Role test-role-2 added to database
[I 2021-09-26 01:03:16.353 JupyterHub app:1893] Not using allowed_users. Any authenticated user will be allowed.
[D 2021-09-26 01:03:16.374 JupyterHub roles:447] Assigning default roles to tokens
[I 2021-09-26 01:03:16.379 JupyterHub roles:451] Added role token to token <APIToken('72e4...', service='jupyterhub-idle-culler', client_id='jupyterhub')>

...

[I 2021-09-26 01:03:27.347 JupyterHub app:3076] JupyterHub is now running at http://:8000
[D 2021-09-26 01:03:27.348 JupyterHub app:2685] It took 11.975 seconds for the Hub to start
[D 2021-09-26 01:03:27.538 JupyterHub base:272] Recording first activity for <APIToken('72e4...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[W 2021-09-26 01:03:27.546 JupyterHub scopes:291] Authenticated with token <APIToken('72e4...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[D 2021-09-26 01:03:27.548 JupyterHub scopes:481] Checking access via scope read:hub
[D 2021-09-26 01:03:27.548 JupyterHub scopes:376] No access to /hub/api/info via read:hub
[W 2021-09-26 01:03:27.548 JupyterHub scopes:489] Not authorizing access to /hub/api/info. Requires any of [read:hub], not derived from scopes [read:users:activity, read:servers, read:users:name, list:users, servers]
[W 2021-09-26 01:03:27.548 JupyterHub web:1787] 403 GET /hub/api/info (127.0.0.1): Action is not authorized with current scopes; requires any of [read:hub]
[E 210926 01:03:27 ioloop:761] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fbb659edaf0>>, <Task finished name='Task-1' coro=<cull_idle() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py:73> exception=HTTP 403: Forbidden>)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 741, in _run_callback
        ret = callback()
      File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 765, in _discard_future_result
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 164, in cull_idle
        resp = await fetch(HTTPRequest(url=url + "/info", headers=auth_header))
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 117, in fetch
        return await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[W 2021-09-26 01:03:27.550 JupyterHub log:189] 403 GET /hub/api/info (jupyterhub-idle-culler@127.0.0.1) 19.92ms
```


Discussion:
I need to add `read:hub` explicitly for jupyterhub-idle-culler it seems.

```
[D 2021-10-12 11:15:00.969 JupyterHub base:272] Recording first activity for <APIToken('fe83...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[W 2021-10-12 11:15:00.973 JupyterHub scopes:298] Authenticated with token <APIToken('fe83...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[D 2021-10-12 11:15:00.975 JupyterHub scopes:488] Checking access via scope read:hub
[D 2021-10-12 11:15:00.975 JupyterHub scopes:383] No access to /hub/api/info via read:hub
[W 2021-10-12 11:15:00.975 JupyterHub scopes:496] Not authorizing access to /hub/api/info. Requires any of [read:hub], not derived from scopes [list:users, delete:servers, read:users:name, servers, read:users:activity, read:servers]
[W 2021-10-12 11:15:00.975 JupyterHub web:1787] 403 GET /hub/api/info (127.0.0.1): Action is not authorized with current scopes; requires any of [read:hub]
[E 211012 11:15:00 ioloop:761] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fd22e157af0>>, <Task finished name='Task-1' coro=<cull_idle() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py:73> exception=HTTP 403: Forbidden>)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 741, in _run_callback
        ret = callback()
      File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 765, in _discard_future_result
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 164, in cull_idle
        resp = await fetch(HTTPRequest(url=url + "/info", headers=auth_header))
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub_idle_culler/__init__.py", line 117, in fetch
        return await client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[W 2021-10-12 11:15:00.977 JupyterHub log:189] 403 GET /hub/api/info (jupyterhub-idle-culler@127.0.0.1) 11.63ms
```

Resolving patch:
diff --git a/jupyterhub/files/hub/jupyterhub_config.py b/jupyterhub/files/hub/jupyterhub_config.py
--- a/jupyterhub/files/hub/jupyterhub_config.py
+++ b/jupyterhub/files/hub/jupyterhub_config.py
@@ -332,9 +332,10 @@ def camelCaseify(s):
     jupyterhub_idle_culler_role = {
         "name": "jupyterhub-idle-culler",
         "scopes": [
+            "read:hub",
             "list:users",
             "read:users:activity",
-            "servers",
+            "delete:servers",
             # "admin:users", # dynamically added if --cull-users is passed
         ],
         # assign the role to a jupyterhub service, so it gains these permissions


Test patch: