Repository: jupyterhub/jupyterhub

Issue:
hub->proxy web-request timebombs - how to not explode?
## Short version based on findings

I've been working on the Z2JH projects CI system a lot recently, which often fails seemingly at random. I figured it out I think.

proxy.py's function `api_request` used to communicate with the proxy can _fail hard_ and crash the whole jupyterhub process. This often happens even though both the proxy and the hub pod in z2jh has become ready in the kubernetes sense, which they only become after they have started listening to web endpoints.

The hub pod has a function (`check_routes`) that is run on startup and is scheduled to run regularly after this, but it is done in an async manner. This leaves the hub able to start operating even before it has ever run a `check_routes` call to completion. The `check_routes` function makes calls to the proxy api, but if for example, the proxy is unavailable somehow, we can end up with a timebomb because the hub will have a async webrequests that in 20 seconds will timeout and fail hard within the `api_request` function in proxy.py.

I consider some action points in https://github.com/jupyterhub/jupyterhub/issues/2745#issuecomment-535190293.

---

## Long version from the start

I'm debugging something in depth, and I believe that that the z2jh CI pipeline is unstable because of a unexpected timing.

From the code block below, the hub pod can and do crash in the z2jh CI pipeline, and I can't figure out why, but end up with some interesting observation I hope someone can shed light on.

https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L758-L774

Looking at the crash logs below. We can see that this code block started running, but only five seconds later it errors because of a 20 second timeout on a request. I don't get that. Is it reusing some connection? What is going on here? I figure 

```
    [I 2019-09-25 12:03:01.117 JupyterHub app:2337] Hub API listening on http://0.0.0.0:8081/hub/

    ### NOTE: The hub started up 20 seconds earlier...

    [I 2019-09-25 12:03:16.796 JupyterHub proxy:281] Removing user testuser-c947c528-06d0-4d0f-be5e-ceb6f1c32dfc from proxy (/user/testuser-c947c528-06d0-4d0f-be5e-ceb6f1c32dfc/)
    ## NOTE: The log line below is generated directly from the code above.
    ##       This is just before making the delete request.
    [D 2019-09-25 12:03:16.796 JupyterHub proxy:765] Proxy: Fetching DELETE http://10.99.19.102:8001/api/routes/user/testuser-c947c528-06d0-4d0f-be5e-ceb6f1c32dfc
    [D 2019-09-25 12:03:16.799 JupyterHub user:724] Stopping testuser-c947c528-06d0-4d0f-be5e-ceb6f1c32dfc
    [I 2019-09-25 12:03:16.800 JupyterHub spawner:1758] Deleting pod jupyter-testuser-2dc947c528-2d06d0-2d4d0f-2dbe5e-2dceb6f1c32dfc
    [E 2019-09-25 12:03:21.132 JupyterHub app:2482]
        Traceback (most recent call last):
        File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 2480, in launch_instance_async
            await self.start()
        File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 2405, in start
            await self.proxy.check_routes(self.users, self._service_map)
        File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 62, in locked_method
            return await method(*args, **kwargs)
        File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 315, in check_routes
            routes = await self.get_all_routes()
        File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 804, in get_all_routes
            resp = await self.api_request('', client=client)
        File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 773, in api_request
            result = await client.fetch(req)
        tornado.curl_httpclient.CurlError: HTTP 599: Connection timed out after 20000 milliseconds
```


## Relevant JupyterHub configuration
Z2JH configures this [in its jupyterhub_config.py](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/d25bcd9e38aca2734f90e51c2bc9a0bf9d90ca3e/images/hub/jupyterhub_config.py#L5-L15) which I think is relevant. The error was after all a `tornado.curl_httpclient.CurlError` error.

```python
from tornado.httpclient import AsyncHTTPClient
from kubernetes import client
from jupyterhub.utils import url_path_join

from z2jh import get_config, set_config_if_not_none

# Configure JupyterHub to use the curl backend for making HTTP requests,
# rather than the pure-python implementations. The default one starts
# being too slow to make a large number of requests to the proxy API
# at the rate required.
AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
```

## Related reference
- https://www.tornadoweb.org/en/stable/httpclient.html#tornado.httpclient.AsyncHTTPClient.configure
- [`request_timeout`](https://www.tornadoweb.org/en/stable/httpclient.html#request-objects) is the configuration defaulting to 20 which makes the request timeout after 20 seconds (20000 milliseconds).
- For a full view of the logs I've been dissecting, see [this TravisCI job's logs](https://travis-ci.org/jupyterhub/zero-to-jupyterhub-k8s/jobs/589410196).

## Background context
Z2JH CI runs a pytest to spawn a user called `test_api_request_user_spawn`, and this fails often for the reason above. The events leading up to this are described below. Currently, this test fails about 50% of the time in my development branch where I use a `kind` k8s cluster instead of a `minikube` k8s cluster.

- The `jupyterhub_user` pytest fixture used by the test function is instantiated and triggers the creation of a JupyterHub user through the JupyterHub API
- `test_api_request_user_spawn` use the JupyterHub API to spawn a user
- `test_api_request_user_spawn` calls `_wait_for_user_to_spawn` that probes the JupyterHub API for a registered server until it finds one
- `test_api_request_user_spawn` cleans up by calling `_delete_server` which calls the JupyterHub API to stop the user server
  - The Hub runs and crashes like described above
- pytest conclude `test_api_request_user_spawn` failed, the function scope is closing down which causes the `jupyterhub_user` fixture finish up by calling the JupyterHub API to delete the user, but JupyterHub has crashed so this function errors.


Discussion:
## My hypothesis

> You need to either increase request_timeout to the amount of time you're willing to wait for the response (including time spent waiting in the queue), or maintain your own queue and feed urls into AsyncHTTPClient gradually (the queue and semaphore classes that are being introduced in the upcoming Tornado 4.2 can help here; until then you can use Toro: http://toro.readthedocs.org/en/stable/examples/web_spider_example.html)
>
> From this very old comment: https://github.com/tornadoweb/tornado/issues/1400#issuecomment-87905311

Combining this knowledge with the fact I've seen many of these failures show up like clockwork 20 seconds after the hub starts makes me quite confident that:
- we have web requests sent from the hub that fails hard, causing the hub to crash, and the requests to the hub from the proxy is returning 503
- the web requests fails because they timeout just like its said, but they timeout mainly because of them being stuck in a queue as indicated by the comment above

With this insight, inspecting the code base for related things led me to find this which strengthens my hypothesis:

https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L423-L444

The only place self.semaphore is actually used for its purpose, is here!

https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L766-L774

Looking back now at the error, I realize the trace is to the `check_routes` function! At some point there was a request made from `check_routes` that never completed, timed out, and brought down the hub in a delayed manner.

## Conclusion
The hub crashes if a single api_request to the proxy times out.
https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L773

It seems very extreme to me that the hub will crash on a single api request timeout. We could and should make this more robust I think, I'm not confident on how to go about it though, hmm...

## Manual stacktrace
__launch_instance_async__
https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/app.py#L2652-L2658
__start__
https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/app.py#L2553
**locked_method / @_one_at_a_time decorator**
https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L51-L64
__check_routes__
https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L309-L315
__get_all_routes__
https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L802-L814
https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/proxy.py#L758-L774
## The Z2JH CI test failures analyzed
The hub pod successfully becomes ready, which is a criteria we have waited for, the hub pod also has a readynessProbe that only makes it become ready when it has started being able to listen to API requests. But, it is a ticking time bomb if it has an initial request to check_routes that got lost somewhere during the simultaneous proxy pod startup.

When the tests initialize before the hub has crashed, but is has a ticking bomb, they will fail to complete in time before the hub explodes.

## Considered action points
- Should we make api_requests not fail so hard as to kill the whole process?
- Should we make check_routes use of api_requests not fail so hard as to kill the whole process?
- Should we lower the connection timeout to explode quicker in general or for certain requests?
- Does it make sense to allow the hub to start working before the initial check_routes finishes? Should we make something in the async startup more sync with regards to waiting for the initial check_routes?
- I think we should make the z2jh hub pods readinessProbe to wait for the initial check_routes to finish, perhaps also for `launch_instance_async` to finish that wraps the initial check_routes call. Currently we await the hubs ability to listen to web request by probing the `/health` endpoint.
  - Is there currently a way for a web request and/or shell script to decide if the hub has finished its startup? Should we implement a way? See [related code](https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/app.py#L2574-L2576).

## Temporary mitigation considerations
- A temporary workaround in z2jh is to increase the hub's readinessProbe duration to be marginally longer than 20 seconds to make sure any initial time bomb explode before it is considered ready.

I think your guidance is required here @minrk! :sunflower:
This might explain why I had to add this 1 minute delay even after the hub was accessible:
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/d25bcd9e38aca2734f90e51c2bc9a0bf9d90ca3e/ci/test.sh#L50-L57
## Verification of hypothesis
I added a delay for the hub pod to become k8s ready of 25 seconds. I then rerun a pipeline of four tests for various k8s version. These have always had at least one failure among them before, but now they didn't! The test suite did not startup if there was a timebomb ticking, it only waited for the hub and proxy to become ready and that was it!

__This hub exploded and would have failed the test suite previously__
![image](https://user-images.githubusercontent.com/3837114/65663205-6e022180-e036-11e9-887a-84d58aa31f47.png)

__This hub didn't explode at all and previous test would succeeded__
![image](https://user-images.githubusercontent.com/3837114/65663344-c76a5080-e036-11e9-97c7-1010159b2e50.png)

Note the difference in time for the first run to complete, it is about 20 seconds, because in one test the hub pod had to restart again after about 20 seconds.
I think it makes sense for the hub startup to crash if it can't reach the proxy. It should also not start serving requests before being done with startup. This way the hub will crash and get restarted which is a simple way to retry the "talk to the proxy" action. If there is stuff the hub could be doing before it can talk to the proxy then we should reconsider the "crash as a way to retry" idea.

I took a look at the code you linked to in `app.py`, I think [this code](https://github.com/jupyterhub/jupyterhub/blob/5b13f96162b77b4ae23306327cd700b8faeb299c/jupyterhub/app.py#L2289) should be `await asyncio.ensure_future()`. This is a bit of a hunch/feeling more than a concrete "this is definitely wrong as is" so needs trying out :-/

Can you easily change that and re-run the CI tests to see if that helps?

---

In general having code like `await a(); await b()` means `b()` won't be called until `a()` has returned. So despite them being "async" they run in order. However it is easy to write `a(); await b()` and not notice (when it isn't a simple example where the code is close to the other code). The result is that both `a()` and `b()` will be called and complete but they won't be ordered any more. If your process only runs for a very short time you might notice this because `a()` hasn't started/finished running by the time the process runs but if you have a long running (several seconds) process it is easy to not notice (I've done that a few times) because `a()` will eventually be called and complete.
I created a related issue: https://github.com/jupyterhub/jupyterhub/issues/2749

@betatim I cannot change that so easy and rerun CI-tests without configuring to use a custom JupyterHub build, which could be done.

I understand the outcome of what it would lead to quite well though, it would block the `initialize` function from returning no matter what, which would lead to `start` not being called even after 10 seconds, which would fail to start and/or configure the proxy through a call to `check_routes`, which would ensure that the hub would fail early to start. But, it would also delete all active userpods as the probes to the users would always have time to fail before the proxy would get configured, which is required for the user pods lifeprobe sent by init_spawners would succeed.

This is complicated =/
I believe the Hub *should* fail to start if it cannot talk to the proxy (in the systemd sense, jupyterhub 'requires' the proxy service). So if the *first* check_routes attempt fails, jupyterhub should rightly crash. The reasoning behind this:

- the hub's principle task is managing the consistency of the proxy
- if it cannot get the initial state of the proxy, it cannot do its main task and cannot function until this initialization succeeds

This is resolved over time in that the Hub will restart if the proxy hasn't started yet, and once the proxy is ready the Hub will start fine.

So the immediate problem is:

- the Hub accepts http connections before the initial check_routes completes
- we use basic HTTP responsiveness as a 'ready' signal
- the hub is not ready until it can talk to the proxy
- which means: responding to http requests doesn't accurately communicate that the Hub is actually ready

One solution to the immediate problem is #2750 - starting the hub http server after the initial check_routes has completed. This will delay the Hub's ready signal by delaying the actual listening of the http server until it's actually ready.

Another solution would be a deeper readiness check that returns actual information that the Hub is ready, which can be after it starts accepting connections.

Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -2452,6 +2452,20 @@ async def start(self):
             loop.stop()
             return
 
+        # start the proxy
+        if self.proxy.should_start:
+            try:
+                await self.proxy.start()
+            except Exception as e:
+                self.log.critical("Failed to start proxy", exc_info=True)
+                self.exit(1)
+        else:
+            self.log.info("Not starting proxy")
+
+        # verify that we can talk to the proxy before listening.
+        # avoids delayed failure if we can't talk to the proxy
+        await self.proxy.get_all_routes()
+
         ssl_context = make_ssl_context(
             self.internal_ssl_key,
             self.internal_ssl_cert,
@@ -2489,16 +2503,6 @@ async def start(self):
             self.log.error("Failed to bind hub to %s", self.hub.bind_url)
             raise
 
-        # start the proxy
-        if self.proxy.should_start:
-            try:
-                await self.proxy.start()
-            except Exception as e:
-                self.log.critical("Failed to start proxy", exc_info=True)
-                self.exit(1)
-        else:
-            self.log.info("Not starting proxy")
-
         # start the service(s)
         for service_name, service in self._service_map.items():
             msg = (


Test patch: