Repository: jupyterhub/jupyterhub

Issue:
Unreliable attempt to add a route for a new user to an external CHP proxy - HTTP 599 Broken pipe (JupyterHub 1.2.0b1 issue)
## Issue background
This issue is migrated from Z2JH where the CI system started having intermittent failures to spawn user pods all the way. The failure wasn't the creation of pods, but when the `hub` pod was speaking with the externally running `proxy` pod (CHP server) and getting `tornado.curlhttpclient.CurlError: HTTP 599: Send failure: Broken pipe` as part of its POST request to add a route for the newly spawned user.

## Logs inspection
The hub errors when speaking with the external CHP proxy pod after the a user pod has successfully been spawned.

```
[I 2020-10-20 16:48:25.540 JupyterHub base:893] User testuser-aa6aa5ea-64bb-4719-93c7-04557f25c3d5 took 2.179 seconds to start
[I 2020-10-20 16:48:25.540 JupyterHub proxy:257] Adding user testuser-aa6aa5ea-64bb-4719-93c7-04557f25c3d5 to proxy /user/testuser-aa6aa5ea-64bb-4719-93c7-04557f25c3d5/ => http://192.168.79.75:8888
[D 2020-10-20 16:48:25.540 JupyterHub proxy:765] Proxy: Fetching POST http://proxy-api:8001/api/routes/user/testuser-aa6aa5ea-64bb-4719-93c7-04557f25c3d5
[I 2020-10-20 16:48:25.541 JupyterHub log:181] 200 GET /hub/api (@10.20.0.120) 1.70ms
[E 2020-10-20 16:48:25.542 JupyterHub base:915] Failed to add testuser-aa6aa5ea-64bb-4719-93c7-04557f25c3d5 to proxy!
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 908, in finish_user_spawn
        await self.proxy.add_user(user, server_name)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 270, in add_user
        await self.add_route(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 781, in add_route
        await self.api_request(path, method='POST', body=body)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 773, in api_request
        result = await client.fetch(req)
    tornado.curl_httpclient.CurlError: HTTP 599: Send failure: Broken pipe
```

If everything were as it should, the proxy should add a network route towards the newly added pod - it is a configurable proxy that JupyterHub controls after all. This is the kind of entry we should see twice because we have two tests adding a pod.

```
16:48:18.515 [ConfigProxy] info: Adding route /user/testuser-c2521e10-e843-40c3-8cb6-dac422b1433a -> http://192.168.79.74:8888
```

It seems like this never happens, and that the proxy pod / CHP doesn't notice any incoming request to add a route, because it doesn't log any incoming POST request either.

## Reference

#### Build failures
[Build failure with logs of hub and proxy pod](https://travis-ci.org/github/jupyterhub/zero-to-jupyterhub-k8s/jobs/737443531) but not for the autohttps pod.
[Build failure with logs of the hub, proxy, and autohttps pod](https://travis-ci.org/github/jupyterhub/zero-to-jupyterhub-k8s/jobs/738037421), this is one of two failures from [the same travis build](https://travis-ci.org/github/jupyterhub/zero-to-jupyterhub-k8s/builds/738037416)!

#### Code that fail to reach return statement
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/2f7135cfeed631553eb3452f6b3f47f27e47c9f9/tests/test_spawn.py#L217-L224

Example user_model data in code linked above
```python
# example user_model that is seen during failures
user_model = {'kind': 'user', 'name': 'testuser-f8ff861a-538a-4c93-a864-ae3345cea907', 'admin': False, 'groups': [], 'server': None, 'pending': None, 'created': '2020-10-20T14:28:03.524748Z', 'last_activity': '2020-10-20T14:28:11.990951Z', 'servers': {}, 'auth_state': None}
```


Discussion:
## Conclusions so far
- The autohttps pod is irrelevant to consider, what fails is the hub pod's request directly to the proxy pod's API without going through the autohttps pod.
- The broken pipe error typically happen when starting a pod as part of the netpol test, which is the second time a pod is started and gets it route added. I have also seen it occur on the first user pod though, so I think that we observe it mostly as part of the netpol test relates to a timing - that it happens a bit later.

My main suspicion is that the hub pod has some kind of connection which is timing out after a while, which at the time cause the broken pipe error. I have some ideas on what can be done about this.

## Solution ideas
1. We make the hub->proxy connection more robust somehow - I lack knowledge about how still but perhaps something about sending keep alive messages?
2. We make the hub->proxy add route operation more robustly - for example by attempting again on 599 failure.
3. We realize the root cause and fix it, I'm not sure how =/

In other words, we need to fix something in jupyterhub to avoid the issue surfacing in our CI tests of Z2JH I think. It is worth noting that I think these kinds of errors may have started to show since we bumped to JupyterHub 1.2.0b1, but I'm not 100% on that.

## Other leads
- __Did a change with tornado cause this?__ I don't think so.
  The Hub pod is using tornado's curl_httpclient, perhaps this is a Tornado issue? We have been using tornado 6.0.4 in 6 months though, and I think this issue has surfaced recently. Tornado 6.0.4 is the most recent version. https://github.com/tornadoweb/tornado/issues/2570#issuecomment-459989339
- __Did a change with CHP (proxy pod), which is running http-proxy, cause this?__ I don't think so.
  The proxy pod is running https://github.com/jupyterhub/configurable-http-proxy, which in turn runs a `http-proxy`. Perhaps there was a change there? http-proxy was updated 5 month ago: https://github.com/http-party/node-http-proxy/tags, and bumped to [use that version 5 months ago](https://github.com/jupyterhub/configurable-http-proxy/blame/master/package-lock.json#L932), but z2jh has kept using 1.18.0 version as of 12 months as we have not cut a CHP release since CHP bumped http-proxy to 1.18.1.
## Inspection of JupyterHub's code

I could not find any obvious change to how calls to the external proxy are made, which can be found here. Changes to this are all ~3 years old.

https://github.com/jupyterhub/jupyterhub/blob/082f6516a15de1c74657a57b5f9fee5a082c0600/jupyterhub/proxy.py#L758-L781

@minrk @GeorgianaElena do you have ideas or feedback for example on my solution ideas? The hub pod doesn't crash, it keeps running, but the user-server spawn fail as the hub deletes the pod when it fails to add the route due to the "HTTP 599, send failure, Broken pipe" errors.

So, can we make the connection between the hub->proxy more robust, to not get the error, and/or can we add some logic to recover if we get the error? Any suggestions on ways forward?
@consideRatio I think you're right that network policy blocking and/or dropping connections (maybe existing connections are dropped as netpol is loaded, even if they are allowed, or some similar startup behavior).

This is likely the most frustrating kind of error, where it may come up often on CI due to everything being started at the same time, but ~never in reality. It costs a lot of time to debug and fix a bug that doesn't actually affect users of the software!

I think the fix is to retry api requests on connection errors with exponential_backoff, like we do many other places. That will fix the cases where this could occur (e.g. proxy pod restarting) even if the hypothesis of network policy setup is wrong, so it's the right thing to do in any case. It has been a low priority because I don't *expect* this comes up ~ever in real deployments, a situation where CI is not very much like production.
@minrk another related error which has been around for very long time, is that when the hub startup alongside the proxy pod, this can quite often happen after 20 seconds.

```
[E 2020-10-23 17:18:44.720 JupyterHub app:2843]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2841, in launch_instance_async
        await self.start()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2614, in start
        await self.proxy.get_all_routes()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 805, in get_all_routes
        resp = await self.api_request('', client=client)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 773, in api_request
        result = await client.fetch(req)
    tornado.curl_httpclient.CurlError: HTTP 599: Connection timed out after 20001 milliseconds
    
[D 2020-10-23 17:18:44.721 JupyterHub application:834] Exiting application: jupyterhub
```

Which makes me see very often that the hub has restarted once and then properly function after that.

```
$ kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
continuous-image-puller-gb66b              1/1     Running   0          3h
hub-796d4d88c-wxjcm                        1/1     Running   1          3h
```

It is the same culprit: proxy.py's `api_request` function!

---

I suggest that we lower the timeout significantly from 20s to more like 1s so that we don't get stuck waiting for something doomed to fail for too long.

Resolving patch:
diff --git a/jupyterhub/proxy.py b/jupyterhub/proxy.py
--- a/jupyterhub/proxy.py
+++ b/jupyterhub/proxy.py
@@ -43,6 +43,7 @@
 from .metrics import CHECK_ROUTES_DURATION_SECONDS
 from .metrics import PROXY_POLL_DURATION_SECONDS
 from .objects import Server
+from .utils import exponential_backoff
 from .utils import make_ssl_context
 from .utils import url_path_join
 from jupyterhub.traitlets import Command
@@ -768,10 +769,36 @@ async def api_request(self, path, method='GET', body=None, client=None):
             method=method,
             headers={'Authorization': 'token {}'.format(self.auth_token)},
             body=body,
+            connect_timeout=3,  # default: 20s
+            request_timeout=10,  # default: 20s
         )
-        async with self.semaphore:
-            result = await client.fetch(req)
-            return result
+
+        async def _wait_for_api_request():
+            try:
+                async with self.semaphore:
+                    return await client.fetch(req)
+            except HTTPError as e:
+                # Retry on potentially transient errors in CHP, typically
+                # numbered 500 and up. Note that CHP isn't able to emit 429
+                # errors.
+                if e.code >= 500:
+                    self.log.warning(
+                        "api_request to the proxy failed with status code {}, retrying...".format(
+                            e.code
+                        )
+                    )
+                    return False  # a falsy return value make exponential_backoff retry
+                else:
+                    self.log.error("api_request to proxy failed: {0}".format(e))
+                    # An unhandled error here will help the hub invoke cleanup logic
+                    raise
+
+        result = await exponential_backoff(
+            _wait_for_api_request,
+            'Repeated api_request to proxy path "{}" failed.'.format(path),
+            timeout=30,
+        )
+        return result
 
     async def add_route(self, routespec, target, data):
         body = data or {}


Test patch: