Repository: jupyterhub/jupyterhub

Issue:
Hub crashes if it tries to delete a proxy route that does not exist
Hub seems to have a lot of trouble with syncing proxy routes to running users, often crashing with something like:

```
E 2018-01-30 10:42:32.806 JupyterHub app:1623]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1621, in launch_instance_async
        yield self.start()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1569, in start
        yield self.proxy.check_routes(self.users, self._service_map)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 366, in check_routes
        yield f
    tornado.httpclient.HTTPError: HTTP 404: Not Found
```

during the startup sequence, while deleting routes that shouldn't exist.


Discussion:
I had to delete all running user pods to bring this back to normal :(
Now getting a lot of:

```
[E 2018-01-30 10:53:26.719 JupyterHub spawner:754] Unhandled error in poll callback for <CustomAttrSpawner object at 0x7f1b34179940>
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/spawner.py", line 752, in poll_and_notify
        yield gen.maybe_future(callback())
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 1206, in user_stopped
        yield self.proxy.delete_user(user, server_name)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 255, in delete_user
        yield self.delete_route(routespec)
    tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20553 milliseconds with 0 bytes received
```
I now suspect it's a bug with a kubespawner update perhaps. It's thrashing heavily with a ton of threads:

```
datahub@k8s-pool2-19522833-2:~$ sudo ls /proc/78309/task/ | wc -l
18929
```
ok, reverting to v0.5 of helm chart fixes this.

```
datahub@k8s-pool1-19522833-12:~$ sudo ls /proc/37483/task/ | wc -l
15
```

Resolving patch:
diff --git a/jupyterhub/proxy.py b/jupyterhub/proxy.py
--- a/jupyterhub/proxy.py
+++ b/jupyterhub/proxy.py
@@ -24,7 +24,7 @@
 from urllib.parse import quote
 
 from tornado import gen
-from tornado.httpclient import AsyncHTTPClient, HTTPRequest
+from tornado.httpclient import AsyncHTTPClient, HTTPRequest, HTTPError
 from tornado.ioloop import PeriodicCallback
 
 
@@ -576,7 +576,16 @@ def add_route(self, routespec, target, data):
 
     def delete_route(self, routespec):
         path = self._routespec_to_chp_path(routespec)
-        return self.api_request(path, method='DELETE')
+        try:
+            return self.api_request(path, method='DELETE')
+        except HTTPError as e:
+            if e.status_code == 404:
+                # Warn about 404s because something might be wrong
+                # but don't raise because the route is gone,
+                # which is the goal.
+                self.log.warning("Route %s already deleted", routespec)
+            else:
+                raise
 
     def _reformat_routespec(self, routespec, chp_data):
         """Reformat CHP data format to JupyterHub's proxy API."""


Test patch: