Repository: jupyterhub/jupyterhub

Issue:
Tracking down 'too many redirects' errors at scale
I've been able to reliably reproduce the 'too many redirect' errors by simply hitting a JupyterHub with enough traffic. 

I've a rough script over at https://github.com/yuvipanda/jupyterhub-loadtest that can do this. 

If I make a simple JupyterHub, with the following config:

```
c.JupyterHub.authenticator_class = 'dummyauthenticator.DummyAuthenticator'
c.JupyterHub.spawner_class = 'simplespawner.SimpleLocalProcessSpawner'
```

and hit it with 50 simultaneous users I get at least a few too many redirect errors.

Going to use this to collect info and try to solve it!


Discussion:
I spent several hours looking at this and chasing down leads. Nowhere exact yet, but I can now reproduce with  confidence that for the users who are stuck in a loop, a POST request adding their route never actually makes it to the proxy, because add_route is never called for them! This would explain all the symptoms too.

Now to track down where add_route is being called and why it isn't called for these.
I was able to reproduce this as well. It's probably a race somewhere, or an uncaught exception when there are too many requests outstanding.
Perhaps completely unrelated. I was playing around with different examples last night and tried to update the `service-whoami` example in the repo. Ran into a bunch of Warnings for potential redirect errors.

```
[02:34:24 carol@cw-pro] jupyterhub on master [?$] $ jupyterhub --ip=127.0.0.1
[I 2017-07-14 02:34:35.320 JupyterHub app:764] Loading cookie_secret from /Users/carol/code/jhub/jupyterhub/jupyterhub_cookie_secret
[W 2017-07-14 02:34:35.333 JupyterHub app:879] No admin users, admin interface will be unavailable.
[W 2017-07-14 02:34:35.333 JupyterHub app:880] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2017-07-14 02:34:35.333 JupyterHub app:907] Not using whitelist. Any authenticated user will be allowed.
[I 2017-07-14 02:34:35.363 JupyterHub app:1435] Hub API listening on http://127.0.0.1:8081/hub/
[W 2017-07-14 02:34:35.364 JupyterHub proxy:383] 
    Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
    Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.
    
[W 2017-07-14 02:34:35.365 JupyterHub proxy:424] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2017-07-14 02:34:35.365 JupyterHub proxy:426] Starting proxy @ http://127.0.0.1:8000/
02:34:35.501 - info: [ConfigProxy] Proxying http://127.0.0.1:8000 to (no default)
02:34:35.504 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[I 2017-07-14 02:34:35.579 JupyterHub proxy:338] Adding default route for Hub: / => http://127.0.0.1:8081
02:34:35.588 - info: [ConfigProxy] Adding route / -> http://127.0.0.1:8081
02:34:35.590 - info: [ConfigProxy] 201 POST /api/routes/ 
02:34:35.594 - info: [ConfigProxy] 200 GET /api/routes 
[I 2017-07-14 02:34:35.596 JupyterHub app:1488] JupyterHub is now running at http://127.0.0.1:8000/
[I 2017-07-14 02:35:10.704 JupyterHub log:122] 302 GET /hub/ → /hub/login (@127.0.0.1) 1.40ms
[I 2017-07-14 02:35:23.365 JupyterHub spawner:843] Spawning jupyterhub-singleuser --port=63644
Failed to set groups [Errno 1] Operation not permitted
[I 2017-07-14 02:35:24.802 SingleUserNotebookApp extension:38] JupyterLab alpha preview extension loaded from /Users/carol/.pyenv/versions/3.6.1/lib/python3.6/site-packages/jupyterlab
JupyterLab v0.24.1
Known labextensions:
[I 2017-07-14 02:35:24.805 SingleUserNotebookApp extension:88] Running the core application with no additional extensions or settings
[I 2017-07-14 02:35:24.814 SingleUserNotebookApp singleuser:362] Starting jupyterhub-singleuser server version 0.8.0.dev
[I 2017-07-14 02:35:24.820 JupyterHub log:122] 200 GET /hub/api (@127.0.0.1) 1.07ms
[I 2017-07-14 02:35:24.821 SingleUserNotebookApp notebookapp:1366] Serving notebooks from local directory: /Users/carol
[I 2017-07-14 02:35:24.821 SingleUserNotebookApp notebookapp:1366] 0 active kernels 
[I 2017-07-14 02:35:24.821 SingleUserNotebookApp notebookapp:1366] The Jupyter Notebook is running at: http://127.0.0.1:63644/user/carol/
[I 2017-07-14 02:35:24.821 SingleUserNotebookApp notebookapp:1367] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2017-07-14 02:35:24.852 SingleUserNotebookApp log:122] 302 GET /user/carol/ → /user/carol/tree? (@127.0.0.1) 0.50ms
[W 2017-07-14 02:35:24.852 JupyterHub _version:23] Single-user server has no version header, which means it is likely < 0.8. Expected 0.8.0.dev
[I 2017-07-14 02:35:24.854 JupyterHub base:336] User carol server took 1.520 seconds to start
[I 2017-07-14 02:35:24.856 JupyterHub proxy:232] Adding user carol to proxy /user/carol/ => http://127.0.0.1:63644
02:35:24.858 - info: [ConfigProxy] Adding route /user/carol -> http://127.0.0.1:63644
02:35:24.858 - info: [ConfigProxy] 201 POST /api/routes/user/carol 
[I 2017-07-14 02:35:24.861 JupyterHub log:122] 302 POST /hub/login?next= → /hub/ (@127.0.0.1) 1646.06ms
[I 2017-07-14 02:35:24.861 JupyterHub login:113] User logged in: carol
[I 2017-07-14 02:35:24.873 JupyterHub log:122] 302 GET /hub/ → /user/carol/ (carol@127.0.0.1) 6.89ms
[I 2017-07-14 02:35:24.876 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.78ms
[W 2017-07-14 02:35:24.881 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.884 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.54ms
[I 2017-07-14 02:35:24.886 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.68ms
[W 2017-07-14 02:35:24.892 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.896 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.63ms
[I 2017-07-14 02:35:24.897 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.67ms
[W 2017-07-14 02:35:24.902 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.905 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.92ms
[I 2017-07-14 02:35:24.906 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.52ms
[W 2017-07-14 02:35:24.910 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.914 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.74ms
[I 2017-07-14 02:35:24.916 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.67ms
[W 2017-07-14 02:35:24.921 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.924 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.36ms
[I 2017-07-14 02:35:24.926 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.78ms
[W 2017-07-14 02:35:24.931 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.936 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 5.42ms
[I 2017-07-14 02:35:24.938 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.62ms
[W 2017-07-14 02:35:24.942 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.946 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.82ms
[I 2017-07-14 02:35:24.948 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.52ms
[W 2017-07-14 02:35:24.951 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.955 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.71ms
[I 2017-07-14 02:35:24.956 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.67ms
[W 2017-07-14 02:35:24.961 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:24.965 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 5.15ms
[I 2017-07-14 02:35:24.967 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.77ms
[W 2017-07-14 02:35:25.031 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.035 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.90ms
[I 2017-07-14 02:35:25.037 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.66ms
[W 2017-07-14 02:35:25.040 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.044 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.33ms
[I 2017-07-14 02:35:25.046 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.56ms
[W 2017-07-14 02:35:25.050 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.053 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.15ms
[I 2017-07-14 02:35:25.055 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.55ms
[W 2017-07-14 02:35:25.059 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.063 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.53ms
[I 2017-07-14 02:35:25.064 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.52ms
[W 2017-07-14 02:35:25.068 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.073 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.39ms
[I 2017-07-14 02:35:25.075 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 1.05ms
[W 2017-07-14 02:35:25.080 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.084 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.92ms
[I 2017-07-14 02:35:25.086 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.53ms
[W 2017-07-14 02:35:25.090 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.093 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.65ms
[I 2017-07-14 02:35:25.094 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.50ms
[W 2017-07-14 02:35:25.098 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.101 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.52ms
[I 2017-07-14 02:35:25.103 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.50ms
[W 2017-07-14 02:35:25.107 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.112 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.63ms
[I 2017-07-14 02:35:25.114 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.82ms
[W 2017-07-14 02:35:25.119 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.123 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.47ms
[I 2017-07-14 02:35:25.124 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.59ms
[W 2017-07-14 02:35:25.128 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:25.132 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.86ms
[W 2017-07-14 02:35:28.436 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.440 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.18ms
[I 2017-07-14 02:35:28.442 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.72ms
[W 2017-07-14 02:35:28.448 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.452 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.64ms
[I 2017-07-14 02:35:28.453 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.52ms
[W 2017-07-14 02:35:28.457 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.462 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.03ms
[I 2017-07-14 02:35:28.464 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.70ms
[W 2017-07-14 02:35:28.468 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.472 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.88ms
[I 2017-07-14 02:35:28.473 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.77ms
[W 2017-07-14 02:35:28.477 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.482 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 5.05ms
[I 2017-07-14 02:35:28.483 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.50ms
[W 2017-07-14 02:35:28.487 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.490 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.82ms
[I 2017-07-14 02:35:28.491 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.56ms
[W 2017-07-14 02:35:28.496 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.499 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.24ms
[I 2017-07-14 02:35:28.500 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.52ms
[W 2017-07-14 02:35:28.504 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.507 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.76ms
[I 2017-07-14 02:35:28.509 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.60ms
[W 2017-07-14 02:35:28.514 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.518 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 4.65ms
[I 2017-07-14 02:35:28.520 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.69ms
[W 2017-07-14 02:35:28.524 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.529 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 5.49ms
[I 2017-07-14 02:35:28.530 JupyterHub log:122] 302 GET /user/carol/ → /hub/user/carol/ (@127.0.0.1) 0.53ms
[W 2017-07-14 02:35:28.534 JupyterHub base:531] 
                        Detected possible direct connection to Hub's private ip: http://127.0.0.1:8081/hub/user/carol/, bypassing proxy.
                        This will result in a redirect loop.
                        Make sure to connect to the proxied public URL http://127.0.0.1:8000/
                        
[I 2017-07-14 02:35:28.537 JupyterHub log:122] 302 GET /hub/user/carol/ → /user/carol/ (carol@127.0.0.1) 3.54ms
[W 2017-07-14 02:35:38.966 JupyterHub base:32] Blocking API request with no referer
[I 2017-07-14 02:35:38.967 JupyterHub log:122] 200 GET /hub/api (@127.0.0.1) 0.86ms
[W 2017-07-14 02:35:44.297 JupyterHub base:32] Blocking API request with no referer
[W 2017-07-14 02:35:44.299 JupyterHub base:32] Blocking API request with no referer
[W 2017-07-14 02:35:44.299 JupyterHub log:122] 403 GET /hub/api/info (@127.0.0.1) 2.38ms
^C
Interrupted
[I 2017-07-14 02:35:56.024 JupyterHub app:1315] Cleaning up single-user servers...
[I 2017-07-14 02:35:56.025 JupyterHub proxy:463] Cleaning up proxy[59731]...
[I 2017-07-14 02:35:56.028 SingleUserNotebookApp notebookapp:1308] Shutting down kernels
[I 2017-07-14 02:35:56.152 JupyterHub app:1346] ...done
```
I spent some time on this, and I think I know roughly the issue. It's a race condition (like everything else) triggered when spawners are slow to start *and* adding routes to the proxy also becomes slow. The solution is not quite clear to me yet.

The relevant moving parts:

1. Spawning a user's server starts in the [Handler.spawn_single_user()](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/handlers/base.py#L318) method.
2. Adding the server to the proxy is handled after `Spawner.spawn()` finished, in [finish_user_spawn](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/handlers/base.py#L326).
3. When the Spawner is prompt, finish is [awaited properly](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/handlers/base.py#L370). No race.
4. However, when the Spawner is slow (e.g. caused by high load), finish is registered [after spawn](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/handlers/base.py#L364), but no longer blocks other things.
5. To deal with requests made while a server is starting, there are [running](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/user.py#L160) and [spawn_pending](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/user.py#L362) flags. `spawn_pending` is cleared (and thereby `running` is set) when `User.spawn` [finishes](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/user.py#L362).
6. When a request is made, [user.running is checked](https://github.com/jupyterhub/jupyterhub/blob/a13e7766fce4da5e556d40628f7a335959f93381/jupyterhub/handlers/pages.py#L97), and if the user's server is running
7. user.running is set when user.spawn finishes, but before the route has been added to the proxy.
8. There is no check for whether the server has finished being added to the proxy.

This leaves a race in the case where adding a route to the proxy is slow (mainly due to a busy queue of tasks in the Hub itself), for the window between when a server becomes responsive and the route being added to the proxy. Requests made to `/hub/spawn` will hit a redirect loop because the Hub redirects assuming that the user has been added to the proxy, but it has only started, and is still waiting to be added to the proxy.

If `user.running` were not set until after the user has been added to the proxy, I think this particular case should be addressed.

@willingc those warning messages look like `127.0.0.1:8080` is being visited directly, rather than the proxied url `127.0.0.1:8000`. Is that happening?

Resolving patch:
diff --git a/jupyterhub/handlers/base.py b/jupyterhub/handlers/base.py
--- a/jupyterhub/handlers/base.py
+++ b/jupyterhub/handlers/base.py
@@ -321,6 +321,7 @@ def spawn_single_user(self, user, options=None):
         tic = IOLoop.current().time()
 
         f = user.spawn(options)
+        user.proxy_pending = True
 
         @gen.coroutine
         def finish_user_spawn(f=None):
@@ -335,8 +336,16 @@ def finish_user_spawn(f=None):
             toc = IOLoop.current().time()
             self.log.info("User %s server took %.3f seconds to start", user.name, toc-tic)
             self.statsd.timing('spawner.success', (toc - tic) * 1000)
-            yield self.proxy.add_user(user)
-            user.spawner.add_poll_callback(self.user_stopped, user)
+            try:
+                yield self.proxy.add_user(user)
+            except Exception:
+                self.log.exception("Failed to add user %s to proxy!", user)
+                self.log.error("Stopping user %s to avoid inconsistent state")
+                yield user.stop()
+            else:
+                user.spawner.add_poll_callback(self.user_stopped, user)
+            finally:
+                user.proxy_pending = False
 
         try:
             yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)
@@ -532,7 +541,7 @@ def get(self, name, user_path):
 
             # logged in as correct user, spawn the server
             if current_user.spawner:
-                if current_user.spawn_pending:
+                if current_user.spawn_pending or current_user.proxy_pending:
                     # spawn has started, but not finished
                     self.statsd.incr('redirects.user_spawn_pending', 1)
                     html = self.render_template("spawn_pending.html", user=current_user)
diff --git a/jupyterhub/singleuser.py b/jupyterhub/singleuser.py
--- a/jupyterhub/singleuser.py
+++ b/jupyterhub/singleuser.py
@@ -4,9 +4,7 @@
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
 
-from distutils.version import LooseVersion as V
 import os
-import re
 from textwrap import dedent
 from urllib.parse import urlparse
 
@@ -349,10 +347,17 @@ def check_hub_version(self):
         - check version and warn on sufficient mismatch
         """
         client = AsyncHTTPClient()
-        try:
-            resp = yield client.fetch(self.hub_api_url)
-        except Exception:
-            self.log.exception("Failed to connect to my Hub at %s. Is it running?", self.hub_api_url)
+        RETRIES = 5
+        for i in range(1, RETRIES+1):
+            try:
+                resp = yield client.fetch(self.hub_api_url)
+            except Exception:
+                self.log.exception("Failed to connect to my Hub at %s (attempt %i/%i). Is it running?",
+                self.hub_api_url, i, RETRIES)
+                yield gen.sleep(min(2**i, 16))
+            else:
+                break
+        else:
             self.exit(1)
         
         hub_version = resp.headers.get('X-JupyterHub-Version')
diff --git a/jupyterhub/user.py b/jupyterhub/user.py
--- a/jupyterhub/user.py
+++ b/jupyterhub/user.py
@@ -110,6 +110,7 @@ def _orm_user_changed(self, change):
     spawner = None
     spawn_pending = False
     stop_pending = False
+    proxy_pending = False
     waiting_for_response = False
 
     @property
@@ -158,8 +159,8 @@ def __repr__(self):
 
     @property # FIX-ME CHECK IF STILL NEEDED
     def running(self):
-        """property for whether a user has a running server"""
-        if self.spawn_pending or self.stop_pending:
+        """property for whether a user has a fully running, accessible server"""
+        if self.spawn_pending or self.stop_pending or self.proxy_pending:
             return False  # server is not running if spawn or stop is still pending
         if self.server is None:
             return False
@@ -392,10 +393,13 @@ def stop(self):
                     self.db.delete(orm_token)
             self.db.commit()
         finally:
-            self.stop_pending = False
             # trigger post-spawner hook on authenticator
             auth = spawner.authenticator
-            if auth:
-                yield gen.maybe_future(
-                    auth.post_spawn_stop(self, spawner)
-                )
+            try:
+                if auth:
+                    yield gen.maybe_future(
+                        auth.post_spawn_stop(self, spawner)
+                    )
+            except Exception:
+                self.log.exception("Error in Authenticator.post_spawn_stop for %s", self)
+            self.stop_pending = False


Test patch: