Repository: jupyterhub/jupyterhub

Issue:
Container dying after startup leads to infinite redirection loop
We are using jupyterhub with the dockerspawner module. Once successfully logged in, we observed that if the docker container is spawned but dies a second or so later (for whatever reason), the redirection from /hub/user/username to /user/username will enter an infinite redirection loop. I suspect the reason is
that there's nothing listening at /user/username (because the container server is not there), so jupyterhub redirects you to /hub/user/username as default action in that case, which in turn tries to send you back to /user/username because you are logged in.

It's an unusual condition and we are running outside the basic design, but if you have any feedback on this it would be great.

jupyterhub version 0.6.1



Discussion:
Something's not right if `/user/username` is going to the Hub and the user's server is not running. That redirect should only happen if the Hub thinks it is running. It's still possible for that to happen if the server has died, but the Hub hasn't noticed, but this should result in a 503 error, not a redirect loop. Can you post debug logs surrounding when this happens?

@minrk Well the problem is that it is running, but dies shortly afterwards.

What do you see in the Hub logs when it does? And what do you see in the single-user container's logs?

@minrk I'll check on Monday.

I'm hitting this with a custom spawner.  I witness the too many redirects once logged in and I first try to launch my server.  I've included my logs below.

I'm on JupyterHub 0.6.1.
- I hit my `/users/jtyberg` endpoint.
- My spawner `poll` method returns an empty string because the container does not exist.
- My spawner `start` method gets invoked and starts the container. 
- My spawner `poll` method gets invoked in spurts every few seconds (over 50 times).  It returns status indicating ready (returns `None`) each time.
- While JupyterHub is waiting, the notebook container is started and ready (see log).
- Despite `poll` returning ready status, the hub eventually gives up and removes the container.  
- The hub then invokes my spawner `start` again.  This time, the container starts and I am routed to my notebook server home page. 

I could be doing something incorrectly, but it seems like the initial startup sequence is not working well if the notebook server container takes a few seconds to respond to the hub.  I have `http_timeout` set to 60 seconds. The [wait_for_http_server](https://github.com/jupyterhub/jupyterhub/blob/a4bb25a75ff449b0d2e4174a5798ac21041d0c68/jupyterhub/utils.py#L64) tries every 0.1 seconds, but then it seems to stop trying and never resumes once the polling mechanism kicks in.  The hub doesn't seem to do anything with the poll ready status either, so eventually the hub just kills the container.

JupyterHub log:

```
[D 2016-08-03 01:36:42.990 JupyterHub application:529] Looking for /srv/jupyterhub/jupyterhub_config in None
[D 2016-08-03 01:36:43.140 JupyterHub application:549] Loaded config file: /srv/jupyterhub/jupyterhub_config.py
[I 2016-08-03 01:36:43.157 JupyterHub app:622] Loading cookie_secret from /data/jupyterhub_cookie_secret
[D 2016-08-03 01:36:43.157 JupyterHub app:694] Connecting to db: sqlite:////data/jupyterhub.sqlite
[W 2016-08-03 01:36:43.201 JupyterHub app:304] 
    Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
    Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.

[D 2016-08-03 01:36:43.225 JupyterHub app:871] Loading state for jtyberg from db
[D 2016-08-03 01:36:43.226 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:36:43.226 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:36:43.474 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: 
[D 2016-08-03 01:36:43.474 JupyterHub app:883] jtyberg not running.
[D 2016-08-03 01:36:43.474 JupyterHub app:888] Loaded users: 
     jtyberg admin
[I 2016-08-03 01:36:43.482 JupyterHub app:1231] Hub API listening on http://0.0.0.0:8080/hub/
[I 2016-08-03 01:36:43.487 JupyterHub app:968] Starting proxy @ http://*:443/
[D 2016-08-03 01:36:43.487 JupyterHub app:969] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '443', '--api-ip', '0.0.0.0', '--api-port', '444', '--default-target', 'http://127.0.0.1:8080', '--error-target', 'http://127.0.0.1:8080/hub/error', '--ssl-key', '/etc/letsencrypt/live/my.domain/privkey.pem', '--ssl-cert', '/etc/letsencrypt/live/my.domain/cert.pem']
01:36:43.638 - info: [ConfigProxy] Proxying https://*:443 to http://127.0.0.1:8080
01:36:43.640 - info: [ConfigProxy] Proxy API at http://0.0.0.0:444/api/routes
[D 2016-08-03 01:36:43.709 JupyterHub app:997] Proxy started and appears to be up
[I 2016-08-03 01:36:43.710 JupyterHub app:1254] JupyterHub is now running at http://127.0.0.1:443/
[I 2016-08-03 01:36:51.838 JupyterHub log:100] 302 GET /user/jtyberg (@10.255.0.3) 2.49ms
[I 2016-08-03 01:36:51.842 JupyterHub log:100] 302 GET /user/jtyberg/api/terminals?_=1470188184437 (@10.255.0.3) 0.76ms
[I 2016-08-03 01:36:51.846 JupyterHub log:100] 302 GET /user/jtyberg/api/sessions?_=1470188184436 (@10.255.0.3) 0.70ms
[D 2016-08-03 01:36:51.857 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:36:51.858 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:36:51.864 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:36:51.864 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:36:51.870 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:36:51.870 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:36:51.872 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: 
[D 2016-08-03 01:36:51.872 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: 
[D 2016-08-03 01:36:51.898 JupyterHub jupyterhub_config:105] Getting service 'notebook-jtyberg'
[E 2016-08-03 01:36:51.907 JupyterHub web:1524] Uncaught exception GET /hub/user/jtyberg/api/terminals?_=1470188184437 (10.255.0.3)
    HTTPServerRequest(protocol='https', host='my.domain', method='GET', uri='/hub/user/jtyberg/api/terminals?_=1470188184437', version='HTTP/1.1', remote_ip='10.255.0.3', headers={'Accept-Language': 'en-US,en;q=0.8', 'Referer': 'https://my.domain/user/jtyberg/tree', 'Connection': 'close', 'Cache-Control': 'no-cache', 'X-Requested-With': 'XMLHttpRequest', 'Accept-Encoding': 'gzip, deflate, sdch, br', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', 'Cookie': 'jupyter-hub-token="2|1:0|10:1470177009|17:jupyter-hub-token|44:YTM1MjMxYTMyYzcyNDQwMzhiOGEyOTYxMjNmODNiOGU=|79fa7dc5a1213a93896104728b02eeb01a4b67d2e7547cac91036e598991b03e"; _ga=GA1.2.897722413.1443065769', 'Accept': 'application/json, text/javascript, */*; q=0.01', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36', 'X-Forwarded-For': '10.255.0.3', 'Host': 'my.domain', 'Pragma': 'no-cache'})
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/tornado/web.py", line 1445, in _execute
        result = yield result
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 494, in get
        yield self.spawn_single_user(current_user)
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 290, in spawn_single_user
        raise RuntimeError("Spawn already pending for: %s" % user.name)
    RuntimeError: Spawn already pending for: jtyberg

[D 2016-08-03 01:36:51.914 JupyterHub base:441] No template for 500
[E 2016-08-03 01:36:51.952 JupyterHub log:99] {
      "Accept-Language": "en-US,en;q=0.8",
      "Referer": "https://my.domain/user/jtyberg/tree",
      "Connection": "close",
      "Cache-Control": "no-cache",
      "X-Requested-With": "XMLHttpRequest",
      "Accept-Encoding": "gzip, deflate, sdch, br",
      "X-Forwarded-Port": "443",
      "X-Forwarded-Proto": "https",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1470177009|17:jupyter-hub-token|44:YTM1MjMxYTMyYzcyNDQwMzhiOGEyOTYxMjNmODNiOGU=|79fa7dc5a1213a93896104728b02eeb01a4b67d2e7547cac91036e598991b03e\"; _ga=GA1.2.897722413.1443065769",
      "Accept": "application/json, text/javascript, */*; q=0.01",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
      "X-Forwarded-For": "10.255.0.3",
      "Host": "my.domain",
      "Pragma": "no-cache"
    }
[E 2016-08-03 01:36:51.952 JupyterHub log:100] 500 GET /hub/user/jtyberg/api/terminals?_=1470188184437 (jtyberg@10.255.0.3) 88.34ms
[D 2016-08-03 01:36:51.952 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: 
[D 2016-08-03 01:36:51.954 JupyterHub jupyterhub_config:186] Creating service notebook-jtyberg with task config {
      "ContainerSpec": {
        "Args": [
          "--debug"
        ],
        "Env": [
          "JPY_HUB_API_URL=http://jupyterhub:8080/hub/api",
          "JPY_API_TOKEN=3ff1dcf7d21f4f60b4dbfffdcb94851e",
          "PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
          "JPY_USER=jtyberg",
          "JPY_BASE_URL=/user/jtyberg",
          "JPY_HUB_PREFIX=/hub/",
          "JPY_COOKIE_NAME=jupyter-hub-token-jtyberg",
          "LANG=C.UTF-8"
        ],
        "Mounts": [
          {
            "ReadOnly": false,
            "VolumeOptions": {
              "DriverConfig": {
                "Name": "convoy"
              }
            },
            "Target": "/home/jovyan/work",
            "Populate": false,
            "Source": "notebook-jtyberg",
            "Type": "volume"
          }
        ],
        "Image": "jupyter/pyspark-notebook:1d374670daaa",
        "Command": [
          "tini",
          "--",
          "start-singleuser.sh"
        ]
      }
    } and kwargs {
      "name": "notebook-jtyberg",
      "networks": [
        {
          "Target": "jupyter"
        }
      ],
      "endpoint_config": {
        "Ports": [
          {
            "Protocol": "tcp",
            "TargetPort": 4040
          },
          {
            "Protocol": "tcp",
            "TargetPort": 8080
          }
        ]
      }
    }
[E 2016-08-03 01:36:51.954 JupyterHub web:1524] Uncaught exception GET /hub/user/jtyberg/api/sessions?_=1470188184436 (10.255.0.3)
    HTTPServerRequest(protocol='https', host='my.domain', method='GET', uri='/hub/user/jtyberg/api/sessions?_=1470188184436', version='HTTP/1.1', remote_ip='10.255.0.3', headers={'Accept-Language': 'en-US,en;q=0.8', 'Referer': 'https://my.domain/user/jtyberg/tree', 'Connection': 'close', 'Cache-Control': 'no-cache', 'X-Requested-With': 'XMLHttpRequest', 'Accept-Encoding': 'gzip, deflate, sdch, br', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https', 'Cookie': 'jupyter-hub-token="2|1:0|10:1470177009|17:jupyter-hub-token|44:YTM1MjMxYTMyYzcyNDQwMzhiOGEyOTYxMjNmODNiOGU=|79fa7dc5a1213a93896104728b02eeb01a4b67d2e7547cac91036e598991b03e"; _ga=GA1.2.897722413.1443065769', 'Accept': 'application/json, text/javascript, */*; q=0.01', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36', 'X-Forwarded-For': '10.255.0.3', 'Host': 'my.domain', 'Pragma': 'no-cache'})
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/tornado/web.py", line 1445, in _execute
        result = yield result
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 494, in get
        yield self.spawn_single_user(current_user)
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 290, in spawn_single_user
        raise RuntimeError("Spawn already pending for: %s" % user.name)
    RuntimeError: Spawn already pending for: jtyberg

[D 2016-08-03 01:36:51.959 JupyterHub base:441] No template for 500
[E 2016-08-03 01:36:51.967 JupyterHub log:99] {
      "Accept-Language": "en-US,en;q=0.8",
      "Referer": "https://my.domain/user/jtyberg/tree",
      "Connection": "close",
      "Cache-Control": "no-cache",
      "X-Requested-With": "XMLHttpRequest",
      "Accept-Encoding": "gzip, deflate, sdch, br",
      "X-Forwarded-Port": "443",
      "X-Forwarded-Proto": "https",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1470177009|17:jupyter-hub-token|44:YTM1MjMxYTMyYzcyNDQwMzhiOGEyOTYxMjNmODNiOGU=|79fa7dc5a1213a93896104728b02eeb01a4b67d2e7547cac91036e598991b03e\"; _ga=GA1.2.897722413.1443065769",
      "Accept": "application/json, text/javascript, */*; q=0.01",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
      "X-Forwarded-For": "10.255.0.3",
      "Host": "my.domain",
      "Pragma": "no-cache"
    }
[E 2016-08-03 01:36:51.967 JupyterHub log:100] 500 GET /hub/user/jtyberg/api/sessions?_=1470188184436 (jtyberg@10.255.0.3) 97.82ms
[I 2016-08-03 01:36:51.967 JupyterHub jupyterhub_config:194] Created service 'notebook-jtyberg' (ID: 7l9qj6pgysukv9zs63nbup0zs) from image jupyter/pyspark-notebook:1d374670daaa
[D 2016-08-03 01:36:51.968 JupyterHub spawner:316] Polling subprocess every 30s
[W 2016-08-03 01:36:51.994 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.098 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.201 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.304 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.407 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.510 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.613 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.716 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.819 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:52.922 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:53.025 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:53.128 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:53.231 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:36:56.333 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno 113] No route to host)
[D 2016-08-03 01:37:01.907 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:37:01.907 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:37:01.912 JupyterHub jupyterhub_config:136] service task status '{
      "Timestamp": "2016-08-03T01:36:55.085236263Z",
      "Message": "started",
      "State": "running",
      "ContainerStatus": {
        "PID": 15290,
        "ContainerID": "8d05009104cf76056f981fa6b2dd537e96e5baf34324084bddd0f6924a669767"
      }
    }'
[D 2016-08-03 01:37:01.913 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: None
[W 2016-08-03 01:37:01.913 JupyterHub base:327] User jtyberg server is slow to become responsive
[D 2016-08-03 01:37:01.913 JupyterHub base:231] Setting cookie for jtyberg: jupyter-hub-token-jtyberg, {'secure': True}
[I 2016-08-03 01:37:01.923 JupyterHub log:100] 302 GET /hub/user/jtyberg (jtyberg@10.255.0.3) 10064.16ms
[I 2016-08-03 01:37:01.948 JupyterHub log:100] 302 GET /user/jtyberg (@10.255.0.3) 0.85ms
[D 2016-08-03 01:37:01.971 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:37:01.972 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:37:01.975 JupyterHub jupyterhub_config:136] service task status '{
      "Timestamp": "2016-08-03T01:36:55.085236263Z",
      "Message": "started",
      "State": "running",
      "ContainerStatus": {
        "PID": 15290,
        "ContainerID": "8d05009104cf76056f981fa6b2dd537e96e5baf34324084bddd0f6924a669767"
      }
    }'
[D 2016-08-03 01:37:01.976 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: None
[D 2016-08-03 01:37:01.977 JupyterHub base:231] Setting cookie for jtyberg: jupyter-hub-token-jtyberg, {'secure': True}
[I 2016-08-03 01:37:01.984 JupyterHub log:100] 302 GET /hub/user/jtyberg (jtyberg@10.255.0.3) 10.47ms
[I 2016-08-03 01:37:02.004 JupyterHub log:100] 302 GET /user/jtyberg (@10.255.0.3) 0.80ms
[D 2016-08-03 01:37:02.023 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:37:02.023 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:37:02.027 JupyterHub jupyterhub_config:136] service task status '{
      "Timestamp": "2016-08-03T01:36:55.085236263Z",
      "Message": "started",
      "State": "running",
      "ContainerStatus": {
        "PID": 15290,
        "ContainerID": "8d05009104cf76056f981fa6b2dd537e96e5baf34324084bddd0f6924a669767"
      }
    }'
[D 2016-08-03 01:37:02.027 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: None
[D 2016-08-03 01:37:02.029 JupyterHub base:231] Setting cookie for jtyberg: jupyter-hub-token-jtyberg, {'secure': True}
[I 2016-08-03 01:37:02.035 JupyterHub log:100] 302 GET /hub/user/jtyberg (jtyberg@10.255.0.3) 10.46ms
[I 2016-08-03 01:37:02.055 JupyterHub log:100] 302 GET /user/jtyberg (@10.255.0.3) 0.79ms
[D 2016-08-03 01:37:02.076 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:37:02.076 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:37:02.079 JupyterHub jupyterhub_config:136] service task status '{
      "Timestamp": "2016-08-03T01:36:55.085236263Z",
      "Message": "started",
      "State": "running",
      "ContainerStatus": {
        "PID": 15290,
        "ContainerID": "8d05009104cf76056f981fa6b2dd537e96e5baf34324084bddd0f6924a669767"
      }
    }'

...skip 50 polls...

[D 2016-08-03 01:37:51.979 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: None
[W 2016-08-03 01:37:56.757 JupyterHub user:264] jtyberg's server never showed up at http://notebook-jtyberg:8888/user/jtyberg after 60 seconds. Giving up
[D 2016-08-03 01:37:56.758 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:37:56.758 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:37:56.762 JupyterHub jupyterhub_config:136] service task status '{
      "Timestamp": "2016-08-03T01:36:55.085236263Z",
      "Message": "started",
      "State": "running",
      "ContainerStatus": {
        "PID": 15290,
        "ContainerID": "8d05009104cf76056f981fa6b2dd537e96e5baf34324084bddd0f6924a669767"
      }
    }'
[D 2016-08-03 01:37:56.762 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: None
[I 2016-08-03 01:37:56.762 JupyterHub jupyterhub_config:229] Removing service notebook-jtyberg (ID: 7l9qj6pgysukv9zs63nbup0zs)
[E 2016-08-03 01:37:56.777 JupyterHub gen:871] Exception in Future <tornado.concurrent.Future object at 0x7f71d7b78f60> after timeout
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 867, in error_callback
        future.result()
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/user.py", line 280, in spawn
        raise e
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/user.py", line 256, in spawn
        yield self.server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/orm.py", line 108, in wait_up
        yield wait_for_http_server(self.url, timeout=timeout)
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/utils.py", line 94, in wait_for_http_server
        **locals()
    TimeoutError: Server at http://notebook-jtyberg:8888/user/jtyberg didn't respond in 60 seconds

[D 2016-08-03 01:38:39.095 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:38:39.095 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:38:39.100 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: 
[D 2016-08-03 01:38:39.122 JupyterHub jupyterhub_config:105] Getting service 'notebook-jtyberg'
[D 2016-08-03 01:38:39.134 JupyterHub jupyterhub_config:186] Creating service notebook-jtyberg with task config {
      "ContainerSpec": {
        "Args": [
          "--debug"
        ],
        "Env": [
          "JPY_HUB_API_URL=http://jupyterhub:8080/hub/api",
          "JPY_API_TOKEN=f2d3cc47126545beb8166162cae98d14",
          "PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
          "JPY_USER=jtyberg",
          "JPY_BASE_URL=/user/jtyberg",
          "JPY_HUB_PREFIX=/hub/",
          "JPY_COOKIE_NAME=jupyter-hub-token-jtyberg",
          "LANG=C.UTF-8"
        ],
        "Mounts": [
          {
            "ReadOnly": false,
            "VolumeOptions": {
              "DriverConfig": {
                "Name": "convoy"
              }
            },
            "Target": "/home/jovyan/work",
            "Populate": false,
            "Source": "notebook-jtyberg",
            "Type": "volume"
          }
        ],
        "Image": "jupyter/pyspark-notebook:1d374670daaa",
        "Command": [
          "tini",
          "--",
          "start-singleuser.sh"
        ]
      }
    } and kwargs {
      "name": "notebook-jtyberg",
      "networks": [
        {
          "Target": "jupyter"
        }
      ],
      "endpoint_config": {
        "Ports": [
          {
            "Protocol": "tcp",
            "TargetPort": 4040
          },
          {
            "Protocol": "tcp",
            "TargetPort": 8080
          }
        ]
      }
    }
[I 2016-08-03 01:38:39.141 JupyterHub jupyterhub_config:194] Created service 'notebook-jtyberg' (ID: dvhzovu8st3hjropht9edg2fx) from image jupyter/pyspark-notebook:1d374670daaa
[D 2016-08-03 01:38:39.141 JupyterHub spawner:316] Polling subprocess every 30s
[W 2016-08-03 01:38:39.154 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.257 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.360 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.463 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.566 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.669 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.772 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.875 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:39.978 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:40.082 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:40.185 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:40.288 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:40.391 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:40.494 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno -2] Name or service not known)
[W 2016-08-03 01:38:43.597 JupyterHub utils:88] Failed to connect to http://notebook-jtyberg:8888/user/jtyberg ([Errno 113] No route to host)
[D 2016-08-03 01:38:43.907 JupyterHub utils:84] Server at http://notebook-jtyberg:8888/user/jtyberg responded with 302
[I 2016-08-03 01:38:43.907 JupyterHub base:306] User jtyberg server took 4.807 seconds to start
[I 2016-08-03 01:38:43.908 JupyterHub orm:159] Adding user jtyberg to proxy /user/jtyberg => http://notebook-jtyberg:8888
[D 2016-08-03 01:38:43.910 JupyterHub orm:146] Fetching POST http://127.0.0.1:444/api/routes/user/jtyberg
[D 2016-08-03 01:38:43.914 JupyterHub base:231] Setting cookie for jtyberg: jupyter-hub-token-jtyberg, {'secure': True}
[I 2016-08-03 01:38:43.923 JupyterHub log:100] 302 GET /hub/user/jtyberg (jtyberg@10.255.0.3) 4829.00ms
[I 2016-08-03 01:38:43.992 JupyterHub log:100] 200 GET /hub/api/authorizations/cookie/jupyter-hub-token-jtyberg/[secret] (jtyberg@10.255.0.9) 11.82ms
[D 2016-08-03 01:38:44.201 JupyterHub log:100] 200 GET /hub/logo (@10.255.0.3) 6.20ms
[D 2016-08-03 01:39:09.147 JupyterHub jupyterhub_config:244] Polling service notebook-jtyberg
[D 2016-08-03 01:39:09.147 JupyterHub jupyterhub_config:124] Getting tasks for service 'notebook-jtyberg'
[D 2016-08-03 01:39:09.153 JupyterHub jupyterhub_config:136] service task status '{
      "Timestamp": "2016-08-03T01:38:42.300474388Z",
      "Message": "started",
      "State": "running",
      "ContainerStatus": {
        "PID": 26935,
        "ContainerID": "2b501c477731b95693e631ad7e8cee0e3beb9f829e49b2621922fa50602ae158"
      }
    }'
[D 2016-08-03 01:39:09.153 JupyterHub jupyterhub_config:255] Status for service notebook-jtyberg: None
```

Notebook container log (before it gets killed and a new one spawned):

```
[D 2016-08-03 01:36:56.300 jtyberg application:178] Searching ['/home/jovyan/work', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2016-08-03 01:36:56.300 jtyberg application:529] Looking for jupyter_config in /etc/jupyter
[D 2016-08-03 01:36:56.300 jtyberg application:529] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2016-08-03 01:36:56.301 jtyberg application:529] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2016-08-03 01:36:56.301 jtyberg application:529] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2016-08-03 01:36:56.301 jtyberg application:529] Looking for jupyter_config in /home/jovyan/work
[D 2016-08-03 01:36:56.304 jtyberg application:529] Looking for jupyter_notebook_config in /etc/jupyter
[D 2016-08-03 01:36:56.304 jtyberg application:529] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2016-08-03 01:36:56.304 jtyberg application:529] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2016-08-03 01:36:56.304 jtyberg application:529] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2016-08-03 01:36:56.305 jtyberg application:549] Loaded config file: /home/jovyan/.jupyter/jupyter_notebook_config.py
[W 2016-08-03 01:36:56.309 jtyberg configurable:169] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[W 2016-08-03 01:36:56.312 jtyberg configurable:169] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2016-08-03 01:36:56.313 jtyberg application:529] Looking for jupyter_notebook_config in /home/jovyan/work
[I 2016-08-03 01:36:56.333 jtyberg notebookapp:503] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[I 2016-08-03 01:36:56.391 jtyberg notebookapp:1128] Serving notebooks from local directory: /home/jovyan/work
[I 2016-08-03 01:36:56.391 jtyberg notebookapp:1128] 0 active kernels 
[I 2016-08-03 01:36:56.391 jtyberg notebookapp:1128] The Jupyter Notebook is running at: http://0.0.0.0:8888/user/jtyberg/
[I 2016-08-03 01:36:56.392 jtyberg notebookapp:1129] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
```

I'm also observing the same behavior as @jtyberg but sudospawner without modification. I'll report back when I have more info.

Maybe a different root cause. I've opened #720 to avoid cluttering up this issue.

I think we've been seeing this as well, occasionally. In our case it seemed to happen when the user's container was slow to start up, or when it started but the singleuser process took a while to come up.

Improving the backing storage performance for docker seemed to help, but we still see it on occasion. I'll try to reproduce it more systematically.


Resolving patch:
diff --git a/jupyterhub/user.py b/jupyterhub/user.py
--- a/jupyterhub/user.py
+++ b/jupyterhub/user.py
@@ -258,7 +258,6 @@ def spawn(self, options=None):
         self.state = spawner.get_state()
         self.last_activity = datetime.utcnow()
         db.commit()
-        self.spawn_pending = False
         try:
             yield self.server.wait_up(http=True, timeout=spawner.http_timeout)
         except Exception as e:
@@ -285,6 +284,7 @@ def spawn(self, options=None):
                 ), exc_info=True)
             # raise original TimeoutError
             raise e
+        self.spawn_pending = False
         return self
 
     @gen.coroutine


Test patch: