Repository: jupyterhub/jupyterhub

Issue:
"ValueError: IOStream is not idle; cannot convert to SSL" while waiting for http server
<!-- Thank you for contributing. These HTML comments will not render in the issue, but you can delete them once you've read them if you prefer! -->
Hi,

in our current JupyterHub the notebook-server starts with a certain delay (they're started on a remote hpc system, which may take a few seconds). The `Spawner.start()` function returns before the notebook-server is fully up and running. Furthermore, we're using `JupyterHub.internal_ssl = true`. 

we're currently using JupyterHub to start Jupyter
### Bug description

While JupyterHub is waiting for the notebook-server to answer, the `tornado.AsyncHTTPClient.fetch()` function in [utils.py](https://github.com/jupyterhub/jupyterhub/blob/0abe517faad95460889724e2599c33b0a1961bb0/jupyterhub/utils.py#L277) throws a `ValueError: IOStream is not idle; cannot convert to SSL`. 

<details><summary>Traceback: (click to expand)</summary>  

``` 
Unhandled error waiting for <name...> server to show up at <address...>: IOStream is not idle; cannot convert to SSL
Traceback (most recent call last):
  File "/src/jupyterhub/jupyterhub/user.py", line 928, in _wait_up
    resp = await server.wait_up(
  File "/src/jupyterhub/jupyterhub/utils.py", line 289, in wait_for_http_server
    re = await exponential_backoff(
  File "/src/jupyterhub/jupyterhub/utils.py", line 221, in exponential_backoff
    ret = await maybe_future(pass_func(*args, **kwargs))
  File "/src/jupyterhub/jupyterhub/utils.py", line 266, in is_reachable
    r = await client.fetch(url, follow_redirects=False)
  File "/usr/local/lib/python3.9/site-packages/tornado/simple_httpclient.py", line 340, in run
    stream = await self.tcp_client.connect(
  File "/usr/local/lib/python3.9/site-packages/tornado/tcpclient.py", line 292, in connect
    stream = await stream.start_tls(
  File "/usr/local/lib/python3.9/site-packages/tornado/iostream.py", line 1246, in start_tls
    raise ValueError("IOStream is not idle; cannot convert to SSL")
ValueError: IOStream is not idle; cannot convert to SSL
```
</details>

Since this error is not catched in `is_reachable`, this will lead to a failed start attempt ([user.py](https://github.com/jupyterhub/jupyterhub/blob/0abe517faad95460889724e2599c33b0a1961bb0/jupyterhub/user.py#L961)).


### How to reproduce

<!-- Use this section to describe the steps that a user would take to experience this bug. -->
I was not able to reproduce this on a staging/devel setup. In our production setup it occured by 230 from 667 start attempts in the last 7 days, which leads to a rather unstable setup.
#### Expected behaviour

JupyterHub should be more robust while waiting for the notebook-server using `internal_ssl=True`, if the server takes a few seconds to start.

<!-- Tell us what you thought would happen. -->

#### Actual behaviour

JupyterHub stops the notebook-server, because of the thrown Exception by tornado.
<!-- Tell us what actually happens. -->

### Your personal set up

Zero2JupyterHub on K8s 1.26.11 , nodes OS: ubuntu-22.04
tornado version of in jupyterhub image: 6.4
<details><summary>Full environment</summary>
<!-- For reproduction, it's useful to have the full environment. For example, the output of `pip freeze` or `conda list` --->

```
aiohttp==3.8.1
aiosignal==1.2.0
alembic==1.13.1
async-generator==1.10
async-timeout==4.0.2
attrs==23.2.0
bcrypt==4.0.0
cachetools==5.3.3
certifi==2024.2.2
certipy==0.1.3
cffi==1.16.0
charset-normalizer==3.3.2
cryptography==42.0.5
datefinder==0.7.3
escapism==1.0.1
frozenlist==1.3.1
google-auth==2.28.2
greenlet==3.0.3
idna==3.6
importlib_metadata==7.0.2
Jinja2==3.1.3
jsonformatter==0.3.2
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter-jsc-custom==0.9.34
jupyter-telemetry==0.1.0
jupyterhub==4.0.2
jupyterhub-firstuseauthenticator==1.0.0
jupyterhub-forwardbasespawner==0.0.22
jupyterhub-hmacauthenticator==1.0
jupyterhub-idle-culler==1.2.1
jupyterhub-kubespawner==4.2.0
jupyterhub-ldapauthenticator==1.3.2
jupyterhub-ltiauthenticator==1.2.0
jupyterhub-nativeauthenticator==1.1.0
jupyterhub-outpostspawner==0.0.29
jupyterhub-tmpauthenticator==0.6
jupyterhub-unicorespawner==0.9.1
kubernetes==29.0.0
kubernetes-asyncio==24.2.2
ldap3==2.9.1
M2Crypto==0.41.0
Mako==1.3.2
MarkupSafe==2.1.5
multidict==6.0.2
mwoauth==0.3.8
nullauthenticator==1.0.0
oauthenticator==16.2.1
oauthlib==3.2.2
onetimepass==1.0.1
packaging==24.0
pamela==1.1.0
prometheus_client==0.20.0
psycopg2-binary==2.9.3
py-spy==0.3.14
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pycurl==7.45.1
PyJWT==2.8.0
PyMySQL==1.0.2
pyOpenSSL==24.1.0
pyparsing==3.0.9
pyrsistent==0.18.1
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
python-slugify==6.1.2
pytz==2024.1
pyunicore==1.0.0
PyYAML==6.0.1
referencing==0.34.0
regex==2023.12.25
requests==2.31.0
requests-oauthlib==1.4.0
rpds-py==0.18.0
rsa==4.9
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
six==1.16.0
SQLAlchemy==2.0.28
sqlalchemy-cockroachdb==1.4.3
statsd==3.3.0
text-unidecode==1.3
tornado==6.4
traitlets==5.14.2
typing_extensions==4.10.0
urllib3==2.2.1
websocket-client==1.7.0
yarl==1.8.1
zipp==3.18.1
```
</details>




Discussion:

Resolving patch:
diff --git a/jupyterhub/utils.py b/jupyterhub/utils.py
--- a/jupyterhub/utils.py
+++ b/jupyterhub/utils.py
@@ -295,6 +295,8 @@ async def is_reachable():
                 errno.ECONNRESET,
             }:
                 app_log.warning("Failed to connect to %s (%s)", url, e)
+        except Exception as e:
+            app_log.warning("Error while waiting for server %s (%s)", url, e)
         return False
 
     re = await exponential_backoff(


Test patch: