Repository: jupyterhub/jupyterhub

Issue:
After db connection times out once, hub is dysfunctional
Our mysql connection has an idle timeout, and once that happens, the hub doesn't work anymore until it is restarted.

```
[E 2016-03-03 14:04:47.056 JupyterHub ioloop:629] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper
 at 0x7f2eddf7e840>, <tornado.concurrent.Future object at 0x7f2eec03ce10>)
    Traceback (most recent call last):
      File "/srv/jupyterhub/lib/python3.4/site-packages/tornado/ioloop.py", line 600, in _run_callback
        ret = callback()
      File "/srv/jupyterhub/lib/python3.4/site-packages/tornado/stack_context.py", line 275, in null_wrapper
        return fn(*args, **kwargs)
      File "/srv/jupyterhub/lib/python3.4/site-packages/tornado/ioloop.py", line 615, in <lambda>
        self.add_future(ret, lambda f: f.result())
      File "/srv/jupyterhub/lib/python3.4/site-packages/jupyterhub/app.py", line 1038, in update_last_activity
        routes = yield self.proxy.get_routes()
      File "/srv/jupyterhub/lib/python3.4/site-packages/jupyterhub/orm.py", line 218, in get_routes
        resp = yield self.api_request('', client=client)
      File "/srv/jupyterhub/lib/python3.4/site-packages/jupyterhub/orm.py", line 162, in api_request
        url = url_path_join(self.api_server.url, path)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
        return self.impl.get(instance_state(instance), dict_)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/attributes.py", line 583, in get
        value = self.callable_(state, passive)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/strategies.py", line 514, in _load_for_state
        passive
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/strategies.py", line 550, in _get_ident_for_use_get
        for pk in self.mapper.primary_key
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/strategies.py", line 550, in <listcomp>
        for pk in self.mapper.primary_key
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/mapper.py", line 2467, in _get_state_attr_by_column
        return state.manager[prop.key].impl.get(state, dict_, passive=passive)
    File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/attributes.py", line 578, in get
        value = state._load_expired(state, passive)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/state.py", line 474, in _load_expired
        self.manager.deferred_scalar_loader(self, toload)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 664, in load_scalar_attributes
        only_load_props=attribute_names)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
        return q.one()
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2693, in one
        ret = list(self)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
        return self._execute_and_instances(context)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 914, in execute
        return meth(self, multiparams, params)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
        compiled_sql, distilled_params
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
        context)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
        exc_info
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb, cause=cause)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 183, in reraise
        raise value.with_traceback(tb)
      File "/srv/jupyterhub/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
        context)
      File "/srv/jupyterhub/lib/python3.4/site-packages/pymysql/cursors.py", line 158, in execute
        result = self._query(query)
      File "/srv/jupyterhub/lib/python3.4/site-packages/pymysql/cursors.py", line 308, in _query
        conn.query(q)
      File "/srv/jupyterhub/lib/python3.4/site-packages/pymysql/connections.py", line 820, in query
        self._affected_rows = self._read_query_result(unbuffered=unbuffered)
      File "/srv/jupyterhub/lib/python3.4/site-packages/pymysql/connections.py", line 1002, in _read_query_result
        result.read()
      File "/srv/jupyterhub/lib/python3.4/site-packages/pymysql/connections.py", line 1285, in read
        first_packet = self.connection._read_packet()
      File "/srv/jupyterhub/lib/python3.4/site-packages/pymysql/connections.py", line 946, in _read_packet
        packet_header = self._read_bytes(4)
      File "/srv/jupyterhub/lib/python3.4/site-packages/pymysql/connections.py", line 982, in _read_bytes
        2013, "Lost connection to MySQL server during query")
    sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: 'SELECT proxies.id AS proxies_id, proxies._public_server_id AS proxies__public_server_id, proxies._api_server_id AS proxies__api_server_id \nFROM proxies \nWHERE proxies.id = %(param_1)s'] [parameters: {'param_1': 1}]
```



Discussion:
```
c.JupyterHub.db_kwargs = {
    'pool_recycle': 60  # Do not keep connections for more than one minute
}
```

might be all I need! Have deployed, let's see how that goes.

That seems to do the trick, but this is basically essential for all mysql installs. Perhaps we could detect that somehow and autoset this? @minrk thoughts?

Sure. What's the default idle timeout?

60s idle timeout is the most common wait_timeout configuration in my
experience.
On Mar 7, 2016 11:56 PM, "Min RK" notifications@github.com wrote:

> Sure. What's the default idle timeout?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/jupyter/jupyterhub/issues/450#issuecomment-193653617.


Resolving patch:
diff --git a/jupyterhub/orm.py b/jupyterhub/orm.py
--- a/jupyterhub/orm.py
+++ b/jupyterhub/orm.py
@@ -378,6 +378,8 @@ def new_session_factory(url="sqlite:///:memory:", reset=False, **kwargs):
     """Create a new session at url"""
     if url.startswith('sqlite'):
         kwargs.setdefault('connect_args', {'check_same_thread': False})
+    elif url.startswith('mysql'):
+        kwargs.setdefault('pool_recycle', 60)
 
     if url.endswith(':memory:'):
         # If we're using an in-memory database, ensure that only one connection


Test patch: