Repository: jupyterhub/jupyterhub

Issue:
An active Jupyterhub session locks Postgres tables from writes.
Noticed while testing permissions issues on #83.  It looks like maintaining a single open session for the entire lifetime of the App results in transactions never being closed, which causes Postgres to write-lock any tables being used by Jupyterhub.  From reading the SQLAlchemy docs, it looks like the expected pattern for webapps is to create a `Session` class at global scope, but create and tear down instances of the session on each request.  



Discussion:
Looks like the current setup also completely locks the sqlite database.  When running with an active jupyterhub process, I get:

```
(jupyterhub)[~/jupyterhub]@(master:db5cf9cf99)$ sqlite jupyterhub.sqlite
SQLite version 2.8.17
Enter ".help" for instructions
sqlite> .tables
Error: database is locked
sqlite>
```

Working on a fix for this now.

Thanks for noting these and updating rapidly. You said you're working a fix right as I was typing this.

Relevant SO post from one of the SQLAlchemy maintainers:
http://stackoverflow.com/questions/12223335/sqlalchemy-creating-vs-reusing-a-session

Makes sense, looking forward to the patch.

More documentation excerpts: http://docs.sqlalchemy.org/en/rel_0_9/dialects/sqlite.html#sqlite-concurrency

Database Locking Behavior / Concurrency

SQLite is not designed for a high level of write concurrency. The database itself, being a file, is locked completely during write operations within transactions, meaning exactly one “connection” (in reality a file handle) has exclusive access to the database during this period - all other “connections” will be blocked during this time.

The Python DBAPI specification also calls for a connection model that is always in a transaction; there is no connection.begin() method, only connection.commit() and connection.rollback(), upon which a new transaction is to be begun immediately. This may seem to imply that the SQLite driver would in theory allow only a single filehandle on a particular database file at any time; however, there are several factors both within SQlite itself as well as within the pysqlite driver which loosen this restriction significantly.

However, no matter what locking modes are used, SQLite will still always lock the database file once a transaction is started and DML (e.g. INSERT, UPDATE, DELETE) has at least been emitted, and this will block other transactions at least at the point that they also attempt to emit DML. By default, the length of time on this block is very short before it times out with an error.

This behavior becomes more critical when used in conjunction with the SQLAlchemy ORM. SQLAlchemy’s Session object by default runs within a transaction, and with its autoflush model, may emit DML preceding any SELECT statement. This may lead to a SQLite database that locks more quickly than is expected. The locking mode of SQLite and the pysqlite driver can be manipulated to some degree, however it should be noted that achieving a high degree of write-concurrency with SQLite is a losing battle.

For more information on SQLite’s lack of write concurrency by design, please see Situations Where Another RDBMS May Work Better - High Concurrency near the bottom of the page.

The following subsections introduce areas that are impacted by SQLite’s file-based architecture and additionally will usually require workarounds to work when using the pysqlite driver.

We should make sure to note that the maximum level of concurrency expected for JupyterHub is one, especially with SQLite. I know you are working on shared-state Hubs with Postgres, and that _may_ be feasible. I don't think we have any need to make concurrent JupyterHub sessions work with SQLite.

Taking Min's comment in another direction, @ssanderson if you take us down the path of doing Postgres well (including a Docker container for the Postgres setup), I'd be _super_ pleased.

@minrk working on tests for this change; is there a reason there's a 5 second wait on the mock app created in `tests/mocking.py`?  It's super annoying to wait 5 seconds for each run...

What 5 second wait? There's a wait on an event with a _timeout_ of 5 seconds, but that should only be hit if something's not working. It only takes me 7 seconds to run the whole test suite.

Ah. The failure I was looking at is currently happening inside the app initialization, which causes the full duration of the wait to happen.

It should probably raise if the timeout is hit, instead of just moving right along after making you wait. Sorry about that.

WIP fix for this is at https://github.com/jupyter/jupyterhub/pull/85.  I'm not super excited about the first cut.  Given the amount of manual management of the Session that this requires of the Hub, I'm inclined to think we'd be better off just using `Engine` and `Connection` directly.  The one thing I'm not sure of w/r/t that approach is how it interacts with Python-level objects that aren't persisted in the database but are stored on the models.  This is most noticeably true of `User.spawner`.

More investigation this morning:  the offending locks  that are causing this issue are being created in `CookieAPIHandler.get`.  When we do

```
orm_token = self.db.query(orm.CookieToken).filter(orm.CookieToken.token == token).first()
```

that actually creates an exclusive lock on the orm_token (and by extension, the user row to which it's related) that's never released because we don't commit or rollback the db session.  When running with SQLite, it locks the entire database from any externel connections.  I **think** this is happening because, even though the entire query has been iterated by the call to `first()`, the object has join attributes that haven't been loaded, so SQLALchemy doesn't consider the transaction completed.  

A simple fix to this is to put a call to one of `self.db.close()`, `self.db.rollback()` or `self.db.commit()` in `BaseHandler.finish`.  This is more dangerous than it sounds though, because the fact that the Hub's maintenance loop shares a Session with the tornado app's handlers means that the close/rollback could splice into the transaction of a running `gen.coroutine`.  (This might happen for example at a yield point like https://github.com/jupyter/jupyterhub/blob/master/jupyterhub/orm.py#L321.)


Resolving patch:
diff --git a/jupyterhub/app.py b/jupyterhub/app.py
--- a/jupyterhub/app.py
+++ b/jupyterhub/app.py
@@ -262,6 +262,7 @@ def _db_url_changed(self, name, old, new):
         help="log all database transactions. This has A LOT of output"
     )
     db = Any()
+    session_factory = Any()
     
     admin_users = Set(config=True,
         help="""set of usernames of admin users
@@ -364,9 +365,13 @@ def init_db(self):
         """Create the database connection"""
         self.log.debug("Connecting to db: %s", self.db_url)
         try:
-            self.db = orm.new_session(self.db_url, reset=self.reset_db, echo=self.debug_db,
+            self.session_factory = orm.new_session_factory(
+                self.db_url,
+                reset=self.reset_db,
+                echo=self.debug_db,
                 **self.db_kwargs
             )
+            self.db = self.session_factory()
         except OperationalError as e:
             self.log.error("Failed to connect to db: %s", self.db_url)
             self.log.debug("Database error was:", exc_info=True)
diff --git a/jupyterhub/handlers/base.py b/jupyterhub/handlers/base.py
--- a/jupyterhub/handlers/base.py
+++ b/jupyterhub/handlers/base.py
@@ -58,6 +58,11 @@ def proxy(self):
     def authenticator(self):
         return self.settings.get('authenticator', None)
 
+    def finish(self, *args, **kwargs):
+        """Roll back any uncommitted transactions from the handler."""
+        self.db.rollback()
+        super(BaseHandler, self).finish(*args, **kwargs)
+
     #---------------------------------------------------------------
     # Login and cookie-related
     #---------------------------------------------------------------
diff --git a/jupyterhub/orm.py b/jupyterhub/orm.py
--- a/jupyterhub/orm.py
+++ b/jupyterhub/orm.py
@@ -381,17 +381,20 @@ class CookieToken(Token, Base):
     __tablename__ = 'cookie_tokens'
 
 
-def new_session(url="sqlite:///:memory:", reset=False, **kwargs):
+def new_session_factory(url="sqlite:///:memory:", reset=False, **kwargs):
     """Create a new session at url"""
     if url.startswith('sqlite'):
         kwargs.setdefault('connect_args', {'check_same_thread': False})
-    kwargs.setdefault('poolclass', StaticPool)
+
+    if url.endswith(':memory:'):
+        # If we're using an in-memory database, ensure that only one connection
+        # is ever created.
+        kwargs.setdefault('poolclass', StaticPool)
+
     engine = create_engine(url, **kwargs)
-    Session = sessionmaker(bind=engine)
-    session = Session()
     if reset:
         Base.metadata.drop_all(engine)
     Base.metadata.create_all(engine)
-    return session
-
 
+    session_factory = sessionmaker(bind=engine)
+    return session_factory


Test patch:
diff --git a/jupyterhub/tests/conftest.py b/jupyterhub/tests/conftest.py
--- a/jupyterhub/tests/conftest.py
+++ b/jupyterhub/tests/conftest.py
@@ -22,7 +22,7 @@ def db():
     """Get a db session"""
     global _db
     if _db is None:
-        _db = orm.new_session('sqlite:///:memory:', echo=True)
+        _db = orm.new_session_factory('sqlite:///:memory:', echo=True)()
         user = orm.User(
             name=getuser_unicode(),
             server=orm.Server(),
diff --git a/jupyterhub/tests/mocking.py b/jupyterhub/tests/mocking.py
--- a/jupyterhub/tests/mocking.py
+++ b/jupyterhub/tests/mocking.py
@@ -1,6 +1,8 @@
 """mock utilities for testing"""
 
+import os
 import sys
+from tempfile import NamedTemporaryFile
 import threading
 
 try:
@@ -56,13 +58,12 @@ def authenticate(self, *args, **kwargs):
 
 class MockHubApp(JupyterHubApp):
     """HubApp with various mock bits"""
-    
+
+    db_file = None
+
     def _ip_default(self):
         return 'localhost'
     
-    def _db_url_default(self):
-        return 'sqlite:///:memory:'
-
     def _authenticator_class_default(self):
         return MockPAMAuthenticator
     
@@ -71,8 +72,10 @@ def _spawner_class_default(self):
     
     def _admin_users_default(self):
         return {'admin'}
-    
+
     def start(self, argv=None):
+        self.db_file = NamedTemporaryFile()
+        self.db_url = 'sqlite:///' + self.db_file.name
         evt = threading.Event()
         def _start():
             self.io_loop = IOLoop.current()
@@ -91,6 +94,6 @@ def _start():
         evt.wait(timeout=5)
     
     def stop(self):
+        self.db_file.close()
         self.io_loop.add_callback(self.io_loop.stop)
         self._thread.join()
-
diff --git a/jupyterhub/tests/test_api.py b/jupyterhub/tests/test_api.py
--- a/jupyterhub/tests/test_api.py
+++ b/jupyterhub/tests/test_api.py
@@ -7,6 +7,30 @@
 from ..utils import url_path_join as ujoin
 from .. import orm
 
+
+def check_db_locks(func):
+    """
+    Decorator for test functions that verifies no locks are held on the
+    application's database upon exit by creating and dropping a dummy table.
+
+    Relies on an instance of JupyterhubApp being the first argument to the
+    decorated function.
+    """
+
+    def new_func(*args, **kwargs):
+        retval = func(*args, **kwargs)
+
+        app = args[0]
+        temp_session = app.session_factory()
+        temp_session.execute('CREATE TABLE dummy (foo INT)')
+        temp_session.execute('DROP TABLE dummy')
+        temp_session.close()
+
+        return retval
+
+    return new_func
+
+
 def find_user(db, name):
     return db.query(orm.User).filter(orm.User.name==name).first()
     
@@ -28,6 +52,7 @@ def auth_header(db, name):
         token = user.api_tokens[0]
     return {'Authorization': 'token %s' % token.token}
 
+@check_db_locks
 def api_request(app, *api_path, **kwargs):
     """Make an API request"""
     base_url = app.hub.server.url
@@ -70,7 +95,6 @@ def test_auth_api(app):
         headers={'Authorization': 'token: %s' % cookie_token.token},
     )
     assert r.status_code == 403
-    
 
 def test_get_users(app):
     db = app.db
@@ -179,4 +203,3 @@ def test_spawn(app, io_loop):
     assert 'pid' not in user.state
     status = io_loop.run_sync(user.spawner.poll)
     assert status == 0
-    
\ No newline at end of file