Repository: CTFd/CTFd

Issue:
SQL "Lost connection to MySQL server during query"
<!--
If this is a bug report please fill out the template below.

If this is a feature request please describe the behavior that you'd like to see.
-->

I have encountered the same bug as in #1395 and #467, which were closed because of lack of reproducibility, however, I've found a way to simulate the conditions under which the issue occurs, which tends to be in more complex deployment, such as my personal deployment under kubernetes.

**Environment**:

  - CTFd Version/Commit: 241467c
  - Operating System: Linux
  - Web Browser and Version: NA


**What happened?**

Occasionally, in production, the SQL database server goes down, for example, during upgrades, or because of network instability. If the user makes a request during this time we expect (and get) a 500 Internal Server error. However, once the SQL database comes back up, the first user request after this will return a 500 Internal Server error (which is not expected).

This issue doesn't often surface because (I think) if the page is cached, then the cached page will be returned, so the SQL server is not contacted in the first place. Therefore the error only appears when the page is not in the cache.

**What did you expect to happen?**

I expected that CTFd would automatically reconnect to the SQL database when it came back.

**How to reproduce your issue**

- Use the latest commit from master (from a fresh start)
- Remove the docker volume mount `.data/redis:/data` from docker-compose.yml
    - This makes it easier to clear the cache later
    - We don't need to do this, but without it makes reproducing the bug slower (as we then need to wait for the cache to clear)
- Start the containers
    ```
    $ docker-compose up -d
    ```
- Check the site is working
    ```
    $ curl http://localhost:8000
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
    <title>Redirecting...</title>
    <h1>Redirecting...</h1>
    <p>You should be redirected automatically to target URL: <a href="/setup">/setup</a>.  If not click the link.
    ```
- Remove the SQL container (to simulate downtime) and the Redis container (to hackily clear the cache)
    ```
    $ docker-compose rm -fsv db cache
    ```
- Bring both containers back up again immediately (notice in the output how the CTFd container has remained up through this)
    ```
    $ docker-compose up -d
    ```
- Wait a few seconds for the database to properly come back up (if the redis volume wasn't removed earlier, then wait a lot longer for the cache to properly clear)
- Check the site (it gives a 500)
    ```
    $ curl http://localhost:8000
    <html>
      <head>
        <title>Internal Server Error</title>
      </head>
      <body>
        <h1><p>Internal Server Error</p></h1>
      </body>
    </html>
    ```
- Check the site again (it works again)
    ```
    $ curl http://localhost:8000`
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
    <title>Redirecting...</title>
    <h1>Redirecting...</h1>
    <p>You should be redirected automatically to target URL: <a href="/setup">/setup</a>.  If not click the link.
    ```

**Any associated stack traces or error logs**

When the 500 internal error occurs we get the following stack trace:

```
ctfd_1   | ERROR [CTFd] Exception on / [GET]
ctfd_1   | Traceback (most recent call last):
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
ctfd_1   |     cursor, statement, parameters, context
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 581, in do_execute
ctfd_1   |     cursor.execute(statement, parameters)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
ctfd_1   |     result = self._query(query)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packlatest docker-composeages/pymysql/cursors.py", line 328, in _query
ctfd_1   |     conn.query(q)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
ctfd_1   |     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
ctfd_1   |     result.read()
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 1075, in read
ctfd_1   |     first_packet = self.connection._read_packet()
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 684, in _read_packet
ctfd_1   |     packet.check_error()
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/pymysql/protocol.py", line 220, in check_error
ctfd_1   |     err.raise_mysql_exception(self._data)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
ctfd_1   |     raise errorclass(errno, errval)
ctfd_1   | pymysql.err.InternalError: (1927, 'Connection was killed')
ctfd_1   | 
ctfd_1   | The above exception was the direct cause of the following exception:

... lots of very messy exception handling ommited for brevity ...

ctfd_1   | Traceback (most recent call last):
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 56, in handle
ctfd_1   |     self.handle_request(listener_name, req, client, addr)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 160, in handle_request
ctfd_1   |     addr)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
ctfd_1   |     respiter = self.wsgi(environ, resp.start_response)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
ctfd_1   |     return self.wsgi_app(environ, start_response)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
ctfd_1   |     return self.app(environ, start_response)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
ctfd_1   |     response = self.handle_exception(e)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/flask_restx/api.py", line 600, in error_router
ctfd_1   |     return original_handler(e)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1878, in handle_exception
ctfd_1   |     server_error = handler(server_error)
ctfd_1   |   File "/opt/CTFd/CTFd/errors.py", line 16, in general_error
ctfd_1   |     return render_template("errors/500.html"), 500
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 138, in render_template
ctfd_1   |     ctx.app.jinja_env.get_or_select_template(template_name_or_list),
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 930, in get_or_select_template
ctfd_1   |     return self.get_template(template_name_or_list, parent, globals)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 883, in get_template
ctfd_1   |     return self._load_template(name, self.make_globals(globals))
ctfd_1   |   File "/opt/CTFd/CTFd/__init__.py", line 85, in _load_template
ctfd_1   |     theme = str(utils.get_config("ctf_theme"))
ctfd_1   |   File "/opt/CTFd/CTFd/utils/__init__.py", line 46, in get_config
ctfd_1   |     value = _get_config(key)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/flask_caching/__init__.py", line 686, in decorated_function
ctfd_1   |     rv = f(*args, **kwargs)
ctfd_1   |   File "/opt/CTFd/CTFd/utils/__init__.py", line 27, in _get_config
ctfd_1   |     Configs.__table__.select().where(Configs.key == key)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
ctfd_1   |     return getattr(self.registry(), name)(*args, **kwargs)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1269, in execute
ctfd_1   |     clause, params or {}
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute
ctfd_1   |     return meth(self, multiparams, params)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
ctfd_1   |     return connection._execute_clauseelement(self, multiparams, params)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement
ctfd_1   |     distilled_params,
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1176, in _execute_context
ctfd_1   |     e, util.text_type(statement), parameters, None, None
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
ctfd_1   |     util.raise_from_cause(sqlalchemy_exception, exc_info)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
ctfd_1   |     reraise(type(exception), exception, tb=exc_tb, cause=cause)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
ctfd_1   |     raise value.with_traceback(tb)
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1171, in _execute_context
ctfd_1   |     conn = self._revalidate_connection()
ctfd_1   |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 457, in _revalidate_connection
ctfd_1   |     "Can't reconnect until invalid "
ctfd_1   | sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
ctfd_1   | [SQL: SELECT config.id, config.`key`, config.value 
ctfd_1   | FROM config 
ctfd_1   | WHERE config.`key` = %(key_1)s]
ctfd_1   | [parameters: [{}]]
ctfd_1   | 192.168.48.1 - - [24/Jun/2020:10:30:56 +0000] "GET / HTTP/1.1" 500 0 "-" "-"
```

**Notes**

I'm not sure what's actually causing this issue, but it looks like the CTFd server is trying to maintain a connection the SQL database across HTTP requests, which, when interrupted, causes an Exception to be thrown.

In the provided docker compose setup, it's not really an issue, as in a local docker setup across localhost, it's very unlikely that the database ever becomes inaccessible at the same time as the cache. Additionally, because the page will be requested often in a real-world setup, then the cache will almost always be full so the issue won't appear frequently. However, in more complex setups, it becomes more likely that the issue will present, especially if the redis container doesn't have an attached volume.

It would be really nice if this issue could be fixed, however, it can be worked around by tweaking the production setup so it doesn't appear as often. However, this issue should be good at least for archival reasons, to demonstrate how to reproduce the problem.


Discussion:
It appears that #1438 solves this issue.
Thanks for doing the research into reproducing this. Non-reproducible bugs will really never get any attention. 

Does the CTFd instance get stuck serving 500s at this point or does SQLAlchemy eventually create new connections? 
You only seem to get that one 500 error at the beginning, and it immediately at the next request, milliseconds later. So yeah, it does seem that SQLAlchemy recovers pretty quickly and starts making new connections.

Resolving patch:
diff --git a/CTFd/config.py b/CTFd/config.py
--- a/CTFd/config.py
+++ b/CTFd/config.py
@@ -340,6 +340,9 @@ class Config(object):
             "max_overflow": int(os.getenv("SQLALCHEMY_MAX_OVERFLOW", 0))
                 or int(empty_str_cast(config_ini["optional"]["SQLALCHEMY_MAX_OVERFLOW"], default=0))  # noqa: E131
                 or 20,  # noqa: E131
+            "pool_pre_ping": process_boolean_str(os.getenv("SQLALCHEMY_POOL_PRE_PING"))
+                or empty_str_cast(config_ini["optional"]["SQLALCHEMY_POOL_PRE_PING"])  # noqa: E131
+                or True,  # noqa: E131
         }
 
     """


Test patch: