Repository: CTFd/CTFd

Issue:
Use sqlalchemy Python script to test availability of database.
**Environment**:

  - CTFd Version/Commit:

2.0.0 branch

  - Operating System:

MacOS X, but building/deploying container images.

  - Web Browser and Version:

N/A

**What happened?**

The check using ``mysqladmin`` in ``docker-entrypoint.sh`` will fail if the target system only has MySQL libs package and doesn't have MySQL clients package.

Further, the wiki says:

> My personal suggestion is MySQL or MariaDB, but PostgreSQL will work just fine.

The ``docker-entrypoint.sh`` script is hard wired to MySQL and expects MySQL clients package to be installed. Would be nice that if also want to use PostgreSQL and only provide alternate ``Dockerfile`` and not change anything else, that this would work and didn't have to rewrite it. Similarly, if use alternate build system like S2I for OpenShift, that can reuse existing scripts unchanged.

**What did you expect to happen?**

Would hope database check is database agnostic and work with only database libs available.

**How to reproduce your issue**

In my case hit it as building from source using S2I (Source-to-Image) builder in OpenShift. The Python base images do not include MySQL clients package.

**Any associated stack traces or error logs**

```
Waiting for ctfd-mariadb to be ready
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
 ...
```

**Solution**

Suggest database agnostic check be used which uses small Python script and ``sqlalchemy``.

```
import os
from sqlalchemy import create_engine
database = create_engine(os.getenv('DATABASE_URL'))
database.execute('select 1')
```

This has added benefit that also checks that required Python packages installed correctly and work before starting up the application.

It should also technically work for SQLite if someone wanted to override ``DATABASE_URL`` and use SQLite with file system path, to directory in a mounted persistent volume, rather than have database created in default location of source code directories where not possible to mounted a persistent volume. Although in SQLite case the database file would technically need to exist first.



Discussion:
The ``docker-entrypoint.sh`` script could use:

```
function database_ready() {
    python -c "import os
from sqlalchemy import create_engine
database = create_engine(os.getenv('DATABASE_URL'))
database.execute('select 1')" > /dev/null 2>&1
}

# Check that the database is available
if [ -n "$DATABASE_URL" ]; then
    database=`echo $DATABASE_URL | awk -F[@//] '{print $4}'`
    echo "Waiting for $database to be ready"
    while ! database_ready; do
        # Show some progress
        echo -n '.';
        sleep 1;
    done
    echo "$database is ready"
    # Give it another second.
    sleep 1;
fi
```

There is another existing problem with that loop for checking the database though.

```
        echo -n '.';
```

This attempt at showing progress will not necessarily work when run in a container as display of output in log system of the container platform can be line buffered. This means you will only see the dots all at once when it finally succeeds, or possibly if some line buffering limit is reached. Would be better to use:

```
        echo 'wait 1 sec for database';
```

and accept that it will display something every second, one per line. If too noisy, increase sleep time to 2 seconds to at least halve it.

This is vaguely reasonable but in the wiki (which needs MAJOR rewriting): 

> Note: Due to development constraints the officially supported database is MySQL/MariaDB although development efforts will focus on feature parity between SQLite, MySQL/MariaDB, and Postgres.

Also SQLAlchemy-Utils has a built in helper function for this: https://sqlalchemy-utils.readthedocs.io/en/latest/database_helpers.html?highlight=database_exists#database-exists




Then can use:

```
function database_ready() {
    python -c "import os, sys
from sqlalchemy_utils.functions import database_exists
database_exists(os.getenv('DATABASE_URL')) or sys.exit(1)" > /dev/null 2>&1
}
```

The docs for ``database_exists()`` leave a bit to be desired because it isn't a simple case of returning ``True`` or ``False``. It can still raise an exception such as in the case of not being able to route to the database. So still have to throw away output to avoid the exception tracebacks filling the logs.
Hi @GrahamDumpleton 
Would you please create a PR from your fix?
I have the same problem and started to fix it, but discovered that you already wrote one...

@tmarlok88 @GrahamDumpleton if either of you wants to PR this I can merge it in. 
I decided to not move forward with the PR I was creating because discussion on other issues was going in a direction where it was unlikely the PR would be accepted. In the end I completely abandoned what I was doing, but you can find the changes I was making in:

* https://github.com/GrahamDumpleton/CTFd/commits/2.0.0

Instead of that I created a completely separate repository with did the build and setup in a way which better aligned with best practices for running in containers and my use case, and wasn't dependent on getting any PRs accepted into the CTFd repository itself. That can be found at:

* https://github.com/openshift-evangelists/ctfd-quickstart

Suggest you take anything you want from the first repository sooner rather than later, as I really should delete that first repository to avoid it causing any confusion, and make sure no one depends on it.
Neither of the proposed scripts actually replicate the original purpose of `mysqladmin ping`. Both of these functions only work if the database inside of the database exists. 


I don't see a great way to solve this and we don't officially support Postgres. If someone wants to figure this out I'm happy to merge in the fix but it should actually work. 
When deploying images for databases in docker containers, you would usually tell it the name of the database to create, eg., using ``MYSQL_DATABASE`` environment variable. You don't want a manual step to create it yourself as then you can't have an out of the box deploy and things would be more difficult to setup. So don't understand why "only work if the database inside of the database exists" would be an issue.
CTFd manually concerns itself with the creation of the database because it prefers to set things properly if it doesn't already exist. Not everyone deploys everything using Docker containers. 

We only need to see if the database server is available, CTFd takes care of the rest. If the script doesn't work beforehand then you get scenarios where it just never works when it would have before. 
let me bring this ancient issue back alive. I couldn't stand the `default-mysql-client` package standing in the way between me and the "totally-no-package-installed image"

```python
import sqlalchemy
from CTFd.config import Config

sqlalchemy.create_engine(Config.DATABASE_URL).raw_connection().ping()
```

I think this does the trick. it's not a perfect copy of the `mysql_ping` C API though (it DOES logs into the database).

~~btw, I think the `docker-entrypoint.sh` should also read configs from `config.ini`.~~
how about letting the pinging script do the job of writing the secret key file, so the entrypoint script wouldn't need to read config items anymore (`entrypoint.py`)?

Resolving patch:
diff --git a/ping.py b/ping.py
new file mode 100644
--- /dev/null
+++ b/ping.py
@@ -0,0 +1,34 @@
+"""
+Script for checking that a database server is available.
+Essentially a cross-platform, database agnostic mysqladmin.
+"""
+import time
+
+from sqlalchemy import create_engine
+from sqlalchemy.engine.url import make_url
+
+from CTFd.config import Config
+
+url = make_url(Config.DATABASE_URL)
+
+# Ignore sqlite databases
+if url.drivername.startswith("sqlite"):
+    exit(0)
+
+# Null out the database so raw_connection doesnt error if it doesnt exist
+# CTFd will create the database if it doesnt exist
+url.database = None
+
+# Wait for the database server to be available
+engine = create_engine(url)
+print(f"Waiting for {url} to be ready")
+while True:
+    try:
+        engine.raw_connection()
+        break
+    except Exception:
+        print(".", end="", flush=True)
+        time.sleep(1)
+
+print(f"{url} is ready")
+time.sleep(1)


Test patch: