Repository: cookiecutter/cookiecutter-django
Pull request: Add Docker support to Travis and GitLab CI
[//]: # (Thank you for helping us out: your efforts mean great deal to the project and the community as a whole!)

[//]: # (Before you proceed:)

[//]: # (1. Make sure to add yourself to `CONTRIBUTORS.rst` through this PR provided you're contributing here for the first time)
[//]: # (2. Don't forget to update the `docs/` presuming others would benefit from a concise description of whatever that you're proposing)


## Description

Docker support for testing CI

* Added lint testing in Travis
* Celerybeat start files wait for table migration so that errors don't occur if a migration does not happen post-build


## Rationale

* Currently running tests is only for non-Docker projects. Otherwise, Docker Travis automatically fail since no .env is loaded.

## Use case(s) / visualization(s)


Don't need to self-configure the project for testing.

Fixes #2100 


Problem statement:
Travis + Docker result in broken build on CI
## What happened?

Try to generate a project with Docker and Tarvis enabled, the tarvis config is not suitable and fails the build:


<details>
  <summary>Build logs</summary>
  
```
[0Ktravis_fold:end:install
[0Ktravis_time:start:183a4568
[0K$ pytest
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/environ/environ.py", line 273, in get_value
    value = self.ENVIRON[var]
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'DATABASE_URL'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.7/bin/pytest", line 10, in <module>
    sys.exit(main())
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/_pytest/config/__init__.py", line 63, in main
    config = _prepareconfig(args, plugins)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/_pytest/config/__init__.py", line 207, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/_pytest/helpconfig.py", line 94, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/_pytest/config/__init__.py", line 687, in pytest_cmdline_parse
    self.parse(args)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/_pytest/config/__init__.py", line 895, in parse
    self._preparse(args, addopts=addopts)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/_pytest/config/__init__.py", line 851, in _preparse
    early_config=self, args=args, parser=self._parser
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pytest_django/plugin.py", line 333, in pytest_load_initial_conftests
    dj_settings.DATABASES
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
    self._setup(name)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/travis/build/browniebroke/cc-dj-tests/config/settings/test.py", line 5, in <module>
    from .base import *  # noqa
  File "/home/travis/build/browniebroke/cc-dj-tests/config/settings/base.py", line 44, in <module>
    DATABASES = {"default": env.db("DATABASE_URL")}
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/environ/environ.py", line 204, in db_url
    return self.db_url_config(self.get_value(var, default=default), engine=engine)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/environ/environ.py", line 277, in get_value
    raise ImproperlyConfigured(error_msg)
django.core.exceptions.ImproperlyConfigured: Set the DATABASE_URL environment variable
travis_time:end:183a4568:start=1559838789069844257,finish=1559838789825801676,duration=755957419
[0K[31;1mThe command "pytest" exited with 1.[0m



Done. Your build exited with 1.
```
</details>

We should provide a suitable config when Docker is selected with Travis.

## How to fix this?

Generate a project with the 2 options enabled, push it to Github with Travis and make the build pass by running the appropriate docker commands instead of bare metals ones.

Hints from issue discussion:
I reverted the database config referenced by @wandering-tales on #1541. 
And added this to .travis.yml
```
before_script:
      - psql -c 'create database {{cookiecutter.project_slug}};' -U postgres
```

After making these changes my travis build passed. 
I have a version of this that works for gitlab ci. Not sure if that would work with travis though.
I would be interested to see your gitlab config
This works with docker-in-docker and probably requires docker to run in privileged mode, and requires an image with docker-compose (see below).

```
image: docker:latest
services:
  - docker:dind

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_DRIVER: overlay2

stages:
  - build
  - start
  - test
  
build:
  stage: build
  image: tiangolo/docker-with-compose
  script:
    - docker-compose -f local.yml build

start:
  stage: start
  image: tiangolo/docker-with-compose
  script:
    - docker-compose -f local.yml up -d
    - docker-compose -f local.yml down -v

test:
  stage: test
  image: tiangolo/docker-with-compose
  script:
    - docker-compose run django coverage run -m pytest
  allow_failure: true
```
> I reverted the database config referenced by @wandering-tales on #1541.
> And added this to .travis.yml
> 
> ```
> before_script:
>       - psql -c 'create database {{cookiecutter.project_slug}};' -U postgres
> ```
> 
> After making these changes my travis build passed.

Actually `pytest` should take care by itself to create a database for testing, if the configured one is missing.

In my `.travis.yml` I didn't have to explicitly create the database. 
Are you sure pytest creates psql databases? Maybe it just falls back to sqlite?
It could never be a SQLite database as in the `DATABASE_URL` setting I'm explicitly using the standard `libpq` connection string to connect to a PostgreSQL database, as I mentioned in https://github.com/pydanny/cookiecutter-django/issues/1541#issuecomment-500194456:

```python
DATABASES = {
    'default': env.db('DATABASE_URL', default='postgres:///{{cookiecutter.project_slug}}'),
```
Therefore the related test database backend is automatically inherited from the default one and can't be anything else (e.g. SQLite).

Besides, if you don't explicitly enable PostgreSQL server in Travis CI build environment the builds will always fail because all the connection attempts to PostgreSQL database fail. (`psycopg2.OperationalIssue`).
Hi!
Here is a project-local solution: add
`DATABASE_URL=postgresql://localhost/{{cookiecutter.project_slug}} `
as environment variable in Travis repo settings.
> Are you sure pytest creates psql databases? Maybe it just falls back to sqlite?

@step21 did you ever get an answer to this?
> > Are you sure pytest creates psql databases? Maybe it just falls back to sqlite?
> 
> @step21 did you ever get an answer to this?

No, sorry.

Commit message: Update beat start files with better? SQL statement

Patch to review:
diff --git a/tests/test_cookiecutter_generation.py b/tests/test_cookiecutter_generation.py
index 2c7e6dc43b..8785d7ce19 100755
--- a/tests/test_cookiecutter_generation.py
+++ b/tests/test_cookiecutter_generation.py
@@ -181,7 +181,15 @@ def test_travis_invokes_pytest(cookies, context):
 
     with open(f"{result.project}/.travis.yml", "r") as travis_yml:
         try:
-            assert yaml.load(travis_yml, Loader=yaml.FullLoader)["script"] == ["pytest"]
+            yml = yaml.load(travis_yml, Loader=yaml.FullLoader)["jobs"]["include"]
+            assert yml[0]["script"] == ["flake8"]
+
+            if context.get("use_docker") == "y":
+                assert yml[1]["script"] == [
+                    "docker-compose -f local.yml run django pytest"
+                ]
+            else:
+                assert yml[1]["script"] == ["pytest"]
         except yaml.YAMLError as e:
             pytest.fail(e)
 
diff --git a/{{cookiecutter.project_slug}}/.gitlab-ci.yml b/{{cookiecutter.project_slug}}/.gitlab-ci.yml
index 611ff202ff..4d67f8a1e6 100644
--- a/{{cookiecutter.project_slug}}/.gitlab-ci.yml
+++ b/{{cookiecutter.project_slug}}/.gitlab-ci.yml
@@ -22,8 +22,19 @@ flake8:
 pytest:
   stage: test
   image: python:3.7
+  {% if cookiecutter.use_docker == 'y' -%}
   tags:
     - docker
+  services:
+    - docker
+  before_script:
+    - docker-compose -f local.yml build
+    - docker-compose -f local.yml up -d
+  script:
+    - docker-compose -f local.yml run django pytest
+  {%- else %}
+  tags:
+    - python
   services:
     - postgres:11
   variables:
@@ -34,4 +45,5 @@ pytest:
 
   script:
     - pytest
+  {%- endif %}
 
diff --git a/{{cookiecutter.project_slug}}/.travis.yml b/{{cookiecutter.project_slug}}/.travis.yml
index 6f75969705..7f90b5f5ec 100644
--- a/{{cookiecutter.project_slug}}/.travis.yml
+++ b/{{cookiecutter.project_slug}}/.travis.yml
@@ -1,17 +1,43 @@
 dist: xenial
-services:
-  - postgresql
-before_install:
-  - sudo apt-get update -qq
-  - sudo apt-get install -qq build-essential gettext python-dev zlib1g-dev libpq-dev xvfb
-  - sudo apt-get install -qq libjpeg8-dev libfreetype6-dev libwebp-dev
-  - sudo apt-get install -qq graphviz-dev python-setuptools python3-dev python-virtualenv python-pip
-  - sudo apt-get install -qq firefox automake libtool libreadline6 libreadline6-dev libreadline-dev
-  - sudo apt-get install -qq libsqlite3-dev libxml2 libxml2-dev libssl-dev libbz2-dev wget curl llvm
+
 language: python
 python:
   - "3.8"
-install:
-  - pip install -r requirements/local.txt
-script:
-  - "pytest"
+
+services:
+  - {% if cookiecutter.use_docker == 'y' %}docker{% else %}postgresql{% endif %}
+jobs:
+  include:
+    - name: "Linter"
+      before_script:
+        - pip install -q flake8
+      script:
+        - "flake8"
+
+    - name: "Django Test"
+    {% if cookiecutter.use_docker == 'y' -%}
+      before_script:
+        - docker-compose -v
+        - docker -v
+        - docker-compose -f local.yml build
+        - docker-compose -f local.yml up -d
+      script:
+        - "docker-compose -f local.yml run django pytest"
+      after_failure:
+        - docker-compose -f local.yml logs
+    {%- else %}
+      before_install:
+        - sudo apt-get update -qq
+        - sudo apt-get install -qq build-essential gettext python-dev zlib1g-dev libpq-dev xvfb
+        - sudo apt-get install -qq libjpeg8-dev libfreetype6-dev libwebp-dev
+        - sudo apt-get install -qq graphviz-dev python-setuptools python3-dev python-virtualenv python-pip
+        - sudo apt-get install -qq firefox automake libtool libreadline6 libreadline6-dev libreadline-dev
+        - sudo apt-get install -qq libsqlite3-dev libxml2 libxml2-dev libssl-dev libbz2-dev wget curl llvm
+      language: python
+      python:
+        - "3.8"
+      install:
+        - pip install -r requirements/local.txt
+      script:
+        - "pytest"
+    {%- endif %}
diff --git a/{{cookiecutter.project_slug}}/compose/local/django/celery/beat/start b/{{cookiecutter.project_slug}}/compose/local/django/celery/beat/start
index c04a7365e9..477dfca305 100644
--- a/{{cookiecutter.project_slug}}/compose/local/django/celery/beat/start
+++ b/{{cookiecutter.project_slug}}/compose/local/django/celery/beat/start
@@ -5,4 +5,41 @@ set -o nounset
 
 
 rm -f './celerybeat.pid'
+
+postgres_ready() {
+python << END
+import sys
+import psycopg2
+try:
+    conn = psycopg2.connect(
+        dbname="${POSTGRES_DB}",
+        user="${POSTGRES_USER}",
+        password="${POSTGRES_PASSWORD}",
+        host="${POSTGRES_HOST}",
+        port="${POSTGRES_PORT}",
+    )
+    # Check if table exists yet.
+    # If not, wait for docker-compose up to migrate all tables.
+    cur = conn.cursor()
+    cur.execute(
+        "select exists(select * from information_schema.tables where table_name=%s)",
+        ('django_celery_beat_periodictask',)
+    )
+    if cur.fetchone()[0] == 1:
+        cur.close()
+        sys.exit(0)
+    else:
+        cur.close()
+        sys.exit(-1)
+except psycopg2.OperationalError:
+    sys.exit(-1)
+
+END
+}
+until postgres_ready; do
+  >&2 echo 'Waiting for celerybeat models to be migrated...'
+  sleep 1
+done
+>&2 echo 'PostgreSQL is ready'
+
 celery -A config.celery_app beat -l INFO
diff --git a/{{cookiecutter.project_slug}}/compose/production/django/celery/beat/start b/{{cookiecutter.project_slug}}/compose/production/django/celery/beat/start
index 20b93123ae..0cc8ae8a44 100644
--- a/{{cookiecutter.project_slug}}/compose/production/django/celery/beat/start
+++ b/{{cookiecutter.project_slug}}/compose/production/django/celery/beat/start
@@ -4,5 +4,41 @@ set -o errexit
 set -o pipefail
 set -o nounset
 
+postgres_ready() {
+python << END
+import sys
+import psycopg2
+try:
+    conn = psycopg2.connect(
+        dbname="${POSTGRES_DB}",
+        user="${POSTGRES_USER}",
+        password="${POSTGRES_PASSWORD}",
+        host="${POSTGRES_HOST}",
+        port="${POSTGRES_PORT}",
+    )
+    # Check if table exists yet.
+    # If not, wait for docker-compose up to migrate all tables.
+    cur = conn.cursor()
+    cur.execute(
+        "select exists(select * from information_schema.tables where table_name=%s)",
+        ('django_celery_beat_periodictask',)
+    )
+    if cur.fetchone()[0] == 1:
+        cur.close()
+        sys.exit(0)
+    else:
+        cur.close()
+        sys.exit(-1)
+except psycopg2.OperationalError:
+    sys.exit(-1)
+
+sys.exit(0)
+END
+}
+until postgres_ready; do
+  >&2 echo 'Waiting for celerybeat models to be migrated...'
+  sleep 1
+done
+>&2 echo 'PostgreSQL is ready'
 
 celery -A config.celery_app beat -l INFO


Code review comments:
diff_hunk: @@ -181,7 +181,15 @@ def test_travis_invokes_pytest(cookies, context):
 
     with open(f"{result.project}/.travis.yml", "r") as travis_yml:
         try:
-            assert yaml.load(travis_yml, Loader=yaml.FullLoader)["script"] == ["pytest"]
+            yml = yaml.load(travis_yml, Loader=yaml.FullLoader)["jobs"]["include"]
+            assert yml[0]["script"] == ["flake8"]
+
+            if context.get("use_docker") == "y":
path: tests/test_cookiecutter_generation.py
text: @user2:
Rather than having this `if` branch, we can utilize [the power of parametrization](https://docs.pytest.org/en/latest/parametrize.html#pytest-mark-parametrize-parametrizing-test-functions):

```python
@user1.mark.parametrize(["use_docker", "expected_test_script"], [
    ("n", "pytest"),
    ("y", "docker-compose -f local.yml run django pytest"),
])
def test_travis_invokes_pytest(cookies, context, use_docker, expected_test_script):
    context.update({"ci_tool": "Travis", "use_docker": use_docker})
    ...
    assert yml[1]["script"] == expected_test_script
```

Currently, I think the non-docker branch is actually never ran. Using parametrization will also have the advantage of generating 2 separate test cases.

Ideally, we would do something similar with `test_gitlab_invokes_flake8_and_pytest`.

diff_hunk: @@ -5,4 +5,41 @@ set -o nounset
 
 
 rm -f './celerybeat.pid'
+
+postgres_ready() {
path: {{cookiecutter.project_slug}}/compose/local/django/celery/beat/start
text: @user1:
Right now this function is duplicated in 3 scripts: `compose/production/django/entrypoint` & in the 2 scripts here.

Is it really required to fix Travis/Gitlab? I'm thinking that we could move this function into a common/util script & source it from the others to avoid duplication.

@author:
Not exactly required. I first used cookiecutter-django with the intention of using Docker, but I also had zero experience with Docker. It'll be nice to have this for those not experienced and wondering why celerybeat fails if you simply do `docker-compose -f local.yml up --build`.

Although, you can see some reminiscence of me original having docker-compose -f local.yml build && docker-compose -f local.yml run --rm django python manage.py migrate && docker-compose -f local.yml up -d`...

We can move this to a different PR (and this definitely was not DRY whatsoever, so I apologize :P). To back up the point of probably needing to move this to a different PR is that connection. There are several connections due to the compose files' &django anchor. Is it possible we can just run several start files under the one Django service?

@user1:
Ideally, that would be nice if it could be part of a separate PR, yes. I think it's significant enough for that, and mix fix other issues (e.g. #1743 maybe).

Also I'm working on generating a changelog from pull requests. As it is currently, the change would not appear.