Advanced Django Development: How to build a professional CMS for any business? (Part #2)
By: Hossein Vahedi
Tags: CMS, Django, Backend Development, Wagtail

This article is based on the previous one.

Overview

In Part #2, we are going to touch briefly some important concepts in backend development with django. We are going to:

Create a wagtail project. “Dockerize” the project and set development and production environments. See an example of writing optimized dockerfiles. Learn how to connect different microservices to start the production environment using docker-compose. learn how to make a microservice wait for another one to be available using wait-for-it. Learn how to serve files using Nginx.

Introduction to Django web framework (Skip if you are already familiar with django)

If you care about backend development, you’ve probably heard about Django:

The web framework for perfectionists with deadlines.

Currently, django’s github repository has got more than 50K stars which shows how popular it is, among other popular backend frameworks like: Ruby on Rails, .NET, Laravel, and …

Django has a very rich ecosystem. You can find many projects that are based on django, each of them has added another “layer of abstraction” to django development. For example:

django-rest-framework which makes django developers able to easily build powerful APIs.

django-allauth which provides a wide set of tools related to authentication and account management.

and account management. Wagtail which lets django developers build professional content management systems. Wagtail has many built-in features out of the box. At the same time, It’s highly customizable and can fit into almost any specific situation. In this article we are going to create a CMS for EngineerX project using wagtail.

…

Create a wagtail project

We will extend this project to build horizontally scalable backend microservices for EngineerX web application. Continue reading this series of tutorials to see How we are going to provision the required EKS, RDS and EFS infrastructure and deploy it using Terraform.

Follow steps below to create a django project using wagtail:

1. Create a python virtual environment (optional but highly recommended):

It’s recommended to create a separate virtual environment for each of your python projects. This way, each of python projects would have it’s own isolated environment with it’s own package manager (pip). You can use “Pipenv: Python Development Workflow for Humans” to easily manage your isolated environments.

2. Install Wagtail:

pip install wagtail

3. Create a Django project using wagtail:

wagtail start engineerx

This command creates a django project named engineerx located at WORKDIR/engineerx . Let’s see some of the created files:

WORKDIR/engineerx/engineerx/settings/base.py : It contains common configurations for the project. For example, you can declare which django applications you want to install by setting the INSTALLED_APPS variable.

: It contains common configurations for the project. For example, you can declare which django applications you want to install by setting the variable. WORKDIR/engineerx/engineerx/settings/dev.py : It extends base.py to properly configure the development environment. For example it sets DEBUG = True which adds more information about the errors and problems that developers may encounter.

: It extends to properly configure the development environment. For example it sets which adds more information about the errors and problems that developers may encounter. WORKDIR/engineerx/engineerx/settings/production.py :It extends base.py to configure the project for production.

4. Install required python packages:

To make sure that all required packages are installed, wagtail has created WORKDIR/engineerx/requirements.txt for us. Run this command to install packages:

pip install -r requirements.txt

We are going to use requirements.txt file to define any required packages and add this file to the git repository.

5. Create and Initialize database:

create a simple Sqlite database and make it ready to work with wagtial:

python manage.py migrate

Create a django superuser:

python manage.py createsuperuser

6. Run the development server:

python manage.py runserver

Now the project is up and running and accessible on 127.0.0.1:8000/. Go to 127.0.0.1:8000/admin/ and login with the username and password provided when initializing the database at step 5.

Dockerize the project

Docker is the most widely used containerization technology. It plays an important role in DevOps culture. Today, most of the software development jobs require the applicants to have a good understanding of docker.

Wagtail has created /WORKDIR/engineerx/Dockerfile for us which is a good starting point for dockerizing a wagtail project. But in a real-world scenario you have to change some parts of it. Please replace /WORKDIR/engineerx/Dockefile with this new Dockerfile:

# Use an official Python runtime based on Debian 10 "buster" as a parent image. FROM python:3.8.1-slim-buster

# Port used by this container to serve HTTP. EXPOSE 8000

# Set environment variables.

# 1. Force Python stdout and stderr streams to be unbuffered.

# 2. Set PORT variable that is used by Gunicorn.

# This should match "EXPOSE" command. ENV PYTHONUNBUFFERED=1 \

PORT=8000

# Install system packages required by Wagtail and Django. RUN apt-get update --yes --quiet

RUN apt-get install --yes --quiet --no-install-recommends build-essential

RUN apt-get install --yes --quiet --no-install-recommends libpq-dev

RUN apt-get install --yes --quiet --no-install-recommends libmariadbclient-dev

RUN apt-get install --yes --quiet --no-install-recommends libjpeg62-turbo-dev

RUN apt-get install --yes --quiet --no-install-recommends zlib1g-dev

RUN apt-get install --yes --quiet --no-install-recommends libwebp-dev

RUN apt-get install --yes --quiet --no-install-recommends python3-opencv

RUN rm -rf /var/lib/apt/lists/*

# Install the application server. RUN pip install "gunicorn==20.0.4"

# Install opencv-python RUN pip install opencv-python

# Install the project requirements. RUN pip install "Django==3.1"

RUN pip install "wagtail==2.11"

RUN pip install psycopg2

RUN pip install Faker # Use /app folder as a directory where the source code is stored. WORKDIR /app

COPY . .

You can also download this Dockerfile from EngineerX official github repository. Also make sure to add this .dockerignore file so that docker’s build context won’t become unnecessarily large.

Some key points about this dockerfile:

apt-get install --yes --quiet --no-install-recommends python3-opencv and pip install opencv-python are required if you want to enable wagtail’s feature detection.

and are required if you want to enable wagtail’s feature detection. pip install "gunicorn==20.0.4" installs gunicorn on the image. Gunicorn is a “Python WSGI HTTP Server for UNIX”. We need gunicorn to serve our django application in production mode. That’s because django itself, isn’t able to serve the application in production mode. You can find more about django deployment on the official documentations.

installs gunicorn on the image. Gunicorn is a “Python WSGI HTTP Server for UNIX”. We need gunicorn to serve our django application in production mode. That’s because django itself, isn’t able to serve the application in production mode. You can find more about django deployment on the official documentations. pip install psycopg2 installs psycopg2 which is required for connecting to a PostgreSQL database. We can continue using Sqlite for development but it cannot be used in production workloads. Django supports almost all types of relational databases (including MySQL) but it’s considered to be the best practice to use it with PostgreSQL. Using Django and PostgreSQL together is the standard way. It’s like using ExpressJs and MongoDB together.

installs psycopg2 which is required for connecting to a PostgreSQL database. We can continue using for development but it cannot be used in production workloads. Django supports almost all types of relational databases (including MySQL) but it’s considered to be the best practice to use it with PostgreSQL. Using Django and PostgreSQL together is the standard way. It’s like using and together. pip install Faker : Faker is a python package which can be used to generate random data. It’s used in two places: 1) To generate fake data for tests. 2) To initialize the database with some fake data for demo: This way you can see how the web application would look like when it becomes full of users and data.

Now we have to build this docker image:

cd WORKDIR/engineerx

docker build . -t engineerx/backend

Note:

This image is very large and the building process takes a considerable amount of time to get finished which mostly depends on your internet connection’s speed.

Run the Dockerized Development Environment

First of all, run the container and publish port 8000:

docker run -it -p 8000:8000 engineerx/backend bash

Now that you can execute commands in the container’s shell, it’s time to create and run migrations on the database:

python manage.py makemigrations && python manage.py migrate

Then start the development server on 0.0.0.0:8000 :

python manage.py runserver 0.0.0.0:8000

Run the Dockerized Production Environment

Some important decisions we have to make:

We cannot use the existing Sqlite database in production mode. We have to choose a production ready relational database like: MySQL , PostgreSQL , … As mentioned earlier, PostgreSQL is the recommended database to be used with django.

database in production mode. We have to choose a production ready relational database like: , , … As mentioned earlier, is the recommended database to be used with django. Django is not able to serve the application in production mode. The solution is to use Gunicorn to serve the application.

Gunicorn doesn’t serve files. To be able to serve files located at /media and /static , we have to use a reverse proxying tool like Nginx.

and , we have to use a reverse proxying tool like Nginx. We have to properly configure and connect these microservices (Gunicorn, PostreSQL and Nginx) to each other. The easiest way is to use docker-compose. Make sure you have docker-compose installed.

Let’s create an Nginx microservice to serve the files:

We can configure the default nginx image to serve requests made on /media/ and /static/ and redirect other requests to Gunicorn web server.

Create a new directory WORKDIR/nginx and put Dcokerfile and nginx.conf in it. Let’s see nginx.conf :

upstream backend {

server backend:8000;

}

listen 80;

location / {

proxy_pass

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $host;

proxy_redirect off;

} server {listen 80;location / {proxy_pass http://backend; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header Host $host;proxy_redirect off; location /static/ {

alias /home/app/web/static/;

} location /media/ {

alias /home/app/web/media/;

}

}

As you can see it redirects anything other than /media/ and /static/ to backend:8000 . backend would get resolved to the IP address of gunicorn webserver.

Build this docker image:

cd WORKDIR/nginx

docker build . -t engineerx/backend-ingress

Sometimes a microservice has to wait for another one:

Obviously, our gunicorn http server depends on PostgreSQL . It has to wait until database is created and ready to accept connections.

Using this amazing wait-for-it bash script, we can wait until PostgreSQL is ready:

./wait-for-it.sh db: 5432

Download the bash script from here and put it in WORKDIR/engineerx . Make it executable so that we can use it later:

chmod +x wait-for-it.sh

Configure the project for production environment:

If you look at manage.py and wsgi.py, you will see that they configure the project for development:

"engineerx.settings.dev"

We have to add manage.production.py and engineerx/wsgi.production.py files to the project for production mode.

Next we have to update production.py:

from .base import * DEBUG = False # SECURITY WARNING: keep the secret key used in production secret!

SECRET_KEY = os.environ['SECRET_KEY'] # SECURITY WARNING: define the correct hosts in production!

ALLOWED_HOSTS = [os.environ['ALLOWED_HOST']] DATABASES = {

'default': {

'ENGINE': 'django.db.backends.postgresql',

'NAME': 'postgres',

'USER': 'postgres',

'PASSWORD': os.environ['POSTGRES_PASSWORD'],

'HOST': 'db',

'PORT': '5432',

}

} try:

from .local import *

except ImportError:

pass

Now add this start.sh shell script to WORKDIR/engineerx . This shell script would start the production gunicorn server. Download it and make it executable:

cd WORKDIR/engineerx

chmod +x start.sh

Re-build the docker image:

We have added some new files to the project. To apply changes, build the docker image again:

cd WORKDIR/engineerx

docker build . -t engineerx/backend

Note that this time, docker build command, finished very fast. That’s because our Dockerfile is optimized.

Docker Compose

Now we are ready to start the production environment with this WORKDIR/docker-compose.yaml:

version: "3"

services:

ingress:

image: engineerx/backend-ingress

ports:

- "8000:80"

networks:

- engineerx-backend

volumes:

- static:/home/app/web/static

- media:/home/app/web/media

depends_on:

- backend

backend:

image: engineerx/backend

ports:

- "8001:8000"

entrypoint: ["/bin/sh","-c"]

command:

- |

./wait-for-it.sh db:5432 -- ./start.sh

environment:

- POSTGRES_PASSWORD

- SECRET_KEY

- ALLOWED_HOST=127.0.0.1

networks:

- engineerx-backend

volumes:

- static:/app/static

- media:/app/media

depends_on:

- db

db:

image: postgres:latest

ports:

- "5432:5432"

environment:

- POSTGRES_PASSWORD

networks:

- engineerx-backend

networks:

engineerx-backend:

driver: bridge

volumes:

static:

driver: local

media:

driver: local

As you can see in docker-compose.yaml, there are a few environment variables that has to be set before starting the production environment:

POSTGRES_PASSWORD : This password is required by PostgreSQL when it creates the database. SECRET_KEY : This secret key has to be kept in a safe place; We don’t want to hard code it. We have to provide it when starting the production environment. Check out the official docs about SECRET_KEY .

Set variables and start the production environment:

cd WORKDIR/ export POSTGRES_PASSWORD=<YOUR_PASSWORD>

export SECRET_KEY=<YOUR_KEY> docker-compose up

Execute commands in the running containers:

Now there are three services up and running: ingress, backend and db. We might want to execute some commands in one of the containers. For example, we might want to create a django super user.

To create a super user, open another terminal and run this command:

docker-compose exec backend python manage.py createsuperuser

Conclusion

We have created a wagtail project, dockerized it and configured it for development and production environments.

In next chapters, we will extend this project, write tests, generate fake data, define some blocks of data for our CMS and …

Next: Learn how to easily generate data in Part #3