A beginner’s guide to deploying a Docker application to production using Docker Compose
By: Uday Hiwarale
Tags: Docker Compose, DevOps, Continuous Integration, Software Development, Docker

Compose File

A Compose file is a YAML file that provides configuration of the services as well as networks, volumes, environment variables, etc. to be used by the services (their containers). This file is usually named docker-compose.yml and placed inside the project directory. When we run the $ docker-compose [options] <subcommand> [suboptions] commands from the project directory, Docker Compose will read this file to analyze services so that it could start, stop, or do other things with them.

💡 Unlike Dockerfile, Compose file provides tons of options which is why it’s very difficult to describe every little thing in just one article. I’ll try my best to cover important topics. For more information, you can always follow this official documentation for the Compose file reference.

A typical docker-compose.yml file looks like the above. The version key specifies the version of the Compose file to use. The current latest version is 3.9 (December 2020). A particular Compose file version is compatible with the particular Docker Engine version(s), here is the compatibility matrix.

💡 A typical Compose file version is in the form of <major>:<minor> release. If we omit the :<minor> part, Docker Compose automatically assumes the 0 minor version which is why we should explicitly mention the minor version.

Using the services field, we define the containers (services) Docker Compose should start. In the above Compose file, we have three services viz. database , frontend , and backend . These service names will be used by the Docker Compose to name images, containers, or volumes it is going to create.

The database service (container) uses the mysql:latest image. We define the image from which the container should be created using the image field of service. Docker downloads this image from Docker Hub and creates a container. This container will expose the 3306 port by default and run the MySQL server as specified by its Dockerfile. This is the minimal configuration of a service in the Compose file.

The backend service uses node:latest image from Docker Hub. However, we have some extra configuration for the container startup. First, we are setting /app as the working directory of the container using the working_dir field. Then we mounting the ./backend directory of the host to /app directory of the container using the volumes field. Then we are executing the node ./server.js command using the command field which will start an HTTP server and expose field specifies the ports of the container to expose.

💡 All relative paths (of the host) used inside the Compose file will be relative to the docker-compose.yml file. The entrypoint , working_dir , command ,and expose field values override the ENTRYPOINT , WORKDIR , CMD and EXPOSE instructions of the image from which the container is being created.

The frontend service doesn’t use a public image. Instead of the image field, we specify a path to the context directory using the build field. Docker Compose will create an image on the fly for the container of this service. You can also specify a custom Dockerfile path and a context directory using an object configuration as discussed here.

The ports field binds the host ports with the exposed container ports. This field provides more configuration options as discussed here. The depends_on field controls how the services are started. In the above case, since the frontend services depends on the database and backend services, they are started first. Once these services (containers) are started, frontend service (container) will be created.

Overriding Compose File

Normally, our application goes through multiple environments. First, we develop the application on the local machine, then we put it on the staging environment for QA, then put it on the CI server for testing, and then it finally goes to the production environment.

Configuration for running application services in these environments won’t be the same always. For example, you might want to expose different ports of a service on the development environment such as 8080 while on the production your ideal choice would be 80 .

We can control these values using the build arguments or the environment variables (discussed next) but sometimes you need a configuration change that can’t be achieved using these variables. So ideal choice would be to have multiple compose files per environment and use the --file or -f flag to decide which config file to use.

However, if your config file is huge and you need to deploy your application on multiple environments then managing multiple config files become a tedious task. There is a better way out. Using the -f flag, we can specify multiple Compose files. Docker Compose will merge these files from right to left and prepare the final Compose configuration.

$ docker-compose -f compose.yml -f compose.dev.yml ...

The $ docker-compose command can accept multiple Compose files. It will then merge configurations from right to left. In the above case, the compose.dev.yml file overrides the configuration of the compose.yml file.

💡 We can have more than one overriding file. Just specify additional overriding files using the -f flag in the $ docker-compose command and the resulting configuration will be created by merging files from right to left.

If no -f flag is provided, Docker Compose looks for the docker-compose.yml file in the current directory. It also looks for the docker-compose.override.yml file in the current directory and if present, it overrides the docker-compose.yml file.

In the override file, the structure of the configuration should be nearly consistent as the file it is overriding. For example, if we want to override a few fields of a service, then services > service_name field should be there. If we introduce a new service in the override file, it will be added to the final configuration. Let’s see how overrides work.

To verify and print the result configuration, you should use the $ docker-compose [options] config command. The final configuration (indicated by the result.yml ) was generated by the command below.

$ docker-compose -f docker-compose.yml -f docker-compose.prod.yml config

Let’s analyze the results. The database service remained as it was since we are not overriding it. The cache service made its way into the final configuration since it was defined in the override file. The image field in the backend service was replaced since it’s a single-value field.

The frontend service configuration is most interesting. A new command property was added in the final configuration from the override. The multi-value fields won’t be overridden. Instead, their values will be merged as you see in the depends_on and the ports fields. However, in the case of volumes , only one mounting point can exist if the container directory is the same.

💡 To know more about overrides, read this documentation.

Using Environment Variables

An environment variable can be used to substitute variables in docker-compose.yml file. This is quite useful while targeting environment-specific configuration changes. Let’s see a small example.

In the above docker-compose.yml file, ${NGINX_VERSION} , ${NGINX_WWW_DIR} and the $NGINX_PORT values will be substituted by the values provided by the environment variables of the same name.

💡 You can use the ${NGINX_VERSION:-latest} syntax to fallback to latest value if the NGINX_VERSION environment variable is not present or empty. To know more about such syntax, follow this documentation.

Values for these variables will be picked up from the shell environment while running the $ docker-compose commands. If an environment variable is not present in the shell environment, then Docker looks for the value of that environment variable inside .env file (if present) in the directory of the Compose file.

# .env file

NGINX_WWW_DIR=./www

NAME_WITH_QUOTES="John Doe"

This file contains the default environment variables for the Compose file. It’s a simple text file with key=value pairs of environment variables. Since this file is not processed by the shell, double or single quotes will be used as part of the values such as "John Doe" in the example above.

💡 You can specify the path to a custom .env file (such as .prod.env ) using --env-file flag with the $ docker-compose command. To know more .env file format, read this documentation.

Environment variables in the docker-compose.yml are not automatically available to the container of a service. Therefore, if you want to pass environment variables present during the execution of $ docker-compose command, you would need to pass them manually to the container using the environment configuration of the service.

In the above docker-compose.yml file, the environment field provides environment variable values to the container. Since the value of LOG_DIRECTORY is fixed, it won’t change. The value of LOG_PREFIX is taken from the environment variables available to the Compose file. If we do not provide a value to an environment variable, such as LOG_LEVEL , its value too will be taken from the environment variables available to the Compose file.

You can also provide service-level .env files using the env_file field of the service configuration. As you can see, there are multiple ways to provide environment variables to a running service (container). When the value of the same environment variable is available in multiple contexts, this how Docker Compose decides which value to use.

1. Value from the compose file (environment)

2. Value from the shell environment (of container)

3. Environment (.env) file (env_file)

4. Dockerfile (ENV instruction)

5. Value is not defined

💡 To know more about the environment variables in Docker compose, read this documentation.

If the image of a service is built on the fly using the build field instead of the image field, then environment variables provided through the environment fields are not available during the build-time. This means if the Dockerfile is doing some variable substitution, the variable values will be empty. For this, we use the build arguments.

Using Build Arguments

Build arguments are the build-time environment variables. These will be available only while building the image from a Dockerfile. We specify a build argument using the ARG instruction in the Dockerfile.

# Dockerfile

ARG tag=latest

FROM node:$tag ARG working_directory

WORKDIR $working_directory

These can be defined with a default value ARG=VAL in the Dockerfile or just with the name. While building an image, we use the --build-arg <key>=<val> flag with the $ docker build command to override or provide values of these variables. In the docker-compose.yml , we provide these values using the args field of the build object. You can either provide a fixed value to a build argument or substitute its value from the environment variable.

In the example above, we have provided a fixed 12.10.0 value for the tag build argument, however, the value for working_directory is taken from the value of FRONTEND_WORDIR environment variable. If we do not provide a value to a build argument such as log_level , its value too will be taken from the environment variable.