Zero Downtime Deployments With Docker Swarm
By: Al-Waleed Shihadeh
Tags: Programming, Docker, Swarm, DevOps, Deployment

The Solution

Let’s start by deploying the Nginx reverse proxy with minimal configurations in a Swarm cluster.

Below is the Docker stack that we’re going to start with. This stack uses the default values for a deployment configuration. As a result of deploying this stack, Swarm will create a new nginx service with only one container and expose it to port 8088 , which is mapped to port 80 inside the docker container:

Deploying this service can be done by simply executing following command, assuming that the stack is stored in a file called nginx.yml :

$> docker stack deploy -c nginx.yml nginx

Redeploying the Nginx service, or updating the service docker image with these configurations, will cause downtime. That’s because we have only one Nginx replica — at some point of time during the rollout of containers, both containers, the old and the new, will be down and any request coming to the service will be rejected. This behavior is a result of the Swarm default configuration of UPDATE_CONFIG . By default Swarm will remove the old container and start the new one.

To enhance the situation we can update the stack file to have the following new configurations:

First, it’s important to provide a health check for the stack services. This allows Swarm to figure out if a service is healthy or not and so it can achieve zero downtime. For instance, in case of deployment of a new Nginx image that contains syntax errors and without the health check, Swarm will start the new Nginx and then remove the old containers. After that, the Nginx docker container with the new image will keep restarting because it is exiting with an error code, but it will never roll back to the old image because the health check is missing. On the other hand, if the health check is present, Swarm will validate if the new docker container is healthy and only if the container is healthy will it continue the update process for the other replicas.

The next important configs section is the update_config . Here we need to change the order of the update process to start-first . This makes Swarm create the new containers first and only do the actual replacement of the containers after they pass the health check.

It’s also important to restore corrupted containers. So, we need to change the failuer_action from the default pause to rollback . This is needed especially in the case of one replica services because Swarm will stop the old containers in the case of pause .

The next section is the rollback_config section. Here we need to rollback all containers simultaneously. therefore we need to set the parallelism to 0 , which means applying the rollback for all affected containers at the same time. In addition, the order of the update should be stop-first to make sure that the corrupted containers are removed as soon as possible.

The third section is the restart_policy section. In this section, we specify the maximum number of restarts in case of a failure during the update. We need to limit the number of the restart to avoid consuming a lot of resources from the host server.

The above compose configuration guarantees zero downtime deployments for applications with one replica. But what about the services where we have multiple replicas? Is that configuration enough?

Luckily the above configurations are also applicable to services that have multiple replicas. However, in that case, we need to be aware of the effect of using parallelism configurations with the update_config section.

The default value of parallelism is 1 which means that Swarm will update the containers one at a given time. On the other hand, setting parallelism to 0 means we want to update the containers simultaneously. Sometimes there’s a need to increase the value parallelism to speed up the process of rolling out or updating services on Swarm clusters. In all these cases the new value shouldn’t be equal to the number of service replicas. Otherwise, the situation will be similar to setting parallelism to 0 , which introduces the risk of having downtime during the deployment again.

Below is a complete Nginx docker stack with the support for zero-downtime deployment configurations. These configurations can be applied for any docker service to achieve the same goal: