Packaging, Deploying and Scaling FAST-CGI Applications in Docker for Kubernetes Done Right
By: Tobias Gurtzick
Tags: Scalability, Docker, PHP, Deployment, Kubernetes

The build process

When building your application and of course your docker container, a few questions arise directly. Should I use variant x or y, in our example now apache or php-fpm? Can the ingress actually reach them? How do I serve my static content? What about the dynamic content? And plugins, like… kinda dynamic ones?

The common idea here seems to build a container containing both nginx and the process executing the code files, to deliver the content. Everyone who is already a bit more familiar with docker and docker best practices knows: Wait, I shouldn’t run more than one process per docker container! So what next? A quick google search brings the solution, create two containers and share a volume to copy files between both. Sounds like a good idea! Is it?

TL;DR; It’s not, it is actually a terrible idea!

To understand how we can properly solve our problem, we need some knowledge about kubernetes and some general architectural principles.

Kubernetes networking basics

If we deploy our applications into two containers, how can we address the other? Or could we maybe directly point the ingress, which in our case will be the nginx-ingress to the fcgi?

The answer to the first question is, in kubernetes we have the concept of pods, so we can just address localhost. This is possible since a pod can contain multiple containers sharing the same NIC. If you want to dive deeper into how Networking works, you can read my in-depth article how kubernetes networking works over here: https://neuvector.com/network-security/advanced-kubernetes-networking/

So first problem solved quite easily, the answer to our second question is two folded. Yes we can, but we may shouldn’t! It always depends on how you deliver your applications. If you deliver all your static content via CDN, you will be fine by pointing the ingress directly. But otherwise, you will have static content anyways, and therefor you will need a solution to deliver them. And this solution is nginx! In short, our secondary problem we didn’t thought about too much yet, the static content, makes our idea of a second container with nginx even more meaningful. It is not anymore just a middleman, it has value itself, by delivering our static content.

Architectural principles for scalable applications

There are some very simple principles to building scalable applications. The first one is to get rid of local state and avoid shared on disk state by all means. This means when you’ve got files to serve, move them to a dedicated service serving files.

Here some examples to this:

For Files : an optimized image serving service, that has low level optimizations like a filesystem optimized by the findings of the haystack design paper: https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf.

: an optimized image serving service, that has low level optimizations like a filesystem optimized by the findings of the haystack design paper: https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf. Moving state : When we talk about cache, keep a small hot in memory LRU and the rest shift off to a dedicated cache service like, memcached or redis.

: When we talk about cache, keep a small hot in memory LRU and the rest shift off to a dedicated cache service like, memcached or redis. Persistent data: use a proper distributed database, which you can horizontally scale, like cockroachdb: https://www.cockroachlabs.com/

Another important principle is: Have a reproducible process, without unnecessary initializations. This will just slow down deployments when you want to scale up. This also means copying over data from one to another place and for avoid multiple processes in the same container, because here the old unix principle still holds true: “Write programs that do one thing and do it well”.

But why exactly do you want to avoid slow deployments?

The answer to that is quite simple: If you’re scaling your services up, this process needs to be as fast as possible. Just think about it, when do you scale and why? The answer is when you need to and if you need to, it means otherwise you loose something. This can be a lower conversion rate due to longer load times and therefor a real loss in money, or even service continuity when you’re in the risk of overloading your whole system and putting it at risk to be in complete denial.

To avoid all this you need to design the whole deployment as fast as possible and you generally already have enough slow components in your chain, so at least do not screw up your own stuff. Take for example your hosting or cloud provider, who delivers a ready to use system for you not quite fast enough (which is true especially for some of the big clouds like azure, which are often terribly slow, 5–15 minutes happened regularly to me already, in delivering your machine).

So always remember, time is money, so predict scaling needs and start scaling early enough and cut the process as short as possible.

The plugin hell

Coming to the next topic, plugins. You will stumble over a problem here right away, which is persistent dynamic content. One of this kind are plugins, which are effectively the reason, why applications like wordpress are hard to be deployed properly, or be scaled across multiple hosts without issues. The ugly truth is, that these applications were never developed with any respect for these things and there is a lot that is “wrong” with it. It only starts with the plugins and continues with wordpress included self upgrade mechanism.

But it is not like you couldn’t install any plugin manually, or that you must use the included mechanism of wordpress to update it instead of doing it by hand. In fact doing it manually is the path you need to go instead. The plugin installation needs to happen in your build step and for the wordpress update, you could create an update routine, that is being run while every rebuild. And luckily this is true for most if not all applications, that have been built for extensibility in that form. This is at least if you want to do it the “right” way. However one problem resides:

You loose comfort!

And this might not be a problem for you, but it is one especially if you just want to host applications such as wordpress for other users, which want to install their own plugins, but are not willing to go through the process of “manual” installation. In this case you’re actually forced to get back to any network file system, which in the best case is a distributed or at least replicated one. This is one of the exceptions where you have basically no choice, but do it and use a shared “local disk” state, you should however instruct your users that have problems while scaling to at least offload all the data away to a CDN or an self hosted S3 Storage like minio and setup the executor of the code files to cache them in memory instead of rereading them from disk.

So by all means, avoid badly designed software. There is software out there, that proved you can do plugins without screwing up scalability. To give an example here: vBulletin Board stores its plugins and the code for it in the database and evaluates it afterwards. While I would never recommend using any evaluation, this is a perfectly fine approach to retain scalability and provide user comfort. So stick with the well designed software instead of staying in the rain afterwards, when it is already to late.

Your own application

But enough of that, let’s assume you’re building your applications with best effort and follow all best practices and principles.

You will quickly experience, that as soon as you’re getting familiar with the proper solutions for storage of different kind, they are getting to be a no brainer and establish a fixed seat in your stack. Here a small bonus for you a small set of examples for different kinds of storage:

Cache

Files

minio (s3) https://www.minio.io/

seaweedfs — a haystack design implementation https://github.com/chrislusf/seaweedfs

Persistent data

Graph data

Equipped with the right tools for the duty, you are ready to build and ship your scalable application.

So looking at our PHP application, we need to decide: PHP-FPM + NGINX or Apache. To make it short, there are many reasons to avoid Apache in favor of NGINX, the biggest probably being the limited scalability of Apache due to its design of opening 1 thread per connection. But unlike Apache, PHP-FPM can not deliver to the client directly, it needs a webserver to process the HTTP request for it and passing it through via FCGI. We will need two containers, one with NGINX and one with PHP-FPM, which seems to be an overhead, but this overhead can be reduced by giving NGINX some task itself instead of being just a pass through. And in web applications we always have this task, which we covered already earlier:

Delivering static content, by building an NGINX container containing this already.

This way we benefit from all sides, the performance of PHP-FPM executing our php files and the performance of NGINX forwarding to PHP-FPM, delivering the static assets and maybe even caching.