ckan/ckan — pull request #1861 by nickstenning (2014-07-29 17:06:47+00:00)
Title: docker: Replace Apache with gunicorn as app server

CKAN's install instructions have, up until now, recommended the use of both Nginx and Apache as part of the default install. While both are venerable pieces of software, using both to deploy a single application
is unusual to say the least.

Nginx is used as a front-line webserver and cache, while Apache is used as a WSGI application server.

Better and simpler WSGI application servers exist, and one of them is [gunicorn](http://gunicorn.org/). Gunicorn is a smaller application, focused on only one thing (WSGI serving) and in my experience has more predictable performance and resource usage characteristics than Apache.

I know this is a contentious change, so please view this PR as much as a place to discuss this rather than a request for a merge right away. In particular, I have no doubt that you'll want the source install instructions updated to recommend Gunicorn as well if you decide that this is the right decision.

I've done some performance testing to demonstrate the differences (particularly with regard to memory consumption) between mod_wsgi and gunicorn. The process in each case was to:

1. Measure system free memory on a VM dedicated only to hosting CKAN/Pg/Solr containers.
2. Start a container with {Apache,gunicorn}. Measure system free memory again.
3. Run [wrk](https://github.com/wg/wrk) in one of two configurations (one single-threaded, one parallel) against the CKAN homepage for 1 minute.
4. Measure system free memory at the end of the test.

The results of this testing -- both the memory measurements and the performance output from wrk -- can be found [in a Google spreadsheet](https://docs.google.com/a/okfn.org/spreadsheets/d/1D-87FA7OQ0zRjYIbUItggmUW0Ro2s58IuHH80fvVqMU/edit#gid=0) and to my mind speak pretty clearly in favour of Gunicorn.

Notable pieces of the data include:

- mod_wsgi running CKAN can leak up to 462KiB of memory per request on average (compared to 54KiB for Gunicorn). Obviously this reflects an underlying problem in CKAN rather than anything wrong with mod_wsgi or Gunicorn *per se*, but mod_wsgi's memory management certainly doesn't seem to be helping matters.
- mod_wsgi running CKAN has wildly inconsistent response times under parallel load (see 99th-percentile response times column -- >6s for mod_wsgi, vs 800ms for Gunicorn)
- mod_wsgi sometimes just completely fails to respond in time, resulting in request timeouts