Create the initial release skeleton

This was generated with a simple `bosh init release --git`. I manually added
some `.gitkeep` files to some of the empty directories so that it would be
easier to see which directories were created at each step.

A fake `final.yml` has been added too. This is required for some BOSH commands
such as creating a release. However, the values of the file are not checked.
Add a README

It outlines the goals and a short description of the project.
Add Redis source code to release

Since we're going to be relying on the `unstable` branch of Redis it's best to
use the source code rather than uploading a release tarball to the blobstore.
The source was added using a git submodule with the command:

    $ git submodule add https://github.com/antirez/redis src/redis

The `unstable` branch was then checked out using the commands:

    $ cd src/redis
    $ git checkout unstable

We're using this branch because it contains the Redis Sentinel work that we're
going to be using later in order to manage automatic failover leader promotion,
monitoring, and error notifications.
Add Redis package

This commit adds the Redis package to the release. A package is used to define
how to compile source code or blobstore artefacts into a form that is usable
in the jobs of the release. A package may also depend on other packages if they
are required for its compilation.

The basic structure of the package was generated using the command:

    $ bosh generate package redis

The `redis` package begins with its specification which is defined in the
`packages/redis/spec` file. This file gives the package a name, any other
package dependencies that are required to compile this package, and a list of
source code or blobs that are required for compilation. These artefacts can
either be from the `blobs` or `src` directories of the release.

The other file in the package is the compilation script which is found at
`packages/redis/packaging`. This file tells BOSH how to take the artefacts
described above and turn them into something that another package or another
job may use. This file begins with setting some Bash settings in order to make
any errors during the compilation phase of this package easier to debug. We
then change into the `redis` directory which we chose for inclusion in the
specification file and the compile Redis (the origin of this command is the
Redis README). We set the `PREFIX` to be `$BOSH_INSTALL_TARGET` for this
command. The `$BOSH_INSTALL_TARGET` is the directory that BOSH expects all of
the compiled artefacts to be in at the end of the compilation. It will then zip
up the contents of this directory before saving the compiled package in its
blobstore for future use.
Add the Redis job

This is nearly the simplest (useful) job possible. It was generated initially
with the command:

    $ bosh generate job redis

The job contains a `monit` file, a `spec` file, and a template (`redis_ctl`)
that we'll combine with Monit to start and manage the job.

The `monit` file contains a Monit job configuration file that tells Monit the
name (that Monit will use), PID file, and start and stop commands for the Redis
job.

The `spec` file contains the specification for the job. This is similar to the
spec file for the package that we saw before. It is a YAML file that contains
the name of the job, the list of templates and where the populated versions
should be placed. By putting the `redis_ctl` file in the `bin/` directory BOSH
will automatically give executable permissions to the file. The last item in
this file are the packages that are required in order to run this job.

The last file is the `redis_ctl` file that is called by Monit when the job is
started and stopped. This file sets up the logging, PID file and starting of
the `redis-server` executable.
Add an example deployment manifest

You can use this manifest to deploy your Redis release to a BOSH running the
Warden CPI.
Scale our deployment to double the size

This commit shows the ease of scaling up a deployment using BOSH. We simply
change the deployment manifest and run `bosh deploy` again. The BOSH director
will realise that the deployment size needs to change and carry out the
appropriate scaling.

In this deployment there are three different values we need to change in order
to scale the deployment.

* The size of the resource pool is increased in order to allow more instances
  to be created inside it.

* The number of instances of the `redis` job is also increased. These instances
  will fill the newly enlarged resource pool.

* Another static IP is added to the job for the newly created instance.

This method will be applicable to any magnitude of scaling. In this commit we
only added a single instance. However, we could have just as easily changed
this number by 10, 100, or even a 1000 and BOSH will do the right thing.
Allow the configuration of the Redis password

BOSH injects configuration onto the virtual machines before it starts the jobs
on them. This commit shows a common method of making a configuration value of
a job be configurable by a release deployer. In this case we will be
configuring the authentication password for the Redis server. There's no real
order to these steps as they all depend on each other in some way.

The `spec` file of the Redis job must be modified by adding entry for the
templating of the Redis configuration file that we are going to add and adding
an entry for the new configurable property we would like people to be able to
configure.

Next, we add the example Redis configuration file from the Redis source code to
the path that we specified in the `spec` (`redis.conf.erb`). This has one
change on lines 310-312 which, if the property `redis.password` is set will
change the configuration of the Redis server to require a password to
authenticate. The `if_p` helper evaluates the block if the property name it is
given as an argument is present. The argument passed to the block is the value
of the property.

The startup script for the job, `redis_ctl`, must also be changed to use the
new configuration file. After BOSH has laid all of the templated files out on
disk the configuration file that we are looking for will be at the path
`/var/vcap/jobs/redis/config/redis.conf`. The last directory and filename in
that path is the same one that we specified in the `spec` file above.

The last change is to add the property to our deployment manifest. This is
where the value of our property originates before being distributed to our
deployment with our next `bosh deploy`. In the deployment manifest you can see
how the property in the YAML file are translated, through the `spec`, to real
values on machines.
Add Master-Slave replication to release

Now things are getting interesting. We've taken our deployment of many distinct
Redis instances and converted them into something that starting to look useful.
After this commit your Redis cluster will replicating from the master out to
N slaves.

We must change the `spec` file for the `redis` job and add 3 new properties:

* `redis.replication.enabled`
  This property must be set to true on each job instance, master or slave, that
  is part of a cluster. It's set to `false` by default to provide a sensible
  initial value.

* `redis.replication.master.enabled`
  This property is set to true on any job instance that should be the master of
  a Redis cluster. It too has a sensible default of false.

* `redis.replication.master.address`
  This property should be set on each slave to tell it the IP address that
  should be used to reach the master node.

Using these new properties we are able to add additional logic to the ERB in
the Redis configuration file to support clustering. We check to see if
replication has been enabled, if we have been given the IP address of the
master, and if this is not the master of the cluster. If all of these
predicates are true then we set the slave to follow a master and authenticate
with it if required.

Finally, we add our new configuration possibilities to the manifest. We now
need two jobs in our deployment: a master and a slave. However, because we did
some more work up front in our templating we are able to use the same
job-template for both of them: `redis`. Properties can either be specified
globally in the `properties` block at the bottom of the deployment manifest or
per-job. We configure the master node at the job level to be the master but
configure the cluster globally using our new properties.
Change Redis password

Uh-oh - someone has leaked our highly secure Redis password! We need to change
it to something else quickly. Luckily since we are using BOSH this change will
be easy. All we need to do is change the value of our `redis.password` value in
the deployment manifest and do another `bosh deploy`. BOSH will go out and
re-template all of the configuration files and then bounce our Redis processes.

Crisis averted...
Scale up the deployment size

We had a higher read load than expected. Let's increase the number of slaves we
have so we can spead some of the load out.
Add HAProxy Package

Our users are getting fed up with having to connect to different IP addresses
to reach the same dataset. It's fragile and confusing. It would be much better
to put some sort of load balancing or routing layer into our release. Let's use
the venerable HAProxy to accomplish this.

We add the HAProxy source code tarball to our BOSH blobs the same as the Redis
package. In the `spec` we specify the name of our job, the fact that it has no
dependencies, and tell it to use the HAProxy blob that we added just now.

In our `packaging` script we step through the motions of compiling the source
into its binary form. All of these steps were taken directly from the HAProxy
documentation.
Add HAProxy Job

Now that we have our package all ready to go let's write the job that sits on
top of it.

This should all look familiar by now. We set up a job and fill in the blanks in
a very similar way to the Redis job. There is a new property called
`haproxy.backend.servers` which enumerates the backend Redis servers that we
are going to be proxying to.
Add HAProxy job to Deployment Manifest

Adding another new job to the deployment is easy as before. We add a new entry
under the `jobs` key in the manifest, tell it to use the `haproxy` job template
and give it a static IP.

After a `bosh deploy` we'll have our new load balanced Redis cluster!
Scale up number of Redis Slaves

It's time to test out our load balancer. We double the size of our deployment
and add the required static IPs. After a `bosh deploy` new instances of Redis
have been spun up and the new instances have been added to our HAProxy
configuration. Web scale.
Allow Redis port number to be configured

We have the same constant repeated all over our templates: the Redis port
number. Similarly to if we were writing code let's extract this number to
a variable and pass it in. We introduce the variable in all of the `spec` files
that care about the value and add the ERB required to inject it. This time we
add a default (the default Redis port number) to the property so that it does
not need to be set in the manifest.

Now we can set the port and change it at a whim as easily as the Redis
password.