travis-ci/travis-build — pull request #138 by rkh (2013-09-20 12:01:14+00:00)
Title: [WiP] implement directory caching

Allows to caching directories on S3 between builds.

## Overview

* [x] rework cache config in a way that's compatible with current apt caching directive
* [x] implement directory caching
* [x] implement bundler specific caching
* [x] implement S3 URL signing
* [ ] pick a S3 path based on repository (do we have the github id?)
* [ ] unit tests
* [ ] test on staging

## Configuration

The current allowed syntax for apt caching is:

``` yaml
cache: apt
```

Or:

``` yaml
cache:
- apt
```

With this patch, those will continue to work and you can add bundler caching:

``` yaml
cache:
- apt
- bundler
```

You can also use a hash form, which will allow the user to explicitly disable caching should it be enabled by default in the future:

``` yaml
cache:
apt: true
bundler: true
```

In addition, it allows listing arbitrary directories that should also be cached:

``` yaml
cache:
apt: true
bundler: true
directories:
- foo
- bar
```

You can access this configuration in a normalised (ie hash) form via `data.cache`.

## Internal API

There is an internal API for caching directories, it can be used from within scripts:

``` ruby
directory_cache.add(sh, "some/directory")
```

The data object has a helper method for parsing the `cache` config to figure out whether a caching feature is enabled:

``` ruby
directory_cache.add(sh, "some/directory") if data.cache? :something
```

The default value for this flag can be stored in `Travis::Build::Data::DEFAULT_CACHES`, but can also be passed as second argument (see the bundler logic for an example).

The payload passed on from [travis-worker](https://github.com/travis-ci/travis-worker) should include a `:cache_options` element:

``` ruby
payload[:cache_options] = {
type: :s3,
s3: {
bucket: "...", # required
access_key_id: "...", # required
secret_access_key: "...", # required
scheme: "...", # optional, defaults to "https"
host: "..." # optional, defaults to "s3.amazonaws.com"
},
fetch_timeout: 10.minutes, # required, in seconds
push_timeout: 80.minutes # required, in seconds
}
```

If the type is `nil` or missing or if `cache_options` is missing, then `directory_cache` will be a dummy object that simply does nothing.

## On the VM

On the VM it will use [casher](https://github.com/rkh/casher):

```
casher fetch FETCH_URL
casher add DIRECTORY
casher push PUSH_URL
```

## Related issues

* https://github.com/travis-ci/travis-ci/issues/834 - Cache installed bundle on S3 (ruby projects)
* https://github.com/travis-ci/travis-ci/issues/1401 - Why installed gems aren't cached?
* https://github.com/travis-ci/travis-ci/issues/1441 - Cache downloaded Maven artifacts