spotify/luigi — pull request #178 by ddaniels888 (2013-09-18 22:39:04+00:00)
Title: Add S3 client and target

This is a boto-powered implementation of the S3 filesystem for luigi. We've been using it for a few weeks now.

We're using it to write files out to S3 and check for existence; we haven't yet used it for read, so that isn't implemented yet. Otherwise, everything seems to work well, and there are tests covering it.

For writes, we use a variant of the "atomic_file", writing to a temp file and moving it on close to S3. We're only doing very simple writes of token, for which this is sufficient, but if others have more complex use cases perhaps they can extend it.