chucknorris/dropkick — pull request #57 by mmooney (2013-03-14 18:46:51+00:00)
Title: Add support for uploading to Amazon S3 bucket

I needed this for another project so I just added in support for uploading files to an Amazon S3 bucket. I'm a little new the innards of the dropkick code, but I copied as much as a I could from the copy directory stuff.

I created a ProtoAmazonS3UploadDirectoryTask DSL class, which derives from BaseProtoTask, and implements FromOptions and a new AmazonS3Options (bucket name, access keys, etc).

I also created a AmazonS3UploadDirectoryTask class which derives from BaseIoTask, which then calls a IAmazonService to perform the uploading.

I uses the AWS SDK, which I added as a NuGet reference. I tried to avoid this at first, to avoid adding new dependencies. There is a partially implemented RestAmazonService in there, but getting all of the authentication signature and XML stuff right was a bit of a nightmare, the AWS SDK made it just a few lines of code. Anyhow, I hid it behind a IAmazonService that is injected into AmazonS3UploadDirectoryTask, so it could be changed later.

I'm using my fork of this for a client now, and it works pretty well. They are uploading 2700+ files, over 29 MB, and it takes about 20 minutes. I'm sure there is more that could be improved (clean destination, parallelizing the AWS calls, multipart uploads, etc), but this was good stable version that pretty much solved the problem.

I tried to put in a lot of testing around it. There are a handful of unit tests, and then the integration tests that actually talk to Amazon are marked as explicit, and read their config from a local file that has been .gitignore'd (the first time you run that test, it will tell you the path that it is looking in, and there is a sample file there too). It looked like there were a few different conventions being used in the tests (TinySpec, etc), and I confess I didn't understand them, so I tried to keep my tests pretty small and straightfoward.

Thanks